Image sensing device and its operating method

Abstract

To provide an electronic device.SOLUTION: An image sensing device 1000 comprises: an image sensor 100 configured to acquire an image including a plurality of pixel data; a memory 300 configured to store reference gain values for vertices defined by a plurality of block areas included in a gain map corresponding to the size of the image; and an image processor 200 configured to calculate gain values included in each of the plurality of block areas using the reference gain values and to output a correction image in which the reference gain values and the gain values are applied to the plurality of pixel data, wherein the plurality of block areas include a first block area and a second block area having a shorter distance from a center of the image than a position of the first block area and having a size greater than that of the first block area.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an electronic device, and more particularly to an image sensing device and an operating method thereof.

Background Art

[0002] An image sensing device is a device that acquires an image. Recently, with the development of the computer industry and the communication industry, the demand for image sensing devices has increased in various electronic devices such as smartphones, digital cameras, game devices, Internet of Things, robots, security cameras, medical cameras, and autonomous driving vehicles.

[0003] The image sensing device can correct an image for improving image quality. This is because the image quality may deteriorate due to the pixel structure of the image sensing device, the characteristics of the wavelength of light, the process error of the lens, the alignment error between the lens and the image sensor, and the like. In particular, when the imbalance between the green channels (for example, the Gb channel and the Gr channel) that represent one of the most sensitive colors in human visual characteristics becomes large, image quality degradation elements such as lattice noise may occur in the image.

[0004] The image sensing device can pre-store the data used for correction in a memory. However, resources such as hardware are limited, and optimization operations such as minimizing the resources required for data processing and storage while maintaining the performance of image quality improvement are required.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Embodiments of the present invention provide an image sensing device and an operating method thereof that can reduce the calculation amount and capacity of data used for image correction.

Means for Solving the Problems

[0006] An image sensing device according to an embodiment of the present invention includes an image sensor that acquires an image containing multiple pixel data, a memory that stores reference gain values ​​for vertices defined by multiple block regions included in a gain map corresponding to the size of the image, and an image processor that uses the reference gain values ​​to calculate the gain values ​​included in each of the multiple block regions and outputs a corrected image in which the reference gain values ​​and the gain values ​​are applied to the multiple pixel data, wherein the multiple block regions may include a first block region and a second block region that is closer to the center of the image than the position of the first block region and is larger in size than the first block region.

[0007] The operation method of the image sensing device according to an embodiment of the present invention includes the steps of: acquiring a calibration image via an image sensor, calculating a reference gain value for vertices defined by a plurality of block regions included in a gain map corresponding to the size of the calibration image; using the reference gain value to calculate the gain value included in each of the plurality of block regions; storing the reference gain value and the gain value in memory; and acquiring an image via the image sensor to output a corrected image obtained by applying the reference gain value and the gain value stored in memory to a plurality of pixel data, wherein the plurality of block regions may include a first block region and a second block region that is closer to the center of the image than the position of the first block region and is larger in size than the first block region. [Effects of the Invention]

[0008] This technology provides an image sensing device and its operating method that can reduce the amount of computation and data used for image correction. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating an image sensing device according to an embodiment of the present invention. [Figure 2] This figure illustrates an image sensor according to an embodiment of the present invention. [Figure 3a] This figure illustrates a pixel array according to an embodiment of the present invention. [Figure 3b] This figure illustrates an image acquired via an image sensor according to an embodiment of the present invention. [Figure 4] This figure illustrates an image processor according to an embodiment of the present invention. [Figure 5] This figure illustrates a calibration image according to an embodiment of the present invention. [Figure 6] This figure illustrates the gain map according to an embodiment of the present invention. [Figure 7] This is a diagram illustrating one of the block regions according to an embodiment of the present invention. [Figure 8a] This figure illustrates a method for calculating a reference gain value according to an embodiment of the present invention. [Figure 8b] This figure illustrates a method for calculating a reference gain value according to an embodiment of the present invention. [Figure 8c] This figure illustrates a method for calculating a reference gain value according to an embodiment of the present invention. [Figure 8d] This figure illustrates a method for calculating a reference gain value according to an embodiment of the present invention. [Figure 9a] This figure illustrates a method for calculating the gain value according to an embodiment of the present invention. [Figure 9b] This figure illustrates a method for calculating the gain value according to an embodiment of the present invention. [Figure 10] This figure illustrates the correction image according to an embodiment of the present invention. [Figure 11] This is a diagram illustrating the operation method of an image sensing device according to an embodiment of the present invention. [Figure 12] This is a diagram illustrating the operation method of an image sensing device according to an embodiment of the present invention. [Figure 13]A diagram for explaining a computing system including an image sensing device according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0010] Specific structural or functional descriptions of embodiments according to the concepts of the present invention disclosed in this specification or application are merely examples for explaining embodiments according to the concepts of the present invention. Embodiments according to the concepts of the present invention may be implemented in various forms and should not be construed as being limited to the embodiments described in this specification or application.

[0011] FIG. 1 is a diagram for explaining an image sensing device according to an embodiment of the present invention.

[0012] Referring to FIG. 1, the image sensing device 1000 can operate according to the control of the host 3000.

[0013] The image sensing device 1000 can acquire an image according to the request of the host 3000. And the image sensing device 1000 can output the image to the host 3000 or a device designated by the host 3000 according to the request of the host 3000. Here, the device designated by the host 3000 may be a memory device for storing data or a display device for outputting data in a visual manner.

[0014] The image sensing device 1000 may be embodied in the form of a packaged module, component, etc. In this case, the image sensing device 1000 may be mounted on the host 3000. Or the image sensing device 1000 may be embodied as an electronic device separate from the host 3000.

[0015] The host 3000 may be implemented by various electronic devices. For example, the host 3000 may be implemented by a digital camera, a mobile device, a smart phone, a PC (Personal Computer), a tablet PC (tablet personal computer), a notebook, a PDA (personal digital assistant), an EDA (enterprise digital assistant), a PMP (portable multimedia player), a wearable device, a drive recorder camera, a robot, an autonomous driving vehicle, etc.

[0016] The image sensing device 1000 may include an image sensor 100, an image processor 200, and a memory 300.

[0017] The image sensor 100 can acquire an image Img. Specifically, when the image sensor 100 receives a command to control to acquire an image from the host 3000, it can acquire the image. The image may include a plurality of pixel data. The plurality of pixel data may be independent of each other. One pixel data can correspond to one pixel. For example, the image sensor 100 can acquire pixel data for each of a plurality of pixels. The image sensor 100 can acquire an image including a plurality of pixel data. The pixel data may include information indicating at least one of a position, a color channel, a pixel value, and an exposure value.

[0018] The position of a pixel can indicate the position where the corresponding pixel is arranged among a plurality of pixels, or the position where the corresponding pixel data is arranged among a plurality of pixel data.

[0019] The color channel of a pixel can indicate the color of light or the color of the pixel data for that pixel. For example, a color channel may include one of a red channel, a green channel, and a blue channel. The green channel may include a first green channel and a second green channel, which are distinguished from each other depending on the position of the pixel (or pixel data). For example, the pixels in the first green channel may be pixels in the same row as the pixels in the blue channel, and the pixels in the second green channel may be pixels in the same row as the pixels in the red channel. As another example, the pixels in the first green channel may be pixels in the same column as the pixels in the blue channel, and the pixels in the second green channel may be pixels in the same column as the pixels in the red channel.

[0020] The pixel value can indicate the brightness of the light detected through that pixel. For example, a larger pixel value indicates brighter light. The pixel exposure value can indicate the time it took for light to be detected through that pixel.

[0021] Therefore, the image sensor 100 may be implemented as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0022] The image processor 200 can correct images. Specifically, when the image processor 200 acquires an image via the image sensor 100, it can generate a correction image (Cor_Img) by applying a gain map to the image. For example, the image processor 200 can generate a correction image by applying the gain values ​​included in the gain map to the pixel data. The image processor 200 can also output the correction image. For example, the image processor 200 can output the correction image to the host 3000 or a device instructed by the host 3000.

[0023] Therefore, the image sensor 100 can acquire a calibration image (Cal_Img). The calibration image is an image used to extract a gain map used to correct the image acquired from the image sensor 100. For example, the calibration image may be an image acquired by the image sensor 100 by photographing a white background in a uniform illumination environment. Due to the structure of the image sensor 100 or the wavelength characteristics of light, distortions such as non-uniform pixel values ​​for each pixel position or color channel may occur in the image acquired through the image sensor 100. Since different distortions may occur for each image sensor 100, an individual gain map for each image sensor 100 may be required. The calibration image may contain multiple pixel data. In one embodiment, the calibration image may be the same size as the image acquired from the image sensor 100.

[0024] A gain map may contain multiple gain values. Gain values ​​are parameters used to correct pixel data (or pixel values) and can represent specific numerical values. The gain map can correspond to the size of the image or calibration image acquired by the image sensor 100. The size of the image or calibration image may be indicated by the number of pixel data points arranged in the column direction and the number of pixel data points arranged in the row direction.

[0025] For example, if the number of pixel data points in an image or calibration image is M x N, the number of gain values ​​in the gain map may also be M x N, where M and N are natural numbers. That is, the number of gain values ​​may be the same as the number of pixel data points. Here, the number of gain values ​​in the gain map may have a one-to-one correspondence with the number of pixel data points in the image. Each gain value can correspond to the position of each pixel data point.

[0026] The number of bits assigned to the data may vary depending on the position of the gain value. A larger number of bits in the data means that it can represent a wider range of numbers or values. For example, when calculating and storing a gain value assigned a small number of bits, the decimal points of the calculated value that are outside the range of bits can be truncated, and the decimal points within the range of bits can be stored in memory 300 as the gain value.

[0027] An image processor 200 according to one embodiment of the present invention can calculate reference gain values ​​included in a gain map using a calibration image and a plurality of block regions. Specifically, the image processor 200 can calculate reference gain values ​​for vertices defined by block regions using a plurality of pixel data included in the calibration image. The calibration image may include a plurality of block regions. Block regions can be defined by the position of the pixel data. This makes it possible to calculate some of the gain values ​​included in the gain map. The image processor 200 can store the reference gain values ​​in memory 300.

[0028] The gain map may include multiple block regions. The gain map may include multiple gain values. The multiple gain values ​​may include a reference gain value and a normal gain value. Each of the multiple block regions may be a rectangular or square region. However, this is only one embodiment, and each of the multiple block regions may be transformed into various shapes of regions.

[0029] A block region may include gain values ​​corresponding to the positions of pixel data. A block region may also include reference gain values. Reference gain values ​​are gain values ​​that correspond to some of the multiple gain values ​​included in the gain map. Each of the reference gain values ​​can correspond to pixel data at a predetermined position. The predetermined positions may correspond to the vertices of the block region. For example, a block region can be defined as a 4x4 region from (1,1) to (4,4). A block region may include gain values ​​corresponding to each position from (1,1) to (4,4). In this case, the positions (1,1), (4,1), (1,4), and (4,4) can be defined as the vertices of the block region. That is, a block region can be defined by connecting the positions (1,1), (4,1), (1,4), and (4,4) with lines. The gain values ​​located at the vertices of the block region, (1,1), (4,1), (1,4), and (4,4), may be reference gain values.

[0030] A plurality of block regions according to one embodiment of the present invention may include a first block region and a second block region. Here, the second block region may be closer to the center of the image than the first block region and may be larger in size than the first block region. In one embodiment, the size of a block region may decrease as it moves further away from the center of the image. For example, within an image, the block region closest to the center of the image may be the largest in size, and the block region closest to the edge of the image may be the smallest in size. Block regions may share vertices with each other. Block regions may not overlap with each other. That is, the pixel data (or gain value) contained in a block region may be different from the pixel data (or gain value) contained in another block region.

[0031] The image processor 200 can then use a reference gain value to calculate the gain values ​​contained in each of the multiple block regions. Here, the calculated gain values ​​are the remaining gain values ​​after subtracting the reference gain value contained in each of the multiple block regions, and for the sake of explanation, each of these remaining gain values ​​will be referred to as the gain value below. In this way, all the gain values ​​contained in the gain map can be calculated. The image processor 200 can then store the gain values ​​contained in the gain map in the memory 300.

[0032] The memory 300 can store reference gain values ​​included in the gain map. In one embodiment of the present invention, the memory 300 can store gain values ​​included in the gain map.

[0033] The memory 300 may be implemented using non-volatile memory elements. For example, the memory 300 may consist of various non-volatile memory elements such as ROM (Read Only Memory) which can only read data, OTP (One Time Programmable) memory which can only be written once, EPROM (Erasable and Programmable ROM) which can erase and write stored data, NAND flash memory, and NOR flash memory.

[0034] According to one embodiment of the present invention, an image sensing device and its operating method can be provided that can reduce the amount of computation and capacity of data used for image correction. A more detailed description will follow below with reference to the attached drawings.

[0035] Figure 2 is a diagram illustrating an image sensor according to an embodiment of the present invention.

[0036] Referring to Figure 2, the image sensor 100 may include an optical lens LS, a pixel array 110, a row decoder 120, a timing generator 130, a signal converter 140, and an output buffer 150.

[0037] The optical lens LS can refract light that has been reflected from the subject object and reached the image sensor. The light refracted by the optical lens LS can then travel to the pixel array 110. The optical lens LS may be one lens or a collection of lenses arranged in the path of light. Furthermore, the optical lens LS may include a collection of microlenses positioned above each pixel of the pixel array 110. The subject object may include at least one of various elements that exist outside the image sensor 100, such as an object, animal, person, or background.

[0038] The pixel array 110 may include a color filter array and a photoelectric conversion layer. The color filter array may be located above the photoelectric conversion layer. The photoelectric conversion layer may be located below the color filter array. Here, the upper and lower parts are determined based on the direction of light propagation, and light can travel from the color filter array towards the photoelectric conversion layer.

[0039] A color filter array may include multiple color filters. For example, each of the multiple color filters may be a red color filter, a green color filter, and a blue color filter. A red color filter can filter incident light to transmit light with wavelengths that represent the red color. A green color filter can filter incident light to transmit light with wavelengths that represent the green color. A blue color filter can filter incident light to transmit light with wavelengths that represent the blue color. However, this is only one embodiment, and the types of color filters that transmit light of a particular color may be varied in many ways.

[0040] The photoelectric conversion layer may include multiple sensing circuits. The sensing circuits may include photodiodes and capacitors. The photodiode can generate an electric current in response to incident light through the photoelectric effect. The capacitor can store charge in accordance with the current generated by the photodiode. Here, the amount of stored charge can correspond to a pixel value representing brightness.

[0041] The row decoder 120 can select a pixel located in the row corresponding to an address in response to the address and control signal output from the timing generator 130. The pixel array 110 can output a signal corresponding to the amount of charge accumulated from the selected pixel and provide it to the signal converter 140.

[0042] The signal converter 140 can acquire pixel data for each of multiple pixels based on each of the signals output from the pixel array 110. Taking one pixel as an example, the signal converter 140 can acquire the pixel value corresponding to the amount of charge stored in the capacitor in the photoelectric conversion layer and the color corresponding to the color filter.

[0043] The output buffer 150 can output an image Img or a calibration image Cal_Img. Specifically, the output buffer 150 may be realized as multiple buffers that store the digital signals output from the signal converter 140. The output buffer 150 can latch and output each column of pixel data provided by the signal converter 140. The output buffer 150 can temporarily store the pixel data output from the signal converter 140 and output the pixel data sequentially according to the control of the timing generator 130.

[0044] Figure 3a is a diagram illustrating a pixel array according to an embodiment of the present invention.

[0045] Referring to Figure 3a, a pixel array 110 according to one embodiment of the present invention may be embodied as a pixel array 310 having a pixel structure as shown in Figure 3a. The pixel array 310 may contain multiple pixels. The multiple pixels may be arranged in the column direction and the row direction. Each of the multiple pixels may contain information about its arranged position. For example, it can be indicated that the pixel CE_xy is the pixel at the x-th position in the column direction and the y-th position in the row direction, where x and y are natural numbers.

[0046] Each of the multiple pixels may include a microlens ML that refracts light, a color filter CF that transmits light of a specific color, and a sensing circuit PD that detects the intensity of light. The multiple pixels may be distinguished by the type of color filter CF. For example, pixels containing a red color filter may be called pixels of the red channels R1 to R4, pixels containing a green color filter may be called pixels of the green channels Gr1 to Gr4 and Gb1 to Gb4, and pixels containing a blue color filter may be called pixels of the blue channels B1 to B4.

[0047] The pixel array 310 may contain multiple pixel groups. These multiple pixel groups may be arranged in both the column and row directions. Each of these multiple pixel groups may contain information about its arranged position. For example, the pixel group CG_XY can be indicated as the pixel group at the Xth position in the column direction and the Yth position in the row direction, where X and Y are natural numbers.

[0048] Each pixel group may contain multiple pixels arranged in a predetermined array pattern. That is, a pixel group can represent a unit region in which an array pattern of multiple pixels is repeated. For example, the predetermined array pattern may be a quad Bayer pattern in which pixels of the 2x2 first green channel Gb1-Gb4, pixels of the 2x2 blue channel B1-B4, pixels of the 2x2 red channel R1-R4, and pixels of the 2x2 second green channel Gr1-Gr4 are arranged in a 4x4 array.

[0049] On the other hand, the above-described embodiment is merely one example, and the pre-set array pattern may be transformed into various patterns, such as a Bayer pattern in which pixels of the first green channel (1x1), the blue channel (1x1), the red channel (1x1), and the second green channel (1x1) are arranged in a 2x2 array.

[0050] Figure 3b is a diagram illustrating an image acquired via an image sensor according to an embodiment of the present invention.

[0051] Referring to Figure 3b, the image Img or calibration image Cal_Img according to one embodiment of the present invention may be an image 320 having a pixel structure as shown in Figure 3b.

[0052] Image 320 can be acquired via the image sensor 100, which includes a pixel array 310.

[0053] Image 320 may contain multiple pixel data. The multiple pixel data contained in Image 320 can correspond to the multiple pixel data contained in the pixel array 310 of the image sensor 100.

[0054] Each of the multiple pixel data points may contain information about its position in the array. For example, pixel data PX_xy may correspond to the pixel at the x-th position in the column direction and the y-th position in the row direction, where x and y are natural numbers. Pixel data PX_xy may also correspond to the pixel CE_xy at the same position. For example, pixel data PX_xy may be pixel data such as the pixel value obtained from pixel CE_xy.

[0055] Image 320 may contain multiple pixel data groups. These multiple pixel data groups may be arranged in both column and row directions. Each of these multiple pixel data groups may contain information about its arranged position. For example, the pixel data group PG_XY can be indicated as the pixel data group at the Xth position in the column direction and the Yth position in the row direction, where X and Y are natural numbers.

[0056] Each pixel data group may contain multiple pixel data arranged in a predetermined array pattern. That is, a pixel data group can represent a unit region in which the array pattern of multiple pixel data is repeated. For example, the predetermined array pattern may be a quad Bayer pattern in which the pixel data of the 2x2 first green channel Gb1-Gb4, the pixel data of the 2x2 blue channel B1-B4, the pixel data of the 2x2 red channel R1-R4, and the pixel data of the 2x2 second green channel Gr1-Gr4 are arranged in a 4x4 array.

[0057] The pixel data contained in Image 320 can be distinguished by color channel or by channel. For example, the pixel data contained in Image 320 may be distinguished by color channel as pixel data for the first green channel Gb1-Gb4, pixel data for the blue channel B1-B4, pixel data for the red channel R1-R4, and pixel data for the second green channel Gr1-Gr4. Alternatively, the pixel data contained in Image 320 may be distinguished by channel as pixel data for the first to fourth channels Gb1-Gb4, pixel data for the fifth to eighth channels Gr1-Gr4, pixel data for the ninth to twelfth channels R1-R4, and pixel data for the thirteenth to sixteenth channels B1-B4.

[0058] On the other hand, the above-described embodiment is merely one example, and the pre-set array pattern may be transformed into various patterns, such as a Bayer pattern in which the pixel data of the first green channel (1x1), the pixel data of the blue channel (1x1), the pixel data of the red channel (1x1), and the pixel data of the second green channel (1x1) are arranged in a 2x2 array.

[0059] Figure 4 is a diagram illustrating an image processor according to an embodiment of the present invention.

[0060] Referring to Figure 4, the image processor 200 may include an image calibrator 210 and an image collector 220.

[0061] The image calibrator 210 can calculate a reference gain value ref_Gain using the calibration image Cal_Img acquired via the image sensor 100.

[0062] Specifically, the image calibrator 210 can select pixel data corresponding to the vertices of block regions from among multiple pixel data contained in the calibration image Cal_Img acquired via the image sensor 100. Below, we will describe an example of calculating a reference gain value ref_Gain for one selected pixel data from among multiple pixel data.

[0063] The image calibrator 210 can calculate the average pixel value of pixel data that have the same color channel as a selected pixel from the pixel data included in the region of interest. Here, the same color channel may be one of the first green channel, second green channel, red channel, and blue channel. In other embodiments, the median value of the pixel data may be used instead of the average pixel value.

[0064] The region of interest may be an area extended to a predetermined size based on the position of the pixel data corresponding to the vertices of the block region. In one embodiment, the regions of interest may be the same size. In other embodiments, the regions of interest may be different sizes depending on the position of the reference pixel data.

[0065] For example, we can assume that the first green channel is the same color channel as the selected pixel data.

[0066] In one embodiment, the average pixel value may be the average of the pixel values ​​of the pixel data having a first green channel among the pixel data included in the region of interest. In another embodiment, the average pixel value may be the average of the pixel values ​​of the pixel data having a first green channel and the pixel values ​​of the pixel data having a second green channel among the pixel data included in the region of interest. The average value can also be calculated in a similar manner when the second green channel is the same color channel as the selected pixel data.

[0067] The image calibrator 210 can calculate a reference gain value ref_Gain corresponding to the selected pixel data using the pixel values ​​of the selected pixel data and the calculated average pixel values. For example, the reference gain value ref_Gain may be the ratio of the pixel values ​​of the selected pixel data to the calculated average pixel values.

[0068] The image calibrator 210 can calculate a reference gain value ref_Gain for each of the remaining pixel data from the selected pixel data in the same manner. The image calibrator 210 can then store the calculated reference gain value ref_Gain in memory 300.

[0069] The image calibrator 210 can calculate the gain value Gain using the reference gain value ref_Gain.

[0070] Specifically, the image calibrator 210 can calculate a gain value Gain corresponding to a selected point based on the distance between a selected point in one of the multiple block regions and a vertex in that block region, and a reference gain value ref_Gain corresponding to a vertex in that block region. Here, the distance between the vertex and the selected point may be the distance between the pixel data corresponding to the vertex and the pixel data corresponding to the selected point. The distance between the pixel data may correspond to the number of pixel data present between those pixel data. The distance may be a concept that includes the length in the row direction or the length in the column direction.

[0071] In one embodiment, the vertices of one block region may include the first, second, third, and fourth vertices of the same block region.

[0072] In this case, the image calibrator 210 can calculate the gain value Gain corresponding to the selected point based on the distance between the selected point and the first vertex, the distance between the selected point and the second vertex, the distance between the selected point and the third vertex, the distance between the selected point and the fourth vertex, the reference gain value corresponding to the first vertex, the reference gain value corresponding to the second vertex, the reference gain value corresponding to the third vertex, and the reference gain value corresponding to the fourth vertex.

[0073] In one embodiment, the multiple block regions may include a first block region and a second block region. Here, the second block region may be closer to the center of the image than the first block region and larger in size than the first block region. This utilizes the phenomenon that the degree of distortion is less in the central part of the image Img or calibration image Cal_Img and the degree of distortion is uniform across regions, while the degree of distortion is worse towards the edges and the degree of distortion is non-uniform across regions. By setting block regions that are relatively larger towards the central part of the image Img or calibration image Cal_Img and relatively smaller towards the edges of the image Img or calibration image Cal_Img, the frequency of calculating the reference gain value ref_Gain included in the block regions closer to the center can be reduced. Although the frequency of calculating the gain value Gain increases, the overall amount of computation can be reduced by calculating the gain value Gain using an algorithm that requires relatively less computation compared to the reference gain value ref_Gain.

[0074] In one embodiment, the gain value included in the first block region may be assigned a larger number of bits than the gain value included in the second block region. Here, the second block region may be closer to the center of the image than the first block region and larger in size than the first block region. The number of bits of data assigned to the gain value may vary depending on the position or region. A larger number of bits in the data means that a wider range of numbers or values ​​can be represented. For example, when calculating and storing a gain value assigned a small number of bits, the decimal part of the calculated value that is outside the range of bits can be truncated, and the decimal part that is within the range of bits can be stored in memory 300 as the gain value.

[0075] When the image collector 220 acquires an image Img via the image sensor 100, it can generate a corrected image Cor_Img by applying a reference gain value ref_Gain and a gain value Gain stored in the memory 300 to multiple pixel data contained in the image. Here, the reference gain value ref_Gain and the gain value Gain may be included in a gain map. The reference gain value ref_Gain and the gain value Gain can correspond to multiple pixel data contained in the image.

[0076] The image collector 220 can output a corrected image, Cor_Img. The image collector 220 can output the corrected image, Cor_Img, to the host 3000 or a device designated by the host 3000. For example, the image collector 220 can output the corrected image, Cor_Img, to a processor, display, or storage device in response to a control command received from the host 3000.

[0077] In one embodiment, the image collector 220 can generate a corrected image Cor_Img using a reference gain value ref_Gain output from memory 300 and a gain value Gain output from image calibrator 210 or volatile memory. That is, the gain value Gain does not need to be stored in memory 300.

[0078] For this purpose, the image calibrator 210 can calculate the gain value Gain using the reference gain value ref_Gain stored in memory 300 whenever a pre-configured event occurs. Here, the pre-configured event may include the event in which the image sensor 100 is turned on. The image calibrator 210 can transmit the calculated gain value Gain to the image collector 220. Alternatively, the image calibrator 210 can store the calculated gain value Gain in volatile memory. The volatile memory may be implemented as SRAM (Static Random Access Memory), DRAM (Dynamic RAM), etc.

[0079] In another embodiment, the image collector 220 can generate a corrected image Cor_Img using the reference gain value ref_Gain and the gain value Gain output from the memory 300. That is, the gain value Gain may be stored in the memory 300 together with the reference gain value ref_Gain.

[0080] Figure 5 is a diagram illustrating a calibration image according to an embodiment of the present invention.

[0081] Referring to Figure 5, the image sensor 100 can acquire image 500. For example, image 500 may be a calibration image Cal_Img acquired by the image sensor 100 by photographing a white background in a uniform illumination environment. Image 500 may be the state before correction is performed. The shading in each part of image 500 can indicate the brightness or pixel value of the pixel data. It can be seen that the difference in shading increases as you move from the center C of image 500 to the edges, which are further away. It can be seen that the degree of distortion increases and becomes more non-uniform as you move from the center C of image 500 to the edges.

[0082] According to an embodiment of the present invention, multiple block regions Blk1 and Blk2 can be set to correct the distortion. Each of the multiple block regions Blk1 and Blk2 may have a different size depending on its distance from the center C of the image 500.

[0083] In one embodiment, the size of each of the multiple block regions Blk1 and Blk2 may decrease as the distance from the center C of image 500 increases. Here, the center C of image 500 may be the point where the length in the column direction and the length in the row direction are halved. For example, the distance between the first block region Blk1 and the center C of image 500 can be defined as the distance d1 between the center c1 of the first block region Blk1 and the center C of image 500. Similarly, the distance between the second block region Blk2 and the center C of image 500 can be defined as the distance d2 between the center c2 of the second block region Blk2 and the center C of image 500.

[0084] On the other hand, the above-described embodiment is merely one embodiment, and the reference position that serves as the basis for determining the size of the block area may change to various positions other than the center C of image 500, such as the vertices of image 500. Furthermore, the basis for determining the size of the block area may be the distance ratio in the column direction and the distance ratio in the row direction. For example, using the first block area Blk1 as a reference, the distance ratio in the column direction may be the first ratio obtained by dividing the distance b1 in the column direction between the center c1 of the first block area Blk1 and the center C of image 500 by the length of image 500 in the column direction. The distance ratio in the row direction may be the second ratio obtained by dividing the distance a1 in the row direction between the center c1 of the first block area Blk1 and the center C of image 500 by the length of image 500 in the row direction. If both the first ratio and the second ratio are greater than a preset value, the size of the block area can be set to the first size, and if both the first ratio and the second ratio are smaller than a preset value, the size of the block area can be set to the second size, which is larger than the first size.

[0085] Figure 6 is a diagram illustrating the gain map according to an embodiment of the present invention.

[0086] Referring to Figure 6, the gain map 600 can correspond to the size of the image Img or calibration image Cal_Img acquired from the image sensor 100.

[0087] The gain map 600 may include multiple block regions Blk1, Blk2, and Blk3. In one embodiment, each of the multiple block regions Blk1, Blk2, and Blk3 may be a square or rectangular region. Here, each block region Blk1, Blk2, and Blk3 is a logically defined region, and each block region Blk1, Blk2, and Blk3 can correspond to a region of the image Img or the calibration image Cal_Img.

[0088] Multiple block regions Blk1, Blk2, and Blk3 may include a first block region Blk1, a second block region Blk2, and a third block region Blk3. The first block region Blk1, the second block region Blk2, and the third block region Blk3 may be arranged to correspond to an image Img or a calibration image Cal_Img. Each of the first block region Blk1, the second block region Blk2, and the third block region Blk3 may be a block region of different sizes. The arrangement position of each of the first block region Blk1, the second block region Blk2, and the third block region Blk3 may change depending on their size. Alternatively, the size of each of the first block region Blk1, the second block region Blk2, and the third block region Blk3 may change depending on their arrangement position.

[0089] For example, the relatively large third block region Blk3 may be positioned at the location that is relatively closest to the center. The medium-sized second block region Blk2 may be positioned at the location that is midway between the center and the center. The relatively smallest first block region Blk1 may be positioned at the location that is relatively farther from the center.

[0090] The image processor 200 can calculate the reference gain value ref_Gain using the pixel values ​​of the pixel data and block regions Blk1, Blk2, and Blk3 contained in the calibration image Cal_Img.

[0091] Specifically, the image processor 200 can select pixel data at a predetermined position from multiple pixel data contained in the calibration image Cal_Img. Here, the pixel data at the predetermined position may be pixel data corresponding to the vertices of block regions Blk1, Blk2, and Blk3. The block regions Blk1, Blk2, and Blk3 can be defined by connecting the vertices with lines in the column and row directions. One reference pixel data group ref_PG_XY may be one pixel data group PG_XY located at a position (X, Y) indicating one vertex of one block region among multiple pixel data groups. One pixel data group PG_XY may contain pixels (or pixel data) with different color channels. On the other hand, one pixel data group ref_PG_XY can be located at the vertices of at least one block region Blk2, Blk3. On the other hand, a vertex of one block region can also be a vertex of another block region. That is, adjacent block regions can share at least one vertex.

[0092] The image processor 200 can select a region of interest (ROI) corresponding to each vertex of block regions Blk1, Blk2, and Blk3. The region of interest (ROI) may be a region of a predetermined size with the position of the reference pixel data group ref_PG_XY (i.e., a vertex of the block region) as the reference point. Here, the reference point may be the center point. The predetermined size may be a single fixed size. However, this is only one embodiment, and the predetermined size may be a variable size depending on the position of the reference pixel data group ref_PG_XY (i.e., a vertex of the block region). The region of interest (ROI) may include pixel data located within the region. A more specific embodiment will be described with reference to Figure 7.

[0093] Figure 7 is a diagram illustrating one of the block regions according to an embodiment of the present invention.

[0094] Referring to Figure 7, the image processor 200 can select regions of interest ROI1 to ROI4, which correspond to each of the vertices Ref_PG1 to Ref_PG4 of any of the block regions BLKi.

[0095] For example, the block region BLKi may be a square or a rectangle. The length of the column direction of the block region BLKi may be W, and the length of the row direction may be H. Here, W and H are natural numbers and may be the same or different values. The vertices Ref_PG1 to Ref_PG4 of the block region BLKi may include the first vertex Ref_PG1, the second vertex Ref_PG2, the third vertex Ref_PG3, and the fourth vertex Ref_PG4. The first vertex Ref_PG1, the second vertex Ref_PG2, the third vertex Ref_PG3, and the fourth vertex Ref_PG4 may be located at a distance of W in the column direction or H in the row direction.

[0096] In this case, the image processor 200 can select a first region of interest (ROI) of a predetermined size centered at the position of the first vertex Ref_PG1, a second region of interest (ROI) of a predetermined size centered at the position of the second vertex Ref_PG2, a third region of interest (ROI) of a predetermined size centered at the position of the third vertex Ref_PG3, and a fourth region of interest (ROI) of a predetermined size centered at the position of the fourth vertex Ref_PG4.

[0097] The image processor 200 can calculate a reference gain value ref_Gain corresponding to multiple vertices Ref_PG1 to Ref_PG4 by using the pixel values ​​of the pixel data contained in the regions corresponding to each of the multiple regions of interest ROI1 to ROI4 in the calibration image Cal_Img. That is, the reference gain value ref_Gain corresponding to multiple vertices Ref_PG1 to Ref_PG4 can be calculated using the pixel values ​​contained in each of the multiple vertices Ref_PG1 to Ref_PG4 and the surrounding pixel values.

[0098] For example, the image processor 200 can calculate a reference gain value corresponding to the first vertex Ref_PG1 of the block region BLKi by using the pixel values ​​of the pixel data contained in the region corresponding to the first region of interest ROI1 in the calibration image Cal_Img. The image processor 200 can calculate a reference gain value corresponding to the second vertex Ref_PG2 of the block region BLKi by using the pixel values ​​of the pixel data contained in the region corresponding to the second region of interest ROI2 in the calibration image Cal_Img. The image processor 200 can calculate a reference gain value corresponding to the third vertex Ref_PG3 of the block region BLKi by using the pixel values ​​of the pixel data contained in the region corresponding to the third region of interest ROI3 in the calibration image Cal_Img. The image processor 200 can calculate a reference gain value corresponding to the fourth vertex Ref_PG4 of the block region BLKi by using the pixel values ​​of the pixel data contained in the region corresponding to the fourth region of interest ROI4 in the calibration image Cal_Img.

[0099] The image processor 200 can calculate the remaining gain value Gain included in the block region BLKi using the reference gain value ref_Gain corresponding to multiple vertices Ref_PG1 to Ref_PG4. Below, a specific example of calculating the reference gain value will be described with reference to Figures 8a to 8d.

[0100] Figures 8a to 8d illustrate a method for calculating a reference gain value according to an embodiment of the present invention.

[0101] Referring to Figure 8a, the image processor 200 can calculate the reference gain value ref_Gain_xy through the mathematical formulas (1-1) to (3) in Figure 8a.

[0102] In the mathematical formulas (1-1) to (3) in Figure 8a, the pixel value ref_PV_xy is the pixel value of the selected pixel data. The selected pixel data is one of the pixel data selected from the pixel data contained in the pixel data group PG_XY at the same array position as the vertex.

[0103] This section describes the case where the selected pixel data is one of the pixel data from the first green channel Gb1 to Gb4.

[0104] In one embodiment, as shown in the mathematical formula in Figure 8a(1-1), the image processor 200 can calculate a reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data and the average pixel value ROI_AvgPV_G for the green channels Gb1~Gb4 and Gr1~Gr4. Here, the average pixel value ROI_AvgPV_G for the green channels Gb1~Gb4 and Gr1~Gr4 may be the average value of the pixel values ​​of the pixel data of the first green channel Gb1~Gb4 and the pixel values ​​of the pixel data of the second green channel Gr1~Gr4 among the multiple pixel data included in the region of interest ROI.

[0105] In other embodiments, as shown in the mathematical formula (1-2) in Figure 8a, the image processor 200 can calculate a reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data and the average pixel value ROI_AvgPV_Gb for the first green channels Gb1 to Gb4. Here, the average pixel value ROI_AvgPV_Gb for the first green channels Gb1 to Gb4 may be the average value of the pixel values ​​of the pixel data of the first green channels Gb1 to Gb4 among the multiple pixel data included in the region of interest ROI.

[0106] This section describes the case where the selected pixel data is one of the pixel data from the second green channel Gr1 to Gr4.

[0107] In one embodiment, as shown in the mathematical formula in Figure 8a(1-1), the image processor 200 can calculate a reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data and the average pixel value ROI_AvgPV_G for the green channels Gb1~Gb4 and Gr1~Gr4.

[0108] In other embodiments, as shown in the mathematical formula (1-3) in Figure 8a, the image processor 200 can calculate a reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data and the average pixel value ROI_AvgPV_Gr for the second green channels Gr1 to Gr4. Here, the average pixel value ROI_AvgPV_Gr for the second green channels Gr1 to Gr4 may be the average value of the pixel values ​​of the pixel data of the second green channels Gr1 to Gr4 among the multiple pixel data included in the region of interest ROI.

[0109] The case where the selected pixel data is one of the pixel data from the red channels R1 to R4 is described below. In this case, as shown in the mathematical formula (2) in Figure 8a, the image processor 200 can calculate the reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data and the average pixel value ROI_AvgPV_R for the red channels R1 to R4. The average pixel value ROI_AvgPV_R for the red channels R1 to R4 may be the average value of the pixel values ​​of the pixel data from the red channels R1 to R4 among the multiple pixel data included in the region of interest ROI.

[0110] The case where the selected pixel data is one of the pixel data from blue channels B1 to B4 is described below. In this case, as shown in the mathematical formula (3) in Figure 8a, the image processor 200 can calculate the reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data and the average pixel value ROI_AvgPV_B for blue channels B1 to B4. The average pixel value ROI_AvgPV_B for blue channels B1 to B4 may be the average value of the pixel values ​​of the pixel data from blue channels B1 to B4 among the multiple pixel data included in the region of interest ROI.

[0111] The following describes how to calculate a single reference gain value with reference to Figures 8b and 8c.

[0112] Referring to Figures 8b and 8c, the image processor 200 can select one pixel data from the pixel data group PG_XY located at the same array position as the vertices of the block region Blki in the calibration images 810 and 820. Here, we assume that the pixel data ref_PX_xy of the first channel Gb1 located at (x, y) is selected.

[0113] The image processor 200 can select a region of interest (ROI) in the calibration images 810 and 820 that has been expanded to a preset size with the pixel data group PG_XY as the central region.

[0114] In one embodiment, as shown in Figure 8b, the image processor 200 can select a pixel data ROI_Avg_Gb from among multiple pixel data included in the region of interest (ROI) of image 810, the pixel data ROI_Avg_Gb having the same color channel as the selected pixel data ref_PX_xy, namely the first green channels Gb1 to Gb4. The image processor 200 can calculate the average pixel value ROI_AvgPV_Gb, which is the average of the pixel values ​​of the selected pixel data ROI_Avg_Gb. As shown in the mathematical formula (1-2) in Figure 8a, the image processor 200 can calculate a reference gain value ref_Gain_xy for the position (x, y) using the pixel value ref_PV_xy and the average pixel value ROI_AvgPV_Gb of the selected pixel data ref_PX_xy from the position (x, y). The reference gain value for another position can be calculated using this method.

[0115] In one embodiment, as shown in Figure 8c, the image processor 200 can select pixel data ROI_Avg_G from among multiple pixel data included in the region of interest (ROI) of image 820, having the same color channels as the selected pixel data ref_PX_xy, namely the first green channels Gb1-Gb4 and the second green channels Gr1-Gr4. The image processor 200 can calculate the average pixel value ROI_AvgPV_G, which is the average of the pixel values ​​of the selected pixel data ROI_Avg_G. As shown in the mathematical formula (1-1) in Figure 8a, the image processor 200 can calculate a reference gain value ref_Gain_xy for the position (x, y) using the pixel value ref_PV_xy and the average pixel value ROI_AvgPV_G of the selected pixel data ref_PX_xy from the position (x, y). A reference gain value for another position can be calculated through this method.

[0116] Furthermore, referring to Figure 6, the smallest first block region Blk1 may be arranged in the edge region 630, which is the furthest from the center among the multiple block regions Blk1, Blk2, and Blk3. According to one embodiment of the present invention, the reference gain value of the first block region Blk1 arranged in the edge region 630 can be calculated using a mirroring method. This will be explained with reference to Figure 8d.

[0117] Referring to Figure 8d, the smallest first block region Blk1 is arranged in the edge region 830, and multiple vertices 850 and 860 of the first block region Blk1 may include a first vertex 850 that is close to the center and a second vertex 860 that is farther from the center. The first vertex 850 and the second vertex 860 may be the most adjacent vertices among all the vertices of the entire block region. That is, the second vertex 860 may be the vertex that is furthest from the center among all the vertices.

[0118] The image processor 200 can calculate a first reference gain value corresponding to the first vertex 850, which is close to the center. Specifically, the image processor 200 can select a reference pixel data group ref_PG_XY corresponding to the position (X, Y) of the first vertex 850 in the calibration image Cal_Img, and select a region of interest 840 based on the selected reference pixel data group ref_PG_XY. The image processor 200 can calculate a first reference gain value corresponding to the position (X, Y) using the pixel value ref_PV_xy of the pixel data included in the reference pixel data group ref_PG_XY and the pixel data included in the region of interest 840.

[0119] Furthermore, the image processor 200 can copy the first reference gain value corresponding to the first vertex 850 as the second reference gain value corresponding to the second vertex 860. That is, each of the second reference gain values ​​may be copied to be the same value as the first reference gain value.

[0120] Figures 9a and 9b illustrate a method for calculating the gain value according to an embodiment of the present invention.

[0121] Referring to Figures 9a and 9b, a single block region Blki can be defined by multiple vertices ref_PX1 to ref_pX4. A single block region Blki may contain points at specific locations. One point can correspond to one pixel data at the same location.

[0122] The image processor 200 can calculate the gain value for each of the multiple points contained within a single block region Blki by utilizing the reference gain values ​​corresponding to multiple vertices ref_PX1 to ref_pX4 that define a single block region Blki. In this case, the image processor 200 can calculate the gain value for the same channel through the reference gain values ​​corresponding to multiple vertices ref_PX1 to ref_pX4 of the same channel. Here, we will describe how to calculate the gain value Gain_P_xy for one of the multiple points contained within a single block region Blki, namely point P_xy. Here, we assume that the channel is the first channel Gb1.

[0123] The image processor 200 can obtain the column-direction distance W and row-direction distance H between multiple vertices ref_PX1 to ref_pX4 that define a single block region Blki. For example, the image processor 200 can obtain the column-direction distance W and row-direction distance H through the positional difference between multiple vertices ref_PX1 to ref_pX4.

[0124] The image processor 200 can obtain the column distance (x1, x2) and row distance (y1, y2) between each of the multiple vertices ref_PX1 to ref_pX4 that define a block region Blki and a selected point P_xy. For example, the image processor 200 can obtain the column distance (x1, x2) and row distance (y1, y2) through the positional difference between each of the multiple vertices ref_PX1 to ref_pX4 and a selected point P_xy.

[0125] The image processor 200 can calculate the gain value Gain_P_xy for a single point P_xy using the mathematical formula shown in Figure 9a, by utilizing the reference gain value, column distance (x1, x2, W), and row distance (y1, y2, H) corresponding to each of the multiple vertices ref_PX1 to ref_pX4.

[0126] Figure 10 is a diagram illustrating the correction image according to an embodiment of the present invention.

[0127] Referring to Figure 10, the image processor 200 can generate a corrected image Cor_Img by applying a gain map to the image Img acquired via the image sensor 100.

[0128] Specifically, the image processor 200 can multiply the pixel value I_xy of the pixel data at a selected (x, y) position from multiple pixel data included in the image Img by the gain value Gain_xy of the (x, y) position selected from multiple reference gain values ​​and multiple gain values ​​included in the gain map. In this case, the image processor 200 can generate a corrected image Cor_Img in which the pixel value is O_xy, which is the result of multiplying the pixel value I_xy and the gain value Gain_xy for each position.

[0129] In other words, the pixel values ​​of the multiple pixel data included in the corrected image Cor_Img may be the result of multiplying the pixel values ​​of the multiple pixel data included in the image Img and the corresponding positions of the multiple reference gain values ​​and multiple gain values ​​included in the gain map.

[0130] Figure 11 is a diagram illustrating the operation method of an image sensing device according to an embodiment of the present invention.

[0131] Referring to Figure 11, the operation method of the image sensing device 1000 according to one embodiment of the present invention is that a calibration image Cal_Img can be acquired via the image sensor 100 S1110.

[0132] S1120 can calculate reference gain values ​​for each vertex of multiple block regions Blk1, Blk2, and Blk3 in the calibration image Cal_Img. That is, it can calculate reference gain values ​​for vertices defined by multiple block regions Blk1, Blk2, and Blk3 included in a gain map corresponding to the size of the calibration image Cal_Img. In one embodiment, each of the multiple block regions Blk1, Blk2, and Blk3 may be a rectangular or square region.

[0133] Here, the multiple block regions Blk1, Blk2, and Blk3 may include the first block region Blk1 and a second block region Blk2 which is closer to the center of the calibration image Cal_Img than the position of the first block region Blk1 and is larger in size than the first block region Blk1.

[0134] In one embodiment, the step of calculating the reference gain value may include the steps of: selecting pixel data corresponding to the vertices of multiple block regions Blk1, Blk2, and Blk3 from multiple pixel data included in the calibration image Cal_Img acquired via the image sensor 100; calculating the average pixel value of the pixel data having a selected color channel from the pixel data included in the region of interest ROI that contains one of the selected pixel data; and calculating one of the reference gain values ​​using the pixel value and the average pixel value of the selected pixel data.

[0135] In one embodiment, the same color channel may be one of the first green channel, second green channel, red channel, and blue channel.

[0136] In one embodiment, the same color channel may be one of the first green channel and the second green channel. In this case, the average pixel value may include the average value of the pixel values ​​of the pixel data having the first green channel and the pixel values ​​of the pixel data having the second green channel among the pixel data included in the region of interest.

[0137] Then, using the reference gain value, S1130 can calculate the gain values ​​for points included in each of the multiple block regions Blk1, Blk2, and Blk3.

[0138] In one embodiment, the step of calculating the gain value may include a step of calculating a gain value corresponding to the selected point based on the distance between a selected point in one of the multiple block regions Blk1, Blk2, and Blk3 and the vertex of that block region Blki, and a reference gain value corresponding to the vertex of that block region Blki.

[0139] Here, the vertices of a block region Blki may include the first, second, third, and fourth vertices of a block region Blki. In this case, the step of calculating the gain value corresponding to the selected location can calculate the gain value corresponding to the selected location based on the distance between the selected location and the first vertex, the distance between the selected location and the second vertex, the distance between the selected location and the third vertex, the distance between the selected location and the fourth vertex, the reference gain value corresponding to the first vertex, the reference gain value corresponding to the second vertex, the reference gain value corresponding to the third vertex, and the reference gain value corresponding to the fourth vertex.

[0140] Furthermore, S1140 allows the reference gain value and the gain value to be stored in memory 300.

[0141] In one embodiment, the number of bits assigned to the gain value in the first block region Blk1 may be greater than the number of bits assigned to the gain value in the second block region Blk2.

[0142] Figure 12 is a diagram illustrating the operation method of an image sensing device according to an embodiment of the present invention.

[0143] Referring to Figure 12, the operation method of the image sensing device 1000 according to one embodiment of the present invention is that an image Img can be acquired via the image sensor 100 S1210. When the image sensing device 1000 receives a capture command from the host 3000, it can acquire an image Img via the image sensor 100.

[0144] Furthermore, S1220 can generate a corrected image, Cor_Img, by applying the reference gain value and gain value stored in memory 300 to multiple pixel data. The reference gain value and gain value stored in memory 300 may be included in a gain map. The gain map can correspond to the size of the calibration image Cal_Img or image Img. The reference gain value and gain value included in the gain map can correspond to multiple pixel data.

[0145] S1230 can output a corrected image Cor_Img. The image sensing device 1000 can output the corrected image Cor_Img to the host 3000. Alternatively, the image sensing device 1000 can output the corrected image Cor_Img to a device instructed by the host 3000. For example, the corrected image Cor_Img may be output to an external device of the image sensing device 1000, such as a storage device, processor, or display.

[0146] Figure 13 illustrates a computing system including an image sensing device according to an embodiment of the present invention.

[0147] Referring to Figure 13, the computing system 2000 may include an image sensor 2010, a processor 2020, a storage device 2030, a memory device 2040, an input / output device 2050, and a display device 2060. Although not shown in Figure 13, the computing system 2000 may further include ports that can communicate with the storage device 2030, the memory device 2040, the input / output device 2050, and the display device 2060, or with external devices.

[0148] The image sensor 2010 can acquire an image or a calibration image. The image sensor 2010 can store a gain map. The image sensor 2010 can generate a corrected image, Cor_Img, by applying the gain map to the acquired image. The image sensor 2010 can communicate with the processor 2020 via an address bus, control bus, data bus, or a different communication link. Here, the image sensor 2010 may be described in the description of the image sensing device 1000 described above.

[0149] The image sensor 2010 may be implemented in various package forms. For example, at least some components of the image sensor 2010 may be implemented using packages such as PoP (Package on Package), Ball grid arrays (BGAs), Chip scale packages (CSPs), Plastic Leaded Chip Carrier (PLCC), Plastic Dual In-Line Package (PDIP), Die in Waffle Pack, Die in Wafer Form, Chip On Board (COB), Ceramic Dual In-Line Package (CERDIP), Plastic Metric Quad Flat Pack (MQFP), Thin Quad Flatpack (TQFP), Small Outline (SOIC), Shrink Small Outline Package (SSOP), Thin Small Outline (TSOP), Thin Quad Flatpack (TQFP), System In Package (SIP), Multi Chip Package (MCP), Wafer-level Fabricated Package (WFP), and Wafer-Level Processed Stack Package (WSP). Depending on the embodiment, the image sensor 2010 may be integrated with the processor 2020 on a single chip, or it may be integrated on separate chips.

[0150] The processor 2020 can control the overall operation of the computing system 2000. The processor 2020 can control the display device 2060 to display the correction image Cor_Img. The processor 2020 can save the correction image Cor_Img to the storage device 2030. Here, the description of the host 3000 described above may apply to the processor 2020.

[0151] The processor 2020 can perform a specific calculation or task. According to an embodiment of the present invention, the processor 2020 may include at least one of the following: a Central Processing Unit (CPU), an Application Processing Unit (APU), a Graphics Processing Unit (GPU), etc.

[0152] The processor 2020 can communicate with the storage device 2030, memory device 2040, and input / output device 2050 via an address bus, control bus, and data bus. According to an embodiment of the present invention, the processor 2020 may be connected to an expansion bus such as a Peripheral Component Interconnect (PCI) bus.

[0153] The storage device 2030 can store data such as a correction image Cor_Img. Here, the data stored in the storage device 2030 may be stored not only when the computing system 2000 is running, but also when it is not running. For example, the storage device 2030 may consist of at least one of any form of non-volatile memory device, such as a flash memory device, a solid state drive (SSD), a hard disk drive (HDD), or an optical disc.

[0154] The memory device 2040 can store data such as a correction image Cor_Img. Here, the data stored in the memory device 2040 may only be stored when the computing system 2000 is running. Alternatively, the data stored in the memory device 2040 may be stored whether the computing system 2000 is running or not. For example, the memory device 2040 may include volatile memory devices such as Dynamic Random Access Memory (DRAM) and Static Random Access Memory (SRAM), and non-volatile memory devices such as Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), and flash memory devices.

[0155] The input / output device 2050 may include an input device and an output device. The input device is a device that can receive user commands through interaction, and may be embodied in, for example, a keyboard, keypad, mouse, microphone, etc. The output device is a device that can output data, and may be embodied in, for example, a printer, speaker, etc.

[0156] The display device 2060 is a device that visually outputs data such as a corrected image. For this purpose, the display device 2060 may be implemented as an LCD (Liquid Crystal Display) that uses a separate backlight unit (e.g., an LED (light-emitting diode)) as a light source and controls the molecular arrangement of the liquid crystal to adjust the degree to which light emitted from the backlight unit is transmitted through the liquid crystal (brightness or intensity of light), or as a display that uses a separate backlight unit or a self-emissive element without liquid crystal (e.g., a mini LED with a size of 100-200um, a micro LED with a size of 100um or less, an OLED (Organic LED), a QLED (Quantum dot LED), etc.) as a light source.

[0157] The display device 2060 may include multiple pixels. The multiple pixels of the display device 2060 may have a positional relationship with the multiple pixels of the correction image Cor_Img that corresponds to each other. The multiple pixels of the display device 2060 can display an image by emitting light with a brightness corresponding to the respective pixel value of the multiple pixels of the correction image Cor_Img. The display device 2060 may include multiple drive circuits corresponding to the multiple pixels. Here, the drive circuits may be implemented in the form of a-Si (amorphous silicon) TFT (thin film transistor), LTPS (low temperature poly silicon) TFT, OTFT (organic TFT), etc.

[0158] In one embodiment, the display device 2060 may be implemented as a flexible display having the characteristic of being able to bend and return to its original shape. In one embodiment, the display device 2060 may be implemented as a transparent display having the characteristic of transmitting light. In one embodiment, the display device 2060 may be implemented as a touch display by being coupled with a touch sensor that identifies the position touched by the user. [Explanation of Symbols]

[0159] 1000 Image Sensing Devices 100 Image Sensors 200 Image Processors 300 memory

Claims

1. An image sensor that acquires an image containing multiple pixel data, A memory that stores reference gain values ​​for vertices defined by multiple block regions included in a gain map corresponding to the size of the image, The image processor includes: calculating the gain value included in each of the plurality of block regions using the reference gain value, and outputting a corrected image obtained by applying the reference gain value and the gain value to the plurality of pixel data, The aforementioned multiple block regions are, It includes a first block region and a second block region that is closer to the center of the image than the position of the first block region and is larger in size than the first block region, Of the plurality of block regions, each of the third block regions that is furthest from the center of the image and has the smallest size includes a first vertex and a second vertex that is closest to the first vertex and further from the center of the image than the first vertex. An image sensing device characterized in that a first reference gain value corresponding to the first vertex is copied to obtain a second reference gain value corresponding to the second vertex.

2. The gain values ​​included in the first block region are The image sensing device according to claim 1, characterized in that a number of bits larger than the gain value included in the second block region is assigned.

3. The aforementioned image processor is From the multiple pixel data included in the calibration image acquired via the image sensor, select the pixel data corresponding to the vertices of the multiple block regions. Among the pixel data included in the region of interest which contains one of the selected pixel data, the average pixel value of the pixel data having the same color channel as the selected pixel data is calculated. The image sensing apparatus according to claim 1, further comprising an image calibrator that calculates one of the reference gain values ​​using the pixel values ​​of the selected pixel data and the average pixel value.

4. The aforementioned same color channel, The image sensing device according to claim 3, characterized in that it is one of the color channels selected from the first green channel, the second green channel, the red channel, and the blue channel.

5. The aforementioned same color channel, It is one of the color channels, the first green channel and the second green channel. The average pixel value is, The image sensing device according to claim 3, characterized in that the pixel data included in the region of interest includes the pixel values ​​of the pixel data having the first green channel and the average value of the pixel values ​​of the pixel data having the second green channel.

6. The aforementioned image processor is The image sensing apparatus according to claim 1, further comprising an image calibrator that calculates a gain value corresponding to a selected point based on the distance between a point selected in one of the plurality of block regions and a vertex of the one block region, and a reference gain value corresponding to a vertex of the one block region.

7. The vertices of the aforementioned block region are, The first vertex, second vertex, third vertex, and fourth vertex of the aforementioned block region, The aforementioned image calibrator is The image sensing device according to claim 6, characterized in that it calculates a gain value corresponding to the selected point based on the distance between the selected point and the first vertex, the distance between the selected point and the second vertex, the distance between the selected point and the third vertex, the distance between the selected point and the fourth vertex, a reference gain value corresponding to the first vertex, a reference gain value corresponding to the second vertex, a reference gain value corresponding to the third vertex, and a reference gain value corresponding to the fourth vertex.

8. The aforementioned image processor is The image sensing device according to claim 1, characterized in that the reference gain value and the gain value calculated using the reference gain value are stored in the memory.

9. The aforementioned image processor is The image sensing apparatus according to claim 8, characterized in that when the image is acquired via the image sensor, it includes an image collector that generates a corrected image by applying the reference gain value stored in the memory and the gain value to the plurality of pixel data included in the image.

10. Each of the aforementioned multiple block regions is, The image sensing device according to claim 1, characterized in that the area is rectangular or square.

11. The steps include: acquiring a calibration image via an image sensor, calculating a reference gain value for vertices defined by multiple block regions included in a gain map corresponding to the size of the calibration image; A step of calculating the gain value included in each of the plurality of block regions using the aforementioned reference gain value, The steps include storing the aforementioned reference gain value and the aforementioned gain value in memory, The process includes the step of acquiring an image via the image sensor and outputting the reference gain value stored in the memory and a corrected image obtained by applying the gain value to multiple pixel data, The aforementioned multiple block regions are, It includes a first block region and a second block region that is closer to the center of the calibration image than the position of the first block region and is larger in size than the first block region, Of the plurality of block regions, each of the third block regions that is furthest from the center of the image and has the smallest size includes a first vertex and a second vertex that is closest to the first vertex and further from the center of the image than the first vertex. A method for operating an image sensing device, characterized in that a first reference gain value corresponding to the first vertex is copied to obtain a second reference gain value corresponding to the second vertex.

12. The gain values ​​included in the first block region are The method for operating an image sensing device according to claim 11, characterized in that a number of bits larger than the gain value included in the second block region is assigned.

13. The step of calculating the aforementioned reference gain value is: The steps include selecting pixel data corresponding to the vertices of the plurality of block regions from a plurality of pixel data included in a calibration image acquired via the image sensor, A step of calculating the average pixel value of pixel data that has the same color channel as the selected pixel data, among the pixel data included in the region of interest which contains one of the selected pixel data, The method for operating an image sensing device according to claim 11, comprising the step of calculating one of the reference gain values ​​using the pixel values ​​of the selected pixel data and the average pixel value.

14. The aforementioned same color channel, The method for operating an image sensing apparatus according to claim 13, characterized in that the color channel is one of the first green channel, the second green channel, the red channel, and the blue channel.

15. The aforementioned same color channel, It is one of the color channels, the first green channel and the second green channel. The average pixel value is, The method for operating an image sensing apparatus according to claim 13, characterized in that the pixel data included in the region of interest includes the pixel values ​​of the pixel data having the first green channel and the average value of the pixel values ​​of the pixel data having the second green channel.

16. The step of calculating the aforementioned gain value is: The method for operating an image sensing device according to claim 11, further comprising the step of calculating a gain value corresponding to a selected point based on the distance between a point selected in one of the plurality of block regions and a vertex of the one block region, and a reference gain value corresponding to a vertex of the one block region.

17. The vertices of the aforementioned block region are, The first vertex, second vertex, third vertex, and fourth vertex of the aforementioned block region, The step of calculating the gain value corresponding to the selected point is: The method for operating an image sensing apparatus according to claim 16, characterized in that it includes calculating a gain value corresponding to the selected point based on the distance between the selected point and the first vertex, the distance between the selected point and the second vertex, the distance between the selected point and the third vertex, the distance between the selected point and the fourth vertex, a reference gain value corresponding to the first vertex, a reference gain value corresponding to the second vertex, a reference gain value corresponding to the third vertex, and a reference gain value corresponding to the fourth vertex.

18. Each of the aforementioned multiple block regions is, The method for operating the image sensing device according to claim 11, characterized in that the area is rectangular or square.

Images