Image signal processor, image system, and operating method of image signal processor

Abstract

An image signal processor that receives an input image from an image sensor includes: a point spread function (PSF) estimation circuit that adjusts a size of a plurality of reference PSFs corresponding to the image sensor based on pre-processed image data corresponding to the input image, and generates a plurality of estimated PSFs indicating estimation results for the plurality of PSFs corresponding to the input image; and a demosaicing circuit generating pre-processed image data based on the input image, and performing interpolation for the input image based on the plurality of estimated PSFs.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2024-0143306, filed on October 18, 2024, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The embodiments of this disclosure described herein relate to an image sensor, and more specifically, to an image signal processor, an image system including the image signal processor, and a method of operating the image signal processor. Background Technology

[0004] An image sensor acquires image information about an external object by converting light reflected from it into electrical signals. Electronic devices that include an image sensor can display images on a display panel using the acquired image information.

[0005] Image sensors can be installed in various types of electronic devices. For example, electronic devices that include image sensors can be included as components in various types of electronic devices such as smartphones, tablet PCs, laptop PCs, and wearable devices. Summary of the Invention

[0006] Generally speaking, this disclosure relates to an image signal processor with improved performance, an image system including the image signal processor, and a method of operating the image signal processor.

[0007] According to some embodiments, an image signal processor that receives an input image from an image sensor includes: a point spread function (PSF) estimation module that adjusts the size of a plurality of reference PSFs corresponding to the image sensor based on preprocessed image data corresponding to the input image, and generates a plurality of estimated PSFs indicating the estimation results of the plurality of PSFs corresponding to the input image; and a demosaicing module that generates preprocessed image data based on the input image, and performs interpolation on the input image based on the plurality of estimated PSFs.

[0008] According to some embodiments, this disclosure relates to an operation method of an image signal processor, comprising: receiving an input image from an image sensor; generating preprocessed image data based on the input image; generating a plurality of estimated PSFs by adjusting the sizes of a plurality of reference point spread functions (PSFs) corresponding to the image sensor based on the preprocessed image data; and generating an interpolated image by performing interpolation on the input image based on the plurality of estimated PSFs, wherein the plurality of estimated PSFs indicate estimation results for the plurality of PSFs corresponding to pixels of the input image respectively.

[0009] According to some embodiments, this disclosure relates to an image system, including: a first image sensor that outputs a first input image; a second image sensor that outputs a second input image; and an image signal processor. The image signal processor includes: a storage device storing first reference PSF data associated with a first reference point spread function (PSF) corresponding to the first image sensor, and second reference PSF data associated with a second reference PSF corresponding to the second image sensor; a PSF estimation module that adjusts the size of the first reference PSF to generate a plurality of first estimated PSFs indicating the estimation result of the PSF corresponding to the first input image, and adjusts the size of the second reference PSF to generate a plurality of second estimated PSFs indicating the estimation result of the PSF corresponding to the second input image; and a de-mosaic module that performs interpolation of the first input image based on the plurality of first estimated PSFs, and performs interpolation of the second input image based on the plurality of second estimated PSFs. Attached Figure Description

[0010] The exemplary embodiments will be more clearly understood from the following detailed description in conjunction with the accompanying drawings.

[0011] Figure 1 This is a diagram illustrating an example of an image system according to some implementation methods.

[0012] Figure 2 The illustration is based on some implementation methods. Figure 1 A block diagram of an example image sensor.

[0013] Figure 3A and Figure 3B This is a diagram illustrating an example of false color phenomena caused by PSF differences for each color channel, according to some implementation methods.

[0014] Figure 4 The illustration is based on some implementation methods. Figure 1 A block diagram of an example image signal processor.

[0015] Figure 5 The illustration is based on some implementation methods. Figure 4 A block diagram of an example of the PSF estimation module and the demosaic module.

[0016] Figure 6 The illustration is based on some implementation methods. Figure 1 A flowchart illustrating an example of the operation of an image signal processor.

[0017] Figure 7 The illustration is generated according to some implementation methods. Figure 6 The flowchart shows an example operation for estimating the PSF set.

[0018] Figure 8 The illustration is generated according to some implementation methods. Figure 7 The flowchart shows an example operation of the first estimated PSF set.

[0019] Figure 9 The illustration is based on some implementation methods. Figure 5 A diagram showing examples of the reference PSF set and the estimated PSF set.

[0020] Figure 10 The illustration is based on some implementation methods. Figure 1 A diagram illustrating an example of an interpolation image generation method for an image signal processor.

[0021] Figure 11 The illustration is based on some implementation methods. Figure 5 A block diagram of an example interpolated image generation unit.

[0022] Figures 12A and 12B are illustrations. Figure 10 A diagram illustrating an example of the operation for generating color information image data, and Figure 12C It is used to describe according to some implementation methods Figure 10 The diagram illustrates the operations for generating interpolated images.

[0023] Figure 13 The illustration is based on some implementation methods. Figure 4 A block diagram of an example of the PSF estimation module and the demosaic module.

[0024] Figure 14 The illustration is based on some implementation methods. Figure 13 An example diagram of an interpolated image generator.

[0025] Figure 15 The illustration is based on some implementation methods. Figure 14 A diagram illustrating the operation of a color information image data generator and an interpolation image generator.

[0026] Figure 16 This is a block diagram illustrating an example of an image system according to some implementation methods.

[0027] Figure 17 This is a block diagram illustrating an example of an image sensor according to some implementation methods.

[0028] Figure 18 This is a block diagram illustrating an example of an electronic device including a multi-camera module according to some embodiments.

[0029] Figure 19 The illustration is based on some implementation methods. Figure 18 A block diagram of an example camera module. Detailed Implementation

[0030] In the following text, exemplary embodiments will be explained in detail with reference to the accompanying drawings.

[0031] In this disclosure, functional blocks corresponding to the terms "block", "unit", "logic", etc., can be implemented in the form of software, hardware, or a combination thereof.

[0032] Figure 1 This is a diagram illustrating an example of an image system according to some implementation methods. Figure 1 In this system, the imaging system 10 may include a lens RS, an image sensor 100, and an image signal processor 200. In some embodiments, the imaging system 10 may be implemented as part of various electronic devices, such as cameras, smartphones, wearable devices, Internet of Things (IoT) devices, home appliances, tablet computers (PCs), personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, drones, advanced driver assistance systems (ADAS), traffic cameras, and CCTV. Furthermore, the imaging system 10 may be installed in electronic devices provided as part of vehicles, furniture, manufacturing equipment, doors, and various measuring instruments.

[0033] Lens RS can correspond to image sensor 100. Lens RS can receive light reflected from an external object. Based on the light received through lens RS, image sensor 100 can generate an electrical image signal. For example, image sensor 100 can be implemented using a complementary metal-oxide-semiconductor (CMOS) image sensor, etc. However, this disclosure is not limited thereto. For example, image sensor 100 can be implemented based on various image sensors such as dynamic vision sensors (DVS) and digital pixel sensors (DPS). Image sensor 100 can output an image generated based on light reflected from an external object as an input image IMG_in.

[0034] Image signal processor 200 can receive an input image IMG_in from image sensor 100 and can perform image signal processing on the received input image IMG_in. Image signal processor 200 can output an output image IMG_out as a result of image signal processing. For example, the output image IMG_out may have improved image quality compared to the input image IMG_in. The output image IMG_out can be provided to an external device (e.g., an application processor (AP), a graphics processing unit (GPU), or a display device).

[0035] The image signal processor 200 may include a point spread function (PSF) estimation module (circuit) 210 and a demosaicing module (circuit) 220. The PSF estimation module 210 can estimate the PSF corresponding to each pixel of the input image IMG_in. The PSF estimation module 210 can adjust the size of a reference PSF corresponding to the image sensor 100 to estimate the PSF corresponding to the input image IMG_in. The PSF estimation module 210 can generate a set of multiple estimated PSFs (EPST) including the estimated PSFs.

[0036] The demosaic module 220 can perform demosaic processing on an input image IMG_in with a non-Bayer pattern (e.g., a tetra pattern or a hexa pattern) to generate a demosaic image in Bayer format. In some embodiments, the demosaic module 220 can perform demosaic processing on an input image IMG_in with either a Bayer pattern or a non-Bayer pattern (e.g., a tetra pattern or a hexa pattern) to generate a demosaic image in RGB format. That is, in this case, the demosaic processing can include all or some of the operations performed for general demosaic processing. The demosaic module 220 can perform interpolation on the input image IMG_in based on the estimated PSF set EPST to generate an interpolated image, and can generate the demosaic image based on the interpolated image.

[0037] For example, the focal position of red light corresponding to the first pixel can be different from that of green light. Therefore, the point spread function (PSF) of red light can be different from that of green light. Due to the PSF differences for each color (or wavelength) of light, false colors may appear in the output image IMG_out. Referring below to Figure 3A and... Figure 3B Describe in detail how the false color phenomenon occurs.

[0038] According to some implementations, when the demosaic module 220 performs interpolation based on the estimated PSF set EPST, the demosaic module 220 can mitigate the occurrence of false colors due to PSF differences for each color (or wavelength) of light. Therefore, the performance of the image system 10 can be improved.

[0039] Figure 2 The illustration is based on some implementation methods. Figure 1 A block diagram of an example image sensor. (See Figure 1 and...) Figure 2 In this process, the image sensor 100 may include a pixel array 110, a line driver 120, an analog-to-digital converter (ADC) 130, an output buffer 140, and a control logic circuit 150.

[0040] Pixel array 110 may include a plurality of pixels. The plurality of pixels may be arranged in both row and column directions. Each pixel of pixel array 110 may output a pixel signal depending on the intensity or amount of light incident from the outside. In this case, the pixel signal may be an analog signal corresponding to the intensity or amount of light incident from the outside. In some embodiments, pixel array 110 may include a color filter array (CFA). The color filter array may be implemented with a Bayer pattern, a four-color pattern, a nine-color pattern, a six-color pattern, a ten-color pattern, or various color patterns. In some embodiments, the input image IMG_in may have the same color pattern as the color filter array of pixel array 110.

[0041] The line driver 120 can provide line control signals (e.g., RST, TX, and SEL) to the pixel array 110. Multiple pixels of the pixel array 110 can operate in response to the line control signals provided from the line driver 120. The analog-to-digital converter 130 can receive pixel signals from the multiple pixels of the pixel array 110, convert the received pixel signals into digital signals, and output them. The output buffer 140 can store the digital signals output from the analog-to-digital converter 130 and can output the stored digital signals as the input image IMG_in. The input image IMG_in can be provided to the image signal processor 200. The control logic circuitry 150 can control all operations of the image sensor 100.

[0042] Reference Figure 2 The illustration describes a schematic configuration of the image sensor 100, and this disclosure is not limited thereto. It will be understood that the image sensor 100 may be implemented in various structures that can be understood by those skilled in the art.

[0043] Figure 3A and Figure 3B This is a diagram illustrating an example of false color phenomena caused by PSF differences in each color channel, according to some implementation methods. Figure 3A In this structure, the focal point of green light reflected from an external object and passing through lens RS can be formed at a first focal point position FP1. Similarly, the focal point of red light reflected from an external object and passing through lens RS can be formed at a second focal point position FP2. In other words, the position of the first pixel PX1 corresponding to the red and green light can be different from the focal points FP1 and FP2 that form the red and green light. Therefore, the red and green light reflected from an external object and passing through lens RS can be expanded in the form of corresponding PSFs and incident on pixel array 110. Furthermore, because the angle at which light is incident on pixel array 110 varies depending on the focal point position, the size of the PSF corresponding to the red light and the size of the PSF corresponding to the green light can be different from each other.

[0044] For example, the PSF of green light corresponding to the first pixel PX1 could be the first PSF (PSF1), the PSF of red light corresponding to the first pixel PX1 could be the second PSF (PSF2), and the PSF of blue light corresponding to the first pixel PX1 could be the third PSF (PSF3). The size of the PSF indicates the effect of light incident on a pixel on the pixel values ​​of surrounding pixels. In other words, the size of the PSF indicates the degree of blur in the image. Therefore, in Figure 3A In this context, the red light incident on the first pixel PX1 can have a greater impact on the pixel values ​​of other pixels than the green light, and the blue light can also have a greater impact than the green light.

[0045] exist Figure 3B In this process, the image signal processor 200 can obtain red image data RID, panchromatic image data PID, and blue image data BID during the interpolation of the input image IMG_in. For example, the input image IMG_in can be an image with a tetra pattern. The red image data RID can be generated by interpolation based on the red pixel values ​​(R) of the input image IMG_in. The panchromatic image data PID can be generated by interpolation based on the green pixel values ​​(G) of the input image IMG_in. The blue image data BID can be generated by interpolation based on the blue pixel values ​​(B) of the input image IMG_in.

[0046] The red image data RID, panchromatic image data PID, and blue image data BID can each contain a number of pixels equal to the number of pixels in the input image IMG_in. In other words, each pixel in the red image data RID, panchromatic image data PID, and blue image data BID can correspond to a pixel in the input image IMG_in.

[0047] Additionally, the panchromatic image data PID can include information about the brightness of the input image IMG_in. The red image data RID and blue image data BID can include information about the color of the input image IMG_in.

[0048] For example, the image signal processor 200 can obtain image data including red information by subtracting the panchromatic image data PID from the red image data RID. The image signal processor 200 can obtain image data including blue information by subtracting the panchromatic image data PID from the blue image data BID. The image signal processor 200 can generate an interpolated image IMG_C by combining the panchromatic image data PID, the image data including red information, and the image data including blue information.

[0049] In this case, as described above, since the PSF size changes for each color of light, for example, the PSF size corresponding to the red image data RID can be larger than the PSF size corresponding to the panchromatic image data PID. Furthermore, the PSF size corresponding to the blue image data BID can be larger than the PSF size corresponding to the panchromatic image data PID. In other words, the red image data RID and blue image data BID can have a greater degree of blurring than the panchromatic image data PID.

[0050] As described above, the image signal processor 200 can acquire image data including red information and image data including blue information without considering the PSF differences between the red image data RID, the panchromatic image data PID, and the blue image data BID. The image signal processor 200 can generate an interpolated image IMG_C based on the image data including the acquired color information and the panchromatic image data PID. The image signal processor 200 can post-process the interpolated image IMG_C to generate an output image IMG_out. In this case, due to the PSF differences between the red image data RID and the panchromatic image data PID, and the PSF differences between the blue image data BID and the panchromatic image data PID, false colors may appear in the output image IMG_out.

[0051] Alternatively, the PSF can be determined based on the characteristics of the lens RS. These characteristics may include the degree of defocusing of the lens RS, the material of the lens RS, the tilt of the lens RS, and the shape of the lens RS. For example, the degree of defocusing of the lens RS can be determined based on the distance from the lens RS to the pixel array 110.

[0052] At the same time, Figure 3B The illustration shows an example in which each of the input image IMG_in, red image data RID, panchromatic image data PID, and blue image data BID comprises 16 pixels, but this disclosure is not limited thereto.

[0053] According to some implementations, the image signal processor 200 can estimate the PSF corresponding to the pixels of the input image IMG_in based on the current characteristics of the lens RS. Specifically, the image signal processor 200 can estimate the PSF corresponding to the pixels of the red image data RID and the blue image data BID, respectively. The image signal processor 200 can perform interpolation on the input image IMG_in based on a set of estimated PSFs including the estimated PSFs. Based on the above description, the image signal processor 200 can reduce the PSF difference between the red image data RID and the panchromatic image data PID, and the PSF difference between the blue image data BID and the panchromatic image data PID. Therefore, the image signal processor 200 can mitigate the occurrence of false colors due to PSF differences. The operation of the image system 10 according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0054] Figure 4 The illustration is based on some implementation methods. Figure 1 A block diagram of an example image signal processor. Figure 4 In this context, the image signal processor 200 may include a PSF estimation module 210, a de-mosaic module 220, a noise reduction module 230, a white balance module 240, and a one-time programmable (OTP) memory 250. However, this disclosure is not limited thereto. For example, the image signal processor 200 may be coupled with... Figure 4 The examples shown are different, and may further include any type of signal processing circuitry or some signal processing modules that may not be included.

[0055] The PSF estimation module 210 can estimate the PSF corresponding to each pixel of the input image IMG_in. The PSF estimation module 210 can generate a set of multiple estimated PSFs, EPST, which includes the estimated PSFs.

[0056] The demosaic module 220 can perform demosaic processing on the input image IMG_in. The demosaic module 220 can perform interpolation on the input image IMG_in based on the estimated PSF set EPST, and can generate an interpolated image. The demosaic module 220 can generate a demosaic image based on the interpolated image.

[0057] The noise reduction module 230 can be configured to remove noise from the input image IMG_in. For example, the noise reduction module 230 can be configured to remove fixed pattern noise or temporal random noise based on the color filter array (CFA) of the image sensor 100. The white balance module 240 can perform white balance. For example, the white balance module 240 can perform white balance on the output of the noise reduction module 230.

[0058] OTP memory 250 can be configured to store reference PSF data DATA_Pref for generating the estimated PSF set EPST. In some embodiments, the reference PSF data DATA_Pref may refer to data in which information about the reference PSF corresponding to the pixels of the input image IMG_in is compressed (or encoded). For example, the reference PSF may be a PSF determined in advance based on the physical characteristics of the lens RS corresponding to the image system 10 during the manufacturing process of the image system 10.

[0059] In some implementations, the OTP memory 250 can be a memory that cannot record additional data once it has been recorded. Furthermore, even if the power supply to the OTP memory 250 is turned off, the data stored in the OTP memory 250 will not be lost. Figure 4 An example is shown where reference PSF data DATA_Pref is stored in OTP memory 250, but this disclosure is not limited thereto. In other words, in some embodiments, reference PSF data DATA_Pref may be stored in any other non-volatile memory other than OTP memory.

[0060] Figure 5 The illustration is based on some implementation methods. Figure 4 A block diagram of an example PSF estimation module and a demosaic module. (See Figure 1.) Figure 4 and Figure 5 In the PSF estimation module (circuit) 210, there may be a defocus calculation unit (circuit) 211, a reference PSF extraction unit (circuit) 212, and an estimated PSF generation unit (circuit) 213. The demosaic module 220 may be an interpolated image generation unit (circuit) 221 and a demosaic image generation unit (circuit) 222.

[0061] The defocus calculation unit 211 can receive preprocessed image data IDAT_P generated based on the input image IMG_in from the interpolation image generation unit 221. The preprocessed image data IDAT_P may include red image data RID, panchromatic image data PID, and blue image data BID. The red image data RID, panchromatic image data PID, and blue image data BID can be respectively associated with… Figure 3B The red image data (RID), panchromatic image data (PID), and blue image data (BID) correspond to each other.

[0062] The defocus calculation unit 211 can receive position information P_info, which indicates information about the position of each pixel in the input image IMG_in. In some embodiments, the defocus calculation unit 211 can receive the position information P_info from the control circuitry controlling the image signal processor 200. In one embodiment, the defocus calculation unit 211 can receive the position information P_info from... Figure 2 The control logic circuit 150 receives the position information P_info.

[0063] The defocus calculation unit 211 can calculate the variance ratio corresponding to the pixels of the input image IMG_in based on the position information P_info and the preprocessed image data IDAT_P. The defocus calculation unit 211 can send the variance ratio data VRD, including the calculated variance ratio, to the estimated PSF generation unit 213. In some embodiments, the calculated variance ratio may include a first variance ratio and a second variance ratio. The first variance ratio may include information about the PSF difference between the red image data RID and the panchromatic image data PID. The second variance ratio may include information about the PSF difference between the blue image data BID and the panchromatic image data PID.

[0064] The reference PSF extraction unit 212 can receive position information P_info and reference PSF data DATA_Pref. In some embodiments, the reference PSF extraction unit 212 can receive reference PSF data DATA_Pref from the OTP memory 250. In some embodiments, the reference PSF extraction unit 212 can receive position information P_info from the control circuit (not shown) controlling the image signal processor 200. In some embodiments, the reference PSF extraction unit 212 can receive... Figure 2 The control logic circuit 150 receives the position information P_info.

[0065] The reference PSF extraction unit 212 can obtain a reference PSF set RPST, which includes reference PSFs corresponding to the pixels of the input image IMG_in, based on the reference PSF data DATA_Pref. Specifically, the reference PSF extraction unit 212 can decompress (or decode) the reference PSF data DATA_Pref to extract the reference PSF set RPST. The reference PSF extraction unit 212 can then send the reference PSF set RPST to the estimated PSF generation unit 213.

[0066] The PSF generation unit 213 can generate multiple estimated PSF sets EPST based on the variance ratio data VRD and the reference PSF set RPST. The multiple estimated PSF sets EPST can include a first estimated PSF set EPST_R and a second estimated PSF set EPST_B. The first estimated PSF set EPST_R can include multiple first estimated PSFs corresponding to pixels of the input image IMG_in, and the second estimated PSF set EPST_B can include multiple second estimated PSFs corresponding to pixels of the input image IMG_in.

[0067] The PSF generation unit 213 can adjust the size of each reference PSF in the reference PSF set RPST based on a first variance ratio, and can generate a first estimated PSF. The PSF generation unit 213 can also adjust the size of each reference PSF in the reference PSF set RPST based on a second variance ratio, and can generate a second estimated PSF. The PSF generation unit 213 can send the estimated PSF set EPST and PSF information PSF_info to the interpolation image generation unit 221. In an embodiment, the PSF information PSF_info may include information about the image data to which the estimated PSF set EPST will be applied.

[0068] As described above, the estimated PSF set EPST can be generated based on the variance ratio data VRD corresponding to the input image IMG_in. Therefore, the estimated PSF set EPST can include PSFs estimated based on the current characteristics (e.g., degree of defocus or tilt) of the lens RS corresponding to the image sensor 100.

[0069] The interpolation image generation unit 221 can interpolate the input image IMG_in to generate preprocessed image data IDAT_P, which includes red image data RID, panchromatic image data PID, and blue image data BID. The interpolation image generation unit 221 can perform interpolation of the input image IMG_in based on the estimated PSF set EPST, and can generate the interpolated image IMG_C.

[0070] For example, the interpolation image generation unit 221 can apply the estimated PSF set EPST to at least some of the red image data RID, panchromatic image data PID, and blue image data BID. In this case, the PSF size of at least some of the red image data RID, panchromatic image data PID, and blue image data BID can be adjusted. The interpolation image generation unit 221 can generate an interpolated image IMG_C by utilizing image data with its PSF adjusted. As described above, the interpolated image IMG_C can be an image in which the occurrence of false color phenomena caused by the PSF differences of the red image data RID, panchromatic image data PID, and blue image data BID is mitigated. In an embodiment, the interpolated image IMG_C can be an image in RGB format, which includes all of the red pixel values ​​(R), green pixel values ​​(G), and blue pixel values ​​(B).

[0071] The demosaic image generation unit 222 can perform post-processing on the interpolated image IMG_C to generate a demosaic image IMG_R. In some embodiments, the demosaic image IMG_R can be an RGB format image. In some embodiments, the demosaic image IMG_R can be a Bayer format image. In this case, the demosaic image generation unit 222 can adjust the arrangement of the interpolated image IMG_C to generate a Bayer format demosaic image IMG_R.

[0072] As described above, according to some implementations, the image signal processor 200 can estimate the PSF corresponding to the input image IMG_in. The image signal processor 200 can adjust the size of the PSF corresponding to the input image IMG_in based on the estimated PSF, and then perform interpolation on the input image IMG_in. According to the above description, the image signal processor 200 can mitigate the occurrence of false colors due to PSF differences in each color channel.

[0073] Figure 6 The illustration is based on some implementation methods. Figure 1 A flowchart illustrating an example of the operation of an image signal processor. (See Figure 1.) Figure 5 and Figure 6 In operation S110, the image signal processor 200 can receive the input image IMG_in.

[0074] In operation S120, the image signal processor 200 can generate preprocessed image data IDAT_P based on the input image. For example, the demosaic module 220 can interpolate the input image IMG_in to generate preprocessed image data IDAT_P that includes red image data RID, panchromatic image data PID, and blue image data BID.

[0075] In operation S130, the image signal processor 200 can obtain a reference PSF set RPST. For example, the PSF estimation module 210 can obtain a reference PSF set RPST that includes reference PSFs corresponding to the pixels of the input image IMG_in, based on the reference PSF data DATA_Pref.

[0076] In operation S140, the image signal processor 200 can adjust the size of the reference PSF to generate estimated PSFs that will be included in the estimated PSF set EPST. For example, the PSF estimation module 210 can generate variance ratio data VRD based on preprocessed image data IDAT_P. The PSF estimation module 210 can generate the estimated PSF set EPST by adjusting the size of the reference PSF based on the variance ratio data VRD.

[0077] In operation S150, the image signal processor 200 can perform interpolation on the input image based on the estimated PSF set EPST, and can generate the interpolated image IMG_C. For example, the demosaic module 220 can generate the interpolated image IMG_C by applying the estimated PSF set EPST to the preprocessed image data IDAT_P.

[0078] In operation S160, the image signal processor 200 can generate a de-mosaic image IMG_R based on the interpolated image IMG_C. For example, the de-mosaic module 220 can perform post-processing on the interpolated image IMG_C to generate a de-mosaic image IMG_R in RGB or Bayer format.

[0079] Figure 7 The illustration is generated according to some implementation methods. Figure 6 A flowchart of an example operation for estimating the PSF set. (See Figure 1 and...) Figures 5 to 7 In operation S141, the image signal processor 200 can adjust the size of the reference PSF to generate a first estimated PSF set EPST_R. For example, the PSF estimation module 210 can generate the first estimated PSF set EPST_R by adjusting the size of the reference PSF of the reference PSF set RPST based on the preprocessed image data IDAT_P.

[0080] For example, the size of the PSF corresponding to each pixel of the red image data RID can be larger than the size of the PSF corresponding to each pixel of the panchromatic image data PID. The first estimated PSF set EPST_R can include a first PSF, where each PSF indicates an estimation result of the PSF for each pixel of the red image data RID. In other words, the first PSF can be the PSF corresponding to the red channel of the input image IMG_in.

[0081] In operation S142, the image signal processor 200 can adjust the size of the reference PSF to generate a second estimated PSF set EPST_B. For example, the PSF estimation module 210 can generate the second estimated PSF set EPST_B by adjusting the size of the reference PSF of the reference PSF set RPST based on the preprocessed image data IDAT_P.

[0082] For example, the size of the PSF corresponding to each pixel of the blue image data BID can be larger than the size of the PSF corresponding to each pixel of the panchromatic image data PID. The second estimated PSF set EPST_B can include a second PSF, each in the second PSF indicating an estimate of the PSF for each pixel of the blue image data BID. In other words, the second PSF can be the PSF corresponding to the blue channel of the input image IMG_in.

[0083] Figure 8 The illustration is generated according to some implementation methods. Figure 7 A flowchart of an example operation for estimating the first PSF set. (See Figure 1 and...) Figures 5 to 8 In operation S141a, the image signal processor 200 can perform luminance normalization on the red image data RID based on the panchromatic image data PID. For example, the defocus calculation unit 211 can generate red-normalized image data NRID by performing luminance normalization on the red image data RID based on Equation 1 below.

[0084] [Equation 1]

[0085]

[0086] In Equation 1, NRID(x, y) is the pixel value of the pixel located at (x, y) in the red-normalized image data NRID, mean(PID) is the average pixel value of the panchromatic image data PID, mean(RID) is the average pixel value of the red image data RID, and RID(x, y) is the pixel value of the pixel located at (x, y) in the red image data RID. The red-normalized image data NRID can indicate the result obtained by adjusting the brightness of the red image data RID based on the brightness of the panchromatic image data PID.

[0087] In operation S141b, the image signal processor 200 can calculate a first variance var1 corresponding to the first pixel of the red image data RID. For example, the defocus calculation unit 211 can calculate the first variance var1 based on the pixel value of the first pixel of the red image data RID and the pixel values ​​of "n" pixels surrounding the first pixel. The first variance var1 may include information about the size of the PSF corresponding to the first pixel of the red image data RID. For example, when the first variance var1 increases, the size of the PSF corresponding to the first pixel of the red image data RID may decrease.

[0088] In operation S141c, the image signal processor 200 can calculate a second variance var2 corresponding to the first pixel of the panchromatic image data PID. In an embodiment, the first pixel of the panchromatic image data PID can refer to the pixel in the panchromatic image data PID whose position corresponds to the position of the first pixel of the red image data RID. For example, the defocus calculation unit 211 can calculate the second variance var2 based on the pixel value of the first pixel of the panchromatic image data PID and the pixel values ​​of "n" pixels surrounding the first pixel. The second variance var2 can include information about the size of the PSF corresponding to the first pixel of the panchromatic image data PID. For example, when the second variance var2 increases, the size of the PSF corresponding to the first pixel of the panchromatic image data PID can decrease.

[0089] In operation S141d, the image signal processor 200 can calculate the first variance ratio based on the first variance var1 and the second variance var2. For example, the defocus calculation unit 211 can calculate the first variance ratio VR1 based on the following equation 2.

[0090] [Equation 2]

[0091]

[0092] The parameters in Equation 2 have already been described above, and further descriptions will be omitted to avoid redundancy. For example, the first variance ratio VR1 may include information about the difference in size between the PSF of the first pixel of the red image data RID and the PSF of the first pixel of the panchromatic image data PID. For example, the first variance var1 may be less than the second variance var2. In this case, the first variance ratio VR1 may indicate how much larger the PSF of the first pixel of the red image data RID is than the PSF of the first pixel of the panchromatic image data PID. For example, the second variance var2 may be less than the first variance var1. In this case, the first variance ratio VR1 may indicate how much smaller the PSF of the first pixel of the red image data RID is than the PSF of the first pixel of the panchromatic image data PID.

[0093] In operation S141e, the image signal processor 200 can generate a first estimated PSF based on a first variance ratio VR1 and a first reference PSF corresponding to a first pixel of the input image IMG_in. For example, the estimated PSF generation unit 213 can obtain the first variance ratio VR1 from the variance ratio data VRD received from the self-defocus calculation unit 211. Similarly, the estimated PSF generation unit 213 can obtain the first reference PSF from the reference PSF set RPST. For example, the estimated PSF generation unit 213 can calculate the variance of the first reference PSF.

[0094] The PSF estimation generation unit 213 can adjust the size of the first reference PSF based on the first variance ratio VR1 and the variance of the first reference PSF, and can generate a first estimated PSF. The first pixel of the input image IMG_in can correspond to the first pixel of the red image data RID and the first pixel of the panchromatic image data PID. For example, the PSF estimation generation unit 213 can determine, based on the first variance ratio VR1 and the variance of the first reference PSF, how much the size of the first reference PSF has increased, how much the size of the first reference PSF has decreased, or whether the size of the first reference PSF has been maintained. For example, the PSF estimation generation unit 213 can check the size of the first reference PSF based on the variance of the first reference PSF. The PSF estimation generation unit 213 can determine, based on the first variance ratio VR1, whether to increase, decrease, or maintain the size of the first reference PSF.

[0095] The PSF estimation generation unit 213 can determine the extent to which the size of the first reference PSF increases or decreases based on the first variance ratio VR1. A value closer to "0" in the first variance ratio VR1 may mean that the PSF difference between the panchromatic image data PID and the red image data RID corresponding to the first pixel becomes larger. Therefore, the PSF estimation generation unit 213 can determine that the closer the value of the first variance ratio VR1 becomes to "0", the greater the increase or decrease. The PSF estimation generation unit 213 can adjust the size of the first reference PSF based on the determined increase or decrease and can generate a first estimated PSF.

[0096] For example, when the first variance is less than the second variance, the size of the first estimated PSF can correspond to the size of the PSF of the first pixel of the red image data RID. For example, when the first variance is greater than the second variance, the size of the first estimated PSF can correspond to the size of the PSF of the first pixel of the panchromatic image data PID.

[0097] For example, when the first variance is less than the second variance, the estimated PSF generation unit 213 can generate PSF information PSF_info, which includes information indicating that the first estimated PSF corresponds to the size of the PSF of the first pixel of the red image data RID.

[0098] In operation S141f, the image signal processor 200 can determine whether the PSF estimation is complete in association with all pixels of the red image data RID and the panchromatic image data PID. For example, if the estimation is not complete, the image signal processor 200 can execute operation S141b. If the estimation is complete, a first estimated PSF corresponding to all pixels of the red image data RID and the panchromatic image data PID can be generated. If the estimation is complete, the image signal processor 200 can execute operation S142.

[0099] As described above, each pixel in the red image data RID and the panchromatic image data PID corresponds to a pixel in the input image IMG_in. Therefore, the first estimated PSF can correspond to a pixel in the input image IMG_in.

[0100] In operation S142, the image signal processor 200 can generate a second estimated PSF set EPST_B in a similar manner to generating the first estimated PSF set EPST_R described above. Specifically, the image signal processor 200 can perform luminance normalization on the blue image data BID based on the panchromatic image data PID, calculate a first variance corresponding to a first pixel of the blue image data BID, calculate a second variance corresponding to a first pixel of the panchromatic image data PID, calculate a first variance ratio based on the first and second variances, and generate the second estimated PSF by adjusting the size of the first reference PSF based on the first variance ratio. The image signal processor 200 can generate a second estimated PSF set EPST_B, which includes the second estimated PSFs corresponding to all pixels of the blue image data BID and the panchromatic image data PID.

[0101] As described above, each pixel in the blue image data BID and the panchromatic image data PID corresponds to a pixel in the input image IMG_in. Therefore, the second estimated PSF can also correspond to a pixel in the input image IMG_in.

[0102] Figure 9 The illustration is based on some implementation methods. Figure 5 A diagram showing examples of the reference PSF set and the estimated PSF set. (See Figure 1 and...) Figures 5 to 9 In the input image, the reference PSF set RPST can include multiple reference PSF RPSFs. The reference PSF set RPST can include reference PSF RPSFs corresponding to pixels in the input image IMG_in. For example, with... Figure 3B Similarly, when the input image IMG_in contains 16 pixels, the reference PSF set RPST can include 16 reference PSFs RPSF. Each reference PSF can correspond to a pixel in the input image IMG_in.

[0103] The first estimated PSF set EPST_R can include multiple first estimated PSFs EPSF_R. (See reference...) Figure 9As described, each of the first estimated PSFs EPSF_R can be generated by adjusting the size of the corresponding reference PSF RPSF based on the variance ratio. Associated with the corresponding pixel, each of the first estimated PSFs EPSF_R can be similar in size to the larger PSF among the pixels of the red image data RID and the pixels of the panchromatic image data PID. The first estimated PSFs EPSF_R can each correspond to a pixel of the input image IMG_in.

[0104] The second estimated PSF set EPST_B can include multiple second estimated PSFs EPSF_B. (See reference...) Figure 9 As described, each of the second estimated PSFs EPSF_B can be generated by adjusting the size of the corresponding reference PSF RPSF based on the variance ratio. Associated with the corresponding pixel, each of the second estimated PSFs EPSF_B can be similar in size to the larger PSF among the pixels of the blue image data BID and the pixels of the panchromatic image data PID. The second estimated PSFs EPSF_B can each correspond to a pixel of the input image IMG_in.

[0105] Figure 10 It is based on some implementation methods Figure 1 An illustration of an example of an interpolation image generation method for an image signal processor. In Figure 1 and... Figures 5 to 10 In operation S210, the image signal processor 200 can generate color information image data based on preprocessed image data IDAT_P and the estimated PSF set EPST. For example, the interpolation image generation unit 221 can adjust the PSF of at least some of the red image data RID, panchromatic image data PID, and blue image data BID based on the estimated PSF set EPST. The interpolation image generation unit 221 can generate color information image data based on the adjusted image data.

[0106] In operation S220, the image signal processor 200 can generate an interpolated image IMG_C based on the color information image data and the panchromatic image data PID. For example, the interpolated image generation unit 221 can generate the interpolated image IMG_C by combining the color information image data and the panchromatic image data PID.

[0107] Figure 11 The illustration is based on some implementation methods. Figure 5 A block diagram of an example interpolated image generation unit. (See Figure 1 and...) Figures 5 to 11In this model, the interpolation image generation unit 221 may include a preprocessor 221_a, a color information image data generator 221_b, and an interpolation image generator 221_c. The preprocessor 221_a can generate preprocessed image data IDAT_P, including red image data RID, panchromatic image data PID, and blue image data BID, based on the input image IMG_in. The preprocessor 221_a can also generate red image data RID, panchromatic image data PID, and blue image data BID based on the pixel values ​​of the input image IMG_in. The preprocessor 221_a can then send the preprocessed image data IDAT_P to the defocus calculation unit 211 and the color information image data generator 221_b.

[0108] The color information image data generator 221_b can generate red information image data RIID and blue information image data BIID based on preprocessed image data IDAT_P, PSF information PSF_info, and estimated PSF set EPST. The color information image data generator 221_b can adjust the size of the PSF corresponding to the red image data RID or the panchromatic image data PID based on the first estimated PSF set EPST_R.

[0109] For example, the size of the PSF corresponding to the red image data RID can be greater than the size of the PSF corresponding to the panchromatic image data PID. In this case, the first estimated PSF set EPST_R can include a first estimated PSF indicating the current PSF of the red image data RID. Similarly, the PSF information PSF_info can include information indicating that the size of the PSF corresponding to the red image data RID is greater than the size of the PSF corresponding to the panchromatic image data PID. The color information image data generator 221_b can check that the size of the PSF corresponding to the red image data RID is greater than the size of the PSF corresponding to the panchromatic image data PID based on the PSF information PSF_info.

[0110] Based on the above description, the color information image data generator 221_b can increase the size of the PSF corresponding to the panchromatic image data PID based on the first estimated PSF set EPST_R. The color information image data generator 221_b can generate red information image data RIID based on the panchromatic image data PID with its PSF size adjusted and the red image data RID. The red information image data RIID can include color information about the red color in the input image IMG_in.

[0111] The color information image data generator 221_b can adjust the PSF size of the blue image data BID or the panchromatic image data PID based on the second estimated PSF set EPST_B. For example, the PSF size corresponding to the blue image data BID can be larger than the PSF size corresponding to the panchromatic image data PID. In this case, the PSF information PSF_info can include information indicating that the PSF size corresponding to the blue image data BID is larger than the PSF size corresponding to the panchromatic image data PID. The color information image data generator 221_b can check that the PSF size corresponding to the blue image data BID is larger than the PSF size corresponding to the panchromatic image data PID based on the PSF information PSF_info.

[0112] In this scenario, similar to the case where the color information image data generator 221_b generates red information image data RIID, the color information image data generator 221_b can increase the size of the PSF corresponding to the panchromatic image data PID based on the second estimated PSF set EPST_B, and can then generate blue information image data BIID. The blue information image data BIID can include color information about the blue color of the input image IMG_in.

[0113] The interpolated image generator 221_c can generate an interpolated image IMG_C by combining the red information image data RIID, the blue information image data BIID, and the panchromatic image data PID.

[0114] As described above, the color information image data generator 221_b can adjust the size of the PSF of the image data based on the estimated PSF set EPST, and then generate color information image data RIID and BIID. Therefore, during the generation of color information image data RIID and BIID, the PSF differences of the red image data RID, blue image data BID, and panchromatic image data PID can be reduced. Thus, the occurrence of false colors on the output image IMG_out generated based on the interpolated image IMG_C can be mitigated.

[0115] Figure 12A and Figure 12B The illustration is generated according to some implementation methods. Figure 10 An example diagram illustrating the manipulation of color information image data, and Figure 12C The illustration is generated according to some implementation methods. Figure 10 Figures 12A to 12B illustrate examples of image interpolation operations. Figure 12C In this context, it is assumed that the size of the PSF corresponding to the red image data RID and the blue image data BID is greater than the size of the PSF corresponding to the panchromatic image data PID.

[0116] exist Figure 12A In the first estimated PSF set EPST_R, there may be multiple first estimated PSFs EPSF_R1 to EPSF_R16. The first estimated PSFs EPSF_R1 to EPSF_R16 may correspond to the PSFs of the pixels in the red image data RID, respectively. The color information image data generator 221_b can perform a convolution between the panchromatic image data PID and the first estimated PSF set EPST_R, and can generate first adjusted panchromatic image data MPID1.

[0117] For example, the color information image data generator 221_b can generate first adjusted panchromatic image data MPID1 by performing a convolution between each green pixel value (G) of the panchromatic image data PID and a first estimated PSF (e.g., one of EPSF_R1 to EPSF_R16) at the corresponding location.

[0118] For example, the color information image data generator 221_b can determine the pixels with smaller PSF among the pixels of the panchromatic image data PID and the red image data RID (in the example above, the pixels of the panchromatic image data PID), and can generate the first adjusted panchromatic image data MPID1 by performing a convolution between the determined pixels and their corresponding first estimated PSF.

[0119] Through convolution, the PSF size of the first adjusted panchromatic image data MPID1 can be similar to the PSF size of the red image data RID. In other words, the PSF size corresponding to the first adjusted panchromatic image data MPID1 can become larger than the PSF size corresponding to the panchromatic image data PID. The color information image data generator 221_b can generate red information image data RIID by subtracting the first adjusted panchromatic image data MPID1 from the red image data RID. The red information image data RIID can be data generated after the PSF difference between the red information image data RIID and the panchromatic image data PID has been corrected. Similarly, the red information image data RIID can be data that includes the red information of the input image IMG_in.

[0120] exist Figure 12B In this context, the second estimated PSF set EPST_B may include multiple second estimated PSFs EPSF_B1 to EPSF_B16. The second estimated PSFs EPSF_B1 to EPSF_B16 may each correspond to the PSF of a pixel in the blue image data BID. The color information image data generator 221_b can perform a convolution between the panchromatic image data PID and the second estimated PSF set EPST_B, and can generate second adjusted panchromatic image data MPID2.

[0121] For example, the color information image data generator 221_b can generate second adjusted panchromatic image data MPID2 by performing a convolution between each green pixel value (G) of the panchromatic image data PID and a second estimated PSF (e.g., one of EPSF_B1 to EPSF_B16) at the corresponding location.

[0122] For example, the color information image data generator 221_b can determine the pixels with smaller PSF among the pixels of the panchromatic image data PID and the blue image data BID (in the example above, the pixels of the panchromatic image data PID), and can generate a second adjusted panchromatic image data MPID2 by performing a convolution between the determined pixels and their corresponding second estimated PSF.

[0123] Through convolution, the PSF size of the second adjusted panchromatic image data MPID2 can be similar to the PSF size of the blue image data BID. In other words, the PSF size corresponding to the second adjusted panchromatic image data MPID2 can become larger than the PSF size corresponding to the panchromatic image data PID. The color information image data generator 221_b can generate blue information image data BIID by subtracting the second adjusted panchromatic image data MPID2 from the blue image data BID. The blue information image data BIID can be data generated after the PSF difference between the blue information image data BIID and the panchromatic image data PID has been corrected. Similarly, the blue information image data BIID can be data that includes the blue information of the input image IMG_in.

[0124] exist Figure 12C In this process, the interpolated image generator 221_c can generate an interpolated image IMG_C by combining red information image data RIID, blue information image data BIID, and panchromatic image data PID. As described above, the interpolated image IMG_C can be generated based on red information image data RIID, in which the PSF difference between red image data RID and panchromatic image data PID is corrected, and blue information image data BIID, in which the PSF difference between blue image data BID and panchromatic image data PID is corrected. Therefore, the output image IMG_out generated based on the interpolated image IMG_C can be image data in which the false color phenomenon caused by PSF difference is mitigated.

[0125] Figure 13 The illustration is based on some implementation methods. Figure 4 Block diagrams of example PSF estimation and demosaic modules. Figure 13In this module, the PSF estimation module 210a may include a defocus calculation unit 211a, a reference PSF extraction unit 212a, and an estimated PSF generation unit 213a. The defocus calculation unit 211a, the reference PSF extraction unit 212a, and the estimated PSF generation unit 213a may respectively correspond to... Figure 5 The defocus calculation unit 211, the reference PSF extraction unit 212, and the estimated PSF generation unit 213 are included. The demosaic module 220a may include an interpolation image generation unit 221a and a demosaic image generation unit 222a. The interpolation image generation unit 221a and the demosaic image generation unit 222a may respectively correspond to... Figure 5 The interpolation image generation unit 221 and the demosaic image generation unit 222.

[0126] Since the operation of the components included in the PSF estimation module 210a and the demosaicing module 220a is described with reference to Figures 5 to 12, the following will mainly describe... Figure 5 and Figure 13 The differences between them.

[0127] exist Figure 13 In this process, the interpolation image generation unit 221a can generate panchromatic image data PID based on the input image IMG_in. Specifically, the interpolation image generation unit 221a can interpolate the input image IMG_in and generate panchromatic image data PID in which pixels have the same red pixel value (R), the same green pixel value (G), and the same blue pixel value (B), and have a saturation of "0". The panchromatic image data PID may include information about the brightness of the input image IMG_in. Figure 5 In different cases, the interpolation image generation unit 221a may send only the panchromatic image data PID to the defocus calculation unit 211a.

[0128] The defocus calculation unit 211a can generate variance ratio data VRD based on the input image IMG_in and the panchromatic image data PID. The variance ratio data VRD may include a first variance ratio, each indicating the PSF difference between the red image data RID and the panchromatic image data PID for each pixel. The variance ratio data VRD may also include a second variance ratio, each indicating the green image data GID for each pixel (see reference). Figure 14 The variance ratio data (VRD) can include a third-party variance ratio, where each third-party variance ratio indicates the PSF difference between the blue image data BID and the panchromatic image data PID for each pixel.

[0129] The defocus calculation unit 211a can calculate a first variance of the pixel-specific PSF size for each indicator of the red image data RID based on the red pixel value (R) of the input image IMG_in. For example, the defocus calculation unit 211a can calculate the pixel value of the red image data RID based on the red pixel value (R) of the input image IMG_in (e.g., by an interpolation algorithm), and then calculate the first variance.

[0130] As described above, the defocus calculation unit 211a can calculate the second variance of the pixel-specific PSF size indicating the green image data GID based on the green pixel values ​​(G) of the input image IMG_in. The green image data GID can be generated by interpolating the green pixel values ​​(G) of the input image IMG_in. In other words, the green image data GID can be... Figure 3B The panchromatic image data PID corresponds to the input image IMG_in. Similarly, the defocus calculation unit 211a can calculate the third variance of the pixel-specific PSF size indicating the blue image data BID based on the blue pixel value (B) of the input image IMG_in. The defocus calculation unit 211a can also calculate the fourth variance of the pixel-specific PSF size indicating the panchromatic image data PID based on the pixel value of the panchromatic image data PID.

[0131] The defocus calculation unit 211a can calculate the first variance ratio based on the first variance and the fourth variance. The defocus calculation unit 211a can calculate the second variance ratio based on the second variance and the fourth variance. The defocus calculation unit 211a can calculate the third variance ratio based on the third variance and the fourth variance.

[0132] The PSF generation unit 213a can adjust the size of the reference PSF based on a first variance ratio and generate a first estimated PSF set EPST_R. The PSF generation unit 213a can adjust the size of the reference PSF based on a second variance ratio and generate a second estimated PSF set EPST_G. The PSF generation unit 213a can adjust the size of the reference PSF based on a third variance ratio and generate a third estimated PSF set EPST_B.

[0133] The interpolation image generation unit 221a can adjust the PSF values ​​of the red image data RID, green image data GID, blue image data BID, and panchromatic image data PID based on the PSF information PSF_info and the estimated PSF set EPST, and can generate the interpolated image IMG_C. As described above, this can mitigate the occurrence of false colors caused by the PSF differences among the red image data RID, green image data GID, blue image data BID, and panchromatic image data PID.

[0134] Figure 14 The illustration is based on some implementation methods. Figure 13 An example diagram of an interpolated image generator. Figure 14 In this model, the interpolation image generation unit 221a may include a preprocessor 221a_a, a color information image data generator 221a_b, and an interpolation image generator 221a_c. The preprocessor 221a_a, the color information image data generator 221a_b, and the interpolation image generator 221a_c can be respectively connected to… Figure 11 The preprocessor 221_a, color information image data generator 221_b, and interpolation image generator 221_c correspond to each other.

[0135] Preprocessor 221a_a can generate preprocessed image data IDAT_P by interpolating the input image IMG_in. The preprocessed image data IDAT_P may include red image data RID, green image data GID, blue image data BID, and panchromatic image data PID. Preprocessor 221a_a can send the preprocessed image data IDAT_P to color information image data generator 221a_b. Preprocessor 221a_a can also send the panchromatic image data PID to color information image data generator 221a and interpolated image generator 221a_c.

[0136] The color information image data generator 221a_b can adjust the PSF of at least some of the red image data RID, green image data GID, blue image data BID, and panchromatic image data PID based on the PSF information PSF_info and the estimated PSF set EPST. The color information image data generator 221a_b can generate the red image data RID, green image data GID, blue image data BID, and panchromatic image data PID by utilizing image data with its PSF adjusted. The red information image data RIID can include information about the red color of the input image IMG_in. The green information image data GIID can include information about the green color of the input image IMG_in. The blue information image data BIID can include information about the blue color of the input image IMG_in.

[0137] The interpolated image generator 221a_c can generate an interpolated image IMG_C by combining red information image data RIID, green information image data GIID, blue information image data BIID, and panchromatic image data PID.

[0138] Figure 15 The illustration is based on some implementation methods. Figure 14 A diagram illustrating the operation of a color information image data generator and an interpolated image generator. Figure 15In this context, it is assumed that the size of the PSF corresponding to the red image data RID and the blue image data BID is greater than the size of the PSF corresponding to the panchromatic image data PID, and the size of the PSF corresponding to the green image data GID is less than the size of the PSF corresponding to the panchromatic image data PID.

[0139] In this scenario, the color information image data generator 221a_b can generate the red information image data RIID by subtracting the result of a convolution performed between the panchromatic image data PID and the first estimated PSF set EPST_R from the red image data RID. Through convolution, the PSF size of the panchromatic image data PID can be increased. Therefore, the red information image data RIID can be data generated after reducing the PSF difference between the red image data RID and the panchromatic image data PID.

[0140] The color information image data generator 221a_b can generate green information image data GIID by subtracting the result of convolution between green image data GID and a second estimated PSF set EPST_G from the panchromatic image data PID. Through convolution, the PSF size of the green image data GID can be increased. Therefore, the green information image data GIID can be data generated after reducing the PSF difference between the green image data GID and the panchromatic image data PID.

[0141] The color information image data generator 221a_b can generate blue information image data BIID by subtracting the result of convolution performed between panchromatic image data PID and the third estimated PSF set EPST_B from blue image data BID. Through convolution, the PSF size of the panchromatic image data PID can be increased. Therefore, the blue information image data BIID can be data generated after reducing the PSF difference between the blue image data BID and the panchromatic image data PID.

[0142] The interpolated image generator 221a_c can generate an interpolated image IMG_C by combining red information image data RIID, green information image data GIID, blue information image data BIID, and panchromatic image data PID. As described above, the red information image data RIID, green information image data GIID, and blue information image data BIID can be data generated by correcting the PSF differences between the image data RID, GID, and BID and the panchromatic image data PID. Therefore, the interpolated image IMG_C can be an image in which the occurrence of false color phenomena caused by PSF differences is mitigated.

[0143] Figure 16 This is a block diagram illustrating an example of an image system according to some implementation methods. Figure 16In this system, the imaging system 20 may include multiple lenses RS1 to RSn, multiple image sensors 310 to 3n0, and an image signal processor 400. A first image sensor 310 may correspond to a first lens RS1, a second image sensor 320 may correspond to a second lens RS2, and an nth image sensor 3n0 may correspond to an nth lens RSn. The multiple image sensors 310 to 3n0 can send multiple input images IMG_in1 to IMG_inn generated by light passing through the corresponding lenses (e.g., RS1 to RSn) to the image signal processor 400.

[0144] The image signal processor 400 can perform signal processing on multiple input images IMG_in1 to IMG_inn. The image signal processor 400 may include a PSF estimation module 410, a demosaic module 420, and an OTP memory 450. The PSF estimation module 410 can estimate the PSF corresponding to each of the multiple input images IMG_in1 to IMG_inn.

[0145] The demosaic module 420 can perform interpolation for each of the multiple input images IMG_in1 to IMG_inn based on the estimated PSF.

[0146] The OTP memory 450 may include reference PSF data DATA_Pref1 to DATA_Prefn, which correspond to a plurality of image sensors 310 to 3n0, respectively. The first reference PSF data DATA_Pref1 may correspond to the first image sensor 310, the second reference PSF data DATA_Pref2 may correspond to the second image sensor 320, and the nth reference PSF data DATA_Prefn may correspond to the nth image sensor 3n0.

[0147] The first reference PSF data DATA_Pref1 can refer to the compressed (or encoded) data containing information about the first reference PSF corresponding to the pixels of the first input image IMG_in1. The second reference PSF data DATA_Pref2 can refer to the compressed (or encoded) data containing information about the second reference PSF corresponding to the pixels of the second input image IMG_in2. The nth reference PSF data DATA_Prefn can refer to the compressed (or encoded) data containing information about the nth reference PSF corresponding to the pixels of the nth input image IMG_inn.

[0148] For example, the first reference PSF may be a PSF pre-generated during the manufacturing process of the imaging system 20 through measurements based on the physical characteristics of the first lens RS1. For example, the second reference PSF may be a PSF pre-generated during the manufacturing process of the imaging system 20 through measurements based on the physical characteristics of the second lens RS2. For example, the nth reference PSF may be a PSF pre-generated during the manufacturing process of the imaging system 20 through measurements based on the physical characteristics of the nth lens RSn.

[0149] The PSF estimation module 410 can, depending on the reference... Figures 1 to 15 The described method or configuration adjusts the size of the first reference PSF to generate a first estimated PSF corresponding to the first input image IMG_in1. The PSF estimation module 410 can adjust the size of the first reference PSF based on the reference PSF. Figures 1 to 15 The described method or configuration adjusts the size of the second reference PSF to generate a second estimated PSF corresponding to the second input image IMG_in1. The PSF estimation module 410 can adjust the size of the second reference PSF based on the reference PSF. Figures 1 to 15 The described method or configuration adjusts the size of the nth reference PSF to generate the nth estimated PSF corresponding to the nth input image IMG_inn.

[0150] In this case, the demosaic module 420 can perform interpolation for the first input image IMG_in1 based on the first estimated PSF, can perform interpolation for the second input image IMG_in2 based on the second estimated PSF, and can perform interpolation for the nth input image IMG_inn based on the nth estimated PSF.

[0151] In other words, according to some implementations, the image signal processor 400 can estimate the PSF corresponding to each of the plurality of input images IMG_in1 to IMG_inn based on a reference PSF corresponding to the plurality of image sensors 310 to 3n0. The image signal processor 400 can then perform interpolation for the input images IMG_in1 to IMG_inn based on the estimated PSF.

[0152] Figure 17 This is a block diagram illustrating an example of an image sensor according to some implementation methods. Figure 17 In this process, the image sensor 500 may include a pixel array 510, peripheral circuitry 520, and an image signal processor 530.

[0153] Pixel array 510 may include multiple pixels. Peripheral circuitry 520 may be configured to process information obtained from the multiple pixels of pixel array 510. In embodiments, peripheral circuitry 520 may include various components necessary for generating image data in image sensor 500, such as line drivers, ADCs, memory, and ramp signal generators.

[0154] The image signal processor 530 can perform image signal processing on the input image obtained by the peripheral circuit 520 and can output an output image IMG_out. In other words, the above describes an implementation of an image signal processor independent of an image sensor, but this disclosure is not limited thereto. For example, as Figure 17 As shown, the entire image signal processor 530 or at least a portion thereof may be included in the image sensor 500.

[0155] Figure 18 This is a block diagram illustrating an example of an electronic device including a multi-camera module according to some embodiments. Figure 19 These are detailed illustrations based on some implementation methods. Figure 18 A block diagram of an example camera module.

[0156] exist Figure 18 In this context, electronic device 1000 may include camera module group 1100, application processor 1200, PMIC 1300, and external memory 1400. Camera module group 1100 may include multiple camera modules 1100a, 1100b, and 1100c. Figure 18 The diagram illustrates an electronic device comprising three camera modules 1100a, 1100b, and 1100c, but this disclosure is not limited thereto. In some embodiments, the camera module group 1100 may be modified to include only two camera modules. Furthermore, in some embodiments, the camera module group 1100 may be modified to include "n" camera modules (n being a natural number of 4 or greater).

[0157] Below, we will refer to Figure 19 The detailed configuration of camera module 1100b is described in more detail below, but the following description can be applied equally to the remaining camera modules 1100a and 1100c.

[0158] exist Figure 19 In this configuration, camera module 1100b may include a prism 1105, an optical path folding element (OPFE) 1110, an actuator 1130, an image sensing device 1140, and a storage device 1150. The prism 1105 may include a reflector 1107 of light-reflecting material and may alter the path of light "L" incident from the outside.

[0159] In some embodiments, prism 1105 can change the path of light "L" incident along the first direction (X) to a second direction (Y) perpendicular to the first direction (X). Furthermore, prism 1105 can change the path of light "L" incident along the first direction (X) to a second direction (Y) perpendicular to the first (X-axis) direction by rotating the reflector 1107 of the light-reflecting material about the central axis 1106 along direction "A" or by rotating the central axis 1106 along direction "B". In this case, OPFE 1110 can move along a third direction (Z) perpendicular to the first direction (X) and the second direction (Y).

[0160] In some implementations, such as Figure 19 As illustrated, the maximum rotation angle of prism 1105 along direction "A" can be equal to or less than 15 degrees in the positive A direction and greater than 15 degrees in the negative A direction, but this disclosure is not limited thereto.

[0161] In some embodiments, prism 1105 can move within about 20 degrees, between 10 and 20 degrees, or between 15 and 20 degrees in the positive or negative B direction; here, prism 1105 can move at the same angle in the positive or negative B direction, or can move at similar angles within about 1 degree.

[0162] In some embodiments, the prism 1105 can move the reflector 1107 of the light-reflecting material in a third direction (e.g., the Z direction) parallel to the direction in which the central axis 1106 extends.

[0163] For example, OPFE 1110 may include optical lenses consisting of "m" groups (m being a natural number). Here, the "m" lenses can be moved along a second direction (Y) to change the optical zoom ratio of camera module 1100b. For example, when the default optical zoom ratio of camera module 1100b is "Z", by moving the "m" optical lenses included in OPFE 1110, the optical zoom ratio of camera module 1100b can be changed to 3Z, 5Z, or 5Z or greater.

[0164] Actuator 1130 can move OPFE 1110 or optical lens (hereinafter referred to as "optical lens") to a specific position. For example, actuator 1130 can adjust the position of optical lens so that image sensor 1142 is placed at the focal length of optical lens for precise sensing.

[0165] Image sensing device 1140 may include image sensor 1142, control logic 1144, and memory 1146. Image sensor 1142 can sense an image of a target using light "L" provided through an optical lens. Control logic 1144 can control the overall operation of camera module 1100b. For example, control logic 1144 can control the operation of camera module 1100b based on control signals provided through control signal line CSLb.

[0166] The memory 1146 can store information required for the operation of the camera module 1100b, such as calibration data 1147. Calibration data 1147 may include information necessary for the camera module 1100b to generate image data using externally supplied light "L". Calibration data 1147 may include, for example, information about the aforementioned rotation, information about the focal length, information about the optical axis, etc. In the case where the camera module 1100b is implemented as a multi-state camera in which the focal length varies depending on the position of the optical lens, calibration data 1147 may include the focal length value for each position (or state) of the optical lens and information about autofocus.

[0167] Storage device 1150 can store image data sensed by image sensor 1142. Storage device 1150 can be arranged outside image sensing device 1140 and can be implemented in the form of a stack of storage device 1150 and sensor chip constituting image sensing device 1140. In some embodiments, storage device 1150 can be implemented with electrically erasable programmable read-only memory (EEPROM), but this disclosure is not limited thereto.

[0168] In Figure 18 and Figure 19 In some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Thus, depending on the operation of the actuator 1130 therein, the same calibration data 1147 or different calibration data 1147 may be included in the plurality of camera modules 1100a, 1100b, and 1100c.

[0169] In some embodiments, one of the plurality of camera modules 1100a, 1100b and 1100c (e.g. 1100b) may be a folded lens shape of a camera module that includes the aforementioned prism 1105 and OPFE 1110, and the remaining camera modules (e.g. 1100a and 1100c) may be vertical shapes in which the aforementioned prism 1105 and OPFE 1110 are not included; however, this disclosure is not limited thereto.

[0170] In some implementations, one of the multiple camera modules 1100a, 1100b, and 1100c (e.g., 1100c) may be a vertically shaped depth camera, for example, that extracts depth information using infrared (IR). In this case, the application processor 1200 may merge image data provided from the depth camera with image data provided from any other camera module (e.g., 1100a or 1100b) and may generate a three-dimensional (3D) depth image.

[0171] In some embodiments, at least two camera modules (e.g., 1100a and 1100b) of the plurality of camera modules 1100a, 1100b and 1100c may have different fields of view. In this case, at least two camera modules (e.g., 1100a and 1100b) of the plurality of camera modules 1100a, 1100b and 1100c may include different optical lenses, but this disclosure is not limited thereto.

[0172] Furthermore, in some embodiments, the fields of view of the multiple camera modules 1100a, 1100b, and 1100c may be different. In this case, the multiple camera modules 1100a, 1100b, and 1100c may include different optical lenses, and are not limited thereto.

[0173] In some implementations, the multiple camera modules 1100a, 1100b, and 1100c can be arranged to be physically separate from each other. In other words, the multiple camera modules 1100a, 1100b, and 1100c may not use the sensing area of ​​a single image sensor 1142, but rather the multiple camera modules 1100a, 1100b, and 1100c may each include an independent image sensor 1142.

[0174] exist Figure 18 In this embodiment, application processor 1200 may include image processing device 1210, memory controller 1220, and internal memory 1230. Application processor 1200 may be implemented separately from multiple camera modules 1100a, 1100b, and 1100c. For example, application processor 1200 and multiple camera modules 1100a, 1100b, and 1100c may be implemented using separate semiconductor chips.

[0175] Image processing device 1210 may include a plurality of sub-image processors 1212a, 1212b, and 1212c, an image generator 1214, and a camera module controller 1216. Image processing device 1210 may include a plurality of sub-image processors 1212a, 1212b, and 1212c in a number corresponding to the plurality of camera modules 1100a, 1100b, and 1100c.

[0176] Image data generated from camera modules 1100a, 1100b, and 1100c can be provided to corresponding sub-image processors 1212a, 1212b, and 1212c via separate image signal lines ISLa, ISLb, and ISLc, respectively. For example, image data generated from camera module 1100a can be provided to sub-image processor 1212a via image signal line ISLa, image data generated from camera module 1100b can be provided to sub-image processor 1212b via image signal line ISLb, and image data generated from camera module 1100c can be provided to sub-image processor 1212c via image signal line ISLc. For example, this image data transmission can be performed using a camera serial interface (CSI) based on MIPI (Mobile Industrial Processor Interface), but this disclosure is not limited thereto.

[0177] Meanwhile, in some implementations, a sub-image processor can be arranged to correspond to multiple camera modules. For example, sub-image processors 1212a and 1212c can be implemented integratedly, rather than as shown in the example. Figure 16 As shown, they are separated from each other; in this case, one of the multiple image data provided from camera module 1100a and camera module 1100c can be selected by selecting an element (e.g., a multiplexer), and the selected image data can be provided to the integrated sub-image processor.

[0178] Image data provided to sub-image processors 1212a, 1212b, and 1212c can be provided to image generator 1214. Image generator 1214 can generate an output image using the image data provided from sub-image processors 1212a, 1212b, and 1212c, depending on the image generation information or mode signal.

[0179] Specifically, image generator 1214 can generate an output image by merging at least a portion of image data generated from camera modules 1100a, 1100b, and 1100c, which have different fields of view, depending on the image generation information (Generation Information) or the mode signal. Furthermore, image generator 1214 can generate an output image by selecting one of the image data generated from camera modules 1100a, 1100b, and 1100c, which have different fields of view, depending on the image generation information (Generation Information) or the mode signal.

[0180] In some implementations, the image generation information may include a zoom signal or zoom factor. Furthermore, in some implementations, the mode signal may be, for example, a signal based on a mode selected by the user.

[0181] When the image generation information is a zoom signal (or zoom factor) and camera modules 1100a, 1100b, and 1100c have different fields of view, the image generator 1214 may perform different operations depending on the type of zoom signal. For example, when the zoom signal is a first signal, the image generator 1214 may merge image data output from camera module 1100a and image data output from camera module 1100c, and may generate an output image by using the merged image signal and image data output from camera module 1100b that was not used in the merging operation. When the zoom signal is a second signal different from the first signal, without an image data merging operation, the image generator 1214 may select one image data from the image data output from camera modules 1100a, 1100b, and 1100c respectively, and may output the selected image data as the output image. However, this disclosure is not limited thereto, and the way image data is processed can be modified without limitation if necessary.

[0182] In some implementations, the image generator 1214 can generate merged image data with increased dynamic range by receiving multiple image data with different exposure times from at least one of multiple sub-image processors 1212a, 1212b and 1212c and performing high dynamic range (HDR) processing on the multiple image data.

[0183] The camera module controller 1216 can provide control signals to camera modules 1100a, 1100b, and 1100c respectively. The control signals generated from the camera module controller 1216 can be provided to the corresponding camera modules 1100a, 1100b, and 1100c respectively through separate control signal lines CSLa, CSLb, and CSLc.

[0184] Depending on the image generation information, including zoom signals or mode signals, one of the multiple camera modules 1100a, 1100b, and 1100c can be designated as the master camera (e.g., 1100b), while the remaining camera modules (e.g., 1100a and 1100c) can be designated as slave cameras. This designation information can be included in control signals, and these control signals, including the designation information, can be provided to the corresponding camera modules 1100a, 1100b, and 1100c respectively via separate control signal lines CSLa, CSLb, and CSLc.

[0185] The camera module operating as a master or slave module can change depending on the zoom factor or operating mode signal. For example, when the field of view of camera module 1100a is wider than that of camera module 1100b and the zoom factor indicates a low zoom ratio, camera module 1100b can operate as the master module, and camera module 1100a can operate as the slave module. Conversely, when the zoom factor indicates a high zoom ratio, camera module 1100a can operate as the master module, and camera module 1100b can operate as the slave module.

[0186] In some implementations, the control signals provided from the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c may include a synchronization enable signal. For example, when camera module 1100b is used as the main camera and camera modules 1100a and 1100c are used as slave cameras, the camera module controller 1216 may send a synchronization enable signal to camera module 1100b. Camera module 1100b, provided with the synchronization enable signal, may generate a synchronization signal based on the provided synchronization enable signal and may provide the generated synchronization signal to camera modules 1100a and 1100c via the synchronization signal line SSL. Camera modules 1100b and 1100a and 1100c may synchronize with the synchronization signal to send image data to the application processor 1200.

[0187] In some embodiments, the control signals provided from the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c may include mode information based on a mode signal. Based on the mode information, the plurality of camera modules 1100a, 1100b, and 1100c may operate in a first operating mode and a second operating mode relating to the sensing speed.

[0188] In the first operating mode, multiple camera modules 1100a, 1100b, and 1100c can generate image signals at a first speed (e.g., generate image signals at a first frame rate), encode the image signals at a second speed (e.g., encode image signals at a second frame rate higher than the first frame rate), and send the encoded image signals to the application processor 1200. In this case, the second speed can be 30 times or less than the first speed.

[0189] Application processor 1200 can store the received image signal, i.e., the encoded image signal, in the memory 1230 provided therein or in a storage device 1400 external to application processor 1200. Then, application processor 1200 can read the encoded image signal from memory 1230 or storage device 1400 and decode it, and can display image data generated based on the decoded image signal. For example, one of the sub-image processors 1212a, 1212b, and 1212c of image processing device 1210 can perform decoding and image processing on the decoded image signal.

[0190] In the second operating mode, multiple camera modules 1100a, 1100b, and 1100c can generate image signals at a third speed (e.g., an image signal with a third frame rate lower than the first frame rate) and send the image signals to the application processor 1200. The image signals provided to the application processor 1200 can be unencoded signals. The application processor 1200 can perform image processing on the received image signals, or it can store the image signals in the memory 1230 or the storage device 1400.

[0191] PMIC 1300 can supply power, for example, to multiple camera modules 1100a, 1100b, and 1100c respectively. For example, under the control of application processor 1200, PMIC 1300 can supply first power to camera module 1100a via power signal line PSLa, second power to camera module 1100b via power signal line PSLb, and third power to camera module 1100c via power signal line PSLc.

[0192] In response to a power control signal PCON from the application processor 1200, the PMIC 1300 can generate a power corresponding to each of the plurality of camera modules 1100a, 1100b, and 1100c, and can adjust the level of that power. The power control signal PCON can include a power adjustment signal for each operating mode of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the operating mode can include a low-power mode. In this case, the power control signal PCON can include information about the camera module operating in low-power mode and the set power level. The power levels provided to the plurality of camera modules 1100a, 1100b, and 1100c respectively can be the same or different from each other. Moreover, the power level can be changed dynamically.

[0193] According to this disclosure, an image signal processor can estimate the point spread function (PSF) corresponding to an input image. The image signal processor can then perform interpolation on the input image based on the estimated PSF. In this case, the occurrence of false colors in the output image can be mitigated. Therefore, an image signal processor with improved performance, an image system including the image signal processor, and a method of operating the image signal processor can be provided.

[0194] While this disclosure contains numerous specific implementation details, these details should not be construed as limiting the scope of the claims, their equivalents, and the claims described below. Certain features described in the context of individual embodiments in this disclosure may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, in some cases, one or more features from that combination may be removed from that combination, and the combination may refer to a sub-combination or a variation of a sub-combination.

[0195] Although this disclosure has been described with reference to different embodiments, it will be apparent to those skilled in the art that various changes and modifications may be made thereto without departing from the spirit and scope of this disclosure as set forth in the following claims.

Claims

1. An image signal processor configured to receive an input image from an image sensor, the image signal processor comprising: The point spread function (PSF) estimation circuit is configured as follows: The sizes of multiple reference PSFs corresponding to the image sensor are adjusted based on preprocessed image data, where the preprocessed image data is based on the input image; and Generate multiple estimated PSFs indicating the estimation results of multiple PSFs corresponding to the input image; and The de-mosaic circuit is configured as follows: Generate preprocessed image data based on the input image; and Interpolation of the input image is performed based on multiple estimated PSFs.

2. The image signal processor according to claim 1, wherein, The preprocessed image data includes first image data, which includes brightness information of the input image.

3. The image signal processor according to claim 2, in, The PSF estimation circuit includes a defocus calculation circuit. The defocus calculation circuit is configured as follows: Based on the first pixel value from the input image, calculate the first variance of the first pixel among multiple pixels in the input image; Based on the second pixel value from the first image data, calculate the second variance of the first pixel; and Calculate the first variance ratio corresponding to the first pixel based on the first variance and the second variance.

4. The image signal processor according to claim 3, wherein, The defocus calculation circuit is also configured as follows: The brightness normalization of the first pixel value is performed based on the first image data to produce a brightness-normalized first pixel value. as well as The first variance is calculated based on the first pixel value after brightness normalization.

5. The image signal processor according to claim 3, wherein, Several estimated PSFs include: Multiple first estimated PSFs, associated with a first color channel of the input image and corresponding to multiple pixels of the input image respectively; and Multiple second estimated PSFs are associated with the second color channels of the input image and correspond to multiple pixels of the input image, respectively.

6. The image signal processor according to claim 3, wherein, The PSF estimation circuit includes: an estimated PSF generation circuit configured to generate a first estimated PSF corresponding to a first pixel by adjusting the size of a first reference PSF among a plurality of reference PSFs based on a first variance ratio, wherein the first reference PSF corresponds to the first pixel.

7. The image signal processor according to claim 1, wherein, The image signal processor includes a memory configured to store reference PSF data associated with a plurality of reference PSFs.

8. The image signal processor according to claim 7, wherein, The memory includes one-time programmable (OTP) memory.

9. The image signal processor according to claim 7, in, The PSF estimation circuit includes a reference PSF extraction circuit. The reference PSF extraction circuit is configured to obtain multiple reference PSFs corresponding to the multiple pixels of the input image, based on the reference PSF data and the position information indicating the position of each pixel in the multiple pixels of the input image.

10. The image signal processor according to claim 2, wherein, The first image data is based on the green pixel values ​​of the input image.

11. The image signal processor according to claim 1, wherein, The PSF estimation circuit is configured to generate multiple PSFs corresponding to the input image based on two or more of the following: the degree of defocus of the lens corresponding to the image sensor, the material of the lens, and the tilt of the lens.

12. The image signal processor according to claim 1, wherein, The de-mosaic circuit includes: An interpolated image generation circuit is configured to generate an interpolated image based on multiple estimated PSFs; and The demosaic image generation circuit is configured to generate a demosaic image based on an interpolated image.

13. The image signal processor according to claim 12, wherein, The interpolation image generation circuit is configured to use multiple estimated PSFs to mitigate false color phenomena.

14. An operation method for an image signal processor, the operation method comprising: Receive input images from the image sensor; Generate preprocessed image data based on the input image; Based on preprocessed image data, multiple estimated PSFs are generated by adjusting the magnitudes of multiple reference point spread functions (PSFs) corresponding to the image sensor; and Based on multiple estimated PSFs, an interpolated image is generated by performing interpolation on the input image. Among them, multiple estimated PSFs indicate the estimation results for multiple PSFs corresponding to pixels of the input image, respectively.

15. The operating method according to claim 14, wherein, The generation of preprocessed image data includes: Generate first image data based on the first pixel value of the input image; Generate second image data based on the second pixel value of the input image; and Generate third image data based on the third pixel value of the input image.

16. The operating method according to claim 15, wherein, Generating multiple estimated PSFs includes: Perform brightness normalization of the first image data based on the second image data; Calculate the first variance corresponding to the first pixel of the first image data; Calculate the second variance corresponding to the first pixel of the second image data; The first variance ratio is calculated based on the first variance and the second variance; and Based on the first variance ratio, a first estimated PSF is generated by adjusting the size of the first reference PSF corresponding to the first pixel of the input image among multiple reference PSFs.

17. The operating method according to claim 16, wherein, Generating the interpolated image includes: when the first variance is less than the second variance, increasing the size of the first PSF corresponding to the first pixel of the second image data in a plurality of PSFs based on the first estimated PSF.

18. The operating method according to claim 15, wherein, The first pixel value includes red pixel values, the second pixel value includes green pixel values, and the third pixel value includes blue pixel values.

19. An image system, comprising: A first image sensor is configured to output a first input image; A second image sensor is configured to output a second input image; as well as Image signal processor, The image signal processor includes: The storage device is configured to store first reference point spread function (PSF) data and second reference PSF data, the first reference PSF data being associated with a first reference PSF corresponding to a first image sensor, and the second reference PSF data being associated with a second reference PSF corresponding to a second image sensor; The PSF estimation circuit is configured as follows: The size of the first reference PSF is adjusted to generate multiple first estimated PSFs, the multiple first estimated PSFs indicating the estimation results for the PSF corresponding to the first input image; and The size of the second reference PSF is adjusted to generate multiple second estimated PSFs, which indicate the estimation results for the PSF corresponding to the second input image; and The de-mosaic circuit is configured as follows: Interpolation of the first input image is performed based on multiple first estimated PSFs; and Interpolation of the second input image is performed based on multiple second estimated PSFs.

20. The imaging system according to claim 19, wherein, The PSF estimation circuit is configured as follows: Based on the first reference PSF data, first reference PSFs corresponding to multiple pixels of the first input image are obtained; and Based on the second reference PSF data, a second reference PSF corresponding to multiple pixels of the second input image is obtained.

Images