Electronic device including image sensor

Abstract

An electronic device includes: a display; a first image sensor configured to output a first image signal based on sensing first light that passes through the display; a second image sensor configured to output a second image signal based on sensing second light that does not pass through the display; and a processor configured to: generate a first optical value and a second optical value based on the second image signal, the second optical value being different from the first optical value; and correct the first image signal by using the second optical value based on the first optical value satisfying a first condition.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and all benefits of Korean Patent Application No. 10-2020-0097880 filed in the Korean Intellectual Property Office on August 5, 2020, the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] The present disclosure relates to an electronic device including an image sensor, and more particularly, to an electronic device including a plurality of image sensors and an image signal processor. Technical Field

[0004] An image sensing device is a semiconductor device that converts optical information into an electrical signal. Examples of image sensing devices may include a charge coupled device (CCD) image sensing device and a complementary metal oxide semiconductor (CMOS) image sensing device.

[0005] A CMOS image sensor (CIS) may include a plurality of pixels arranged two-dimensionally. Each of the plurality of pixels may include, for example, a photodiode (PD). The photodiode may be used to convert incident light into an electrical signal.

[0006] In recent years, with the development of the computer industry and the telecommunications industry, the demand for image sensors with improved performance has increased in various fields such as digital cameras, camcorders, smartphones, gaming devices, security cameras, medical miniature cameras, robots, autonomous vehicles, drones, etc.

[0007] Recently, an image sensor disposed below the display panel of an electronic device has been provided. When the image sensor senses light passing through the display panel, information regarding the sensed light may differ from information regarding the actual light. For example, the quality of an image output by an image sensor sensing light passing through the display panel may be degraded compared to the quality of an image output by an image sensor sensing light that did not pass through the display panel. Therefore, a method for addressing this issue is needed in image sensors disposed below the display panel of electronic devices. Summary of the Invention

[0008] One or more example embodiments of the present disclosure provide an electronic device including an image sensor and a processor, in which product reliability is improved by reducing degradation of image quality.

[0009] One or more example embodiments of the present disclosure also provide an electronic device including an image sensor and a processor, wherein product reliability is improved by enhancing image quality.

[0010] However, the aspects of the present disclosure are not limited to the contents set forth herein. The above and other aspects of the present disclosure will become more apparent to those skilled in the art to which the present disclosure pertains by referring to the detailed description of the present disclosure given below.

[0011] According to an aspect of an example embodiment, an electronic device is provided, including: a display; a first image sensor, the first image sensor being configured to output a first image signal based on sensing first light passing through the display; a second image sensor, the second image sensor being configured to output a second image signal based on sensing second light that does not pass through the display; a processor, the processor being configured to: generate a first optical value and a second optical value based on the second image signal, the second optical value being different from the first optical value; and correct the first image signal by using the second optical value based on the first optical value satisfying a first condition.

[0012] The first optical value may include at least one of a brightness value or an illuminance value generated based on the second image signal, and the second optical value may include a color temperature value generated based on the second image signal.

[0013] The processor may be further configured to correct the first image signal by using the color temperature value based on the first optical value satisfying the first condition and further based on the color temperature value being less than the first color temperature or being equal to or greater than a second color temperature, the second color temperature being higher than the first color temperature.

[0014] The processor may be further configured to generate a third optical value and a fourth optical value based on the first image signal, the fourth optical value being different from the third optical value.

[0015] The third optical value may include at least one of a brightness value or an illuminance value generated based on the first image signal, and the fourth optical value may include a color temperature value generated based on the first image signal.

[0016] The first condition may be that the first optical value is equal to or greater than a threshold value.

[0017] The processor may be further configured to correct the first image signal by using an optical value generated based on the first image signal, based on the first optical value not satisfying the first condition.

[0018] The optical value generated based on the first image signal may include a color temperature value generated based on the first image signal.

[0019] The first condition may be that a difference between the first optical value and the third optical value is smaller than a threshold value.

[0020] The processor may be further configured to correct the first image signal by using an optical value generated based on the first image signal, based on a difference between the first optical value and the third optical value being equal to or greater than a threshold value.

[0021] The display may be further configured to output an image generated based on the corrected first image signal.

[0022] The processor may be further configured to output a third image signal by performing automatic white balance on the first image signal, and the processor may be further configured to output a fourth image signal obtained by correcting the third image signal using the second optical value based on the first optical value satisfying the second condition.

[0023] The display may be further configured to output an image generated based on at least one of the third image signal and the fourth image signal.

[0024] According to an aspect of an example embodiment, an electronic device is provided, comprising: a first image sensor configured to output a first image signal based on sensing first light incident on a front surface of the electronic device; a second image sensor configured to output a second image signal based on sensing second light incident on a rear surface of the electronic device; a processor configured to receive the first image signal and the second image signal; and a display disposed on the front surface and configured to output an image generated based on the first image signal, wherein the processor is further configured to generate a first color temperature value based on the second image signal, and based on a first condition being satisfied, correct the first image signal by using the first color temperature value.

[0025] The display may further be configured to cover the first image sensor.

[0026] The processor may be further configured to generate a second color temperature value based on the first image signal, and generate at least one of a brightness value or a luminance value based on the second image signal.

[0027] The first condition may be that at least one of the brightness value or the illuminance value is equal to or greater than a threshold value.

[0028] The processor may be further configured to correct the first image signal by using a second color temperature value based on the first condition not being satisfied.

[0029] The display may be further configured to output an image generated based on the corrected first image signal.

[0030] According to one aspect of an example embodiment, an electronic device is provided, including: a display; a first camera module, the first camera module including a first image sensor, the first image sensor being configured to output a first image signal based on sensing first light passing through the display; a second camera module, the second camera module including a second image sensor, the second image sensor being configured to output a second image signal based on sensing second light that does not pass through the display; and an application processor, the application processor being provided independently of the first camera module and the second camera module and including an image signal processor, wherein the image signal processor is configured to: receive the first image signal from the first camera module through a first camera serial interface; receive the second image signal from the second camera module through a second camera serial interface; generate a first color temperature value based on the first image signal; generate at least one of a luminance value or an illuminance value and a second color temperature value based on the second image signal; correct the first image signal based on the second color temperature value based on at least one of the luminance value or the illuminance value being equal to or greater than a threshold value; and correct the first image signal based on the first color temperature value based on at least one of the luminance value or the illuminance value being less than the threshold value. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other aspects, features and advantages of certain example embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0032] Figure 1 is a perspective view illustrating an electronic device according to some example embodiments;

[0033] Figure 2 It is along Figure 1 A partial cross-sectional view of the electronic device taken along line AA;

[0034] Figure 3 is the viewing angle along the first direction, Figure 1 a perspective view of the electronic device shown;

[0035] Figure 4 It is along Figure 3 A partial cross-sectional view of the electronic device taken along line BB;

[0036] Figure 5 is a block diagram illustrating an image sensing system according to some example embodiments;

[0037] Figure 6 is used to describe Figure 5 Illustration of the conceptual layout of the image sensor in;

[0038] Figure 7 It shows Figure 5 a block diagram of a first image signal processor and a second image signal processor in FIG.

[0039] Figure 8 It shows Figure 7 The block diagram of the post-processing circuit in FIG.

[0040] Figure 9 It shows Figure 8 The block diagram of the post-processing circuit in FIG.

[0041] Figure 10 is a flowchart illustrating an image compensation method by a post-processing circuit according to some example embodiments;

[0042] Figure 11 It shows Figure 8 The block diagram of the post-processing circuit in FIG.

[0043] Figure 12 is a diagram illustrating image compensation conditions according to some example embodiments;

[0044] Figure 13 is a diagram describing the operation of an electronic device according to some example embodiments;

[0045] Figure 14 is a block diagram illustrating an image sensing system according to some other example embodiments;

[0046] Figure 15 is a block diagram for describing an electronic device including a multi-camera module according to some example embodiments; and

[0047] Figure 16 yes Figure 15 Detailed block diagram of the camera module in Figure 1. DETAILED DESCRIPTION

[0048] Hereinafter, example embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0049] The following will refer to Figures 1 to 4 An electronic device including a first image sensor and a second image sensor is described.

[0050] Figure 1 is a perspective view illustrating an electronic device according to some example embodiments. Figure 2 It is along Figure 1 A partial cross-sectional view of the electronic device taken along line AA. Figure 3 is the viewing angle along the first direction, Figure 1 A perspective view of the electronic device shown. Figure 4 It is along Figure 3 A partial cross-sectional view of the electronic device taken along line BB.

[0051] refer to Figures 1 to 4, the electronic device 1 may include a cover glass 10 , a display 20 , a rear glass 30 , a rear cover 40 , a first image sensor 100 , a second image sensor 200 , and the like.

[0052] In some example embodiments, electronic device 1 may include a housing. For example, the housing may include cover glass 10 facing a first direction D1 and rear glass 30 facing a fourth direction D4 opposite to first direction D1. Furthermore, the housing may include, for example, a connection portion connecting cover glass 10 to rear glass 30. The housing may be used to protect components within electronic device 1 from external impact.

[0053] The cover glass 10 may include a transparent material so that the electrical information signal displayed by the display 20 can be recognized from the outside. For example, the cover glass 10 may include glass or plastic.

[0054] The cover glass 10 and the back glass 30 may have a flat shape. For example, each of the cover glass 10 and the back glass 30 may have a length in the second direction D2 and a length in the third direction D3, where the length in the third direction is greater than the length or thickness in the first direction D1. Thus, the electronic device 1 may have a flat shape. However, exemplary embodiments according to the technical spirit of the present disclosure are not limited thereto.

[0055] The surface of the electronic device 1 facing the first direction D1 may be the front surface of the electronic device 1, and the surface of the electronic device 1 facing the fourth direction D4 may be the rear surface of the electronic device 1. However, this is merely an example and does not limit the exemplary embodiments according to the technical spirit of the present disclosure.

[0056] refer to Figure 1 and Figure 2 , the cover glass 10 may cover the display 20. For example, the display 20 may be arranged along the fourth direction D4 from the cover glass 10.

[0057] The display 20 may include a plurality of pixels arranged in rows and columns. For example, the display 20 may include an organic light emitting diode display (OLED), a liquid crystal display (LCD), a plasma display panel (PDP), an electrochromic display (ECD), a digital mirror device (DMD), a driven mirror device (AMD), a grating value (GLV) device, an electroluminescent display (ELD), etc.

[0058] The display 20 may output an image provided by the electronic device 1. For example, signals are sent to a plurality of pixels, and the display 20 may output a corresponding image based on these signals. For example, the display 20 may include a touch screen or a touch panel that displays an image and / or receives a user's touch input.

[0059] The first image sensor 100 may sense a portion of light incident from outside the electronic device 1 through the display 20 and / or light output from the display 20. For example, the first image sensor 100 may be covered by the display 20 and may sense light incident from outside the electronic device 1 in the electronic device 1. For example, the first image sensor 100 may sense light output from the display 20 and reflected by the cover glass 10.

[0060] For example, the first image sensor 100 may sense light from the cover glass 10 and pixels of the display 20 , as well as light passing through gaps between pixels of the display 20 .

[0061] Accordingly, the brightness and color temperature of light passing through the display 20 may be changed by the cover glass 10 and the pixels of the display 20. For example, the brightness and color temperature generated as a result of processing a signal output by the first image sensor 100 sensing light passing through the display 20 may be smaller than the actual brightness and color temperature. However, the embodiments of the present disclosure are not limited thereto.

[0062] In addition, according to some example embodiments, the brightness value may be used together with the luminance value, or the luminance value may be used instead of the brightness value. Hereinafter, in an example embodiment according to the present disclosure, the use of the brightness value is described, but the luminance value may also be used in the same manner as the brightness value.

[0063] The first image sensor 100 may be covered by the display 20 and surrounded by the back cover 40. For example, the first image sensor 100 may be disposed in an opening formed in the back cover 40.

[0064] The back cover 40 can prevent light generated inside the electronic device 1 from affecting the display 20. In addition, the back cover 40 can prevent light output from the display 20 from entering the interior of the electronic device 1.

[0065] Despite Figure 2 Not shown in the figure, the back glass 30 may be disposed on the back cover 40 and the first image sensor 100 in the fourth direction D4 and protect the back cover 40 and the first image sensor 100 from external impact.

[0066] refer to Figure 3 and Figure 4 , the rear glass 30 may be provided on the rear surface of the electronic device 1 (e.g., the surface of the electronic device 1 in the fourth direction D4). Furthermore, the second image sensor 200 may be provided on the rear surface of the electronic device 1 (e.g., the surface of the electronic device 1 in the fourth direction D4). However, the embodiments of the present disclosure are not limited thereto, and the second image sensor 200 may be provided at any other location in the electronic device 1, such as on the side or front surface of the electronic device 1.

[0067] The rear glass 30 may be disposed to surround the second image sensor 200. For example, the second image sensor 200 may be disposed in an opening formed in the rear glass 30. A surface of the second image sensor 200 may face the fourth direction D4. For example, the second image sensor 200 may have a light receiver facing the fourth direction D4 (e.g., a direction toward the rear surface).

[0068] The second image sensor 200 can sense light incident on the rear surface of the electronic device 1. For example, the second image sensor 200 can sense light that does not pass through the display 20 provided on the front surface of the electronic device 1. That is, although the first image sensor 100 senses light that passes through the display 20, the light receivers of the second image sensor 200 can sense light that does not pass through the display 20 because the light receivers of the second image sensor 200 facing the fourth direction D4 (e.g., the direction toward the rear surface) are not covered by the display 20 or another component of the electronic device 1.

[0069] Although one second image sensor 200 is shown, the second image sensor 200 may include a plurality of image sensors. For example, the second image sensor 200 may include a normal camera, a wide-angle camera, and a telephoto camera.

[0070] The rear glass 30 may prevent light from entering the electronic device 1 from the outside. That is, light may be prevented from entering a portion other than the opening of the rear glass 30 where the second image sensor 200 is provided.

[0071] Despite Figure 3 Although not shown in the drawing, the cover glass 10 , the display 20 , and the back cover 40 may be disposed on the back glass 30 in the first direction D1 .

[0072] The following will refer to Figures 5 to 9 An image sensing system 2 including the first image sensor 100 , the second image sensor 200 , and the application processor 300 will be described.

[0073] Figure 5 is a block diagram illustrating an image sensing system according to some example embodiments.

[0074] refer to Figure 5 , the image sensing system 2 may include a first image sensor 100 , a second image sensor 200 , and an application processor 300 .

[0075] The first image sensor 100 may generate a first image signal S1 by sensing an image of a sensing target using incident light. For example, the first image sensor 100 may sense light passing through the display 20 and generate a first image signal S1.

[0076] The second image sensor 200 may generate the second image signal S2 by sensing an image of a sensing target using incident light. For example, the second image sensor 200 may generate the second image signal S2 by sensing light that does not pass through the display 20.

[0077] The first image signal S1 and the second image signal S2 may be provided to and processed by the application processor (AP) 300. For example, the first image signal S1 may be provided to and processed by the first image signal processor 310 of the AP 300, and the second image signal S2 may be provided to and processed by the second image signal processor 330 of the AP 300. However, the embodiments of the present disclosure are not limited thereto. For example, the first image signal processor 310 and the second image signal processor 330 may be implemented as one image signal processor.

[0078] The first image signal processor 310 may receive the first image signal S1 output from the buffer 170 of the first image sensor 100 and process the received first image signal S1 to make it suitable for display.

[0079] The second image signal processor 330 may receive the second image signal S2 output from the buffer 270 of the second image sensor 200 and process the received second image signal S2 to make it suitable for display.

[0080] In some example embodiments, the first image signal processor 310 and the second image signal processor 330 may perform digital merging on the first image signal S1 and the second image signal S2, respectively. The first image signal S1 and the second image signal S2 may be original image signals from the pixel arrays 140 and 240, respectively, on which analog merging has not been performed, or may be the first image signal S1 and the second image signal S2, respectively, on which analog merging has been performed.

[0081] In some example embodiments, the first image sensor 100, the second image sensor 200, and the application processor 300 may be provided separately from each other. For example, the first image sensor 100 and the second image sensor 200 may be mounted on a first chip, the application processor 300 may be mounted on a second chip, and the first image sensor 100 and the second image sensor 200 may communicate with the application processor 300 via an interface. However, example embodiments of the present disclosure are not limited thereto. For example, the first image sensor 100, the second image sensor 200, and the application processor 300 may be implemented in a single package, such as a multi-chip package (MCP).

[0082] The first image sensor 100 may include a control register block 110 , a timing generator 120 , a row driver 130 , a pixel array 140 , a readout circuit 150 , a ramp signal generator 160 , and a buffer 170 .

[0083] The control register block 110 may control the overall operation of the image sensor 100. Specifically, the control register block 110 may directly transmit an operating signal to the timing generator 120, the ramp signal generator 160, and the buffer 170.

[0084] The timing generator 120 may generate a reference signal for operation timing of each component of the image sensor 100. The reference signal for operation timing generated by the timing generator 120 may be transmitted to the row driver 130, the readout circuit 150, the ramp signal generator 160, and the like.

[0085] The ramp signal generator 160 may generate and transmit a ramp signal used in the readout circuit 150. For example, the readout circuit 150 may include a correlated double sampler (CDS), a comparator, etc., and the ramp signal generator 160 may generate and transmit a ramp signal used in the correlated double sampler (CDS), the comparator, etc.

[0086] The buffer 170 may include, for example, a latch unit, and the buffer 170 may temporarily store the first image signal S1 to be provided to the outside, and transmit the first image signal S1 to an external memory or an external device.

[0087] The pixel array 140 may sense an external image. The pixel array 140 may include a plurality of pixels (or unit pixels). The row driver 130 may selectively activate rows of the pixel array 140. The pixel array 140 may collect light incident through the display 20.

[0088] The readout circuit 150 may sample the pixel signal provided from the pixel array 140 , compare the sampled pixel signal with the ramp signal, and convert the analog image signal (data) into a digital image signal (data) based on the comparison result.

[0089] Figure 6 is used to describe Figure 5 Illustration of the conceptual layout of an image sensor.

[0090] refer to Figure 6 The first image sensor 100 may include a first region R1 and a second region R2 stacked in a first direction (e.g., a vertical direction). As shown in the figure, the first region R1 and the second region R2 may extend in a second direction (e.g., a horizontal direction) intersecting the first direction and a third direction intersecting both the first and second directions. Figure 5The blocks shown in FIG. 1 may be disposed in the first region R1 and the second region R2 .

[0091] Although not shown in the figure, the third region in which the memory is arranged may be provided above or below the second region R2. In this case, the memory provided in the third region may receive image data from the first region R1 and the second region R2, store or process the image data, and resend the image data to the first region R1 and the second region R2. In this case, the memory may include storage elements such as dynamic random access memory (DRAM), static random access memory (SRAM), spin transfer torque magnetic random access memory (STT-MRAM), and flash memory. When the memory includes, for example, DRAM, the memory can receive and process image data at a relatively high speed. In addition, in some example embodiments, the memory may be provided in the second region R2.

[0092] The first region R1 may include a pixel array region PA and a first peripheral region PH1, and the second region R2 may include a logic circuit region LC and a second peripheral region PH2. The first region R1 and the second region R2 may be sequentially stacked vertically.

[0093] In the first region R1, the pixel array region PA may be a region in which a pixel array (eg, Figure 5 The pixel array area PA may include a plurality of unit pixels arranged in a matrix. Each pixel may include a photodiode and a transistor.

[0094] The first peripheral area PH1 may include a plurality of pads and may be disposed around the pixel array area PA. The plurality of pads may transmit and / or receive electrical signals to and / or from an external device.

[0095] In the second region R2, the logic circuit region LC may include electronic components including a plurality of transistors. The electronic components included in the logic circuit region LC may be electrically connected to the pixel array region PA to provide signals to each unit pixel PX of the pixel array region PA or control the output signal of each unit pixel PX.

[0096] For example, reference Figure 5 The control register block 110, timing generator 120, row driver 130, readout circuit 150, ramp signal generator 160 and buffer 170 are arranged in the logic circuit region LC. Figure 5 Among the illustrated blocks, blocks other than the pixel array 140 may be disposed in the logic circuit region LC.

[0097] In addition, in the second region R2, the second peripheral region PH2 may be provided in a region corresponding to the first peripheral region PH1 of the first region R1, but the embodiment is not limited thereto. The second peripheral region PH2 may include a plurality of pads that may transmit and / or receive electrical signals to and / or from external devices.

[0098] The first image sensor 100 has been described above, but the same or similar description may also be applied to the second image sensor 200. For example, the configuration and operation of the control register block 110, the timing generator 120, the row driver 130, the pixel array 140, the readout circuit 150, the ramp signal generator 160, and the buffer 170 of the first image sensor 100 may be substantially the same as or similar to the configuration and operation of the control register block 210, the timing generator 220, the row driver 230, the pixel array 240, the readout circuit 250, the ramp signal generator 260, and the buffer 270 of the second image sensor 200.

[0099] In addition, one or more of the control register block 210 , the timing generator 220 , the row driver 230 , the readout circuit 250 , the ramp signal generator 260 , the buffer 270 , and the like may be provided in the logic circuit region LC of the second image sensor 200 .

[0100] However, the difference between the first image sensor 100 and the second image sensor 200 is that the first image sensor 100 outputs a first image signal S1, and the second image sensor 200 outputs a second image signal S2. In addition, the pixel array 240 is different from the pixel array 140 in that the pixel array 240 can collect light that does not pass through the display 20.

[0101] Figure 7 It shows Figure 5 Block diagram of the first image signal processor and the second image signal processor in FIG.

[0102] refer to Figure 7 , the first image signal processor 310 may receive the first image signal S1 from the first image sensor 100 , and the second image signal processor 330 may receive the second image signal S2 from the second image sensor 200 .

[0103] The first image signal processor 310 may include a pre-processing circuit 311, an image memory 317, a post-processing circuit 321, a conversion circuit 327, a data compression circuit 328, and a memory 329. The second image signal processor 330 may include a pre-processing circuit 331, an image memory 337, a post-processing circuit 341, a conversion circuit 347, a data compression circuit 348, and a memory 349.

[0104] In some example embodiments, the preprocessing circuit 311 may perform preprocessing on the first image signal S1. The preprocessing circuit 311 may include a black level correction circuit 312, a defective pixel correction circuit 313, a shading correction circuit 314, an automatic exposure (AE) evaluation value calculator 315, and an automatic white balance (AWB) evaluation value calculator 316.

[0105] The first image signal S1 output from the first image sensor 100 can be corrected so that the black level becomes constant through black level correction processing performed by the black level correction circuit 312. When a pixel defect exists in the first image signal S1, the defective pixel correction circuit 313 can interpolate information about the first image signal S1 based on information surrounding the defective pixel. Furthermore, in the first image signal S1, shading correction circuit 314 can correct brightness differences between pixels caused by brightness omissions around the pixel.

[0106] The AE evaluation value calculator 315 may calculate the AE evaluation value AE1 based on the first conversion signal S1a that has been corrected by the black level correction circuit 312, the defective pixel correction circuit 313, and the shading correction circuit 314. For example, the AE evaluation value calculator 315 may calculate the AE evaluation value AE1 that represents brightness obtained by integrating brightness values sensed by the first image sensor 100. For example, the AE evaluation value AE1 may include the brightness value sensed by the first image sensor 100.

[0107] The AWB evaluation value calculator 316 may calculate an AWB evaluation value AWB1 using a specific algorithm based on the AE evaluation value AE1 corrected by the black level correction circuit 312, the defective pixel correction circuit 313, and the shading correction circuit 314, and the first conversion signal S1a. For example, the AWB evaluation value AWB1 may include a white balance gain to be used in the white balance compensation process. For example, the AWB evaluation value AWB1 may include a color temperature value sensed by the first image sensor 100.

[0108] In this case, the color temperature value means expressing the color of the sensed light as a temperature. For example, in the case of red light, the color temperature value may be about 2,000K, and in the case of blue light, the color temperature value may be about 10,000K, but the embodiments of the present disclosure are not limited thereto.

[0109] The first conversion signal S1a that has been corrected by the black level correction circuit 312, the defective pixel correction circuit 313, and the shading correction circuit 314; the AE evaluation value AE1 output from the AE evaluation value calculator 315; and the AWB evaluation value AWB1 output from the AWB evaluation value calculator 316 may be temporarily stored in the image memory 317 and transmitted to the post-processing circuit 321.

[0110] However, embodiments of the present disclosure are not limited thereto, and the first conversion signal S1 a , the AE evaluation value AE1 , and the AWB evaluation value AWB1 may be transmitted to the post-processing circuit 321 without being stored in the image memory 317 .

[0111] The post-processing circuit 321 may perform post-processing on the first conversion signal S1a that has been pre-processed by the pre-processing circuit 311. The post-processing circuit 321 may include a demosaicing processor 322, an edge emphasis processor 323, a gamma compensation processor 324, a white balance compensation processor 325, a color compensation processor 326, and the like.

[0112] The demosaicing processor 322 may perform a demosaicing process (e.g., a Bayer color interpolation process) on the first conversion signal S1a transmitted from the pre-processing circuit 311. The edge emphasis processor 323 may perform an edge emphasis process on the first conversion signal S1a transmitted from the pre-processing circuit 311. The gamma compensation processor 324 may perform gamma compensation on the first conversion signal S1a transmitted from the pre-processing circuit 311.

[0113] The white balance compensation processor 325 may receive the first conversion signal S1a and the AWB evaluation value AWB1 from the pre-processing circuit 311. The white balance compensation processor 325 may perform white balance compensation processing on the first conversion signal S1a using a white balance gain of the AWB evaluation value AWB1.

[0114] The color compensation processor 326 may receive the first conversion signal S1a and the AWB evaluation value AWB1 from the pre-processing circuit 311. The color compensation processor 326 may perform color compensation processing on the first conversion signal S1a using the color temperature value sensed by the first image sensor 100 and included in the AWB evaluation value AWB1. Furthermore, the color compensation processor 326 may perform color compensation processing on the signal that has undergone white balance compensation processing by using the color temperature value sensed by the first image sensor 100 and included in the AWB evaluation value AWB1.

[0115] However, embodiments of the present disclosure are not limited thereto, and the color compensation processor 326 may perform color compensation by using other methods.

[0116] The first conversion signal S1a that has been post-processed by the post-processing circuit 321 can be sent to the conversion circuit 327. The conversion circuit 327 can convert the RGB image signal into a YCbCr (YCC) image signal. Through the conversion process, the color space of the captured image can be converted from the RGB color space to the YCC (YCrCb) color space.

[0117] The YCC image signal converted by the conversion circuit 327 may be sent to the data compression circuit 328. The data compression circuit 328 may compress the YCC image signal using a compression format such as Joint Photographic Experts Group (JPEG). The compressed YCC image signal may be stored in the memory 329. Alternatively, the compressed YCC image signal may be processed by the application processor 300, and the processed image may be output through the display 20.

[0118] Although the first image signal processor 310 and the components included therein have been described as an example, the same or similar description may also be applied to the second image signal processor 330. That is, the configuration and operation of the pre-processing circuit 311, image memory 317, post-processing circuit 321, conversion circuit 327, data compression circuit 328, and memory 329 of the first image signal processor 310 may be the same as or substantially similar to the configuration and operation of the pre-processing circuit 331, image memory 337, post-processing circuit 341, conversion circuit 347, data compression circuit 348, and memory 349 of the second image signal processor 330.

[0119] However, the difference between first image signal processor 310 and second image signal processor 330 is that first image signal processor 310 processes first image signal S1 to generate data, while second image signal processor 330 processes second image signal S2 to generate data. That is, first image signal processor 310 may process first image signal S1 output by sensing incident light passing through display 20, while second image signal processor 330 may process second image signal S2 output by sensing incident light not passing through display 20.

[0120] For example, the pre-processing circuit 331 of the second image signal processor 330 may receive the second image signal S2.

[0121] The AE evaluation value calculator 335 of the pre-processing circuit 331 may calculate an AE evaluation value AE2 based on the signal that has been corrected by the black level correction circuit 332, the defective pixel correction circuit 333, and the shading correction circuit 334. For example, the AE evaluation value calculator 335 may calculate the AE evaluation value AE2 representing brightness obtained by integrating brightness values sensed by the second image sensor 200. For example, the AE evaluation value AE2 may include the brightness value sensed by the second image sensor 200.

[0122] The AE evaluation value calculator 335 may transmit the calculated AE evaluation value AE2 to the post-processing circuit 321 of the first image signal processor 310 .

[0123] The AWB evaluation value calculator 336 may calculate the AWB evaluation value AWB2 using a specific algorithm based on the AE evaluation value AE2 and the signal corrected by the black level correction circuit 332, the defective pixel correction circuit 333, and the shading correction circuit 334. For example, the AWB evaluation value AWB2 may include a white balance gain to be used in the white balance compensation process. For example, the AWB evaluation value AWB2 may include a color temperature value sensed by the second image sensor 200.

[0124] The AWB evaluation value calculator 336 may transmit the calculated AWB evaluation value AWB2 to the post-processing circuit 321 of the first image signal processor 310 .

[0125] In the drawings, the first image signal processor 310 and the second image signal processor 330 are shown as being separated from each other, but this is for descriptive purposes only and the present disclosure is not limited thereto. For example, the AE evaluation values AE1 and AE2 and the AWB evaluation values AWB1 and AWB2 may be generated and shared by one processor (e.g., the application processor 300).

[0126] The following will refer to Figure 8 To describe in more detail Figure 7 The post-processing circuit 321 in.

[0127] Figure 8 It shows Figure 7 Block diagram of the post-processing circuit in .

[0128] In some embodiments, the post-processing circuit 321 may include a white balance compensation processor 325 and a color compensation processor 326 .

[0129] The first conversion signal S1a transmitted to the post-processing circuit 321 may be a signal transmitted from the pre-processing circuit 311 or a signal obtained by compensating the signal transmitted from the pre-processing circuit 311 via the demosaicing processor 322 , the edge emphasis processor 323 , and the gamma compensation processor 324 .

[0130] The white balance compensation processor 325 may generate a second conversion signal S1b by performing a white balance compensation process on the first conversion signal S1a using a white balance gain of the AWB evaluation value AWB1 and may transmit the generated second conversion signal S1b to the first determination unit 318 .

[0131] The color compensation processor 326 may receive the second conversion signal S1b from the white balance compensation processor 325. However, the present disclosure is not limited thereto, and the color compensation processor 326 may receive the first conversion signal S1a that is not white balance compensated, instead of the second conversion signal S1b.

[0132] The color compensation processor 326 may receive the AE evaluation value AE2 and the AWB evaluation value AWB2 from the second image signal processor 330. For example, the color compensation processor 326 may receive the AE evaluation value AE2 and the AWB evaluation value AWB2 of the signal sensed and output by the second image sensor 200 from the preprocessing circuit 331.

[0133] The color compensation processor 326 may generate a third conversion signal S1c by performing color compensation processing based on at least one of the received AE evaluation values AE1 and AE2, the AWB evaluation values AWB1 and AWB2, and the second conversion signal S1b, and may transmit the generated third conversion signal S1c to the first determination unit 318.

[0134] First determination unit 318 may select and output at least one of the received second conversion signal S1b and third conversion signal S1c. For example, fourth conversion signal S1d may include at least one of second conversion signal S1b and third conversion signal S1c. Fourth conversion signal S1d may be transmitted to other components of application processor 300. For example, image data processed using fourth conversion signal S1d may be output to display 20.

[0135] The following will refer to Figures 9 to 12 To describe in more detail Figure 8 The post-processing circuit 321 in.

[0136] Figure 9 It shows Figure 8 Block diagram of the post-processing circuit in . Figure 10 is a flowchart illustrating an image compensation method implemented by a post-processing circuit according to some example embodiments. Figure 11 It shows Figure 8 Block diagram of the post-processing circuit in . Figure 12 is a diagram illustrating image compensation conditions according to some example embodiments.

[0137] refer to Figure 9 The post-processing circuit 321 may further include a second determination unit 319. For example, the AWB evaluation values AWB1 and AWB2 may be sent to the second determination unit 319 before being sent to the color compensation processor 326. The AE evaluation value AE2 may also be sent to the second determination unit 319.

[0138] In this case, the AE evaluation value AE1 may include a luminance value of light sensed by the first image sensor 100 after passing through the display 20 , and the AE evaluation value AE2 may include a luminance value of light sensed by the second image sensor 200 without passing through the display 20 .

[0139] Furthermore, the AWB evaluation value AWB1 may include a color temperature value of light sensed by the first image sensor 100 after passing through the display 20 , and the AWB evaluation value AWB2 may include a color temperature value of light sensed by the second image sensor 200 without passing through the display 20 .

[0140] refer to Figure 9 and Figure 10 , the post-processing circuit 321 may receive signals from the first image sensor 100 and the second image sensor 200 (step S350). For example, the second determination unit 319 may receive the AE evaluation value AE2 and the AWB evaluation value AWB2 generated as a result of processing the signal sensed by the second image sensor 200. For example, the second determination unit 319 may receive the AWB evaluation value AWB1 and the second conversion signal S1b (or the first conversion signal S1a) generated as a result of processing the signal sensed by the first image sensor 100.

[0141] The second determining unit 319 may determine whether the AE evaluation value AE2 is greater than a threshold value (step S351). In this case, the AE evaluation value AE2 may include a brightness value of light sensed by the second image sensor 200 that does not pass through the display 20. That is, when the environment in which the electronic device 1 is exposed is bright, the AE evaluation value AE2 may have a higher value.

[0142] For example, reference Figure 12 If the threshold is X1 lux and the AE evaluation value AE2 is greater than X1 lux (Yes in step S351), the determination unit 319 may transmit the AWB evaluation value AWB2 to the color compensation processor 326 (step S352). In other words, when the environment in which the electronic device 1 is exposed to is sufficiently bright, the color compensation processor 326 may perform color compensation using the AWB evaluation value AWB2 generated by sensing by the second image sensor 200, rather than using the AWB evaluation value AWB1 generated by sensing by the first image sensor 100.

[0143] When the first image sensor 100 senses light passing through the display 20, information about the light may be distorted by the display 20, etc. Therefore, when the AE evaluation value AE2 is greater than the threshold value, the color compensation processor 326 may perform color compensation using the AWB evaluation value AWB2 generated by sensing by the second image sensor 200. In one exemplary embodiment, the color compensation processor 326 may perform color compensation using the AWB evaluation value AWB2 only when the AE evaluation value AE2 is greater than the threshold value.

[0144] Since the AWB evaluation value AWB2 is used, a more accurate color temperature value can be used, thereby improving image quality by the color compensation processor 326 .

[0145] Reference again Figure 10 If the AE evaluation value AE2 is not greater than the threshold value ("No" in step S351), the determination unit 319 may transmit the AWB evaluation value AWB1 to the color compensation processor 326 (step S353). In other words, when the environment in which the electronic device 1 is located is not bright enough, the light sensed by the first image sensor 100 and the light sensed by the second image sensor 200 may differ significantly. Therefore, in this case, color compensation can be performed using the AWB evaluation value AWB1 generated by the sensing of the first image sensor 100.

[0146] After receiving one of the AWB evaluation values AWB1 and AWB2 from the determination unit 319 , the color compensation processor 326 may compensate the second conversion signal S1 b using the one of the AWB evaluation values AWB1 and AWB2 .

[0147] The color compensation processor 326 may determine whether the color temperature value derived from the corresponding one of the AWB evaluation values AWB1 and AWB2 is within the boundary area (step S354). For example, when the color compensation processor 326 receives the AWB evaluation value AWB1 from the determination unit 319, the color compensation processor 326 may determine whether the color temperature value derived from the AWB evaluation value AWB1 is within the boundary area; and when the color compensation processor 326 receives the AWB evaluation value AWB2 from the determination unit 319, the color compensation processor 326 may determine whether the color temperature value derived from the AWB evaluation value AWB2 is within the boundary area.

[0148] refer to Figure 12 When the color temperature value derived from a corresponding one of the AWB evaluation values AWB1 and AWB2 is less than the first temperature T1 or greater than the second temperature T2, the color compensation processor 326 may compensate the second conversion signal S1b.

[0149] Reference again Figure 10When the color temperature value derived from the corresponding one of the AWB evaluation values AWB1 and AWB2 is less than the first temperature T1 or greater than the second temperature T2 (step S354: "No"), the color compensation processor 326 may perform color compensation (step S355). For example, when the color temperature value derived from the corresponding AWB evaluation value AWB1 or AWB2 is less than approximately 3,000K, the color compensation processor 326 may compensate the second converted signal S1b with a redder color. Furthermore, for example, when the color temperature value derived from the corresponding AWB evaluation value AWB1 or AWB2 is greater than approximately 10,000K, the color compensation processor 326 may compensate the second converted signal S1b with a bluer color. However, embodiments of the present disclosure are not limited thereto, and the color compensation method of the color compensation processor 326 may be performed in various manners.

[0150] When the color temperature value derived from the corresponding one of the AWB evaluation values AWB1 and AWB2 is between the first temperature T1 and the second temperature T2 (“Yes” in step S354 ), the color compensation processor 326 may complete the process without performing color compensation.

[0151] By having the color compensation processor 326 perform color compensation on the second conversion signal S1b, the quality of the image corresponding to the second conversion signal S1b can be improved. Since the first image signal processor 310 performs color compensation using the AWB evaluation value AWB2 generated by the second image sensor 200 and the second image signal processor 330 only under specific conditions (e.g., in a sufficiently bright outdoor environment), accurate compensation can be performed.

[0152] Reference again Figure 9 , the color compensation processor 326 can output a third conversion signal S1c that has been color compensated. The third conversion signal S1c can be sent to Figure 8 The first determining unit 318 in .

[0153] refer to Figure 11 , the post-processing circuit 321 may further include a third determining unit 320. For example, the reference Figure 9 and Figure 10 The signal output by the described second determination unit 319 is sent to the third determination unit 320. In addition, the AWB evaluation value AWB1 may be sent to the third determination unit 320.

[0154] The third determination unit 320 may determine which of the AWB evaluation value AWB1 and the AWB evaluation value AWB2 is to be transmitted to the color compensation processor 326 based on a difference between the AE evaluation values AE1 and AE2 (eg, |AE1−AE2|).

[0155] For example, when the AE evaluation value AE2 is greater than the AE evaluation value AE1 by a certain difference or more (for example, when the brightness of the light incident on the rear surface of the electronic device 1 is greater than the brightness of the light incident on the front surface of the electronic device 1), the third determination unit 320 may transmit the AE evaluation value AE1 to the color compensation processor 326. In addition, when the AE evaluation value AE2 is almost the same as the AE evaluation value AE1 (for example, the brightness of the light incident on the rear surface of the electronic device 1 is similar to the brightness of the light incident on the front surface of the electronic device 1), the third determination unit 320 may transmit the AE evaluation value AE2 received from the second determination unit 319 to the color compensation processor 326.

[0156] refer to Figure 12 For example, when the difference between the AE evaluation value AE2 and the AE evaluation value AE1 is equal to or greater than X2 lux, the third determining unit 320 may transmit the AE evaluation value AE1 to the color compensation processor 326. Alternatively, when the difference between the AE evaluation value AE2 and the AE evaluation value AE1 is less than X2 lux, the third determining unit 320 may transmit the AE evaluation value AE2 received from the second determining unit 319 to the color compensation processor 326.

[0157] That is, when the difference between the brightness of the light incident on the front surface measured by the first image sensor 100 and the brightness of the light incident on the rear surface measured by the second image sensor 200 is higher than a certain difference or more, the color compensation processor 326 may perform color compensation using the AE evaluation value AE1 instead of the AE evaluation value AE2. In this way, color compensation can be performed on the image by using the AE evaluation value AE2 when a preset condition is satisfied.

[0158] Figure 13 are diagrams describing operations of an electronic device according to some example embodiments.

[0159] refer to Figure 13 , signals may be transmitted between the first image sensor 100 , the second image sensor 200 , the white balance compensation processor 325 , the color compensation processor 326 , the application processor 300 , and the display 20 included in the electronic device 1 .

[0160] The first image sensor 100 may provide the AWB evaluation value AWB1, the AE evaluation value AE1, and the first conversion signal S1a to the white balance compensation processor 325 (step S360). For example, the AWB evaluation value AWB1, the AE evaluation value AE1, and the first conversion signal S1a may be generated by the pre-processing circuit 311 and sent to the white balance compensation processor 325 of the post-processing circuit 321.

[0161] The white balance compensation processor 325 receives the AWB evaluation value AWB1, the AE evaluation value AE1, and the first conversion signal S1a and performs white balance compensation on the first conversion signal S1a based on the AWB evaluation value AWB1 to generate a second conversion signal S1b (step S361).

[0162] The white balance compensation processor 325 may provide the generated second conversion signal S1b to the color compensation processor 326 (step S362).

[0163] The first image sensor 100 may provide the AWB evaluation value AWB1 and the AE evaluation value AE1 to the color compensation processor 326 (step S363 ).

[0164] The second image sensor 200 may provide the AWB evaluation value AWB2 and the AE evaluation value AE2 to the color compensation processor 326 (step S364). For example, the AWB evaluation value AWB2 and the AE evaluation value AE2 may be generated by the pre-processing circuit 331 and sent to the color compensation processor 326 of the post-processing circuit 321.

[0165] The color compensation processor 326 may receive the AE evaluation values AE1 and AE2 , the AWB evaluation values AWB1 and AWB2 , and the second conversion signal S1 b (step S365 ).

[0166] The color compensation processor 326 may perform color compensation on the second conversion signal S1b based on the received signal (step S366). For example, the reference signal may be used. Figures 8 to 12 The color compensation processor 326 performs the color compensation processing described above. Thus, the color compensation processor 326 can generate a third conversion signal S1c.

[0167] Color compensation processor 326 may provide third conversion signal S1c to application processor 300 (step S367). For example, application processor 300 may receive third conversion signal S1c. For example, a display driving circuit included in application processor 300 may use third conversion signal S1c to generate an image signal.

[0168] The application processor 300 may provide the generated image signal to the display 20 (step S368). For example, the image signal generated by the display driving circuit may be transmitted to the display 20. As a result, the display 20 may output an image based on the image signal.

[0169] The following will refer to Figure 14 Image sensing systems according to some other example embodiments are described.

[0170] Figure 14is a block diagram showing an image sensing system according to some other embodiments. Figures 1 to 13 Duplicate descriptions will be abbreviated or omitted.

[0171] refer to Figure 14 , the image sensing system 3 may include a first image sensor 100 , a second image sensor 200 and an application processor 300 .

[0172] The first image sensor 100 may generate a first image signal S1 by sensing an image of a sensing target using incident light. For example, the first image sensor 100 may sense light passing through the display 20 and generate a first image signal S1.

[0173] The second image sensor 200 may generate the second image signal S2 by sensing an image of a sensing target using incident light. For example, the second image sensor 200 may generate the second image signal S2 by sensing light that does not pass through the display 20.

[0174] The first image signal S1 and the second image signal S2 may be provided to and processed by the application processor 300. For example, the first image signal S1 and the second image signal S2 may be provided to and processed by the image signal processor 390 included in the application processor 300.

[0175] The image signal processor 390 can process the reference Figures 5 to 13 The first image signal S1 and the second image signal S2.

[0176] For example, when processing the first image signal S1, the signal generated as a result of processing the second image signal S2 may be used under certain conditions. That is, when processing the first image signal S1, the signal generated by processing the second image signal S2 may be shared within the image signal processor 390. Furthermore, when processing the second image signal S2, the signal generated by processing the first image signal S1 may be shared within the image signal processor 390.

[0177] That is to say, by reference Figures 5 to 13 The processing steps performed by the first image signal processor 310 and the second image signal processor 330 of the image sensing system 2 described above can be performed by Figure 14 The image signal processor 390 of the image sensing system 3 is executed.

[0178] In this case, the shared signal is not limited to a signal generated based on the first image signal S1 and the second image signal S2 and may include various other information. For example, the shared signal may include a signal for performing high dynamic range (HDR) processing on a plurality of image data.

[0179] The following will refer to Figures 15 to 16 Electronic devices according to some other embodiments are described.

[0180] Figure 15 is a block diagram for describing an electronic device 1000 including a multi-camera module according to some embodiments. Figure 16 yes Figure 15 Detailed block diagram of the camera module in the . For ease of description, refer to Figures 1 to 14 Duplicate descriptions will be abbreviated or omitted.

[0181] refer to Figure 15 , the image sensing system 4 includes an electronic device 1000 , and the electronic device 1000 may include a camera module group 1100 , an application processor 1200 , a power management integrated circuit (PMIC) 1300 , an external memory 1400 , and a display 1500 .

[0182] The camera module group 1100 may include a plurality of camera modules 1100a, 1100b, and 1100c. Although the accompanying drawings illustrate an embodiment in which three camera modules 1100a, 1100b, and 1100c are arranged, the embodiments are not limited thereto. In some embodiments, the camera module group 1100 may be modified to include only two camera modules. Furthermore, in some embodiments, the camera module group 1100 may be modified to include n (n is a natural number greater than 3) camera modules.

[0183] In this embodiment, one of the three camera modules 1100a, 1100b, and 1100c may be a camera module including a reference Figures 1 to 14 The camera module of the first image sensor 100 described above may include a camera module. For example, the camera module 1100b may be a camera module including the first image sensor 100 and may be disposed toward the front surface of the electronic device 1 or 4. In addition, the camera module 1100b may be covered by the display 20 or the display 1500 and may sense incident light passing through the display 20 or 1500.

[0184] The following will refer to Figure 16 A detailed configuration of the camera module 1100b is described. The following description can be equally applied to the other camera modules 1100a and 1100c according to the embodiments.

[0185] refer to Figure 16 , the camera module 1100 b may include a prism 1105 , an optical path folding element (hereinafter referred to as “OPFE”) 1110 , an actuator 1130 , an image sensing unit 1140 and a storage unit 1150 .

[0186] The prism 1105 may include a reflective surface 1107 having a light reflective material and change a path of light L incident from the outside.

[0187] In some embodiments, the prism 1105 can change the path of light L incident in a first direction X to a second direction Y perpendicular to the first direction X. Furthermore, the prism 1105 can rotate the reflective surface 1107 having a light-reflecting material in direction A relative to the central axis 1106, or rotate the central axis 1106 in direction B, thereby changing the path of light L incident in the first direction X to a second direction Y perpendicular thereto. In this case, the OPFE 1110 can also move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0188] In some embodiments, as shown, the maximum rotation angle of the prism 1105 in the A direction may be 15 degrees or less in the positive (+) A direction and may be greater than 15 degrees in the negative (-) A direction. However, embodiments are not limited thereto.

[0189] In some embodiments, the prism 1105 can be moved in the positive (+) or negative (-) B direction by about 20 degrees, between 10 degrees and 20 degrees, or between 15 degrees and 20 degrees. In this case, the movement angle can be the same in the positive (+) or negative (-) B direction, or almost the same in the positive (+) or negative (-) B direction, with a difference of about 1 degree.

[0190] In some embodiments, the prism 1105 may move the reflective surface 1107 having the light reflective material in a third direction (eg, direction Z) parallel to the extension direction of the central axis 1106 .

[0191] The OPFE 1110 may include, for example, m (m is a natural number) optical lenses. The m lenses may be moved in the second direction Y to change the optical zoom ratio of the camera module 1100b. For example, assuming that the base optical zoom ratio of the camera module 1100b is Z, the optical zoom ratio of the camera module 1100b may be changed to 3Z, 5Z, or more by moving the m optical lenses included in the OPFE 1110.

[0192] The actuator 1130 can move the optical lens or OPFE 1110 (hereinafter referred to as "optical lens") to a specific position. For example, the actuator 1130 can adjust the position of the optical lens so that the image sensor 1142 can be positioned at the focal length of the optical lens for accurate sensing.

[0193] The image sensing unit 1140 may include an image sensor 1142, control logic 1144, and memory 1146. The image sensor 1142 may sense an image of a sensing target using light L provided through an optical lens. In some embodiments, the image sensor 1142 may include at least one of the above-described image sensors 100 and 200.

[0194] The control logic 1144 may control the overall operation of the camera module 1100b. For example, the control logic 1144 may control the operation of the camera module 1100b according to a control signal provided through the control signal line CSLb.

[0195] The memory 1146 may store information such as calibration data 1147 for the operation of the camera module 1100b. The calibration data 1147 may include information for the camera module 1100b to generate image data using light L provided from the outside. The calibration data 1147 may include, for example, the aforementioned information regarding the optical axis, information regarding the degree of rotation, and information regarding the focal length. When the camera module 1100b is implemented as a multi-state camera in which the focal length varies depending on the position of the optical lens, the calibration data 1147 may include information regarding autofocus and the focal length value for each position (or state) of the optical lens.

[0196] The storage unit 1150 may store image data sensed by the image sensor 1142. The storage unit 1150 may be provided outside the image sensing unit 1140 and may be implemented in a form of being stacked with a sensor chip constituting the image sensing unit 1140. In some embodiments, the storage unit 1150 may be implemented as an electrically erasable programmable read-only memory (EEPROM), but the embodiment is not limited thereto.

[0197] Common Reference Figure 15 and Figure 16 In some embodiments, each of the camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Accordingly, the camera modules 1100a, 1100b, and 1100c may each include the same or different calibration data 1147 depending on the operation of the actuator 1130 included therein.

[0198] In some embodiments, one camera module (e.g., camera module 1100b) among the camera modules 1100a, 1100b, and 1100c may be a folded lens type camera module including the above-described prism 1105 and OPFE 1110, and the remaining camera modules (e.g., camera modules 1100a and 1100c) may be vertical type camera modules that do not include the prism 1105 and OPFE 1110. However, embodiments are not limited thereto.

[0199] In some embodiments, one of the camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100c) may be a vertical-type depth camera that extracts depth information using, for example, infrared (IR). In this case, the application processor 1200 may combine image data provided from the depth camera with image data provided from another camera module (e.g., camera module 1100a or 1100b) to generate a three-dimensional (3D) depth image.

[0200] In some embodiments, at least two of the camera modules 1100a, 1100b, and 1100c (e.g., camera modules 1100a and 1100c) may have different fields of view (perspectives). In this case, for example, at least two of the camera modules 1100a, 1100b, and 1100c (e.g., camera modules 1100a and 1100c) may have different optical lenses, but the present invention is not limited thereto.

[0201] Furthermore, in some embodiments, the camera modules 1100a, 1100b, and 1100c may have different viewing angles. In this case, the optical lenses included in the respective camera modules 1100a, 1100b, and 1100c may also be different, but the present disclosure is not limited thereto.

[0202] In some embodiments, the camera modules 1100a, 1100b, and 1100c may be physically separated from each other. That is, the sensing area of one image sensor 1142 is not divided and used by all of the camera modules 1100a, 1100b, and 1100c, but an independent image sensor 1142 may be provided within each of the camera modules 1100a, 1100b, and 1100c.

[0203] In some embodiments, for example, the image sensor 1142 included in the camera module 1100 a may have the above-described image sensor 200 , and the image sensor 1142 included in the camera module 1100 b may have the above-described image sensor 100 .

[0204] Reference again Figure 15 , the application processor 1200 may include an image processing unit 1210, a memory controller 1220, and an internal memory 1230. The application processor 1200 may be implemented separately from the camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the camera modules 1100a, 1100b, and 1100c may be implemented as separate semiconductor chips, respectively.

[0205] The image processing unit 1210 may include a plurality of sub-image processors 1212a, 1212b, and 1212c, an image generator 1214, and a camera module controller 1216. In some embodiments, the image processing unit 1210 may include the first image signal processor 310, the second image signal processor 330, and the image signal processor 390 described above.

[0206] The image processing unit 1210 may include sub-image processors 1212a, 1212b, and 1212c, the number of which corresponds to the number of camera modules 1100a, 1100b, and 1100c. In some embodiments, each of the sub-image processors 1212a, 1212b, and 1212c may include one of the first image signal processor 310, the second image signal processor 330, and the image signal processor 390 described above.

[0207] Image data generated from the respective camera modules 1100a, 1100b, and 1100c can be provided to the corresponding sub-image processors 1212a, 1212b, and 1212c via separate image signal lines ISLa, ISLb, and ISLc. For example, image data generated from camera module 1100a can be provided to sub-image processor 1212a via image signal line ISLa, image data generated from camera module 1100b can be provided to sub-image processor 1212b via image signal line ISLb, and image data generated from camera module 1100c can be provided to sub-image processor 1212c via image signal line ISLc. Such image data transmission can be performed using a camera serial interface (CSI) based on a mobile industry processor interface (MIPI), for example, but embodiments are not limited thereto.

[0208] In some embodiments, one sub-image processor may be arranged to correspond to multiple camera modules. For example, sub-image processor 1212a and sub-image processor 1212c may be integrated into one sub-image processor rather than being separated from each other as shown, and image data provided from camera module 1100a and camera module 1100c may be selected by a data selector (e.g., a multiplexer) and then provided to the integrated sub-image processor.

[0209] The image data provided to each of the sub-image processors 1212a, 1212b, and 1212c may be provided to the image generator 1214. The image generator 1214 may use the image data provided by each of the sub-image processors 1212a, 1212b, and 1212c to generate an output image based on the image generation information or mode signal. In some embodiments, the image generator 1214 may include the post-processing circuit 321 described above. The image generator 1214 may perform the compensation process described above.

[0210] Specifically, based on the image generation information or the mode signal, the image generator 1214 may generate an output image by combining at least some of the image data generated from the camera modules 1100a, 1100b, and 1100c having different viewing angles. Furthermore, based on the image generation information or the mode signal, the image generator 1214 may generate an output image by selecting any one of the image data generated from the camera modules 1100a, 1100b, and 1100c having different viewing angles.

[0211] In some embodiments, the image generation information may include a zoom signal (or zoom factor).Furthermore, in some embodiments, the mode signal may be a signal based on, for example, a mode selected by a user.

[0212] When the image generation information is a zoom signal (zoom factor) and the camera modules 1100a, 1100b, and 1100c have different fields of view (angles of view), the image generator 1214 may perform different operations depending on the type of zoom signal. For example, when the zoom signal is a first signal, the image data output from the camera module 1100a may be merged with the image data output from the camera module 1100c, and then the output image may be generated using the merged image signal and the image data output from the camera module 1100b that is not used for merging. When the zoom signal is a second signal different from the first signal, the image generator 1214 does not perform image data merging, but may generate an output image by selecting any one of the image data output from the camera modules 1100a, 1100b, and 1100c. However, the embodiment is not limited thereto, and various modifications may be made to the method of processing image data as needed.

[0213] In some embodiments, the image generator 1214 may receive multiple image data with different exposure times from at least one of the sub-image processors 1212a, 1212b, and 1212c, and perform high dynamic range (HDR) processing on the multiple image data to generate merged image data with increased dynamic range.

[0214] The camera module controller 1216 may provide control signals to each of the camera modules 1100a, 1100b, and 1100c. The control signals generated from the camera module controller 1216 may be provided to the corresponding camera modules 1100a, 1100b, and 1100c through separate control signal lines CSLa, CSLb, and CSLc.

[0215] Any one of the camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100a) can be designated as a master camera based on a mode signal or image generation information including a zoom signal, and the remaining camera modules (e.g., camera modules 1100b and 1100c) can be designated as slave cameras. Such information can be included in the control signals provided to the corresponding camera modules 1100a, 1100b, and 1100c via separate control signal lines CSLa, CSLb, and CSLc.

[0216] The camera module that operates as the master or slave device can be changed based on the zoom factor or operation mode signal. For example, when the angle of view of camera module 1100a is greater than that of camera module 1100c and the zoom factor indicates a low zoom ratio, camera module 1100c can operate as the master device and camera module 1100a can operate as the slave device. Conversely, when the zoom factor indicates a high zoom ratio, camera module 1100a can operate as the master device and camera module 1100c can operate as the slave device.

[0217] In some embodiments, the control signal provided from camera module controller 1216 to each of camera modules 1100a, 1100b, and 1100c may include a synchronization enable signal. For example, when camera module 1100b is the master camera and camera modules 1100a and 1100c are slave cameras, camera module controller 1216 may transmit the synchronization enable signal to camera module 1100b. Camera module 1100b, having received the synchronization enable signal, may generate a synchronization signal based on the received synchronization enable signal and transmit the generated synchronization signal to camera modules 1100a and 1100c via a synchronization signal line SSL. Based on the synchronization signal, camera modules 1100a, 1100b, and 1100c may synchronously transmit image data to application processor 1200.

[0218] In some embodiments, the control signal provided to the camera modules 1100a, 1100b, and 1100c from the camera module controller 1216 may include mode information according to the mode signal. Based on the mode information, the camera modules 1100a, 1100b, and 1100c may operate in the first operating mode or the second operating mode with respect to the sensing rate.

[0219] In the first operating mode, camera modules 1100a, 1100b, and 1100c may generate image signals at a first rate (e.g., a first frame rate), encode the image signals at a second rate higher than the first rate (e.g., a second frame rate higher than the first frame rate), and transmit the encoded image signals to application processor 1200. In this case, the second rate may be 30 times or less than the first rate.

[0220] Application processor 1200 may store the received image signal, i.e., the encoded image signal, in memory 1230 provided therein or in memory 1400 provided external to application processor 1200. Application processor 1200 may then read the encoded image signal from memory 1230 or memory 1400 to decode the encoded image signal and display image data generated based on the decoded image signal. For example, corresponding sub-processors 1212a, 1212b, and 1212c of image processing unit 1210 may perform decoding and may also perform image processing on the decoded image signal. For example, image data generated based on the decoded image signal may be displayed on display 1500.

[0221] In the second operating mode, camera modules 1100a, 1100b, and 1100c may generate image signals at a third rate lower than the first rate (e.g., a third frame rate lower than the first frame rate) and transmit the image signals to application processor 1200. The image signals provided to application processor 1200 may be unencoded signals. Application processor 1200 may perform image processing on the received image signals or may store the image signals in memory 1230 or memory 1400.

[0222] The PMIC 1300 may supply power, such as a power supply voltage, to each of the camera modules 1100a, 1100b, and 1100c. For example, under the control of the application processor 1200, the PMIC 1300 may supply a first power to the camera module 1100a via a power signal line PSLa, supply a second power to the camera module 1100b via a power signal line PSLb, and supply a third power to the camera module 1100c via a power signal line PSLc.

[0223] The PMIC 1300 can generate power corresponding to each of the camera modules 1100a, 1100b, and 1100c in response to a power control signal PCON from the application processor 1200, and can also adjust the power level. The power control signal PCON can include a power adjustment signal for each operating mode of the camera modules 1100a, 1100b, and 1100c. For example, the operating mode may include a low power mode, and in this case, the power control signal PCON may include information about the camera module to operate in the low power mode and at a set power level. The power levels provided to the respective camera modules 1100a, 1100b, and 1100c can be the same or different. In addition, the power levels can be changed dynamically.

[0224] The present disclosure can be implemented as computer-readable codes written on a computer-readable recording medium. The computer-readable recording medium may be any type of recording device that stores data in a computer-readable manner.

[0225] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and carrier waves (e.g., data transmission via the Internet). The computer-readable recording medium can be distributed across multiple computer systems connected to a network so that computer-readable code can be written thereto and executed therefrom in a decentralized manner. Those skilled in the art can easily infer functional programs, codes, and code segments for implementing the present disclosure.

[0226] According to an example embodiment, at least one of the components, elements, modules or units described herein can be implemented as various quantities of hardware, software and / or firmware structures that perform the above-mentioned various functions. For example, at least one of these components, elements or units can use a direct circuit structure that can perform the corresponding function by the control of one or more microprocessors or other control devices, such as a memory, a processor, a logic circuit, a lookup table, etc. In addition, at least one of these components, elements or units can be specifically implemented by a part of a module, program or code, and a part of this module, program or code includes one or more executable instructions for performing a specified logical function and performed by one or more microprocessors or other control devices. In addition, at least one of these components, elements or units can further include a processor, such as a central processing unit (CPU), a microprocessor, etc. that performs the corresponding function, or is implemented by it. Two or more of these components, elements or units can be combined into a single component, element or unit, and this single component, element or unit performs all operations or functions of the two or more components, elements or units that are combined. In addition, at least a portion of the functions of at least one of these components, elements or units can be performed by another of these components, elements or units. Furthermore, although a bus is not shown in the block diagram, communication between components, elements, or units can be performed via a bus. The functional aspects of the above example embodiments can be implemented as algorithms executed on one or more processors. Furthermore, the components, elements, or units represented by blocks or processing operations can employ any number of related art techniques for electronic configuration, signal processing and / or control, data processing, and the like.

[0227] While the present disclosure has been particularly shown and described with reference to embodiments thereof, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. An electronic device including an image sensor, comprising: monitor; a first image sensor configured to output a first image signal based on sensing first light passing through the display; a second image sensor configured to output a second image signal based on sensing a second light that does not pass through the display; as well as a processor configured to: generating a first optical value and a second optical value based on the second image signal, the second optical value being different from the first optical value, and Based on the first optical value satisfying a first condition, the first image signal is corrected using the second optical value.

2. The electronic device according to claim 1 , wherein the first optical value comprises at least one of a brightness value or an illuminance value generated based on the second image signal, and The second optical value includes a color temperature value generated based on the second image signal.

3. The electronic device of claim 2 , wherein the processor is further configured to correct the first image signal by using the color temperature value based on the first optical value satisfying the first condition and also based on the color temperature value being less than a first color temperature or being equal to or greater than a second color temperature, the second color temperature being higher than the first color temperature. 4 . The electronic device of claim 1 , wherein the processor is further configured to generate a third optical value and a fourth optical value based on the first image signal, the fourth optical value being different from the third optical value.

5. The electronic device according to claim 4 , wherein the third optical value comprises at least one of a brightness value or an illuminance value generated based on the first image signal, and The fourth optical value includes a color temperature value generated based on the first image signal. The electronic device according to claim 1 , wherein the first condition is that the first optical value is equal to or greater than a threshold value. 7 . The electronic device of claim 1 , wherein the processor is further configured to correct the first image signal by using an optical value generated based on the first image signal, based on the first optical value not satisfying the first condition. 8 . The electronic device of claim 7 , wherein the optical value generated based on the first image signal comprises a color temperature value generated based on the first image signal. 9 . The electronic device according to claim 4 , wherein the first condition is that a difference between the first optical value and the third optical value is smaller than a threshold value.

10. The electronic device of claim 9, wherein the processor is further configured to correct the first image signal by using an optical value generated based on the first image signal based on a difference between the first optical value and the third optical value being equal to or greater than the threshold. 11 . The electronic device according to claim 1 , wherein the display is further configured to output an image generated based on the corrected first image signal.

12. The electronic device according to claim 1, wherein the processor is further configured to: output a third image signal by performing automatic white balance on the first image signal; and wherein the processor is further configured to output a fourth image signal obtained by correcting the third image signal using the second optical value based on the first optical value satisfying a second condition. 13 . The electronic device of claim 12 , wherein the display is further configured to output an image generated based on at least one of the third image signal or the fourth image signal.

14. An electronic device including an image sensor, comprising: a first image sensor configured to output a first image signal based on sensing first light incident on a front surface of the electronic device; a second image sensor configured to output a second image signal based on sensing second light incident on a rear surface of the electronic device; a processor configured to receive the first image signal and the second image signal; as well as a display provided on the front surface and configured to output an image generated based on the first image signal, The processor is further configured to generate a first color temperature value based on the second image signal, and correct the first image signal by using the first color temperature value based on a first condition being satisfied. 15 . The electronic device of claim 14 , wherein the display is further configured to cover the first image sensor. 16 . The electronic device of claim 14 , wherein the processor is further configured to generate a second color temperature value based on the first image signal, and to generate at least one of a brightness value or a luminance value based on the second image signal. 17 . The electronic device according to claim 16 , wherein the first condition is that at least one of the brightness value or the illuminance value is equal to or greater than a threshold value. 18 . The electronic device of claim 16 , wherein the processor is further configured to correct the first image signal by using the second color temperature value based on the first condition not being satisfied. 19 . The electronic device of claim 14 , wherein the display is further configured to output an image generated based on the corrected first image signal.

20. An electronic device including an image sensor, comprising: monitor; a first camera module comprising a first image sensor configured to output a first image signal based on sensing first light passing through the display; a second camera module including a second image sensor configured to output a second image signal based on sensing a second light that does not pass through the display; as well as an application processor provided separately from the first camera module and the second camera module, the application processor including an image signal processor, The image signal processor is configured to: receiving the first image signal from the first camera module via a first camera serial interface, receiving the second image signal from the second camera module via a second camera serial interface, generating a first color temperature value based on the first image signal, generating one of a brightness value or an illuminance value, and a second color temperature value based on the second image signal, and Based on the luminance value or at least one of the luminance values being equal to or greater than a threshold, the first image signal is corrected based on the second color temperature value, and based on the luminance value or at least one of the luminance values being less than the threshold, the first image signal is corrected based on the first color temperature value.

Images