Image sensor and electronic device including the same

Abstract

An image sensor includes: a pixel array including a plurality of pixels arranged in a first direction and a second direction; and a logic circuit configured to drive the plurality of pixels. The logic circuit is configured to generate first image data having a first resolution based on pixel data corresponding to each of the plurality of pixels, generate second image data having a second resolution based on merged pixel data corresponding to each of the plurality of merged pixels after generating the first image data, and generate the second image data having a third resolution based on the merged pixel data corresponding to each of the plurality of merged pixels after generating the second image data. A plurality of merging pixels merging two or more pixels adjacent to each other among the plurality of pixels, the second resolution being lower than the first resolution; and outputting the first image data and the second image data. A first number of frames per second of the first image data is lower than a second number of frames per second of the second image data.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10 - 2024 - 0000727, filed on January 3, 2024, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] One or more example embodiments of the present disclosure relate to an image sensor and an electronic device including the image sensor.

[0004] An image sensor can not only image an object to generate a two - dimensional image, but also be applied to various fields to measure the distance to the object or generate a three - dimensional image. In particular, in recent years, active research has been conducted to implement the function of capturing an image by imaging an object and the function of tracking the movement of the object or measuring the distance to the object using a single image sensor. Summary of the Invention

[0005] One or more example embodiments of the present disclosure provide an image sensor and an electronic device including the image sensor, the image sensor being capable of generating image data with different resolutions and different numbers of frames per second from a single pixel array, thereby outputting a video with a high resolution and a high number of frames per second, or reducing the jitter of an object.

[0006] According to an aspect of an example embodiment of the present disclosure, there is provided an image sensor including: a pixel array including a plurality of pixels arranged along a first direction and a second direction, the second direction intersecting the first direction; and a logic circuit configured to drive the plurality of pixels, wherein the logic circuit is further configured to: generate first image data having a first resolution based on pixel data corresponding to each of the plurality of pixels, and after generating the first image data, generate second image data having a second resolution based on binning pixel data corresponding to each of a plurality of binning pixels, the plurality of binning pixels binning two or more adjacent pixels among the plurality of pixels, the second resolution being lower than the first resolution; and output the first image data and the second image data, and wherein a first number of frames per second of the first image data is lower than a second number of frames per second of the second image data.

[0007] According to one aspect of an example embodiment of the present disclosure, an electronic device is provided, including: an image sensor configured to sequentially output first image data having a first resolution and second image data having a second resolution, the second resolution being lower than the first resolution; and a processor configured to output a result image using the first image data and the second image data, wherein the image sensor includes a pixel array including a plurality of pixels having the same structure, and wherein the image sensor is further configured to generate the first image data based on pixel data obtained from each of the plurality of pixels exposed to light for a first exposure time, and generate the second image data based on combined pixel data obtained from each of the plurality of combined pixels formed by combining two or more adjacent pixels among the plurality of pixels exposed to light for a second exposure time, the second exposure time being shorter than the first exposure time.

[0008] According to one aspect of an example embodiment of the present disclosure, a processor is provided, including: an interface configured to receive image data from an image sensor; and an image signal processor configured to process the image data to generate a result image, wherein the interface is configured to receive first image data having a first resolution and a first frames per second, and second image data having a second resolution and a second frames per second, wherein the first resolution is higher than the second resolution, and the first frames per second is slower than the second frames per second, and wherein the image signal processor is configured to use the second image data to process the first image data to generate a result image. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other aspects, features, and advantages of certain example embodiments of the present disclosure will be more clearly understood from the following detailed description in conjunction with the accompanying drawings, in which:

[0010] Figure 1 is a block diagram schematically showing an electronic device including an image sensor according to one or more example embodiments.

[0011] Figure 2 is a view schematically showing a part of a pixel array included in an image sensor according to one or more example embodiments.

[0012] Figure 3 is a circuit diagram schematically showing a pixel disposed in a pixel array of an image sensor according to one or more example embodiments.

[0013] Figure 4 is a view schematically showing a part of a pixel array included in an image sensor according to one or more example embodiments.

[0014] Figure 5is a circuit diagram schematically showing a pixel placed in a pixel array of an image sensor according to one or more example embodiments.

[0015] Figure 6 is a flowchart showing the operation of an image sensor according to one or more example embodiments.

[0016] Figure 7 is a flowchart showing the operation of an electronic device according to one or more example embodiments.

[0017] Figure 8 and Figure 9 is a view showing the operation of an image sensor according to one or more example embodiments.

[0018] Figures 10 to 12 is a view showing the operation of an electronic device according to one or more example embodiments.

[0019] Figures 13 to 15 is a view showing the operation of an electronic device according to one or more example embodiments.

[0020] Figure 16 is a flowchart showing the operation of an electronic device according to one or more example embodiments.

[0021] Figure 17 and Figure 18 is a view showing the output data of an image sensor according to one or more example embodiments.

[0022] Figure 19 and Figure 20 is a view showing the operation of an electronic device according to one or more example embodiments.

[0023] Figure 21 is a view schematically showing an image sensor according to one or more example embodiments.

[0024] Figure 22 is a view schematically showing a processor connected to an image sensor according to one or more example embodiments.

[0025] Figure 23 is a view schematically showing an electronic device including an image sensor according to one or more example embodiments. Detailed Description

[0026] Hereinafter, example embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0027] Figure 1 is a block diagram schematically showing an electronic device including an image sensor according to one or more example embodiments.

[0028] Reference Figure 1 As shown in Figure 1 , the electronic device 1 may include an image sensor 10 and a processor 40. The image sensor 10 may include a pixel array 20 and a peripheral circuit 30. In the pixel array 20, a plurality of pixels PX may be arranged in an array form along a plurality of rows and a plurality of columns. The plurality of pixels PX may have the same structure. Each of the plurality of pixels PX may include at least one photoelectric conversion element that generates charge in response to incident light. The photoelectric conversion element may be a photodiode.

[0029] In addition to the photodiode, each of the plurality of pixels PX may further include a pixel circuit that converts the charge generated by the light incident on the photodiode into an electrical signal. In an embodiment, the pixel circuit may include a plurality of transistors, and according to an embodiment, two or more adjacent pixels may share some of the plurality of transistors. The structure and operation of the pixel circuit will be described later.

[0030] The peripheral circuit 30 may include a circuit for controlling the pixel array 20. For example, the peripheral circuit 30 may also include a row driver 31, a readout circuit 32, a data input / output circuit 33, a control logic 34, etc. The row driver 31 may simultaneously drive at least a part of the plurality of pixels PX arranged in the pixel array 20. For example, the row driver 31 may provide a predetermined control signal to a part of the transistors included in the corresponding selected pixels arranged along the selected row line.

[0031] The readout circuit 32 may be connected to the pixels through column lines. The readout circuit 32 may read pixel signals from the selected pixels through the column lines, and the selected pixels receive control signals from the row driver 31. In an embodiment, the pixel signal may correspond to the difference between the reset voltage and the pixel voltage detected at each selected pixel. For example, the readout circuit 32 may convert the pixel signal into a digital signal to generate pixel data.

[0032] The pixel data generated by the readout circuit 32 may be sent to the data output circuit 33, and the data output circuit 33 may output image data including the pixel data through a predetermined interface. For example, the data output circuit 33 may send the image data to the processor 40. In addition to a central processing unit, a graphics processing unit, and a power supply unit, the processor 40 may further include an image signal processor (ISP) that receives the image data and generates a result image to be output to the user. According to an embodiment, the image signal processor may not be included in the processor 40, but may be packaged together with the image sensor 10.

[0033] The control logic 34 may include a timing controller for controlling the operation timing of the line driver 31, the readout circuit 32, the data output circuit 33, etc. According to an embodiment, the image sensor 10 may further include a memory for storing image data, and in this case, the control logic 34 may include a circuit for controlling the memory. For example, the memory may be implemented as a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.

[0034] In an embodiment, the image sensor 10 may generate image data with different resolutions. For example, the image sensor 10 may generate first image data and second image data. The first image data includes pixel data obtained from each of a plurality of pixels PX, and the second image data includes merged pixel data obtained from a plurality of merged pixels respectively defined by merging two or more of the plurality of pixels PX. The first resolution of the first image data may be higher than the second resolution of the second image data. According to an embodiment, the first image data may be RGB image data, and the second image data may be black-and-white image data.

[0035] The first image data with a relatively high resolution may inevitably have a limitation in frames per second (fps). Therefore, when a resultant image in video format is generated from the first image data, a resultant image with a high resolution may be output, but the frames per second of the resultant image may be limited. Since the second image data may be generated from merged pixel signals obtained from a plurality of merged pixels, the second image data may have a lower resolution and may increase the frames per second compared to the first image data.

[0036] In an embodiment, second image data with a lower resolution but a higher frames per second may be generated compared to the first image data, and the second image data may be processed to reduce the number of bits of the merged pixel data corresponding to each of the plurality of merged pixels. For example, among the frames included in the second image data, the difference between the merged pixel data included in a pair of consecutively output frames may be calculated, and the calculated difference may be compared with a threshold to reduce the number of bits of the merged pixel data.

[0037] For example, an image signal processor that has received the first image data and the second image data may use the second image data to up-convert the frames per second of the first image data in video format. Alternatively, the quality of the first image data in still image format may be improved by deblurring the first image data with reference to the second image data.

[0038] In the first image data having a high resolution, the number of bits of each pixel data can be relatively large, and thus, the bandwidth required to output the first image data can be greater than the bandwidth required to output the second image data. In an embodiment, as described above, the number of bits of the merged pixel data included in the second image data can be reduced to minimize an increase in the bandwidth required to output the first image data and the second image data. In addition, a memory for temporarily storing the second image data generated at a high number of frames per second can be implemented with a small capacity.

[0039] Figure 2 is a view schematically showing a part of a pixel array included in an image sensor according to one or more example embodiments.

[0040] Reference Figure 2 , the pixel array 50 of the image sensor according to an embodiment may include a plurality of pixels 51 to 53 arranged along a first direction (e.g., the X-axis direction) and a second direction (e.g., the Y-axis direction). For example, the pixel array 50 may include red pixels 51, green pixels 52, and blue pixels 53. Each red pixel 51 may include a red color filter, each green pixel 52 may include a green color filter, and each blue pixel 53 may include a blue color filter.

[0041] According to an embodiment, the color filters of the pixels 51 to 53 included in the pixel array 50 may be implemented in more various ways. For example, at least a part of the green pixels 52 may include a white color filter that allows white light to pass through, instead of a green color filter. Alternatively, at least a part of the green pixels 52 may include a yellow color filter that allows yellow light to pass through, instead of a green color filter.

[0042] Figure 3 is a circuit diagram of a pixel placed in a pixel array of an image sensor schematically showing according to one or more example embodiments.

[0043] Reference Figure 3 , the pixel 60 placed in the pixel array may include a photodiode PD, a transfer transistor TX, a reset transistor RX, a driving transistor DX, a selection transistor SX, etc. A node to which the transfer transistor TX, the reset transistor RX, and the driving transistor DX are connected to each other may be provided as a floating diffusion FD, and when the transfer transistor TX is turned on, the charge generated from the photodiode PD may move to the floating diffusion FD.

[0044] The readout operation of obtaining a pixel signal from pixel 60 and converting the pixel signal into pixel data may start from the reset operation of pixel 60. For example, when transfer transistor TX and reset transistor RX are turned on according to transfer control signal TG and reset control signal RG, respectively, floating diffusion FD may be connected to a power node supplying power supply voltage VDD and may be reset. Driving transistor DX operating as a source-follower amplifier may amplify the voltage of reset floating diffusion FD, and may output the reset voltage to column line COL by turning on selection transistor SX according to selection control signal SEL.

[0045] After floating diffusion FD is reset, photodiode PD may be exposed to light and may generate charges during a predetermined exposure time. When the exposure time ends, transfer transistor TX may be turned on, and the charges of photodiode PD may move to floating diffusion FD. When the main charge carriers of photodiode PD may be electrons, the voltage of floating diffusion FD may decrease. Driving transistor DX may amplify the voltage of floating diffusion FD, and when selection transistor SX is turned on, the pixel voltage may be output to column line COL. A readout circuit connected to column line COL may digitally convert the difference between the reset voltage and the pixel voltage to generate pixel data corresponding to pixel 60.

[0046] Figure 4 is a view schematically showing a part of a pixel array included in an image sensor according to one or more example embodiments.

[0047] Reference Figure 4 , the pixel array 70 of the image sensor according to an embodiment may include a plurality of pixel groups 71, 73, and 75 arranged in a first direction (e.g., X-axis direction) and a second direction (e.g., Y-axis direction). In Figure 4 the illustrated embodiment, each of the plurality of pixel groups 71, 73, and 75 may include two or more pixels (e.g., 72, 74, and 76), and the two or more pixels may be adjacent to each other in at least one of the first direction or the second direction and share color filters of the same color.

[0048] In each of the multiple pixel groups 71, 73, and 75, two or more pixels (e.g., 72, 74, and 76) may include color filters of the same color. For example, the first pixel group 71 may include red pixels 72 each having a red color filter, and the second pixel group 73 may include green pixels 74 each having a green color filter. The third pixel group 75 may include blue pixels 76 each having a blue color filter. The color of the color filter may vary according to the embodiment. For example, each of the second pixel groups 73 alternately arranged with the first pixel group 71 in the first direction may include green pixels 74 having green color filters, while each of the second pixel groups 73 alternately arranged with the third pixel group 75 in the second direction may include white pixels having white color filters.

[0049] In each of the multiple pixel groups 71, 73, and 75, two or more pixels (e.g., 72, 74, and 76) may be arranged in a 2×2 form. Each of the pixels (e.g., 72, 74, and 76) may include a photodiode and a transfer transistor. In an embodiment, the pixels (e.g., 72, 74, and 76) included in each of the pixel groups 71, 73, and 75 may share a floating diffusion, a reset transistor, a driving transistor, and a selection transistor.

[0050] Figure 5 is a circuit diagram schematically showing pixels placed in a pixel array of an image sensor according to one or more example embodiments.

[0051] Figure 5 may be a circuit diagram schematically showing a pixel circuit corresponding to a pixel group 80 included in a pixel array according to an embodiment (e.g., the pixel array 70 as shown above Figure 4 ). As previously referenced Figure 4 , the pixel group 80 may include a first pixel to a fourth pixel arranged in a 2×2 form. Referring to Figure 5 , the first pixel may include a first photodiode PD1 and a first transfer transistor TX1, and the second pixel may include a second photodiode PD2 and a second transfer transistor TX2. The third pixel may include a third photodiode PD3 and a third transfer transistor TX3, and the fourth pixel may include a fourth photodiode PD4 and a fourth transfer transistor TX4.

[0052] In an embodiment, the first pixel to the fourth pixel may share a floating diffusion FD, a reset transistor RX, a driving transistor DX, and a selection transistor SX, and may be connected to a column line COL. Thus, in the readout operation of the pixel group 80, pixel data may be sequentially obtained from each of the first pixel to the fourth pixel.

[0053] For example, after turning on the first transfer transistor TX1 and the reset transistor RX to reset the floating diffusion FD and the first photodiode PD1, the first photodiode PD1 can be exposed to light for a predetermined exposure time to generate charges. When the exposure time has passed, the first transfer transistor TX1 can be turned on to move the charges of the first photodiode PD1 to the floating diffusion FD, and the selection transistor SX can be turned on. The readout circuit of the image sensor can obtain a pixel signal corresponding to the difference between the reset voltage and the pixel voltage through the column line COL, and can convert the pixel signal into pixel data. The above operations can be sequentially performed for the second photodiode PD2 to the fourth photodiode PD4 to perform a readout operation for the pixel group 80.

[0054] Figure 6 is a flowchart illustrating the operation of an image sensor according to one or more example embodiments.

[0055] Refer to Figure 6 , the image sensor according to an embodiment can generate first image data having a first resolution (S10). For example, the first image data can be image data generated from pixel data obtained from a plurality of pixels arranged in a pixel array in the image sensor, respectively.

[0056] When the first image data is generated, the image sensor can perform binning setting (S11). The binning setting can be performed to generate one piece of binned pixel data based on pixel data obtained from two or more pixels arranged adjacent to each other in the pixel array. Therefore, the effect of obtaining pixel data from fewer pixels than all the pixels arranged in the pixel array can be achieved through the binning setting.

[0057] After the binning is set, the image sensor can generate second image data having a second resolution (S12). One piece of binned pixel data can be generated according to the pixel data obtained from two or more pixels through the binning setting, and the binned pixel data can be determined by the pixel value of each of the binned pixels included in the second image data. Therefore, the second resolution can be lower than the first resolution.

[0058] The image sensor can output the first image data and the second image data (S13). For example, the first image data can be output before the second image data because the first image data is generated before the second image data. Alternatively, according to an embodiment, the second image data can be output before the first image data. However, the present disclosure is not limited thereto, and the first image data and the second image data can be output in different ways. For example, the first image data and the second image data can be output together.

[0059] A processor connected to an image sensor may use first image data and second image data to generate a result image. According to an embodiment, various result images may be generated from the first image data and the second image data, which will be described later with reference to Figure 7 be described.

[0060] Figure 7 is a flowchart showing operations of an electronic device according to one or more example embodiments.

[0061] Referring to Figure 7 , an electronic device according to an embodiment may perform a camera function (S20). When the camera function is activated in the electronic device, an operation mode such as a capture mode or a video mode may be selected as the camera function, and operations in the image sensor and a processor connected to the image sensor may be performed according to the selected operation mode.

[0062] The capture mode may be an operation mode for capturing a still image (e.g., a photo). When it is confirmed that the capture mode has been selected as the camera function (S21), the image sensor may perform binning setting (S22). As previously described with reference to Figure 6 , when binning is set, the image sensor may use pixel data obtained from two or more adjacent pixels to generate binned pixel data. The binned pixel data may be data corresponding to a binned pixel defined by two or more adjacent pixels.

[0063] For example, binned pixels may be defined as pixels adjacent to each other and arranged in an M×M form. According to an embodiment, binned pixel data may be generated using pixel data obtained from four pixels adjacent to each other and arranged in a 2×2 form, or binned pixel data may be generated using pixel data obtained from 16 pixels adjacent to each other and arranged in a 4×4 form.

[0064] After binning is set, the image sensor may acquire image data in a relatively short exposure time (S23). For example, the image sensor may set the exposure time of the binned pixels to a second exposure time, which is a relatively short time compared to a first exposure time to be described later, and may acquire binned pixel data from the binned pixels to generate image data. When image data corresponding to the short exposure time is acquired, binning setting may be deactivated in the image sensor (S24).

[0065] When the binning setting is deactivated, the image sensor can acquire image data with a long exposure time (S25). For example, the image sensor can set the exposure time of each of the multiple pixels arranged in the pixel array to a first exposure time longer than the second exposure time, and can acquire pixel data from each of the multiple pixels to generate image data. When generating image data based on the pixel data obtained from each of the multiple pixels, the image sensor can perform the binning setting again (S26).

[0066] When the binning setting is completed, the image sensor can expose the binned pixels to light for the second exposure time and can acquire image data (S27). The image data generated by the image sensor when the binning setting is deactivated may have a relatively lower resolution than the image data generated by the image sensor when the binning setting is activated. Hereinafter, for convenience of explanation, the high-resolution image data generated in S25 may be referred to as first image data, the low-resolution image data generated in S27 may be referred to as second image data, and the low-resolution image data generated in S23 may be referred to as third image data.

[0067] The image sensor can send the first image data to the third image data generated in S22 to S27 to the processor, and each of the first image data to the third image data sent to the processor can be a single frame. The processor can perform a deblurring process using the first image data to the third image data received from the image sensor (S28). In the deblurring process, the processor can remove the blur present in the high-resolution first image data with reference to the second image data and the third image data.

[0068] For example, the processor can calculate a motion vector representing the movement of an object between the time point of capturing the object for generating the second image data and the time point of capturing the object for generating the third image data with reference to the second image data and the third image data. Based on the calculated motion vector, the processor can perform a deblurring process to remove the blurring phenomenon that appears on the object in the first image data.

[0069] The second image data and the third image data can be used to determine the movement of the object and can thus be generated as black-and-white image data, which is different from the first image data that is RGB image data. In addition, according to an embodiment, the second image data and the third image data can be generated before the first image data. The processor can determine the movement of the object based on the image data generated before the first image data and / or based on the image data generated after the first image data, and can perform a deblurring process for correcting the jitter of the object according to the first image data with reference to the image data generated before and / or after the first image data.

[0070] When the video mode is selected as the camera function, the image sensor can output image data having a predetermined number of frames per second (FPS). For example, the image data output by the image sensor in the video mode can include dozens of frames or more per second.

[0071] When it is confirmed that the video mode has been selected (S29), the image sensor can generate first image data having a first resolution without the binning setting (S30). The first image data generated in S30 can include a plurality of frames, and the resolution of each of the plurality of frames can be the first resolution. When generating the first image data, the binning setting can be activated in the image sensor (S31).

[0072] When binning is set, the image sensor can use pixel data obtained from two or more adjacent pixels to generate binned pixel data. As described above, the binned pixel data can be data corresponding to the binned pixel defined by two or more adjacent pixels. The image sensor can generate second image data having a second resolution lower than the first resolution using the binned pixel data (S32).

[0073] The second image data can be image data generated by the image sensor in a state where binning is set. Therefore, the second image data having a relatively low resolution can be generated at a higher number of frames per second than the first image data having a high resolution.

[0074] The processor can receive the first image data and the second image data from the image sensor, and can upscale the first image data using the second image data (S33). Through the upscaling operation in S33, the number of frames per second of the first image data can be increased. Therefore, in the video mode, the processor can produce a resultant image having a high resolution and a high number of frames per second.

[0075] When deblurring is completed in the capture mode or upscaling can be completed in the video mode, the processor can output the resultant image (S34). In the electronic device according to the embodiment, the processor can use the image data generated by the image sensor at different resolutions using one pixel array to correct the blur of an object appearing in a still image or increase the number of frames per second. Therefore, in an image sensor including one pixel array, a high-quality resultant image can be provided to the user.

[0076] Figure 8 and Figure 9 are views showing the operation of an image sensor according to one or more example embodiments.

[0077] Reference Figure 8 and Figure 9, the image sensor 100 according to an embodiment may include a pixel array 110, a row driver 120, a readout circuit 130, and a control logic 140. Figure 8 is a view showing the operation of the image sensor 100 when the binning setting is deactivated. Figure 9 is a view showing the operation of the image sensor 100 when the binning setting is activated.

[0078] First, referring to Figure 8 , a plurality of red pixels 111, a plurality of green pixels 112, and a plurality of blue pixels 113 may be placed in the pixel array 110. The arrangement of the pixels (e.g., 111 to 113) is not limited to Figure 8 the arrangement shown in, and may be modified in various ways according to the embodiment. The row driver 120 may be connected to the plurality of pixels (111 to 113) through a plurality of row lines ROW1 to ROW6, and the readout circuit 130 may be connected to the plurality of pixels (111 to 113) through a plurality of column lines COL1 to COL6.

[0079] Each of the plurality of pixels (111 to 113) may include a photodiode and a pixel circuit, and the pixel circuit may include at least one transistor. According to an embodiment, each of the row lines ROW1 to ROW6 may include a plurality of control signal lines for transmitting signals for controlling the transistors included in the pixel circuit.

[0080] When the binning setting is not activated, the row driver 120 may sequentially drive the plurality of row lines ROW1 to ROW6, and the readout circuit 130 may obtain a pixel signal corresponding to the difference between the reset voltage and the pixel voltage from the plurality of pixels (111 to 113). The readout circuit 130 may convert the pixel signal into pixel data, and may output the pixel data, and the pixel data may be digital data. Therefore, in Figure 8 the embodiment where the binning setting is not activated, the first image data output by the image sensor 100 may have a first resolution corresponding to the number of pixels (111 to 113) arranged in the pixel array 110.

[0081] When the binning setting is activated, as shown in Figure 9 , two or more pixels (111 to 113) adjacent to each other in the pixel array 110 may be defined as a binned pixel 115. For example, the binned pixel 115 may be defined by a plurality of pixels arranged in an M×M form, and in the embodiment shown in Figure 9 , may be defined by four pixels (111 to 113) adjacent to each other in a 2×2 form. The number and arrangement of the pixels included in the binned pixel 115 may vary according to the embodiment. For example, the binned pixel 115 may be defined to include 16 pixels arranged in a 4×4 form.

[0082] In a state where the merging setting is activated, the row driver 120 may drive the pixel array 110 in units of merged pixels 115. In the embodiment described in Figure 9 , the row driver 120 may select and drive the merged pixels 115 arranged along the first row using the first row line ROW1 and the second row line ROW2.

[0083] The readout circuit 130 may obtain merged pixel data from the selected merged pixels 115. For example, the row driver 120 may drive the merged pixels 115 in a charge merging method using control signals respectively input to the first row line ROW1 and the second row line ROW2. In the charge merging method, charges generated in each of the four pixels (111 to 113) included in each of the merged pixels 115 arranged in the first row are simultaneously accumulated in the floating diffusion. The readout circuit 130 may generate merged pixel data determined by the charges generated in the four pixels (111 to 113) included in each of the merged pixels 115.

[0084] Since one merged pixel 115 may be defined by four adjacent pixels (111 to 113), the second image data output by the image sensor 100 when the merging setting is activated has a second resolution, which may be lower than the first resolution of the first image data corresponding to the number of pixels (111 to 113) arranged in the pixel array 110. Since each of the merged pixels 115 has a relatively large area compared to each of the pixels (111 to 113), the exposure time of each of the merged pixels 115 may be set shorter than the exposure time of each of the pixels (111 to 113). Therefore, the second image data output by the image sensor 100 when the merging setting is activated may have a higher frames per second than the first image data output by the image sensor 100 when the merging setting is deactivated.

[0085] A processor connected to the image sensor 100 may receive the first image data and the second image data from the image sensor 100. The processor may include a neural engine, an image signal processor, etc. for image processing, and may perform post-processing work to increase the frames per second of the first image data by using the second image data having a higher frames per second than the first image data, correct the jitter of an object appearing in the first image data, etc. Therefore, by using only the image sensor 100 including one pixel array 110 in which the same type of pixels (111 to 113) are arranged, a high-quality result image may be provided to the user.

[0086] Figures 10 to 12 is a view showing the operation of an electronic device according to one or more example embodiments.

[0087] Figure 10 is a view schematically showing first image data 210 output from an image sensor and received by a processor, and Figure 11 is a view schematically showing second image data 220 output from an image sensor and received by a processor. The first image data 210 and the second image data 220 may be data output from an image sensor operating in a video mode. The first image data 210 and the second image data 220 may be data generated by the image sensor, and the image sensor may include only one pixel array.

[0088] First, referring to Figure 10 , the first image data 210 may include a plurality of first image frames 211 to 214. One frame of the plurality of first image frames 211 to 214 may be output per first time period T1, and the first frames per second may be determined by the first time period T1, which may be the frames per second of the first image data 210. Each of the plurality of first image frames 211 to 214 may include first pixels 215, and a first resolution that may be the resolution of the first image data 210 may be determined by the number of the plurality of first pixels 215.

[0089] Next, referring to Figure 11 , the second image data 220 may include a plurality of second image frames 221 to 224, and one frame of the plurality of second image frames 221 to 224 may be output per second time period T2. The second time period T2 may be shorter than the first time period T1, and thus, a second frames per second of the second image data 220 may be higher than the first frames per second of the first image data 210. Since the second image data 220 may be image data generated in a state where a merging setting is activated, a size of corresponding second pixels 225 included in the second image data 220 may be larger than a size of the corresponding first pixels 215. Therefore, a second resolution that may be the resolution of the second image data 220 may be lower than the first resolution of the first image data 210.

[0090] A processor connected to the image sensor may receive the first image data 210 and the second image data 220, and may increase the frames per second of the first image data 210 based on the second image data 220 to generate a result image 200. Figure 12 is a view schematically showing the result image 200 generated by the processor. Referring to Figure 12 , the result image 200 may include a plurality of frames 201 to 203 and 211 to 214, and a time interval between the plurality of frames 201 to 203 and 211 to 214 may be shorter than the first time period T1.

[0091] In addition to the plurality of first frames 211 to 214 included in the first image data 210, the resultant image 200 may further include additional frames 201 to 203 inserted by the processor. For example, one or two or more of the additional frames 201 to 203 may be inserted between a pair of the plurality of first frames 211 to 214 output in sequence at intervals of a first time period T1. Accordingly, the resultant image 200 may have a higher number of frames per second than the first image data 210.

[0092] In an embodiment, the resultant image 200 may have a resolution equal to a first resolution of the first image data 210. Referring Figure 12 , each of the plurality of frames 201 to 203 and 211 to 214 included in the resultant image 200 may include a plurality of pixels 205, and a size of each of the plurality of pixels 205 may be equal to a size of each of the plurality of first pixels 215 included in the first image data 210. Accordingly, a high-quality resultant image having a high resolution and a high number of frames per second may be obtained by using the electronic device.

[0093] According to an embodiment, the second pixels 225 included in the second image data 220 may include event information according to movement of an object or the like. For example, the event information may include whether an event has occurred, and in an embodiment, the merged pixel data obtained from each of the merged pixels may be converted to generate second pixel data corresponding to each of the second pixels 225.

[0094] For example, the first pixel data corresponding to each of the first pixels 215 may be RGB data, and the second pixel data corresponding to each of the second pixels 225 may be event data. Accordingly, the number of bits of the first pixel data may be greater than the number of bits of the second pixel data. In an embodiment, the image sensor may calculate a difference between a pair of merged pixel data corresponding to each of the merged pixels in a pair of consecutive second image frames 211 to 214 among the plurality of second image frames 211 to 214 to obtain the second pixel data for each of the second pixels 225.

[0095] To calculate the difference between a pair of merged pixel data corresponding to each of the merged pixels in a pair of second image frames 211 to 214, the image sensor may include a memory that stores the merged pixel data generated for each of the merged pixels. The memory may be implemented with DRAM, SRAM, or the like, may be formed on a wafer separate from the image sensor, and may then be bonded to the image sensor.

[0096] In an embodiment, the merged pixel data obtained from each of the merged pixels may be converted into a log scale value, and the difference in the log scale values between a pair of consecutive second image frames 211 to 214 among the plurality of second image frames 211 to 214 may be calculated. The image sensor may compare the calculated difference with a predetermined threshold to determine the second pixel data of each of the second pixels 225, and may generate second image data 210 based on the comparison result.

[0097] For example, for the second pixels 225 with a difference less than the threshold, the image sensor may determine the second pixel data as 0, and for the second pixels 225 with a difference greater than the threshold, the image sensor may determine the second pixel data as 1. Accordingly, the number of bits of the second pixel data corresponding to each of the second pixels 225 may be less than the number of bits of the first pixel data corresponding to each of the first pixels 215. An example of a method for generating the second image data 220 will be described later with reference to Figure 16 an example of a method for generating the second image data 220 will be described later with reference to

[0098] Figures 13 to 15 is a view showing the operation of an electronic device according to one or more example embodiments.

[0099] Figures 13 to 15 may be a view showing the operation of the electronic device in a capture mode for taking a still image. First, with reference to Figure 13 When the shooting function is selected in the capture mode, the image sensor may generate output data 300 including a plurality of image data 301 to 303. In an embodiment, each of the plurality of image data 301 to 303 may include one frame.

[0100] The first image data 301 may be image data obtained by imaging an object at a high resolution. The first image data 301 may be generated using the pixel data obtained from each of the plurality of pixels without a merging setting. In the operation of generating the first image data 301, each of the plurality of pixels may receive light and may generate charge during a first exposure time EIT1.

[0101] The second image data 302 and the third image data 303 may be data generated by the image sensor in a state where the merging setting is activated. For example, the image sensor may define merged pixels by merging two or more adjacent pixels arranged in a pixel array, and may generate the second image data 302 and the third image data 303 using the merged pixel data obtained from each of the merged pixels. Accordingly, the resolution of the second image data 302 and the resolution of the third image data 303 may be lower than the resolution of the first image data 301.

[0102] In the operations of generating the second image data 302 and the third image data 303, each of the plurality of merged pixels may receive light within a second exposure time EIT2 shorter than the first exposure time EIT1 to generate charges. The third image data 303 may be generated earlier than the first image data 301, and the second image data 302 may be generated later than the first image data 301. However, this is merely an example, and the present disclosure is not limited thereto.

[0103] The image sensor may output the first image data 301 to the third image data 303 to the processor. The processor may identify an object in each of the second image data 302 and the third image data 303, and may generate a motion vector based on the movement of the object. Depending on the magnitude of the motion vector representing the movement of the object between the second image data 302 and the third image data 303, a blurring phenomenon may occur in the first image data 301 where the object is not accurately represented. The processor may perform a deblurring process based on the motion vector to compensate for the jitter of the object, and thus may provide a high-quality result image to the user.

[0104] Figure 14 It may be a view showing a method in which a processor generates a motion vector based on image data generated earlier and later than the first image data. Refer to Figure 14 , in the image data captured earlier than the first image data, an object such as a first edge 401 (represented by a solid line) may be identified. In the image data captured later than the first image data, an object such as a second edge 402 (represented by a dashed line) may be identified. Each of the image data captured earlier and later than the first image data may be a frame, and may be generated by an image sensor in which the binning setting is activated. Therefore, the pixel 405 may correspond to a merged pixel in which the binning setting is activated.

[0105] The processor may calculate a motion vector 403 indicating the movement from the first edge 401 to the second edge 402, and may perform a deblurring operation based on this to remove the jitter of the object included in the first image data. The deblurring operation may be performed in various ways, and for example, the deconvolution operation may be performed by using the motion vector and the region identified as indicating the object in the first image data to compensate for the jitter of the object in the first image data.

[0106] Figure 15It may be a view showing an example in which image data 503 and 502 captured earlier and later than the first image data 501 are used to compensate for the jitter of an object appearing in the first image data 501. Different from the first image data 501 captured for a relatively long exposure time, little jitter of the object may appear in the second image data 502 and the third image data 503 captured for a short exposure time.

[0107] The processor may identify an object in each of the second image data 502 and the third image data 503, and may generate a motion vector indicating the movement between them. The motion vector may be used to compensate for the jitter in the object of the first image data 501, and thus a result image 500 after the deblurring process is completed may be generated, as Figure 15 shown. The resolution of the result image 500 may be equal to the resolution of the first image data 501.

[0108] Figure 16 It is a flowchart showing the operation of an electronic device according to one or more example embodiments.

[0109] An electronic device according to an embodiment may include an image sensor and a processor. When the video mode is selected in the camera function of the electronic device, the image sensor may generate first image data having a first resolution and a first number of frames per second, generate second image data having a second resolution and a second number of frames per second, and send the first image data and the second image data to the processor. For example, the image sensor may sequentially send the first image data and the second image data to the processor. The first resolution may be higher than the second resolution, and the first number of frames per second may be lower than the second number of frames per second.

[0110] The processor may perform an up-conversion operation to increase the number of frames per second of the first image data, thereby generating a high-quality result image having a high resolution and a high number of frames per second. The processor may increase the number of frames per second of the first image data with reference to the second image data, and will refer to Figure 16 A method of generating the second image data will be described in more detail.

[0111] Refer to Figure 16, the image sensor of the electronic device according to the embodiment may obtain merged pixel data from each of the plurality of merged pixels (S40). Each of the plurality of merged pixels may be defined as including two or more adjacent pixels among the plurality of pixels included in the pixel array of the image sensor. In an embodiment, the image sensor may simultaneously drive two or more pixels included in each of the plurality of merged pixels to obtain merged pixel data from each of the merged pixels. The merged pixel data may be N-bit data (N is a natural number of 2 or greater) output by the readout circuit of the image sensor, and may have the same number of bits as the pixel data obtained by the readout circuit from each pixel in a state where the merging setting is deactivated.

[0112] The image sensor may use the merged pixel data to generate a plurality of image frames at a second frame rate per second, and may store the plurality of image frames in a memory (S41). The memory may be implemented as DRAM, SRAM, etc., and according to an embodiment, may be implemented as a resistive memory such as RRAM, PRAM, FeRAM, etc.

[0113] The image sensor may convert the merged pixel data into a logarithmic scale value in each of the plurality of image frames stored in the memory (S42). The image sensor may calculate the difference between the logarithmic scale values corresponding to each of the merged pixels in a consecutive pair of frames (S43).

[0114] For example, when the second image data includes a consecutive first frame and a second frame, the image sensor may calculate the difference between the logarithmic scale value obtained by converting the merged pixel data of the first merged pixel of the first frame and the logarithmic scale value obtained by converting the merged pixel data of the first merged pixel of the second frame. The operations described above may be applied to each of the merged pixels, and the difference between the logarithmic scale values may be calculated and compared with a threshold value (S44).

[0115] By comparing the difference between the logarithmic scale value and the threshold value, the image sensor may select, among the merged pixels included in each of the plurality of frames, the merged pixels in which an event has occurred. For example, the merged pixels in which the difference between the logarithmic scale values is greater than the threshold value may be determined as the merged pixels in which an event has occurred, and the merged pixels in which the difference between the logarithmic scale values is less than the threshold value may be determined as the merged pixels in which an event has not occurred.

[0116] The image sensor may assign different values to the merged pixels with a difference in logarithmic scale values greater than a threshold and the merged pixels with a difference in logarithmic scale values less than the threshold, thereby generating second image data at a second frames per second (S45). For example, a first digital value may be assigned to the merged pixels with a difference in logarithmic scale values greater than the threshold, and a second digital value complementary to the first digital value may be assigned to the merged pixels with a difference in logarithmic scale values less than the threshold. According to the operations described above, as above, an event-based image rather than a general two-dimensional image may be generated from a plurality of merged pixels. Therefore, each of the plurality of frames included in the second image data may be generated as an event-based image.

[0117] The characteristics of each of the plurality of frames included in the second image data may be changed according to the threshold applied in S44. For example, when the threshold is decreased, the number of merged pixels in which the occurrence of an event is recognized may increase, and the influence of noise may increase. When the threshold is increased, the influence of noise may be reduced, but the number of merged pixels in which the occurrence of an event is recognized may decrease, so that the contour or movement of an object in the event-based image cannot be accurately tracked. Therefore, it may be important to select an appropriate threshold.

[0118] After generating the first image data at a high resolution and a low frames per second, the image sensor may generate second image data including frames as event-based images, as in the embodiments described in the reference Figure 16 Therefore, in the frame unit, the second image data may have a relatively small data size compared to the first image data. The processor may up-convert the frames per second of the first image data with reference to the second image data, and may generate a high-quality result image at a high resolution and a high frames per second.

[0119] Figure 17 and Figure 18 are views showing output data of an image sensor according to one or more example embodiments.

[0120] Reference Figure 17 shows that output data 600 output from the image sensor and sent to the processor may include first image data 601 and second image data 602. In an embodiment, the first image data 601 and the second image data 602 may be output alternately. The plurality of first frames included in the first image data 601 may each have a first resolution, and the plurality of first frames may be generated at a first frames per second. Data corresponding to one pixel in each of the plurality of first frames may be data of N bits or more, for example, data of 8 bits or more.

[0121] The second image data 602 may include a plurality of second frames, and each of the plurality of second frames may have a second resolution lower than the first resolution. Each of the plurality of second frames may be an event-based image generated according to the method described above with reference to Figure 16 Thus, the data corresponding to one pixel in each of the plurality of second frames may be data less than N bits, for example, 1 or 2 bits of data.

[0122] The plurality of second frames may be generated at a second frames per second rate faster than the first frames per second rate. For example, the first frames per second rate may be 60 fps, and the second frames per second rate may be 960 fps. For example, the first exposure time set to generate each of the plurality of first frames may be longer than the second exposure time set to generate each of the plurality of second frames.

[0123] According to a merging operation performed earlier than the generation of the second image data 602, the difference between the first resolution and the second resolution may be changed. For example, when one merged pixel is defined by 16 pixels arranged adjacent to each other in a 4×4 form, the first resolution may be 16 times the second resolution.

[0124] The processor may refer to the second image data 602 to increase the frames per second of the first image data 601 from the first frames per second to the second frames per second. For example, the frames per second of the first image data output by the image sensor immediately before the second image data 602 may be increased with reference to the second image data 602.

[0125] Depending on the interface connecting the image sensor and the processor, it may not be possible for the image sensor to send image data including frames generated at the first resolution and the second frames per second rate to the processor. In an embodiment, when the image sensor sends the first image data with a high resolution but a low frames per second rate and the second image data with a low resolution but a high frames per second rate and sufficient to track the movement of an object to the processor, the processor may refer to the second image data to increase the frames per second of the first image data to generate a resulting image. Thus, a high-quality resulting image with a high resolution and a high frames per second rate can be generated regardless of the data transfer rate between the image sensor and the processor.

[0126] Referring to Figure 18 , the output data 610 sent from the image sensor to the processor may include the first image data 611 and the second image data 612. In an embodiment, the first image data 611 and the second image data 612 may be output alternately, and additional image data 613 may be output therebetween. The first image data 611 and the second image data 612 may be similar to the first image data 601 and the second image data 602 described above with reference to Figure 17 .

[0127] The additional image data 613 may be a frame. The additional image data 613 may have the same resolution as the second image data and may be generated at the same first frames per second as the first image data 611. The additional image data 613 may be a general two-dimensional image of an object rather than an event-based image.

[0128] Figure 19 and Figure 20 are views showing operations of an electronic device according to one or more example embodiments.

[0129] In Figure 19 and Figure 20 shown example embodiments, the first images 700 and 710 may be original images obtained by imaging an object with an image sensor, and the second to fourth images 701 to 703 and 711 to 713 may be event-based images respectively generated from the original images 700 and 710. For example, pixel data corresponding to pixels in the original images 700 and 710 may be converted to logarithmic scale values, differences between the logarithmic scale values obtained by converting pixel data of the immediately preceding frames of the original images 700 and 710 may be calculated, and the calculated differences may be compared with a threshold to generate the event-based images 701 to 703 and 711 to 713.

[0130] As Figure 19 and Figure 20 shown, the first to third event-based images 701 to 703 and 711 to 713 may each be generated by comparing the difference calculated for the logarithmic scale value with different thresholds. First, referring to Figure 19 , the first threshold applied to the first event-based image 701 may be 48%, the second threshold applied to the second event-based image 702 may be 74%, and the third threshold applied to the third event-based image 703 may be 104%. Next, referring to Figure 20 , the first threshold applied to the first event-based image 711 may be 27%, the second threshold applied to the second event-based image 712 may be 70%, and the third threshold applied to the third event-based image 713 may be 99%.

[0131] As explained with reference to Figure 19 and Figure 20 , as the threshold increases, the influence of noise may decrease, but the contrast of the edges representing the contour of the object may also decrease, making it possible that the object may not be accurately recognized. As the threshold decreases, the influence of noise may increase, such that the edges representing the contour of the object may not be accurately recognized. An appropriate threshold may be changed according to the object and the background, and an optimal threshold may be determined according to the object and the background.

[0132] Figure 21 is a view schematically showing an image sensor according to one or more example embodiments.

[0133] Referring Figure 21 , the image sensor 800 may include a first layer 810, a second layer 820, and a third layer 830 stacked in sequence. The first layer 810 may include a pixel array 811, and the second layer 820 may include logic circuitry 821. The pixel array 811 may also be connected to the logic circuitry 821 through a plurality of row lines and a plurality of column lines.

[0134] The third layer 830 may include a memory element array 831, and a memory (such as DRAM, SRAM, FeRAM, RRAM, PRAM, etc.) may be placed in the memory element array 831. The circuitry for controlling the memory may be placed in the third layer 830 together with the memory element array 831, or the circuitry for controlling the memory may be placed in the second layer 820 together with the logic circuitry 821.

[0135] The logic circuitry 821 may obtain pixel data from pixels arranged in the pixel array 811 in a state where the merging setting is deactivated to generate first image data. Additionally, the logic circuitry 821 may activate the merging setting to define merged pixels using the pixels of the pixel array 811, and may generate second image data using the merged pixel data obtained from the merged pixels.

[0136] The logic circuitry 821 may reduce the size of the merged pixel data to generate second image data. For example, the logic circuitry 821 may obtain merged pixel data while driving the pixel array 811 at a high frames per second to generate a plurality of frames, and the merged pixel data constituting the plurality of frames may first be stored in the memory element array 831. The logic circuitry 821 may convert the merged pixel data stored in the memory element array 831 into logarithmic scale values.

[0137] The logic circuitry 821 may calculate the difference between a pair of logarithmic scale values corresponding to a pair of merged pixels included in a pair of consecutive frames among the plurality of frames, and may compare the difference with a threshold value to generate second image data having a small size. For example, when the calculated difference is greater than the threshold value, the pixel value of the corresponding merged pixel may be defined as '1', and when the calculated difference is less than the threshold value, the pixel value of the corresponding merged pixel may be defined as '0'. Through this operation, the data of each of the merged pixels included in each of the plurality of frames included in the second image data can be reduced to 1 bit. Accordingly, the image sensor 800 may output second image data having a high frames per second to an external processor.

[0138] Figure 22 is a view schematically showing a processor connected to an image sensor according to one or more example embodiments.

[0139] Referring Figure 22 , a processor 900 according to an embodiment may include an interface 910 configured to receive image data from an externally located image sensor, an image signal processor 920 configured to process the image data to generate a resultant image, a neural processor unit 930, a central processing unit 940, etc. In an embodiment, the interface 910 may receive the image data from the image sensor communicatively according to the Mobile Industry Processor Interface (MIPI) standard.

[0140] According to the operation mode of the image sensor, the interface 910 may receive image data having different frames per second and resolutions. For example, the interface 910 may receive first image data having a high resolution and a low frames per second from the image sensor, and second image data having a low resolution and a high frames per second.

[0141] When the image sensor operates in a capture mode, the image signal processor 920 may generate a motion vector representing the movement of an object based on the second image data, and may perform deblurring on the first image data using the motion vector. When the image sensor operates in a video mode, the image signal processor 920 may up-convert the frames per second of the first image data using the second image data. For example, to up-convert the frames per second of the first image data, the image signal processor 920 may use the neural processor unit 930.

[0142] Figure 23 is a view schematically showing an electronic device including an image sensor according to one or more example embodiments.

[0143] Referring Figure 23 , an electronic device 1000 may include a camera module group 1100, an application processor 1200, a power management integrated circuit (PMIC) 1300, and an external memory 1400.

[0144] The camera module group 1100 may include a plurality of camera modules 1100a, 1100b, and 1100c. In Figure 23In the illustrated embodiments, although three (3) camera modules 1100a, 1100b, and 1100c are shown arranged, the embodiments are not limited thereto. In some embodiments, the camera module group 1100 may be modified to include only two (2) camera modules. In some embodiments, the camera module group 1100 may be modified to include n camera modules (where n is a natural number of 4 or greater). In an embodiment, at least one of the plurality of camera modules 1100a, 1100b, and 1100c included in the camera module group 1100 may include an image sensor according to one of the one or more exemplary embodiments described above with reference to Figures 1 to 21 One of the image sensors of the described one or more example embodiments.

[0145] In an embodiment, at least two of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., 1100a, 1100b) may have different fields of view (or viewing angles). In this case, for example, the optical lenses of at least two of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., 1100a and 1100b) may be different from each other, but the present disclosure is not limited thereto.

[0146] In some embodiments, the viewing angle of each of the plurality of camera modules 1100a, 1100b, and 1100c may be different. In this case, the optical lenses included in each of the plurality of camera modules 1100a, 1100b, and 1100c may also be different from each other, but the present disclosure is not limited thereto.

[0147] In some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may be arranged to be physically separated from each other. For example, instead of dividing the sensing area of one image sensor into the plurality of camera modules 1100a, 1100b, and 1100c, an independent image sensor may be placed within each of the plurality of camera modules 1100a, 1100b, and 1100c.

[0148] Referring to Figure 23 , the application processor 1200 may include an image signal processor 1210, a memory controller 1220, an internal memory 1230, etc. The application processor 1200 may be implemented separately from the plurality of camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the plurality of camera modules 1100a, 1100b, and 1100c may be implemented separately from each other as separate semiconductor chips.

[0149] The image signal processor 1210 may include a plurality of sub-processors 1212a, 1212b, and 1212c, an image generator 1214, and a camera module controller 1216.

[0150] The image signal processor 1210 may include a plurality of sub-processors 1212a, 1212b, and 1212c corresponding to the plurality of camera modules 1100a, 1100b, and 1100c.

[0151] The image data generated from each of the camera modules 1100a, 1100b, and 1100c may be provided to the corresponding sub-image processors 1212a, 1212b, and 1212c through separate image signal lines ISLa, ISLb, and ISLc. For example, the image data generated from the first camera module 1100a may be provided to the first sub-processor 1212a through the first image signal line ISLa, the image data generated from the second camera module 1100b may be provided to the second sub-processor 1212b through the second image signal line ISLb, and the image data generated from the third camera module 1100c may be provided to the third sub-processor 1212c through the third image signal line ISLc. Such transmission of image data may be performed using, for example, a camera serial interface (CSI) based on the Mobile Industry Processor Interface (MIPI), but embodiments are not limited thereto.

[0152] In an embodiment, the sub-processors may be placed to correspond to two or more camera modules. For example, as shown in the figure, the first sub-processor 1212a and the third sub-processor 1212c may not be implemented as separate from each other, but may be implemented as integrated into a single sub-image processor, and the image data provided from the camera module 1100a and the camera module 1100c may be selected by a selection element (e.g., a multiplexer) or the like and then provided to the integrated sub-image processor.

[0153] The image data provided to each of the sub-processors 1212a, 1212b, and 1212c may be provided to the image generator 1214. The image generator 1214 may use the image data provided from each of the sub-processors 1212a, 1212b, and 1212c according to image generation information or a mode signal to generate an output image.

[0154] In an embodiment, the image generator 1214 may combine at least a part of the image data generated from the camera modules 1100a, 1100b, and 1100c having different perspectives according to image generation information or a mode signal to generate an output image. Additionally, the image generator 1214 may generate an output image by selecting any one of the image data generated from the camera modules 1100a, 1100b, and 1100c having different perspectives according to image generation information or a mode signal.

[0155] In an embodiment, the image generation information may include a zoom signal or a zoom factor. Additionally, in an embodiment, the mode signal may be a signal based on a user-selected mode, for example.

[0156] When the image generation information is a zoom signal (e.g., a zoom factor) and each of the camera modules 1100a, 1100b, and 1100c has a different field of view (e.g., viewing angle), the image generator 1214 may operate differently according to the type of the zoom signal. For example, when the zoom signal is a first signal, after combining the image data output from the camera module 1100a and the image data output from the camera module 1100c, the combined image signal and the image data output from the camera module 1100b (not used in the combination) may be used to generate an output image. When the zoom signal is a second signal different from the first signal, the image generator 1214 may not perform such image data combination and may select any one of the image data output from each of the camera modules 1100a, 1100b, and 1100c to create an output image. The embodiment is not limited thereto, and the method of processing image data may be modified and executed as needed.

[0157] In an embodiment, the image generator 1214 may receive multiple pieces of image data having different exposure times from at least one of the multiple sub-image processors 1212a, 1212b, and 1212c, and may process high dynamic range (HDR) with respect to the multiple pieces of image data to generate combined image data with an increased dynamic range.

[0158] The camera module controller 1216 may provide control signals to each of the camera modules 1100a, 1100b, and 1100c. The control signals generated from the camera module controller 1216 may be provided to the corresponding camera modules 1100a, 1100b, and 1100c through separate control signal lines CSLa, CSLb, and CSLc.

[0159] According to the image generation information including a zoom signal or a mode signal, any one of the multiple camera modules 1100a, 1100b, and 1100c may be designated as a main camera (e.g., 1100b), and the remaining camera modules (e.g., 1100a and 1100c) may be designated as slave cameras. Such information may be included in the control signal and may be provided to the corresponding camera modules 1100a, 1100b, and 1100c through separate control signal lines CSLa, CSLb, and CSLc.

[0160] The camera modules operating as the master and slave devices can be changed according to the zoom factor or the operation mode signal. For example, when the viewing angle of the camera module 1100a is wider than that of the camera module 1100b and the zoom factor indicates a low zoom magnification, the camera module 1100b can operate as the master device, and the camera module 1100a can operate as the slave device. When the zoom factor indicates a high zoom magnification, the camera module 1100a can operate as the master device, and the camera module 1100b can operate as the slave device.

[0161] In some embodiments, the control signals provided from the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c may include a synchronization enable signal. For example, when the camera module 1100b is the main camera and the camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 may send a synchronization enable signal to the camera module 1100b. The camera module 1100b that receives such a synchronization enable signal may generate a synchronization signal based on the synchronization enable signal and may send the generated synchronization signal to the camera modules 1100a and 1100c through the synchronization signal line SSL. The camera module 1100b and the camera modules 1100a and 1100c may be synchronized with the synchronization signal to send image data to the application processor 1200.

[0162] In some embodiments, the control signals provided from the camera module controller 1216 to the plurality of camera modules 1100a, 1100b, and 1100c may include mode information according to the mode signal. Based on this mode information, the plurality of camera modules 1100a, 1100b, and 1100c may operate in a second operation mode and a first operation mode related to the sensation rate.

[0163] In the first operation mode, at least one of the plurality of camera modules 1100a, 1100b, and 1100c may generate first image data at a first resolution and a first number of frames per second, and generate second image data at a second resolution and a second number of frames per second. The first resolution may be higher than the second resolution, and the first number of frames per second may be lower than the second number of frames per second.

[0164] The pixel value corresponding to each pixel in a separate frame included in the first image data may be RGB data, and the pixel value corresponding to each pixel in a separate frame of the second image data may be event data. The number of bits of the RGB data may be greater than the number of bits of the event data. Additionally, due to the merging setting, the size of each pixel included in a separate frame of the first image data may be smaller than the size of each pixel included in a separate frame of the second image data. Thus, in an embodiment, an image sensor including the same type of pixels may be used to generate the first image data including RGB data and the second image data including event data.

[0165] The sub-processors 1212a, 1212b, and 1212c of the application processor 1200 and the image generator 1214 may perform de-blurring on the first image data using the second image data, or increase the number of frames per second of the first image data to generate a result image.

[0166] The PMIC 1300 may supply power, such as a power supply voltage, to each of the plurality of camera modules 1100a, 1100b, and 1100c. For example, under the control of the application processor 1200, the PMIC 1300 may supply first power to the camera module 1100a through the power signal line PSLa, supply second power to the camera module 1100b through the power signal line PSLb, and supply third power to the camera module 1100c through the power signal line PSLc.

[0167] The PMIC 1300 may generate power corresponding to each of the plurality of camera modules 1100a, 1100b, and 1100c in response to a power control signal PCON from the application processor 1200, and may also adjust the level of the power. The power control signal PCON may include power adjustment signals for each operation mode of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the operation mode may include a low-power mode. In this case, the power control signal PCON may include information about the camera module operating in the low-power mode and the level of the power to be set. The levels of the power supplied to each of the plurality of camera modules 1100a, 1100b, and 1100c may be the same as or different from each other. Additionally, the power level may be dynamically changed.

[0168] According to an embodiment, the logic circuit of the image sensor may obtain first image data having a first resolution from pixels arranged in a pixel array, and may obtain second image data having a second resolution lower than the first resolution from binned pixels that bin two or more adjacent pixels to each other. The exposure time of each of the binned pixels may be set to be shorter than the exposure time of each of the pixels, and thus the number of frames per second of the second image data may be higher than the number of frames per second of the first image data. The second image data may be used to track the movement of an object included in the first image data to compensate for jitter of the object in the first image data, or the number of frames per second of the first image data may be up-converted using the second image data to generate a high-quality video.

[0169] The various advantages and effects of the present disclosure are not limited to the above, and may be more easily understood during the process of describing the exemplary embodiments of the present disclosure.

[0170] Although the exemplary embodiments have been shown and described above, it will be apparent to those skilled in the art that modifications and variations can be made without departing from the scope of the present disclosure defined by the appended claims.

Claims

1. An image sensor, comprising: A pixel array including a plurality of pixels arranged along a first direction and a second direction, the second direction intersecting the first direction; And A logic circuit configured to drive the plurality of pixels, Wherein the logic circuit is further configured to: Generate first image data having a first resolution based on pixel data corresponding to each of the plurality of pixels, After generating the first image data, generate second image data having a second resolution based on combined pixel data corresponding to each of a plurality of combined pixels, the plurality of combined pixels combining two or more adjacent pixels among the plurality of pixels, the second resolution being lower than the first resolution; and Output the first image data and the second image data, and Wherein a first frames per second of the first image data is lower than a second frames per second of the second image data.

2. The image sensor according to claim 1, wherein, Each of the plurality of combined pixels corresponds to adjacent pixels among the plurality of pixels arranged in M×M, where M is a natural number of 2 or greater.

3. The image sensor according to claim 1, wherein, Alternately repeat a first time period during which the first image data is output and a second time period during which the second image data is output.

4. The image sensor according to claim 3, wherein, The logic circuit is further configured to output an image frame having the second resolution between the alternately repeated first time period and second time period.

5. The image sensor according to claim 4, wherein, The one image frame is generated at a rate corresponding to the first frames per second.

6. The image sensor according to claim 1, further comprising: A memory configured to store the combined pixel data, the combined pixel data corresponding to the intensity of light incident on each of the plurality of combined pixels and generated at the second frames per second, Wherein the logic circuit is further configured to obtain a difference between a pair of the combined pixel data corresponding to each of the plurality of combined pixels in a pair of consecutive frames to generate the second image data.

7. The image sensor according to claim 6, wherein, The logic circuit is further configured to convert the combined pixel data corresponding to each of the plurality of combined pixels into logarithmic scale values, and obtain a difference between the logarithmic scale values corresponding to each of the plurality of combined pixels in the pair of consecutive frames to generate the second image data.

8. The image sensor according to claim 7, wherein, The logic circuit is further configured to compare the difference of the logarithmic scale values with a predetermined threshold to generate the second image data.

9. The image sensor according to claim 8, wherein, The logic circuit is further configured to: for each of the plurality of combined pixels, set a pixel value to a first logical value based on the difference of the logarithmic scale values being greater than the predetermined threshold, and set the pixel value to a second logical value based on the difference of the logarithmic scale values being less than the predetermined threshold, and Wherein the first logical value and the second logical value are complementary to each other.

10. The image sensor according to claim 1, wherein, The logic circuit is further configured to set an exposure time of each of the plurality of pixels to be longer than an exposure time of each of the plurality of combined pixels.

11. The image sensor according to claim 1, wherein, The logic circuit is further configured to: before generating the first image data, generate third image data having the second resolution at the second frames per second, and wherein the logic circuit is further configured to sequentially output the third image data, the first image data, and the second image data.

12. An electronic device, comprising: an image sensor configured to sequentially output first image data having a first resolution and second image data having a second resolution, the second resolution being lower than the first resolution; and a processor configured to output a result image using the first image data and the second image data, wherein the image sensor includes a pixel array, the pixel array including a plurality of pixels having the same structure, and wherein the image sensor is further configured to generate the first image data based on pixel data obtained from each of the plurality of pixels exposed to light for a first exposure time, and generate the second image data based on combined pixel data obtained from each of a plurality of combined pixels, the plurality of combined pixels combining two or more adjacent pixels among the plurality of pixels, the two or more pixels being exposed to light for a second exposure time shorter than the first exposure time.

13. The electronic device according to claim 12, wherein, The first image data includes a plurality of first frames generated at a first frames per second, and the second image data includes a plurality of second frames generated at a second frames per second, the second frames per second being higher than the first frames per second, and wherein the processor is further configured to generate the result image by up-converting the first frames per second of the first image data based on the second image data.

14. The electronic device according to claim 13, wherein, The result image has the first resolution and the second frames per second.

15. The electronic device according to claim 12, wherein, The image sensor is further configured to output third image data having the second resolution before outputting the first image data, and wherein the processor is further configured to obtain a motion vector from the second image data and the third image data, and generate the result image based on the motion vector to compensate for movement of an object included in the first image data.

16. A processor, comprising: an interface configured to receive image data from an image sensor; and an image signal processor configured to process the image data to generate a result image, wherein the interface is configured to receive first image data having a first resolution and a first frames per second, and second image data having a second resolution and a second frames per second, wherein the first resolution is higher than the second resolution, and the first frames per second is slower than the second frames per second, and wherein the image signal processor is configured to process the first image data using the second image data to generate the result image.

17. The processor according to claim 16, wherein, The size of the first image data is larger than the size of the second image data.

18. The processor according to claim 16, wherein, The first image data includes a plurality of first frames, and the second image data includes a plurality of second frames, and Among them, each pixel in each of the plurality of first frames has a greater number of bits of data than each pixel in each of the plurality of second frames.

19. The processor according to claim 18, wherein, Each pixel in each of the plurality of first frames has RGB data, and each pixel in each of the plurality of second frames has event data.

20. The processor according to claim 16, wherein, The interface is configured to alternately receive the first image data and the second image data.

Images