Image sensor and merging method thereof

Abstract

A merging method for an image sensor, comprising: reading out a plurality of pixel signals from at least two rows of each of a plurality of regions of a pixel array at a time, each of the plurality of regions comprising a plurality of pixels arranged in a 2n×2n matrix, wherein n is an integer greater than or equal to 2; generating first image data by performing analog-to-digital conversion on the plurality of pixel signals; based on the first image data, generating a first sum value for each of a plurality of merging regions based on two pixel values ​​corresponding to the same color in each of a plurality of merging regions, the plurality of merging regions corresponding to the plurality of regions of the pixel array; and generating a second sum value for each of the two merging regions based on two first sum values ​​corresponding to the same color in the two merging regions, the two merging regions being adjacent to each other in a column direction among the plurality of merging regions.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2020-0078813 filed in the Korean Intellectual Property Office on June 26, 2020, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0003] Example embodiments of the present disclosure relate to image sensors, and more particularly, to a merging method of an image sensor and an image sensor for performing the method. Background Art

[0004] As the resolution of image sensors increases, the size of image data generated by the image sensors also increases. However, because the size of image data generated by the image sensors increases, it is difficult to maintain a high frame rate in video mode, and power consumption increases. For example, the frame rate can be a frame rate that can be achieved based on the circuit interface bandwidth and the computational bandwidth associated with avoiding artifacts such as zigzag artifacts or false colors when changing the sampling rate. This is a problem because the achievable frame rate is limited when the amount of computation required to avoid artifacts is high and / or the circuit interface bandwidth is not high. Binning techniques are used to increase the frame rate and maintain image quality. Summary of the invention

[0005] Example embodiments provide a merging method of an image sensor and an image sensor for performing the method, by which a frame rate is improved, a data size is reduced, and image quality is maintained.

[0006] According to one aspect of an example embodiment, a merging method of an image sensor is provided. The merging method includes: reading out a plurality of pixel signals from at least two rows of each of a plurality of regions of a pixel array at a time, each of the plurality of regions including a plurality of pixels arranged in a 2n×2n matrix, where n is an integer greater than or equal to 2; generating first image data by performing analog-to-digital conversion on the plurality of pixel signals; based on the first image data, generating a first sum value of each of the plurality of merging regions based on two pixel values ​​corresponding to the same color in each of the plurality of merging regions, the plurality of merging regions corresponding to the plurality of regions of the pixel array; and generating a second sum value of each of the two merging regions based on two first sum values ​​corresponding to the same color in the two merging regions, the two merging regions being adjacent to each other in a column direction in the plurality of merging regions.

[0007] According to one aspect of an example embodiment, there is provided an image sensor, comprising: a pixel array divided into a plurality of regions having a quadrilateral shape, each of the plurality of regions comprising pixels arranged in a 2nx2n matrix, wherein n is an integer greater than or equal to 2; an analog-to-digital conversion circuit configured to read out a plurality of pixel signals and convert the plurality of pixel signals into first image data, the first image data comprising a plurality of pixel values, and the plurality of pixel signals being received from the pixel array via a plurality of column lines; a row driver configured to provide a control signal via a plurality of row lines connected to the pixel array, the control signal being configured to control pixel signals of at least two rows of the pixel array to be output simultaneously; a row buffer configured to store the first image data in specific row units; and a processor configured to perform merging on the first image data stored in the row buffer.

[0008] According to one aspect of an example embodiment, there is provided an image processing system, comprising: an image sensor configured to sense a light signal and generate image data; and a first processor configured to receive and process the image data from the image sensor, wherein the image sensor comprises: a pixel array divided into a plurality of regions having a quadrilateral shape, each of the plurality of regions comprising pixels arranged in a 4x4 matrix; an analog-to-digital conversion circuit configured to read out a plurality of pixel signals and convert the plurality of pixel signals into first image data, the first image data comprising a plurality of pixel values, and the plurality of pixel signals being received from the pixel array via a plurality of column lines; a row driver configured to provide a control signal via a plurality of row lines connected to the pixel array, the control signal being configured to control pixel signals of at least two rows of the pixel array to be output simultaneously; a row buffer configured to store the first image data in specific row units; and a second processor configured to perform merging on the first image data stored in the row buffer. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 is a schematic block diagram of an image sensor according to an example embodiment;

[0011] Figure 2 Show Figure 1 Examples of patterns of pixel arrays in;

[0012] Figure 3A is a flow chart illustrating vertical analog summation and interpolation of an image sensor according to example embodiments;

[0013] Figure 3Bis a flow chart of a merging method of an image sensor according to an example embodiment;

[0014] Figure 4A and Figure 4B is a diagram for describing a readout method according to an example embodiment;

[0015] Figure 5A , Figure 5B and Figure 5C is a schematic diagram of first image data generated by a readout method according to an example embodiment;

[0016] Figure 6 A first merging performed on each of a plurality of merging regions of first image data based on a pixel value in each merging region in a merging method according to an example embodiment is shown;

[0017] Figure 7 illustrates interpolation performed based on pixel values ​​of two adjacent merging areas in a merging method according to an example embodiment;

[0018] Figure 8 is a flow chart of a merging method of an image sensor according to an example embodiment;

[0019] Fig. 9 is a diagram for describing a second merging applied to a green pixel in a merging method according to example embodiments;

[0020] Fig.10 shows an example of a pixel according to an example embodiment;

[0021] Fig.11A A pixel array having a quaternary pattern is shown; Fig. 11B An example of applying a pixel array having a quaternary pattern to an image sensor according to example embodiments is shown;

[0022] Fig.12 shows an example of a pixel according to an example embodiment;

[0023] Fig.13 is a block diagram of an electronic device including a multi-camera module using an image sensor according to example embodiments; and

[0024] Fig.14 yes Fig.13 Detailed block diagram of the camera module in Figure 1. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[0026] Figure 1 is a schematic block diagram of an image sensor according to example embodiments.

[0027] The image sensor 100 may be mounted on an electronic device having an image or optical sensing function. For example, the image sensor 100 may be mounted on an electronic device such as a camera, a smart phone, a wearable device, an Internet of Things (IoT) device, a home appliance, a tablet personal computer (PC), a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a drone, or an advanced driver assistance system (ADAS). The image sensor 100 may also be mounted on an electronic device used as a component of a vehicle, furniture, a manufacturing facility, a door, or various measuring devices.

[0028] refer to Figure 1 , the image sensor 100 may include a pixel array 110 , a row driver 120 , an analog-to-digital converter (ADC) circuit 130 , a ramp signal generator 140 , a timing controller 150 , a row buffer 160 , and a processor 170 .

[0029] The pixel array 110 may include a plurality of pixels PX in a matrix and a plurality of row lines RL and column lines CL connected to the pixels PX.

[0030] Each pixel PX may include at least one photoelectric conversion element (or photosensitive device). The photoelectric conversion element may sense light and convert the light into photocharges. For example, the photoelectric conversion element may include a photosensitive device, such as an inorganic photodiode, an organic photodiode, a perovskite photodiode, a phototransistor, a photogate, or a pinned photodiode, which includes an organic or inorganic material. In an embodiment, each pixel PX may include a plurality of photoelectric conversion elements. A plurality of photosensitive devices may be arranged in the same layer or stacked on each other in a vertical direction.

[0031] A microlens for light collection may be disposed above each pixel PX, or disposed above a pixel group including adjacent pixels PX. Each pixel PX may sense light in a specific spectrum from the light received through the microlens. For example, the pixel array 110 may include a red pixel that converts light in a red spectrum into an electrical signal, a green pixel that converts light in a green spectrum into an electrical signal, and a blue pixel that converts light in a blue spectrum into an electrical signal. A color filter that transmits light in a specific spectrum may be disposed above each pixel PX. However, embodiments are not limited thereto, and the pixel array 110 may include pixels that convert light in other spectrums other than the red, green, and blue spectrums.

[0032] In one embodiment, the pixel PX may have a multi-layer structure. Each pixel PX having a multi-layer structure may include stacked photosensitive devices, each of which converts light in a different spectrum into an electrical signal, so that electrical signals corresponding to different colors can be generated from the photosensitive device. In other words, electrical signals corresponding to different colors can be output from a single pixel PX.

[0033] The pixel array 110 may have a Bayer pattern in which first, second, and third pixels sense signals of different colors and are repeatedly arranged in a column direction and a row direction.

[0034] Figure 2 Show Figure 1 An example of a pattern of pixel array 110 in FIG.

[0035] refer to Figure 2 , the pixel array 110 may include a plurality of pixels PX arranged in a row direction (e.g., an X-axis direction) and a column direction (e.g., a Y-axis direction), and the plurality of pixels PX may include a red pixel PX_R, a green pixel PX_Gr and PX_Gb, and a blue pixel PX_B. In the pixel array 110, a row including a red pixel PX_R and a green pixel (e.g., a first green pixel PX_Gr) alternates with a row including another green pixel (e.g., a second green pixel PX_Gb) and a blue pixel PX_B; and the green pixel (e.g., the first green pixel PX_Gr and the second green pixel PX_Gb) may be on an oblique line. The green pixel (e.g., the first green pixel PX_Gr or the second green pixel PX_Gb) is closely related to brightness, and is therefore arranged in each row; and the red pixel PX_R and the blue pixel PX_B are alternately arranged on different rows.

[0036] This pattern may be referred to as an RGB Bayer pattern. In the following, it is assumed that the pixel array 110 has an RGB Bayer pattern. However, the embodiment is not limited thereto. Various patterns having the following structure may be applied to the pixel array 110: pixels of at least three colors are repeatedly arranged, and a second pixel (e.g., a pixel related to brightness) is arranged in each row and forms an oblique line with another second pixel of an adjacent row. For example, an RYYB pattern including one red pixel, two yellow pixels, and one blue pixel may be applied to the pixel array 110.

[0037] The pixel array 110 may be divided into a plurality of regions AR. Each region AR may include pixels PX arranged in a 2n×2n matrix, where “n” is an integer of at least 2. For example, each region AR may include pixels PX arranged in a 4×4 matrix. At this time, according to an embodiment, each region AR is a basic unit to which a readout method is applied when the image sensor 100 operates in a first mode that performs merging. The regions AR may respectively correspond to a plurality of merging regions of image data generated based on the readout signals. According to the readout method of an example embodiment, a plurality of pixel signals of at least two rows in each region AR may be read out at a time. For example, a plurality of pixel signals of at least two rows of pixels may be read out in a single horizontal period. Reference will be made to FIG. 4A to FIG. 5B A readout method of an example embodiment is described.

[0038] When the image sensor 100 operates in the second mode (eg, a normal mode in which binning is not performed), the pixel array 110 may sequentially read out pixel signals row by row.

[0039] Go back for reference Figure 1 Each row line RL may extend in a row direction and may be connected to pixels PX of one row. For example, each row line RL may transmit a control signal from the row driver 120 to a plurality of elements (eg, transistors) included in each pixel PX.

[0040] Each column line CL may extend in a column direction and may be connected to a column of pixels PX. Each column line CL may transmit pixel signals (e.g., reset signals and sensing signals) from pixels PX of each row of the pixel array 110 to the ADC circuit 130. When the image sensor 100 operates in the first mode, as described above, some of the column lines CL may transmit pixel signals of at least two rows at a time.

[0041] The timing controller 150 may control the timing of other elements of the image sensor 100 (e.g., the row driver 120, the ADC circuit 130, the ramp signal generator 140, the row buffer 160, and the processor 170). The timing controller 150 may provide a timing signal indicating an operation timing to each of the row driver 120, the ADC circuit 130, the ramp signal generator 140, the row buffer 160, and the processor 170.

[0042] Under the control of the timing controller 150, the row driver 120 may generate a control signal for driving the pixel array 110 and provide the control signal to the pixels PX of the pixel array 110 through the row lines RL. The row driver 120 may control a plurality of pixels of the pixel array 110 to sense incident light simultaneously or row by row. The row driver 120 may select one or at least two rows of pixels PX, and may control the selected pixels PX to output pixel signals through the column lines CL.

[0043] The ramp signal generator 140 may generate a ramp signal RAMP that increases or decreases with a specific slope, and provide the ramp signal RAMP to the ADC circuit 130 .

[0044] The ADC circuit 130 may receive pixel signals read out from pixels PX of a row selected from among a plurality of pixels PX of the pixel array 110 by the row driver 120 and convert the pixel signals into pixel values ​​corresponding to digital data.

[0045] The ADC circuit 130 may generate and output first image data IDT1 (eg, raw image data) in row units by converting pixel signals received from the pixel array 110 through the column lines CL into digital data based on the ramp signal RAMP from the ramp signal generator 140 .

[0046] The ADC circuit 130 may include a plurality of ADCs corresponding to the column lines CL, respectively. Each ADC may compare a pixel signal received through a corresponding column line CL with the ramp signal RAMP, and may generate a pixel value based on the comparison result. For example, the ADC may remove a reset signal from a sensing signal using correlated double sampling (CDS), and generate a pixel value indicating the amount of light sensed by the pixel PX.

[0047] The row buffer 160 may include a plurality of row memories, and store a plurality of pixel values ​​output from the ADC circuit 130 in a specific row unit. In other words, the row buffer 160 may store the first image data IDT1 output from the ADC circuit 130 in a specific row unit. For example, the row buffer 160 may include three row memories corresponding to three rows of the pixel array 110, respectively, and store a plurality of pixel values ​​corresponding to the three rows of the first image data IDT1 output from the ADC circuit 130 in the three row memories.

[0048] The processor 170 may process a plurality of pixel values ​​corresponding to a plurality of rows of the first image data IDT1 stored in the row buffer 160. The processor 170 may perform image quality compensation, merging, size reduction, etc. on the first image data IDT1 stored in a specific row unit in the row buffer 160. Therefore, the output image data OIDT generated by the image processing may be generated and output in a specific row unit.

[0049] In one embodiment, the processor 170 may process the first image data IDT1 by color. For example, when the first image data IDT1 includes red pixel values, green pixel values, and blue pixel values, the processor 170 may process the red pixel values, green pixel values, and blue pixel values ​​in parallel or in series. In one embodiment, the processor 170 may include a plurality of processing circuits for processing different colors in parallel. However, the embodiment is not limited thereto, and a single processing circuit may be used repeatedly.

[0050] The processor 170 may generate the output image data OIDT having a smaller data size by performing a merging method according to example embodiments described below.

[0051] The output image data OIDT may be output to an external processor, such as an application processor. The application processor may store, perform image processing, or display the output image data OIDT. Fig.13 and Fig.14 This will be described later.

[0052] According to example embodiments, when the image sensor 100 operates in the first operation mode, a plurality of pixel signals of at least two rows may be read out simultaneously and may be subjected to analog summation in a vertical direction (e.g., a column direction). According to the analog vertical summation, at least two rows are read out simultaneously during a single horizontal period so that the frame rate may be increased by at least two times.

[0053] The first image data IDT1 may be generated according to the analog vertical summation, and the processor 170 may perform merging to generate the output image data OIDT. Therefore, the size of the output image data OIDT may be reduced, and the occurrence of zigzag noise and false colors caused by the sampling frequency difference may be reduced, thereby providing good image quality and a high frame rate.

[0054] Figure 3A is a flow chart illustrating vertical analog summation and interpolation of an image sensor according to example embodiments. Figure 3A The merging method can be obtained by Figure 1 The image sensor 100 is executed.

[0055] At operation S010, the pixel generates photocharge. See below for further details. Fig.12 discussion.

[0056] At operation S020, photocharges are summed on the column conductors in an analog manner. Figure 1 The photocharges are summed on the column lines CL. Figure 3A Also see IDT0. Fig.12 .

[0057] At operation S030, the voltage generated by the charge summation is converted to a digital value by analog-to-digital conversion ("ADC"). Figure 1 These digital values ​​generated by the ADC circuit 130 form Figure 1 Image IDT1. See e.g. Figure 4B and subsequent discussion.

[0058] At operation S040, pixels from IDT1 are interpolated using weights. See e.g. Figure 6 and subsequent discussion.

[0059] At operation S050, the generated interpolated digital value is, for example, Figure 1 The output image data OIDT outputs.

[0060] Figure 3B is a flowchart of a merging method of an image sensor according to example embodiments. Figure 3B The merging method can be obtained by Figure 1 The image sensor 100 is executed.

[0061] refer to Figures 1 to 3B In operation S110, the image sensor 100 may simultaneously read out a plurality of pixel signals of at least two rows in each area AR of the pixel array 110. Therefore, as described above, at least two pixel signals output from at least two pixels in one column and at least two rows may be subjected to analog summation.

[0062] In operation S120, the image sensor 100 may generate first image data IDT1 by performing analog-to-digital conversion on the read pixel signal. For example, the ADC circuit 130 may generate first image data IDT1 by performing analog-to-digital conversion on the pixel signal received through the column line CL. Then, digital binning may be performed.

[0063] The first image data IDT1 may be divided into a plurality of merging regions. In operation S130, the image sensor 100 may perform a first merge on each merging region of the first image data IDT1 based on the pixel values ​​of each merging region. The image sensor 100 may perform a weighted summation on the pixel values ​​having the same color in each merging region. In the present disclosure, the weighted summation indicates that a set weight may be applied to each pixel value, and the weighted values ​​may be summed (or summed and averaged).

[0064] In operation S140, the image sensor 100 may perform interpolation based on pixel values ​​of two merging areas adjacent to each other in the column direction. The image sensor 100 may perform weighted summation on pixel values ​​having the same color in the two merging areas.

[0065] Therefore, in operation S150 , second image data having a smaller size than the first image data IDT1 may be output. For example, when each merge region includes pixel values ​​arranged in a 4×4 matrix, the size of the second image data may correspond to ¼ of the resolution of the pixel array 110 .

[0066] The following will refer to FIG. 4A to FIG. 9 A merging method according to example embodiments is described in detail.

[0067] Figure 4A and Figure 4B is a diagram for describing a readout method according to example embodiments. FIG. 5A to FIG. 5C is a schematic diagram of first image data generated by a readout method according to example embodiments.

[0068] Figure 4A and Figure 4B It is shown that a region AR of the pixel array 110 is read out. The region AR may include a plurality of pixels PX arranged in a 4×4 matrix.

[0069] refer to Figure 4A , the first row Row1 and the third row Row3 can be read out simultaneously during the first horizontal period. The pixel signals of the first green pixels Gr1 and Gr3 in the first column C1 can be read out through the first column line CL1, and the pixel signals of the red pixels R2 and R4 in the fourth column C4 can be output through the fourth column line CL4.

[0070] When the pixel signals of two pixels (e.g., the first green pixels Gr1 and Gr3) are read out through the first column line CL1, the pixel signals may be summed. However, when the pixel signal is output from the pixel PX, the pixel PX may operate as a source follower. Because of the parasitic resistance of the pixel PX, the pixel signal having a higher value among the pixel signals of the first green pixels Gr1 and Gr3 may be provided to the ADC circuit 130 through the first column line CL1 as a sum signal corresponding to the first green pixels Gr1 and Gr3.

[0071] In the red pixels R1 and R3 of the second column C2, the pixel signal of the red pixel R1 in the outer area of ​​the area AR can be output through the second column line CL2. In the first green pixels Gr2 and Gr4 in the third column C3, the pixel signal of the first green pixel Gr2 in the outer area of ​​the area AR can be output through the third column line CL3. In other words, in each of the second column C2 and the third column C3, the pixel signals of the pixels in the outer area among the pixels with the same color in the first row Row1 and the third row Row3 can be read out. The pixel signals of the pixels in the inner area may not be read out.

[0072] The ADC circuit 130 may convert the pixel signals into digital values, for example, pixel values ​​PGr13, PR1, PGr2, and PR24. In one embodiment, the pixel values ​​PGr13, PR1, PGr2, and PR24 generated during the first horizontal period may be stored in one line memory (e.g., the first line memory LM1 of the line buffer 160) and may form part of the first image data IDT1. However, the pixel values ​​PGr13, PR1, PGr2, and PR24 may not correspond to the same line of the first image data IDT1.

[0073] refer to Figure 4B, the second row Row2 and the fourth row Row4 can be simultaneously read out during the second horizontal period after the first horizontal period. The pixel signals of the blue pixels B1 and B3 in the first column C1 can be output through the first column line CL1, and the pixel signals of the second green pixels Gb2 and Gb4 in the fourth column C4 can be output through the fourth column line CL4. At this time, as described above with reference to Figure 4A As described above, two pixel signals output through the same column line may be summed, and the sum signal may be provided to the ADC circuit 130 .

[0074] In the second green pixels Gb1 and Gb3 in the second column C2, the pixel signal of the second green pixel Gb3 in the outer area of ​​the area AR can be output through the second column line CL2. In the blue pixels B2 and B4 in the third column C3, the pixel signal of the blue pixel B4 in the outer area of ​​the area AR can be output through the third column line CL3.

[0075] The ADC circuit 130 may convert the pixel signals into digital values, such as pixel values ​​PB13, PGb3, PB4, and PGb24. In one embodiment, the pixel values ​​PB13, PGb3, PB4, and PGb24 generated during the second horizontal period may be stored in a line memory (e.g., the second line memory LM2 of the line buffer 160) and may form part of the first image data IDT1. In one embodiment, the pixel values ​​PGr13, PR1, PGr2, and PR24 generated during the first horizontal period may be moved from the first line memory LM1 to the second line memory LM2, and the pixel values ​​PB13, PGb3, PB4, and PGb24 generated during the second horizontal period may be stored in the first line memory LM1. The line buffer 160 may also include a third line buffer LM3, which is used in a continuous manner with LM2 and LM1, similar to the discussion of LM1 and LM2 above.

[0076] refer to Figure 5A , the pixel values ​​PGr13, PR1, PGr2, and PR24 stored in the first row memory LM1 can form the first row Row1 and the second row Row2 in the merge area BA of the first image data IDT1. Each of the pixel values ​​PGr13 and PR24 is two pixels in the first row Row1 and the third row Row3 of the area AR of the pixel array 110, respectively (for example, Figure 4A The pixel values ​​PR1 and PGr2 are the sum (or average) of the pixel signals of the first green pixels Gr1 and Gr3 or the red pixels R2 and R4 in the image, and thus can represent the pixel value of the sampling position corresponding to the midpoint between the two pixels. The pixel values ​​PR1 and PGr2 can respectively represent the corresponding pixels (i.e., Figure 4A The pixel values ​​at the positions of the red pixel R1 and the first green pixel Gr2).

[0077] refer to Figure 5B , the pixel values ​​PB13, PGb3, PB4, and PGb24 stored in the second row memory LM2 can form the third row Row3 and the fourth row Row4 in the merge area BA of the first image data IDT1. Each of the pixel values ​​PB13 and PGb24 is two pixels in the second row Row2 and the fourth row Row4 of the area AR of the pixel array 110, respectively (for example, Figure 4B The pixel values ​​PGb3 and PB4 are the sum (or average) of the pixel signals of the blue pixels B1 and B3 or the second green pixels Gb2 and Gb4 in the image, and thus can represent the pixel value of the sampling position corresponding to the midpoint between the two pixels. The pixel values ​​PGb3 and PB4 can respectively represent the corresponding pixels (i.e., Figure 4B The pixel values ​​at the positions of the second green pixel Gb3 and the blue pixel B4) in .

[0078] When reading out is performed on the area AR of the pixel array 110 according to example embodiments, it may be performed as follows Figure 5C The pixel values ​​of the merge area BA of the first image data IDT1 are determined as shown.

[0079] Figure 6 A first merging performed on each of a plurality of merging regions of first image data based on a pixel value in each merging region in a merging method according to example embodiments is illustrated.

[0080] refer to Figure 6 , the first merging may be performed by summing pixel values ​​corresponding to the same color in the merging area BA.

[0081] For example, the pixel value corresponding to the sampling position S11 can be calculated by summing the pixel values ​​PGr13 and PGr2 corresponding to the first green color. At this time, a preset weight can be applied to each of the pixel values ​​PGr13 and PGr2, and the weighted values ​​can be summed. The weight can be preset in consideration of the sampling position. In other words, the weight can be set so that the sum of the weighted values ​​is located at the sampling position S11. For example, when the distance between the centers of the pixels represented by the pixel values ​​PGr13 and PGr2 is 10 and the sampling position S11 is located at a distance of 3 from the pixel value PGr13, the ratio of the weights applied to the pixel values ​​PGr13 and PGr2, respectively, can be 7:3. In other words, a higher weight can be applied to the pixel value PGr13.

[0082] In the same manner, weights may be applied to pixel values ​​corresponding to the same color so that the color is located at sampling position S12, S13, or S14, and the weighted values ​​may be summed. Thus, pixel values ​​PGr_b, PR_b, PB_b, and PGb_b corresponding to sampling positions S11, S12, S13, and S14, respectively, may be calculated.

[0083] Figure 7 Interpolation performed based on pixel values ​​of two adjacent merging areas in a merging method according to example embodiments is illustrated.

[0084] refer to Figure 7 , the first image data IDT1 may include a plurality of merged areas BAn-1, BAn, and BAn+1. Interpolation may be performed on the merged areas BAn-1, BAn, and BAn+1, so that pixel values ​​PGr_t, PR_t, PB_t, and PGb_t corresponding to target sampling positions TS1, TS2, TS3, and TS4, respectively, may be generated.

[0085] The generation of pixel values ​​corresponding to the target sampling positions in the merge area BAn will be described exemplarily. Figure 6 As described, pixel values ​​PGr_b, PR_b, PB_b, and PGb_b corresponding to sampling positions S11, S12, S13, and S14 are calculated in the merged area BAn, and each of the pixel values ​​PGr_b, PR_b, PB_b, and PGb_b can be added to the pixel value corresponding to the nearest pixel among the pixel values ​​of the same color in the adjacent merged areas BAn-1 and BAn+1. A weight can be applied to each pixel value. The weight can be set in consideration of the position of each pixel value and the target sampling position. The smaller the distance between the position corresponding to the pixel value and the target sampling position, the larger the weight can be. As described above, pixel values ​​PGr_t, PR_t, PB_t, and PGb_t corresponding to the target sampling positions TS1, TS2, TS3, and TS4 in the merged area BAn, respectively, can be calculated by interpolation.

[0086] By analog vertical summing and merging of readings from a plurality of pixels PX arranged in a 4×4 matrix, second image data IDT2 including pixel values ​​arranged in a 2×2 matrix may be generated. Figure 7 The right part of .

[0087] Figure 8 is a flowchart of a merging method of an image sensor according to example embodiments. Figure 8 The merging method can be obtained by Figure 1 Operations S210, S220, S230, and S240 are performed with the image sensor 100. Figure 3BOperations S110, S120, S130, and S140 in FIG. 1 are the same. Therefore, redundant descriptions will be omitted.

[0088] After generating the first image data IDT1 in operation S220, the image sensor 100 may perform a second merging on the green pixels in operation S250. The image sensor 100 may sum at least two green pixel values ​​of each of the plurality of merging areas of the first image data IDT1 and green pixel values ​​in adjacent merging areas. For example, the adjacent merging areas may be adjacent to each merging area in a column direction. A weight may be applied to each pixel value in consideration of a sampling position, and the weighted values ​​may be summed. The reference Fig. 9 The second merge is described in detail.

[0089] Fig. 9 is a diagram for describing a second merger applied to a green pixel in a merger method according to example embodiments.

[0090] refer to Fig. 9 , the merged area BAn and the merged area BAn-1 may be most adjacent to each other in the column direction.

[0091] The pixel values ​​PGr13 and PGr2 corresponding to the first green pixel in the merged area BAn and the pixel value PGb3 in the merged area BAn-1 can be summed, and the pixel value PGb3 corresponds to the second green pixel that is most adjacent to the first green pixel in the merged area BAn. At this time, a weight can be applied to each of the pixel values ​​PGr13, PGr2, and PGb3 so that the sum value is located at the first target sampling position TS1, and the weighted values ​​can be summed. Therefore, the pixel value PGr_t' of the green pixel corresponding to the first target sampling position TS1 can be generated. In a similar manner, the pixel value PGb_t' of the green pixel corresponding to the second target sampling position TS2 can be generated. In this way, the pixel value of the green pixel at the target sampling position corresponding to the green pixel of the merged area BAn is determined, and thus the third image data IDT3 including the pixel value of the green pixel can be generated.

[0092] During the interpolation performed in operation S240, the pixel values ​​to be summed are far away from each other, such as Figure 7 However, according to the second merging, the pixel value of the green pixel closest to the merging area BAn and the pixel value of the green pixel in the merging area BAn can be summed. Therefore, merging can be performed based on the pixel values ​​corresponding to the adjacent pixels.

[0093] Go back for reference Figure 8In operation S270, the second image data and the third image data may be combined with each other based on the pixel value difference between the first green pixel and the second green pixel. As described above, the third image data may include the pixel value of the green pixel. Therefore, the pixel value of the green pixel of the second image data may be combined with the pixel value of the green pixel of the third image data. At operation S280, the combined image data is output.

[0094] At this time, the second image data and the third image data may be combined with each other based on a difference between pixel values ​​corresponding to two most adjacent pixels among pixel values ​​of the first green pixel and the second green pixel used during the second combination period.

[0095] For example, when the difference between the pixel values ​​is less than the first reference value, that is, when the difference between the pixel values ​​is small, the pixel value of the green pixel of the third image data may be applied to the output image data. In other words, the pixel values ​​of the red pixel and the blue pixel of the second image data and the pixel value of the green pixel of the third image data may be included in the output image data. When the difference between the pixel values ​​exceeds the second reference value, that is, when the difference between the pixel values ​​is large, the second image data may be selected as the output image data. In other words, the green pixel value of the third image data may not be reflected in the output image data.

[0096] Comparison of the difference to a threshold provides a non-linear step size, which is useful for reducing artifacts (eg, zigzag artifacts and false color artifacts).

[0097] Otherwise, when the difference between the pixel values ​​is greater than or equal to the first reference value and less than the second reference value, the difference may be converted to a value less than or equal to 1 based on the first reference value and the second reference value, a weight may be applied to each of the second image data and the third image data based on the value generated by the conversion, and the weighted values ​​may be summed. For example, when the difference between the pixel values ​​is converted to a value of 0.6, a weight of 0.4 may be applied to the pixel value of the green pixel of the second image data and a weight of 0.6 may be applied to the pixel value of the green pixel of the third image data, and the weighted values ​​may be summed. Based on the summed pixel value for the green pixel and the pixel values ​​of the red pixel and the blue pixel of the second image data, output image data may be generated.

[0098] Fig.10 An example of a pixel according to example embodiments is illustrated.

[0099] refer to Fig.10 , the pixel PX may include a photoelectric conversion unit 11 and a pixel circuit 12. The pixel circuit 12 may include a plurality of transistors, for example, a transfer transistor TX controlled by a signal TS, a reset transistor RX, a drive transistor DX, and a selection transistor SX.

[0100] The photoelectric conversion unit 11 may include a photodiode. The photodiode may generate photocharges that vary with the intensity of incident light. The transfer transistor TX may be configured to generate photoelectric charges according to the intensity of incident light. Figure 1 The transmission control signal TS provided by the row driver 120 of the floating diffusion node FD is used to transfer the photocharge to the floating diffusion node FD. The driving transistor DX can amplify the voltage corresponding to the photocharge accumulated in the floating diffusion node FD. The driving transistor DX can operate as a source follower. When the drain node of the selection transistor SX is connected to the source node of the driving transistor DX and the selection transistor SX is turned on in response to the selection signal SEL output from the row driver 120, the pixel signal APS corresponding to the voltage level of the floating diffusion node FD can be output to the column line CL connected to the pixel PX. The reset transistor RX can reset the floating diffusion node FD based on the power supply voltage VDD in response to the reset signal RS provided from the row driver 120.

[0101] As referenced above Figure 4A and Figure 4B As described above, at least two rows of the pixel array 110 can be read out. At this time, the pixels in the central part of the area AR of the pixel array 110 are not read out. Therefore, when two rows are read out at the same time, the pixels in the central part may not be selected. The pixels to be read out can be connected to the column line CL in response to the first selection signal SEL1 at the effective level, and the pixels not to be read out can be disconnected from the column line CL in response to the second selection signal SEL2 at the invalid level. Therefore, even if the pixels are in the same row, the pixel signal of each pixel can be selectively output or not output.

[0102] Fig.11A A pixel array having a quaternary pattern is shown. Fig. 11B An example of applying a pixel array having a quadruple pattern to an image sensor according to example embodiments is illustrated.

[0103] refer to Fig.11A , the pixel array 110a has a quaternary pattern. The red pixels PX_R may be arranged in a 2x2 matrix, the first green pixels PX_Gr may be arranged in a 2x2 matrix, the second green pixels PX_Gb may be arranged in a 2×2 matrix, and the blue pixels PX_B may be arranged in a 2x2 matrix. This pattern may be repeated in a matrix form. Such a pattern may be referred to as a quaternary Bayer pattern.

[0104] In an example, each pixel in a 2×2 matrix may include a photoelectric conversion element and may share a floating diffusion node and a pixel circuit with each other, such as Fig.12 Therefore, the pixels in the 2×2 matrix can be operated as a single large pixel, for example, Fig. 11BPX_R1, PX_Gr1, PX_Gb1, or PX_B1 as shown. Large pixels may form a Bayer pattern. Therefore, as described above, the merging method according to the example embodiment may be applied. The first row Row1 and the third row Row3 may be simultaneously read out during the first horizontal period, and the second row Row2 and the fourth row Row4 may be simultaneously read out during the second horizontal period.

[0105] Fig.12 An example of a pixel according to example embodiments is illustrated.

[0106] refer to Fig.12 , the pixel PXa may include a plurality of photoelectric conversion elements 22a, 22b, 22c, and 22d and a pixel circuit 12. For example, the pixel PXa may include four photoelectric conversion elements 22a, 22b, 22c, and 22d. In some embodiments, the photoelectric conversion elements 22a, 22b, 22c, and 22d may include photodiodes PD1A, PD1B, PD1C, and PD1D, respectively. A microlens may be arranged above each photoelectric conversion element 22a, 22b, 22c, and 22d. Therefore, the combination of a microlens and a photoelectric conversion element may be referred to as a single pixel, and therefore, Fig.12 The pixel PXa can be regarded as four pixels.

[0107] The pixel circuit 12 may include: four transfer transistors TX1 to TX4 respectively connected to the photoelectric conversion elements 22 a , 22 b , 22 c , and 22 d ; a reset transistor RX1 ; a drive transistor DX1 ; and a selection transistor SX1 .

[0108] The floating diffusion node FD1 can be shared by four photoelectric conversion elements 22a, 22b, 22c and 22d and four transfer transistors TX1 to TX4. The reset transistor RX1 can be turned on in response to the reset signal RS1 to reset the floating diffusion node FD1 with the power supply voltage VDD. Each transfer transistor TX1 to TX4 can connect a corresponding one of the photodiodes PD1A, PD1B, PD1C and PD1D to the floating diffusion node FD1 or disconnect them according to the voltage of a corresponding one of the transfer gates TG1, TG2, TG3 and TG4.

[0109] Light incident to each of the photodiodes PD1A, PD1B, PD1C, and PD1D may be accumulated therein as charges by photoelectric conversion. When the charges accumulated in each of the photodiodes PD1A, PD1B, PD1C, and PD1D are transferred to the floating diffusion node FD1, the charges may be output as a first analog voltage V1out via the drive transistor DX1 and the select transistor SX1. The first analog voltage V1out corresponding to the voltage change in the floating diffusion node FD1 may be sent to an external readout circuit (not shown).

[0110] Pixel PXa can be applied to Fig.11A For example, the four photoelectric conversion elements 22a, 22b, 22c and 22d of the pixel PXa may correspond to the pixels in the 2x2 matrix, respectively. In other words, the pixels in the 2x2 matrix may share the floating diffusion node FD1, and Fig.12 When the transfer transistors TX1 to TX4 are turned on or off at the same time, the pixels in the 2x2 matrix can be operated as a single large pixel, such as Fig. 11B In one embodiment, when the pixel PXa operates as a large pixel, only some of the transfer transistors TX1 to TX4 may be turned on or off, while the other transfer transistors TX1 to TX4 remain off.

[0111] Fig.13 is a block diagram of an electronic device including a multi-camera module using an image sensor according to example embodiments. Fig.14 yes Fig.13 Detailed block diagram of the camera module in Figure 1.

[0112] refer to Fig.13 , the electronic device 1000 may include a camera module group 1100 , an application processor 1200 , a power management integrated circuit (PMIC) 1300 , and an external memory 1400 .

[0113] The camera module group 1100 may include a plurality of camera modules 1100a, 1100b, and 1100c. Fig.13 1100a, 1100b, and 1100c are shown, but example embodiments are not limited thereto. In some embodiments, camera module group 1100 may be modified to include only two camera modules. In some embodiments, camera module group 1100 may be modified to include "n" camera modules, where "n" is a natural number of at least 4.

[0114] Will be referenced below Fig.14 A detailed configuration of the camera module 1100b is described. The following description may also be applied to the other camera modules 1100a and 1100c.

[0115] refer to Fig.14 , the camera module 1100 b may include a prism 1105 , an optical path folding element (OPFE) 1110 , an actuator 1130 , an image sensing device 1140 , and a storage 1150 .

[0116] The prism 1105 may include a reflective surface 1107 of a light reflective material and may change a path of light L incident from the outside.

[0117] In some embodiments, the prism 1105 may change the path of the light L incident in the first direction X to a second direction Y perpendicular to the first direction X. The prism 1105 may rotate the reflective surface 1107 of the light reflective material around the central axis 1106 in the direction A or around the central axis 1106 in the direction B so that the path of the light L incident in the first direction X is changed to the second direction Y perpendicular to the first direction X. At this time, the OPFE 1110 may move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0118] In some embodiments, the A-direction maximum rotation angle of the prism 1105 may be less than or equal to 15 degrees in the positive (+) A direction and greater than 15 degrees in the negative (−) A direction, but example embodiments are not limited thereto.

[0119] In a certain example, the prism 1105 may move about 20 degrees or an angle in the range from about 10 degrees to about 20 degrees or from about 15 degrees to about 20 degrees in the positive B direction or the negative B direction. At this time, the angle at which the prism 1105 moves in the positive B direction may be the same as or similar to the angle at which the prism 1105 moves in the negative B direction within a difference of about 1 degree.

[0120] In some embodiments, the prism 1105 may move the reflective surface 1107 of the light reflective material in a third direction Z that is parallel to the extension direction of the central axis 1106 .

[0121] The OPFE 1110 may include, for example, "m" optical lenses, where "m" is a natural number. The "m" lenses may move in the second direction Y and change the optical zoom ratio of the camera module 1100b. For example, when the default optical zoom ratio of the camera module 1100b is Z, the optical zoom ratio of the camera module 1100b may be changed to 3Z, 5Z, or more by moving the "m" optical lenses included in the OPFE 1110.

[0122] The actuator 1130 may move the OPFE 1110 or the optical lens to a specific position. For example, the actuator 1130 may adjust the position of the optical lens so that the image sensor 1142 is positioned at the focal length of the optical lens for accurate sensing.

[0123] The image sensing device 1140 may include an image sensor 1142, a control logic unit 1144, and a memory 1146. The image sensor 1142 may sense an image of an object using light L provided through an optical lens. Figure 1 The image sensor 100 of the embodiment of the present invention can be used as the image sensor 1142. Therefore, when the image sensing device 1140 is operated in the first mode, the frame rate and image quality can be improved and the size of the image data generated by the image sensing device 1140 can be reduced. For example, the frame rate can be a frame rate that can be achieved based on the circuit interface bandwidth and the computing bandwidth. The image sensor 100 (providing example details of the image sensor 1142) allows the frame rate to be a high frame rate.

[0124] The control logic 1144 may generally control the operation of the camera module 1100b. For example, the control logic 1144 may control the operation of the camera module 1100b according to a control signal provided through the control signal line CSLb.

[0125] The memory 1146 may store information such as calibration data 1147 required for the operation of the camera module 1100b. The calibration data 1147 may include information required by the camera module 1100b for generating image data using the light L provided from the outside. For example, the calibration data 1147 may include information related to the angle of the above-mentioned rotation, information related to the focal length, information related to the optical axis, etc. When the camera module 1100b is implemented as a multi-state camera having a focal length that varies with the position of the optical lens, the calibration data 1147 may include a value of the focal length for each position (or state) of the optical lens and information related to autofocus.

[0126] The storage 1150 may store image data sensed by the image sensor 1142. The storage 1150 may be provided outside the image sensing device 1140 and may form a stack with a sensor chip of the image sensing device 1140. In some embodiments, the storage 1150 may include an electrically erasable programmable read-only memory (EEPROM), but the embodiment is not limited thereto.

[0127] refer to Fig.13 and Fig.14 In some embodiments, each camera module 1100a, 1100b, and 1100c may include an actuator 1130. Therefore, the camera modules 1100a, 1100b, and 1100c may include calibration data 1147, and the calibration data 1147 in the camera modules 1100a, 1100b, and 1100c may be the same or different depending on the operation of the actuator 1130 included in each camera module 1100a, 1100b, and 1100c.

[0128] In some embodiments, one of the camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100b) may be a folded lens type including a prism 1105 and an OPFE 1110, and the other camera modules (e.g., camera modules 1100a and 1100c) may be a vertical type not including the prism 1105 and the OPFE 1110. However, embodiments are not limited thereto.

[0129] In some embodiments, one of the camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100c) may include a vertical depth camera that extracts depth information using infrared (IR). In this case, the application processor 1200 may generate a three-dimensional (3D) depth image by combining image data provided from the depth camera with image data provided from another camera module (e.g., camera module 1100a or 1100b).

[0130] In some embodiments, at least two camera modules (e.g., 1100a and 1100b) of the camera modules 1100a, 1100b, and 1100c may have different fields of view. In this case, two camera modules (e.g., 1100a and 1100b) of the camera modules 1100a, 1100b, and 1100c may have different optical lenses, respectively, but the embodiment is not limited thereto.

[0131] In some embodiments, the camera modules 1100a, 1100b, and 1100c may have different fields of view from each other. In this case, the camera modules 1100a, 1100b, and 1100c may have different optical lenses, respectively, but the embodiment is not limited thereto.

[0132] In some embodiments, the camera modules 1100a, 1100b, and 1100c may be physically isolated from each other. In other words, the sensing area of ​​the image sensor 1142 is not divided and used by the camera modules 1100a, 1100b, and 1100c, but the image sensor 1142 may be independently included in each camera module 1100a, 1100b, and 1100c.

[0133] Go back for reference Fig.13 , the application processor 1200 may include an image processing unit 1210, a memory controller 1220, and an internal memory 1230. The application processor 1200 may be implemented separately from the camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the camera modules 1100a, 1100b, and 1100c may be implemented in different semiconductor chips.

[0134] The image processing unit 1210 may include: a plurality of sub-image processors 1212 a , 1212 b , and 1212 c ; an image generator 1214 ; and a camera module controller 1216 .

[0135] The image processing unit 1210 may include as many sub-image processors 1212a, 1212b, and 1212c as the camera modules 1100a, 1100b, and 1100c.

[0136] The image data generated from each camera module 1100a, 1100b, and 1100c may be provided to a corresponding one of the sub-image processors 1212a, 1212b, and 1212c through a corresponding one of the image signal lines ISLa, ISLb, and ISLc separated from each other. For example, the image data generated from the camera module 1100a may be provided to the sub-image processor 1212a through the image signal line ISLa, the image data generated from the camera module 1100b may be provided to the sub-image processor 1212b through the image signal line ISLb, and the image data generated from the camera module 1100c may be provided to the sub-image processor 1212c through the image signal line ISLc. Such image data transmission may be performed using, for example, a camera serial interface (CSI) based on a mobile industry processor interface (MIPI), but the embodiment is not limited thereto.

[0137] In some embodiments, a single sub-image processor may be provided for multiple camera modules. Fig.13 Differently, the sub-image processors 1212a and 1212c may not be separated but may be integrated into a single sub-image processor, and the image data provided from the camera module 1100a or the camera module 1100c may be selected by a selection element (e.g., a multiplexer) and then provided to the integrated sub-image processor.

[0138] The image data provided to each of the sub image processors 1212a, 1212b, and 1212c may be provided to the image generator 1214. The image generator 1214 may generate an output image using the image data provided from each of the sub image processors 1212a, 1212b, and 1212c according to the image generation information or mode signal.

[0139] Specifically, the image generator 1214 may generate an output image by combining at least a portion of each image data generated from the camera modules 1100a, 1100b, and 1100c having different fields of view, respectively, according to the image generation information or the mode signal. Alternatively, the image generator 1214 may generate an output image by selecting one of each image data generated from the camera modules 1100a, 1100b, and 1100c having different fields of view, respectively, according to the image generation information or the mode signal.

[0140] In some embodiments, the image generation information may include a zoom signal or a zoom factor.In some embodiments, the mode signal may be based on a mode selected by a user.

[0141] When the image generation information includes a zoom signal or a zoom factor and the camera modules 1100a, 1100b, and 1100c have different fields of view, the image generator 1214 may perform different operations according to different kinds of zoom signals. For example, when the zoom signal is a first signal, the image generator 1214 may combine the image data output from the camera module 1100a with the image data output from the camera module 1100c, and may generate an output image using the combined image signal and the image data output from the camera module 1100b that is not used during the combination. When the zoom signal is a second signal different from the first signal, the image generator 1214 may generate an output image by selecting one of the respective image data outputted from the camera modules 1100a, 1100b, and 1100c, respectively, instead of performing a combination. However, the embodiment is not limited thereto, and the method of processing the image data may be changed when necessary.

[0142] In some embodiments, the image generator 1214 may receive a plurality of image data having different exposure times from at least one of the sub-image processors 1212a, 1212b, and 1212c, and perform high dynamic range (HDR) processing on the respective image data, thereby generating combined image data having a higher dynamic range.

[0143] The camera module controller 1216 may provide a control signal to each camera module 1100a, 1100b, and 1100c. The control signal generated by the camera module controller 1216 may be provided to a corresponding one of the camera modules 1100a, 1100b, and 1100c through a corresponding one of control signal lines CSLa, CSLb, and CSLc separated from each other.

[0144] One of the camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100b) may be designated as a master camera according to a mode signal or an image generation signal including a zoom signal, and the other camera modules (e.g., 1100a and 1100c) may be designated as slave cameras. Such designation information may be included in a control signal and provided to each of the camera modules 1100a, 1100b, and 1100c through a corresponding one of control signal lines CSLa, CSLb, and CSLc separated from each other.

[0145] The camera module operating as a master camera module or a slave camera module may be changed according to a zoom factor or an operation mode signal. For example, when the field of view of the camera module 1100a is greater than the field of view of the camera module 1100b and the zoom factor indicates a low zoom ratio, the camera module 1100b may operate as a master camera module and the camera module 1100a may operate as a slave camera module. On the contrary, when the zoom factor indicates a high zoom ratio, the camera module 1100a may operate as a master camera module and the camera module 1100b may operate as a slave camera module.

[0146] In some embodiments, the control signal provided from the camera module controller 1216 to each camera module 1100a, 1100b, and 1100c may include a synchronization enable signal. For example, when the camera module 1100b is a master camera module and the camera module 1100a is a slave camera module, the camera module controller 1216 may send a synchronization enable signal to the camera module 1100b. The camera module 1100b provided with the synchronization enable signal may generate a synchronization signal based on the synchronization enable signal, and may provide the synchronization signal to the camera modules 1100a and 1100c through the synchronization signal line SSL. The camera modules 1100a, 1100b, and 1100c may be synchronized using the synchronization signal and may send image data to the application processor 1200.

[0147] In some embodiments, the control signal provided to each camera module 1100a, 1100b, and 1100c from the camera module controller 1216 may include mode information consistent with the mode signal. The camera modules 1100a, 1100b, and 1100c may operate in a first operation mode or a second operation mode related to the sensing speed based on the mode information.

[0148] In the first operation mode, the camera modules 1100a, 1100b, and 1100c may generate image signals at a first speed (e.g., at a first frame rate), encode the image signals at a second speed higher than the first speed (e.g., at a second frame rate higher than the first frame rate), and transmit the encoded image signals to the application processor 1200. At this time, the second speed may be up to 30 times the first speed.

[0149] The application processor 1200 may store the received image signal (i.e., the encoded image signal) in its internal memory 1230 or in an external memory 1400 outside the application processor 1200. Then, the application processor 1200 may read the encoded image signal from the internal memory 1230 or the external memory 1400, decode the encoded image signal, and display the generated image data based on the decoded image signal. For example, a corresponding one of the sub-image processors 1212a, 1212b, and 1212c of the image processing unit 1210 may perform decoding and may also perform image processing on the decoded image signal.

[0150] In the second operating mode, camera modules 1100a, 1100b, and 1100c may generate image signals at a third speed lower than the first speed (e.g., at a third frame rate lower than the first frame rate) and transmit the image signals to application processor 1200. The image signals provided to application processor 1200 may not have been encoded. Application processor 1200 may perform image processing on the image signals, or store the image signals in internal memory 1230 or external memory 1400.

[0151] The PMIC 1300 may provide power (e.g., a power supply voltage) to each of the camera modules 1100a, 1100b, and 1100c. For example, under the control of the application processor 1200, the PMIC 1300 may provide a first power to the camera module 1100a through a power signal line PSLa, a second power to the camera module 1100b through a power signal line PSLb, and a third power to the camera module 1100c through a power signal line PSLc.

[0152] The PMIC 1300 may generate power corresponding to each camera module 1100a, 1100b, and 1100c, and adjust the level of power in response to a power control signal PCON from the application processor 1200. The power control signal PCON may include a power adjustment signal for each operating mode of the camera modules 1100a, 1100b, and 1100c. For example, the operating mode may include a low power mode. At this time, the power control signal PCON may include information related to the camera module for operating in a low power mode and a power level to be set. The same or different levels of power may be provided to the camera modules 1100a, 1100b, and 1100c, respectively. The level of power may be changed dynamically.

[0153] While example embodiments have been particularly shown and described, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the following claims.

Claims

1. A method for merging an image sensor, the method comprising: Reading out a plurality of pixel signals from at least two rows of each of a plurality of regions of a pixel array at a time, each of the plurality of regions including a plurality of pixels arranged in a 2n×2n matrix, where n is an integer greater than or equal to 2; generating first image data by performing analog-to-digital conversion on the plurality of pixel signals; generating, based on the first image data, a first sum value for each of the plurality of merging areas based on two pixel values ​​corresponding to the same color in each of the plurality of merging areas, the plurality of merging areas corresponding to the plurality of areas of the pixel array; as well as A second sum value of each of two merged areas is generated based on two first sum values ​​corresponding to the same color in two merged areas, the two merged areas being adjacent to each other in a column direction among the plurality of merged areas.

2. The merging method according to claim 1, wherein reading out the plurality of pixel signals comprises: simultaneously reading out pixel signals from at least two first pixels having a first color, the at least two first pixels being in a first column of the pixel array; And reading out a pixel signal from one second pixel of at least two second pixels having a second color, the at least two second pixels being in a second column of the pixel array. 3 . The merging method according to claim 2 , wherein among the at least two second pixels, the one second pixel is in an outer area of ​​each of the plurality of areas.

4. The merging method according to claim 2, wherein generating the first image data comprises: Receiving a sum signal by the analog-to-digital conversion circuit through the first column line, the sum signal corresponding to the sum of the pixel signals of the at least two first pixels; as well as A pixel value for a first sampling position is generated by the analog-to-digital conversion circuit by performing analog-to-digital conversion on the sum signal, the first sampling position corresponding to a midpoint between the at least two first pixels.

5. The merging method according to claim 1, wherein generating the first sum value comprises: A weight is applied to each of the two pixel values ​​and the weighted values ​​are summed. 6 . The merging method according to claim 5 , wherein the weight applied to each of the two pixel values ​​is set based on a second sampling position where the first sum value is located.

7. The merging method according to claim 1, wherein generating the second sum value comprises: applying a weight to each of the two first sum values ​​and summing the weighted values; as well as The second sum value is generated as a pixel value for a third sampling position.

8. The merging method according to claim 1, further comprising: generating a third sum value based on at least two first pixel values ​​of each of the plurality of merged regions of the first image data and a first pixel value of an adjacent merged region adjacent to each of the plurality of merged regions; as well as For at least two first pixels, the second sum value is combined with the third sum value.

9. The merging method according to claim 8, wherein the combining is performed based on a difference between a first pixel value of each of the plurality of merging regions and a first pixel value of the adjacent merging region, wherein the third sum value is included in the output image data when the difference is less than a first threshold value, including the second sum value in the output image data when the difference exceeds a second threshold, and When the difference is greater than or equal to the first threshold and less than or equal to the second threshold: applying a weight to each of the second sum and the third sum based on the difference, and The sum of the weighted values ​​is included in the output image data.

10. An image sensor, comprising: A pixel array divided into a plurality of regions having a quadrilateral shape, each of the plurality of regions including pixels arranged in a 2n×2n matrix, where n is an integer greater than or equal to 2; The analog-to-digital conversion circuit is configured as: Reading out a plurality of pixel signals, and converting the plurality of pixel signals into first image data, the first image data comprising a plurality of pixel values, and the plurality of pixel signals being received from the pixel array via a plurality of column lines; A row driver configured to provide a control signal through a plurality of row lines connected to the pixel array, wherein the control signal is configured to control pixel signals of at least two rows of the pixel array to be output simultaneously; a line buffer configured to store the first image data in specific line units; as well as The processor is configured as: performing merging on the first image data stored in the line buffer; generating a first sum value for each of a plurality of merged areas based on two pixel values ​​corresponding to the same color in each of a plurality of merged areas, the plurality of merged areas being included in the first image data; as well as A second sum value of each of two merged areas is generated based on two first sum values ​​corresponding to the same color in two merged areas, the two merged areas being adjacent to each other in a column direction among the plurality of merged areas.

11. The image sensor according to claim 10, wherein the row driver is further configured to: control at least two first pixels to output pixel signals simultaneously, and control one of at least two second pixels to output pixel signals, wherein the at least two first pixels are in the first column of the at least two rows of the pixel array, and the at least two second pixels are in the second column of the at least two rows of the pixel array.

12. The image sensor according to claim 11, wherein the analog-to-digital conversion circuit is further configured to: The sum signal is received via the first column line, and A pixel value for a first sampling position is generated by performing analog-to-digital conversion on the sum signal, the sum signal corresponding to a sum of pixel signals of the at least two first pixels, and the first sampling position corresponding to a midpoint between the at least two first pixels.

13. The image sensor according to claim 10, wherein the processor is further configured to: applying a weight to each of the two pixel values; and The weighted values ​​are summed, the weights being set based on the second sampling position at which the first sum value is located.

14. An image sensor according to claim 13, wherein the processor is further configured to: calculate a third sum value based on at least two first pixel values ​​of each of the multiple merged areas of the first image data and first pixel values ​​of adjacent merged areas adjacent to each of the multiple merged areas; and combine the second sum value with the third sum value for at least two first pixels. 15 . The image sensor of claim 10 , wherein each of the plurality of regions includes a Bayer pattern having one red pixel, two green pixels, and one blue pixel repeatedly arranged.

16. An image processing system, comprising: an image sensor configured to sense the light signal and generate image data; as well as a first processor configured to receive and process the image data from the image sensor, The image sensor comprises: a pixel array divided into a plurality of regions having a quadrilateral shape, each of the plurality of regions including pixels arranged in a 4×4 matrix; The analog-to-digital conversion circuit is configured as follows: Reading out a plurality of pixel signals, and converting the plurality of pixel signals into first image data, the first image data comprising a plurality of pixel values, the plurality of pixel signals being received from the pixel array via a plurality of column lines; A row driver configured to provide a control signal through a plurality of row lines connected to the pixel array, wherein the control signal is configured to control pixel signals of at least two rows of the pixel array to be output simultaneously; a line buffer configured to store the first image data in specific line units; and The second processor is configured as follows: performing merging on the first image data stored in the line buffer; generating a first sum value for each of a plurality of merged areas based on two pixel values ​​corresponding to the same color in each of a plurality of merged areas, the plurality of merged areas being included in the first image data; and A second sum value of each of two merged areas is generated based on two first sum values ​​corresponding to the same color in two merged areas, the two merged areas being adjacent to each other in a column direction among the plurality of merged areas.

17. The image processing system according to claim 16, wherein the second processor is further configured to: calculating a third sum value based on at least two first pixel values ​​of each of the plurality of merged regions of the first image data and a first pixel value of another merged region adjacent to each of the plurality of merged regions; For at least two first pixels, the second sum value is combined with the third sum value.

Images