Image signal processor, image signal processing method, and electronic device thereof

Abstract

An image signal processor, an electronic device, and an image signal processing method are provided. The image signal processor includes: a white balance block that performs white balance on a Bayer pattern raw RGB image received from an image sensor based on a core or in a core unit; a green generation block that performs cross-merging on the white-balanced G pixels to generate first green pixels, and adds high-frequency components applied with a preset weight to generate merged green pixels; a red and blue generation block that generates U pixels and V pixels indicating directionality based on the merged green pixels, the white-balanced R pixels, and the white-balanced B pixels, and fuses the merged green pixels with each of the U pixels and V pixels to generate merged red pixels and merged blue pixels; and an inverse white balance block that performs inverse white balance on the merged red pixels, the merged green pixels, and the merged blue pixels to output a final merged image.

Description

Technical Field

[0001] The present invention relates to an image signal processor, and more particularly, to an image signal processor, an image signal processing method, and an image sensing device that perform a merging operation. Background Art

[0002] An image sensing device may be used in a mobile device such as a smart phone, a tablet PC, and a digital camera or various electronic devices. The image sensing device is generally configured in a form in which fine pixels are two-dimensionally integrated, converts an electrical signal corresponding to the luminance of incident light into a digital signal, and outputs the digital signal. For example, the image sensing device may be configured in a Bayer pattern and may provide Bayer image data corresponding to the luminance of light incident on the Bayer pattern.

[0003] On the other hand, with the development of technology, the number of pixels (i.e., resolution) in the image sensing device may be increased, but increasing the resolution of the pixels increases the amount of data to be processed. Accordingly, the image sensing device may perform merging. The merging operation may include an operation of generating an image not using information of all pixels of the image sensing device, but collecting information of adjacent pixels to generate a single type of information, and then using the collected information to generate a target image to be processed. Summary of the Invention

[0004] Aspects of the present invention provide an image signal processor that reduces zigzag noise and slows down false color defects to output a merged image having improved image quality.

[0005] Aspects of the present invention also provide an electronic device that reduces zigzag noise and slows down false color defects to output a merged image having improved image quality.

[0006] Aspects of the present invention also provide an image signal processing method that reduces zigzag noise and slows down false color defects to output a merged image having improved image quality.

[0007] One aspect of the present invention provides an image signal processor, comprising: a white balance circuit configured to perform white balance on a raw RGB image in Bayer pattern received from an image sensor to generate a white balanced G pixel, a white balanced R pixel, and a white balanced B pixel; a green generation circuit configured to perform cross-merging on the white balanced G pixel to generate a first green pixel, and add a high-frequency component with a preset weight to the first green pixel to generate a merged green pixel; a red / blue generation circuit configured to generate U pixels and V pixels indicating directionality based on the merged green pixel, the white balanced R pixel, and the white balanced B pixel, and fuse the merged green pixel with each of the U pixels and V pixels to generate a merged red pixel and a merged blue pixel; and an inverse white balance circuit configured to perform inverse white balance on the merged red pixel, the merged green pixel, and the merged blue pixel to output a final merged image.

[0008] One aspect of the present invention provides an electronic device, comprising: an image sensor configured to detect light reflected from an object and output a raw RGB image in Bayer pattern; and an image signal processor configured to perform merging on the raw RGB image to generate a merged image, wherein the image signal processor is configured to perform an operation including the following steps: perform white balance on the raw RGB image to generate a white balanced G pixel, a white balanced R pixel, and a white balanced B pixel; perform a green preprocessing operation on the white balanced G pixel; add a high-frequency component to generate at least one merged green pixel; generate U pixels and V pixels indicating directionality based on the merged green pixel, the white balanced R pixel, and the white balanced B pixel; fuse the merged green pixel with each of the U pixels and V pixels to generate a merged red pixel and a merged blue pixel; and perform inverse white balance on the merged red pixel, the merged green pixel, and the merged blue pixel to output the merged image.

[0009] One aspect of the present invention provides an image signal processing method, comprising: receiving a raw RGB image in Bayer pattern; performing white balance based on a kernel to generate a white balanced G pixel, a white balanced R pixel, and a white balanced B pixel; performing cross-merging on the white balanced G pixel to generate a first green pixel; adding a high-frequency component to the first green pixel to generate at least one merged green pixel; generating U pixels and V pixels indicating vertical cost and horizontal cost based on the merged green pixel, the white balanced R pixel, and the white balanced B pixel; fusing the merged green pixel with each of the U pixels and V pixels to generate a merged image in Bayer pattern including a merged red pixel and a merged blue pixel; and performing inverse white balance on the merged image to output a final merged image.

[0010] However, aspects of the present invention are not limited to those set forth herein. By reference to the detailed description of the present invention given below, the above and other aspects of the present invention will become apparent to those of ordinary skill in the art to which the present invention pertains. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram for illustrating an image signal processor according to some embodiments of the present invention.

[0012] Figure 2 is for illustrating Figure 1 a block diagram of the green generation block shown in

[0013] Figure 3 is for illustrating Figure 1 a block diagram of the red and blue generation block shown in

[0014] Figures 4 to 9 is a diagram for illustrating the operation of an image signal processor according to some embodiments of the present invention.

[0015] Figures 10 to 14 is a diagram for illustrating the operation of an image signal processor according to some embodiments of the present invention.

[0016] Figure 15 is a diagram showing an image sensing device according to some embodiments of the present invention.

[0017] Figure 16 is showing Figure 15 a diagram of the image sensing device shown in

[0018] Figure 17 is a block diagram of an electronic device including a multi-camera module according to some embodiments.

[0019] Figure 18 is Figure 17 a detailed block diagram of the camera module of DETAILED DESCRIPTION

[0020] Hereinafter, various embodiments of the present invention will be described with reference to the drawings.

[0021] Terms such as "unit", "module", and "block" used in the detailed description or functional blocks shown in the drawings may be implemented in the form of hardware, software, or a combination thereof, where the hardware includes an electronic circuit configured to perform a specific function. Terms such as "first", "second", "third", etc. may be used herein to distinguish one element from another.

[0022] As used herein, a pixel or pixel value may refer to information or a value output or obtained from an optical signal generated by a physical pixel element (e.g., of an optical sensor) that defines a Bayer color filter. Hereinafter, an original image may refer to an image on which image signal processing is to be performed per unit original pixel. Hereinafter, a merged image or a merged image may refer to an image generated by performing image signal processing on an original image.

[0023] The original image has a Bayer pattern in which are alternately placed: (i) rows in which Gr pixels and R pixels are sequentially placed, and (ii) rows in which Gb pixels and B pixels are sequentially placed (see Figure 4 ). In this specification, although R refers to a red pixel, B refers to a blue pixel, and Gr pixels and Gb pixels refer to green pixels, a green pixel belonging to or set in a row having a red pixel is expressed as Gr, and a green pixel belonging to or set in a row having a blue pixel is expressed as Gb.

[0024] The merged image is a reduced image, and each of the merged pixels is expressed as an R' pixel, a G' pixel, and a B' pixel. The merged image also has a Bayer pattern in which are alternately placed: rows in which G' pixels and R' pixels are sequentially placed, and rows in which G' pixels and B' pixels are sequentially placed.

[0025] Figure 1 is a block diagram for explaining an image signal processor according to some embodiments of the present invention. Figure 2 is for explaining Figure 1 a block diagram of the green generation block shown in Figure 3 is for explaining Figure 1 a block diagram of the red / blue generation block shown in Figures 4 to 9 is a diagram for explaining the operation of an image signal processor according to some embodiments of the present invention.

[0026] Referring to Figure 1 , the image signal processor 1 may receive a Bayer pattern image to perform a merging operation and output a merged image. That is, the image signal processor 1 may perform image signal processing on the original image to generate a merged image. According to some embodiments, the image signal processor 1 may include a white balance block 10, a green generation block 20, a red / blue generation block 30, and an inverse white balance block 40.

[0027] The image signal processor 1 may perform image signal processing on a kernel basis. For example, in Figures 4 to 9 the embodiment of 11 to 55 the pixels of Figures 4 to 9 are included in kernel X in Figures 4 to 9Only one kernel X is shown and described, but the present invention is not limited thereto, and it can be applied to all pixels included in the original image.

[0028] The white balance block 10 performs white balance for each original pixel. White balance is an operation of subtracting a component corresponding to the characteristics of the light source from the sensed original image to make the color of the object itself clearer. For example, if a first blue light with a high color temperature flashes on the object, the color of the original image is bluer than the color of the object, but the actual color of the object may be a color that does not include blue. Therefore, the white balance block 10 can adjust the sensitivity difference of each color according to the characteristics of the light source.

[0029] The green generation block 20 can generate reduced green pixels from the white-balanced original image. Referring together Figure 2 to, the green generation block 20 may include a green preprocessing block 21, an HF weight calculation block 23, and an HF adder HF block 25 according to some embodiments.

[0030] Referring to Figure 4 in, in the kernel X, G pixels are alternately placed with R pixels or B pixels, and the G pixels in any one row and the G pixels in the next row are also alternately placed.

[0031] The green preprocessing block 21 can generate a first green pixel value from the white-balanced green pixel values, and the white-balanced green pixel values are placed vertically and horizontally around the position (hereinafter referred to as the target position) where the combined green pixel value is generated (i.e., adjacent to each other above and below the target position and to the left and right of the target position). In this specification, although the operation of preprocessing the white-balanced green pixel values placed adjacent to each other above and below the target position and to the left and right of the target position will be described as diamond merging, cross merging, or translational merging according to some embodiments, the green preprocessing operation is not limited to these terms and can be referred to by other terms or be called by other terms.

[0032] Since the number of green pixels in the kernel X is greater than the number of pixels of each of the other colors, it can affect the image resolution more greatly than the pixels of the other colors according to the type of image signal processing. Therefore, the green preprocessing block 21 performs vertical and horizontal merging on four green pixels adjacent to the target position, so that the combined green pixels G' are placed at the same interval without overlapping each other. For example, the combined green pixels G' are not calculated based on the target position of other combined green pixels G'. The basic pixels performing cross merging according to some embodiments can be equidistantly spaced diagonally from each other.

[0033] For example, Figure 5 in, it can be based on Equation 1 based on the placement at P23 , P 32 , P 34 and P 43 's green pixels at position P 33 generate the first green pixel.

[0034] <Equation 1>

[0035] G1_P 33 = (Gr_P 23 + Gb_P 32 + Gb_P 34 + Gr_P 43 ) / 4

[0036] In Equation 1, G1_P 33 refers to the merged first green pixel value generated at position P 33 , Gr_P 23 , Gb_P 32 , Gb_P 34 , Gr_P 43 respectively refer to the green pixel values of the white balance based on the target position P 33 placed at the upper position P 23 , the left position P 32 , the right position P 34 and the lower position P 43 .

[0037] Similarly, in Figure 6 , the green preprocessing block 21 can perform merging based on the green pixel values placed at P 00 , P 01 , P 12 and P 21 to generate G1_P 11 . Additionally, the green preprocessing block 21 can generate other merged first green pixel values G1_P 13 , G1_P 15 , G1_P 31 , G1_P 35 , G1_P 51 , G1_P 53 and G1_P 55 from the green pixels of the adjacent upper, lower, left, and right white balances (the positions of the vertices represented by the dotted lines) according to Equation 1.

[0038] The HF weight calculation block 23 calculates a weight W that reflects the high-frequency components of the Gr and Gb pixels based on the R and B pixels in kernel X. The HF adder block 25 adds the high-frequency component values of the green pixels reflecting the weight to the first green pixel value G1 to generate a merged green pixel G'. The values used in the HF adder block 25 are the white balance values in the green preprocessing block 21. The high-frequency component can be a value obtained by subtracting the average B (or R) pixel value from the original B (or R) pixel value at the center position Pcenter of the kernel.

[0039] Specifically, the merged green pixel value G' can be generated according to Equation 2.

[0040] <Equation 2>

[0041] G' = G1_P target + (B Raw - B Aver ) × W

[0042] In Equation 2, G' represents the merged green pixel value, G1 represents the first green pixel of cross-merging, B Raw represents the original B pixel value at the center position of the kernel, B Aver represents the average value of the original B pixels included in the kernel, and W represents the weight of the reflectance of the high-frequency component. According to some embodiments, W can be set as a user parameter according to the system and can be a value modified by reflecting the operation of the image signal processor according to some embodiments. B Aver is the average value of the B pixels included in kernel X. B Aver .

[0043] <Equation 3>

[0044]

[0045] In Equation 3, B_P ij is the B pixel at the P ij coordinate belonging to the kernel, and ∑B_P ij is the sum of the B pixel values in the kernel.

[0046] In Figure 7 embodiments, although nine original B pixels are shown based on the center position P 33 of kernel X, when the position of the kernel changes according to other embodiments, the original pixels that become the basis for high-frequency component calculation can be changed. In the illustrated embodiments, although the original B pixels in the 5×5 kernel based on the center position P33 where the merged green pixel G' is generated are shown, the original R pixels can also be used according to the center position.

[0047] The red-blue generation block 30 generates a combined red pixel R' and a combined blue pixel B' based on R pixels and B pixels. Since red, blue, and green have independent bands and also include high-frequency components, compared to the case of applying general image signal processing to the original pixels themselves, the case of generating the combined red pixel R' and blue pixel B' (i.e., U pixels and V pixels as the difference between color channels) based on color channel differences that do not include high-frequency components can be stronger in terms of noise.

[0048] As Figure 3 shown, the red-blue generation block 30 according to some embodiments may include a direction calculation block 31, a UV generation block 33, and a Bayer pattern generation block 35.

[0049] The direction calculation block 31 calculates the horizontal cost and the vertical cost in the kernel X.

[0050] According to some embodiments, the horizontal cost Hcost is a reciprocal value obtained by summing the luminance differences between horizontally adjacent pixels of the same color in the kernel, and the vertical cost Vcost may be a reciprocal value obtained by summing the luminance differences between vertically adjacent pixels of the same color in the kernel. Specifically, the vertical cost Vcost and the horizontal cost Hcost can be calculated as in Equation 4. In this case, the vertical cost Vcost and the horizontal cost Hcost can be inversely proportional to each other.

[0051] <Equation 4>

[0052]

[0053]

[0054] In Equation 4, Hcost is the horizontal cost, ∑(R H Diff) is a value obtained by summing the differences between horizontally adjacent R pixels, ∑(B H Diff) is a value obtained by summing the differences between horizontally adjacent B pixels, ∑(Gr H Diff) is a value obtained by summing the differences between horizontally adjacent Gr pixels, ∑(Gb H Diff) is a value obtained by summing the differences between horizontally adjacent Gb pixels. Vcost is the vertical cost, ∑(R V Diff) is a value obtained by summing the differences between vertically adjacent R pixels, ∑(B V Diff) is a value obtained by summing the differences between vertically adjacent B pixels, ∑(Gr V Diff) is a value obtained by summing the differences between vertically adjacent Gr pixels, ∑(Gb VDiff) is a value obtained by summing the differences between vertically adjacent Gb pixels.

[0055] More specifically, referring to Figure 7 , the direction calculation block 31 calculates the sum of the differences between all B_Ps included in the kernel X for ∑(B H Diff), the difference between B_P 11 and B_P 13 (hereinafter, the difference is expressed as B_P 11 :B_P 13 ), and B_P 13 :B_P 15 , B_P 31 :B_P 33 , B_P 33 :B_P 35 , B_P 51 :B_P 53 and B_P 53 :B_P 55 to obtain a value. Similarly, the direction calculation block 31 can also calculate ∑(R H Diff), ∑(Gr H Diff), and ∑(Gb H Diff) for the Gr, Gb, and R color channels, and can calculate the horizontal cost Hcost as the reciprocal of the value obtained by summing ∑(B H Diff), ∑(R H Diff), ∑(Gr H Diff), and ∑(Gb H Diff).

[0056] If the luminance difference between adjacent G pixels, adjacent R pixels, or adjacent B pixels placed in the same row in the kernel X is large, the horizontal cost Hcost increases. That is, when there are pixel values with a large difference in the vertical direction, it can be determined that the directionality of the pixel values has a small luminance value difference in the horizontal direction, and the vertical cost increases.

[0057] The UV generation block 33 generates UV pixels as a color difference signal based on the vertical cost Vcost and the horizontal cost Hcost. In Figure 7 , since the R pixels in the kernel X are placed on the same horizontal line as the Gr pixels and on the same vertical line as the Gb pixels, the UV generation block 33 reflects the direction cost of that direction according to the vertical color difference or the horizontal color difference when calculating the color difference.

[0058] Specifically, the UV generation block 33 can reflect each direction cost on the horizontal color difference between R - Gr and the vertical color difference between R - Gb to generate the V value as shown in Equation 5.

[0059] The V pixel can be a value obtained by adding the color difference value between the average R pixel value reflecting the horizontal cost and the average Gr pixel value and the color difference value between the average R pixel value reflecting the vertical cost and the average Gb pixel value. According to some embodiments, as shown in Equation 5, the V pixel can be calculated through the following steps: from the average R of all R pixels included in kernel X Aver subtract the average Gr of the Gr pixels placed on the same horizontal line (e.g., the same row) as the R pixel Aver then multiply by the horizontal cost Hcost, from the average R of all R pixels included in kernel X Aver subtract the average Gb of the Gb pixels placed on the same vertical line (e.g., the same column) as the R pixel Aver then multiply by the vertical cost Vcost and add. At this time, kernel X has the coordinates for generating the V pixel as the center position. Since Figure 7 shows P 33 as the center position, so Equation 5 for generating the V pixel can be applied to, for example, a kernel with V_P 13 as the center position.

[0060] <Equation 5>

[0061] V = (R Aver - Gr Aver ) × Hcost + (R Aver - Gb Aver ) × Vcost

[0062] <Equation 6>

[0063]

[0064] <Equation 7>

[0065]

[0066] <Equation 8>

[0067]

[0068] In Equations 6 to 8, R_P ij is the R pixel belonging to the kernel at the coordinates of P ij , ∑R_P ij is the sum of the R pixel values in the kernel, Gr_P ij is the Gr pixel belonging to the kernel at the coordinates of P ij , ∑Gr_P ij is the sum of the Gr pixel values in the kernel, Gb_P ij is the Gb pixel belonging to the kernel at the coordinates of P ijGb pixel at the coordinate, ∑Gb_P ij is the sum of the Gb pixel values in the kernel. Here, i and j are integers and are indices representing the coordinates in the kernel (e.g., along the rows and columns of the kernel respectively).

[0069] Referring to Equation 9, the U pixel can be calculated through the following steps: from the average value B of all B pixels included in kernel X Aver subtract the average value Gb of the Gb pixels placed on the same horizontal line (e.g., the same row) as the B pixels Aver then multiply by the horizontal cost Hcost, and from the average value B of all B pixels included in kernel X Aver subtract the average value Gr of the Gr pixels placed on the same vertical line (e.g., the same column) as the B pixels Aver then multiply by the vertical cost Vcost and add them up.

[0070] <Equation 9>

[0071] U = (B Aver - Gb Aver ) × Hcost + (B Aver - Gr Aver ) × Vcost

[0072] In Equation 9, the average value B of the B pixels Aver can be the average value of all B pixels included in kernel X, as in Equation 10.

[0073] <Equation 10>

[0074]

[0075] As Figure 8 shown, the generated U pixels and V pixels are placed at positions P 13 、P 53 、P 31 and P 35 .

[0076] The Bayer pattern generation block 35 can generate a combined Bayer pattern image by fusing the G’ pixels generated in the green generation block 20 with the U pixels and V pixels generated in the UV generation block 33. Referring to Figure 6 、 Figure 8 and Figure 9 , the pattern generation block 35 adds the UV image ( Figure 8 ) to the G’ image ( Figure 6 ) to generate the R’G’B’ image ( Figure 9 ).

[0077] That is, the Bayer pattern generation block 35 adds the merged green pixels G’ to the U pixels and V pixels generated from the color difference information between R-G and B-G to generate the merged red pixels R’ and the merged blue pixels B’. At this time, the merged R’ pixels, G’ pixels, and B’ pixels can be placed in the Bayer pattern at a reduced interval of the original image. The interval between the R’ pixels, G’ pixels, and B’ pixels can be changed according to the reduction ratio. Another reduced embodiment will be described in Figure 10 below.

[0078] Since the white balance block performs white balance to perform the merging of the original image ( Figure 2 10), inverse white balance is performed on the image including the R’ pixels, G’ pixels, and B’ pixels output from the Bayer pattern generation block 35 ( Figure 2 40). That is, the optical characteristics of each color channel are reflected again, and the final merged pattern is output.

[0079] In this way, when converting the original image into a merged image, merging is performed to have uniform sampling points centered on green, and the merged image is generated by reflecting or indicating directionality using the UV pixel values, while reducing the occurrence of false colors. Therefore, image signal processing can be performed without substantial loss of image details. That is, even for an image sensor with high resolution, a high-resolution image can be obtained while eliminating the jagged artifacts and false colors generated according to image signal processing.

[0080] Figures 10 to 14 is a diagram for explaining the operation of an image signal processor according to some embodiments of the present invention.

[0081] Figures 4 to 9 is an example in which the original image is reduced at a ratio of 1 / 4, Figures 10 to 14 is an example in which the original image is reduced at a ratio of 1 / 16. For ease of explanation, mainly the differences from Figures 4 to 9 will be described, and the remaining configurations can be processed in the same or similar manner as the embodiments of Figures 4 to 9 .

[0082] For ease of explanation, although Figures 10 to 14 shows an 8×7 pixel array and is described as a 5×5 kernel X, the embodiments of the present invention are not limited thereto, the kernel size can be enlarged, or this can also be applied to adjacent pixel arrays.

[0083] Referring to Figure 10 , according to some embodiments, the image signal processor 1 receives an input signal (i.e., the original image in Bayer pattern), and can perform white balance on the original pixel array of the Bayer pattern.

[0084] Referring to Figure 11, the image signal processor 1 performs green preprocessing operations such that the sampling (pixel positions) becomes uniform based on or relative to the center position P44 of the kernel X. That is, the green preprocessing operations can be performed to have uniform sampling points such that there is no overlap between the first green pixels after the preprocessing operations. The green pixels as inputs for the green preprocessing operations can be four white-balanced green pixels set above, below, to the left, and to the right of the target position where the first green pixels are located. Although according to some embodiments, the green preprocessing operations can also be referred to by other names or be called by other names, such as diamond merging, cross merging, or fan merging, it is not limited by the terminology.

[0085] According to the illustrated embodiment, if cross merging of G pixels is performed at positions D1, D2, D3, and D4, then the first green pixels can be generated at positions P Figure 11 as shown in 22 , P 26 , P 62 and P 66 respectively.

[0086] The image signal processor 1 can generate a merged G' pixel on the first green pixel according to Equation 2. The image signal processor can add a weighted high-frequency component to the first green pixel to generate the merged G' pixel as shown in Figure 12 . According to some embodiments, the image signal processor 1 can generate a merged G' pixel from the first green pixel according to Equation 2.

[0087] As shown in Figure 13 , the image signal processor 1 can generate U pixel values and V pixel values respectively based on the G' pixels as well as the R pixels and B pixels of the original image.

[0088] Specifically, as in Equation 5, the V pixel can be generated by the following steps: subtracting the average value Gr Aver of the Gr pixels placed on the same horizontal line (e.g., row) as the R pixels from the average value R Aver of all the R pixels included in the kernel X, then multiplying by the horizontal cost Hcost, subtracting the average value Gb Aver of the Gb pixels placed on the same vertical line (e.g., column) as the R pixels from the average value R Aver of all the R pixels included in the kernel X, and then multiplying by the vertical cost Vcost and adding them together.

[0089] Referring to Equation 9, the U pixel can be calculated by the following steps: subtracting the average value Gb Aver of the Gb pixels placed at the same horizontal line (e.g., row) as the B pixels from the average value B Aver, and then multiply by the horizontal cost Hcost, from the average value B of all B pixels included in the core X Aver Subtract the average value Gr of the Gr pixels placed on the same vertical line (e.g., column) as the B pixels Aver , and then multiply by the vertical cost Vcost and add them up.

[0090] Refer to Figure 14 , the image signal processor 1 can fuse the U pixel value and the V pixel value with the G' pixel to generate an R' pixel and a B' pixel respectively. The image signal processor 1 can perform reverse white balance on the fused image to output a combined R'G'B' pixel image.

[0091] In Figure 14 the output combined image of, compared with the combined image of Figure 9 , according to the reduction ratio of the same core size, the difference in the interval between the combined pixels can occur.

[0092] Figure 15 is a diagram showing an image sensing device according to some embodiments of the present invention.

[0093] Refer to Figure 15 , according to some embodiments, Figure 1 the image signal processor of can be included in the image sensing device. The image sensor 200 includes a pixel array 210, a row driver 220, a correlated double sampling (CDS) block 230, an analog-to-digital converter (ADC) 240, a ramp generator 260, a timing generator 270, a control register block 280, and a buffer 290.

[0094] The image sensor 200 detects or senses an object 400 photographed through a lens 500 under the control of a digital signal processor (DSP) 150, and the DSP 150 can output the image sensed and output by the image sensor 200 to a display unit 300. At this time, the display unit 300 includes all devices capable of outputting or displaying the image. For example, the display unit 300 can refer to a computer, a mobile communication device, and / or other video output terminals.

[0095] The DSP 150 includes a camera control device 110, an image signal processor (ISP) 100, and an interface (I / F) 120.

[0096] The camera control device 110 controls the operation of the control register block 280. Although the camera control device 110 can use I2C (Inter-Integrated Circuit) to control the operation of the image sensor 200 (i.e., the control register block 280), the embodiments of the present invention are not limited thereto.

[0097] The ISP 100 can include according to some embodiments such asFigure 1 The ISP 100 receives the image data output from the buffer 290, processes or manipulates the received image data, and outputs the processed or manipulated image data to the display unit 300 through the I / F 120.

[0098] Despite Figure 15 , the ISP 100 is shown as being included in the DSP 150, but the ISP 100 may be additionally or alternatively included in the image sensor 200 in some embodiments. In addition, the image sensor 200 and the ISP 100 may be implemented as a single package (e.g., as an MCP (multi-chip package)).

[0099] The pixel array 210 may be implemented as a plurality of optical sensing elements, such as photodiodes or pinned diodes. The raw image data output from the pixel array 210 through the CDS block 230 and the ADC 240 may be Bayer image data formed in a Bayer format.

[0100] The Bayer image data is processed into RGB format data (red, green, and blue format data) by the ISP 100 , and output to the display unit 300 .

[0101] Figure 16 It is shown Figure 15 FIG. 1 is a diagram of an image sensing device shown in FIG.

[0102] Reference Figure 16 , the image sensor 700 may be implemented as a plurality of stacked layers. According to some embodiments, the pixel array 210 is implemented in the first layer (Layer 1), and the remaining configuration (ie, logic circuit) may be implemented in the second layer (Layer 2). Figure 15 In addition to the pixel array 210 in the image sensor 200 shown in FIG. 2 , the logic circuit may further include the remaining components 220 to 290. That is, the pixel array region and the logic circuit region may be stacked at the wafer level.

[0103] The first layer (Layer 1) may include a sensing area SA including a plurality of pixels PX and a first pad area PA1 disposed around the sensing area SA. The first pad area PA1 includes a plurality of upper pads PAD1, and the plurality of upper pads PAD1 may be connected to pads PAD21 and PAD22 disposed in a second pad area PA2 of the second layer Layer 2 and a logic circuit LOGIC, for example, through conductive vias or the like.

[0104] According to some embodiments, pads PAD21 and PAD22 may be input interface pads, and pad PAD23 may be an output interface pad.

[0105] Figure 17 is a block diagram of an electronic device including a multi-camera module according to some embodiments. Figure 18 is Figure 17 a detailed block diagram of the camera module.

[0106] Referring to Figure 17 , the electronic device 1000 may include a camera module group 1100, an application processor 1200, a PMIC 1300, and an external memory 1400.

[0107] The camera module group 1100 may include a plurality of camera modules 1100a, 1100b, and 1100c. Although the drawings illustrate an embodiment in which three camera modules 1100a, 1100b, and 1100c are placed, the embodiment is not limited thereto, and there may be fewer or more camera modules. In some embodiments, the camera module group 1100 may be implemented by being modified to include only two camera modules. Additionally, in some embodiments, the camera module group 1100 may be modified to include n (n is a natural number equal to or greater than 4) camera modules.

[0108] Hereinafter, the detailed configuration of the camera module 1100b will be described in more detail with reference to Figure 18 , but the following description may also be similarly applied to other camera modules 1100a and 1100c according to embodiments.

[0109] Referring to Figure 18 , the camera module 1100b includes a prism 1105, an optical path folding element (hereinafter referred to as "OPFE") 1110, an actuator 1130, an image sensing device 1140, and a storage device 1150.

[0110] The prism 1105 may include a reflection surface 1107 of a light reflecting material to deform the path of light L (referred to herein as incident light L) incident from outside the camera module 1100b.

[0111] In some embodiments, the prism 1105 may change or redirect the path of the incident light L incident in the first direction X to, for example, a second direction Y perpendicular to the first direction X. Additionally, the prism 1105 may rotate the reflection surface 1107 of the light reflecting material in the direction A around the central axis 1106, and / or may rotate the central axis 1106 in the direction B to change or redirect the path of the light L incident in the first direction X in the vertical second direction Y. At this time, the OPFE 1110 may also move in a third direction Z (e.g., perpendicular to the first direction X and the second direction Y).

[0112] In some embodiments, as shown, although the maximum rotation angle of the prism 1105 in the A direction can be 15 degrees or less in the positive (+) A direction and can be greater than 15 degrees in the negative (-) A direction, the embodiments are not limited thereto.

[0113] In some embodiments, the prism 1105 can be moved approximately 20 degrees in the positive (+) B direction or the negative (-) B direction, or can be moved between 10 degrees and 20 degrees, or can be moved between 15 degrees and 20 degrees. Here, the movement angle can be moved by the same angle in the positive (+) B direction or the negative (-) B direction, or can be moved by an almost similar angle within a range of approximately 1 degree.

[0114] In some embodiments, the prism 1105 can move the reflection surface 1107 of the light reflecting material in a third direction (e.g., direction Z) parallel to the extension direction of the central axis 1106.

[0115] The OPFE 1110 can include, for example, optical lenses including m (where m is a natural number) groups. The m lenses can be moved in the second direction Y to change the optical zoom ratio of the camera module 1100b. For example, when the basic optical zoom ratio of the camera module 1100b is defined as z, if the m optical lenses included in the OPFE 1110 are moved, the optical zoom ratio of the camera module 1100b can be changed to 3z or 5z or a larger optical zoom ratio.

[0116] The actuator 1130 can move the OPFE 1110 or the optical lenses (collectively referred to as optical lenses) to a specific position. For example, the actuator 1130 can adjust the position of the optical lenses so that the image sensor 1142 is located at the focal length of the optical lenses for accurate sensing.

[0117] The image sensing device 1140 can include an image sensor 1142, control logic 1144, and a memory 1146. The image sensor 1142 can detect or sense an image of a sensing target using the light L provided by the optical lenses. The control logic 1144 can control the overall operation of the camera module 1100b. For example, the control logic 1144 can control the operation of the camera module 1100b according to a control signal provided through the control signal line CSLb.

[0118] The memory 1146 may store information for the operation of the camera module 1100b, such as calibration data 1147. The calibration data 1147 may include information used by the camera module 1100b to generate image data using incident light L. The calibration data 1147 may include, for example, information about the degree of rotation, information about the focal length, information about the above optical axis, etc. When the camera module 1100b is implemented as a multi-state camera in which the focal length can be changed according to the position of the optical lens, the calibration data 1147 may include the focal length values for each position (or each state) of the optical lens and information about autofocus.

[0119] The storage device 1150 may store the image data sensed by the image sensor 1142. The storage device 1150 may be implemented outside the image sensing device 1140 and may be implemented in the form of being stacked with the sensor chip constituting the image sensing device 1140. In some embodiments, although the storage device 1150 may be implemented as an EEPROM (Electrically Erasable Programmable Read-Only Memory), the embodiments of the present invention are not limited thereto.

[0120] Referring Figure 17 and Figure 18 , in some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Accordingly, each of the plurality of camera modules 1100a, 1100b, and 1100c may include the same or different calibration data 1147 according to the operation of the actuator 1130 included therein.

[0121] In some embodiments, one camera module (e.g., 1100b) of the plurality of camera modules 1100a, 1100b, and 1100c may be a folded lens type camera module including the prism 1105 and the OPFE 1110 described above, and the remaining camera modules (e.g., 1100a and 1100c) may be vertical type camera modules that do not include the prism 1105 and the OPFE 1110, but the embodiments of the present invention are not limited thereto.

[0122] In some embodiments, one camera module (e.g., 1100c) of the plurality of camera modules 1100a, 1100b, and 1100c may be a vertical type depth camera that uses IR (infrared ray) or additionally uses infrared light to extract depth information. In this case, the application processor 1200 may fuse the image data provided from such a depth camera with the image data provided from another camera module (e.g., 1100a or 1100b) to generate a 3D depth image.

[0123] In some embodiments, at least two of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., 1100a, 1100b) may have different fields of view from each other. In this case, for example, although the optical lenses of at least two of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., 1100a and 1100b) may be different from each other, embodiments of the present invention are not limited thereto.

[0124] In addition, in some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may have a different field of view from each other. In this case, although the optical lenses included in each of the plurality of camera modules 1100a, 1100b, and 1100c may also be different from each other, embodiments of the present invention are not limited thereto.

[0125] In some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may be placed physically separated from each other. That is, instead of dividing or sharing respective portions of the sensing area of an image sensor 1142, independent image sensors 1142 may be included in each of the plurality of camera modules 1100a, 1100b, and 1100c.

[0126] Referring again to Figure 17 , the application processor 1200 may include an image processing device 1210, a memory controller 1220, and an internal memory 1230. The application processor 1200 may be implemented separately from the plurality of camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the plurality of camera modules 1100a, 1100b, and 1100c may be implemented separately using separate semiconductor chips or distributed in separate semiconductor chips.

[0127] The image processing device 1210 may include a plurality of sub-image processors 1212a, 1212b, and 1212c, an image generator 1214, and a camera module controller 1216.

[0128] The image processing device 1210 may include a plurality of sub-image processors 1212a, 1212b, and 1212c corresponding to the number of the plurality of camera modules 1100a, 1100b, and 1100c.

[0129] The image data generated by each of the camera modules 1100a, 1100b, and 1100c can be provided to the corresponding sub-image processors 1212a, 1212b, and 1212c through image signal lines ISLa, ISLb, and ISLc that are separate or distinguishable from each other. For example, the image data generated by the camera module 1100a can be provided to the sub-image processor 1212a through the image signal line ISLa, the image data generated by the camera module 1100b can be provided to the sub-image processor 1212b through the image signal line ISLb, and the image data generated by the camera module 1100c can be provided to the sub-image processor 1212c through the image signal line ISLc. Although such image data transmission can be performed using, for example, a camera serial interface (CSI) based on MIPI (Mobile Industry Processor Interface), the embodiments are not limited thereto.

[0130] Meanwhile, in some embodiments, one sub-image processor can be placed corresponding to multiple camera modules. For example, the sub-image processors 1212a and 1212c may not be implemented separately from each other as shown, but may be implemented as a fused single sub-image processor, and the image data provided from the camera modules 1100a and 1100c can be provided to the fused sub-image processor after being selected by a selection element (e.g., a multiplexer).

[0131] The image data provided to each of the sub-image processors 1212a, 1212b, and 1212c can be provided to the image generator 1214. The image generator 1214 can generate an output image using the image data provided from each of the sub-image processors 1212a, 1212b, and 1212c according to image generation information or a mode signal.

[0132] Specifically, the image generator 1214 can fuse at least some of the image data generated by the camera modules 1100a, 1100b, and 1100c having different fields of view according to image generation information or a mode signal to generate an output image. Additionally, the image generator 1214 can select any one of the image data generated by the camera modules 1100a, 1100b, or 1100c having different fields of view according to image generation information or a mode signal to generate an output image.

[0133] In some embodiments, the image generation information can include a zoom signal (or a zoom factor). Additionally, in some embodiments, the mode signal can be, for example, a signal based on a mode selected by a user.

[0134] When the image generation information is a zoom signal (zoom factor) and each of the camera modules 1100a, 1100b, and 1100c has a respective field of view (angle of view) different from each other, the image generator 1214 may perform different operations according to the type of the zoom signal. For example, when the zoom signal is a first signal, after fusing the image data output from the camera module 1100a and the image data output from the camera module 1100c, the fused image signal may be used to generate an output image, and the image data not used for fusion may be output from the camera module 1100b. If the zoom signal is a second signal different from the first signal, the image generator 1214 may not fuse the image data and may select any one of the image data output from the camera modules 1100a, 1100b, or 1100c to generate an output image. However, embodiments of the present invention are not limited thereto, and the method for processing image data may be modified and implemented as needed.

[0135] In some embodiments, the image generator 1214 receives image data with different exposure times from at least one of the plurality of sub-image processors 1212a, 1212b, and 1212c, and performs HDR (High Dynamic Range) processing on the plurality of image data, thereby generating fused image data with an increased dynamic range.

[0136] Figure 1 The image signal processor 1 shown in may be used in various applications according to various embodiments. According to some embodiments, the image signal processor 1 may be implemented in the logic 1144 in the Figure 18 camera module 1100b, or the image signal processor 1 may be implemented in the Figure 17 sub-image processors 1212a, 1212b, 1212c, or the image signal processor 1 may be implemented in the image generator 1214.

[0137] The camera module controller 1216 may provide control signals to each of the camera modules 1100a, 1100b, and 1100c. The control signals generated from the camera module controller 1216 may be provided to the corresponding camera modules 1100a, 1100b, and 1100c through separate control signal lines CSLa, CSLb, and CSLc.

[0138] Any one of the plurality of camera modules 1100a, 1100b, and 1100c may be designated as a main camera (e.g., 1100b) according to image generation information or a mode signal including a zoom signal, and the remaining camera modules (e.g., 1100a and 1100c) may be designated as slave cameras. Such information is included in a control signal and may be provided to the corresponding camera modules 1100a, 1100b, and 1100c through control signal lines CSLa, CSLb, and CSLc that are separate from or distinguishable from each other.

[0139] The camera modules operating as the main camera and the slave cameras may change or vary according to a zoom factor or an operation mode signal. For example, when the viewing angle of camera module 1100a is wider than the viewing angle of camera module 1100b and the zoom factor indicates a low zoom ratio, camera module 1100b may operate as the main camera and camera module 1100a may operate as the slave camera. Conversely, when the zoom factor indicates a high zoom ratio, camera module 1100a may operate as the main camera and camera module 1100b may operate as the slave camera.

[0140] In some embodiments, the control signals provided from the slave camera module controller 1216 to the respective camera modules 1100a, 1100b, and 1100c may include a synchronization enable signal. For example, when camera module 1100b is the main camera and camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 may send a synchronization enable signal to camera module 1100b. The camera module 1100b provided with such a synchronization enable signal generates a synchronization signal based on the provided synchronization enable signal and may provide the generated synchronization signal to camera modules 1100a and 1100c through a synchronization signal line SSL. Camera module 1100b and camera modules 1100a and 1100c may transmit image data to the application processor 1200 in synchronization with such a synchronization signal.

[0141] In some embodiments, the control signals provided from the slave camera module controller 1216 to the plurality of camera modules 1100a, 1100b, and 1100c may include mode information according to a mode signal. Based on the mode information, the plurality of camera modules 1100a, 1100b, and 1100c may operate in a first operation mode or a second operation mode with respect to a sensing speed.

[0142] Multiple camera modules 1100a, 1100b, and / or 1100c generate image signals at a first speed in a first operation mode (e.g., generate image signals at a first frame rate), encode the image signals at a second speed higher than the first speed (e.g., encode image signals at a second frame rate higher than the first frame rate), and may send the encoded image signals to the application processor 1200. At this time, the second speed may be equal to or less than 30 times the first speed.

[0143] The application processor 1200 stores the received image signals (i.e., the encoded image signals) in the internal memory 1230 provided inside the application processor 1200 or in the external memory 1400 outside. After that, the application processor 1200 may read or decode the encoded image signals from the internal memory 1230 or the external memory 1400, and may display the image data generated based on the decoded image signals. For example, the corresponding sub - processors among the multiple sub - processors 1212a, 1212b, and 1212c of the image processing device 1210 may perform decoding and may perform image processing on the decoded image signals.

[0144] In a second operation mode, multiple camera modules 1100a, 1100b, and / or 1100c generate image signals at a third speed lower than the first speed (e.g., generate image signals at a third frame rate lower than the first frame rate), and may send the image signals to the application processor 1200. The image signals provided to the application processor 1200 may be non - encoded signals. The application processor 1200 may perform image processing on the received image signals or store the image signals in the internal memory 1230 or the external memory 1400.

[0145] The PMIC 1300 may supply power (e.g., a power supply voltage) to each of the multiple camera modules 1100a, 1100b, and 1100c. For example, under the control of the application processor 1200, the PMIC 1300 may supply a first power level to the camera module 1100a through the power signal line PSLa, supply a second power level to the camera module 1100b through the power signal line PSLb, and supply a third power level to the camera module 1100c through the power signal line PSLc.

[0146] The PMIC 1300 generates power corresponding to each of the plurality of camera modules 1100a, 1100b, and 1100c in response to a power control signal PCON from the application processor 1200, and may adjust the level of the power. The power control signal PCON may include power adjustment signals for each operation mode of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the operation mode may include a low power mode, and at this time, the power control signal PCON may include information about the camera module operating in the low power mode and the power level to be set. The power levels supplied to each of the plurality of camera modules 1100a, 1100b, and 1100c may be the same as or different from each other. Additionally, the power levels may be changed dynamically.

[0147] Upon concluding the detailed description, those skilled in the art will understand that many changes and modifications can be made to the preferred embodiments without materially departing from the principles of the present invention. Accordingly, the disclosed preferred embodiments of the present invention are used only in a general and descriptive sense and not for purposes of limitation.

Claims

1. An image signal processor, comprising: A white balance circuit configured to perform white balance on a raw RGB image in a Bayer pattern received from an image sensor to generate white-balanced G pixels, white-balanced R pixels, and white-balanced B pixels; A green generation circuit configured to perform cross-merging on the white-balanced G pixels to generate first green pixels, and add a high-frequency component with a preset weight to the first green pixels to generate merged green pixels; A red / blue generation circuit configured to generate U pixels and V pixels indicating directionality based on the merged green pixels, the white-balanced R pixels, and the white-balanced B pixels, and fuse the merged green pixels with each of the U pixels and the V pixels to respectively generate merged red pixels and merged blue pixels; And An inverse white balance circuit configured to perform inverse white balance on the merged red pixels, the merged green pixels, and the merged blue pixels to output a final merged image.

2. The image signal processor according to claim 1, wherein, The white-balanced G pixels include respective white-balanced G pixels, and the cross-merging generates the first green pixels based on the respective white-balanced G pixels vertically and horizontally set with respect to a target position of the merged green pixels.

3. The image signal processor according to claim 2, wherein, The cross-merging generates the first green pixels with uniform sampling points based on a center position of a kernel unit.

4. The image signal processor according to claim 3, wherein: The kernel unit includes a subset of pixels of the Bayer pattern of the raw RGB image; The respective white-balanced G pixels include a first pixel and a second pixel, the first pixel is in the same row of the kernel unit as the target position, the second pixel is in the same column of the kernel unit as the target position, one of the first pixel and the second pixel includes a Gb pixel, and the other of the first pixel and the second pixel includes a Gr pixel; and The U pixels and the V pixels respectively indicate the directionality of the Gb pixels and the Gr pixels.

5. The image signal processor according to claim 1, wherein, The high-frequency component is obtained by subtracting an average B pixel value from the white-balanced B pixels or by subtracting an average R pixel value from the white-balanced R pixels.

6. The image signal processor according to claim 1, wherein, The V pixels are obtained by adding a color difference value between an average R pixel value indicating a first cost and an average Gr pixel value and a color difference value between an average R pixel value indicating a second cost and an average Gb pixel value.

7. The image signal processor according to claim 1, wherein, The U pixels include a value obtained by adding a color difference value between an average B pixel value indicating a first cost and an average Gb pixel value and a color difference value between an average B pixel value indicating a second cost and an average Gr pixel value.

8. The image signal processor according to claim 5, wherein, The average B pixel value is an average of a plurality of B pixels included in the kernel unit, and wherein the average R pixel value is an average of a plurality of R pixels included in the kernel unit.

9. The image signal processor according to claim 6, wherein, The first cost is a reciprocal of a value obtained by summing luminance differences between horizontally adjacent pixels of the same color in the kernel unit, and The second cost is a reciprocal of a value obtained by summing luminance differences between vertically adjacent pixels of the same color in the kernel unit.

10. The image signal processor according to claim 1, wherein, The red and blue generation circuit is configured to add the merged green pixel to the V pixel to generate the merged red pixel, and add the merged green pixel to the U pixel to generate the merged blue pixel.

11. An electronic device, comprising: An image sensor configured to detect light reflected from an object and output a raw RGB image in a Bayer pattern; And An image signal processor configured to perform merging on the raw RGB image to generate a merged image, wherein the image signal processor is configured to perform an operation including the following steps: Perform white balance on the raw RGB image to generate a white-balanced G pixel, a white-balanced R pixel, and a white-balanced B pixel; Perform a green preprocessing operation on the white-balanced G pixel and add a high-frequency component to generate a merged green pixel; Generate a U pixel and a V pixel indicating directionality based on the merged green pixel, the white-balanced R pixel, and the white-balanced B pixel; Fuse the merged green pixel with each of the U pixel and the V pixel to respectively generate a merged red pixel and a merged blue pixel; and Perform inverse white balance on the merged red pixel, the merged green pixel, and the merged blue pixel to output the merged image.

12. The electronic device according to claim 11, wherein, The electronic device includes at least one camera module, and wherein the camera module includes the image sensor and the image signal processor.

13. The electronic device according to claim 11, wherein,The image sensor includes a plurality of image sensors, wherein the image signal processor includes a plurality of image signal processors, and wherein the electronic device further includes: A plurality of physically separated camera modules, wherein each of the plurality of camera modules includes a corresponding image sensor of the plurality of image sensors and is configured to output the raw RGB image; and An application processor including an image processing device, a memory controller, and an internal memory, wherein the image processing device includes a plurality of sub-image processors corresponding to the plurality of camera modules respectively, and wherein each of the plurality of sub-image processors includes a corresponding image signal processor of the plurality of image signal processors.

14. The electronic device according to claim 11, wherein, The white-balanced G pixel includes respective white-balanced G pixels, and the green preprocessing operation generates a first green pixel based on the respective white-balanced G pixels arranged vertically and horizontally adjacent to each other around the target position of the merged green pixel.

15. The electronic device according to claim 14, wherein, The high-frequency component is obtained by applying a preset weight to a value obtained by subtracting an average B pixel value from the white-balanced B pixel or by subtracting an average R pixel value from the white-balanced R pixel.

16. The electronic device according to claim 11, wherein: The U pixel is obtained by adding a color difference value between the average B pixel value indicating a first cost and the average Gb pixel value and a color difference value between the average B pixel value indicating a second cost and the average Gr pixel value, and The V pixel is obtained by adding a color difference value between the average R pixel value indicating the first cost and the average Gr pixel value and a color difference value between the average R pixel value indicating the second cost and the average Gb pixel value.

17. The electronic device according to claim 16, wherein, The first cost is the reciprocal value obtained by summing the luminance differences between horizontally adjacent pixels of the same color in the kernel, and the second cost is the reciprocal value obtained by summing the luminance differences between vertically adjacent pixels of the same color in the kernel.

18. The electronic device according to claim 16, wherein: The average Gb pixel value is the average of the Gb pixels of the white balance that are in the same row as each B pixel in the kernel, and the average Gr pixel value is the average of the Gr pixels of the white balance that are in the same row as each R pixel in the kernel.

19. The electronic device according to claim 14, wherein: The kernel includes a subset of the pixels of the Bayer pattern of the original RGB image; Each G pixel of the white balance includes a first pixel that is in the same row of the kernel as the target position and a second pixel that is in the same column of the kernel as the target position. One of the first pixel and the second pixel includes a Gb pixel, and the other of the first pixel and the second pixel includes a Gr pixel; and The U pixel and the V pixel respectively indicate the directionality of the Gb pixel and the Gr pixel.

20. An image signal processing method, comprising: Receiving an original RGB image in a Bayer pattern; Performing white balance on the original RGB image based on the kernel to generate G pixels of the white balance, R pixels of the white balance, and B pixels of the white balance; Performing cross-merging on the G pixels of the white balance to generate first green pixels; Adding a high-frequency component to the first green pixels to generate merged green pixels; Generating U pixels and V pixels indicating vertical cost and horizontal cost based on the merged green pixels, the R pixels of the white balance, and the B pixels of the white balance; Fusing the merged green pixels with each of the U pixels and the V pixels to generate a merged image in a Bayer pattern including merged red pixels and merged blue pixels; and Performing inverse white balance on the merged image to output a final merged image.

Images