Electronic device, method for generating image data, and non-transitory computer-readable medium

Abstract

An electronic device includes a camera assembly, and the camera assembly includes an image sensor configured to capture an image of an object and generate color image data. Among them, the image sensor has an array arranged in a Bayer format at each pixel position for generating a green unit block, a blue unit block, and a red unit block of color image data. The green unit block, the blue unit block, and the red unit block each include four physical pixel units, and among them, the green unit block includes four green physical pixel units, the blue unit block includes two blue physical pixel units and two green physical pixel units, and the red unit block includes two red physical pixel units and two green physical pixel units. The electronic device further includes a main processor that performs image processing.

Description

Technical Field

[0001] The present disclosure relates to a method for generating image data, an electronic device for implementing the method, and a non-transitory computer-readable medium including program instructions stored thereon for executing the method. Background Art

[0002] Electronic devices such as smartphones and tablet terminals are widely used in our daily lives. Nowadays, many electronic devices are equipped with camera components to capture images. Some electronic devices are portable and thus easy to carry. Therefore, by using the camera components of electronic devices, users of electronic devices can easily capture images of objects anytime and anywhere.

[0003] There are many formats for capturing images of objects and thereby generating target image data. One of the well-known formats is the Bayer format including sparse image data.

[0004] Here, Figure 9 is a diagram showing an example of the prior art of performing Bayer Binning on four pixels. Figure 10 is a diagram showing an example of performing Bayer Binning on four pixels to generate merged image data in the prior art shown in Figure 9 .

[0005] For example, as shown in Figure 9 , due to resource limitations, sensor analog binning of the pixel array is performed in the same manner for all pixels (pixel positions) in the sensor. In the pixel array, the noise quality of using 4-pixel binning is better ( Figure 10 ). However, no G (green) information is obtained at the R (red) and B (blue) pixel positions.

[0006] On the other hand, Figure 11 is a diagram showing another example of the prior art of performing merging on two diagonally arranged pixels. Figure 12 is a diagram showing an example of merging two diagonally arranged pixels to generate merged data in the prior art shown in Figure 11 .

[0007] As shown in Figure 11As shown, in a color filter sensor with a CFA (Color Filter Array) where G (green) is scattered in R-G blocks and B-G blocks, only the analog merging of 2 out of 4 pixels is performed in a merging block with 4 pixels to avoid color mixing between the R-G block at the pixel position and the B-G block at the pixel position. In CFA merging, these G pixels improve the resolution on the G channel. Therefore, the resolution of CFA merging is better than that of the merged Bayer CFA. However, in CFA merging, as Figure 12 shown, due to the number of accumulated electrons, the noise quality of merging 2 pixels is worse than that of merging 4 pixels.

[0008] Next, Figure 13 is a diagram showing an example of the prior art for converting GYCG image data obtained by using a GYCG sensor into RGB image data.

[0009] As Figure 13 shown, there are sensors with different color arrays similar to GYCG (Y = yellow, G = green, C = cyan), and these sensors obtain YGC images. The standard image format requires RGB (R = red, G = green, B = blue) images. Therefore, it is necessary to convert the color images obtained by the sensors using a color correction matrix, and the color correction matrix is as follows:

[0010]

[0011] However, the exact intensity of each color depends on the spectrum and object reflection. For example, the cyan intensity is not equal to the sum of the green intensity and the blue intensity. Therefore, it cannot clearly separate the complementary color intensity into two primary color components.

[0012] Therefore, the present invention proposes to perform 4-pixel merging in a new color filter array to reduce the effect of color mixing and improve the noise quality. Summary of the Invention

[0013] The present disclosure aims to solve at least one of the above technical problems. Therefore, the present disclosure needs to provide a method for generating target image data and an electronic device for implementing the method.

[0014] According to the present disclosure, the electronic device may include:

[0015] A camera component includes an image sensor configured to capture an image of an object and generate color image data. The image sensor has a green unit block, a blue unit block, and a red unit block arranged in a Bayer pattern at each pixel position for generating color image data. The green unit block, the blue unit block, and the red unit block each include four physical pixel units. The green unit block includes four green physical pixel units, the blue unit block includes two blue physical pixel units and two green physical pixel units, and the red unit block includes two red physical pixel units and two green physical pixel units. And

[0016] A main processor that performs image processing,

[0017] wherein the main processor obtains the combined image data of the unit blocks, and the combined image data is generated by the camera component by combining the charges of the four physical pixel units of the unit blocks;

[0018] The main processor calculates the estimated combined green image data corresponding to the two green physical pixel units based on the combined image data of the green unit block;

[0019] The main processor calculates the estimated combined blue image data by subtracting the estimated combined green image data from the combined image data of the blue unit block; and

[0020] The main processor calculates the estimated combined red image data by subtracting the estimated combined green image data from the combined image data of the red unit block.

[0021] In some embodiments, the main processor obtains the combined image data of the green unit block, the estimated combined blue image data, and the estimated combined red image data as image data conforming to the Bayer pattern.

[0022] In some embodiments, wherein,

[0023] The green unit block, the blue unit block, and the red unit block have a rectangular shape;

[0024] In the green unit block, the four green physical pixel units are respectively arranged corresponding to the four corners,

[0025] In the blue unit block, the two blue physical pixel units are located on the first diagonal line, the two green physical pixel units are located on the second diagonal line, and each of the two blue physical pixel units and the two green physical pixel units corresponds to one of the four corners; and

[0026] In the red unit block, two red physical pixel units are located on the first diagonal line, two green physical pixel units are located on the second diagonal line, and each of the two red physical pixel units and the two green physical pixel units corresponds to one of the four corners respectively.

[0027] In some embodiments,

[0028] The main processor calculates the estimated blue merged image data by subtracting the estimated green merged image data from the merged image data of the blue unit block; and

[0029] The main processor calculates the estimated red merged image data by subtracting the estimated green merged image data from the merged image data of the red unit block.

[0030] In some embodiments,

[0031] The main processor calculates the estimated green merged image data corresponding to the two green physical pixel units based on the average value calculated from the merged image data of multiple green unit blocks.

[0032] In some embodiments,

[0033] The main processor calculates the estimated green merged image data based on the average value of the merged image data of two or four green unit blocks adjacent to the blue unit block, and this estimated green merged image data is used to calculate the estimated blue merged image data; and

[0034] The main processor calculates the estimated green merged image data based on the average value of the merged image data of two or four green unit blocks adjacent to the red unit block, and this estimated green merged image data is used to calculate the estimated red merged image data.

[0035] In some embodiments,

[0036] The main processor calculates the total average value of the merged image data of four green unit blocks adjacent to the blue unit block in the first direction and in the second direction orthogonal to the first direction;

[0037] The main processor calculates the first average value of the merged image data of two green unit blocks adjacent to the blue unit block in the first direction;

[0038] The main processor calculates the second average value of the merged image data of two green unit blocks adjacent to the blue unit block in the second direction;

[0039] The main processor calculates the first absolute value of the difference between the merged image data of two green unit blocks adjacent to the blue unit block in the first direction;

[0040] The main processor calculates the second absolute value of the difference between the combined image data of two green unit blocks adjacent to the blue unit block in the second direction;

[0041] The main processor calculates a reference value based on the ratio of the first absolute value and the second absolute value, and

[0042] wherein,

[0043] when the reference value is less than a preset threshold, the main processor sets half of the total average value as the estimated green combined image data;

[0044] when the reference value is equal to or greater than the threshold and the first absolute value is less than the second absolute value, the main processor sets half of the first average value as the estimated green combined image data; and

[0045] when the reference value is equal to or greater than the threshold and the first absolute value is equal to or greater than the second absolute value, the main processor sets half of the second average value as the estimated green combined image data.

[0046] In some embodiments, wherein,

[0047] The main processor calculates the total average value of the combined image data of four green unit blocks adjacent to the red unit block in the first direction and in the second direction orthogonal to the first direction;

[0048] The main processor calculates the first average value of the combined image data of two green unit blocks adjacent to the red unit block in the first direction;

[0049] The main processor calculates the second average value of the combined image data of two green unit blocks adjacent to the red unit block in the second direction;

[0050] The main processor calculates the first absolute value of the difference between the combined image data of two green unit blocks adjacent to the red unit block in the first direction;

[0051] The main processor calculates the second absolute value of the difference between the combined image data of two green unit blocks adjacent to the red unit block in the second direction;

[0052] The main processor calculates a reference value based on the ratio of the first absolute value and the second absolute value, and

[0053] wherein,

[0054] when the reference value is less than a preset threshold, the main processor sets half of the total average value as the estimated green combined image data;

[0055] when the reference value is equal to or greater than the threshold and the first absolute value is less than the second absolute value, the main processor sets half of the first average value as the estimated green combined image data; and

[0056] When the reference value is equal to or greater than the threshold value and the first absolute value is equal to or greater than the second absolute value, the main processor sets half of the second average value as the estimated green merged image data.

[0057] According to the present disclosure, a method for generating image data includes:

[0058] Obtaining merged image data of a unit block, the merged image data being generated by combining charges of four physical pixel units of the unit block, wherein the unit block includes a green unit block, a blue unit block, and a red unit block arranged in a Bayer format at each pixel position for generating color image data, the green unit block, the blue unit block, and the red unit block each include four physical pixel units, and wherein the green unit block includes four green physical pixel units, the blue unit block includes two blue physical pixel units and two green physical pixel units, and the red unit block includes two red physical pixel units and two green physical pixel units;

[0059] Calculating estimated green merged image data corresponding to two green physical pixel units based on the merged image data of the green unit block;

[0060] Calculating estimated blue merged image data by subtracting the estimated green merged image data from the merged image data of the blue unit block; and

[0061] Calculating estimated red merged image data by subtracting the estimated green merged image data from the merged image data of the red unit block.

[0062] According to the present disclosure, a non - transitory computer - readable medium includes program instructions stored thereon for performing at least the following operations:

[0063] Obtaining merged image data of a unit block, the merged image data being generated by combining charges of four physical pixel units of the unit block, wherein the unit block includes a green unit block, a blue unit block, and a red unit block arranged in a Bayer format at each pixel position for generating color image data, the green unit block, the blue unit block, and the red unit block each include four physical pixel units, and wherein the green unit block includes four green physical pixel units, the blue unit block includes two blue physical pixel units and two green physical pixel units, and the red unit block includes two red physical pixel units and two green physical pixel units;

[0064] Calculating estimated green merged image data corresponding to two green physical pixel units based on the merged image data of the green unit block;

[0065] The estimated blue merged image data is calculated by subtracting the estimated green merged image data from the merged image data of the blue unit blocks; and

[0066] The estimated red merged image data is calculated by subtracting the estimated green merged image data from the merged image data of the red unit blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] These and / or other aspects and advantages of the embodiments of the present disclosure will become apparent and be more readily understood from the following description with reference to the accompanying drawings, in which:

[0068] Figure 1 A plan view of a first side of an electronic device according to an embodiment of the present disclosure is shown;

[0069] Figure 2 A plan view of a second side of an electronic device according to an embodiment of the present disclosure is shown;

[0070] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0071] Figure 4 is a diagram illustrating an example of a configuration of an image sensor of a camera module that performs Bayer merging on four pixels according to an embodiment of the present disclosure;

[0072] Figure 5 is a diagram illustrating an example of performing Bayer merging on four pixels to generate merged image data according to an embodiment of the present disclosure;

[0073] Figure 6 is a diagram illustrating a specific example of a process for generating estimated blue merged image data as Figure 5 shown according to an embodiment of the present disclosure;

[0074] Figure 7 is a diagram illustrating a specific example of a process for generating estimated red merged image data according to the present disclosure;

[0075] Figure 8 is a diagram for explaining the effects according to an embodiment of the present disclosure;

[0076] Figure 9 is a diagram illustrating an example of the prior art of performing Bayer merging on four pixels;

[0077] Figure 10 is a diagram illustrating Figure 9 an example of performing Bayer merging on four pixels to generate merged image data in the prior art as

[0078] Figure 11A diagram showing another example of the prior art in which merging is performed on two diagonally arranged pixels;

[0079] Figure 12 is a diagram showing an example in the prior art shown in Figure 11 where two diagonally arranged pixels are merged to generate merged data; and

[0080] Figure 13 is a diagram showing an example of the prior art for converting GYCG image data obtained by using a GYCG sensor into RGB image data. DETAILED DESCRIPTION

[0081] Embodiments of the present disclosure will be described in detail, and examples of the embodiments will be shown in the drawings. Throughout the description, the same or similar elements and elements having the same or similar functions will be denoted by the same reference numerals. Here, the embodiments described with reference to the drawings are explanatory and are intended to illustrate the present disclosure, but should not be construed as limiting the present disclosure.

[0082] Figure 1 A plan view of a first side of an electronic device 10 according to an embodiment of the present disclosure is shown; Figure 2 A plan view of a second side of the electronic device 10 according to an embodiment of the present disclosure is shown. The first side may be referred to as the rear side of the electronic device 10, and the second side may be referred to as the front side of the electronic device 10.

[0083] As Figure 1 and Figure 2 shown, the electronic device 10 may include a display 20 and a camera assembly 30. In the present embodiment, the camera assembly 30 includes a first main camera 32, a second main camera 34, and a sub-camera 36. The first main camera 32 and the second main camera 34 may capture images in the first side of the electronic device 10, and the sub-camera 36 may capture images in the second side of the electronic device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called outer cameras, and the sub-camera 36 is so-called inner camera. As an example, the electronic device 10 may be a mobile phone, a tablet computer, a personal digital assistant, etc.

[0084] Although the electronic device 10 according to the present embodiment has three cameras, the electronic device 10 may have fewer than three cameras or more than three cameras. For example, the electronic device 10 may have two cameras, four cameras, five cameras, and so on.

[0085] Figure 3 A block diagram of the electronic device 10 according to the present embodiment is shown. As Figure 3As shown, in addition to the display 20 and the camera assembly 30, the electronic device 10 may include a main processor 40, an image signal processor 42, a memory 44, a power circuit 46, and a communication circuit 48. The display 20, the camera assembly 30, the main processor 40, the image signal processor 42, the memory 44, the power circuit 46, and the communication circuit 48 are connected to each other via a bus 50.

[0086] The main processor 40 executes one or more programs stored in the memory 44. The main processor 40 implements various applications and data processing of the electronic device 10 by executing the programs. The main processor 40 may be one or more computer processors. The main processor 40 is not limited to one CPU core, but may have multiple CPU cores. The main processor 40 may be the main CPU of the electronic device 10, an image processing unit (IPU), or a DSP equipped in the camera assembly 30.

[0087] The image signal processor 42 controls the camera assembly 30 and processes various image data captured by the camera assembly 30 to generate target image data. For example, the image signal processor 42 may perform demosaicing processing, noise reduction processing, automatic exposure processing, automatic focusing processing, automatic white balance processing, high dynamic range processing, etc. on the image data captured by the camera assembly 30.

[0088] In this embodiment, the main processor 40 and the image signal processor 42 cooperate with each other to generate target image data of an object captured by the camera assembly 30. That is, both the main processor 40 and the image signal processor 42 are configured to capture an image of an object through the camera assembly 30 and perform various image processing on the captured image data.

[0089] The memory 44 stores programs executed by the main processor 40 and various data. For example, the data of the captured image is stored in the memory 44.

[0090] The memory 44 may include a high-speed RAM memory and / or a non-volatile memory such as a flash memory and a disk memory. That is, the memory 44 may include a non-transitory computer-readable medium storing programs.

[0091] The power circuit 46 may have a battery, such as a lithium-ion rechargeable battery, and a battery management unit (BMU) for managing the battery.

[0092] The communication circuit 48 is configured to receive and transmit data to communicate with a base station of a telecommunication network system, the Internet, or other devices via wireless communication. The wireless communication may employ any communication standard or protocol, including but not limited to GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), LTE (Long Term Evolution), LTE-Advanced, and fifth-generation communication technology (5G). The communication circuit 48 may include an antenna and RF (Radio Frequency) circuitry.

[0093] Here, Figure 4 is a diagram showing an example of the configuration of an image sensor of a camera component that performs Bayer merging of four pixels according to an embodiment of the present disclosure.

[0094] As Figure 4 shown, the camera component 30 includes an image sensor that captures an image of an object and generates color image data.

[0095] Then, for example, as Figure 4 shown, the image sensor has a green unit block GK, a blue unit block BK, and a red unit block RK arranged in a Bayer format at each pixel position to generate color image data. The green unit block GK, the blue unit block BK, and the red unit block RK each contain four physical pixel units. The green unit block GK, the blue unit block BK, and the red unit block RK have a rectangular shape. That is, the pixel array of the present embodiment employs a 2X2 merging technique.

[0096] As Figure 4 shown, the green unit block GK includes four green physical pixel units G. In the green unit block GK, the four green physical pixel units G are arranged corresponding to the four corners.

[0097] For example, the green signal value (merged image data) 4GBIN is generated by combining the four charges in the four green physical pixel units G.

[0098] In particular, in the present embodiment, as Figure 4 shown, the blue unit block BK includes two blue physical pixel units B and two green physical pixel units G. In the blue unit block BK, the two blue physical pixel units B are located on the first diagonal line, and the two green physical pixel units G are located on the second diagonal line. Each of the two blue physical pixel units B and the two green physical pixel units G corresponds to one of the four corners.

[0099] For example, the cyan signal value (merged image data) 4CBIN(BG) is generated by combining the four charges in the two blue physical pixel units B and the two green physical pixel units G.

[0100] Specifically, in the present embodiment, as Figure 4 shown, the red unit block RK includes two red physical pixel units R and two green physical pixel units G. In the red unit block RK, the two red physical pixel units R are located on the first diagonal line, the two green physical pixel units G are located on the second diagonal line, and each of the two red physical pixel units and the two green physical pixel units G corresponds to one of the four corners respectively.

[0101] For example, a yellow signal value (merged image data) 4YBIN(RG) is generated by combining four charges in the two red physical pixel units R and the two green physical pixel units G.

[0102] Next, an example of an operation including image processing will be described, in which the electronic device 10 having the above configuration acquires image data conforming to the Bayer format. Figure 5 is a diagram showing an example of performing Bayer merging on four pixels to generate merged image data according to an embodiment of the present disclosure. Although Figure 5 an example of calculating the estimated blue merged image data EB is shown, the case of calculating the estimated red merged image data ER is also shown in the same way.

[0103] In the present embodiment, for example, the main processor 40 performs image generation processing to generate image data.

[0104] In addition, in the present embodiment, program instructions for image generation processing are stored in a non-transitory computer-readable medium of the memory 44. When the program instructions are read out from the memory 44 and executed in the main processor 40, the main processor 40 implements the image generation processing.

[0105] First, the main processor 40 acquires the merged image data 4GBIN, 4CBIN, and 4YBIN of the unit blocks GK, BK, and RK. The merged image data 4GBIN, 4CBIN, and YBIN are generated by the camera assembly 30 by combining the charges of the four physical pixel units of the unit blocks GK, BK, and RK.

[0106] Then, the main processor 40 calculates the estimated green merged image data G corresponding to the two green physical pixel units G based on the merged image data 4GBIN of the green unit block GK 2pix .

[0107] Next, the main processor 40 calculates the estimated blue merged image data EB by subtracting the estimated green merged image data G from the merged image data 4CBIN(BG) of the blue unit block BK 2pix .

[0108] Similarly, the main processor 40 calculates the estimated red merged image data ER by subtracting the estimated green merged image data G from the merged image data 4YBIN(RG) of the red unit block RK. 2pix , to calculate the estimated red merged image data ER.

[0109] Here, a specific example of the process for generating the estimated blue merged image data as shown in Figure 5 the embodiments of the present invention will be described. Figure 6 FIG. Figure 5 is a diagram showing a specific example of the process for generating the estimated blue merged image data as shown in

[0110] For example, in Figure 6 step S1, as shown in the following formula, the main processor 40 calculates the merged image data G of the four green unit blocks GK adjacent to the blue unit block BK in the first direction and in the second direction orthogonal to the first direction a , G b , G c , and G d (4GBIN) to obtain the total average value G ave4 .

[0111] G ave4 = (G a + G b + G c + G d ) / 4

[0112] In addition, as shown in the following formula, the main processor 40 calculates the first average value G a of the merged image data G of the two green unit blocks GK adjacent to the blue unit block BK in the first direction c and G aveV .

[0113] G aveV = (G a + G c ) / 2

[0114] In addition, as shown in the following formula, the main processor 40 calculates the second average value G b of the merged image data G of the two green unit blocks GK adjacent to the blue unit block BK in the second direction d and G aveH .

[0115] G aveH = (G b + G d ) / 2

[0116] In addition, as shown in the following formula, the main processor 40 calculates the combined image data G of two green unit blocks GK adjacent to the blue unit block BK in the first direction a and G c The first absolute value G of the difference between difV .

[0117] G difV = |G a - G c |

[0118] In addition, as shown in the following formula, the main processor 40 calculates the second absolute value G of the difference between the combined image data G of two green unit blocks GK adjacent to the blue unit block BK in the second direction b and G d The second absolute value G of the difference between difH .

[0119] G difH = |G b - G d |

[0120] In addition, as shown in the following formula, the main processor 40 calculates the reference value G based on the ratio of the first absolute value G difV and the second absolute value G difH . dir .

[0121] G dir = |1 - G difV / G difH |

[0122] Here, as shown in the following formula, when the reference value G dir is less than the preset threshold T dir , the main processor 40 sets the total average value G ave4 as the interpolated green value G 4pix .

[0123] On the other hand, when the reference value G dir is equal to or greater than the threshold T dir and the first absolute value G difV is less than the second absolute value G difH , the main processor 40 sets the first average value G aveV as the interpolated green value G 4pix .

[0124] On the other hand, when the reference value G dir is equal to or greater than the threshold T dir and the first absolute value G difV is equal to or greater than the second absolute value G difH , the main processor 40 sets the second average value G aveHSet to interpolated green value G 4pix .

[0125]

[0126] Next, in Figure 6 step S2, as shown in the following formula, the main processor 40 uses half of the interpolated green value G 4pix as the estimated green merged image data G 2pix . That is, based on the merged G (G 4pix ) of 4 pixels, 2 pixels of merged G (G 2pix ) are estimated.

[0127] G 2pix = G 4pix / 2

[0128] In this way, the main processor 40 calculates the estimated green merged image data G 2pix corresponding to two green physical pixel units G based on the average value calculated from the merged image data of multiple green unit blocks GK.

[0129] Specifically, the main processor 40 calculates the estimated green merged image data G 4pix , G 2pix for calculating the estimated blue merged image data EB based on the average value G 2pix of the merged image data of two or four green unit blocks GK adjacent to the blue unit block BK.

[0130] Next, in Figure 6 step S3, as shown in the following formula, the main processor 40 calculates the estimated blue merged image data EB by subtracting the estimated green merged image data G 2pix adjacent to the blue unit block BK from the merged image data 4CBIN(BG) of the blue unit block BK.

[0131] EB = BG - G 2pix

[0132] On the other hand, Figure 7 is a diagram showing a specific example of the process for generating the estimated red merged image data of the present disclosure. Figure 7 The process for generating the estimated red merged image data shown is the same as the process shown above Figure 6 .

[0133] For example, in Figure 7 step S11, as shown in the following formula, the main processor 40 calculates the merged image data G a of four green unit blocks GK adjacent to the red unit block RK in the first direction and in the second direction orthogonal to the first direction, G b , G c and G d (4GBIN), the total average value G ave4 .

[0134] G ave4 = (G a + G b + G c + G d ) / 4

[0135] In addition, as shown in the following formula, the main processor 40 calculates the first average value G a and G c of the combined image data of two green unit blocks GK adjacent to the red unit block RK in the first direction aveV .

[0136] G aveV = (G a + G c ) / 2

[0137] In addition, as shown in the following formula, the main processor 40 calculates the second average value G b and G d of the combined image data of two green unit blocks GK adjacent to the red unit block RK in the second direction aveH .

[0138] G aveH = (G b + G d ) / 2

[0139] In addition, as shown in the following formula, the main processor 40 calculates the first absolute value G a and G c of the difference between the combined image data of two green unit blocks GK adjacent to the red unit block RK in the first direction difV .

[0140] G difV = |G a - G c |

[0141] In addition, as shown in the following formula, the main processor 40 calculates the second absolute value G b and G d of the difference between the combined image data of two green unit blocks GK adjacent to the red unit block RK in the second direction difH .

[0142] G difH = |G b - G d |

[0143] In addition, as shown in the following formula, the main processor 40 calculates a reference value G based on the ratio of the first absolute value G difV and the second absolute value G difH . dir .

[0144] G dir = |1 - G difV / G difH |

[0145] Here, as shown in the following formula, when the reference value G dir is less than a preset threshold value T dir , the main processor 40 sets the total average value G ave4 to the interpolated green value G 4pix .

[0146] On the other hand, when the reference value G dir is equal to or greater than the threshold value T dir and the first absolute value G difV is less than the second absolute value G difH , the main processor 40 sets the first average value G aveV to the interpolated green value G 4pix .

[0147] On the other hand, when the reference value G dir is equal to or greater than the threshold value T dir and the first absolute value G difV is equal to or greater than the second absolute value G difH , the main processor 40 sets the second average value G aveH to the interpolated green value G 4pix .

[0148]

[0149] Next, in step S12 of Figure 7 , as shown in the following formula, the main processor 40 uses half of the interpolated green value G 4pix as the estimated green merged image data G 2pix . That is, the merged G (G 4pix ) of 4 pixels is used to estimate the merged G (G 2pix ) of 2 pixels.

[0150] G 2pix = G 4pix / 2

[0151] In this way, the main processor 40 calculates the estimated green merged image data G corresponding to two green physical pixel units G based on the average value calculated from the merged image data of multiple green unit blocks GK2pix 。

[0152] Specifically, the main processor 40 calculates the estimated green merged image data G based on the average value G of the merged image data of two or four green unit blocks GK adjacent to the red unit block RK. 4pix to calculate the estimated green merged image data G 2pix , G 2pix for calculating the estimated red merged image data ER.

[0153] Next, in Figure 7 step S13, as shown in the following formula, the main processor 40 calculates the estimated red merged image data ER by subtracting the estimated green merged image data G adjacent to the red unit block RK from the merged image data 4YBIN(RG) of the red unit block RK. 2pix to calculate the estimated red merged image data ER.

[0154] ER = RG - G 2pix

[0155] According to the above process, the main processor 40 of the electronic device 10 obtains the merged image data 4GBIN of the green unit block GK, the estimated blue merged image data EB, and the estimated red merged image data ER as image data conforming to the Bayer format.

[0156] Then, the main processor 40 outputs the obtained image data conforming to the Bayer format to, for example, the image signal processor 42. Then, the image signal processor 42 performs predetermined image processing on the input image data. Here, Figure 8 is a diagram for explaining the effects of the embodiments according to the present disclosure.

[0157] As described above, the electronic device 10 according to the present embodiment includes a camera assembly 30. The camera assembly includes an image sensor configured to capture an image of an object and generate color image data. The image sensor has a green unit block, a blue unit block, and a red unit block arranged in a Bayer format at each pixel position for generating color image data. The green unit block, the blue unit block, and the red unit block each include four physical pixel units. The green unit block includes four green physical pixel units, the blue unit block includes two blue physical pixel units and two green physical pixel units, and the red unit block includes two red physical pixel units and two green physical pixel units. The electronic device 10 further includes a main processor 40 that performs image processing. The main processor 40 obtains the combined image data of the unit blocks, and the combined image data is generated by the camera assembly 30 by combining the charges of the four physical pixel units of the unit blocks. The main processor 40 calculates the estimated green combined image data corresponding to the two green physical pixel units based on the combined image data of the green unit block. The main processor 40 calculates the estimated blue combined image data by subtracting the estimated green combined image data from the combined image data of the blue unit block, and the main processor 40 calculates the estimated red combined image data by subtracting the estimated green combined image data from the combined image data of the red unit block.

[0158] Therefore, as Figure 8 shown, the present invention proposes to perform 4-pixel binning in a new color filter array to reduce the color mixing effect and improve the noise quality of 2-pixel binning.

[0159] In the description of the embodiments of the present disclosure, it should be understood that terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", and "counterclockwise" should be interpreted as referring to the directions or positions described or shown in the accompanying drawings being discussed. These relative terms are only used to simplify the description of the present disclosure and do not indicate or imply that the devices or elements mentioned must have a specific orientation, or be constructed or operated in a specific orientation. Therefore, these terms should not be considered as limiting the present disclosure.

[0160] In addition, terms such as "first" and "second" are used herein for descriptive purposes and are not intended to indicate or imply relative importance or significance, or imply the number of the indicated technical features. Therefore, the features defined by "first" and "second" may include one or more of such features. In the description of the present disclosure, "a plurality" means two or more unless otherwise specified.

[0161] In the description of the embodiments of the present disclosure, unless otherwise specified or restricted, terms such as "installed", "connected", "coupled", etc. are widely used and can be, for example, fixedly connected, detachably connected or integrally connected; they can also be mechanical or electrical connections; they can also be direct connections or indirect connections through intermediate structures; they can also be internal communications between two elements, which can be understood by those skilled in the art according to specific situations.

[0162] In the embodiments of the present disclosure, unless otherwise specified or restricted, a structure in which a first feature is "above" or "below" a second feature may include an embodiment in which the first feature is in direct contact with the second feature, and may also include an embodiment in which the first feature and the second feature are not in direct contact with each other, but are in contact through additional features formed therebetween. In addition, the first feature being "above", "on" or "at the top of" the second feature may include: an embodiment in which the first feature is vertically or obliquely "above", "on" or "at the top of" the second feature, or simply means that the first feature is at a height higher than that of the second feature; while the first feature being "below", "under" or "at the bottom of" the second feature may include: an embodiment in which the first feature is vertically or obliquely "below", "under" or "at the bottom of" the second feature, or simply means that the first feature is at a height lower than that of the second feature.

[0163] In the above description, various embodiments and examples are provided to implement different structures of the present disclosure. To simplify the present disclosure, specific units and settings are described above. However, these units and settings are only examples and are not intended to limit the present disclosure. In addition, in different examples of the present disclosure, the drawing numbers and / or reference letters may be repeated. This repetition is for simplicity and clarity and does not refer to the relationship between different embodiments and / or settings. In addition, examples of different processes and materials are provided in the present disclosure. However, those skilled in the art will understand that other processes and / or materials may also be applied.

[0164] Throughout the specification, references to "embodiments", "some embodiments", "exemplary embodiments", "exemplary examples", "specific examples" or "some examples" mean that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present disclosure. Thus, the appearance of the above phrases throughout the specification is not necessarily referring to the same embodiments or examples of the present disclosure. In addition, in one or more embodiments or examples, the specific features, structures, materials or characteristics may be combined in any suitable manner.

[0165] Any process or method described in a flowchart or otherwise described herein can be understood to include one or more modules, segments, or portions of code of executable instructions for implementing specific logical functions or steps in the process, and the scope of the preferred embodiments of the present disclosure includes other implementations, where those skilled in the art should understand that the functions can be implemented in a sequence different from that shown or discussed, including in substantially the same sequence or the reverse sequence.

[0166] The logic and / or steps described otherwise herein or shown in a flowchart, for example, a specific sequence list of executable instructions for implementing a logical function, can be embodied in any computer-readable medium that will be used by an instruction execution system, apparatus, or device (e.g., a computer-based system that includes a processor or other system capable of obtaining instructions from and executing instructions of the instruction execution system, apparatus, and device), or will be used in conjunction with the instruction execution system, apparatus, and device. For the purposes of this specification, a "computer-readable medium" can be any device suitable for including, storing, communicating, propagating, or transporting a program that will be used by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of computer-readable media include, but are not limited to: an electronic connection (electronic device) having one or more wires, a portable computer case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which a program can be printed, because, for example, when the program needs to be obtained electronically, the paper or other suitable media can be optically scanned and then edited, decrypted, or processed by other suitable methods, and then the program can be stored in a computer memory.

[0167] It should be understood that each part of the present disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by an appropriate instruction execution system. For example, if implemented by hardware, in another embodiment as well, the steps or methods can be implemented by one or a combination of the following techniques known in the art: discrete logic circuits having logic gate circuits for implementing the logical functions of data signals, application-specific integrated circuits having appropriate combinations of logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), and the like.

[0168] Those skilled in the art should understand that all or part of the steps in the above exemplary methods of the present disclosure can be implemented by using program commands to relevant hardware. These programs can be stored in a computer-readable storage medium, and when running on a computer, these programs include one or a combination of the steps in the method embodiments of the present disclosure.

[0169] In addition, each functional unit of the embodiments of the present disclosure can be integrated into a processing module, or these units can exist separately physically, or two or more units can be integrated into a processing module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, the integrated module can be stored in a computer-readable storage medium.

[0170] The above storage medium can be a read-only memory, a disk, a CD, etc.

[0171] Although the embodiments of the present disclosure have been shown and described, those skilled in the art will understand that these embodiments are illustrative and should not be construed as limiting the present disclosure, and changes, modifications, substitutions, and variations can be made to the embodiments without departing from the scope of the present disclosure.

Claims

1. An electronic device, comprising: a camera component including an image sensor configured to capture an image of an object and generate color image data, wherein the image sensor has an array arranged in a Bayer format at each pixel position for generating a green unit block, a blue unit block, and a red unit block of the color image data, the green unit block, the blue unit block, and the red unit block each include four physical pixel units, and wherein the green unit block includes four green physical pixel units, the blue unit block includes two blue physical pixel units and two green physical pixel units, and the red unit block includes two red physical pixel units and two green physical pixel units; and a main processor that performs image processing, wherein the main processor obtains combined image data of the unit blocks, the combined image data being generated by the camera component by combining charges of the four physical pixel units of the unit blocks; the main processor calculates estimated green combined image data corresponding to the two green physical pixel units based on the combined image data of the green unit block; the main processor calculates estimated blue combined image data by subtracting the estimated green combined image data from the combined image data of the blue unit block; and the main processor calculates estimated red combined image data by subtracting the estimated green combined image data from the combined image data of the red unit block, wherein the green unit block, the blue unit block, and the red unit block have a rectangular shape; in the green unit block, the four green physical pixel units are respectively arranged corresponding to the four corners; in the blue unit block, the two blue physical pixel units are located on a first diagonal line, the two green physical pixel units are located on a second diagonal line, and each of the two blue physical pixel units and the two green physical pixel units respectively corresponds to one of the four corners; and in the red unit block, the two red physical pixel units are located on a first diagonal line, the two green physical pixel units are located on a second diagonal line, and each of the two red physical pixel units and the two green physical pixel units respectively corresponds to one of the four corners.

2. The electronic device according to claim 1, wherein, the main processor obtains the combined image data of the green unit block, the estimated blue combined image data, and the estimated red combined image data as image data conforming to the Bayer format.

3. The electronic device according to claim 1, wherein, the main processor calculates the estimated green combined image data corresponding to the two green physical pixel units based on an average value calculated from the combined image data of a plurality of green unit blocks.

4. The electronic device according to claim 3, wherein, The main processor calculates the estimated green merged image data based on the average value of the merged image data of two or four of the green unit blocks adjacent to the blue unit block, and the estimated green merged image data is used to calculate the estimated blue merged image data; And The main processor calculates the estimated green merged image data based on the average value of the merged image data of two or four of the green unit blocks adjacent to the red unit block, and the estimated green merged image data is used to calculate the estimated red merged image data.

5. The electronic device according to claim 4, wherein The main processor calculates the total average value of the merged image data of the four green unit blocks adjacent to the blue unit block in a first direction and in a second direction orthogonal to the first direction; The main processor calculates a first average value of the merged image data of the two green unit blocks adjacent to the blue unit block in the first direction; The main processor calculates a second average value of the merged image data of the two green unit blocks adjacent to the blue unit block in the second direction; The main processor calculates a first absolute value of the difference between the merged image data of the two green unit blocks adjacent to the blue unit block in the first direction; The main processor calculates a second absolute value of the difference between the merged image data of the two green unit blocks adjacent to the blue unit block in the second direction; The main processor calculates a reference value based on the ratio of the first absolute value and the second absolute value, and wherein When the reference value is less than a preset threshold, the main processor sets half of the total average value as the estimated green merged image data; When the reference value is equal to or greater than the threshold and the first absolute value is less than the second absolute value, the main processor sets half of the first average value as the estimated green merged image data; and When the reference value is equal to or greater than the threshold and the first absolute value is equal to or greater than the second absolute value, the main processor sets half of the second average value as the estimated green merged image data.

6. The electronic device according to claim 4, wherein The main processor calculates the total average value of the merged image data of the four green unit blocks adjacent to the red unit block in a first direction and in a second direction orthogonal to the first direction; The main processor calculates a first average value of the merged image data of the two green unit blocks adjacent to the red unit block in the first direction; The main processor calculates a second average value of the merged image data of the two green unit blocks adjacent to the red unit block in the second direction; The main processor calculates a first absolute value of the difference between the merged image data of the two green unit blocks adjacent to the red unit block in the first direction; The main processor calculates a second absolute value of the difference between the combined image data of the two green unit blocks adjacent to the red unit block in the second direction; The main processor calculates a reference value based on the ratio of the first absolute value and the second absolute value, and wherein, when the reference value is less than a preset threshold, the main processor sets half of the total average value as the estimated green combined image data; when the reference value is equal to or greater than the threshold and the first absolute value is less than the second absolute value, the main processor sets half of the first average value as the estimated green combined image data; and when the reference value is equal to or greater than the threshold and the first absolute value is equal to or greater than the second absolute value, the main processor sets half of the second average value as the estimated green combined image data.

7. A method for generating image data, comprising: obtaining combined image data of unit blocks, the combined image data being generated by combining charges of four physical pixel units of the unit blocks, wherein the unit blocks include green unit blocks, blue unit blocks, and red unit blocks arranged in a Bayer pattern at each pixel position for generating color image data, the green unit blocks, the blue unit blocks, and the red unit blocks each include four physical pixel units, and wherein the green unit blocks include four green physical pixel units, the blue unit blocks include two blue physical pixel units and two green physical pixel units, and the red unit blocks include two red physical pixel units and two green physical pixel units; calculating estimated green combined image data corresponding to the two green physical pixel units based on the combined image data of the green unit blocks; calculating estimated blue combined image data by subtracting the estimated green combined image data from the combined image data of the blue unit blocks; and calculating estimated red combined image data by subtracting the estimated green combined image data from the combined image data of the red unit blocks, wherein the green unit blocks, the blue unit blocks, and the red unit blocks have a rectangular shape; in the green unit blocks, the four green physical pixel units are respectively arranged corresponding to the four corners, in the blue unit blocks, the two blue physical pixel units are located on the first diagonal line, the two green physical pixel units are located on the second diagonal line, and each of the two blue physical pixel units and the two green physical pixel units corresponds to one of the four corners; and in the red unit blocks, the two red physical pixel units are located on the first diagonal line, the two green physical pixel units are located on the second diagonal line, and each of the two red physical pixel units and the two green physical pixel units corresponds to one of the four corners.

8. A non-transitory computer-readable medium, including program instructions stored on the medium for at least performing the following operations: Obtain the combined image data of the unit block, where the combined image data is generated by combining the charges of four physical pixel units of the unit block. Wherein, The unit block includes a green unit block, a blue unit block, and a red unit block arranged in a Bayer format at each pixel position to generate color image data. The green unit block, the blue unit block, and the red unit block each include four physical pixel units. And wherein, the green unit block includes four green physical pixel units, the blue unit block includes two blue physical pixel units and two green physical pixel units, and the red unit block includes two red physical pixel units and two green physical pixel units; Based on the combined image data of the green unit block, calculate the estimated combined green image data corresponding to the two green physical pixel units; Calculate the estimated combined blue image data by subtracting the estimated combined green image data from the combined image data of the blue unit block; and Calculate the estimated combined red image data by subtracting the estimated combined green image data from the combined image data of the red unit block. Wherein, the green unit block, the blue unit block, and the red unit block have a rectangular shape; In the green unit block, the four green physical pixel units are respectively arranged corresponding to the four corners. In the blue unit block, the two blue physical pixel units are located on the first diagonal line, the two green physical pixel units are located on the second diagonal line, and each of the two blue physical pixel units and the two green physical pixel units respectively corresponds to one of the four corners; and In the red unit block, the two red physical pixel units are located on the first diagonal line, the two green physical pixel units are located on the second diagonal line, and each of the two red physical pixel units and the two green physical pixel units respectively corresponds to one of the four corners.

Images