Electronic device, method of generating image data, and non-transitory computer readable medium

Abstract

An electronic device comprising: a camera assembly comprising an image sensor configured to capture an image of a subject and generate color image data, wherein the image sensor has green element blocks, blue element blocks, and red element blocks arranged in an array in a Bayer format at each pixel location to generate the color image data, the green element blocks, the blue element blocks, and the red element blocks each comprising a plurality of physical pixel elements, the green element blocks comprising two green physical pixel elements and two white physical pixel elements, the blue element blocks comprising two blue physical pixel elements and two white physical pixel elements, and the red element blocks comprising two red physical pixel elements and two white physical pixel elements; and a main processor that performs image processing.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a method of generating image data, an electronic device implementing such a method, and a non-transitory computer readable medium including program instructions stored thereon for performing such a method. BACKGROUND

[0002] Electronic devices such as smartphones and tablet terminals are widely used in our daily life. Nowadays, many electronic devices are equipped with a camera assembly to take an image. Some electronic devices are portable, and thus are easy to carry. Therefore, a user of an electronic device can easily take a picture of a subject anytime and anywhere by using the camera assembly of the electronic device.

[0003] There are many formats to take an image of a subject and generate its target image data. One of the well-known formats is a Bayer format including sparse image data.

[0004] In addition to the sparse image data, in order to improve the quality of the image of the subject based on the target image data, when the camera assembly takes the subject, dense image data is also generated.

[0005] In this document, Figure 8 is a schematic diagram illustrating an example of a conventional technique for acquiring BGB image data and white image data using an RGB camera and a monochrome camera.

[0006] As Figure 8 shown, in the prior art, the output of the special format sensor requires special processing. For high resolution images, an RGB sensor of an RGB camera and a W sensor of a monochrome camera are proposed, the RGB sensor and the W sensor output an RGB (Bayer) image and a white image. Therefore, the system of the prior art requires more data transmission resources and special processing (e.g., including position fitting) of the white image.

[0007] On the other hand, Figure 9 is a schematic diagram illustrating an example of a conventional technique for acquiring BGB image data and white image data using a pixel array conforming to a Bayer format.

[0008] As Figure 9 shown, in the prior art, the image sensor includes a clear filter (w: white) pixel on a typical RGB sparse (Bayer RAW) image sensor. In this case, an RGB Bayer image and a white image are available.

[0009] However, the system of the prior art requires more data transmission resources and special processing of the white image. SUMMARY

[0010] The present disclosure aims to solve at least one of the above technical problems. Therefore, the present disclosure needs to provide a method of generating image data and an electronic device implementing such a method.

[0011] According to the present disclosure, an electronic device can include: a camera assembly including an image sensor configured to capture an image of an object and generate color image data, wherein the image sensor has a green element block, a blue element block, and a red element block arranged in an array in a Bayer format at each pixel position so as to generate color image data, the green element block, the blue element block, and the red element block each including a plurality of physical pixel elements, the green element block including two green physical pixel elements and two white physical pixel elements, the blue element block including two blue physical pixel elements and two white physical pixel elements, and the red element block including two red physical pixel elements and two white physical pixel elements; and a main processor performing image processing, wherein: the camera assembly acquires green merged image data and white merged image data of the green element block, red merged image data and white merged image data of the red element block, and blue merged image data and white merged image data of the blue element block generated by pixel binning processing of the camera assembly, the camera assembly calculates a green-to-white ratio of the green merged image data and the white merged image data of the green element block, the camera assembly acquires white residual data based on a difference between white merged image data of a red element block or a blue element block at a first pixel position at which estimated green residual data should be estimated and white merged image data of a green element block at a second pixel position adjacent to the first position, and the camera assembly estimates estimated green residual data corresponding to the first pixel position based on a green-to-white ratio corresponding to the green element block at the second pixel position and the white residual data.

[0012] In some embodiments, wherein the camera assembly acquires the white residual data by subtracting the white merged image data of the green element block at the second pixel position from the white merged image data of the red element block or the blue element block at the first pixel position.

[0013] In some embodiments, wherein the camera assembly acquires the estimated green residual data by multiplying the green-to-white ratio corresponding to the green element block at the second pixel position by the white residual data.

[0014] In some embodiments, the green element block, the blue element block, and the red element block have a rectangular shape, and wherein: in the green element block, the two green physical pixel elements are located on the first diagonal, and the two white physical pixel elements are located on the second diagonal, thus corresponding to each of the four corners respectively; in the blue element block, the two blue physical pixel elements are located on the first diagonal, and the two white physical pixel elements are located on the second diagonal, thus corresponding to each of the four corners respectively; and in the red element block, the two red physical pixel elements are located on the first diagonal, and the two white physical pixel elements are located on the second diagonal, thus corresponding to each of the four corners respectively.

[0015] In some embodiments, the camera assembly generates the green merged image data and the white merged image data of the green element block, the red merged image data and the white merged image data of the red element block, and the blue merged image data and the white merged image data of the blue element block by using the image sensor for pixel merging processing.

[0016] In some embodiments, the camera component uses the green merged image data of the green element blocks, the red merged image data of the red element blocks, and the blue merged image data of the blue element blocks as sparse image data conforming to the Bayer format.

[0017] In some embodiments, the camera component generates embedded sparse image data from the sparse image data by embedding the estimated green residual data into the sparse image data at the first pixel location.

[0018] In some embodiments, the electronic device further includes an image signal processor that processes the sparse image data in the embedded sparse image data to generate target image data, wherein the camera component inputs the embedded sparse image data to the image signal processor.

[0019] In some embodiments, after the embedded sparse image data has been input to the image signal processor, the main processor obtains the embedded sparse image data from the image signal processor, the main processor extracts the estimated green residual data from the embedded sparse image data obtained by the image signal processor, and the main processor reconstructs dense image data based on the estimated green residual data.

[0020] According to this disclosure, a method for generating image data may include: acquiring green merged image data and white merged image data of green element blocks, red merged image data and white merged image data of red element blocks, and blue merged image data and white merged image data of blue element blocks generated by pixel merging processing of a camera component, wherein the camera component includes an image sensor configured to capture an image of an object and generate color image data, wherein the image sensor has the green element blocks, the blue element blocks, and the red element blocks, the green element blocks, the blue element blocks, and the red element blocks are arranged in a Bayer-format array at each pixel location to generate color image data, the green element blocks, the blue element blocks, and the red element blocks each include a plurality of physical pixel elements, the green element blocks including The blue element block comprises two green physical pixel elements and two white physical pixel elements, and the red element block comprises two red physical pixel elements and two white physical pixel elements; the green-to-white ratio of the green merged image data and the white merged image data of the green element block is calculated; white residual data is obtained based on the difference between the white merged image data of the red element block or blue element block at a first pixel position and the white merged image data of the green element block at a second pixel position, where the green residual data should be estimated at the first pixel position, and the second position is adjacent to the first position; and the estimated green residual data corresponding to the first pixel position is estimated based on the green-to-white ratio and the white residual data corresponding to the green element block at the second pixel position.

[0021] According to this disclosure, a non-transitory computer-readable medium may include program instructions stored on the non-transitory computer-readable medium, the program instructions being configured to at least perform the following operations: acquire green merged image data and white merged image data of green element blocks, red merged image data and white merged image data of red element blocks, and blue merged image data and white merged image data of blue element blocks generated by pixel merging processing of a camera component, wherein the camera component includes an image sensor configured to capture an image of an object and generate color image data, wherein the image sensor has the green element blocks, the blue element blocks, and the red element blocks, the green element blocks, the blue element blocks, and the red element blocks are arranged in a Bayer-formatted array at each pixel location to generate color image data, the green element blocks, the blue element blocks, and the red element blocks respectively The method includes multiple physical pixel elements, wherein the green element block includes two green physical pixel elements and two white physical pixel elements, the blue element block includes two blue physical pixel elements and two white physical pixel elements, and the red element block includes two red physical pixel elements and two white physical pixel elements; the method calculates the green-to-white ratio of the green merged image data and the white merged image data of the green element block; obtains white residual data based on the difference between the white merged image data of the red element block or blue element block at a first pixel position and the white merged image data of the green element block at a second pixel position, where the green residual data should be estimated at the first pixel position, and the second position is adjacent to the first position; and estimates the estimated green residual data corresponding to the first pixel position based on the green-to-white ratio and the white residual data corresponding to the green element block at the second pixel position. Attached Figure Description

[0022] These and / or other aspects and advantages of the embodiments of this disclosure will become apparent and more readily understood from the following description taken with reference to the accompanying drawings, wherein:

[0023] Figure 1 A plan view of a first side of an electronic device according to an embodiment of the present disclosure is shown;

[0024] Figure 2 A plan view of the second side of an electronic device according to an embodiment of the present disclosure is shown;

[0025] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0026] Figure 4 A portion of the pixel array of an image sensor of a camera assembly according to an embodiment of the present disclosure is shown;

[0027] Figure 5A This is a schematic diagram illustrating an example of a process for generating image data conforming to the Bayer format according to embodiments of the present disclosure;

[0028] Figure 5B This illustrates an embodiment according to the present disclosure. Figure 5A A schematic diagram illustrating an example of the process used to generate the target image data;

[0029] Figure 6 It shows the use of in Figure 5A The diagram illustrates an example of a process in image data processing that generates embedded sparse image data for a camera component to be input to an image signal processor.

[0030] Figure 7A It is used for explanation Figure 5A The diagram illustrates the calculation of white residual data from adjacent green and red pixel blocks in a white image.

[0031] Figure 7B This is a schematic diagram illustrating the configuration for estimating green residual data based on white residual data from white image data;

[0032] Figure 8 This is a schematic diagram illustrating examples of conventional techniques for acquiring BGB image data and white image data using RGB cameras and monochrome cameras; and

[0033] Figure 9 This is a schematic diagram illustrating an example of a conventional technique for acquiring BGB image data and white image data using a pixel array conforming to the Bayer format. Detailed Implementation

[0034] Embodiments of this disclosure will be described in detail, and examples of embodiments will be shown in the accompanying drawings. Throughout the description, the same or similar elements and elements having the same or similar functions are denoted by the same reference numerals. The embodiments described herein with reference to the accompanying drawings are illustrative and intended to illustrate this disclosure, but should not be construed as limiting this disclosure.

[0035] Figure 1 A plan view of the first side of an electronic device 10 according to an embodiment of the present disclosure is shown, and Figure 2 A plan view of a second side of an electronic device 10 according to an embodiment of the present disclosure is shown. The first side may refer to the rear side of the electronic device 10, while the second side may refer to the front side of the electronic device 10.

[0036] like Figure 1 and Figure 2As shown, the electronic device 10 may include a display 20 and a camera assembly 30. In this embodiment, the camera assembly 30 includes a first main camera 32, a second main camera 34, and a secondary camera 36. The first main camera 32 and the second main camera 34 can capture images on a first side of the electronic device 10, while the secondary camera 36 can capture images on a second side of the electronic device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called external cameras, while the secondary camera 36 is a so-called built-in camera. As an example, the electronic device 10 may be a mobile phone, a tablet computer, a personal digital assistant, etc.

[0037] Although the electronic device 10 according to this embodiment has three cameras, the electronic device 10 may have fewer than three cameras or more than three cameras. For example, the electronic device 10 may have two cameras, four cameras, five cameras, etc.

[0038] Figure 3 A block diagram of an electronic device 10 according to this embodiment is shown. (As shown) Figure 3 As shown, in addition to the display 20 and camera assembly 30, the electronic device 10 may include a main processor 40, an image signal processor 42, a memory 44, a power supply circuit 46, and a communication circuit 48. The display 20, camera assembly 30, main processor 40, image signal processor 42, memory 44, power supply circuit 46, and communication circuit 48 are interconnected via a bus 50.

[0039] The main processor 40 executes one or more programs stored in the memory 44. The main processor 40 implements various applications and data processing (including image data processing) of the electronic device 10 by running programs. The main processor 40 can be one or more computer processors. The main processor 40 is not limited to a single CPU core, but can have multiple CPU cores. The main processor 40 can be the main CPU of the electronic device 10, an image processing unit (IPU), or a DSP with a camera assembly 30.

[0040] The image signal processor 42 controls the camera assembly 30 and processes various image data captured by the camera assembly 30 to generate target image data. For example, the image signal processor 42 can perform demosaic processing, noise reduction processing, automatic exposure processing, automatic focus processing, automatic white balance processing, high dynamic range processing, etc. on the image data captured by the camera assembly 30.

[0041] In this embodiment, the main processor 40 and the image signal processor 42 cooperate to generate target image data of the object captured by the camera assembly 30. That is, the main processor 40 and the image signal processor 42 are configured to capture images of the object through the camera assembly 30 and perform various image processing on the captured image data.

[0042] The memory 44 stores programs to be executed by the main processor 40 and the image signal processor 42, as well as various data. For example, the data of the captured images is stored in the memory 44.

[0043] Memory 44 may include high-speed RAM and / or non-volatile memory such as flash memory and disk storage. That is, memory 44 may include non-transitory computer-readable media that stores programs.

[0044] The power supply circuit 46 may have a battery such as a lithium-ion rechargeable battery and a battery management unit (BMU) for managing the battery.

[0045] Communication circuit 48 is configured to receive and transmit data to communicate wirelessly with base stations of telecommunications network systems, the Internet, or other devices. The wireless communication can employ any communication standard or protocol, including but not limited to GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), LTE (Long Term Evolution), LTE-Advanced (evolution of LTE), and 5G. Communication circuit 48 may include an antenna and RF (radio frequency) circuitry.

[0046] here, Figure 4 A portion of the pixel array of the image sensor of a camera assembly 30 according to an embodiment of the present disclosure is shown. Figure 4 For example, eight pixel positions are shown in the image.

[0047] like Figure 4 As shown, the camera assembly 30 includes an image sensor that captures images of objects and generates color image data.

[0048] Then, for example, such as Figure 4 As shown, the image sensor has a green element block GK, a blue element block BK, and a red element block RK. The green element block GK, the blue element block BK, and the red element block RK are arranged in a Bayer-formatted array at each pixel location to generate color image data. In the Bayer format, in sparse image data, the number of green pixels is twice the number of red pixels or blue pixels. The green element block GK, blue element block BK, and red element block RK each contain a plurality of physical pixel elements (in...). Figure 4In the example, there are four physical pixel elements. The green element block GK, the blue element block BK, and the red element block RK have rectangular shapes. That is, as shown... Figure 4 As shown, the pixel array in this embodiment uses a 2×2 merging technique.

[0049] like Figure 4 As shown, the green element block GK consists of two green physical pixel elements G and two white physical pixel elements W. In the green element block GK, the two green physical pixel elements G are located on the first diagonal, while the two white physical pixel elements W are located on the second diagonal, thus corresponding to each of the four corners respectively.

[0050] For example, a green signal value is generated by combining two charges from two green physical pixel elements G (green merged image data). Similarly, a white signal value is generated by combining two charges from two white physical pixel elements W (white merged image data).

[0051] Furthermore, in this embodiment, such as Figure 4 As shown, the blue element block BK consists of two blue physical pixel elements B and two white physical pixel elements W. In the blue element block BK, the two blue physical pixel elements B are located on the first diagonal, while the two white physical pixel elements W are located on the second diagonal, thus corresponding to each of the four corners respectively.

[0052] For example, a blue signal value is generated by combining two charges from two blue physical pixel elements B (blue merged image data). Similarly, a white signal value is generated by combining two charges from two white physical pixel elements W (white merged image data).

[0053] Additionally, in this embodiment, as Figure 4 As shown, the red element block RK consists of two red physical pixel elements R and two white physical pixel elements W. In the red element block RK, the two red physical pixel elements R are located on the first diagonal, while the two white physical pixel elements W are located on the second diagonal, thus corresponding to each of the four corners respectively.

[0054] For example, a red signal value is generated by combining two charges from two red physical pixel elements R (red merged image data). Similarly, a white signal value is generated by combining two charges from two white physical pixel elements W (white merged image data).

[0055] Next, an example of an operation including image processing will be described below, in which the electronic device 10 with the above configuration acquires image data conforming to the Bayer format.

[0056] here,Figure 5A This is a schematic diagram illustrating an example of a process for generating image data conforming to the Bayer format according to an embodiment of the present disclosure. Figure 5B This illustrates an embodiment according to the present disclosure. Figure 5A A schematic diagram illustrating an example of the process used to generate target image data. Figure 6 It is shown in Figure 5A The diagram illustrates an example of a process in image data processing where embedded sparse image data is generated for a camera component and input to an image signal processor. Figure 7A It is used for explanation Figure 5A The diagram shows the calculation of the white residual data of the white image data of adjacent green and red pixel blocks. Figure 7B This is a schematic diagram illustrating the arrangement of green residual data estimated from white residual data based on white image data. It should be noted that... Figure 7A and Figure 7B An example is shown where the red element block RK is located at the first pixel position, but the same applies when the blue element block BK is located at the first pixel position.

[0057] In this embodiment, for example, the main processor 40 performs the target image generation process to generate target image data. However, the main processor 40 cooperates with the image signal processor 42 to generate the target image data. Therefore, the main processor 40 and the image signal processor 42 constitute the image processor in this embodiment.

[0058] Furthermore, in this embodiment, the program instructions for target image generation processing are stored in a non-transitory computer-readable medium in memory 44. When the program instructions are read from memory 44 and executed in main processor 40, main processor 40 implements... Figure 5A , Figure 5B and Figure 6 The target image generation process is shown.

[0059] First, such as Figure 5A As shown, the camera assembly 30 generates green merged image data and white merged image data for green element block GK, red merged image data and white merged image data for red element block RK, and blue merged image data and white merged image data for blue element block BK by using an image sensor for pixel merging processing.

[0060] More specifically, the camera assembly 30 generates green merged image data by combining the charges of two green physical pixel elements of the green element block GK and white merged image data by combining the charges of two white physical pixel elements of the green element block GK through pixel merging processing.

[0061] Similarly, the camera assembly 30 generates blue merged image data by combining the charges of two blue physical pixel elements of blue element block BK and white merged image data by combining the charges of two white physical pixel elements of blue element block BK through pixel merging processing.

[0062] Similarly, the camera assembly 30 generates red merged image data by combining the charges of two red physical pixel elements of the red element block RK and white merged image data by combining the charges of two white physical pixel elements of the red element block RK through pixel merging processing.

[0063] Next, the camera component 30 acquires green merged image data and white merged image data of the green element block GK, red merged image data and white merged image data of the red element block RK, and blue merged image data and white merged image data of the blue element block BK generated by the pixel merging process of the camera component 30.

[0064] Then, as Figure 5A As shown, camera component 30 uses green merged image data of green element block GK, red merged image data of red element block RK, and blue merged image data of blue element block BK as sparse image data RX, GX, and BX conforming to the Bayer format.

[0065] Next, as Figure 6 As shown in step S1, camera component 30 calculates the green-to-white ratio R of the green merged image data and the white merged image data of the green element block GK. gw ( Figure 5A (1) in the middle.

[0066] For example, in focusing on Figure 7A In the case of the two pixel positions shown, as shown in the following formula, the camera component 30 calculates the green-to-white ratio R of the green merged image data (G1+G2) and the white merged image data (W1+W2) of the green element block GK. gw .

[0067] R gw = (G1+G2) / (W1+W2)

[0068] Next, as Figure 6 As shown in step S2, the camera component 30 estimates the green residual data D. g The white residual data D should be obtained by comparing the white merged image data of the red element block RK (or blue element block BK) at the first pixel position with the white merged image data of the green element block GK at the second pixel position adjacent to the first pixel position. w (Figure 5A (2) in the middle.

[0069] More specifically, the camera assembly 30 obtains white residual data D by subtracting the white merged image data of the green element block GK located at the second pixel position from the white merged image data of the red element block RK (or blue element block BK) at the first pixel position. w .

[0070] For example, in focusing on Figure 7A In the case of the two pixel positions shown, as shown in the formula below, the camera component 30 obtains the white residual data D by subtracting the white merged image data (W1+W2) of the green element block GK located at the second pixel position from the white merged image data (W3+W4) of the red element block RK at the first pixel position. w .

[0071] D w = (W3+W4)-(W1+W2)

[0072] Next, as Figure 6 As shown in step S3, the camera component 30 bases the green-to-white ratio R on the green element block GK corresponding to the second pixel position. gw and white residual data D w ( Figure 5A (3) is used to estimate the estimated green residual data D corresponding to the first pixel position. g .

[0073] More specifically, as shown in the formula below, the camera component 30 uses the green-to-white ratio R corresponding to the green element block GK located at the second pixel position. gw Multiply by the white residual data to obtain the estimated green residual data D g .

[0074] D g =D w ×R gw

[0075] Then, as Figure 6 As shown in step S4, the camera component 30 estimates the green residual data D at the first pixel location. g (or based on estimated green residual data D) g The data is embedded in sparse image data RX and BX. Figure 5A In (4), embedded sparse image data ESD is generated from sparse image data.

[0076] For example, in focusing on Figure 7AIn the case of the two pixel positions shown, assume that the green element block EGK used for dense image data is located at the first pixel position ( Figure 7B In this case, the green residual data D is estimated as shown in the formula below. g It is the difference between the estimated green merged image data (G3+G4) of the assumed green element block EGK and the green merged image data (G1+G2) of the green element block GK.

[0077] D g = Estimate ((G3+G4)-(G1+G2))

[0078] Next, as Figure 5B As shown, the camera assembly 30 inputs embedded sparse image data ESD to the image signal processor 42. Figure 5B (5)). The image signal processor 42 processes the sparse image data in the embedded sparse image data to generate the target image data (RGB processing).

[0079] Then, as Figure 5B As shown, after the embedded sparse image data has been input to the image signal processor, the main processor 40 obtains the embedded sparse image data ESD from the image signal processor 42. That is, the image signal processor 42 has one or more data output ports for outputting various data during processing and one or more data input ports for inputting various data to the image signal processor 42. Therefore, the main processor 40 obtains the embedded sparse image data via one of the data output ports of the image signal processor 42.

[0080] Then, as Figure 5B As shown, the main processor 40 extracts estimated green residual data D from the embedded sparse image data obtained by the image signal processor. g ( Figure 5B (6) in the middle.

[0081] Then, as Figure 5B As shown, the main processor 40 is based on the estimated green residual data D g Reconstructing dense image data ( Figure 5B (7) in the middle.

[0082] Then, the main processor 40 performs a predetermined plane process on the dense image data. Figure 5B (8) in the middle.

[0083] If needed, for example, the main processor 40 can base its calculations on the estimated green residual data D. g To generate compressed data. There are several methods to compress residual data to reduce the number of bits in the residual data.

[0084] In this case, if needed, the main processor 40 can use the compressed estimated green residual data D g The obtained compressed data is embedded in the sparse image data corresponding to the first pixel position to generate embedded sparse image data. Figure 5A (4)). Furthermore, in this case, the main processor expands the compressed data generated from the segmented data into embedded sparse image data from the image signal processor. Then, the main processor reconstructs dense image data based on the residual data reconstructed from the compressed data. Figure 5B (6) in the middle.

[0085] Additionally, for example, the main processor 40 obtains image data generated based on sparse image data from one of the data output ports of the image signal processor 42. The image data generated during the processing based on sparse image data can be obtained from the image signal processor 42.

[0086] Next, for example, the main processor 40 combines the reconstructed dense image data and the generated image data to generate combined image data.

[0087] Next, for example, the main processor 40 inputs the combined image data to one of the data input ports of the image signal processor 42 to improve the resolution. Figure 5B (9)). Subsequently, the image signal processor 42 continues to process the combined image data and finally outputs the target image data from the image signal processor 42. Figure 5B (10) in the middle.

[0088] For example, an image to be displayed on the display 20 can be generated based on the target image data. Optionally, the target image data can be stored in the memory 44. The target image data can be in various formats. For example, the target image data can be in the formats of JPEG, TIFF, GIF, etc.

[0089] As described above, according to the electronic device 10 of this embodiment, dense image data can be embedded as residual data into sparse image data input to the image signal processor 42, and then the dense image can be reconstructed based on the residual data embedded in the sparse image data. Therefore, by combining image data generated based on sparse image data and dense image data reconstructed from the residual data embedded in the sparse image data, an image based on dense image data can be regenerated, and the quality of the target image data can be improved.

[0090] Furthermore, since the format of the embedded sparse image data is the same as that of the sparse image data, the common image signal processor used for sparse image data can still be used as the image signal processor 42 for the embedded sparse image data. Therefore, it is not necessary to develop a new image signal processor 42 to process the embedded sparse image data of this embodiment to generate the target image data.

[0091] In the description of embodiments of this disclosure, it should be understood that terms such as “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “rear,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” should be interpreted as referring to the direction or position as described or shown in the accompanying drawings. These relative terms are used only to simplify the description of this disclosure and do not indicate or imply that the mentioned devices or elements must have a specific orientation, or be constructed or operated in a specific orientation. Therefore, these terms should not be construed as limiting the present application.

[0092] Furthermore, the terms such as “first” and “second” are used herein for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature. In the description of this disclosure, “a plurality of” means at least two, unless otherwise explicitly specified.

[0093] In the description of the embodiments of this disclosure, unless otherwise specified or limited, the terms “installed,” “connected,” and “linked” should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection; as a mechanical connection or an electrical connection; as a direct or indirect connection through an intermediate structure; or as internal communication between two elements, which can be understood by those skilled in the art based on the specific circumstances.

[0094] In embodiments of this disclosure, unless otherwise expressly specified and limited, "above" or "below" the second feature can include embodiments in which the first and second features are in direct contact, or embodiments in which the first and second features are not in direct contact but are in contact through another feature between them. Furthermore, "above," "over," and "on top" the second feature includes embodiments in which the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is located at a height higher than the second feature. Conversely, "below," "below," and "at the bottom" the second feature can include embodiments in which the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is located at a height lower than the second feature.

[0095] Various embodiments and examples have been provided in the foregoing description to implement different structures of this disclosure. Certain elements and arrangements have been described above to simplify this disclosure. However, these elements and arrangements are merely examples and are not intended to limit this disclosure. Furthermore, reference numerals and / or reference letters may be repeated in different examples of this disclosure. This repetition is for simplification and clarity and does not indicate a relationship between different embodiments and / or arrangements. Additionally, examples of different processes and materials are provided in this disclosure. However, those skilled in the art will understand that other processes and / or materials may also be applied.

[0096] Throughout this specification, references to "embodiment," "some embodiments," "example embodiments," "example," "specific example," or "some examples" mean that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Therefore, the foregoing terms appearing throughout this specification do not necessarily refer to the same embodiment or example of this disclosure. Furthermore, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0097] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more operable instructions for implementing a particular logical function or step in a process, and the scope of the preferred embodiments of this application includes other embodiments, wherein those skilled in the art will understand that functions may be performed not in the order shown or discussed, including in substantially the same order or in reverse order.

[0098] The logic and / or steps (e.g., a specific list of executable instructions for implementing logical functions) represented in the flowchart or otherwise described herein can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system capable of fetching and executing instructions from and executing instructions from an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means suitable for containing, storing, communicating, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of computer-readable media include, but are not limited to: electronic connections (electronic devices) having one or more wires, portable computer disk enclosures (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0099] It should be understood that the various parts of this disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, the steps or methods can be implemented by any one or a combination of the following techniques known in the art, as in another embodiment: discrete logic circuits having logic gates for implementing data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0100] Those skilled in the art will understand that all or part of the steps in the exemplary methods described above can be implemented by instructing related hardware using a program. The program can be stored in a computer-readable storage medium, and when run on a computer, the program includes one or a combination of steps from the method embodiments of this disclosure.

[0101] Furthermore, the functional units of the embodiments of this disclosure can be integrated into the processing module, or these units can be separate physical entities, or two or more units can be integrated into the processing module. The integrated module can be implemented in hardware or as a software functional module. When the integrated module is implemented as a software functional module and sold or used as a standalone product, the integrated module can be stored in a computer-readable storage medium.

[0102] The aforementioned storage media can be read-only memory, disk, CD, etc.

[0103] Although embodiments of the present disclosure have been shown and described, those skilled in the art should understand that these embodiments are illustrative and should not be construed as limiting the present disclosure, and that changes, modifications, substitutions and variations may be made to these embodiments without departing from the scope of the present disclosure.

Claims

1. An electronic device comprising: a camera assembly including an image sensor configured to capture an image of an object and generate color image data, wherein the image sensor has a green element block, a blue element block, and a red element block arranged in an array in a Bayer format at each pixel location so as to generate color image data, the green element block, the blue element block, and the red element block each including a plurality of physical pixel elements, the green element block including two green physical pixel elements and two white physical pixel elements, the blue element block including two blue physical pixel elements and two white physical pixel elements, and the red element block including two red physical pixel elements and two white physical pixel elements; and a main processor performing image processing, wherein: the camera assembly acquires green binning image data and white binning image data of the green element block, red binning image data and white binning image data of the red element block, and blue binning image data and white binning image data of the blue element block generated by pixel binning processing of the camera assembly, the camera assembly calculates a green-to-white ratio of the green binning image data and the white binning image data of the green element block, the camera assembly acquires white residual data based on a difference between white binning image data of a red element block or a blue element block at a first pixel location at which estimated green residual data is supposed to be estimated and white binning image data of a green element block at a second pixel location adjacent to the first pixel location, and the camera assembly estimates estimated green residual data corresponding to the first pixel location based on a green-to-white ratio corresponding to the green element block at the second pixel location and the white residual data.

2. The electronic device of claim 1, wherein, the camera assembly acquires the white residual data by subtracting the white binning image data of the green element block at the second pixel location from the white binning image data of the red element block or the blue element block at the first pixel location. 3.The electronic device of claim 1, wherein, the camera assembly acquires the estimated green residual data by multiplying the green-to-white ratio corresponding to the green element block at the second pixel location by the white residual data.

4. The electronic device according to claim 1, wherein the green element block, the blue element block, and the red element block have a rectangular shape, and wherein: in the green element block, the two green physical pixel elements are located on a first diagonal line and the two white physical pixel elements are located on a second diagonal line, thereby respectively corresponding to each of four corners, in the blue element block, the two blue physical pixel elements are located on the first diagonal line and the two white physical pixel elements are located on the second diagonal line, thereby respectively corresponding to each of the four corners, and in the red element block, the two red physical pixel elements are located on the first diagonal line and the two white physical pixel elements are located on the second diagonal line, thereby respectively corresponding to each of the four corners. In the red elemental block, the two red physical pixel elements are located on a first diagonal line and the two white physical pixel elements are located on a second diagonal line, thereby respectively corresponding to each of four corners. 5.The electronic device of claim 1, wherein, The camera assembly generates the green bin image data and the white bin image data of the green elemental block, the red bin image data and the white bin image data of the red elemental block, and the blue bin image data and the white bin image data of the blue elemental block by using the image sensor from pixel binning processing.

6. The electronic device of claim 1, wherein, The camera assembly uses the green bin image data of the green elemental block, the red bin image data of the red elemental block, and the blue bin image data of the blue elemental block as sparse image data conforming to the Bayer format.

7. The electronic device of claim 6, wherein, The camera assembly generates embedded sparse image data from the sparse image data by embedding the estimated green residual data into the sparse image data at the first pixel location.

8. The electronic device of claim 7, wherein, The electronic device further includes an image signal processor that processes the sparse image data in the embedded sparse image data to generate target image data, wherein the camera assembly inputs the embedded sparse image data to the image signal processor.

9. The electronic device of claim 8, wherein: after the embedded sparse image data has been input to the image signal processor, the main processor obtains the embedded sparse image data from the image signal processor, the main processor extracts the estimated green residual data from the embedded sparse image data obtained by the image signal processor, and the main processor reconstructs dense image data based on the estimated green residual data.

10. A method of generating image data, comprising: obtaining green bin image data and white bin image data of a green elemental block, red bin image data and white bin image data of a red elemental block, and blue bin image data and white bin image data of a blue elemental block generated by pixel binning processing of a camera assembly, wherein the camera assembly includes an image sensor configured to capture an image of an object and generate color image data, wherein the image sensor has the green elemental block, the blue elemental block, and the red elemental block arranged in an array of a Bayer format at each pixel location in order to generate color image data, so as to generate color image data, the green elemental block, the blue elemental block, and the red elemental block respectively include a plurality of physical pixel elements, the green elemental block includes two green physical pixel elements and two white physical pixel elements, the blue elemental block includes two blue physical pixel elements and two white physical pixel elements, and the red elemental block includes two red physical pixel elements and two white physical pixel elements; calculating a green-to-white ratio of the green bin image data and the white bin image data of the green elemental block; obtaining white residual data based on a difference between white merged image data of a red element block or a blue element block at a first pixel location at which estimated green residual data should be estimated and white merged image data of a green element block at a second pixel location adjacent to the first pixel location; and estimating estimated green residual data corresponding to the first pixel location based on a green-to-white ratio corresponding to the green element block at the second pixel location and the white residual data.

11. A non-transitory computer readable medium comprising program instructions stored on the non-transitory computer readable medium for performing at least the following: acquiring green binning image data and white binning image data of a green element block, red binning image data and white binning image data of a red element block, and blue binning image data and white binning image data of a blue element block generated by a pixel binning process of the camera assembly, wherein, The camera assembly includes an image sensor configured to capture an image of an object and generate color image data, wherein the image sensor has the green element block, the blue element block, and the red element block arranged in an array in a Bayer format at each pixel location to generate color image data, the green element block, the blue element block, and the red element block each including a plurality of physical pixel elements, the green element block including two green physical pixel elements and two white physical pixel elements, the blue element block including two blue physical pixel elements and two white physical pixel elements, and the red element block including two red physical pixel elements and two white physical pixel elements; calculating a green-to-white ratio of the green merged image data and the white merged image data of the green element block; obtaining white residual data based on a difference between white merged image data of a red element block or a blue element block at a first pixel location at which estimated green residual data should be estimated and white merged image data of a green element block at a second pixel location adjacent to the first pixel location; and estimating estimated green residual data corresponding to the first pixel location based on a green-to-white ratio corresponding to the green element block at the second pixel location and the white residual data.

Images