Method for generating target image data, electronic device and non-transitory computer-readable medium

Abstract

A method for generating target image data includes: acquiring sparse image data and dense image data, wherein the sparse image data includes at least a plurality of pixels among first color pixels, second color pixels and third color pixels, and the dense image data includes a plurality of pixels among the first color pixels; generating residual data based on the dense image data by calculating the difference between every two adjacent pixels in the dense image data; compressing the residual data to generate compressed data to reduce the data volume of the residual data; segmenting each pixel of the compressed data to generate segmented data, wherein the segmented data includes a first data portion and a second data portion; and embedding the segmented data into the sparse image data to generate embedded sparse image data.

Description

Technical Field

[0001] The present invention relates to a method for generating target image data, an electronic device for implementing such a method, and a non-transitory computer-readable medium including program instructions stored thereon for executing such a method. Background Art

[0002] Electronic devices such as smart phones and tablet terminals are widely used in our daily lives. Now, many electronic devices are equipped with camera components to capture images. Some electronic devices are portable and easy to carry, so users of electronic devices can easily take photos of objects anytime and anywhere by using the camera components of the electronic devices.

[0003] There are many formats to acquire an image of an object and generate its target image data. One of the widely known formats is the Bayer format, which includes sparse image data.

[0004] In addition to the sparse image data, in order to improve the image quality of the object based on the target image data, dense image data is also generated when the camera assembly captures the object. In this case, the sparse image data and the dense image data are used to generate the target image data to be displayed on the display or to be stored in the memory of the electronic device. However, ordinary image signal processors cannot process these two types of image data. Summary of the invention

[0005] The present invention aims to solve at least one of the above technical problems. Therefore, the present invention needs to provide a method for generating target image data and an electronic device for implementing the method.

[0006] According to the present invention, the method for generating target image data may include:

[0007] Acquire sparse image data and dense image data, wherein the sparse image data includes at least a plurality of pixels among first color pixels, second color pixels, and third color pixels, and the dense image data includes a plurality of pixels among first color pixels;

[0008] generating residual data based on the dense image data by calculating a difference between every two adjacent pixels in the dense image data;

[0009] compressing the residual data to generate compressed data, so as to reduce the data volume of the residual data;

[0010] Segmenting each pixel of the compressed data to generate segmented data, wherein the segmented data includes a first data portion and a second data portion; and

[0011] The segmentation data is embedded into the sparse image data to generate embedded sparse image data.

[0012] In some embodiments, the method may further include inputting the embedded sparse image data into an image signal processor, the image signal processor processing the sparse image data in the embedded sparse image data to generate the target image data.

[0013] In some embodiments, every two adjacent pixels in the dense image data may constitute a first group, and the first group includes a first value of the first color pixel and a second value of the first pixel.

[0014] In some embodiments, every two adjacent pixels in the sparse image data may constitute a second group, and the second group includes the third value of the first color pixel and the fourth value of the second color pixel, or includes the third value of the first color pixel and the fourth value of the third color pixel.

[0015] In some embodiments, the first value of the first color pixel in the first group may correspond to a third value of the first color pixel located in the second group, and the position of the first group of dense image data corresponds to the position of the second group of sparse image data.

[0016] In some embodiments, generating the residual data may include subtracting a second value of the first color pixel from a first value of the first color pixel in the first group.

[0017] In some embodiments, generating compressed data may include reducing the number of bits of the residual data.

[0018] In some embodiments, reducing the number of bits of the residual data may include converting the residual data into the compressed data based on a compression curve defining a relationship between pixel values ​​of the residual data and pixel values ​​of the compressed data.

[0019] In some embodiments, each of the first color pixels, each of the second color pixels, and each of the third color pixels in the sparse image data may have a spare space in which the sparse image data is not stored.

[0020] In some embodiments, when generating the segmentation data, the size of the first data portion and the second data portion of the segmentation data can match the size of the spare space in each of the first color pixels, each of the second color pixels, and each of the third color pixels in the sparse image data.

[0021] In some embodiments, generating the embedded sparse image data may include embedding the first data portion and the second data portion of the first group into a spare space of a second group of the sparse image data.

[0022] In some embodiments, the method may further include:

[0023] After the embedded sparse image data is input to the image signal processor, acquiring the embedded sparse image data from the image signal processor;

[0024] extracting segmentation data from embedded sparse image data acquired by an image signal processor to generate compressed data;

[0025] expanding the compressed data generated from the segmented data to reconstruct the residual data; and

[0026] Dense image data is reconstructed based on the residual data reconstructed from the compressed data.

[0027] In some embodiments, the method may further include:

[0028] acquiring generated image data during processing to generate target image data based on the sparse image data from the image signal processor; and

[0029] The generated image data and dense image data reconstructed from the residual data are combined to generate combined image data.

[0030] In some embodiments, the method may further comprise inputting the combined image data into the image signal processor.

[0031] In some embodiments, the first color may be green, the second color may be red, and the third color may be blue.

[0032] In some embodiments, the sparse image data may conform to a Bayer format.

[0033] According to the present invention, an electronic device may include:

[0034] a camera assembly configured to acquire an image of a subject and generate sparse image data and dense image data; and

[0035] The main processor is configured as:

[0036] Acquire sparse image data and dense image data, wherein the sparse image data includes at least a plurality of pixels among first color pixels, second color pixels, and third color pixels, and the dense image data includes a plurality of pixels among first color pixels;

[0037] generating residual data based on the dense image data by calculating a difference between every two adjacent pixels in the dense image data;

[0038] compressing the residual data to generate compressed data, so as to reduce the data volume of the residual data;

[0039] Segmenting each pixel of the compressed data to generate segmented data, wherein the segmented data includes a first data portion and a second data portion; and

[0040] The segmentation data is embedded into the sparse image data to generate embedded sparse image data.

[0041] In some embodiments, the main processor may be further configured to input the embedded sparse image data into an image signal processor, and the image signal processor processes the sparse image data in the embedded sparse image data to generate the target image data.

[0042] In some embodiments, every two adjacent pixels in the dense image data may constitute a first group, and the first group may include a first value of the first color pixel and a second value of the first pixel.

[0043] In some embodiments, every two adjacent pixels in the sparse image data may constitute a second group, and the second group may include a third value of the first color pixel and a fourth value of the second color pixel, or may include a third value of the first color pixel and a fourth value of the third color pixel.

[0044] In some embodiments, the first value of the first color pixel in the first group may correspond to a third value of the first color pixel located in the second group, wherein the location of the first group of dense image data corresponds to the location of the second group of sparse image data.

[0045] In some embodiments, the residual data may be generated by subtracting the second value of the first color pixel from the first value of the first color pixel in the first group.

[0046] In some embodiments, the number of bits of the residual data may be reduced to generate compressed data.

[0047] In some embodiments, the number of bits of the residual data may be reduced by converting the residual data into compressed data based on a compression curve that defines a relationship between pixel values ​​of the residual data and pixel values ​​of the compressed data.

[0048] In some embodiments, each of the first color pixels, each of the second color pixels, and each of the third color pixels in the sparse image data may have a spare space in which the sparse image data is not stored.

[0049] In some embodiments, when the segmentation data is generated, the size of the first data portion and the second data portion of the segmentation data can match the size of the spare space in each of the first color pixels, each of the second color pixels, and each of the third color pixels in the sparse image data.

[0050] In some embodiments, the first data portion and the second data portion of the first group may be embedded into a spare space of the second group of the sparse image data to generate the embedded sparse image data.

[0051] In some embodiments, the main processor may be further configured to:

[0052] After the embedded sparse image data is input to the image signal processor, acquiring the embedded sparse image data from the image signal processor;

[0053] extracting segmented data from embedded sparse image data acquired by an image signal processor to generate compressed data;

[0054] expanding the compressed data generated from the segmented data to reconstruct the residual data; and

[0055] The dense image data is reconstructed based on the residual data reconstructed from the compressed data.

[0056] In some embodiments, the main processor may be further configured to:

[0057] acquiring generated image data during processing to generate target image data based on the sparse image data from the image signal processor; and

[0058] The generated image data and the dense image data reconstructed from the residual data are combined to generate combined image data.

[0059] In some embodiments, the main processor may be further configured to input the combined image data to the image signal processor.

[0060] In some embodiments, the first color may be green, the second color may be red, and the third color may be blue.

[0061] In some embodiments, the sparse image data may conform to a Bayer format.

[0062] According to the present invention, a non-transitory computer readable medium may include program instructions stored thereon for performing at least the following operations:

[0063] Acquire sparse image data and dense image data, wherein the sparse image data includes at least a plurality of pixels among first color pixels, second color pixels, and third color pixels, and the dense image data includes a plurality of pixels among first color pixels;

[0064] generating residual data based on the dense image data by calculating a difference between every two adjacent pixels in the dense image data;

[0065] compressing the residual data to generate compressed data, so as to reduce the data volume of the residual data;

[0066] Segmenting each pixel of the compressed data to generate segmented data, wherein the segmented data includes a first data portion and a second data portion; and

[0067] The segmentation data is embedded into the sparse image data to generate embedded sparse image data to generate target image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] These and / or other advantages of the embodiments of the present invention will become more apparent and easier to understand based on the following description in conjunction with the accompanying drawings, in which:

[0069] Figure 1 shows a plan view of a first side of an electronic device according to one embodiment of the present invention;

[0070] Figure 2 shows a plan view of a second side of an electronic device according to an embodiment of the present invention;

[0071] Figure 3 A block diagram of an electronic device according to an embodiment of the present invention is shown;

[0072] Figure 4 A flow chart of a target image generation process (Part 1) performed by an electronic device according to an embodiment of the present invention is shown;

[0073] Figure 5 A flow chart of a target image generation process (Part 2) performed by an electronic device according to an embodiment of the present invention is shown;

[0074] Figure 6 A schematic diagram for explaining a mechanism for generating embedded sparse image data to be input to an image signal processor in an embodiment of the present invention is shown;

[0075] Figure 7 A schematic diagram for explaining how to generate residual data and compressed data in an embodiment of the present invention is shown;

[0076] Figure 8One of the examples of compressing residual data to generate a compression curve of compressed data in an embodiment of the present invention is shown;

[0077] Fig. 9 A schematic diagram for explaining how to generate segmented data from compressed data in an embodiment of the present invention is shown;

[0078] Fig.10 A schematic diagram for explaining a mechanism for generating target image data in an embodiment of the present invention is shown;

[0079] Fig.11 A schematic diagram for explaining how to reconstruct pixel values ​​of residual data and dense image data in an embodiment of the present invention is shown; and

[0080] Fig.12 An example of generated image data based on sparse image data, dense image data reconstructed based on embedded sparse image data obtained from an image signal processor, and combined image data generated by combining the generated image data and the dense image data in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0081] Embodiments of the present invention will be described in detail below, and examples of the embodiments will be shown in the accompanying drawings. Throughout the description, identical or similar elements and elements having identical or similar functions are represented by identical reference numerals. The embodiments described herein with reference to the accompanying drawings are illustrative and intended to illustrate the present invention, but should not be construed as limiting the present invention.

[0082] Figure 1 A plan view of a first side of an electronic device 10 according to an embodiment of the present invention is shown, and Figure 2 A plan view of a second side of the electronic device 10 is shown according to an embodiment of the present invention. The first side may be referred to as a rear side of the electronic device 10, and the second side may be referred to as a front side of the electronic device 10.

[0083] like Figure 1 and 2 As shown, the electronic device 10 may include a display 20 and a camera assembly 30. In the present embodiment, the camera assembly 30 includes a first main camera 32, a second main camera 34, and a sub-camera 36. The first main camera 32 and the second main camera 34 capture an image of a first side of the electronic device 10, and the sub-camera 36 may capture an image of a second side of the electronic device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called external cameras, and the sub-camera 36 is a so-called built-in camera. As an example, the electronic device 10 may be a mobile phone, a tablet computer, a personal digital assistant, etc.

[0084] Although the electronic device 10 according to the present embodiment has three cameras, the electronic device 10 may have less than three cameras or more than three cameras. For example, the electronic device 10 may have two, four, five, etc. cameras.

[0085] Figure 3 1 shows a block diagram of an electronic device 10 according to this embodiment. Figure 3 As shown, in addition to the display 20 and the camera assembly 30, the electronic device 10 may include a main processor 40, an image signal processor 42, a memory 44, a power circuit 46, and a communication circuit 48. The display 20, the camera assembly 30, the main processor 40, the image signal processor 42, the memory 44, the power circuit 46, and the communication circuit 48 are connected to each other via a bus 50.

[0086] The main processor 40 executes one or more programs stored in the memory 44. The main processor 40 implements various applications and data processing of the electronic device 10 by executing the program. The main processor 40 can be one or more computer processors. The main processor 40 is not limited to one CPU core, and it can have multiple CPU cores. The main processor 40 can be the main CPU of the electronic device 10, an image processing unit (IPU), or a digital signal processor (DSP) provided with the camera assembly 30.

[0087] The image signal processor 42 controls the camera assembly 30 and processes various image data captured by the camera assembly 30 to generate target image data. For example, the image signal processor 42 can perform demosaic processing, noise reduction processing, automatic exposure processing, automatic focus processing, automatic white balance processing, high dynamic range processing, etc. on the image data captured by the camera assembly 30.

[0088] In the present embodiment, the main processor 40 and the image signal processor 42 cooperate with each other to generate target image data of the object captured by the camera assembly 30. That is, the main processor 40 and the image signal processor 42 are configured to capture an image of the object through the camera assembly 30 and perform various image processing on the captured image data.

[0089] The memory 44 stores programs and various data to be executed by the main processor 40. For example, data of a captured image is stored in the memory 44.

[0090] The memory 44 may include a high-speed RAM memory and / or a non-volatile memory such as a flash memory and a disk memory. That is, the memory 44 may include a non-transitory computer-readable medium storing a program.

[0091] The power circuit 46 may have a battery such as a lithium-ion rechargeable battery and a battery management unit (BMU) for managing the battery.

[0092] The communication circuit 48 is configured to receive and send data to communicate with a base station of a telecommunication network system, the Internet or other devices via wireless communication. The wireless communication may adopt any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), LTE-Advanced, Fifth Generation (5G). The communication circuit 48 may include an antenna and a radio frequency (RF) circuit.

[0093] Figure 4 and Figure 5 A flow chart of a target image generation process performed by the electronic device 10 according to the present embodiment is shown. In the present embodiment, for example, the target image generation process is performed by the main processor 40 to generate target image data. However, the main processor 40 cooperates with the image signal processor 42 to generate the target image data. Therefore, the main processor 40 and the image signal processor 42 constitute the image processor in the present embodiment.

[0094] In addition, in the present embodiment, the program instructions of the target image generation process are stored in a non-transitory computer readable medium of the memory 44. When the program instructions are read out from the memory 44 and executed in the main processor 40, the main processor 40 implements Figure 4 and Figure 5 The target image generation process shown.

[0095] like Figure 4 As shown, for example, the main processor 40 acquires sparse image data and dense image data (step S10). In the present embodiment, the main processor 40 acquires sparse image data and dense image data from the camera assembly 30. That is, the camera assembly captures an image of an object and generates sparse image data and dense image data. In the present embodiment, the sparse image data includes a plurality of pixels consisting of green pixels, red pixels, and blue pixels. On the other hand, the dense image data includes a plurality of pixels of green pixels.

[0096] In order to generate sparse image data and dense image data with the camera assembly 30, the camera assembly 30 may have a dedicated image sensor to perform a single imaging operation to capture an image of an object with a single camera and generate sparse image data and dense image data. In this case, for example, the first main camera 32 may perform a single imaging operation to capture an image of an object and generate sparse image data and dense image data.

[0097] On the other hand, the camera assembly 30 can use two cameras to capture an image of the object and perform a single imaging operation to generate sparse image data and dense image data. In this case, for example, the first main camera 32 captures an image of the object and generates sparse image data, while the second main camera 34 captures an image of the object and generates dense image data.

[0098] On the other hand, the camera assembly 30 can perform two imaging operations to capture an image of an object with a single camera and generate sparse image data and dense image data. For example, the sub-camera 36 performs a first imaging operation to acquire an image of an object to generate sparse image data, and then the sub-camera 36 performs a second imaging operation immediately after the first imaging operation to capture an image of the object to generate dense image data.

[0099] Figure 6 A schematic diagram for explaining a mechanism for generating embedded sparse image data to be input to the image signal processor 42 is shown.

[0100] like Figure 6 As shown, the sparse image data conforms to the Bayer format. Therefore, the green, red, and blue colors in the color filters of the image sensor of the camera assembly 30 that captures the image of the object are set to conform to the Bayer setting. In the Bayer format, the number of green pixels in the sparse image data is twice the number of red pixels or blue pixels. The sparse image data may also be referred to as raw (RAW) data from the camera assembly 30.

[0101] The dense image data consists of green pixels. This is because the brightness of green is more sensitive to the human eye than the brightness of red or blue. In this embodiment, the dense image is captured to adjust the brightness of the target image data.

[0102] Next, if Figure 4 As shown, for example, the main processor 40 generates residual data based on the dense image data (step S12). That is, in this embodiment, in order to reduce the amount of data, the difference between every two adjacent pixels in the dense image data is calculated to generate residual data.

[0103] Figure 7 FIG. 2 shows a schematic diagram illustrating how to generate residual data and compressed data. Figure 6 and Figure 7 As shown, a plurality of green pixels P1 are included in both the sparse image data and the dense image data. On the other hand, a plurality of green pixels P2 are included in the dense image data, but they are not included in the sparse image data. Figure 7 , one of the green pixels P1 and one of the green pixels P2 are shown as examples.

[0104] Usually, the brightness of every two adjacent pixels is similar to or the same as the brightness of another pixel. That is, the difference between the value of green pixel P1 and the value of green pixel P2 adjacent to green pixel P1 is usually small. Therefore, in this embodiment, in order to reduce the amount of data, the value of green pixel P2 is subtracted from the value of green pixel P1 adjacent to green pixel P2 to obtain the difference between the value of green pixel P1 and the value of green pixel P2 adjacent to green pixel P1.

[0105] In other words, every two adjacent pixels in the dense image data form a first group to generate residual data. Figure 6 In the example of sparse image data in , there are 12 (3×4) first groups to calculate residual data. After the residual data is generated, the number of pixels of the residual data is half the number of pixels of the dense image data.

[0106] For example, in this embodiment, one pixel of dense image data consists of 10 bits. That is, one pixel value of dense image data is between 0 and 1023. On the other hand, one pixel of residual data consists of 11 bits because one pixel value of residual data is between +1023 and -1024.

[0107] Next, if Figure 4 As shown, for example, the main processor 40 generates compressed data based on the residual data (step S14). There are various methods for compressing the residual data to reduce the number of bits of the residual data. Therefore, an example of a method for compressing the residual data will be described here.

[0108] Figure 8 One of the examples of a compression curve for compressing residual data to generate compressed data is shown. That is, the residual data is converted into compressed data based on the compression curve. The compression curve is also called a tone curve, which is used to compress various data and defines the relationship between the pixel value of the residual data and the pixel value of the compressed data. The number of bits of the pixel value of the compressed data is less than the number of bits of the pixel value of the residual data.

[0109] like Figure 7 and Figure 8 As shown, in this embodiment, one pixel of 11 bits of residual data is compressed into one pixel of 8 bits of compressed data. That is, one pixel value of residual data is between -1024 and +1023, while one pixel value of compressed data is between -128 and +127.

[0110] As described above, since the difference between each two adjacent pixel values ​​of dense image data is generally small, the compression curve is substantially linear in a range where the absolute value of the pixel of the residual data is small. On the other hand, in a range where the absolute value of the pixel of the residual data is large, the compression curve is substantially flat or constant. As a result, the compression curve is S-shaped.

[0111] based on Figure 8 The residual data is compressed using the compression curve shown in FIG. 1 , and the 11-bit pixel of the residual data can be compressed into the 8-bit pixel of the compressed data. For example, if the pixel value of the residual data is 10, the pixel value of the residual data is also 10. Therefore, if the pixel value of the residual data is expanded, the pixel value of the residual data can be returned to 10. That is, within the range where the absolute value of the pixel of the residual data is small, the compressed data can be returned to the residual data that is substantially the same as the original data.

[0112] On the other hand, if the pixel value of the residual data is 1023, the pixel value of the compressed data is 127. In addition, if the pixel value of the residual data is 850, the pixel value of the compressed data is 126. That is, in the range where the absolute value of the pixel of the residual data is large, the compressed data cannot be returned to the same residual data as the original data. In other words, when the original absolute value of the pixel of the residual data is large, the pixel value of the residual data expanded based on the compressed data is rough. However, since the possibility that the absolute value of the pixel of the residual data is large is lower than the possibility that the absolute value of the pixel of the residual data is small, the low reproducibility of the value in the range where the absolute value of the pixel of the residual data is large is acceptable.

[0113] Next, if Figure 4 As shown, the main processor 40 generates segmentation data based on the compressed data (step S16). That is, since the size of the pixel of the compressed data is too large to be embedded into the sparse image data, in this embodiment, each pixel of the compressed data is divided into two pieces of data, i.e., a first data portion and a second data portion.

[0114] Fig. 9 A schematic diagram illustrating how segmented data is generated from compressed data is shown. Fig. 9 The upper part shows a comparative example of the related art, Fig. 9 The lower part of FIG. 1 shows a description of this embodiment.

[0115] like Fig. 9 As shown, the pixel value of the compressed data is represented by 8 bits and is divided into a first data part of 4 bits and a second data part of 4 bits.

[0116] In the present embodiment, the available space in the image signal processor 42 consists of 14 bits for each pixel of the sparse image data, but each pixel of the sparse image data requires 10 bits. Therefore, 4 bits of the 14 bits are reserved bits and are not used in the image signal processor 42. That is, the space of 4 bits of the 14 bits is a spare space that does not store the sparse image data.

[0117] Therefore, in this embodiment, in order to be able to insert the 8-bit pixel value of the compressed data into the 4 reserved bits of the sparse image data, the 8-bit pixel value of the compressed data is divided into two 4-bit data as segmentation data. As a result, the size of the first data portion and the size of the second data portion match the size of the spare space of the sparse image data. In this embodiment, each pixel of 8 bits is divided into 4 bits of the first part data and 4 bits of the second part data.

[0118] Next, if Figure 4 As shown, for example, the main processor 40 embeds the segmentation data into the sparse image data to generate embedded sparse image data (step S18). Fig. 9 and Figure 6 As shown, 4 bits of segmentation data are embedded into 4 reserved bits of sparse image data. More specifically, each red pixel, each blue pixel, and each green pixel of the sparse image data has 4 reserved bits, which is a spare space for the image signal processor 42. Hereinafter, the sparse image data in which the segmentation data has been embedded is also referred to as embedded sparse image data.

[0119] In the sparse image data of the Bayer format, the red pixel R1 and the green pixel P1 may constitute a second group, and the blue pixel B1 and the green pixel P1 may also constitute a second group. In the Bayer format, every two adjacent pixels include a green pixel P1 and a red pixel R1 or a blue pixel B1.

[0120] The green pixel P1 in the second group corresponds to the green pixel P1 in the first group located at a position corresponding to the first group of dense image data. That is, when the position of the second group of sparse image data is the same as the position of the first group of dense image data, the value of the green pixel P1 in the second group of sparse image data is substantially the same as the value of the green pixel P1 in the first group of dense image data.

[0121] In this embodiment, the first data portion of the segmented data is embedded into the 4-bit spare space of the second group of green pixels P1, and the second data portion of the segmented data is embedded into the 4-bit spare space of red pixels R1 or blue pixels B1.

[0122] That is, the first data portion and the second data portion of the segmented data are embedded into every two adjacent red and green pixels R1 and P1 of the second group or every two adjacent blue and green pixels B1 and P1 of the second group. In the present embodiment, all the first data portions and the second data portions of the segmented data are embedded into the spare space of the sparse image data.

[0123] In this embodiment, the first group of dense image data and the second group of sparse image data have a one-to-one correspondence. Therefore, the first data portion and the second data portion are embedded into two adjacent pixels of the second group corresponding to the original first group to calculate their first data portion and second data portion. That is, the first data portion and the second data portion are inserted into the second group at the position corresponding to the original first group.

[0124] However, the segmentation data may be embedded in the spare space of the sparse image data in any manner if the segmentation data can specify the positions at which the first data portion and the second data portion of the segmentation data are to be embedded in the sparse image data.

[0125] exist Fig. 9 In the comparative example shown in the upper part, the information of the green pixel P2 is discarded when the sparse image data is input to the image signal processor 42. However, in this embodiment, the data of the green pixel P2 can also be embedded in the sparse image data, so the information of the green pixel P2 is not discarded.

[0126] Next, if Figure 4 As shown, for example, the main processor 40 inputs the embedded sparse image data to the image signal processor 42 (step S20). That is, the embedded sparse image data including the sparse image data and the segmentation data is input to the image signal processor 42 to generate the target image data. Thereafter, the image signal processor 42 starts to process the sparse image data in the embedded sparse image data to obtain the target image data.

[0127] Next, if Figure 5 As shown, for example, the main processor 40 acquires the embedded sparse image data from the image signal processor 42 (step S30). That is, the image signal processor 42 has one or more data output ports for outputting various data during processing and one or more data input ports for inputting various data to the image signal processor 42. Therefore, the main processor 40 acquires the embedded sparse image data via one of the data output ports of the image signal processor 42.

[0128] Fig.10 FIG. 2 shows a schematic diagram for explaining a mechanism for generating target image data in this embodiment. Fig.10 As shown, the embedded sparse image data may be acquired from the image signal processor 42, and the embedded sparse image data includes the sparse image data and the segmentation data.

[0129] Incidentally, when the embedded sparse image data is during processing, the embedded sparse image data obtained from the image signal processor 42 may not be the same as the embedded sparse image data input to the image signal processor 42. However, since the segmentation data is stored in the spare space in the sparse image data, it is acceptable for the target image generation process disclosed herein.

[0130] Next, if Figure 5 As shown, for example, the main processor 40 extracts segmentation data from the embedded sparse image data (step S32). In this embodiment, each pixel of the sparse image data includes 4 bits of segmentation data. Fig. 9 The 4 bits of segmentation data per pixel shown in are extracted from the embedded sparse image data.

[0131] Next, if Figure 5 As shown, for example, the main processor 40 combines the first data part and the second data part of the segmented data to obtain compressed data (step S34). As described above, when generating the segmented data, the 8-bit pixel value of the compressed data is divided into a 4-bit first data part and a 4-bit second data part. Therefore, by combining the first data part and the second data part extracted from the same second group, the 8-bit pixel value of the segmented data can be reconstructed. By combining the first data part and the second data part of each pixel of the segmented data, the 8-bit pixel value of the segmented data can be obtained again as shown in FIG. Fig. 9 The compressed data is shown.

[0132] Next, if Figure 5 As shown, for example, the main processor 40 expands the compressed data to reconstruct the residual data (step S36). As described above, the pixels of the 11-bit residual data have been compressed to the 8-bit pixels of the compressed data. Therefore, the compressed data can be expanded to reconstruct the 11-bit pixels of the residual data.

[0133] Fig.11 A schematic diagram for explaining how to reconstruct pixel values ​​of residual data and dense image data is shown. Fig.11 Shown is a reference Figure 7 The reverse process of generating residual data and compressed data is described.

[0134] like Fig.11 As shown, the compressed data can be used Figure 8 The compression curve shown in is expanded to obtain the residual data again. That is, when generating the compressed data, Figure 8 The compression curve shown converts the compressed data from the residual data. Therefore, using Figure 8 The compression curve shown is used to inversely transform the compressed data to obtain the residual data again.

[0135] For example, if the pixel value of the compressed data is 10, the pixel value of the residual data is 10. Also, if the pixel value of the compressed data is 127, the pixel value of the residual data is 1023. However, if the pixel value of the compressed data is 126, the pixel value of the residual data is 850.

[0136] As described above, since the pixel value of the compressed data is represented by 8 bits, and the pixel value of the residual data is represented by 11 bits, if the pixel value of the residual data is large, the reproducibility of the value is not so high. However, if the pixel value of the residual data is small, the reproducibility of the value is high.

[0137] Next, if Figure 5 As shown, for example, the main processor 40 reconstructs the dense image data based on the residual data (step S38). Fig.11 As shown, in order to calculate the value of the green pixel P2, the value of the green pixel P1 of the sparse image data is added to the value of the green pixel P2 of the residual data.

[0138] As described above, the dense image data includes a plurality of first groups, each of which includes a green pixel P1 and a green pixel P2. In addition, each pixel value of the residual data indicates the difference between the value of the green pixel P1 and the value of the green pixel P2 in each first group. The value of the green pixel P1 can be obtained from the sparse image data from the image signal processor 42, and thus the value of the green pixel P2 can be calculated by adding it to the value of the green pixel P2 of the residual data.

[0139] Thereafter, the value of green pixel P1 of the sparse image data and the value of green pixel P2 calculated by adding the residual data and the value of green pixel P1 can be combined to obtain a first group of dense image data. This process is repeated and applied to each first group to regenerate dense image data.

[0140] Next, if Figure 5 As shown, for example, the main processor 40 obtains the generated image data from one of the data output ports of the image signal processor 42 based on the sparse image data (step S40). Fig.10 As shown, image data generated during processing based on sparse image data may be acquired from the image signal processor 42 .

[0141] Next, if Figure 5 As shown, for example, the main processor 40 combines the dense image data reconstructed in step 38 and the generated image data acquired in step 40 to generate combined image data (step S42).

[0142] Fig.12 An example of image data generated based on the sparse image data and dense image data reconstructed in step S38 is shown. Fig.12 As shown, in the image signal processor 42, the generated image data is generated based on the sparse image data. As a result, even if an image of the generated image data is displayed on the display 20, the brightness of the image may be slightly rough, but it is a full-color image. On the contrary, if an image of dense image data is displayed on the display 20, the brightness of the image can be good enough because the color of the dense image data is green, to which the human eye is sensitive to light. Therefore, in the present embodiment, the dense image data is combined with the generated image data based on the sparse image data to generate combined image data.

[0143] Next, if Figure 5 and Fig.10 As shown, for example, the main processor 40 inputs the combined image data to one of the data input ports of the image signal processor 42 (step S44). Thereafter, the image signal processor continues to process the combined image data, and finally outputs the target image data from the image signal processor 42.

[0144] For example, an image to be displayed on the display 20 may be generated based on the target image data. Alternatively, the target image data may be stored in the memory 44. The target image data has a variety of formats. For example, the format of the target image data is JPEG, TIFF, GIF, etc.

[0145] As described above, according to the electronic device 10 of the present embodiment, dense image data can be embedded as segmentation data into the sparse image data input to the image signal processor 42, and then the dense image can be reconstructed based on the segmentation data embedded in the sparse image data. As a result, by combining the image data generated based on the sparse image data and the dense image data reconstructed from the segmentation data in the embedded sparse image data, an image based on the dense image data can be reproduced, and the quality of the target image data can be improved.

[0146] Furthermore, since the format of the embedded sparse image data is the same as that of the sparse image data, a common image signal processor for sparse image data can still be used as the image signal processor 42 for the embedded sparse image data. Therefore, there is no need to newly develop the image signal processor 42 to process the embedded sparse image data of the present embodiment to generate the target image data.

[0147] Incidentally, in the above-described embodiment, although dense image data is generated in green, other colors may be used to generate dense image data. For example, yellow may be used to generate dense image data. In this case, the color filter of the image sensor of the camera assembly 30 is composed of red, yellow, and blue (RYB), the sparse image data is composed of red, yellow, and blue, and the dense image data is composed of yellow.

[0148] In addition, the sparse image data may include more than three colors. For example, the sparse image data may include green pixels, red pixels, blue pixels, and yellow pixels. That is, the sparse image data may include multiple pixels of at least three colors.

[0149] In the description of the embodiments of the present invention, it should be understood that terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "rear", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise" and "counterclockwise" should be interpreted as referring to directions or positions as described or shown in the drawings discussed. These related terms are only used to simplify the description of the present invention and do not indicate or imply that the mentioned devices or elements must have a specific orientation, or be constructed or operated in a specific orientation. Therefore, these terms cannot be constructed to limit the present invention.

[0150] In addition, the use of terms such as "first" and "second" herein is for descriptive purposes and is not intended to indicate or imply relative importance or significance, or to imply the number of technical features indicated. Therefore, features defined using "first" and "second" may include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0151] In the description of the embodiments of the present invention, unless otherwise specified or limited, the terms "installation", "connection", "coupling" and the like are widely used, and may be, for example, a fixed connection, a detachable connection or an integral connection; it may also be a mechanical or electrical connection; it may also be a direct connection or an indirect connection through an intermediate structure; it may also be an internal communication between two elements, which can be understood by those skilled in the art according to the specific circumstances.

[0152] In the embodiments of the present invention, unless otherwise specified or limited, the structure in which the first feature is "above" or "below" the second feature may include an embodiment in which the first feature is in direct contact with the second feature, and may also include an embodiment in which the first feature and the second feature are not in direct contact with each other, but are in contact through an additional feature formed therebetween. In addition, the first feature "located "above", "above" or "top" of the second feature may include an embodiment in which the first feature is located directly "above", "above" or "top" of the second feature, or an embodiment in which it is obliquely "above", "above" or "top", or simply means that the height of the first feature is higher than the second feature; and the first feature "located "below", "below" or "bottom" of the second feature may include an embodiment in which the first feature is located directly "below", "below" or "bottom" of the second feature, or obliquely "below", "below" or "bottom", or simply means that the height of the first feature is lower than the second feature.

[0153] Various embodiments and examples are provided in the above description to realize different structures of the present invention. In order to simplify the present invention, certain elements and settings are described above. However, these elements and settings are only used as examples and are not intended to limit the present invention. In addition, in different examples of the present invention, reference numerals and / or letters of the drawings may be repeated. This repetition is for simplification and clarity, and does not involve the relationship between different embodiments and / or settings. In addition, examples of different processes and materials are provided in the present invention. However, it will be appreciated by those skilled in the art that other processes and / or materials may also be applied.

[0154] References in this specification to "an embodiment," "some embodiments," "an exemplary embodiment," "an example," "a specific example," or "some examples" mean that a particular feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the invention. Thus, the appearance of the above phrases throughout this specification does not necessarily refer to the same embodiment or example of the invention. Furthermore, in one or more embodiments or examples, the particular features, structures, materials, or characteristics may be combined in any suitable manner.

[0155] Any process or method described in a flowchart or otherwise described in the text may be understood to include one or more modules, segments or portions of executable instruction code for implementing specific logical functions or steps in the process, and the scope of the preferred embodiments of the present invention includes other implementations, in which those skilled in the art should understand that the functions may be implemented in an order different from that shown or discussed, including in substantially the same order or in reverse order.

[0156] The logic and / or steps otherwise described herein or shown in a flowchart, for example, a specific sequence list of executable instructions for implementing a logical function, may be specifically implemented in any computer-readable medium to be used by an instruction execution system, apparatus or device (e.g., a computer-based system including a processor or other system capable of obtaining instructions from and executing instructions to an instruction execution system, apparatus or device), or to be used in combination with an instruction execution system, apparatus or device. With respect to this specification, a "computer-readable medium" may be any device suitable for containing, storing, communicating, propagating or transmitting a program to be used by or in combination with an instruction execution system, apparatus or device. More specific examples of computer-readable media include, but are not limited to: an electronic connection with one or more wires (electronic devices), a portable computer housing (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, because, for example, the paper or other suitable medium can be optically scanned and then edited, decrypted or processed using other suitable methods if necessary to electronically obtain the program, which can then be stored in a computer memory.

[0157] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by an appropriate instruction execution system. For example, in another embodiment, if implemented by hardware, the steps or methods can be implemented by one or a combination of the following technologies known in the art: a discrete logic circuit with a logic gate circuit for implementing the logic function of a data signal, a dedicated integrated circuit with an appropriate combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0158] Those skilled in the art will appreciate that all or part of the steps in the above exemplary method of the present invention may be implemented by commanding related hardware with a program. The program may be stored in a computer-readable storage medium, and when running on a computer, the program includes one or a combination of the steps in the method embodiment of the present invention.

[0159] In addition, each functional unit of the embodiment of the present invention can be integrated in the processing module, or these units can be separate physical existences, or two or more units are integrated in the processing module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, the integrated module can be stored in a computer-readable storage medium.

[0160] The above storage medium may be a read-only memory, a disk, a CD, etc.

[0161] Although embodiments of the present invention have been shown and described, those skilled in the art will appreciate that these embodiments are illustrative and are not to be construed as limiting the present invention and that changes, modifications, substitutions and variations may be made in the embodiments without departing from the scope of the present invention.

Claims

1. A method for generating target image data, comprising: Acquire sparse image data and dense image data, wherein the sparse image data includes at least a plurality of pixels among first color pixels, second color pixels, and third color pixels, and the dense image data includes a plurality of pixels among the first color pixels; generating residual data based on a difference between every two adjacent pixels in the dense image data; compressing the residual data to generate compressed data, so as to reduce the data volume of the residual data; Segmenting each pixel of the compressed data to generate segmented data, wherein the segmented data includes a first data portion and a second data portion; and The first data portion and the second data portion of the first group are embedded into a spare space of the second group of the sparse image data to generate embedded sparse image data, wherein every two adjacent pixels in the dense image data constitute a first group, and every two adjacent pixels in the sparse image data constitute a second group, and the position of the first group of the dense image data corresponds to the position of the second group of the sparse image data.

2. The method according to claim 1, wherein: The method further includes inputting the embedded sparse image data to an image signal processor, the image signal processor processing the sparse image data in the embedded sparse image data to generate the target image data.

3. The method according to claim 1, wherein: The first group includes a first value for the first color pixel and a second value for the first color pixel.

4. The method according to claim 3, wherein: The second group includes the third value of the first color pixel and the fourth value of the second color pixel, or includes the third value of the first color pixel and the fourth value of the third color pixel.

5. The method according to claim 4, wherein: The first value of the first color pixel in the first group corresponds to the third value of the first color pixel in the second group.

6. The method according to claim 5, wherein: The generating residual data includes subtracting the second value of the first color pixel from the first value of the first color pixel in the first group.

7. The method according to claim 6, wherein: Generating the compressed data includes reducing the number of bits of the residual data.

8. The method according to claim 7, wherein: Reducing the number of bits of the residual data includes converting the residual data into the compressed data based on a compression curve, the compression curve being used to define a relationship between pixel values ​​of the residual data and pixel values ​​of the compressed data.

9. The method according to claim 8, wherein: Each of the first color pixels, each of the second color pixels, and each of the third color pixels in the sparse image data has a spare space in which the sparse image data is not stored.

10. The method according to claim 9, wherein: When generating the segmentation data, the sizes of the first data portion and the second data portion of the segmentation data match the size of the spare space in each of the first color pixels, each of the second color pixels, and each of the third color pixels in the sparse image data.

11. The method according to claim 2, wherein: The method further comprises: after the embedded sparse image data is input to the image signal processor, obtaining the embedded sparse image data from the image signal processor; extracting the segmented data from the embedded sparse image data acquired by the image signal processor to generate the compressed data; expanding the compressed data generated from the segmented data to reconstruct the residual data; and The dense image data is reconstructed based on the residual data reconstructed from the compressed data.

12. The method according to claim 11, wherein: The method further comprises: acquiring generated image data during processing to generate the target image data based on the sparse image data from the image signal processor; and The generated image data and the dense image data reconstructed from the residual data are combined to generate combined image data.

13. The method according to claim 12, wherein: The method also includes inputting the combined image data to the image signal processor.

14. The method according to any one of claims 1 to 13, wherein: The first color is green, the second color is red, and the third color is blue.

15. The method according to claim 14, wherein: The sparse image data conforms to a Bayer format.

16. An electronic device, comprising: a camera assembly configured to capture an image of a subject and generate sparse image data and dense image data; as well as The main processor is configured as: Acquire the sparse image data and the dense image data, wherein the sparse image data includes at least a plurality of pixels among first color pixels, second color pixels, and third color pixels, and the dense image data includes a plurality of pixels among the first color pixels; generating residual data based on a difference between every two adjacent pixels in the dense image data; compressing the residual data to generate compressed data, so as to reduce the data volume of the residual data; Segmenting each pixel of the compressed data to generate segmented data, wherein the segmented data includes a first data portion and a second data portion; and The first data portion and the second data portion of the first group are embedded into a spare space of the second group of the sparse image data to generate embedded sparse image data, wherein every two adjacent pixels in the dense image data constitute a first group, and every two adjacent pixels in the sparse image data constitute a second group, and the position of the first group of the dense image data corresponds to the position of the second group of the sparse image data.

17. The electronic device according to claim 16, wherein: The main processor is further configured to input the embedded sparse image data to an image signal processor, which processes the sparse image data in the embedded sparse image data to generate target image data.

18. The electronic device according to claim 16, wherein: The first group includes a first value for the first color pixel and a second value for the first color pixel.

19. The electronic device according to claim 18, wherein: The second group includes the third value of the first color pixel and the fourth value of the second color pixel, or includes the third value of the first color pixel and the fourth value of the third color pixel.

20. The electronic device according to claim 19, wherein: The first value of the first color pixel in the first group corresponds to the third value of the first color pixel in the second group.

21. The electronic device according to claim 20, wherein: The second value of the first color pixel is subtracted from the first value of the first color pixel in the first group to generate residual data.

22. The electronic device according to claim 21, wherein: The number of bits of the residual data is reduced to generate the compressed data.

23. The electronic device according to claim 22, wherein: The residual data is converted into the compressed data based on a compression curve to reduce the number of bits of the residual data, wherein the compression curve is used to define a relationship between pixel values ​​of the residual data and pixel values ​​of the compressed data.

24. The electronic device according to claim 23, wherein: Each of the first color pixels, each of the second color pixels, and each of the third color pixels in the sparse image data has a spare space in which the sparse image data is not stored.

25. The electronic device according to claim 24, wherein: When generating the segmented data, the sizes of the first data portion and the second data portion of the segmented data match the size of spare space in each of the first color pixels, each of the second color pixels, and each of the third color pixels in the sparse image data.

26. The electronic device according to claim 17, wherein: The main processor is also configured to: after the embedded sparse image data is input to the image signal processor, obtaining the embedded sparse image data from the image signal processor; extracting the segmented data from the embedded sparse image data acquired by the image signal processor to generate the compressed data; expanding the compressed data generated from the segmented data to reconstruct the residual data; as well as The dense image data is reconstructed based on the residual data reconstructed from the compressed data.

27. The electronic device according to claim 26, wherein: The main processor is also configured to: acquiring generated image data during processing to generate the target image data based on the sparse image data from the image signal processor; as well as The generated image data and the dense image data reconstructed from the residual data are combined to generate combined image data.

28. The electronic device according to claim 27, wherein: The main processor is also configured to input the combined image data to the image signal processor.

29. The electronic device according to any one of claims 16 to 28, wherein: The first color is green, the second color is red, and the third color is blue.

30. The electronic device according to claim 29, wherein: The sparse image data conforms to a Bayer format.

31. A non-transitory computer readable medium comprising program instructions stored thereon, which, when executed by a host processor, are used to perform at least the following operations: Get sparse image data and dense image data, where, The sparse image data includes at least a plurality of pixels among first color pixels, second color pixels, and third color pixels, and the dense image data includes a plurality of pixels among first color pixels; generating residual data based on a difference between every two adjacent pixels in the dense image data; compressing the residual data to generate compressed data, so as to reduce the data volume of the residual data; Segmenting each pixel of the compressed data to generate segmented data, wherein the segmented data includes a first data portion and a second data portion; and The first data portion and the second data portion of the first group are embedded into a spare space of the second group of the sparse image data to generate embedded sparse image data, wherein every two adjacent pixels in the dense image data constitute a first group, and every two adjacent pixels in the sparse image data constitute a second group, and the position of the first group of the dense image data corresponds to the position of the second group of the sparse image data.

Images