Method for generating target image data, electronic device, and non-transitory computer readable medium

Abstract

A method includes obtaining embedded sparse image data for generating target image data from an image signal processor, wherein the embedded sparse image data includes sparse image data, the sparse image data includes pixels, and the pixels include at least first color pixels, second color pixels, and third color pixels; extracting split data including first and second data parts from the sparse image data; merging the first and second data parts to obtain compressed data; inversely transforming the compressed data according to a compression curve to obtain inversely transformed residual data; adding random values ​​within an error range to the inversely transformed residual data to obtain reconstructed residual data; and reconstructing dense image data based on the reconstructed residual data and the sparse image data, wherein the dense image data includes pixels, and the pixels include first color pixels.

Description

Technical Field

[0001] The present application relates to a method for generating target image data, an electronic device for implementing the method, and a non-transitory computer-readable medium including program instructions stored thereon for executing the method. Background Art

[0002] Electronic devices such as smart phones and tablet terminals are widely used in our daily lives. Nowadays, many electronic devices are equipped with camera components to capture images. Some electronic devices are portable and therefore easy to carry. Therefore, users of electronic devices can easily take pictures of objects using the camera components of the electronic devices anytime and anywhere.

[0003] There are various formats for capturing an image of an object and generating its target image data. One widely known format is the Bayer format including sparse image data.

[0004] In addition to the sparse image data, in order to improve the image quality of the object in the target image data, the camera assembly also generates dense image data when capturing an image of the object. In this case, the sparse image data and the dense image data are used to generate the target image data for display on a display or storage in a memory of the electronic device. In order to reduce the development cost of the electronic device, it is preferred to use a general-purpose image signal processor to generate the target image data based on the sparse image data and the dense image data. Summary of the invention

[0005] The present application aims to solve at least one of the above technical problems. Therefore, the present application needs to provide a method for generating target image data, an electronic device implementing the method, and a non-transitory computer-readable medium including program instructions stored thereon for executing the method.

[0006] According to the present application, a method for generating target image data may include:

[0007] Acquire embedded sparse image data for generating target image data from an image signal processor, wherein the embedded sparse image data comprises sparse image data, and the sparse image data comprises a plurality of pixels, the plurality of pixels comprising at least a first color pixel, a second color pixel, and a third color pixel;

[0008] extracting split data including a first data portion and a second data portion from the sparse image data;

[0009] combining the first data portion and the second data portion to obtain compressed data;

[0010] Inversely transform the compressed data according to the compression curve to obtain inversely transformed residual data;

[0011] adding a random value within an error range to the inversely transformed residual data to obtain reconstructed residual data; and

[0012] Dense image data is reconstructed based on the reconstructed residual data and the sparse image data, wherein the dense image data includes a plurality of pixels including first color pixels.

[0013] According to the present application, an electronic device may include:

[0014] an image signal processor configured to generate target image data based on the embedded sparse image data; and

[0015] Main processor, configured as:

[0016] Acquire the embedded sparse image data for generating the target image data from the image signal processor, wherein the embedded sparse image data comprises sparse image data, and the sparse image data comprises a plurality of pixels, the plurality of pixels comprising at least a first color pixel, a second color pixel, and a third color pixel;

[0017] extracting split data including a first data portion and a second data portion from the sparse image data;

[0018] combining the first data portion and the second data portion to obtain compressed data;

[0019] Inversely transform the compressed data according to the compression curve to obtain inversely transformed residual data;

[0020] adding a random value within an error range to the inversely transformed residual data to obtain reconstructed residual data; and

[0021] Dense image data is reconstructed based on the reconstructed residual data and the sparse image data, wherein the dense image data includes a plurality of pixels including first color pixels.

[0022] According to the present application, a non-transitory computer-readable medium may store program instructions, and the program instructions are used to at least perform the following operations:

[0023] Acquire embedded sparse image data for generating target image data from an image signal processor, wherein the embedded sparse image data comprises sparse image data, and the sparse image data comprises a plurality of pixels, the plurality of pixels comprising at least a first color pixel, a second color pixel, and a third color pixel;

[0024] extracting split data including a first data portion and a second data portion from the sparse image data;

[0025] combining the first data portion and the second data portion to obtain compressed data;

[0026] Inversely transform the compressed data according to the compression curve to obtain inversely transformed residual data;

[0027] adding a random value within an error range to the inversely transformed residual data to obtain reconstructed residual data; and

[0028] Dense image data is reconstructed based on the reconstructed residual data and the sparse image data, wherein the dense image data includes a plurality of pixels including first color pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] These and / or other aspects and advantages of the embodiments of the present application will become more apparent and more easily understood from the following description with reference to the accompanying drawings.

[0030] Figure 1 is a plan view of a first side of an electronic device according to an embodiment of the present application;

[0031] Figure 2 is a plan view of a second side of the electronic device according to an embodiment of the present application;

[0032] Figure 3 is a block diagram of an electronic device according to a first embodiment of the present application;

[0033] Figure 4 is a flowchart of a target image generation process performed by an electronic device according to an embodiment of the present application (part 1);

[0034] Figure 5 is a flowchart of a target image generation process performed by an electronic device according to an embodiment of the present application (Part 2);

[0035] Figure 6 is a schematic diagram of a mechanism for generating embedded sparse image data input to an image signal processor in an embodiment of the present application;

[0036] Figure 7 is a schematic diagram of an embodiment of the present application explaining how to calculate original residual data and compressed data based on two adjacent pixels in a first pair in dense image data;

[0037] Figure 8 One of the examples of the compression curve for compressing original residual data to generate compressed data in the embodiment of the present application;

[0038] Fig. 9 is a schematic diagram explaining how to split the original residual data to generate split data and embed the split data into sparse image data in an embodiment of the present application;

[0039] Fig.10 is a schematic diagram explaining how to generate combined image data based on reconstructed dense image data and generated image data in an embodiment of the present application;

[0040] Fig.11 is a schematic diagram explaining how to reconstruct two adjacent pixels in a first pair in dense image data based on compressed data;

[0041] Fig.12 It is a locally enlarged compression curve graph when the difference between two adjacent pixels in the first pair is relatively small;

[0042] Fig.13 is a locally enlarged compression curve graph when the difference between two adjacent pixels in the first pair is large; and

[0043] Fig.14 Schematic diagram of a mechanism for generating target image data based on embedded sparse image data in an embodiment of the present application. DETAILED DESCRIPTION

[0044] Embodiments of the present application will be described in detail, and examples of the embodiments will be shown in the accompanying drawings. Throughout the description, the same or similar elements and elements having the same or similar functions are represented by the same reference numerals. The embodiments described herein in conjunction with the accompanying drawings are illustrative and intended to illustrate the present disclosure, but should not be construed as limiting the present disclosure.

[0045] (First embodiment)

[0046] Figure 1 is a plan view of a first side of an electronic device 10 according to a first embodiment of the present application, Figure 2 1 is a plan view of the second side of the electronic device 10 of the first embodiment of the present application. The first side may be referred to as the rear side of the electronic device 10, and the second side may be referred to as the front side of the electronic device 10.

[0047] like Figure 1 and Figure 2 As shown, the electronic device 10 may include a display 20 and a camera assembly 30. In this embodiment, the camera assembly 30 includes a first main camera 32, a second main camera 34, and a secondary camera 36. The first main camera 32 and the second main camera 34 can capture images of a first side of the electronic device 10, and the secondary camera 36 can capture images of a second side of the electronic device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called out-cameras, and the secondary camera 36 is a so-called in-camera. For example, the electronic device 10 may be a mobile phone, a tablet computer, a personal digital assistant, etc.

[0048] Each of the first main camera 32, the second main camera 34, and the sub-camera 36 has an image sensor that converts light that has passed through the color filter into an electrical signal, the signal value of which depends on the amount of light that has passed through the color filter.

[0049] Although the electronic device 10 according to the present embodiment has three cameras, the electronic device 10 may have less than three or more than three cameras. For example, the electronic device 10 may have two, four, five, etc. cameras.

[0050] Figure 3 FIG. 1 is a block diagram of the electronic device 10 of the present embodiment. Figure 3 As shown, in addition to the display 20 and the camera assembly 30, the electronic device 10 may further include a main processor 40, an image signal processor 42, a memory 44, a power circuit 46, and a communication circuit 48. The display 20, the camera assembly 30, the main processor 40, the image signal processor 42, the memory 44, the power circuit 46, and the communication circuit 48 are interconnected via a bus 50.

[0051] The main processor 40 executes one or more program instructions stored in the memory 44. The main processor 40 implements various applications and data processing of the electronic device 10 by executing the program instructions. The main processor 40 can be one or more computer processors. The main processor 40 is not limited to one CPU core, and can also have multiple CPU cores. The main processor 40 can be the main CPU of the electronic device 10, an image processing unit (IPU) or a DSP provided with a camera assembly 30.

[0052] The image signal processor 42 controls the camera assembly 30 and processes various image data captured by the camera assembly 30 to generate target image data. For example, the image signal processor 42 may apply demosaic processing, noise reduction processing, automatic exposure processing, automatic focus processing, automatic white balance processing, high dynamic range processing, etc. to the image data captured by the camera assembly 30.

[0053] In the present embodiment, the main processor 40 and the image signal processor 42 cooperate with each other to generate target image data of the object captured by the camera assembly 30. That is, the main processor 40 and the image signal processor 42 are configured to capture an image of the object through the camera assembly 30 and perform various image processing on the captured image data.

[0054] The memory 44 stores program instructions and various data to be executed by the main processor 40. For example, data of captured images is also stored in the memory 44.

[0055] The memory 44 may include a high-speed RAM memory and / or a non-volatile memory such as a flash memory and a disk memory. That is, the memory 44 may include a non-transitory computer-readable medium in which program instructions are stored.

[0056] The power circuit 46 may have a battery such as a lithium-ion rechargeable battery and a battery management unit (BMU) for managing the battery.

[0057] The communication circuit 48 is used to receive and send data to communicate with a base station of a telecommunication network system, the Internet or other devices through wireless communication. The wireless communication may adopt any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), LTE-Advanced, 5G. The communication circuit 48 may include an antenna and a radio frequency circuit.

[0058] like Figure 4 and Figure 5 1 is a flowchart of the target image generation processing of the electronic device 10 of the present embodiment. In the present embodiment, for example, the target image generation processing is performed by the main processor 40 to generate target image data. However, the main processor 40 can also cooperate with the image signal processor 42 to generate the target image data. Therefore, the main processor 40 and the image signal processor 42 constitute the image processor in the present embodiment.

[0059] In addition, in the present embodiment, the program instructions of the target image generation process are stored in a non-transitory computer-readable medium of the memory 44. Therefore, for example, when the program instructions are read from the memory 44 and executed in the main processor 40, the main processor 40 executes Figure 4 and Figure 5 The target image generation process shown.

[0060] Reference Figure 4 , for example, the main processor 40 obtains sparse image data and dense image data (step S10). In the present embodiment, the main processor 40 obtains the sparse image data and dense image data from the camera assembly 30. That is, the camera assembly 30 captures an image of an object and generates sparse image data and dense image data. In the present embodiment, the sparse image data includes a plurality of green pixels, a plurality of red pixels, and a plurality of blue pixels. On the other hand, the dense image data includes a plurality of green pixels.

[0061] In order to generate sparse image data and dense image data using the camera assembly 30, the camera assembly 30 may have a dedicated image sensor to capture an image of an object and generate sparse image data and dense image data by performing a single imaging operation using a single camera. In this case, for example, the first main camera 32 may capture an image of an object and generate sparse image data and dense image data by performing a single imaging operation.

[0062] On the other hand, the camera assembly 30 can use two cameras to capture an image of the object and generate sparse image data and dense image data by performing a single imaging operation. In this case, for example, the first main camera 32 captures an image of the object and generates sparse image data, while the second main camera 34 captures an image of the object and generates dense image data.

[0063] On the other hand, the camera assembly 30 can use a single camera to capture an image of the object by performing two imaging operations and generate sparse image data and dense image data. For example, the sub-camera 36 captures an image of the object by performing a first imaging operation to generate sparse image data, and then the sub-camera 36 captures an image of the object by performing a second imaging operation immediately after the first imaging operation to generate dense image data.

[0064] Figure 6 is a schematic diagram explaining a mechanism of generating embedded sparse image data to be input to the image signal processor 42.

[0065] like Figure 6 As shown, the sparse image data conforms to the Bayer format. Therefore, the arrangement of the green, red, and blue color filters of the image sensor of the camera assembly 30 for capturing the subject image conforms to the Bayer arrangement. In the Bayer format, the number of green pixels G1 is twice the number of red pixels R1 or the number of blue pixels B1 in the sparse image data. The sparse image data may also be referred to as raw data (RAW data) from the camera assembly 30.

[0066] The dense image data consists of green pixels G2. This is because the human eye is more sensitive to the brightness of green than to the brightness of red or blue. In this embodiment, a dense image is captured to adjust the brightness of the target image data.

[0067] Next, if Figure 4 As shown, for example, the main processor 40 selects two green pixels G2 in the dense image data and generates the original residual data of the selected two green pixels G2 (step S12). That is, in this embodiment, in order to reduce the amount of data, the original residual data is generated by calculating the difference between two adjacent pixels in the dense image data.

[0068] Figure 7 is a diagram explaining how to generate the original residual data, compressed data, and split data. Figure 6 and Figure 7 As shown, the green pixel G1 in the sparse image data corresponds to the green pixel G2a in the dense image data. That is, the value of the green pixel G1 at the same position is substantially equal to the value of the green pixel G2a.

[0069] On the other hand, the green pixel G2b is included in the dense image data but not in the sparse image data. That is, the sparse image data does not include a green pixel corresponding to the green pixel G2b in the dense image data. Figure 7 In the example, a green pixel G2a and a green pixel G2b adjacent to each other constituting a first pair are shown. In other words, for example, the main processor 40 selects one of the first pairs in the dense image data in step S12, and repeats steps S12 to S20 described later until all the first pairs are processed.

[0070] Generally speaking, the brightness of two adjacent pixels is similar or the same. That is, the difference between the value of the green pixel G2a and the value of the green pixel G2b adjacent to the green pixel G2a is usually small. Therefore, in this embodiment, in order to reduce the amount of data, the difference between the value of the green pixel G2a and the value of the green pixel G2b adjacent to the green pixel G2a is obtained by subtracting the value of the green pixel G2b from the value of the green pixel G2a adjacent to the green pixel G2b.

[0071] In other words, two adjacent pixels G2a and G2b in the dense image data form a first pair to generate the original residual data. Figure 6 In the example of sparse image data in , there are 12 (3×4) first pairs used to calculate the original residual data. In other words, the number of first pairs is half the number of green pixels G2 in the dense image data.

[0072] In addition, for example, in the present embodiment, one green pixel G2 is composed of 10 bits. That is, the value of one green pixel G2 of the dense image data is between 0 and 1023. Since the value of the original residual data becomes between +1023 and -1024, the original residual data is composed of 11 bits.

[0073] Next, refer to Figure 4 For example, the main processor 40 generates compressed data based on the original residual data (step S14). There are many ways to compress the original residual data to reduce the number of bits of the original residual data. Here, an example of a way to compress the original residual data will be explained.

[0074] Figure 8is one of the examples of a compression curve for compressing the original residual data to generate compressed data. That is, the original residual data is converted into compressed data according to the compression curve. The compression curve is also called a tone curve to compress various data and defines the relationship between the value of the original residual data and the value of the compressed data. The value of the bit of the compressed data is less than the value of the bit of the original residual data.

[0075] For example, Figure 7 and Figure 8 As shown, in this embodiment, the original residual data consisting of 11 bits is compressed into compressed data consisting of 8 bits. That is, the value of the original residual data is between -1024 and +1023, while the value of the compressed data is between -128 and +127.

[0076] As described above, since the difference between the values ​​of two adjacent green pixels G2a and G2b of the dense image data is generally small, the compression curve increases and decreases substantially linearly in a range where the absolute value of the original residual data is small. On the other hand, the compression curve is substantially flat or constant in a range where the absolute value of the original residual data is large. As a result, the compression curve is S-shaped.

[0077] Based on Figure 8 The compression curve shown compresses the original residual data, and the original residual data consisting of 11 bits can be compressed into compressed data consisting of 8 bits. For example, if the value of the original residual data is 10, the value of the compressed data is also 10. Therefore, if the value of the original residual data is expanded according to the compression curve, the value of the residual data can be returned to 10. That is, within the range where the absolute value of the original residual data is small, the compressed data can be returned to residual data that is substantially the same as the original residual data.

[0078] On the other hand, if the value of the original residual data is 1023, the value of the compressed data is 127. In addition, if the value of the original residual data is 980, the value of the compressed data is 126. That is, in the range where the absolute value of the original residual data is large, the compressed data cannot be restored to the same residual data as the original residual data. In other words, when the absolute value of the original residual data is large, the extended value of the residual data based on the compressed data is rough. However, since the possibility that the absolute value of the original residual data is large is lower than the possibility that the absolute value of the original residual data is small, the low reproducibility of the value range where the absolute value of the original residual data is large is acceptable.

[0079] Next, if Figure 4 As shown, the main processor 40 generates split data based on the compressed data (step S16). That is, since the pixel size of the compressed data is too large to be embedded in the sparse image data, in this embodiment, the compressed data is split into two parts, namely the first data part and the second data part.

[0080] Fig. 9 is a diagram explaining how to generate split data from compressed data. Fig. 9 The upper part shows a comparative example of the related art, Fig. 9 The lower part of FIG. 1 shows a description of this embodiment.

[0081] like Fig. 9 As shown, the value of the compressed data is represented by 8 bits and is divided into a first data part of 4 bits and a second data part of 4 bits.

[0082] In the present embodiment, although the available space for each pixel of the corresponding sparse image data in the image signal processor 42 consists of 14 bits, each pixel of the sparse image data requires 10 bits. Therefore, 4 bits of the 14 bits are reserved bits and are not used in the image signal processor 42. That is, the space of 4 bits of the 14 bits is an empty space in which the sparse image data is not stored.

[0083] Therefore, in this embodiment, in order to be able to insert the compressed data consisting of 8 bits into the 4 reserved bits of the sparse image data, the 8-bit compressed data is divided into two 4-bit parts as split data. Therefore, the size of the first data part and the size of the second data part match the size of the free space of the sparse image data. Therefore, in this embodiment, the 8-bit compressed data is divided into a 4-bit first data part and a 4-bit second data part.

[0084] Next, if Figure 4 As shown, for example, the main processor 40 embeds the split data into the sparse image data to generate embedded sparse image data (step S18). Fig. 9 and Figure 6 As shown, the split data is embedded into two groups of 4 reserved bits of the sparse image data.

[0085] More specifically, in the present embodiment, each red pixel R1, each blue pixel B1, and each green pixel G1 of the sparse image data has 4 reserved bits as free space of the image signal processor 42. Hereinafter, the sparse image data in which the split data is embedded is also referred to as embedded sparse image data.

[0086] like Figure 6 As shown, in the sparse image data of the Bayer format, the red pixel R1 and the green pixel G1 can form a second pair, and the blue pixel B1 and the green pixel G1 can also form a second pair. In the Bayer format, each pair of two adjacent pixels includes a green pixel G1 and a red pixel R1, or includes a green pixel G1 and a blue pixel B1. Figure 6In the example of the sparse image data in , the sparse image data includes 12 (3×4) second pairs, each pair consisting of a green pixel G1 and a red pixel R1, or a green pixel G1 and a blue pixel B1.

[0087] The green pixel G1 in the second pair corresponds to the green pixel G2 in the first pair located at a position corresponding to the first pair in the dense image data. That is, when the position of the second pair of the sparse image data is the same as the position of the first pair of the dense image data, the value of the green pixel G1 of the second pair in the sparse image data is substantially the same as the value of the green pixel G2 of the first pair in the dense image data.

[0088] In this embodiment, the first data portion of the split data is embedded into the 4-bit free space in the second pair of green pixels G1, and the second data portion of the split data is embedded into the 4-bit free space in the second pair of red pixels R1 or blue pixels B1.

[0089] That is, the first data portion and the second data portion of the split data are embedded into the two adjacent red pixels R1 and green pixels G1 of the second pair or the two adjacent blue pixels B1 and green pixels G1 of the second pair. Therefore, in this embodiment, the first data portion and the second data portion of the split data can both be embedded into the free space of the sparse image data.

[0090] Furthermore, in the present embodiment, the first pair of dense image data has a one-to-one correspondence with the second pair of sparse image data. Therefore, the first data portion and the second data portion are embedded into two adjacent pixels of the second pair, and the two adjacent pixels of the second pair correspond to the first pair where the first data portion and the second data portion were originally calculated. That is, the first data portion and the second data portion are inserted into the second pair at the position corresponding to the original first pair.

[0091] However, the split data may be embedded into the free space of the sparse image data in any manner that allows specification of the embedding positions of the first data portion and the second data portion of the split data in the sparse image data.

[0092] exist Fig. 9 In the comparative example shown in the upper part of FIG. 4 , the information of the green pixel G2b is discarded when the sparse image data is input to the image signal processor 42. However, in the present embodiment, the data of the green pixel G2b can also be embedded in the sparse image data, so the information of the green pixel G2b is not discarded.

[0093] Next, if Figure 4As shown, for example, the main processor 40 determines whether all first pairs in the dense image data have been processed and completed (step S20). That is, in this embodiment, each original residual data based on the first pair should be embedded in the sparse image data.

[0094] If all the first pairs in the dense image data have not been processed and completed (step S20: No), the target image generation process returns to step 12 and repeats the process from step S12 to step S18. That is, the process from step S12 to step S18 is repeated until all the original residual data of the first pairs are embedded in the sparse image data.

[0095] On the other hand, for example, if all the first pairs in the dense image data have been processed and completed (step S20: Yes), the main processor 40 inputs the embedded sparse image data to the image signal processor 42 (step S22). That is, the embedded sparse image data including the sparse image data and the split data is input to the image signal processor 42 to generate the target image data. Thereafter, the image signal processor 42 starts processing the sparse image data in the embedded sparse image data to generate the target image data.

[0096] Next, if Figure 5 As shown, for example, the main processor 40 obtains the embedded sparse image data from the image signal processor 42 (step S30). That is, the image signal processor 42 has one or more data output ports for outputting various data during processing, and one or more data input ports for inputting various data into the image signal processor 42. Therefore, the main processor 40 obtains the embedded sparse image data through one data output port of the image signal processor 42.

[0097] Fig.10 FIG. 2 is a schematic diagram illustrating the mechanism of generating target image data in this embodiment. Fig.10 As shown, embedded sparse image data may be obtained from the image signal processor 42, the embedded sparse image data including the sparse image data and the segmentation data.

[0098] Incidentally, while the embedded sparse image data is being processed, the embedded sparse image data obtained from the image signal processor 42 may be different from the embedded sparse image data input to the image signal processor 42. However, since the split data is stored in the free space in the sparse image data and is not destroyed, the target image generation process disclosed herein is acceptable.

[0099] Next, if Figure 5As shown, for example, the main processor 40 extracts the split data from the embedded sparse image data (step S32). That is, the main processor 40 extracts one split data of the second pair in the sparse image data. In the present embodiment, each pixel of the sparse image data includes a first data portion of 4 bits of split data or a second data portion of 4 bits of split data. Therefore, the split data including the first data portion and the second data portion are extracted from the embedded sparse image data. In the present embodiment, for example, the main processor 40 extracts the split data from the second pair in the sparse image data in step S32, and repeats step S32 to step S40 mentioned below until all the split data in the second pair are processed.

[0100] Next, if Figure 5 As shown, for example, the main processor 40 merges the first data part and the second data part of the split data together to obtain compressed data (step S34). As mentioned above, when generating the split data, the 8-bit compressed data has been split into a 4-bit first data part and a 4-bit second data part. Therefore, the compressed data can be reconstructed by merging the first data part and the second data part that have been extracted from the same second pair. By merging the first data part and the second data part in the same second pair, the compressed data can be obtained again. Fig. 9 The compressed data is shown.

[0101] Next, if Figure 5 As shown, for example, the main processor 40 performs an inverse conversion on the compressed data to obtain inversely converted residual data (step S36). As mentioned above, in step S14, the 11-bit original residual data has been compressed into 8-bit compressed data according to the compression curve. Therefore, the inversely converted residual data consisting of 11 bits can be obtained from the compressed data consisting of 8 bits.

[0102] Fig.11 is a diagram explaining how to reconstruct dense image data from split data. That is, Fig.11 Shown with reference Figure 7 The process of generating the original residual data and the compressed data is reversed.

[0103] like Fig.11 As shown, you can use Figure 8 The compression curve shown in FIG. 1 is used to inversely transform the compressed data to obtain the inversely transformed residual data. That is, when generating the compressed data, the compressed data has been generated by using Figure 8 The compression curve shown is transformed from the original residual data. Therefore, by inversely transforming the compressed data, the inversely transformed residual data is the same as or close to the original residual data.

[0104] Next, if Figure 5As shown, for example, the main processor 40 adds a random value within the error range to the inversely transformed residual data to obtain reconstructed residual data (step S38). Since the inversely transformed residual data contains a certain amount of error relative to the original residual data, the main processor adds a random value within the error range to the inversely transformed residual data to hide the error relative to the original residual data.

[0105] Fig.12 It is a locally enlarged compressed graph where the difference between the green pixels G2a and G2b in the first pair is relatively small. Fig.13 A locally enlarged compressed graph is shown where the difference between the green pixels G2a and G2b in the first pair is relatively large.

[0106] like Fig.12 As shown, if the value of the compressed data is 10, the value of the inversely transformed residual data is also 10. That is, the inversely transformed residual data can be accurately obtained, and the inversely transformed residual data is equal to the original residual data.

[0107] On the other hand, Fig.13 As shown in the figure, if the compressed data is 110, the value of the inverse-converted residual data is 810. However, the original residual data may be 810, 811, 812, or 813. That is, when the original residual data is compressed, if the original residual data is 810, 811, 812, and 813, the compressed data becomes 110. Therefore, it is impossible to obtain exactly the same original residual data by inverse-converting the compressed data. It will cause aliasing and lines to appear in the target image data.

[0108] Therefore, in this embodiment, for example, the main processor 40 adds a random value within the error range to the inverse-converted residual data. Fig.13 As shown, the main processor 40 adds 0, 1, 2, or 3 to 810. In this case, the values ​​of 0, 1, 2, and 3 are random values ​​within the error range, and 810 is the inverse-converted residual data. That is, the random value within the error range is one of the difference values ​​between the value that the original residual data can take in the compression curve and the value of the inverse-converted residual data.

[0109] according to Figure 8 For the compression curve in , the random value of the error range depends on the value of the residual data. If the residual data is small, the error range is narrow. For example, if the compression data is 10, the random value of the error range can only be 0, because the residual data of the inverse transformation can only be 10.

[0110] On the other hand, if the residual data is large, the error range is wide. For example, if the compressed data is 110, the random value of the error range is 0, 1, 2, or 3, because the inverse-transformed residual data is 810, and the original residual data may be 810, 811, 812, or 813.

[0111] In this embodiment, if the compressed data is greater than 110, the error range is greater than 4.

[0112] The original residual data is unknown, therefore, in this embodiment, a random value within the error range is added to the inversely transformed residual data to obtain the reconstructed residual data. In other words, the reconstructed residual data can be randomly dispersed in the possible original residual data.

[0113] Next, if Figure 5 As shown, for example, the main processor 40 reconstructs the dense image data based on the reconstructed residual data (step S40). Fig.11 As shown, to calculate the value of the green pixel G2b, the reconstructed residual data is added to the value of the green pixel G2a.

[0114] As described above, the dense image data includes a plurality of first pairs, each of which includes a green pixel G2a and a green pixel G2b. In addition, the value of the residual data represents the difference between the value of the green pixel G2a in the first pair and the value of the green pixel G2b. The value of the green pixel G2a can be obtained from the embedded sparse image data from the image signal processor 42, because the value of the green pixel G2a is substantially the same as the value of the green pixel G1 in the sparse image data. Therefore, the value of the green pixel G2b can be calculated by adding the value of the residual data to the value of the green pixel G2a.

[0115] Thereafter, a first pair of dense image data may be obtained by merging the value of the green pixel G1 of the sparse image data and the value of the green pixel G2b calculated by adding the value of the residual data to the value of the green pixel G2a.

[0116] Next, if Figure 5 As shown, for example, the main processor 40 determines whether all split data in the embedded sparse image data have been processed and completed (step S42). That is, in this embodiment, each split data should be extracted from the embedded sparse image data and processed to reconstruct all first pairs of green pixels G2b in the dense image data.

[0117] If the split data in the embedded sparse image data has not been processed completely (step S42: No), the target image generation process returns to step 32 and repeats the process from step S32 to step S40. That is, the process from step S32 to step S40 is repeated until all the first pairs of green pixels G2b in the dense image data are reconstructed.

[0118] On the other hand, if all the split data in the embedded sparse image data have been processed (step S42: yes), it means that the dense image data consisting of the first pair of green pixel G2a and green pixel G2b has been reconstructed. Therefore, for example, the main processor 40 obtains the generated image data based on the sparse image data from a data output port of the image signal processor 42 (step S44). Fig.10 As shown, generated image data can be obtained from the image signal processor 42 as a result of processing starting with sparse image data.

[0119] Next, if Figure 5 As shown, for example, the main processor 40 combines the reconstructed dense image data with the generated image data obtained in step 44 to generate combined image data (step S46).

[0120] Fig.14 is an example of generating image data and reconstructing dense image data based on sparse image data in step S46. Fig.14 As shown, generated image data is generated based on the sparse image data in the image signal processor 42. Therefore, even if the image of the generated image data is displayed on the display 20, the brightness of the image may be slightly rough, but it is a full-color image. On the contrary, if the image of the dense image data is displayed on the display 20, the brightness of the image can be satisfactory because the color of the dense image data is green, which is a photosensitive color of the human eye. Therefore, in the present embodiment, the dense image data is synthesized with the generated image data based on the sparse image data to generate combined image data.

[0121] Next, if Figure 5 and Fig.10 As shown, for example, the main processor 40 inputs the combined image data to a data input port of the image signal processor 42 (step S48). Thereafter, the image signal processor 42 continues to process the combined image data, and finally outputs the target image data from the image signal processor 42.

[0122] For example, an image to be displayed on the display 20 may be generated based on the target image data. Alternatively, the target image data may be stored in the memory 44. The target image data has a variety of formats. For example, the target image data may be formatted as JPEG, TIFF, GIF, etc.

[0123] As described above, according to the electronic device 10 of this embodiment, the reconstructed residual data is obtained by adding the random value within the error range to the inversely transformed residual data. Therefore, the reconstructed residual data can be randomly dispersed in the possible original residual data, thereby hiding the aliasing in the target image data.

[0124] Incidentally, in the above-described embodiment, although the compressed data is split into the first data portion and the second data portion, the compressed data need not necessarily be split if the free space in the sparse image data is sufficient to accommodate the non-split compressed data. In this case, the compressed data can be directly embedded in the sparse image data, and thus the compressed data can be directly extracted from the embedded sparse image data obtained by the image signal processor 42.

[0125] In addition, in the above-described embodiment, although dense image data is generated in green, another color may be used to generate dense image data. For example, yellow may be used to generate dense image data. In this case, the color filter of the image sensor of the camera assembly 30 is composed of red, yellow, and blue (RYB), the sparse image data is composed of red, yellow, and blue, and the dense image data is composed of yellow.

[0126] In addition, the sparse image data may include more than three colors. For example, the sparse image data may include green pixels, red pixels, blue pixels, and yellow pixels. That is, the sparse image data may include multiple pixels of at least three colors.

[0127] In the description of the embodiments of the present application, it should be understood that terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise" and "counterclockwise" should be interpreted as referring to the direction or position shown in the described or discussed figure. These relative terms are only used to simplify the description of the present application and do not indicate or imply that the mentioned device or element must have a specific direction, or be constructed or operated in a specific direction. Therefore, these terms cannot be interpreted as limiting the present disclosure.

[0128] In addition, terms such as "first" and "second" are used herein for descriptive purposes and are not intended to indicate or imply relative importance or significance or to imply the number of technical features indicated. Therefore, features defined using "first" and "second" may include one or more of such features. In the description of this application, "plurality" means two or more, unless otherwise specified.

[0129] In the description of the embodiments of the present application, unless otherwise specified or limited, the terms "installed", "connected", "coupled" and the like are used in a broad sense, for example, it may be a fixed connection, a detachable connection or an integral connection; it may be a mechanical connection or an electrical connection; it may be a direct connection or an indirect connection through an intermediate structure; or two elements may be internally connected. Those skilled in the art may understand this according to the specific circumstances.

[0130] In the embodiments of the present application, unless otherwise specified or limited, the first feature being "above" or "below" the second feature may include an embodiment in which the first feature is in direct contact with the second feature; it may also include an embodiment in which the first feature and the second feature are not in direct contact with each other, but are in contact through an additional feature formed between them. In addition, the first feature being "above", "above" or "on top" of the second feature may include an embodiment in which the first feature is directly above or above the second feature, or simply indicates that the height of the first feature is higher than the height of the second feature; and the first feature being "below", "below" or "at the bottom" of the second feature may include an embodiment in which the first feature is directly below or below the second feature, or simply indicates that the height of the first feature is lower than the height of the second feature.

[0131] Various embodiments and examples are provided in the above description to realize different structures of the present application. In order to simplify the present application, certain elements and settings are described above. However, these elements and settings are only used as examples and are not intended to limit the present application. In addition, reference numerals and / or reference letters may be repeated in different examples of the present application. This repetition is for the purpose of simplification and clarity, rather than referring to the relationship between different embodiments and / or settings. In addition, examples of different processes and materials are provided in the present application. However, it will be appreciated by those skilled in the art that other processes and / or materials may also be applied.

[0132] References throughout the specification to "an embodiment," "some embodiments," "an exemplary embodiment," "an example," "a specific example," or "some examples" mean that a particular feature, structure, material, or characteristic described herein in conjunction with the embodiment or example is included in at least one embodiment or example of the present application. Therefore, the appearance of the above phrases throughout the specification does not necessarily refer to the same embodiment or example of the present application. Furthermore, the particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0133] Any process or method described in a flowchart or otherwise described herein may be understood to include one or more modules, code segments or portions of executable instructions for implementing specific logical functions or steps in the process, and the scope of the preferred embodiments of the present application includes other implementations. Those skilled in the art should understand that the functions may be implemented in an order different from that shown or discussed, including in substantially the same order or in reverse order.

[0134] The logic and / or steps otherwise described herein or shown in the flowchart, such as a specific sequence list of executable instructions for implementing the logic function, can be specifically implemented in any computer-readable medium used by an instruction execution system, device or apparatus (such as a computer-based system that includes a processor or other system capable of obtaining instructions from the instruction execution system, device and apparatus and executing the instructions), or used in combination with the instruction execution system, device and apparatus. For the purpose of this specification, "computer-readable medium" can be any device suitable for containing, storing, communicating, propagating or transmitting a program for use by an instruction execution system, device or apparatus, or used in combination with the instruction execution system, device or apparatus. More specific examples of computer-readable media include, but are not limited to: an electronic connector with one or more wires (electronic device), a portable computer housing (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, because, for example, the paper or other suitable medium can be optically scanned and then edited, decrypted or processed by other suitable methods if necessary to electronically obtain the program, and then the program can be stored in the computer memory.

[0135] It should be understood that various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by an appropriate instruction execution system. For example, if it is implemented by hardware, in another embodiment, these steps or methods can be implemented by one or a combination of the following technologies known in the art: a discrete logic circuit with a logic gate circuit for implementing the logic function of a data signal, a dedicated integrated circuit with an appropriate combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0136] Those skilled in the art should understand that all or part of the steps in the above-mentioned exemplary method of the present application can be implemented by program instruction related hardware. The program can be stored in a computer-readable storage medium, and when the program is run on a computer, the program includes a combination of one or more steps in the method embodiment of the present application.

[0137] In addition, the various functional units of the embodiments of the present application can be integrated in the processing module, or these units can exist physically separately, or two or more units can be integrated in the processing module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, the integrated module can be stored in a computer-readable storage medium.

[0138] The above storage medium may be a read-only memory, a magnetic disk, an optical disk, etc.

[0139] Although the embodiments of the present application have been shown and described, those skilled in the art should understand that these embodiments are illustrative and cannot be construed as limiting the present application, and that the embodiments of the present application may be changed, modified, substituted and deformed without departing from the scope of the present application.

Claims

1. A method for generating target image data, characterized in that: include: Acquire embedded sparse image data from an image signal processor, wherein the embedded sparse image data comprises sparse image data, and the sparse image data comprises a plurality of pixels, the plurality of pixels comprising at least a first color pixel, a second color pixel, and a third color pixel; extracting split data including a first data portion and a second data portion from the sparse image data; combining the first data portion and the second data portion to obtain compressed data; Inversely transform the compressed data according to the compression curve to obtain inversely transformed residual data; adding a random value within an error range to the inversely transformed residual data to obtain reconstructed residual data; reconstructing dense image data based on the reconstructed residual data and the sparse image data, wherein the dense image data includes a plurality of pixels, and the plurality of pixels includes the first color pixels; acquiring generated image data in a process of generating target image data based on the sparse image data from the image signal processor; combining the generated image data and the dense image data reconstructed from the reconstructed residual data to generate combined image data; and The combined image data is input to the image signal processor to generate the target image data.

2. The method according to claim 1, characterized in that The random value within the error range is one of the difference values ​​between the value taken by the original residual data in the compression curve and the value of the inverse-converted residual data.

3. The method according to claim 1, characterized in that Two adjacent pixels in the dense image data form a first pair, and the first pair includes a first value of the first color pixel and a second value of the first color pixel.

4. The method according to claim 3, characterized in that The reconstructing the dense image data comprises: obtaining a first value of the first color pixel in the first pair from the sparse image data; The reconstructed residual data is added to the first value of the first color pixel to obtain a second value of the first color pixel.

5. The method according to claim 1, characterized in that: The compression curve is substantially S-shaped, wherein the compression ratio of larger residual data is greater than the compression ratio of smaller residual data.

6. The method according to claim 3, characterized in that Further including: Acquire the sparse image data and the dense image data from a camera assembly capturing an object; generating raw residual data based on the dense image data by calculating a difference between a first value of the first color pixel in the first pair in the dense image data and a second value of the first color pixel; Compressing the original residual data according to the compression curve to generate the compressed data, so as to reduce the data volume of the original residual data; generating the split data by splitting the compressed data into the first data portion and the second data portion; and The split data is embedded into the sparse image data to generate the embedded sparse image data.

7. The method according to claim 6, characterized in that Two adjacent pixels in the sparse image data form a second pair, and the first data portion and the second data portion are embedded in two adjacent pixels of the second pair located at positions corresponding to the first pair of the dense image data.

8. The method according to claim 6, characterized in that Further including: The embedded sparse image data is input to the image signal processor.

9. The method according to any one of claims 1 to 8, characterized in that The first color is green, the second color is red, and the third color is blue.

10. The method according to claim 9, characterized in that The sparse image data conforms to a Bayer format.

11. An electronic device, characterized in that: include: an image signal processor configured to generate target image data based on the embedded sparse image data; and Main processor, configured as: Retrieving the embedded sparse image data from the image signal processor, wherein the embedded sparse image data comprises sparse image data, and the sparse image data comprises a plurality of pixels, the plurality of pixels comprising at least a first color pixel, a second color pixel, and a third color pixel; extracting split data including a first data portion and a second data portion from the sparse image data; combining the first data portion and the second data portion to obtain compressed data; Inversely transform the compressed data according to the compression curve to obtain inversely transformed residual data; adding a random value within an error range to the inversely transformed residual data to obtain reconstructed residual data; reconstructing dense image data based on the reconstructed residual data and the sparse image data, wherein the dense image data includes a plurality of pixels, and the plurality of pixels includes first color pixels; acquiring generated image data in the process of generating the target image data based on the sparse image data from the image signal processor; combining the generated image data and the dense image data reconstructed from the reconstructed residual data to generate combined image data; and The combined image data is input to the image signal processor to generate the target image data.

12. A non-transitory computer readable medium having program instructions stored thereon, the program instructions causing a computer to at least perform the following operations: Get embedded sparse image data from an image signal processor, where The embedded sparse image data includes sparse image data, and the sparse image data includes a plurality of pixels, the plurality of pixels including at least a first color pixel, a second color pixel, and a third color pixel; extracting split data including a first data portion and a second data portion from the sparse image data; combining the first data portion and the second data portion to obtain compressed data; Inversely transform the compressed data according to the compression curve to obtain inversely transformed residual data; adding a random value within an error range to the inversely transformed residual data to obtain reconstructed residual data; reconstructing dense image data based on the reconstructed residual data and the sparse image data, wherein the dense image data includes a plurality of pixels, and the plurality of pixels includes first color pixels; acquiring generated image data in a process of generating target image data based on the sparse image data from the image signal processor; combining the generated image data and the dense image data reconstructed from the reconstructed residual data to generate combined image data; and The combined image data is input to the image signal processor to generate the target image data.

Images