Sensor, electronic device, and non-transitory computer readable medium

Abstract

An imaging sensor (60) includes a first circuit (62) and a second circuit (64). The first circuit (62) acquires first pixel data and outputs the first pixel data from the imaging sensor (60) as conventionally processed image data, wherein the first pixel data is data of conventional gain pixels. The second circuit (64) acquires second pixel data and performs normalization processing on the second pixel data to obtain first normalized pixel data, wherein the second pixel data is data of low gain pixels; if the conventionally processed image data is not saturated, the first normalized pixel data is output from the imaging sensor (60) as conventionally processed image data, and if the conventionally processed image data is saturated, the second normalized pixel data is clipped from the first normalized pixel data and output from the imaging sensor (60) as conventionally processed image data.

Description

Technical Field

[0001] This invention relates to a sensor, an electronic device, and a non-transitory computer-readable medium. Background Technology

[0002] Smartphones and tablets are widely used in our daily lives. Many electronic devices today are equipped with camera components for capturing images. Some are portable, making them easy to carry. Therefore, users of these devices can easily take photos of objects anytime, anywhere using their built-in camera components.

[0003] Many electronic devices can generate High Dynamic Range (HDR) image data, which represents a wide brightness range from dark pixels to bright pixels. In this case, the sensitivity level of the pixel data acquired by the imaging sensor of the camera component differs between normal-gain pixels and low-gain pixels. Therefore, a specific process is needed to normalize the pixel data of low-gain pixels acquired by the imaging sensor. This specific process is necessary even if HDR image data is not required. Furthermore, specific hardware architectures and / or software processes are required to perform this specific process. Therefore, the cost of developing such specific hardware architectures and / or specific software increases. Summary of the Invention

[0004] The present invention aims to solve at least one of the above-mentioned technical problems. Therefore, the present invention provides an imaging sensor, an electronic device, and a non-transitory computer-readable medium.

[0005] According to the present invention, an imaging sensor may include a first circuit and a second circuit.

[0006] The first circuit is configured to: acquire first acquired pixel data and output the first acquired pixel data from the imaging sensor as conventional processed image data, wherein the first acquired pixel data is data of conventional gain pixels.

[0007] The second circuit is configured to: acquire second acquired pixel data and normalize the second acquired pixel data to obtain first normalized pixel data, wherein the second acquired pixel data is low-gain pixel data.

[0008] The second circuit is also configured to output first normalized pixel data from the imaging sensor as conventional image data when conventional processed image data is input into the conventional image link, if the conventional processed image data is not saturated.

[0009] The second circuit is also configured to: when conventionally processed image data is input into the conventional image link, if the conventionally processed image data is saturated, then extract second normalized pixel data from the first normalized pixel data and output the second normalized pixel data from the imaging sensor as conventionally processed image data.

[0010] According to the present invention, an electronic device may include the imaging sensor described above and a third circuit.

[0011] The third circuit is configured to collect multiple embedded data slices in the spare space of the conventional image data of the low-gain pixel itself and in the spare space of another conventional image data of another pixel to obtain compressed data.

[0012] According to the present invention, a non-transitory computer-readable medium is provided thereon storing program instructions for performing at least the following operations.

[0013] Multiple data slices are collected in the spare space for the regular image data of the low-gain pixel itself and in another spare space for the regular image data of the other pixel. The regular image data is output from the imaging sensor.

[0014] Compressed data is obtained based on the collected data fragments.

[0015] The compressed data is expanded to obtain the first normalized pixel data for reconstruction. Attached Figure Description

[0016] The foregoing and / or other aspects and advantages of the embodiments of the present invention will become more apparent and easier to understand in conjunction with the accompanying drawings and the following detailed description.

[0017] Figure 1 A plan view of the first side of an electronic device according to an embodiment of the present invention is shown.

[0018] Figure 2 A plan view of the second side of an electronic device according to an embodiment of the present invention is shown.

[0019] Figure 3 A block diagram of an electronic device according to an embodiment of the present invention is shown.

[0020] Figure 4 An example of a pixel array of an imaging sensor in a camera assembly of an electronic device according to an embodiment of the present invention is shown.

[0021] Figure 5 Another example of a pixel array of an imaging sensor in a camera assembly of an electronic device according to an embodiment of the present invention is shown.

[0022] Figure 6The structure of an imaging sensor for a camera assembly according to an embodiment of the present invention is shown, as well as the process performed in the imaging sensor to generate conventionally processed image data for a conventional image link of an electronic device.

[0023] Figure 7 A flowchart illustrating the conventional image data generation process for generating conventional image data for a conventional image link of an electronic device according to an embodiment of the present invention is shown.

[0024] Figure 8 The structure of an imaging sensor for a camera assembly according to an embodiment of the present invention is shown, as well as the process performed in the imaging sensor to generate conventionally processed image data for an electronic device's HDR image link.

[0025] Figure 9 A flowchart illustrating the conventional image data generation process for generating conventional image data for an electronic device's HDR image link according to an embodiment of the present invention is shown.

[0026] Figure 10 An example of compressing a tone curve of first normalized pixel data by reducing the number of bits in the first normalized pixel data is shown.

[0027] Figure 11 The relationship between the first normalized pixel data and its output from the imaging sensor is shown if the output is linear.

[0028] Figure 12 The bit format for conventional image data processing in an electronic device according to the present invention is shown.

[0029] Figure 13 This illustrates an example of conventionally processed image data where four data slices are embedded into a specific pixel array.

[0030] Figure 14 The first normalized pixel data recovery process is shown, which reconstructs the first normalized pixel data based on conventionally processed image data output from the second circuit.

[0031] Figure 15 It shows from Figure 13 The image shown is an example of collecting four data slices from conventionally processed image data in a pixel array.

[0032] Figure 16 The reconstruction of the first normalized pixel data to be input into the HDR image link is shown.

[0033] Figure 17 An extended tone curve is shown for expanding compressed data obtained by collecting multiple data slices from conventionally processed image data.

[0034] Figure 18 The invention illustrates how conventional image links and HDR image links are utilized in an electronic device according to the invention. Detailed Implementation

[0035] Embodiments of the present invention will be described in detail, and examples of embodiments will be shown in the accompanying drawings. Throughout the specification, the same or similar elements and elements having the same or similar functions may be designated by similar reference numerals. The embodiments described herein with reference to the accompanying drawings are illustrative and intended to illustrate the invention, but should not be construed as limiting the invention.

[0036] Figure 1 This is a plan view of the first side of the electronic device 10 according to an embodiment of the present invention. Figure 2 This is a plan view of the second side of the electronic device 10 according to an embodiment of the present invention. The first side may be referred to as the rear side of the electronic device 10, and the second side may be referred to as the front side of the electronic device 10.

[0037] like Figure 1 and Figure 2 As shown, the electronic device 10 may include a display 20 and a camera assembly 30. In this embodiment, the camera assembly 30 includes a first main camera 32, a second main camera 34, and a sub-camera 36. The first main camera 32 and the second main camera 34 can capture images from a first side of the electronic device 10, and the sub-camera 36 can capture images from a second side of the electronic device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called external cameras, while the sub-camera 36 is a so-called internal camera. As an example, the electronic device 10 may be a mobile phone, a tablet computer, a personal digital assistant, etc.

[0038] Each of the first main camera 32, the second main camera 34, and the sub-camera 36 has an imaging sensor that converts light that has passed through a color filter into an electrical signal. The signal value of the electrical signal depends on the amount of light that has passed through the color filter.

[0039] Although the electronic device 10 according to this embodiment has three cameras, the electronic device 10 may have fewer than three cameras or more than three cameras. For example, the electronic device 10 may have two cameras, four cameras, five cameras, and so on.

[0040] Figure 3 This is a block diagram of the electronic device 10 according to this embodiment. (As shown...) Figure 3As shown, in addition to the display 20 and camera assembly 30, the electronic device 10 may include a main processor 40, an image signal processor 42, a memory 44, a power supply circuit 46, and a communication circuit 48. The display 20, camera assembly 30, main processor 40, image signal processor 42, memory 44, power supply circuit 46, and communication circuit 48 are interconnected via a bus 50.

[0041] The main processor 40 executes one or more program instructions stored in the memory 44. The main processor 40 implements various applications and data processing of the electronic device 10 by executing program instructions. The main processor 40 can be one or more computer processors. The main processor 40 is not limited to a single Central Processing Unit (CPU) core, but can have multiple CPU cores. The main processor 40 can be the main CPU of the electronic device 10, an Image Processing Unit (IPU), or a Digital Signal Processor (DSP) provided with the camera assembly 30.

[0042] The image signal processor 42 controls the camera assembly 30 and processes various image data acquired by the camera assembly 30 to generate target image data. For example, the image signal processor 42 can apply de-mosaic processing, noise reduction processing, automatic exposure processing, automatic focus processing, automatic white balance processing, high dynamic range processing, etc. to the image data acquired by the camera assembly 30.

[0043] In this embodiment, the main processor 40 and the image signal processor 42 cooperate with each other to generate target image data of the object acquired by the camera assembly 30. That is, the main processor 40 and the image signal processor 42 are configured to acquire images of the object by means of the camera assembly 30 and apply various image processing techniques to the acquired image data.

[0044] Memory 44 stores program instructions and various data that will be executed by the main processor 40. For example, data for acquiring images is also stored in memory 44.

[0045] Memory 44 may include high-speed RAM and / or non-volatile memory (such as flash memory and disk storage). In other words, memory 44 may include non-transitory computer-readable media that stores program instructions.

[0046] The power supply circuit 46 may have a battery such as a lithium-ion rechargeable battery and a battery management unit (BMU) for managing the battery.

[0047] The communication circuit 48 is configured to receive and transmit data for communication with base stations of telecommunications network systems, the Internet, or other devices via wireless communication. The wireless communication can employ any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), LTE-Advanced, and 5th generation (5G). The communication circuit 48 may include an antenna and radio frequency (RF) circuitry.

[0048] Figure 4 An example of the pixel array of the imaging sensor 60 in the camera assembly 30 is shown. In other words, the camera assembly 30 has the imaging sensor 60 to acquire images of objects. Figure 4 The pixel array shown includes green, red, and blue pixels. There are two types of green pixels: regular gain green pixels (GR) and low gain green pixels (GL). The sensitivity of low gain green pixels (GL) is lower than that of regular gain green pixels (GR). For example, the analog gain of low gain green pixels (GL) is lower than that of regular gain green pixels (GR). In another example, the exposure time of low gain green pixels (GL) is shorter than that of regular gain green pixels (GR).

[0049] Similarly, there are two types of red pixels: regular gain red pixels (RR) and low gain red pixels (RL). The sensitivity of the low gain red pixel (RL) is lower than that of the regular gain red pixel (RR). Additionally, there are two types of blue pixels: regular gain blue pixels (BR) and low gain blue pixels (BL). The sensitivity of the low gain blue pixel (BL) is lower than that of the regular gain blue pixel (BR).

[0050] Figure 5 Another example of the pixel array of the imaging sensor 60 in the camera assembly 30 is shown. In this example, the imaging sensor 60 also has a normal gain green pixel GR, a low gain green pixel GL, a normal gain red pixel RR, a low gain red pixel RL, a normal gain blue pixel BR, and a low gain blue pixel BL. However, Figure 5 The arrangement of pixels in the pixel array is different from that in other pixels. Figure 4 The arrangement of pixels in a pixel array.

[0051] Furthermore, the arrangement of the pixel array is optional. Therefore, any pixel array arrangement can be applied to the imaging sensor 60. Moreover, the color of the pixel array of the imaging sensor 60 is not limited to green, red, and blue. For example, the color of the pixel array of the imaging sensor 60 can include red, yellow, and blue (RYB). More specifically, the imaging sensor can have regular gain red pixels, low gain red pixels, regular gain yellow pixels, low gain yellow pixels, regular gain blue pixels, and low gain blue pixels. Therefore, in the following text, one or more pixels can indicate one or more pixels of any color.

[0052] Figure 6 The structure of the imaging sensor 60 of the camera assembly 30 is shown, as well as the process performed in the imaging sensor 60 to generate conventional processed image data for the conventional image link 70. Figure 6 The imaging sensor 60 generates conventionally processed image data and outputs it to a conventional image link 70. For example, the image signal processor 42 includes a conventional image link 70 to process the conventional image data within the conventionally processed image data. In this embodiment, the conventional image link 70 can process 12 bits of conventional image data, which is the standard bit width of the conventional image link 70.

[0053] like Figure 6 As shown, the imaging sensor 60 has a first circuit 62 for conventional gain pixels of the pixel array and a second circuit 64 for low gain pixels of the pixel array. The functions of the first circuit 62 and the second circuit 64 will be explained below using a single pixel as an example.

[0054] The first circuit 62 acquires first pixel data, which is data from a conventional gain pixel. The value of the first pixel data depends on the amount of light passing through the color filter. In other words, the amount of electrons accumulated in a conventional gain pixel is proportional to the intensity of light passing through the color filter. The first circuit 62 converts the accumulated electrons into a digital value for the first pixel data.

[0055] Subsequently, the first circuit 62 outputs the first acquired pixel data as conventional processed image data from the imaging sensor 60 to the conventional image link 70. That is, since the first acquired pixel data is obtained from conventional gain pixels, its sensitivity is also conventional; therefore, the first circuit 62 does not modify the first acquired pixel data. Thus, the first circuit 62 outputs the first acquired pixel data itself as conventional processed image data from the imaging sensor 60 to the conventional image link 70. The conventional image processing data is compatible with the conventional image link 70.

[0056] For example, in this embodiment, the conventionally processed image data consists of 16 bits, and the first normalized pixel data consists of 12 bits. Therefore, the first normalized pixel data can be contained within the conventionally processed image data. In other words, the conventional image link 70 accepts 12 bits of conventional image data, which is the maximum number of bits to be processed in the conventional image link 70. That is, in this embodiment, the standard bit width of the conventional image link 70 is 12 bits.

[0057] On the other hand, the second circuit 64 obtains second acquired pixel data, which is data of low-gain pixels, and processes the second acquired pixel data to generate conventionally processed image data. Figure 7 A flowchart illustrating the conventional image data generation process for generating conventional processed image data for conventional image link 70 is shown. In this embodiment, the conventional processed image data generation process is performed in the second circuit 64 of the imaging sensor 60, which is constructed from hardware.

[0058] like Figure 7 As shown, firstly, the second circuit 64 obtains second acquired pixel data from the pixel array (step S10). The value of the second acquired pixel data depends on the amount of light passing through the color filter. In other words, the amount of electrons accumulated in the low-gain pixel is proportional to the intensity of the light passing through the color filter. The second circuit 64 converts the accumulated electrons into a digital value of the second acquired pixel data.

[0059] After that, as Figure 7 As shown, the second circuit 64 normalizes the second acquired pixel data to obtain first normalized pixel data (step S12). As mentioned above, the sensitivity of low-gain pixels is lower than that of conventional-gain pixels. Therefore, it is necessary to normalize the values ​​of the second acquired pixel data to obtain the first normalized pixel data.

[0060] like Figure 7 As shown, in step S12, the value of the second acquired pixel data is converted to a normal gain level. In other words, the value of the first normalized pixel data is essentially equal to the value when the sensitivity of the low-gain pixel is equal to the sensitivity of the normal-gain pixel. Therefore, the normalization process is a compensation process that compensates for the low sensitivity of the low-gain pixel.

[0061] After that, as Figure 7 As shown, the second circuit 64 determines whether the value of the conventionally processed pixel data is saturated, in order to determine whether the first normalized pixel data should be output as conventionally processed image data to be input into the conventional image link 70 (step S14).

[0062] For example, in this embodiment, the first normalized pixel data consists of 18 bits, and the conventional image link 70 can accept 12-bit image data. Therefore, if the value of the first normalized pixel data is equal to or less than 4095 (2 12 If the value of the first normalized pixel data is equal to or greater than 0 (up to equal to or less than 4095), then the normalized processing data is not saturated when it is input into the normalized image link 70. In this case, the second circuit 64 determines whether the value of the first normalized pixel data is equal to or less than 4095 (2). 12 It can represent values ​​from 0 to 4095.

[0063] If the conventionally processed image data is not saturated (step S14: No), then the second circuit 64 outputs the first normalized pixel data from the imaging sensor 60 as the conventionally processed image data (step S16). That is, since the first normalized pixel data is equal to or less than 4095, the conventionally processed image data is not saturated, and the second circuit 64 uses the first normalized pixel data itself as the conventionally processed image data and outputs it from the imaging sensor 60 to the conventional image link 70.

[0064] Since the first normalized pixel data is equal to or less than 4095, it can be represented using 12 bits. Therefore, the 12-bit first normalized pixel data can be contained within the 16-bit conventional processed image data.

[0065] The 12-bit image data corresponding to the first normalized pixel data is input into the conventional image link 70. That is, the conventional image link 70 processes the 12-bit image data of the conventional processed image data output from the second circuit 64 in the same way as it processes the conventional image data of the conventional processed image data output from the first circuit 62.

[0066] On the other hand, if the image data is saturated during conventional processing (step S14: Yes), the second circuit 64 clips out the second normalized pixel data from the first normalized pixel data (step S18). For example, in this embodiment, the first normalized pixel data consists of 18 bits. In this case, the second circuit 64 clips the lower 12 bits of the 18-bit first normalized pixel data as the second normalized pixel data. That is, the maximum value of the second normalized pixel data is 12 bits, so the value of the second normalized pixel data is saturated at 4095.

[0067] Subsequently, the second circuit 64 outputs the second normalized pixel data as conventional processed image data from the imaging sensor 60 to the conventional image link 70 (step S20). For example, in this embodiment, the second normalized pixel data is at 4095 (2 12At point ), 12 bits of conventional processed image data are input into the conventional image link 70. That is, the conventional image link 70 processes the 12 bits of conventional processed image data output from the second circuit 64 in the same way as it processes the conventional processed image data output from the first circuit 62.

[0068] After step S20 is completed, the routine image data generation process is finished. However, Figure 7 The conventional image data generation process shown is a process for one pixel. Therefore, the second circuit 64 of the imaging sensor 60 will perform a process such as... for each low-gain pixel. Figure 7 The image data generation process shown is a conventional processing method.

[0069] based on Figure 6 and Figure 7 The process of generating conventional processed image data for a conventional image link 70 has been explained. However, the electronic device 10 according to this embodiment also has a High Dynamic Range (HDR) image link. Therefore, the imaging sensor 60 also needs to acquire and output HDR image data for the HDR image link.

[0070] Since the image signal processor 42 has both a conventional image link 70 and an HDR image link, the conventional image link 70 can also be referred to as a non-HDR image link for processing non-HDR image data.

[0071] Figure 8 The process of generating conventionally processed image data for HDR image link 72, performed in imaging sensor 60, is illustrated. Figure 8 The HDR image link 72 shown is Figure 6 The image shown corresponds to the conventional image link 70.

[0072] Figure 8 The first circuit 62 in the middle is related to the function of the HDR image link 72. Figure 8 The first circuit 62 in the circuit has the same function. Therefore, the first circuit 62 is equivalent to... Figure 6 The first acquired pixel data is obtained in the same way as shown, and the first acquired pixel data is output as regular processed image data.

[0073] On the other hand, with Figure 6 Compared to the function of the second circuit 64, Figure 8 The second circuit 64 for the HDR image link 72 has additional functionality. That is, in Figure 8In the second circuit 64, saturation information of the first normalized pixel data, consisting of 18 bits, is embedded into the conventionally processed image data to enable the recovery of saturation information. That is, the second circuit 64 embeds the saturation information of the first normalized pixel data into the conventionally processed image data itself and another conventionally processed image data, so that the first normalized pixel data can be reconstructed later.

[0074] Figure 9 A flowchart illustrating the conventional image data generation process for generating conventional processed image data for HDR image link 72 is shown. In this embodiment, this conventional image data generation process is performed in the second circuit 64 of the imaging sensor 60, which is constructed of hardware.

[0075] Figure 9 The process from step S10 to step S20 is as follows Figure 7 The same applies. Therefore, it will not be described in detail here. After step S20, the second circuit 64 compresses the first normalized pixel data to reduce the data volume and obtain compressed data (step S30). The first normalized pixel data is the data generated in step S12. For example, in this embodiment, the first normalized pixel data consisting of 18 bits is compressed into compressed data consisting of 16 bits.

[0076] There are various methods to compress the first normalized pixel data. For example, in this embodiment, the second circuit 64 compresses the first normalized pixel data based on a compressed tone curve. In the compressed tone curve, the larger the first normalized pixel data, the higher the compression ratio. That is, the smaller the first normalized pixel data, the lower the compression ratio.

[0077] Figure 10 An example of compressing a tonal curve of first normalized pixel data by reducing the number of bits in the first normalized pixel data is shown. Figure 11 The relationship between the first normalized pixel data and the linear output of the first normalized data is shown, and Figure 12 The bit format of conventional image data processing in the electronic device 10 of this embodiment is shown.

[0078] like Figure 12 As shown, in this embodiment, conventionally processed image data consists of 16 bits, but since the conventional image link 70 accepts 12-bit image data, the effective bits of the image data are 12 bits. The first normalized pixel data consists of 18 bits, and the second normalized pixel data consists of 12 bits. That is, in step S12, the second circuit 64 performs normalization processing on the second acquired pixel data to obtain the first normalized pixel data consisting of 18 bits.

[0079] The remaining 4 bits of the 16-bit conventional image data constitute a spare space. In this embodiment, the spare space is used to hold the saturation information of the first normalized pixel data. Since the spare space consists of only 4 bits, the saturation information of the first normalized pixel data is sparsely embedded into multiple spare spaces of the conventional image data.

[0080] like Figure 11 As shown, the first normalized pixel data, consisting of 18 bits, can represent values ​​from 0 to 262143. If the value of the first normalized pixel data is equal to or greater than 0 and equal to or less than 4095, it can be represented by 12 bits. Therefore, this value can be accommodated in the second normalized pixel data, which can be accommodated in the conventional processed image data consisting of 16 bits.

[0081] On the other hand, if the value of the first normalized pixel data is equal to or greater than 4096 and equal to or less than 262143, then the value cannot be represented by 12 bits. Therefore, this value cannot be contained in the second normalized pixel data, which can be contained in the conventionally processed image data. Therefore, the second circuit 64 embeds the saturation information of the first normalized pixel data into the conventionally processed image data.

[0082] In step 30, the second circuit 64 uses... Figure 10 The compressed tone curve shown compresses the first normalized pixel data to obtain compressed data and reduce the amount of data.

[0083] For example, based on Figure 10 In the compressed tone curve, when the first normalized pixel data is 0, the allocated compressed data is also 0. On the other hand, based on Figure 10 In the compressed tone curve, when the first normalized pixel data is 262143, the allocated compressed data is 65535. The higher the value of the first normalized pixel data, the higher the compression ratio.

[0084] If the image data is saturated during normal processing, it means that the pixels are very bright. The brighter the pixels, the more difficult it is for the human eye to distinguish pixels of different brightness. Therefore, in this embodiment, the compression ratio increases as the value of the first normalized pixel data increases.

[0085] As a result of the compression process, the first normalized pixel data consisting of 18 bits can be converted into compressed data consisting of 16 bits, which can represent a value that is equal to or greater than 0 and equal to or less than 65535.

[0086] Subsequently, as Figure 9As shown, the second circuit 64 divides the compressed data into multiple data slices and embeds the multiple data slices into the spare space of the conventional processed image data of the low-gain pixel itself and the spare space of another conventional processed image data of another pixel (step S32).

[0087] In this embodiment, since the compressed data consists of 16 bits, it is divided into four data segments, each consisting of 4 bits. For example... Figure 12 As shown, the spare space for the 16-bit conventional image data consists of 4 bits. Therefore, the second circuit 64 needs to embed the four data slices (each consisting of 4 bits) separately into the four spare spaces of the conventional image data.

[0088] Figure 13 An example of conventionally processed image data is shown, in which four data slices are embedded into a specific pixel array. In this example, the second circuit 64 embeds the first data slice into spare space of the conventionally processed image data of the low-gain red pixel RL itself, where compressed data has already been generated from the low-gain red pixel RL.

[0089] Furthermore, the second circuit 64 embeds the second data slice into the spare space of the conventional processed image data of the conventionally processed red pixel RR adjacent to the right of the low-gain red pixel RL. Furthermore, the second circuit 64 embeds the third data slice into the spare space of the conventionally processed image data of the conventionally processed red pixel RR adjacent to the lower right of the low-gain red pixel RL. Furthermore, the second circuit 64 embeds the fourth data slice into the spare space of the conventionally processed image data of the conventionally processed red pixel RR adjacent to the lower bottom of the low-gain red pixel RL.

[0090] However, the method of embedding data slices into spare space for conventional image data processing is not limited to Figure 13 Examples exist. Various methods exist for sparsely embedding data slices into compressed data. Furthermore, data slices can be embedded into regular processed image data containing pixels of the same or different colors as the original pixels that generated the compressed data.

[0091] Subsequently, as Figure 9 As shown, the second circuit 64 outputs second normalized pixel data and data slices embedded in the spare space from the imaging sensor 60 as regular image data (step S34).

[0092] like Figure 12As shown, the second circuit 64 stores the 12-bit second normalized pixel data in the valid bits of the conventional image data and stores the 4-bit data slice in the spare space of the conventional image data. Therefore, the 16-bit conventional image data is output from the imaging sensor 60 to the conventional image link 70 and the HDR image link. That is, in this embodiment, the conventional image data includes the second normalized pixel data and the compressed data slice.

[0093] After executing step S34, the routine image data generation process is completed. However, Figure 9 The conventional image data generation process shown is performed on a per-pixel basis. Therefore, the second circuit 64 of the imaging sensor 60 will perform the following for each low-gain pixel: Figure 9 The image data generation process shown is a conventional processing method.

[0094] like Figure 6 As shown, conventional processed image data based on first acquired pixel data from the first circuit 62, conventional processed image data based on first normalized pixel data from the second circuit 64, and conventional processed image data based on second normalized pixel data can be directly input into the conventional image link 70.

[0095] On the other hand, such as Figure 8 As shown, in order to obtain the first normalized pixel data of the HDR image data input into the HDR image link 72, the electronic device 10 must reconstruct the first normalized pixel data based on multiple conventionally processed image data.

[0096] Figure 14 The first normalized pixel data recovery process is illustrated, which reconstructs the first normalized pixel data based on conventionally processed image data output from the second circuit 64.

[0097] like Figure 8 As shown, the first normalized pixel data recovery process is executed by the third circuit 66. For example, the third circuit 66 can be composed of a main processor 40, which executes a software program to implement the first normalized pixel data recovery process. The instructions of the software program can be stored on a non-transitory computer-readable medium, and the main processor 40 reads and executes the instructions of the software program from the non-transitory computer-readable medium to perform the first normalized pixel data recovery process.

[0098] In addition, the third circuit 66 can be composed of a combination of the main processor 40 and the image signal processor 42, or the third circuit 66 can be implemented using an application-specific integrated circuit (ASIC) to realize the first normalized pixel data recovery process.

[0099] like Figure 14 As shown, the third circuit 66 collects multiple data slices in the spare space of the conventional image data of the low-gain pixel itself and in the spare space of another conventional image data of another pixel to obtain compressed data (step S40).

[0100] Figure 15 It shows from Figure 13 The image shown is an example of collecting four data slices from a typical processed image data in a pixel array. Figure 16 The reconstruction of the first normalized pixel data of the HDR image data input to the HDR image link 70 is shown.

[0101] like Figure 15 and Figure 16 As shown, in this embodiment, the third circuit 66 collects a first data slice from the spare space of the conventional processed image data of the low-gain red pixel RL, a second data slice from the spare space of the conventional processed image data of the conventional-gain red pixel RR on the right, a third data slice from the spare space of the conventional processed image data of the conventional-gain red pixel RR on the lower right, and a fourth data slice from the spare space of the conventional processed image data of the conventional-gain red pixel RR below.

[0102] In this embodiment, the spare space consists of 4 bits, so the compressed data consists of 16 bits (4 by 4 bits). In other words, the third circuit 66 connects the four collected data slices together to reconstruct the compressed data.

[0103] Subsequently, as Figure 14 As shown, the third circuit 66 expands the compressed data collected in step S40 to obtain the reconstructed first normalized pixel data (step S42). In this embodiment, the second circuit 64 uses... Figure 10 The compressed tone curve shown compresses the first normalized pixel data. Therefore, the third circuit 66 uses... Figure 17 The extended tone curve shown expands the compressed data. Figure 17 The extended hue curve in the middle is Figure 10 The mirror image of the compressed tone curve in the third circuit 66 is obtained by using... Figure 17 The extended tone curve in the image sensor reverse-converts the compressed data into reconstructed first normalized pixel data, which is almost equal to the first normalized pixel data in the imaging sensor 60.

[0104] like Figure 17 As shown, the value of the compressed data is equal to or greater than 0 and equal to or less than 65535(2). 16The value of the first normalized pixel data is equal to or greater than 0 and equal to or less than 262143 (2). 18 Therefore, it is impossible to accurately recover the value of the first normalized pixel data. However, this is not a major problem, because the brightness of the first normalized pixel data with larger values ​​is almost saturated, and the human eye cannot perceive the compression error.

[0105] Next, as Figure 14 As shown, the third circuit 66 outputs the reconstructed first normalized pixel data to the HDR image link 72 (step S44). In this embodiment, the reconstructed first normalized pixel data consists of 18 bits. Therefore, the reconstructed first normalized pixel data can be input into the HDR image link 72 to process HDR image data.

[0106] After executing step S44, the first normalized pixel data recovery process is completed. However, Figure 14 The first normalized pixel data recovery process shown is for one pixel. Therefore, the third circuit 66 will perform the following for each low-gain pixel: Figure 14 The first normalized pixel data recovery process is shown.

[0107] Figure 18 The diagram illustrates the use of a conventional image link 70 and an HDR image link 72. The conventional image link 70 can process conventional image data quickly, but the quality of the conventional image data is not very high. On the other hand, the HDR image link 72 can generate high-quality images based on HDR image data, but this processing is not as fast.

[0108] For example, in this embodiment, the electronic device 10 uses a conventional image link 70 to generate a moving image (video) using clipped second normalized pixel data. In this case, due to the characteristics of moving images, the electronic device 10 needs to process many images within a certain time.

[0109] On the other hand, the electronic device 10 utilizes the HDR image link 72 to generate a still image using the reconstructed first normalized pixel data. In this case, the electronic device 10 has sufficient processing time to reconstruct the first normalized pixel data and process the reconstructed first normalized pixel data.

[0110] In another example, electronic device 10 can switch between conventional image link 70 and HDR image link 72 based on the frame rate of the moving image. For example, if the frame rate of the moving image is equal to or greater than a certain threshold (e.g., 60 images per second), electronic device 10 uses conventional image link 70 to generate the moving image based on clipped second normalized pixel data.

[0111] On the other hand, if the frame rate of the motion image is less than a certain threshold, the electronic device 10 uses the HDR image link 72 to generate the motion image based on the reconstructed first normalized pixel data.

[0112] Of course, the electronic device 10 can select one of the conventional image link 70 and the HDR image link 72 based on various factors to smoothly process image data and obtain high-quality images for the user.

[0113] As described above, the electronic device 10 according to this embodiment, such as Figure 6 As shown, the electronic device 10 can process HDR image data just like it processes regular image data. Therefore, the electronic device 10 can process regular image data smoothly and quickly.

[0114] In addition, such as Figure 8 As shown, the electronic device 10 can reconstruct HDR image data based on embedded data slices of compressed data from the first normalized pixel data. Therefore, the electronic device 10 can process and manipulate HDR image data to obtain high-quality images for the user. Thus, the user can obtain unsaturated images and enjoy HDR images.

[0115] Furthermore, although the first normalized pixel data was compressed in the above embodiments to reduce the amount of data, it is not always necessary to compress the first normalized pixel data. In this case, the second circuit 64 of the imaging sensor 60 divides the first normalized pixel data into multiple data slices and embeds these data slices into the conventional processed image data of the low-gain pixel itself and another conventional processed image data of another pixel.

[0116] Furthermore, if the length of the compressed data is equal to or less than the length of the spare space for the conventional image data, the second circuit 64 does not need to divide the compressed data into multiple data slices. In this case, the second circuit 64 embeds the compressed data into the spare space of the conventional image data for the low-gain pixels themselves.

[0117] In the description of embodiments of the present invention, it should be understood that terms such as “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “rear,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” should be interpreted as referring to the direction or position as described or shown in the accompanying drawings. These relative terms are used only to simplify the description of the invention and do not indicate or imply that the mentioned devices or elements must have a particular orientation, or be constructed or operated in a particular orientation. Therefore, these terms should not constitute a limitation on the invention.

[0118] Furthermore, the use of terms such as “first” and “second” herein is for descriptive purposes only and is not intended to indicate or imply relative importance or significance, nor to imply the number of technical features indicated. Therefore, a feature defined by “first” and “second” may include one or more of that feature. In the description of this invention, unless otherwise stated, “a plurality” means two or more.

[0119] In the description of embodiments of the present invention, unless otherwise specified or limited, terms such as “installed,” “connected,” and “coupled” are widely used and can be, for example, a fixed connection, a detachable connection, or an integral connection; it can also be a mechanical connection or an electrical connection; it can also be a direct connection or an indirect connection through an intermediate structure; it can also be internal communication between two elements, the meaning of which can be understood by those skilled in the art according to the specific circumstances.

[0120] In embodiments of the present invention, unless otherwise specified or limited, the structure of the first feature "above" or "below" the second feature may include the following embodiments: the first feature and the second feature are in direct contact, or the first feature and the second feature are not in direct contact with each other, but rather make contact through an additional feature contact between them. Furthermore, the first feature "above," "over," or "on" the second feature may include the following embodiments: the first feature is directly / obliquely above, below, or "on" the second feature, or simply means that the height of the first feature is higher than the height of the second feature; while the first feature "below," "below," or "below" the second feature may include the following embodiments: the first feature is directly / obliquely below, below, or "below" the second feature, or simply means that the height of the first feature is lower than the height of the second feature.

[0121] Various embodiments and examples have been provided in the foregoing description to implement different structures of the present invention. To simplify the invention, certain elements and arrangements have been described above. However, these elements and arrangements are merely examples and are not intended to limit the invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the invention. This repetition is for simplification and clarity, and does not indicate a relationship between different embodiments and / or arrangements. In addition, examples of different processes and materials are provided in the present invention. However, those skilled in the art will understand that other processes and / or materials may also be applied.

[0122] Throughout this specification, references to "embodiment," "some embodiments," "exemplary embodiments," "example," "specific example," or "some examples" refer to specific features, structures, materials, or properties described in connection with an embodiment or example that are included in at least one embodiment or example of the invention. Therefore, the appearance of these phrases throughout this specification does not necessarily refer to the same embodiment or example of the invention. Furthermore, in one or more embodiments or examples, specific features, structures, materials, or properties may be combined in any suitable manner.

[0123] Any process or method described in the flowchart or otherwise herein can be understood as including one or more modules, segments, or portions of code comprising executable instructions for implementing specific logical functions or steps within that process, and the scope of preferred embodiments of the invention includes other embodiments. Those skilled in the art will understand that functions may be implemented in an order different from the order shown or discussed, including in substantially the same order or in reverse order.

[0124] The logic and / or steps otherwise described herein or shown in flowcharts, such as a specific list of executable instructions for implementing logical functions, may be embodied in any computer-readable medium for use by, or in combination with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a processor-integrated system, or other system capable of obtaining and executing instructions from and with an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any device that adaptively includes, stores, communicates, propagates, or transmits a program to be used by or in combination with an instruction execution system, apparatus, or device. More specific examples of computer-readable media include, but are not limited to: electronic connections (electronic devices) having one or more wires, portable computer casings (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), fiber optic devices, and compact disk read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because, for example, when a program needs to be obtained electronically, the paper or other suitable medium can be optically scanned, then edited, decrypted or processed by other suitable methods, and then the program can be stored in computer memory.

[0125] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in memory, and these steps or methods can be executed by a suitable instruction execution system. For example, if implemented in hardware, similarly in another embodiment, the steps or methods can be implemented by one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing data signals, application-specific integrated circuits (ASICs) having suitable combinations of logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0126] Those skilled in the art will understand that all or part of the steps in the exemplary methods described above can be implemented by instructing related hardware using programs. These programs can be stored in a computer-readable storage medium, and when run on a computer, they include one or a combination of the steps in the method embodiments of the present invention.

[0127] Furthermore, each functional unit in the embodiments of the present invention can be integrated into a processing module, or these units can exist physically separately, or two or more units can be integrated into a processing module. The integrated module can be implemented in hardware or as a software functional module. When the integrated module is implemented as a software functional module and sold or used as a standalone product, the integrated module can be stored in a computer-readable storage medium.

[0128] The aforementioned storage media can be read-only memory, disk, CD, etc.

[0129] Although embodiments of the invention have been shown and described, those skilled in the art will understand that these embodiments are illustrative and should not be construed as limiting the invention, and that changes, modifications, substitutions and variations may be made in the embodiments without departing from the scope of the invention.

Claims

1. An imaging sensor, comprising: A first circuit is configured to: acquire first acquired pixel data and output the first acquired pixel data from the imaging sensor as first conventionally processed image data, wherein the first acquired pixel data is data of conventionally gained pixels; and The second circuit is configured to: acquire second acquired pixel data and normalize the second acquired pixel data to obtain first normalized pixel data, wherein the second acquired pixel data is data of low-gain pixels; The second circuit is further configured to: when the first conventionally processed image data is input into the conventional image link, if the first conventionally processed image data is not saturated, output the first normalized pixel data from the imaging sensor as the second conventionally processed image data; and The second circuit is further configured to: when the first conventionally processed image data is input into the conventional image link, if the first conventionally processed image data is saturated, then extract second normalized pixel data from the first normalized pixel data, and output the second normalized pixel data from the imaging sensor as the second conventionally processed image data.

2. The imaging sensor according to claim 1, wherein, The first conventionally processed image data and the second conventionally processed image data are input into the conventional image link that processes non-high dynamic range HDR image data.

3. The imaging sensor according to claim 1, wherein, The second circuit is further configured to embed saturation information of the first normalized pixel data into the second conventionally processed image data itself and another conventionally processed image data.

4. The imaging sensor according to claim 3, wherein, The second circuit is further configured to compress the first normalized pixel data to reduce the amount of data and obtain compressed data.

5. The imaging sensor according to claim 4, wherein, The second circuit is configured to divide the compressed data into multiple data slices and embed the multiple data slices into the spare space of the second conventionally processed image data of the low-gain pixel itself and the spare space of another conventionally processed image data of another pixel, wherein the spare space refers to the remaining space of the second conventionally processed image data after the second normalized pixel data is contained in the second conventionally processed image data.

6. The imaging sensor according to claim 5, wherein, The second circuit is further configured to compress the first normalized pixel data by reducing the number of bits of the first normalized pixel data based on a compressed tone curve.

7. The imaging sensor according to claim 6, wherein, The first normalized pixel data is compressed based on the compressed tone curve. The larger the first normalized pixel data is, the higher the compression ratio of the compressed tone curve.

8. The imaging sensor according to claim 7, wherein, The plurality of data slices are embedded into the spare space of the second conventionally processed image data of the low-gain pixel itself and the spare space of the conventionally processed image data of the pixels adjacent to the low-gain pixel.

9. The imaging sensor according to claim 8, wherein, The second circuit is further configured to output the second conventionally processed image data, comprising the second normalized pixel data and the compressed data, from the imaging sensor.

10. An electronic device, comprising: The imaging sensor according to any one of claims 5-9; as well as A third circuit is configured to collect the embedded plurality of data slices in the spare space of the second conventionally processed image data of the low-gain pixel itself and the spare space of the other conventionally processed image data of the other pixel to obtain the compressed data.

11. The electronic device according to claim 10, wherein, The third circuit is further configured to expand the compressed data obtained by collecting the plurality of data slices to obtain reconstructed first normalized pixel data.

12. The electronic device according to claim 11, wherein, The third circuit is also configured to output the reconstructed first normalized pixel data.

13. The electronic device according to claim 12, wherein, The reconstructed first normalized pixel data is input into the HDR image link to process the reconstructed first normalized pixel data.

14. A computer-readable medium having program instructions stored thereon for performing at least the following operations: In the spare space of the regular image data of the low-gain pixel itself and in another spare space of the regular image data of another pixel, multiple embedded data slices are collected, wherein, The image data being processed is output from the imaging sensor; Compressed data is obtained based on the collected data fragments; as well as The compressed data is expanded to obtain reconstructed first normalized pixel data.

Images