Image compression method using saturated pixels, encoder and electronic device

Abstract

An image compression method for compressing image data generated by an image sensor is provided, the image compression method including: detecting saturated pixels among a plurality of pixels included in a pixel group included in the image data, the saturated pixels having pixel values exceeding a threshold value, and the plurality of pixels being adjacent to each other and having the same color as each other; generating a saturation flag indicating a position of the saturated pixels; compressing the image data by comparing a reference pixel with at least one non-saturated pixel among the plurality of pixels included in the pixel group; and outputting a bitstream including the saturation flag, a compression result, and a compression method.

Description

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10-2020-0088453 filed with the Korean Intellectual Property Office on July 16, 2020 and Korean Patent Application No. 10-2021-0008910 filed with the Korean Intellectual Property Office on January 21, 2021, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] The exemplary embodiments of this disclosure relate to image compression methods, and more specifically, to image compression methods, encoders, and electronic devices using saturated pixels. Background Technology

[0004] As the demand for high-resolution images increases, the size of image data generated by image sensors is also increasing. Since the size of image data is related to the data transmission rate, an efficient method for compressing image data is needed.

[0005] To reduce the size of image data, a method is used to compress the data based on the difference between the target pixel to be compressed and the reference pixel used for compression. However, excessive light reception leads to saturation of pixels, resulting in large pixel values, and thus, a decrease in the compression ratio of the compression method based on the difference between the target and reference pixels. Summary of the Invention

[0006] One or more example embodiments provide a method for effectively compressing saturated pixels.

[0007] According to one aspect of an example embodiment, an image compression method is provided for compressing image data generated by an image sensor. The image compression method includes: detecting saturated pixels among a plurality of pixels included in a pixel group included in the image data, the saturated pixels having a pixel value exceeding a threshold, and the plurality of pixels being adjacent to each other and having the same color; generating a saturation flag indicating the position of the saturated pixels; compressing the image data by comparing a reference pixel with at least one unsaturated pixel among the plurality of pixels included in the pixel group; and outputting a bitstream including the saturation flag, a compression result, and a compression method.

[0008] According to another aspect of example embodiments, there is provided an encoder for processing image data generated by an image sensor, the encoder configured to: detect saturated pixels among a plurality of pixels included in a pixel group, the saturated pixels having pixel values exceeding a threshold, and the plurality of pixels being adjacent to each other and having a same color as each other; generate a saturation flag indicating a location of the saturated pixels; and compress the image data by comparing a reference pixel with at least one non-saturated pixel among the plurality of pixels included in the pixel group.

[0009] According to another aspect of example embodiments, there is provided an electronic device for capturing an image, the electronic device including: an image sensor including a pixel array and configured to output image data; an image signal processor including an encoder configured to: detect saturated pixels among a plurality of pixels included in a pixel group, the saturated pixels having pixel values exceeding a threshold, and the plurality of pixels being adjacent to each other and having a same color as each other; generate a saturation flag indicating a location of the saturated pixels; compress the image data by comparing a reference pixel with at least one non-saturated pixel among the plurality of pixels included in the pixel group; and output a bitstream including a compression result, the saturation flag, and a compression method; and an application processor including a decoder configured to reconstruct the image data by decoding the bitstream. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and / or other aspects, features, and advantages of the example embodiments will be more apparent from the following detailed description, taken in conjunction with the accompanying drawings, which are by way of illustration, in which:

[0011] Figure 1 is a block diagram of an electronic device according to example embodiments;

[0012] Figure 2 is a block diagram of an encoder according to example embodiments;

[0013] Figure 3 is a block diagram of a decoder according to example embodiments;

[0014] Figure 4 is a flowchart of an image compression method according to example embodiments;

[0015] Figure 5A and Figure 5B is a conceptual diagram illustrating a structure of a pixel according to example embodiments;

[0016] Figure 6 is a conceptual diagram illustrating a structure of a bitstream in a differential pulse code modulation (DPCM);

[0017] Figure 7 , Figure 8and Figure 9 is a conceptual diagram illustrating a structure of a bitstream having different bit allocations according to a number of saturated pixels according to an example embodiment;

[0018] Figure 10 is a conceptual diagram illustrating a structure of a pixel according to an example embodiment;

[0019] Figure 11 is a conceptual diagram illustrating a structure of a bitstream in DPCM;

[0020] Figure 12 is a conceptual diagram illustrating a structure of a bitstream generated according to an occurrence of a saturated pixel according to an example embodiment;

[0021] Figure 13A and Figure 13B is a table of compressed information according to an example embodiment;

[0022] Figure 14A and Figure 14B is a block diagram of an electronic device each including an image signal processor according to an example embodiment;

[0023] Figure 15 is a block diagram of an electronic device according to an example embodiment; and

[0024] Figure 16 is a detailed block diagram of a camera module in Figure 15 . DETAILED DESCRIPTION

[0025] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[0026] The term "pixel", "pixel group", "color pixel", or "sub-pixel" can refer to a physical area in which a photosensitive device that senses an object is located, or can refer to a data value corresponding to an electrical signal generated by a photosensitive device of a portion of the sensing object. In addition to the description of the image sensor, the term "color pixel" can be considered as a data value corresponding to an electrical signal generated by a photosensitive device. "Pixel" is used as an inclusive term including color pixels and sub-pixels.

[0027] As the integration density and processing speed of a semiconductor device increase, the number of images that can be captured per second also increases, and thus the size of image data stored and processed by an electronic device gradually increases. Accordingly, a technology for compressing image data for efficient image data processing is desired.

[0028] As the number of photosensitive devices that convert an optical signal into an electrical signal increases, it is easier to generate a high-resolution image. As the integration density of the photosensitive devices increases, the physical distance between the photosensitive devices can decrease, and thus noise such as crosstalk can occur and cause degradation of image quality.

[0029] It is more difficult to input a significant light signal to a light-sensitive device at a low light level, which can be an obstacle to generating a high-resolution image. In order to generate a high-resolution image at a low light level, a plurality of sub-pixels can be used to express a single color pixel by dividing each of the color pixels sharing a single color filter into a plurality of sub-pixels. When the electrical signals generated by the sub-pixels are added, a large amount of light can be ensured. According to an example embodiment, a single color pixel can be divided into four sub-pixels or nine sub-pixels. However, the number of sub-pixels is merely an example. The color pixel can be divided into a square number such as 16 or 25 or a matrix form such as m x n, where "m" and "n" are integers of at least 2.

[0030] In order to more effectively compress image data, satisfactory compression efficiency and low data loss are required. In a method of compressing an image, a target pixel to be compressed can be determined, a reference pixel can be selected from candidate pixels adjacent to the target pixel, and the image data can be compressed based on a difference between the target pixel and the reference pixel. The compression method based on the pixel difference value can be referred to as differential pulse code modulation (DPCM).

[0031] The compression method can also be used when compressing a Bayer image in which color pixels each including a plurality of sub-pixels sharing one color filter are arranged in a specific color pattern. In an example embodiment, the Bayer pattern can include a known pattern in which green, red, blue, and green are sequentially arranged in a matrix.

[0032] Figure 1 is a block diagram of an electronic device 10 according to an example embodiment.

[0033] The electronic device 10 can sense an image of an object using a solid-state image sensor such as a charge-coupled device (CCD) image sensor or a complementary metal-oxide semiconductor (CMOS) image sensor, process or store the sensed image in a memory, and store the processed image in the memory. According to an example embodiment, the electronic device 10 can include, for example, a digital camera, a digital camcorder, a mobile phone, a desktop computer, or a portable electronic device. The portable electronic device can include a laptop computer, a mobile phone, a smart phone, a tablet personal computer (PC), a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital camcorder, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an electronic book, or a wearable device. The electronic device 10 can also be mounted as a component on an electronic device such as a drone and an advanced driver assistance system (ADAS), a vehicle, furniture, a manufacturing facility, a door, and / or various measuring equipment. However, embodiments are not limited thereto.

[0034] Referring to Figure 1 The electronic device 10 can include an image sensor 100, an image signal processor (ISP) 200, an application processor (AP) 300, and a memory subsystem 400. The ISP 200 and the AP 300 can include an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a dedicated microprocessor, a general-purpose processor, or the like.

[0035] The image sensor 100 can convert a light signal of an object received through an optical lens into an electrical signal and generate and output image data IDTA based on the electrical signal. The image sensor 100 can be mounted on an electronic device having an image sensing function or a light sensing function. For example, the image sensor 100 can be mounted on an electronic device such as a camera, a smart phone, a wearable device, an Internet of Things (IoT) device, a tablet PC, a PDA, a PMP, a navigation device, a drone, and an ADAS. The image sensor 100 can also be mounted on an electronic device serving as a component of a vehicle, furniture, a manufacturing facility, a door, and / or various measuring equipment. The image sensor 100 can be controlled by the ISP 200 or the AP 300 to sense an object captured through a lens.

[0036] The image sensor 100 can include a pixel array 110.

[0037] The pixel array 110 can include a plurality of row lines, a plurality of column lines, a plurality of pixels arranged in a matrix form, and a plurality of color filters arranged corresponding to the respective pixels. Each of the pixels can be connected to the row lines and the column lines.

[0038] Each of the pixels can include a photosensitive device. The photosensitive device can sense light and convert the light into a pixel signal as an electrical signal. For example, the photosensitive device can include a photodiode, a phototransistor, a photogate, a pinned photodiode (PPD), or a combination thereof. The photosensitive device can have a four-transistor structure including a photodiode, a transfer transistor, a reset transistor, an amplifier transistor, and a selection transistor. According to an example embodiment, the photosensitive device can have, for example, a one-transistor structure, a three-transistor structure, a four-transistor structure, or a five-transistor structure, or can have a structure in which some transistors are shared by a plurality of pixels.

[0039] The color filter can be arranged to correspond to one of the pixels included in the pixel array 110, and can transmit only light of a specific wavelength incident to the photosensitive device. In an example embodiment, the color filter can include a Bayer color filter. The Bayer pattern is based on the assumption that the human eye derives most of the luminance data from the green component of an object. Half of the pixels included in the Bayer color filter can detect a green signal, a quarter of the pixels can detect a red signal, and the remaining quarter of the pixels can detect a blue signal. According to an example embodiment, the Bayer color filter can have a configuration in which 2x2 color pixels including a red pixel, a blue pixel, and two green pixels are repeatedly arranged. According to an example embodiment, the Bayer color filter can have a configuration in which 2x2 color pixels including a red pixel, a blue pixel, and two wide green pixels are repeatedly arranged. In an example embodiment, an RGB color filter can be used in which a green color filter is arranged for two of the four pixels, and a blue color filter and a red color filter are arranged for the other two pixels, respectively. However, embodiments are not limited thereto. For example, a CYGM color filter in which a cyan color filter, a yellow color filter, a green color filter, and a magenta color filter are arranged for the four pixels, respectively, can be used. In addition, a cyan, yellow, green, and key (CYMK) color filter can also be used. A plurality of color filters can form a single color filter layer. The Bayer pattern is used as an example, but embodiments are not limited to the Bayer pattern. Various patterns including white or yellow or merging at least two color regions can be used.

[0040] The image sensor 100 can further include a plurality of modules that process the pixel signals generated by the pixel array 110. According to an example embodiment, the modules can include additional components for processing light signals or enhancing image sensing capabilities, such as a row driver, a ramp signal generator, a timing generator, an analog-to-digital converter, and a readout circuit. For example, the readout circuit can generate raw data based on the electrical signals of the pixel array 110, and output the raw data as is or the raw data that has been pre-processed such as bad pixel removal as image data IDTA. The image sensor 100 can be implemented in a semiconductor chip or package including the pixel array 110 and the readout circuit.

[0041] The image sensor 100 can output image data IDTA by processing the pixel signals generated by the pixel array 110 using a plurality of modules.

[0042] The image data IDTA is a result of processing the pixel signals using a plurality of modules (e.g., a ramp signal generator and a readout circuit). In an example embodiment, the image data IDTA can include binary codes. In an example embodiment, the image data IDTA can include pixel information and pixel values of an object. For example, the image data IDTA can include pixel information such as a location of a photosensitive device in the pixel array 110 that senses a specific portion of an object, a color of a pixel as a type of color filter, etc. In an example embodiment, the image data IDTA can include pixel values involved in a dynamic range of data that can be processed by the image sensor 100. For example, since a dynamic range of a single pixel is ten bits, the image data IDTA can include a sensed pixel value among pixel values 0 to 1023 as data about a single pixel.

[0043] In an example embodiment, a set of pixel values can be referred to as a Bayer image. At this time, the Bayer image can be distinguished from a color filter that physically transmits light of a specific wavelength. In an example embodiment, the Bayer image can correspond to an abstract shape, and an image sensed by the image sensor 100 is recognized as such an abstract shape by the ISP 200, the AP 300, or a user interface. The Bayer image can be image data including pixel information of a complete image, which is regarded as a single processing unit in the electronic device 10. The term "Bayer image" can be used in the present specification to describe image data, but it will be considered that embodiments can use color filters having various patterns, without being limited to color filters having a Bayer pattern.

[0044] The ISP 200 can include a central processing unit (CPU), a microprocessor, or a microcontroller unit (MCU). The processing performed by the ISP 200 can refer to applying an image enhancement algorithm to an image artifact. For example, the ISP 200 can perform white balancing, denoising, demosaicking, lens shading, gamma correction, etc. on a received image frame, but is not limited thereto. The ISP 200 can perform various types of image post-processing.

[0045] The ISP 200 can perform image processing on the image data IDTA. For example, the ISP 200 can perform image processing on the image data IDTA to change a data format (e.g., to change a Bayer pattern to a YUV format or an RGB format), or to perform image processing for enhancing image quality such as denoising, brightness adjustment, and / or sharpness adjustment. The ISP 200 can form hardware of the electronic device 10. Although the ISP 200 is described as being included in the electronic device 10 in the present specification, the ISP 200 can be included in a separate device connected to the electronic device 10. Figure 1The ISP 200 is separate from the image sensor 100, but embodiments are not limited thereto. The ISP 200 can be integrally disposed with the image sensor 100 or the AP 300.

[0046] According to an example embodiment, the ISP 200 can receive image data IDTA as an output signal of the image sensor 100, and generate encoded data ED as a result of processing the image data IDTA. The encoded data ED can be provided to the outside through the first interface 250.

[0047] The ISP 200 can include an encoder 210, a mode selector 230, and a first interface 250. According to an example embodiment, the ISP 200 can receive image data IDTA and output encoded data ED.

[0048] The encoder 210 can reduce the data size by compressing the image data IDTA, and can encode the image data IDTA according to an image standard so that the image data IDTA can be processed by a processor (e.g., the AP 300). According to an example embodiment, the encoder 210 can generate compressed data CD by compressing the image data IDTA, and output the compressed data CD to the mode selector 230. According to an example embodiment, the encoder 210 can receive a mode signal MODE from the mode selector 230, and generate encoded data ED by performing encoding according to a compression mode.

[0049] According to an example embodiment, when the encoder 210 compresses the image data IDTA, the encoder 210 can compare all pixels of the image data IDTA, except for saturated pixels SP that receive excessive light, with reference pixels having the same color information as the saturated pixels SP and adjacent to the saturated pixels SP.

[0050] According to an example embodiment, the saturated pixels SP can correspond to data having a pixel value generated by receiving excessive light. According to an example embodiment, the saturated pixels SP include data of a pixel that represents a bright light such that a human eye cannot clearly distinguish a difference from a maximum brightness. The saturated pixels SP can have a pixel value slightly different from a maximum pixel value that a pixel can represent, but have little effect on resolution of an image.

[0051] In an example embodiment, a saturated pixel SP can have a pixel value greater than a threshold in a dynamic range, which is a data size that a single pixel can have. According to an example embodiment, the threshold can correspond to a pixel value close to an upper limit of the dynamic range of a single pixel. For example, when the dynamic range of a single pixel is ten bits, i.e., when a pixel includes ten bits of information, a single pixel can have a pixel value of 0 to 1023, and the threshold can be 1000. At this time, a pixel having a pixel value greater than or equal to 1000 can be classified as a saturated pixel SP. For example, the threshold can correspond to 95% of the upper limit of the dynamic range of a single pixel, but is not limited thereto.

[0052] The threshold based on which a pixel is classified as a saturated pixel SP can vary with a change in the dynamic range of a single pixel, an image capturing environment, or a required resolution.

[0053] According to an example embodiment, as the dynamic range of a single pixel decreases, the threshold can vary in proportion to a decrease in the upper limit of a pixel value. For example, when the dynamic range is eight bits, the dynamic range of a single pixel can be 0 to 255, and the threshold can be 230. For example, when the dynamic range is twelve bits, the dynamic range of a single pixel can be 0 to 4191, and the threshold can be 4000.

[0054] According to an example embodiment, when an image capturing environment generates many saturated pixels, the threshold can be increased. For example, when the image capturing environment is outdoors or there is a lot of backlight, the threshold can be 1020. For example, when the image capturing environment is indoors or there is almost no backlight, the threshold can be 950.

[0055] According to an example embodiment, when high resolution is required, the threshold can be set closer to the upper limit of the dynamic range of a single pixel. For example, when high resolution is required, the threshold of a 10-bit pixel can be 1020. For example, when low resolution is required, the threshold of a 10-bit pixel can be 980.

[0056] The threshold is not limited to the specific numbers given above, and can be variably changed to achieve the best compression rate.

[0057] In an example embodiment, the reference pixel can be one of a plurality of pixels. The reference pixel can have the same color as a target pixel to be compressed. The reference pixel can be compressed before the target pixel. The reference pixel can be reconstructed before the target pixel and thus be referenced when the target pixel is decoded. The reference pixel can be located in a predetermined direction with respect to the target pixel and a predetermined distance from the target pixel.

[0058] In an example embodiment, a reference pixel may include a dummy pixel with a specific value. For example, a reference pixel may refer to a dummy pixel having the average pixel value of a plurality of pixels or the median pixel value among the individual pixel values ​​of a pixel, wherein the individual pixel values ​​are sorted in ascending (or descending) order. Figure 5A and Figure 5B Describe in detail the relationship between the reference pixel and the target pixel.

[0059] Encoder 210 can generate encoded data ED by encoding image data ITA. The encoded data ED can be output as a bitstream. ISP 200 can perform reading and writing of data stored in memory subsystem 400 while encoding image data ITA.

[0060] Encoder 210 can reduce the size of the Bayer image by compressing image data IDTA, which corresponds to the raw pixel data obtained through the Bayer color filter. The Bayer image can refer to the Bayer pattern pixel data obtained through pixel array 110.

[0061] According to an example embodiment, encoder 210 can classify pixels in image data IDTA with pixel values ​​greater than a threshold as saturated pixels SP. According to an example embodiment, encoder 210 can determine a reference pixel that is physically adjacent to the saturated pixel SP among candidate pixels that include the same color information as the saturated pixel SP. According to an example embodiment, encoder 210 can compare the pixel value of the target pixel to be compressed with the pixel value of the reference pixel to reduce the amount of data.

[0062] When an image of an object is captured, a large pixel value difference may exist between the target pixel to be compressed and candidate or reference pixels located near the target pixel. The following can lead to large pixel value differences: blotchy pixels appearing when the image sensed by image sensor 100 has a relatively small size compared to other surrounding objects and has high contrast in brightness or color; bad pixels appearing when there is a sensing error in image sensor 100; edge pixels in the corners of Bayer images; situations with significant backlighting in the image capture environment; or outdoor image capture environments. In these cases, the compression ratio may be significantly reduced or image quality may degrade due to the large differences between the candidate pixel used for compression and its neighboring pixels. Therefore, compression must be performed taking into account the large pixel value difference between the target pixel and each of its other neighboring pixels. Image quality degradation can be prevented by considering flags during compression. A method of compressing data based on comparisons (e.g., subtraction between the target pixel and the reference pixel) can be called DPCM.

[0063] Because the image data IDTA is encoded or decoded row by row from left to right, the reference pixel to be compared with the target pixel must be encoded or decoded beforehand. Therefore, according to the example embodiment, the reference pixel can be located to the left or above the target pixel. However, the embodiment is not limited to this. A pixel adjacent to the saturated pixel and located in a region that has already undergone encoding or decoding according to the encoding or decoding order can be determined as the reference pixel. For example, among pixels with the same color information as the saturated pixel SP, the pixel located to the left of the saturated pixel SP and closest to the saturated pixel SP can be determined as the reference pixel. At this time, the relative position between the saturated pixel SP and the reference pixel or the relative position between the target pixel and the reference pixel can be determined. When the reference pixel is determined, the relative position of the reference pixel relative to the target pixel is also shared by the decoder 310. The position of the reference pixel can be related to the position of the saturated pixel SP.

[0064] According to an example embodiment, encoder 210 can compress data in units of pixel groups, where each pixel group includes a set of pixels. According to an example embodiment, when a pixel group includes saturated pixels, encoder 210 can generate only the location information of the saturated pixels without compressing them. According to an example embodiment, when encoder 210 only stores the location information of saturated pixels without compressing them (which results in a reduced compression ratio), encoder 210 can reallocate the space used to store the saturated pixels to another pixel.

[0065] According to an example embodiment, encoder 210 can generate a bitstream by compressing pixel groups. According to an example embodiment, the bitstream may include: a header indicating the compression method, compression mode, compression ratio, loss information, etc.; a saturation flag including the position information of saturated pixels; and residual information indicating the pixel value difference between the target pixel to be compressed and a reference pixel. According to an example embodiment, when the pixel value difference between the saturated pixel and the reference pixel is not stored in the bitstream, the limited space of the bitstream can be allocated to unsaturated pixels, thus reducing the data loss rate of unsaturated pixels. (Refer to...) Figure 6 to Figure 11 Describe the structure of the bit stream.

[0066] Reference Figure 2 as well as Figure 6 to Figure 12 The process of encoding image data IDTA using encoder 210 is described in detail.

[0067] The mode selector 230 can receive compressed data CDs and determine the compression mode for encoding image data ITA from a plurality of compression modes. According to an example embodiment, the ISP 200 may have various compression modes based on the target to be compressed, compression ratio, error rate, and / or loss information. According to an example embodiment, the mode selector 230 can examine the compression ratio, error rate, and loss information of the compressed data CD in the current compression mode and compare the compression ratio, error rate, and loss information of the compressed data CD in the current compression mode with the compression ratio, error rate, and loss information in other compression modes.

[0068] According to an example embodiment, when the compressed data CD shows better performance compared to other compression modes, the mode selector 230 can output a mode signal MODE to the encoder 210, indicating that the image data IDTA should be encoded using the compression mode used to generate the compressed data CD. In an example embodiment, when the compressed data CD does not show better performance compared to other compression modes, the mode selector 230 can output a mode signal MODE, indicating that the image data IDTA should be encoded using a compression mode that is better than the compression mode used to generate the compressed data CD. As a compression mode, a saturation mode that does not compress saturated pixels in a pixel group or a bad pixel mode that detects bad pixels can be used. (Refer to...) Figure 13A and Figure 13B Describe the different compression modes in detail.

[0069] The first interface 250 can support interaction between devices, enabling the transmission of encoded data ED according to a standard suitable for another device or module. An interface is a physical protocol or standard that allows data and / or signals to be smoothly transmitted between devices with different standards and configurations.

[0070] In an example embodiment, to transmit the encoded data ED to AP 300, the first interface 250 may use the same interface supported by AP 300 to transmit the data. In an example embodiment, to store the encoded data ED in memory subsystem 400, the first interface 250 may use the same interface supported by memory subsystem 400 to transmit the data.

[0071] Camera module 50 may include image sensor 100 and ISP 200. Encoded data ED generated by camera module 50 may be output as a bitstream. The encoded data ED may be decoded by a device (e.g., decoder 310) that shares the same protocol as camera module 50. The protocol includes rules governing compression algorithms, such as compression methods, compression order, compression bits, and the position of reference pixels, and may specify mutual agreements that allow decoding to be performed using the same principles as encoding. In an example embodiment, the encoded data ED generated by camera module 50 may be decoded using the exact same protocol as that applied to encoder 210 of camera module 50; therefore, decoder 310 does not need to be mounted on the same semiconductor chip as encoder 210. According to the example embodiment, the manufacturer of camera module 50 may be different from the manufacturer of AP 300.

[0072] Despite Figure 1 The camera module 50 does not include memory, but the embodiments are not limited thereto. According to an example embodiment, the camera module 50 may include a portion of the memory of the memory subsystem 400. For example, the camera module 50 may include dynamic random access memory (DRAM) as part of the memory of the memory subsystem 400. However, the embodiments are not limited thereto. For example, the camera module 50 may include various types of memory that support high-speed data access, such as static RAM (SRAM). When memory is included in the camera module 50, the camera module 50 may have a 3-stacked structure.

[0073] AP 300 may include a CPU, microprocessor, or MCU. AP 300 can perform post-processing on the decoded bitstream output from decoder 310. Post-processing may refer to applying image enhancement algorithms to image artifacts. For example, AP 300 can perform white balance, denoising, de-stitching, lens shading, gamma correction, etc., on the decoded bitstream, but is not limited to these. AP 300 can perform various functions to enhance image quality.

[0074] AP 300 may include a decoder 310 and a second interface 330.

[0075] The second interface 330 can receive encoded data ED as a bitstream generated by encoder 210. The second interface 330 can receive data according to the same protocol as the first interface 250. In an example embodiment, the first interface 250 and the second interface 330 can use the Mobile Industry Processor Interface (MIPI) Alliance. MIPI is a protocol agreed upon by the MIPI Alliance and is a communication protocol jointly defined by mobile device manufacturers regarding interface methods and specifications. However, the interface method is not limited to this, and various data communication and signal input / output protocols can be used.

[0076] The second interface 330 can interact with the encoded data ED in a data communication format suitable for AP 300 and provide the encoded data ED to the decoder 310.

[0077] Decoder 310 can decode the encoded data ED. In an example embodiment, decoder 310 can reconstruct the original pixel values ​​of the compressed target pixels by performing a series of processes that have already been performed by encoder 210 to encode image data IDTA in reverse order. For example, decoder 310 can reconstruct image data IDTA that has been compressed by encoder 210.

[0078] In an example embodiment, decoder 310 may use the same protocol as encoder 210. Decoder 310 may use a decoding method based on an algorithm corresponding to the encoding method used by encoder 210. According to the example embodiment, the relative position of a reference pixel relative to a target pixel may be preset in decoder 310, the relative position being determined by encoder 210. For example, when a pixel that is to the left of and closest to saturated pixel SP, which includes the same color information as saturated pixel SP, is determined as a reference pixel, decoder 310 may decode the target pixel based on the pixel value of the reference pixel, which has the same color information as the target pixel and is to the left of the target pixel. According to the example embodiment, decoder 310 may reconstruct the target pixel using a relatively small amount of data by decoding image data IDTA based on the relative position information between the target pixel and the reference pixel. Figure 3 Describe the decoding operation of decoder 310 in detail.

[0079] The memory subsystem 400 can store image data ITA, compressed data CD, and / or encoded data ED provided from the image sensor 100 or ISP 200. The memory subsystem 400 can store reconstructed data generated by decoding the encoded data ED. The memory subsystem 400 can provide the stored data to other components of the electronic device 10. The memory subsystem 400 can also store various types of system or user data required to drive the electronic device 10.

[0080] The memory subsystem 400 may include volatile memory or non-volatile memory. For example, the memory subsystem 400 may include non-volatile memory that stores various types of information in a non-volatile manner, and volatile memory to which information such as firmware related to the driving electronics 10 is loaded. Volatile memory may include DRAM, SRAM, etc. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc.

[0081] In the example embodiment, a portion of the memory subsystem 400 may store data of the ISP 200, and other portions of the memory subsystem 400 may store data of the AP 300 and provide cache memory functionality. Although in Figure 1 The diagram shows that the memory subsystem 400 supports both ISP 200 and AP 300, but this is only for conceptual and functional description of the memory subsystem 400 supporting memory or cache memory functions. ISP 200, AP 300 and memory subsystem 400 do not necessarily have to be mounted on a single semiconductor chip.

[0082] In the example embodiment, a portion of the memory subsystem 400 may be used as a memory device for storing data of the ISP 200, and may also be used as a buffer for temporarily storing at least a portion of the image data IDTA.

[0083] According to an example embodiment, the memory subsystem 400 can buffer data corresponding to the first row of image data IDTA. Subpixels can be arranged in a matrix, and the encoder 210 can compress the data only after receiving all pixel information for each pixel. Therefore, it is not necessary to process the data corresponding to the first row of image data IDTA immediately.

[0084] When the data corresponding to the second row of image data IDTA is provided to encoder 210, ISP 200 can load the data corresponding to the first row of image data IDTA into memory subsystem 400. Encoder 210 can compress pixel data based on the data corresponding to the first row and the second row of image data IDTA. The compressed data can be stored in memory subsystem 400.

[0085] ISP 200 and AP 300 may be implemented as processing circuitry (such as hardware components including logic circuitry) or as a combination of hardware and software (such as a processor executing software that performs compression). Specifically, the processing circuitry may include, but is not limited to, a CPU, an arithmetic logic unit that performs arithmetic and logical operations, bit shifting, etc., a digital signal processor (DSP), a microprocessor, an application-specific integrated circuit (ASIC), etc.

[0086] According to an example embodiment, the image compression method can increase the number of bits allocated to unsaturated pixels by not compressing saturated pixels, which would reduce the compression ratio. As the number of bits allocated to unsaturated pixels increases, image quality degradation can be reduced, and the compression ratio can be improved.

[0087] Figure 2 This is a block diagram of encoder 210 according to an example embodiment. Reference will also be made to... Figure 1 .

[0088] Reference Figure 2 The encoder 210 may include an encoding detector 211, a saturation flag generator 213, a compressor 215, an encoding reconstructor 217, and an encoding reference buffer 219. The encoder 210 may also include a central processing unit, which typically controls the encoding detector 211, encoding reference buffer 219, saturation flag generator 213, compressor 215, and encoding reconstructor 217. Each of the encoding detector 211, encoding reference buffer 219, saturation flag generator 213, compressor 215, and encoding reconstructor 217 can be operated by its respective processor, and these processors can operate mutually and organically, allowing the encoder 210 to operate as a whole. The encoding detector 211, encoding reference buffer 219, saturation flag generator 213, compressor 215, and encoding reconstructor 217 may be controlled by a processor external to the encoder 210. (See reference...) Figure 1 The encoder 210 described can be applied to Figure 2 Encoder 210. Unless otherwise specified. Figure 1 Those that are different will be given; otherwise, redundant descriptions will be omitted.

[0089] Encoder 210 can reduce the amount of data information by compressing image data ITA from image sensor 100. According to an example embodiment, encoder 210 can compress image data ITA by comparing a target pixel to be compressed in a Bayer image with a plurality of candidate pixels located near the target pixel.

[0090] Encoding detector 211 can receive image data ITA generated by image sensor 100. Encoding detector 211 can divide the pixels of image data ITA into groups, each with a specific arrangement and a specific number of pixels. A group of pixels can be referred to as a pixel group, and encoder 210 can process data on a pixel group basis. For example, encoding detector 211 can compress image data ITA pixel by pixel group. Each pixel group can have a two-dimensional array such as a 3×3 array or a 5×5 array, or a 1×8 array with eight pixels arranged in rows. 3×3 arrays, 5×5 arrays, and / or 1×8 arrays are merely examples, where the 3, 5, and 1 before the "×" are the number of columns, and the 3, 5, and 8 after the "×" are the number of rows. Embodiments are not limited to the arrays and numbers given above.

[0091] According to an example embodiment, the encoding detector 211 can detect a target pixel to be compressed and multiple candidate pixels located near the target pixel. The number of pixels detected at one time can vary with the configuration and size of the pixel group. Candidate pixels can be located to the left or above the target pixel, but the position of the candidate pixels can change according to the encoding order.

[0092] According to an example embodiment, the encoding detector 211 can detect the presence of saturated pixels in a pixel group. According to an example embodiment, the encoding detector 211 can determine pixels in the pixel group that have pixel values ​​exceeding a threshold as... Figure 1 The threshold for a saturated pixel SP is determined as follows: For example, when ten bits are allocated to each pixel and the dynamic range of the pixel is 0 to 1023, the threshold can be set to 1000. In this case, the encoding detector 211 can identify pixels with pixel values ​​exceeding 1000 as saturated pixels SP. The encoding detector 211 can use attribute information (e.g., a flag indicating a saturated pixel SP) to mark pixels identified as saturated pixels SP.

[0093] According to an example embodiment, the encoding detector 211 can detect the presence of at least two saturated pixels in a pixel group, and the encoder 210 can compress image data in saturation mode. For example, when at least two saturated pixels are detected in a pixel group, the encoding detector 211 can send a signal to the encoder 210, causing the encoder 210 to operate in saturation mode, regardless of the mode signal MODE. In an example embodiment, when a new target pixel following an already encoded old target pixel is encoded, the encoding detector 211 can use information about candidate pixels that have been reconstructed by the encoding reconstructor 217 and stored in the encoding reference buffer 219. According to an example embodiment, when compressing or encoding a new target pixel, previously encoded old target pixels can be used as candidate pixels.

[0094] The saturation flag generator 213 can generate position information of saturated pixels in a pixel group in the form of a flag. According to an example embodiment, when a saturated pixel is included in a pixel group as a processing unit, the saturation flag generator 213 can generate a flag that represents the position of the saturated pixel in the pixel group using bits. This flag may include values ​​obtained by numbering the direction information of the target pixel and the reference pixel, or by mapping the direction information to specific bit values.

[0095] According to an example embodiment, the saturation flag generator 213 can process unsaturated pixels among a plurality of pixels included in a pixel group as bit 0 and saturated pixels as bit 1. For example, when a pixel group includes four pixels and the second pixel in the pixel group is a saturated pixel, the saturation flag generator 213 can generate flag 0100. However, processing unsaturated pixels as bit 0 and saturated pixels as bit 1 is merely an example. The saturation flag generator 213 can allow various methods to distinguish between unsaturated and saturated pixels. For example, the saturation flag generator 213 can process unsaturated pixels among a plurality of pixels included in a pixel group as bit 1 and saturated pixels as bit 0. Hereinafter, the above description will be applied to the bit allocation involved in distinguishing between unsaturated and saturated pixels.

[0096] Compressor 215 can encode a header indicating the compression method, compression mode, compression ratio, loss information, etc., including a saturation flag containing the position information of saturated pixels, and residual information indicating the pixel value difference between the target pixel and the reference pixel to be compressed, and can include the encoding result in a bitstream. The bitstream can be a sequence of encoded bits. In an example embodiment, the bitstream may include a header field storing header information, a saturation flag field storing the saturation flag, and a residual field storing residual information indicating the difference between the reference pixel and the target pixel.

[0097] Because image data IDTA is encoded or decoded line by line from left to right, reference pixels to be compared with the target pixel must be encoded or decoded beforehand. According to an example embodiment, encoder 210 may identify a pixel that includes the same color information as the target pixel and is to the left of and closest to the target pixel as a reference pixel. In an example embodiment, decoder 310, receiving encoded data ED including a flag, may immediately identify the reference pixel while decoding the target pixel. However, the embodiments are not limited to the positional relationships described above. It will be understood that a pixel adjacent to a saturated pixel and in a region that has already undergone encoding or decoding according to the encoding or decoding order may be identified as a reference pixel.

[0098] In an example embodiment, compressor 215 may compress a target pixel based on predetermined relative position information between a target pixel and a reference pixel. In an example embodiment, compressor 215 may compare the target pixel to be compressed with a reference pixel providing reference values ​​for compression, the relative position information being predetermined as an attribute or labeled onto pixel information. In an example embodiment, compressor 215 may position the reference pixel at a predetermined position from the target pixel's position and compare the pixel value of the target pixel with the pixel value of the reference pixel. For example, compressor 215 may perform a difference operation on the pixel values ​​of the target pixel and the reference pixel, where the reference pixel is directly above the target pixel and includes the same color information as the target pixel. As a result of performing the difference operation, residual information may be generated. Although a difference operation is given as an example of a comparison operation, the embodiment is not limited to this, and various operations can be used. The method of comparing a target pixel with candidate pixels is not limited to this. Various comparison methods, such as using the average of the pixel values ​​of multiple candidate pixels, can be used.

[0099] According to an example embodiment, compressor 215 may not perform differential operations on saturated pixels SP in image data IDTA. According to an example embodiment, compressor 215 may generate residual information as a result of performing differential operations on a reference pixel and unsaturated pixels among a plurality of pixels included in a pixel group. However, when the pixel group includes saturated pixels SP, compressor 215 may include only the saturation flag generated by saturation flag generator 213 in the bitstream without performing differential operations on the reference pixel and saturated pixels SP. According to an example embodiment, since saturated pixels SP are not included in the bitstream, storage space can be allocated for the results of performing differential operations on saturated pixels SP and reference pixels in the bitstream for unsaturated pixels. Therefore, compared to the case where the residual information of saturated pixels SP is included in the bitstream, more space can be used to represent the residual information of unsaturated pixels, and thus, the loss rate involved in the compression of unsaturated pixels can be reduced.

[0100] Compressor 215 can compress image data IDTA based on a mode signal MODE. According to an example embodiment, the mode signal MODE can instruct compressor 215 to compress image data IDTA differently based on compression mode, compression method, compression ratio, and / or loss information.

[0101] In an example embodiment, as a result of evaluating the compression ratio, error rate, and loss information of the compressed data CD, a mode signal MODE can be generated by mode selector 230, and the mode signal MODE can indicate one of a plurality of compression modes.

[0102] According to an example embodiment, the compression method may include DPCM and averaging. In DPCM, encoding is performed based on a difference operation between the pixel value of the target pixel and a reference value determined from a reference pixel. In averaging, encoding is performed based on the average of the pixel values ​​of the original pixels. Old target pixels that have already undergone encoding can be used as candidate pixels in the DPCM mode, and information about candidate pixels for use as encoded target pixels in the DPCM mode can be stored in the encoding reference buffer 219. Figure 13A and Figure 13B Provide a detailed description of various compression methods.

[0103] In an example embodiment, compressor 215 may include residual information in a compressed field of the bitstream. In this case, the data size of the residual information is larger than the size of the memory allocated to the compressed field, which may result in data loss or image quality degradation when compressing the image data ITA. To prevent image quality degradation, compressor 215 can adjust the data size to the size of the allocated memory by performing bit shifting on the residual information included in the compressed field.

[0104] Compressor 215 can output the encoded data ED as a bitstream. Compressor 215 can also provide the encoded data ED to encoder-reconstructor 217 for decoding the encoded target pixels.

[0105] The encoder-reconstructor 217 can generate candidate pixels by reconstructing the encoded data ED output from the compressor 215. In an example embodiment, the encoder-reconstructor 217 can be configured to... Figure 1 The decoder 310 performs decoding by decoding the encoded target pixels in a configuration similar to that in the encoder 210. While the encoder 210 has all the original pixel information, as well as its encoded and decoded pixel information, of the old target pixels, the decoder 310 does not have the original pixel information of the old target pixels. For example, when the encoder 210 uses old original pixels as candidate pixels and the decoder 310 does not have information about the old original pixels referenced by the encoder 210, errors may occur in the reconstruction result, and therefore, inconsistencies may exist between the encoder 210 and the decoder 310. Therefore, pixels obtained by reconstructing old target pixels that have already undergone encoding can be used as candidate pixels. The encoding reconstructor 217 can store the reconstructed pixels in the memory subsystem 400. The encoding reconstructor 217 can also store the reconstructed pixels directly in the encoding reference buffer 219.

[0106] The encoding reference buffer 219 provides information about candidate pixels as data for encoding a target pixel to the encoding detector 211. Candidate pixels are neighboring pixels adjacent to the target pixel and may have the same color information as the target pixel. According to an example embodiment, the encoding reference buffer 219 may include line memory storing the pixel values ​​of the target pixel's neighboring pixels, which are necessary for encoding the target pixel. According to an example embodiment, the encoding reference buffer 219 may include, but is not limited to, volatile memory such as DRAM or SRAM.

[0107] According to an example embodiment, the image compression method can increase the number of bits allocated to unsaturated pixels by not compressing saturated pixels that would reduce the compression ratio. As the number of bits allocated to unsaturated pixels increases, image quality degradation can be reduced and the compression ratio can be improved. Furthermore, according to the example embodiment, the image compression method can achieve lossless compression even when many saturated pixels are present.

[0108] Figure 3 This is a block diagram of decoder 310 according to an example embodiment. Figure 1 and Figure 2 will also work with Figure 3 They were referenced together.

[0109] Decoder 310 can decode the encoded data ED. In an example embodiment, decoder 310 can decode the data ED by performing the decoding in reverse order. Figure 2 The encoder 210 performs a series of processes to encode the image data ITA to reconstruct the original pixel values ​​of the compressed target pixels or to decode the original pixel values ​​of the compressed target pixels. Therefore, the decoder 310 can reconstruct the image data ITA that has been compressed by the encoder 210.

[0110] In an example embodiment, decoder 310 may use the same protocol as encoder 210. Decoder 310 may use a decoding method based on an algorithm corresponding to the encoding method used by encoder 210. According to the example embodiment, the relative position of a reference pixel relative to a target pixel, determined by encoder 210, may be preset in decoder 310. For example, decoder 310 may decode the target pixel to be reconstructed by referring to data reconstructed from the predetermined position of the reference pixel.

[0111] In the example embodiment, the decoder can reconstruct image data IDTA that has been compressed in units of pixel groups. A pixel group is the processing unit used by the encoder 210. For example, when the encoder 210 uses pixel groups having a 1×8 array comprising eight pixels arranged in rows, the decoder 310 can also use pixel groups having a 1×8 array.

[0112] Decoder 310 may include a decoding detector 311, a mode determiner 313, a decompressor 315, a decoding reconstructor 317, and a decoding reference buffer 319. Decoder 310 may also include a central processing unit (CPU), which typically controls the decoding detector 311, mode determiner 313, decompressor 315, decoding reconstructor 317, and decoding reference buffer 319. Each of the decoding detector 311, mode determiner 313, decompressor 315, decoding reconstructor 317, and decoding reference buffer 319 can be operated by its respective processor, and the processors can operate mutually and organically, allowing decoder 310 to operate as a whole. The decoding detector 311, mode determiner 313, decompressor 315, decoding reconstructor 317, and decoding reference buffer 319 may be controlled by a processor external to decoder 310. Because Figure 3 Each of the decoder 310's decoder detector 311, decoder reconstructor 317, and decoder reference buffer 319 is configured to perform a decoding operation with the decoder. Figure 2 The encoder 210 has a similar function to the encoder detector 211, encoder reconstructor 217 and encoder reference buffer 219, or performs the inverse operation of the algorithm performed by the encoder detector 211, encoder reconstructor 217 and encoder reference buffer 219, so redundant descriptions will be omitted.

[0113] The decoding detector 311 can detect groups of pixels that have been used as encoding units in the encoded data ED. In an example embodiment, the decoding detector 311 can detect groups of pixels as decoding units in the encoded data ED in the form of a bitstream having binary code.

[0114] The mode determiner 313 can decode the header of the bitstream and identify the compression mode, compression method, compression ratio, and loss information. According to an example embodiment, as a result of decoding the header, the mode determiner 313 can identify that DPCM has been used as the compression method and Figure 1 In the example embodiment, the saturated pixel SP is not compressed in a saturated mode, which has been used as a compressed mode. In the example embodiment, the mode determiner 313 can identify that a saturated mode has been used as a compressed mode by checking a saturation flag. In the example embodiment, the mode determiner 313 can determine the number of bit shift operations and the sign of the comparison result from the header, and can determine a method for comparing the target pixel with a reference pixel to generate the reconstructed pixel.

[0115] The decompressor 315 can reconstruct the target pixel based on the determined compression mode, compression method, compression ratio, and loss information. According to an example embodiment, the decompressor 315 can reconstruct the target pixel by identifying the position of the reference pixel based on predetermined relative position information between the target pixel and the reference pixel, and adding residual information to the pixel value of the reference pixel. According to an example embodiment, the residual information may include relatively important data, and therefore, the reconstruction rate of the target pixel can be improved.

[0116] According to an example embodiment, the decompressor 315 can examine the saturation flag in the encoded data ED and identify the location of the saturated pixels.

[0117] According to an example embodiment, decompressor 315 can decompress pixels in a pixel group that are determined to be at positions not in a saturated pixel. In the example embodiment, decompressor 315 can reconstruct a target pixel by performing addition on the unsaturated pixel and a reference pixel. For example, when the first pixel corresponds to bit 0 in the saturation flag, the pixel value of the reference pixel can be added to the residual information corresponding to the first pixel to reconstruct the target pixel. The reconstructed result can be output as reconstructed data RIDTA.

[0118] According to an example embodiment, when a pixel is determined to be the location of a saturated pixel in a pixel group, the decompressor 315 can skip the reconstruction of the target pixel by adding the pixel value to a reference pixel value. For example, when the second pixel corresponds to bit 1 in the saturation flag, the target pixel can be reconstructed to have a predetermined saturation value. For example, the second pixel can be reconstructed to have a saturation value of 1000 as its pixel value.

[0119] According to an example embodiment, the decompressor 315 can reconstruct saturated pixels into pixel values ​​with predetermined reconstruction values. According to an example embodiment, the threshold used by the decoder 310 may be the same as the threshold used by the encoder 210 to determine saturated pixels in a pixel group. According to an example embodiment, the threshold used by the decoder 310 may be the average of the threshold used by the encoder 210 to determine saturated pixels and the maximum value of the dynamic range of a single pixel. In addition to the values ​​described above, various reconstruction values ​​can be used to improve the reconstruction rate.

[0120] The decoder reconstructor 317 can reconstruct the target pixel to be reconstructed subsequently by rearranging the reconstructed data RIDTA. In an example embodiment, the decoder reconstructor 317 can reconstruct the (N+1)th row based on the reconstructed data RIDTA of the Nth row, where N is a natural number.

[0121] The decoder reconstructor 317 can reconstruct the data corresponding to the reference pixel earlier than the target pixel to be reconstructed. Because the decoder 310 reconstructs the data sequentially line by line, the data corresponding to the reference pixel needs to be reconstructed before the target data to be reconstructed.

[0122] The decoding reference buffer 319 can buffer data corresponding to the first row of encoded data ED. While the data corresponding to the first row of encoded data ED is being buffered, the decoding detector 311 can detect data corresponding to the second row of encoded data ED. When the decoding detector 311 processes the data corresponding to the second row of encoded data ED, the data corresponding to the first row of encoded data ED can be loaded from the decoding reference buffer 319 into the decoding detector 311, and therefore, the corresponding images of the first and second rows of encoded data ED can be determined simultaneously. The decoder 310 can buffer the image data of the first row without immediately decoding the encoded data ED, and can decode the encoded data ED corresponding to the first and second rows simultaneously while decoding the encoded data ED of the second row, and therefore, the power consumed by decoding can be reduced.

[0123] Figure 4 This is a flowchart of an image compression method according to an example embodiment. See also... Figure 1 .

[0124] In operation S110, Figure 1 The encoder 210 can detect saturated pixels in a group of pixels included in image data IDTA. In an example embodiment, the encoder 210 can detect saturated pixels with pixel values ​​exceeding a threshold in a group of pixels that are adjacent to each other and have the same color. The threshold can be 95% of the upper limit of the dynamic range of pixel values ​​that each pixel can have, but is not limited to this. For example, when ten bits are allocated to a pixel value, the threshold can be 1000, but is not limited to this.

[0125] In operation S120, encoder 210 can generate a saturation flag indicating the position of a saturated pixel. Encoder 210 can set the position of a saturated pixel to bit 1 and the position of a non-saturated pixel to bit 0, or it can set the position of a saturated pixel to bit 0 and the position of a non-saturated pixel to bit 1. All four bits can be assigned to the saturation flag, but the embodiment is not limited thereto.

[0126] In operation S130, encoder 210 compresses image data IDTA by comparing each of all pixels except saturated pixels with a reference pixel. The reference pixel may be predetermined. The reference pixel may be compressed before the target pixel. The reference pixel may be located before the target pixel. For example, the reference pixel may be located in a row preceding the target pixel. For example, the reference pixel may be located to the left of the target pixel. For example, the reference pixel may correspond to a pixel value that has been compressed and reconstructed earlier than the target pixel. However, the above-described position of the reference pixel applies to cases where compression is performed sequentially row by row from left to right, and can be changed depending on the compression order and direction. Differential compression can be used as a compression method, but the embodiment is not limited to this. Various compression methods can be used.

[0127] Differential compression can be used. Differential compression compresses data by calculating the difference between the pixel value of a reference pixel and the pixel value of a target pixel. The example embodiment assumes that one of a plurality of pixels is selected as the reference pixel, but the embodiment is not limited thereto. For example, the reference pixel can refer to a dummy pixel having the average pixel value of a plurality of pixels or the median pixel value among the individual pixel values, where the individual pixel values ​​are sorted in ascending or descending order.

[0128] In operation S140, encoder 210 can output a bitstream including a saturation flag, a compression result, and compression information. According to an example embodiment, the compression information can be included in the header field of the bitstream, the saturation flag can be included in the saturation flag field of the bitstream, and the compression result can be included in the residual field of the bitstream.

[0129] According to the example embodiment, whether to output a bitstream can be determined based on the mode signal MODE. For example, encoder 210 can output a bitstream including a saturation flag, compression result, and compression information indicating the saturation mode based on the mode signal MODE indicating a saturation mode. For example, encoder 210 can output a bitstream with a different structure based on the mode signal MODE indicating another compression mode different from the saturation mode.

[0130] According to the example embodiment, encoder 210 can detect two or more saturated pixels. When at least two saturated pixels are present, encoder 210 can output a bitstream corresponding to the saturation mode, regardless of the mode signal MODE.

[0131] Figure 5A and Figure 5B This is a conceptual diagram illustrating the structure of pixels according to an example embodiment. Reference will also be made to... Figure 1 .

[0132] Reference Figure 5A ,Depend on Figure 1The image data IDTA generated by the image sensor 100 can correspond to the pixel information of objects in the Bayer pattern. The image data in the Bayer pattern can be referred to as a Bayer image.

[0133] Image data IDTA may include Bayer pixels 111. Bayer pixels 111 may contain a set of pixels including all red, green, and blue color information. Bayer pixels 111 may include pixels arranged in a Bayer pattern. Bayer pixels 111 may be the basic unit of color information for a portion of a displayed object.

[0134] Bayer pixel 111 may include pixel group 113. In an example embodiment, Bayer pixel 111 may include a red pixel group, two green pixel groups, and a blue pixel group. Bayer pixel 111 may include color information of a portion of a sensed object and may correspond to... Figure 1 It is part of the pixel array 110.

[0135] Pixel group 113 may include multiple sub-pixels 115. Multiple sub-pixels 115 included in a pixel group 113 are generated by passing through the same color filter, so multiple sub-pixels 115 included in a pixel group 113 may have the same color information as each other.

[0136] Subpixels 115 can be arranged in a matrix. Although in Figure 5A The sub-pixels 115 are arranged in a 2×2 matrix, but the embodiment is not limited to this. The sub-pixels 115 can be arranged in an M×N matrix, such as a 3×3 matrix, where M and N are natural numbers.

[0137] In an example embodiment, the green pixel group may include four green sub-pixels Gr1, Gr2, Gr3, and Gr4. Similarly, in an example embodiment, the blue pixel group may include four blue sub-pixels B1, B2, B3, and B4, and the red pixel group may include four red sub-pixels R1, R2, R3, and R4. Since the Bayer pixel 111 conforms to the Bayer pattern, the Bayer pixel 111 may include two green pixel groups. Therefore, the other green pixel group may include four green sub-pixels Gb1, Gb2, Gb3, and Gb4, which are different from the green sub-pixels Gr1, Gr2, Gr3, and Gr4.

[0138] Two green pixels included in Bayer pixel 111 have the same color information as each other, but these two green pixels are physically arranged to process different characteristics and are therefore substantially distinguishable. In the example embodiment, the green sub-pixels Gr1 to Gr4 in the first and second rows of Bayer pixel 111 can be associated with the characteristics of red pixels, and the green sub-pixels Gb1 to Gb4 in the third and fourth rows of Bayer pixel 111 can be associated with the characteristics of blue pixels.

[0139] Because multiple sub-pixels 115 generate sub-pixel signals of the same color, even at low light levels, Figure 1 The electronic device 10 can also generate an electrical signal corresponding to the result of sensing light by adding the sub-pixel signals.

[0140] Reference Figure 5B The green sub-pixels Gr1 to Gr4 in the first and second rows of Bayer pixel 111 can correspond to the reference pixel RP, and the green sub-pixels Gb1 to Gb4 in the third and fourth rows of Bayer pixel 111 can correspond to the target pixel TP to be compressed.

[0141] In the example embodiment, the position of the reference pixel RP can be predetermined. Figure 1 The position of the reference pixel RP predetermined in the encoder 210 can be compared with that in Figure 1 The position of the target pixel TP is the same as that of the pre-determined reference pixel RP in the decoder 310. The decoder 310 can reconstruct the target pixel TP by referring to the data corresponding to the position of the reference pixel RP.

[0142] A reference pixel RP can be compressed before the target pixel TP, and the position of the reference pixel RP can logically and temporally precede the position of the target pixel TP. For example, the reference pixel RP can be located to the left of the target pixel TP. For example, the reference pixel RP can correspond to a pixel value that has been compressed and reconstructed before the target pixel TP. For example, the reference pixel RP can be located in a row preceding the target pixel TP. However, the above-described position of the reference pixel applies to the case where compression is performed sequentially from left to right, and can be changed according to the compression order and direction, and the embodiments are not limited thereto.

[0143] Although it is assumed that one of multiple pixel groups (e.g., pixel group 113) Figure 5A Selected as Figure 5B The reference pixel RP is used, but the embodiments are not limited to this. For example, the reference pixel RP may be a virtual pixel having the average pixel value of the sub-pixels forming a pixel group. For example, the reference pixel RP may be a virtual pixel having the median pixel value among the individual pixel values ​​of the sub-pixels forming a pixel group, wherein the individual pixel values ​​are sorted in ascending or descending order.

[0144] According to an example embodiment, the target pixel TP may include a saturated pixel SP. The saturated pixel SP has a pixel value exceeding a threshold in the dynamic range corresponding to the data size that each sub-pixel may have. For example, each of the sub-pixels Gb2 and Gb3 of the target pixel TP may have a pixel value exceeding the threshold.

[0145] According to the example embodiment, each pixel value of sub-pixels Gb2 and Gb3 can be compressed based on the result of comparing each of the pixel values ​​of sub-pixels Gb2 and Gb3 with the pixel value of reference pixel RP. For example, encoder 210 can calculate the difference between the pixel value of sub-pixel Gb2 and the pixel value of reference pixel RP, and the difference between the pixel value of sub-pixel Gb3 and the pixel value of reference pixel RP, and can output a bitstream as the result of compressing the difference.

[0146] Figure 6 This is a conceptual diagram illustrating the structure of a bitstream in a DPCM. Figure 6 It can be shown Figure 4 The bitstream involved in the compression method. Figure 6 The assumption is that the bitstream is the result of compressing four pixels, each representing data in ten bits.

[0147] Reference Figure 1 , Figure 4 and Figure 6 The bitstream includes a header field, a reference field, and a residual field. According to an example embodiment, four bits can be allocated to the header field to represent header information H. Header information H refers to an encoded set of bits indicating a compression method (e.g., information about compression algorithms such as DPCM or PCM). As a result of allocating four bits to header information H, 2... 4 (=16) segments of compressed information. According to the example embodiment, the decoder 310 may refer to the header information H of the method for compressing subpixels and decode the bitstream using the same method.

[0148] According to the example embodiment, Figure 1 The encoder 210 can set a group of pixels or sub-pixels at a predetermined position as a reference pixel RP. The reference pixel RP can refer to the pixel value of the corresponding pixel, the average pixel value of the sub-pixels forming the pixel group, or the median pixel value among the pixel values ​​of the sub-pixels. The reference pixel RP can be a comparison reference used for data compression.

[0149] According to an example embodiment, four bits can be allocated to the reference pixel RP. Because allocating four bits to the data space for the reference pixel RP while it can have a 10-bit pixel value might be insufficient, the data space R could be inadequate. Therefore, encoder 210 can remove a portion of the data from the reference pixel RP. According to an example embodiment, encoder 210 can remove the lower six bits from the ten bits of the reference pixel RP and include only the higher four bits of the reference pixel RP in the reference field of the bitstream. Depending on the desired performance (e.g., compression ratio, data loss rate, or power consumption of encoder 210), any number of lower bits can be removed from the bits of the reference pixel RP.

[0150] According to the example embodiment, three bits can be assigned to a single residual field RESIDUAL1 to RESIDUAL4. Pixel group (e.g., Figure 5A Pixel group 113 in the image may include four sub-pixels (e.g., Figure 5A The data in the bitstream of the data transmission pixel group 113 can be allocated to sub-pixels 115, and a total of twelve bits 3BIT_1, 3BIT_2, 3BIT_3, and 3BIT_4 can be allocated to the space for a total of four sub-pixels 115. The data included in the residual field can correspond to the difference between the pixel value of the reference pixel RP and the pixel value of the target pixel TP.

[0151] As a result, the total 40 bits of data from the four sub-pixels 115 can be compressed into 16 bits. When the four bits allocated to the header containing the compression information are added to these 16 bits, the 40 bits of data can be compressed into 20 bits, achieving a compression ratio of 50%. However, the embodiments are not limited to this. Depending on the desired performance (e.g., compression ratio, data loss rate, or power consumption), the compressed data can have various sizes, such as 10 bits (corresponding to a 75% compression ratio) and 30 bits (corresponding to a 25% compression ratio).

[0152] Figure 7 to Figure 9 This is a conceptual diagram illustrating the structure of a bitstream with different bit allocations based on the number of saturated pixels SP according to an example embodiment. Figure 7 to Figure 9 The assumption is that the bitstream is caused by compressing four pixels, each of which represents data with ten bits. Each of the four pixels (e.g., first pixel P1, second pixel P2, third pixel P3, and fourth pixel P4) can refer to the average or median pixel value of a subpixel or group of pixels.

[0153] Reference Figure 1 , Figure 5B and Figure 7 , Figure 1 The encoder 210 can encode the first pixel P1, the second pixel P2, the third pixel P3, and the fourth pixel P4. The encoder 210 can aim for a compression rate of 50%, and the total data size of the bitstream to be achieved as a result of compression can be 20 bits.

[0154] Encoder 210 can determine the fourth pixel P4 as Figure 1 The saturated pixel SP. The saturated pixel SP has a pixel value exceeding a threshold due to excessive light reception.

[0155] Encoder 210 Comparable Figure 5B The target pixel TP and Figure 5B Reference pixel RP in the data, and compress it. Figure 1The image data ITA is contained within. According to an example embodiment, encoder 210 can compare a reference pixel RP with each of the unsaturated pixels (e.g., first to third pixels P1, P2, and P3) in the pixel group, excluding the saturated pixel SP (e.g., the fourth pixel P4). For example, encoder 210 can calculate the difference between the pixel value of each of the unsaturated pixels (e.g., first to third pixels P1, P2, and P3) and the pixel value of the reference pixel RP.

[0156] The encoder 210 may include the compression method in a header that is allocated four bits. For example, the header may be expressed as bits 0111.

[0157] Encoder 210 can include the position of the saturated pixel SP (e.g., the fourth pixel P4) in a saturation flag field that is assigned four bits. For example, since only the fourth pixel P4 is a saturated pixel SP, the saturation flag can be expressed as bit 0001.

[0158] The encoder 210 can include the compression result (e.g., the difference) in a residual space that is allocated four bits. For example, the result of compressing the first pixel P1 can be included in the 4-bit residual space P1_1, the result of compressing the second pixel P2 can be included in the 4-bit residual space P2_1, and the result of compressing the third pixel P3 can be included in the 4-bit residual space P3_1.

[0159] According to an example embodiment, encoder 210 may not compare the saturated pixel SP (e.g., the fourth pixel P4) with the reference pixel RP, and may include the result of each of only the unsaturated pixels (e.g., the first to third pixels P1, P2, and P3) in the compressed pixel group 113 in the residual field. Therefore, with Figure 6 Compared to the case where the residual field includes data for four pixels, encoder 210 can increase the amount of data representing unsaturated pixels (e.g., first pixel P1, second pixel P2, or third pixel P3) from three bits to four bits, indicating an increase in dynamic range. Therefore, image quality degradation caused by data loss during compression can be reduced. Furthermore, compared to compression methods with the same compression ratio, images with higher resolution can be obtained.

[0160] Reference Figure 1 , Figure 5B and Figure 8 The encoder 210 can encode the first pixel P1, the second pixel P2, the third pixel P3, and the fourth pixel P4. (See above for reference.) Figure 7 The encoder 210 can target a compression rate of 50%, and the total data size of the bitstream to be achieved as a result of compression can be 20 bits. (Referencing omitted)Figure 7 The given description is redundant.

[0161] Encoder 210 can determine the third pixel P3 and the fourth pixel P4 as saturated pixels SP. Encoder 210 can compare the target pixel TP with the reference pixel RP and compress the image data IDTA.

[0162] According to an example embodiment, encoder 210 can compare a reference pixel RP with each of the unsaturated pixels (e.g., first pixel P1 and second pixel P2) in the pixel group, excluding saturated pixels SP (e.g., third pixel P3 and fourth pixel P4). For example, encoder 210 can calculate the difference between the pixel value of each of the unsaturated pixels (e.g., first pixel P1 and second pixel P2) and the pixel value of the reference pixel RP.

[0163] Encoder 210 can include the position of each of the saturated pixels SP (e.g., third pixel P3 and fourth pixel P4) in a saturation flag field that is assigned four bits. For example, since third pixel P3 and fourth pixel P4 are saturated pixels SP, the saturation flag can be expressed as bit 0011.

[0164] The encoder 210 can include the compression result (e.g., the difference) in a residual space that is allocated six bits. For example, the result of compressing the first pixel P1 can be included in the 6-bit residual space P1_2, and the result of compressing the second pixel P2 can be included in the 6-bit residual space P2_2.

[0165] According to an example embodiment, encoder 210 may not compare saturated pixels SP (e.g., third pixel P3 and fourth pixel P4) with reference pixels RP, and may include the result of each of only unsaturated pixels (e.g., first pixel P1 and second pixel P2) in the compressed pixel group 113 in the residual field. Therefore, with Figure 7 Compared to the case where the residual field includes data for three pixels, encoder 210 can increase the amount of data representing unsaturated pixels (e.g., first pixel P1 or second pixel P2) from four bits to six bits, indicating an increase in dynamic range. Therefore, image quality degradation caused by data loss during compression can be reduced. Furthermore, compared to compression methods with the same compression ratio, images with higher resolution can be obtained.

[0166] According to the example embodiment, the more saturated pixels SP detected, the higher the compression ratio can be and the less the image quality deterioration can be.

[0167] Reference Figure 1 , Figure 5B and Figure 9The encoder 210 can encode the first pixel P1, the second pixel P2, the third pixel P3, and the fourth pixel P4. (See above for reference.) Figure 7 The encoder 210 can target a compression rate of 50%, and the total data size of the bitstream to be achieved as a result of compression can be 20 bits. (Referencing omitted) Figure 7 The given description is redundant.

[0168] Encoder 210 can determine the second pixel P2, the third pixel P3, and the fourth pixel P4 as saturated pixels SP. Encoder 210 can compare the target pixel TP with the reference pixel RP and compress the image data IDTA.

[0169] According to an example embodiment, encoder 210 can compare a reference pixel RP with a non-saturated pixel (e.g., a first pixel P1) in a pixel group, excluding saturated pixels SP (e.g., second pixel P2, third pixel P3, and fourth pixel P4). For example, encoder 210 can calculate the difference between the pixel value of the non-saturated pixel (e.g., first pixel P1) and the pixel value of the reference pixel RP.

[0170] Encoder 210 can include the position of each of the saturated pixels SP (e.g., second pixel P2, third pixel P3, and fourth pixel P4) in a saturation flag field that is assigned four bits. For example, since second pixel P2, third pixel P3, and fourth pixel P4 are saturated pixels SP, the saturation flag can be expressed as bit 0111.

[0171] Encoder 210 can include the compression result (e.g., the difference) in a residual space allocated twelve bits. For example, the result of compressing the first pixel P1 can be included in a 10-bit residual space P1_3 within a 12-bit residual space. An additional two-bit data field can be stored as a pseudo-field not used for compression.

[0172] According to an example embodiment, encoder 210 may not compare saturated pixels SP (e.g., second pixel P2, third pixel P3, and fourth pixel P4) with reference pixels RP, and may include the result of compressing only unsaturated pixels (e.g., first pixel P1) in pixel group 113 in the residual field. Therefore, with Figure 8 Compared to the case where the residual field includes data for two pixels, encoder 210 can increase the amount of data representing unsaturated pixels (e.g., the first pixel P1) from six bits to twelve bits, indicating an increase in dynamic range. Therefore, lossless compression without data loss during compression can be achieved. According to the example embodiment, the more saturated pixels SP detected, the higher the compression ratio and the less image quality degradation.

[0173] Figure 10 This is a conceptual diagram illustrating the structure of pixels according to an example embodiment.

[0174] Reference Figure 10 , Figure 1 The image data IDTA in the image can include Bayer pixels 112, which is a set of pixels that includes all red, green, and blue color information. Bayer pixels 112 can be the basic unit for color information of a portion of the displayed object.

[0175] Bayer pixel 112 may include pixel group 114. In an example embodiment, Bayer pixel 112 may include a red pixel group, two green pixel groups, and a blue pixel group. Bayer pixel 112 may include color information of a portion of a sensed object and may correspond to... Figure 1 It is part of the pixel array 110.

[0176] Pixel group 114 may include multiple sub-pixels 116. Multiple sub-pixels 116 included in a pixel group 114 are generated by passing through the same color filter, so multiple sub-pixels 116 included in a pixel group 114 may have the same color information as each other.

[0177] The subpixels can be arranged in a matrix of 116. Figure 5A The sub-pixel 115 is different in Figure 10 The sub-pixels 116 are arranged in a 3×3 matrix, but the embodiment is not limited to this. It will be understood that the sub-pixels 116 can be arranged in an M×N matrix, where M and N are natural numbers.

[0178] In an example embodiment, the green pixel group may include nine green sub-pixels Gr1, Gr2, Gr3, Gr4, Gr5, Gr6, Gr7, Gr8, and Gr9. Similarly, in an example embodiment, the blue pixel group may include nine blue sub-pixels B1, B2, B3, B4, B5, B6, B7, B8, and B9, and the red pixel group may include nine red sub-pixels R1, R2, R3, R4, R5, R6, R7, R8, and R9. Since the Bayer pixel 112 conforms to the Bayer pattern, the Bayer pixel 112 may include two green pixel groups. Therefore, the other green pixel group may include nine green sub-pixels Gb1, Gb2, Gb3, Gb4, Gb5, Gb6, Gb7, Gb8, and Gb9, which are different from the green sub-pixels Gr1, Gr2, Gr3, Gr4, Gr5, Gr6, Gr7, Gr8, and Gr9.

[0179] In the following text, it is assumed that the first green pixel group, which includes green sub-pixels Gr1, Gr2, Gr3, Gr4, Gr5, Gr6, Gr7, Gr8, and Gr9, is the reference pixel RP', and the second green pixel group, which includes green sub-pixels Gb1, Gb2, Gb3, Gb4, Gb5, Gb6, Gb7, Gb8, and Gb9, is the target pixel TP'.

[0180] Figure 11 This is a conceptual diagram illustrating the structure of a bitstream in a DPCM. Figure 11 Provided as a description and Figure 10 The bitstream corresponding to the pixel structure. Figure 11 The bitstream is assumed to be the result of compressing nine pixels, each representing data in ten bits. For example, it is assumed that a single subpixel includes 10 bits of data, but the embodiment is not limited to this. A single subpixel may include data of different numbers of bits, such as 8 bits, 11 bits, or 12 bits.

[0181] Encoder 210 can allocate data space for the bitstream, allowing 90 bits of data from nine pixels to be compressed into 45 bits. When compressing 90 bits of data into 45 bits, the compression rate is 50%.

[0182] Reference Figure 1 , Figure 10 and Figure 11 The bitstream includes a header field, a reference field, and a residual field.

[0183] According to the example embodiment, four bits can be allocated to the header. As mentioned above, the header information refers to an encoded set of bits indicating the compression method (e.g., information about compression algorithms such as DPCM or PCM), and as a result of allocating four bits to the header information, 2 can be transmitted via the header information. 4 (=16) segments of compressed information.

[0184] A reference pixel can be determined based on the pixel value of one of the multiple sub-pixels of pixel group 114, or based on the pixel values ​​of each sub-pixel. For example, encoder 210 can determine a sub-pixel 116 having an average pixel value or a median pixel value, or a predetermined sub-pixel 116 among multiple sub-pixels 116. Figure 10 The reference pixel RP' in the image.

[0185] According to the example embodiment, five bits can be allocated to the reference field used for the reference pixel RP'. Because allocating five bits to the data space used for the reference pixel RP' while it can have a 10-bit pixel value might result in insufficient data space. Therefore, the encoder 210 can remove a portion of the data from the reference pixel RP'. According to the example embodiment, the encoder 210 can remove the lower five bits from the ten bits of the reference pixel RP' and only include the higher five bits of the reference pixel RP' in the bitstream. As mentioned above, the data removed from the sub-pixel is not limited to this.

[0186] According to the example embodiment, the compression of multiple pixel groups 114 Figure 10 The result of the target pixel TP' to be compressed can be included in the residual field. The target pixel TP' can include nine green subpixels Gb1, Gb2, Gb3, Gb4, Gb5, Gb6, Gb7, Gb8, and Gb9. The bitstream length can be 45 bits. When four bits are allocated to the header and five bits are allocated to the reference pixel RP' in the bitstream, 36 bits can be allocated to the residual field.

[0187] According to the example embodiment, since the target pixel TP' comprises nine green sub-pixels Gb1, Gb2, Gb3, Gb4, Gb5, Gb6, Gb7, Gb8, and Gb9, four bits (4BIT_1, 4BIT_2, ..., 4BIT_9) can be allocated to each of the nine green sub-pixels Gb1, Gb2, Gb3, Gb4, Gb5, Gb6, Gb7, Gb8, and Gb9. For example, the result of compressing the green sub-pixel Gb1 can be included in the first residual field RESIDUAL1, the result of compressing the green sub-pixel Gb2 can be included in the second residual field RESIDUAL2, and the result of compressing the green sub-pixel Gb9 can be included in the ninth residual field RESIDUAL9.

[0188] Therefore, the total 90 bits of data from the nine sub-pixels 116 can be compressed into 41 bits. When the four bits allocated to the header containing the compression information are added to these 41 bits, the 90 bits can be compressed into 45 bits, achieving a compression ratio of 50%. Although it is assumed that the size of the compressed data is 45 bits, the embodiments are not limited to this. Depending on the desired performance (e.g., compression ratio, data loss rate, or power consumption), the compressed data can have various sizes.

[0189] Figure 12 This is a conceptual diagram illustrating the structure of a bitstream generated based on the presence of saturated pixels according to an example embodiment.

[0190] Reference Figure 10 to Figure 12 ,Figure 1 The encoder 210 can encode the first pixel G1 to the ninth pixel G9. The encoder 210 can aim for a compression rate of 50%, and the total data size of the bitstream to be achieved as a result of compression can be 45 bits.

[0191] Encoder 210 can determine the second pixel G2 to the eighth pixel G8 as Figure 1 The saturated pixel SP. The saturated pixel SP has a pixel value exceeding a threshold due to excessive light reception.

[0192] Encoder 210 Comparable Figure 10 The target pixel TP' in Figure 10 Reference pixel RP' in the middle, and compressed Figure 1 The image data ITA is contained within. According to an example embodiment, encoder 210 can compare a reference pixel RP' with each of the unsaturated pixels (e.g., first pixel G1 and ninth pixel G9) in the pixel group, excluding saturated pixels SP (e.g., second pixels G2 to eighth pixels G8). For example, encoder 210 can calculate the difference between the pixel value of each of the unsaturated pixels (e.g., first pixels G1 and ninth pixels G9) and the pixel value of the reference pixel RP'.

[0193] The encoder 210 may include the compression method in a header that is allocated four bits. For example, the header may be expressed as bits 0111.

[0194] Encoder 210 can include the position of each of the saturated pixels SP (e.g., second pixels G2 to eighth pixels G8) in a saturation flag field that is assigned nine bits. For example, the saturation flag can be expressed as bit 011111110.

[0195] The encoder 210 can include the compression result (e.g., the difference) in a residual space allocated 32 bits. For example, the result of compressing the first pixel G1 can be included in a 10-bit residual space P4_1, and the result of compressing the ninth pixel G9 can be included in a 10-bit residual space P4_9. An additional twelve bits of data field can be stored as a pseudo-field not used for compression.

[0196] According to an example embodiment, encoder 210 may not compare saturated pixels SP (e.g., second pixels G2 to eighth pixels G8) with reference pixels RP', and may include the result of each of only unsaturated pixels (e.g., first pixel G1 and ninth pixel G9) in the compressed pixel group 114 in the residual field. Therefore, with Figure 11Compared to the case where the residual field includes data for nine pixels, encoder 210 can increase the amount of data representing unsaturated pixels (e.g., the first pixel G1 or the ninth pixel G9) from four bits to ten bits, indicating an increase in dynamic range. Therefore, lossless compression without data loss during compression can be achieved. According to the example embodiment, the more saturated pixels SP detected, the higher the compression ratio and the less image quality degradation.

[0197] Figure 13A and Figure 13B This is a table of compression information based on the example embodiment. Figure 13A and Figure 13B Provided as a description of a compression mode or method according to the standards recommended by the MIPI Alliance. Also referenced. Figure 1 .

[0198] Reference Figure 13A It can compress image data IDTA in Bayer patterns according to various compression modes. Figure 1 Compression modes may include: pixel-based directional difference (PD) mode, diagonal-based difference (DGD) mode, extended tilt horizontal or vertical difference (eSHV) mode, outlier compensation (OUT) mode, and fixed quantization and no reference (FNR) mode.

[0199] PD mode allows DPCM to be performed on image data with Bayer patterns (IDTA). Depending on the specific implementation algorithm, PD modes can be classified as MODE0, MODE1, MODE2, MODE3, MODE12, and MODE13. Because four bits can be allocated to the header indicating the compression method, the sixteen compression modes can be expressed differently as bits in the header information. For example, MODE0 can be expressed as bit 0000, MODE1 as bit 0001, MODE2 as bit 0010, MODE3 as bit 0011, MODE12 as bit 1100, and MODE13 as bit 1101.

[0200] DGD mode allows DPCM to be performed on image data IDTA with a diagonal structure. Depending on the specific implementation algorithm, DGD mode can be classified as MODE4 (bit 0100), MODE5 (bit 0101), MODE8 (bit 1000), MODE9 (bit 1001), MODE10 (bit 1010), and MODE11 (bit 1011).

[0201] Similarly, the eSHV mode may include MODE14 (bit 1110) and MODE15 (bit 1111), the OUT mode may include MODE7 (bit 0111), and the FNR mode may include MODE6 (bit 0110). According to an example embodiment, MODE7 may be referred to as the OUT mode or the saturation mode, where the OUT mode includes a bad pixel mode for processing bad pixels. Depending on the operating environment, either the saturation mode or the OUT mode can be selected.

[0202] In the example embodiment, Figure 1 The mode selector 230 can sequentially evaluate PD mode, DGD mode, eSHV mode, OUT mode, and FNR mode, and select the optimal mode based on a compression evaluation index such as compression ratio or data loss rate. However, the embodiments are not limited to the mode evaluation order suggested above.

[0203] Reference Figure 13B Image data IDTA can be compressed in various compression modes, in which Bayer pixels (e.g., Figure 5A The pixel group (e.g., Bayer pixel 111) in the Bayer pixel group (e.g., Figure 5A The pixel group 113 in the matrix includes multiple sub-pixels (e.g., four sub-pixels in a 2×2 matrix). The compression modes may include: average-based directional difference (AD) mode, extended horizontal or vertical difference (eHVD) mode, tilt-based difference (OD) mode, extended multi-pixel difference (eMPD) mode, extended horizontal or vertical average difference (eHVA) mode, extended outlier compensation (eOUT) mode, and FNR mode.

[0204] AD mode allows DPCM to be performed on image data ITA, in which pixel group 113 forming the Bayer pattern comprises multiple sub-pixels. Depending on the specific implementation algorithm, AD mode can be classified as MODE0, MODE1, MODE2, and MODE3. Because four bits can be allocated to the header indicating the compression method, the sixteen compression modes can be expressed differently as bits in the header information. For example, MODE0 can be expressed as bit 0000, MODE1 as bit 0001, MODE2 as bit 0010, and MODE3 as bit 0011.

[0205] OD mode allows for the compression of image data IDTA with a diagonal structure. Depending on the specific implementation algorithm, OD mode can be classified as MODE4 (bit 0100) and MODE5 (bit 0101).

[0206] Similarly, eMPD modes may include MODE8 (bit 1000), MODE9 (bit 1001), MODE10 (bit 1010), and MODE11 (bit 1011); eHVD modes may include MODE12 (bit 1100) and MODE13 (bit 1101); eHVA modes may include MODE14 (bit 1110); (e)OUT modes (e.g., eOUT mode or OUT mode) may include MODE15 (bit 1111) and MODE7 (bit 0111); and FNR modes may include MODE6 (bit 0110). According to an example embodiment, MODE7 may be referred to as (e)OUT mode or saturation mode, where (e)OUT mode includes a bad pixel mode for processing bad pixels. Depending on the operating environment, either saturation mode or (e)OUT mode may be selected.

[0207] In an example embodiment, the mode selector 230 may sequentially evaluate AD mode, eHVD mode, OD mode, eMPD mode, eHVA mode, eOUT mode, and FNR mode, and select the optimal mode based on a compression evaluation index such as compression ratio or data loss rate. However, the embodiments are not limited to the mode evaluation order suggested above.

[0208] Figure 14A and Figure 14B This is a block diagram of electronic devices 1a and 1b, respectively including ISP 20a and ISP 20b, according to an example embodiment. (Refer to...) Figure 1 The described electronic device 10 can be applied to Figure 14A and Figure 14B .

[0209] Reference Figure 14A The electronic device 1a may include an image sensor 10a, an ISP 20a, a display device 50a, an AP 30a, a working memory 40a, a storage device 60a, a user interface 70a, and a wireless transceiver 80a. The ISP 20a may be implemented as an integrated circuit separate from the AP 30a. Figure 1 The image sensor 100 in the middle can be operated as Figure 14A The image sensor 10a in the middle, and Figure 1 The ISP 200 in the middle can be used as Figure 14A ISP 20a.

[0210] Image sensor 10a can generate image data (e.g., raw image data) based on received light signals and provide binary data to ISP 20a. AP 30a can be provided as a system-on-a-chip (SoC) that controls all operations of electronic device 1a and runs applications, operating systems, etc. AP 30a can control the operation of ISP 20a and can provide the converted image data generated by ISP 20a to display device 50a or store the converted image data in memory 60a.

[0211] Display device 50a may include any device capable of outputting images. For example, display device 50a may include a computer, mobile phone, and other image output terminals. Display device 50a may be an example of an output device. Other examples of output devices include graphics display devices, computer screens, alarm systems, computer-aided design / computer-aided manufacturing (CAD / CAM) systems, video game stations, smartphone displays, and other types of data output devices.

[0212] Working memory 40a may store programs and / or data processed or executed by AP 30a. Storage 60a may include non-volatile memory such as NAND flash memory or resistive memory. For example, storage 60a may be provided as a memory card, such as a multimedia card (MMC), embedded MMC (eMMC), secure digital card (SD) card, or microSD card. Storage 60a may store data and / or programs related to the execution algorithms controlling the image processing operations of ISP 20a. When image processing operations are performed, data and / or programs may be loaded into working memory 40a.

[0213] User interface 70a may include various devices capable of receiving user input, such as a keyboard, keypad, touch panel, fingerprint sensor, and microphone. User interface 70a can receive user input and provide signals corresponding to the user input to AP 30a. Wireless transceiver 80a may include modem 81a, transceiver 82a, and antenna 83a.

[0214] Reference Figure 14B The electronic device 1b may include an image sensor 10b, an ISP 20b, a display device 50b, an AP 30b, a working memory 40b, a storage device 60b, a user interface 70b, and a wireless transceiver 80b. Figure 1 The image sensor 100 in the middle can be operated as Figure 14B The image sensor 10b in the middle, and Figure 1 The ISP 200 in the middle can be used as Figure 14B ISP 20b.

[0215] AP 30b may include ISP 20b. ISP 20b may be provided as a sub-element of AP 30b, rather than being implemented by separate hardware or a combination of hardware and software. Figure 14B Other components and Figure 14A The components in the text are similar, so redundant descriptions have been omitted.

[0216] Figure 15 This is a schematic block diagram of an electronic device 20 according to an example embodiment. Figure 16 yes Figure 15 Detailed block diagram of camera module 1100b.

[0217] For example, Figure 14A Electronic device 1a or Figure 14B A portion of the electronic device 1b is shown as Figure 15 Electronic device 20 in the middle. Figure 14A or Figure 14B The text shows, but Figure 15 The components omitted in the text may be included in the electronic device 20.

[0218] Reference Figure 15 The electronic device 20 may include a multi-camera module 1100, an AP 1300, and a memory 1400. The memory 1400 can perform operations related to... Figure 14A Working memory 40a or Figure 14B The working memory 40b in the middle has the same function, so redundant descriptions will be omitted.

[0219] Electronic device 20 may use a CMOS image sensor to capture and / or store images of objects, and may include a mobile phone, desktop computer, or portable electronic device. Portable electronic devices may include laptop computers, mobile phones, smartphones, tablet PCs, or wearable devices. Electronic device 20 may include at least one camera module and an AP 1300 for processing image data generated by the camera module.

[0220] The multi-camera module 1100 may include a first camera module 1100a, a second camera module 1100b, and a third camera module 1100c. The multi-camera module 1100 can perform operations related to... Figure 1 The camera module 50 in the example has the same function. Although three camera modules are shown (e.g., first camera module 1100a, second camera module 1100b, and third camera module 1100c), the embodiment is not limited thereto. Various numbers of camera modules may be included in the multi-camera module 1100.

[0221] The following will refer to Figure 16The detailed configuration of the second camera module 1100b is described below. The following description can also be applied to other camera modules, such as the first camera module 1100a and the third camera module 1100c.

[0222] Reference Figure 16 The second camera module 1100b may include a prism 1105, an optical path folding element (OPFE) 1110, an actuator 1130, an image sensing device 1140, an encoder 1145, and a storage device 1150.

[0223] The prism 1105 may include a reflective surface 1107 of light-reflecting material and may change the path of light L incident from the outside.

[0224] According to the example embodiment, prism 1105 can change the path of light L incident along the first direction X to a second direction Y perpendicular to the first direction X. Prism 1105 can rotate the reflective surface 1107 of the light-reflecting material about the central axis 1106 along direction A, or rotate the central axis 1106 along direction B, such that the path of light L incident along the first direction X is changed to a second direction Y perpendicular to the first direction X. In this case, OPFE 1110 can move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0225] In the example embodiment, the maximum rotation angle of prism 1105 in direction A can be less than or equal to 15 degrees in the positive (+)A direction and greater than 15 degrees in the negative (-)A direction, but the embodiment is not limited to this.

[0226] In the example embodiment, prism 1105 can be moved by an angle of approximately 20 degrees in the positive B direction or the negative B direction, or within a range of approximately 10 degrees to approximately 20 degrees or from approximately 15 degrees to approximately 20 degrees. In this case, the angle by which prism 1105 moves in the positive B direction can be the same as or similar to the angle by which prism 1105 moves in the negative B direction (within a difference of approximately 1 degree).

[0227] In some example embodiments, the prism 1105 may move the reflective surface 1107 of the light-reflecting material in a third direction Z parallel to the extension direction of the central axis 1106.

[0228] OPFE 1110 may include, for example, "m" optical lenses, where "m" is a natural number. The "m" lenses can be moved in the second direction Y, and can change the optical zoom ratio of the second camera module 1100b. For example, when the default optical zoom ratio of the second camera module 1100b is Z, by moving the "m" optical lenses included in OPFE 1110, the optical zoom ratio of the second camera module 1100b can be changed to 3Z, 5Z, or greater.

[0229] Actuator 1130 can move OPFE 1110 or optical lens to a position. Hereinafter, OPFE 1110 and optical lens are collectively referred to as optical lens. For example, actuator 1130 can adjust the position of optical lens so that image sensor 1142 is positioned at the focal length of optical lens for accurate sensing.

[0230] Image sensing device 1140 may include image sensor 1142, control logic 1144, encoder 1145, and memory 1146. Image sensor 1142 can use light L provided through an optical lens to sense an image of an object. Figure 16 The image sensor 1142 in the middle is functionally similar to Figure 1 The image sensor 100 is used, so redundant descriptions will be omitted. Control logic 1144 typically controls the operation of the second camera module 1100b. For example, control logic 1144 can control the operation of the second camera module 1100b based on control signals provided via control signal line CSLb.

[0231] Encoder 1145 can encode the sensed image data. Encoder 1145 can perform operations similar to those referenced... Figure 1 to Figure 14B The encoder 210 described has the same function, therefore redundant descriptions will be omitted. (Similar to...) Figure 1 The encoder 210 in the middle is different. Figure 16 Encoder 1145 is not included. Figure 1 The encoder 1145 is located in the ISP 200, but may be included in a camera module (e.g., a second camera module 1100b). Although the encoder 1145 is shown as a functional block, the embodiment is not limited thereto. The encoder 1145 may be executed by control logic 1144 to compress and encode image data.

[0232] The memory 1146 may store information necessary for the operation of the second camera module 1100b, such as calibration data 1147. Calibration data 1147 may include information necessary for the second camera module 1100b to generate image data using externally provided light L. For example, calibration data 1147 may include information about the degree of rotation, information about the focal length, information about the optical axis, etc. When the second camera module 1100b is implemented as a multi-state camera with a focal length that varies with the position of the optical lens, calibration data 1147 may include the focal length value for each position or state of the optical lens and information about autofocus.

[0233] The memory 1150 can store image data sensed by the image sensor 1142. The memory 1150 can be disposed externally to the image sensing device 1140 and can form a stack with the sensor chip of the image sensing device 1140. In an example embodiment, the memory 1150 may include an electrically erasable programmable read-only memory (EEPROM), but the embodiment is not limited thereto.

[0234] Reference Figure 15 and Figure 16 In an example embodiment, among multiple camera modules (e.g., first camera module 1100a, second camera module 1100b, and third camera module 1100c), one camera module (e.g., first camera module 1100a) may include, for example, a color pixel of a Tetra unit, which includes four sub-pixels that are adjacent to each other and share the same color information, and another camera module (e.g., second camera module 1100b) may include, for example, a color pixel of a Nona unit, which includes nine sub-pixels that are adjacent to each other and share the same color information, but the embodiment is not limited thereto.

[0235] In an example embodiment, each of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c may include an actuator 1130. Therefore, the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c may include calibration data 1147, which may be the same or different depending on the operation of the actuator 1130 included in each of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c.

[0236] In an example embodiment, among the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c, one camera module (e.g., the second camera module 1100b) may be a folding lens type including prism 1105 and OPFE 1110, while the other camera modules (e.g., the first camera module 1100a and the third camera module 1100c) may be vertical types excluding prism 1105 and OPFE 1110. However, the embodiments are not limited to this.

[0237] In an example embodiment, one of the camera modules 1100a, 1100b, and 1100c (e.g., 1100c) may include a vertical depth camera that extracts depth information using infrared (IR). In this case, AP 1300 can generate a three-dimensional (3D) depth image by merging image data from the depth camera with image data from another camera module (e.g., 1100a or 1100b).

[0238] In an example embodiment, at least two camera modules (e.g., the first camera module 1100a and the second camera module 1100b) of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c may have different fields of view (FOV). In this case, at least two camera modules (e.g., the first camera module 1100a and the second camera module 1100b) of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c may each have different optical lenses, but the embodiment is not limited to this. For example, the first camera module 1100a of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c may have a lower FOV than the second camera module 1100b and the third camera module 1100c. However, the embodiment is not limited to this. The multi-camera module 1100 may also include a camera module having a higher FOV than the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c.

[0239] In some example embodiments, the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c may have different fields of view from each other. In this case, the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c may each have different optical lenses, but the embodiments are not limited thereto.

[0240] In some example embodiments, the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c may be physically separated from each other. For example, the sensing area of ​​the image sensor 1142 may not be divided and used by the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c, but the image sensor 1142 may be independently included in each of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c.

[0241] AP 1300 may include multiple subprocessors (e.g., first subprocessor 1311, second subprocessor 1312 and third subprocessor 1313), decoder 1320, camera module controller 1330, memory controller 1340 and internal memory 1350.

[0242] AP 1300 can be implemented separately from the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c. For example, AP 1300 and the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c can be implemented in different semiconductor chips.

[0243] Image data generated from each of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c can be provided to a corresponding one of the first subprocessor 1311, the second subprocessor 1312, and the third subprocessor 1313 via a corresponding one of the first image signal lines ISLa, ISLb, and ISLc, which are separate from each other. For example, image data generated from the first camera module 1100a can be provided to the first subprocessor 1311 via the first image signal line ISLa, image data generated from the second camera module 1100b can be provided to the second subprocessor 1312 via the second image signal line ISLb, and image data generated from the third camera module 1100c can be provided to the third subprocessor 1313 via the third image signal line ISLc. Such image data transmission can be performed using, for example, a MIPI-based camera serial interface (CSI), but the embodiments are not limited thereto.

[0244] In an example embodiment, a single sub-processor can be provided for multiple camera modules. For example, with Figure 15 In contrast, the first subprocessor 1311 and the third subprocessor 1313 may not be separate, but may be integrated into a single subprocessor, and image data provided from the first camera module 1100a and the third camera module 1100c may be selected by a selection element (e.g., a multiplexer) and then provided to the integrated subprocessor.

[0245] Decoder 1320 can decode the bitstream provided to each of the first subprocessor 1311, the second subprocessor 1312 and the third subprocessor 1313. Figure 15 The decoder 1320 in the middle can perform with Figure 1The decoder 1320 functions similarly to the decoder 310 in the previous example, therefore redundant descriptions will be omitted. Although the decoder 1320 is shown as a functional block separate from the first subprocessor 1311, the second subprocessor 1312, and the third subprocessor 1313, the embodiment is not limited thereto. The decoder 1320 may be included in each of the first subprocessor 1311, the second subprocessor 1312, and the third subprocessor 1313. For example, the decoder 1320 may decode the bitstream in each of the first subprocessor 1311, the second subprocessor 1312, and the third subprocessor 1313.

[0246] The camera module controller 1330 can provide control signals to each of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c. The control signals generated by the camera module controller 1330 can be provided to a corresponding one of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c via one of the separate control signal lines CSLa, CSLb, and CSLc.

[0247] One of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c (e.g., the second camera module 1100b) may be designated as the main camera based on a mode signal or an image generation signal including a zoom signal, and other camera modules (e.g., the first camera module 1100a and the third camera module 1100c) may be designated as slave cameras. Such designation information may be included in control signals and provided to each of the first camera module 1100a, the second camera module 1100b, and the third camera module 1100c via a corresponding one of the mutually separate control signal lines CSLa, CSLb, and CSLc.

[0248] The camera module can be changed to operate as either a master or slave camera under the control of the camera module controller 1330. For example, when the field of view of the first camera module 1100a is larger than that of the second camera module 1100b and the zoom factor indicates a low zoom ratio, the second camera module 1100b can operate as the master camera, and the first camera module 1100a can operate as the slave camera. When the zoom factor indicates a high zoom ratio, the first camera module 1100a can operate as the master camera, and the second camera module 1100b can operate as the slave camera.

[0249] In an example embodiment, control signals provided from camera module controller 1330 to each of the first camera module 1100a, second camera module 1100b, and third camera module 1100c may include a synchronization enable signal. For example, when the second camera module 1100b is the main camera and the first camera module 1100a and the third camera module 1100c are the slave cameras, camera module controller 1330 may send a synchronization enable signal to the second camera module 1100b. The second camera module 1100b, which provides the synchronization enable signal, may generate a synchronization signal based on the synchronization enable signal and provide the synchronization signal to the first camera module 1100a and the third camera module 1100c via the synchronization signal line SSL. The first camera module 1100a, second camera module 1100b, and third camera module 1100c may synchronize using the synchronization signal and may send image data to AP 1300.

[0250] AP 1300 can store encoded image signals in internal memory 1350 or in external memory 1400. AP 1300 can then read the encoded image signals from internal memory 1350 or memory 1400, decode the encoded image signals, and display image data generated based on the decoded image signals. Memory controller 1340 typically controls internal memory 1350 and memory 1400, causing image data to be stored in or loaded into internal memory 1350 and / or memory 1400.

[0251] According to an exemplary embodiment, at least one of the components, elements, modules, or units (collectively referred to as "units" in this paragraph) represented by the boxes in the accompanying drawings (e.g., Figure 1The mode selector 230, encoder 210, first interface 250, second interface 330, and decoder 310 can be presented as various numbers of hardware, software, and / or firmware structures performing the functions described above. For example, at least one of these units can use a direct circuit structure, such as a memory, processor, logic circuit, lookup table, etc., which can perform the corresponding function under the control of one or more microprocessors or other control devices. Furthermore, at least one of these units can be specifically presented as a module, program, or part of code containing one or more executable instructions for performing a specified logical function, and executed by one or more microprocessors or other control devices. Additionally, at least one of these units can include a processor, microprocessor, etc., performing the corresponding function, such as a central processing unit (CPU), or can be implemented by a processor, microprocessor, etc., performing the corresponding function. Two or more of these units can be combined into a single unit, which performs all the operations or functions of the combined two or more units. Furthermore, at least a portion of the function of at least one of these units can be performed by another of these units. Moreover, although a bus is not shown in the above block diagram, communication between units can be performed via a bus. The functional aspects of the exemplary embodiments described above can be implemented in algorithms that execute on one or more processors. Furthermore, the components represented by blocks or processing steps can employ any number of related techniques used for electronic configuration, signal processing and / or control, data processing, etc.

[0252] Although exemplary embodiments have been shown and described above, it will be apparent to those skilled in the art that modifications and changes may be made without departing from the scope defined by the appended claims and their equivalents.

Claims

1. An image compression method for compressing image data generated by an image sensor, the image compression method comprising: Detect saturated pixels among a plurality of pixels in a pixel group included in the image data, wherein the saturated pixels have pixel values ​​exceeding a threshold, and the plurality of pixels are adjacent to each other and have the same color; Generate a saturation flag indicating the position of the saturated pixel; The image data is compressed by comparing a reference pixel with at least one unsaturated pixel among the plurality of pixels included in the pixel group; and The output includes the saturation flag, the compression result, and the bitstream of the compression method. The compression result included in the bitstream only includes the compression result of the at least one unsaturated pixel.

2. The image compression method according to claim 1, wherein, Generating the saturation flag includes: Set the position of the saturated pixel to bit 1; and Set the position of the at least one unsaturated pixel to bit 0.

3. The image compression method according to claim 1, wherein, The position of the reference pixel is predetermined.

4. The image compression method according to claim 3, wherein, The reference pixel is compressed before the at least one unsaturated pixel.

5. The image compression method according to claim 1, wherein, Compressing the image data includes: obtaining the difference between the unsaturated pixel value of the at least one unsaturated pixel and the reference pixel value of the reference pixel.

6. The image compression method according to claim 5, wherein, The reference pixel value corresponds to the average pixel value of adjacent pixels that have the same color.

7. The image compression method according to claim 5, wherein, The reference pixel value corresponds to the median pixel value of adjacent pixels that have the same color.

8. The image compression method according to claim 1, wherein... The bitstream includes a header field, and The header field includes the compression method.

9. The image compression method according to claim 1, wherein, The threshold is at least 95% of the information represented by the plurality of pixels.

10. The image compression method according to claim 1, wherein, Outputting the bitstream includes determining whether to output the saturation flag and the compression result based on a pattern signal indicating the plurality of pixels included in the image data other than the saturated pixels.

11. The image compression method according to claim 10, wherein, The output of the bitstream also includes: outputting the bitstream based on the detection of at least two saturated pixels, regardless of the mode signal.

12. An encoder for processing image data generated by an image sensor, the encoder comprising a memory and a processor, the memory storing one or more executable instructions, the one or more executable instructions, when executed by the processor, causing the processor to: Detect saturated pixels among a plurality of pixels in a pixel group, wherein the saturated pixels have pixel values ​​exceeding a threshold, and the plurality of pixels are adjacent to each other and have the same color; Generate a saturation flag indicating the position of the saturated pixel; and The image data is compressed by comparing a reference pixel only with at least one unsaturated pixel other than the saturated pixel in the plurality of pixels included in the pixel group.

13. The encoder according to claim 12, wherein, The position of the reference pixel is predetermined.

14. The encoder according to claim 12, wherein, The pixel group consists of four pixels.

15. The encoder according to claim 12, wherein, The pixel group comprises nine pixels.

16. The encoder according to claim 12, wherein, The encoder is further configured to determine whether to compress the image data based on a pattern signal indicating that the plurality of pixels included in the image data, excluding the saturated pixels, are to be compressed.

17. The encoder according to claim 12, wherein, The threshold is at least 95% of the information represented by the plurality of pixels.

18. An electronic device for capturing images, the electronic device comprising: An image sensor, which includes a pixel array and is configured to output image data; An image signal processor includes an encoder, the encoder being configured to: Detect saturated pixels among a plurality of pixels in a pixel group, wherein the saturated pixels have pixel values ​​exceeding a threshold, and the plurality of pixels are adjacent to each other and have the same color; Generate a saturation flag indicating the position of the saturated pixel; The image data is compressed by comparing a reference pixel with at least one unsaturated pixel among the plurality of pixels included in the pixel group; and The output includes the compression result, the saturation flag, and the bitstream of the compression method; as well as An application processor includes a decoder configured to reconstruct the image data by decoding the bitstream. The compression result included in the bitstream only includes the compression result of the at least one unsaturated pixel.

19. The electronic device according to claim 18, wherein, The decoder is further configured to decode the at least one unsaturated pixel based on data reconstructed at the predetermined location of the reference pixel.

20. The electronic device according to claim 18, wherein, The decoder is also configured to reconstruct the saturated pixels to a predetermined saturation value.

Images