Image sensor module, image processing system, and image compression method

Abstract

An image sensor module, an image processing apparatus, and an image compression method are provided. The image compression method includes receiving pixel values of a target pixel group to be compressed in image data and reference values of reference pixels to be used for compressing the target pixel group; generating a virtual reference map by applying an offset value to each of the reference values; compressing the pixel values of the target pixel group based on the virtual reference map; and generating a bitstream including a compression result and compression information based on the virtual reference map.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2020-0088451 filed on July 16, 2020 and Korean Patent Application No. 10-2021-0029046 filed on March 4, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to an image sensor, and more specifically, to an image sensor module, an image processing system, and an image compression method for compressing pixel data in isolated regions. Background Technology

[0004] With increasing interest in high-quality images and videos, the number of sensing pixels in the pixel array contained in image sensors and the size of image data generated by image sensors have increased. Image data can be transmitted to an image processing device where it can be compressed to improve transmission efficiency, and compressed image data can be transmitted to the image processing device. Image data can include various two-dimensional or multi-dimensional image patterns. When compressing pixel data within a specific region of the image pattern, compression methods that can improve compression efficiency and reduce compression loss can be applied. Summary of the Invention

[0005] An image sensor module, an image processing system, and an image compression method are provided that can effectively compress pixel data in isolated regions.

[0006] According to embodiments of this disclosure, an image compression method is provided, comprising: receiving pixel values ​​of a target pixel group to be compressed in image data and reference values ​​of reference pixels to be used for compressing the target pixel group; generating a virtual reference map by applying offset values ​​to each of the reference values; and compressing the pixel values ​​of the target pixel group based on the virtual reference map; and

[0007] A bitstream including compression results and compression information is generated based on a virtual reference map.

[0008] According to another embodiment of this disclosure, an image sensor module is provided, comprising: an image sensor configured to generate image data including a plurality of pixels; an encoder configured to generate compressed data including a plurality of bitstreams by sequentially compressing the image data generated by the image sensor in units of pixel groups, and configured to compress target pixel groups to be compressed according to at least one of a plurality of encoding methods; and

[0009] an interface configured to output the compressed data to an external image processing device, wherein the encoder generates a virtual reference map by applying the offset value to each of the reference values of the reference pixels adjacently arranged with the target pixel group among a plurality of encoding methods, and compresses the target pixel group based on the virtual reference map.

[0010] According to another embodiment of the disclosure, there is provided an image processing system including an image sensor configured to generate image data by sensing a received optical signal, an encoder configured to generate a plurality of bitstreams by sequentially compressing a plurality of pixel groups of the image data, and a decoder configured to reconstruct the image data by decompressing the plurality of bitstreams, wherein the encoder generates a virtual reference map by applying an offset value to each of reference values of reference pixels adjacently arranged with a target pixel group to be compressed, and compresses the target pixel group based on the virtual reference map. BRIEF DESCRIPTION OF DRAWINGS

[0011] Embodiments of the disclosure will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a block diagram illustrating an image processing system according to an embodiment of the disclosure;

[0013] Figure 2 is a mixed diagram illustrating an example of a reference map according to an embodiment of the disclosure;

[0014] Figure 3A is a block diagram illustrating an encoder according to an embodiment of the disclosure;

[0015] Figure 3B is a block diagram illustrating an encoder according to an embodiment of the disclosure;

[0016] Figure 4A is a mixed diagram illustrating an example of a virtual reference map according to an embodiment of the disclosure;

[0017] Figure 4B is a mixed diagram illustrating an example of a virtual reference map according to an embodiment of the disclosure;

[0018] Figure 5A is a conceptual diagram illustrating a compression method according to an embodiment of the disclosure;

[0019] Figure 5B is a conceptual diagram illustrating a compression method according to an embodiment of the disclosure;

[0020] Figure 6A is a conceptual diagram illustrating a bitstream according to an embodiment of the disclosure;

[0021] Figure 6B is a conceptual diagram illustrating a bitstream according to an embodiment of the disclosure;

[0022] Figure 7 is a conceptual diagram illustrating a compression method according to an embodiment of the disclosure;

[0023] Figure 8 is a mixed diagram illustrating an example of image data and a virtual reference mapping according to an embodiment of the disclosure;

[0024] Figure 9 is a conceptual diagram illustrating a compression method according to an embodiment of the disclosure;

[0025] Figure 10 is a flowchart illustrating a method of compressing image data according to an embodiment of the disclosure;

[0026] Figure 11A is a block diagram of an image sensor module according to an embodiment of the disclosure;

[0027] Figure 11B is a block diagram of an image sensor module according to an embodiment of the disclosure;

[0028] Figure 12 is a block diagram of a decoder according to an embodiment of the disclosure;

[0029] Figure 13 is a block diagram of an image processing system according to an embodiment of the disclosure;

[0030] Figure 14 is a table diagram illustrating compression information according to an embodiment of the disclosure;

[0031] Figure 15A is a block diagram of an electronic device including a multi-camera module;

[0032] Figure 15B is a mixed block diagram of a camera module of Figure 15A ; and

[0033] Figure 16 is a block diagram schematically illustrating an electronic device according to an embodiment of the disclosure. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the disclosure will be described in detail with reference to the accompanying drawings.

[0035] Figure 1 illustrates an image processing system according to an embodiment of the disclosure, Figure 2 illustrates a pixel array and image data applied to an image sensor module according to an embodiment of the disclosure.

[0036] The image processing system 10 can sense an image of an object or an object, store the sensed image in a memory and / or process the sensed image and store the processed image in the memory. According to an embodiment, the image processing system 10 can include a digital camera, a digital camcorder, a mobile phone, a tablet, or a portable electronic device. The portable electronic device can include a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital camera, a digital camcorder, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a palm game console, an electronic book, a wearable device, etc. Also, the image processing system 10 can be installed as a component in an electronic device such as a drone or an advanced driver assistance system (ADAS), a vehicle, furniture, a manufacturing facility, a door, various measuring devices, etc.

[0037] Referring to Figure 1 , the image processing system 10 can include an image sensor module 100 and an image processing device 200. In an embodiment, the image sensor module 100 can include an image sensor 110, an encoder 120, and an interface (I / F) 130. In an embodiment, the image sensor module 100 can include a plurality of semiconductor chips. For example, a pixel array PXA (in Figure 2 ) of the image sensor 110 can be integrated on one semiconductor chip, logic circuits of the image sensor 110, the encoder 120, and the interface 130 can be integrated on another semiconductor chip, the plurality of semiconductor chips can be electrically connected to each other through a connection member, or the plurality of semiconductor chips can be stacked to be electrically connected to each other through a via metal passing therethrough. However, the disclosure is not limited thereto, and the image sensor module 100 can include a single semiconductor chip.

[0038] In an embodiment, the image processing device 200 can include an interface (I / F) 210, a memory 220, a decoder 230, and an image signal processor 240.

[0039] The image sensor module 100 can image an external object or an object and generate image data IDT. The image sensor module 100 can include the image sensor 110 capable of converting an optical signal of an object incident thereon through the lens LS into an electrical signal.

[0040] The image sensor 110 can include a pixel array PXA in which a plurality of sensing pixels SPX are two-dimensionally arranged, and output image data IDT including a plurality of pixel values respectively corresponding to the plurality of sensing pixels of the pixel array PXA. Figure 2

[0041] ​The pixel array PXA can include a plurality of row lines, a plurality of column lines, and a plurality of sensing pixels SPX arranged in a matrix form, the plurality of sensing pixels SPX each connected to the row lines and the column lines.

[0042] Each of the plurality of sensing pixels SPX of the pixel array PXA can sense an optical signal of at least one color among a plurality of reference colors. For example, the plurality of reference colors can include red, green, and blue, or can include red, green, blue, and white, and can further include other colors such as ultraviolet or infrared, but is not limited thereto. For example, the plurality of reference colors can include cyan, yellow, green, and magenta. The pixel array PXA can generate a pixel signal including information on the reference color of each of the plurality of sensing pixels SPX.

[0043] For example, as shown in FIG. 1A, the pixel array PXA can include red sensing pixels SPX_R, blue sensing pixels SPX_B, and two types of green sensing pixels SPX_Gr and SPX_Gb. The green sensing pixels arranged in the same row as the red sensing pixels SPX_R can be referred to as first green sensing pixels PX_Gr, and the green sensing pixels arranged in the same row as the blue sensing pixels PX_B can be referred to as second green sensing pixels PX_Gb. Figure 2

[0044] For example, as shown in FIG. 1A, the pixel array PXA can include red sensing pixels SPX_R, blue sensing pixels SPX_B, and two types of green sensing pixels SPX_Gr and SPX_Gb. The green sensing pixels arranged in the same row as the red sensing pixels SPX_R can be referred to as first green sensing pixels PX_Gr, and the green sensing pixels arranged in the same row as the blue sensing pixels PX_B can be referred to as second green sensing pixels PX_Gb.

[0045] For example, as shown in FIG. 1A, the pixel array PXA can include red sensing pixels SPX_R, blue sensing pixels SPX_B, and two types of green sensing pixels SPX_Gr and SPX_Gb. The green sensing pixels arranged in the same row as the red sensing pixels SPX_R can be referred to as first green sensing pixels PX_Gr, and the green sensing pixels arranged in the same row as the blue sensing pixels PX_B can be referred to as second green sensing pixels PX_Gb. Figure 2 Figure 8 ​​). Alternatively, the pixel pattern PT can include red sensing pixels SPX_R arranged in an n x n matrix (where n is an integer of 3 or more), blue sensing pixels SPX_B arranged in an n x n matrix, first green sensing pixels SPX_Gr arranged in an n x n matrix, and second green sensing pixels SPX_Gb arranged in an n x n matrix.

[0046] Image data IDT can be generated based on the pixel signals output from the pixel array PXA. The image data IDT can have a color pattern corresponding to the pixel pattern PT of the pixel array PXA. In one example, when the pixel array PXA has a Bayer pattern, the image data IDT can also have a Bayer pattern. In another example, when the pixel array PXA has a four-pixel pattern, the image data IDT can have a four-pixel pattern, or it can have a Bayer pattern.

[0047] For example, when the pixel array PXA has a four-pixel pattern, one pixel signal can be output from four sensing pixels SPX of the same color included in the pixel pattern PT, or four pixel signals can be output from the four sensing pixels SPX. When one pixel signal is output, the image data IDT can have a Bayer pattern, and when four pixel signals are output, the image data IDT can have a four-pixel pattern as shown in FIG. 4B. Figure 2

[0048] The image data IDT can include red pixels PX_R, blue pixels PX_B, first green pixels PX_Gr, and second green pixels PX_Gb arranged repeatedly. The pixels PX of the image data IDT can indicate data (i.e., pixel data) corresponding to the sensing pixels SPX of the pixel array PXA. The red pixels PX_R, the blue pixels PX_B, the first green pixels PX_Gr, and the second green pixels PX_Gb can correspond to the red sensing pixels SPX_R, the blue sensing pixels SPX_B, the first green sensing pixels SPX_Gr, and the second green sensing pixels SPX_Gb of the pixel array PXA.

[0049] The image data IDT can include a plurality of pixel groups PG, which can be arranged in a matrix order according to a color pattern of the image data IDT, can be set to include a preset number of pixels PX arranged in one direction, or can be set to correspond to the same reference color and include pixels PX adjacent to each other.

[0050] For example, as shown in FIG. 4A, the image data IDT can include a plurality of pixel groups PG, each of which includes a red pixel PX_R, a blue pixel PX_B, a first green pixel PX_Gr, and a second green pixel PX_Gb. Figure 2 ​As illustrated in FIG. 1, when the image data IDT has a four-pixel pattern, the pixel group PG can be set to correspond to the same reference color (e.g., red, blue, green, etc.) and include four pixels PX adjacent to each other. In another example, when the image data IDT has a Bayer pattern, the pixel group PG can be set to include a preset number (e.g., 4) of pixels PX arranged in a matrix.

[0051] Referring back to Figure 1 Each of the plurality of sensing pixels SPX can include at least one light sensing or photoelectric conversion element. The light sensing element can sense light and convert the sensed light into an electrical signal. For example, the light sensing element can be a photodiode, a phototransistor, a photogate, a pinned photodiode (PPD), or a combination thereof.

[0052] Each of the plurality of sensing pixels SPX can include at least one light sensing element and a pixel circuit for outputting a pixel signal corresponding to an electrical signal generated by the light sensing element. For example, the pixel circuit can have a four-transistor structure including a transfer transistor, a reset transistor, an amplification transistor, and a selection transistor. However, the disclosure is not limited thereto, and the pixel circuit can have a one-transistor structure, a three-transistor structure, a four-transistor structure, a five-transistor structure, or a structure in which a plurality of pixels share some transistors. In an embodiment, each pixel circuit can have a memory or an analog-to-digital converter.

[0053] In an embodiment, a plurality of color filters through which optical signals of a specific waveband, such as optical signals of a specific color, are transmitted can be arranged on the plurality of pixels to respectively correspond to the plurality of pixels of the pixel array PXA, and the at least one light sensing element included in the pixel can convert the optical signals transmitted through the corresponding color filter into an electrical signal. Accordingly, each of the plurality of sensing pixels SPX of the pixel array PXA can output at least one pixel signal for at least one assigned reference color. However, the disclosure is not limited thereto, and the at least one light sensing element included in the sensing pixel SPX can also selectively convert optical signals of a specific waveband among incident light beams into an electrical signal.

[0054] In an embodiment, the image data IDT can include raw image data including a plurality of pixel values obtained by digitally-analog converting a plurality of pixel signals output from the pixel array PXA, or can include image data obtained by pre-processing the raw image data.

[0055] The image sensor module 100 can compress the image data IDT using the encoder 120 to increase the data transmission speed, to reduce power consumption according to data transmission, and to efficiently use data storage space, and can transmit the compressed data CDT to the image processing device 200.

[0056] The encoder 120 can receive the image data IDT from the image sensor 110, and compress the image data IDT to generate the compressed data CDT. The compressed data CDT can have a form of an encoded bitstream. Hereinafter, the encoded bitstream can be referred to as a bitstream. The bitstream can include a compression result and compression information (e.g., mode information indicating a compression method).

[0057] The encoder 120 can generate the compressed data CDT by encoding the image data IDT in units of the pixel group PG. The encoder 120 can generate one bitstream by encoding one pixel group PG, and generate the compressed data CDT based on the bitstream of some or all of the pixel groups PG in the image data IDT. The pixel group PG is compressed by being encoded, and in the following disclosure, encoding can be used in the same meaning as compression.

[0058] The encoder 120 can perform compression by using a reference map generated based on pixel values corresponding to pixels that are compressed earlier than a pixel group PG to be compressed (i.e., a target pixel group). The encoder 120 can compress a pixel value of a target pixel in the target pixel group based on a reference value of at least one reference pixel adjacent to the target pixel. The reference value can be generated based on a pixel value of the reference pixel, for example, the reference value can be a value generated when the pixel value of the reference pixel is compressed and then decompressed.

[0059] It is likely but not necessary that the pixel value of the target pixel and the pixel value of the adjacent reference pixel have values similar to each other. In addition, the pixel values of the target pixels in the target pixel group can likely have values similar to each other. Accordingly, the encoder 120 can compress the target pixel group by using such a differential pulse code modulation (DPCM) method that encodes the target pixels of the target pixel group based on a difference between adjacent pixels (e.g., a difference value between the pixel value of the target pixel and the reference value of the adjacent reference pixel or a difference value between the pixel value of the target pixel and the pixel value of another target pixel in the target pixel group). Accordingly, the compression efficiency (or compression rate) can be improved, and data loss due to compression can be reduced.

[0060] However, the pixel values of the pixels of the target pixel group included in the isolated region in the image data IDT can be significantly different from the pixel values of the reference pixels. In other words, the correlation between the pixel value of the target pixel and the reference value of the reference pixel can be reduced.

[0061] Here, the isolated region indicates a region corresponding to an edge region in at least two directions (e.g., two directions orthogonal to each other) among edge regions of a two-dimensional or multi-dimensional image pattern generated in the image data IDT.

[0062] When there is a large difference between the pixel value of the target pixel and the reference value of the reference pixel, the encoder 120 according to an embodiment of the disclosure can generate a virtual reference map (VRM) by applying an offset value to each of the reference values included in the reference map, and can compress the target pixel group based on the VRM by using a DPCM method or another encoding method. The VRM can include compensated reference values in which the difference between the pixel value of the target pixel and the reference value of the reference pixel is reduced due to the offset value. As such, the encoding method based on the virtual reference map according to an embodiment of the disclosure can be referred to as an offset encoding method. The offset encoding method can be described in detail with reference to Figures 4A-10

[0063] When the encoder 120 compresses the pixel group of the isolated region by using the above-described DPCM method, or when the encoder 120 compresses the pixel group of the isolated region by using a method of performing encoding based on some of the upper data bits among the plurality of data bits representing the pixel value of each target pixel, a large amount of data loss can occur. Accordingly, the quality of the reconstructed image data generated by decompressing the compressed data CDT can be reduced, and artifacts can occur in the image data. However, as described above, when the encoder 120 compresses the pixel group in the isolated region by using the offset encoding method according to an embodiment of the disclosure, the compression efficiency can be improved, and the data loss can be reduced.

[0064] The encoder 120 can provide the compressed data CDT to the image processing apparatus 200 through the interface 130. For example, the interface 130 can include a camera serial interface (CSI) based on a mobile industry processor interface (MIPI). However, the interface 130 is not limited thereto, and can be implemented according to various protocol standards.

[0065] The image processing apparatus 200 can generate an image to be displayed on a display (not shown) by converting the compressed data CDT received from the image sensor module 100. The image processing apparatus 200 can receive the compressed data CDT from the image sensor module 100, decompress the compressed data CDT to generate decompressed data DDT (e.g., reconstructed image data), and perform image processing on the decompressed data DDT.

[0066] In an embodiment, the image processing apparatus 200 can receive the compressed data CDT from the image sensor module 100 through the interface 210. The interface 210 can include an MIPI (such as the interface 130 included in the image sensor module 100), but is not limited thereto. The image processing apparatus 200 can store the received compressed data CDT in the memory 220.

[0067] ​The memory 220 is a storage location for storing data. The compressed data CDT can be stored in the memory 220. In addition, the memory 220 can store other data, for example, an operating system (OS), various programs, and various data (for example, the compressed data CDT). The memory 220 can include a volatile memory such as a dynamic random access memory (DRAM) or a static RAM (SRAM) or a non-volatile memory such as a phase change RAM (PRAM), a resistive RAM (ReRAM), and a magnetic RAM (MRAM) flash memory. In addition, although Figure 1 Although it is shown that the memory 220 is included in the image processing apparatus 200, the present disclosure is not limited thereto, and the memory 220 can be separately provided outside the image processing apparatus 200.

[0068] The decoder 230 can read the compressed data CDT from the memory 220 and decompress the compressed data CDT to generate the decompressed data DDT. The decoder 230 can provide the decompressed data DDT to the image signal processor 240.

[0069] The decoder 230 can decompress the compressed data CDT in units of the pixel group PG by using a decompression method (or a decoding method) according to a compression method (or an encoding method) performed by the encoder 120 of the image sensor module 100. In this case, the decoder 230 can determine the compression method applied to the pixel group PG based on the compression information included in the bitstream of the compressed data CDT. The decoder 230 can decompress a target pixel of a target pixel group based on a reference map including a reference value corresponding to a pixel (that is, a reference pixel) that is decompressed earlier than the target pixel group to be decompressed.

[0070] In an embodiment, the decoder 230 can generate a virtual reference map (VRM) by applying an offset value to each of the reference values of the reference pixels, and can decompress the target pixel group based on the VRM. In this case, the offset value can be equal to the offset value used by the encoder 120.

[0071] The image signal processor 240 can perform various image processing on the received decompressed data DDT. In a non-limiting example, the image signal processor 240 can perform at least one of bad pixel correction, offset correction, lens distortion correction, color gain correction, shading correction, gamma correction, denoising, and sharpening among the image processing on the decompressed data DDT. In an embodiment, some of the above-described image processing can be omitted according to the performance of the image sensor module 100. For example, when the image sensor module 100 includes an image sensor 110 having high quality, bad pixel correction (in particular, static bad pixel correction), offset correction, and the like among the image processing can be omitted.

[0072] Each of the encoder 120 and decoder 230 can be implemented by software, hardware, or a combination of software (such as firmware) and hardware. When the encoder 120 and decoder 230 are implemented in software, each of the above-described functions can be implemented using programmed source code to be stored in the storage medium included in each of the image sensor module 100 and the image processing device 200. The functions of the encoder 120 and decoder 230 can be executed by a processor (e.g., an image processing processor) included in each of the image sensor module 100 and the image processing device 200. When the encoder 120 and decoder 230 are implemented in hardware, the encoder 120 and decoder 230 can include logic circuitry and registers to perform each of the above-described functions based on register settings.

[0073] In addition, although Figure 1 The image processing system 10 shown includes an image sensor module 100 and an image processing device 200, but this disclosure is not limited thereto. For example, the image processing system 10 may include a portion of the image sensor module 100 and the image processing device 200 without necessarily including all of them, or it may include multiple image sensor modules 100. Furthermore, although... Figure 1 The decoder 230 and image signal processor 240 are shown to have separate configurations, but this disclosure is not limited thereto. For example, the image signal processor 240 may include the decoder 230.

[0074] Figure 3A and Figure 3B An encoder according to an embodiment of the present disclosure is shown. Figure 3A and Figure 3B It shows Figure 1 Example of encoder 120.

[0075] Reference Figure 1 and Figure 3A The encoder 120 may include a reference pixel detector 121, a compression circuit 122, a mode selector 123, a reconstructed image generator 124, and a reference buffer 125.

[0076] Reference pixel detector 121 can be from Figure 1 Image sensor 110 receives image data IDT and can receive a reference map from reference buffer 125 including reference values ​​of reference pixels for compressing the target pixel group. Reference pixel detector 121 can detect reference values ​​of reference pixels adjacent to the target pixel group from the reconstructed image data (i.e., the reconstructed pixel values ​​of the reference pixels) stored in reference buffer 125, and can receive the reference values ​​from reference buffer 125 as a reference map. Reference pixel detector 121 can provide the reference map of the target pixel group and image data IDT to compression circuit 122.

[0077] The compression circuit 122 can compress the target pixel group based on the reference map. The compression circuit 122 can include an offset mode compressor OMC and a normal mode compressor NMC, which can be arranged in parallel to each other without being limited thereto. The offset mode compressor OMC and the normal mode compressor NMC can compress the target pixel group by using different compression or encoding methods, such as by performing these methods in parallel to each other but not limited thereto.

[0078] As described above with reference to Figure 1 The offset mode compressor OMC can encode the target pixel group based on the VRM by using an offset encoding method. For example, the offset mode compressor OMC can generate the VRM by applying an offset value to the reference values of the reference map, and can encode the target pixel group based on the VRM by using a DPCM method. After compressing the target pixel group, the VRM can be removed.

[0079] The normal mode compressor NMC can encode the target pixel group based on the reference map by using a DPCM method. The normal mode compressor NMC can output first encoded data EDT1, and the offset mode compressor OMC can output second encoded data EDT2.

[0080] Although Figure 3A Although it is shown that the compression circuit 122 includes the offset mode compressor OMC and the normal mode compressor NMC, the disclosure is not limited thereto, and the compression circuit 122 can further include a compressor for encoding the target pixel group by using an encoding method different from the offset mode compressor OMC and the normal mode compressor NMC.

[0081] The mode selector 123 can select the compressed data (e.g., one of the first encoded data EDT1 and the second encoded data EDT2) received from the compression circuit 122, and can output the selected encoded data as the compressed data CDT.

[0082] The mode selector 123 can decode the first encoded data EDT1 and the second encoded data EDT2 according to the encoding method by using a decoding method, and can select one of the first encoded data EDT1 and the second encoded data EDT2 according to the decoding based on an error rate. The error rate refers to a difference between the decoded data (i.e., the decoded pixel value) and the pixel value that is not decoded, and the smaller the difference, the lower the error rate. As the error rate decreases, the quality of the decompressed data DDT generated by the image processing apparatus 200 (i.e., the quality of the reconstructed image data) decreases. Accordingly, the mode selector 123 can select the encoded data having a low error rate among the first encoded data EDT1 and the second encoded data EDT2 as the compressed data CDT, and can output the compressed data CDT.

[0083] In an embodiment, the normal mode compressor NMC (or another compressor) can generate encoding data (e.g., first encoding data EDT1) by encoding the target pixel group, and the offset mode compressor OMC can operate when an error rate of the first encoding data EDT1 exceeds a reference error rate. The offset mode compressor OMC can generate second encoding data EDT2 by encoding the target pixel group. When the error rate of the first encoding data EDT1 is less than or equal to the reference error rate, the mode selector 123 can output the first encoding data EDT1 as the compressed data CDT. When the error rate of the first encoding data EDT1 exceeds the reference error rate, the mode selector 123 can output the second encoding data EDT2 received from the offset mode compressor OMC as the compressed data CDT.

[0084] The reconstructed image generator 124 can generate reconstructed image data by decoding the compressed data CDT. The reconstructed image generator 124 can reconstruct pixel values of the target pixel group by decoding each of a plurality of bitstreams included in the compressed data CDT using a decoding method corresponding to the encoding method. Pixels corresponding to the reconstructed pixel values can be used as reference pixels of another target pixel group to be compressed.

[0085] The reference buffer 125 can store the reconstructed image data and can provide a reference value for compressing a reference pixel of the target pixel group to the reference pixel detector 121. In an embodiment, the reference buffer 125 can include a line memory and can store reference pixels positioned around a target pixel of the target pixel group. In an embodiment, the reference buffer 125 can include a volatile memory such as a DRAM or an SRAM. In addition, the disclosure is not limited thereto, and the reference buffer 125 can include a non-volatile memory such as a ReRAM or a PRAM.

[0086] Referring to Figure 3B , the encoder 120a can include the reference pixel detector 121, the pre-detector 126, the compression circuit 122, the mode selector 123, the reconstructed image generator 124, and the reference buffer 125. Compared to the encoder 120 of Figure 3A , the encoder 120a can further include the pre-detector 126. The operations of the reference pixel detector 121, the compression circuit 122, the mode selector 123, the reconstructed image generator 124, and the reference buffer 125 have been described with reference to Figure 3A , and thus, a redundant description thereof is omitted.

[0087] The pre-detector 126 can enable or disable the offset mode compressor OMC. In an embodiment, the pre-detector 126 can enable or disable the offset mode compressor OMC based on a reference value of a reference pixel. For example, when a difference between the reference value and a pixel value of a target pixel is greater than or equal to a threshold value (or a certain code value), the offset mode compressor OMC can be enabled, and when the difference is less than the threshold value, the offset mode compressor OMC can be disabled. In an embodiment, the pre-detector 126 can include a register, and can enable or disable the offset mode compressor OMC based on a control signal stored in the register. For example, the control signal can be received from the image processing apparatus 200.

[0088] When the offset mode compressor OMC is disabled, another compressor (such as the normal mode compressor NMC) included in the compression circuit 122 can encode the target pixel group, and can output the encoded data (i.e., the first encoded data EDT1) as the compressed data CDT.

[0089] When the offset mode compressor OMC is enabled, at least some of the compressors included in the compression circuit 122 and the offset mode compressor OMC respectively encode the target pixel group, and the mode selector 123 can output the encoded data having the lowest error rate among the encoded data as the compressed data CDT.

[0090] In an embodiment, even when the offset mode compressor OMC is enabled, a priority of the offset mode compressor OMC among the compressors included in the compression circuit 122 can be lowered. For example, the normal mode compressor NMC first encodes the target pixel group to generate the first encoded data EDT1, and the offset mode compressor OMC can operate when an error rate of the first encoded data EDT1 exceeds a reference error rate. The offset mode compressor OMC can generate the second encoded data EDT2 by encoding the target pixel group. When the error rate of the first encoded data EDT1 is less than or equal to the reference error rate, the mode selector 123 can output the first encoded data EDT1 as the compressed data CDT. When the error rate of the first encoded data EDT1 exceeds the reference error rate, the mode selector 123 can output the second encoded data EDT2 received from the offset mode compressor OMC as the compressed data CDT.

[0091] In an embodiment, when the offset mode compressor OMC is enabled, another compressor (e.g., the normal mode compressor NMC) included in the compression circuit 122 is disabled, and the second encoded data EDT2 generated by the offset mode compressor OMC can be output as the compressed data CDT. Accordingly, power consumption of the encoder 120 can be reduced.

[0092] Figure 4A and Figure 4BAn example of reference mapping and VRM according to an embodiment of the disclosure is illustrated. In Figure 4A and Figure 4B In, the numbers in the brackets indicate the pixel values (i.e., pixel values, reference values, or compensated reference values).

[0093] Referring to Figure 4A , the image data IDT can include a plurality of pixel groups PG, and the plurality of pixel groups PG can be sequentially compressed according to a set direction. In an embodiment, the image data IDT can be sequentially compressed in units of pixel groups PG in a left-to-right direction and an up-to-down direction. However, the disclosure is not limited thereto, and the image data IDT can be sequentially compressed in a right-to-left direction or a down-to-up direction.

[0094] A target pixel group TG, such as target pixels T0, T1, T2, and T3 of the target pixel group TG, can be compressed. The target pixel group TG can be compressed based on pixels in pixel groups PG corresponding to the same color among adjacent compressed pixel groups PG. The adjacent pixels for compressing the target pixel group TG can be referred to as reference pixels.

[0095] Reference values of the reference pixels adjacent to the target pixel group TG among the reconstructed image data RIDT stored in the reference buffer 125 can be generated as the reference mapping RM. Here, the reference value refers to a value generated when a pixel value is compressed and then decompressed. For example, the reference mapping RM can include reference values of the reference pixels (e.g., R11, R12, R13, R14, R21, R22, R23, R24, R31, R32, R33, and R34).

[0096] The target pixel group TG can be compressed based on the reference mapping RM. For example, the target pixel group TG or the target pixels T0 of the target pixel group TG can be encoded based on more relevant reference values among the reference values of the adjacent reference pixels R13, R14, R23, R24, R32, and R34 or the reference values of the reference pixels adjacent to each other in a preset direction.

[0097] In addition, the difference between the pixel values of the target pixels T0, T1, T2, and T3 of the target pixel group TG and the reference values of the reference pixels can be large. For example, the pixel value of the target pixel T0 is 283, the reference values of the adjacent reference pixels R14, R23, and R32 are 137, 148, and 127, respectively, and when the threshold value is set to 125, the difference between the pixel value and the reference value is greater than or equal to 125, and thus, the target pixel group TG can correspond to an isolated region. The target pixel group TG can be compressed based on the VRM in Figure 4B For example, the offset mode compressor OMC of FIG. 3 can generate the VRM based on the reference mapping RM and compress the target pixel group TG based on the VRM.

[0098] Referring to Figure 4B VRM can be generated by applying an offset value to each of the reference values of the reference pixels (e.g., R11, R12, R13, R14, R21, R22, R23, R24, R31, R32, R33, and R34). It is assumed that the offset value is 128. The offset value can be a positive value or a negative value.

[0099] VRM can include a compensated reference value, which can be generated by adding an offset value to each of the reference values of the reference pixels (e.g., R11, R12, R13, R14, R21, R22, R23, R24, R31, R32, R33, and R34). The pixel value of the target pixel can be similar to the compensated reference value, and the correlation between the pixel value and the compensated reference value can be improved.

[0100] As described above, when the target pixel group TG corresponds to the isolated region and has a low correlation with the reference map RM, the offset mode compressor OMC can generate a VRM having a high correlation by applying an offset value to the reference map RM, and can compress the target pixel group TG based on the VRM.

[0101] Figure 5A and Figure 5B A compression method according to an embodiment of the disclosure is illustrated. The compression method to be described in the disclosure can be performed by an offset mode compressor OMC of Figure 3A and Figure 3B and can be performed based on a DPCM method.

[0102] Referring to Figure 5A and Figure 5B , the offset mode compressor OMC can compress the target pixels T0, T1, T2, and T3 of the target pixel group TG based on a compensated reference value of at least one reference pixel included in the VRM.

[0103] Referring to Figure 5A , the reference pixel R32 of the VRM can be used to compress the target pixel group TG. For example, the reference pixel R32 can correspond to the same color as the target pixel group TG, and can be a reference pixel included in a pixel group immediately before which compression is performed.

[0104] The offset mode compressor OMC can calculate a difference value d0 between the compensated reference value of the reference pixel R32 and the pixel value of the target pixel T0. For example, the compensated reference value 255 can be subtracted from the pixel value 283, and the difference value d0 can be 28. In this case, when the pixel value is less than the compensated reference value, the difference value d0 can have a negative value. Alternatively, the difference value d0 can be generated by subtracting the pixel value from the compensated reference value.

[0105] The offset mode compressor OMC can calculate the difference values d1, d2, and d3 between the target pixels T0, T1, T2, and T3. The offset mode compressor OMC can generate the bitstream BS for the target pixel group TG based on the difference value d0 between the compensated reference value of the reference pixel R32 and the pixel value of the target pixel T0 and the difference values d1, d2, and d3 between the target pixels T0, T1, T2, and T3.

[0106] Referring to Figure 5B , the plurality of reference pixels (e.g., the reference pixels R31, R32, R33, and R34) of the VRM can be used to compress the target pixel group TG. For example, the reference pixels R31, R32, R33, and R34 can correspond to the same color as the target pixel group TG, and can be reference pixels included in a pixel group immediately before which compression is performed.

[0107] The offset mode compressor OMC can calculate the difference values d0, d1, d2, d3, and d4 between the compensated reference values of the reference pixels R31, R32, R33, and R34 and the pixel values of the target pixels T0, T1, T2, and T3 of the target pixel group TG.

[0108] The offset mode compressor OMC can calculate the bitstream BS for the target pixel group TG based on the difference values d0, d1, d2, d3, and d4 between the compensated reference values of the reference pixels R31, R32, R33, and R34 and the pixel values of the target pixels T0, T1, T2, and T3.

[0109] The bitstream BS can include a header HD and a data block DB. The header HD can include compression information, for example, mode information including an encoding method (e.g., a DPCM method, an offset mode method, etc.), a compression rate, loss information, etc. used for compression, and the data block DB can include information, for example, a plurality of difference values d0, d1, d2, and d3, according to the pixel values of the target pixels T0, T1, T2, and T3.

[0110] When the bitstream BS is transmitted to the image processing apparatus 200 in Figure 1 as compressed data CDT, Figure 1 the decoder 230 in can determine the compression method (i.e., the encoding method) of the bitstream BS based on the mode information included in the header HD, and can decompress the compressed data CDT by decoding the bitstream BS based on a decompression method (i.e., a decoding method) corresponding to the compression method and the information included in the data block DB.

[0111] Figure 6A and Figure 6B A bitstream according to an embodiment of the disclosure is illustrated. Figure 6A and Figure 6BAn embodiment example of the bitstream BS of FIG. 5 is shown. The image data IDT before the compression (i.e., the pixel values (e.g., T0, T1, T2, and T3) of the target pixels) is represented as data including 10 bits, and since the target pixel group TG is compressed at a compression rate of 50%, it is assumed that the bitstream BS is data including 20 bits.

[0112] Referring to Figure 6A and Figure 6B The data blocks DB of the bitstreams BSa and BSb can be allocated to the first to sixteenth bits B0 to B15, and the header HD can be allocated to the seventeenth to twentieth bits B16 to B19.

[0113] As described above, the header HD can include mode information. For example, since 4 bits are allocated to the header HD, the header HD can include 2 4 (= 16) pieces of mode information.

[0114] The data blocks DB can include a plurality of residual regions, for example, first to fourth residual regions RD0, RD1, RD2, and RD3. For example, the difference values d0, d1, d2, and d3 calculated in FIG. 5 can be included in the first to fourth residual regions RD0, RD1, RD2, and RD3, respectively. In the disclosure, the values (e.g., difference values) included in the bitstream BSa can be represented as negative and positive numbers by a most significant sign bit notation, without being limited thereto. For example, when the difference value is represented as data of four bits, the most significant bit can indicate a sign, and the remaining 3 bits can indicate an absolute value. The difference value can be represented as a 4-bit binary code representing integers from -8 to 7. However, the disclosure is not limited thereto, and a 1's complement method, an absolute value method, etc. can be used.

[0115] In an embodiment, the first to fourth residual regions RD0, RD1, RD2, and RD3 can be allocated to the same number of bits. For example, the fourth residual region RD3 can be allocated to the first to fourth bits B0 to B3, the third residual region RD2 can be allocated to the fifth to eighth bits B4 to B7, the second residual region RD1 can be allocated to the ninth to twelfth bits B8 to B11, and the first residual region RD0 can be allocated to the thirteenth to sixteenth bits B12 to B15.

[0116] Referring to Figure 6BThe data block DB may include multiple residual regions, such as the first to fourth residual regions RD0, RD1, RD2, and RD3, and an additional region SUB. In an embodiment, the first to fourth residual regions RD0, RD1, RD2, and RD3 may be allocated different numbers of bits. For example, the fourth residual region RD3 may be allocated to bits B0 to B2, the third residual region RD2 may be allocated to bits B3 to B5, the second residual region RD1 may be allocated to bits B6 to B9, and the first residual region RD0 may be allocated to bits B10 to B14. Figure 5A and Figure 5B In this context, the difference d0 between the compensation reference value of reference pixel R32 and the pixel value of target pixel T0 can be greater than the differences d1, d2, and d3 between relatively adjacent target pixels T0, T1, T2, and T3. Therefore, the maximum number of bits can be allocated to the first residual region RD0.

[0117] The supplementary region SUB can be assigned to the sixteenth bit B15. The supplementary region SUB can include code indicating the amount of shift required (e.g., the amount of right or left shift) when the differences d0, d1, d2, and d3 included in the first to fourth residual regions RD0, RD1, RD2, and RD3 are decoded. For example, when the code of the supplementary region SUB is "0", this indicates no shift; when the code of the supplementary region SUB is "1", the differences d0, d1, d2, and d3 are set to shift left twice; if the code of the supplementary region SUB (e.g., binary code) is "1" and the difference d0 is "110100", then the code value included in the second residual region RD1 could be "1101". In this example, the most significant bit represents the sign, therefore, no shift occurs.

[0118] When reconstructing the compressed data CDT, the image processing device 200 ( Figure 1 decoder 230 (in Chinese) Figure 1 The decoder 230 can calculate the value by performing two left shift operations on the values ​​read from the first residual region to the fourth residual region RD0, RD1, RD2, and RD3 as differences d0, d1, d2, and d3, taking into account the code of the additional region SUB. For example, the decoder 230 can generate the code value "110100" by performing two left shift operations on "101" obtained by excluding the most significant bit of the code value "1101" included in the second residual region RD1. The decoder 230 can determine -20 as the difference d, representing the code value "110100".

[0119] Reference Figure 6A and Figure 6BAn example of the bitstream BS of FIG. 5 is described. However, the disclosure is not limited thereto, and the number of bits and the structure of the bitstream BS can be changed according to a compression method, a compression rate, etc. In addition, the number of bits allocated to each region (e.g., the header HD and the data block DB) of the bitstream BS or the number of bits allocated to each region (e.g., the residual regions RD0, RD1, RD2, and RD3 and the additional region SUB) of the data block BD can also be changed.

[0120] Figure 7 A compression method according to an embodiment of the disclosure is illustrated. The compression method of Equation 1 can be performed by the offset mode compressor OMC of FIG. 3. Figure 7

[0121] Referring to Equation 1, Figure 7 the offset mode compressor OMC can compress the target pixels T0, T1, T2, and T3 of the target group TG based on the offset reference value of at least one reference pixel included in the VRM. For example, the reference pixel R32 of the VRM can be used to compress the target group TG.

[0122] The offset mode compressor OMC can calculate the average value of the pixel values of the target pixels T0, T1, T2, and T3. Thereafter, the offset mode compressor OMC can calculate the difference value d0 between the offset reference value and the average value of the reference pixel R32. For example, the average value of the target pixels T0, T1, T2, and T3 can be calculated as 342, the offset reference value 255 can be subtracted from 342, and thus the difference value d0 can be 87. When the pixel value is less than the offset reference value, the difference value d0 can have a negative value. The offset mode compressor OMC can calculate the difference values d1, d2, d3, and d4 between the average value and the pixel values of the target pixels T0, T1, T2, and T3. The offset mode compressor OMC can generate the bitstream BSc for the target group TG based on the difference value d0 and the difference values d1, d2, d3, and d4.

[0123] The data block DB can include a plurality of residual regions, for example, the first to fourth residual regions RD0, RD1, RD2, and RD3 and the average region AVE. The difference value d0 can be included in the average region AVE, and the difference values d1, d2, d3, and d4 can be included in the first to fourth residual regions RD0, RD1, RD2, and RD3, respectively.

[0124] Figure 8 An image data and a VRM according to an embodiment of the disclosure are illustrated.

[0125] In Figure 8 ​In an embodiment, the image data IDTa can include a Bayer pattern. In an embodiment, the pixel group PG can be set in units of four pixels arranged in series. In an embodiment, the pixel group PG can include two red pixels and two green pixels, or can include two blue pixels and two green pixels.

[0126] Compression can be performed in units of the pixel group PG, the target pixel group TG can be compressed first, and the target pixel group TG can be compressed based on reference pixels (e.g., R11, R12, R13, R14, R21, R22, R23, R24, R31, R32, R33, and R34) of the pixel group PG corresponding to the same color among the adjacent pixel groups PG. The pixel values of the target pixel group TG and the reference pixels R11, R12, R13, R14, R21, R22, R23, R24, R31, R32, R33, and R34 can have a significant difference, and the target pixel group TG can be determined to correspond to the isolated region. Accordingly, the target pixel group TG can be compressed by using the offset encoding method.

[0127] When the offset value is 128 and the pixel values of the reference pixels are compressed and then decompressed, it is assumed that the pixel values before compression are the same as the pixel values (i.e., reference values) after compression. Figure 3A The offset mode compressor OMC in the VRM can generate the VRG by adding the offset value 128 to each of the reference values of the reference mapping. The reference values of the reference pixels R11, R12, R13, R14, R21, R22, R23, R24, R31, R32, R33, and R34 can become similar to the pixel values of the target pixels T0, T1, T2, and T3. Figure 3A The offset mode compressor OMC in the VRM can compress the target pixel group TG based on the VRM.

[0128] Figure 9 A compression method according to an embodiment of the disclosure is illustrated. The compression method can be performed on the image data IDTa of FIG. 1 by using the VRM. Figure 8 The compression method of FIG. 1 can be performed on the image data IDTa of FIG. 1. Figure 9 The compression method of FIG. 1 can be performed on the image data IDTa of FIG. 1.

[0129] Referring to FIG. 1, Figure 9 , Figure 3A The offset mode compressor OMC in the VRM can compress the target pixels T0, T1, T2, and T3 of the target pixel group TG based on the compensation reference values of at least one reference pixel included in the VRM. For example, the reference pixels R33 and R34 of the VRM can be used to compress the target pixel group TG. The reference pixels having the same color as the target pixels can be used for compression.

[0130] The offset mode compressor OMC can calculate a difference d0 between the compensated reference value of the reference pixel R33 and the target pixel T0, and also calculate a difference d1 between the pixel values of the target pixels T0 and T2. In addition, the offset mode compressor OMC can calculate a difference d2 between the compensated reference value of the reference pixel R34 and the target pixel T1, and calculate a difference d3 between the pixel values of the target pixels T2 and T3. The offset mode compressor OMC can generate a bitstream BS based on the differences d0, d1, d2, and d3. The differences d0, d1, d2, and d3 can be included in the data block DB of the bitstream BS.

[0131] Figure 10 A method of compressing image data according to an embodiment of the disclosure is illustrated. Figure 10 The compression method can include the offset encoding method described above, and can be performed by Figure 3A and Figure 3B the offset mode compressor OMC. The compression method based on the offset encoding method described above can be applied to the present embodiment.

[0132] Referring to Figure 3A and Figure 10 , the offset mode compressor OMC can receive pixel values of a target pixel group and reference pixel values of reference pixels (S110). In other words, the offset mode compressor OMC can receive image data IDT including the target pixel group and a reference map including reference values of the reference pixels. The target pixel group can include target pixels to be compressed as a pixel group to be compressed, and can receive pixel values corresponding to the target pixels. The reference pixels can be neighboring pixels of the target pixels to be used for compressing the target pixel group. The pixel values of the neighboring pixels can be compressed and then reconstructed to generate the reference values.

[0133] The offset mode compressor OMC can generate a VRM by applying an offset value to each of the reference values of the reference pixels (S120). The compensated reference values can be generated by adding a positive offset value or a negative offset value to each of the reference values. The difference between the compensated reference values and the pixel values of the target pixels can be smaller than the difference between the reference values and the pixel values of the target pixels. The VRM can include a plurality of the compensated reference values.

[0134] In an embodiment, a preset offset value can be applied to each of the reference values by the offset mode compressor OMC. In a non-limiting example, when the pixel values or the reference values are represented as data (e.g., binary code data) including 8 bits capable of representing 256 values (e.g., 0 code to 255 code), the offset value can be set to a maximum pixel value, for example, a value (e.g., 127) corresponding to half of 255. The offset value can include 1 bit of data (e.g., MSB) representing a sign and 7 bits of data representing 127. However, the disclosure is not limited thereto, and the offset value can be preset in various values and methods.

[0135] In an embodiment, the offset mode compressor OMC can include determination logic for determining the offset value, and the determination logic can determine the offset value based on the reference value. For example, the offset value can be determined according to a gray scale region including the reference value. If the gray scale region including the reference value corresponds to a low gray scale region, a positive candidate offset value corresponding to the gray scale region among the positive candidate offset values can be determined as the offset value, and if the gray scale region including the reference value is a high gray scale region, a negative candidate offset value corresponding to the gray scale region among the negative candidate offset values can be determined as the offset value. In another example, a candidate offset value corresponding to an actual difference (in other words, a difference between the pixel value of the current target pixel and the reference value of the current reference pixel) among a plurality of candidate offset values set based on a difference between the reference value and the pixel value of the target pixel can be determined as the offset value. In addition thereto, the offset value can be determined according to various methods.

[0136] The offset mode compressor OMC can compress the pixel values (i.e., the pixel values of the target pixels) of the target pixel group based on the VRM (S130). For example, the offset mode compressor OMC can compress the target pixel group according to the compression method (such as an encoding method) described with reference to FIG. 5, Figure 7 and Figure 9 The offset mode compressor OMC can generate a bitstream including the compression result and the compression information (S140). As described above, the bitstream can include a header and a data block, the header can include the compression information, and the data block can include the compression result (e.g., a difference between the reference value of the reference pixel and the pixel value of the target pixel or a difference between the pixel values of the target pixels).

[0137] The offset mode compressor OMC can generate a bitstream including the compression result and the compression information (S140). As described above, the bitstream can include a header and a data block, the header can include the compression information, and the data block can include the compression result (e.g., a difference between the reference value of the reference pixel and the pixel value of the target pixel or a difference between the pixel values of the target pixels).

[0138] In addition, as described with reference to Figure 3A the compression method can further include the steps of generating reconstructed pixel values by decompressing the bitstream after the bitstream is generated, and generating reconstructed image data including a reference pixel to be used for compressing a next target pixel group to be compressed after the target pixel group based on the reconstructed pixel values.

[0139] Figure 11A and Figure 11B An image sensor module according to an embodiment of the disclosure is illustrated.

[0140] Referring to Figure 11A , the image sensor module 100 can include an image sensor 110, processing logic 150, an encoder 120, and an interface (IF) 130.

[0141] The image sensor 110 can include a pixel array PXA and a driving and reading circuit DRC. As illustrated above, the pixel array PXA can include a plurality of pixels PX arranged in a matrix. The driving and reading circuit DRC can control the pixel array PXA and convert a pixel signal received from the pixel array PXA into a pixel value. The driving and reading circuit DRC can generate raw image data RDT including a pixel value corresponding to each of the received pixel signals.

[0142] The processing logic 150 can perform processing on the raw image data RDT. For example, the processing can include image processing such as bad pixel correction, crosstalk correction, noise removal, merging, size change, and color space conversion.

[0143] The encoder 120 can compress the image data IDT (or the raw image data RDT) received from the processing logic 150 to generate compressed data CDT. The encoder 120 can compress the image data IDT in units of a pixel group, and can compress a target pixel group to be compressed by using adjacent pixel groups on which compression has been performed. As described above, when the target pixel group corresponds to an isolated region, the encoder 120 can generate a VRM based on this and compress the target pixel group.

[0144] The compressed data CDT can be provided to the interface 130, which can transmit the compressed data CDT to Figure 1 the image processing apparatus 200.

[0145] Referring to Figure 11B , the image sensor module 100a can further include a memory 160. The memory 160 can include a volatile memory such as DRAM and SRAM or a non-volatile memory such as PRAM, ReRAM, MRAM, and a flash memory. The compressed data CDT generated by the encoder 120 can be stored in the memory 160. The compressed data CDT can be read from the memory 160 and output through the interface 130.

[0146] Figure 12 A decoder according to an embodiment of the disclosure is illustrated.

[0147] The decoder 230 can perform decoding by performingFigure 1 The encoder 120 in the image data IDT encodes and generates a series of processes to decompress the compressed data CDT in the opposite order, for example, the reconstructed image data.

[0148] The decoder 230 can decompress the compressed data CDT by using a decoding method corresponding to the encoding method used by the encoder 120. The decoder 230 can decode the compressed data CDT in units of a bitstream.

[0149] The decoder 230 can include a reference pixel detector 231, a mode decoder 232, a decompressor 233, and a reference buffer 234.

[0150] The reference pixel detector 231 can receive the compressed data CDT, and can receive a reference map to be used for decompressing a target bitstream to be decompressed among the bitstreams included in the image data CDT from the reference buffer 234. The reference map includes neighboring pixels of a pixel group related to a reference target bitstream (i.e., a reference value of a reference pixel).

[0151] The reference pixel detector 231 can detect a reference value of a reference pixel (i.e., a reconstructed pixel value of a reference pixel) adjacent to a target pixel group in a position from the reconstructed image data stored in the reference buffer 225, and can receive the reference value as a reference map from the reference buffer 225. The reference pixel detector 121 can provide the target bitstream and the reference map of the compressed data CDT to the mode decoder 232 or the decompressor 233.

[0152] The mode decoder 232 can decode a header of the bitstream, and determine mode information, a compression rate, loss information, etc. as a decoding result. According to an embodiment of the disclosure, the mode decoder 232 can check that the compression is performed by using an offset encoding method or another encoding method (e.g., a DPCM method) with the decoding result of the header.

[0153] The decompressor 233 can reconstruct a target pixel from the bitstream based on the determined compression mode, compression rate, loss information, etc. According to an embodiment of the disclosure, the decompressor 233 can generate a VRM by applying an offset to the reference map, and decode the bitstream based on the VRM. The decompressor 233 can calculate a compensation reference value by applying an offset value used by the encoder 120 to each of the reference values included in the reference map. The VRM can include the compensation reference value. The decompressor 233 can decode the bitstream by using at least one compensation reference value of the VRM or a compensation reference value for each pixel group. The pixel group generated by decoding the bitstream can be output as decompressed data DDT.

[0154] The reference buffer 234 can store the decompressed data DDT (i.e., the reconstructed image data). In an embodiment, the reference buffer 234 can store a group of pixels corresponding to the next bitstream to be decompressed among the reconstructed image data. In an embodiment, the memory or buffer (e.g., the memory 160) in the image processing apparatus 200 included in Figure 1 may be used as the reference buffer 234.

[0155] Figure 13 An image processing system according to an embodiment of the disclosure is illustrated. Figure 13 A modified embodiment of the image processing system 10 of Figure 1 is illustrated.

[0156] Referring to Figure 13 , the image processing system 10b can include an image sensor module 100b and an image processing apparatus 200b. The image sensor module 100b can include an image sensor 110 and an interface (I / F) 130. The image sensor module 100b can further include a memory. The image processing apparatus 200b can include an interface (I / F) 210, an encoder 250, a decoder 230, an image signal processor 240, and a memory 220. Figure 13 The encoder 250 of Figure 1 may correspond to the encoder 120 of

[0157] Comparing the image processing system 10b of Figure 13 with the image processing system 10 of Figure 1 , the image processing apparatus 200b, not the image sensor module 100b, can include the encoder 250, and the image processing apparatus 200b can compress the image data IDT. The remaining configurations can be substantially similar to each other. Among the configurations of the image processing system 10b, the description of the configurations overlapping with the configurations of the image processing system 10 of Figure 1 is omitted.

[0158] Referring to Figure 13 , the image sensor 110b can generate the image data IDT (raw image data or pre-processed image data). The image data IDT can be transmitted to the image processing apparatus 200b through the interface 130. The image processing apparatus 200b can receive the image data IDT from the image sensor module 100b, compress the image data IDT, and store the compressed data CDT in the memory 260. Thereafter, the decoder 230 can read the compressed data CDT stored in the memory 260 and decompress the compressed data CDT. The decoder 230 can provide the decompressed data CDT (e.g., the reconstructed image data) to the image signal processor 240.

[0159] As shown above, compression and decompression can be performed for each pixel group, and a pixel group corresponding to an isolated region of image data CDT can be compressed or decompressed based on a VRM.

[0160] Figure 14 Compression information according to embodiments of the disclosure is shown. Figure 14 Examples of compression modes or compression methods according to embodiments applicable to a proposed Mobile Industry Processor Interface (MIPI) standard are shown.

[0161] Referring to Figure 14 Image data IDT of a four-pixel pattern can be compressed according to various compression modes. Figure 2 In addition, the disclosure is not limited thereto, and image data in which a red pixel group, a blue pixel group, a first green pixel group, and a second green pixel group each including pixels arranged in an n x n matrix are repeatedly arranged can also be compressed according to various compression modes.

[0162] Compression modes include an Average-based Directional Difference (AD) mode, an Extended Multi-pixel Difference (eMPD) mode, an Extended Horizontal or Vertical Directional Difference (eHVD) mode, an Extended Horizontal or Vertical Average Difference (eHVA) mode, an Orientation Directional (OD) mode, an Extended Outlier Compensation (eOUT) mode, an OUT mode, and a Fixed Quantization and No Reference (FNR) mode. The names of the above-described compression modes represent examples, but the disclosure is not limited to the above-described examples.

[0163] In the AD mode, encoding can be performed on a target pixel group by using a DPCM method. For example, a bitstream (e.g., BSc) can be generated based on a difference between an average value of pixel values of the target pixel group and a reference value of a reference pixel and a difference between each of the pixel values and the average value. Figure 7

[0164] The AD mode can be divided into MODE0, MODE1, MODE2, and MODE3 according to detailed execution algorithms. Since 4 bits can be allocated to a header indicating a compression method, 16 compression modes can each represent header information as different bits. For example, MODE0 can be represented by bits of 0000, MODE1 by bits of 0001, MODE2 by bits of 0010, and MODE3 by bits of 0011.

[0165] In the OD mode, image data IDT having a diagonal line structure can be compressed. The OD mode can be divided into MODE4 (bits of 0100) and MODE5 (bits of 0101) according to detailed execution algorithms.

[0166] ​Similarly, the eMPD mode can include MODE8 (bits of 1000), MODE9 (bits of 1001), MODE10 (bits of 1010), and MODE11 (bits of 1011), the eHVD mode can include MODE12 (bits of 1100) and MODE13 (bits of 1101), the eHVA mode can include MODE14 (bits of 1110), the eOUT mode can include MODE15 (bits of 1111), the OUT mode can include MODE7 (bits of 0111). The FNR mode can include MODE6 (bits of 0110). In an embodiment, MODE7 (bits of 0111) can be included in the eOUT mode according to a value stored in a register.

[0167] According to an embodiment of the disclosure, the offset encoding method can correspond to the eOUT mode, Figure 3A The offset mode compressor OMC in Equation 1 can generate a bitstream including a header indicating the eOUT mode.

[0168] In an embodiment, Figure 3A and Figure 3B The mode selector 123 in Equation 1 can sequentially evaluate the AD mode, the eMPD mode, the eHVD mode, the eHVA mode, the OD mode, the eOUT mode, and the FNR mode, and can select the optimal mode according to a compression evaluation index such as a compression rate and loss information. However, the technical idea of the disclosure is not limited to the given mode evaluation order.

[0169] Figure 15A An electronic device including a multi-camera module is illustrated, Figure 15B is Figure 15A a detailed block diagram of the camera module of

[0170] Referring to Figure 15A , the electronic device 1000 can include a camera module group 1100, an application processor 1200, a power management integrated circuit (PMIC) 1300, and an external memory 1400.

[0171] The camera module group 1100 can include a plurality of camera modules 1100a, 1100b, and 1100c. Although Figure 15A An embodiment in which three camera modules 1100a, 1100b, and 1100c are arranged is illustrated, but the embodiment is not limited thereto. In some embodiments, for example, the camera module group 1100 can include two camera modules, or can include k (where k is a natural number greater than or equal to 4) camera modules.

[0172] Hereinafter, referring to Figure 15BThe detailed configuration of the camera module 1100b is described in more detail, and the following description can be equally applied to other camera modules 1100a and 1100b according to embodiments.

[0173] Referring to Figure 15B , the camera module 1100b can include a prism 1105, an optical path folding element (hereinafter, referred to as "OPFE") 1110, an actuator 1130, an image sensing device 1140, and a storage 1150.

[0174] The prism 1105 can include a reflection surface 1107 of a light-reflecting material, and change a path of light L incident from the outside.

[0175] In some embodiments, the prism 1105 can change a path of light L incident in a first direction X to a second direction Y perpendicular to the first direction X. In addition, the prism 1105 can rotate about a central axis 1106 of the reflection surface 1107 of the light-reflecting material in an A direction or a B direction, thereby changing a path of light L incident in the first direction X to the second direction Y perpendicular to the first direction X. In this case, the OPFE 1110 can also move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0176] In some embodiments, a maximum rotation angle of the prism 1105 in the A direction is less than 15 degrees in the +A direction, and can be greater than 15 degrees in the -A direction, as shown in Figure 15B , but embodiments are not limited thereto.

[0177] In some embodiments, the prism 1105 can move in a range of about 20 degrees, or can move in the +B or -B direction by between 10 degrees and 20 degrees or between 15 degrees and 20 degrees, and the angle of movement can be the same as each other in the +B or -B direction, or can be in a range of 1 degree.

[0178] In some embodiments, the reflection surface 1106 of the light-reflecting material of the prism 1105 can move in a third direction (for example, the Z direction) parallel to an extension direction of the central axis 1106.

[0179] In some embodiments, the camera module 1100b can include two or more prisms, thereby changing a path of light L incident in a first direction X differently to a second direction Y perpendicular to the first direction X, to the first direction X or a third direction Z, and then to the second direction Y.

[0180] The OPFE 1110 can include, for example, optical lenses including m (m is a natural number) groups. The m lenses can be moved in the second direction Y to change the optical zoom ratio of the camera module 1100b. For example, when the base optical zoom ratio of the camera module 1100b is referred to as Z, and when the m optical lenses included in the OPFE 1110 are moved, the optical zoom ratio of the camera module 1100b can become 3Z, 5Z, or more.

[0181] The actuator 1130 can move the OPFE 1110 or the optical lens to a specific position. For example, the actuator 1130 can adjust the position of the optical lens for accurate sensing so that the image sensor 1142 is positioned at the focal length of the optical lens.

[0182] The image sensing device 1140 can include an image sensor 1142, control logic 1144, and a memory 1146. Figure 1 The image sensor module 100b of FIG. 1 Figure 13 The image sensor module 100b of FIG. 1 can be applied as the image sensing device 1140.

[0183] The image sensor 1142 can sense an image of a sensing target by using light L provided through the optical lens. The control logic 1144 can control the operation of the camera module 1100b and process the sensed image. For example, the control logic 1144 can control the operation of the camera module 1100b according to a control signal provided through a control signal line CSLb, and can extract image data corresponding to a specific image in the sensed image (for example, a face, an arm, a leg, etc. of the image), or perform image processing such as noise removal.

[0184] In an embodiment, the control logic 1144 can include Figure 1 the encoder 120 in FIG. 1, and can encode the image that has undergone image processing. As shown above, the encoder 120 can compress the image in units of pixel groups, and can compress the pixel groups in the isolated area according to the offset encoding method.

[0185] The memory 1146 can store information such as calibration data 1147 for the operation of the camera module 1100b. The calibration data 1147 can be information for the camera module 1100b to generate image data by using light L provided from the outside, and can include, for example, information about a degree of rotation, information about a focal length, information about an optical axis, etc. When the camera module 1100b includes a multi-state camera whose focal length changes according to the position of the optical lens, the calibration data 1147 can include information about the focal length value for each position (or state) of the optical lens and about auto focus.

[0186] In some embodiments, the compressed data can be stored in the memory 1146. Also, the memory 1146 can be used as a reference buffer 125 of the encoder 120.

[0187] The storage 1150 can store image data sensed by the image sensor 1142. The storage 1150 can be disposed outside the image sensing device 1140, and can be implemented in a form of a sensor chip stack constituting the image sensing device 1140. In some embodiments, the image sensor 1142 can include a first chip, and the control logic 1144, the storage 1150, and the memory 1146 can include a second chip, such that the two chips can be stacked.

[0188] In some embodiments, the storage 1150 can include an electrically erasable programmable read-only memory (EEPROM), but embodiments are not limited thereto. In some embodiments, the image sensor 1142 can include a pixel array, and the control logic 1144 can include an analog-to-digital converter and an image signal processor for processing a sensed image.

[0189] Referring to Figure 15A and Figure 15B In some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c can include the actuator 1130. Accordingly, the plurality of camera modules 1100a, 1100b, and 1100c can include calibration data 1147 which are the same as or different from each other according to operations of the actuator 1130 included therein.

[0190] In some embodiments, one camera module (e.g., 1100b) among the plurality of camera modules 1100a, 1100b, and 1100c can be a folded lens type camera module including the above-described prism 1105 and the OPFE 1110, and the other camera modules (e.g., 1100a and 1100c) can be vertical type camera modules not including the prism 1105 and the OPFE 1110, but are not limited thereto.

[0191] In some embodiments, one camera module (e.g., 1100c) among the plurality of camera modules 1100a, 1100b, and 1100c can be a vertical shaped depth camera for extracting depth information by using, for example, infrared (IR). In this case, the application processor 1200 can merge image data provided from the depth camera with image data provided from another camera module (e.g., 1100a or 1100b), and provide a three-dimensional (3D) depth image.

[0192] In some embodiments, at least two camera modules (e.g., 1100a and 1100b) among the plurality of camera modules 1100a, 1100b, and 1100c can have different field of view angles. In this case, for example, the optical lenses of at least two camera modules (e.g., 1100a and 1100b) among the plurality of camera modules 1100a, 1100b, and 1100c can be different from each other, but are not limited thereto.

[0193] In addition, in some embodiments, the field of view angles of each of the plurality of camera modules 1100a, 1100b, and 1100c can be different from each other. For example, the camera module 1100a can be an ultra-wide camera, the camera module 1100b can be a wide camera, and the camera module 1100c can be a telephoto camera, but are not limited thereto. In this case, the optical lenses included in each of the plurality of camera modules 1100a, 1100b, and 1100c can also be different from each other, but the present disclosure is not limited thereto.

[0194] In some embodiments, the plurality of camera modules 1100a, 1100b, and 1100c can be physically separated and arranged from each other. That is, the sensing area of one image sensor 1142 is not divided by the plurality of camera modules 1100a, 1100b, and 1100c, and independent image sensors 1142 can be arranged inside each of the plurality of camera modules 1100a, 1100b, and 1100c.

[0195] Referring back to Figure 15A , the application processor 1200 can include an image processing device 1210, a memory controller 1220, and an internal memory 1230. The application processor 1200 can be configured as a semiconductor chip that is separate from the plurality of camera modules 1100a, 1100b, and 1100c, for example.

[0196] The image processing device 1210 can include a plurality of sub-image processors 1212a, 1212b, and 1212c, an image generator 1214, and a camera module controller 1216.

[0197] The image processing device 1210 can include a plurality of sub-image processors 1212a, 1212b, and 1212c corresponding to the plurality of camera modules 1100a, 1100b, and 1100c, respectively.

[0198] Image data generated by the camera module 1100a can be supplied to the sub image processor 1212a through an image signal line ISLa, image data generated by the camera module 1100b can be supplied to the sub image processor 1212b through an image signal line ISLb, and image data generated by the camera module 1100c can be supplied to the sub image processor 1212c through an image signal line ISLc. Such image data transmission can be performed by using, for example, a camera serial interface (CSI) based on a mobile industry processor interface (MIPI), but is not limited thereto.

[0199] In an embodiment, at least one of the plurality of sub image processors 1212a, 1212b, and 1212c can include a decoder 230 in the image processing apparatus 200a. When the camera module 1100a corresponding to the plurality of sub image processors 1212a, 1212b, and 1212c each includes an encoder 120 in the image processing apparatus 200a, the plurality of sub image processors 1212a, 1212b, and 1212c can include the decoder 230 to decompress the compressed image data. Figure 1 Figure 1 In an embodiment, at least one of the plurality of sub image processors 1212a, 1212b, and 1212c can include a decoder 230 in the image processing apparatus 200a. When the camera module 1100a corresponding to the plurality of sub image processors 1212a, 1212b, and 1212c each includes an encoder 120 in the image processing apparatus 200a, the plurality of sub image processors 1212a, 1212b, and 1212c can include the decoder 230 to decompress the compressed image data.

[0200] In some embodiments, the image processing apparatus 200b can include at least one of the plurality of sub image processors 1212a, 1212b, and 1212c, and at least one of the plurality of sub image processors 1212a, 1212b, and 1212c can include an encoder 250 in the image processing apparatus 200b and a decoder 230 in the image processing apparatus 200b. Figure 13 Figure 13 In some embodiments, the image processing apparatus 200b can include at least one of the plurality of sub image processors 1212a, 1212b, and 1212c, and at least one of the plurality of sub image processors 1212a, 1212b, and 1212c can include an encoder 250 in the image processing apparatus 200b and a decoder 230 in the image processing apparatus 200b. Figure 13

[0201] In some embodiments, one sub image processor can be arranged to correspond to a plurality of camera modules. For example, the sub image processor 1212a and the sub image processor 1212c can be integrated into one sub image processor without being separated from each other as shown in the image processing apparatus 200a, and image data provided from the camera modules 1100a and 1100c can be selected by a selection element (e.g., a multiplexer) or the like and then supplied to the integrated sub image processor. In this case, the sub image processor 1212b can receive image data from the camera module 1100b without being integrated into the camera module 1100b. Figure 15A

[0202] ​​​​Further, in some embodiments, image data generated by the camera module 1100a can be supplied to the sub-image processor 1212a through an image signal line ISLa, image data generated by the camera module 1100b can be supplied to the sub-image processor 1212b through an image signal line ISLb, and image data generated by the camera module 1100c can be supplied to the sub-image processor 1212c through an image signal line ISLc. Further, image data processed by the sub-image processor 1212b can be directly supplied to the image generator 1214, and image data processed by the sub-image processors 1212a and 1212c can be selected by a selection element (e.g., a multiplexer) or the like and then supplied to the image generator 1214.

[0203] Each of the sub-image processors 1212a, 1212b, and 1212c can perform image processing such as bad pixel correction, 3A adjustment such as auto focus correction, auto white balance, and auto exposure, noise reduction, sharpening, gamma control, and remosaicking for image data supplied from the camera modules 1100a, 1100b, and 1100c.

[0204] In some embodiments, remosaicking signal processing can be performed for each of the camera modules 1100a, 1100b, and 1100c, and then the results of the remosaicking signal processing can be supplied to the sub-image processors 1212a, 1212b, and 1212c.

[0205] Image data processed by each of the sub-image processors 1212a, 1212b, and 1212c can be supplied to the image generator 1214. The image generator 1214 can generate an output image by using image data supplied from each of the sub-image processors 1212a, 1212b, and 1212c in accordance with image generation information or a mode signal.

[0206] The image generator 1214 can generate an output image by merging at least some of image data generated by the image processors 1212a, 1212b, and 1212c in accordance with image generation information or a mode signal. Further, the image generator 1214 can generate an output image by selecting any one of image data generated by the image processors 1212a, 1212b, and 1212c in accordance with image generation information or a mode signal.

[0207] In some embodiments, the image generation information can include a zoom signal or a zoom factor. Further, in some embodiments, the mode signal can be a signal based on a mode selected by a user, for example.

[0208] When the image generation information is a zoom signal (zoom factor), and when the camera modules 1100a, 1100b, and 1100c have different fields of view (viewing angles), the image generator 1214 can perform different operations according to the type of the zoom signal. For example, when the zoom signal is a first signal, the image generator 1214 can generate an output image by using image data output from the sub-image processors 1212a and 1212b among the image data output from the sub-image processors 1212a to 1212c. When the zoom signal is a second signal different from the first signal, the image generator 1214 can generate an output image by using image data output from the sub-image processors 1212c and 1212b among the image data output from the sub-image processors 1212a to 1212c. If the zoom signal is a third signal different from the first and second signals, the image generator 1214 does not perform image data merging, and generates an output image by selecting any one of the image data output from each of the sub-image processors 1212a, 1212b, and 1212c. However, embodiments are not limited thereto, and the method of processing image data can be modified to satisfy application standards.

[0209] In some embodiments, the image processing apparatus 1210 can further include a selector (not shown) that selects the outputs of the sub-image processors 1212a, 1212b, and 1212c and transmits the selected outputs to the image generator 1214.

[0210] In this case, the selector can perform different operations according to the zoom signal or zoom factor. For example, when the zoom signal is a fourth signal (e.g., a zoom ratio is a first ratio), the selector can select any one of the outputs of the sub-image processors 1212a, 1212b, and 1212c and transmit the selected output to the image generator 1214.

[0211] In addition, when the zoom signal is a fifth signal (e.g., a zoom ratio is a second ratio) different from the fourth signal, the selector can sequentially transmit p (p is a natural number greater than or equal to 2) outputs among the outputs of the sub-image processors 1212a, 1212b, and 1212c to the image generator 1214. For example, the selector can sequentially transmit the output of the sub-image processor 1212b and the output of the sub-image processor 1212c to the image generator 1214. In addition, the selector can sequentially transmit the output of the sub-image processor 1212a and the output of the sub-image processor 1212b to the image generator 1214. The image generator 1214 can generate one output image by merging the p outputs received sequentially.

[0212] Here, the sub image processors 1212a, 1212b, and 1212c perform image processing such as re-mosaicking, downscaling to a video / preview resolution size, gamma correction, and high dynamic range (HDR) processing, and then transmit the processed image data to the image generator 1214. Accordingly, even when the processed image data is provided to the image generator 1214 through the selector and one signal line, the image merging operation of the image generator 1214 can be performed at a high speed.

[0213] In some embodiments, the image generator 1214 can receive a plurality of pieces of image data having different exposure times from at least one of the plurality of sub image processors 1212a, 1212b, and 1212c, and perform high dynamic range (HDR) processing on the plurality of pieces of image data, thereby generating merged image data having an increased dynamic range.

[0214] The camera module controller 1216 can provide a control signal to the camera modules 1100a, 1100b, and 1100c. The control signal generated by the camera module controller 1216 can be provided to the corresponding camera modules 1100a, 1100b, and 1100c through the control signal lines CSLa, CSLb, and CSLc, which are separated from each other.

[0215] Any one of the plurality of camera modules 1100a, 1100b, and 1100c can be designated as a master camera (e.g., 1100b) and the other camera modules (e.g., 1100a and 1100c) can be designated as slave cameras according to image generation information including a zoom signal or a mode signal. Such information can be included in the control signal and provided to the corresponding camera modules 1100a, 1100b, and 1100c through the control signal lines CSLa, CSLb, and CSLc, which are separated from each other.

[0216] The camera modules operating as the master camera and the slave camera can be changed according to a zoom factor or an operation mode signal. For example, when the field of view angle of the camera module 1100a is wider than that of the camera module 1100b and its zoom factor indicates a low zoom ratio, the camera module 1100a can operate as a master camera and the camera module 1100b can operate as a slave camera. In contrast, when the zoom factor indicates a high zoom ratio, the camera module 1100b can operate as a master camera and the camera module 1100a can operate as a slave camera.

[0217] In some embodiments, the control signal provided from the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c can include a synchronization enable signal. For example, when the camera module 1100b is a master camera and the camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 can transmit a synchronization enable signal to the camera module 1100b. Upon receiving the synchronization enable signal, the camera module 1100b can generate a synchronization signal based on the provided synchronization enable signal, and transmit the generated synchronization signal to the camera modules 1100a and 1100c through a synchronization signal line SSL. The camera module 1100b as well as the camera modules 1100a and 1100c can synchronize with the synchronization signal to transmit image data to the application processor 1200.

[0218] In some embodiments, the control signal provided from the camera module controller 1216 to the plurality of camera modules 1100a, 1100b, and 1100c can include mode information according to a mode signal. The plurality of camera modules 1100a, 1100b, and 1100c can operate in a first operation mode and a second operation mode related to a sensing speed based on the mode information.

[0219] The plurality of camera modules 1100a, 1100b, and 1100c can generate image signals at a first speed (e.g., generate image signals at a first frame rate) in the first operation mode, and encode the image signals at a second speed higher than the first speed (e.g., encode the image signals at a second frame rate higher than the first frame rate), and transmit the encoded image signals to the application processor 1200. In this case, the second speed can be lower than or equal to 30 times the first speed.

[0220] The application processor 1200 can store the received image signals (i.e., the encoded image signals) in an internal memory 1230 included therein or an external memory 1400 external to the application processor 1200, then read the encoded image signals from the internal memory 1230 or the external memory 1400, and decode the encoded image signals, and display image data generated based on the decoded image signals. For example, a corresponding sub image processor among the plurality of sub image processors 1212a, 1212b, and 1212c of the image processing device 1210 can perform decoding, and can also perform image processing on the decoded image signals.

[0221] The plurality of camera modules 1100a, 1100b, and 1100c can each generate an image signal at a third speed lower than the first speed (e.g., an image signal at a third frame rate lower than the first frame rate) in the second operation mode, and transmit the image signal to the application processor 1200. The image signal provided to the application processor 1200 can be an unencoded signal. The application processor 1200 can perform image processing on the received image signal, or can store the image signal in the internal memory 1230 or the external memory 1400.

[0222] The PMIC 1300 can supply power (e.g., a power voltage) to the plurality of camera modules 1100a, 1100b, and 1100c. For example, the PMIC 1300 can supply first power to the camera module 1100a through a power signal line PSLa, second power to the camera module 1100b through a power signal line PSLb, and third power to the camera module 1100c through a power signal line PSLc under the control of the application processor 1200.

[0223] The PMIC 1300 can generate power corresponding to each of the plurality of camera modules 1100a, 1100b, and 1100c in response to a power control signal PCON from the application processor 1200, and can also adjust the level of the power. The power control signal PCON can include a power adjustment signal for each operation mode of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the operation mode can include a low power mode, and in this case, the power control signal PCON can include information about the camera module operating in the low power mode and the power level to be set. The power levels provided to the plurality of camera modules 1100a, 1100b, and 1100c can be the same as or different from each other. In addition, the power level can be dynamically changed.

[0224] Figure 16 An electronic device according to an embodiment of the disclosure is illustrated. Figure 16 The electronic device 2000 can include a portable terminal.

[0225] Referring to Figure 16 The electronic device 2000 can include an application processor 2100, a camera module 2200, a working memory 2300, a storage 2400, a display device 2600, a user interface 2700, and a wireless transmission / reception unit 2500.

[0226] The application processor 2100 can control the operation of the electronic device 2000 and can include a system on chip (SoC) that drives an application program, an operating system, etc. The application processor 2100 can provide image data provided from the camera module 2200 to the display device 2600 or can store the image data in the storage 2400.

[0227] Referring to Figures 1-11B The described image sensor module 100 can be applied to the camera module 2200. The image sensor 2200 can include an encoder 2210 that can compress image data to generate compressed data and transmit the compressed data to the application processor 2100. As described above, when at least some (e.g., at least one pixel group) of the image data corresponds to the isolated area, the encoder 2210 can compress the pixel group by using an offset encoding method.

[0228] The application processor 2100 can include a decoder 2110 that decompresses the compressed data by using a compression method of the encoder 2210 (e.g., a decoding method corresponding to the encoding method). The decoder 2110 can decompress the compressed data received from the camera module 2200 to generate reconstructed image data, and the application processor 2100 can perform image processing on the reconstructed image data. The application processor 2100 can display the reconstructed image data or the image data subjected to the image processing on the display device 2600 or can store the reconstructed image data or the image data subjected to the image processing in the storage 2400.

[0229] The working memory 2300 can include a volatile memory such as a DRAM or an SRMA or a non-volatile resistive memory such as a ferroelectric random access memory (FeRAM), a resistive random access memory (RRAM), or a PRAM. The working memory 200 can store programs executed by the application processor 2100 and / or data processed by the application processor 2100.

[0230] The storage 2400 can include a non-volatile memory device such as a NAND flash or a resistive memory, for example, the storage 2400 can include a memory card (a multimedia card (MMC), an embedded MMC (eMMC), a secure digital (SD), a micro-SD, etc.). The storage 2400 can store image data received from the camera module 2200 or data processed or generated by the application processor 2100.

[0231] The user interface 2700 can include various means by which user input can be received, such as a keyboard, a wall keypad, a touch panel, a fingerprint sensor, and a microphone. The user interface 2700 can receive user input and provide a signal corresponding to the received user input to the application processor 2100.

[0232] The wireless transmission / reception unit 2500 can include a transceiver 2510, a modem 2520, and an antenna 2530.

[0233] While the disclosure has been particularly shown and described with reference to embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details can be made therein without departing from the spirit and scope of the appended claims.

Claims

1. An image compression method, comprising: Receive the pixel values ​​of the target pixel group to be compressed and the reference values ​​of the reference pixels to be used to compress the target pixel group in the image data; A virtual reference map is generated by applying offset values ​​to each of the reference values; The pixel values ​​of the target pixel group are compressed based on the virtual reference mapping; as well as A bitstream including compression results and compression information is generated based on the virtual reference mapping. The compression of the pixel value includes calculating a first difference between a compensation reference value included in the virtual reference map and a first value of at least one of the pixel values, and calculating a second difference between the first value and other values ​​of the pixel values.

2. The image compression method according to claim 1, wherein: The image data is generated by an image sensor. The reference values ​​correspond to the reconstructed pixel values ​​of reference pixels that were compressed before the target pixel group, and the difference between the pixel values ​​of pixels included in the target pixel group and the reference values ​​is greater than or equal to a threshold.

3. The image compression method according to claim 1, wherein, Generating the virtual reference map includes determining the offset value based on the reference value.

4. The image compression method according to claim 1, wherein, The first value is the pixel value of the first pixel among the plurality of pixels in the target pixel group.

5. The image compression method according to claim 4, wherein, The bitstream includes data indicating the first difference and the second difference, as well as mode information indicating the compression method.

6. The image compression method according to claim 5, wherein, The number of bits allocated to the region that includes the first difference is greater than the number of bits allocated to the region that includes the second difference.

7. The image compression method according to claim 5, wherein, The bitstream also includes additional information indicating the amount of bit shift or the offset value of the data.

8. The image compression method according to claim 1, wherein, The first value is the average value of the pixel values.

9. The image compression method according to claim 1, further comprising: Reconstructed pixel values ​​are generated by decompressing the bitstream after it has been generated. as well as A reconstructed image is generated based on the reconstructed pixel values, the reconstructed image including reference pixels to be used for compressing the next target pixel group to be compressed after the target pixel group.

10. The image compression method according to claim 1, wherein, The target pixel group comprises four pixels corresponding to the same color and arranged in two rows and two columns.

11. An image sensor module, comprising: An image sensor configured to generate image data comprising multiple pixels; The encoder is configured to generate compressed data comprising multiple bit streams by sequentially compressing image data generated by the image sensor in units of pixel groups, and is configured to compress the target pixel group to be compressed according to at least one of a variety of encoding methods. as well as An interface configured to output the compressed data to an external image processing device. The encoder generates a virtual reference map by applying an offset value to each of the reference values ​​of reference pixels arranged adjacent to the target pixel group according to a first encoding method among the multiple encoding methods, and compresses the target pixel group based on the virtual reference map. The compression of the target pixel group includes calculating a first difference between a compensation reference value included in the virtual reference map and a first value of at least one of the pixel values ​​of the pixels in the target pixel group, and calculating a second difference between the first value and other values ​​among the pixel values.

12. The image sensor module according to claim 11, wherein, The encoder determines the offset value based on the reference value.

13. The image sensor module according to claim 11, wherein, The encoder adds a preset offset value to each of the reference values.

14. The image sensor module according to claim 11, wherein, The encoder encodes the target pixel group based on the virtual reference mapping using a differential pulse code modulation method.

15. The image sensor module according to claim 11, wherein, The encoder generates multiple encoded data by compressing the target pixel group according to the multiple encoding methods, and selects the encoded data with the lowest error rate from the multiple encoded data as the compressed data for the target pixel group.

16. The image sensor module according to claim 11, wherein, When the difference between the pixel value of the target pixel in the target pixel group and the reference value is greater than or equal to a threshold, the encoder compresses the target pixel group according to the first encoding method among the multiple encoding methods.

17. The image sensor module according to claim 11, wherein, The encoder generates encoded data by compressing the target pixel group using an encoding method with higher priority among the multiple encoding methods, and compresses the target pixel group using the first encoding method when the error rate of the encoded data is less than or equal to a reference value.

18. An image processing system, comprising: An image sensor, configured to generate image data by sensing received optical signals; An encoder configured to generate multiple bitstreams by sequentially compressing multiple groups of pixels in the image data; as well as A decoder configured to reconstruct the image data by decompressing the plurality of bitstreams. The encoder generates a virtual reference map by applying the same offset value to all reference values ​​of reference pixels arranged adjacent to the target pixel group to be compressed, and compresses the target pixel group based on the virtual reference map. The compression of the target pixel group includes calculating a first difference between a compensation reference value included in the virtual reference map and a first value of at least one of the pixel values ​​of the pixels in the target pixel group, and calculating a second difference between the first value and other values ​​among the pixel values.

19. The image processing system according to claim 18, wherein, When the difference between the pixel value of the target pixel in the target pixel group and the reference value is greater than or equal to a threshold, the encoder compresses the target pixel group based on the virtual reference mapping.

20. The image processing system according to claim 18, wherein, The multiple bit streams are transmitted from the encoder to the decoder via a camera serial interface based on a mobile industrial processor interface.

Images