Image sensor module, image processing system and image compression method

Abstract

An image sensor module, an image processing system, and an image compression method are provided. The image compression method for compressing image data generated by an image sensor includes: receiving pixel values ​​of a target pixel group of image data to be compressed and reference values ​​of reference pixels to be used in the compression of the target pixel group; determining an averaging direction in which averaging calculations will be performed on the target pixel values; averaging the pixel values ​​of the target pixels along the averaging direction; generating balance information based on the reference pixels, including compensation values ​​to be applied to the average value; and generating a bitstream based on the average value, the balance information, and the compression information.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2020-0088452 filed on July 16, 2020, and Korean Patent Application No. 10-2021-0029045 filed on March 4, 2021, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure relates generally to image sensors, and more specifically to an image sensor module, an image processing system, and an image compression method for compressing data in an isolated region. Background Technology

[0004] With increasing interest in high-quality and high-resolution images, the number of sensing pixels in the pixel array of image sensors and the size of image data generated by image sensors have increased. Image data can be sent to an image processing device, where it can be compressed to improve transmission efficiency, and compressed image data can be sent to the image processing device. Image data can include various two-dimensional or multi-dimensional image patterns. To compress pixel data contained in specific areas of the image pattern, compression methods can be applied to improve compression efficiency and reduce compression loss. Summary of the Invention

[0005] This disclosure provides an image sensor module, an image processing system, and an image compression method for effectively compressing pixel data in isolated regions.

[0006] According to embodiments of this disclosure, an image compression method for compressing image data generated by an image sensor is provided, comprising: receiving pixel values ​​of a target pixel group of image data to be compressed and reference values ​​of a reference pixel to be used in the compression of the target pixel group; determining an averaging direction in which an averaging calculation is performed on the target pixel values; averaging the pixel values ​​of the target pixel in the averaging direction; generating balance information based on the reference pixel, including a compensation value to be applied to the average value; and generating a bitstream based on the average value, the balance information, and the compression information.

[0007] According to another embodiment of this disclosure, an image sensor module is provided, comprising: an image sensor configured to generate image data including a plurality of pixels; an encoder configured to generate compressed data including a plurality of bitstreams by sequentially compressing the image data generated by the image sensor in units of pixel groups, and the compression is performed on target pixel groups compressed according to at least one of a plurality of encoding methods; and an interface configured to output the compressed data to an external image processing device, wherein the encoder generates an average value based on the pixel value of the target pixel according to a first encoding method of the plurality of encoding methods, generates balance information including a compensation value to be applied to the average value, and generates a bitstream including the average value, the balance information, and the compression information.

[0008] According to another embodiment of this disclosure, an image processing system is provided, comprising: an image sensor configured to sense received optical signals and generate image data; an encoder configured to sequentially compress a plurality of pixel groups of the image data and generate a plurality of bitstreams; and a decoder configured to decompress the plurality of bitstreams and recover the image data, wherein the encoder generates balance information including an average value of pixel values ​​based on target pixels and a compensation value to be applied to the average value, and generates a bitstream including the average value, the balance information, and compression information. Attached Figure Description

[0009] Embodiments of this disclosure will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 This is a block diagram of an image processing system according to an embodiment;

[0011] Figure 2 This is a conceptual diagram of a pixel array and image data applied to an image sensor module according to an embodiment;

[0012] Figure 3A This is a block diagram of an encoder according to an embodiment;

[0013] Figure 3B This is a block diagram of an encoder according to an embodiment;

[0014] Figure 4 It is a hybrid concept diagram of the reference mapping according to the embodiments;

[0015] Figure 5 This is a flowchart of the compression method according to an embodiment;

[0016] Figure 6A It is a description Figure 5 A conceptual diagram of the operation of the compression method;

[0017] Figure 6B It is a description Figure 5 A conceptual diagram of the operation of the compression method;

[0018] Figure 6C It is a description Figure 5 A conceptual diagram of the operation of the compression method;

[0019] Figure 6D It is a description Figure 5 A conceptual diagram of the operation of the compression method;

[0020] Figure 7A This is a conceptual diagram illustrating a bitstream according to an embodiment;

[0021] Figure 7B This is a conceptual diagram illustrating a bitstream according to an embodiment;

[0022] Figure 8 This is a conceptual diagram illustrating a compression method using a comparative example according to an embodiment;

[0023] Figure 9 This is a conceptual diagram illustrating image data and reference mapping according to an embodiment;

[0024] Figure 10 This is a conceptual diagram describing a compression method according to an embodiment;

[0025] Figure 11A This is a block diagram illustrating an image sensor module according to an embodiment;

[0026] Figure 11B This is a block diagram illustrating an image sensor module according to an embodiment;

[0027] Figure 12 This is a schematic block diagram of a decoder according to an embodiment;

[0028] Figure 13 This is a block diagram of an image processing system according to an embodiment;

[0029] Figure 14 This is a tabular diagram describing compression information according to an embodiment;

[0030] Figure 15A It is a block diagram of an electronic device that includes multiple camera modules;

[0031] Figure 15B yes Figure 15A A block diagram of the camera module in the diagram; and

[0032] Figure 16 This is a schematic block diagram of an electronic device according to an embodiment. Detailed Implementation

[0033] The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0034] Figure 1 An image processing system 10 according to an embodiment is shown. Figure 2 A pixel array and image data applied to an image sensor module according to an embodiment are shown.

[0035] The image processing system 10 can sense images of objects or devices, store the sensed images in a memory, and / or process the sensed images, and can also store the processed images in the memory. According to embodiments, the image processing system 10 can be implemented as a digital camera, digital camcorder, mobile phone, tablet computer, or portable electronic device. Portable electronic devices may include laptops, mobile phones, smartphones, tablet PCs, personal digital assistants (PDAs), enterprise digital assistants (EDAs), digital still cameras, digital camcorders, audio devices, portable multimedia players (PMPs), personal navigation devices (PNDs), MP3 players, handheld game consoles, e-readers, wearable devices, etc. Additionally, the image processing system 10 can be installed as a component in electronic devices (such as drones, advanced driver assistance systems (ADAS), vehicles, furniture, manufacturing facilities, doors, various measuring devices, etc.).

[0036] Reference Figure 1 The image processing system 10 may include an image sensor module 100 and an image processing device 200. In an embodiment, the image sensor module 100 may include an image sensor 110, an encoder 120, and an interface (I / F) 130. In an embodiment, the image sensor module 100 may be implemented as a plurality of semiconductor chips. For example, the pixel array of the image sensor 110 (e.g., Figure 2 The PXA (Image Sensor Module 100) can be integrated into a single semiconductor chip, or into other semiconductor chips in which the logic circuitry of the image sensor 110, encoder 120, and I / F 130 differs. These semiconductor chips can be electrically connected to each other via interconnects or via stacked through-holes. However, embodiments are not limited thereto; the image sensor module 100 can be implemented as a single semiconductor chip, and is not limited thereto.

[0037] In an embodiment, the image processing device 200 may include an I / F 210, a memory 220, a decoder 230, and an image signal processor 240.

[0038] Image sensor module 100 can capture images of external targets, objects, or entities and generate image data IDT. Image sensor module 100 may include image sensor 110, which is capable of converting optical signals from a target incident through lens LS into electrical signals.

[0039] Image sensor 110 may include a plurality of sensing pixels (e.g., Figure 2 A two-dimensional array of pixels arranged in SPX (e.g., Figure 2 The PXA in the pixel array can output image data IDT, which includes multiple pixel values ​​corresponding to multiple sense pixels SPX in the pixel array PXA.

[0040] The pixel array PXA may include multiple row lines, multiple column lines, and multiple sensing pixels SPX connected to each of the multiple row lines and each of the multiple column lines and arranged in a matrix.

[0041] Each of the plurality of sensing pixels SPX in a pixel array PXA can sense an optical signal of at least one of a plurality of reference colors. For example, the plurality of reference colors may include red, green, and blue, or red, green, blue, and white, and may also include colors other than these. For example, the plurality of reference colors may include cyan, yellow, green, and magenta. The pixel array PXA can generate pixel signals that include information about the reference color of each of the plurality of sensing pixels SPX.

[0042] For example, such as Figure 2 As shown, the pixel array PXA may include a red sensing pixel SPX_R, a blue sensing pixel SPX_B, a first green sensing pixel SPX_Gr, and a second green sensing pixel SPX_Gb. The green sensing pixel arranged in the same row as the red sensing pixel SPX_R may be referred to as the first green sensing pixel SPX_Gr, and the green sensing pixel arranged in the same row as the blue sensing pixel SPX_B may be referred to as the second green sensing pixel SPX_Gb.

[0043] The red sensing pixel SPX_R, the blue sensing pixel SPX_B, the first green sensing pixel SPX_Gr, and the second green sensing pixel SPX_Gb can be arranged in multiple rows and columns, and this arrangement can be called a pixel pattern PT. Multiple pixel patterns PT can be repeatedly arranged in the pixel array PXA.

[0044] For example, such as Figure 2As shown, a pixel pattern PT may include red sensing pixels SPX_R arranged in a 2×2 matrix, blue sensing pixels SPX_B arranged in a 2×2 matrix, a first green sensing pixel SPX_Gr arranged in a 2×2 matrix, and a second green sensing pixel SPX_Gb arranged in a 2×2 matrix. This type of pixel pattern PT may be referred to as a four-grid pattern. However, the technical concept of this disclosure is not limited thereto, and the pixel pattern PT may include red sensing pixels SPX_R, blue sensing pixels SPX_B, a first green sensing pixel SPX_Gr, and a second green sensing pixel SPX_Gb arranged in a 2×2 matrix, and this type of pixel pattern PT may be referred to as a Bayer pattern (e.g., see...). Figure 9 Alternatively, the pixel pattern PT may include red sensing pixels SPX_R arranged in an n×n matrix (where n is an integer equal to or greater than 3), blue sensing pixels SPX_B arranged in an n×n matrix, a first green sensing pixel SPX_Gr arranged in an n×n matrix, and a second green sensing pixel SPX_Gb arranged in an n×n matrix.

[0045] Image data IDT can be generated based on the pixel signals output from the pixel array PXA. The image data IDT may have a color pattern corresponding to the pixel pattern PT of the pixel array PXA. As an example, when the pixel array PXA has a Bayer pattern, the image data IDT may also have a Bayer pattern. As another example, when the pixel array PXA has a four-cell pattern, the image data IDT may have either a four-cell pattern or a Bayer pattern.

[0046] For example, when the pixel array PXA has a four-cell pattern, a single pixel signal can be output from four sensing pixels SPX of the same color included in the pixel pattern PT, or four pixel signals can be output along with the output pixel signal from each of the four sensing pixels SPX. When outputting a single pixel signal, the image data IDT can have a Bayer pattern, and when outputting four pixel signals, such as... Figure 2 As shown, image data IDT can have a four-grid pattern.

[0047] The image data IDT may include alternately arranged red pixels PX_R, blue pixels PX_B, a first green pixel PX_Gr, and a second green pixel PX_Gb. Pixels PX in the image data IDT may indicate data (i.e., pixel data) corresponding to the sensing pixels SPX of the pixel array PXA. The red pixels PX_R, blue pixels PX_B, first green pixels PX_Gr, and second green pixels PX_Gb may correspond to the red sensing pixels SPX_R, blue sensing pixels SPX_B, first green sensing pixels SPX_Gr, and second green sensing pixels SPX_Gb, respectively.

[0048] Image data IDT may include multiple pixel groups PG, and in this case, according to the color pattern PT of image data IDT, pixel group PG may be set to include a predetermined number of pixels PX arranged in multiple rows and columns or arranged in one direction, or may include adjacent pixels PX corresponding to the same reference color.

[0049] For example, such as Figure 2 As shown, when the image data IDT has a four-grid pattern, the pixel group PG can be set to correspond to the same reference color (e.g., red, blue, and green) and include four adjacent pixels PX. As another example, when the image data IDT has a Bayer pattern, the pixel group PG can be set to include a predetermined number (e.g., four) of pixels PX of different colors arranged in a matrix.

[0050] Refer to Figure 1 Each of the multiple sensing pixels (SPX) may include at least one photosensitive element or photoelectric conversion element. The photosensitive element can sense light and convert the sensed light into an electrical signal. For example, the photosensitive element may include a photodiode, a phototransistor, a photogate, a pinned photodiode (PPD), or a combination thereof.

[0051] Each of the plurality of sensing pixels (SPXs) may include a photosensitive element and a pixel circuit for outputting a pixel signal corresponding to the electrical signal generated by the photosensitive element. For example, the pixel circuit may have a four-transistor structure including a transfer transistor, a reset transistor, an amplification transistor, and a selection transistor. However, embodiments are not limited thereto, and the pixel circuit may have a one-transistor structure, a three-transistor structure, a four-transistor structure, a five-transistor structure, or a structure in which multiple pixels (PXs) share some transistors. In embodiments, each of the pixel circuits may be equipped with a memory and / or an analog-to-digital converter.

[0052] In an embodiment, multiple color filters transmitting optical signals of a specific wavelength (e.g., optical signals of a specific color) can be arranged on multiple pixels PX, each corresponding to a plurality of pixels PX in a pixel array PXA, and the optical signals transmitted through the color filters corresponding to at least one photosensitive element disposed in a pixel PX can be converted. Therefore, each of the plurality of sensing pixels SPX can output at least one pixel signal corresponding to at least one assigned reference color. However, the embodiment is not limited thereto. For example, the optical signal of light of a specific wavelength incident on at least one photosensitive element disposed in the sensing pixel SPX can be selectively converted into an electrical signal.

[0053] In an embodiment, the image data IDT may include raw image data or image data in which preprocessing operations have been performed on the raw image data, wherein the raw image data includes the plurality of pixel values ​​in which the plurality of pixel signals output by the pixel array PXA have been digital-to-analog converted.

[0054] Regarding data transmission speed, reduced power consumption during data transmission, and efficient data storage, the image sensor module 100 can compress the image data IDT using the encoder 120 and send the compressed data CDT to the image processing device 200.

[0055] Encoder 120 can receive image data IDT from image sensor 110, compress the image data IDT, and generate compressed data CDT. The compressed data CDT can be implemented as an encoded bitstream. Hereinafter, the encoded bitstream may be simply referred to as a bitstream. The bitstream may include the compression result and compression information (e.g., mode information indicating the compression method).

[0056] Encoder 120 can generate compressed data CDT by encoding image data IDT in units of pixel groups PG. Encoder 120 can generate a bitstream by encoding a pixel group PG, and generate compressed data CDT based on the bitstream of all pixel groups PG in the image data IDT. As pixel groups PG are encoded, pixel groups PG can be compressed; hereinafter, in this disclosure, encoding and compression can be used in the same sense.

[0057] Encoder 120 can perform compression by utilizing a reference map generated based on pixel values ​​corresponding to pixels PX that have been previously compressed before the pixel group PG (i.e., the target pixel group) to which compression will be performed. Encoder 120 can compress the pixel value of the target pixel based on a reference value of at least one reference pixel in the reference map that is adjacent to at least one target pixel in the target pixel group. The reference value can be generated based on the pixel value of the reference pixel; for example, the reference value can be a value generated as the pixel value of the reference pixel is compressed and decompressed.

[0058] The pixel value of a target pixel can be similar to the pixel value of its neighboring reference pixel. Furthermore, the pixel values ​​of target pixels within a target pixel group can be similar to each other. Therefore, the encoder 120 can compress the target pixel group using a differential pulse code modulation (DPCM)-based encoding method, in which encoding is performed based on the differences between the target pixel and its neighboring pixels (such as, for example, the difference between the pixel value of the target pixel and the reference value of the reference pixel, or the difference between the pixel value of the target pixel and the pixel values ​​of other target pixel groups). Therefore, compression efficiency or compression ratio can be improved, and thus data loss due to compression can be reduced.

[0059] In an image data IDT, the difference between the pixel value of the target pixel group and the pixel value of the reference pixel within the isolated region can be large. In other words, the correlation between the pixel value of the target pixel and the reference value of the reference pixel can be low.

[0060] In this context, the isolation zone can indicate the zone corresponding to the peripheral zone in at least two directions (e.g., two directions orthogonal to each other) of the peripheral zone of a two-dimensional or multi-dimensional image pattern generated in the image data IDT, such as the zone corresponding to the edge of the image pattern.

[0061] The encoder 120 according to an embodiment may include a Balanced Mode Compressor (BMC), and when the difference between the pixel value of a target pixel and the reference value of a reference pixel is large, the BMC may compress the target pixel using a dedicated encoding method such as, for example, a horizontal or vertical (HV) balanced encoding method. The BMC may generate an average value by averaging the pixel value of the target pixel in the horizontal or vertical direction, or in the vertical or horizontal direction, and generate a bitstream based on the average value and balance information. In this case, the balance information may include information for compensating for the difference between the average value and the pixel value, and may include a selection value and a slope value, the selection value indicating whether to recover the pixel value by applying the difference between the reference values ​​of adjacent pixels to the average value as a compensation value during the encoding operation, and the slope value indicating whether, relative to a particular pixel, the difference or a preset default value is added to the average value or subtracted from the average value. See also... Figures 5 to 10 This HV balanced coding method will be described in more detail.

[0062] When encoder 130 compresses pixel groups in the isolation region using the DPCM method described above, or compresses pixel groups in the isolation region based on the encoding method of the pixel groups in the isolation region using some high data bits including the most significant bit (MSB) of the plurality of data bits indicating the pixel value of each of the target pixels, a large amount of data loss may occur. Therefore, image degradation may occur in the recovered image data generated by decompressing the compressed data CDT, and artifacts may occur in the image data. However, as described above, encoder 120 according to the embodiment can compress pixel groups in the isolation region using a dedicated encoding method (e.g., HV balanced encoding method). Therefore, compression efficiency can be improved, and data loss can be reduced.

[0063] Encoder 120 can provide compressed data CDT to image processing device 200 via I / F 130. For example, I / F 130 can be implemented as a Camera Serial Interface (CSI) based on Mobile Industrial Processor Interface (MIPI). However, the type of I / F 130 is not limited to this and can be implemented according to various protocol standards.

[0064] The image processing apparatus 200 can generate an image to be displayed on a monitor by converting compressed data CDT received from the image sensor module 100. The image processing apparatus 200 can receive compressed data CDT from the image sensor module 100, generate decompressed data DDT (such as, for example, recovered image data) by decompressing the compressed data CDT, and perform image processing operations on the decompressed data DDT.

[0065] In this embodiment, the image processing apparatus 200 can receive compressed data CDT from the image sensor module 100 via I / F 210. Like the I / F 130 provided in the image sensor module 100, I / F 210 can be implemented using MIPI, but is not limited thereto. The image processing apparatus 200 can store the received compressed data CDT in memory 220.

[0066] Memory 220 can be a storage location for storing data. Compressed data CDT can be stored in memory 220. Furthermore, memory 220 can also store other data such as, for example, an operating system (OS), various programs, and various types of data (e.g., compressed data CDT). Memory 220 can include volatile memory (such as random access memory (RAM), dynamic random access memory (DRAM), and / or static RAM (SRAM)) or non-volatile memory (such as phase-change RAM (PRAM), resistive RAM (ReRAM), magnetic RAM (MRAM), and / or flash memory). Figure 1 In the image processing device 200, memory 220 is shown as being included in the image processing device 200, but is not limited thereto, and memory 220 may be disposed separately outside the image processing device 200.

[0067] Decoder 230 can read compressed data CDT from memory 220 and generate decompressed data DDT by decompressing the compressed data CDT. Decoder 230 can provide the decompressed data DDT to image signal processor 240.

[0068] Decoder 230 can decompress compressed data CDT in units of pixel groups PG by utilizing a decompression or decoding method based on a compression or encoding method executed by encoder 120 of image sensor module 100. Decoder 230 can determine the decompression method applied to pixel groups PG based on compression information included in the bitstream of compressed data CDT. Decoder 230 can decompress target pixels of target pixel groups based on pixels that have been decompressed before the target pixel group to be decompressed (i.e., including reference mappings of reference values ​​corresponding to reference pixels).

[0069] In this embodiment, decoder 230 may decompress the target pixel group using a decoding method based on the HV balanced encoding method. Decoder 230 may store pixel values ​​based on the balance information by adjusting the average value included in the bitstream.

[0070] The image signal processor 240 can perform various image processing operations on the received decompressed data DDT. As a non-limiting example, the image signal processor 240 can perform at least one of the following image processing operations on the decompressed data DDT: bad pixel compensation, bias compensation, lens distortion compensation, color gain compensation, shading compensation, gamma compensation, denoising, and / or sharpening. In embodiments, some of the above image processing operations can be omitted depending on the performance of the image sensor module 100. For example, when the image sensor module 100 includes a high-quality image sensor 110, bad pixel compensation (e.g., static bad pixel compensation) or bias compensation, etc., can be omitted.

[0071] On the other hand, each of the encoder 120 and decoder 230 can be implemented as software or hardware, or a combination of software and hardware such as firmware. When the encoder 120 and decoder 230 are implemented as software, each of the above-described functions can be implemented as programmed source code and can be loaded into a storage medium provided in each of the image sensor module 100 and image processing device 200. The functions of the encoder 120 and decoder 230 can be implemented as the processor (e.g., an image processor) provided in each of the image sensor module 100 and image processing device 200 executes the software. When the encoder 120 and decoder 230 are implemented as hardware, the encoder 120 and decoder 230 can include logic circuitry and registers, and each of the above-described functions can be executed based on register settings.

[0072] exist Figure 1 In this embodiment, the image processing system 10 is shown to include an image sensor module 100 and an image processing device 200, but the embodiment is not limited thereto. For example, the image processing system 10 may include some of the image sensor modules 100 and the image processing device 200, or it may be implemented to include multiple image sensor modules 100. Additionally, in Figure 1 In this embodiment, decoder 230 and image signal processor 240 are shown as separate components, but the embodiments are not limited thereto. For example, image signal processor 240 may be implemented to include decoder 230.

[0073] Figure 3A and Figure 3B An encoder according to an embodiment is shown. Figure 3A and Figure 3B It shows Figure 1 Example of encoder 120 in the example.

[0074] Reference Figure 1 and Figure 3A The encoder 120 may include a reference pixel detector 121, a compression circuit 122, a mode selector 123, a restored image generator 124, and a reference buffer 125.

[0075] The reference pixel detector 121 can be obtained from an image sensor (e.g., Figure 1 The reference pixel detector 121 receives image data IDT and a reference map including reference values ​​of reference pixels for compressing the target pixel group from the reference buffer 125. The reference pixel detector 121 can detect reference values ​​(i.e., recovered pixel values ​​of reference pixels) of reference pixels adjacent to the target pixel group in the recovered image data stored in the reference buffer 125, and can receive these reference values ​​as a reference map from the reference buffer 125. The reference pixel detector 121 can provide the reference map of the target pixel group and the image data IDT to the compression circuit 122.

[0076] Compression circuit 122 can compress the target pixel group based on a reference map. Compression circuit 122 may include a balanced mode compressor (BMC) and a normal mode compressor (NMC), and the balanced mode compressor (BMC) and the normal mode compressor (NMC) can compress the target pixel group by utilizing different compression methods or encoding methods. The normal mode compressor (NMC) can output first encoded data EDT1, and the balanced mode compressor (BMC) can output second encoded data EDT2.

[0077] The Normal Mode Compressor (NMC) can encode target pixel groups based on a reference map using the DPCM method. In an embodiment, the NMC can generate the difference between a reference value of a reference pixel and the pixel value of a target pixel, as well as the difference between the pixel values ​​of the target pixels, and can generate a bitstream based on these differences. In another embodiment, the NMC can generate the difference between the average pixel value of the target pixels and the reference value of a reference pixel, as well as the difference between the average pixel value and the pixel value of the target pixels, and can generate a bitstream based on these differences. The bitstream generated in the NMC can be output as first encoded data EDT1.

[0078] The balanced mode compressor BMC can be referenced above. Figure 1 The HV balanced coding method described above encodes target pixel groups. Figure 1 The described method generates an average value by averaging the pixel values ​​of the target pixels in the vertical or longitudinal direction, and generates a bitstream based on balance information used to compensate for the difference between the average value and the pixel value in each average value. The bitstream generated by the Balanced Mode Compressor (BMC) can be output as second encoded data EDT2.

[0079] In this scenario, the Balanced Mode Compressor (BMC) can determine whether to average the pixel values ​​of target pixels arranged continuously from top to bottom, or whether to average the pixel values ​​of target pixels arranged continuously from left to right (i.e., the direction in which the averaging calculation will be performed). The Balanced Mode Compressor (BMC) can determine the direction in which the averaging calculation will be performed based on reference values ​​from a reference map.

[0080] exist Figure 3A In the figure, compression circuit 122 is shown to include a balanced mode compressor (BMC) and a normal mode compressor (NMC), but the embodiment is not limited thereto. Compression circuit 122 may also include a compressor that encodes the target pixel group by using an encoding method different from the encoding method of the balanced mode compressor (BMC) and / or the normal mode compressor (NMC).

[0081] The mode selector 123 can select compressed data received from the compression circuit 122 (e.g., one of the first encoded data EDT1 and the second encoded data EDT2) and can output the selected encoded data as compressed data CDT.

[0082] The mode selector 123 can decode the first encoded data EDT1 and the second encoded data EDT2 according to the decoding method of decoding each of the first encoded data EDT1 and the second encoded data EDT2, and can select one of the first encoded data EDT1 and the second encoded data EDT2 based on the decoding error rate. The error rate can indicate the difference between the decoded data (i.e., the decoded pixel value and the pixel value before decoding), and the smaller the difference, the lower the error rate. When the error rate is lower, the decompressed data DDT (i.e., the image degradation of the recovered image data) generated by the image processing device 200 can be less. Therefore, the mode selector 123 can select the encoded data with the lower error rate of the first encoded data EDT1 and the second encoded data EDT2 as the compressed data CDT, and can output the compressed data CDT.

[0083] In an embodiment, the normal mode compressor (NMC) or another separate compressor can generate encoded data by encoding a target pixel group (e.g., first encoded data EDT1), and the balanced mode compressor (BMC) can operate when the error rate of the first encoded data EDT1 exceeds a reference error rate. The balanced mode compressor (BMC) can generate second encoded data EDT2 by encoding the target pixel group. When the error rate of the first encoded data EDT1 is equal to or less than the reference error rate, the mode selector 123 can output the first encoded data EDT1 as compressed data CDT. When the error rate of the first encoded data EDT1 exceeds the error rate of the second encoded data EDT2, the mode selector 123 can output the second encoded data EDT2 received from the balanced mode compressor (BMC) as compressed data CDT.

[0084] The restored image generator 124 can generate a restored image as data by decoding the compressed data CDT. The restored image generator 124 can recover the pixel values ​​of the target pixel group by decoding according to a decoding method corresponding to the encoding method of each of the plurality of bitstreams included in the compressed data CDT. The pixel corresponding to the recovered pixel value can be used as a reference pixel for other target pixel groups to which compression will be performed.

[0085] Reference buffer 125 can store recovered image data and provide reference values ​​for reference pixels used to compress the target pixel group to reference pixel detector 121. In an embodiment, reference buffer 125 may include line memory and store reference pixels surrounding the target pixels of the target pixel group. In an embodiment, reference buffer 125 may be implemented as a volatile memory such as DRAM and SRAM. However, embodiments are not limited thereto, and reference buffer 125 may be implemented as a non-volatile memory such as ReRAM and PRAM.

[0086] Reference Figure 3B The encoder 120a may include a reference pixel detector 121, a pre-detector 126, a compression circuit 122, a mode selector 123, a reconstructed image generator 124, and a reference buffer 125. Figure 3A Compared to encoder 120, encoder 120a may further include pre-detector 126. (See already referenced...) Figure 3A The operation of the reference pixel detector 121, compression circuit 122, mode selector 123, image recovery generator 124, and reference buffer 125 is described, and repeated descriptions thereof are omitted.

[0087] The pre-detector 126 can enable or disable the Balanced Mode Compressor (BMC). In an embodiment, the pre-detector 126 can enable or disable the BMC based on a reference value of a reference pixel. For example, the BMC can be enabled when the difference between the reference value and the pixel value of the target pixel is equal to or greater than a threshold or a specific code value, and disabled when it is less than the threshold. In an embodiment, the pre-detector 126 may include a register, and the BMC can be enabled or disabled based on a control signal stored in the register. For example, the control signal can be received from the image processing device 200.

[0088] When the Balanced Mode Compressor (BMC) is disabled, another compressor (e.g., the Normal Mode Compressor (NMC) equipped in the compression circuit 122 can encode the target pixel group and output encoded data (e.g., first encoded data EDT1) as compressed data CDT.

[0089] When the Balanced Mode Compressor (BMC) is enabled, at least some of the compressors in the compression circuit 122 and the Balanced Mode Compressor (BMC) can encode the target pixel group respectively, and the mode selector 123 can output the encoded data with the lowest error rate as compressed data CDT.

[0090] In this embodiment, although the Balanced Mode Compressor (BMC) is enabled, the BMC in the compressor of the compression circuit 122 may have a lower priority. For example, the Normal Mode Compressor (NMC) may first encode the target pixel group to generate first encoded data EDT1. When the error rate of the first encoded data EDT1 exceeds the reference error rate, the Balanced Mode Compressor (BMC) may operate. The Balanced Mode Compressor (BMC) may generate second encoded data EDT2 by encoding the target pixel group. When the error rate of the first encoded data EDT1 is equal to or less than the reference error rate, the mode selector 123 may output the first encoded data EDT1 as compressed data CDT. When the error rate of the first encoded data EDT1 exceeds the second encoded data EDT2, the mode selector 123 may output the second encoded data EDT2 received from the Balanced Mode Compressor (BMC) as compressed data CDT.

[0091] In this embodiment, when the Balanced Mode Compressor (BMC) is enabled, another compressor (e.g., the Normal Mode Compressor (NMC) in the compression circuit 122 can be disabled, and the second encoded data EDT2 generated by the Balanced Mode Compressor (BMC) can be output as compressed data CDT. Therefore, the power consumption of the encoder 120a can be reduced.

[0092] Figure 4 A reference mapping according to an embodiment is shown. Figure 4 In this context, the reference index for a pixel is an alphanumeric character that begins with a letter. In this embodiment, the three-digit number associated with a pixel indicates the value of that pixel (in other words, the pixel value or reference value). For example, the value of the pixel indicated as T0 is 283.

[0093] Reference Figure 4 Image data IDT may include multiple pixel groups PG, and the multiple pixel groups PG may be compressed sequentially in the direction in which they are set. In an embodiment, the image data IDT may be compressed sequentially in the left-to-right direction and in the top-to-bottom direction, and in units of pixel groups PG. However, the embodiment is not limited to this, and the image data IDT may be compressed in the right-to-left direction or the bottom-to-top direction.

[0094] Compression can be performed on the target pixel group TG or on the target pixels T0, T1, T2, and T3 of the target pixel group TG. The target pixel group TG can be compressed based on pixels included in the pixel group PG corresponding to a neighboring pixel group PG of the same color that has already undergone compression. The neighboring pixels used to compress the target pixel group TG are referred to as reference pixels.

[0095] A reference value can be generated as a reference map RM for a reference pixel adjacent to the target pixel group TG of the recovered image data RIDT from the reference buffer 125. In this case, the reference value can indicate the value generated after the pixel value is compressed and decompressed. For example, the reference map RM may include reference values ​​of reference pixels (e.g., R11, R12, R13, R14, R21, R22, R23, R24, R31, R32, R33, and R34).

[0096] The target pixel group TG can be compressed based on the reference map RM. For example, the target pixel group TG or the target pixel T0 of the target pixel group TG can be encoded based on the reference value with the highest relative correlation among the reference values ​​of the neighboring reference pixels R13, R14, R23, R24, R32 and R34, or the reference value of the neighboring reference pixels in a preset direction.

[0097] The differences between the pixel values ​​of target pixels T0, T1, T2, and T3 in the target pixel group TG can be relatively large. For example, when the pixel value of target pixel T0 is 283, the reference values ​​of the neighboring reference pixels R14, R23, and R32 are 137, 148, and 127, respectively, and the threshold is set to 125. The difference between the pixel value and the reference value can be equal to or greater than 125, and the target pixel group TG can correspond to an isolation zone. In this way, when the target pixel group TG corresponds to an isolation zone, it can be compressed based on the HV balanced coding method. For example, the balanced mode compressor BMC in Figure 3 can compress the target pixel group TG according to the HV balanced coding method, and in this case, the balanced mode compressor BMC can use the neighboring pixels of the target pixel group TG.

[0098] Figure 5 A compression method according to an embodiment is shown. Figure 6A , Figure 6B , Figure 6C and Figure 6D It shows Figure 5 The operation of compression methods.

[0099] It can be done Figure 3A and Figure 3B The balanced mode compressor BMC is used to perform this. Figure 5 and Figures 6A to 6D Compression methods.

[0100] The pixel values ​​of target pixels T0, T1, T2, and T3 can be similar to each other. The Balanced Mode Compressor (BMC) can compress target pixels T0, T1, T2, and T3 according to the HV balanced mode encoding method by utilizing the similarity of their pixel values. In this case, a reference map (e.g., ...) can be used. Figure 4 At least one pixel group included in the RM (in the image).

[0101] The compression method according to an embodiment is described by assuming that the balanced mode compressor BMC compresses the target pixel group TG using reference pixels R31, R32, R33, and R34 of the pixel group of the reference map RM that are closest to the left of the target pixel group TG. The embodiment is not limited thereto.

[0102] Reference Figure 5 The Balanced Mode Compressor (BMC) can receive pixel values ​​of the target pixel group and reference pixel values ​​of the reference pixel (S110). In other words, the Balanced Mode Compressor (BMC) can receive image data IDT including the target pixel group TG and a reference map RM including the reference values ​​of the reference pixels. The target pixel group TG (the pixel group to which compression is performed) may include target pixels T0, T1, T2, and T3, and the BMC can receive the pixel values ​​of each of the target pixels T0, T1, T2, and T3. The reference pixel may include neighboring pixels of the target pixels used to compress the target pixel group. The pixel values ​​of the neighboring pixels can be compressed, then recovered, and generated as reference values.

[0103] The Balanced Mode Compressor (BMC) can determine the direction in which averaging calculations are to be performed according to the HV balanced mode encoding method (hereinafter referred to as the averaging direction) (S120). See reference... Figure 6A The Balanced Mode Compressor (BMC) can determine whether to perform averaging calculations in the horizontal direction (HD) or the vertical direction (VD).

[0104] In an embodiment, the Balanced Mode Compressor (BMC) can determine the averaging direction based on reference pixels R31, R32, R33, and R34. For example, when the difference RDh between the reference pixels in the horizontal direction HD (e.g., the difference between the reference values ​​of reference pixels R31 and R32) is equal to or less than the difference RDv between the reference pixels in the vertical direction VD (e.g., the difference between the reference values ​​of reference pixels R33 and R34), the Balanced Mode Compressor (BMC) can determine the horizontal direction HD as the averaging direction. Conversely, when the difference RDh in the horizontal direction HD is greater than the difference RDv in the vertical direction VD, the Balanced Mode Compressor (BMC) can determine the vertical direction VD as the averaging direction. As another example, when the difference between the reference pixel difference RDh in the horizontal direction HD and the target pixel difference TDh in the horizontal direction HD (e.g., the difference between the pixel values ​​of target pixels T0 and T1) (e.g., the absolute value of (RDh-TDh)) is equal to or less than the difference between the reference pixel difference RDv in the vertical direction HV and the target pixel difference TDv in the vertical direction HV (e.g., the difference between the pixel values ​​of target pixels T0 and T3) (e.g., the absolute value of (RDv-TDv)), the Balanced Mode Compressor BMC can determine the horizontal direction HD as the averaging direction; conversely, the vertical direction VD can be determined as the averaging direction. However, the embodiments are not limited to this, and the Balanced Mode Compressor BMC can determine the averaging direction in various ways. In one embodiment, the Balanced Mode Compressor BMC can determine a preset direction as the averaging direction.

[0105] Refer to Figure 5 The Balanced Mode Compressor (BMC) can average the pixel value of the target pixel according to the determined averaging direction (S130).

[0106] like Figure 6B As shown, when the horizontal direction HD is determined as the averaging direction, the Balanced Mode Compressor BMC can average the pixel values ​​of target pixels T0 and T1, as well as the pixel values ​​of target pixels T2 and T3. Therefore, the average value AVGh0 of target pixels T0 and T1 (which is 287 (=(283+291) / 2)) and the average value AVGh1 of target pixels T2 and T3 (which is 300 ( (298+301) / 2)).

[0107] like Figure 6C As shown, when the vertical direction VD is determined as the averaging direction, the Balanced Mode Compressor BMC can average the pixel values ​​of target pixels T0 and T2, as well as the pixel values ​​of target pixels T1 and T3. Therefore, the average value AVGv0 of target pixels T0 and T2 (which is 291) can be calculated. (283+298) / 2)) and the average value of target pixels T1 and T3, AVGv1 (which is 296 (=(291+301) / 2)).

[0108] The Balanced Mode Compressor (BMC) can generate balance information based on reference pixels (S140).

[0109] A difference may exist between the average value and the pixel value. The Balanced Mode Compressor (BMC) can generate balance information to compensate for this difference. The balance information may include information for compensating for the difference between the average value and the pixel value, and may include a selection value and a slope value. The selection value indicates whether, during encoding operations, the pixel value is recovered by applying the difference between reference values ​​of adjacent pixels to the average value (i.e., the absolute value of the difference). The slope value indicates whether the difference (or a preset default value) is added to the average value relative to a specific pixel or subtracted from the average value (or a preset default value).

[0110] Reference Figure 6D When the averaging direction is determined to be the horizontal direction HD, the Balanced Mode Compressor BMC can calculate the difference Rd0 between reference pixels R31 and R32 and the difference Td0 between target pixels (e.g., T0 and T1), and can determine a first selection value by comparing the difference Rd0 with the difference Td0, the first selection value indicating whether to apply different values ​​as compensation values ​​when decoding target pixels T0 and T1.

[0111] For example, when the difference Rd0 of the reference pixel is not '0' and is equal to or less than the difference Td0 of the target pixel, the Balanced Mode Compressor (BMC) can determine a first selection value, such as the code value '1', which indicates that the difference Rd0 of the reference pixel should be applied as a compensation value.

[0112] When the difference Rd0 of the reference pixels is equal to or greater than the difference Td0 of the target pixels and is '0', the Balanced Mode Compressor (BMC) can determine, for example, a first selection value with the code value '0', indicating that the difference Rd0 of the reference pixels should not be used as a compensation value. In this case, a preset default value can be applied as the compensation value. Depending on the degree of loss of the average value, an optimal compensation value can be set as the default value. For example, when four bits of the average value are lost in the bitstream, the default value can be set to 4.

[0113] When a reference pixel difference Rd0, which is greater than the target pixel difference Td0, is applied as a compensation value during decoding, the difference between the recovered pixel value of the target pixel (i.e., the decoded pixel value) and the original pixel value (i.e., the pixel value before encoding) can be larger than the difference between the average value and the original pixel value. Conversely, when the reference pixel difference Rd0 is '0' and is applied as a compensation value, the difference between the average value and the original pixel value does not need to be compensated. Therefore, when the reference pixel difference Rd0 is greater than the target pixel difference Td0 or is '0', the Balanced Mode Compressor (BMC) can determine the first selection value as the code value '0', and during decoding, it can apply the default value instead of the reference pixel difference Rd0 as a compensation value based on the code value '0' of the first selection value.

[0114] The Balanced Mode Compressor (BMC) determines a first slope value that indicates which of the target pixels T0 and T1 has a larger value. In other words, the first slope value indicates whether, for a specific pixel in target pixels T0 and T1, the difference is added to the average or subtracted from the average. For example, when the first slope value is code value '0', the pixel value of target pixel T0 can be greater than the pixel value of target pixel T1; when the first slope value is code value '1', the pixel value of target pixel T1 can be greater than the pixel value of target pixel T0. Therefore, when the first slope value is '0', during decoding, the compensation value (difference Rd0 or the default value) can be restored and added to the average (e.g., ...). Figure 6B The value of AVGh0 in the mean value can be used as the pixel value of the target pixel T0, and the value of the mean value AVGh0 minus the compensation value can be recovered as the pixel value of the target pixel T1.

[0115] Similar to the above, the Balanced Mode Compressor (BMC) can calculate the difference Rd1 between reference pixels R33 and R34 and the difference Td1 between target pixels T2 and T3. When decoding target pixels T2 and T3, it can determine a second selection value indicating whether to apply the difference Rd1 by comparing Rd1 with Td1. Additionally, the Balanced Mode Compressor (BMC) can determine a second slope value indicating the slope of target pixels T2 and T3.

[0116] When the averaging direction is determined to be the vertical direction VD, the Balanced Mode Compressor BMC can generate balance information in a manner similar to that described above.

[0117] Refer to Figure 5 The Balanced Mode Compressor (BMC) can generate a bitstream that includes average value, balance information, and compression information (S150).

[0118] On the other hand, as mentioned above Figure 3AThe compression method described may further include: generating recovered pixel values ​​by decompressing the bitstream after generating the bitstream; and generating recovered image data based on the recovered pixel values, the recovered image data including reference pixels to be used for compressing the next target pixel group to be compressed after the target pixel group.

[0119] Figure 7A and Figure 7B A bitstream according to an embodiment is shown.

[0120] It can be assumed that the target pixels before compression (e.g., Figure 4 The pixel values ​​of T0, T1, T2 or T3 in the bitstream are represented by 10 bits of data, and the bitstreams BSa and BSB each have 20 bits of data as the target pixel group TG is compressed at a compression rate of about 50%.

[0121] Reference Figure 7A and Figure 7B Bitstreams BSa and BSB may include a header HD and a data block DB. The header HD may include mode information, compression ratio, loss information, etc., with an encoding method for compression (e.g., DPCM method, HV balanced mode encoding method, etc.), and the data block DB may include information based on the pixel values ​​of target pixels T0, T1, T2, and T3, such as the multiple average values ​​and balance information related to the average values.

[0122] When bitstream BSa or BSB is sent to an image processing device (e.g., Figure 1 When 200 in the code is used as compressed data in CDT, the decoder (e.g., Figure 1 230) can determine the compression method (e.g., encoding method) based on the mode information included in the header HD, and can decompress the compressed data CDT by decoding the bit streams BSa and BSB based on the information included in the decompression method (e.g., decoding method) corresponding to the compression method and the data block DB.

[0123] The header HD can be allocated from bit 17 (B16) to bit 20 (B19). Since four bits are allocated to the header HD, the header HD can include one of 4×4 (=16) pattern information.

[0124] Data block DB is allocated to bits 1 through 16 (B15). Data block DB may include a first averaging area AVE0, a second averaging area AVE1, and a balance information area (BIF). Each of the first averaging area AVE0 and the second averaging area AVE1 may be allocated six bits. The first averaging area AVE0 may be allocated to bits 11 through 16 (B15), and the second averaging area AVE1 may be allocated to bits 5 (B4) through 10 (B9). The first averaging area AVE0 and the second averaging area AVE1 may include the average values ​​of the target pixels AVGh0 and AVGh1 (or AVGv0 and AVGv1).

[0125] The Balanced Information Area (BIF) can be allocated to four bits, namely, the first bit B0 to the fourth bit B3.

[0126] Reference Figure 7A The Balance Information Area (BIF) may include a first slope value S0, a second slope value S1, a first selection value b0, and a second selection value b1. In this case, information about the averaging direction can be included in the header HD as pattern information.

[0127] Reference Figure 7B The Balance Information Area (BIF) may include information such as the averaging direction. The BIF may include information about the averaging direction (D), a slope value (S), a first selection value (b0), and a second selection value (b1). The same slope value (S) can be applied to target pixels T0 and T1 (or T0 and T2) and target pixels T2 and T3 (or T1 and T3). For example, when the slope value (S) is the code value '1', during encoding, for target pixel T1 in target pixels T0 and T1, a compensation value (e.g., ...) can be applied. Figure 6D The difference Rd0 or the default value is added to the average (e.g., AVGh0), and for target pixel T0, the compensation value can be subtracted from the average. Additionally, for target pixel T3 in target pixels T2 and T3, the compensation value (e.g., ...) can be added to the average. Figure 6D The difference Rd1 or the default value is added to the average value (e.g., AVGh1), ​​and for the target pixel T2, the compensation value can be subtracted from the average value.

[0128] Figure 8 An embodiment of a compression method according to a comparative example of a compression method is shown, wherein substantially repetitive descriptions may be omitted. In comparison, although... Figures 6A to 6D and Figures 7A to 7B A compression method based on the HV method can be shown, but the embodiments are not limited thereto. For example, Figure 8 This illustrates a compression method based on the DPCM method utilizing reference mapping RM, for example, which can be achieved using a normal mode compressor (e.g., Figure 3A and Figure 3BThe NMC in the middle performs the compression method.

[0129] Reference Figure 8 The normal mode compressor NMC can compress target pixels T0, T1, T2, and T3 based on the reference value of at least one reference pixel included in the reference map RM. For example, reference pixel R32 of the reference map RM can be used for compression of target pixel group TG.

[0130] The normal mode compressor NMC can calculate the difference d0 between the reference value of reference pixel R32 and the pixel value of target pixel T0. For example, the reference value 127 can be subtracted from the pixel value 283, so the difference d0 can be 156. In this case, the difference d0 can be negative when the pixel value is less than the compensation reference value. Alternatively, the difference d0 can be generated by subtracting the pixel value from the compensation reference value.

[0131] The normal mode compressor NMC can calculate the differences d1, d2, and d3 between target pixels T0, T1, T2, and T3. The differences d1, d2, and d3 can be 78 (=361-283), -41 (=320-361), and 82 (=402-320), respectively.

[0132] The Balanced Mode Compressor (BMC) can generate a bitstream BS' for the target pixel group TG based on the difference d0 between the reference value of the reference pixel R32 and the pixel value of the target pixel T0, as well as the differences d1, d2, and d3 between the target pixels T0, T1, T2, and T3.

[0133] The bitstream BS' may include a header HD and a data block DB, and the data block DB may include multiple remaining regions, such as first remaining regions to fourth remaining regions RD0, RD1, RD2, and RD3. For example, the first remaining regions to fourth remaining regions RD0, RD1, RD2, and RD3 may include references... Figure 8 The calculated differences are d0, d1, d2, and d3. As a non-restrictive example, the values ​​included in the bitstream BS' can be represented as negative and positive values ​​using the most significant bit notation.

[0134] Each of the first to fourth remaining regions RD0, RD1, RD2, and RD3 can be allocated the same number of bits. For example, the fourth remaining region RD3 can be allocated to bits B0 to B3, the third remaining region RD2 can be allocated to bits B4 to B7, the second remaining region RD1 can be allocated to bits B8 to B11, and the first remaining region RD0 can be allocated to bits B12 to B15.

[0135] On the other hand, the difference d0 can be 156, and the differences d1, d2, and d3 can be 78, -41, and 82, respectively. The differences d1, d2, and d3 can be represented by a similar or equal number of bits (e.g., eight bits, including seven bits representing the absolute value and one bit representing the sign). However, the difference d0 can be significantly different from the differences d1, d2, and d3, and may require eight bits to represent the absolute value of d0. However, because the first to fourth residual regions RD0, RD1, RD2, and RD3 are allocated the same number of bits, significant data loss occurs when the normal mode compressor NMC performs compression on the pixel group PG of the isolated region based on the compression method according to the current comparison example.

[0136] Therefore, when the target pixel group TG corresponds to the isolation region, the image sensor module according to the embodiment (e.g., Figure 1 The 100 in the middle can reduce compression loss and improve compression efficiency by using the HV balanced coding method to compress the target pixel group TG.

[0137] Figure 9 Image data and reference mapping according to an embodiment are shown.

[0138] exist Figure 9 In this context, the image data IDTa may include a Bayer pattern. In one embodiment, a pixel group PG can be configured using four consecutively arranged pixel units. In another embodiment, the pixel group PG may include two red pixels and two green pixels, or two blue pixels and two green pixels.

[0139] Compression can be performed on a pixel group PG basis, and reference values ​​(e.g., R11, R12, R13, R14, R21, R22, R23, R24, R31, R32, R33, and R34) of the pixel group PG corresponding to the pixel group PG of the same color that is compressed before and adjacent to the target pixel group TG can be generated as a reference map RM for compressing the target pixel group TG.

[0140] There can be a significant difference (e.g., 125 or greater, but not limited to) between the target pixel group TG and the pixel values ​​of the reference pixels R11, R12, R13, R14, R21, R22, R23, R24, R31, R32, R33, and R34, and the target pixel group TG can be identified as corresponding to an isolation region. Therefore, the target pixel group TG can be compressed based on the HV balanced coding method.

[0141] Figure 10 A compression method according to an embodiment is illustrated. Compression can be achieved by utilizing a reference mapping (RM). Figure 9 Image data IDTa execution Figure 10Compression methods.

[0142] Reference Figure 10 Balanced mode compressor (e.g., Figure 3A The BMC (Block Matrix Map) in the reference map can compress target pixels T0, T1, T2, and T3 based on reference values ​​of at least one pixel group included in the reference map RM. For example, reference pixels R31, R32, R33, and R34 can be used to compress target pixel group TG. During compression, reference pixels with the same color as the target pixels can be used.

[0143] Pixel group PG comprises four pixels arranged consecutively in the horizontal direction HD, and the Balanced Mode Compressor (BMC) can determine the horizontal direction HD as the averaging direction. The Balanced Mode Compressor (BMC) can calculate a first average value AVGh0 by averaging the pixel values ​​corresponding to target pixels T0 and T2 of the same color, and calculate a second average value AVGh1 by averaging the pixel values ​​corresponding to target pixels T1 and T3 of the same color.

[0144] The Balanced Mode Compressor (BMC) can generate balance information (e.g., a first selection value and a first slope value) for target pixels T0 and T2 based on the difference Rd0 between the reference values ​​of reference pixels R31 and R33 and the difference Td0 between the pixel values ​​of target pixels T0 and T2. The Balanced Mode Compressor (BMC) can also generate balance information (e.g., a second selection value and a second slope value) for target pixels T1 and T3 based on the difference Rd1 between the reference values ​​of reference pixels R32 and R34 and the difference Td1 between the pixel values ​​of target pixels T1 and T3.

[0145] The Balanced Mode Compressor (BMC) can calculate the difference d0 between the compensation reference value of reference pixel R33 and the pixel value of target pixel T0. Furthermore, it calculates the difference d1 between the pixel values ​​of target pixels T0 and T2. Additionally, the BMC can calculate the difference d2 between the compensation reference value of reference pixel R34 and the pixel value of target pixel T1, and the difference d3 between the pixel values ​​of target pixels T1 and T3. The BMC can generate the bitstream BS based on the differences d0, d1, d2, and d3 (see, for example, [link to documentation]). Figure 7A Bsa, Figure 7B BSb and / or Figure 8 (The BS' is not limited to this). Differences d0, d1, d2, and d3 may be included in the data block DB of the bitstream BS.

[0146] The balanced mode compressor (BMC) can generate a bitstream (BS) based on a first average value AVGh0, a second average value AVGh1, and balance information. Figure 7A BSA or Figure 7B (BSb).

[0147] Figure 11A and Figure 11B An image sensor module according to an embodiment is shown.

[0148] Reference Figure 11A The image sensor module 100' may include substantially the same as Figure 1 The image sensor 100 is identical to the image sensor 110, encoder 120, I / F 130, and processing logic 150. Repeated descriptions may be omitted.

[0149] Image sensor 110 may include a pixel array PXA and a driving and readout circuitry DRC. As described above, the pixel array PXA may include a plurality of pixels PX arranged in multiple rows and columns. The driving and readout circuitry DRC can control the pixel array PXA and convert pixel signals received from the pixel array PXA into pixel values. The driving and readout circuitry DRC can generate raw image data RDT including pixel values ​​corresponding to the received pixel signals.

[0150] Processing logic 150 can perform preprocessing on the raw image data RDT. For example, preprocessing may include image processing such as bad pixel correction, crosstalk compensation, noise reduction, binning, resizing, and color space conversion.

[0151] Encoder 120 can generate compressed data CDT by compressing image data IDT (or raw image data RDT) received from processing logic 150. Encoder 120 can compress image data IDT on a pixel-by-pixel basis and compress target pixel groups to be compressed by utilizing neighboring pixel groups that have already been compressed. As described above, when the target pixel group corresponds to an isolation zone, encoder 120 can compress the target pixel group by utilizing the HV balanced coding method.

[0152] The compressed data CDT can be provided to I / F 130, and I / F 130 can send the compressed data CDT to an image processing device (e.g., Figure 1 (200 in the middle).

[0153] Reference Figure 11B The image sensor module 100a may also include a memory 160. The memory 160 may include volatile memory such as DRAM and SRAM, or non-volatile memory such as PRAM, ReRAM, MRAM, and flash memory. Compressed data CDT generated by the encoder 120 may be stored in the memory 160. The compressed data CDT can be read from the memory 160 and output via I / F 130.

[0154] Figure 12A decoder 220 according to an embodiment is shown.

[0155] Decoder 220 can be generated by encoder (e.g., in reverse order) Figure 1 120) performs a series of operations for encoding image data IDT, producing decompressed data DDT (e.g., recovered image data) by decompressing compressed data CDT.

[0156] Decoder 220 can decompress compressed data CDT by utilizing a decoding method corresponding to the encoding method used by encoder 120. Decoder 220 can decode compressed data CDT in units of bit stream.

[0157] Decoder 220 may include reference pixel detector 221, pattern decoder 222, decompressor 223 and reference buffer 224.

[0158] Reference pixel detector 221 can receive compressed data CDT and receive from reference buffer 224 a reference map RM of the target bitstream to be decompressed, which will be used to decompress the bitstream included in the compressed data CDT. The reference map RM may include reference values ​​of neighboring pixels of the pixel group associated with the bitstream to be referenced (i.e., reference values ​​of reference pixels).

[0159] Reference pixel detector 221 can detect reference values ​​(i.e., the recovered pixel values ​​of reference pixels) at reference pixel locations adjacent to the target pixel group in the recovered image data stored in the reference buffer, and can receive the reference values ​​from the reference buffer 224 as a reference map RM. Reference pixel detector 221 can provide the target bitstream of compressed data CDT and the reference map RM to the pattern decoder 222 or the decompressor 223.

[0160] The pattern decoder 222 can decode the header HD of the bitstream and determine pattern information, compression ratio, lost information, etc., by using the decoding result. According to an embodiment, the pattern decoder 222 can verify that compression has been performed in the HV balanced coding method or another coding method (e.g., the DPCM method) by using the result of decoding the header HD.

[0161] The decompressor 223 can recover target pixels from the bitstream based on a determined compression mode, compression ratio, lost information, etc. According to an embodiment, when it is determined by utilizing the result of decoding the header HD that a bitstream has been generated in the HV balanced encoding method, the decompressor 223 can verify the average value and balance information from the data blocks of the bitstream, and can recover the pixel values ​​of the compressed pixels by applying compensation values ​​to the average value based on the balance information. The pixel group generated by decoding the bitstream can be output as decompressed data DDT.

[0162] Reference buffer 224 can store decompressed data DDT, i.e., the recovered image data. In an embodiment, reference buffer 224 can store pixel groups corresponding to the next bitstream to be decompressed from the recovered image data. In an embodiment, an image processing device (e.g., Figure 1 The memory or buffer configured in 200) (e.g., memory 220) can be used as reference buffer 224.

[0163] Figure 13 An image processing system 10b according to an embodiment is shown. The image processing system 10b is... Figure 1 Modifiable embodiments of the image processing system 10.

[0164] Reference Figure 13 The image processing system 10b may include an image sensor module 100b and an image processing device 200b. The image sensor module 100b may include an image sensor 110 and an I / F 130. The image sensor module 100b may also include a memory. The image processing device 200b may include an I / F 210, an encoder 250, a decoder 230, an image signal processor 240, and a memory 220. Figure 13 The encoder 250 in the middle can correspond to Figure 1 Encoder 120 in the middle.

[0165] Will Figure 13 Image processing system 10b and Figure 1 Compared to the image processing system 10, the image processing device 200b, which differs from the image sensor module 100b, may include an encoder 250, and the image processing device 200b can compress image data IDT. Other components may be substantially the same as each other. (The last sentence appears to be incomplete and possibly refers to a different image processing system.) Figure 1 The components of the image processing system 10 are substantially similar to those of the components of the image processing system 10b, and are thus described repeatedly.

[0166] Reference Figure 13 The image sensor module 100b can generate image data IDT (raw image data or preprocessed image data). The image data IDT can be transmitted to the image processing device 200b via I / F 130. The image processing device 200b can receive the image data IDT from the image sensor module 100b, encode the image data IDT to form compressed data CDT, and store the compressed data CDT in memory 220. Then, the decoder 230 can read the compressed data CDT stored in memory 220 and decompress the compressed data CDT. The decoder 230 can provide the compressed data CDT (e.g., recovered image data) to the image signal processor 240.

[0167] As mentioned above, compression and decompression can be performed on a pixel-by-pixel basis, and pixel groups corresponding to the isolated regions of the compressed data CDT can be compressed or decompressed based on the HV balanced coding method.

[0168] Figure 14 Compression information according to an embodiment is shown. Figure 14 An example of a compression pattern (or compression method) based on a standard proposed by the Mobile Industrial Processor Interface (MIPI) Alliance is shown.

[0169] Reference Figure 14 Image data of four-grid patterns can be compressed according to various compression modes (e.g., Figure 2 (IDT in the example). However, the embodiments are not limited to this, and image data in which red pixel groups, blue pixel groups, first green pixel groups and second green pixel groups, which are alternately arranged in an n×n matrix, can be compressed according to various compression modes.

[0170] As compression modes, the following can be used: Average-based directional difference (AD) mode, Extended Multi-Pixel Difference (eMPD) mode, Extended Horizontal or Vertical Difference (eHVD) mode, Extended Horizontal or Vertical Average Difference (eHVA) mode, Tilt-based Difference (OD) mode, Extended Outlier Compensation (eOUT) mode, Outlier Compensation (OUT) mode, and Fixed Quantization and No Reference (FNR) mode. However, the names of the above compression modes may be merely examples, and the embodiments are not limited thereto.

[0171] In AD mode, the target pixel group can be encoded using the DPCM method. For example, a bitstream can be generated based on the difference between the average values ​​of the pixel values ​​in the target pixel group and the difference between each pixel value and the average value (e.g., Figure 7A (BSa in the middle).

[0172] Depending on the specific algorithm implemented, AD modes can be divided into MODE0, MODE1, MODE2, and MODE3. Since four bits can be allocated to the header representing the compression method, the sixteen compression modes can represent header information using different bits from each other. For example, MODE0 can be represented by bit 0000, MODE1 by bit 0001, MODE2 by bit 0010, and MODE3 by bit 0011.

[0173] In OD mode, diagonal image data IDT can be compressed. Depending on the specific algorithm implemented, OD mode can be divided into MODE4 (e.g., bit 0100) and MODE5 (e.g., bit 0101). Similarly, eMPD mode can include MODE8 (e.g., bit 1000), MODE9 (e.g., bit 1001), MODE10 (e.g., bit 1010), and MODE11 (e.g., bit 1011), and eHVD mode can include MODE12 (e.g., bit 1100) and MODE13 (e.g., bit 1101).

[0174] The eHVA mode may include MODE14 (e.g., bit 1110). The HV balanced mode according to an embodiment may correspond to the eHVA mode, and the balanced mode compressor (e.g., Figure 3A The BMC in the middle can generate a bit stream that includes a header HD indicating the eHVA mode.

[0175] The eOUT mode may include MODE15 (e.g., bit 1111), and the OUT mode may include MODE7 (e.g., bit 0111). The FNR mode may include MODE6 (e.g., bit 0110). In an embodiment, MODE7 (e.g., 0111) may be included in the eOUT mode depending on the value stored in the register.

[0176] In an embodiment, a mode selector (e.g., Figure 3A and Figure 3B (123) can evaluate AD mode, eMPD mode, eHVD mode, eHVA mode, OD mode, eOUT mode, OUT mode, and FNR mode in sequence, and the optimal mode can be selected based on compression evaluation metrics such as compression ratio and lost information. However, this disclosure is not limited to the described mode evaluation order.

[0177] Figure 15A An electronic device 1000 including a multi-camera module with multiple camera modules is shown. Figure 15B It shows Figure 15A The camera module in the system.

[0178] Reference Figure 15A The electronic device 1000 may include a multi-camera module 1100, an application processor 1200, a PMIC 1300, and an external memory 1400.

[0179] The multi-camera module 1100 may include multiple camera modules 1100a, 1100b, and 1100c. Although the accompanying drawings show an embodiment in which three camera modules 1100a, 1100b, and 1100c are arranged, the embodiments are not limited thereto. In an embodiment, the multi-camera module 1100 may include only two camera modules, or it may be modified and implemented to include k (where k is a natural number equal to or greater than 4) camera modules.

[0180] In the following text, refer to Figure 15B The detailed construction of camera module 1100b will be described below, but the following description can be equally applied to other camera modules 1100a and 1100c according to the embodiments.

[0181] Reference Figure 15B The camera module 1100b may include a prism 1105, an optical path folding element (hereinafter referred to as OPFE) 1110, an actuator 1130, an image sensing device 1140, and a storage unit 1150.

[0182] The prism 1105 can change the path of light L incident from the outside, and includes a reflective surface 1107 of reflective material.

[0183] In several embodiments, prism 1105 can change the path of light L incident in the first direction X to a second direction Y perpendicular to the first direction X. Additionally, prism 1105 can rotate the reflective surface 1107 of the reflective material about its central axis 1106 to direction A, or change the path of light L incident in the first direction X to the second direction Y by rotating the central axis 1106 to direction B. In this case, OPFE 1110 can be moved to a third direction Z perpendicular to both the first and second directions X and Y.

[0184] In several embodiments, as shown, the maximum rotation angle of prism 1105 in direction A may be equal to or less than about 15 degrees in the positive (+) direction A and greater than about 15 degrees in the negative (-) direction A, but the embodiments are not limited thereto.

[0185] In several embodiments, the prism 1105 may be moved by an angle of approximately 20 degrees in the positive (+) direction B or the negative (-) direction B, or between approximately 10 degrees and approximately 20 degrees, or between approximately 15 degrees and approximately 20 degrees. In this case, the angle of movement may be the same angle in the positive (+) direction B or the negative (-) direction B, or nearly similar within a range of approximately 1 degree.

[0186] In several embodiments, the prism 1105 can move the reflective surface 1107 to a third direction (e.g., the Z direction) parallel to the extension direction of the central axis 1106.

[0187] In several embodiments, the camera module 1100b may include two or more prisms, and the path of light L incident in the first direction X through these prisms may be changed in different ways, such as changing to a second direction Y perpendicular to the first direction X, changing back to the first direction X or a third direction Z, and returning to the second direction Y.

[0188] For example, OPFE 1110 may include optical lenses comprising m groups (where m is a natural number). The m lenses can move in the second direction Y, changing the optical zoom ratio of camera module 1100b. For example, when the basic optical zoom ratio of camera module 1100b is Z, and the m optical lenses included in OPFE 1110 move, the optical zoom ratio of camera module 1100b can be changed to 3Z, 5Z, or a larger optical zoom ratio.

[0189] Actuator 1130 can move OPFE 1110 or the optical lens (hereinafter referred to as the optical lens) to a specific position. For example, actuator 1130 can adjust the position of the optical lens so that image sensor 1142 is located at the focal length of the optical lens for accurate sensing.

[0190] The image sensing device 1140 may include an image sensor 1142, control logic 1144, and memory 1146. It can be applied... Figure 1 Image sensor module 100 or Figure 13 The image sensor module 100b in the image sensor module 1140 serves as the image sensing device 1140.

[0191] Image sensor 1142 can sense an image of a target by utilizing light L provided through an optical lens. Control logic 1144 can control all operations of camera module 1100b and process the sensed images. For example, control logic 1144 can control the operation of camera module 1100b according to control signals provided via control signal line CSLb, and can perform image processing (such as extracting image data corresponding to a specific image from the sensed image (e.g., a person's face, arms, legs, etc. in the image)) and noise reduction.

[0192] In an embodiment, control logic 1144 may include an encoder (e.g., Figure 1 The encoder 120 (as described above) can compress (or encode) the sensed or image-processed image. As stated above, the encoder 120 can compress the image in pixel groups and compress the pixel groups of the isolated region according to the HV balanced coding method.

[0193] The memory 1146 may store information required for the operation of the camera module 1100b, such as calibration data 1147. Calibration data 1147 may be information required to generate image data using light L provided externally to the camera module 1100b, and may include information such as rotation angle, focal length, and optical axis. When the camera module 1100b is implemented as a multi-state camera in which the focal length varies according to the position of the optical lens, the calibration data 1147 may include the focal length value for each position (or state) of the optical lens and information regarding autofocus.

[0194] In several embodiments, compressed data may be stored in memory 1146. Additionally, memory 1146 may be used as a reference buffer 125 for encoder 120.

[0195] The storage unit 1150 can store image data sensed by the image sensor 1142. The storage unit 1150 can be disposed outside the image sensing device 1140 and can be implemented as sensor chips constituting the image sensing device 1140 in a stacked manner. In various embodiments, the image sensor 1142 may include a first chip, and the control logic 1144, the storage unit 1150, and the memory 1146 may include a second chip; therefore, they can all be implemented in the form of two chips stacked together.

[0196] In several embodiments, the storage unit 1150 may be implemented as an electrically erasable programmable read-only memory (EEPROM), but the embodiments are not limited thereto. In several embodiments, the image sensor 1142 may include a pixel array, and the control logic 1144 may include an analog-to-digital converter and an image signal processor for processing the sensed image.

[0197] Refer to together Figure 15A and Figure 15B In several embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Therefore, each of the plurality of camera modules 1100a, 1100b, and 1100c may include calibration data 1147 that may be the same as or different from each other, depending on the operation of the included actuator 1130.

[0198] In several embodiments, one of the multiple camera modules 1100a, 1100b and 1100c (e.g. 1100b) may include a folding lens type camera module having the prism 1105 and OPFE 1110 described above, and the other camera modules (e.g. 1100a and 1100c) may include a vertical type camera module without the prism 1105 and OPFE 1110, but the embodiments are not limited thereto.

[0199] In several embodiments, one of the multiple camera modules 1100a, 1100b, and 1100c (e.g., 1100c) may include a vertical depth camera that extracts depth information using, for example, infrared (IR). In this case, the application processor 1200 can generate a three-dimensional (3D) depth image by merging image data provided by the vertical depth camera with image data provided by other camera modules (e.g., 1100a or 1100b).

[0200] In several embodiments, at least two camera modules (e.g., 1100a and 1100b) of the plurality of camera modules 1100a, 1100b, and 1100c may have different fields of view from each other. In this case, for example, the optical lenses of at least two camera modules (e.g., 1100a and 1100b) of the plurality of camera modules 1100a, 1100b, and 1100c may be different from each other, but the embodiments are not limited thereto.

[0201] Furthermore, in several embodiments, the field of view of each of the plurality of camera modules 1100a, 1100b, and 1100c may be different from each other. For example, camera module 1100a may include an ultra-wide-angle camera, camera module 1100b may include a wide-angle camera, and camera module 1100c may include a telephoto camera, but the embodiments are not limited thereto. In this case, the optical lenses included in each of the plurality of camera modules 1100a, 1100b, and 1100c may also be different from each other, but the embodiments are not limited thereto.

[0202] In several embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may be physically separated from each other. In other words, the sensing area of ​​an image sensor 1142 may not be divided and used by the plurality of camera modules 1100a, 1100b, and 1100c, and an independent image sensor 1142 may be arranged within each of the plurality of camera modules 1100a, 1100b, and 1100c.

[0203] Refer to Figure 15A The application processor 1200 may include an image processing device 1210, a memory controller 1220, and internal memory 1230. The application processor 1200 and the plurality of camera modules 1100a, 1100b, and 1100c may be implemented as separate semiconductor chips, for example, separate semiconductor chips.

[0204] The image processing apparatus 1210 may include a plurality of sub-image processors 1212a, 1212b and 1212c, an image generator 1214 and a camera module controller 1216.

[0205] The image processing apparatus 1210 may include a plurality of sub-image processors 1212a, 1212b and 1212c corresponding to the number of a plurality of camera modules 1100a, 1100b and 1100c.

[0206] Image data generated by camera module 1100a can be provided to sub-image processor 1212a via image signal line ISL1, image data generated by camera module 1100b can be provided to sub-image processor 1212b via image signal line ISL1, and image data generated by camera module 1100c can be provided to sub-image processor 1212c via image signal line ISL1. Image data transmission can be performed using, for example, MIPI-based CSI, but the embodiments are not limited thereto.

[0207] In an embodiment, at least one of the plurality of sub-image processors 1212a, 1212b, and 1212c may include a decoder (e.g., Figure 1 (230 in the text). When the corresponding camera modules 1100a, 1100b and 1100c include an encoder (e.g., Figure 1 In the case of 120), the multiple sub-image processors 1212a, 1212b and 1212c may include a decoder 230 for decompressing compressed image data.

[0208] In several embodiments, Figure 13 The image processing device 200b can be implemented as at least one of a plurality of sub-image processors 1212a, 1212b, and 1212c, and at least one of the plurality of sub-image processors 1212a, 1212b, and 1212c may include an encoder (e.g., Figure 13 (250 in the middle) and decoder (e.g., Figure 13 (230 in the middle).

[0209] On the other hand, in several embodiments, a sub-image processor may be arranged to correspond to multiple camera modules. For example, sub-image processors 1212a and 1212c may not be implemented as separate from each other as shown, but may instead be integrated into a single sub-image processor, and image data provided by camera modules 1100a and 1100c may be provided to the integrated sub-image processor after selection by a selected element (e.g., a multiplexer). In this case, sub-image processor 1212b may not be integrated and may receive image data from camera module 1100b.

[0210] In several embodiments, image data generated by camera module 1100a can be provided to sub-image processor 1212a via image signal line ISL1, image data generated by camera module 1100b can be provided to sub-image processor 1212b via image signal line ISL1b, and image data generated by camera module 1100c can be provided to sub-image processor 1212c via image signal line ISL1c. Alternatively, image data processed by sub-image processor 1212b can be directly provided to image generator 1214; however, image data processed by sub-image processors 1212a and 1212c can be provided to image generator 1214 after either a selected element (e.g., a multiplexer) is selected.

[0211] Each of the sub-image processors 1212a, 1212b and 1212c can perform image processing on image data provided by the multiple camera modules 1100a, 1100b and 1100c, such as bad pixel correction, autofocus correction, auto white balance, auto exposure (3A adjustment), noise reduction, sharpening, gamma control and re-mosaic.

[0212] In several embodiments, the re-stitching signal processing can be provided to the sub-image processors 1212a, 1212b and 1212c after being performed by each of the camera modules 1100a, 1100b and 1100c.

[0213] Image data processed by each of the sub-image processors 1212a, 1212b, and 1212c can be provided to the image generator 1214. The image generator 1214 can generate an output image by utilizing the image data provided by each of the sub-image processors 1212a, 1212b, and 1212c, based on image generation information or pattern signals.

[0214] Based on image generation information or a pattern signal, image generator 1214 can generate an output image by merging at least a portion of the image data generated by image processors 1212a, 1212b, and 1212c. Alternatively, based on image generation information or a pattern signal, image generator 1214 can generate an output image by selecting any one of the image data generated by image processors 1212a, 1212b, and 1212c.

[0215] In several embodiments, the image generation information may include a zoom signal or a zoom factor. Additionally, in several embodiments, the mode signal may, for example, include a signal based on a user-selected mode.

[0216] When the image generation information includes a zoom signal (zoom factor), and camera modules 1100a, 1100b, and 1100c have different fields of view, image generator 1214 can perform different operations depending on the type of zoom signal. For example, when the zoom signal includes a first signal, an output image can be generated by utilizing image data output by sub-image processor 1212a and sub-image processor 1212c, including image data output from sub-image processor 1212a and sub-image processor 1212b. When the zoom signal includes a second signal different from the first signal, an output image can be generated by utilizing image data output by sub-image processor 1212a and sub-image processor 1212c, including image data output from sub-image processor 1212c and sub-image processor 1212b. When the zoom signal includes a third signal that is different from the first and second signals, the image generator 1214 may not perform this image data merging and may generate an output image by selecting any one of the image data output by each of the sub-image processors 1212a, 1212b, and 1212c. However, the embodiments are not limited thereto, and the method of processing image data may be modified and executed as needed.

[0217] In several embodiments, the image processing apparatus 1210 may further include a selector that selects the outputs of sub-image processors 1212a, 1212b and 1212c and sends them to the image generator 1214.

[0218] In this case, the selector can perform different operations depending on the zoom signal or the zoom factor. For example, when the zoom signal includes a fourth signal (e.g., the zoom ratio includes a first ratio), the selector can select any one of the outputs from the sub-image processors 1212a, 1212b, and 1212c and send it to the image generator 1214.

[0219] Additionally, when the zoom signal includes a fifth signal different from the fourth signal (e.g., the zoom ratio includes a second ratio), the selector can send p (where p is a natural number equal to or greater than 2) outputs from sub-image processors 1212a, 1212b, and 1212c to image generator 1214 in sequence. For example, the selector can send the outputs from sub-image processor 1212b and sub-image processor 1212c to image generator 1214 in sequence. Alternatively, the selector can send the outputs from sub-image processor 1212a and sub-image processor 1212b to image generator 1214 in sequence. Image generator 1214 can generate an output image by combining the p sequentially provided outputs.

[0220] In this configuration, image processing (such as re-stitching, video / preview resolution downsizing, gamma control, and high dynamic range (HDR) processing) can be pre-executed by sub-image processors 1212a, 1212b, and 1212c. The processed image data can then be sent to image generator 1214. Therefore, even though the processed image data is provided to image generator 1214 as a single signal line via a selector, image merging operations can be performed by image generator 1214 at high speed.

[0221] In several embodiments, image generator 1214 may receive multiple image data with different exposure times from at least one of sub-image processors 1212a, 1212b and 1212c, perform HDR processing on the multiple image data, and then generate merged image data with increased dynamic range.

[0222] The camera module controller 1216 can provide control signals to each of the camera modules 1100a, 1100b, and 1100c. The control signals generated by the camera module controller 1216 can be provided to the corresponding camera modules 1100a, 1100b, and 1100c via separate control signal lines CSLa, CSLb, and CSLc.

[0223] Based on image generation information or mode signals, including zoom signals, any one of the multiple camera modules 1100a, 1100b, and 1100c can be designated as a master camera (e.g., 1100b), and other camera modules (e.g., 1100a and 1100c) can be designated as slave cameras. This information can be included in control signals and provided to the corresponding camera modules 1100a, 1100b, and 1100c via separate control signal lines CSL1, CSLb, and CSLc.

[0224] Depending on the zoom factor or operating mode signal, the camera module operating as the master camera or slave camera can be changed. For example, when the field of view of camera module 1100a is wider than that of camera module 1100b, and the zoom factor indicates a lower zoom ratio, camera module 1100a can operate as the master camera, and camera module 1100b can operate as the slave camera. Conversely, when the zoom factor indicates a high zoom ratio, camera module 1100b can operate as the master camera, and camera module 1100a can operate as the slave camera.

[0225] In several embodiments, the control signals provided by the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c may include a synchronization enable signal. For example, when camera module 1100b is the main camera and camera modules 1100a and 1100c are the slave cameras, the camera module controller 1216 may send a synchronization enable signal to camera module 1100b. Having received the synchronization enable signal, camera module 1100b may generate a synchronization signal based on the provided synchronization enable signal and may provide the generated synchronization signal to camera modules 1100a and 1100c via the synchronization enable signal line SSL. Camera modules 1100b, 1100a, and 1100c may synchronize with the synchronization signal and send image data to the application processor 1200.

[0226] In several embodiments, the control signals provided by the camera module controller 1216 to the plurality of camera modules 1100a, 1100b, and 1100c may include mode information based on mode signals. Based on the mode information, the plurality of camera modules 1100a, 1100b, and 1100c may operate in a first operating mode and a second operating mode with respect to sensing speed.

[0227] Multiple camera modules 1100a, 1100b, and 1100c can generate image signals at a first speed (e.g., generate image signals at a first frame rate), encode the generated image signals at a second speed higher than the first speed (e.g., encode image signals at a second frame rate higher than the first frame rate), and send the encoded image signals to the application processor 1200. In this case, the second speed can be equal to or less than 30 times the first speed.

[0228] The application processor 1200 can store the received image signal (i.e., the encoded image signal) in its internal memory 1230 or in an external memory 1400 outside the application processor 1200. It can then read and decode the encoded signal from the internal memory 1230 or the external memory 1400, and can display image data generated based on the decoded image signal. For example, corresponding subprocessors of the plurality of subprocessors 1212a, 1212b, and 1212c of the image processing device 1210 can perform decoding, and additionally, image processing can be performed on the decoded image signal.

[0229] Multiple camera modules 1100a, 1100b, and 1100c can generate image signals at a third speed lower than the first speed (e.g., generate image signals at a third frame rate lower than the first frame rate) in a second operating mode and send the image signals to application processor 1200. The image signals provided to application processor 1200 can be unencoded signals. Application processor 1200 can perform image processing on the received image signals or store the image signals in internal memory 1230 or external memory 1400.

[0230] PMIC 1300 can supply power (e.g., power supply voltage) to each of the multiple camera modules 1100a, 1100b, and 1100c. For example, under the control of application processor 1200, PMIC 1300 can supply first power to camera module 1100a via power signal line PSLa, second power to camera module 1100b via power signal line PSLb, and third power to camera module 1100c via power signal line PSLc.

[0231] The PMIC 1300 can generate power corresponding to each of the plurality of camera modules 1100a, 1100b, and 1100c in response to a power control signal PCON from the application processor 1200, and can also adjust the power level. The power control signal PCON can include a power adjustment signal for each operating mode of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the operating mode can include a low-power mode, and in this case, the power control signal PCON can include information about the camera module operating in low-power mode and the set power level. The power levels supplied to each of the plurality of camera modules 1100a, 1100b, and 1100c can be the same or different from each other. Furthermore, the power levels can be changed dynamically.

[0232] Figure 16 An electronic device 2000 according to an embodiment is shown. Figure 16 The electronic device 2000 may include a mobile terminal.

[0233] Reference Figure 16 The electronic device 2000 may include an application processor 2100, a camera module 2200, a working memory 2300, a storage unit 2400, a display device 2600, a user interface 2700, and a wireless transceiver 2500.

[0234] The application processor 2100 can control the operation of the electronic device 2000 and can be implemented as a system-on-a-chip (SoC) that drives applications, operating systems, etc. The application processor 2100 can provide image data provided by the camera module 2200 to the display device 2600 or store it in the storage unit 2400.

[0235] The above is for reference only. Figures 1 to 11B The described image sensor modules 100 and 100a can be applied to camera module 2200. Camera module 2200 may include encoder 2210, which can generate compressed data by compressing image data and send the compressed data to application processor 2100. As described above, when at least some image data (e.g., when at least one group of pixels corresponds to an isolation area), encoder 2210 can compress the group of pixels by using an HV balanced encoding method.

[0236] Application processor 2100 may include decoder 2110, which decompresses compressed data using a decoding method corresponding to the compression method (e.g., encoding method) of encoder 2210. Decoder 2110 can decompress compressed data received from camera module 2200 and generate restored image data, and application processor 2100 can perform image processing on the restored image data. Application processor 2100 can display the restored image data on display device 2600 or store the image data or image-processed image data in storage unit 2400.

[0237] The working memory 2300 can be implemented as volatile memory (such as DRAM and SRAM) or non-volatile memory (such as FeRAM, RRAM and PRAM). The working memory 2300 can store programs and / or data executed or processed by the application processor 2100.

[0238] Storage unit 2400 can be implemented as non-volatile memory (such as NAND flash memory and resistive memory), and storage unit 2400 can be provided as, for example, a memory card, such as a multimedia card (MMC), an embedded MMC (eMMC), a security card (SD), and a micro SD. Storage unit 2400 can store image data received from camera module 2200 or data processed or generated by application processor 2100.

[0239] User interface 2700 can be implemented as various devices capable of receiving user input, such as a keyboard, a curtain keypad, a touch panel, a fingerprint sensor, and a microphone. User interface 2700 can receive user input and provide the application processor 2100 with signals corresponding to the received user input.

[0240] The wireless transceiver 2500 may include a transceiver 2510, a modem 2520, and an antenna 2530.

[0241] Although this disclosure has been specifically shown and described with reference to embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the appended claims.

Claims

1. An image compression method, comprising: Receive the pixel values ​​of the target pixel group of image data to be compressed, and the reference values ​​of the reference pixels to be used in the compression of the target pixel group; Determine the averaging direction between the horizontal and vertical directions; The pixel values ​​of each pair of at least two pairs of target pixels that are adjacent to each other in the averaging direction are averaged to produce at least two average values. Balance information, including compensation values ​​to be applied to the at least two average values, is generated based on the reference pixels; as well as Generate a bitstream that includes the at least two average values, the balance information, and the averaging direction information.

2. The image compression method according to claim 1, wherein: The image data is generated by an image sensor. The reference value corresponds to the recovered pixel value of the reference pixel that has been compressed before the target pixel group, and the difference between at least one of the pixel values ​​and the reference value is equal to or greater than a threshold.

3. The image compression method according to claim 1, wherein, The averaging process includes: calculating a first average value and a second average value for two pairs of target pixels in the averaging direction within the target pixels.

4. The image compression method according to claim 3, wherein, The step of generating the balance information includes: generating balance information about the first pair of target pixels based on the difference between the first pair of target pixels in the two pairs of target pixels and the difference between the first pair of reference pixels in the reference pixels.

5. The image compression method according to claim 4, wherein, The balance information includes a first selection value and a first slope value for the first pair of target pixels. Wherein, the first selection value indicates whether to apply the difference between the first pair of reference pixels to the first average value, and The first slope value indicates the target pixel with the larger pixel value in the first pair of target pixels.

6. The image compression method according to claim 5, wherein, When the first selection value indicates that the difference between the first pair of reference pixels should not be applied to the first average value of the first pair of target pixels, the preset default value is set to be applied to the first average value.

7. The image compression method according to claim 4, wherein, The balance information includes selection values ​​and slope values ​​for each of the two pairs of target pixels.

8. The image compression method according to claim 4, wherein, The balance information includes selection values ​​for each of the two pairs of target pixels and a slope value applied together to the two pairs of target pixels.

9. The image compression method according to claim 1, wherein, The bitstream includes a header and data blocks, and The averaged direction information is included in the header or the data block.

10. The image compression method according to claim 1, further comprising: After the bitstream is generated, the recovered pixel values ​​are generated by decompressing the bitstream; as well as Based on the recovered pixel values, a recovered image is generated, the recovered image including reference pixels for compression of the next target pixel group after the target pixel group.

11. The image compression method according to claim 1, wherein, The target pixel group comprises four pixels of the same color arranged in a 2×2 matrix.

12. An image sensor module, comprising: An image sensor configured to generate image data comprising multiple pixels; An encoder is configured to generate compressed data comprising multiple bit streams by sequentially compressing image data generated by the image sensor in units of pixel groups, and the compression is performed on target pixel groups according to at least one of a variety of encoding methods. as well as An interface configured to output the compressed data to an external image processing device. The encoder determines an averaging direction in the horizontal and vertical directions according to a first encoding method among the multiple encoding methods, generates at least two average pixel values ​​for at least two pairs of target pixels that are adjacent to each other in the averaging direction in the target pixel group, generates compensation values ​​that will be applied to the at least two average pixel values, and generates a bit stream that includes the at least two average pixel values, the compensation values, and averaging direction information.

13. The image sensor module according to claim 12, wherein, The encoder determines the averaging direction based on a reference value, in which the averaging calculation will be performed on the pixel value of the target pixel, the reference value corresponding to the recovered pixel value of a reference pixel that has been compressed before the target pixel group.

14. The image sensor module according to claim 13, wherein, The encoder determines the selection value and slope value for the first pair of target pixels based on the difference between the first pair of target pixels in the averaging direction and the difference between the first pair of reference pixels in the reference pixels.

15. The image sensor module according to claim 12, wherein, The encoder generates multiple encoded data by compressing the target pixel group according to the multiple encoding methods, and selects the encoded data with the smallest error rate from the multiple encoded data as the compressed data for the target pixel group.

16. The image sensor module according to claim 12, wherein, When the difference between the pixel value of the target pixel in the target pixel group and the reference value is equal to or greater than a threshold, the encoder compresses the target pixel group according to the first encoding method among the multiple encoding methods.

17. The image sensor module according to claim 12, wherein, At least one of the plurality of bitstreams includes a difference, a selection value, and a slope value for the target pixel.

18. An image processing system, comprising: An image sensor is configured to sense received optical signals and generate image data; An encoder configured to sequentially compress multiple groups of pixels in the image data and generate multiple bitstreams; as well as A decoder configured to decompress the plurality of bitstreams and recover the image data. The encoder determines an averaging direction in the horizontal and vertical directions, generates at least two average pixel values ​​for at least two pairs of target pixels that are adjacent to each other in the averaging direction in the target pixel group, generates balance information to be applied to the at least two average pixel values, and generates a bit stream including the at least two average pixel values, the balance information, and the averaging direction information.

19. The image processing system according to claim 18, wherein, The image data includes four pixels corresponding to the same color and arranged in a 2×2 matrix.

20. The image processing system according to claim 18, wherein, The multiple bit streams are transmitted from the encoder to the decoder via a camera serial interface based on a mobile industrial processor interface.

Images