Camera module, image processing system and image compression method

Abstract

An image compression camera module, an image processing system, and a method of compressing each of a plurality of pixel groups forming image data are provided. The method includes detecting a bad pixel among a plurality of pixels in a pixel group, generating a marker indicating position information about the bad pixel, calculating a first difference between a pixel value of a pixel other than the bad pixel among the plurality of pixels and a reference pixel value, and generating a bitstream including the marker and the first difference.

Description

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10-2020-0088450 filed on July 16, 2020, and Korean Patent Application No. 10-2021-0001057 filed on January 5, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to a camera module, an image processing system, and an image compression method, and more specifically, to a camera module that compresses image data based on the detection of bad pixels. Background Technology

[0004] Recently, with the increasing demand for high-quality and high-resolution photos and images, the pixel size in image sensors has decreased, and more pixels have been integrated. However, due to certain circumstances (such as manufacturing process issues), image sensor pixels can include one or more bad pixels of arbitrary shapes at arbitrary locations. Since bad pixels of arbitrary shapes are not used to generate photos, images, etc., they degrade the performance of the image sensor. Therefore, it is necessary to correct the image data output from bad pixels. Summary of the Invention

[0005] One or more embodiments provide a camera module and image processing system for compressing image data based on the detection of bad pixels, and an image compression method thereof.

[0006] According to an embodiment, an image compression method is provided for compressing each of a plurality of pixel groups forming image data. The method includes: detecting bad pixels among the plurality of pixels in the pixel group; generating an indication indicating location information about the bad pixels; calculating a first difference between pixel values ​​of pixels other than the bad pixels and reference pixel values; and generating a bitstream including the indication and the first difference.

[0007] According to an embodiment, a camera module is provided, the camera module comprising: an image sensor configured to generate image data including a plurality of pixels; an encoder configured to divide the plurality of pixels into a plurality of pixel groups and sequentially compress the plurality of pixel groups to generate compressed data including a plurality of bitstreams; and a memory storing reference information including pixel values ​​of pixels compressed by the encoder, wherein the encoder is further configured to: detect bad pixels in a first pixel group; generate a first bitstream corresponding to the first pixel group by compressing first pixel values ​​of a plurality of first pixels in the first pixel group based on the result of detecting bad pixels and the reference information; and update the reference information based on corrected pixel values ​​obtained by correcting the pixel values ​​of the bad pixels.

[0008] According to an embodiment, an image processing system is provided, comprising: an image sensor configured to generate image data including a plurality of pixels; an encoder configured to generate a plurality of bitstreams by sequentially compressing a plurality of pixel groups in the image data; and a decoder configured to recover the image data by decompressing the plurality of bitstreams, wherein the encoder is further configured to: detect bad pixels in each of the plurality of pixel groups; compress pixel values ​​of a second pixel group in the plurality of pixel groups according to reference information; generate the reference information based on pixel values ​​of a first pixel group in the plurality of pixel groups that were compressed prior to the second pixel group; and update the reference information based on the result of detecting bad pixels. Attached Figure Description

[0009] Embodiments of the inventive concept will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 This is a diagram illustrating an image processing system according to an embodiment;

[0011] Figure 2 This is a diagram illustrating an encoder according to an embodiment;

[0012] Figure 3 This is a conceptual diagram illustrating an image compression method according to an embodiment;

[0013] Figure 4 This is a conceptual diagram illustrating an image compression method according to an embodiment;

[0014] Figure 5A and Figure 5B This is a diagram illustrating a bitstream according to an embodiment;

[0015] Figure 6 This is a diagram illustrating a bitstream according to an embodiment;

[0016] Figure 7 This is a flowchart illustrating an image compression method according to an embodiment;

[0017] Figure 8 This is a diagram illustrating a decoder according to an embodiment;

[0018] Figure 9A and Figure 9B This is a conceptual diagram illustrating an image decompression method according to an embodiment;

[0019] Figure 10 This is a diagram illustrating an image processing system according to an embodiment;

[0020] Figure 11 This is a diagram illustrating an encoder according to an embodiment;

[0021] Figure 12A and Figure 12B This is a table showing compression information according to an embodiment;

[0022] Figure 13 This is a diagram illustrating an image processing system according to an embodiment;

[0023] Figure 14 This is a diagram illustrating an electronic device according to an embodiment;

[0024] Figure 15 This is a diagram illustrating a portion of an electronic device according to an embodiment; and

[0025] Figure 16 This is a diagram illustrating a detailed configuration of the camera module according to an embodiment. Detailed Implementation

[0026] The embodiments will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 This is a diagram illustrating an image processing system 10 according to an embodiment. (Refer to...) Figure 1 The image processing system 10 may include a camera module 100 and an image processing device 200. The camera module 100 may include an image sensor 110, an encoder 120, a memory 130, and an interface (I / F).

[0028] 140. The image processing apparatus 200 may include an interface 210, a memory 220, a decoder 230, and an image signal processor (ISP) 240. However, one or more embodiments are not limited thereto, and the camera module and image processing apparatus may include more or fewer components or parts.

[0029] The image processing system 10 can be implemented as a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. Portable electronic devices can include laptops, mobile phones, smartphones, tablet PCs, personal digital assistants (PDAs), enterprise digital assistants (EDAs), digital cameras, digital video cameras, audio devices, portable multimedia players (PMPs), personal navigation devices (PNDs), MP3 players, handheld game consoles, e-readers, wearable devices, etc. Furthermore, the image processing system 10 can be installed in electronic devices such as drones, advanced driver assistance systems (ADAS), or provided as components for vehicles, furniture, manufacturing facilities, doors, various measuring devices, etc.

[0030] Camera module 100 can capture images of an object and generate image data IDT. In some embodiments, camera module 100 may include an image sensor 110 for converting light signals from the object into electrical signals. For example, image sensor 110 may include a plurality of pixels arranged in a two-dimensional manner and may include a pixel array 111. One of a plurality of reference colors may be assigned to each of the plurality of pixels. For example, the plurality of reference colors may include red, green, and blue (RGB) or red, green, blue, and white (RGBW). Reference colors may also include colors other than those listed above. For example, the plurality of reference colors may include cyan, yellow, green, and magenta. Pixel array 111 may generate pixel signals that include information about the reference color for each of the plurality of pixels.

[0031] Pixel array 111 may include multiple row lines, multiple column lines, and multiple pixels connected to the row lines and column lines and arranged in a matrix. Multiple color filters may be arranged to correspond to multiple pixels respectively. For example, see reference... Figure 1 Each of the multiple color filters can have a 2×2 unit, which includes a repeating arrangement of red pixels R, blue pixels B, and two green pixels Gr and Gb. This pattern can be called a Bayer pattern.

[0032] As another example, a color filter can have a configuration in which groups of pixels, each corresponding to one of the multiple reference colors, are arranged in a repeating pattern. For example, a color filter can have a configuration in which a red pixel group comprising a 2×2 arrangement of red pixels R, a first green pixel group comprising a 2×2 arrangement of first green pixels Gr, a blue pixel group comprising a 2×2 arrangement of blue pixels B, and a second green pixel group comprising a 2×2 arrangement of green pixels Gb are arranged in a repeating pattern. Such a pattern can be referred to as a four-grid pattern or a four-grid unit.

[0033] As another example, a color filter may have a configuration in which a red pixel group comprising a 3×3 arrangement of red pixels R, a first green pixel group comprising a 3×3 arrangement of first green pixels Gr, a blue pixel group comprising a 3×3 arrangement of blue pixels B, and a second green pixel group comprising a 3×3 arrangement of green pixels Gb are repeated. This pattern may be referred to as a nine-grid pattern or a nine-grid unit.

[0034] One or more embodiments are not limited thereto, and the color filter may also have a configuration in which red pixel groups, blue pixel groups, first green pixel groups and second green pixel groups are repeatedly arranged in a 2n×2n or 3n×3n (where n is a positive integer) arrangement of pixels.

[0035] In a non-limiting example embodiment, the image sensor 110 can be implemented using a charge-coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) image sensor, and can be implemented as various types of optoelectronic devices. The image sensor 110 can output image data IDT in which preprocessing has been performed on the pixel signals generated by the pixel array 111.

[0036] While it has been described that the image data IDT includes information about a reference color (e.g., RGB information), one or more embodiments are not limited thereto. For example, the image sensor 110 can convert the RGB information of each pixel into YUV information including information about brightness and color difference through color space conversion. Therefore, the image data IDT can include YUV information corresponding to each pixel. Even when the image data IDT includes YUV information, the inventive concept can be applied substantially equivalently.

[0037] The camera module 100 can compress image data IDT using encoder 120 to reduce power consumption during data transmission and improve the efficiency of data storage. Specifically, encoder 120 can receive image data IDT from image sensor 110 and generate compressed data CDT by compressing the image data IDT. The compressed data CDT can be output as an encoded bitstream. Hereinafter, the encoded bitstream will simply be referred to as a bitstream.

[0038] Encoder 120 can compress image data IDT in units of pixel groups. Here, a pixel group can be set to include a predetermined number of pixels arranged in order according to the pattern of the image data IDT, or it can be set to include pixels corresponding to the same reference color and adjacent to each other. For example, when the image data IDT is a Bayer pattern, a pixel group can be set to include a predetermined number (e.g., four) of pixels arranged horizontally and vertically in order. As another example, when the image data IDT is a four-grid or nine-grid pattern, a pixel group can be set to include four or nine pixels corresponding to the same reference color (e.g., red, blue, green, etc.) and adjacent to each other. Encoder 120 can generate a bitstream by compressing a pixel group, and can generate compressed data CDT based on the bitstream of all pixel groups in the image data IDT.

[0039] The encoder 120 can compress a specific pixel group based on a first reference map RM1 generated corresponding to the pixel values ​​of pixels compressed before the corresponding pixel group. For example, the encoder 120 can determine a reference value based on the pixel value of at least one pixel of the target pixel to be compressed in a pixel group adjacent to the first reference map RM1, and compress the pixel value of the target pixel based on the reference value and the pixel value of the target pixel.

[0040] When compressing a pixel group, encoder 120 can generate (or update) a new first reference map RM1 by adding the pixel values ​​of the compressed pixel group to the existing first reference map RM1. Alternatively, encoder 120 can compress pixel groups in the next sequence using the new first reference map RM1.

[0041] The pixel value of the target pixel and the pixel values ​​of its neighboring pixels are likely to have similar values. Therefore, when compressed data CDT is generated by compressing image data IDT according to the method described above, compression efficiency can be improved and data loss can be reduced. Hereinafter, for ease of explanation, information including pixel values ​​corresponding to pixels that have been compressed before the pixel group is compressed will be referred to as the first reference map RM1, and the inventive concept is not limited thereto. The first reference map RM1 may also be referred to by other names, such as reference information.

[0042] Image data IDT may include pixel values ​​of bad pixels. Here, bad pixels may include static bad pixels that are continuously switched on or off, and dynamic bad pixels that are randomly switched on or off. Because the position of static bad pixels is fixed, the pixel values ​​of static bad pixels can be corrected by preprocessing operations of the image sensor 110, which allows for simple operation. Since the position of dynamic bad pixels is not fixed, correction requires more than simple operations. Therefore, it is more difficult to correct the pixel values ​​of dynamic bad pixels by preprocessing operations of the image sensor 110.

[0043] Because encoder 120 is configured to compress image data IDT rather than correct it, encoder 120 can compress image data IDT even if it includes bad pixels. Therefore, when encoder 120 performs compression as described above, the first reference map RM1 can include the pixel values ​​of bad pixels. Because the pixel values ​​of bad pixels differ significantly from those of neighboring pixels, compression efficiency is reduced, and data loss occurs during compression using the first reference map RM1.

[0044] To improve compression efficiency and prevent data loss, encoder 120 can detect bad pixels among the plurality of pixels included in the first reference map RM1. When the first reference map RM1 includes bad pixels, a reference value can be determined based on the pixel values ​​of pixels other than the bad pixels (e.g., normal pixels). Because the above detection operation requires a large number of operations when detecting one or more bad pixels for each of the compression operations for all pixels, it increases power consumption.

[0045] According to an embodiment, in the above method, during the compression process, the encoder 120 can correct the pixel values ​​of bad pixels (specifically, dynamic bad pixels) and include the corrected pixel values ​​in the first reference map RM1. Therefore, when the first reference map RM1 does not include the pixel values ​​of bad pixels, the bad pixel detection operation for the first reference map RM1 can be omitted.

[0046] For example, memory 130 can be configured to store image data IDT or a first reference map RM1. Memory 130 can be a volatile memory such as dynamic random access memory (DRAM) or static random access memory (SRAM) or a non-volatile memory such as phase-change random access memory (PRAM), resistive random access memory (ReRAM), or flash memory.

[0047] The encoder 120 can provide the generated compressed data CDT to the image processing device 200 via interface 140. For example, interface 140 can be implemented as a camera serial interface (CSI) based on Mobile Industrial Processor Interface (MIPI). However, the type of interface 140 is not limited to this and can be implemented according to various protocol standards.

[0048] The image processing apparatus 200 can convert compressed data CDT received from the camera module 100 to generate an image that will be displayed on a monitor. Specifically, the image processing apparatus 200 can receive compressed data CDT from the camera module 100, generate decompressed data DDT by decompressing the compressed data CDT, and generate a final image by performing image processing operations based on the decompressed data DDT.

[0049] In some embodiments, the image processing apparatus 200 may receive compressed data CDT from the camera module 100 via interface 210. Interface 210 may be implemented as MIPI, like interface 140, but is not limited thereto. The image processing apparatus 200 may store the received compressed data CDT in memory 220.

[0050] For example, memory 220 can be configured to store an operating system (OS), various programs, and various data (e.g., compressed data CDT). Memory 220 can be a volatile memory such as DRAM or SRAM, or a non-volatile memory such as PRAM, ReRAM, or flash memory.

[0051] Decoder 230 can read compressed data CDT from memory 220 and decompress compressed data CDT to generate decompressed data DDT. Decoder 230 can output decompressed data DDT to ISP 240.

[0052] In some embodiments, decoder 230 may decompress compressed data CDT on a pixel-by-pixel basis. Here, decoder 230 may decompress pixel groups by utilizing a second reference map RM2 generated based on pixel values ​​corresponding to pixels that were decompressed before the corresponding pixel group.

[0053] According to an embodiment, like the first reference map RM1 described above, the second reference map RM2 may include corrected pixel values ​​of bad pixels (specifically, dynamically bad pixels). That is, when bad pixels are present in the decompressed pixel group, the decoder 230 can correct the pixel values ​​of the bad pixels and generate a new second reference map RM2 that includes the corrected pixel values. In other words, the decoder 230 can update the second reference map RM2 with the corrected pixel values ​​of the bad pixels. Furthermore, the decoder 230 can decompress the pixel group in the next order based on the new second reference map RM2. The second reference map RM2 may be stored in the memory 220. Hereinafter, for ease of explanation, information including pixel values ​​corresponding to pixels that have been decompressed before the pixel group to be decompressed is referred to as the second reference map RM2, but one or more embodiments are not limited thereto.

[0054] Each of encoder 120 and decoder 230 can be implemented as software or hardware, or a combination of software and hardware such as firmware. When encoder 120 and decoder 230 are implemented as software, each of the above-described functions can be implemented as programmable source code and can be loaded into the storage medium included in each of camera module 100 and image processing device 200. The processor (e.g., microprocessor) included in each of camera module 100 and image processing device 200 can execute the software, thus implementing the functions of encoder 120 and decoder 230. When encoder 120 and decoder 230 are implemented as hardware, encoder 120 and decoder 230 can include logic circuitry and registers, and the above-described functions can be performed based on register settings.

[0055] The ISP 240 can perform various image processing operations on the received decompressed data DDT. In a non-limiting example, the ISP 240 can perform at least one of the following image processing operations on the decompressed data DDT: bad pixel correction, offset correction, lens distortion correction, color gain correction, shading correction, gamma correction, noise reduction, and sharpening. In some embodiments, some of the above image processing operations can be omitted depending on the performance of the camera module 100. For example, when the camera module 100 includes a high-quality image sensor 110, bad pixel correction (specifically, static bad pixel correction) or offset correction in the image processing operations can be omitted.

[0056] exist Figure 1 In the illustration, the image processing system 10 is shown to include a camera module 100 and an image processing device 200, but the inventive concept is not limited thereto. For example, the image processing system 10 may include only some of the camera modules 100 and the image processing device 200, or it may include multiple camera modules 100. Furthermore, although... Figure 1 The decoder 230 and ISP 240 are shown as separate components, but the inventive concept is not limited thereto. For example, the ISP 240 can be combined with the decoder 230 as a single component.

[0057] Furthermore, although memory 130 and memory 220 are shown as being included in camera module 100 and image processing device 200 respectively, in Figure 1 However, the present invention is not limited thereto. For example, each of the memory 130 and the memory 220 may be located outside the camera module 100 or the image processing device 200.

[0058] The image processing system 10 of this invention can significantly reduce the number of bad pixel detection operations by generating a reference map for compression and decompression based on the detection of bad pixels and compressing or decompressing image data based on the generated reference map. Therefore, the power consumption of the image processing system can be reduced.

[0059] Figure 2 This is a diagram illustrating an encoder 120 according to an embodiment. (Refer to...) Figure 1 and Figure 2 The encoder 120 may include a bad pixel (BP) detector 121, a compressor 123, and a first reference map (RM1) generator 125.

[0060] The bad pixel detector 121 can receive image data IDT from the image sensor 110. The bad pixel detector 121 can perform bad pixel detection on the pixel group to be compressed based on the received image data IDT. (Refer to the following...) Figure 3 and Figure 4 This will be described in detail. The bad pixel detector 121 can send bad pixel information BP, including the detection results, to the compressor 123 and / or the first reference map generator 125.

[0061] Compressor 123 can generate compressed data CDT by compressing image data IDT using bad pixel information BP and a first reference map RM1. Specifically, compressor 123 can check bad pixels in the pixel group to be compressed in the image data IDT based on the bad pixel information BP received from bad pixel detector 121. In addition, compressor 123 can compress the pixel values ​​of pixels other than bad pixels in the pixel group (e.g., normal pixels) by utilizing the first reference map RM1.

[0062] For example, compressor 123 can determine a reference value based on the pixel value of at least one pixel in the pixel group that is adjacent to the target pixel or the pixel value of a normal pixel in the first reference map RM1. Alternatively, compressor 123 can generate a bitstream BS corresponding to the pixel group by compressing the pixel group based on the reference value and the pixel values ​​of normal pixels (not bad pixels). The following will refer to... Figure 3 and Figure 4 Please describe this in detail.

[0063] Compressor 123 can generate a bitstream BS corresponding to each pixel group by repeating the above-described operation for each pixel group in the image data IDT. Additionally, compressor 123 can generate compressed data CDT based on the bitstream BS.

[0064] Compressor 123 can send the generated compressed data CDT to interface 140. Interface (I / F) 140 can send the compressed data CDT to image processing device 200. In addition, compressor 123 can send the bitstream BS to the first reference map generator 125 based on the generated bitstreams BS.

[0065] The first reference map generator 125 can generate a new first reference map RM1 (NEW) based on the received bad pixel information BP and bitstream BS. Specifically, the first reference map generator 125 can recover the pixel values ​​of the pixel group by first decoding the bitstream BS. Furthermore, the first reference map generator 125 can check for bad pixels in the pixel group based on the bad pixel information BP. Additionally, the first reference map generator 125 can correct the pixel values ​​of the bad pixels in the recovered pixels to values ​​similar to those of neighboring pixels. Furthermore, the first reference map generator 125 can generate a new first reference map RM1 (NEW) by adding the recovered pixel values ​​of the pixels in the pixel group (excluding bad pixels) and the corrected pixel values ​​of the bad pixels to the existing first reference map RM1. The following will refer to... Figure 3 and Figure 4 Please describe this in detail.

[0066] As described above, when compressing a pixel group, the first reference map generator 125 can generate a new first reference map RM1(NEW) by adding pixel values ​​(e.g., recovered pixel values ​​corresponding to the compressed pixel group and / or corrected pixel values ​​of bad pixels) to the existing first reference map RM1 for use in compressing the pixel group in the next order. The first reference map generator 125 can store the generated new first reference map RM1(NEW) in memory 130. The compressor 123 can read the new first reference map RM1(NEW) stored in memory 130 and perform compression on the pixel group in the next order based on the read new first reference map RM1(NEW).

[0067] Each of the bad pixel detector 121, compressor 123, and first reference map generator 125 can be implemented as software or hardware, or a combination of software and hardware such as firmware. When the bad pixel detector 121, compressor 123, and first reference map generator 125 are implemented as software, each of the above functions can be implemented as programmed source code and can be loaded into a storage medium included in the camera module 100. A processor (e.g., a microprocessor) included in the camera module 100 can execute software, thus implementing the functions of the bad pixel detector 121, compressor 123, and first reference map generator 125. When the bad pixel detector 121, compressor 123, and first reference map generator 125 are implemented as hardware, the bad pixel detector 121, compressor 123, and first reference map generator 125 can include logic circuitry and registers, and can perform the above functions based on register settings.

[0068] Figure 3 This is a conceptual diagram illustrating an image compression method according to an embodiment. Specifically, Figure 3This diagram illustrates a method for compressing image data IDT according to a Bayer pattern using encoder 120. In the following text, it is assumed that a pixel group comprises four pixels arranged in sequence. Furthermore, an embodiment in which encoder 120 compresses a pixel group PG comprising pixel R7 (a bad pixel) and pixels Gr7, R8, and Gr8 (normal pixels). For ease of explanation, the pixel in pixel group PG that is the target to be processed is referred to as the target pixel.

[0069] The bad pixel detector 121 can detect one or more bad pixels among the plurality of pixels in the pixel group PG. Specifically, the bad pixel detector 121 can detect whether a target pixel is a bad pixel based on the pixel value of at least one first pixel in the image data IDT that is adjacent to the target pixel. In some embodiments, the bad pixel detector 121 can detect whether a target pixel is a bad pixel based on the pixel value of at least one first pixel that corresponds to the same reference color as the target pixel and is adjacent to the target pixel. In this case, the criteria for determining pixels adjacent to the target pixel can be set differently. For example, pixels that are in direct contact with the target pixel or pixels located within a certain distance from the target pixel can be determined as pixels adjacent to the target pixel.

[0070] Specifically, when there are multiple first pixels, the bad pixel detector 121 can calculate the average pixel value of the multiple first pixels and calculate the difference between the average pixel value and the pixel value of the target pixel. When the calculated difference exceeds a threshold, the bad pixel detector 121 can detect the target pixel as a bad pixel. Here, the threshold can be preset by the user or the manufacturer, and can have different values ​​for each image data IDT. The bad pixel detector 121 can generate bad pixel information by repeatedly performing the above detection operation for each pixel in the pixel group PG. In addition, the bad pixel information may include the location information of the bad pixels.

[0071] For example, refer to Figure 3 The bad pixel detector 121 can calculate the average pixel value of a plurality of first pixels (e.g., R3, R6, R8, and R11) in the image data IDT that correspond to the same red color as pixel R7 and are adjacent to pixel R7, where pixel R7 is the target pixel. Additionally, the bad pixel detector 121 can calculate the difference between the calculated average pixel value and the pixel value of pixel R7. When the calculated difference exceeds a threshold, the bad pixel detector 121 can identify pixel R7 as a bad pixel.

[0072] The bad pixel detection operation of the bad pixel detector 121 is not limited to the example described above, and various methods can be applied. For example, regardless of the reference color, the bad pixel detector 121 can detect whether a target pixel is a bad pixel based on the pixel value of at least one first pixel adjacent to the target pixel.

[0073] For example, refer to Figure 3 The bad pixel detector 121 can detect whether pixel R7 is a bad pixel based on the pixel values ​​of a plurality of first pixels (e.g., R6, Gr6, Gr7, and R8) that are horizontally adjacent to pixel R7 in the image data IDT. As another example, the bad pixel detector 121 can detect whether pixel R7 is a bad pixel based on the pixel values ​​of a plurality of first pixels (e.g., R3, Gb3, Gb7, and R11) that are vertically adjacent to pixel R7. As yet another example, the bad pixel detector 121 can detect whether pixel R7 is a bad pixel based on the pixel values ​​of other pixels (e.g., Gr7, R8, and Gr8) included in the pixel group PG that includes pixel R7.

[0074] Compressor 123 can identify bad pixels in a pixel group based on bad pixel information received from bad pixel detector 121, and can compress the pixel values ​​of pixels other than bad pixels.

[0075] Specifically, compressor 123 can identify at least one second pixel (which is not a bad pixel) in the first reference map RM1 that corresponds to the same reference color as the target pixel and is adjacent to the target pixel (or pixel group PG). Furthermore, compressor 123 can determine a reference value based on the pixel value of the identified at least one second pixel. Additionally, compressor 123 can compress the pixel value of a bad pixel based on the reference value and the pixel value of the bad pixel. In this case, the criteria for determining the second pixel adjacent to the target pixel can be set differently.

[0076] In some embodiments, when there are multiple second pixels, the compressor 123 can determine the pixel value of the pixel located at a specific position (e.g., to the left, above, or diagonally opposite to the target pixel) among the multiple second pixels as a reference value. Alternatively, the compressor 123 can calculate the average pixel value of the identified multiple second pixels and determine the calculated average pixel value as the reference value. Additionally, the compressor 123 can calculate the difference RES between the reference value and the pixel value of the target pixel.

[0077] Compressor 123 can calculate multiple differences RES by repeating the above operation on all pixels in pixel group PG, and can generate a bit stream BS including multiple differences RES. As described above, the compression method performed based on the differences RES can be called Differential Pulse Code Modulation (DPCM).

[0078] For example, refer to Figure 3The compressor 123 can identify in the first reference map RM1 a plurality of second pixels (e.g., Gb3, Gb4, Gb7, Gb8, Gr3, Gr6, and Gr8) that correspond to the same reference color as pixel Gr7 and are adjacent to pixel Gr7, where pixel Gr7 is the target pixel. Additionally, the compressor 123 can determine the pixel value (e.g., Gr6) located to the left of pixel Gr7 among the plurality of second pixels (e.g., Gb3, Gb4, Gb7, Gb8, Gr3, Gr6, and Gr8) as a reference value. Furthermore, the compressor 123 can calculate the difference RES between the reference value and the pixel value of pixel Gr7, and can generate a bitstream BS including the calculated difference RES.

[0079] When compressing pixel group PG, the first reference map generator 125 can generate a new first reference map RM1 (NEW) by adding the pixel values ​​corresponding to the compressed pixel group PG to the existing first reference map RM1. For example, the first reference map generator 125 can recover the pixel values ​​of pixel group PG by decoding the bitstream BS of pixel group PG received from compressor 123, and can generate a new first reference map RM1 (NEW) by adding the recovered pixel values ​​to the existing first reference map RM1. The method of recovering pixel values ​​based on the bitstream BS performed by the first reference map generator 125 can be substantially the same as the decompression method of decoder 230 described below. As another example, the first reference map generator 125 can also add the pixel values ​​corresponding to pixel group PG in image data IDT to the existing first reference map RM1.

[0080] According to the present invention, when a pixel group PG includes bad pixels, the first reference map generator 125 can correct the pixel value of the bad pixels, and then add the corrected pixel value of the bad pixels to the existing first reference map RM1, thereby generating a new first reference map RM1 (NEW).

[0081] In some embodiments, the first reference map generator 125 can correct the pixel value of a bad pixel based on the pixel values ​​of a plurality of third pixels adjacent to the bad pixel using various methods. Specifically, the first reference map generator 125 can identify the plurality of third pixels that correspond to the same reference color as the bad pixel and are adjacent to the bad pixel. Furthermore, the first reference map generator 125 can correct the pixel value of the bad pixel based on the pixel values ​​of the plurality of third pixels.

[0082] For example, refer to Figure 3The first reference mapping generator 125 can correct the pixel value of pixel R7 to the pixel value of a third pixel (e.g., R3) that is the same red as pixel R7 and is located above pixel R7, where pixel R7 is a bad pixel. As another example, the first reference mapping generator 125 can correct the pixel value of pixel R7 to the pixel value of a third pixel (e.g., R6) located to the left of pixel R7. As another example, the first reference mapping generator 125 can correct the pixel value of pixel R7 to the pixel value of a third pixel (e.g., R8) located to the right of pixel R7. As another example, the first reference mapping generator 125 can correct the pixel value of pixel R7 to the average pixel value of a plurality of third pixels (e.g., R2 and R4) located diagonally opposite pixel R7.

[0083] Additionally, the first reference map generator 125 can add the corrected pixel value R7b of pixel R7 to the existing first reference map RM1, wherein the pixel value of pixel R7 is corrected to the corrected pixel value R7b based on any of the methods described above. Furthermore, the first reference map generator 125 can also add the pixel values ​​of pixels Gr7, R8, and Gr8, which are normal pixels in pixel group PG, to the existing first reference map RM1 to generate a new first reference map RM1 (NEW).

[0084] Figure 4 This is a conceptual diagram illustrating an image compression method according to an embodiment. Specifically, Figure 4 This diagram illustrates a method for compressing image data IDT according to a four-grid pattern performed by encoder 120. In the following text, it is assumed that a pixel group is defined as comprising pixels corresponding to the same reference color and being adjacent to each other. For example, a pixel group PG comprising pixels Gb5, Gb6, Gb7, and Gb8 has the same reference color, green. In this pixel group PG, pixel Gb5 may be a bad pixel, and pixels Gb6, Gb7, and Gb8 may be normal pixels. References to the above will be omitted. Figure 3 The provided description is duplicated. Figure 4 The description.

[0085] The bad pixel detector 121 can detect whether a target pixel is a bad pixel based on the pixel value of at least one first pixel that is adjacent to the target pixel in the image data IDT. In some embodiments, the bad pixel detector 121 can detect whether a target pixel is a bad pixel based on the pixel value of at least one first pixel that corresponds to the same reference color as the target pixel and is adjacent to the target pixel.

[0086] For example, refer to Figure 4The bad pixel detector 121 can calculate the average pixel value of a plurality of first pixels (e.g., Gb6, Gb7, and Gb8) in the image data IDT that correspond to the same green color as pixel Gr5 and are adjacent to pixel Gr5, where pixel Gr5 is the target pixel. Additionally, the bad pixel detector 121 can calculate the difference between the calculated average pixel value and the pixel value of the target pixel. When the calculated difference exceeds a threshold, the bad pixel detector 121 can determine that the target pixel is a bad pixel. The bad pixel detection operation of the bad pixel detector 121 is not limited to the example described above, and various methods can be applied.

[0087] Compressor 123 can identify bad pixels in a pixel group based on bad pixel information received from bad pixel detector 121, and can compress the pixel values ​​of pixels other than bad pixels.

[0088] Specifically, compressor 123 can identify at least one second pixel (which is not a bad pixel) in the first reference map RM1 that corresponds to the same reference color as the target pixel and is adjacent to the target pixel (or pixel group PG). Furthermore, compressor 123 can determine a reference value based on the pixel value of the identified at least one second pixel. Additionally, compressor 123 can compress the pixel value of the bad pixel based on the reference value and the pixel value of the bad pixel.

[0089] In some embodiments, when there are multiple second pixels, the compressor 123 may determine the pixel value of a pixel at a specific location (e.g., to the left, top, or diagonally opposite to the target pixel) among the multiple second pixels as a reference value. Alternatively, the compressor 123 may calculate the average pixel value of pixel groups compressed before the currently compressed pixel group PG and determine the calculated average pixel value as the reference value. For example, assuming that the first pixel group PG1 is compressed before the second pixel group PG2, the compressor 123 may use the average pixel value calculated for the first pixel group PG1 as the reference value when compressing the second pixel group PG2. In addition, the compressor 123 may calculate the difference RES between the reference value and the pixel value of the target pixel and generate a bitstream BS including the calculated difference RES.

[0090] Compressor 123 can calculate multiple differences RES by repeating the above operation for all pixels in pixel group PG, and can generate a bit stream BS that includes multiple differences RES.

[0091] For example, refer to Figure 4The compressor 123 can identify, in the first reference map RM1, a plurality of second pixels (e.g., Gb1, Gb2, Gb3, and Gb4) corresponding to the same reference color as pixel Gb6 and adjacent to pixel Gb6, wherein pixel Gb6 is not a bad pixel. Furthermore, the compressor 123 can determine the average pixel value of the plurality of second pixels (e.g., Gb2, Gb4, and Gr7) as a reference value. Additionally, the compressor 123 can calculate the difference RES between the reference value and the pixel value of pixel Gb6, and can generate a bitstream BS including the calculated difference RES.

[0092] When compressing pixel group PG, the first reference map generator 125 can generate a new first reference map RM1 (NEW) by adding the pixel values ​​corresponding to the compressed pixel group PG to the existing first reference map RM1. For example, the first reference map generator 125 can recover the pixel values ​​of pixel group PG by decoding the bitstream BS received from the compressor 123, and can generate a new first reference map RM1 (NEW) by adding the recovered pixel values ​​to the existing first reference map RM1.

[0093] That is, the first reference map generator 125 can correct the pixel values ​​of bad pixels in the pixel group PG, and then add the corrected pixel values ​​of the bad pixels to the existing first reference map RM1.

[0094] In some embodiments, the first reference map generator 125 can correct the pixel value of a bad pixel based on the pixel values ​​of a plurality of third pixels adjacent to the bad pixel using various methods. Specifically, the first reference map generator 125 can identify the plurality of third pixels that correspond to the same reference color as the bad pixel and are adjacent to the bad pixel. Furthermore, the first reference map generator 125 can correct the pixel value of the bad pixel based on the pixel values ​​of the plurality of third pixels.

[0095] For example, refer to Figure 4The first reference mapping generator 125 can correct the pixel value of pixel Gb5 to the pixel value of a third pixel (e.g., Gb2) that corresponds to the same green color as pixel Gb5 and is located to the left of pixel Gb5, where pixel Gb5 is a bad pixel. As another example, the first reference mapping generator 125 can correct the pixel value of pixel Gb5 to the pixel value of a third pixel located above pixel Gb5. As another example, the first reference mapping generator 125 can correct the pixel value of pixel Gb5 to the pixel value of a third pixel (e.g., Gb6) located to the right of pixel Gb5. As another example, the first reference mapping generator 125 can correct the pixel value of pixel Gb5 to the average pixel value of pixels located diagonally opposite pixel Gb5 (e.g., Gr4 and Gr7). As another example, the first reference mapping generator 125 can correct the pixel value of pixel Gb5 to the average pixel value of other pixels (e.g., Gb6, Gb7, and Gb8) excluding bad pixels in pixel group PG.

[0096] Additionally, the first reference map generator 125 can add the corrected pixel value Gb5b of the target pixel Gb5 to the existing first reference map RM1, wherein the pixel value of pixel Gb5 is corrected to the corrected pixel value Gb5b based on any of the methods described above. Furthermore, the first reference map generator 125 can also add the pixel values ​​of pixels Gb6, Gb7, and Gb8, which are normal pixels among the plurality of pixels in pixel group PG, to the existing first reference map RM1 to generate a new first reference map RM1 (NEW).

[0097] Although reference Figure 4 As described above, it is assumed that the image data IDT has a four-grid pattern, but the inventive concept is not limited to this. For example, the above method can also be applied when the image data IDT has a nine-grid pattern.

[0098] Figure 5A and Figure 5B This is a diagram illustrating the encoded bitstream according to an embodiment. Specifically, Figure 5A and Figure 5B These are diagrams illustrating bitstreams BS generated by compressing the pixel values ​​of the pixels included in a pixel group PG. For ease of understanding, an embodiment will be described where one pixel group PG includes four pixels (e.g., first pixel P1 to fourth pixel P4) and each pixel includes 10 bits.

[0099] Reference Figure 5A and Figure 5BThe bitstream BS may include 20 bits, and may include 4 bits for the header, 4 bits for the bad pixel flag (BP flag), and 12 bits for the difference (RES). The header may include information related to the compression of the image data IDT. For example, the header may include information such as the compression method, compression mode, compression ratio, and missing information. The decoder 230 may examine the header and compress the bitstream BS according to the method corresponding to the header.

[0100] The bad pixel label (BP) flag can include information indicating bad pixels in pixel group PG. Each of the 4 bits forming the bad pixel label (BP) flag can correspond to each of the first pixel P1 through the fourth pixel P4 in pixel group PG. For example, in the bad pixel label (BP) flag, the first bit can correspond to the first pixel P1, the second bit can correspond to the second pixel P2, the third bit can correspond to the third pixel P3, and the fourth bit can correspond to the fourth pixel P4. Furthermore, when each of the 4 bits of the bad pixel label (BP) flag has a specific value (e.g., 1 or 0), it can mean that the pixel corresponding to that bit is a bad pixel.

[0101] The difference value RES can include the difference between normal pixels and a reference value in pixel group PG. Here, the difference value of normal pixels is indicated based on the above reference. Figure 3 and Figure 4 The first reference map RM1 determines the difference between a reference value and the pixel value of a target pixel that can be considered a normal pixel or a bad pixel. When the difference between the reference value and the pixel value of the target pixel is not included in the range of bit representations that can be allocated in the bit stream BS, the least significant bit (LSB) of the difference can be removed. When the difference is included in the range of bit representations that can be allocated in the bit stream BS, the difference can be included in the bit stream BS.

[0102] According to the present invention, as the number of bad pixels in pixel group PG increases, the number of bits allocated to the difference between normal pixels can also increase. For example, refer to... Figure 5A Because there is one bad pixel in pixel group PG (e.g., the fourth pixel P4), the differences RES1, RES2, and RES3 between the other three normal pixels P1, P2, and P3 can be formed using 12 bits. For example, 4 bits can be allocated to each of the differences RES1, RES2, and RES3. (See reference...) Figure 5BBecause there are two bad pixels in pixel group PG (they are the third pixel P3 and the fourth pixel P4), the differences RES1 and RES2 between the other two normal pixels P1 and P2 can be formed by 12 bits. For example, 6 bits can be allocated to each of the differences RES1 and RES2. Because the range of the differences increases with the number of bits allocated, data loss can be reduced. The number of bits allocated to each of the normal pixels can be the same or different.

[0103] Decoder 230 can recover the pixel values ​​of pixel group PG based on bitstream BS. First, decoder 230 can recover the pixel values ​​of normal pixels based on difference RES. Additionally, decoder 230 can recover the pixel values ​​of bad pixels based on the recovered pixel values ​​of normal pixels. The following will explain... Figure 9A and Figure 9B The description of the recovery of decoder 230 is detailed in the text.

[0104] The number of bits for each of the header HEADER, bad pixel flag (BP FLAG), and difference value (RES) is not limited to the example above, and the number of bits can be set differently depending on the embodiment.

[0105] Figure 6 This is a diagram illustrating the encoded bitstream according to an embodiment. Specifically, Figure 6 It is shown Figure 5A and Figure 5B The diagram shows a modified embodiment. References above will be omitted. Figure 5A and Figure 5B The descriptions made were repetitive. Figure 6 The description.

[0106] Reference Figure 6 The bit stream BS may include 20 bits, and may include 4 bits indicating the header, 2 bits indicating the bad pixel flag (BP FLAG), and 14 bits indicating the difference (RES).

[0107] When a pixel group PG includes a bad pixel, it can be applied Figure 6 The bitstream BS is used to form a bad pixel flag BPFLAG, which has 2 bits representing the bad pixel among four pixels in pixel group PG (e.g., first pixel P1 to fourth pixel P4). For example, when the first pixel P1 is a bad pixel, the bad pixel flag BPFLAG can have a value of 00. When the second pixel P2 is a bad pixel, the bad pixel flag BPFLAG can have a value of 01. When the third pixel is a bad pixel, the bad pixel flag BPFLAG can have a value of 10. When the fourth pixel is a bad pixel, the bad pixel flag BPFLAG can have a value of 11.

[0108] According to the embodiment, because only 2 bits are allocated to the bad pixel flag (BP FLAG), therefore... Figure 5A Compared to the previous implementation, more bits can be allocated to the difference RES. Therefore, data loss in the difference RES can be reduced. Although Figure 6 The illustration shows 5 bits allocated to difference RES1, 5 bits allocated to difference RES2, and 4 bits allocated to difference RES3, but the inventive concept is not limited thereto. The number of bits can be set differently depending on the embodiment.

[0109] according to Figures 5A to 6 Because the 40 bits forming the pixel group PG are compressed into a 20-bit bitstream BS, the image data IDT can be compressed at a compression rate of 50%. However, the inventive concept is not limited to this. The number of bits in the bitstream BS can be set to less than or greater than 20 bits, thus increasing or decreasing the compression rate.

[0110] Figure 7 This is a flowchart illustrating an image compression method according to an embodiment. Specifically, Figure 7 It is shown Figure 1 A flowchart of an image compression method in the image processing system 10. This method can be executed via encoder 120. Figure 7 At least one of the operations.

[0111] Reference Figure 1 and Figure 7 In operation S110, encoder 120 can detect bad pixels in each of the multiple pixel groups forming the image data IDT. Encoder 120 can detect bad pixels based on at least one of the pixel values ​​of the pixel group in the image data IDT and the pixel values ​​of pixels adjacent to the pixel group. For example, to detect whether a target pixel is a bad pixel in the pixel group, encoder 120 can calculate the average pixel value of the pixels adjacent to the target pixel. Additionally, encoder 120 can detect whether a target pixel is a bad pixel based on the difference between the calculated average pixel value and the pixel value of the target pixel. For example, when the difference between the average pixel value and the pixel value of the target pixel exceeds a threshold, encoder 120 can detect the target pixel as a bad pixel.

[0112] Additionally, in operation S120, encoder 120 can generate a marker indicating the location information of a bad pixel. This marker can have a value corresponding to the location of at least one bad pixel included in the pixel group. Furthermore, in operation S130, encoder 120 can calculate the difference between the pixel values ​​of pixels in the pixel group other than the bad pixel and a reference pixel value. Encoder 120 can determine the reference pixel value based on reference information (e.g., a first reference map RM1) including pixel values ​​corresponding to pixels compressed before the pixel group. For example, encoder 120 can determine the reference pixel value based on the pixel values ​​of at least one pixel adjacent to the pixel group in the reference information. (Reference...) Figure 3 and Figure 4 A detailed description of the method for determining the reference pixel value is provided; therefore, its redundant description is omitted.

[0113] In operation S140, encoder 120 can generate a bitstream including identifiers and differences. Encoder 120 can generate compression information indicating a compression method applied to a group of pixels. Additionally, encoder 120 can include the compression information generated in the bitstream. For example, encoder 120 can include compression information from the header of the bitstream.

[0114] Furthermore, after generating the bitstream, encoder 120 can update the reference information based on the pixel values ​​corresponding to pixels in the pixel group. Specifically, encoder 120 can correct the pixel values ​​of bad pixels in the pixel group. For example, encoder 120 can correct the pixel values ​​of bad pixels based on the pixel values ​​of pixels adjacent to the bad pixel. (Already referred to...) Figure 3 and Figure 4 A detailed description of the method for correcting the pixel values ​​of bad pixels is provided; therefore, its redundant description is omitted. Additionally, the encoder 120 can update the reference information by adding the pixel values ​​of pixels other than bad pixels in the pixel group and the corrected pixel values ​​of the bad pixels to the reference information.

[0115] Figure 8 This is a diagram illustrating the decoder 230 according to an embodiment. Figure 8 It is shown Figure 1 The diagram of decoder 230. (Refer to...) Figure 1 and Figure 8 The decoder 230 may include a decompressor 231 and a second reference map generator 233.

[0116] Decompressor 231 can receive compressed data CDT and a second reference map RM2 from memory 220, and decompress the compressed data CDT using the second reference map RM2 to generate decompressed data DDT. Decompressor 231 can sequentially decompress multiple bitstreams BS included in the compressed data CDT using the second reference map RM2 generated based on the pixel values ​​of previously decompressed pixels. (Refer to the following...) Figure 9A and Figure 9B Provide a detailed description of this.

[0117] When decompressing a bitstream BS, the decompressor 231 can provide pixel information DP, which includes the recovered pixel values ​​of the decompressed pixels, to the second reference map generator 233. Additionally, the decompressor 231 can identify bad pixels using the bad pixel flag BP of the bitstream BS and provide bad pixel information BP about the identified bad pixels to the second reference map generator 233.

[0118] The second reference map generator 233 can generate a new second reference map RM2(NEW) based on the recovered pixel information DP and bad pixel information BP received from the decompressor 231. That is, the second reference map generator 233 can identify bad pixels in the recovered pixel group based on the bad pixel information BP. In addition, the second reference map generator 233 can correct the pixel values ​​of bad pixels in the recovered pixel values ​​to values ​​similar to the pixel values ​​of neighboring pixels. Therefore, the second reference map generator 233 can generate a new second reference map RM2(NEW) by adding or replacing the recovered pixel values ​​of normal pixels in the pixel group with the corrected pixel values ​​of bad pixels to the second reference map RM2. The following will refer to... Figure 9A and Figure 9B Provide a detailed description of this.

[0119] The second reference map generator 233 can store the generated new second reference map RM2(NEW) in memory 220. The decompressor 231 can read the new second reference map RM2(NEW) stored in memory 220 and perform decompression on the pixel group in the next order based on the read new second reference map RM2(NEW).

[0120] Each of the decompressor 231 and the second reference map generator 233 can be implemented as software or hardware, or a combination of software and hardware such as firmware. When the decompressor 231 and the second reference map generator 233 are implemented as software, each of the above-described functions can be implemented as programmed source code and can be loaded into a storage medium included in the image processing apparatus 200. The processor (e.g., a microprocessor) included in the image processing apparatus 200 can execute software, and therefore can implement the functions of the decompressor 231 and the second reference map generator 233. When the decompressor 231 and the second reference map generator 233 are implemented as hardware, the decompressor 231 and the second reference map generator 233 can include logic circuitry and registers, and can perform the above-described functions based on register settings.

[0121] Figure 9A and Figure 9B This is a conceptual diagram illustrating an image decompression method according to an embodiment. Specifically, Figure 9A and Figure 9B These are diagrams illustrating methods for decompressing image data IDT according to the Bayer pattern performed by decoder 230. In the following, it is assumed that a pixel group comprises four pixels arranged in sequence. Furthermore, an embodiment in which decoder 230 decompresses the sixth bitstream BS6 corresponding to pixel group PG comprising pixels R7, Gr7, R8, and Gr8, and where pixel R7 is a bad pixel, will be described.

[0122] Reference Figure 9A The decompressor 231 can decompress the sixth bitstream BS6 by utilizing the second reference map RM2. Here, the second reference map RM2 may include the pixel values ​​of the first bitstream BS1 to the fifth bitstream BS5 that were decompressed before the sixth bitstream BS6.

[0123] The decompressor 231 can identify bad pixels based on the bad pixel flag (BP FLAG) of the sixth bitstream BS6. Additionally, the decompressor 231 can perform decompression starting from normal pixels (which are not bad pixels).

[0124] In some embodiments, the decompressor 231 may select at least one pixel or a target pixel (which is a normal pixel) of the pixel group PG from the second reference map RM2. Additionally, the decompressor 231 may determine a reference value based on the pixel value of the selected pixel. Here, the method of determining the reference value may correspond to the method in which the compressor 123 determines the reference value during the operation of generating the sixth bitstream BS6. For example, when the compressor 123 determines the pixel value of a pixel located at a specific position (e.g., to the left of the target pixel) among the pixels adjacent to the target pixel in the first reference map RM1 as the reference value, the decompressor 231 may determine the reference value by referring to the pixel value of a pixel located at a specific position (e.g., to the left of the target pixel) among the pixels adjacent to the target pixel in the second reference map RM2. According to embodiments, information regarding the method in which the compressor 123 determines the reference value may be implemented as being included in the header HEADER of the bitstream BS.

[0125] Furthermore, the decompressor 231 can recover the pixel value of the target pixel based on a determined reference value and the difference RES between the target pixels included in the sixth bitstream BS6. That is, the decompressor 231 can recover the pixel value of the target pixel by adding the difference RES between the target pixels to the determined reference value. For example, when the target pixel is pixel Gr7, the decompressor 231 can recover the pixel value of pixel Gr7 by adding the difference RES between pixel Gr7 and the reference value.

[0126] When normal pixels are recovered, decompressor 231 can recover the pixel values ​​of bad pixels. The difference RES of bad pixels may not be included in the sixth bitstream BS6. Therefore, decoder 230 can calculate the average pixel value of the recovered normal pixel values ​​and add the calculated average pixel value to a threshold to recover the pixel values ​​of bad pixels. The threshold can be referenced above. Figure 3 The bad pixel detection operation described uses the same threshold, but the inventive concept is not limited to this, and various values ​​can be set as the threshold.

[0127] For example, when the bad pixel is pixel R7, the decoder 230 can calculate the average pixel value of pixels Gr7, R8, and Gr8 as recovered normal pixels, and add the calculated average pixel value to a threshold to recover the pixel value of the bad pixel R7. The method by which the decoder 230 recovers bad pixels is not limited to the example above, and bad pixels can be recovered using various methods.

[0128] When decompressing pixel group PG, the second reference map generator 233 can generate a new second reference map RM2 (NEW) by adding the pixel values ​​of the decompressed pixel group PG to the existing second reference map RM2.

[0129] Specifically, when the pixel group does not include bad pixels, the second reference map generator 233 can generate a new second reference map RM2 (NEW) by adding the decompressed pixel values ​​to the existing second reference map RM2.

[0130] Conversely, when a group of pixels includes bad pixels, the second reference map generator 233 can correct the pixel values ​​of the bad pixels in the recovered pixel values ​​based on the pixel values ​​of pixels adjacent to the bad pixels. This can be compared with the above reference... Figure 3 The method for correcting the pixel values ​​of bad pixels described is similar to the method for correcting the pixel values ​​of bad pixels.

[0131] For example, refer to Figure 9B The second reference map generator 233 can correct the pixel value of pixel R7 to the pixel value of a pixel that is the same red as pixel R7 and is located above pixel R7 (e.g., R3), where pixel R7 is the target pixel with bad pixels. As another example, the second reference map generator 233 can correct the pixel value of pixel R7 to the pixel value of a pixel located to the left of pixel R7 (e.g., R6). As another example, the second reference map generator 233 can correct the pixel value of pixel R7 to the pixel value of a pixel located to the right of pixel R7 (e.g., R8). As another example, the second reference map generator 233 can correct the pixel value of pixel R7 to the average pixel value of pixels located diagonally opposite pixel R7 (e.g., R2 and R4). The corrected value of the target pixel R7 can be the pixel value R7b shown in the new second reference map RM2 (NEW).

[0132] Additionally, the second reference map generator 233 can generate a new second reference map RM2 (NEW) by adding the pixel values ​​of the recovered normal pixels and the corrected pixel values ​​of the bad pixels to the existing second reference map RM2.

[0133] As described above, when decompressing a pixel group PG, the second reference map generator 233 can generate a new second reference map RM2(NEW) by adding the pixel values ​​corresponding to the pixel group PG (i.e., the pixel values ​​of the recovered normal pixels and the corrected pixel values ​​of the bad pixels of the pixel group PG) to the existing second reference map RM2, so as to decompress the pixel group in the next order.

[0134] Figure 10 This is a diagram illustrating an image processing system 10a according to an embodiment. Specifically, Figure 10 It is shown Figure 1 A diagram of another embodiment of the image processing system 10 shown.

[0135] Reference Figure 10The image processing system 10a may include a camera module 100a and an image processing device 200a. The camera module 100a may include an image sensor 110a, an encoder 120a, a memory 130a, and an interface 140a. The image processing device 200a may include an interface 210a, a memory 220a, a decoder 230a, and an ISP 240a.

[0136] and Figure 1 Compared to the image processing system 10, Figure 10 The image processing system 10a also includes a mode selector 127a in the encoder 120a, and its other configurations can be integrated with... Figure 1 The configuration of the image processing system 10 is basically the same as that of the image processing system 10a. In the configuration of the image processing system 10a, the details will be omitted. Figure 1 The configuration of the image processing system 10 is described repeatedly.

[0137] According to this embodiment, encoder 120a can generate multiple compressed data CDTs by compressing image data IDT according to multiple compression modes. Here, each of the multiple compression modes can be set based on compression targets, compression ratios, error rates, and / or loss information. Additionally, encoder 120a may include mode information in the header of the bitstream during compression. The following will refer to… Figure 12A and Figure 12B A detailed description of the various compression modes is provided.

[0138] For example, encoder 120a can generate a first reference map RM1 that includes pixel values ​​of uncorrected bad pixels, and can compress image data IDT according to a first mode in which a bad pixel detection operation for the first reference map RM1 is performed during compression using the first reference map RM1. In this case, information indicating the first mode can be included in the header of the bitstream.

[0139] Additionally, encoder 120a can be referenced above. Figures 1 to 9B The method generates a first reference map RM1 that includes corrected pixel values ​​of bad pixels, and the image data IDT can be compressed according to a second mode in which compression operations are performed without performing bad pixel detection operations for the first reference map RM1. In this case, information indicating the second mode can be included in the header of the bitstream.

[0140] The present invention is not limited thereto; encoder 120a can compress image data IDT according to three or more modes. Furthermore, the compression method for each of the multiple compression modes can be configured differently from the above embodiments.

[0141] The mode selector 127a of encoder 120a can select one of multiple modes and sends compressed data CDT corresponding to the selected compression mode to image processing device 200a via interface 140a. Here, mode selector 127a can be a switch implemented in software and / or hardware. Furthermore, mode selector 127a can select a mode from the multiple modes based on compression target, compression ratio, error rate, and / or loss information, and the mode information can be included in the header of the bitstream. For example, the compression ratio can be set based on user input, manufacturing specifications, etc., and mode selector 127a can select the multiple modes based on a predetermined compression ratio. In some embodiments, mode selector 127a can select one of the multiple modes based on error data of the multiple modes. Here, error data refers to data representing the difference between the data obtained by recovering the compressed data CDT according to a specific mode and the image data IDT. Mode selector 127a can select the mode with the least error based on the error data of the multiple modes and send compressed data CDT corresponding to the selected mode to image processing device 200a.

[0142] According to the present invention, decoder 230a can generate multiple decompressed data DDTs by decompressing compressed data CDT according to multiple modes. Specifically, decoder 230a can check mode information based on the header of the bitstream included in the compressed data CDT. In addition, decoder 230a can decompress the bitstream according to the decompression method corresponding to the checked mode information.

[0143] For example, when the header of the bitstream includes mode information indicating the first mode, the decoder 230a can decompress the bitstream according to the decompression method corresponding to the first mode. Similarly, when the header of the bitstream includes mode information indicating the second mode, the decoder 230a can decompress the bitstream according to the decompression method corresponding to the second mode.

[0144] Figure 11 This is a diagram illustrating encoder 120a according to an embodiment.

[0145] Figure 11 Showing more details Figure 10 Encoder 120a. (Refer to...) Figure 10 and Figure 11 The encoder 120a may include a bad pixel detector 121a, a compressor 123a, a first reference map generator 125a, and a mode selector 127a.

[0146] Reference Figure 11The bad pixel detector 121a, compressor 123a, and first reference map generator 125a can generate multiple compressed data CDTs by performing compression on the image data IDT according to a compression method corresponding to each of the multiple modes. Additionally, compressor 123a can generate error data ED based on the multiple compressed data CDTs. In some embodiments, the bad pixel detector 121a, compressor 123a, and / or first reference map generator 125a can compress the image data IDT according to each of the multiple modes under the control of mode selector 127a.

[0147] Compressor 123a can generate recovered data by recovering each of the multiple compressed data CDTs, relative to each of the multiple compressed data CDTs, and calculate the error between the recovered data and the image data IDT. Additionally, compressor 123a can generate error data ED that includes the errors of the multiple modes. Furthermore, compressor 123a can send the error data ED to mode selector 127a.

[0148] The mode selector 127a can identify the mode with the lowest error based on the error data ED, and send the compressed data CDT corresponding to the identified mode to the image processing device 200a through the interface 140a.

[0149] According to this embodiment, compressor 123a can generate error data ED for each pixel group. That is, compressor 123a can calculate the error between each of the n pixel groups (where n is a positive integer) forming image data IDT and each of the n pixel groups forming the recovered data. Furthermore, compressor 123a can generate error data ED for each pixel group, the error data ED including errors of multiple modes. Mode selector 127a can identify the mode with the lowest error for each pixel group based on the error data ED, and send the bitstream corresponding to the identified mode to image processing device 200a. For example, mode selector 127a determines the mode in which decoder 230a can decompress compressed data CDT, so that when decompressed, decompressed data DDT has the lowest error rate. Here, the error can be referred to as the error generated during the recovery of compressed image data CDT by decoder 230a. However, the error is not limited to this, but may include other aspects or causes in which distortion may occur relative to the original image.

[0150] In the above embodiments, although it is described that the bad pixel detector 121a, compressor 123a, and first reference map generator 125a generate multiple compressed data CDTs by performing compression according to a compression method corresponding to each of the multiple modes, the inventive concept is not limited thereto. For example, encoder 120a may be implemented to include bad pixel detector 121a, compressor 123a, and / or first reference map generator 125a corresponding to each of the multiple modes. For example, encoder 120a may be implemented to include bad pixel detector 121a, compressor 123a, and / or first reference map generator 125a corresponding to a first mode, and to include bad pixel detector 121a, compressor 123a, and / or first reference map generator 125a corresponding to a second mode.

[0151] Although reference Figure 10 and Figure 11 The mode selector 127a is shown and described as being included in the encoder 120a, but the inventive concept is not limited thereto. For example, the mode selector 127a may be implemented in a configuration separate from the encoder 120a in the camera module 100a.

[0152] Figure 12A and Figure 12B This is a table explaining the compression information of an exemplary embodiment based on the concept of the present invention. In detail, Figure 12A and Figure 12B This illustrates the compression mode (compression method) according to the standard proposed by the MIPI Association. (Refer to...) Figure 10 and Figure 11 Let's describe them together.

[0153] Reference Figure 12A It can compress Bayer pattern according to various compression modes. Figure 3 Image data IDT. As examples of compression modes, pixel-based directional difference (PD), diagonal-based difference (DGD), extended tilted horizontal or vertical difference (eSHV), OUTlier compensation (OUT) mode, and / or fixed quantization and no-reference (FNR) mode can be used. However, the compression mode is not limited to these and may include any other compression modes capable of compressing data.

[0154] In PD mode, DPCM can be performed on image data IDT following the Bayer pattern. PD mode can be divided into MODE0, MODE1, MODE2, MODE3, MODE12, and MODE13 according to the detailed implementation algorithm. Because 4 bits can be allocated to the header indicating the compression method, sixteen compression modes are represented in headers with different bit values. For example, MODE0 can be represented as bit 0000, MODE1 as bit 0001, MODE2 as bit 0010, MODE3 as bit 0011, MODE12 as bit 1100, and MODE13 as bit 1101.

[0155] In DGD mode, DPCM can be performed on image data IDT with a diagonal structure. DGD mode can be divided into MODE4 (bit 0100), MODE5 (bit 0101), MODE8 (bit 1000), MODE9 (bit 1001), MODE10 (bit 1010), and MODE11 (bit 1011) according to the detailed implementation algorithm.

[0156] Similarly, the eSHV mode may include MODE14 (bit 1110) and MODE15 (bit 1111), the OUT mode may include MODE7 (bit 0111), and the FNR mode may include MODE6 (bit 0110). According to an embodiment, MODE7 may refer to either the OUT mode, which includes a bad pixel (BP) mode for processing bad pixels, or the saturation mode, but one of the two modes (saturation mode or OUT mode) may be selected depending on the operating environment.

[0157] In an embodiment, the mode selector 127a can evaluate PD mode, DGD mode, eSHV mode, OUT mode, and FNR mode in sequence, and can select the optimal mode based on a compression evaluation index that includes indices such as compression ratio and lost information. However, the inventive concept is not limited to the above-described mode evaluation order.

[0158] Reference Figure 12B Image data in a four-grid pattern ( Figure 4 The image data (IDT) can be compressed according to various compression modes. The inventive concept is not limited to this; image data in which red pixel groups, blue pixel groups, first green pixel groups, and second green pixel groups are repeatedly arranged can also be compressed according to various compression modes. For example, the image data may have red pixel groups, blue pixel groups, first green pixel groups, and second green pixel groups comprising pixels arranged in a 2n×2n or 3n×3n pattern (where n is a positive integer).

[0159] As compression modes, the following modes are used: average-based directional difference (AD) mode, extended horizontal or vertical directional difference (eHVD) mode, tilt-based difference (OD) mode, extended multi-pixel-based difference (eMPD) mode, extended horizontal or vertical average-based difference (eHVA) mode, extended OUTlier compensation (eOUT) mode, or FNR mode.

[0160] In AD mode, DPCM can be performed on image data IDT comprising multiple pixels in a Bayer group (PG) formed from one pixel group (PG). AD mode can be categorized into MODE0, MODE1, MODE2, and MODE3 based on the detailed implementation algorithm. Because 4 bits can be allocated to the header indicating the compression method, the sixteen compression modes can be represented by different bits in the header information. For example, MODE0 can be represented as bit 0000, MODE1 as bit 0001, MODE2 as bit 0010, and MODE3 as bit 0011.

[0161] In OD mode, compression can be performed on diagonally structured image data using IDT. OD mode can be divided into MODE4 (bit 0100) and MODE5 (bit 0101) depending on the detailed implementation algorithm.

[0162] Similarly, eMPD may include MODE8 (bit 1000), MODE9 (bit 1001), MODE10 (bit 1010), and MODE11 (bit 1011); eHVD may include MODE12 (bit 1100) and MODE13 (bit 1101); eHVA may include MODE14 (bit 1110); (e)OUT mode (eOUT mode or OUT mode) may include MODE15 (bit 1111) and MODE7 (bit 0111); and FNR mode may include MODE6 (bit 0110). According to an embodiment, MODE7 may refer to either (e)OUT mode including BP mode for handling bad pixels or saturation mode, but one of the two modes (saturation mode or (e)OUT mode) may be selected depending on the operating environment.

[0163] In an embodiment, the mode selector 127a can evaluate AD mode, eHVD mode, OD mode, eMPD mode, eHVA mode, eOUT mode, and FNR mode in sequence, and can select the optimal mode based on a compression evaluation index that includes indices such as compression ratio and lost information. However, the inventive concept is not limited to the described mode evaluation order.

[0164] Figure 13 This is a diagram illustrating an image processing system 10b according to an embodiment. Figure 13 It shows Figure 1 Another embodiment of the image processing system 10.

[0165] Reference Figure 13 The image processing system 10b may include a camera module 100b and an image processing device 200b. The camera module 100b may include an image sensor 110b and an interface 140b. The image processing device 200b may include an interface 210b, a memory 220b, a decoder 230b, an ISP 240b, and an encoder 250b. Figure 13 The encoder 250b can correspond to Figure 1 encoder 120 or Figure 10 The encoder 120a.

[0166] Will Figure 13 Image processing system 10b and Figure 1 Compared to the image processing system 10, the difference lies in the inclusion of an encoder 250b in the image processing device 200b instead of the camera module 100b, while its other configurations can be substantially the same. In the configuration of the image processing system 10b, the encoder 250b will be omitted. Figure 1 The configuration of the image processing system 10 is described repeatedly.

[0167] Reference Figure 13 Image sensor 110b can generate image data IDT. The image data IDT can be sent to image processing device 200b via interface 140b. According to an embodiment, camera module 100b may also include an image processing unit (ISP). The image data IDT can be processed by the ISP, and the processed image data IDT can be sent to image processing device 200b.

[0168] Image processing device 200b can receive image data IDT via interface 210b. Encoder 250b can generate compressed data CDT by compressing the image data IDT using a first reference map RM1 stored in memory 220b. The method by which encoder 250b generates compressed data CDT has been described above, therefore redundant descriptions will be omitted. The compressed data CDT can be stored in memory 220b and can be read from memory 220b by decoder 230b. Decoder 230b can generate decompressed data DDT by decompressing the image data IDT using a second reference map RM2 stored in memory 220b. The method by which decoder 230b generates decompressed data DDT is the same as described above. Figures 10 to 11 The descriptions are basically the same, therefore, redundant descriptions will be omitted.

[0169] Figure 14 This is a diagram illustrating an electronic device 1000 according to an embodiment.

[0170] Reference Figure 14 The electronic device 1000 may include a camera module 1100, an application processor (AP) 1200, a display device 1300, a memory 1400, a storage unit 1500, a user interface 1600, and a wireless transceiver 1700. Figure 14 The camera module 1100 can correspond to Figure 1 Camera module 100 Figure 10 Camera module 100a or Figure 13 Camera module 100b. Figure 14 The AP1200 may include Figure 1 Image processing device 200 Figure 10 Image processing device 200a or Figure 13 Image processing device 200b. (Omitted from the above reference) Figure 1 , Figure 10 and Figure 13 The redundant descriptions.

[0171] The AP 1200 controls the overall operation of the electronic device 1000 and can be configured as a system-on-a-chip (SoC) to drive applications, operating systems, etc.

[0172] The memory 1400 can store programs and / or data processed or executed by the AP 1200. The storage unit 1500 can be implemented as a non-volatile memory device such as NAND flash memory or resistive memory. For example, the storage unit 1500 can be configured as a memory card (Multimedia Card (MMC), Embedded MMC (eMMC), Secure Digital (SD), Micro SD, etc.). The storage unit 1500 can store data and / or programs for executing algorithms that control image processing operations of the AP 1200, and the data and / or programs can be loaded into the memory 1400 when image processing operations are performed.

[0173] User interface 1600 can be various devices capable of receiving user input, such as a keyboard, curtain keypad, touch panel, fingerprint sensor, microphone, etc. User interface 1600 can receive user input and provide signals corresponding to the received user input to AP 1200. Wireless transceiver 1700 may include modem 1710, transceiver 1720, and antenna 1730.

[0174] Figure 15 This is a diagram showing a portion of an electronic device 2000 according to an embodiment. Figure 16 This is a diagram illustrating a detailed configuration of a camera module according to an embodiment. Specifically, Figure 15 It is shown as Figure 14 A diagram of a portion of electronic device 1000 and electronic device 2000. Figure 16 It is shown Figure 15 A diagram showing the detailed configuration of the second camera module 2100b.

[0175] Reference Figure 15 The electronic device 2000 may include a multi-camera module 2100, an AP 2200, and a memory 2300. The memory 2300 can be connected to... Figure 14 The memory 1400 shown performs the same function, therefore, redundant descriptions are omitted.

[0176] Electronic device 2000 can capture and / or store images of objects using a CMOS image sensor, and can be implemented as a mobile phone, tablet computer, or portable electronic device. Portable electronic devices may include laptop computers, mobile phones, smartphones, tablet PCs, wearable devices, etc.

[0177] The multi-camera module 2100 may include a first camera module 2100a, a second camera module 2100b, and a third camera module 2100c. The multi-camera module 2100 can be used with... Figure 1 Camera module 100 Figure 10 Camera module 100a or Figure 13 The camera module 100b performs the same function. Figure 15 Although the multi-camera module 2100 is shown as including three camera modules (i.e., the first camera module 2100a, the second camera module 2100b, and the third camera module 2100c), the inventive concept is not limited thereto, and the multi-camera module 2100 may include a variety of numbers of camera modules.

[0178] In the following text, reference will be made to Figure 16 The detailed configuration of the second camera module 2100b is described in more detail, but the following description is equally applicable to other camera modules (i.e., the first camera module 2100a and the third camera module 2100c) according to the embodiments.

[0179] Reference Figure 16 The second camera module 2100b may include a prism 2105, an optical path folding element (hereinafter, "OPFE") 2110, an actuator 2130, an image sensing device 2140, and a storage unit 2150.

[0180] The prism 2105 may include a reflective surface 2107 of a light-reflecting material to alter the path of light L incident from the outside.

[0181] According to an embodiment, prism 2105 can change the path of light L incident in the first direction X to a second direction Y perpendicular to the first direction X. Furthermore, prism 2105 can rotate the reflective surface 2107 of the light-reflecting material about its central axis 1106 in either direction A or B, thereby changing the path of light L incident in the first direction X to a second direction Y perpendicular to the first direction X. At this time, OPFE 2110 can also move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0182] For example, OPFE 2110 may include an optical lens having m groups (where m is a natural number). The m lenses can be moved in the second direction Y and change the optical zoom ratio of camera module 2100b. For example, when the basic optical zoom ratio of camera module 2100b is Z and the m optical lenses included in OPFE 2110 are moved, the optical zoom ratio of camera module 2100b can be changed to 3Z, 5Z, or an optical zoom ratio greater than 5Z.

[0183] Actuator 2130 can move OPFE 2110 or some optical lenses (hereinafter referred to as optical lenses) to a specific position. For example, actuator 2130 can adjust the position of the optical lenses so that image sensor 2142 is positioned at the focal length of the optical lenses for accurate sensing.

[0184] Image sensing device 2140 may include image sensor 2142, logic unit 2144, encoder 2145, and memory 2146. Image sensor 2142 can sense an image of an object using light L provided through an optical lens. Figure 16 The image sensor 2142 can function similarly to Figure 1 Image sensor 110 Figure 10 Image sensor 110a or Figure 13 The functionality of the image sensor 110b is omitted here, therefore its redundant description will be omitted. Logic unit 2144 can control the overall operation of the second camera module 2100b.

[0185] The encoder 2145 can encode the sensed image data. Figure 16 The encoder 2145 can perform with Figure 1 Encoder 120, Figure 10 encoder 120a or Figure 13 The encoder 250b has similar functionality, therefore, redundant descriptions will be omitted. Figure 16 In this context, encoder 2145 is shown as included in logic element 2144, but is not limited thereto, and may be implemented as a single functional unit different from logic element 2144.

[0186] The memory 2146 may store information required for the operation of the second camera module 2100b, such as calibration data 2147. Calibration data 2147 may include information required by the second camera module 2100b to generate image data using light L provided externally. For example, calibration data 2147 may include information about the aforementioned rotation, information about the focal length, information about the optical axis, etc. When the second camera module 2100b is implemented as a multi-state camera in which the focal length changes according to the position of the optical lens, calibration data 2147 may include focal length values ​​for each position (or state) of the optical lens and information related to autofocus.

[0187] Storage unit 2150 can store image data sensed by image sensor 2142. In some embodiments, storage unit 2150 can store compressed data generated by encoder 2145. Storage unit 2150 can be disposed outside image sensing device 2140 and can be implemented in the form of stacking with sensor chip forming image sensing device 2140. In some embodiments, storage unit 2150 can be implemented as electrically erasable programmable read-only memory (EEPROM), but the inventive concept is not limited thereto.

[0188] Reference Figure 15 and Figure 16 In some embodiments, each of the first camera module 2100a, the second camera module 2100b, and the third camera module 2100c may include an actuator 2130 (also collectively referred to as "camera modules 2100a, 2100b, and 2100c"). Therefore, each of the camera modules 2100a, 2100b, and 2100c may include calibration data 2147, wherein the calibration data 2147 of the camera modules 2100a, 2100b, and 2100c may be the same as or different from each other depending on the operation of the actuator 2130 included in each of the camera modules 2100a, 2100b, and 2100c.

[0189] In an example embodiment, one of the camera modules 2100a, 2100b, and 2100c (e.g., the second camera module 2100b) may be a folding lens type camera module including the prism 2105 and OPFE 2110 as described above, while the other camera modules (e.g., the first camera module 2100a and the third camera module 2100c) may be vertical type camera modules without the prism 2105 and OPFE 2110. However, one or more embodiments are not limited thereto.

[0190] In an embodiment, one of the camera modules 2100a, 2100b, and 2100c (e.g., the third camera module 2100c) may be, for example, a vertical depth camera that extracts depth information using infrared (IR). In this case, the AP 2200 can generate a 3D depth image by merging image data provided from such a depth camera with image data provided from another camera module (e.g., the first camera module 2100a or the second camera module 2100b).

[0191] In an embodiment, at least two camera modules (e.g., the first camera module 2100a and the second camera module 2100b) from camera modules 2100a, 2100b, and 2100c may have different fields of view (FOV). In this case, for example, at least two camera modules (e.g., the first camera module 2100a and the second camera module 2100b) may have different optical lenses, but the inventive concept is not limited thereto. For example, the first camera module 2100a may have a smaller FOV than the second camera module 2100b and the third camera module 2100c.

[0192] In some embodiments, camera modules 2100a, 2100b, and 2100c may have different FOVs from each other. In this case, the optical lenses included in camera modules 2100a, 2100b, and 2100c may also be different from each other, but the inventive concept is not limited thereto.

[0193] In some embodiments, camera modules 2100a, 2100b, and 2100c may be physically separated from each other. In other words, camera modules 2100a, 2100b, and 2100c may not be divided and use the sensing area of ​​a single image sensor 2142. Instead, an independent image sensor 2142 may be provided within each of the camera modules 2100a, 2100b, and 2100c.

[0194] AP 2200 may include multiple subprocessors (e.g., a first subprocessor 2210a, a second subprocessor 2210b, and a third subprocessor 2210c), an image generator 2200, a camera module controller 2230, a memory controller 2240, and internal memory 2250. AP 2200 may be implemented separately from camera modules 2100a, 2100b, and 2100c. For example, AP 2200 and camera modules 2100a, 2100b, and 2100c may be implemented separately as separate semiconductor chips.

[0195] Image data or compressed data generated by the first camera module to the third camera modules 2100a, 2100b, and 2100c can be provided to the corresponding subprocessors 2210a, 2210b, and 2210c respectively via separate first image signal lines to third image signal lines ISLa, ISLb, and ISLc. For example, first image data generated from the first camera module 2100a can be provided to the first subprocessor 2210a via the first image signal line ISLa, second image data generated from the second camera module 2100b can be provided to the second subprocessor 2210b via the second image signal line ISLb, and third image data generated from the third camera module 2100c can be provided to the third subprocessor 2210c via the third image signal line ISLc. Image data transmission can be performed using a MIPI-based camera serial interface, but the embodiments are not limited thereto.

[0196] In an embodiment, a sub-processor may be provided to process image data output from multiple camera modules. For example, the first sub-processor 2210a and the third sub-processor 2210c may be implemented as a single sub-processor rather than separately, and the image data provided from the first camera module 2100a and the third camera module 2100c may be selected by a selection element (e.g., a multiplexer) and provided to the integrated sub-image processor.

[0197] Each of the first to third subprocessors 2210a, 2210b and 2210c may include Figure 1 Decoder 230, Figure 10 decoder 230a or Figure 13 The decoder 230b. The first to third subprocessors 2210a, 2210b, and 2210c can decompress the received compressed data to generate decompressed data, and output the generated decompressed data to the image generator 2220. The image generator 2220 can correspond to... Figure 1 ISP240, Figure 10 ISP 240a or Figure 13 ISP 240b.

[0198] The camera module controller 2230 can provide control signals to each of the camera modules 2100a, 2100b, and 2100c. The control signals generated from the camera module controller 2230 can be provided to the corresponding camera modules 2100a, 2100b, and 2100c via separate control signal lines CSLa, CSLb, and CSLc.

[0199] Although the inventive concept has been specifically shown and described with reference to the above embodiments, it will be understood that various changes in form and detail may be made without departing from the spirit and scope of this disclosure as defined in the appended claims.

Claims

1. An image compression method, the method compressing each of a plurality of pixel groups forming image data, the method comprising: Detect bad pixels among multiple pixels in a pixel group; Generate an indication of the location information of the bad pixel; Calculate the first difference between the pixel value of each pixel (excluding the bad pixel) and the reference pixel value; as well as A bit stream including the identifier and the first difference is generated.

2. The image compression method according to claim 1, wherein, Detecting the bad pixels includes: Whether a target pixel is a bad pixel is determined based on at least one of the pixel values ​​of the plurality of pixels in the pixel group and at least one pixel value of at least one pixel adjacent to the pixel group.

3. The image compression method according to claim 2, wherein, Determining whether the target pixel is a bad pixel includes: Calculate the average pixel value of one or more pixels adjacent to the target pixel in the pixel group; and Calculate the second difference between the average pixel value and the pixel value of the target pixel.

4. The image compression method according to claim 1 further includes: The reference pixel value is determined based on reference information including the pixel values ​​of pixels that were compressed before the pixel group.

5. The image compression method according to claim 4, wherein, Determining the reference pixel value includes: The reference pixel value is determined based on the pixel value of at least one pixel adjacent to the pixel group in the reference information.

6. The image compression method according to claim 4 further includes: After the bitstream is generated, the reference information is updated based on the pixel values ​​of the plurality of pixels in the pixel group.

7. The image compression method according to claim 6, wherein, The updated reference information includes: Correcting the pixel values ​​of the bad pixels in the pixel group; and Update the pixel values ​​of all pixels in the reference information except for the bad pixels in the pixel group, and the corrected pixel values ​​of the bad pixels.

8. The image compression method according to claim 7, wherein, Correcting the pixel value of the bad pixel in the pixel group further includes: correcting the pixel value of the bad pixel based on the pixel value of at least one pixel adjacent to the bad pixel.

9. The image compression method according to claim 1, further comprising: Generate compression information indicating the compression method applied to the pixel group. Generating the bitstream includes generating a bitstream that includes the compressed information.

10. A camera module, comprising: An image sensor configured to generate image data comprising multiple pixels; An encoder configured to divide the plurality of pixels into a plurality of pixel groups and compress the plurality of pixel groups in sequence to produce compressed data comprising a plurality of bit streams; as well as The memory stores reference information including pixel values ​​of pixels compressed by the encoder. The encoder is further configured as follows: Detect bad pixels in the first pixel group; Based on the detection results of the bad pixels and the reference information, a first bitstream corresponding to the first pixel group is generated by compressing the first pixel values ​​of a plurality of first pixels in the first pixel group. The first bitstream includes the difference between the pixel values ​​of pixels other than the bad pixels among the plurality of first pixels and the reference pixel value determined based on the reference information; and The reference information is updated based on the corrected pixel values ​​obtained by correcting the pixel values ​​of the bad pixels.

11. The camera module according to claim 10, wherein, The encoder is also configured to: The pixel value of the bad pixel is corrected based on the pixel value of at least one pixel adjacent to the bad pixel; as well as The reference information is updated by adding the pixel values ​​of one or more first pixels (excluding the bad pixel) and the corrected pixel value of the bad pixel to the reference information.

12. The camera module according to claim 11, wherein, The encoder is also configured to: The bad pixel is detected based on at least one of the pixel values ​​of the plurality of first pixels and the pixel values ​​of the target pixel adjacent to the plurality of first pixels.

13. The camera module according to claim 11, wherein, The encoder is also configured to: The reference pixel value is determined based on at least one pixel that is adjacent to the plurality of first pixels in the reference information; Calculate the difference between the pixel value of each pixel (excluding the bad pixel) among the plurality of first pixels and the reference pixel value; as well as The first bitstream including the difference is generated.

14. The camera module according to claim 13, wherein, The encoder is also configured to: Generate an indicator indicating the location information of the bad pixel and generate a first bitstream including the indicator.

15. The camera module according to claim 14, wherein, The encoder also includes a mode switch configured to generate multiple compressed data by compressing image data multiple times according to multiple modes corresponding to multiple compression methods, and to select, based on the multiple compressed data, a mode among the multiple modes that has the least error in the image data.

16. The camera module according to claim 15, wherein, The encoder is also configured to: Generate compressed information including information about the selected mode among the multiple modes, and generate a first bit stream including the generated compressed information.

17. An image processing system, comprising: An image sensor configured to generate image data comprising multiple pixels; An encoder configured to generate multiple bitstreams by sequentially compressing multiple groups of pixels in the image data; as well as A decoder configured to recover the image data by decompressing the plurality of bitstreams. The encoder is further configured as follows: Detect bad pixels in each of the plurality of pixel groups; The reference information is generated based on the pixel values ​​of a second pixel group in the plurality of pixel groups that were compressed prior to the second pixel group, according to reference information. The reference information is updated based on the result of detecting the bad pixels, and wherein the bitstream includes the difference between the pixel values ​​of the pixels in the plurality of pixels other than the bad pixels in the second pixel group and the reference pixel values ​​determined based on the reference information.

18. The image processing system according to claim 17, wherein, Detecting the bad pixels includes: The bad pixel is detected based on at least one pixel value of the pixels included in the second pixel group and the pixels adjacent to the target pixel of the second pixel group.

19. The image processing system according to claim 18, wherein, Compressing the pixel values ​​includes: Generate an indication of the location information of the bad pixel; Calculate the difference between the pixel values ​​of the pixels in the plurality of pixels, excluding the bad pixels in the second pixel group, and the reference pixel value determined based on the reference information; and Generate the bit stream that includes the identifier and the difference.

20. The image processing system according to claim 19, wherein, The updated reference information includes: The pixel value of the bad pixel is corrected based on at least one pixel value of a pixel adjacent to the bad pixel; and The corrected pixel values ​​of the bad pixels and the pixel values ​​of the pixels other than the bad pixels are updated to the reference information.

Images