Camera module, image processing device, and image compression method

Abstract

A camera module includes a compressor configured to divide a plurality of pixels included in image data into a plurality of pixel groups, calculate, for each of the plurality of pixel groups into which the plurality of pixels are divided, a representative pixel value of the corresponding pixel group based on pixel values of the plurality of pixels included in the corresponding pixel group, generate first compressed data based on the calculated representative pixel value of each of the plurality of pixel groups, calculate, for each of the plurality of pixel groups into which the plurality of pixels are divided, a residual value representing a difference between the pixel values of the plurality of pixels included in the corresponding pixel group and the representative pixel value of the corresponding pixel group, and generate second compressed data based on the calculated residual value of each of the plurality of pixel groups.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Korean Patent Application Nos. 10-2020-0050355 and 10-2021-0014395, filed with the Korean Intellectual Property Office on April 24, 2020 and February 1, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to camera modules, image processing devices, and image compression methods, and more specifically, to a camera module, image processing device, and image compression method using multi-resolution compressed data. Background Technology

[0004] Image compression can refer to the process of generating compressed image data with a smaller size than the original image data. Similarly, image decompression can refer to the process of generating reconstructed image data by decompressing compressed image data. Depending on the compression and decompression methods, the reconstructed image data can be the same as or different from the original image data. For example, reconstructed image data can include preview image data with a lower resolution than the original image data, as well as full-resolution image data with the same resolution as the original image data. Summary of the Invention

[0005] A camera module, an image processing device, and an image compression method are provided. The camera module is used to generate and use compressed data to generate images with multiple resolutions.

[0006] Additional aspects will be set forth in part in the description which follows, and will also be apparent in part from the description itself, or may be learned by practice of the embodiments presented.

[0007] According to an embodiment, a camera module is provided, including: an image sensor configured to generate image data including a plurality of pixels; a compressor configured to divide the plurality of pixels included in the generated image data into a plurality of pixel groups, and for each pixel group into which the plurality of pixels are divided, to calculate a representative pixel value of the corresponding pixel group based on the pixel values ​​of the plurality of pixels included in the corresponding pixel group, to generate first compressed data based on the calculated representative pixel value of each pixel group, and for each pixel group into which the plurality of pixels are divided, to calculate a residual value representing the difference between the pixel values ​​of the plurality of pixels included in the corresponding pixel group and the representative pixel value of the corresponding pixel group, and to generate second compressed data based on the calculated residual value of each pixel group. The camera module further includes: an interface configured to output the generated first compressed data and the generated second compressed data.

[0008] According to an embodiment, an image processing apparatus is provided, comprising: an interface configured to receive compressed data, the compressed data being obtained by compressing image data comprising a plurality of pixel groups in units of pixel groups; and a memory for storing the received compressed data. The image processing apparatus further comprises: a decompressor configured to decompress at least a portion of the stored compressed data to generate decompressed data; and an image signal processor configured to perform image processing based on the generated decompressed data to generate reconstructed image data. The reconstructed image data includes first image data having a first resolution and second image data having a second resolution, and the compressed data includes first compressed data for generating the first image data and the second image data, and second compressed data for generating the second image data.

[0009] According to an embodiment, an image compression method is provided for image data including multiple pixel groups. The image compression method includes: for each pixel group, calculating a representative pixel value of the corresponding pixel group based on the pixel values ​​of the multiple pixels included in the corresponding pixel group; and generating first compressed data based on the calculated representative pixel value of each pixel group. The image compression method further includes: for each pixel group, calculating a residual value representing the difference between the pixel values ​​of the multiple pixels included in the corresponding pixel group and the representative pixel value of the corresponding pixel group; and generating second compressed data based on the calculated residual value of each pixel group. Attached Figure Description

[0010] The above and other aspects, features, and advantages of embodiments of the present disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, in which:

[0011] Figure 1 This is a diagram illustrating an image processing system according to an embodiment;

[0012] Figure 2A and Figure 2B This is a diagram illustrating a pixel array according to an embodiment;

[0013] Figure 3 This is a flowchart illustrating an image compression method according to an embodiment;

[0014] Figure 4 This is a diagram illustrating a method for calculating representative pixel values ​​of a pixel group according to an embodiment;

[0015] Figure 5 This is a diagram illustrating a method for generating first compressed data using representative pixel values ​​according to an embodiment;

[0016] Figure 6This is a flowchart illustrating a method for generating first compressed data according to an embodiment;

[0017] Figure 7 This is a diagram illustrating the configuration of the first compressed data according to an embodiment;

[0018] Figure 8 This is a diagram illustrating a method for calculating the loss value and reference value of the first compressed data according to an embodiment;

[0019] Figure 9 This is a diagram illustrating a method for generating first compressed data using representative pixel values ​​according to an embodiment;

[0020] Figure 10 This is a diagram illustrating a method for calculating the flags of a pixel group according to an embodiment;

[0021] Figure 11 This is a diagram illustrating the configuration of the first compressed data according to an embodiment;

[0022] Figure 12 This is a diagram illustrating a method for generating second compressed data using residual values ​​according to an embodiment;

[0023] Figure 13 This is a diagram illustrating the configuration of the second compressed data according to an embodiment;

[0024] Figure 14 This is a diagram illustrating a bitstream including first compressed data and second compressed data according to an embodiment;

[0025] Figure 15 This is a diagram illustrating an image reading operation using a bitstream according to an embodiment;

[0026] Figure 16 This is a flowchart illustrating a method for generating a low-resolution image according to an embodiment;

[0027] Figure 17 This is a diagram illustrating a decompression operation using the first compressed data according to an embodiment;

[0028] Figure 18 This is a flowchart illustrating a method for generating a high-resolution image according to an embodiment;

[0029] Figure 19 This is a diagram illustrating a camera module according to an embodiment;

[0030] Figure 20 This is a diagram illustrating an image processing apparatus according to an embodiment;

[0031] Figure 21 This is a diagram illustrating a camera module according to an embodiment;

[0032] Figure 22 This is a diagram illustrating a camera module according to an embodiment;

[0033] Figure 23 This is a diagram illustrating an electronic device according to an embodiment;

[0034] Figure 24 This is a diagram illustrating a portion of an electronic device according to an embodiment;

[0035] Figure 25 This is a diagram illustrating a detailed configuration of the camera module according to an embodiment;

[0036] Figure 26 This is a diagram illustrating an electronic device according to an embodiment; and

[0037] Figure 27 This is a diagram illustrating an electronic device according to an embodiment. Detailed Implementation

[0038] Figure 1 This is a diagram illustrating an image processing system 10 according to an embodiment.

[0039] refer to Figure 1 The image processing system 10 may include a camera module 100, an image processing device 200, and a display 300. In one embodiment, the camera module 100 may include an image sensor 110, a compressor 120, and an interface (IF) 130. In another embodiment, the image processing device 200 may include an interface (IF) 210, a memory 220, a decompressor 230, and an image signal processor (ISP) 240.

[0040] As an example, the image processing system 10 can be implemented as a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. Portable electronic devices can be laptops, mobile phones, smartphones, tablet PCs, personal digital assistants (PDAs), enterprise digital assistants (EDAs), digital cameras, digital camcorders, audio equipment, portable multimedia players (PMPs), personal navigation devices (PNDs), MP3 players, handheld game consoles, e-book readers, wearable devices, etc. Furthermore, the image processing system 10 can be installed on electronic devices such as drones and advanced driver assistance systems (ADAS), or on electronic devices that are components of vehicles, furniture, manufacturing facilities, doors, various measuring devices, etc.

[0041] Camera module 100 can capture images of external objects and generate image data IDT. For example, camera module 100 may include image sensor 110 for converting the object's light signals into electrical signals. Image sensor 110 may include a plurality of pixels arranged in a two-dimensional manner and may include a pixel array 111. One of a plurality of reference colors may be assigned to each of the plurality of pixels. For example, the plurality of reference colors may include red, green, and blue (RGB), or red, green, blue, and white (RGBW).

[0042] As a non-limiting example, the image sensor 110 can be implemented using a charge-coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) image sensor, and can be implemented as various types of optoelectronic devices. In some embodiments, the image sensor 110 can output image data IDT, in which preprocessing (e.g., correction of defective pixels) has been performed on the pixel signals generated by the pixel array 111.

[0043] In one embodiment, the camera module 100 may use compressor 120 to compress image data IDT to reduce power consumption during data transmission and improve the efficiency of data storage. Compressor 120 may receive image data IDT from image sensor 110 and compress the image data IDT to generate compressed data.

[0044] Compressor 120 can generate first compressed data CDT1 and second compressed data CDT2 based on image data IDT, which are used to generate images with different resolutions. For example, compressor 120 can generate first compressed data CDT1, which is used to generate both a first image with a first resolution and a second image with a second resolution. Additionally, compressor 120 can generate second compressed data CDT2, which is used to generate the second image. The second resolution can be greater than the first resolution. In some embodiments, the second resolution can be the same as the resolution of the image data IDT generated by camera module 100.

[0045] In some embodiments, a first image having a first resolution may be referred to as a low-resolution image or a preview image, while a second image having a second resolution may be referred to as a high-resolution image or a full-resolution image. In this specification, for ease of understanding, the first image is referred to as a low-resolution image and the second image as a high-resolution image, but the embodiments are not limited thereto.

[0046] The compressor 120 can calculate representative pixel values ​​for multiple pixel groups constituting the image data IDT, and can generate first compressed data CDT1 based on the calculated representative pixel values. Additionally, the compressor 120 can generate second compressed data CDT2 based on the image data IDT and the representative pixel values. The first compressed data CDT1 and the second compressed data CDT2 can be implemented in the form of a bitstream.

[0047] Compressor 120 can provide the generated first compressed data CDT1 and second compressed data CDT2 to image processing device 200 via interface 130. For example, interface 130 can be implemented as a Camera Serial Interface (CSI) based on Mobile Industrial Processor Interface (MIPI). However, the type of interface 130 is not limited to this and can be implemented according to various protocol standards.

[0048] Image processing device 200 can convert image data received from camera module 100 to generate image data to be displayed on display 300. Image processing device 200 can receive first compressed data CDT1 and second compressed data CDT2 from camera module 100, and perform image processing operations based on any one or any combination of the first compressed data CDT1 and the second compressed data CDT2 to generate image data. Here, the image data may include low-resolution image data LIDT and high-resolution image data HIDT.

[0049] In some embodiments, the image processing device 200 can generate low-resolution image data LIDT based on the first compressed data CDT1, or it can generate high-resolution image data HIDT based on the first compressed data CDT1 and the second compressed data CDT2. The image processing device 200 can receive the first compressed data CDT1 and the second compressed data CDT2 from the camera module 100 through interface 210. Like interface 130, interface 210 can be implemented as MIPI, but is not limited thereto. The image processing device 200 can store the received first compressed data CDT1 and the second compressed data CDT2 in memory 220.

[0050] Memory 220 is a storage device for storing data and can store, for example, an operating system (OS), various programs, and various types of data (e.g., image data). Memory 220 can be a volatile memory such as dynamic random access memory (DRAM) or static RAM (SRAM), or a non-volatile memory such as phase-change RAM (PRAM), resistive RAM (ReRAM), or flash memory.

[0051] When the image processing device 200 receives a request for a low-resolution image, the decompressor 230 can read first compressed data CDT1 from the memory 220 and decompress the first compressed data CDT1 to generate first decompressed data DDT1. Conversely, when the image processing device 200 receives a request for a high-resolution image, the decompressor 230 can read first compressed data CDT1 and second compressed data CDT2 from the memory 220 and decompress both to generate second decompressed data DDT2. The decompressor 230 can output either the generated first decompressed data DDT1 or the second decompressed data DDT2 to the image signal processor 240. The first decompressed data DDT1 may correspond to low-resolution image data LIDT, and the second decompressed data DDT2 may correspond to high-resolution image data HIDT.

[0052] Each of the compressor 120 and decompressor 230 can be implemented as software or hardware, or as a combination of software and hardware such as firmware. When the compressor 120 and decompressor 230 are implemented as software, each of the above functions can be implemented as programmed source code and can reside in the storage medium included in each of the camera module 100 and the image processing device 200. The processor (e.g., microprocessor) included in each of the camera module 100 and the image processing device 200 can execute the software and thus implement the functions of the compressor 120 and decompressor 230. When the compressor 120 and decompressor 230 are implemented in hardware, the compressor 120 and decompressor 230 can include logic circuitry and registers, and can perform the above functions based on register settings.

[0053] The image signal processor 240 can perform various image processing operations on the received image signal. In a non-limiting example, the image signal processor 240 can perform at least one of the following image processing operations on the image signal: defect pixel correction, offset correction, lens distortion correction, color gain correction, shadow correction, gamma correction, noise reduction, and sharpening. In some embodiments, some of the above image processing operations can be omitted depending on the performance of the camera module 100. For example, when the camera module 100 includes a high-quality image sensor 110, defect pixel correction or offset correction in the image processing operations can be omitted.

[0054] When the first decompressed data DDT1 is received, the image signal processor 240 can perform image processing on the first decompressed data DDT1 to generate low-resolution image data LIDT. Furthermore, when the second decompressed data DDT2 is received, the image signal processor 240 can perform image processing on the second decompressed data DDT2 to generate high-resolution image data HIDT. The image signal processor 240 can output either the low-resolution image data LIDT or the high-resolution image data HIDT to the display 300.

[0055] Display 300 can display various content (e.g., text, images, videos, icons, or symbols) to a user based on low-resolution image data LIDT and / or high-resolution image data HIDT received from image processing device 200. For example, display 300 may include liquid crystal display (LCD), light-emitting diode (LED) display, organic light-emitting diode (OLED) display, microelectromechanical system (MEMS) display, or electronic paper display.

[0056] exist Figure 1 In this embodiment, the image processing system 10 is shown to include a camera module 100, an image processing device 200, and a display 300, but the embodiment is not limited thereto. For example, the image processing system 10 may include only some of the camera module 100, image processing device 200, and display 300, or it may include multiple camera modules 100. Furthermore, although... Figure 1 The decompressor 230 and the image signal processor 240 are shown as separate components, but the embodiment is not limited thereto. For example, the image signal processor 240 may include the decompressor 230.

[0057] A first compressed data CDT1 for generating a low-resolution image can be generated, and a second compressed data CDT2 for generating a high-resolution image can be generated based on the first compressed data CDT1. Therefore, compared to generating compressed data for low-resolution and high-resolution images separately, the amount of data in the first compressed data CDT1 and the second compressed data CDT2 can be reduced. This reduces power consumption caused by transmitting and receiving compressed data between various devices. Furthermore, the efficiency of storage space for storing compressed data can be improved.

[0058] Figure 2A and Figure 2B This is a diagram illustrating pixel arrays 113 and 115 according to an embodiment. Figure 2A This is a diagram showing the pixel array 113 of the TETRA pattern. Figure 2B This is a diagram showing the pixel array 115 of the NONA pattern.

[0059] Pixel arrays 113 and 115 may include multiple row lines, multiple column lines, multiple pixels PX connected to the row lines and column lines respectively and arranged in a matrix, and multiple color filters arranged to correspond to the multiple pixels PX respectively.

[0060] Color filters can be applied in the form of Bayer color filters. The Bayer pattern is based on the assumption that the human eye receives most of the brightness data from the green component of an object. Half of the pixels (PX) included in a Bayer color filter can detect green signals, another quarter can detect red signals, and the remaining quarter can detect blue signals.

[0061] For example, a Bayer color filter can be configured such that 2×2 cells, comprising a red pixel, a blue pixel, and two green pixels, are repeatedly arranged. In another example, a Bayer color filter can be configured such that 2×2 cells, comprising a red pixel, a blue pixel, and two wide green pixels (W pixels), are repeatedly arranged. That is, the following RGB color filter method can be primarily chosen: a green color filter is arranged in two of the four pixels (PX), and a blue color filter and a red color filter are arranged in the remaining two pixels, respectively.

[0062] However, the types of color filters are not limited to the examples above. A color filter can have the following configuration: in this configuration, groups of pixels (PGs) corresponding to a reference color are repeatedly arranged. For example, a reference... Figure 2A The color filter can be configured such that a red pixel group, including red pixels R arranged in a 2×2 pattern, a first green pixel group, including first green pixels G arranged in a 2×2 pattern, a blue pixel group, including blue pixels B arranged in a 2×2 pattern, and a second green pixel group, including green pixels G arranged in a 2×2 pattern, are repeatedly arranged. Such a pattern can be called a TETRA pattern.

[0063] As another example, see Figure 2B The color filter can have the following configuration: in this configuration, a red pixel group including a red pixel R arranged in 3×3, a first green pixel group including a first green pixel G arranged in 3×3, a blue pixel group including a blue pixel B arranged in 3×3, and a second green pixel group including a green pixel G arranged in 3×3 are repeatedly arranged. Such a pattern can be called a NONA pattern.

[0064] Figure 1Image sensor 110 can generate image data IDT, which includes pixel values ​​of pixels PX constituting pixel arrays 113 and 115. The pixel values ​​of pixels PX can include information about a reference color (e.g., red, blue, green, etc.) corresponding to the corresponding pixel PX. Image data IDT can be compressed in units of pixel groups PG. Here, pixel groups PG can include pixels arranged adjacent to each other and corresponding to the same reference color. However, embodiments are not limited to this; when pixel array 113 includes the Bayer color filter described above, pixel groups PG can include a predetermined number (e.g., 4) of pixels PX arranged adjacent to each other.

[0065] For example, refer to Figure 2A Because pixel array 113 is a TETRA pattern, the image data IDT can be compressed in units of pixel groups PG, which include four pixels PX corresponding to red, blue, or green. As another example, see [reference]. Figure 2B Because the pixel array 115 is a NONA pattern, the image data IDT can be compressed in units of pixel groups PG, which include 9 pixels PX corresponding to red, blue or green.

[0066] Although Figure 2A and Figure 2B The diagram illustrates pixel arrays 113 and 115 for an RGB color filter method, but the embodiment is not limited thereto. For example, a cyan, yellow, green, and magenta (CYGM) color filter method can be applied, wherein cyan, yellow, green, and magenta filters are provided on at least one pixel. Furthermore, a CYMK color filter method can be applied. Additionally, although Bayer patterns are shown in the embodiment for ease of description, the embodiment is not limited to Bayer patterns, and it should be understood that color filters including white or yellow, or color filters with various patterns in which two or more color regions are combined, can be applied.

[0067] Additionally, although reference Figure 2A and Figure 2B The image data IDT is described as including information about a reference color (e.g., RGB information), but embodiments are not limited thereto. Image sensor 110 can convert the RGB information of each pixel PX into YUV information, including information about brightness and color difference, through color space conversion. Therefore, the image data IDT can include YUV information corresponding to each pixel PX.

[0068] Additionally, as in the embodiments described above, the image data IDT including YUV information can also be compressed on a per-pixel-group (PG) basis. However, since the image data IDT including YUV information does not include information about the reference color, the pixel group (PG) can be classified only based on the position of the pixel (PX).

[0069] Image data IDT, which includes YUV information, can classify adjacent pixels (PX) into a pixel group (PG). For example, as referenced above. Figure 2A and Figure 2B As described, pixels arranged in a 2×2 or 3×3 configuration can be grouped into a pixel group (PG). Furthermore, image data IDT, including YUV information, can be compressed at the pixel group (PG) level.

[0070] Figure 3 This is a flowchart illustrating an image compression method according to an embodiment. Figure 3 It is shown Figure 1 A flowchart of the image compression method of the image processing system 10. Figure 3 Any one or any combination of operations can be performed by the camera module 100 or the image processing device 200.

[0071] refer to Figures 1 to 3 For each pixel group PG in a plurality of pixel groups of image data IDT, image processing system 10 can calculate a representative pixel value for the corresponding pixel group PG based on the pixel values ​​of the pixels PX included in the corresponding pixel group PG (operation S100). Image processing system 10 can classify image data IDT including information about a reference color (e.g., RGB information) into pixel groups PG, where pixel groups PG include pixels that are adjacent to each other and have information about the same reference color. However, embodiments are not limited to this; when image data IDT includes YUV information, image processing system 10 can classify image data IDT into pixel groups PG that include pixels that are adjacent to each other, without distinguishing the reference color. In addition, image processing system 10 can calculate a representative pixel value representing each pixel group PG in a plurality of pixel groups PG. Reference will be made later. Figure 4 Provides a detailed description of the method for calculating representative pixel values.

[0072] Additionally, the image processing system 10 can generate first compressed data CDT1 based on representative pixel values ​​of multiple pixel groups PG (operation S200). The image processing system 10 can generate first compressed data CDT1 by compressing representative pixel values ​​of multiple pixel groups PG. See later... Figures 5 to 11 A detailed description of the method for generating the first compressed data CDT1 is provided.

[0073] Furthermore, for each pixel group in the plurality of pixel groups PG, the image processing system 10 can calculate a residual value, which indicates the difference between the pixel value of pixel PX included in pixel group PG and the representative pixel value of the corresponding pixel group PG (operation S300). See later for further details. Figure 12 A detailed description of the method for calculating the residual values ​​is provided. Additionally, the image processing system 10 can generate second compressed data CDT2 based on the residual values ​​of multiple pixel groups PG (operation S400). See later for further details. Figure 13 and Figure 14 A detailed description of the method for generating the second compressed data CDT2 is provided.

[0074] Figure 4 This is a diagram illustrating a method for calculating representative pixel values ​​of a pixel group according to an embodiment. In the following description, for ease of explanation, the following assumptions are made: [like...] Figure 2A Like pixel array 113, pixel array 111a has a TETRA pattern, and the image data IDT includes information about the reference color. Figure 4 The pixel array 111a may include adjacent red pixel groups PG(R1), PG(R2), and PG(R3) that correspond to the color red. For example, the first red pixel group PG(R1) may include four red pixels R11, R12, R13, and R14; the second red pixel group PG(R2) may include four red pixels R21, R22, R23, and R24; and the third red pixel group PG(R3) may include four red pixels R31, R32, R33, and R34.

[0075] Compressor 120 can calculate the average pixel value of each pixel group in the red pixel groups PG(R1), PG(R2), and PG(R3). For example, compressor 120 can calculate the average pixel value R1 of red pixels R11, R12, R13, and R14 included in the first red pixel group PG(R1), the average pixel value R2 of red pixels R21, R22, R23, and R24 included in the second red pixel group PG(R2), and the average pixel value R3 of red pixels R31, R32, R33, and R34 included in the third red pixel group PG(R3). Compressor 120 can determine the calculated average pixel values ​​R1, R2, and R3 as representative pixel values ​​of the red pixel groups PG(R1), PG(R2), and PG(R3), respectively.

[0076] although Figure 4 The method for calculating representative pixel values ​​for the red pixel group is shown, but the above embodiment can also be applied to the green and blue pixel groups. Furthermore, although... Figure 4The illustration shows that compressor 120 determines the average pixel value of pixel group PG as the representative pixel value, but the embodiment is not limited to this. For example, compressor 120 may determine the median value among the pixel values ​​of pixel group PG as the representative pixel value, or it may determine the pixel value of the pixel at a fixed position in pixel group PG (as a non-limiting example, the pixel located at the top left of the four pixels included in pixel group PG) as the representative pixel value. Additionally, although... Figure 4 The illustration shows pixel array 111a as a TETRA pattern, but the embodiment is not limited to this; pixel array 111a may have a NONA pattern. Even in this case, compressor 120 may determine the representative pixel value in substantially the same manner as in the embodiment described above. For example, compressor 120 may determine the representative pixel value as the average pixel value of the nine pixels included in pixel group PG, or it may determine the representative pixel value as the pixel value of a pixel at a fixed position (as a non-limiting example, the pixel located at the center of pixel group PG).

[0077] Furthermore, even when the image data IDT includes YUV information, the compressor 120 can determine the representative pixel value for all pixel groups PG of the image data IDT in essentially the same manner as in the embodiments described above. That is, for each pixel group PG of the image data IDT that includes YUV information, the compressor 120 can determine the average pixel value as the representative pixel value, or it can determine the pixel value of the pixel at a fixed position in the pixel group PG as the representative pixel value.

[0078] Figure 5 This is a diagram illustrating a method for generating first compressed data CDT1 using representative pixel values ​​according to an embodiment.

[0079] refer to Figure 4 and Figure 5 The compressor 120 can generate first compressed data CDT1 by compressing the representative pixel values ​​of pixel groups corresponding to the same reference color. For example, the compressor 120 can compress the representative pixel values ​​of the red pixel group, the blue pixel group, and the green pixel group to generate the first compressed data CDT1.

[0080] Furthermore, compressor 120 can sequentially compress representative pixel values ​​of pixel groups corresponding to the same reference color (hereinafter referred to as same-color pixel groups). Compressor 120 can calculate the difference DIFF between the representative pixel value REP of the target pixel group to be compressed and the predicted value PRE of the target pixel group. Here, the predicted value PRE of the target pixel group can be determined based on the pixel values ​​of the same-color pixel groups that were previously compressed before the target pixel group. Furthermore, compressor 120 can generate an n-bit bitstream (n is a positive integer) based on the calculated difference DIFF. In addition, compressor 120 can generate first compressed data CDT1 including this n-bit bitstream.

[0081] In some embodiments, the predicted value PRE of the target pixel group can be the representative pixel value REP of a previously compressed group of the same color that immediately precedes the target pixel group. Because the target pixel group and the previously compressed group of the same color are adjacent to each other, they are very likely to have similar pixel values. Therefore, to improve the compression ratio, the pixel values ​​of the immediately preceding compressed group of the same color can be used to determine the predicted value PRE of the target pixel group.

[0082] For example, refer to Figure 4 and Figure 5 The representative pixel value REP of the second red pixel group PG(R2) is R2, and the predicted value PRE is R1, where R1 is the representative pixel value REP of the first red pixel group PG(R1) that is immediately preceding it in the compression. Therefore, compressor 120 can calculate D2, which is the difference DIFF between R2 (the representative pixel value REP of the second red pixel group PG(R2)) and R1 (the predicted value PRE), and compressor 120 can generate an n-bit bitstream based on D2 (the calculated difference DIFF). See later... Figure 6 Provides a detailed description of a method for generating an n-bit bitstream based on DIFF (Differential Interference).

[0083] When the target pixel group is the first pixel group in a group of pixels of the same color, the predicted value PRE of that target pixel group has a default value because there is no immediately preceding compressed group of pixels of the same color. The default value can be preset by the user or the manufacturer, but the embodiments are not limited to this, and different default values ​​can be used for each image data IDT.

[0084] In another embodiment, the predicted value PRE of the target pixel group can be the pixel value of a pixel at a fixed position in a previously compressed group of pixels of the same color. For example, the predicted value PRE of the target pixel group can be the bottom-right pixel in a group of pixels immediately preceding the target pixel group. (See reference...) Figure 4 and Figure 5The predicted value PRE of the second red pixel group PG(R2) can be the pixel value of the fourth pixel R14 of the first red pixel group PG(R1). The fixed position is not limited to the example above and can vary depending on the embodiment. For example, if the fixed position is within a group of pixels of the same color that has been compressed before the target pixel group, then that position is sufficient.

[0085] Compressor 120 can generate first compressed data CDT1 by compressing the representative pixel values ​​of all the multiple pixel groups PG that constitute the image data IDT according to the method described above. Therefore, for an image data IDT comprising m pixel groups PG (m is a positive integer), compressor 120 can generate first compressed data CDT1 comprising at least m×n bits.

[0086] According to a modified embodiment, the image data IDT may include YUV information. Even in this case, the compressor 120 can generate the first compressed data CDT1 by compressing the image data IDT in substantially the same manner as in the embodiments described above. As in the embodiments described above, for the image data IDT including YUV information, the compressor 120 can compress the representative pixel values ​​of each pixel group PG corresponding to the same reference color.

[0087] However, since the image data IDT, which includes YUV information, does not include information about the reference color, according to an embodiment, compressor 120 can sequentially compress the representative pixel values ​​of pixel group PG without distinguishing the reference color. For example, reference... Figure 5 The compressor 120 can compress representative pixel values ​​in the following order: a first pixel group including pixels R11, R12, R13 and R14; a second pixel group including pixels G11a, G12a, G13a and G14a; and a third pixel group including pixels R21, R22, R23 and R24.

[0088] Figure 6 This is a flowchart illustrating a method for generating first compressed data CDT1 according to an embodiment. Figure 6 This is a flowchart illustrating a method performed by compressor 120 to generate first compressed data CDT1 by compressing the representative pixel values ​​REP of pixel group PG.

[0089] refer to Figures 4 to 6 First, compressor 120 can calculate the difference DIFF by subtracting the predicted value PRE from the representative pixel value REP of pixel group PG (operation S210). See above for reference. Figure 5The method for determining the predicted value PRE is described, therefore redundant descriptions are omitted. Additionally, compressor 120 can check whether the difference DIFF is included within a predetermined range (operation S220). Here, the predetermined range refers to the range of integers that can be represented by k bits (k is a positive integer), including the sign bit. Furthermore, k is less than... Figure 5 The integer n in the example can be set to a different integer depending on the implementation.

[0090] When the difference DIFF is not included in the predetermined range (operation S220 - No), the compressor 120 may shift the difference DIFF to the right once (operation S230). Additionally, the compressor 120 may change the loss value (operation S240). Here, the loss value is a value corresponding to the number of times the difference DIFF is shifted to the right, and may initially have a default value. For example, the default value of the loss value may be set to 0. However, the embodiment is not limited to this, and the default value of the loss value may be set to a value other than 0 (e.g., 1). Each time a right shift is performed on the difference DIFF, the compressor 120 may increase the loss value. For example, each time a right shift is performed, the compressor 120 may increase the loss value by a value (e.g., 1). Alternatively, according to an embodiment, the compressor 120 may be implemented to decrease the loss value by that value (e.g., 1) each time a right shift is performed. The method of changing the loss value may be set differently by the manufacturer or the user. However, in the following description, for ease of explanation, it is assumed that the loss value is initially set to the value 0 and increases by 1 each time a right shift is performed.

[0091] In some embodiments, the loss value can be represented by nk bits. However, the embodiments are not limited to this; the loss value can be initially set to a value other than 0 (e.g., 1), and can be represented by a number of bits less than nk bits. Additionally, the compressor 120 can perform operation S220 again.

[0092] When the difference DIFF is within a predetermined range (operation S220 - Yes), compressor 120 can generate first compressed data CDT1 based on the difference DIFF and the loss value (operation S250). Compressor 120 can generate an n-bit bitstream including the difference DIFF and the loss value. The difference DIFF in this operation can be referred to as the reference value. (The reference will be discussed later.) Figure 7 Describe the configuration of an n-bit bitstream.

[0093] As described above, compressor 120 can generate compressed data that includes the difference DIFF between representative pixel values ​​REP, rather than compressed data that includes representative pixel values ​​REP of multiple pixel groups PG, thereby improving the compression ratio first. Furthermore, when the difference DIFF is large, compressor 120 can generate compressed data by shifting the difference DIFF to the right, thereby further improving the compression ratio.

[0094] Figure 7 This is a diagram illustrating the configuration of the first compressed data CDT1 according to an embodiment. Figure 7 This is a diagram showing the first compressed data CDT1 corresponding to a pixel group PG.

[0095] refer to Figure 7 The first compressed data CDT1 corresponding to a pixel group PG may include 8 bits, including 4 bits H1, H2, H3, and H4 constituting the HEADER, and 4 bits REF1, REF2, REF3, and REF4 constituting the REFERENCE. The HEADER may include the above references. Figure 6 The description includes the loss value LOSS, and according to an embodiment, in addition to the loss value LOSS, it may also include additional information about compression. Furthermore, the reference value REFERENCE represents the above reference... Figure 6 The operation S250 is described as the difference DIFF. That is, the reference value REFERENCE represents the difference DIFF within an integer range that can be represented using 4 bits, including the sign bit.

[0096] The number of bits constituting the HEADER and the number of bits constituting the REFERENCE are not limited to the examples above, and according to embodiments, the number of bits constituting the HEADER or REFERENCE can be set in various ways.

[0097] Figure 8 This is a diagram illustrating a method for calculating the loss value (LOSS) and reference value (REFERENCE) of the first compressed data CDT1 according to an embodiment. Figure 8 This shows the calculation. Figure 7 A graph showing the method for calculating the loss value (LOSS) and reference value (REFERENCE) of the first compressed data CDT1.

[0098] For example, refer to Figure 7 And refer to Figure 8 In the upper part, the representative pixel value REP of pixel group PG can be 10, and the predicted value PRE can be 9. Compressor 120 can calculate 1 as the difference DIFF between the representative pixel value REP and the predicted value PRE. Since 1 as the difference DIFF is within the range of integers that can be represented by 4 bits (including the sign bit) (i.e., equal to or greater than -8 and equal to or less than 7), the right shift of the difference DIFF can be omitted. Therefore, the loss value LOSS can be 0000 indicating the initial value, and the reference value REF can be determined as 0001 indicating 1. Compressor 120 can generate an 8-bit bitstream (00000001) including the loss value LOSS and the reference value REF.

[0099] As another example, see Figure 7 And refer to Figure 8 In the lower part, the representative pixel value REP of pixel group PG can be 10, and the predicted value PRE can be 1. Compressor 120 can calculate 9 as the difference DIFF between the representative pixel value REP and the predicted value PRE. Because 9 as the difference DIFF is not included in the range of integers that can be represented by 4 bits (including the sign bit) (i.e., equal to or greater than -8 and equal to or less than 7), compressor 120 can shift the difference DIFF to the right and increase the loss value LOSS by 1. Because the binary representation of 9 is 1001, the difference DIFF can be shifted to 0100, and the loss value LOSS can be changed from the binary representation 0000 to 0001. In addition, because the binary representation 0100 is the decimal representation 4, which is included in the range of integers, the additional right shift of the difference DIFF can be omitted. Therefore, the loss value LOSS can be 0001, and the reference value REF can be determined to be 0100. Compressor 120 can generate an 8-bit bitstream 00010100, including the loss value LOSS and the reference value REF.

[0100] Figure 9 This is a diagram illustrating a method for generating first compressed data CDT1 using representative pixel values ​​REP according to an embodiment. Figure 9 It is shown to be with Figure 5 The representative pixel value REP is calculated in the same way, but with... Figure 5 A diagram illustrating different methods for generating the first compressed data CDT1.

[0101] As referenced above Figure 5 The compressor 120 generates first compressed data CDT1 by compressing representative pixel values ​​of the same color pixel group. Furthermore, the compressor 120 can sequentially compress representative pixel values ​​of the same color pixel group. The representative pixel value can be determined as the average value of the pixel values ​​of the pixel group. The compressor 120 can calculate the difference DIFF between the representative pixel value REP of the target pixel group to be compressed and the predicted value PRE of the target pixel group. Here, the predicted value PRE of the target pixel group can be determined based on the pixel values ​​of the same color pixel group compressed before the target pixel group.

[0102] exist Figure 9 In some embodiments, the predicted value PRE of the target pixel group can be determined as the pixel value most similar to the pixel value REP among the pixel values ​​of the same color pixel group compressed before the target pixel group. In some embodiments, the predicted value PRE of the target pixel group can be determined as the pixel value most similar to the pixel value REP among the pixel values ​​of the same color pixel group immediately preceding the target pixel group.

[0103] For example, refer to Figure 9 The representative pixel value REP of the second red pixel group PG(R2) can be determined as R2, and the predicted value PRE can be determined as R13, where R13 is the most similar pixel value among the pixel values ​​of the first red pixel group PG(R1) that is immediately preceding it. Therefore, the compressor 120 can calculate the difference D2 (i.e., the difference DIFF) between R2 and R13, where R2 is the representative pixel value REP of the second red pixel group PG(R2), and R13 is the predicted value PRE.

[0104] exist Figure 9 In one embodiment, compressor 120 may additionally generate a flag indicating information about the relative position of pixels with a predicted value PRE. Additionally, compressor 120 may generate an n-bit bitstream based on the difference DIFF and the flag. For example, refer to... Figure 9 Compressor 120 can generate an n-bit bitstream based on D2 and F2, where D2 is the difference DIFF and F2 is a flag indicating information about the relative position of pixel R13. (See later...) Figure 10 Provide a detailed description of the method for generating a flag.

[0105] According to a modified embodiment, the image data IDT may include YUV information. Even in this case, the compressor 120 can determine the predicted value PRE of the target pixel group in substantially the same manner as in the embodiment described above. As in the embodiment described above, the compressor 120 can determine the predicted value PRE from among the pixel values ​​of the pixel group immediately preceding the compressed pixel group, the pixel value that corresponds to the same reference color as the target pixel group and is most similar to the representative pixel value REP.

[0106] However, since the image data IDT, which includes YUV information, does not include information about the reference color, according to an embodiment, the compressor 120 can determine the predicted value PRE from the pixel values ​​of the pixel group that is most similar to the representative pixel value REP among the pixel values ​​of the pixel group that is compressed immediately before the target pixel group, without distinguishing the reference color.

[0107] Figure 10 This is a diagram illustrating a method for calculating a flag for a group of pixels according to an embodiment. Figure 10 It shows based on Figure 9 A diagram illustrating a method for generating a flag in an embodiment.

[0108] A flag can indicate the relative position of a pixel with a predicted value PRE relative to a target pixel group. For example, when compressor 120 determines the predicted value PRE among pixel values ​​in a group of pixels of the same color that were compressed immediately preceding the target pixel group, the flag can have a value corresponding to the position of a pixel value in the same color pixel group that was compressed immediately preceding the target pixel group.

[0109] For example, refer to Figure 9 and Figure 10 When the target pixel group is the second red pixel group PG (R2), the flag can have a value corresponding to the position of one of the pixels R11, R12, R13, and R14 included in the first red pixel group PG (R1). When the predicted value PRE is the pixel value of pixel R11, the flag can have 00, indicating the position of the first pixel (e.g., the top-left pixel) of the previous pixel group. Additionally, when the predicted value PRE is the pixel value of pixel R12, the flag can have 01, indicating the position of the second pixel (e.g., the top-right pixel) of the previous pixel group. Furthermore, when the predicted value PRE is the pixel value of pixel R13, the flag can have 10, indicating the position of the third pixel (e.g., the bottom-left pixel) of the previous pixel group. Finally, when the predicted value PRE is the pixel value of pixel R14, the flag can have 11, indicating the position of the fourth pixel (e.g., the bottom-right pixel) of the previous pixel group.

[0110] Although Figure 9 and Figure 10 In the illustration, pixel array 111 is shown with a TETRA pattern, but the embodiment is not limited to this; pixel array 111 could be pixel array 115 with a NONA pattern. Even in this case, compressor 120 can determine the predicted value PRE of the target pixel group in substantially the same manner as in the embodiment described above, and generate a flag corresponding to the position of the pixel with the predicted value PRE. For example, compressor 120 can determine the predicted value PRE from nine pixel values ​​of the same color pixel group that is compressed immediately preceding the target pixel group, and generate a flag having a value indicating the position of the pixel with the predicted value PRE.

[0111] In addition, although in Figure 9 and Figure 10As shown, compressor 120 determines the prediction value PRE among the pixel values ​​of the same color pixel group that is compressed immediately preceding the target pixel group, but the embodiment is not limited to this. For example, it is sufficient if the prediction value PRE is determined among the same color pixel group that is compressed preceding the target pixel group. Alternatively, a sufficient number of bits can be allocated to the flag FLAG to indicate the relative position of the pixel with the prediction value PRE.

[0112] Figure 11 This is a diagram illustrating the configuration of the first compressed data CDT1 according to an embodiment. Figure 11 It shows the basis Figure 9 and Figure 10 A diagram of the first compressed data CDT1 corresponding to a pixel group PG in an embodiment.

[0113] refer to Figure 11 The first compressed data CDT1 corresponding to a pixel group PG may include 8 bits, including 3 bits H1, H2 and H3 constituting the HEADER, 3 bits REF1, REF2 and REF3 constituting the REFERENCE, and 2 bits FLG1 and FLG2 constituting the FLAG.

[0114] The header can include the loss value. Here, when... Figure 11 Loss value LOSS Figure 6 and Figure 7 When comparing the loss value (LOSS), since only the number of allocated bits differs, the loss value (LOSS) can be calculated in essentially the same way. Therefore, redundant descriptions of the method for calculating the loss value (LOSS) are omitted. According to an embodiment, in addition to the loss value (LOSS), the header (HEADER) may also include additional information about compression. Furthermore, when comparing the reference value (REFERENCE) with... Figure 6 and Figure 7 When comparing the reference value REFERENCE, since only the number of allocated bits differs, the reference value REFERENCE can be calculated in essentially the same way. Therefore, redundant descriptions of the method for calculating the reference value REFERENCE are omitted.

[0115] The number of bits constituting the HEADER, the number of bits constituting the REFERENCE, and the number of bits constituting the FLAG are not limited to the examples above, and the number of bits can be set in various ways according to the embodiments.

[0116] Figure 12 This is a diagram illustrating a method for generating second compressed data CDT2 using residual values ​​according to an embodiment. In the following description, for ease of description, it is assumed that pixel array 111a is an image... Figure 2AThe pixel array 113 is described with a TETRA pattern, but the embodiments are not limited thereto; even when the pixel array 111a is like... Figure 2B When pixel array 115 has the same NONA pattern, pixel array 111a can also be applied.

[0117] Compressor 120 can generate second compressed data CDT2 based on image data IDT and representative pixel values ​​of pixel groups. For each of the multiple pixel groups PG in the image data IDT, compressor 120 can calculate a residual value indicating the difference between the pixel value of a pixel included in the corresponding pixel group and the representative pixel value of the corresponding pixel group. Additionally, compressor 120 can generate second compressed data CDT2 based on the residual values ​​of the multiple pixel groups PG.

[0118] refer to Figure 12 The compressor 120 can calculate the differences between the pixel values ​​R21, R22, R23, and R24 of the second red pixel group PG(R2) and the representative pixel value R2. For example, the compressor 120 can calculate a first residual value r21 by subtracting the representative pixel value R2 from the first pixel value R21 of the second red pixel group PG(R2). Furthermore, the compressor 120 can calculate a second residual value r22 by subtracting the representative pixel value R2 from the second pixel value R22 of the second red pixel group PG(R2). Furthermore, the compressor 120 can calculate a third residual value r23 by subtracting the representative pixel value R2 from the third pixel value R23 of the second red pixel group PG(R2). Furthermore, the compressor 120 can calculate a fourth residual value r24 by subtracting the representative pixel value R2 from the fourth pixel value R24 of the second red pixel group PG(R2).

[0119] Furthermore, compressor 120 can generate an i-bit bitstream based on the residual values ​​r21, r22, r23, and r24 of the pixel values ​​R21, R22, R23, and R24 corresponding to the second red pixel group PG(R2), respectively. Additionally, compressor 120 can generate second compressed data CDT2, which includes the i-bit bitstream.

[0120] Compressor 120 can generate second compressed data CDT2 by calculating residual values ​​for all pixel groups PG constituting image data IDT according to the method described above. Therefore, for image data IDT comprising m pixel groups PG (m is a positive integer), compressor 120 can generate second compressed data CDT2 comprising at least m×i bits.

[0121] Figure 13 This is a diagram illustrating the configuration of the second compressed data CDT2 according to an embodiment. Figure 13 This is a diagram showing the second compressed data CDT2 corresponding to a pixel group PG.

[0122] refer to Figure 13 The second compressed data CDT2 corresponding to a pixel group PG may include 12 bits and may include four residual values ​​RESIDUAL included in the pixel group PG. Each residual value RESIDUAL may include 3 bits. The first residual value RES1 indicates the residual value of the first pixel of the pixel group PG, the second residual value RES2 indicates the residual value of the second pixel of the pixel group PG, the third residual value RES3 indicates the residual value of the third pixel of the pixel group PG, and the fourth residual value RES4 indicates the residual value of the fourth pixel of the pixel group PG.

[0123] Despite Figure 13 In this example, since pixel array 113 has a TETRA pattern, the second compressed data CDT2 is shown to include four residual values ​​(RESIDUAL), but the embodiment is not limited to this. For example, when image data IDT of pixel array 115 with a NONA pattern is compressed, the second compressed data CDT2 may include nine residual values ​​(RESIDUAL).

[0124] In addition, although in Figure 13 The diagram shows that each of the four residual values ​​RESIDUAL included in the second compressed data CDT comprises 3 bits, but the embodiments are not limited thereto, and the number of bits allocated to each residual value RESIDUAL can be the same or different from each other. For example, 3 bits can be allocated to the first residual value RES1, 4 bits can be allocated to the second residual value RES2, 2 bits can be allocated to the third residual value RES3, and 3 bits can be allocated to the fourth residual value RES4.

[0125] Figure 14 This is a diagram illustrating a bitstream including first compressed data CDT1 and second compressed data CDT2 according to an embodiment. Figure 14 It shows including Figure 7 The first compressed data CDT1 and Figure 13 The second compressed data, CDT2, is included in the bitstream BS.

[0126] refer to Figure 14 Compressor 120 can generate including Figure 7 The first compressed data CDT1 and Figure 13 The second compressed data, CDT2, is a 20-bit bitstream BS. However, the embodiment is not limited to this; the compressor 120 can generate a bitstream BS including... Figure 11 The first compressed data CDT1 and Figure 13 The second compressed data, CDT2, is a 20-bit bitstream. One bitstream BS can correspond to one pixel group PG.

[0127] When a pixel PX is represented as a 10-bit pixel value, a pixel group PG according to the image data IDT of the TETRA pattern can have a 40-bit pixel value. The first compressed data CDT1 and the second compressed data CDT2 can be represented as a total of 20 bits of pixel value, therefore, the first compressed data CDT1 and the second compressed data CDT2 can have a compression ratio of 50%.

[0128] Although Figure 14 The illustration shows a bitstream BS comprising 20 bits, but the embodiment is not limited to this. Depending on the number of bits in the first compressed data CDT1 and the second compressed data CDT2, a bitstream BS may include fewer or more than 20 bits. As the number of bits in the bitstream BS decreases, the compression ratio can be increased.

[0129] Figure 15 This is a diagram illustrating an image reading operation using a bitstream according to an embodiment. Figure 15 It shows the use Figure 14 A diagram of the image reading operation of the bitstream BS.

[0130] Compressor 120 can send a bitstream BS, including first compressed data CDT1 and second compressed data CDT2, to image processing device 200. Image processing device 200 can store the received bitstream BS in memory 220. In some embodiments, image processing device 200 can store bitstream BS in memory 220, but uses multiple addresses to manage the storage location of bitstream BS. Image processing device 200 can use a first address indicating the storage location of first compressed data CDT and a second address indicating the storage location of second compressed data CDT2 to manage the storage location of bitstream BS.

[0131] When a request for a low-resolution image, such as a preview image, is received, the image processing device 200 can read the first compressed data CDT1 from the memory 220. For example, refer to Figure 15 The image processing device 200 can read 8 bits (header and reference value) corresponding to the first compressed data CDT1 from the memory 220 based on the first address. However, it does not read 12 bits (residual value) corresponding to the second compressed data CDT2 from the memory 220.

[0132] When a request for a high-resolution image, such as a full-resolution image (or a captured image CAPTURE), is received, the image processing device 200 can read first compressed data CDT1 and second compressed data CDT2 from memory 220. For example, refer to Figure 15The image processing device 200 can read first compressed data CDT1 from memory 220 based on a first address, and read second compressed data CDT2 from memory 220 based on a second address.

[0133] As described above, regardless of the resolution of the requested image, the image processing device 200 can read the first compressed data CDT1, and when the resolution of the requested image is that of a high-resolution image, it can additionally read the second compressed data CDT2. That is, even if the received request changes from a low-resolution image request to a high-resolution image request, the image processing device 200 can generate a high-resolution image by only additionally reading the second compressed data CDT2. Therefore, in response to a request for a high-resolution image, the image processing device 200 can generate a high-resolution image at high speed.

[0134] Figure 16 This is a flowchart illustrating a method for generating a low-resolution image according to an embodiment. Figure 16 It is shown Figure 1 A flowchart of a method for an image processing system 10 to generate a low-resolution image (e.g., a preview image). Figure 16 Any one or any combination of operations can be performed by the decompressor 230 or the image signal processor 240 of the image processing device 200.

[0135] refer to Figure 1 , Figure 15 and Figure 16 The image processing system 10 can read the first compressed data CDT1 (operation S510). When a request for a low-resolution image is received, the image processing system 10 can read the first compressed data CDT1 from the memory 220.

[0136] Additionally, the image processing system 10 can check whether the loss value LOSS of the first compressed data CDT1 is a default value (operation S520). For example, the default value of the loss value can be set to 0. In this case, the image processing system 10 can check whether the bit corresponding to the loss value LOSS in the first compressed data CDT1 represents 0. However, the embodiment is not limited to this, and the default value of the loss value can be set to a value other than 0 (e.g., 1). If the loss value LOSS has a default value, the loss value LOSS indicates that the reference value REFERENCE does not need to be shifted left. Conversely, when the loss value LOSS does not have a default value, the loss value LOSS indicates that the reference value REFERENCE needs to be shifted left.

[0137] If the loss value LOSS is not the default value (operation S520 - No), the image processing system 10 can shift the reference value REFERENCE of the first compressed data CDT1 to the left once (operation S530). Alternatively, the image processing system 10 can change the loss value LOSS of the first compressed data CDT1 (operation S540). Each time the reference value REFERENCE is shifted to the left, the image processing system 10 can decrease the loss value LOSS. For example, the image processing system 10 can decrease the loss value LOSS by one value (e.g., 1) each time a left shift is performed. In this case, it is assumed that during the generation of the first compressed data CDT1, the method of increasing the loss value LOSS is applied whenever data loss occurs. Alternatively, according to an embodiment, the image processing system 10 can be implemented to increase the loss value LOSS by one value (e.g., 1) each time a left shift is performed. In this case, it is assumed that during the generation of the first compressed data CDT1, the method of decreasing the loss value LOSS is applied whenever data loss occurs. The method of changing the loss value LOSS can be set differently by the manufacturer or the user. However, for ease of explanation in the following text, it is assumed that the default value of the loss value LOSS is set to 0, and is decreased by 1 each time a left shift is performed. Thus, when the loss value LOSS does not have a default value, the image processing system 10 can shift the reference value REFERENCE to the left and gradually change the loss value LOSS to a value similar to the default value.

[0138] In addition, the image processing system 10 can perform operation S520 again.

[0139] When the loss value LOSS is the default value (operation S520 - Yes), the image processing system 10 can calculate the representative pixel value REP of the pixel group PG based on the reference value REFERENCE and the predicted value PRE of the first compressed data CDT1 (operation S550). The image processing system 10 can check the predicted value PRE of the target pixel group to be decompressed. Alternatively, the image processing system 10 can calculate the representative pixel value REP of the target pixel group by adding the predicted value PRE of the target pixel group to the reference value REP. Here, the predicted value PRE of the target pixel group to be decompressed can be determined based on the pixel values ​​of the pixel groups that were decompressed before the target pixel group. In addition, the predicted value PRE of the target pixel group can be determined in a manner corresponding to the method of generating the first compressed data CDT1.

[0140] For example, when according to Figure 5 When generating the first compressed data CDT1 in the embodiment, the predicted value PRE of the target pixel group to be decompressed can be determined as the representative pixel value of the same color pixel group that was decompressed immediately before the target pixel group. As another example, when according to Figure 9When generating the first compressed data CDT1 in the embodiment, the predicted value PRE of the target pixel group to be decompressed can be determined as the pixel value of the pixel identified by the flag FLAG. The method for calculating the predicted value PRE of the target pixel group is not limited to the example above, and various methods can be applied.

[0141] According to a modified embodiment, in operation S520, it can be checked whether the loss value LOSS has a predetermined value other than zero. Here, the predetermined value can be set to a value that indicates that a left shift operation is not required.

[0142] Additionally, the image processing system 10 can generate low-resolution image data LIDT based on the representative pixel value REP of the pixel group PG (operation S560). The image processing system 10 can generate low-resolution image data LIDT including the representative pixel value REP.

[0143] The image processing system 10 can provide the generated low-resolution image data LIDT to the display 300. Additionally, the display 300 can display a low-resolution image (e.g., a preview image) based on the received low-resolution image data LIDT.

[0144] Figure 17 This is a diagram illustrating the decompression operation using the first compressed data CDT1 according to an embodiment. Figure 17 It is shown that by using according to Figure 8 The first compressed data CDT1 generated by the embodiment is used for execution. Figure 16 The diagram shows an example of operation S520 to S550.

[0145] For example, refer to Figure 8 and Figure 17 Before compression, the representative pixel value REP of pixel group PG can be 10, and the predicted value PRE can be 9. Additionally, in the first compressed data CDT1, the loss value LOSS of pixel group PG can be 0000, and the reference value REF can be 0001. Because the loss value LOSS of image processing system 10 is 0, the left shift of the reference value REF 0001 can be omitted. The reference value REF when the loss value LOSS is 0 can be referred to as the difference DIFF, indicating the difference between the representative pixel value REP and the predicted value PRE of the target pixel group. Image processing system 10 can calculate 10 as the representative pixel value REP' by adding 1 as the difference DIFF and 9 as the predicted value PRE; this is the result of decompression.

[0146] As another example, see Figure 8 and Figure 17Before compression, the representative pixel value REP of pixel group PG can be 10, and the predicted value PRE can be 1. Additionally, in the first compressed data CDT1, the loss value LOSS of pixel group PG can be 0001, and the reference value REF can be 0100. Because the loss value LOSS is not 0, the image processing system 10 can shift the reference value REF to the left and decrease the loss value LOSS by 1. The reference value REF can be changed to 1000 by shifting it to the left, and the loss value LOSS can be changed from 0001 to 0000. Therefore, the difference DIFF can be represented as 1000 in binary and 8 in decimal. The image processing system 10 can calculate 9 as the representative pixel value REP' by adding the 8 as the difference DIFF and the 1 as the predicted value PRE; this is the result of decompression.

[0147] Figure 18 This is a flowchart illustrating a method for generating a high-resolution image according to an embodiment. Figure 18 It is shown Figure 1 A flowchart of a method for an image processing system 10 to generate high-resolution images (e.g., full-resolution images). Figure 18 Any one or any combination of operations can be performed by the decompressor 230 or the image signal processor 240 of the image processing device 200.

[0148] refer to Figure 16 and Figure 18 The image processing system 10 can read the first compressed data CDT1 and the second compressed data CDT2 (operation S610). When a request for a high-resolution image is received, the image processing system 10 can read the first compressed data CDT1 and the second compressed data CDT2 from the memory 220.

[0149] Additionally, the image processing system 10 can check whether the loss value LOSS of the first compressed data CDT1 is the default value (operation S620). When the loss value LOSS is not the default value (operation S620 - No), the image processing system 10 can shift the reference value REFERENCE of the first compressed data CDT1 one position to the left (operation S630). Alternatively, the image processing system 10 can change the loss value LOSS of the first compressed data CDT1 (operation S640). For example, when the method of increasing the loss value LOSS is applied whenever data loss occurs during the generation of the first compressed data CDT1, the image processing system 10 can decrease the loss value LOSS. Alternatively, when the method of decreasing the loss value LOSS is applied whenever data loss occurs during the generation of the first compressed data CDT1, the image processing system 10 can increase the loss value LOSS. When the loss value LOSS is the default value (operation S620 - Yes), the image processing system 10 can calculate the representative pixel value REP of the pixel group PG based on the reference value REFERENCE and the predicted value PRE of the first compressed data CDT1 (operation S650). Figure 18 Operations S620 to S650 can be respectively connected with Figure 17 The operations S520 to S550 are basically the same, so their detailed description is omitted.

[0150] Additionally, the image processing system 10 can calculate pixel values ​​based on the representative pixel value REP and the residual value RESIDUAL of the second compressed data CDT2 (operation S660). The image processing system 10 can calculate the pixel value of the target pixel group PG by adding the representative pixel value REP of the target pixel group PG to each residual value RESIDUAL of the target pixel group PG to be decompressed.

[0151] Additionally, the image processing system 10 can generate high-resolution image data HIDT based on the pixel values ​​of pixel group PG (operation S670). The image processing system 10 can generate high-resolution image data HIDT including the calculated pixel values.

[0152] The image processing system 10 can provide the generated high-resolution image data (HIDT) to the display 300. Additionally, the display 300 can display a high-resolution image (e.g., a full-resolution image) based on the received high-resolution image data (HIDT).

[0153] Figure 19 This is a diagram illustrating a camera module 100a according to an embodiment. Figure 19 It is shown as Figure 1 A figure of a camera module 100a of an image processing system 10a of a modifiable embodiment of an image processing system 10a.

[0154] refer to Figure 19 The image processing system 10a may include a camera module 100a and an image processing device 200a. In an embodiment, the camera module 100a may include an image sensor 110a, a compressor 120a, a memory 140a, and an interface (IF) 130a. When used with... Figure 1 Compared to the camera module 100, Figure 19 The camera module 100a may further include a memory 140a. Figure 19 The image sensor 110a, compressor 120a, and interface 130a can be respectively similar to Figure 1 The image sensor 110, compressor 120, and interface 130 operate together, therefore details related to the reference are omitted. Figure 1 The described content is a repetitive description.

[0155] Compressor 120a can generate first compressed data CDT1 and second compressed data CDT2, and store the generated first compressed data CDT1 and second compressed data CDT2 in memory 140a. When using a low-resolution image (e.g., a preview image), the first compressed data CDT1 can be read from memory 140a and output through interface 130a. Conversely, when using a full-resolution image, the first compressed data CDT1 and second compressed data CDT2 can be read from memory 140a and output through interface 130a.

[0156] That is, according to Figure 19 In one embodiment, the first compressed data CDT1 and the second compressed data CDT2 can be co-stored in the memory 140a, and can be selectively read from the memory 140a according to the type of image data.

[0157] Figure 20 This is a diagram illustrating an image processing apparatus 200a according to an embodiment. Figure 20 It is shown Figure 19 The image processing system 10a and the image processing device 200a are shown in the figure.

[0158] refer to Figure 19 and Figure 20 The image processing device 200a may include an interface (IF) 210a, a memory 220a, a decompressor 230a, and an image signal processor 240a. Figure 19 Image processing device 200a includes with Figure 1 The image processing device 200 uses the same components and can operate similarly, therefore the components described above are omitted. Figure 1 The description in the text is a repetitive description.

[0159] Image processing device 200a can be used from Figure 19The camera module 100a receives first compressed data CDT1 and second compressed data CDT2. According to this embodiment, upon receiving a request for a low-resolution image (e.g., a preview image), the image processing device 200a can request the first compressed data CDT1 from the camera module 100a. After receiving the first compressed data CDT1 via interface 210a, the image processing device 200a can decompress the first compressed data CDT1 via decompressor 230a to generate first decompressed data DDT1. Furthermore, the image processing device 200a can perform image processing on the first decompressed data DDT1 via image signal processor 240a to generate low-resolution image data LIDT. Additionally, the image processing device 200a can provide the low-resolution image data LIDT to the display 300.

[0160] Furthermore, upon receiving a request for a full-resolution image, the image processing device 200a can request first compressed data CDT1 and second compressed data CDT2 from the camera module 100a. Additionally, the image processing device 200a can receive the first compressed data CDT1 and second compressed data CDT2 via interface 210a. Furthermore, the image processing device 200a can decompress the first compressed data CDT1 and second compressed data CDT2 via decompressor 230a to generate second decompressed data DDT2. Furthermore, the image processing device 200a can perform image processing on the second decompressed data DDT2 via image signal processor 240a to generate high-resolution image data HIDT. Finally, the image processing device 200a can provide the high-resolution image data HIDT to the display 300.

[0161] That is, according to Figure 20 In one embodiment, the image processing device 200a can request compressed data in real time and receive the requested compressed data from the camera module 100a to generate low-resolution image data or high-resolution image data. In the above embodiment, the image processing device 200a can be configured to store first compressed data CDT1 and second compressed data CDT2 in the memory 220a.

[0162] Figure 21 This is a diagram illustrating a camera module 100b according to an embodiment. Figure 21 It is shown as Figure 1 A figure of the camera module 100b of the image processing system 10b of a modifiable embodiment of the image processing system 10.

[0163] refer to Figure 21The image processing system 10b may include a camera module 100b and an image processing device 200b. In an embodiment, the camera module 100b may include an image sensor 110b, a first compressor 150b, a second compressor 120b, and an interface (IF) 130b. Figure 1 Compared to the camera module 100, Figure 21 The camera module 100b may also include a first compressor 150b. Figure 21 The image sensor 110b, the second compressor 120b, and the interface 130b can be respectively similar to Figure 1 The image sensor 110, compressor 120, and interface 130 are used for operation, therefore, references are omitted. Figure 1 The description is a repetitive description.

[0164] In some embodiments, the pixel array 111b of the image sensor 110b may include a color filter with a DECAHEXA pattern. The color filter of the pixel array 111b may be configured such that a red pixel group comprising a 4×4 arrangement of red pixels R, a first green pixel group comprising a 4×4 arrangement of first green pixels G, a blue pixel group comprising a 4×4 arrangement of blue pixels B, and a second green pixel group comprising a 4×4 arrangement of green pixels G are repeatedly arranged. Additionally, the image sensor 110b may generate image data IDT with a DECAHEXA pattern.

[0165] The first compressor 150b can generate a second image data IDT2 with a TETRA pattern based on the image data IDT. In some embodiments, the first compressor 150b can be configured as described above. Figure 5 The method described in the text compresses image data in IDT. Figure 21 Each pixel group PG1 (pixel group PG1 consists of pixels arranged in a 4×4 pattern) is used to generate the second image data IDT2 of the TETRA pattern, which includes Figure 21 The pixel group PGI' (pixel group PG1' includes pixels arranged in a 2×2 pattern).

[0166] For each pixel group PG1 comprising 4×4 arranged pixels in the image data IDT, the first compressor 150b can calculate a representative pixel value (e.g., average or median) for each 2×2 arranged pixel subgroup within pixel group PG1, as the pixel value for each 2×2 arranged pixel in pixel group PG1'. In this case, the first compressor 150b can also calculate the residual value between the representative pixel value and the 2×2 arranged pixels. Furthermore, the first compressor 150b can generate a second image data IDT2 with the calculated representative pixel values ​​arranged in a 2×2 configuration. Moreover, the first compressor 150b can generate a third compressed data CDT3 based on the residual value. Additionally, the first compressor 150b can send the generated second image data IDT2 and third compressed data CDT3 to the second compressor 120b.

[0167] The second compressor 120b can generate first compressed data CDT1 and second compressed data CDT2 based on the second image data IDT2, and send the first compressed data CDT1, the second compressed data CDT2, and the third compressed data CDT3 received from the first compressor 150b to the image processing device 200b.

[0168] The decompressor 230b can generate low-resolution or high-resolution images based on the received first to third compressed data CDT1 to CDT3. For example, the decompressor 230b can generate a low-resolution image based on the first compressed data CDT1, a medium-resolution image with a higher resolution than the low-resolution image based on the first compressed data CDT1 and the second compressed data CDT2, and a high-resolution image based on the first to third compressed data CDT1 to CDT3. The method of generating a high-resolution image based on the third compressed data CDT3 executed by the decompressor 230b can be consistent with the above-mentioned reference. Figure 18 The methods of description are basically the same, so redundant descriptions are omitted.

[0169] refer to Figure 22 The image sensor 110b generates image data IDT of a DECAHEXA pattern, which is shown and described, but the embodiments are not limited thereto. For example, the image sensor 110b may generate image data IDT in which a red pixel group arranged in a 2n (n is a positive integer) × 2n arrangement, two green pixel groups arranged in a 2n × 2n arrangement, and a blue pixel group arranged in a 2n × 2n arrangement are repeatedly arranged.

[0170] Alternatively, image sensor 110b can generate image data IDT, in which a red pixel group arranged in a 3n×3n configuration (n is a positive integer), two green pixel groups arranged in a 3n×3n configuration, and a blue pixel group arranged in a 3n×3n configuration are repeatedly arranged. In this case, first compressor 150b can generate third compressed data CDT with NONA mode based on image data IDT using the method described above.

[0171] Figure 22 This is a diagram illustrating a camera module 100c according to an embodiment. Figure 22 It is shown as Figure 1 A figure of a camera module 100c of an image processing system 10c of a modifiable embodiment of an image processing system 10.

[0172] refer to Figure 22 The image processing system 10c may include a camera module 100c and an image processing device 200c. In an embodiment, the camera module 100c may include an image sensor 110c, a reordering module 160c, a compressor 120c, and an interface (IF) 130c. Figure 1 Compared to the camera module 100, Figure 22 The camera module 100c may also include a reordering module 160c. Figure 22 The image sensor 110c, compressor 120c, and interface 130c can be respectively similar to Figure 1 The image sensor 110, compressor 120, and interface 130 are used for operation, therefore, references are omitted. Figure 1 The description is a repetitive description.

[0173] In some embodiments, the pixel array 111b of the image sensor 110b may include the pixel array 111b described above. Figure 22 The color filter for the DECAHEXA pattern described herein. Additionally, the image sensor 110b can generate image data IDT of the DECAHEXA pattern.

[0174] The reordering module 160c can generate a second image data IDT2 with a TETRA pattern by rearranging the image data IDT of the DECAHEXA pattern. The reordering module 160c does not compress the image data IDT; it only performs repositioning. For example, the reordering module 160c can generate the second image data IDT2 by rearranging the image data IDT, which includes a 4×4 arrangement of red pixel groups, two green pixel groups, and a blue pixel group, into a TETRA pattern, in which a 2×2 arrangement of red pixel groups, two green pixel groups, and a blue pixel group is repeatedly arranged. The reordering module 160c can send the generated second image data IDT2 to the compressor 120c. The compressor 120c can generate a first compressed data CDTI and a second compressed data CDT2 based on the second image data IDT2. (Reference) Figure 23 The image sensor 110c generates image data IDT of a DECAHEXA pattern, which is shown and described, but the embodiments are not limited thereto. For example, the image sensor 110c can generate image data IDT in which a red pixel group arranged in a 2n (n is a positive integer) × 2n arrangement, two green pixel groups arranged in a 2n × 2n arrangement, and a blue pixel group arranged in a 2n × 2n arrangement are repeatedly arranged.

[0175] Alternatively, the image sensor 110c can generate image data IDT, in which a red pixel group arranged in a 3n×3n pattern, two green pixel groups arranged in a 3n×3n pattern, and a blue pixel group arranged in a 3n×3n pattern are repeatedly arranged. In this case, the reordering module 160c can generate a second image data IDT with a NONA pattern based on the image data IDT using the method described above.

[0176] Figure 23 This is a diagram illustrating an electronic device 1000 according to an embodiment.

[0177] refer to Figure 23 Electronic device 1000 may include camera module 1100, application processor (AP) 1200, display 1300, memory 1400, storage device 1500, user interface 1600 and wireless transceiver 1700. Figure 23 The camera module 1100 can correspond to Figure 1 Camera module 100 Figure 19 Camera module 100a, Figure 21 Camera module 100b or Figure 22 The camera module 100c. Figure 23 Application processor 1200 may include Figure 1 Image processing device 200 or Figure 19 Image processing device 200a. In Figure 1 , Figure 19 , Figure 21 and Figure 22 The descriptions that are repeated above have been omitted.

[0178] Application processor 1200 controls the overall operation of electronic device 1000 and can be configured as a system-on-a-chip (SoC) to drive applications, operating systems, etc. Application processor 1200 can receive first compressed data for generating a low-resolution image (e.g., a preview image) and second compressed data for generating a high-resolution image (e.g., a full-resolution image) from camera module 1100, generating low-resolution image data based on the first compressed data, or generating high-resolution image data based on the first and second compressed data. In some embodiments, application processor 1200 can store the first and second compressed data in memory 1400 or storage device 1500. In some embodiments, application processor 1200 can store both low-resolution image data and high-resolution image data in memory 1400 or storage device 1500.

[0179] The memory 1400 may store programs and / or data processed or executed by the application processor 1200. The storage device 1500 may be implemented as a non-volatile storage device such as NAND flash memory or resistive memory. For example, the storage device 1500 may be configured as a memory card (MMC, eMMC, SD, micro SD, etc.). The storage device 1500 may store data and / or programs for executing algorithms that control image processing operations of the application processor 1200, and the data and / or programs may be loaded into the memory 1400 when image processing operations are performed.

[0180] User interface 1600 can be implemented as various devices capable of receiving user input, such as a keyboard, keypad, touchpad, fingerprint sensor, microphone, etc. User interface 1600 can receive user input and provide signals corresponding to the received user input to application processor 1200. Wireless transceiver 1700 may include modem 1710, transceiver 1720, and antenna 1730.

[0181] Figure 24 This is a diagram illustrating a portion of an electronic device 2000 according to an embodiment. Figure 25 This is a diagram showing the detailed configuration of the camera module 2100b according to an embodiment. Figure 24 It is shown as Figure 23 A diagram of electronic equipment 1000, which is a part of electronic equipment 2000. Figure 25 It is shown Figure 24 A diagram showing the detailed configuration of the 2100b camera module.

[0182] refer to Figure 24 The electronic device 2000 may include a multi-camera module 2100, an application processor (AP) 2200, and a memory 2300. The memory 2300 can perform operations related to… Figure 23 The memory 1400 shown has the same function, so redundant descriptions are omitted.

[0183] Electronic device 2000 can capture and / or store images of objects using a CMOS image sensor, and can be implemented as a mobile phone, tablet computer, or portable electronic device. Portable electronic devices can include laptop computers, mobile phones, smartphones, tablet PCs, wearable devices, etc.

[0184] The multi-camera module 2100 may include a first camera module 2100a, a second camera module 2100b, and a third camera module 2100c. The multi-camera module 2100 can perform operations related to... Figure 1 Camera module 100 Figure 19 Camera module 100a, Figure 21 Camera module 100b or Figure 22 The camera module 100c has the same function. Although in Figure 24 The multi-camera module 2100 is shown as including three camera modules 1100a to 1100c, but the embodiments are not limited thereto, and the multi-camera module 2100 may include a variety of numbers of camera modules.

[0185] In the following text, reference will be made to Figure 25 The detailed configuration of camera module 2100b is described in more detail, but the following description can be applied equally to other camera modules 2100a and 2100c according to the embodiments.

[0186] refer to Figure 25 The second camera module 2100b may include a prism 2105, an optical path folding element (hereinafter referred to as "OPFE") 2110, an actuator 2130, an image sensing device 2140, and a storage device 2150.

[0187] The prism 2105 may include a reflective surface 2107 of a light-reflecting material to change the path of light L incident from the outside.

[0188] According to the example embodiment, prism 2105 can change the path of light L incident in the first direction X to a second direction Y perpendicular to the first direction X. Furthermore, prism 2105 can rotate the reflective surface 2107 of the light-reflecting material about the central axis 1106 in direction A or direction B, thereby changing the path of light L incident in the first direction X to a second direction Y perpendicular to the first direction X. At this time, OPFE 2110 can also move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0189] In the example embodiment, the maximum rotatable angle of prism 2105 in direction A may be less than or equal to 15 degrees in the positive (+)A direction and greater than 15 degrees in the negative (-)A direction. However, the embodiment is not limited to this.

[0190] In the example embodiment, prism 2105 can rotate approximately 20 degrees, between 10 and 20 degrees, or between 15 and 20 degrees in the positive (+) or negative (-) B direction. Here, prism 2105 can rotate by the same angle or by similar angles differing from each other by approximately 1 degree in the positive (+) B and negative (-) B directions.

[0191] In an example embodiment, the prism 2105 can move the reflective surface 2106 of the light-reflecting material in a third direction (e.g., the Z direction) parallel to the direction in which the central axis 2106 extends.

[0192] For example, OPFE 2110 may include m (where m is a natural number) optical lenses. The m lenses can move in the second direction Y and change the optical zoom ratio of camera module 2100b. For example, when the basic optical zoom ratio of camera module 2100b is Z and the m optical lenses included in OPFE 2110 move, the optical zoom ratio of camera module 2100b can be changed to 3Z, 5Z, or an optical zoom ratio higher than 5Z.

[0193] Actuator 2130 can move OPFE 2110 or optical lens (hereinafter referred to as optical lens) to a position. For example, actuator 2130 can adjust the position of optical lens so that image sensor 2142 is located at the focal length of optical lens for accurate sensing.

[0194] Image sensing device 2140 may include image sensor 2142, control logic 2144, compressor 2145, and memory 2146. Image sensor 2142 can use light L provided through an optical lens to sense an image of the target. Figure 25 The image sensor 2142 can be functionally similar to Figure 1 Image sensor 110 or Figure 19The image sensor 110a is used, therefore redundant descriptions are omitted. Control logic 2144 can control the overall operation of the second camera module 2100b. For example, control logic 2144 can control the operation of the second camera module 2100b based on control signals provided via control signal line CSLb.

[0195] Compressor 2145 can compress the sensed image data. Figure 25 The compressor 2145 can perform with Figure 1 Compressor 120, Figure 19 Compressor 120a, Figure 21 Compressor 120b Figure 22 The compressor 2145 functions similarly to the compressor 120c, so redundant descriptions are omitted. For ease of description, the compressor 2145 is illustrated as a separate functional unit distinct from the other functional units, but it is not limited thereto and may be included in the control logic 2144 for compressing and encoding image data.

[0196] The memory 2146 may store information for the operation of the second camera module 2100b, such as calibration data 2147. Calibration data 2147 may include information for the second camera module 2100b to generate image data using light L provided externally. Calibration data 2147 may include, for example, information about the aforementioned rotation angle, information about the focal length, information about the optical axis, etc. When the second camera module 2100b is implemented as a multi-state camera in which the focal length changes according to the position of the optical lens, calibration data 2147 may include focal length values ​​for each position (or state) of the optical lens, and information related to autofocus.

[0197] Storage device 2150 can store image data sensed by image sensor 2142. Storage device 2150 can be disposed externally to image sensing device 2140 and can be stacked together with the sensor chip constituting image sensing device 2140. In an example embodiment, storage device 2150 can be implemented using EEPROM, but the embodiment is not limited thereto.

[0198] Let's refer to each other. Figure 24 and Figure 25 In the example embodiment, among the plurality of camera modules 2100a, 2100b, and 2100c, one camera module (e.g., the first camera module 2100a) may include four sub-pixels (i.e., TETRA units) that are adjacent to each other and share the same color information in a color pixel, and another camera module (e.g., the second camera module 2100b) may include nine sub-pixels (i.e., NONA units) that are adjacent to each other and share the same color information in a color pixel. However, the embodiments are not limited thereto.

[0199] In the example embodiment, camera modules 2100a, 2100b, and 2100c may each include an actuator 2130. Therefore, camera modules 2100a, 2100b, and 2100c may include the same or different calibration data 2147 depending on the operation of the included actuator 2130.

[0200] In the example embodiment, among camera modules 2100a, 2100b, and 2100c, one camera module (e.g., the second camera module 2100b) may be a folding lens type camera module including the prism 2105 and OPFE 2110 as described above, while the other camera modules (e.g., 2100a and 2100c) may be vertical camera modules without the prism 2105 and OPFE 2110. However, the embodiments are not limited to this.

[0201] In an example embodiment, for instance, one of the camera modules 2100a, 2100b, and 2100c (e.g., the third camera module 2100c) may be a vertical depth camera that extracts depth information using infrared (IR). In this case, the AP 2200 can generate a 3D depth image by merging image data provided from such a depth camera with image data provided from another camera module (e.g., the first camera module 2100a or the second camera module 2100b).

[0202] In the example embodiment, at least two camera modules among camera modules 2100a, 2100b, and 2100c (e.g., the first camera module 2100a and the second camera module 2100b) may have different fields of view (FOV). In this case, for example, at least two camera modules among camera modules 2100a, 2100b, and 2100c (e.g., the first camera module 2100a and the second camera module 2100b) may have different optical lenses, but the embodiment is not limited to this. For example, the first camera module 2100a among camera modules 2100a, 2100b, and 2100c may have a smaller FOV than the second camera module 2100b and the third camera module 2100c. However, the embodiment is not limited to this, and the multi-camera module 2100 may further include camera modules with a larger FOV than the originally used camera modules 2100a, 2100b, and 2100c.

[0203] Furthermore, in some embodiments, camera modules 2100a, 2100b, and 2100c may have different FOVs from each other. In this case, the optical lenses included in camera modules 2100a, 2100b, and 2100c may also be different from each other, but the embodiments are not limited thereto.

[0204] In some embodiments, camera modules 2100a, 2100b, and 2100c may be physically separated from each other. In other words, camera modules 2100a, 2100b, and 2100c do not divide and use the sensing area of ​​a single image sensor 2142. Instead, an independent image sensor 2142 may be disposed within each of the camera modules 2100a, 2100b, and 2100c.

[0205] AP 2200 may include multiple subprocessors 2210a, 2210b, and 2210c, a camera module controller 2230, a memory controller 2240, and internal memory 2250. AP 2200 may be implemented separately from camera modules 2100a, 2100b, and 2100c. For example, AP 2200 and camera modules 2100a, 2100b, and 2100c may be implemented as separate semiconductor chips.

[0206] Image data generated by camera modules 2100a, 2100b, and 2100c can be provided to corresponding subprocessors 2210a, 2210b, and 2210c via separate image signal lines ISLa, ISLb, and ISLc, respectively. For example, image data generated from the first camera module 2100a can be provided to the first subprocessor 2210a via the first image signal line ISLa, image data generated from the second camera module 2100b can be provided to the second subprocessor 2210b via the second image signal line ISLb, and image data generated from the third camera module 2100c can be provided to the third subprocessor 2210c via the third image signal line ISLc. Image data transmission can be performed using a MIPI-based camera serial interface, but the embodiments are not limited to this.

[0207] In an example embodiment, a sub-processor may be provided to correspond to multiple camera modules. For example, the first sub-processor 2210a and the third sub-processor 2210c may be implemented as a single sub-processor instead of being implemented separately, and image data provided from the first camera module 2100a and the third camera module 2100c may be selected by a selection element (e.g., a multiplexer) and provided to the integrated sub-image processor.

[0208] Each of the subprocessors 2210a, 2210b and 2210c may include Figure 1 decompressor 230 Figure 19 decompressor 230a or Figure 21The decompressor 230b. Subprocessors 2210a, 2210b, and 2210c can decompress the received compressed data to generate decompressed data, and output the generated decompressed data to the image generator 2220. The image generator 2220 can correspond to Figure 1 Image signal processor 240 or Figure 19 The image signal processor 240a. That is, the image generator 2220 can generate low-resolution image data or high-resolution image data based on decompressed data.

[0209] The camera module controller 2230 can provide control signals to each of the camera modules 2100a, 2100b, and 2100c. The control signals generated from the camera module controller 2230 can be provided to the corresponding camera modules 2100a, 2100b, and 2100c via separate control signal lines CSLa, CSLb, and CSLc.

[0210] Figure 26 This is a diagram illustrating an electronic device 1000a according to an embodiment. In detail, Figure 26 It is shown Figure 23 A diagram of a modifiable embodiment of the electronic device 1000.

[0211] refer to Figure 26 The electronic device 1000a may include a camera module 1100a, an application processor (AP) 1200a, a display 1300a, a memory 1400a, a storage device 1500a, a user interface 1600a, a wireless transceiver 1700a, and a compression module 1800a. The wireless transceiver 1700a may include a modem 1710a, a transceiver 1720a, and an antenna 1730a. Figure 26 The display 1300a, memory 1400a, storage device 1500a, user interface 1600a, and wireless transceiver 1700a can respectively correspond to Figure 23 The display 1300, memory 1400, storage device 1500, user interface 1600, and wireless transceiver 1700 are included.

[0212] In Figure 23 Electronic devices 1000 and Figure 26 When comparing it with the 1000a electronic device, the difference is that... Figure 26 The electronic device 1000a includes an image sensor for photographing the object and a compressor for compressing the image data generated by the image sensor, as separate configurations.

[0213] Figure 23 The camera module 1100 can capture images of objects to generate image data using an internal compressor (e.g., ...). Figure 25The compressor 2145 generates compressed data based on the image data to generate images with different resolutions, and sends the generated compressed data to the application processor 1200.

[0214] Figure 26 The camera module 1100a can capture images of an object to generate image data, and can send the generated image data to the compression module 1800a. The compression module 1800a may include a compressor, and can be used to generate compressed data based on the received image data for generating images with different resolutions. For example, the compression module 1800a may include any one or any combination of the following: Figure 1 Compressor 120, Figure 19 Compressor 120a, Figure 21 First compressor 150b and second compressor 120b Figure 22 The reordering module 160c and compressor 120c. (Omitted from the above reference) Figure 1 , Figure 19 , Figure 21 and Figure 22 The description is a repetitive description.

[0215] Additionally, the compression module 1800a can send the generated compressed data to the application processor 1200a. The application processor 1200a may include a decompressor, and can use the decompressor to generate decompressed data based on the received compressed data. For example, the application processor 1200a may include... Figure 1 decompressor 230 Figure 20 The decompressor 230a and Figure 21 Any one or any combination of the decompressors 230b. (Omitted from the above reference) Figure 1 , Figure 20 and Figure 21 The description is redundant. Additionally, the application processor 1200a can generate low-resolution or high-resolution image data based on the decompressed data.

[0216] However, the embodiments are not limited thereto, and according to the embodiments, the compression module 1800a may be implemented to include not only a compressor but also a decompressor. For example, the compression module 1800a may be implemented to further include Figure 1 decompressor 230 Figure 20 The decompressor 230a and Figure 21The compression module 1800a can use any one or any combination of the decompressors 230b. In this case, the compression module 1800a can use the decompressor to decompress the compressed data to generate decompressed data, and send the generated decompressed data to the application processor 1200a. The application processor 1200a can generate low-resolution image data or high-resolution image data based on the decompressed data.

[0217] In this embodiment, the camera module 1100a, the compression module 1800a, and the application processor 1200a can be implemented as separate semiconductor chips.

[0218] Figure 27 This is a diagram illustrating an electronic device 1000b according to an embodiment. Specifically, Figure 27 It is shown Figure 26 A diagram of a modified embodiment of the electronic device 1000a.

[0219] refer to Figure 27 The electronic device 1000b may include a camera module 1100b, an application processor (AP) 1200b, a display 1300b, a memory 1400b, a storage device 1500b, a user interface 1600b, and a wireless transceiver 1700b. Additionally, the application processor 1200b may include a compression module 1800b. The wireless transceiver 1700b may include a modem 1710b, a transceiver 1720b, and an antenna 1730b. Figure 27 The camera module 1100b, display 1300b, memory 1400b, storage device 1500b, user interface 1600b, and wireless transceiver 1700b can respectively correspond to camera module 1100a, display 1300a, memory 1400a, storage device 1500a, user interface 1600a, and wireless transceiver 1700a.

[0220] In Figure 26 Electronic equipment 1000a and Figure 27 When compared to the 1000b electronic device, Figure 27 The difference between the electronic device 1000b and the other device is that the compression module 1800b is included in the application processor 1200b.

[0221] Figure 27 The camera module 1100b can capture images of objects to generate image data, and can send the generated image data to the application processor 1200b. Additionally, the application processor 1200b can use the compression module 1800b to generate compressed data based on the received image data for generating images with different resolutions. The compression module 1800b can correspond to... Figure 26 The compression module 1800a may include a compressor. Figure 26 In the text, descriptions that are repeated above have been omitted.

[0222] The compression module 1800b may include not only a compressor but also a decompressor that generates decompressed data based on the compressed data. In this case, the application processor 1200b can use the compression module 1800b to decompress the compressed data to generate decompressed data. Additionally, the application processor 1200b can generate low-resolution or high-resolution image data based on the decompressed data. (The reference above is omitted.) Figure 1 , Figure 20 and Figure 21 The description is a repetitive description.

[0223] In this embodiment, the camera module 1100b and the application processor 1200b can be implemented as separate semiconductor chips.

[0224] Although the inventive concept has been shown and described with reference to embodiments thereof, it should be understood that various changes in form and detail may be made without departing from the spirit and scope of the appended claims.

Claims

1. A camera module comprising: an image sensor configured to generate image data including a plurality of pixels; a compressor configured to: divide the plurality of pixels included in the generated image data into a plurality of pixel groups; for each of the plurality of pixel groups into which the plurality of pixels is divided, determine a representative pixel value of the corresponding pixel group based on pixel values of the plurality of pixels included in the corresponding pixel group; generate first compressed data based on the determined representative pixel value of each of the plurality of pixel groups; for each of the plurality of pixel groups into which the plurality of pixels is divided, calculate a residual value indicating a difference between the pixel values of the plurality of pixels included in the corresponding pixel group and the representative pixel value of the corresponding pixel group; and generate second compressed data based on the calculated residual value of each of the plurality of pixel groups; and an interface configured to output the generated first compressed data and the generated second compressed data, wherein the representative pixel value of the corresponding pixel group includes an average value of the pixel values of the plurality of pixels included in the corresponding pixel group, a median value of the pixel values of the plurality of pixels included in the corresponding pixel group, or a pixel value of a pixel at a fixed position in the corresponding pixel group. The compressor is further configured to: divide the plurality of pixels into the plurality of pixel groups, the plurality of pixel groups being disposed adjacent to each other and respectively including pixels having the same color information; 2. The camera module of claim 1, wherein, and for each of the plurality of pixel groups into which the plurality of pixels is divided: calculate an average value of the pixel values of the plurality of pixels included in the corresponding pixel group; and determine the calculated average value as the representative pixel value of the corresponding pixel group. The compressor is further configured to: for each of the plurality of pixel groups into which the plurality of pixels is divided, determine a prediction value of the corresponding pixel group based on any one or any combination of the pixel values of the pixels included in the pixel groups compressed before the corresponding pixel group; and generate the first compressed data based on the target pixel value and the determined prediction value of each of the plurality of pixel groups. The compressor is further configured to, for each of the plurality of pixel groups into which the plurality of pixels is divided, determine the representative pixel value of the pixel groups compressed before the corresponding pixel group as the prediction value of the corresponding pixel group.

3. The camera module of claim 1, wherein, The compressor is further configured to: for each of the plurality of pixel groups into which the plurality of pixels is divided: calculate a difference value indicating a difference between the prediction value of the corresponding pixel group and the target pixel value; convert the calculated difference value into a reference value represented in a predetermined number of bits; and 4. The camera module of claim 3, wherein, calculate a loss value indicating a degree of data loss due to the conversion of the difference value into the reference value; and 5. The camera module of claim 3, wherein, generate the first compressed data including the calculated loss value of each of the plurality of pixel groups and the reference value of each of the plurality of pixel groups. The compressor is further configured to, for each of the plurality of pixel groups into which the plurality of pixels is divided: check whether the calculated difference value is included in an integer range represented in the predetermined number of bits; ​ ​ ​ ​ ​ 6. The camera module of claim 5, wherein, ​ ​ generating first compression data based on the checking that the difference value is included in the integer range; and based on the checking that the difference value is not included in the integer range, right-shifting the difference value and increasing the calculated loss value.

7. The camera module of claim 3, wherein, The compressor is further configured to, for each of the plurality of pixel groups into which the plurality of pixels is divided: among the pixel values of the pixel groups compressed before the corresponding pixel group, checking a pixel value most similar to the representative pixel value of the corresponding pixel group; and determining the checked pixel value as a prediction value of the corresponding pixel group.

8. The camera module of claim 3, wherein, The compressor is further configured to: for each of the plurality of pixel groups into which the plurality of pixels is divided: generating a flag indicating a relative position between the corresponding pixel group and a pixel having the prediction value; calculating a difference value indicating a difference between the prediction value of the corresponding pixel group and a target pixel value; converting the calculated difference value into a reference value represented by a predetermined number of bits; and calculating a loss value indicating a degree of data loss due to the conversion of the difference value into the reference value; and generating first compression data including the calculated loss value of each of the plurality of pixel groups, the generated flag of each of the plurality of pixel groups, and the reference value of each of the plurality of pixel groups.

9. The camera module of claim 1, wherein, The interface is further configured to: based on receiving a request for an image having a first resolution, output the generated first compression data; and based on receiving a request for an image having a second resolution higher than the first resolution, output the generated first compression data and the generated second compression data.

10. The camera module of claim 1, further comprising: a memory storing the generated first compression data and the generated second compression data, wherein the interface is further configured to: based on receiving a request for an image having a first resolution, read the stored first compression data from the memory and output the read first compression data; and based on receiving a request for an image having a second resolution higher than the first resolution, read the stored first compression data and the stored second compression data from the memory and output the read first compression data and the read second compression data.

11. An image processing apparatus comprising: an interface configured to receive compression data obtained by compressing image data including a plurality of pixel groups in units of pixel groups; a memory storing the received compression data; a decompressor configured to decompress at least a part of the stored compression data to generate decompressed data; and an image signal processor configured to perform image processing based on the generated decompressed data to generate reconstructed image data, wherein the reconstructed image data includes first image data having a first resolution and second image data having a second resolution, and wherein the compression data includes first compression data for generating the first image data and the second image data, and second compression data for generating the second image data, The first compressed data is based on a representative pixel value of each of the plurality of pixel groups, the representative pixel value being determined based on pixel values of a plurality of pixels included in the corresponding pixel group, and including an average of the pixel values of the plurality of pixels included in the corresponding pixel group, a median of the pixel values of the plurality of pixels included in the corresponding pixel group, or a pixel value of a pixel at a fixed position in the corresponding pixel group.

12. The image processing device according to claim 11, wherein The decompressor is further configured to, based on receiving a request for the first image data, decompress the first compressed data among the stored compressed data to generate first decompressed data, and The image signal processor is further configured to, based on the generated first decompressed data, perform image processing to generate the first image data.

13. The image processing device according to claim 12, wherein The first compressed data includes a loss value indicating a degree of data loss due to compression of the representative pixel value of each of the plurality of pixel groups, and a reference value as a result of compression of the representative pixel value of each of the plurality of pixel groups, and The decompressor is further configured to: calculate the representative pixel value of each of the plurality of pixel groups based on the loss value and the reference value of each of the plurality of pixel groups, and generate the first decompressed data including the calculated representative pixel value of each of the plurality of pixel groups.

14. The image processing device according to claim 13, wherein The decompressor is further configured to, based on receiving a request for the second image data, decompress the first compressed data and the second compressed data among the stored compressed data to generate second decompressed data, and The image signal processor is further configured to, based on the generated second decompressed data, perform image processing to generate the second image data.

15. The image processing device according to claim 14, wherein For each of the plurality of pixel groups, the second compressed data includes a residual value representing a difference between a pixel value of a pixel included in the corresponding pixel group and the representative pixel value of the corresponding pixel group, and The decompressor is further configured to: calculate the pixel value of the pixel included in each of the plurality of pixel groups based on the calculated representative pixel value and the calculated residual value of each of the plurality of pixel groups, and generate the second decompressed data including the calculated pixel value of each of the plurality of pixel groups.

16. An image compression method for image data including a plurality of pixel groups, the image compression method comprising: For each of the plurality of pixel groups, determining a representative pixel value of the corresponding pixel group based on pixel values of a plurality of pixels included in the corresponding pixel group; generating first compressed data based on the determined representative pixel value of each of the plurality of pixel groups; For each of the plurality of pixel groups, calculating a residual value representing a difference between a pixel value of a pixel included in the corresponding pixel group and the representative pixel value of the corresponding pixel group; and generating second compressed data based on the calculated residual value of each of the plurality of pixel groups, The representative pixel value of the corresponding pixel group includes an average of pixel values of the plurality of pixels included in the corresponding pixel group, a median of pixel values of the plurality of pixels included in the corresponding pixel group, or a pixel value of a pixel at a fixed position in the corresponding pixel group.

17. The image compression method of claim 16, wherein, Determining the representative pixel value for each of the plurality of pixel groups includes: calculating an average of pixel values of the plurality of pixels included in the corresponding pixel group; and determining the calculated average as the representative pixel value of the corresponding pixel group.

18. The image compression method of claim 16, wherein, Generating the first compressed data includes: determining, for each of the plurality of pixel groups, a prediction value of the corresponding pixel group based on any one or any combination of pixel values of pixels included in pixel groups compressed before the corresponding pixel group; and generating the first compressed data based on the target pixel value and the determined prediction value of each of the plurality of pixel groups.

19. The image compression method of claim 18, wherein, Determining the prediction value includes, for each of the plurality of pixel groups, determining a representative pixel value of a pixel group compressed before the corresponding pixel group as the prediction value of the corresponding pixel group.

20. The image compression method of claim 18, wherein, Generating the first compressed data further includes: for each of the plurality of pixel groups: calculating a difference value indicating a difference between the prediction value of the corresponding pixel group and the target pixel value; converting the calculated difference value into a reference value represented by a predetermined number of bits; and calculating a loss value indicating a degree of data loss due to the conversion of the difference value into the reference value; and generating the first compressed data including the calculated loss value of each of the plurality of pixel groups and the reference value of each of the plurality of pixel groups.

Images