Image compression method, encoder, and camera module including the encoder

Abstract

An image compression method, an encoder, and a camera module including the encoder are provided. The image compression method includes: classifying image data corresponding to a plurality of sub-pixels that are arranged adjacent to each other and generate first color information as first color pixels; determining a reference value as a standard for compressing the image data based on pixel values of the plurality of sub-pixels; comparing the pixel values of the plurality of sub-pixels with the reference value; and outputting a comparison result and the reference value.

Description

[0001] This application claims priority to Korean Patent Application No. 10-2020-0052891, filed on Apr. 29, 2020, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The inventive concept relates to image compression, and more particularly, to an image compression method, an encoder for compressing an image, and a camera module including the encoder for compressing an image. Background Art

[0003] As dual cameras are installed in various electronic devices, or the number of images that can be captured per second increases, and the demand for high-resolution images increases, the size of image data stored in electronic devices is increasing.

[0004] To reduce the size of image data, high spatial similarity may be utilized to compress data based on the difference between a target pixel to be compressed and a reference pixel. Summary of the Invention

[0005] The present disclosure provides an image compression method, an encoder, and an image capturing device for compressing image data with reference to pixels having high spatial similarity.

[0006] According to an exemplary embodiment of the present disclosure, there is provided an image compression method for compressing image data generated through a pixel array, the method including: classifying image data corresponding to a plurality of sub-pixels that are adjacent to each other and generate first color information as first color pixels; determining a reference value as a standard for compressing the image data based on pixel values of the plurality of sub-pixels; comparing the pixel values of the plurality of sub-pixels with the reference value; and outputting a comparison result and the reference value.

[0007] According to an exemplary embodiment of the present disclosure, an encoder is configured to: classify image data corresponding to a plurality of sub-pixels that are adjacent to each other and generate first color information as first color pixels; determine a reference value as a standard for compressing the image data based on pixel values of the plurality of sub-pixels; compare the pixel values of the plurality of sub-pixels with the reference value; and output a comparison result and the reference value.

[0008] According to an exemplary embodiment of the present disclosure, an image sensor is provided. The image sensor includes a pixel array having a plurality of pixels arranged in a matrix shape and generating pixel signals respectively. The image sensor is configured to: convert light into an electrical signal using a photoelectric conversion element and output image data; an encoder is configured to: output a bitstream generated by compressing the image data, wherein the encoder classifies a plurality of sub-pixels that are adjacent to each other and generate first color information as first color pixels, determines a reference value as a standard for compressing the image data based on pixel values of the plurality of sub-pixels, compares the pixel values of the plurality of sub-pixels with the reference value, and outputs a comparison result and the reference value. Description of the Drawings

[0009] Embodiments of the present disclosure will be more clearly understood from the following detailed description in conjunction with the accompanying drawings, wherein:

[0010] Figure 1 is a flowchart of an image compression method according to an exemplary embodiment;

[0011] Figure 2 is a block diagram showing an image capturing device according to an exemplary embodiment;

[0012] Figure 3 is a schematic diagram of a unit in which color pixels including a plurality of sub-pixels are arranged in a Bayer pattern according to an exemplary embodiment;

[0013] Figure 4A and Figure 4B is a schematic diagram showing a pixel array according to an exemplary embodiment;

[0014] Figure 5A 、 Figure 5B and Figure 5C is a hybrid schematic diagram showing a process for determining a reference value according to an exemplary embodiment;

[0015] Figure 6A 、 Figure 6B and Figure 6C is a hybrid schematic diagram showing another process for determining a reference value according to an exemplary embodiment;

[0016] Figure 7A and Figure 7B is a hybrid schematic diagram showing a configuration of a bitstream according to an exemplary embodiment;

[0017] Figure 8 is a hybrid schematic diagram showing another configuration of a bitstream according to an exemplary embodiment;

[0018] Figure 9 is a schematic diagram showing a pixel array according to an exemplary embodiment;

[0019] Figure 10 and Figure 11 is a flowchart of an image compression method according to an exemplary embodiment;

[0020] Figure 12A and Figure 12B is a block diagram showing a camera module according to an exemplary embodiment;

[0021] Figure 13 is a block diagram showing an application processor (AP) according to an exemplary embodiment;

[0022] Figure 14 is a flowchart of an image compression method according to an exemplary embodiment;

[0023] Figure 15A and Figure 15B is a block diagram showing an electronic device according to an exemplary embodiment;

[0024] Figure 16 is a view showing a part of an electronic device according to an exemplary embodiment;

[0025] Figure 17 is a detailed block diagram of a camera module according to an exemplary embodiment. Detailed Description of the Invention

[0026] In the present disclosure, the term "pixel" or "sub - pixel" represents a physical area where a light - sensing element for sensing an object is located, and in a representative sense, represents information about a part of the sensing object corresponding to an electrical signal generated from the light - sensing element of a part of the sensing object.

[0027] Figure 1 Shows an image compression method according to an exemplary embodiment of the present disclosure.

[0028] Due to the increase in the integration degree and processing speed of semiconductor devices, the number of images that can be captured per second can also increase. The size of image data stored and processed in an electronic device gradually increases. Therefore, a technique for compressing image data is provided to effectively process image data.

[0029] The larger the number of light - sensing elements that convert light signals into electrical signals, the easier it is to generate a high - resolution image. As the integration degree of light - sensing elements increases, the physical distance between light - sensing elements can become narrower. As a result, noises such as crosstalk may occur, deteriorating the image quality.

[0030] At low brightness, it is relatively difficult to input sufficient optical signals into a light sensing element, and insufficient optical signals can be an obstacle to generating a high-resolution image. In order to generate a high-resolution image even at low brightness, color pixels sharing the same color can each be divided into a plurality of sub-pixels, and thus, the plurality of sub-pixels can be used to represent a monochromatic pixel. When all the electrical signals generated by the plurality of sub-pixels are combined, a sufficient amount of light can be collected.

[0031] For effective compression of image data, high compression efficiency and little data loss are desired. As a method of compressing an image, a reference pixel among a target pixel to be compressed and candidate pixels adjacent to the target pixel can be determined, and the image data can be compressed based on the pixel difference between the target pixel and the reference pixel. The method of compressing an image based on the pixel difference can be performed on the premise that the target pixel and the reference pixel are sensed at a spatially adjacent distance from each other. In other words, the method of compressing an image based on the pixel difference can utilize the physical property that a specific part of an object is sensed by at least one light sensing element located in a corresponding part of an image sensor. Therefore, in order to reduce data loss, the high spatial similarity between pixels corresponding to the light sensing elements for sensing an object is utilized.

[0032] The above method of compressing an image can also be applied to the case of compressing a Bayer image, in which color pixels including a plurality of sub-pixels arranged through the same color filter are arranged according to a specific color pattern. In one exemplary embodiment, the Bayer pattern can be a pattern known in the art, in which (stated in the order from left to right and from top to bottom, for example) green, red, blue, and another green are arranged in a matrix form. In order to refer to pixel information regarding the same color pixel, a target sub-pixel can refer to pixel information regarding another color pixel that crosses other color pixels adjacent to the target sub-pixel. For example, when the target sub-pixel to be compressed passes through a green color filter (i.e., the target sub-pixel contains information regarding green), the candidate sub-pixel to be referred to for compressing the target sub-pixel can be a green pixel located around the target sub-pixel. Since the sub-pixels are arranged in a Bayer pattern, the target sub-pixel is surrounded by red pixels and / or blue pixels, and thus, in order to determine the candidate sub-pixel and the reference sub-pixel, pixel information regarding another green pixel other than one red pixel and / or one blue pixel can be referred to. When crossing other color pixels, the spatial similarity between the target sub-pixel and the reference sub-pixel may be relatively low.

[0033] The image compression method according to the inventive concept may include operations S100 to S300 as follows.

[0034] In operation S100, image data corresponding to a plurality of sub-pixels that are arranged adjacent to each other and generate pixel values or pixel signals including substantially the same color information may be classified as the same color pixel. The image data may include information from a plurality of color pixels arranged in various combinations of color patterns or color regions. In one exemplary embodiment, the color pixels may include red pixels, green pixels, or blue pixels, and the pixel values output from the sub-pixels included in one color pixel may all include the same color information. Each of the color pixels may include a plurality of sub-pixels arranged to pass through the same color filter. The sub-pixels may be arranged in a matrix form. For example, the sub-pixels may be arranged in the form of an M×N matrix (where M and N are natural numbers greater than 0) that constitutes one color pixel (such as, for example, a 2×2 matrix, a 3×3 matrix, a 3×2 matrix, etc.). In one exemplary embodiment, for example, a plurality of sub-pixels that generate pixel signals generally including green information may be classified as green pixels.

[0035] In operation S200, data including information on a reference value for compressing image data may be generated based on the respective pixel values of the sub-pixels. In the image compression method according to the inventive concept, a reference sub-pixel may be determined based on the pixel values of a plurality of candidate sub-pixels sharing the same color information (operation S200). The pixel information on the reference sub-pixel (e.g., the data value of the reference sub-pixel) may be referred to as a reference value. Alternatively, the reference value may be a value calculated from a plurality of pixel data values. According to an exemplary embodiment, the reference value may be used as a criterion for compressing image data.

[0036] In operation S300, a difference may be calculated by comparing the pixel values of a plurality of sub-pixels included in the same color pixel with the reference value, and the comparison result and the reference value may be output.

[0037] The target sub-pixel to be compressed may be any one of the sub-pixels, and the reference value may also be a value determined based on at least any one of the sub-pixels. Thus, as a comparison result, there may be a target sub-pixel having the same value as the reference value. In an alternative embodiment, the reference value may be the median or average of the pixel values of a plurality of sub-pixels included in the same color pixel, rather than the data value of a single reference sub-pixel.

[0038] In the image compression method according to the present disclosure, since the target sub-pixels to be compressed and the reference sub-pixels to be referred to for compression can be in the same color pixel, the spatial similarity between the target sub-pixels and the reference sub-pixels can be relatively high. In other words, in the above image compression method, the sub-pixels in the same color pixel are referred to for compressing the image data, and thus, the spatial similarity may be high. Therefore, the compression efficiency of the image data can be higher than that in the case of using farther color pixels for reference. As the spatial similarity increases, the compression loss of the image data can be reduced and / or the compression efficiency can be improved.

[0039] Figure 2 FIG. 1 shows an image capturing device 1 according to an exemplary embodiment of the present disclosure. Figure 2 The image capturing device 1 can execute the image compression method described above with reference to Figure 1 FIG. 1.

[0040] With reference to Figure 2 FIG. 1, the image capturing device 1 may include a camera module 3 and an application processor (AP) 50. The camera module 3 may include an image sensor 10 and an image signal processor (ISP) 30. The image sensor 10 may include a pixel array 100, the ISP 30 may include an encoder 310, and the AP 50 may include a decoder 510.

[0041] The image capturing device 1 can capture and / or store an image of an object by using a solid-state image sensor (such as, a charge-coupled device, a complementary metal-oxide semiconductor (CMOS) sensor, etc.), and can be implemented as a digital camera, a digital video camera, a mobile phone, a tablet computer, a portable electronic device, etc. The portable electronic device may include a laptop computer, a mobile phone, a smart phone, a tablet computer, or a personal computer (PC), a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital camera, a digital video camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an e-book device, a wearable device, etc. In addition, the image capturing device 1 can be mounted on another device (such as, a drone, an advanced driver assistance system (ADAS)), or mounted on an electronic device provided as a component of a vehicle, furniture, a manufacturing facility, a door, various measuring devices, etc.

[0042] The image sensor 10 can convert an optical signal of an object incident through an optical lens into an electrical signal by using a photoelectric conversion element, generate image data IDTA based on the electrical signal, and output the image data IDTA. The image sensor 10 can be mounted on an electronic device having a function of sensing an image or light. For example, the image sensor 10 can be mounted on an electronic device (such as a camera, a smart phone, a wearable device, an Internet of Things (IoT) device, a tablet computer or PC, a PDA, a PMP, a navigation device, a drone, and / or an ADAS). In addition, the image sensor 10 can be mounted on an electronic device provided as a component of a vehicle, furniture, a manufacturing facility, a door, and various measuring devices. The image sensor 10 can sense an object photographed through a lens under the control of the ISP 30 or the AP 50.

[0043] The image sensor 10 can include a pixel array 100 and a plurality of functional units that process the received optical signal. The functional units can include additional components (such as, for example, a row driver, a ramp signal generator, a timing generator, an analog-to-digital converter, a readout circuit, etc.) for processing the optical signal or improving the image sensing sensitivity.

[0044] The pixel array 100 can include a plurality of row lines, a plurality of column lines, a plurality of pixels each connected to the row lines and the column lines and arranged in a matrix form, and a plurality of color filters arranged corresponding to the respective pixels.

[0045] Each pixel can include a light sensing element (such as a photoelectric conversion element), detect light by using the light sensing element, and convert the detected light into a pixel signal that can be an electrical signal. For example, the light sensing element can be a photodiode, a phototransistor, a photogate, a pinned photodiode (PPD), or a combination thereof. Each of the plurality of light sensing elements can have a 4-transistor structure including a photodiode, a transfer transistor, a reset transistor, an amplification transistor, and a selection transistor. According to an alternative embodiment, each of the light sensing elements can have a 1-transistor structure, a 3-transistor structure, a 5-transistor structure, or a structure in which some transistors are shared by a plurality of pixels.

[0046] The image sensor 10 can obtain pixel data from the optical signal that has passed through the color filter. The pixel data can represent the incident light intensity or the pixel value of the pixel. The color filter can be arranged corresponding to each pixel in the pixel array, and thus serves as a filter that transmits only light of a specific wavelength among the light incident on the light sensing element of the image sensor. One color pixel can include a plurality of sub-pixels, and each sub-pixel can operate as a unit element for processing one color.

[0047] The color filter can be applied in the form of a Bayer color filter. The Bayer pattern is based on the premise that the human eye obtains most of the luminance data from the green component of an object. Half of the pixels included in the Bayer color filter can detect green signals, a quarter of the pixels can detect red signals, and another quarter of the pixels can detect blue signals. According to one embodiment, the Bayer color filter may have the following configuration: a 2×2 unit including a red (R) pixel, a blue (B) pixel, and two green (G) pixels is repeatedly arranged. According to another embodiment, the Bayer color filter may have the following configuration: a 2×2 unit including a red pixel, a blue pixel, and two wide green (W) pixels is repeatedly arranged. In other words, an RGB color filter scheme in which green color filters are provided for two of the four pixels and blue and red color filters are provided for the other two pixels may be adopted. In addition, in addition to the RGB color filter scheme, a CYGM color filter scheme in which cyan, yellow, green, and magenta color filters are respectively arranged in the four pixels is also adopted. In addition, a cyan, magenta, yellow, black (CYMK) color filter scheme may be applied. A plurality of color filters may be considered to constitute a color filter layer. Although the Bayer pattern is shown for ease of explanation, the present disclosure is not limited to the Bayer pattern, and color filters including white color filters or yellow color filters, or color filters having various patterns that combine two or more color regions may be applied.

[0048] The pixel data or a set of pixel values generated through the color filter may be referred to as a Bayer image. Here, the Bayer image is distinguished from the color filter that physically transmits light of a specific wavelength, and the Bayer image may correspond to the image sensed by the image sensor 10, or be recognized as the conceptual image shape recognized by the ISP 30, the AP 50, or in the user environment. In other words, the Bayer image may represent the image data including the pixel information of the complete image regarded as one processing unit in the image capturing device 1. For ease of explanation, the image data will be described below by using the term "Bayer image". However, the present disclosure is not limited to the color filter of the Bayer pattern, and color filters having various patterns may be applied.

[0049] The image sensor 10 may output image data IDTA generated by preprocessing (such as defective pixel removal, etc.) the pixel signals generated by the pixel array 100. The image sensor 10 may be implemented as a semiconductor chip, a package including a pixel array, etc. The pixel array 100 may include a plurality of units 110. In one exemplary embodiment, the unit 110 may include two green sub-pixels (G1, G2), one red sub-pixel (R), and one blue sub-pixel (B).

[0050] The ISP 30 can receive the image data IDTA as an output signal from the image sensor 10, process the image data IDTA into an image recognizable by humans, and output the processed image to a display. Optionally, the ISP 30 can receive a control signal from an external host through an interface (I / F), and correspondingly provide the processed image to the external host.

[0051] The ISP 30 can perform image processing on the image data IDTA. For example, the ISP 30 can perform image processing for changing the data format of the image data IDTA (such as, taking the example of changing the image data in Bayer pattern to image data in YUV format or RGB format), and image processing for improving image quality (such as, noise reduction, brightness adjustment, and / or sharpness adjustment). The ISP 30 can be implemented as a hardware component of the image capture device 1. Although Figure 2 it is shown that the ISP 30 is provided independently of the image sensor 10, the present disclosure is not limited thereto, and the ISP 30 can be located inside the image sensor 10 or inside the AP 50 without limitation.

[0052] The ISP 30 can include an encoder 310. The encoder 310 can encode the image data IDTA and output the encoded image data IDTA in the form of a bitstream BS.

[0053] The encoder 310 can reduce the data size by compressing the image data IDTA corresponding to the pixel data obtained through the color filter. The image data can represent the pixel data of various patterns obtained through the color filter. Since the original pixel data is the pixel data or pixel value of the pixel to be compressed, the original pixel data can be referred to as the pixel value of the target sub-pixel. According to an embodiment of the present disclosure, the encoder 310 can generate encoded data of the pixel data based on a plurality of sub-pixels included in the pixel. The following can be referred to Figures 2 to 14 for a more detailed description of the process for compressing pixel data based on a plurality of sub-pixels.

[0054] The camera module 3 can include the image sensor 10 and the ISP 30. In other words, the combination of the image sensor 10 and the ISP 30 that processes the image data IDTA output from the image sensor 10 can be referred to as the camera module 3. The camera module 3 can compress the captured image and output the compression result as a bitstream BS.

[0055] The bitstream BS generated by the camera module 3 can be decoded by a device such as or including a decoder (e.g., a decoder sharing the same protocol with the camera module 3). The protocol is a shared communication principle for compression algorithms (such as compression methods, compression sequences, and compressed bits), and can indicate a bilateral protocol for implementing decoding by applying the same mechanism as the encoding mechanism. In one exemplary embodiment, as long as the protocol applied to the encoder 310 included in the camera module 3 is the same as the protocol supported by the AP 50 and / or the decoder 510, the bitstream BS generated by the camera module 3 can be decoded, so the decoder does not need to be installed on the same semiconductor chip as the encoder 310.

[0056] The AP 50 can be a central processing unit (CPU), a microprocessor, or a microcontroller unit (MCU). The AP 50 can perform post-processing on the bitstream BS. Post-processing can represent applying an image enhancement algorithm to image artifacts. For example, the AP 50 can perform white balance correction, autofocus correction, autoexposure correction, denoising, demosaicing filtering, lens shading correction, and / or gamma correction on the received bitstream BS. However, the present disclosure is not limited thereto, and various functions for improving image quality can be performed.

[0057] The decoder 510 can receive the bitstream BS including the encoded data generated by the encoder 310 and decode the received bitstream BS. The decoder 510 can perform a series of processes opposite to the processes performed by the encoder 310 for encoding the image data IDTA in the reverse order, thereby restoring the original pixels corresponding to the original image data. In other words, the decoder 510 can restore or decompress the compressed image data IDTA received from the encoder 310. In other words, the encoder 310 and the decoder 510 can use the same protocol.

[0058] Figure 2 Each module shown (i.e., the image sensor 10, the ISP 30, and the AP 50) can be implemented by a single processor, or multiple modules can be independently implemented by different processors respectively. Hereinafter, the term "pixel value" can represent a value obtained by processing a signal corresponding to a light signal input to a photosensing element on the pixel array 100 (such as through a color filter) as image data.

[0059] The ISP 30 and the AP 50 can be implemented as processing circuits (such as hardware including logic circuits) or can be implemented as a combination of hardware and software (such as a processor executing software for performing a compression operation). More specifically, the processing circuit can be implemented by a central processing unit (CPU), an arithmetic logic unit (ALU) that performs arithmetic and logical operations and shifts, a digital signal processor (DSP), a microprocessor, or an application-specific integrated circuit (ASIC), but the present disclosure is not limited thereto.

[0060] Figure 3 Unit 110 showing a larger pixel array in which color pixels 130 including a plurality of sub-pixels 150 are arranged in a Bayer pattern. Reference may be made to Figure 2 description Figure 3 and repeated descriptions may be omitted.

[0061] In the image data generated by an image sensor ( Figure 2 100 and 110), pixel information about an object may be arranged in a Bayer pattern. As described above, the image data in the Bayer pattern may be referred to as a Bayer image.

[0062] The image data may include a plurality of units 110. Each unit 110 may include pixels containing color information. In one exemplary embodiment, unit 110 may include one red pixel, two green pixels, and one blue pixel. Unit 110 may include color information about a part of the sensed object and may correspond to a part of the pixel array ( Figure 2 100). Unit 110 may be a basic unit for displaying color information about a part of an object.

[0063] Unit 110 may include a plurality of color pixels 130. In one exemplary embodiment, according to the arrangement of the color pixels included in pixel array 100, the color pixels 130 of unit 110 may include two green pixels, one red pixel, and one blue pixel.

[0064] Each color pixel 130 may include a plurality of sub-pixels 150 of the same color. Since the sub-pixels 150 included in one color pixel 130 are formed through the same color filter, the sub-pixels 150 may have the same color information.

[0065] The sub-pixels 150 may be arranged in a matrix form. Although Figure 3 the arrangement of the sub-pixels 150 is shown as an example of a 2×2 sub-pixel matrix within a larger 2×2 pixel matrix, the present disclosure is not limited thereto, and the sub-pixels 150 may be arranged in an M×N matrix (where M and N are natural numbers greater than 0) (such as, for example, a 2×3 matrix or a 3×3 matrix) without limitation. In addition, in an alternative embodiment, each unit of the sub-pixels does not need to be configured as a tetra cell and may be, for example, a tri cell and / or a hexa cell for a planar image sensor, or a penta cell for a hyperbolic or spherical image sensor, but is not limited thereto. In addition, in an alternative embodiment, the green sub-pixels may be arranged in a hexa cell, while the red sub-pixels and the blue sub-pixels may be arranged in a tri cell.

[0066] In an exemplary embodiment, a green pixel may include four green sub-pixels G1a, G2a, G3a, and G4a. Similarly, according to an exemplary embodiment, a blue pixel may include four blue sub-pixels B1, B2, B3, and B4, a red pixel may include four red sub-pixels R1, R2, R3, and R4, and the image data may include two green pixels, so another green pixel may include four different green sub-pixels G1b, G2b, G3b, and G4b.

[0067] Although the two green pixels included in one unit 110 have the same color information, the two green pixels may be physically arranged to process different characteristics, and thus, the two green pixels may be distinguishable from each other. In an exemplary embodiment, the green pixel in the first row of the unit 110 may be related to the characteristics of the red pixel, and the green pixel in the second row may be related to the characteristics of the blue pixel, but the present disclosure is not limited thereto.

[0068] When the sub-pixels 150 generate sub-pixel signals of the same color, the image capturing device (see Figure 2 and Figure 1 ) sums the sub-pixel signals, so that sufficient electrical signals as a result of the sensed light can be generated even at a low illumination level.

[0069] Figure 4A and Figure 4B show examples of pixel arrays 100a and 100b. Reference may be made to Figure 2 and Figure 3 for description Figure 4A and Figure 4B , and repeated descriptions may be omitted.

[0070] When only one reference sub-pixel among the target sub-pixel to be compressed and the candidate sub-pixels adjacent to the target sub-pixel is determined, and only the pixel difference between the target sub-pixel and the reference sub-pixel is stored or processed, the amount of data to be processed and / or stored can be reduced. For example, a predictive coding method, a differential pulse code modulation (DPCM) method, or a similar method may be used.

[0071] Referring to Figure 4A , according to an exemplary embodiment, it is assumed that the sub-pixels 150 included in the color pixel 130 are arranged in a 2×2 matrix. When the number of sub-pixels is 4, the unit including the sub-pixels may be referred to as a four-unit.

[0072] To compress one sub-pixel 151 (i.e., the target sub-pixel) in the first row of the pixel array 100a, the camera module (e.g., Figure 2The camera module 3) can search for reference sub-pixels including substantially the same color information. Any one of the candidate sub-pixels can be determined as the reference sub-pixel, or the reference sub-pixel can be determined based on the values of the candidate sub-pixels. In an alternative embodiment, the reference sub-pixel can be, for example, a virtual sub-pixel based on the average of the sub-pixels in the unit.

[0073] The sub-pixel that senses color information substantially the same as the target sub-pixel 151 and is closest to the target sub-pixel 151 can be determined as the reference sub-pixel 132. To represent pixel information, the candidate sub-pixels can be read or processed (such as compressed) first before the target sub-pixel 151, so there is no need to refer to the rows below the row of the target sub-pixel 151. Additionally, if the target sub-pixel is red or blue in this example, since the green sub-pixels in the row directly above the row of the target sub-pixel 151 may have different characteristics from the target sub-pixel 151, there is no need to refer to the green sub-pixels. As a result, the reference sub-pixel 132 can be the pixel of substantially the same color closest to the target sub-pixel 151.

[0074] Since the target sub-pixel 151 (if green) is surrounded by red pixels and blue pixels, the red pixels and blue pixels are sensed before the target sub-pixel 151, and the pixel distance P between the reference sub-pixel 132, which is the closest to the target sub-pixel 151 among the pixels sensing color information substantially the same as the target sub-pixel 151, and the target sub-pixel 151 distance can be 3. Hereinafter, it is assumed that the width of one sub-pixel is 1.

[0075] Similarly, the target sub-pixel 153 included in the blue pixel can determine the pixel having substantially the same color information "blue" and closest to the target sub-pixel 153 as the reference sub-pixel 133. The pixel distance P between the target sub-pixel 153 and the reference sub-pixel 133 distance can also be 3.

[0076] Referring to Figure 4B , according to an exemplary embodiment, assume a pixel array 100b in which the sub-pixels 150 are arranged in a 3×3 matrix. When the number of sub-pixels is 9, the unit including the sub-pixels can be referred to as a nona cell.

[0077] To compress a sub-pixel 155 that is a target sub-pixel (and hereinafter referred to as the target sub-pixel) in the first row of the pixel array 100b, the camera module 3 may search for reference sub-pixels including substantially the same color information. As a result, a pixel having substantially the same color information "green" as the target sub-pixel 155 and closest to the target sub-pixel 155 may be determined as the reference sub-pixel 134. The distance between the target sub-pixel 155 and the reference sub-pixel 134 may additionally include 3 as the width of one color pixel and the sum of the widths of a plurality of sub-pixels. As a result, the pixel distance P between the target sub-pixel 155 and the reference sub-pixel 134 distance may also be 4.

[0078] Similarly, to compress the target sub-pixel 157 included in the blue pixel, a pixel having substantially the same color information "blue" as the target sub-pixel 157 and closest to the target sub-pixel 157 may be determined as the reference sub-pixel 137. The pixel distance P between the target sub-pixel 157 and the reference sub-pixel 137 distance may also be 4.

[0079] Figure 5A 、 Figure 5B and Figure 5C illustrate a process for determining a reference value according to an exemplary embodiment of the present disclosure. Reference may be made to Figure 2 、 Figure 3 and Figure 4A for description Figure 5A 、 Figure 5B and Figure 5C , and repeated descriptions may be omitted. In one exemplary embodiment, it is assumed that the pixel array 100a in which the sub-pixels 150 are arranged in a 2×2 matrix.

[0080] In the image compression method according to the present disclosure, a reference value may be determined based on a plurality of sub-pixels (i.e., candidate sub-pixels) sharing substantially the same color information. Specifically, a reference sub-pixel may be determined within a color pixel 131 including the target sub-pixel to be compressed ( Figure 4A 151) and including color information about green, or a reference sub-pixel may be determined based on at least one of a plurality of sub-pixels included in the color pixel 131.

[0081] According to an exemplary embodiment, the color pixel 131 ( Figure 5A 131) including the target sub-pixel may include four sub-pixels G1a, G2a, G3a, and G4a. In other words, the target sub-pixel 151 may be any one of the four sub-pixels G1a, G2a, G3a, and G4a.

[0082] Referring to Figure 5A , the camera module ( Figure 2 3, specifically,Figure 2 The encoder 310) can calculate the average of the pixel values of the four sub-pixels G1a, G2a, G3a, and G4a included in the color pixel 131. The camera module 3 can determine the average pixel value of the four sub-pixels G1a, G2a, G3a, and G4a as a reference value. When there is no sub-pixel having the average pixel value of the four sub-pixels G1a, G2a, G3a, and G4a, it can be understood that due to the mathematical characteristics of the four sub-pixels G1a, G2a, G3a, and G4a, a virtual sub-pixel having the average pixel value of the four sub-pixels G1a, G2a, G3a, and G4a can be virtually located at the center of the matrix of the four sub-pixels G1a, G2a, G3a, and G4a. In one exemplary embodiment, a sub-pixel having the average pixel value as the pixel value can virtually exist.

[0083] The camera module 3 can compare the pixel value of each of the four sub-pixels G1a, G2a, G3a, and G4a with the reference value. In one exemplary embodiment, the camera module 3 can perform subtraction between the reference value and the pixel value of each of the four sub-pixels G1a, G2a, G3a, and G4a. In one exemplary embodiment, a residual sub-pixel r1a having the difference between the pixel value of the sub-pixel G1a and the reference value Gra can be generated, a residual sub-pixel r2a having the difference between the pixel value of the sub-pixel G2a and the reference value Gra can be generated, a residual sub-pixel r3a having the difference between the pixel value of the sub-pixel G3a and the reference value Gra can be generated, and a residual sub-pixel r4a having the difference between the pixel value of the sub-pixel G4a and the reference value Gra can be generated. The order of the subtraction operations is not restricted, and the reference value can be subtracted from the sub-pixel value. In one exemplary embodiment, the order of the subtraction operations can be predetermined, and the decoder ( Figure 2 of 510) can decode the encoded data with reference to the order of the subtraction operations predetermined by the encoder 310. Although the determination of the residual sub-pixel values is shown here as a subtraction operation, in an alternative embodiment, the residual sub-pixels r1a, r2a, r3a, and r4a can be determined by dividing one of the target or reference of each target sub-pixel of each pixel unit by the other of the target or reference, respectively.

[0084] ISP( Figure 2 of 30) can compress the image data by generating the respective pixel values of the reference value and the residual sub-pixels as a bitstream BS, instead of transmitting the pixel values of the sub-pixels included in the same color pixel.

[0085] It can be assumed that the residual sub-pixels r1a, r2a, r3a, and r4a are arranged in a 2×2 matrix similar to the four sub-pixels G1a, G2a, G3a, and G4a. In this case, it can be understood that the set of the residual sub-pixels r1a, r2a, r3a, and r4a is the color pixel 131' that generates green information. The reference sub-pixel 151' having the average pixel value as the pixel value can be virtually located at the center of the matrix of the residual sub-pixels r1a, r2a, r3a, and r4a.

[0086] Assuming that the width of the sub-pixel is 1, the pixel distance P between the reference sub-pixel 151' and each of the residual sub-pixels r1a, r2a, r3a, and r4a distance can be 1 / √2. Figure 5A Compared with Figure 4A the pixel distance P distance can be reduced from 3 to 1 / √2. It can be understood that the reduction of the pixel distance P distance corresponds to increased spatial similarity. As the spatial similarity increases, the image compression efficiency of the image compression method according to the present disclosure can be improved, and the compression loss of the image data can be reduced.

[0087] Referring to Figure 5B , the camera module 3 (specifically, the encoder 310) can calculate the median of the pixel values of the four sub-pixels G1a, G2a, G3a, and G4a included in the color pixel 131.

[0088] Specifically, the camera module 3 can sort the pixel values of the four sub-pixels G1a, G2a, G3a, and G4a according to the magnitudes of the pixel values of the four sub-pixels G1a, G2a, G3a, and G4a. The camera module 3 can determine the sub-pixel G1a having the second smallest pixel value among the multiple pixel values as the reference sub-pixel 151″. Generally, when the number of pixel values is even, the median can represent the average of the two values close to the middle. However, according to the present disclosure, it can be understood that the median represents the smaller value of the two pixel values. However, the present disclosure is not limited thereto, and the median can represent the larger value of the two pixel values. The pixel value of the reference sub-pixel 151″ can be determined as the reference value Gra. In other words, the reference sub-pixel 151″ can be a virtual sub-pixel included in the color pixel 131.

[0089] The camera module 3 may compare the pixel value of each of the four sub-pixels G1a, G2a, G3a, and G4a with a reference value. In an exemplary embodiment, the camera module 3 may perform a subtraction between the reference value Gra and the pixel value of each of the four sub-pixels G1a, G2a, G3a, and G4a. In an exemplary embodiment, since the reference value Gra is the same as the pixel value of the sub-pixel G1a, the result of performing the subtraction operation may be zero (0). Since the residual sub-pixel r1a has zero data, it has no physical meaning. However, the residual sub-pixel r1a may be abstractly assumed. Similarly, a residual sub-pixel r2a having a sub-pixel value the same as the difference between the reference value Gra and the pixel value of the sub-pixel G2a may be generated, a residual sub-pixel r3a having a sub-pixel value the same as the difference between the reference value Gra and the pixel value of the sub-pixel G3a may be generated, and a residual sub-pixel r4a having a sub-pixel value the same as the difference between the reference value Gra and the pixel value of the sub-pixel G4a may be generated. As already stated above, the order of the subtraction operation is not restricted. The ISP ( Figure 2 of 30) may compress the image data by generating the reference value and the corresponding pixel values of the residual sub-pixels r2a, r3a, and r4a as a bit stream BS, instead of transmitting the pixel values of the sub-pixels included in the same color pixel.

[0090] It can be understood that the set of the residual sub-pixels r1a, r2a, r3a, and r4a is the color pixel 131 that generates green information. The reference sub-pixel 151” may correspond to the sub-pixel G1a. In an exemplary embodiment, the pixel distance P distance between the reference sub-pixel 151” and the residual sub-pixels r2a and r3a may be 1, and the pixel distance P distance between the reference sub-pixel 151” and the residual sub-pixel r4a may be √2. Comparing Figure 5B with Figure 4A , the pixel distance P distance may be reduced from 3 to 1 or from 3 to √2. Similarly, it can be understood that the reduction of the pixel distance P distance corresponds to increased spatial similarity.

[0091] Referring to Figure 5C , the camera module 3 (specifically, the encoder 310) may determine any one of the four sub-pixels G1a, G2a, G3a, and G4a included in the color pixel 131 (the sub-pixel G2a) as the reference sub-pixel 151”'. The position of the sub-pixel to be determined as the reference sub-pixel 151”' may be predetermined. For example, the sub-pixel to be determined as the reference sub-pixel 151”' may be the sub-pixel set in the first row and the second column among the multiple sub-pixels in the color pixel. The position of the predetermined sub-pixel may be referred to by the encoder 310 and the decoder 510.

[0092] The camera module 3 may compare the pixel value of each of the four sub-pixels G1a, G2a, G3a, and G4a with a reference value. In an exemplary embodiment, the camera module 3 may perform a subtraction between the reference value and the pixel value of each of the four sub-pixels G1a, G2a, G3a, and G4a. In an exemplary embodiment, since the reference value Gra is the same as the pixel value of the sub-pixel G2a, the result of performing the subtraction operation may be zero (0). Since the residual sub-pixel r2a has zero data, it has no physical meaning. However, the residual sub-pixel r2a may be abstractly assumed. Similarly, a residual sub-pixel r1a having a sub-pixel value the same as the difference between the reference value Gra and the pixel value of the sub-pixel G1a may be generated, a residual sub-pixel r3a having a sub-pixel value the same as the difference between the reference value Gra and the pixel value of the sub-pixel G3a may be generated, and a residual sub-pixel r4a having a sub-pixel value the same as the difference between the reference value Gra and the pixel value of the sub-pixel G4a may be generated. As already stated above, the order of the subtraction operation is not restricted. The ISP ( Figure 2 30) may compress the image by generating the residual sub-pixels r1a, r3a, and r4a having pixel values that are the differences between the reference value and the pixel values of the sub-pixels as a bitstream BS, instead of transmitting the pixel values of all the sub-pixels.

[0093] It can be understood that the set of the residual sub-pixels r1a, r2a, r3a, and r4a is the color pixel 131”' that generates green information. The reference sub-pixel 151”' may correspond to the sub-pixel G2a. In an exemplary embodiment, the pixel distance P distance between the reference sub-pixel 151”' and the residual sub-pixels r1a and r4a may be 1, and the pixel distance P distance between the reference sub-pixel 151”' and the residual sub-pixel r3a may be √2. Comparing Figure 5C with Figure 4A , the pixel distance P distance may be reduced from 3 to 1 or from 3 to √2. Similarly, it can be understood that the reduction of the pixel distance P distance corresponds to increased spatial similarity.

[0094] Figure 6A 、 Figure 6B and Figure 6C show the process for determining a reference value according to an exemplary embodiment of the present disclosure. Reference may be made to Figure 2 、 Figure 3 、 Figure 4B and Figures 5A to 5C for a description Figure 6A 、 Figure 6B and Figure 6C, and repeated descriptions may be omitted. In one exemplary embodiment, a pixel array 100b in which sub-pixels 150 are arranged in a 3×3 matrix is ​​assumed.

[0095] Except that the number of sub-pixels included in one color pixel 135 is 9 and the sub-pixels 150 are arranged in a 3×3 matrix, Figures 6A to 6C The image compression method shown in Figures 5A to 5C The image compression method shown in FIG.

[0096] Reference Figure 6A , Camera Module( Figure 2 3. Specifically, the encoder ( Figure 2 310)), an average value of pixel values ​​of nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a included in the color pixel 135 may be calculated. The camera module 3 may determine the average pixel value of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a as a reference value. The reference value may be any one of the pixel values ​​of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a, or may not be one of the pixel values ​​of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a. In other words, a sub-pixel having the same pixel value as the average pixel value of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a may exist and may or may not be included in the color pixel 135. When there is no reference sub-pixel 155' having the same pixel value as the average pixel value of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a, it may be understood that the average pixel value of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a may be virtually located at any position in the matrix of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a due to mathematical properties thereof. In one exemplary embodiment, the reference sub-pixel 155 ′ having the average pixel value as a pixel value may exist virtually.

[0097] The camera module 3 can compare the pixel value of each of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a with a reference value. In an exemplary embodiment, the camera module 3 can perform a subtraction between the reference value and the pixel value of each of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a. In an exemplary embodiment, a residual sub-pixel r1a of the sub-pixel value can be generated as the difference between the pixel value of the pixel having the sub-pixel G1a and the reference value Gra. Similarly, residual sub-pixels r2a, r3a, r4a, r5a, r6a, r7a, r8a, and r9a can be generated as the results of the subtraction operations between the pixel values of the sub-pixels G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a and the reference value Gra, respectively. As already stated above, the order of the subtraction operations is not limited. The ISP ( Figure 2 of 30) can compress the image by generating the difference between the reference value and the pixel value of the sub-pixel as a bitstream BS, instead of transmitting the pixel values of all the sub-pixels.

[0098] It can be assumed that the residual sub-pixels r1a, r2a, r3a, r4a, r5a, r6a, r7a, r8a, and r9a are arranged in a 3×3 matrix in a similar manner to the sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a. In this case, it can be understood that the set of the residual sub-pixels r1a, r2a, r3a, r4a, r5a, r6a, r7a, r8a, and r9a is a color pixel 135' including green information.

[0099] Assume that the position of the reference sub-pixel 155' having an average pixel value as the pixel value is the position of the residual sub-pixel r7a.

[0100] Since the width of the sub-pixel is 1, the pixel distance P distance The maximum value of can be 2√2 as the distance between the sub-pixel r7a and the residual sub-pixel r3a. Figure 6A Compare with Figure 4B The pixel distance P distance can be reduced from 4 to 2√2. It can be understood that the reduction of the pixel distance P distance corresponds to an increased spatial similarity. As the spatial similarity increases, the image compression efficiency of the image compression method according to the present disclosure can be improved, and the compression loss of the image data can be reduced.

[0101] Refer to Figure 6B, the camera module 3 (specifically, the encoder 310) may calculate the median of the pixel values of nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a included in the color pixel 135.

[0102] Specifically, the camera module 3 may sort the pixel values of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a according to the magnitudes of the pixel values of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a. The camera module 3 may determine the sub-pixel G1a having the fifth smallest pixel value among the multiple pixel values as the reference sub-pixel 155″. The fifth smallest pixel value among the nine pixel values may simultaneously be the fifth largest pixel value and thus may be the median of the nine pixel values. The pixel value of the reference sub-pixel 155″ may be determined as the reference value Gra. In other words, the reference sub-pixel 155″ may be included in the color pixel 135″.

[0103] The camera module 3 may compare the pixel value of each of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a with the reference value Gra. In one exemplary embodiment, the camera module 3 may perform a subtraction between the reference value and the pixel value of each of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a. In one exemplary embodiment, since the reference value Gra is the same as the pixel value of the sub-pixel G1a, the result of performing the subtraction operation may be zero (0). Since the residual sub-pixel r1a has zero data, it has no physical meaning. However, the residual sub-pixel r1a may be abstractly assumed. Similarly, a residual sub-pixel r2a having a sub-pixel value equal to the difference between the reference value Gra and the pixel value of the sub-pixel G2a may be generated, and a residual sub-pixel r3a having a sub-pixel value equal to the difference between the reference value Gra and the pixel value of the sub-pixel G3a may be generated. Similarly, residual sub-pixels r3a, r4a, r5a, r6a, r7a, r8a, and r9a having the differences between the reference value Gra and the pixel values of the sub-pixels G3a, G4a, G5a, G6a, G7a, G8a, and G9a as sub-pixel values may be respectively generated. As already stated above, the order of the subtraction operations is not restricted. The ISP ( Figure 2 30) may compress the image by generating the residual sub-pixels r2a, r3a, r4a, r5a, r6a, r7a, r8a, and r9a having sub-pixel values that are the differences between the reference value Gra and the pixel values of the sub-pixels as the bit stream BS, instead of transmitting the pixel values of all the sub-pixels.

[0104] It is understandable that the set of residual sub-pixels r1a, r2a, r3a, r4a, r5a, r6a, r7a, r8a and r9a is the color pixel 135″ including green information. The reference sub-pixel 155″ may correspond to the sub-pixel G1a. In an exemplary embodiment, the pixel distance P distance between the reference sub-pixel 155″ and the residual sub-pixels r2a, r3a, r4a, r5a, r6a, r7a, r8a and r9a Figure 6B can be Figure 4B compared, and the pixel distance P distance can be reduced from 4 to 1, from 4 to √2, from 4 to √5, or from 4 to 2√2. Similarly, it is understandable that the reduction of the pixel distance P distance corresponds to increased spatial similarity.

[0105] Referring to Figure 6C , the camera module 3 (specifically, the encoder 310) may determine any one of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a and G9a included in the color pixel 135 (sub-pixel G3a) as the reference sub-pixel 155”'. The position of the sub-pixel to be determined as the reference sub-pixel may be predetermined. The position of the predetermined sub-pixel may be referred to by the encoder 310 and the decoder 510. The reference sub-pixel 155”' may be included in the color pixel 135.

[0106] The camera module 3 can compare the pixel value of each of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a with the reference value Gra. In an exemplary embodiment, the camera module 3 can perform a subtraction between the reference value Gra and the pixel value of each of the nine sub-pixels G1a, G2a, G3a, G4a, G5a, G6a, G7a, G8a, and G9a. In an exemplary embodiment, since the reference value Gra is the same as the pixel value of the sub-pixel G3a, the result of performing the subtraction operation can be zero (0). Since the residual sub-pixel r3a has zero data, it has no physical meaning. However, the residual sub-pixel r3a can be abstractly assumed. Similarly, a residual sub-pixel r1a having a sub-pixel value the same as the difference between the reference value Gra and the pixel value of the sub-pixel G1a can be generated, and a residual sub-pixel r2a having a sub-pixel value the same as the difference between the reference value Gra and the pixel value of the sub-pixel G2a can be generated. Similarly, residual sub-pixels r4a, r5a, r6a, r7a, r8a, and r9a having sub-pixel values that are the differences between the reference value Gra and the pixel values of the sub-pixels G4a, G5a, G6a, G7a, G8a, and G9a, respectively, can be generated. As already stated above, the order of the subtraction operation is not restricted. The ISP ( Figure 2 30) can compress the image by generating the residual sub-pixels r1a, r2a, r4a, r5a, r6a, r7a, r8a, and r9a having pixel values that are the differences between the reference value Gra and the pixel values of the sub-pixels as a bitstream BS, instead of transmitting the pixel values of all the sub-pixels.

[0107] It can be understood that the set of the residual sub-pixels r1a, r2a, r3a, r4a, r5a, r6a, r7a, r8a, and r9a is the color pixel 135”' including green information. The reference sub-pixel 155”' can correspond to the sub-pixel G3a. In an exemplary embodiment, the pixel distance P distance between the reference sub-pixel 155”' and the residual sub-pixels r1a, r2a, r3a, r4a, r5a, r6a, r7a, r8a, and r9a distance can be 1, √2, 2, √5, or 2√2. In other words, the maximum value of the pixel distance P Figure 6C between the reference sub-pixel 155”' and the residual sub-pixels r1a, r2a, r3a, r4a, r5a, r6a, r7a, r8a, and r9a Figure 4B can be 2√2. Comparing distance with distance , the maximum value of the pixel distance P

[0108] Figure 7A and Figure 7B illustrates a configuration of a bitstream BS according to an exemplary embodiment of the present disclosure.

[0109] Figures 7A to 7B The illustrated bitstream BS may be a result of compressing image data corresponding to pixel values generated from the four sub-pixels shown in Figure 5A . According to an exemplary embodiment, the image data corresponding to the pixel value output from one sub-pixel may be 10 bits. Therefore, the image data corresponding to the pixel value output from one color pixel ( Figure 3 130) including four sub-pixels may include a total of 40-bit data. Although, for ease of explanation, the data included in one sub-pixel is shown as 10 bits herein, the present disclosure is not limited thereto, and one sub-pixel may include data of various bits (e.g., 8 bits, 11 bits, 12 bits, etc.).

[0110] The camera module ( Figure 2 3) may allocate a data space of the bitstream BS to compress 40-bit data into 20-bit data. When 40-bit data is compressed into 20-bit data, the compression ratio of the data is 50%.

[0111] Figure 7A may be the bitstream BS according to the Figure 5A compression result. Referring to Figure 7A and Figure 5A , according to an exemplary embodiment, 4 bits may be allocated to the header information (e.g., “H” as shown in Figure 7A ). The header information may be a set of bits generated by encoding compression information about a compression method (e.g., information about compression algorithms such as DPCM method and PCM method). As a result of allocating 4 bits to the header information, 2^4 (=16) pieces of compression information may be transmitted through the header information. According to an exemplary embodiment, the decoder 510 may refer to the compression method applied to the sub-pixels through the header information and decode the bitstream BS by using the same compression method.

[0112] According to an exemplary embodiment, the camera module ( Figure 2 3) may determine the average pixel value of the sub-pixels as a reference value Gra (i.e., the pixel value of a reference sub-pixel 151' that virtually exists). The sub-pixel 151' may be a comparison standard for compressing data.

[0113] According to an exemplary embodiment, 4 bits may be allocated to the reference value Gra (i.e., the pixel value of a reference sub-pixel 151' that virtually exists) (e.g., as shown in Figure 7A("R" shown in). Since the reference sub-pixel 151' is the average of sub-pixels each containing 10-bit data, the reference sub-pixel 151' can contain a pixel value of 10 bits. However, since only 4-bit data space is allocated, the data space may be insufficient. To solve the insufficiency of the data space, the encoder ( Figure 2 of 310) may remove a part of the data of the reference sub-pixel 151'. According to an exemplary embodiment, the encoder 310 may remove the lower 6 bits from the 10 bits of the reference sub-pixel 151' and include only the upper 4 bits in the bitstream BS. The data of the sub-pixel to be removed is not limited thereto, and any number of lower bits may be removed according to the required performance (compression ratio, data loss rate, power, etc.).

[0114] According to an exemplary embodiment, 3 bits may be allocated for one residual sub-pixel. One color pixel 131' may include four residual sub-pixels, and a total of 12-bit space 3BIT_1, 3BIT_2, 3BIT_3, and 3BIT_4 may be allocated for the four residual sub-pixels in the bitstream BS for transmitting data regarding one color pixel 131'. Since the pixel value of the residual sub-pixel is the difference between the reference value Gra and the pixel value of the sub-pixel in one color pixel 131, a space of 3 bits less than 4 bits may be allocated as the space for the residual sub-pixel.

[0115] As a result, 40 bits for four sub-pixels can be compressed into 16 bits. When adding 4 bits allocated for the header information including the compression information, 40 bits can be compressed to 20 bits, and a compression ratio of 50% can be achieved. Although for ease of explanation, it is assumed that the size of the compressed data is 20 bits, the present disclosure is not limited thereto, and the data can be compressed into various sizes (such as 10 bits (75% compression ratio) and 30 bits (25% compression ratio)) according to the required performance (compression ratio, data loss rate, power, etc.).

[0116] Figure 7B will be referred to in conjunction with Figure 5B According to an exemplary embodiment, the camera module 3 may determine the sub-pixel G1a having the median of the sub-pixels (i.e., the pixel value of the sub-pixel G1a is the median of the pixel values of the plurality of sub-pixels) as the reference sub-pixel 151", or may determine any one predetermined sub-pixel (sub-pixel G2a) among the sub-pixels as the reference sub-pixel 151".

[0117] According to an exemplary embodiment, 4 bits may be allocated to the reference sub-pixel 151" or 151"'. Although the reference sub-pixel 151" or 151"' may include a 10-bit pixel value, the data space allocated to the reference sub-pixel 151" or 151"' is only 4 bits, and therefore, the data space may be insufficient. In order to address the insufficient data space, the encoder 310 may remove a portion of the data of the reference sub-pixel 151" or 151"'. According to an exemplary embodiment, the encoder 310 may remove the lower 6 bits from the 10 bits of the reference sub-pixel 151" or 151"' and include only the upper 4 bits in the bitstream BS. As described above, the data of the sub-pixel to be removed is not limited thereto.

[0118] According to an exemplary embodiment, 4 bits may be allocated to each of the remaining sub-pixels except for the pixel determined as the reference sub-pixel 151" or 151"'. One color pixel 131" or 131"' may include four sub-pixels, and in a bit stream BS for transmitting data about one color pixel 131" or 131"', a total of 12 bits of space 4BIT_2, 4BIT_3, and 4BIT_4 may be allocated to the remaining three sub-pixels except for the reference sub-pixel 151" or 151"'.

[0119] As a result, 40 bits for four sub-pixels can be compressed to 16 bits. As described above, when 4 bits allocated to header information including compression information are added, 40 bits can be compressed to 20 bits, and a compression rate of 50% can be achieved.

[0120] Figure 8 A configuration of a bitstream BS according to an exemplary embodiment of the present disclosure is shown. Figure 2 , Figure 4B and Figure 6B will with Figure 8 Refer to it together.

[0121] Figure 8 The bit stream shown in BS can be used with Figure 6B According to an exemplary embodiment, one sub-pixel may include 10-bit data. A color pixel ( Figure 6B 135) may include a total of 90 bits of data. The present disclosure may relate to a method for compressing 90 bits of data included in nine sub-pixels. Although for ease of explanation, it is shown here that the data included in one sub-pixel is 10 bits, the present disclosure is not limited thereto, and one sub-pixel may include data of various bits (e.g., 8 bits, 11 bits, 12 bits, etc.).

[0122] Camera Module( Figure 2 3) The data space of the bit stream BS can be allocated to compress 90-bit data into 45-bit data. When 90-bit data is compressed into 45-bit data, the compression rate of the data is 50%.

[0123] As described above, according to an exemplary embodiment, 4 bits may be allocated to the header information, and the header information may be a set of bits generated by encoding compressed information including a compression method.

[0124] The camera module 3 may determine the sub-pixel G1a having the median of the sub-pixels as the reference sub-pixel 155", or may determine any one of the predetermined sub-pixels (sub-pixel G2a) in the sub-pixels as the reference sub-pixel 155"'. According to an exemplary embodiment, 5 bits may be allocated to the reference sub-pixel 155" or 155"'. Although the reference sub-pixel 155" or 155"' may include a pixel value of 10 bits, the data space allocated to it is only 5 bits. Therefore, the data space may be insufficient. To solve the shortage of the data space, the encoder 310 may remove a part of the data of the reference sub-pixel 155" or 155"'. According to an exemplary embodiment, the encoder 310 may remove the lower 5 bits from the 10 bits of the reference sub-pixel 155" or 155"', and include only the upper 5 bits in the bitstream BS. As described above, the data of the sub-pixels to be removed is not limited to this.

[0125] According to an exemplary embodiment, 5 bits may be allocated to each of the remaining sub-pixels except for the pixels determined as the reference sub-pixel 155" or 155"'. One color pixel 135" or 135"' may include nine sub-pixels, and in the bitstream BS for transmitting data about one color pixel 135" or 135"', a total of 40 bits of space may be allocated to the remaining eight sub-pixels except for the reference sub-pixel 155" or 155"'.

[0126] According to an exemplary embodiment, when the fifth sub-pixel G5a is determined as the reference sub-pixel 155" or 155"', as a result of the subtraction operation between the reference value Gra and the first sub-pixel G1a, the first residual sub-pixel r1a may be generated. At this time, 5 bits (5BIT_1) may be allocated to the first residual sub-pixel r1a. As a result of the subtraction operation between the reference value Gra and the second sub-pixel G2a, the second residual sub-pixel r2a may be generated, and 4 bits (4BIT_2) may be allocated to the second residual sub-pixel r2a. Optionally, 5 bits (5BIT_3) may be allocated to the third residual sub-pixel r3a, and 4 bits (4BIT_4) may be allocated to the fourth residual sub-pixel r4a, and repeatedly, 4 bits (4BIT_8) may be allocated to the eighth residual sub-pixel r8a, and 5 bits (5BIT_9) may be allocated to the ninth residual sub-pixel r9a.

[0127] As a result, 90 bits of the nine sub-pixels may be compressed to 41 bits. As described above, when adding the 4 bits allocated to the header information including the compressed information, 90 bits may be compressed to 45 bits, and a compression rate of 50% may be achieved.

[0128] Figure 9 FIG. 100c shows a pixel array according to another exemplary embodiment of the present disclosure.

[0129] Assume that sub-pixels 150 are arranged in a 4×4 matrix in pixel array 100c. When the number of sub-pixels is 16, the unit including the sub-pixels may be referred to as a hexadeca cell.

[0130] In an alternative embodiment, the pixel array may be configured in a hybrid octa-cell - tetra-cell layout (such as having green sub-pixels arranged as octa-cells and red and blue sub-pixels having sub-pixels arranged as alternating tetra-cells). In this case, for the pixel pitch P between the octa-cells (e.g., green sub-pixels) and the tetra-cells (e.g., red sub-pixels and blue sub-pixels) distance may be different.

[0131] To compress one sub-pixel 161 (i.e., the target sub-pixel) in the first row of pixel array 100c, the camera module ( Figure 2 3) may search for a reference sub-pixel including substantially the same color information. The pixel having substantially the same color information as the target sub-pixel 161 and closest to the target sub-pixel 161 may be determined as the reference sub-pixel 140. The reference sub-pixel 140 may be the pixel closest to the target sub-pixel 161.

[0132] Since the target sub-pixel 161 is surrounded by red and blue pixels, the distance between the target sub-pixel 161 and the reference sub-pixel 140 may additionally include 4 as the width of one color pixel. As a result, the pixel pitch P between the target sub-pixel 161 and the reference sub-pixel 140 distance may also be 5.

[0133] The 4×4 matrix arrangement may be divided into a plurality of 2×2 matrices. According to an exemplary embodiment, the 4×4 matrix array may be divided into a first partial color pixel 141, a second partial color pixel 143, a third partial color pixel 145, and a fourth partial color pixel 147.

[0134] The plurality of partial color pixels 141, 143, 145, and 147 as 2×2 matrices may be compressed in a manner similar to that of the sub-pixels arranged as a 2×2 matrix described above in Figures 5A to 5C . As a result, the pixel pitch of the pixel array 100c of the 4×4 matrix array may be √2, and the spatial similarity may be higher than the case where the pixel pitch is 5.

[0135] In a similar manner, a P×P matrix arrangement may be divided into a plurality of Q×Q matrix arrangements (where P is a non-prime natural number of 4 or greater, Q is a natural number of 2 or greater, and P>Q).

[0136] Figure 10 and Figure 11 illustrates an image compression method according to an exemplary embodiment of the present disclosure.

[0137] Figure 10 will be referred to in conjunction with Figure 2 、 Figures 5A to 5C and Figures 6A to 6C together.

[0138] In operation S200 after operation S100, a camera module (e.g., Figure 2 the camera module 3) may determine a reference value Gra based on pixel values of a plurality of sub-pixels (operation S200).

[0139] Specifically, a case where the reference value is determined by an average value of a plurality of sub-pixels and a case where the reference value is determined by a pixel value of any one of the sub-pixels may be distinguished.

[0140] In operation S210, when the reference value is determined by an average value of a plurality of sub-pixels, an encoder ( Figure 2 310) may calculate an average pixel value of all sub-pixels (i.e., virtual sub-pixels 151' or 155') (operation S210). The encoder 310 may determine the calculated average value as the reference value Gra (operation S230).

[0141] In operation S250, when the reference value is determined by a pixel value of any one of the sub-pixels, the encoder 310 may determine a pixel value of a first sub-pixel 151”, 151”', 155” or 155”' among the sub-pixels as the reference value (operation S250).

[0142] When the reference value is determined, the method may proceed to operation S300 to compare the pixel value of the sub-pixel with the reference value.

[0143] will be referred to in conjunction with Figure 2 、 Figures 5A to 5C and Figures 6A to 6C together Figure 11 , and repeated descriptions may be omitted. Figure 11 illustrates an operation in which, when the reference value is determined by a pixel value of any one of the sub-pixels, the encoder 310 determines a pixel value of a first sub-pixel 151”, 151”', 155” or 155”' among the sub-pixels as the reference value.

[0144] Specifically, Figure 11 the case illustrated in

[0145] In operation S251, when using the median value, the encoder 310 may sort the pixel values of the sub-pixels according to the magnitudes of the pixel values of the sub-pixels.

[0146] Thereafter, in operation S253, the encoder 310 may determine whether the number of sub-pixels is odd or even.

[0147] In operation S255, when the number of sub-pixels is odd, the sub-pixel having the median value of the pixel values may be identified. For example, when there are nine sub-pixels, the sub-pixel having the pixel value that is the fifth largest (i.e., the fifth smallest) among the nine sub-pixel values may be identified.

[0148] In operation S257, when the number of sub-pixels is even, the sub-pixel having the smaller value of the two median values of the sorted pixel values may be identified. However, the present disclosure is not limited thereto, and the larger value of the two median values may be determined as a reference value according to the design concept of the encoder 310.

[0149] In operation S259, the encoder 310 may determine the identified sub-pixel as the first sub-pixel, and may use the pixel value of the determined first sub-pixel as a reference value.

[0150] In operation S252, when using the sub-pixel at a predetermined position, the encoder 310 may determine the sub-pixel at the predetermined position among the sub-pixels as the first sub-pixel.

[0151] Figure 12A and Figure 12B FIGS. 3A and 3B illustrate camera modules 3a and 3b according to exemplary embodiments of the present disclosure.

[0152] Referring Figure 12A , the camera module 3a may include an image sensor 10a, an ISP 30a, and a Mobile Industry Processor Interface (MIPI) interface 40a. The ISP 30a may include an encoder 310a.

[0153] As a result of optically sensing an object, the image sensor 10a may provide image data IDTA to the ISP 30a. The ISP 30a may transmit a bitstream BS through the MIPI interface 40a. According to an exemplary embodiment, the bitstream BS may be provided to the AP 50.

[0154] The MIPI interface 40a can interface between the ISP 30a and the AP 50. Specifically, the MIPI interface 40a can represent a physical protocol or specification that facilitates data transmission and / or signal transmission between the ISP 30a and the AP 50 with different specifications and configurations. The MIPI interface 40a can adopt any channel specification determined by an organization of product manufacturers (such as the MIPI Alliance interface, which is a common interface specification elaborated by mobile device manufacturers). However, the present disclosure is not limited thereto, and there can be various arbitrary signal input / output protocols implemented instead of or in addition to such a MIPI interface.

[0155] Although Figure 12A it is shown that the MIPI interface 40a is included in the camera module 3a, due to the nature of the interface device, it can be understood that the MIPI interface 40a is also included in the AP 50 for receiving the bit stream (BS).

[0156] Referring to Figure 12B , the camera module 3b can include an image sensor 10b, an ISP 30b, and a MIPI interface 40b, and the ISP 30b can include an encoder 310b. The camera module 3b can also include a memory buffer 20b. Since the image sensor 10b, the ISP 30b, and the MIPI interface 40b are similar to Figure 12A the image sensor 10a, the ISP 30a, and the MIPI interface 40a, the same description as that given above can be omitted.

[0157] The memory buffer 20b can be used as a buffer for receiving the image data IDTA and temporarily storing at least a part of the image data IDTA.

[0158] According to an exemplary embodiment, the memory buffer 20b can buffer data corresponding to the first row of the image data IDTA. Sub-pixels can be arranged in a matrix form, and the encoder 310b can compress the data only after all pixel information of the color pixels including the sub-pixels is provided. Therefore, it is not necessary to immediately process the data corresponding to the first row of the image data IDTA.

[0159] When the data corresponding to the second row of the image data IDTA is provided, the memory buffer 20b can load the buffered image data BDTA of the first row into the ISP 30b. The encoder 310b can compress the data corresponding to the color pixels based on the loaded buffered image data BDTA and the provided data corresponding to the second row. The encoded image data EDTA as the compressed data can be stored in the memory buffer 20b.

[0160] The bitstream BS generated as a result of compression can be provided to the MIPI interface 40b. The data generated as a result of encoding can be directly provided from the ISP 30b to the MIPI interface 40b, or can be provided from the memory buffer 20b to the MIPI interface 40b based on the encoded image data EDTA stored in the memory buffer 20b.

[0161] The camera module 3b can buffer the data corresponding to the first row of the image data IDTA without immediately processing the data, and when reading the data corresponding to the second row of the image data IDTA, compress the data corresponding to the first row of the image data IDTA and the data corresponding to the second row of the image data IDTA at once. Therefore, the speed for compressing the image data IDTA can be increased. In addition, since there are rows that are not immediately compressed and buffered, the power consumption can be reduced compared to the case of compressing all rows.

[0162] The memory buffer 20b can be implemented using a volatile memory or a non-volatile memory. The volatile memory can include a dynamic random access memory (DRAM), a static RAM (SRAM), etc., and the non-volatile memory can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc.

[0163] The structure in which the image sensor 10b, the ISP 30b, and the memory buffer 20b are included in the camera module 3b can be referred to as a 3-stacked-layer.

[0164] Figure 13 An AP 50 according to an exemplary embodiment of the present disclosure is shown.

[0165] In addition to the decoder 510, the AP 50 may further include a memory 530.

[0166] The memory 530 can be implemented using a volatile memory or a non-volatile memory. The volatile memory can include a dynamic random access memory (DRAM), a static RAM (SRAM), etc., and the non-volatile memory can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc.

[0167] The AP 50 may temporarily store the provided bitstream BS in the memory 530. In addition, the AP 50 may directly provide the bitstream BS to the decoder 510.

[0168] According to an exemplary embodiment, the bitstream corresponding to the first row of the image data IDTA among the bitstreams BS may be stored in the memory 530, and the bitstream BS corresponding to the second row of the image data may be provided to the decoder 510.

[0169] When the decoder 510 processes the bitstream corresponding to the second row of the image data IDTA among the bitstreams BS, the bitstream LBS corresponding to the first row stored in the memory 530 may be loaded. The decoder 510 may output RIDTA as the decoding result.

[0170] The AP 50 may buffer the data corresponding to the first row of the image data IDTA without immediately decoding the data, and when decoding the data corresponding to the second row of the image data IDTA, decode the data corresponding to the first row of the image data IDTA and the data corresponding to the second row of the image data IDTA at one time. Therefore, the speed of decoding the image data IDTA may be increased.

[0171] Figure 14 An image compression method according to an exemplary embodiment of the present disclosure is shown.

[0172] The operation S100 for classifying a plurality of sub-pixels including first color information into first color pixels may be subdivided.

[0173] The values of the sub-pixels located in the first row among the sub-pixels may be buffered (operation S110). Thereafter, the values of the sub-pixels located in the second row among the sub-pixels may be read (operation S130), and based on the values of the sub-pixels located in the second row and the buffered values of the sub-pixels located in the first row, the image data IDTA may be compressed.

[0174] Thereafter, the method may proceed to operation S200 to determine a reference value based on the sub-pixels.

[0175] Figure 15A and Figure 15B Electronic devices 10000a and 10000b according to an exemplary embodiment of the present disclosure are shown.

[0176] Refer to Figure 15A, the electronic device 10000a according to an exemplary embodiment of the present disclosure may include an image sensor 1000a, an ISP 2000a, a display device 3000a, an AP 4000a, a working memory 5000a, a storage device 6000a, a user interface 7000a, and a wireless transceiver 8000a. Among them, the ISP 2000a may be implemented as an integrated circuit separate from the AP 4000a. Figure 2 The image sensor 10 can be used as Figure 15A the image sensor 1000a, and Figure 2 the ISP 30 can be applied as Figure 15A the ISP 2000a.

[0177] The image sensor 1000a may generate image data (such as raw image data) based on the received optical signal and provide binary data to the ISP 2000a. The AP 4000a controls the overall operation of the electronic device 10000a and may be set as a system-on-chip (SoC) that drives application programs, operating systems, etc. The AP 4000a may control the operation of the ISP 2000a and provide the converted image data generated by the ISP 2000a to the display device 3000a, or store the converted image data in the storage device 6000a.

[0178] The memory 5000a may store programs and / or data processed or executed by the AP 4000a. The storage device 6000a may be implemented using a non-volatile memory device (such as NAND flash or resistive memory). For example, the storage device 6000a may be set as a memory card (MMC, eMMC, SD, micro SD, etc.). The storage device 6000a may store data and / or programs regarding the execution algorithm for controlling the image processing operation of the ISP 2000a, and when the image processing operation is executed, the data and / or programs may be loaded into the working memory 5000a.

[0179] The user interface 7000a may be implemented using various devices capable of receiving user input (such as a keyboard, a curtain key panel, a touch panel, a fingerprint sensor, a microphone, etc.). The user interface 7000a may receive user input and provide a signal corresponding to the received user input to the AP 4000a. The wireless transceiver 8000a may include a modem 8100a, a transceiver 8200a, and an antenna 8300a.

[0180] Refer to Figure 15B, the electronic device 10000b according to an exemplary embodiment of the present disclosure may include an image sensor 1000b, an ISP 2000b, a display device 3000b, an AP 4000b, a working memory 5000b, a storage device 6000b, a user interface 7000b, and a wireless transceiver 8000b, where Figure 2 the image sensor 10 may be used as Figure 15B the image sensor 1000b, and Figure 2 the ISP 30 may be applied as Figure 15B the ISP 2000b.

[0181] The AP 4000b may include the ISP 2000b. The ISP 2000b may be set as a sub-component of the AP 4000b, rather than being configured as a separate hardware or a combination of hardware and software. The wireless transceiver 8000b may include a modem 8100b, a transceiver 8200b, and an antenna 8300b. Figure 15B the other components of Figure 15A are similar to those components of

[0182] Figure 16 and thus the same description as that given above may be omitted. Figure 17 shows Figure 16 a camera module of Figure 16 For the sake of convenience of explanation, Figure 15A a part of the electronic device 10000a shown in Figure 15B or the electronic device 10000b shown in Figure 15A is shown as the electronic device 20000, but Figure 16 the components omitted in

[0183] or 15B may be included in the electronic device 20000 and implement the present disclosure as a whole. Figure 16 Referring to Figure 15A the memory 5000a shown in Figure 15B or the memory 5000b shown in

[0184] The electronic device 20000 can capture and / or store an image of an object by using a CMOS image sensor, and can be implemented as a mobile phone, a tablet computer, or a portable electronic device. The portable electronic device may include a laptop computer, a mobile phone, a smart phone, a tablet PC, a wearable device, and the like. The electronic device 20000 may include one or more camera modules and an AP that processes image data generated by the one or more camera modules.

[0185] The multi-camera module 1100 may include a first camera module 1100a, a second camera module 1100b, and a third camera module 1100c. The multi-camera module 1100 may perform the same functions as the Figure 2 camera module 3. Although three camera modules 1100a to 1100c are shown for ease of explanation, the present disclosure is not limited thereto, and various numbers of camera modules may be included in the multi-camera module 1100.

[0186] Hereinafter, the detailed configuration of the second camera module 1100b will be described in more detail with reference to Figure 17 , but the following description may also be applied to other camera modules 1100a and 1100c according to embodiments.

[0187] Referring to Figure 17 , the second camera module 1100b may include a prism 1105, an optical path folding element (hereinafter, "OPFE") 1110, an actuator 1130, an image sensing device 1140, and a storage device 1150.

[0188] The prism 1105 may include a reflective surface 1107 of a light-reflective material to modify the path of light L incident from the outside.

[0189] According to an exemplary embodiment, the prism 1105 may change the path of light L incident in a first direction X to a second direction Y perpendicular to the first direction X. In addition, the prism 1105 may rotate the reflective surface 1107 of the light-reflective material about a central axis 1106 in the A direction or the B direction, thereby changing the path of light L incident in the first direction X to a second direction Y perpendicular to the first direction X. At this time, the OPFE 1110 may also move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0190] In one exemplary embodiment, as shown in Figure 17 , the maximum rotatable angle of the prism 1105 in the A direction may be less than or equal to 15 degrees in the positive (+) A direction and greater than 15 degrees in the negative (-) A direction. However, the embodiments are not limited thereto.

[0191] In an exemplary embodiment, the prism 1105 may be rotated about 20 degrees in the positive (+) or negative (-) B direction, or rotated between 10 degrees and 20 degrees, or rotated between 15 degrees and 20 degrees. Here, the prism 1105 may be rotated the same angle or similar angles differing from each other by about 1 degree in the positive (+) B direction and the negative (-) B direction.

[0192] In an exemplary embodiment, the prism 1105 may move the reflective surface 1106 of the light reflecting material in a third direction (e.g., the Z direction) parallel to the direction in which the central axis 1106 extends.

[0193] For example, the OPFE 1110 may include optical lenses including m groups (where m is a natural number greater than 0). The m lenses may move in the second direction Y and change the optical zoom ratio of the camera module 1100b. For example, when the basic optical zoom ratio of the camera module 1100b is Z and the m optical lenses included in the OPFE 1110 move, the optical zoom ratio of the camera module 1100b may be changed to 3Z, 5Z, or an optical zoom ratio higher than 5Z.

[0194] The actuator 1130 may move the OPFE 1110 or the optical lens (hereinafter referred to as the optical lens) to a specific position. For example, the actuator 1130 may adjust the position of the optical lens so that the image sensor 1142 is positioned at the focal length of the optical lens for accurate sensing.

[0195] The image sensing device 1140 may include an image sensor 1142, a control logic 1144, an encoder 1145, and a memory 1146. The image sensor 1142 may sense an image of a sensing target by using the light L provided through the optical lens. Since Figure 17 the image sensor 1142 may be functionally similar to Figure 2 the image sensor 10, the description same as that already given above may be omitted. The control logic 1144 may control the overall operation of the second camera module 1100b. For example, the control logic 1144 may control the operation of the second camera module 1100b according to a control signal provided through the control signal line CSLb.

[0196] The encoder 1145 may encode the sensed image data. Since Figure 17 the encoder 1145 may perform a function similar to that of Figure 2 the encoder 310, the description same as that already given above may be omitted. Different from the encoder 310 shown in Figure 2 the encoder 1145 shown in Figure 17 may not be included in the ISP ( Figure 2in 30), and may be included in a camera module (e.g., 1100b). Although, for ease of explanation, the encoder 1145 is shown as a separate functional unit separated from other functional units, the present disclosure is not limited thereto, and the encoder 1145 may be included in the control logic 1144 to compress and encode image data.

[0197] The memory 1146 may store information required for the operation of the second camera module 1100b (e.g., calibration data 1147). The calibration data 1147 may include information required for the second camera module 1100b to generate image data by using the light L provided from the outside. The calibration data 1147 may include, for example, information about the degree of rotation described above, information about the focal length, information about the optical axis, etc. When the second camera module 1100b is implemented in the form of a multi-state camera in which the focal length changes according to the position of the optical lens, the calibration data 1147 may include the focal length values of the respective positions or respective states of the optical lens and information related to autofocus.

[0198] The storage device 1150 may store the image data sensed by the image sensor 1142. The storage device 1150 may be provided outside the image sensing device 1140 and may be stacked with the sensor chip constituting the image sensing device 1140. In one exemplary embodiment, the storage device 1150 may be implemented using an EEPROM, but the embodiment is not limited thereto.

[0199] Referring together Figure 16 and Figure 17 , in one exemplary embodiment, one of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., the first camera module 1100a) may include four sub-pixels that are adjacent to each other and share the same color information in one color pixel (i.e., a four-unit pixel), and another camera module (e.g., the second camera module 1100b) may include nine sub-pixels that are adjacent to each other and share the same color information in one color pixel (i.e., a nine-unit pixel). However, the embodiment is not limited thereto.

[0200] In one exemplary embodiment, the camera modules 1100a, 1100b, and 1100c may all include an actuator 1130. Accordingly, the camera modules 1100a, 1100b, and 1100c may include the same or different calibration data 1147 according to the operation of the actuator 1130 included therein.

[0201] In one exemplary embodiment, one of the camera modules 1100a, 1100b, and 1100c (e.g., the second camera module 1100b) may be a folded lens type camera module including the prism 1105 and the OPFE 1110 as described above, and the other camera modules (e.g., 1100a and 1100c) may be vertical camera modules without the prism 1105 and the OPFE 1110. However, the embodiments are not limited thereto.

[0202] In one exemplary embodiment, for example, one of the camera modules 1100a, 1100b, and 1100c (e.g., the third camera module 1100c) may be a vertical depth camera that extracts depth information by using infrared rays (IR). In this case, the AP 4000 may generate a 3D depth image by combining the image data provided from such a depth camera with the image data provided from another camera module (e.g., the first camera module 1100a or the second camera module 1100b).

[0203] In one exemplary embodiment, at least two of the camera modules 1100a, 1100b, and 1100c (e.g., the first camera module 1100a and the second camera module 1100b) may have different fields of view (FOVs). In this case, for example, at least two of the camera modules 1100a, 1100b, and 1100c (e.g., the first camera module 1100a and the second camera module 1100b) may have different optical lenses, but the present disclosure is not limited thereto. For example, the first camera module 1100a among the camera modules 1100a, 1100b, and 1100c may have a smaller FOV than the second camera module 1100b and the third camera module 1100c. However, the present disclosure is not limited thereto, and the multi-camera module 1100 may further include a camera module having a larger FOV than the initially used camera modules 1100a, 1100b, and 1100c.

[0204] In addition, in some embodiments, the camera modules 1100a, 1100b, and 1100c may have different FOVs from each other. In this case, the optical lenses included in the camera modules 1100a, 1100b, and 1100c may also be different from each other, but the present disclosure is not limited thereto.

[0205] In some embodiments, the camera modules 1100a, 1100b, and 1100c may be physically separated from each other. In other words, the camera modules 1100a, 1100b, and 1100c do not divide and use the sensing area of one image sensor 1142. Instead, independent image sensors 1142 may be provided inside each of the camera modules 1100a, 1100b, and 1100c.

[0206] The AP 4000 may include a plurality of sub-processors 4100a, 4100b, and 4100c, a decoder 4200, a camera module controller 4300, a memory controller 4400, and an internal memory 4500.

[0207] The AP 4000 may be implemented separately from the camera modules 1100a, 1100b, and 1100c. For example, the AP 4000 and the camera modules 1100a, 1100b, and 1100c may be implemented as separate semiconductor chips separately from each other.

[0208] The image data generated by the camera modules 1100a, 1100b, and 1100c may be provided to the corresponding sub-processors 4100a, 4100b, and 4100c respectively through separate image signal lines ISLa, ISLb, and ISLc. For example, the image data generated by the first camera module 1100a may be provided to the first sub-processor 4100a through the first image signal line ISLa, the image data generated by the second camera module 1100b may be provided to the second sub-processor 4100b through the second image signal line ISLb, and the image data generated by the third camera module 1100c may be provided to the third sub-processor 4100c through the third image signal line ISLc. The transmission of the image data may be performed by using a MIPI-based camera serial interface, but the embodiments are not limited thereto.

[0209] In one exemplary embodiment, one sub-processor may be set to correspond to a plurality of camera modules. For example, the first sub-processor 4100a and the third sub-processor 4100c may be integrally implemented as a single sub-processor instead of being implemented separately from each other, and the image data provided from the first camera module 1100a and the third camera module 1100c may be selected by a selection element (e.g., a multiplexer) and provided to the integrated sub-image processor.

[0210] The decoder 4200 may decode the bitstreams provided to the sub-processors 4100a, 4100b, and 4100c. Since the decoder 4200 in FIG. 15 may perform a function similar to that of the Figure 2 decoder 510, the same description as that already given above may be omitted. Although for the sake of convenience of explanation, Figure 16 and Figure 17 show that the decoder 4200 and the sub-processors 4100a, 4100b, and 4100c are separate functional units, the present disclosure is not limited thereto, and the decoder 4200 may be included in each of the sub-processors 4100a, 4100b, and 4100c according to the embodiments. In other words, the decoder 4200 may decode the bitstreams inside the sub-processors 4100a, 4100b, and 4100c.

[0211] The camera module controller 4300 can provide control signals to each of the camera modules 1100a, 1100b, and 1100c. The control signals generated from the camera module controller 4300 can be provided to the corresponding camera modules 1100a, 1100b, and 1100c through the separate control signal lines CSLa, CSLb, and CSLc.

[0212] Any one of the camera modules 1100a, 1100b, and 1100c is designated as a main camera (e.g., 1100b) according to the image generation information or a mode signal including a zoom signal, and the remaining camera modules (e.g., 1100a and 1100c) can be designated as slave cameras. This information is included in the control signal and can be provided to the corresponding camera modules 1100a, 1100b, and 1100c through the separate control signal lines CSLa, CSLb, and CSLc.

[0213] Under the control of the camera module controller 4300, the camera modules 1100a, 1100b, and 1100c serving as the main camera and the slave cameras can be changed. For example, when the FOV of the first camera module 1100a is wider than the FOV of the second camera module 1100b and the zoom factor of the first camera module 1100a indicates a lower zoom ratio, the second camera module 1100b can be used as the main camera, and the first camera module 1100a can be used as the slave camera. On the contrary, when the zoom factor of the first camera module 1100a indicates a higher zoom magnification, the first camera module 1100a can be used as the main camera, and the second camera module 1100b can be used as the slave camera.

[0214] In an exemplary embodiment, the control signals provided from the camera module controller 4300 to the camera modules 1100a, 1100b, and 1100c can include a synchronization enable signal. For example, when the second camera module 1100b is the main camera and the first camera module 1100a and the third camera module 1100c are slave cameras, the camera module controller 4300 can transmit the synchronization enable signal to the second camera module 1100b. The second camera module 1100b provided with the synchronization enable signal generates a synchronization signal based on the provided synchronization enable signal and provides the generated synchronization signal to the first camera module 1100a and the third camera module 1100c through the synchronization signal line SSL. The first camera module 1100b and the second and third camera modules 1100a and 1100c can be synchronized using the synchronization signal and transmit the image data to the AP 4000.

[0215] The AP 4000 can store the encoded image data in the internal memory 4500 or the memory 5000 external to the AP 4000, read the encoded image signal from the internal memory 4500 or the memory 5000, decode the encoded image signal, and display the image data generated based on the decoded image signal. The memory controller 4400 can overall control the internal memory 4500 and the memory 5000 such that the image data can be stored or loaded between the internal memory 4500 and the memory 5000.

[0216] Although the present disclosure has been specifically shown and described with reference to exemplary embodiments thereof, those of ordinary skill in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of the inventive concept as set forth in the claims.

Claims

1. An image compression method for compressing image data generated by a pixel array, the method comprises: classifying the image data corresponding to a plurality of sub-pixels that are arranged adjacent to each other and generate first color information as first color pixels; determining a reference value as a criterion for compressing the image data based on the pixel values of at least one of the plurality of sub-pixels; and comparing the pixel value of each of the plurality of sub-pixels with the reference value, and outputting a comparison result.

2. The method according to claim 1, wherein, the step of determining the reference value comprises: calculating an average pixel value of the pixel values of the plurality of sub-pixels, and determining the average pixel value as the reference value.

3. The method according to claim 1, wherein, the step of determining the reference value comprises: determining the pixel value of a first sub-pixel among the plurality of sub-pixels as the reference value.

4. The method according to claim 3, wherein, the step of determining the pixel value of the first sub-pixel as the reference value comprises: sorting the pixel values of the plurality of sub-pixels according to the pixel values of the plurality of sub-pixels; and determining the pixel value of the first sub-pixel having the median of the pixel values of the plurality of sub-pixels as the reference value.

5. The method according to claim 4, wherein, in response to the number of the plurality of sub-pixels being even, the first sub-pixel is the sub-pixel having the smaller pixel value among the two medians of the sorted pixel values.

6. The method according to claim 3, wherein, the first sub-pixel is the sub-pixel arranged at a predetermined position within a pixel unit among the plurality of sub-pixels.

7. The method according to any one of claims 1 to 6, wherein, the step of classifying the image data as first color pixels comprises: buffering the pixel value of each of the plurality of sub-pixels located in the first row; and receiving the image data corresponding to the sub-pixels located in the second row among the plurality of sub-pixels.

8. The method according to any one of claims 1 to 6, wherein, the number of the plurality of sub-pixels is four, and the plurality of sub-pixels are arranged in a matrix form in the first color pixel.

9. The method according to any one of claims 1 to 6, wherein, the number of the plurality of sub-pixels is nine, and the plurality of sub-pixels are arranged in a matrix form in the first color pixel.

10. The method according to any one of claims 1 to 6, wherein, the first color information indicates any one of red, green, and blue.

11. The method according to claim 1, wherein, the step of comparing the pixel value of each of the plurality of sub-pixels with the reference value comprises: calculating the difference between the reference value and the pixel value of each of the plurality of sub-pixels; outputting each difference as a comparison result; and outputting the reference value.

12. An encoder for compressing image data generated by a pixel array, wherein, the encoder is configured to: classify the image data corresponding to a plurality of sub-pixels that are arranged adjacent to each other and generate first color information as first color pixels; Determine a reference value as a standard for compressing image data based on the pixel value of each of the plurality of sub-pixels; Compare the pixel value of each of the plurality of sub-pixels with the reference value, and output the comparison result and the reference value.

13. The encoder according to claim 12, wherein, The reference value is the average value of the pixel values of the plurality of sub-pixels.

14. The encoder according to claim 12 or 13, wherein, The first color pixels are arranged in a 2×2 matrix, and The reference value is the second smallest pixel value among the pixel values of the plurality of sub-pixels.

15. The encoder according to claim 12 or 13, wherein, The first color pixels are arranged in a 3×3 matrix, and The reference value is the median value among the pixel values of the plurality of sub-pixels.

16. The encoder according to claim 12, wherein, The reference value is the pixel value of the sub-pixel arranged at a predetermined position among the plurality of sub-pixels.

17. A camera module, comprising: An image sensor, including a pixel array having a plurality of pixels arranged in a matrix and generating pixel signals respectively, the image sensor being configured to convert light into an electrical signal by using a photoelectric conversion element and output image data; and An encoder, configured to output a bitstream generated by compressing image data, wherein the encoder classifies a plurality of sub-pixels arranged adjacent to each other and generating first color information as first color pixels, determines a reference value as a standard for compressing image data based on the pixel value of each of the plurality of sub-pixels, compares the pixel value of each of the plurality of sub-pixels with the reference value, and outputs the comparison result and the reference value.

18. The camera module according to claim 17, wherein, The reference value is any one of the average value and the median value of the pixel values of the plurality of sub-pixels.

19. The camera module according to claim 17, wherein, The reference value is the pixel value of the sub-pixel arranged at a predetermined position among the plurality of sub-pixels.

20. The camera module according to any one of claims 17 to 19 further includes a memory, wherein, The memory buffers image data corresponding to the sub-pixels located in the first row among the plurality of sub-pixels, and wherein the encoder receives image data corresponding to the sub-pixels located in the second row among the plurality of sub-pixels, and compresses the image data based on the reception result and the buffer result.

Images