Encoder, decoder, and image processing system

Abstract

This invention provides an encoder, decoder, and image processing system that improve the compression or decompression efficiency of HDR (High Dynamic Range) image data. [Solution] An encoder 120 for compressing image data generated by an image sensor, comprising: a bit depth control circuit that generates first image data based on a first plurality of bits extracted from the image data using at least one encoding scheme, and generates second image data based on a second plurality of bits corresponding to the remaining image data that is different from the first image data; an encoding circuit that compresses the first image data to generate partially compressed data; a register for storing the second image data; and a packing circuit that packs the second image data into the partially compressed data to generate compressed data and outputs the compressed data.

Description

Technical Field

[0001] The present invention relates to an encoder, a decoder, and an image processing system for compressing and decompressing image data.

Background Art

[0002] Recently, due to the increasing demand for high-quality and high-definition photos, images, etc., the number of sensing pixels in the pixel array of an image sensor has been increasing. As a result, the bit-depth of the image data generated by the image sensor has also been increasing. The image data is transmitted to an image processing device, and at this time, the image data is compressed by an encoder in order to improve the transmission efficiency. The image processing device performs various image processes after decompressing the compressed image data using a decoder. With the development of HDR (High Dynamic Range) technology, an encoder, a decoder, and an image processing system including the same are required to efficiently process high-bit-depth image data.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The present invention has been made in view of the above prior art, and an object of the present invention is to provide an encoder, a decoder, and an image processing system including the same for efficiently processing high-bit-depth image data (e.g., HDR image data).

Means for Solving the Problems

[0004] An encoder for compressing image data generated by an image sensor according to one aspect of the present invention, made to achieve the above objective, comprises: a bit depth control circuit that generates first image data based on a first plurality of bits extracted from the image data using at least one encoding scheme, and generates second image data based on a second plurality of bits corresponding to the remaining image data of the image data that is different from the first image data; an encoding circuit that compresses the first image data to generate partially compressed data; a register for storing the second image data; and a packing circuit that packs the second image data into the partially compressed data to generate compressed data and outputs the compressed data.

[0005] In one embodiment, an encoder for compressing image data generated by an image sensor includes: a bit depth control circuit that generates first image data based on a first plurality of bits extracted from the image data using at least one encoding scheme, and generates second image data based on a portion of the remaining bits of the image data excluding the first image data; an encoding circuit that compresses the first image data to generate partially compressed data; a register for storing the second image data; and a packing circuit that packs the second image data into the partially compressed data to generate compressed data and outputs the compressed data.

[0006] A decoder for decompressing compressed data according to one aspect of the present invention, made to achieve the above objective, comprises: a bit depth control circuit that separates the compressed data into partially compressed data and the remaining compressed data based on the compressed data using at least one decoding scheme; a register for storing the remaining compressed data; a decoding circuit that decompresses the partially compressed data to generate partially decompressed data; and a dithering circuit that generates random data, packs a portion of the remaining compressed data into the partially decompressed data based on the at least one decoding scheme to generate packing data, and packs the random data into the packing data to generate decompression data.

[0007] An image processing system according to one aspect of the present invention, made to achieve the above objective, comprises: an image sensor that senses a received optical signal and generates image data; an encoder that sequentially compresses a plurality of pixel groups included in the image data to generate a plurality of bitstreams; and a decoder that decompresses the plurality of bitstreams to restore the image data, wherein the encoder generates first image data and second image data based on the image data using at least one encoding scheme, compresses the first image data to generate partially compressed data, and packs the second image data into the partially compressed data to generate the plurality of bitstreams. [Effects of the Invention]

[0008] According to the encoder, decoder, and image processing system of the present invention, the compression efficiency or decompression efficiency of HDR image data can be improved by performing an encoding operation or a decoding operation based on partially compressed data or partially decompressed data generated by compressing or decompressing at least a portion of the image data.

[0009] According to the encoder, decoder, and image processing system of the present invention, by improving the compression or decompression efficiency of HDR image data, it becomes possible to compress or decompress HDR image data without additional hardware logic enhancements (e.g., adding an encoding / decoding IP (intellectual property), increasing the IP size, etc.), and furthermore, it is possible to reduce the power consumption for compressing or decompressing HDR image data. [Brief explanation of the drawing]

[0010] [Figure 1A] This figure shows an example of the compression operation of HDR image data in the image processing system according to the first comparative embodiment. [Figure 1B] This figure shows an example of the compression operation of HDR image data in the image processing system according to the second comparative embodiment. [Figure 2A] This is an image processing system according to one embodiment of the present invention. [Figure 2B] This figure illustrates a pixel array and image data applied to an image sensor module according to one embodiment of the present invention. [Figure 3] This is a block diagram of an encoder based on one embodiment of the present invention. [Figure 4] This is a block diagram showing a decoder according to one embodiment of the present invention. [Figure 5] This diagram illustrates the operation of an encoder of a first encoding scheme according to one embodiment of the present invention. [Figure 6] This diagram illustrates the operation of a decoder using a first decoding method according to one embodiment of the present invention. [Figure 7] This diagram illustrates the operation of an encoder of a second encoding scheme according to one embodiment of the present invention. [Figure 8] This diagram illustrates the operation of a decoder using a second decoding method according to one embodiment of the present invention. [Figure 9]It is a diagram for explaining the operation of an encoder of a third encoding method according to an embodiment of the present invention. [Figure 10] It is a diagram for explaining the operation of a decoder of a third decoding method according to an embodiment of the present invention. [Figure 11] It is a block diagram of an electronic device including a multi-camera module according to an embodiment of the present invention. [Figure 12] It is a detailed block diagram of a camera module according to an embodiment of the present invention. [Figure 13] It is a block diagram schematically showing an electronic device according to an embodiment of the present invention.

Mode for Carrying Out the Invention

[0011] Hereinafter, specific examples of embodiments for carrying out the present invention will be described in detail with reference to the drawings.

[0012] Embodiments of the present invention are illustrated in the drawings and detailed descriptions related thereto are provided, but this is not intended to limit various embodiments of the present invention to a specific form. For example, it is obvious to those skilled in the art that embodiments of the present invention can be variously modified.

[0013] The expression "at least one of" used in this specification means that it modifies the entire list of elements listed after the expression and does not modify only individual elements in the list. For example, the expression "at least one of A, B, and C" should be interpreted to include cases where only A is included, cases where only B is included, cases where only C is included, cases where A and B are included, cases where A and C are included, cases where B and C are included, or cases where all of A, B, and C are included.

[0014] FIGS. 1A and 1B are diagrams showing an example of a compression operation of HDR image data of an image processing system according to a comparative embodiment.

[0015] Referring to Figure 1A, the encoder of the image processing system according to the first comparative example compresses image data 101 having a size of, for example, 40 bits, with a compression ratio of 50% to generate compressed data 102 having a size of, for example, 20 bits. The image data 101 includes the pixel values ​​of multiple pixels (e.g., pixel G0, pixel R1, pixel G2, pixel R3). For example, the compressed data 102 (e.g., 20 bits) includes, for example, a 4-bit header 103 and, for example, a 16-bit payload 104. The header 103 includes information about the compression method of the image data 101, and the payload 104 includes information about the compressed pixel values ​​of multiple pixels (e.g., pixel G0, pixel R1, pixel G2, pixel R3) (also called pixel data). According to one embodiment, the compression method refers to the type of compression used when generating or acquiring compressed data, and may also be called a compression process, compression scheme, or compression technique. Within the payload 104, space is allocated for each pixel to store, for example, compressed pixel data of 4 bits in size. The bit depth of the image data 101 in Figure 1A is 10 bits. For example, each pixel included in the pixel data has a size of 10 bits, and the decoder according to the first comparative example obtains a 16-bit payload 104 by compressing each pixel to a size of 4 bits using lossy compression.

[0016] Referring to FIG. 1B, the encoder of the image processing system according to the second comparative example compresses image data 105 having, for example, a 48-bit size at a compression rate of 50% to generate compressed data 106 having, for example, a 24-bit size. The image data 105 consists of pixel values of a plurality of pixels (for example, pixel G0, pixel R1, pixel G2, pixel R3). For example, the compressed data 106 (for example, 24 bits) includes a header 107 having a size of, for example, 4 bits and a payload 108 having a size of, for example, 20 bits. The header 107 includes information regarding the compression method of the image data 105, and the payload 108 includes information regarding the compressed pixel values (also referred to as pixel data) of a plurality of pixels (for example, pixel G0, pixel R1, pixel G2, pixel R3). In the payload, for each pixel, a space for storing compressed pixel data having a size of, for example, 5 bits is allocated. The bit depth of the image data 105 in FIG. 1B is 12 bits. For example, each pixel included in the pixel data has a size of 10 bits, and the decoder according to the second comparative example performs lossy compression to compress each pixel to a 5-bit size to obtain a 20-bit size payload 108.

[0017] Comparing and referring to the first comparative example and the second comparative example, it can be seen that although the size of one pixel data has increased by 2 bits from 10 bits to 12 bits, the payload space of the compressed data CDT has only increased by 1 bit from 4 bits to 5 bits. In other words, even though the bit depth has increased by 2 bits and the amount of information of the pixel data that can be represented (for example, pixel code value) has increased by four times, the space of the payload for storing this has only doubled. The encoder according to the comparative embodiment has a problem that it has to perform lossy compression with a larger amount of image quality degradation. Furthermore, with the development of recent HDR (High Dynamic Range) technology, when compressing image data with a higher bit depth (for example, 14 bits, etc.), there is a problem that the problems of image quality degradation and reduction in compression efficiency due to the above problems become more serious.

[0018] Accordingly, the present invention discloses an encoder, decoder, and an image processing system including the same that efficiently compress (or decompress) high-bit-depth image data without excessive configuration changes (e.g., the addition of new logic or an increase in the size of existing logic). A specific explanation of this will be given later in Figures 2 to 13.

[0019] Figure 2A is a block diagram showing an image processing system according to one embodiment of the present invention, and Figure 2B is a diagram illustrating a pixel array and image data applied to an image sensor module according to one embodiment of the present invention.

[0020] The image processing system 10 senses an image related to a subject, processes the sensed image or stores it in memory, and stores the processed image in memory. According to one embodiment, the image processing system 10 is embodied in a digital camera, digital camcorder, mobile phone, tablet computer, or portable electronic device. Portable electronic devices include laptop computers, mobile phones, smartphones, tablet PCs, PDAs (personal digital assistants), EDAs (enterprise digital assistants), digital still cameras, digital video cameras, audio equipment, PMPs (portable multimedia players), PNDs (personal navigation devices), MP3 players, handheld game consoles, e-books, wearable devices, etc. The electronic device 10 is also mounted as a component in electronic devices such as drones and advanced driver assistance systems (ADAS), or in vehicles, furniture, manufacturing equipment, doors, various measuring instruments, etc.

[0021] Referring to Figure 2A, the image processing system 10 comprises an image sensor module 100 and an image processing device 200. In this embodiment, the image sensor module 100 comprises an image sensor 110, an encoder 120, and an interface (I / F) 130. In one embodiment, the image sensor module 100 is realized by multiple semiconductor chips. For example, the pixel array of the image sensor 110 (PXA in Figure 2B) is integrated on one semiconductor chip, and the logic circuit of the image sensor 110, the encoder 120, and the interface 130 are integrated on other semiconductor chips, with the multiple semiconductor chips electrically connected via connecting members, or the multiple semiconductor chips are stacked and electrically connected to each other via through vias. However, it is not limited to this, and the image sensor module 100 may be realized on a single semiconductor chip.

[0022] In this embodiment, the image processing device 200 includes an interface (I / F) 210, a memory 220, a decoder 230, and an image signal processor 240.

[0023] The image sensor module 100 captures an external subject (or object) and generates image data IDT. The image sensor module 100 includes an image sensor 110 that converts the optical signal of the subject incident through the lens LS into an electrical signal.

[0024] The image sensor 110 includes a pixel array (PXA in Figure 2B) in which multiple sensing pixels (SPX in Figure 2B) are arranged two-dimensionally, and outputs image data IDT which includes multiple pixel values ​​corresponding to each of the multiple sensing pixels SPX in the pixel array PXA.

[0025] The pixel array PXA comprises multiple row lines, multiple column lines, and multiple sensing pixels SPX, each connected to the row and column lines and arranged in a matrix.

[0026] Each of the multiple sensing pixels SPX in the pixel array PXA senses an optical signal of at least one of several reference colors. For example, the multiple reference colors may include red, green, and blue, or red, green, blue, and white, but may also include other colors. For example, the multiple reference colors may include cyan, yellow, green, and magenta. The pixel array PXA generates a pixel signal that contains information about each of the multiple sensing pixels SPX's reference colors.

[0027] For example, as shown in Figure 2B, the pixel array PXA comprises a red sensing pixel SPX_R, a blue sensing pixel SPX_B, and two green sensing pixels SPX_Gr and SPX_Gb. The green sensing pixel located in the same row as the red sensing pixel SPX_R is referred to as the first green sensing pixel PX_Gr, and the green sensing pixel located in the same row as the blue sensing pixel PX_B is referred to as the second green sensing pixel PX_Gb.

[0028] Red sensing pixels SPX_R, blue sensing pixels SPX_B, first green sensing pixels SPX_Gr, and second green sensing pixels SPX_Gb are arranged in a matrix, which is referred to as the pixel pattern PT. Multiple pixel patterns PT are arranged repeatedly within the pixel array PXA.

[0029] For example, as shown in Figure 2B, the pixel pattern PT comprises a red sensing pixel SPX_R arranged in a 2x2 matrix, a blue sensing pixel SPX_B arranged in a 2x2 matrix, a first green sensing pixel SPX_Gr arranged in a 2x2 matrix, and a second green sensing pixel SPX_Gb arranged in a 2x2 matrix. Such a pixel pattern PT is referred to as a tetra pattern. However, the technical concept of the present invention is not limited thereto, and the pixel pattern PT comprises a red sensing pixel SPX_R, a blue sensing pixel SPX_B, a first green sensing pixel SPX_Gr, and a second green sensing pixel SPX_Gb arranged in a 2x2 matrix, and such a pixel pattern PT is referred to as a Bayer pattern. Alternatively, the pixel pattern PT may comprise red sensing pixels SPX_R arranged in an n x n matrix (where n is an integer greater than or equal to 3), blue sensing pixels SPX_B arranged in an n x n matrix (where n is an integer greater than or equal to 3), first green sensing pixels SPX_Gr arranged in an n x n matrix, and second green sensing pixels SPX_Gb arranged in an n x n matrix.

[0030] Image data IDT is generated based on the pixel signals output from the pixel array PXA. The image data IDT has a color pattern corresponding to the pixel pattern PT of the pixel array PXA. For example, if the pixel array PXA has a Bayer pattern, the image data IDT also has a Bayer pattern. As another example, if the pixel array PXA has a tetrahedron pattern, the image data IDT has either a tetrahedron pattern or a Bayer pattern.

[0031] For example, if the pixel array PXA has a tetrahedron pattern, either one pixel signal is output from four sensing pixels SPX of the same color contained in the pixel pattern SPX, or four pixel signals are output, with each of the four sensing pixels SPX outputting a pixel signal. When one pixel signal is output, the image data IDT has a Bayer pattern, and when four pixel signals are output, the image data IDT has a tetrahedron pattern, as shown in Figure 2.

[0032] The image data IDT comprises a repeating arrangement of red pixels PX_R, blue pixels PX_B, a first green pixel PX_Gr, and a second green pixel PX_Gb. The pixel PX in the image data IDT refers to the data corresponding to the sensing pixel SPX of the pixel array PXA, in other words, the pixel data. The red pixel PX_R, blue pixel PX_B, first green pixel PX_Gr, and second green pixel PX_Gb correspond to the red sensing pixel SPX_R, blue sensing pixel SPX_B, first green sensing pixel SPX_Gr, and second green sensing pixel SPX_Gb of the pixel array PXA, respectively.

[0033] The image data IDT includes multiple pixel groups PG, where each pixel group PG is configured to have a predetermined number of pixels PX arranged sequentially in a matrix according to the color pattern of the image data IDT, or arranged in one direction, or to have pixels PX that correspond to the same reference color and are adjacent to each other.

[0034] For example, as shown in Figure 2B, if the image data IDT has a tetrahedron pattern, the pixel group PG is configured to have four adjacent pixels PX that correspond to the same reference color (e.g., red, blue, green, etc.). As another example, if the image data IDT has a Bayer pattern, the pixel group PG is configured to have a predetermined number (e.g., four) of pixels PX arranged in a matrix.

[0035] Next, referring to Figure 2A, each of the multiple sensing pixels SPX comprises at least one photosensing element (or photoelectric conversion element). The photosensing element senses light and converts the sensed light into an electrical signal. For example, the photosensing element is a photodiode, phototransistor, photogate, pinned photodiode (PPD), or a combination thereof.

[0036] Each of the multiple sensing pixel SPX comprises at least one photosensitive element and a pixel circuit that outputs a pixel signal corresponding to the electrical signal generated by the photosensitive element. For example, the pixel circuit has a 4-transistor structure comprising a transmission transistor, a reset transistor, an amplification transistor, and a selection transistor. However, it is not limited to these, and the pixel circuit may have a 1-transistor structure, a 3-transistor structure, a 4-transistor or 5-transistor structure, or a structure in which multiple pixels share some transistors. In one embodiment, each of the pixel circuits may include a memory or an analog-to-digital converter.

[0037] In one embodiment, multiple color filters that transmit optical signals in a specific wavelength band (in other words, optical signals of a specific hue) are arranged on multiple pixels of a pixel array PXA, corresponding to each of the multiple pixels, and at least one photosensing element provided on the pixel converts the optical signal transmitted through the corresponding color filter into an electrical signal. As a result, each of the multiple sensing pixels SPX of the pixel array PXA outputs at least one pixel signal for at least one assigned reference color. However, it is not limited to this, and at least one photosensing element provided on the sensing pixel SPX may selectively convert optical signals in a specific wavelength band from the incident light into electrical signals.

[0038] In one embodiment, the image data IDT includes raw image data (row image data) which contains multiple pixel values ​​obtained by digital-to-analog conversion of multiple pixel signals output from a pixel array PXA, or image data which has been preprocessed from the raw image data.

[0039] The image sensor module 100 compresses the image data IDT using the encoder 120 to improve data transmission speed, reduce power consumption during data transmission, and optimize data storage space, and transmits the compressed data CDT to the image processing device 200.

[0040] The encoder 120 receives image data IDT from the image sensor 110 and compresses the image data IDT to generate compressed data CDT. The compressed data CDT is realized in the form of an encoded bitstream. Hereinafter, the encoded bitstream will be simply referred to as the bitstream. The bitstream includes the compression result and compression information (e.g., mode information indicating the compression method).

[0041] The encoder 120 encodes the image data IDT in units of pixel groups PG to generate compressed data CDT. The encoder 120 encodes one pixel group PG to generate one bitstream, and then generates compressed data CDT based on the bitstreams of all pixel groups PG in the image data IDT. By encoding the pixel groups PG, the pixel groups PG are compressed; therefore, in the present invention, encoding is used interchangeably with compression.

[0042] The encoder 120 performs compression based on the pixel values ​​corresponding to the pixel group PG being compressed, in other words, pixels that have been compressed before the target pixel group. Specifically, the encoder 120 compresses the pixel values ​​of the target pixels based on the reference values ​​of at least one reference pixel adjacent to at least one target pixel within the target pixel group. The reference value is generated based on the pixel value of the reference pixel; for example, the reference value is the value generated when the pixel value of the reference pixel is compressed and then decompressed.

[0043] The pixel value of a target pixel and the pixel value of an adjacent reference pixel are likely to be similar. Also, the pixel values ​​of target pixels within a target pixel group are likely to be similar. Therefore, the encoder 120 compresses the target pixel group using a differential pulse code modulation (DPCM) method, which encodes the target pixels in the target pixel group based on the difference with surrounding pixels, for example, the difference between the pixel value of the target pixel and the reference value of an adjacent reference pixel, or the difference between the pixel value of the target pixel and the pixel value of other target pixels within the target pixel group. This increases the compression efficiency (or compression ratio) and reduces data loss due to compression.

[0044] The encoder 120 according to this embodiment compresses at least a portion of the image data (e.g., first image data) to generate partially compressed data, and then packs at least a portion of the remaining image data (e.g., second image data) into the partially compressed data to generate compressed data CDT. As a result, the encoder 120 according to this embodiment can improve compression efficiency even when compressing images with high bit depth without excessive deformation of the IP included in the image sensor module 100 (e.g., increasing the size of each IP or adding new IP), and can reduce the power consumption that accompanies this. A detailed explanation of the encoder 140 will be given later in Figure 3.

[0045] The encoder 120 provides compressed data CDT to the image processing unit 200 via interface 130. For example, interface 130 is implemented by a Camera Serial Interface (CSI) based on MIPI (Mobile Industry Processor Interface). However, the type of interface 130 is not limited to this and can also be implemented by various protocol standards.

[0046] The image processing device 200 converts the compressed data CDT received from the image sensor module 100 to generate an image to be displayed on a display (not shown). Specifically, the image processing device 200 receives the compressed data CDT from the image sensor module 100, decompresses the compressed data CDT to generate decompressed data DDT, for example, restored image data, and processes the decompressed data DDT as an image.

[0047] In one embodiment, the image processing device 200 receives compressed data CDT from the image sensor module 100 via interface 210. Interface 210 is implemented as MIPI, such as interface 130 provided on the image sensor module 100, but is not limited to this. The image processing device 200 stores the received compressed data CDT in memory 220.

[0048] Memory 220 is a storage location for saving data. Compressed data CDT is stored in memory 220. In addition, memory 220 stores other data, such as the OS (Operating System), various programs, and various other data (e.g., compressed data CDT). Memory 220 includes volatile memory such as DRAM (Dynamic Random Access Memory) and SRAM (Static RAM), or non-volatile memory such as PRAM (Phase Change RAM), ReRAM (Resistive RAM), MRMA (Magnetic RAM), and flash memory. On the other hand, although memory 220 is shown as being located within the image processing unit 200 in Figure 2A, it is not limited to this, and memory 220 may be located separately outside the image processing unit 200.

[0049] The decoder 230 reads the compressed data CDT from the memory 220, decompresses the compressed data CDT, and generates decompressed data DDT. The decoder 230 provides the decompressed data DDT to the image signal processor 240.

[0050] The decoder 230 decompresses the compressed data CDT in units of pixel groups PG using a decompression method (or decoding method) based on the compression method (or encoding method) performed by the encoder 120 of the image sensor module 100. At this time, the decoder 230 determines the compression method applied to the pixel group PG based on the compression information contained in the bitstream of the compressed data CDT. The decoder 230 decompresses the target pixels of the target pixel group based on the reference value corresponding to the pixels that were decompressed before the target pixel group to be decompressed, in other words, the reference pixels.

[0051] As described above, according to one embodiment, the compression method refers to the type of compression used to generate or acquire compressed data, and is also called the compression process, compression scheme, or compression technique. Similarly, the decompression method refers to the type of decompression used to generate or acquire data to be decompressed, and is also called the decompression process, decompression scheme, or decompression technique. In one embodiment, encoding is called compression, and conversely, compression is called encoding. Similarly, decoding is called decompression, and decompression is called decoding.

[0052] The decoder 230 according to this embodiment decompresses partially compressed data separated from the compressed data CDT to generate partially decompressed data, and then packs the remaining data and random data from the compressed data CDT into the partially decompressed data to generate decompressed data DDT. As a result, the decoder 230 according to this embodiment can improve decompression efficiency even when decompressing images with high bit depth without excessive modification of the IPs included in the image processing module 200 (for example, increasing the size of each IP or adding new IPs), and reduce the power consumption that accompanies this. A detailed explanation of the decoder 230 will be given later in Figure 4.

[0053] The image signal processor 240 performs various image processing on the received decompressed data DDT. As an unrestricted example, the image signal processor 240 performs at least one of the following image processing on the decompressed data DDT: defective pixel correction, offset correction, lens distortion correction, color gain correction, shading correction, gamma correction, noise reduction, and sharpening. In one embodiment, some of the above-described image processing may be omitted depending on the performance of the image sensor module 100. For example, if the image sensor module 100 is equipped with a high-quality image sensor 110, defective pixel correction (especially static defective pixel correction) or offset correction may be omitted from the image processing.

[0054] On the other hand, the encoder 120 and decoder 230 are each implemented in software or hardware, or in a combination of software and hardware such as firmware. When the encoder 120 and decoder 230 are implemented in software, the respective functions described above are implemented in programmed source code, which is loaded onto storage media provided in the image sensor module 100 and image processing device 200, respectively, and the functions of the encoder 120 and decoder 230 are implemented when a processor (e.g., an image processing processor) provided in the image sensor module 100 and image processing device 200 executes the software. When the encoder 120 and decoder 230 are implemented in hardware, the encoder 120 and decoder 230 include logic circuits and registers, and perform the respective functions described above based on the settings of the registers.

[0055] On the other hand, although Figure 2A illustrates that the image processing system 10 comprises an image sensor module 100 and an image processing device 200, the present invention is not limited thereto. For example, the image processing system 10 may be implemented comprising only a part of the image sensor module 100 and the image processing device 200, or comprising multiple image sensor modules 100. Also, although Figure 2A illustrates that the decoder 230 and the image signal processor 240 are configured separately, the present invention is not limited thereto. For example, the image signal processor 240 may be implemented comprising the decoder 230.

[0056] Figure 3 is a block diagram showing an encoder according to one embodiment of the present invention.

[0057] In detail, Figure 3 is a block diagram showing the configuration included in the encoder 120 of Figure 2A.

[0058] Referring to Figure 3, the encoder 120 according to this embodiment includes a bit depth control circuit 121, a register 123, an encoding circuit 125, and a packing circuit 127.

[0059] In one embodiment, the bit depth control circuit 121 receives image data IDT from the image sensor. The bit depth control circuit 121 receives control data according to the image sensor's operating scenario via the Advanced Peripheral Bus (APB). The control data includes information regarding the compression ratio required by the encoder 120 according to the image sensor's operating scenario (e.g., 50%), the bit depth (or bit size) of the image data IDT to be compressed by the encoder 120 (e.g., 10 bits, 12 bits, etc.), and an encoding scheme selected from at least one encoding scheme. The control data is transmitted to the encoder 120 via the APB and stored in the Special Function Register (SFR) set.

[0060] In one embodiment, the bit depth control circuit 121 generates first image data IDT_1 from N bits extracted from image data IDT based on the received control data, and generates second image data IDT_2 from some bits of the remaining image data. Here, the remaining image data refers to the image data excluding the first image data IDT_1 from the image data IDT. The number of bits extracted from image data IDT (e.g., N bits) is determined based on the control data received via APB. The bit depth control circuit 121 generates first image data IDT_1 and second image data IDT_2 in various ways depending on the encoding scheme. Various embodiments of the method for generating first image data IDT_1 and second image data IDT_2 will be described later in Figures 5, 7, and 9.

[0061] In one embodiment, the bit depth control circuit 121 transmits the first image data IDT_1 to the encoding circuit 125 and the second image data IDT_2 to the register 123.

[0062] In one embodiment, register 123 is a FIFO (First-In First-Out) register that stores the second image data IDT_2. Register 123 transmits the second image data IDT_2 to the packing circuit 127 through signaling with the packing circuit 127.

[0063] In one embodiment, the encoding circuit 125 compresses the first image data IDT_1 to generate partially compressed data Parial CDT. The encoder 120 according to this embodiment can effectively compress HDR image data without any additional modifications to the encoding circuit 125 (i.e., while maintaining the configuration of the encoding circuit 125 that compresses image data with a low bit depth) by controlling the bit depth of the image data to be compressed (e.g., the first image data IDT_1) via the bit depth control circuit 121. The encoding circuit 125 transmits the generated partially compressed data Parial CDT to the packing circuit 127.

[0064] In one embodiment, the packing circuit 127 packs the partially compressed data Parial CDT and the second image data IDT_2 to generate compressed data CDT for the image data IDT. For example, the packing circuit 127 generates compressed data CDT by outputting the second image data IDT_2 together with the output timing of the partially compressed data Parial CDT. The packing circuit 127 outputs the generated compressed data CDT to the image processing module. Here, the compressed data CDT includes a header, a payload, and the second image data IDT_2, and the header includes information about the encoding scheme of the image data IDT and information about the number of quantizations in the compression process of the image data IDT.

[0065] The encoder 120 according to this embodiment can control the bit depth of the image data via the bit depth control circuit 121, and therefore can effectively compress HDR image data (i.e., image data with a high bit depth) without any additional modifications to the encoder 120 (e.g., the encoding circuit 125).

[0066] Furthermore, the encoder 120 according to this embodiment has the technical effect of reducing the power consumption required during the compression process by improving the compression efficiency of HDR image data.

[0067] Figure 4 is a block diagram showing a decoder according to one embodiment of the present invention.

[0068] In detail, Figure 4 is a block diagram showing the configuration included in the decoder 230 of Figure 2A.

[0069] Referring to Figure 4, the decoder 230 according to this embodiment includes a bit depth control circuit 231, a register 233, a decoding circuit 235, and a dithering circuit 237.

[0070] In one embodiment, the bit depth control circuit 231 receives compressed data CDT from the image sensor module (100 in Figure 2A). The compressed data CDT includes a header, a payload, and second image data, the header including information about the encoding scheme of the image data IDT and information about the number of quantizations in the compression process of the image data IDT. Based on the information stored in the header of the compressed data CDT, the bit depth control circuit 231 determines one of at least one decoding schemes as the decoding scheme for the compressed data CDT.

[0071] In one embodiment, the bit depth control circuit 231 generates partially compressed data (Parial CDT) and the remaining compressed data based on the compressed data (CDT) using a defined decoding scheme. Here, the remaining compressed data is the compressed data remaining after subtracting the partially compressed data (Parial CDT) from the compressed data (CDT), and corresponds to the second image data (IDT_2) packed into the compressed data (CDT) in Figure 3. The bit depth control circuit 231 generates the partially compressed data (Parial CDT) and the remaining compressed data in various ways depending on the decoding scheme. Various embodiments of the method for generating the partially compressed data (Parial CDT) and the remaining compressed data will be described later in Figures 6, 8, and 10.

[0072] In one embodiment, the bit depth control circuit 231 transmits the partially compressed data Parial CDT to the decoding circuit 235 and the remaining compressed data to the register 233.

[0073] In one embodiment, register 233 is a FIFO (First-In First-Out) register that stores the remaining compressed data. Register 233 transmits the remaining compressed data to the dithering circuit 237 through signaling with the dithering circuit 237.

[0074] In one embodiment, the decoding circuit 235 decompresses the partially compressed data (Parial CDT) to generate partially decompressed data (Partial DDT). The encoder 230 according to this embodiment can effectively decompress HDR image data without any additional modifications to the decoding circuit 235 (i.e., while maintaining the configuration of the decoding circuit 235 that decompresses image data with a low bit depth) by controlling the bit depth of the compressed data (e.g., partially compressed data (Partial CDT)) via the bit depth control circuit 231. The decoding circuit 235 transmits the generated partially decompressed data (Partial DDT) to the dithering circuit 237.

[0075] In one embodiment, the dithering circuit 237 includes a random number generator. The dithering circuit 237 generates random data based on the random number generator.

[0076] In one embodiment, the dithering circuit 237 packs the partially decompressed data (Partial DDT), the remaining compressed data, and random data to generate decompressed data (DDT) for the image data (IDT). For example, the dithering circuit 237 generates decompressed data (DDT) by outputting the remaining compressed data and random data together in time with the output timing of the partially decompressed data (Partial DDT). The dithering circuit 237 outputs the generated decompressed data (DDT) to a display, application AP, etc.

[0077] The decoder 230 according to this embodiment can control the bit depth of the compressed data via the bit depth control circuit 231, and therefore can effectively decompress HDR image data (i.e., image data with a high bit depth) without any additional modifications to the decoder 230 (e.g., the decoding circuit 235).

[0078] Furthermore, the decoder 230 according to this embodiment has the technical effect of reducing the power consumption required during the decompression process by improving the decompression efficiency of HDR image data.

[0079] On the other hand, although Figure 4 shows the bit depth control circuit 231 and register 233 being provided separately outside the decoding circuit 235, the system is not limited to this, and the bit depth control circuit 231 and register 233 can be provided inside the decoding circuit 235.

[0080] Figure 5 is a diagram illustrating the operation of an encoder according to one embodiment of the present invention.

[0081] In detail, Figure 5 illustrates the compression operation of the encoder 120 on the image data IDT 500 using the first encoding scheme. In Figure 5, it is assumed that the image data IDT 500 is image data (48 bits in total) corresponding to four pixels (e.g., pixels G0, R1, G2, R3).

[0082] Referring to Figure 5, the bit depth control circuit 121 generates first image data 501 and second image data 503 based on image data (hereinafter referred to as pixel data) 500-1 (12 bits) corresponding to one pixel (e.g., G0). The bit depth control circuit 121 extracts a predetermined number of bits (e.g., 10 bits) of MSB (most significant bit) from the image data 500-1 (12 bits) using a first encoding scheme among at least one encoding scheme, and generates it as first image data 501. The bit depth control circuit 121 generates second image data 503 using the first encoding scheme by extracting the MSB (e.g., 1 bit) from the remaining image data 502. Here, the remaining image data 502 refers to the image data remaining after the first image data 501 has been extracted from the pixel data 500-1. The bit depth control circuit 121 repeats the above process to generate first image data 501 and second image data 503 for each pixel data 500-1. The bit depth control circuit 121 combines the first image data 501 for each pixel data 500-1 to generate a first bitstream (40 bits) 505 and transmits it to the encoding circuit 125. The bit depth control circuit 121 also combines the second image data 503 for each pixel data 500-1 to generate a second bitstream (4 bits) 504 and transmits it to the register 123.

[0083] Register 123 receives and stores the second bitstream 504 from the bit depth control circuit 121. Register 123 transmits the second bitstream 504 to the packing circuit 127 via signaling with the packing circuit 127. Register 123 is a FIFO (First-In First-Out) register.

[0084] The encoding circuit 125 receives the first bitstream (40 bits) 505 from the bit depth control circuit 121 and compresses the first bitstream 505 according to the compression ratio of the first encoding scheme. For example, if the compression ratio of the first encoding scheme is 50%, the encoding circuit 125 compresses the first bitstream (40 bits) 505 to generate partially compressed data (20 bits) 506. At this time, the partially compressed data 506 includes a header (HEADER) containing information about the encoding scheme of the first bitstream 505 and information about the number of quantizations during the compression process of the first bitstream 505, and a payload (PAYLOAD) which stores compressed / combined pixel data for each pixel (for example, pixel data corresponding to pixels G0, R1, G2, R3) (i.e., stores the compressed pixel data). The encoding circuit 125 transmits the partially compressed data 506 generated by signaling with the packing circuit 127 to the packing circuit 127.

[0085] The packing circuit 127 packs the partially compressed data 506 and the second bitstream 504 to generate compressed data 507. For example, the packing circuit 127 outputs the second bitstream 504 together with the output timing of the partially compressed data 506 to generate compressed data 507. The packing circuit 127 outputs the generated compressed data 507 to the image processing module.

[0086] Figure 6 is a diagram illustrating the operation of a decoder according to one embodiment of the present invention.

[0087] In detail, Figure 6 is a diagram illustrating the decompression operation of the decoder 230 on the compressed data CDT 600 using the first decoding method. In Figure 6, the compressed data CDT 600 corresponds to the compressed data CDT 507 in Figure 5.

[0088] Referring to Figure 6, the bit depth control circuit 231 separates the compressed data CDT 600 into partially compressed data PARIAL CDT 601 and the remaining compressed data 602 using a first decoding scheme. For example, the bit depth control circuit 231 separates / generates the compressed data CDT 600 into partially compressed data PARIAL CDT 601 and the remaining compressed data 602 using a first decoding scheme. The partially compressed data PARIAL CDT 601 includes a header containing information about the encoding scheme of the image data and information about the number of quantizations in the image data compression process, and a payload in which the compressed pixel data is stored. Here, the first decoding scheme is determined based on the header of the partially compressed data PARIAL CDT 601 from at least one decoding scheme.

[0089] The bit depth control circuit 231 transmits the partially compressed data Parial CDT 601 to the decoding circuit 235 and the remaining compressed data 602 to the register 233. For example, the remaining compressed data 602 corresponds to the second bitstream 504 in Figure 5.

[0090] Register 233 receives and stores the remaining compressed data 602 from the bit depth control circuit 231. Register 233 transmits the remaining compressed data 602 to the dithering circuit 237 via signaling with the dithering circuit 237. Register 233 is a FIFO (First-In First-Out) register.

[0091] The decoding circuit 235 receives the partially compressed data Parallel CDT 601 from the bit depth control circuit 231 and decompresses the partially compressed data Parallel CDT 601 using the first decoding method. For example, if the partially compressed data Parallel CDT 601 is compressed to a compression ratio of 50%, the decoding circuit 231 decompresses the partially compressed data Parallel CDT (20 bits) 601 to generate a partially decompressed bitstream (40 bits) 604. At this time, the partially decompressed bitstream (40 bits) 604 is a bitstream generated by combining the partially decompressed data Parallel DDT (10 bits) 603 for each pixel data. For example, the partially decompressed bitstream (40 bits) 604 is a bitstream that includes the partially decompressed data Parallel DDT corresponding to pixel G0, pixel R1, pixel G2, and pixel R3. The decoding circuit 235 transmits the partially decompressed bitstream (40 bits) 604 to the dithering circuit 237 through signaling with the dithering circuit 237.

[0092] The dithering circuit 237 includes a random number generator. The dithering circuit 237 generates random data 607 based on the random number generator.

[0093] The dithering circuit 237 packs the partially decompressed data Parial DDT (10 bits) 603, a portion of the remaining compressed data 602 605, and random data 607 to generate decompressed data DDT 608 for each pixel. The dithering circuit 237 outputs the portion of the remaining compressed data 602 605 together with the output timing of the partially decompressed data Parial DDT 603, thereby packing the partially decompressed data Parial DDT 603 and the portion of the remaining compressed data 602 605 to generate packed data 606. The dithering circuit 237 outputs the random data 607 together with the output timing of the packed data 606, thereby packing the packed data 606 and the random data 607 to generate decompressed data DDT 608. For example, the dithering circuit 237 generates packing data 606 using a first decoding method, where the partially decompressed data Partial DDT 603 is the MSB (10 bits) and the remaining compressed data 605 is the LSB (1 bit). The dithering circuit 237 generates decompressed data DDT (12 bits) 608 using the first decoding method, where the packing data 606 is the MSB (11 bits) and the random data 607 is the LSB (1 bit). The dithering circuit 237 repeats the above method for each pixel to generate decompressed data 608 corresponding to each pixel. The dithering circuit 237 combines the decompressed data 608 corresponding to each pixel to generate a decompressed bitstream (48 bits) 610 and outputs the generated decompressed bitstream (48 bits) 610.

[0094] Figure 7 is a diagram illustrating the operation of an encoder according to one embodiment of the present invention.

[0095] In detail, Figure 7 illustrates the compression operation of the encoder 120 on the image data IDT 700 using the second encoding scheme. In Figure 7, it is assumed that the image data IDT 700 is image data (48 bits in total) corresponding to four pixels (for example, pixels G0, R1, G2, and R3).

[0096] Referring to Figure 7, the bit depth control circuit 121 generates first image data 701 and second image data 703 based on image data (hereinafter referred to as pixel data) 700-1 (12 bits) corresponding to one pixel (e.g., G0). The bit depth control circuit 121 extracts a predetermined number of bits (e.g., 10 bits) of LSB (Least Significant Bit) from the image data 700-1 (12 bits) using the second encoding scheme of at least one encoding scheme, and generates it as first image data 701. The bit depth control circuit 121 generates second image data 703 using the LSB (e.g., 1 bit) from the remaining image data 702 using the second encoding scheme. Here, the remaining image data 702 refers to the image data remaining after the first image data 701 has been extracted from the pixel data 700-1. The bit depth control circuit 121 repeats the above process to generate first image data 701 and second image data 703 for each pixel data 700-1. The bit depth control circuit 121 combines the first image data 701 for each pixel data 700-1 to generate a first bitstream (40 bits) 705 and transmits it to the encoding circuit 125. The bit depth control circuit 121 also combines the second image data 703 for each pixel data 700-1 to generate a second bitstream (4 bits) 704 and transmits it to the register 123.

[0097] Register 123 receives and stores the second bitstream 704 from the bit depth control circuit 121. Register 123 transmits the second bitstream 704 to the packing circuit 127 via signaling with the packing circuit 127. Register 123 is a FIFO (First-In First-Out) register.

[0098] The encoding circuit 125 receives the first bitstream (40 bits) 705 from the bit depth control circuit 121 and compresses the first bitstream 705 according to the compression ratio of the second encoding scheme. For example, if the compression ratio of the second encoding scheme is 50%, the encoding circuit 125 compresses the first bitstream (40 bits) 705 to generate partially compressed data PARIAL CDT (20 bits) 706. At this time, the partially compressed data PARIAL CDT 706 includes a header containing information about the encoding scheme of the first bitstream 705 and information about the number of quantizations during the compression process of the first bitstream 705, and a payload that compresses / combines and stores the pixel data for each pixel (for example, pixel data corresponding to pixels G0, R1, G2, R3) (i.e., stores the compressed pixel data). The encoding circuit 125 transmits the partially compressed data PARIAL CDT 706 generated by signaling with the packing circuit 127 to the packing circuit 127.

[0099] The packing circuit 127 packs the partially compressed data Parial CDT 706 and the second bitstream 704 to generate compressed data CDT 707. The packing circuit 127 outputs the second bitstream 704 together with the output timing of the partially compressed data Parial CDT 706 to generate compressed data CDT 707. The packing circuit 127 outputs the generated compressed data CDT 707 to the image processing module.

[0100] Figure 8 is a diagram illustrating the operation of a decoder according to one embodiment of the present invention.

[0101] In detail, Figure 8 illustrates the decompression operation of the decoder 230 on the compressed data CDT 800 using the second decoding method. In Figure 8, the compressed data CDT 800 corresponds to the compressed data CDT 707 in Figure 7.

[0102] Referring to Figure 8, the bit depth control circuit 231 separates the compressed data CDT 800 into partially compressed data Parallel CDT 801 and the remaining compressed data 802 using a second decoding scheme. For example, the bit depth control circuit 231 separates and generates the compressed data CDT 800 into partially compressed data Parallel CDT 801 and the remaining compressed data 802 using a second decoding scheme. The partially compressed data Parallel CDT 801 includes a header containing information about the encoding scheme of the image data and information about the number of quantizations in the image data compression process, and a payload in which the compressed pixel data is stored. Here, the second decoding scheme is determined based on the header of the partially compressed data Parallel CDT 801 from at least one decoding scheme.

[0103] The bit depth control circuit 231 transmits the partially compressed data Parial CDT 801 to the decoding circuit 235 and the remaining compressed data 802 to the register 233. For example, the remaining compressed data 802 corresponds to the second bitstream 704 in Figure 7.

[0104] Register 233 receives and stores the remaining compressed data 802 from the bit depth control circuit 231. Register 233 transmits the remaining compressed data 602 to the dithering circuit 237 via signaling with the dithering circuit 237. Register 233 is a FIFO (First-In First-Out) register.

[0105] The decoding circuit 235 receives the partially compressed data Parallel CDT 801 from the bit depth control circuit 231 and decompresses the partially compressed data Parallel CDT 801 using the second decoding method. For example, if the partially compressed data Parallel CDT 801 is compressed to a compression ratio of 50%, the decoding circuit 231 decompresses the partially compressed data Parallel CDT (20 bits) 801 to generate a partially decompressed bitstream (40 bits) 804. At this time, the partially decompressed bitstream (40 bits) 804 is a bitstream generated by combining the partially decompressed data Parallel DDT (10 bits) 803 for each pixel data. For example, the partially decompressed bitstream (40 bits) 804 is a bitstream that includes the partially decompressed data Parallel DDT corresponding to pixel G0, pixel R1, pixel G2, and pixel R3. The decoding circuit 235 transmits the partially decompressed bitstream (40 bits) 804 to the dithering circuit 237 through signaling with the dithering circuit 237.

[0106] The dithering circuit 237 includes a random number generator. The dithering circuit 237 generates random data 807 based on the random number generator.

[0107] The dithering circuit 237 packs the partially decompressed data Parial DDT (10 bits) 803, a portion of the remaining compressed data 802 805, and random data 807 to generate decompressed data DDT 808 for each pixel. The dithering circuit 237 outputs the portion of the remaining compressed data 802 805 together with the output timing of the partially decompressed data Parial DDT 803, thereby packing the partially decompressed data Parial DDT 803 and the portion of the remaining compressed data 802 805 to generate packed data 806. The dithering circuit 237 outputs the random data 807 together with the output timing of the packed data 806, thereby packing the packed data 806 and the random data 807 to generate decompressed data DDT 808. For example, the dithering circuit 237 generates packing data 806 using a second decoding method, where a portion of the remaining compressed data 805 is the MSB (1 bit) and the partially decompressed data Partial DDT 803 is the LSB (10 bits). The dithering circuit 237 generates decompressed data DDT (12 bits) 808 using a second decoding method, where the random data 607 is the LSB (1 bit) and the packing data 806 is the LSB (11 bits). The dithering circuit 237 repeats the above method for each pixel to generate decompressed data 808 corresponding to each pixel. The dithering circuit 237 combines the decompressed data 808 corresponding to each pixel to generate a decompressed bitstream (48 bits) 810 and outputs the generated decompressed bitstream (48 bits) 810.

[0108] Figure 9 is a diagram illustrating the operation of an encoder according to one embodiment of the present invention.

[0109] In detail, Figure 9 illustrates the compression operation of the encoder 120 on the image data IDT 900 using the third encoding scheme. In Figure 9, it is assumed that the image data IDT 900 is image data (48 bits in total) corresponding to four pixels (e.g., pixels G0, R1, G2, R3).

[0110] Referring to Figure 9, the bit depth control circuit 121 generates first image data 901 and second image data 903 based on image data (hereinafter referred to as pixel data) 900-1 (12 bits) corresponding to one pixel (e.g., G0). The bit depth control circuit 121 generates the MSB (e.g., 1 bit) of image data-1 as second image data 903 using a third encoding scheme, which is one of at least one encoding schemes. The bit depth control circuit 121 extracts a predetermined number of bits (e.g., 10 bits) of LSB (Least Significant Bit) from image data 900-1 (12 bits) using the third encoding scheme and generates it as first image data 701. The bit depth control circuit 121 generates the MSB (e.g., 10 bits) of the remaining image data 902 as first image data 901 using the third encoding scheme. Here, the remaining image data 902 refers to the image data remaining after the second image data 903 has been extracted from the pixel data 900-1. The bit depth control circuit 121 repeats the process described above to generate a first image data 901 and a second image data 903 for each pixel data 900-1. The bit depth control circuit 121 combines the first image data 901 for each pixel data 900-1 to generate a first bitstream (40 bits) 905 and transmits it to the encoding circuit 125. The bit depth control circuit 121 also combines the second image data 903 for each pixel data 900-1 to generate a second bitstream (4 bits) 904 and transmits it to the register 123.

[0111] Register 123 receives and stores the second bitstream 904 from the bit depth control circuit 121. Register 123 transmits the second bitstream 904 to the packing circuit 127 via signaling with the packing circuit 127. Register 123 is a FIFO (First-In First-Out) register.

[0112] The encoding circuit 125 receives the first bitstream (40 bits) 905 from the bit depth control circuit 121 and compresses the first bitstream 905 according to the compression ratio of the third encoding scheme. For example, if the compression ratio of the third encoding scheme is 50%, the encoding circuit 125 compresses the first bitstream (40 bits) 905 to generate partially compressed data, Parial CDT (20 bits) 906. At this time, the partially compressed data, Parial CDT 906, includes a header containing information about the encoding scheme of the first bitstream 905 and information about the number of quantizations during the compression process of the first bitstream 905, and a payload that compresses / combines and stores the pixel data for each pixel (for example, pixel data corresponding to pixels G0, R1, G2, R3) (i.e., stores the compressed pixel data). The encoding circuit 125 transmits the partially compressed data, Parial CDT 906, generated by signaling with the packing circuit 127 to the packing circuit 127.

[0113] The packing circuit 127 packs the partially compressed data Parial CDT 906 and the second bitstream 904 to generate compressed data CDT 907. The packing circuit 127 outputs the second bitstream 904 together with the output timing of the partially compressed data Parial CDT 906 to generate compressed data CDT 907. The packing circuit 127 outputs the generated compressed data CDT 907 to the image processing module.

[0114] Figure 10 is a diagram illustrating the operation of a decoder according to one embodiment of the present invention.

[0115] In detail, Figure 10 is a diagram illustrating the decompression operation of the decoder 230 using the third decoding method on the compressed data CDT 1000. In Figure 10, the compressed data CDT 1000 corresponds to the compressed data CDT 907 in Figure 9.

[0116] Referring to Figure 10, the bit depth control circuit 231 separates the compressed data CDT 1000 into partially compressed data Parallel CDT 1001 and the remaining compressed data 1002 using a third decoding scheme. For example, the bit depth control circuit 231 separates the compressed data CDT 1000 into partially compressed data Parallel CDT 1001 and the remaining compressed data 1002 using a third decoding scheme. The partially compressed data Parallel CDT 1001 includes a header containing information about the encoding scheme of the image data and information about the number of quantizations in the image data compression process, and a payload in which the compressed pixel data is stored. Here, the third decoding scheme is determined based on the header of the partially compressed data Parallel CDT 1001 from at least one decoding scheme.

[0117] The bit depth control circuit 231 transmits the partially compressed data Parial CDT 1001 to the decoding circuit 235 and the remaining compressed data 1002 to the register 233. For example, the remaining compressed data 802 corresponds to the second bitstream 904 in Figure 9.

[0118] Register 233 receives and stores the remaining compressed data 1002 from the bit depth control circuit 231. Register 233 transmits the remaining compressed data 1002 to the dithering circuit 237 via signaling with the dithering circuit 237. Register 233 is a FIFO (First-In First-Out) register.

[0119] The decoding circuit 235 receives the partially compressed data Parallel CDT 1001 from the bit depth control circuit 231 and decompresses the partially compressed data Parallel CDT 1001 using the third decoding method. For example, if the partially compressed data Parallel CDT 1001 is compressed to a compression ratio of 50%, the decoding circuit 231 decompresses the partially compressed data Parallel CDT (20 bits) 1001 to generate a partially decompressed bitstream (40 bits) 1004. At this time, the partially decompressed bitstream (40 bits) 1004 is a bitstream generated by combining the partially decompressed data Parallel DDT (10 bits) 1003 for each pixel data. For example, the partially decompressed bitstream (40 bits) 1004 is a bitstream that includes the partially decompressed data Parallel DDT 1003 corresponding to pixels G0, R1, G2, and R3. The decoding circuit 235 transmits the partially decompressed bitstream (40 bits) 1004 to the dithering circuit 237 through signaling with the dithering circuit 237.

[0120] The dithering circuit 237 includes a random number generator. The dithering circuit 237 generates random data 1007 based on the random number generator.

[0121] The dithering circuit 237 packs the partially decompressed data Parial DDT (10 bits) 1003, a portion of the remaining compressed data 1002 1005, and random data 1007 to generate decompressed data DDT 1008 for each pixel. The dithering circuit 237 outputs the portion of the remaining compressed data 1002 1005 together with the output timing of the partially decompressed data Parial DDT 1003, thereby packing the partially decompressed data Parial DDT 1003 and the portion of the remaining compressed data 1005 to generate packing data 1006. The dithering circuit 237 outputs the random data 1007 together with the output timing of the packing data 1006, thereby packing the packing data 1006 and the random data 1007 to generate decompressed data DDT 1008. For example, the dithering circuit 237 generates packing data 1006 using a third decoding method, where a portion of the remaining compressed data 1005 is the MSB (1 bit) and the partially decompressed data Partial DDT 1003 is the LSB (10 bits). The dithering circuit 237 generates decompressed data DDT (12 bits) 1008 using a third decoding method, where the packing data 1006 is the MSB (11 bits) and the random data 1007 is the LSB (1 bit). The dithering circuit 237 repeats the above method for each pixel to generate decompressed data 1008 corresponding to each pixel. The dithering circuit 237 combines the decompressed data 1008 corresponding to each pixel to generate a decompressed bitstream (48 bits) 1010 and outputs the generated decompressed bitstream (48 bits) 1010.

[0122] Figure 11 is a block diagram of an electronic device including a multi-camera module according to one embodiment of the present invention, and Figure 12 is a detailed block diagram of the camera module of Figure 11 according to one embodiment of the present invention.

[0123] Referring to Figure 11, the electronic device 10000 comprises a plurality of camera modules 1100, an application processor 1200, a PMIC (power module integrated circuit) 1300, and an external memory 1400.

[0124] Multiple camera modules 1100 comprise camera modules 1100. The drawings show an embodiment in which three camera modules 1100 are arranged, but the embodiment is not limited thereto. In one embodiment, multiple camera modules 1100 may be modified to comprise only two camera modules, or k camera modules (where k is a natural number of 4 or more).

[0125] The detailed configuration of camera module 1100b will be described in more detail below with reference to Figure 12, but the following description also applies to other camera modules 1100a and 1100b depending on the embodiment.

[0126] Referring to Figure 12, the camera module 1100b comprises a prism 1105, an optical path folding element (OPFE) 1110, an actuator 1130, an image sensing device 1140, and a storage unit 1150.

[0127] The prism 1105 is equipped with a reflective surface 1107 made of a light-reflecting material, which deforms the path of light L incident from the outside.

[0128] In one embodiment, the prism 1105 changes the path of light L incident in a first direction (X) to a second direction (Y) perpendicular to the first direction (X). The prism 1105 also changes the path of light L incident in a first direction (X) to a second direction (Y) perpendicular to the first direction (X) by rotating the reflective surface 1107 of the light-reflecting material in direction A about the central axis 1106 or by rotating the central axis 1106 in direction B. At this time, the OPFE 1110 also moves in a third direction (Z) perpendicular to the first direction (X) and the second direction (Y).

[0129] In one embodiment, as shown in the figure, the maximum rotation angle of the prism 1105 in the A direction is 15° or less in the positive (+) A direction and greater than 15° in the negative (-) A direction, but the embodiment is not limited thereto.

[0130] In one embodiment, the prism 1105 moves within approximately 20° in the positive (+) or negative (-)B direction, or between 10° and 20°, or between 15° and 20°, where the angle of movement is either the same angle in the positive (+) or negative (-)B direction, or to a substantially similar angle within approximately 1°.

[0131] In one embodiment, the prism 1105 moves the reflective surface 1107 of the light-reflecting material in a third direction (for example, the Z direction) parallel to the extension of the central axis 1106.

[0132] In one embodiment, the camera module 1100b is composed of two or more prisms, which change the path of light L incident in a first direction (X) through them to a second direction (Y) perpendicular to the first direction (X), then back to the first direction (X) or a third direction (Z), and then back to the second direction (Y), and so on.

[0133] The OPFE 1110 includes, for example, m (where m is a natural number) groups of optical lenses. The M lenses move in a second direction (Y) to change the optical zoom magnification of the camera module 1100b. For example, if the basic optical zoom magnification of the camera module 1100b is Z, then when the m optical lenses provided in the OPFE 1110 are moved, the optical zoom magnification of the camera module 1100b is changed to 3Z, 5Z, or 5Z or greater.

[0134] The actuator 1130 moves the OPFE 1110 or optical lens (hereinafter referred to as the optical lens) to a specific position. For example, the actuator 1130 adjusts the position of the optical lens so that the image sensor 1142 is positioned at the focal length of the optical lens for accurate sensing.

[0135] The image sensing device 1140 comprises an image sensor 1142, control logic 1144, and memory 1146. The image sensor module 100 in Figure 1, or the camera module 1100b in Figure 11, can be used as the image sensing device 1140.

[0136] The image sensor 1142 senses an image of the object to be sensed using light L provided through an optical lens. The control logic 1144 controls the overall operation of the camera module 1100b and processes the sensed image. For example, the control logic 1144 controls the operation of the camera module 1100b by control signals provided through the control signal line CSLb, and extracts image data corresponding to a specific image (e.g., a person's face, arms, legs, etc. in the image) from the sensed image or performs image processing such as noise reduction.

[0137] In one embodiment, the control logic 1144 includes an encoder (120 in Figure 1) that compresses the sensed image or the image processed. As described above, the encoder 120 compresses the image on a pixel group basis and compresses the pixel groups of isolated regions using an offset coding scheme.

[0138] Memory 1146 stores information necessary for the operation of the camera module 1100b, such as calibration data 1147. Calibration data 1147 is information necessary for the camera module 1100b to generate image data using light L supplied from an external source, and includes, for example, information on rotation, focal length, and optical axis. If the camera module 1100b is implemented as a multi-state camera configuration in which the focal length changes depending on the position of the optical lens, the calibration data 1147 includes the focal length values ​​for each position (or state) of the optical lens and information on autofocusing.

[0139] In one embodiment, compressed data is stored in memory 1146. Memory 1146 is also used as a reference buffer 125 for encoder 120.

[0140] The storage unit 1150 stores image data sensed through the image sensor 1142. The storage unit 1150 is located outside the image sensing device 1140 and is implemented in a stacked configuration with the sensor chips that make up the image sensing device 1140. In one embodiment, the image sensor 1142 is made up of the first chip, and the control logic 1144, storage unit 1150, and memory 1146 are made up of the second chip, and the two chips are implemented in a stacked configuration.

[0141] In one embodiment, the storage unit 1150 is implemented as an EEPROM (Electrically Erasable Programmable Read-Only memory), but this embodiment is not limited thereto. In one embodiment, the image sensor 1142 is configured as a pixel array, and the control logic 1144 comprises an analog-to-digital converter and an image signal processing unit for processing the sensed image.

[0142] Referring to both Figures 11 and 12, in one embodiment, each of the multiple camera modules 1100 is provided with an actuator 1130. Thus, each of the multiple camera modules 1100 is provided with equal or different calibration data 1147 by the operation of the actuator 1130 located inside it.

[0143] In one embodiment, one of the multiple camera modules 1100 (e.g., 1100b) is a folded lens camera module equipped with the prism 1105 and OPFE 1110 described above, while the remaining camera modules (e.g., 1100a and 1100c) may be vertical camera modules without the prism 1105 and OPFE 1110, but are not limited thereto.

[0144] In one embodiment, one of the multiple camera modules 1100 (e.g., 1100c) is a vertical depth camera that extracts depth information using, for example, IR (Infrared Ray). In this case, the application processor 1200 merges the image data provided by such a depth camera with the image data provided by other camera modules (e.g., 1100a or 1100b) to generate a three-dimensional depth image.

[0145] In one embodiment, at least two of the multiple camera modules 1100 (e.g., 1100a and 1100b) have different fields of view (angles of view). In this case, for example, the optical lenses of at least two of the multiple camera modules 1100 (e.g., 1100a and 1100b) are different from each other, but are not limited to this.

[0146] Furthermore, in one embodiment, the field of view of each of the multiple camera modules 1100 is different. For example, camera module 1100a is an ultrawide camera, camera module 1100b is a wide camera, and camera module 1100c is a telephoto camera, but is not limited to this. In this case, the optical lenses provided in each of the multiple camera modules 1100 are also different, but is not limited to this.

[0147] In one embodiment, the multiple camera modules 1100 are physically separated from each other. That is, the sensing area of ​​a single image sensor 1142 is not divided and used by the multiple camera modules 1100; rather, an independent image sensor 1142 is arranged inside each of the multiple camera modules 1100.

[0148] Referring again to Figure 11, the application processor 1200 comprises an image processing unit 1210, a memory controller 1220, and internal memory 1230. The application processor 1200 and the multiple camera modules 1100 may be implemented separately from each other, for example, by separate semiconductor chips.

[0149] The image processing unit 1210 comprises a plurality of sub-image processors (1212a, 1212b, 1212c), an image generator 1214, and a camera module controller 1216.

[0150] The image processing device 1210 includes multiple sub-image processors (1212a, 1212b, 1212c) corresponding to the number of camera modules 1100.

[0151] Image data generated from camera module 1100a is provided to sub-image processor 1212a via image signal line ISLa, image data generated from camera module 1100b is provided to sub-image processor 1212b via image signal line ISLb, and image data generated from camera module 1100c is provided to sub-image processor 1212c via image signal line ISLc. Such image data transmission is performed using, for example, a Camera Serial Interface (CSI) based on MIPI (Mobile Industry Processor Interface), but is not limited to this.

[0152] In one embodiment, at least one of the multiple sub-image processors (1212a, 1212b, 1212c) includes a decoder (230 in Figure 1). The multiple sub-image processors (1212a, 1212b, 1212c) include a decoder 230 to decompress compressed image data when the corresponding camera module 1100 includes an encoder (120 in Figure 1).

[0153] In one embodiment, the image processing device 200 in Figure 1 is embodied in at least one of a plurality of sub-image processors (1212a, 1212b, 1212c), and at least one of the plurality of sub-image processors (1212a, 1212b, 1212c) comprises an encoder (120 in Figure 1) and a decoder (230 in Figure 1).

[0154] In one embodiment, a sub-image processor corresponding to an encoder (120 in Figure 1) among a plurality of sub-image processors (1212a, 1212b, 1212c) generates first image data and second image data based on image data using at least one encoding scheme, compresses the first image data to generate partially compressed data, and packs the second image data into the partially compressed data to generate a plurality of bitstreams (i.e., data streams of compressed image data).

[0155] In one embodiment, a sub-image processor corresponding to an encoder (120 in Figure 1) extracts a predetermined number of MSBs from the image data and generates them as first image data, generates the MSBs of the remaining image data as second image data, and generates multiple bitstreams based on these.

[0156] In one embodiment, a sub-image processor corresponding to an encoder (120 in Figure 1) extracts a predetermined number of LSBs from the image data and generates them as first image data, generates the LSBs of the remaining image data as second image data, and generates multiple bitstreams based on these.

[0157] In one embodiment, a sub-image processor corresponding to an encoder (120 in Figure 1) generates the MSB of the image data as second image data in the case of a third encoding scheme among at least one encoding scheme, extracts an MSB of a predetermined number of bits from the remaining image data and generates it as first image data, and generates multiple bitstreams based on this.

[0158] For the sake of explanation, the compression operation of the sub-image processor corresponding to the encoder (120 in Figure 1) using the first to third encoding schemes has been described, but it is not limited to this. The sub-image processor corresponding to the encoder (120 in Figure 1) according to this embodiment can generate first and second image data using various encoding schemes and generate multiple bitstreams, which are compressed data based on this data.

[0159] In one embodiment, a sub-image processor corresponding to a decoder (230 in Figure 1) among a plurality of sub-image processors (1212a, 1212b, 1212c) separates a plurality of bitstreams using at least one decoding scheme to generate partially compressed data and second image data, decompresses the partially compressed data to restore the first image data, and packs the restored first image data, second image data, and random data based on at least one decoding scheme to restore the image data. At this time, the sub-image processor corresponding to the decoder (230 in Figure 1) includes a random number generator and generates random data based on it.

[0160] In one embodiment, a sub-image processor corresponding to a decoder (230 in Figure 1) among a plurality of sub-image processors (1212a, 1212b, 1212c) generates packing data in which the partially decompressed data is the MSB and the remaining part of the compressed data is the LSB, using a first decoding method of at least one decoding method, and generates decompressed data in which the packing data is the MSB and the random data is the LSB, using the first decoding method. Furthermore, the sub-image processor corresponding to the decoder (230 in Figure 1) combines the decompressed data for each pixel to generate a plurality of bitstreams (i.e., data streams obtained by decompressing compressed data).

[0161] In one embodiment, a sub-image processor corresponding to a decoder (230 in Figure 1) among a plurality of sub-image processors (1212a, 1212b, 1212c) generates packing data in which a portion of the remaining compressed data is the MSB and the partially decompressed data is the LSB, using a second decoding method of at least one decoding method, and generates decompressed data in which the random data is the MSB and the packing data is the LSB, using the second decoding method. Furthermore, the sub-image processor corresponding to the decoder (230 in Figure 1) combines the decompressed data for each pixel to generate a plurality of bitstreams (i.e., data streams obtained by decompressing compressed data).

[0162] In one embodiment, a sub-image processor corresponding to a decoder (230 in Figure 1) among a plurality of sub-image processors (1212a, 1212b, 1212c) generates packing data in which a portion of the remaining compressed data is the MSB and the partially decompressed data is the LSB, using a third decoding method among at least one decoding method, and generates decompressed data in which the packing data is the MSB and the random data is the LSB, using the third decoding method. Furthermore, the sub-image processor corresponding to the decoder (230 in Figure 1) combines the decompressed data for each pixel to generate a plurality of bitstreams (i.e., data streams obtained by decompressing compressed data).

[0163] For the sake of explanation, the compression operation of the sub-image processor corresponding to the decoder (230 in Figure 1) using the first to third encoding schemes has been described, but it is not limited to this. The sub-image processor corresponding to the decoder (230 in Figure 1) according to this embodiment can generate partially decompressed data, packing data, and random data using various decoding schemes, and based on these, generate multiple bitstreams that are decompressed data.

[0164] On the other hand, in one embodiment, one sub-image processor may be arranged to correspond to multiple camera modules. For example, sub-image processors 1212a and 1212c are not realized separately from each other as shown in the figure, but are realized integrated into a single sub-image processor, and the image data provided from camera modules 1100a and 1100c are selected through a selection element (e.g., a multiplexer) and then provided to the integrated sub-image processor. In this case, sub-image processor 1212b is not integrated, and image data is provided from camera module 1100b.

[0165] In one embodiment, image data generated from camera module 1100a is provided to sub-image processor 1212a via image signal line ISLa, image data generated from camera module 1100b is provided to sub-image processor 1212b via image signal line ISLb, and image data generated from camera module 1100c is provided to sub-image processor 1212c via image signal line ISLc. The image data processed by sub-image processor 1212b is then immediately provided to image generator 1214, but one of the image data processed by sub-image processor 1212a and the image data processed by sub-image processor 1212c is selected via a selection element (e.g., a multiplexer) before being provided to image generator 1214.

[0166] Each sub-image processor (1212a, 1212b, 1212c) performs image processing on the image data provided by the camera module 1100, including faulty pixel correction, 3A adjustment (auto-focus correction, auto-white balance, auto-exposure), noise reduction, sharpening, gamma adjustment, and remosaic.

[0167] In one embodiment, remosaic signal processing is performed in each camera module 1100 and then provided to sub-image processors (1212a, 1212b, 1212c).

[0168] The image data processed by each sub-image processor (1212a, 1212b, 1212c) is provided to the image generator 1214. The image generator 1214 generates an output image using the image data provided by each sub-image processor (1212a, 1212b, 1212c) based on image generation information or mode signals.

[0169] Specifically, the image generator 1214 generates an output image by merging at least a portion of the image data generated by the sub-image processors (1212a, 1212b, 1212c) based on image generation information or mode signals. Alternatively, the image generator 1214 may select one of the image data generated by the sub-image processors (1212a, 1212b, 1212c) based on image generation information or mode signals to generate an output image.

[0170] In one embodiment, the image generation information includes a zoom signal (or zoom factor). In another embodiment, the mode signal is a signal based on a mode selected by the user, for example.

[0171] If the image generation information is a zoom signal (zoom factor), and each camera module 1100 has a different field of view (field of view angle), the image generator 1214 will perform different operations depending on the type of zoom signal. For example, if the zoom signal is a first signal, the image generator 1214 will generate an output image using the image data output from sub-image processor 1212a and the image data output from sub-image processor 1212b, from the image data output from sub-image processor 1212a and the image data output from sub-image processor 1212c. If the zoom signal is a second signal different from the first signal, the image generator 1214 will generate an output image using the image data output from sub-image processor 1212c and the image data output from sub-image processor 1212b, from the image data output from sub-image processor 1212a and the image data output from sub-image processor 1212c. If the zoom signal is a third signal different from the first and second signals, the image generator 1214 does not perform such image data merging, but instead selects one of the image data output from each sub-image processor (1212a, 1212b, 1212c) to generate the output image. However, this embodiment is not limited thereto, and the method of processing the image data can be modified in various ways as needed.

[0172] In one embodiment, the image processing device 1210 further includes a selection unit (not shown) that selects the output of sub-image processors (1212a, 1212b, 1212c) and transmits it to the image generator 1214.

[0173] In this case, the selection unit performs different operations depending on the zoom signal or zoom factor. For example, if the zoom signal is the fourth signal (for example, the zoom magnification is the first magnification), the selection unit selects one of the outputs of the sub-image processors (1212a, 1212b, 1212c) and transmits it to the image generator 1214.

[0174] Furthermore, if the zoom signal is a fifth signal different from the fourth signal (for example, a zoom magnification of 2x), the selection unit sequentially transmits p outputs (where p is a natural number greater than or equal to 2) from the outputs of the sub-image processors (1212a, 1212b, 1212c) to the image generator 1214. For example, the selection unit sequentially transmits the outputs of sub-image processors 1212b and 1212c to the image generator 1214. Alternatively, the selection unit sequentially transmits the outputs of sub-image processors 1212a and 1212b to the image generator 1214. The image generator 1214 merges the sequentially provided p outputs to generate a single output image.

[0175] Here, image processing such as demosaicing, downscaling to the video / preview resolution size, gamma correction, and HDR (High Dynamic Range) processing is performed in advance by sub-image processors (1212a, 1212b, 1212c), and then the processed image data is transmitted to the image generator 1214. Therefore, even if the processed image data is provided to the image generator 1214 via a single signal line through the selection unit, the image merging operation of the image generator 1214 can be performed at high speed.

[0176] In one embodiment, the image generator 1214 receives multiple image data with different exposure times from at least one of a plurality of sub-image processors (1212a, 1212b, 1212c), and generates merged image data with an increased dynamic range by performing HDR (high dynamic range) processing on the multiple image data.

[0177] The camera module controller 1216 provides control signals to each camera module 1100. The control signals generated by the camera module controller 1216 are provided to the corresponding camera module 1100 through mutually isolated control signal lines (CSLa, CSLb, CSLc).

[0178] One of the multiple camera modules 1100 is designated as the master camera (e.g., 1100b) by image generation information including a zoom signal or a mode signal, while the remaining camera modules (e.g., 1100a and 1100c) are designated as slave cameras. This information is included in the control signals and provided to the corresponding camera modules 1100 through separate control signal lines (CSLa, CSLb, CSLc).

[0179] The camera module that operates as master and slave changes depending on the zoom factor or operating mode signal. For example, if the field of view of camera module 1100a is wider than that of camera module 1100b, and the zoom factor indicates a low zoom magnification, camera module 1100a operates as the master and camera module 1100b operates as the slave. Conversely, if the zoom factor indicates a high zoom magnification, camera module 1100b operates as the master and camera module 1100a operates as the slave.

[0180] In one embodiment, the control signals provided from the camera module controller 1216 to each camera module 1100 include a sync enable signal. For example, if camera module 1100b is the master camera and camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 transmits a sync enable signal to camera module 1100b. Camera module 1100b, having received such a sync enable signal, generates a sync signal based on the provided sync enable signal and provides the generated sync signal to camera modules 1100a and 1100c through the sync signal line SSL. Camera modules 1100b, as well as camera modules 1100a and 1100c, are synchronized by such sync signals and transmit image data to the application processor 1200.

[0181] In one embodiment, the control signals provided from the camera module controller 1216 to the multiple camera modules 1100 include mode information via mode signals. Based on such mode information, the multiple camera modules 1100 operate in a first operating mode and a second operating mode depending on the sensing speed.

[0182] Multiple camera modules 1100 generate an image signal at a first speed in a first operating mode (for example, an image signal at a first frame rate), encode it at a second speed higher than the first speed (for example, encode an image signal at a second frame rate higher than the first frame rate), and transmit the encoded image signal to the application processor 1200. At this time, the second speed is 30 times or less the first speed.

[0183] The application processor 1200 stores the received image signal, in other words, the encoded image signal, in its internal memory 1230 or its external memory 1400. Then it reads the encoded image signal from the internal memory 1230 or external memory 1400, decodes it, and displays the image data generated based on the decoded image signal. For example, one of the multiple sub-image processors (1212a, 1212b, 1212c) of the image processing device 1210 performs decoding and also performs image processing on the decoded image signal.

[0184] Multiple camera modules 1100 generate image signals in a second operating mode at a third speed lower than the first speed (for example, generating image signals at a third frame rate lower than the first frame rate) and transmit the image signals to the application processor 1200. The image signals provided to the application processor 1200 are unencoded signals. The application processor 1200 either performs image processing on the received image signals or stores the image signals in internal memory 1230 or external memory 1400.

[0185] The PMIC 1300 supplies power, such as a power supply voltage, to each of the multiple camera modules 1100. For example, under the control of the application processor 1200, the PMIC 1300 supplies first power to camera module 1100a through power signal line PSLa, second power to camera module 1100b through power signal line PSLb, and third power to camera module 1100c through power signal line PSLc.

[0186] The PMIC 1300 generates and adjusts the power levels corresponding to each of the multiple camera modules 1100 in response to a power control signal PCON from the application processor 1200. The power control signal PCON includes power adjustment signals according to the operating mode of the multiple camera modules 1100. For example, the operating mode includes a low-power mode, in which case the power control signal PCON includes information about the camera modules operating in low-power mode and the power levels to be set. The power levels provided to each of the multiple camera modules 1100 are equal or different. Furthermore, the power levels change dynamically.

[0187] Figure 13 is a schematic block diagram showing an electronic device according to one embodiment of the present invention. The electronic device 2000 in Figure 13 is a portable terminal.

[0188] The encoder 2210 and decoder 2110 in Figure 13 correspond to the encoder 120 and decoder 230 in Figures 1 to 11.

[0189] Referring to Figure 13, the electronic device 2000 comprises an application processor 2100, a camera module 2200, a working memory 2300, a storage unit 2400, a display device 2600, a user interface 2700, and a wireless transceiver unit 2500.

[0190] The application processor 2100 is implemented as a system-on-chip SoC that controls the overall operation of the image processing system 2000 and drives application programs, operating systems, etc. The application processor 2100 provides image data from the camera module 2200 to the display device 2600 or stores it in the storage unit 2400.

[0191] The image sensor module 100, described with reference to Figures 1 to 12, is applied to the camera module 2200. The image sensor 2200 includes an encoder 2210, which compresses image data to generate compressed data and transmits the compressed data to the application processor 2100. As described above, the encoder 2210 according to this embodiment generates first image data and second image data based on image data, compresses the first image data to generate partially compressed data, and packs the second image data into the partially compressed data to generate multiple bitstreams (i.e., data streams of compressed image data).

[0192] The application processor 2100 includes a decoder 2110 that decompresses the compressed data using a decoding method corresponding to the compression method, for example, the encoding method, of the encoder 2210. The decoder 2110 decompresses the compressed data received from the camera module 2200 to generate restored image data, and the application processor 2100 processes the restored image data.

[0193] The decoder 2110 according to this embodiment separates multiple bitstreams using at least one decoding method to generate partially compressed data and the remaining compressed data, decompresses the partially compressed data to generate partially decompressed data, and packs the partially decompressed data, a portion of the remaining compressed data, and random data using at least one decoding method to restore (i.e., decompress) the image data.

[0194] The application processor 2100 displays the restored image data or the image-processed image data on the display device 2600 or stores it in the storage unit 2400.

[0195] The working memory 2300 is implemented by volatile memory such as DRAM or SRMA, or non-volatile memory such as FeRAM, ReRAM, or PRAM. The working memory 200 stores programs and / or data that the application processor 2100 processes or executes.

[0196] The storage unit 2400 is embodied in a non-volatile memory device such as NADN flash or resistive memory, and is provided by, for example, a memory card (MMC, eMMC, SD, micro SD). The storage unit 2400 stores image data received from the camera module 2200 or data processed or generated by the application 2100.

[0197] The user interface 2700 is embodied in various devices that receive user input, such as a keyboard, curtain key panel, touch panel, fingerprint sensor, and microphone. The user interface 2700 receives user input and provides signals corresponding to the received user input to the application processor 2100.

[0198] The wireless transceiver unit 2500 includes a transceiver 2510, a modem 2520, and an antenna 2530.

[0199] Although embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the embodiments described above, and can be modified and implemented in various ways without departing from the technical spirit of the present invention. [Explanation of symbols]

[0200] 10 Image Processing Systems 100 Image Sensor Modules 101, 105 Image data 102, 106 Compressed data Header 103, 107 104, 108 payloads 110 Image Sensor 120, 2210 encoders 121, 231-bit depth control circuits Registers 123, 233 125 Encoding circuit 127 Packing Circuit 130, 210 Interface (I / F) 200, 1210 Image Processing Equipment 220 memory 230, 2110 Decoder 235 Decoding circuit 237 Dithering Circuit 240 Image Signal Processors 500, 700, 900 image data (IDT) 500-1, 700-1, 900-1 Image Data (Pixel Data) 501, 701, 901 First Image Data 502, 702, 902 Remaining image data 503, 703, 903 Second Image Data 504, 704, 904 Second bitstream 505, 705, 905 First bitstream 506, 706, 906 Partial Compression Data (Parial CDT) 507, 707, 907 compressed data (CDT) 600, 800, 1000 compressed data (CDT) 601, 801, 1001 Partial Compression Data (Parial CDT) 602, 802, 1002 Remaining compressed data 603, 803, 1003 Partial Decompression Data (Parial DDT) 604, 804, 1004 partial decompression bitstream 605, 805, 1005: Part of the remaining compressed data 606, 806, 1006 Packing Data 607, 807, 1007 Random Data 608, 808, 1008 decompressed data (DDT) 610, 810, 1010 decompression bitstream 1100, 1100a, 1100b, 1100c camera modules 1105 Prism 1106 Center axis 1107 Reflective surface 1110 Optical Path Bend Element (OPFE) 1130 Actuator 1140 Image Sensing Device 1142 Image Sensor 1144 Control Logic 1146 memory 1147 Calibration data 1150, 2400 storage section 1200 application processors 1212a, 1212b, 1212c sub-image processors 1214 Image Generator 1216 Camera Module Controller 1220 Memory Controller 1230 internal memory 1300 PMIC 1400 External memory 2000, 10000 electronic devices 2100 Application Processors 2200 Camera Module 2300 Working Memory 2500 Wireless Transceiver Unit 2510 Transceiver 2520 Modem 2530 Antenna 2600 Display Devices 2700 User Interface CDT compressed data CSLa, CSLb, CSLc control signal lines DDT decompressed data IDT Image Data IDT_1, IDT_2: First and second image data ISLa, ISLb, ISLc image signal lines LS Lens PCON Power Control Signal PG Pixel Group PSLa, PSLb, PSLc power signal lines PT Pixel Pattern PX pixels PXA Pixel Array SPX Sensing Pixel SPX_B Blue Sensing Pixel SPX_Gb, SPX_Gr 2nd and 1st Green Sensing Pixels SPX_R Red Sensing Pixel SSL synchronous signal line

Claims

1. An encoder for compressing image data generated by an image sensor, A bit depth control circuit generates first image data based on a first plurality of bits extracted from the image data using at least one encoding scheme, and generates second image data based on a second plurality of bits corresponding to the remaining image data from the image data that is different from the first image data, An encoding circuit that compresses the first image data to generate partially compressed data, A register for storing the aforementioned second image data, An encoder characterized by comprising a packing circuit that packs the second image data into the partially compressed data to generate compressed data and outputs the compressed data.

2. The bit depth control circuit is, Using the first encoding method among the at least one encoding method, a predetermined number of first multiple MSBs (Most Significant Bits) are extracted from the image data, and the first multiple MSBs are generated as the first image data. The encoder according to claim 1, characterized in that the first encoding method is used to obtain a second plurality of MSBs from the remaining image data and generate the second plurality of MSBs as the second image data.

3. The bit depth control circuit is, Using the second encoding method among the at least one encoding method described above, a predetermined number of first LSBs (Least Significant Bits) are extracted from the image data, and the first LSBs are generated as the first image data. The encoder according to claim 1, characterized in that the second encoding method is used to obtain a second plurality of LSBs from the remaining image data and generate the second plurality of LSBs as the second image data.

4. The bit depth control circuit is, Using the third encoding method among the at least one encoding method, a first plurality of MSBs are obtained from the image data and the first plurality of MSBs are generated as the second image data. The encoder according to claim 1, characterized in that, using the third encoding method, a predetermined number of second multiple MSBs are extracted from the image data excluding the second image data, and the second multiple MSBs are generated as the first image data.

5. The bit depth control circuit generates the first image data and the second image data based on the control signal received from the image sensor. The encoder according to claim 1, characterized in that the control signal includes information regarding the number of bits of the image data output from the image sensor, the compression ratio of the encoder, and an encoding method selected from a plurality of encoding methods.

6. The encoder according to claim 1, characterized in that the packing circuit outputs the second image data together with the output timing of the partially compressed data to generate the compressed data in order to generate the compressed data.

7. The encoder according to claim 1, characterized in that the register includes a FIFO (First-In First-Out) register.

8. The compressed data includes a header, a payload, and the second image data. The encoder according to claim 1, characterized in that the header includes information regarding the encoding method of the image data and information regarding the number of quantizations in the image data compression process.

9. A decoder that decompresses compressed data, A bit depth control circuit that separates the compressed data into partially compressed data and the remaining compressed data based on the compressed data using at least one decoding method, A register for storing the remaining compressed data, A decoding circuit that decompresses the aforementioned partially compressed data and generates partially decompressed data, A decoder comprising: a dithering circuit that generates random data, packs a portion of the remaining compressed data into the partially decompressed data based on at least one decoding scheme to generate packed data, and packs the random data into the packed data to generate decompression data.

10. The aforementioned dithering circuit is The packing data is generated by the first decoding method of the at least one decoding method such that the partially decompressed data is included in the MSB of the packing data and a portion of the remaining compressed data is included in the LSB of the packing data. The decoder according to claim 9, characterized in that the first decoding method generates the decompressed data such that the packing data is included in the MSB of the decompressed data and the random data is included in the LSB of the decompressed data.

11. The aforementioned dithering circuit is The packing data is generated by the second decoding method of the at least one decoding method such that a portion of the remaining compressed data is included in the MSB of the packing data and the partially decompressed data is included in the LSB of the packing data. The decoder according to claim 9, characterized in that the second decoding method generates the decompressed data such that the random data is included in the MSB of the decompressed data and the packing data is included in the LSB of the decompressed data.

12. The aforementioned dithering circuit is The packing data is generated by the third decoding method among the at least one decoding method such that a portion of the remaining compressed data is included in the MSB of the packing data and the partially decompressed data is included in the LSB of the packing data. The decoder according to claim 9, characterized in that the third decoding method generates the decompressed data such that the packing data is included in the MSB of the decompressed data and the random data is included in the LSB of the decompressed data.

13. The dithering circuit includes a random number generator, The decoder according to claim 9, wherein the dithering circuit is further configured to generate the random data based on the random number generator.

14. The decoder according to claim 9, characterized in that the register includes a FIFO (First-In First-Out) register.

15. The compressed data includes a header, a payload, and the remaining compressed data. The decoder according to claim 9, characterized in that the header includes information regarding the encoding scheme of the image data and information regarding the number of quantization cycles in the compression process of the image data.

16. The decoder according to claim 15, characterized in that the at least one decoding method is determined based on the header of the compressed data.

17. An image processing system, An image sensor that senses the received optical signal and generates image data, An encoder that sequentially compresses multiple pixel groups contained in the image data to generate multiple bitstreams, The system includes a decoder that decompresses the plurality of bitstreams and restores the image data, The encoder described above is A first image data and a second image data are generated based on the image data using at least one encoding scheme. The first image data is compressed to generate partially compressed data. An image processing system characterized by generating the plurality of bitstreams by packing the second image data into the partially compressed data.

18. The encoder is further configured to operate according to one of the first encoding scheme, the second encoding scheme, and the third encoding scheme. In the case of the first encoding method, the MSB of a predetermined number of bits is extracted from the image data to generate the first image data, and the MSB of the remaining image data is set as the second image data. In the case of the second encoding method, the first image data is generated by extracting a predetermined number of LSB bits from the image data, and the LSB of the remaining image data is set as the second image data. The image processing system according to claim 17, characterized in that, in the case of the third encoding method, the MSB of the image data is set as the second image data, and the MSB of a predetermined number of bits is extracted from the remaining image data to generate the first image data.

19. The decoder mentioned above is Based on at least one decoding scheme, the partially compressed data and the second image data are generated from the plurality of bitstreams. The partially compressed data is decompressed to restore the first image data. The image processing system according to claim 17, characterized in that it packs the recovered first image data, the second image data, and random data based on at least one decoding method to restore the image data.

20. The decoder further includes a random number generator, The image processing system according to claim 19, characterized in that the decoder generates the random data based on the random number generator.

Images