Image Encoder, Image Sensing Device, and Method of Operating an Image Encoder

Abstract

The present disclosure provides an image encoder, an image sensing device, and an operating method of the image encoder. The image encoder is configured to encode an original image and reduce compression loss. The image encoder includes an image signal processor and a compressor. The image signal processor is configured to receive a first frame image and a second frame image, and generate a compressed image of the second frame image using a boundary pixel image of the first frame image. The image signal processor may include a memory configured to store first reference pixel data as the first frame image. The compressor is configured to receive the first reference pixel data from the memory and generate a bitstream obtained by encoding the second frame image based on the difference between the first reference pixel data and the second frame image. The image signal processor generates a compressed image of the second frame image using the bitstream generated by the compressor.

Description

[0001] This application claims priority to Korean Patent Application No. 10-2020-0051927, filed on Apr. 29, 2020, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure relates to an image encoder, an image sensing device, and a method of operating an image encoder. Background Art

[0003] Modern computers use image compression as a method of reducing resources for image storage or data transmission. Storage devices in computers can store more compressed image data than uncompressed image data. Additionally, by transmitting compressed image files, wireless or wired data transmission is faster and more reliable.

[0004] Image compression is a process of generating encoded image data using less computational storage compared to the original image data. Additionally, image decompression is a process of decoding the encoded image data to generate reconstructed image data. Depending on the encoding and decoding methods, the reconstructed image data may be different from the original image data.

[0005] Differential Pulse Code Modulation (DPCM) is an encoding method that compresses original image data using surrounding pixel values. However, since there are no peripheral pixels to be referenced, boundary pixels located at the edges of the original image may not be able to perform DPCM compression.

[0006] Additionally, if the difference between the boundary pixel value and the surrounding pixel values is large, the error may affect the DPCM of the pixel values of the boundary pixel and other pixels. Therefore, there is a need in the art for a compression method that considers various pixel information when compressing an image when peripheral pixel data is not available. Summary of the Invention

[0007] Aspects of the present disclosure provide an image encoder configured to encode an original image. Aspects of the present disclosure also provide an image encoder capable of reducing compression loss.

[0008] However, aspects of the present disclosure are not limited to the aspects set forth herein. The above and other aspects of the present disclosure will become more apparent to those of ordinary skill in the art to which the present disclosure pertains by reference to the following detailed description of the present disclosure. Details of the embodiments are included in the detailed description and the drawings.

[0009] According to one aspect of the present disclosure, there is provided an image sensing device including: an image signal processor configured to receive a first frame image and a second frame image (e.g., temporally after the first frame image), and generate a compressed image of the second frame image based on a boundary pixel image of the first frame image. The image signal processor includes: a memory configured to store first reference pixel data that is a boundary pixel image of the first frame image; and a compressor configured to receive the first reference pixel data from the memory and generate a bitstream obtained by encoding the second frame image based on a difference between the first reference pixel data and original pixel data of the second frame image. The image signal processor generates the compressed image of the second frame image based on (e.g., using) the bitstream.

[0010] According to another aspect of the present disclosure, there is provided an image encoder configured to receive a first frame image and a second frame image temporally after the first frame image. The image encoder includes: a memory configured to store first reference pixel data that is a boundary pixel image of the first frame image; a compressor configured to receive the first reference pixel data from the memory, generate a first bitstream obtained by encoding the original pixel data of the second frame image based on a difference between the first reference pixel data and the original pixel data of the second frame image, and output the generated first bitstream; and a reconstructor configured to reconstruct the first bitstream to generate second reference pixel data, where the second reference pixel data is a boundary pixel image of the second frame image.

[0011] According to another aspect of the present disclosure, there is provided a method for operating an image encoder. The method includes: receiving a first frame image and a second frame image, where the second frame image is received at a different time from when the first frame image is received; storing first reference pixel data that is a first boundary pixel image of the first frame image; generating a bitstream obtained by encoding the original pixel data of the second frame image based on a difference between the stored first reference pixel data and the original pixel data of the second frame image; outputting the generated bitstream; and reconstructing the bitstream to generate second reference pixel data, where the second reference pixel data is a second boundary pixel image of the second frame image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other aspects and features of the present disclosure will become more apparent by describing embodiments of the present disclosure in detail with reference to the accompanying drawings, in which:

[0013] Figure 1 is a block diagram of an electronic device having an image encoder for illustrating some embodiments according to the present disclosure.

[0014] Figure 2 is a diagram showing a Bayer image obtained by a color filter according to some embodiments of the present disclosure.

[0015] Figure 3 And Figure 4 is a diagram for explaining boundary pixel data according to some embodiments of the present disclosure.

[0016] Figure 5 And Figure 6 is a block diagram for explaining an encoder according to some embodiments of the present disclosure.

[0017] Figure 7 is for explaining the operation Figure 6 of an encoder. The flowchart of the method

[0018] Figure 8 is a diagram for explaining a method of compressing raw pixel data according to some embodiments of the present disclosure.

[0019] Figure 9 And Figure 10 is a diagram for explaining a method of compressing raw pixel data according to some embodiments of the present disclosure.

[0020] Figure 11 is a diagram for explaining a method of compressing raw pixel data according to some embodiments of the present disclosure.

[0021] Figure 12 is a diagram for explaining a method of compressing raw pixel data according to some embodiments of the present disclosure.

[0022] Figure 13 is a diagram for explaining a method of compressing raw pixel data according to some embodiments of the present disclosure.

[0023] Figure 14 is a diagram for explaining a method of compressing raw pixel data according to some embodiments of the present disclosure.

[0024] Figure 15 is a block diagram for explaining an encoder according to some embodiments of the present disclosure.

[0025] Figure 16 is for explaining the operation Figure 15 of an encoder. The flowchart of the method

[0026] Figure 17 is a block diagram for explaining an electronic device having an image encoder according to some embodiments of the present disclosure.

[0027] Figure 18 is a block diagram for explaining an electronic device having an image encoder according to some embodiments of the present disclosure.

[0028] Figure 19It is a block diagram for explaining an electronic device having a multi-camera module according to some embodiments of the present disclosure.

[0029] Figure 20 is Figure 19 a detailed block diagram of the camera module of. Detailed implementation

[0030] Image compression is a process of generating encoded image data (e.g., compressed encoded image data) using less computational storage compared to the original image data (e.g., original uncompressed image data). In addition, image decompression is a process of decoding the encoded image data to generate reconstructed image data. Depending on the encoding method and decoding method (e.g., and the amount of compression loss), the reconstructed image data may be different from the original image data. For example, differential pulse code modulation (DPCM) is an encoding method for compressing the original image data using surrounding pixel values.

[0031] In a scenario where the original pixel data associated with the boundary condition (e.g., uncompressed pixel data) does have data for referring to the original data, compression of the image may be difficult or errors may occur. For example, boundary pixels located at the edge of the original image may not be able to perform DPCM compression because there are no surrounding pixels to be referred to. Generally, the boundary condition may refer to a condition where the original pixel data does not have reference data in the current frame image (e.g., a condition where the original pixel data is located at the boundary of the frame image such as the upper boundary or the left boundary).

[0032] The present disclosure generally relates to an image encoder, an image sensing device, and a method of operating the image encoder. More specifically, embodiments of the present disclosure relate to a method of compressing original image data associated with a boundary condition (e.g., compressing the original image data when there is no corresponding boundary data in the original image data). In some embodiments, the present disclosure performs image compression by referring to pixel data surrounding the reference pixel data of the previous frame image corresponding to the original image data to be compressed.

[0033] The image encoder of the present disclosure is configured to encode the original image while reducing compression loss. Reducing the compression loss of the original pixel data associated with the boundary condition can be achieved by using the reference pixels of the boundary image associated with the previous frame image. Such reference pixels may be associated with pixel values close to the pixel values of the original pixel data. In some instances, the average value of the pixel values in the boundary image of the previous frame image may be used as the reference pixel, so that the stored reference pixel data can be reduced and the compression reliability can be improved. Embodiments according to the technical concept of the present disclosure are described with reference to the accompanying drawings.

[0034] The following will refer to Figures 1 to 4Describe the electronic device 1 having an image encoder.

[0035] Figure 1 is a block diagram for explaining an electronic device having an image encoder according to some embodiments of the present disclosure. Figure 2 is a diagram showing a Bayer image obtained by a color filter according to some embodiments of the present disclosure. Figure 3 and Figure 4 is a diagram for explaining boundary pixel data according to some embodiments of the present disclosure.

[0036] The electronic device 1 may be an electronic device that captures and stores an image of an object using a solid-state image sensor (e.g., such as complementary metal oxide semiconductor (CMOS)). For example, the electronic device 1 may include a digital camera, a digital video camera, a mobile phone, and a tablet computer.

[0037] Referring to Figure 1 , the electronic device 1 may include a color filter 100, an encoder 200, a decoder 600, and an application processor 700.

[0038] The color filter 100 may obtain raw pixel data from an optical signal. The raw pixel data may represent the pixel values of raw pixels. Half of the pixels in the color filter 100 may detect green signals. One quarter of the pixels in the color filter 100 may detect red signals, and one quarter of the pixels in the color filter 100 may detect blue signals. For example, the color filter 100 may have a structure in which units of 2×2 size with one red (R) pixel, one blue (B) pixel, and two green (G) pixels arranged repeatedly. However, according to the technical idea of the present disclosure, the embodiments are not limited thereto. For example, the color filter 100 may have a structure in which units of 2×2 size with one red (R) pixel, one blue (B) pixel, and two wide green (G) pixels arranged repeatedly.

[0039] A pixel (or picture element) refers to the smallest addressable element in a display device and the smallest controllable element of a picture represented on the device. In some cases, each pixel may represent a sample point of an original image. The color and intensity of each pixel are variable. In a color imaging system, the color may be represented by three component intensities or four component intensities (such as red, green, and blue, or cyan, magenta, yellow, and black).

[0040] The encoder 200 may compress the raw pixel data 510 provided from the color filter 100 to reduce the image data size. Referring to Figure 2, the Bayer image 500 may include the raw pixel data 510 obtained by the color filter 100. In some embodiments, the encoder 200 may generate the encoded data of the raw pixel data 510 using the reference pixel data 520. However, embodiments according to the technical concept of the present disclosure are not limited thereto. The encoder 200 may generate the encoded data of the raw pixel data 510 without using the reference pixel data 520. The encoded data of the raw pixel data 510 may be stored in the bitstream generated by the encoder 200.

[0041] Refer to Figure 3 , the Bayer image 500 may include first boundary pixel data 530. The first boundary pixel data 530 may include the boundary pixel image of the Bayer image 500. For example, the first boundary pixel data 530 may include the pixel image of the first row of the boundary of the Bayer image 500. For example, the first boundary pixel data 530 may include the pixel image provided from one row and column, where a 2×2 size unit having one red (R) pixel, one blue (B) pixel, and two green (G) pixels is arranged in a row. In another example, the first boundary pixel data 530 may include the pixel image of the first column of the boundary of the Bayer image 500.

[0042] Refer to Figure 4 , the Bayer image 500 may include second boundary pixel data 540. The second boundary pixel data 540 may include the boundary pixel image of the Bayer image 500. For example, the second boundary pixel data 540 may include the pixel images of the first and second rows of the boundary of the Bayer image 500. For example, the second boundary pixel data 540 may include the pixel image provided from two rows and columns, where a 2×2 size unit having one red (R) pixel, one blue (B) pixel, and two green (G) pixels is arranged in a row. As another example, the second boundary pixel data 540 may include the pixel images of the first and second columns of the Bayer image 500.

[0043] Refer to again Figure 1 , the decoder 600 may receive the bitstream generated by the encoder 200. The decoder 600 may decode the received bitstream to generate decoded data. The decoder 600 may provide the data obtained by decoding the bitstream to the application processor 700.

[0044] The application processor 700 may include a central processing unit (CPU), a microprocessor, or an MCU (microcontroller unit), and may perform post - processing on the decoded data received from the decoder 600. The post - processing may include applying an image enhancement algorithm to image artifacts. For example, although the application processor 700 may perform white balance, denoising, demosaicing, lens shading, gamma correction, etc. on the received decoded data, embodiments according to the technical concept of the present disclosure are not limited thereto.

[0045] Generally, a processor may refer to an intelligent hardware device (e.g., a general - purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application - specific integrated circuit (ASIC), a field - programmable gate array (FPGA), a programmable logic device, discrete gate or transistor logic components, discrete hardware components, or any combination thereof). In some cases, the processor is configured to operate a memory array using a memory controller. In other cases, the memory controller is integrated into the processor. In some cases, the processor is configured to execute computer - readable instructions stored in a memory to perform various functions. In some embodiments, the processor includes dedicated components for image processing, modem processing, baseband processing, digital signal processing, or transmission processing.

[0046] The image signal processor 900 may include an encoder 200 and a decoder 600. The image signal processor 900 may receive raw image data from the color filter 100 and provide the data to the application processor 700 through the encoder 200 and the decoder 600. Since the raw image data is compressed by the encoder 200 to reduce the size of the image data, the storage space efficiency and bandwidth efficiency of the electronic device 1 can be enhanced.

[0047] The following will refer to Figures 5 to 7 Describe the encoder 200.

[0048] Figure 5 And Figure 6 Are block diagrams for illustrating an encoder according to some embodiments of the present disclosure. Figure 7 Is for illustrating the operation Figure 6 Of the encoder.

[0049] Refer to Figure 5 According to, the encoder 200 may include a bad pixel detector 210, a compressor 220, a reconstructor 230, and a buffer 300. The encoder 200 may compress the provided raw pixel data to output a bitstream with encoded data.

[0050] The bad pixel detector 210 may detect bad pixels in pixel data obtained by a Bayer color filter. The bad pixels may include static bad pixels caused by physical errors at the positions of the Bayer color filter and dynamically generated bad pixels that irregularly occur. In some cases, the bad pixel detector 210 may compare the signal levels of a plurality of pixels horizontally located around the pixel to be inspected. The comparison may be used to determine whether the pixel to be inspected is included in an edge region of the entire image, and if the pixel to be inspected is not included in the edge region, to determine whether the pixel to be inspected is defective. In some embodiments, bad pixels may be detected by comparing the signal levels of the peripheral pixels of a target pixel. The bad pixel detector 210 may mark the pixel determined to be a bad pixel with attribute information (e.g., a flag) indicating the bad pixel.

[0051] The compressor 220 may perform encoding of the original pixel data. In some embodiments, the compressor 220 may receive the original pixel data indicating that bad pixels may be provided from the bad pixel detector 210. The compressor 220 may generate a bitstream having the encoded data of the original pixel data. For example, the compressor 220 may perform differential pulse code modulation (DPCM) that performs encoding based on the difference between the original pixel data and reference pixel data to generate a bitstream. However, embodiments according to the technical concept of the present disclosure are not limited thereto, and a bitstream may be generated in another method. The compressor 220 may provide the generated bitstream to the reconstructor 230.

[0052] The reconstructor 230 may receive the bitstream from the compressor 220, and may reconstruct the bitstream to generate reference pixel data. The reference pixel data may correspond to the original pixel data. The reconstructor 230 may provide the reference pixel data to the buffer 300.

[0053] Buffer 300 can receive and store the reference pixel data reconstructed from reconstructor 230. The memory may include, but is not limited to, volatile memories (such as dynamic random access memory (DRAM) and static random access memory (SRAM)), and may also include non-volatile memories (such as flash memory, PRAM (phase change random access memory), MRAM (magnetic random access memory), ReRAM (resistive random access memory), and FRAM (ferroelectric random access memory)). Examples of memory devices include solid state memory and hard disk drives. In some examples, the memory is used to store computer-readable software including instructions, computer-executable software, where the instructions, when executed, cause the processor to perform the various functions described herein. In some cases, among other things, the memory contains a basic input / output system (BIOS) that controls basic hardware or software operations (such as interaction with peripheral components or devices). In some cases, a storage controller operates the memory cells. For example, the storage controller may include a row decoder, a column decoder, or both a row decoder and a column decoder. In some cases, the memory cells within the memory store information in the form of logical states.

[0054] Buffer 300 can provide the reference pixel data for encoding the original pixel data to compressor 220. The reference pixel data may be pixel data located around the original pixel data. Additionally or alternatively, the reference pixel data may be pixel data located around the original pixel data of a previous frame image.

[0055] Referring to Figure 6 and Figure 7 , buffer 300 may include a reference pixel buffer 310, a reference boundary pixel buffer 320, a classifier 330, and a latch 340. The reconstructed reference pixel data may be provided from reconstructor 230 to buffer 300. The classifier 330 may receive the reference pixel data.

[0056] The classifier 330 can determine whether the reconstructed reference pixel data corresponds to the boundary pixel data (S250). For example, the classifier 330 can determine whether the reconstructed reference pixel data is included in the boundary pixel image of the Bayer image 500. For example, referring to Figure 3 , the classifier 330 can determine whether the reconstructed reference pixel data is included in the first boundary pixel data 530. For example, referring to Figure 4 , the classifier 330 can determine whether the reconstructed reference pixel data is included in the second boundary pixel data 540.

[0057] Referring again to Figure 7, if the reconstructed reference pixel data corresponds to the boundary pixel data (S250 - Yes), the reconstructed reference pixel data can be stored in the reference boundary pixel buffer 320 (S251). For example, the reference boundary pixel data can be stored in the reference boundary pixel buffer 320. The reference boundary pixel buffer 320 can provide the stored reference boundary pixel data to the compressor 220 through the latch 340 (S252). For example, the reference boundary pixel data provided to the latch can be provided to the compressor 220 with a delay. In some cases, the latch may include a 1-bit memory cell. The latch can allow the circuit to store data and deliver the data at a later time (e.g., when providing the reference boundary pixel data, the latch can introduce a delay), rather than delivering the data when it is obtained. Therefore, the reference pixel data of the first boundary image of the previous frame image can be delayed to be delivered at substantially the same time as the reference pixel data of the second boundary pixel image of the current frame image.

[0058] If the reconstructed reference pixel data does not correspond to the boundary pixel data (S250 - No), the reconstructed reference pixel data can be stored in the reference pixel buffer 310 (S253). The reference pixel buffer 310 can provide the stored reference pixel data to the compressor 220 (S254). For example, different from the reference boundary pixel data, the reference pixel data can be provided to the compressor 220 without delay.

[0059] Since the reference boundary pixel data is provided with a delay and the reference pixel data is provided without delay, the reference boundary pixel data can include the image of the previous frame. Therefore, the reference pixel data can include the image of the current frame.

[0060] Figure 8 is a diagram for explaining a method of compressing raw pixel data according to some embodiments of the present disclosure.

[0061] The encoder 200 can encode the raw pixel data based on at least one reference pixel data. For example, the encoder 200 can encode the raw pixel data 510 based on the reference pixel data 520. Here, the raw pixel data 510 and the reference pixel data 520 may not be included in the boundary pixel image of the Bayer image 500. For example, the raw pixel data 510 and the reference pixel data 520 are not limited to the boundary pixel image of the Bayer image 500. For example, the pixels two rows and two columns above the raw pixel data 510 can be used as the reference pixel data 520.

[0062] In some embodiments, the original pixel data 510 may include a first original pixel GT0 that is a green pixel, a second original pixel RT0 that is a red pixel, a third original pixel GT1 that is a green pixel, and a fourth original pixel RT1 that is a red pixel. In other embodiments, although the second original pixel RT0 and the fourth original pixel RT1 may be blue pixels, for the sake of convenience of explanation, it is assumed that the second original pixel RT0 and the fourth original pixel RT1 are red pixels. Each of the original pixel data 510 is a pixel before being compressed. Each pixel value may be represented by a value greater than or equal to 0 and less than 1024.

[0063] The bitstream B1 generated by encoding the original pixel data 510 may include pixel regions DPCM1, DPCM2, DPCM3, and DPCM4 that store information about the pixel values. For example, the first pixel region DPCM1 may store the encoded data of the first original pixel GT0. The second pixel region DPCM2 may store the encoded data of the second original pixel RT0. The third pixel region DPCM3 may store the encoded data of the third original pixel GT1. The fourth pixel region DPCM4 may store the encoded data of the fourth original pixel RT1.

[0064] The encoder 200 may store the difference d1 between the pixel value of the first original pixel GT0 and the reference value in the first pixel region DPCM1. According to some embodiments, the reference value may include the average value between the pixel value of the reference pixel G0 located diagonally above and to the left of the first original pixel GT0 and the pixel value of the reference pixel G1 located diagonally above and to the right of the first original pixel GT0. The difference d1 may be defined by Equation 1 below.

[0065] d1 = (G0 + G1) / 2 - GT0 (1)

[0066] The encoder 200 may store the difference d3 between the pixel value of the second original pixel RT0 and the reference value in the second pixel region DPCM2. According to some embodiments, the reference value may include the pixel value of the reference pixel R1 located one row above two rows of the second original pixel RT0. The difference d3 may be defined by Equation 2 below.

[0067] d3 = R1 - RT0 (2)

[0068] The encoder 200 may store the difference d2 between the pixel value of the third original pixel GT1 and the reference value in the third pixel region DPCM3. According to some embodiments, the reference value may include the average value between the pixel value of the reference pixel G1 located diagonally above and to the left of the third original pixel GT1 and the pixel value of the reference pixel G2 located diagonally above and to the right. The difference d2 may be defined by Equation 3 below.

[0069] d2 = (G1 + G2) / 2 - GT1 (3)

[0070] The encoder 200 may store the difference d4 between the pixel value of the fourth original pixel RT1 and the reference value in the fourth pixel region DPCM4. According to some embodiments, the reference value may include the pixel value of the reference pixel R2 located two rows and two columns above the fourth original pixel RT1. The difference d4 may be defined by Equation 4 below.

[0071] d4 = R2 - RT1 (4)

[0072] In the encoding of the original pixel data referred to Figure 8 The positions of the reference pixel data referred to in the encoding of the original pixel data are examples and may be changed.

[0073] Hereinafter, reference will be made to Figure 9 and Figure 10 to describe a method of compressing the original pixel data when the reference pixel data is included in the border pixel image of the Bayer image 500.

[0074] Figure 9 and Figure 10 are diagrams for illustrating a method of compressing the original pixel data according to some embodiments of the present disclosure.

[0075] The encoder 200 may encode the original pixel data based on at least one reference pixel data. For example, referring to Figure 9 , the encoder 200 may encode the original pixel data 550 based on the first reference border pixel data 560. Here, although the original pixel data 550 may be included in the border pixel image of the Bayer image 500, it is not limited thereto according to the embodiments of the present disclosure. In addition, the first reference border pixel data 560 may be included in the border pixel image of the Bayer image 500. For example, the first reference border pixel data 560 may be included in the border pixel image of the Bayer image 500 of the previous frame image.

[0076] Referring to Figure 6 , the original pixel data 550 may be stored in the reference pixel buffer 310 and provided to the compressor 220 from the reference pixel buffer 310. The first reference border pixel data 560 may be stored in the reference border pixel buffer 320 and provided to the compressor 220 with a delay. For example, the first reference border pixel data 560 may be included in the border pixel image of the Bayer image 500 of the previous frame image of the original pixel data 550 to be compressed.

[0077] When compressing the boundary pixel image of the Bayer image 500 by encoding the original pixel data 550 using the first reference boundary pixel data 560 corresponding to the previous frame, compression loss can be reduced by using a pixel value having a value closest to the pixel value as the reference pixel data.

[0078] Referring Figure 10 , the original pixel data 550 may include a first original pixel GB0 that is a green pixel, a second original pixel RB0 that is a red pixel, a third original pixel GB1 that is a green pixel, and a fourth original pixel RB1 that is a red pixel. In some other embodiments, although the second original pixel RB0 and the fourth original pixel RB1 may be blue pixels, for ease of explanation, it is assumed that the second original pixel RB0 and the fourth original pixel RB1 are red pixels. Each of the original pixel data 550 is a pixel before being compressed, and each pixel value may be represented by a value greater than or equal to 0 and less than 1024.

[0079] The bitstream B2 generated by encoding the original pixel data 550 may include pixel regions DPCM5, DPCM6, DPCM7, and DPCM8 that store information about the pixel values.

[0080] The encoder 200 may store the difference d5 between the pixel value of the first original pixel GB0 and the reference value in the fifth pixel region DPCM5. According to some embodiments, the reference value may include the average value between the pixel value of the reference pixel G8 located diagonally above and to the left of the first original pixel GB0 and the pixel value of the reference pixel G9 located diagonally above and to the right of the first original pixel GB0. The difference d5 may be defined by Equation 5 below.

[0081] D5 = (G8 + G9) / 2 - GB0 (5)

[0082] The encoder 200 may store the difference d7 between the pixel value of the second original pixel RB0 and the reference value in the sixth pixel region DPCM6. According to some embodiments, the reference value may include the pixel value of the reference pixel R5 located two rows and two columns above the second original pixel RB0. The difference d7 may be defined by Equation 6 below.

[0083] d7 = R5 - RB0 (6)

[0084] The encoder 200 may store the difference d6 between the pixel value of the third original pixel GB1 and the reference value in the seventh pixel region DPCM7. According to some embodiments, the reference value may include the average value between the pixel value of the reference pixel G9 located diagonally above and to the left of the third original pixel GB1 and the pixel value of the reference pixel G10 located diagonally above and to the right of the third original pixel GB1. The difference d6 may be defined by Equation 7 below.

[0085] d6 = (G9 + G10) / 2 - GB1 (7)

[0086] The encoder 200 may store the difference d8 between the pixel value of the fourth original pixel RB1 and the reference value in the eighth pixel region DPCM8. According to some embodiments, the reference value may include the pixel value of the reference pixel R6 located two rows and two columns above the fourth original pixel RB1. The difference d8 may be defined by Equation 8 below.

[0087] d8 = R6 - RB1 (8)

[0088] In the reference Figure 10 The position of the reference pixel data referred to in the encoding of the original pixel data described is an example and may be changed.

[0089] Figure 11 is a diagram for explaining a method of compressing original pixel data according to some embodiments of the present disclosure. For ease of explanation, the repeated parts of the content using Figures 1 to 10 will be briefly described or omitted.

[0090] Referring to Figure 11 , the encoder 200 may encode the original pixel data 550 based on the second reference boundary pixel data 570. The second reference boundary pixel data 570 may be included in the boundary pixel image of the Bayer image 500. For example, the second reference boundary pixel data 570 may be included in the boundary pixel image of the Bayer image 500 of the previous frame image. Referring to Figure 4 , the second reference boundary pixel data 570 may be included in the second boundary pixel data 540. For example, the second reference boundary pixel data 570 may include the pixel images of the first and second rows and the first and second columns of the boundary of the Bayer image 500.

[0091] Figure 12 is a diagram for explaining a method of compressing original pixel data according to some embodiments of the present disclosure. For ease of explanation, the repeated parts of the content using Figures 1 to 10 will be briefly described or omitted.

[0092] Referring to Figure 12, the encoder 200 may encode the original pixel data 550 based on the third reference boundary pixel data 580. The third reference boundary pixel data 580 may be included in the boundary pixel image of the Bayer image 500. The third reference boundary pixel data 580 may include the pixel images of the first row and the pixel images of the first column of the boundary of the Bayer image 500. Additionally or alternatively, the third reference boundary pixel data 580 may further include the image of the overlapping portion of the pixel images of the first row and the pixel images of the first column of the boundary of the Bayer image 500. For example, the encoder 200 may encode the original pixel data 550 based on the pixel data located in the upper left of the original pixel data 550 in the third reference boundary pixel data 580.

[0093] Figure 13 is a diagram for explaining a method of compressing original pixel data according to some embodiments of the present disclosure. For ease of explanation, the repeated parts of the content using Figures 1 to 10 will be briefly described or omitted.

[0094] Referring to Figure 13 , the encoder 200 may encode the original pixel data 550 based on the fourth reference boundary pixel data 590. The fourth reference boundary pixel data 590 may be the average value of the pixel values included in the boundary pixel image of the Bayer image 500 of the previous frame image. Since the fourth reference boundary pixel data 590 includes the average value of the pixel values, the stored data can be reduced and the reliability can be improved.

[0095] Figure 14 is a diagram for explaining a method of compressing original pixel data according to some embodiments of the present disclosure. For ease of explanation, the repeated parts of the content using Figures 1 to 10 will be briefly described or omitted.

[0096] Referring to Figure 14 , the Bayer image 500 may include a first region image 501 and a second region image 502. The first region image 501 and the second region image 502 may include different regions from each other. The first region image 501 may include the original pixel data 551. The second region image 502 may include the original pixel data 552.

[0097] The encoder 200 may encode the original pixel data 551 of the first region image 501 based on the fifth reference boundary pixel data 591, and the fifth reference boundary pixel data 591 may be the boundary pixel image of the first region image 501 corresponding to the previous frame. The encoder 200 may encode the original pixel data 552 of the second region image 502 based on the sixth reference boundary pixel data 592, and the sixth reference boundary pixel data 592 may be the boundary pixel image of the second region image 502 corresponding to the previous frame. By encoding other regions in the Bayer image 500, the compression reliability can be improved and the compression loss can be reduced.

[0098] Hereinafter, reference will be made to Figure 15 and Figure 16 to describe the encoder 200. For ease of explanation, the repeated parts of the content using Figures 1 to 10 will be briefly described or omitted.

[0099] Figure 15 is a block diagram for explaining an encoder according to some embodiments of the present disclosure. Figure 16 is for explaining the operation of Figure 15 the encoder.

[0100] Referring to Figure 15 , the encoder 200 may include a motion detector 240. The encoder 200 may use the motion detector 240 to adjust the data provided from the buffer 300 to the compressor 220.

[0101] Referring to Figure 16 , the motion detector 240 may detect the motion of the electronic device 1 having the encoder 200 (S260). The motion detector 240 may determine whether the motion of the electronic device 1 having the encoder 200 is greater than a reference value (S261).

[0102] If the detected movement of the electronic device 1 is greater than a reference value (S261 - Yes), the encoder 200 may use a predetermined value as reference pixel data (S262). When the detected movement of the electronic device 1 is not greater than the reference value (S261 - No), the encoder 200 may use boundary pixel data as reference pixel data (S263). If the movement of the electronic device 1 is not smooth, since the pixel values of the previous frame may not be similar to the pixel values of the current frame, the predetermined value may be used as reference pixel data to provide enhanced compressibility to the encoder 200. For example, the motion detector 240 may detect whether the electronic device 1 has moved to or repositioned to an extent where the subsequent frame image may not have a boundary image similar to the previously captured frame image. In some examples, the motion detector 240 may include an accelerometer, a gyroscope, an inertial measurement unit (IMU), etc. In some examples, the motion detector 240 may be any sensor capable of determining that the pixel values of the previous frame may not be similar to the corresponding pixel values of the current frame.

[0103] Figure 17 is a block diagram illustrating an electronic device having an image encoder according to some embodiments of the present disclosure. For ease of explanation, repeated parts of the content described using Figures 1 to 16 will be briefly described or omitted.

[0104] Referring to Figure 17 , the electronic device 2 may include an image sensing device 800, a memory 820, and an application processor 700.

[0105] The image sensing device 800 may include an encoder 200, a decoder 600, and a storage controller 810. The bitstream generated by the encoder 200 may be sent to the storage controller 810.

[0106] The storage controller 810 may control the input and output operations of the encoded data of the memory 820. The bitstream generated by the encoder 200 may be input to the memory 820 under the control of the storage controller 810. The storage controller 810 may include dedicated logic circuits (e.g., FPGA, ASIC, etc.) that perform various operations for controlling the overall operation within the memory 820.

[0107] The memory 820 is connected to the image sensing device 800 and may store image frames. The memory 820 may store the encoded data generated by the encoder 200 instead of the original data of the image frames. Therefore, the number of image frames stored in the memory 820 may increase compared to the case where the original data is stored in the memory 820.

[0108] The memory 820 may output a bitstream having encoded data to the decoder 600 under the control of the storage controller 810. The decoder 600 may perform an operation of decoding the bitstream. For example, the decoder 600 may generate reconstructed image data based on the bitstream received from the storage controller 810.

[0109] Figure 18 is a block diagram of an electronic device having an image encoder according to some embodiments of the present disclosure. For ease of explanation, repeated portions of the content using Figures 1 to 16 will be briefly described or omitted.

[0110] Referring to Figure 18 , the electronic device 3 may include an application processor 700 and a display device 910.

[0111] The application processor 700 may include an encoder 200 and a decoder 600. The encoder 200 in the application processor 700 may encode and compress the original image data. The decoder 600 in the application processor 700 may decode the encoded bitstream to output image data.

[0112] The application processor 700 may transmit the image data output from the decoder 600 to the display device 910. The application processor 700 compresses the image data input to the application processor 700 through the encoder 200 and the decoder 600 losslessly, decompresses the image data losslessly, and transmits the image data to the display device 910 to display the image data. The display device 910 may include a conventional monitor, a monitor coupled to an integrated display, an integrated display (e.g., an LCD display), or other devices for viewing associated data or processing information. Output devices other than a display (such as a printer, other computers, or data storage devices, and computer networks) may be used.

[0113] Hereinafter, referring to Figure 19 and Figure 20 an electronic device having multiple camera modules 1100a, 1100b, and 1100c will be described. For ease of explanation, repeated portions of the content using Figures 1 to 16 will be briefly described or omitted. Each of the camera modules 1100a, 1100b, and 1100c may include the same encoder 200 as the encoder described using Figures 1 to 16 .

[0114] Figure 19 is a block diagram of an electronic device having multiple camera modules according to some embodiments of the present disclosure. Figure 20 is Figure 19 a detailed block diagram of the camera module.

[0115] Referring toFigure 19 The electronic device 4 may include a camera module group 1100, an application processor 1200, a power management integrated circuit (PMIC) 1300, and an external memory 1400.

[0116] The camera module group 1100 may include a plurality of camera modules 1100a, 1100b, and 1100c. Although an example in which three camera modules 1100a, 1100b, and 1100c are arranged is shown in the drawings, embodiments are not limited thereto. In some embodiments, the camera module group 1100 may include two camera modules. Further, in some embodiments, the camera module group 1100 may include n (n is a natural number equal to or greater than 4) camera modules.

[0117] Hereinafter, although Figure 20 the detailed configuration of the camera module 1100b will be described more specifically, according to embodiments, the following description may be similarly applied to the other camera modules 1100a and 1100c.

[0118] Referring to Figure 20 , the camera module 1100b may include a prism 1105, an optical path folding element (hereinafter referred to as "OPFE") 1110, an actuator 1130, an image sensing device 1140, and a memory 1150.

[0119] The prism 1105 may include a reflective surface 1107 of a light reflecting material to deform the path of light L incident from the outside.

[0120] In some embodiments, the prism 1105 may change the path of the light L incident in the first direction X to a second direction Y perpendicular to the first direction X. Further, the prism 1105 may rotate the reflective surface 1107 of the light reflecting material about the central axis 1106 in the direction A or rotate the central axis 1106 in the direction B, thereby changing the path of the light L incident in the first direction X. The OPFE 1110 may also move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0121] In some embodiments, as shown in the figure, although the maximum rotation angle of the prism 1105 in the direction A is equal to or less than 15 degrees in the positive (+) direction A and may be greater than 15 degrees in the negative (-) direction A, embodiments are not limited thereto.

[0122] In some embodiments, the prism 1105 may move about 20 degrees in the positive (+) or negative (-) direction B, or move between 10 degrees and 20 degrees, or move between 15 degrees and 20 degrees. Here, the moving angle may move the same angle in the positive (+) or negative (-) direction B, or may move to almost the same angle within a range of about 1 degree.

[0123] In some embodiments, the prism 1105 may move the reflective surface 1107 of the light reflecting material in a third direction (e.g., the direction Z) parallel to the extending direction of the central axis 1106.

[0124] The OPFE 1110 may include, for example, m (where m is a natural number) sets of optical lenses. The m lenses may be moved in the second direction Y to change the optical zoom ratio of the camera module 1100b. For example, when the basic optical zoom ratio of the camera module 1100b is set to Z, if the m optical lenses included in the OPFE 1110 are moved, the optical zoom ratio of the camera module 1100b may be changed to 3Z or 5Z or an optical zoom ratio greater than 5Z.

[0125] The actuator 1130 may move the OPFE 1110 or the optical lens (hereinafter referred to as the optical lens) to a specific position. For example, the actuator 1130 may adjust the position of the optical lens such that the image sensor 1142 is located at the focal length of the optical lens for precise sensing.

[0126] The image sensing device 1140 may include an image sensor 1142, a control logic 1144, and a memory 1146. Although the image sensing device 1140 may include the same encoder 200 as described with the encoder 200, embodiments according to the technical concept of the present disclosure are not limited thereto, and the encoder 200 may be included in other configurations of the camera module 1100b. The image sensor 1142 may sense an image of a sensing target using the light L provided by the optical lens. The control logic 1144 may control the overall operation of the camera module 1100b. For example, the control logic 1144 may control the operation of the camera module 1100b according to a control signal provided through a control signal line (CSL) (such as CSLb). Figures 1 to 16 The memory 1146 may store information (such as calibration data 1147) for the operation of the camera module 1100b. The calibration data 1147 may include information for the camera module 1100b to generate image data using the light L provided from the outside. The calibration data 1147 may include, for example, information about the above-mentioned degree of rotation, information about the focal length, information about the optical axis, etc. If the camera module 1100b is implemented in the form of a multi-state camera in which the focal length changes according to the position of the optical lens, the calibration data 1147 may include the focal length value and autofocus information for each position (each state) of the optical lens.

[0127] The memory 1146 may store information (such as calibration data 1147) for the operation of the camera module 1100b. The calibration data 1147 may include information for the camera module 1100b to generate image data using the light L provided from the outside. The calibration data 1147 may include, for example, information about the above-mentioned degree of rotation, information about the focal length, information about the optical axis, etc. If the camera module 1100b is implemented in the form of a multi-state camera in which the focal length changes according to the position of the optical lens, the calibration data 1147 may include the focal length value and autofocus information for each position (each state) of the optical lens.

[0128] The memory 1150 may store image data sensed by the image sensor 1142. For example, the memory 1150 may store the image data (e.g., bitstream) encoded by the encoder 200. The memory 1150 may be disposed outside the image sensing device 1140 and may be implemented in a form stacked with the sensor chip constituting the image sensing device 1140. In some embodiments, although the memory 1150 may be implemented as an EEPROM (electrically erasable programmable read-only memory), the embodiments are not limited thereto.

[0129] Referring together Figure 19 and Figure 20 , in some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Accordingly, each of the plurality of camera modules 1100a, 1100b, and 1100c may include the same or different calibration data 1147 according to the operation of the actuator 1130 included therein.

[0130] In some embodiments, one of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., 1100b) may be a folded lens type camera module having the prism 1105 and the OPFE 1110 described above, and the remaining camera modules (e.g., 1100a and 1100c) may be vertical type camera modules that do not include the prism 1105 and the OPFE 1110. However, the embodiments are not limited thereto.

[0131] In some embodiments, one of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., 1100c) may be a vertical depth camera that extracts depth information using, for example, infrared (IR) rays. In this case, the application processor 1200 may merge the image data provided from such a depth camera and the image data provided from other camera modules (e.g., 1100a or 1100b) to generate a 3D depth image.

[0132] In some embodiments, at least two of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., 1100a and 1100b) may have different fields of view from each other. In this case, for example, at least two of the plurality of camera modules 1100a, 1100b, and 1100c (e.g., 1100a and 1100b) may have different optical lenses, but the present disclosure is not limited thereto.

[0133] In addition, in some embodiments, the fields of view of each of the plurality of camera modules 1100a, 1100b, and 1100c may be different from each other. In this case, although the optical lenses included in each of the plurality of camera modules 1100a, 1100b, and 1100c may be different from each other, the embodiments are not limited thereto.

[0134] In some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may be positioned to be physically separated from each other. In other words, instead of using a sensing area of a single image sensor 1142 alone, independent image sensors 1142 may be disposed inside each of the plurality of camera modules 1100a, 1100b, and 1100c.

[0135] Referring again to Figure 19 , the application processor 1200 may include an image processor 1210, a memory controller 1220, and an internal memory 1230. The application processor 1200 may be implemented separately from the plurality of camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the plurality of camera modules 1100a, 1100b, and 1100c may be implemented by separate semiconductor chips from each other.

[0136] The image processor 1210 may include a plurality of sub-image processors 1212a, 1212b, and 1212c, an image generator 1214, and a camera module controller 1216.

[0137] The image processor 1210 may include a plurality of sub-image processors 1212a, 1212b, and 1212c corresponding to the number of the plurality of camera modules 1100a, 1100b, and 1100c.

[0138] The image data generated from each of the camera modules 1100a, 1100b, and 1100c can be provided to the corresponding sub-image processors 1212a, 1212b, and 1212c through separate image signal lines (such as ISLa, ISLb, and ISLc). The image data provided through the image signal lines ISLa, ISLb, and ISLc can include the bitstream output from the encoder 200. For example, the image data generated from the camera module 1100a can be provided to the sub-image processor 1212a through the image signal line ISLa. The image data generated from the camera module 1100b can be provided to the sub-image processor 1212b through the image signal line ISLb. The image data generated from the camera module 1100c can be provided to the sub-image processor 1212c through the image signal line ISLc. Although the image data transmission can be performed, for example, using a camera serial interface (CSI) based on the Mobile Industry Processor Interface (MIPI), the embodiments are not limited thereto.

[0139] In addition, in some embodiments, one sub-image processor can be arranged to correspond to multiple camera modules. For example, the sub-image processors 1212a and 1212c are not implemented separately from each other as shown in the figure. The sub-image processors 1212a and 1212c can be implemented by integrating them into a single sub-image processor, selecting the image data provided from the camera modules 1100a and 1100c through a selection element (such as a multiplexer), etc., and then the image data can be provided to the integrated sub-image processor.

[0140] The image data provided to each of the sub-image processors 1212a, 1212b, and 1212c can be provided to the image generator 1214. The image generator 1214 can generate an output image using the image data provided from the respective sub-image processors 1212a, 1212b, and 1212c according to the image generation information or the mode signal.

[0141] The image generator 1214 can merge at least some of the image data generated from the camera modules 1100a, 1100b, and 1100c having different fields of view according to the image generation information or the mode signal to generate an output image. Additionally or alternatively, the image generator 1214 can select one of the image data generated from the camera modules 1100a, 1100b, and 1100c having different fields of view to generate an output image according to the image generation information or the mode signal.

[0142] In some embodiments, the image generation information can include a zoom signal or a zoom factor. In addition, in some embodiments, the mode signal can be, for example, a signal based on the mode selected by the user.

[0143] When the image generation information is a zoom signal (zoom factor) and each of the camera modules 1100a, 1100b, and 1100c has a different viewing view (field of view) from each other, the image generator 1214 may perform different operations from each other according to the type of the zoom signal. For example, if the zoom signal is a first signal, after combining the image data output from the camera module 1100a and the image data output from the camera module 1100c, the combined image signal and the image data not used for combination output from the camera module 1100b may be used to generate an output image. If the zoom signal is a second signal different from the first signal, the image generator 1214 does not perform image data combination, but may select one of the image data output from the respective camera modules 1100a, 1100b, and 1100c to generate an output image. However, the embodiments are not limited thereto, and the method of processing the image data may be variously modified and implemented as needed.

[0144] In some embodiments, the image generator 1214 may receive a plurality of image data having different exposure times from at least one of the plurality of sub-image processors 1212a, 1212b, and 1212c, and perform high dynamic range (HDR) processing on the plurality of image data, thereby generating combined image data having an increased dynamic range.

[0145] The camera module controller 1216 may provide a control signal to each of the camera modules 1100a, 1100b, and 1100c. The control signal generated from the camera module controller 1216 may be provided to the respective camera modules 1100a, 1100b, and 1100c through separate control signal lines CSLa, CSLb, and CSLc.

[0146] One of the plurality of camera modules 1100a, 1100b, and 1100c may be designated as a main camera (e.g., 1100b) according to the image generation information having a zoom signal or a mode signal, and the remaining camera modules (e.g., 1100a and 1100b) may be designated as slave cameras. This information is included in the control signal and may be provided to the respective camera modules 1100a, 1100b, and 1100c through separate control signal lines CSLa, CSLb, and CSLc.

[0147] The camera modules operating as the main camera module and the slave camera module can change according to the zoom factor or the operation mode signal. For example, when the field of view of camera module 1100a is wider than that of camera module 1100b and the zoom factor indicates a low zoom rate, camera module 1100b can operate as the main camera module, and camera module 1100a can operate as the slave camera module. Additionally or alternatively, when the zoom factor indicates a high zoom rate, camera module 1100a can operate as the main camera module, and camera module 1100b can operate as the slave camera module.

[0148] In some embodiments, the control signals provided from the camera module controller 1216 to the respective camera modules 1100a, 1100b, and 1100c may include a synchronization enable signal. For example, when camera module 1100b is the main camera and camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 may send a synchronization enable signal to camera module 1100b. The camera module 1100b provided with the synchronization enable signal generates a synchronization signal based on the provided synchronization enable signal, and may provide the generated synchronization signal to camera modules 1100a and 1100c through the synchronization signal line SSL. Camera module 1100b and camera modules 1100a and 1100c may send image data to the application processor 1200 synchronously with such a synchronization signal.

[0149] In some embodiments, the control signals provided from the camera module controller 1216 to the multiple camera modules 1100a, 1100b, and 1100c may include mode information according to the mode signal. The multiple camera modules 1100a, 1100b, and 1100c may operate in a first operation mode and a second operation mode related to the sensing speed based on the mode information.

[0150] The multiple camera modules 1100a, 1100b, and 1100c generate an image signal at a first speed in the first operation mode (e.g., generate an image signal at a first frame rate). The multiple camera modules 1100a, 1100b, and 1100c also encode the image signal at a second speed higher than the first speed (e.g., encode the image signal at a second frame rate higher than the first frame rate). Additionally or alternatively, the multiple camera modules 1100a, 1100b, and 1100c may send the encoded image signal to the application processor 1200. The second speed may be equal to or less than 30 times the first speed.

[0151] The application processor 1200 stores the received image signal (e.g., the encoded image signal) in the memory 1230 disposed inside the external memory 1400 of the application processor 1200 or in the external memory 1400 of the application processor 1200, then reads the encoded image signal from the memory 1230 or the memory 1400 and decodes the encoded image signal, and can display the image data generated based on the decoded image signal. For example, the corresponding sub-processors among the plurality of sub-processors 1212a, 1212b, and 1212c of the image processor 1210 can perform decoding and can perform image processing on the decoded image signal.

[0152] The plurality of camera modules 1100a, 1100b, and 1100c generate image signals (e.g., generate image signals at a third frame rate lower than the first frame rate) at a third speed lower than the first speed in the second operation mode and can send the image signals to the application processor 1200. The image signals provided to the application processor 1200 can be non-encoded signals. The application processor 1200 can perform image processing on the received image signals or can store the image signals in the memory 1230 or the memory 1400.

[0153] The PMIC 1300 can supply power (e.g., a power supply voltage) to each of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the PMIC 1300 can supply a first power to the camera module 1100a through the power signal line PSLa, a second power to the camera module 1100b through the power signal line PSLb, and a third power to the camera module 1100c through the power signal line PSLc under the control of the application processor 1200.

[0154] The PMIC 1300 generates the power corresponding to each of the plurality of camera modules 1100a, 1100b, and 1100c in response to the power control signal PCON from the application processor 1200 and can adjust the power level. The power control signal PCON can include power adjustment signals for each operation mode of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the operation mode can include a low power mode, and the power control signal PCON can include information about the camera module operating in the low power mode and the power level to be set. The power levels provided to each of the plurality of camera modules 1100a, 1100b, and 1100c can be the same as or different from each other. In addition, the power level can be dynamically changed.

[0155] In summarizing the detailed description, those skilled in the art will appreciate that many variations and modifications can be made to the embodiments without substantially departing from the principles of the present disclosure. Accordingly, the disclosed embodiments of the present disclosure are used only in a general and descriptive sense and not for purposes of limitation.

Claims

1. An image sensing device, comprising: An image signal processor configured to receive a first frame image and a second frame image temporally after the first frame image, and generate a compressed image of the second frame image based on a boundary pixel image of the first frame image, wherein the image signal processor includes: A memory configured to store first reference pixel data as a boundary pixel image of the first frame image; and A compressor configured to receive the first reference pixel data from the memory and generate a bitstream obtained by encoding the second frame image based on a difference between the first reference pixel data and original pixel data of the second frame image, wherein the image signal processor generates a compressed image of the second frame image based on the bitstream.

2. The image sensing device according to claim 1, further comprising: A reconstructor configured to reconstruct the bitstream to generate second reference pixel data, wherein the second reference pixel data is a second boundary pixel image of the second frame image, wherein the memory is configured to store the second reference pixel data and provide the second reference pixel data to the compressor.

3. The image sensing device according to claim 2, wherein, The compressor is configured to generate a bitstream obtained by encoding the second frame image based on a second difference between the second frame image and the second reference pixel data provided from the memory.

4. The image sensing device according to claim 2, wherein, The memory includes a reference pixel buffer memory and a reference boundary pixel buffer memory, and The first reference pixel data is stored in the reference boundary pixel buffer memory, and the second reference pixel data is stored in the reference pixel buffer memory.

5. The image sensing device according to claim 4, wherein, The first reference pixel data provided from the reference boundary pixel buffer memory is provided through a latch.

6. The image sensing device according to claim 4, wherein, The memory is configured to provide the first reference pixel data and the second reference pixel data to the compressor simultaneously.

7. The image sensing device according to claim 1, wherein, The compressor is configured to generate a bitstream based on a second difference between the second reference pixel data and the first reference pixel data provided from the memory, wherein the second reference pixel data is a second boundary pixel image of the second frame image.

8. The image sensing device according to claim 1, wherein, The boundary pixel image of the first frame image includes: an image included in the first row of the first frame image.

9. The image sensing device according to claim 1, wherein, The boundary pixel image of the first frame image includes: an image included in the first column of the first frame image.

10. The image sensing device according to claim 1, wherein, The memory is configured to store third reference pixel data as a boundary pixel image of the first frame image, wherein the third reference pixel data is different from the first reference pixel data, and The compressor is configured to generate a bitstream based on a second difference between the original pixel data of the second frame image and an average value of the first reference pixel data and the third reference pixel data provided from the memory.

11. The image sensing device according to claim 1, further comprising: An action detector configured to detect whether an action of the image sensing device is greater than a reference value, wherein, based on the action detector detecting that the action of the image sensing device is greater than the reference value, the compressor generates a bitstream based on a second difference between the original pixel data of the second frame image and a constant reference pixel data.

12. The image sensing device according to claim 1, further comprising: An image decoder configured to decode the bitstream.

13. An image encoder configured to receive a first frame image and a second frame image temporally after the first frame image, the image encoder comprising: a memory configured to store first reference pixel data as a boundary pixel image of the first frame image; a compressor configured to receive the first reference pixel data from the memory, generate a first bitstream obtained by encoding the original pixel data of the second frame image based on a difference between the first reference pixel data and the original pixel data of the second frame image, and output the generated first bitstream; and a reconstructor configured to reconstruct the first bitstream to generate second reference pixel data, wherein the second reference pixel data is a second boundary pixel image of the second frame image.

14. The image encoder according to claim 13, wherein, The memory includes a reference pixel buffer memory and a reference boundary pixel buffer memory, and the first reference pixel data is stored in the reference boundary pixel buffer memory, and the second reference pixel data is stored in the reference pixel buffer memory.

15. The image encoder according to claim 14, wherein, The first reference pixel data is provided with a delay from the reference boundary pixel buffer memory.

16. The image encoder according to claim 14, wherein, The memory is configured to provide the first reference pixel data and the second reference pixel data to the compressor simultaneously.

17. The image encoder according to claim 13, wherein, The memory is configured to store the second reference pixel data, and the compressor is configured to receive the second reference pixel data from the memory and generate a second bitstream, wherein the second bitstream is obtained by encoding the second original pixel data of the third frame image based on a second difference between the second reference pixel data and the second original pixel data of the third frame image temporally after the second frame image.

18. The image encoder according to claim 17, wherein, The reconstructor is configured to reconstruct the second bitstream to generate third reference pixel data, wherein the third reference pixel data is a third boundary pixel image of the third frame image.

19. A method for operating an image encoder, the method comprising: receiving a first frame image and a second frame image, wherein the second frame image is received at a different time from the reception of the first frame image; storing first reference pixel data as a first boundary pixel image of the first frame image; generating a bitstream obtained by encoding the original pixel data of the second frame image based on a difference between the stored first reference pixel data and the original pixel data of the second frame image; outputting the generated bitstream; and reconstructing the bitstream to generate second reference pixel data, wherein the second reference pixel data is a second boundary pixel image of the second frame image.

20. The method for operating an image encoder according to claim 19, further comprising: generating a bitstream based on a second difference between the second original pixel data of the second boundary pixel image and the stored first reference pixel data.

Images