Frame buffer compression circuit and image processing apparatus

Abstract

Frame buffer compression circuitry and image processing apparatus are provided. The image processing apparatus includes a memory configured to store compressed data, and frame buffer compression circuitry including an encoder circuit configured to compress at least some of source data to generate compressed data and send the compressed data to the memory, and a decoder circuit configured to read the compressed data from the memory and decompress the compressed data, wherein the compressed data includes a header and a payload including actual compressed data and a flag, the frame buffer compression circuitry is configured to reflect, in the flag, a result obtained by comparing an accumulated compression rate corresponding to the compressed data with a reference compression rate, and configured to perform compression or decompression in a lossy mode or a lossless mode according to the flag.

Description

Technical Field

[0001] The present invention relates to an image processing device. Background Technology

[0002] With the increasing demand for high-resolution and high-frame-rate video images, the amount of multimedia IP access memory (i.e., bandwidth) in image processing devices has increased significantly.

[0003] When bandwidth increases, the processing capacity of image processing devices reaches its limit, which can lead to a decrease in speed during video image recording or playback operations.

[0004] Therefore, the type of data compression when accessing storage via multimedia IP is being considered. For example, data can be compressed before being written to storage, and the compressed data can be decompressed before being read from storage. Summary of the Invention

[0005] An aspect of the present invention provides an image processing apparatus with improved image quality and / or resolution, and a frame buffer compression circuit used in the image processing apparatus.

[0006] One aspect of the present invention provides an image processing apparatus comprising: a multimedia IP for processing raw data to generate source data and receiving and using output data; a frame buffer compression circuit that operates in a lossy or lossless mode to compress the source data into compressed data or decompress the compressed data into output data; and a memory for storing the compressed data and accessed by the multimedia IP, wherein the frame buffer compression circuit selects a lossy or lossless mode based on the cumulative compression ratio of the compressed data and performs compression or decompression based on the selected mode.

[0007] Another aspect of the present invention provides a frame buffer compression circuit, the frame buffer compression circuit comprising: an encoder circuit for receiving source data and generating compressed data; and a decoder circuit for decompressing the compressed data and outputting output data, wherein when the cumulative compression ratio of the compressed data exceeds a reference compression ratio, the encoder circuit performs compression in lossy mode; otherwise, the encoder circuit performs compression in lossless mode, wherein the decoder circuit performs decompression according to the compression mode corresponding to the compressed data.

[0008] Other aspects of the present invention provide an image processing apparatus comprising: a memory for storing compressed data; and a frame buffer compression circuit including an encoder circuit and a decoder circuit, the encoder circuit being configured to compress at least some of the source data to generate compressed data and send the compressed data to the memory, the decoder circuit being configured to read the compressed data from the memory and decompress the compressed data, wherein the compressed data includes a payload and a header, the header including actual compressed data and a flag, wherein the frame buffer compression circuit reflects in the flag a result obtained by comparing a cumulative compression ratio corresponding to the compressed data with a reference compression ratio, and performs compression or decompression in a lossy mode or a lossless mode according to the flag.

[0009] However, the aspects of the inventive concept are not limited to those set forth herein. These and other aspects of the inventive concept will become more apparent to those skilled in the art by referring to the specific embodiments of the inventive concept given below. Attached Figure Description

[0010] Figure 1 These are block diagrams illustrating an image processing apparatus based on some exemplary embodiments of the present invention.

[0011] Figure 2 It is used for detailed explanation Figure 1 A block diagram of a frame buffer compressor.

[0012] Figure 3 It is shown Figure 2 The flowchart shows the operation method of the encoder.

[0013] Figure 4 This is a diagram showing the cell blocks included in a single frame of source data.

[0014] Figure 5 It is shown Figure 4 A diagram of the unit block.

[0015] Figure 6 It is shown Figure 4 A diagram showing multiple unit blocks.

[0016] Figure 7 This is for illustrative purposes based on some example embodiments. Figure 2 A block diagram of the encoder.

[0017] Figure 8 This is for illustrative purposes based on some example embodiments. Figure 2 A block diagram of the encoder.

[0018] Figure 9 It is shown Figure 2 The flowchart shows the operation method of the decoder.

[0019] Figure 10 This is for illustrative purposes based on some example embodiments. Figure 2 A block diagram of the decoder.

[0020] Figure 11 These are block diagrams illustrating an image processing apparatus based on some exemplary embodiments of the present invention.

[0021] Figure 12 These are block diagrams illustrating an image processing apparatus based on some exemplary embodiments of the present invention.

[0022] Figure 13 These are block diagrams illustrating an image processing apparatus based on some exemplary embodiments of the present invention. Detailed Implementation

[0023] In the following text, reference will be made to Figures 1 to 13 An image processing apparatus according to some exemplary embodiments of the present invention is described.

[0024] Figure 1 These are block diagrams illustrating an image processing apparatus based on some exemplary embodiments of the present invention. Figure 2 It is used for detailed explanation Figure 1 A block diagram of a frame buffer compressor.

[0025] Reference Figure 1 and Figure 2 The image processing apparatus according to some example embodiments of the present invention includes a multimedia IP (intellectual property) (or a multimedia IP block) 100, a frame buffer compressor 200, a memory 300 and / or a system bus 400.

[0026] Multimedia IP 100 can be the part that directly performs image processing on the image processing device. That is, Multimedia IP 100 can represent various modules for recording and playing back images (such as CAM encoding and playback of video images).

[0027] Multimedia IP 100 can receive raw data from an external source, such as a camera, and convert the raw data into source data 10. The raw data can be raw motion image data or raw image data. Source data 10 can also include data generated and processed by Multimedia IP 100. That is, Multimedia IP 100 can repeatedly store data obtained by processing raw data through multiple operations in memory 300 and update the data again. Source data 10 can include all data during this operation. However, because source data 10 can be stored in memory 300 as compressed data 20, the source data 10 in memory 300 can represent data prior to storage in memory 300 or data after being read from memory 300. This will be explained in more detail below.

[0028] For example, multimedia IP 100 may include an image signal processor (ISP) 110, an image stabilization module (G2D) 120, a multi-format codec (MFC) 130, a GPU 140, and / or a display 150. However, the example embodiment is not limited thereto. That is, multimedia IP 100 may include at least some of the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140, and display 150 described above. In other words, multimedia IP 100 may represent a processing module that needs to access memory 300 to process moving images or images.

[0029] The image signal processor 110 can receive raw data, preprocess the raw data, and convert the raw data into source data 10. At this time, the raw data can be RGB type image raw data. For example, the image signal processor 110 can convert RGB type raw data into YUV type source data 10.

[0030] In this context, RGB data represents a data format based on the three primary colors of light. That is, it's a type of image representation that uses three colors: red, green, and blue. Conversely, YUV data represents a data format where luminance (i.e., the luminance signal and chrominance signal) are represented separately. Specifically, Y represents the luminance signal, and U (Cb) and V (Cr) represent the chrominance signals. U represents the difference between the luminance signal and the blue signal component, and V represents the difference between the luminance signal and the red signal component.

[0031] YUV format data can be obtained from RGB format data by using conversion formulas (such as Y = 0.3R + 0.59G + 0.11B, U = (BY) × 0.493, V = (RY) × 0.877).

[0032] Because the human eye is more sensitive to luminance signals but less sensitive to color signals, YUV data can be compressed more easily than RGB data. As a result, the image signal processor 110 can convert raw RGB data into YUV source data 10.

[0033] The image signal processor 110 can convert the raw data into source data 10, and then store the source data in the memory 300.

[0034] Image stabilization module 120 can perform image stabilization on both still image and moving image data. Image stabilization module 120 can perform image stabilization by reading raw data or source data 10 stored in memory 300. In this case, image stabilization refers to the operation of detecting and removing camera shake from the moving image data.

[0035] The image stabilization module 120 can correct jitter in the original data or source data 10, update or generate new source data 10, and store the new source data 10 in the memory 300.

[0036] The multi-format codec 130 can be a codec for compressing moving image data. Typically, because moving image data is very large, a compression module is needed to reduce its size. Moving image data can be compressed through the correlation between multiple frames, and this compression can be performed by the multi-format codec 130. The multi-format codec 130 can read and compress the raw data or the source data 10 stored in the memory 300.

[0037] The multi-format codec 130 can compress the original data or source data 10 to generate new source data 10 or update the source data 10, and can store the new source data 10 in the memory 300.

[0038] GPU (Graphics Processing Unit) 140 can process and generate 2D or 3D graphics. GPU 140 can perform arithmetic operations on raw data or source data 10 stored in memory 300. GPU 140 is dedicated to processing graphics data and can process graphics data in parallel.

[0039] GPU 140 can compress the original data or source data 10 to generate new source data 10 or update the source data 10, and can store the new source data 10 in memory 300.

[0040] Display 150 can display source data 10 stored in memory 300 on the screen. Display 150 can also display image data (i.e., source data 10) processed by other components of multimedia IP 100 (i.e., image signal processor 110, image stabilization module 120, multi-format codec 130, and GPU 140). However, the example embodiment is not limited thereto.

[0041] The image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140, and display 150 of the multimedia IP 100 can each operate independently. That is, each of the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140, and display 150 can independently access memory 300 to read or write data.

[0042] Before the multimedia IP 100 accesses the memory 300 independently, the frame buffer compressor 200 compresses at least some of the source data 10 and converts it into compressed data 20. The frame buffer compressor 200 can then send the compressed data 20 to the memory 300.

[0043] Therefore, the compressed data 20 compressed by the frame buffer compressor 200 can be stored in the memory 300. Conversely, when loaded by the multimedia IP 100, the compressed data 20 stored in the memory 300 can be sent to the frame buffer compressor 200. The frame buffer compressor 200 can decompress the compressed data 20 and convert it into output data 30. The frame buffer compressor 200 can then send the output data 30 back to the multimedia IP 100. Although the output data 30 should in principle be identical to the source data 10, it can be changed during the compression and decompression processes.

[0044] Whenever the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140, and display 150 of the multimedia IP 100 individually access the memory 300, the frame buffer compressor 200 may compress at least some of the source data 10 into compressed data 20 and send the compressed data 20 to the memory 300. Conversely, whenever there is a request for raw data from the memory 300 by the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140, and display 150 of the multimedia IP 100, the frame buffer compressor 200 may decompress the compressed data 20 into output data 30 and send the output data to the component of the multimedia IP 100 that requested the raw data.

[0045] The memory 300 can store the compressed data 20 generated by the frame buffer compressor 200, and can provide the stored compressed data 20 to the frame buffer compressor 200 so that the frame buffer compressor 200 can decompress the stored compressed data 20.

[0046] The frame buffer compressor 200 and the memory 300 can be connected to the system bus 400 respectively. For example, the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140 and display 150 of the multimedia IP 100 can be individually connected to the system bus 400 via the frame buffer compressor 200.

[0047] Reference Figure 2 The frame buffer compressor 200 may include an encoder 210 and a decoder 220.

[0048] Encoder 210 can receive source data 10 from multimedia IP 100 to generate compressed data 20. At this time, the source data 10 can be sent from the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140, and display 150 of multimedia IP 100, respectively. Compressed data 20 can be sent to memory 300 via multimedia IP 100 and system bus 400.

[0049] Conversely, decoder 220 can decompress compressed data 20 stored in memory 300 into output data 30. Output data 30 can be sent to multimedia IP 100. At this time, output data 30 can be sent to the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140 and display 150 of multimedia IP 100 respectively.

[0050] Figure 3 It is shown Figure 2 A flowchart of the encoder's operation method. Figure 4 This is a diagram showing the cell blocks included in a single frame of source data. Figure 5 It is shown Figure 4 A diagram of the unit block. Figure 6 It is shown Figure 4 A diagram showing multiple unit blocks. (Refer to...) Figures 3 to 6 The following description is provided.

[0051] The encoder 210 of the frame buffer compressor 200 first receives source data 10 on a frame basis (S10). The encoder 210 can compress the source data 10 in a lossy compression mode (S21) and in a lossless compression mode (S25). The source data 10 includes at least two frames, and each frame can include at least two blocks arranged in rows and columns.

[0052] Encoder 210 can compare the cumulative compression ratio of at least two or more compressed data with a reference compression ratio (S30). The compressed data can be lossless or lossy compressed data. Lossless compression means compression without data loss and indicates a type of compression ratio that varies depending on the data. Conversely, lossy compression involves some loss of data, has a higher compression ratio than lossless compression, and can have a preset, fixed compression ratio.

[0053] If, in operation S30, it is determined that the cumulative compression ratio of the lossy compressed source data in operation S21 is less than the reference compression ratio, then encoder 210 selects a lossy path and outputs lossy compressed data (S40). However, when the cumulative compression ratio of the lossy compressed source data is greater than the reference compression ratio, encoder 210 can select a lossless path and output lossless compressed data (S40).

[0054] In this case, the cumulative compression ratio can be a value obtained by adding the compression ratios of at least two blocks requested by the Multimedia IP 100. Furthermore, the compression ratio of each block can represent the percentage of compressed data in the total source data.

[0055] According to some example embodiments, at least two blocks that may be the basis for calculating the cumulative compression ratio may be included in columns (i.e., row cells). Alternatively, according to some example embodiments, at least two blocks that may be the basis for the cumulative compression ratio may be at least one block cell. According to some example embodiments, at least two blocks that may be the basis for the cumulative compression ratio may be at least two or more blocks arranged consecutively among a plurality of blocks, or at least two or more blocks arranged randomly.

[0056] Suppose that if the cumulative compression ratio of the lossless compressed data in each block is less than the reference compression ratio, a lossless path is selected, and if the cumulative compression ratio of the lossless compressed data is greater than the reference compression ratio, a lossy path is selected. In some example embodiments, since the compression ratio is calculated independently for each block, the overall compression ratio is increased. However, if the compression ratio in a particular block is low and the cumulative compression ratio of the compressed data is greater than the reference compression ratio, image quality degradation may occur due to that particular block when a lossy path is selected based on the original rules.

[0057] Therefore, by comparing the total compression ratio (i.e., cumulative compression ratio) of each block with a reference compression ratio, image quality degradation of only specific blocks can be reduced or prevented, and the number of blocks processed as lossy compression and the overall loss can also be reduced.

[0058] For detailed explanation, please refer to Figure 4 Provided. According to some example embodiments, it is assumed that a frame consists of fifteen blocks of 5×3.

[0059] In the example shown, the first block is the first block, which is either lossy or lossless compressed. The cumulative compression ratio of the first and second blocks is the value obtained by adding the first block (calculated above) to the second block (which is either lossy or lossless compressed) (= first block + second block). The cumulative compression ratio from the first block to the third block is the value obtained by adding the cumulative value (first block + second block) calculated above to the third block (which is either lossy or lossless compressed) (= first block + second block + third block). The cumulative compression ratio in each block (such as the fourth and fifth blocks) can be calculated in the same way, and in the case of the Nth block, the value obtained by adding the compression ratios from the first block to the Nth block (= first block + second block + third block + ... + Nth block) can be the cumulative compression ratio of the Nth block.

[0060] The cumulative compression ratio calculated in this way is compared with the reference compression ratio K, and the encoder 210 can determine whether to select a lossy path or a lossless path based on the comparison result.

[0061] The reference compression ratio K is a compression ratio defined based on the bandwidth of the system bus 400 that connects the frame buffer compressor 200 to the memory 300, and is a value that can be changed according to the design. As an example, depending on the configuration of the multimedia IP 100 and the frame buffer compressor 200, the reference compression ratio K can be defined as 50% of the bandwidth of the system bus 400, and as another example, it can be defined as 40%, or as yet another example, it can be defined as 60%.

[0062] Reference Figure 5 According to some example embodiments, the compressed data 20 corresponding to each block included in the frame may include a payload and a header.

[0063] The payload includes flags, the actual compressed data, and the values ​​required for decompression. The header is the part that indicates the compression ratio of the compressed data and can be stored as a header index.

[0064] According to some example embodiments, there may be a situation where the source data 10 is M times the burst length BL (M is a natural number greater than zero). In this case, the header index may include M / 2 bits. According to some example embodiments, the header index may indicate the size of the compressed data. The encoder 210 may read only the size of the compressed data based on the header index, thus reducing the actual bandwidth used.

[0065] The compressed data 20 can become shorter than the source data 10. In this case, the payload may include a flag F indicating whether the compressed data 20 is due to a lossy path or a lossless path. The flag is 1 bit and can be included in the compressed data 20. For example, a flag of 0 can indicate a lossy path, and a flag of 1 can indicate a lossless path, and according to another example, the reverse is also possible.

[0066] Reference Figure 4 and Figure 6 Multiple blocks are included in a single frame, and the compressed data for each block (e.g., 1st BkComp_data to Nth Bk Comp_data) includes a payload and a header. The cumulative compression ratio can be the cumulative sum of the compression ratios of the compressed data (1st Bk Comp_data) of at least one requested block. In some example embodiments, as shown, the header exists separately from the compressed data, and the length of the header, etc., does not affect the calculation of the cumulative compression ratio.

[0067] Encoder 210 can repeatedly perform operation S21 or S25 to S40 for the requested number of blocks until the end of a frame (S50).

[0068] Figure 7 This is for illustrative purposes based on some example embodiments. Figure 2 A block diagram of the encoder.

[0069] Reference Figure 7 The encoder 210 includes a first mode selector 219, a quantization module 211, a prediction module 213, an entropy coding module 215, a padding module 216, and a compression management module 218.

[0070] The first mode selector 219 can determine whether the encoder 210 operates in lossless mode or lossy mode based on the control signal from the entropy encoding module 215. When the encoder 210 operates in lossless mode, the source data 10 can be transmitted along... Figure 3 The lossless path (lossless) is compressed. When encoder 210 operates in lossy mode, source data 10 can be compressed along a lossy path (lossy).

[0071] The first mode selector 219 can receive a signal from the multimedia IP 100 to determine whether to perform lossless compression or lossy compression. Lossless compression indicates compression without data loss and represents a type of compression ratio that varies depending on the data. Conversely, lossy compression involves some loss of data, has a higher compression ratio than lossless compression, and can have a preset, fixed compression ratio.

[0072] When encoder 210 operates in lossless mode, first mode selector 219 can guide the data stream along the lossless path (lossless) to prediction module 213, entropy encoding module 215, and padding module 216. When encoder 210 operates in lossy mode, first mode selector 219 can guide the data stream along the lossy path (lossy) to quantization module 211, prediction module 213, and entropy encoding module 215.

[0073] As an example, when encoder 210 operates in lossless mode, first mode selector 219 can send the source data 10 input to encoder 210 to prediction module 213. As another example, when encoder 210 operates in lossy mode, first mode selector 219 can send the source data 10 input to encoder 210 to quantization module 211.

[0074] The quantization module 211 can perform quantization on the source data 10 input to the frame buffer compressor 200 using predefined or desired quantization coefficients to generate reconstructed data. For example, the quantization module 211 can quantize each of a plurality of source pixel data using predefined or desired quantization coefficients to generate reconstructed data.

[0075] Data removed during quantization will not be recovered later. Therefore, the quantization module 211 can be used only in lossy mode. However, compared to lossless mode, lossy mode can have a relatively high compression ratio and can have a preset fixed compression ratio.

[0076] The compressed data 20 may include quantization coefficients used by the quantization module 211. For example, the encoder 210 may add quantization coefficients used in the quantization module 211 to the compressed data 20.

[0077] Prediction module 213 can perform intra-prediction on source data 10 or reconstructed data to generate predicted data. According to some example embodiments, the predicted data can be a single dataset obtained by reducing the size of multiple reconstructed data sets. As an example, prediction module 213 can reduce the data size by representing some of the multiple reconstructed pixel data sets using residual pixel data.

[0078] The prediction module 213 can compress the size of the data by generating prediction data, which includes reference pixel data and residual pixel data, from the source data 10 or the reconstructed data. Furthermore, the prediction module 213 can perform intra-frame prediction based on multiple source pixel data of the source data 10 or multiple reconstructed pixel data of the reconstructed data to generate prediction data.

[0079] According to some example embodiments of this application, the quantization module 211 and the prediction module 213 can perform quantization and intra-frame prediction in parallel for each of a plurality of source pixel data. For example, the quantization module 211 can simultaneously perform quantization of all source pixel data to generate reconstructed data. Furthermore, the prediction module 213 can generate prediction data, including reference pixel data and prediction pixel data, in parallel from the source data or the reconstructed data. In this way, according to some example embodiments of the present invention, the quantization module 211 and the prediction module 213 can perform quantization and intra-frame prediction in parallel, respectively.

[0080] Entropy encoding module 215 can perform entropy encoding on the prediction data compressed by prediction module 213 to generate entropy data. Entropy encoding module 215 calculates the compression ratio of the generated entropy data and compares it with a reference compression ratio to output a control signal to the first mode selector 219. This control signal becomes the basis for selecting a lossy path or a lossless path. In other words, it can correspond to... Figure 3 Operation S30.

[0081] According to some example embodiments, the entropy coding module 215 can determine the entropy pixel data based on the entropy frequency.

[0082] In addition, the entropy encoding module 215 can perform entropy encoding on the prediction data generated from the source data 10 to generate entropy data.

[0083] In some example embodiments, the entropy coding module 215 may use Huffman coding to compress the prediction data. Alternatively, the entropy coding module 215 may use exponential Golomb coding or Golomb Rice coding to compress the prediction data.

[0084] The padding module 216 can perform padding on the entropy data generated by the entropy encoding module 215 to generate padding data. For example, the padding module 216 can add meaningless data (e.g., zero data) to the entropy data to generate padding data with a predefined or expected size.

[0085] As an example, when encoder 210 operates in lossy mode, the size of the padding data can be defined based on the size of the source data and a fixed compression ratio. For example, when the size of source data 10 is 100 and the fixed compression ratio is 50%, the size of the padding data can be defined as 50. On the other hand, the size of the entropy data compressed by quantization module 211, prediction module 213, and entropy encoding module 215 can be less than 50. In some example embodiments, padding module 216 can add zero data to the entropy data to generate padding data with a predefined or desired size of 50.

[0086] As another example, when encoder 210 operates in lossless mode, the size of the padding data can be defined based on the size of the source data. For example, when the size of the source data 10 is 100, the size of the padding data can be defined as 100. On the other hand, the size of the entropy data compressed by prediction module 213 and entropy encoding module 215 can be less than 100. In some example embodiments, padding module 216 can add zero data to the entropy data to generate padding data with a predefined or desired size of 100.

[0087] In this way, the filling module 216 can generate filling data of a predefined or desired size from the entropy data compressed by other modules of the encoder 210. The encoder 210 can output the filling data as compressed data 20. That is, the compressed data 20 stored in the memory 300 can be made to have a specific size by the filling module 216.

[0088] Compression management module 218 can control the compression of source data 10 in quantization module 211 and entropy encoding module 215 based on determined quantization coefficients and entropy tables.

[0089] The compression management module 218 can determine the quantization coefficients used in the quantization module 211. For example, when the encoder 210 operates in lossy mode, the source data 10 is compressed along... Figure 3 The lossy path (lossy) is compressed. At this point, the compression management module 218 may include a QP table containing quantization coefficients. For example, the QP table may include one or more entries, and each entry may include the quantization coefficients used in the quantization module 211.

[0090] Furthermore, the compression management module 218 may determine an entropy table representing the entropy frequency of the entropy pixel data used for each entropy encoding. For example, the compression management module 218 may include an entropy table. The entropy table represents multiple code tables for performing the entropy encoding algorithm, determined by the value of k, and the entropy table that may be used in some exemplary embodiments of the inventive concept may include at least one of exponential Golomb code and Golomb Rice code.

[0091] Subsequently, the frame buffer compressor 200 can write the compressed data 20 generated from the encoder 210 into the memory 300. Furthermore, the frame buffer compressor 200 can read the compressed data 20 from the memory 300, decompress the read compressed data 20, and provide it to the multimedia IP 100.

[0092] Figure 8 This is for illustrative purposes based on some example embodiments. Figure 2 A block diagram of the encoder. For ease of explanation, the main focus will be on...Figure 7 The differences.

[0093] Reference Figure 8 According to some example embodiments, encoder 210 may also include CRC (Cyclic Redundancy Check) module 217.

[0094] CRC module 217 can directly receive source data 10 that has not been compressed by the previous module. CRC module 217 can perform CRC calculations using a pre-stored polynomial, thereby generating CRC bits. The CRC bits are then appended to the compressed data 20 and can be used as a means to check for errors in the compression and decompression process when the compressed data 20 is later decompressed and becomes output data 30.

[0095] Figure 9 It is shown Figure 2 The flowchart shows the operation method of the decoder.

[0096] Reference Figure 9 According to some example embodiments, decoder 220 can receive accumulated compressed data (S110). Decoder 220 checks the flag F on the received compressed data 20 to determine whether to select a lossless path or a lossy path (S120).

[0097] In the case of a lossy path, decoder 220 can perform unpadding of zero data along the lossy path based on padding data of a predefined or expected size (S130). In the case of a lossless path, decoder 220 can perform unpadding of zero data along the lossless path based on padding data of a predefined or expected size (S130).

[0098] Decoder 220 can perform entropy decoding (S140) on the depuffed entropy data. For example, entropy decoding can extract prediction data from the entropy data through Huffman coding, exponential Golomb coding, or Columbus coding.

[0099] Decoder 220 can perform lossy path decompression (S151) or lossless path decompression (S152) on the extracted prediction data. Lossy path decompression S151 can be performed by inverse quantization after prediction compensation is performed on the inverse prediction data. Inverse quantization can be performed on the prediction compensation data using the quantization coefficients of the compressed data, and the result is output as the output data (S131).

[0100] Figure 10 This is for illustrative purposes based on some example embodiments. Figure 2 A block diagram of the decoder.

[0101] Reference Figure 10The decoder 220 includes a second mode selector 229, a de-padding module 226, an entropy decoding module 225, a prediction compensation module 223, and an inverse quantization module 221.

[0102] The second mode selector 229 can select whether the compressed data 20 stored in the memory 300 is lossless or lossy compressed. As an example, such as... Figure 5 and Figure 6 As shown, the second mode selector 229 can determine which mode, lossless or lossy, the compressed data 20 is compressed by using the flag F of the compressed data.

[0103] In lossless mode, the second mode selector 229 can guide the compressed data 20 along the lossless path (lossless) to the de-padding module 226, the entropy decoding module 225, and the prediction compensation module 223. Conversely, in lossy mode, the second mode selector 229 can guide the compressed data 20 along the lossy path (lossy) to the de-padding module 226, the entropy decoding module 225, the prediction compensation module 223, and the inverse quantization module 221.

[0104] The depadding module 226 can remove meaningless data (e.g., zero data) added by the padding module 216 of the encoder 210. For example, the depadding module 226 can remove zero data from the compressed data 20 to generate entropy data. The entropy data generated by the depadding module 226 can be sent to the entropy decoding module 225.

[0105] The entropy decoding module 225 can decompress the data compressed by the entropy encoding module 215 of the encoder 210. That is, the entropy decoding module 225 can generate prediction data based on the entropy data sent from the de-filling module 226.

[0106] According to some example embodiments, the entropy decoding module 225 can use an entropy table to generate residual pixel data and reference pixel data corresponding to each entropy pixel data. That is, the entropy decoding module 225 can generate prediction data including residual pixel data and reference pixel data. The prediction data generated from the entropy decoding module 225 can be sent to the prediction compensation module 223.

[0107] In some example embodiments, the entropy decoding module 225 may perform decompression via Huffman coding, exponential Golomb coding, or Columbus coding. Huffman coding, exponential Golomb coding, or Columbus coding are common techniques, and their details will be omitted.

[0108] According to some example embodiments, when generating prediction data from entropy data, the entropy decoding module 225 checks the flag F included in the compressed data and may notify the second mode selector 229 of the flag. According to one example, after performing both decompression according to a lossy mode and decompression according to a lossless mode, the flag may also be notified to the second mode selector 229, causing the second mode selector 229 to select one of the two results. According to another example, after the second mode selector 229 selects one of the lossy or lossless modes by the flag, decompression may also be performed along the selected path.

[0109] The prediction compensation module 223 can perform intra-frame prediction compensation on the prediction data to decompress the prediction data sent from the entropy decoding module 225. That is, the prediction compensation module 223 can perform intra-frame prediction compensation on the prediction data to generate reconstructed data or output data 30. For example, if the compressed data 20 is compressed in lossy mode, the prediction compensation module 223 can generate reconstructed data and send it to the inverse quantization module 221. Furthermore, when the compressed data 20 is compressed in lossless mode, the prediction compensation module 223 can generate output data 30. In some example embodiments, the decoder can send the output data 30 to the multimedia IP 100.

[0110] For example, the prediction compensation module 223 can decompress the prediction data in the reverse order of intra-frame prediction performed by the prediction module 213. The prediction data may include multiple prediction pixel data. Each of the multiple prediction pixel data may be reference pixel data or residual pixel data.

[0111] As an example, the prediction compensation module 223 can use reference pixel data or residual pixel data to generate multiple reconstructed pixel data. For instance, the prediction compensation module 223 can use reference pixel data to generate reconstructed pixel data corresponding to the reference pixel data. Furthermore, the prediction compensation module 223 can use both reference pixel data and residual pixel data to generate reconstructed pixel data corresponding to the residual pixel data. As explained above, the prediction compensation module 223 can send reconstructed data including multiple reconstructed pixel data to the inverse quantization module 221.

[0112] As another example, output data 30 may include output pixel data corresponding to each of a plurality of predicted pixel data. Prediction compensation module 223 may use reference pixel data or residual pixel data to generate the plurality of output pixel data. For example, prediction compensation module 223 may use reference pixel data to generate output pixel data corresponding to the reference pixel data. Furthermore, prediction compensation module 223 may use both reference pixel data and residual pixel data to generate output pixel data corresponding to the residual pixel data. As explained above, decoder 220 may send output data 30, including the plurality of output pixel data, to multimedia IP 100.

[0113] In this way, the prediction compensation module 223 can recover the intra-frame prediction performed pixel by pixel by the prediction module 213.

[0114] In lossy mode, the inverse quantization module 221 can generate output data 30 based on the quantization coefficients from the compressed data 20 and the reconstructed data sent from the prediction compensation module 223. That is, the inverse quantization module 221 can perform inverse quantization on the reconstructed data using the quantization coefficients and generate output data 30 as the result. For example, the reconstructed data may include a QP table determined by the compression management module 218 of the encoder 210. The inverse quantization module 221 can determine the quantization coefficients from the QP table.

[0115] For example, output data 30 may include multiple output pixel data corresponding to each of the multiple reconstructed pixel data. The output pixel data can be generated by performing inverse quantization on the corresponding reconstructed pixel data. For example, the output pixel data can be generated by multiplying the corresponding reconstructed pixel data by a quantization factor.

[0116] In this way, the inverse quantization module 221 can generate output data 30 from the reconstructed data. At this time, the output data 30 generated by the inverse quantization module 221 may differ from the source data 10 input to the encoder 210. This is because when the quantization module 211 of the encoder 210 performs quantization on the source data 10, data smaller than the quantization coefficient may be lost and cannot be recovered. Therefore, the inverse quantization module 221 can only be used in lossy mode.

[0117] When decompressing compressed data 20, the decompression management module 228 can execute the above-mentioned... Figure 8 The combination of the QP table and entropy table determined by the compression management module 218 for performing compression of the source data 10 can be appropriately reflected in the operation.

[0118] On the other hand, although not shown, decoder 220 may also include a CRC check module following inverse quantization module 221.

[0119] The CRC check module 224 can receive CRC bits from the second mode selector 229. The CRC check module 224 can perform CRC calculation on the output data 30 decompressed by the previous module using a pre-stored polynomial to generate comparison CRC bits.

[0120] The CRC check module 224 compares the CRC bits with the comparison CRC bits. If the CRC bits and the comparison CRC bits are the same, the output data 30 can be exactly the same as the source data 10. That is, in such an example embodiment, it can be confirmed that there are no errors in the compression and decompression processes.

[0121] Conversely, if the CRC bits and the comparison CRC bits are different from each other, the output data 30 may differ from the source data 10. That is, in such an example embodiment, it can be confirmed that an error occurred during the compression and decompression process. In such an example embodiment, the CRC check module 224 may perform error marking on the output data 30. Error marking may be a display that allows the user to know that an error occurred during the compression or decompression process. For example, the CRC check module 224 may assign a specific color to the output data 30 so that the user knows that the specific colored portion is the error portion.

[0122] In the following text, reference will be made to Figure 11 An image processing apparatus according to some exemplary embodiments of the present invention will be described. Descriptions of identical portions thereof will be simplified or omitted. Figure 11 These are block diagrams illustrating an image processing apparatus based on some exemplary embodiments of the present invention.

[0123] Reference Figure 11 In some exemplary embodiments of the image processing apparatus according to the present invention, the frame buffer compressor 200, the multimedia IP 100, and the memory 300 can each be directly connected to the system bus 400.

[0124] The frame buffer compressor 200 is not directly connected to the multimedia IP 100, but can be connected to the multimedia IP via the system bus 400. For example, the multimedia IP 100 can send data to and receive data from the frame buffer compressor 200 via the system bus 400. That is, during the compression process, the multimedia IP 100 can send source data 10 to the frame buffer compressor 200 via the system bus 400. Subsequently, the frame buffer compressor 200 can generate compressed data 20 based on the source data 10, and can send the compressed data 20 to the memory 300 again via the system bus 400.

[0125] Furthermore, during the decompression process, the compressed data 20 stored in the memory 300 is sent to the frame buffer compressor 200 via the system bus 400, and the compressed data 20 can be decompressed into output data 30. Subsequently, the frame buffer compressor 200 can send the output data 30 to the multimedia IP 100 via the system bus 400.

[0126] According to some example embodiments, even if the frame buffer compressor is not separately connected to the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140 and display 150 of the multimedia IP 100, the hardware configuration can be simplified and / or the operating speed can be improved because the frame buffer compressor can be connected to the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140 and display 150 of the multimedia IP 100 via the system bus.

[0127] In the following text, reference will be made to Figure 12 An image processing apparatus according to some exemplary embodiments of the present invention will be described. Descriptions of identical portions thereof will be simplified or omitted. Figure 12 These are block diagrams illustrating an image processing apparatus based on some exemplary embodiments of the present invention.

[0128] Reference Figure 12 In some example embodiments of the image processing apparatus according to the present invention, the system bus 400 is directly connected to the multimedia IP 100 and the frame buffer compressor 200, and the memory 300 can be connected to the system bus 400 via the frame buffer compressor 200.

[0129] In other words, the memory 300 is not directly connected to the system bus 400, but can be connected to the system bus 400 only via the frame buffer compressor 200. Furthermore, the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140, and display 150 of the multimedia IP 100 can be directly connected to the system bus 400. Therefore, the image signal processor 110, image stabilization module 120, multi-format codec 130, GPU 140, and display 150 of the multimedia IP 100 can access the multimedia memory 300 only through the frame buffer compressor 200.

[0130] In some example embodiments, since the frame buffer compressor 200 involves all access to the memory 300, when the frame buffer compressor 200 is directly connected to the system bus 400 and the memory 300 is connected to the system bus 400 via the frame buffer compressor 200, data transmission errors can be reduced and / or speed can be increased.

[0131] In the following text, reference will be made to Figure 13 An image processing apparatus according to some exemplary embodiments of the present invention will be described. Descriptions of identical portions thereof will be simplified or omitted. Figure 13 These are block diagrams illustrating an image processing apparatus based on some exemplary embodiments of the present invention.

[0132] Reference Figure 13 In an image processing apparatus according to some exemplary embodiments of the present invention, a system bus 400 can be directly connected to a multimedia IP 100 and a memory 300. A frame buffer compressor 200 can be connected to the multimedia IP 100. The frame buffer compressor 200 can receive source data 10 from the multimedia IP 100. The frame buffer compressor 200 can compress the source data 10 to generate compressed data 20, and can send the compressed data 20 back to the multimedia IP 100. The multimedia IP 100 can store the compressed data 20 in the memory 300 via the system bus 400.

[0133] During decompression, the multimedia IP 100 can receive compressed data 20 from the memory 300 via the system bus 400. The multimedia IP 100 can send the compressed data 20 to the frame buffer compressor 200. The frame buffer compressor 200 can decompress the compressed data 20 to generate output data 30, and can send the output data back to the multimedia IP 100.

[0134] Any element disclosed above may include one or more circuits (such as hardware including logic circuits, hardware / software combinations (such as a processor executing software), or combinations thereof), or be implemented in one or more circuits (such as hardware including logic circuits, hardware / software combinations (such as a processor executing software), or combinations thereof). For example, the circuits may more specifically include, but are not limited to, central processing units (CPUs), arithmetic logic units (ALUs), digital signal processors, microcomputers, field-programmable gate arrays (FPGAs), system-on-a-chip (SoCs), programmable logic units, microprocessors, application-specific integrated circuits (ASICs), etc.

[0135] In summarizing the specific embodiments, those skilled in the art will understand that many variations and modifications can be made to the preferred exemplary embodiments without substantially departing from the principles of the inventive concept. Therefore, the preferred exemplary embodiments of the disclosed inventive concept are used only in a general and descriptive sense and not for limiting purposes.

Claims

1. An image processing apparatus, the image processing apparatus comprising: The multimedia intellectual property block is configured to: process raw data to generate source data, and receive and use output data; The frame buffer compression circuit is configured to operate in lossy or lossless mode to compress source data into compressed data or decompress compressed data into output data. as well as The memory is configured to store compressed data and is accessed by the multimedia intellectual property block. The frame buffer compression circuit is configured to select either a lossy or lossless mode based on the cumulative compression ratio of the compressed data, and to perform compression or decompression based on the selected mode. The frame buffer compression circuit is configured to compare the cumulative compression ratio of the compressed data with a reference compression ratio, and select either a lossy mode or a lossless mode based on the comparison result. The source data includes at least one frame containing multiple blocks. The cumulative compression ratio is a value obtained by adding the compression ratios of at least two of the plurality of blocks, and the reference compression ratio is a compression ratio defined based on the bandwidth of the system bus that connects the frame buffer compression circuit to the memory.

2. The image processing device as described in claim 1, in, When the cumulative compression ratio of at least two of the plurality of blocks exceeds the reference compression ratio, the frame buffer compression circuit is configured to operate in lossy mode; otherwise, the frame buffer compression circuit is configured to operate in lossless mode.

3. The image processing apparatus as described in claim 2, wherein, The at least two blocks include all of the plurality of blocks.

4. The image processing apparatus as described in claim 2, wherein, The at least two blocks are blocks included in at least one row among the plurality of blocks.

5. The image processing apparatus as described in claim 2, wherein, The at least two blocks are at least two or more blocks that are placed consecutively among the plurality of blocks.

6. The image processing apparatus as claimed in claim 2, wherein, The at least two blocks are at least two or more blocks that are randomly placed among the plurality of blocks.

7. The image processing apparatus according to any one of claims 1 to 6, wherein, The compressed data also includes: The header indicates the percentage of compressed data in the bandwidth; and Net load includes a flag indicating whether the cumulative compression ratio exceeds the reference compression ratio, actual compression data, and the value required for decompression.

8. A frame buffer compression circuit, the frame buffer compression circuit comprising: The encoder circuit is configured to receive source data and generate compressed data; as well as The decoder circuit is configured to decompress the compressed data and output the output data. The encoder circuit is configured to compare the cumulative compression ratio of the compressed data with a reference compression ratio, and select either a lossy mode or a lossless mode based on the comparison result. The source data includes at least one frame containing multiple blocks. The cumulative compression ratio is a value obtained by adding the compression ratios of at least two of the plurality of blocks, while the reference compression ratio is a compression ratio defined based on the bandwidth of the system bus. Specifically, when the cumulative compression ratio of the compressed data exceeds the reference compression ratio, the encoder circuit is configured to perform compression in lossy mode; otherwise, the encoder circuit is configured to perform compression in lossless mode. The decoder circuit is configured to perform decompression according to the compression mode corresponding to the compressed data.

9. The frame buffer compression circuit as described in claim 8, wherein, The compressed data includes: The header indicates the percentage of compressed data in the bandwidth; and The payload includes flags, actual compressed data, and the values ​​required for decompression.

10. The frame buffer compression circuit as described in claim 9, wherein, The encoder circuit includes: The quantization circuit is configured to quantize the source data according to the quantization coefficients; Prediction circuitry is configured to perform predictions on source data or quantized data to generate predicted data; and An entropy coding circuit is configured to perform entropy coding on the prediction data to generate entropy data, and The entropy coding circuit is configured to: calculate the compression ratio of the source data based on blocks and accumulate the compression ratio of the source data, and The entropy coding circuit is configured to reflect in the flag the result obtained by comparing the cumulative compression ratio of the at least two or more blocks with a reference compression ratio.

11. The frame buffer compression circuit as described in claim 10, wherein, The encoder circuit also includes a first mode selector circuit, which is configured to select a lossy mode or a lossless mode based on a flag provided by the entropy encoding circuit. The first mode selector circuit is located before the quantization circuit and is configured to send source data to the prediction circuit when the lossless mode is selected, and to send source data to the quantization circuit when the lossy mode is selected.

12. The frame buffer compression circuit as described in claim 10, wherein, The encoder circuit also includes a cyclic redundancy check circuit, configured to add cyclic redundancy check bits generated based on the source data to the compressed data.

13. The frame buffer compression circuit according to any one of claims 9 to 12, wherein, The decoder circuit includes: The second mode selector circuit is configured to select either a lossy mode or a lossless mode based on a flag. The entropy decoding circuit is configured to perform entropy decoding of compressed data. The prediction compensation circuit is configured to perform prediction compensation on the entropy-decoded data; and The inverse quantization circuit is configured to inverse quantize the prediction-compensated data in lossy mode, and is configured to output the inverse quantized data as the output data. In the lossless mode, the prediction compensation circuit outputs the predicted and compensated data as the output data.

14. An image processing apparatus, the image processing apparatus comprising: The memory is configured to store compressed data; as well as The frame buffer compression circuit includes an encoder circuit and a decoder circuit. The encoder circuit is configured to compress at least some of the source data to generate compressed data and send the compressed data to a memory. The decoder circuit is configured to read the compressed data from the memory and decompress the compressed data. The compressed data includes the payload and header; the payload includes the actual compressed data and flags. The frame buffer compression circuit is configured to reflect the result obtained by comparing the cumulative compression ratio corresponding to the compressed data with a reference compression ratio in a flag, and is configured to perform compression or decompression in lossy or lossless mode according to the flag. The frame buffer compression circuit is configured to compare the cumulative compression ratio corresponding to the compressed data with a reference compression ratio, and select either a lossy mode or a lossless mode based on the comparison result. The source data includes at least one frame containing multiple blocks. The cumulative compression ratio is a value obtained by adding the compression ratios of at least two of the plurality of blocks, and the reference compression ratio is a compression ratio defined based on the bandwidth of the system bus that connects the frame buffer compression circuit to the memory.

15. The image processing apparatus of claim 14, wherein, The encoder circuit includes: The quantization circuit is configured to quantize the source data according to the quantization coefficients; Prediction circuitry is configured to perform predictions on source data or quantized data to generate predicted data; and Entropy encoding circuitry is configured to perform entropy encoding on the prediction data to generate entropy data. The entropy coding circuit is configured to calculate the compression ratio of entropy data based on blocks and accumulate the compression ratio of the entropy data. The entropy coding circuit is configured to generate a flag by comparing the cumulative compression ratio of the at least two or more blocks with a reference compression ratio.

16. The image processing apparatus of claim 15, wherein, The encoder circuit also includes a first mode selector circuit configured to operate the encoder circuit in lossy or lossless mode according to a flag.

17. The image processing apparatus according to any one of claims 14 to 16, wherein, The decoder circuitry is configured to select between lossy and lossless modes based on flags in the compressed data being read. The decoder circuitry is configured to perform depadding of the compressed data with zero data. The decoder circuitry is configured to perform entropy decoding on the de-padding compressed data, and When the lossy mode is selected, the decoder circuit is configured to inverse quantize the entropy-decoded data and output the inverse-quantized data as output data. When lossless mode is selected, the decoder circuit is configured to output the data after output entropy decoding as the output data.

18. The image processing apparatus according to claim 17, wherein, The header indicates the percentage of compressed data in the bandwidth between the memory and the frame buffer compression circuitry.

19. The image processing apparatus of claim 15, wherein, The at least two blocks are blocks included in at least one row of the source data.

20. The image processing apparatus of claim 15, wherein, The at least two blocks are at least two or more consecutively placed blocks among a plurality of blocks of source data.

Images