Frame buffer compression and decompression method and device

Abstract

The invention discloses a frame buffer compression and decompression method and device, and belongs to the technical field of electronic equipment.The method comprises the steps that firstly, original image data are obtained, dynamic compression and filtering processing are conducted on the original image data, and decomposition is conducted to obtain a plurality of sub-images; performing lossless compression on each sub-image to generate a plurality of code streams, recording compression information and position information of each code stream, and generating a lookup table; storing the plurality of code streams and the lookup table in a memory, wherein the memory is divided into a lookup table area and a data area; according to application requirements, a target code stream is determined through the bandwidth control module in combination with the lookup table, the target code stream is read, decompressed and synthesized to obtain output image data, the identification detection type application correspondingly decompresses all the code streams to obtain lossless output image data, and the display type application can decompress part of the code streams to obtain visual lossless output image data.

Description

Technical Field

[0001] This invention relates to the field of electronic devices, and more specifically to compression and decompression techniques in image processing. Background Technology

[0002] In multimedia applications, an increasing number of applications require high-resolution and high-frame-rate images, but this also leads to significant bandwidth consumption. For example, displaying a single frame of 4K resolution YUV420 video image consumes 3840*2160*1.5*60=712MB / s of bandwidth, while UI data in ARGB8888 format consumes 1898MB / s. Considering the reception and processing of video signals by camera, encoding / decoding modules, and other related modules, the system's bandwidth consumption for display is substantial.

[0003] Therefore, it is essential to introduce frame buffer compression and decompression technology into the system.

[0004] Common video compression techniques are lossy compression based on both temporal and spatial constraints, and the process is relatively complex, making them unsuitable as a compression method for frame buffers in a system. Generally, intra-frame spatial compression methods are used instead.

[0005] One approach is to use intra-frame lossy compression (or visually lossless), which offers high compression efficiency and controllable bandwidth. However, it also has application limitations. For example, in common image recognition and detection, if lossy or hybrid compression methods are used on the original video data, due to the complexity of the recognition and detection algorithms and application scenarios, it is impossible to determine how the compression loss will affect the subsequent recognition and detection results. Generally, it is not possible to perform processing that results in a loss of accuracy for recognition and detection applications. If the bandwidth of the display application needs to be reduced, two frame buffers need to be stored, one for the recognition and detection application and the other for the display output application.

[0006] Another approach is to use intra-frame lossless compression. Since the original data is preserved, it is suitable for any application. However, the overall compression bandwidth is not as high as lossy compression, and the overall bandwidth consumption is not low enough.

[0007] In response, some new technologies employ adaptive content control compression, employing either lossy block compression or a hybrid lossless block compression method. However, lossy data can also lead to problems unsuitable for certain applications. Summary of the Invention

[0008] The purpose of this invention is to provide a method and apparatus for frame buffer compression and decompression, which solves the problems that existing intra-frame lossy compression cannot be adapted to identification and detection applications, and intra-frame lossless compression is difficult to reduce bandwidth consumption, resulting in the system needing to store multiple frame buffers for different applications, and the heavy burden of resources and bandwidth.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] In a first aspect, the present invention provides a frame buffer compression and decompression method, comprising the following steps:

[0011] Obtain the raw image data;

[0012] The original image data is filtered and decomposed to obtain multiple sub-images;

[0013] The multiple sub-images are each subjected to lossless compression to obtain multiple bitstreams;

[0014] Record the compression and location information of the multiple bitstreams and generate a lookup table;

[0015] The target bitstream is read according to the lookup table, the target bitstream is decompressed, and the output image data is synthesized.

[0016] In one possible implementation, when filtering and decomposing the original image data, the original image data is gradually decomposed into multiple sub-images according to the characteristics of human vision; when performing lossless compression on each of the multiple sub-images, lossless compression is performed on each sub-image in blocks of a specific size and in a certain order to obtain the corresponding bitstream.

[0017] In one possible implementation, after the lookup table is generated, the multiple bitstreams and the lookup table are stored in a memory in a specific format, and the memory is divided into a lookup table area and a data area; when reading the target bitstream according to the lookup table, the bandwidth control circuit, in conjunction with the information in the lookup table, determines the bitstream and corresponding block data that need to be read or skipped.

[0018] In one possible implementation, when determining the bitstream and corresponding block data that need to be read or skipped, it is possible to select to read or skip all blocks of the entire image, or dynamically select to read or skip blocks at certain locations in the image; when decompressing the target bitstream, one or more target bitstreams are decompressed, and then the output image data is obtained through a synthesis operation.

[0019] In one possible implementation, all the bitstreams are decompressed and synthesized to obtain lossless output image data; a portion of the bitstreams are decompressed and synthesized to obtain visually lossless output image data; after synthesizing the output image data, the output image data is transmitted to the corresponding application module to adapt to different application requirements such as recognition detection or display output.

[0020] Secondly, the present invention provides a frame buffer compression and decompression apparatus, comprising a video signal receiving unit, an image / video encoder, a graphics / image processing unit, a display output control unit, a data bus port, a compression unit, a decompression unit, and a memory. The video signal receiving unit is signal-connected to the memory via the data bus port, and is also signal-connected to the compression unit. The compression unit is signal-connected to the memory via the data bus port. The decompression unit is signal-connected to the memory via the data bus port. The image / video encoder is signal-connected to both the memory and the compression unit via the data bus port. The graphics / image processing unit is signal-connected to the memory, the compression unit, and the decompression unit via the data bus port. The display output control unit is signal-connected to the decompression unit via the data bus port.

[0021] In one possible implementation, the compression unit is used to compress the original data into blocks of a specific size and in a certain order, and output one or more bitstreams. The compression unit also decomposes the original image into different sub-images in a manner that conforms to the characteristics of human vision, compresses each sub-image separately to generate the bitstream, and generates a lookup table containing the position information of each block in the corresponding bitstream when outputting multiple bitstreams. Both the bitstream and the lookup table are transmitted to the memory for storage.

[0022] In one possible implementation, the memory is divided into a lookup table area and a data area. The lookup table is stored in the lookup table area, and the bitstream is stored in the data area. The arrangement of the lookup table area and the data area facilitates skipping data blocks that do not need to be read. The decompression unit is connected to the lookup table signal. After reading the lookup table and the corresponding bitstream in the memory, the decompression unit decompresses one or more bitstreams and outputs image data.

[0023] In one possible implementation, the device further includes a bandwidth control circuit connected to the decompression unit. The bandwidth control circuit is used to control the decompression unit to read or skip the bitstream of corresponding position blocks during decompression. It can control the decompression unit to read or skip all blocks in the entire image, or control the decompression unit to dynamically select to read or skip blocks at certain positions in the image.

[0024] In one possible implementation, the decompression unit synthesizes output data by decompressing one or more of the bitstreams; the decompression unit synthesizes lossless output data by decompressing all of the bitstreams; the decompression unit synthesizes lossy output data by decompressing a portion of the bitstreams; the output data is transmitted to the display output control unit or the graphics processing unit; the video signal receiving unit is used to receive video image signals from outside the system; and the display output control unit is used to convert the received output data into a signal suitable for the display device to receive.

[0025] Compared with existing technologies, the advantages of this invention are as follows: This invention generates multiple bitstreams through lossless compression after filtering and decomposing the original image data, and records the compression and location information of each bitstream to create a lookup table, enabling the same frame buffer format to adapt to different application scenarios. It eliminates the need to store multiple frame buffers separately for different applications such as recognition, detection, and display output, effectively saving memory space and reducing bandwidth consumption during data transmission and storage, thus alleviating the overall bandwidth burden of the system.

[0026] In the decompression stage, this invention uses a bandwidth control module combined with a lookup table to determine the target bitstream. It can select to decompress the entire bitstream to obtain lossless output image data according to application requirements, thus meeting the data accuracy requirements of recognition and detection applications; or it can select to decompress a portion of the bitstream to obtain visually lossless output image data, adapting to the image quality requirements of display applications.

[0027] This invention stores multiple bitstreams and lookup tables in a specific format in a memory divided into a lookup table area and a data area. Through a clear partitioned storage method, combined with the bitstream location information recorded in the lookup table, the target bitstream can be quickly located and read, reducing the location time during data reading, improving operational convenience, and avoiding the reading of unnecessary data blocks, further helping to reduce bandwidth consumption. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A multimedia system architecture diagram illustrating the application of the frame buffer compression and decompression method provided in this embodiment of the invention;

[0030] Figure 2 A flowchart of the compression encoding process of the frame buffer compression unit provided in an embodiment of the present invention;

[0031] Figure 3 This is a flowchart of the decompression process of the frame buffer decompression unit provided in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram showing the distribution of multiple bitstreams after image region compression, as provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0034] Example:

[0035] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0036] This implementation method focuses on the compression and decompression processing of the frame buffer, in conjunction with the appendix. Figure 1 A multimedia system contains multiple functional modules and multiple compression and decompression units. The video signal receiving module 101 receives video signals from the video interface outside the system. After acquiring the video signal, it transmits it to the compression unit 106, which performs lossless compression on the video signal. After compression, the compressed video data is transmitted to the memory 113 for storage via the memory bus interface 112 for subsequent system retrieval.

[0037] When the image and video encoder 102 needs to process video data, it first performs lossless decompression on the video data stored in the memory 113 through the decompression unit 107. After the data is restored to its original precision, the decompressed video data is then processed by image and video encoding to ensure that the data does not deviate in subsequent encoding processing.

[0038] Before performing identification and detection, the identification and detection module 103 reads the compressed video data from the memory 113 through the memory bus interface 112, and then transmits the read compressed data to the decompression unit 108. The decompression unit 108 performs lossless decompression to obtain the original precision video data before performing the identification and detection function, ensuring that the identification and detection results are not affected by the data precision.

[0039] When the video image processor 104 processes video data, it first performs lossless decompression on the video data in the memory 113 through the decompression unit 109, obtains the decompressed video data, and performs internal calculations. After the calculations are completed, the processed video data is transmitted to the compression unit 110, which performs lossless compression on it. Finally, the compressed video data is written back to the memory 113 through the memory bus interface 112.

[0040] When the display output controller 105 needs to output an image, it decompresses the video data in the memory 113 through the decompression unit 111. Operators can configure parameters to allow the decompression unit 111 to perform either lossless or lossy decompression. After the video data is restored, it is output to the external display device. Through the operation of the above modules, different modules can use the same compression format frame buffer. Lossless compression of the video source data reduces the data bandwidth written to the memory 113, while in display processing-related operations, the output signal can be decompressed using either lossy or lossless methods, meeting the needs of different applications while minimizing bus bandwidth consumption.

[0041] This implementation describes the compression encoding process of the frame buffer, in conjunction with the appendix. Figure 2 The specific operation process is as follows.

[0042] First, the video signal receiving module receives externally transmitted data from the system's video interface, obtaining the original image 201. After acquiring the original image 201, dynamic compression and filtering are performed on it. This processing adjusts the format and details of the image data, generating image 203. Subsequently, image 203 is input into the lossless compression unit for lossless compression. After compression, bitstream 0 is generated, corresponding to data 205. After compression, bitstream 0 is transmitted to the system's memory for storage, for later retrieval.

[0043] While obtaining image 203, the difference between the original image 201 and image 203 is calculated to obtain the image data difference between the two. For this difference, dynamic compression and filtering 1 are performed to optimize the structure of the difference data and generate image 207. Next, image 207 is losslessly compressed to generate bitstream 1, which corresponds to data 209. Consistent with the storage method of bitstream 0, bitstream 1 is also written into the aforementioned memory to achieve centralized storage of multiple bitstreams.

[0044] Next, the difference between the original image 201 and image 203 is calculated first, and then this difference is calculated with image 207. Data 210 is obtained through these two difference calculations. Lossless compression processing is performed on data 210 to generate bitstream 2, which corresponds to data 212. Bitstream 2 is also transferred to the memory for storage, and together with bitstream 0 and bitstream 1, it constitutes the multi-bitstream data corresponding to the original image 201.

[0045] During the entire process of generating bitstreams 0, 1, and 2, the system synchronously records compression information such as compression parameters and data size for each bitstream. Based on this recorded compression information and the image region location information corresponding to each bitstream, a lookup table 213 is generated. Through all the above steps, an image framebuffer is finally compressed into multiple bitstreams, bitstream 0, bitstream 1, and bitstream 2, which, together with the generated lookup table 213, complete the compression encoding process of the framebuffer.

[0046] This implementation describes the decompression process of the frame buffer, in conjunction with the appendix. Figure 3 The specific operation process is as follows.

[0047] Appendix Figure 3 In the table, 301 to 303 correspond to bitstreams 0 to 2 generated during the compression encoding process, respectively. 304 is the lookup table 213 corresponding to these bitstreams. Regardless of whether lossy or lossless decompression is performed subsequently, the contents of lookup table 304 must be read first. After reading lookup table 304, bitstream 0 is read and decompressed using the information in lookup table 304 to obtain image 306; simultaneously, bitstream 1 is read and decompressed using the information in lookup table 304 to obtain image 309; and bitstream 2 is read and decompressed using the information in lookup table 304 to obtain image 311. The bandwidth control module 307 first obtains the bandwidth limitation parameters, then combines them with the compression information recorded in lookup table 304 to generate and output a control signal. This control signal is transmitted to the decoding module corresponding to the bitstream, i.e., the decoding module responsible for processing images 306, 309, and 311. The control signal controls whether each decoding module needs to read data from specific image positions in the corresponding bitstream.

[0048] Combined with appendix Figure 4As shown, a certain region of an image is processed into multiple bitstreams during compression. Due to the differences in content between different image regions, the sizes of the bitstreams generated after compression vary. When using bitstream data, the various application modules in the system can control the bandwidth by configuring parameters on the decoding unit. If at least one bitstream of the image block at the current position is prevented from being read and decoded, the image data obtained after decompression of that bitstream will not participate in the synthesis of the output image. If all bitstreams are read and decoded, the image data after decompression of all bitstreams will participate in the synthesis of the output image, thus adapting to the different application requirements for data precision and bandwidth.

[0049] In the process of processing video data in a multimedia system, the compression and decompression operations of the frame buffer need to be adapted to different scenarios according to the needs of different functional modules. The specific processing methods are as follows.

[0050] After receiving external video data, the video signal receiving module 101 transmits the data to the compression unit 106. The compression unit 106 performs lossless compression on the received data. After compression, the compressed data is stored in the memory 113 via the memory bus interface 112, providing a data basis for subsequent calls to various functional modules. For the image and video encoder 102, the video image processor 104, and the recognition and detection module 103, these modules all need to obtain image data with the original precision when processing video data. Therefore, they read the compressed data from the memory 113 through their respective decompression units (decompression unit 107 for image and video encoder 102, decompression unit 109 for video image processor 104, and decompression unit 108 for recognition and detection module 103), and perform lossless decoding on the compressed data. This operation restores the original data to meet the data precision requirements of encoding processing, pre-display image processing, and recognition and detection functions.

[0051] The display output controller 105 is mainly used to output image data to an external display device. In display application scenarios, image quality can tolerate a certain degree of loss, and display applications usually require a high display frame rate. In this case, the display output controller 105 will perform lossy decompression on the compressed data in the memory 113 through the corresponding decompression unit 111. By appropriately reducing the precision of the image data, more system bandwidth is obtained, ensuring the smoothness of high frame rate display. For example, when displaying 4K@60Hz YUV420 format video data, the original data bandwidth of the video data is 712MB / s. After lossless compression processing, the data bandwidth can be reduced to 400MB / s. If lossy compression processing is used, the data bandwidth can be further reduced to 250MB / s, significantly reducing the system bandwidth burden.

[0052] It should be noted that the appendix Figure 2 The corresponding 200 embodiments (compression encoding process) and appendices Figure 3 In the corresponding 300 implementation (decompression process), the scheme of decomposing the original image data into 3 bitstreams is only a specific example. In practical applications, the original image data can be decomposed into 2 or more bitstreams according to the different requirements of the system for compression efficiency, data accuracy and bandwidth consumption. The compression processing method of each bitstream after decompression and the synthesis logic during decompression are consistent with the principle when decomposing into 3 bitstreams. Both can achieve lossless compression and flexible decompression.

[0053] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0054] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A frame buffer compression decompression method, characterized by, The method comprises the following steps: Obtaining original image data; Filtering and decomposing the original image data to obtain a plurality of sub-images; Respectively losslessly compressing the plurality of sub-images to obtain a plurality of code streams; Recording compression information and position information of the plurality of code streams to generate a lookup table; Reading a target code stream according to the lookup table, decompressing the target code stream, and synthesizing to obtain output image data.

2. The frame buffer compression decompression method of claim 1, wherein, When filtering and decomposing the original image data, the original image data is gradually decomposed into a plurality of sub-images according to human visual characteristics; when respectively losslessly compressing the plurality of sub-images, a lossless compression operation is performed on each sub-image in a certain size block and in a certain order to obtain a corresponding code stream.

3. The frame buffer compression decompression method of claim 1, wherein, After generating the lookup table, the plurality of code streams and the lookup table are stored in a storage in a specific format, the storage is divided into a lookup table area and a data area; when reading the target code stream according to the lookup table, the information in the lookup table is combined through a bandwidth control circuit to determine the code stream and the corresponding block data that need to be read or skipped.

4. The frame buffer compression decompression method of claim 3, wherein, When determining the code stream and the corresponding block data that need to be read or skipped, all blocks of the entire image can be selected to be read or skipped, or blocks at partial positions in the image can be dynamically selected to be read or skipped; when decompressing the target code stream, one or more target code streams are decompressed, and output image data is obtained through a synthesis operation.

5. The frame buffer compression decompression method of claim 3, wherein, All the code streams are decompressed and a synthesis operation is performed to obtain lossless output image data; part of the code streams are decompressed and a synthesis operation is performed to obtain visually lossless output image data; after the output image data is synthesized, the output image data is transmitted to a corresponding application module to adapt to different application requirements such as identification detection or display output.

6. An apparatus for frame buffer compression and decompression, comprising a video signal receiving unit, an image video encoder, a graphic image processing unit, a display output control unit, a data bus port, a compression unit, a decompression unit and a storage, the video signal receiving unit is signal connected with the storage through the data bus port, and the video signal receiving unit is also signal connected with the compression unit; the compression unit is signal connected with the storage through the data bus port; the decompression unit is signal connected with the storage through the data bus port; the image video encoder is signal connected with the storage and the compression unit through the data bus port respectively; the graphic image processing unit is signal connected with the storage, the compression unit and the decompression unit through the data bus port respectively; the display output control unit is signal connected with the decompression unit through the data bus port.

7. The apparatus for frame buffer compression decompression of claim 6, wherein, The compression unit is used for compressing original data in a certain size block and in a certain order and outputting one or more code streams, the compression unit also decomposes original image into different sub-images in a manner consistent with human visual characteristics, compresses each sub-image to generate the code stream, and generates a lookup table containing position information of each block in the corresponding code stream when outputting a plurality of code streams, the code stream and the lookup table are both transmitted to the storage for storage.

8. The apparatus for frame buffer compression decompression of claim 7, wherein, The memory is divided into a lookup table area and a data area, the lookup table is stored in the lookup table area, the code stream is stored in the data area, and the arrangement of the lookup table area and the data area facilitates skipping of data blocks that do not need to be read; the decompression unit is connected to the lookup table; after reading the lookup table and the corresponding code stream in the memory, the decompression unit decompresses one or more code streams and outputs image data.

9. The apparatus for frame buffer compression decompression of claim 8, wherein, The device further comprises a bandwidth control circuit connected to the decompression unit, which is used to control the decompression unit to read or skip the code stream of the block at the corresponding position during decompression, and can control the decompression unit to read or skip all blocks in the entire image, or control the decompression unit to dynamically select to read or skip the blocks at certain positions in the image.

10. The apparatus for frame buffer compression decompression of claim 9, wherein, The decompression unit synthesizes output data by decompressing one or more code streams, decompresses all code streams to synthesize lossless output data, or decompresses part of the code streams to synthesize lossy output data; the output data is transmitted to the display output control unit or the graphic image processing unit; the video signal receiving unit is used to receive a video image signal from outside the system; and the display output control unit is used to convert the received output data into a signal suitable for a display device.

Images