A frame buffer compression and decompression method and apparatus

Abstract

The application discloses a frame buffer compression and decompression method and device, and belongs to the technical field of electronic equipment. The method comprises the following steps: acquiring original image data, performing dynamic compression and filtering processing on the original image data, and decomposing to obtain a plurality of sub-images; performing lossless compression on each sub-image to generate a plurality of code streams, and recording compression information and position information of each code stream to generate a lookup table; storing the plurality of code streams and the lookup table in a memory, wherein the memory is divided into a lookup table area and a data area; determining a target code stream by combining the lookup table through a bandwidth control module according to application requirements, reading the target code stream and decompressing the target code stream to obtain output image data, wherein all code streams are decompressed to obtain lossless output image data for an identification detection type application, and a display type application can decompress part of the code streams to obtain visually lossless output image data.

Description

Technical Field

[0001] This invention relates to the field of electronic devices, and more specifically to compression and decompression techniques in image processing. Background Technology

[0002] In multimedia applications, an increasing number of applications require high-resolution and high-frame-rate images, but this also leads to significant bandwidth consumption. For example, displaying a single frame of 4K resolution YUV420 video image consumes 3840*2160*1.5*60=712MB / s of bandwidth, while UI data in ARGB8888 format consumes 1898MB / s. Considering the reception and processing of video signals by camera, encoding / decoding modules, and other related modules, the system's bandwidth consumption for display is substantial.

[0003] Therefore, it is essential to introduce frame buffer compression and decompression technology into the system.

[0004] Common video compression techniques are lossy compression based on both temporal and spatial constraints, and the process is relatively complex, making them unsuitable as a compression method for frame buffers in a system. Generally, intra-frame spatial compression methods are used instead.

[0005] One approach is to use intra-frame lossy compression (or visually lossless), which offers high compression efficiency and controllable bandwidth. However, it also has application limitations. For example, in common image recognition and detection, if lossy or hybrid compression methods are used on the original video data, due to the complexity of the recognition and detection algorithms and application scenarios, it is impossible to determine how the compression loss will affect the subsequent recognition and detection results. Generally, it is not possible to perform processing that results in a loss of accuracy for recognition and detection applications. If the bandwidth of the display application needs to be reduced, two frame buffers need to be stored, one for the recognition and detection application and the other for the display output application.

[0006] Another approach is to use intra-frame lossless compression. Since the original data is preserved, it is suitable for any application. However, the overall compression bandwidth is not as high as lossy compression, and the overall bandwidth consumption is not low enough.

[0007] In response, some new technologies employ adaptive content control compression, employing either lossy block compression or a hybrid lossless block compression method. However, lossy data can also lead to problems unsuitable for certain applications. Summary of the Invention

[0008] The purpose of this invention is to provide a method and apparatus for frame buffer compression and decompression, which solves the problems that existing intra-frame lossy compression cannot be adapted to identification and detection applications, and intra-frame lossless compression is difficult to reduce bandwidth consumption, resulting in the system needing to store multiple frame buffers for different applications, and the heavy burden of resources and bandwidth.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] In a first aspect, the present invention provides a frame buffer compression and decompression method, comprising the following steps:

[0011] Obtain the raw image data;

[0012] The original image data is filtered and decomposed to obtain multiple sub-images;

[0013] The multiple sub-images are each subjected to lossless compression to obtain multiple bitstreams;

[0014] Record the compression and location information of the multiple bitstreams and generate a lookup table;

[0015] The target bitstream is read according to the lookup table, the target bitstream is decompressed, and the output image data is synthesized.

[0016] In one possible implementation, when filtering and decomposing the original image data, the original image data is gradually decomposed into multiple sub-images according to the characteristics of human vision; when performing lossless compression on each of the multiple sub-images, lossless compression is performed on each sub-image in blocks of a specific size and in a certain order to obtain the corresponding bitstream.

[0017] In one possible implementation, after the lookup table is generated, the multiple bitstreams and the lookup table are stored in a memory in a specific format, and the memory is divided into a lookup table area and a data area; when reading the target bitstream according to the lookup table, the bandwidth control circuit, in conjunction with the information in the lookup table, determines the bitstream and corresponding block data that need to be read or skipped.

[0018] In one possible implementation, when determining the bitstream and corresponding block data that need to be read or skipped, it is possible to select to read or skip all blocks of the entire image, or dynamically select to read or skip blocks at certain locations in the image; when decompressing the target bitstream, one or more target bitstreams are decompressed, and then the output image data is obtained through a synthesis operation.

[0019] In one possible implementation, all the bitstreams are decompressed and synthesized to obtain lossless output image data; a portion of the bitstreams are decompressed and synthesized to obtain visually lossless output image data; after synthesizing the output image data, the output image data is transmitted to the corresponding application module to adapt to different application requirements such as recognition detection or display output.

[0020] Secondly, the present invention provides a frame buffer compression and decompression apparatus, comprising a video signal receiving unit, an image / video encoder, a graphics / image processing unit, a display output control unit, a data bus port, a compression unit, a decompression unit, and a memory. The video signal receiving unit is signal-connected to the memory via the data bus port, and is also signal-connected to the compression unit. The compression unit is signal-connected to the memory via the data bus port. The decompression unit is signal-connected to the memory via the data bus port. The image / video encoder is signal-connected to both the memory and the compression unit via the data bus port. The graphics / image processing unit is signal-connected to the memory, the compression unit, and the decompression unit via the data bus port. The display output control unit is signal-connected to the decompression unit via the data bus port.

[0021] In one possible implementation, the compression unit is used to compress the original data into blocks of a specific size and in a certain order, and output one or more bitstreams. The compression unit also decomposes the original image into different sub-images in a manner that conforms to the characteristics of human vision, compresses each sub-image separately to generate the bitstream, and generates a lookup table containing the position information of each block in the corresponding bitstream when outputting multiple bitstreams. Both the bitstream and the lookup table are transmitted to the memory for storage.

[0022] In one possible implementation, the memory is divided into a lookup table area and a data area. The lookup table is stored in the lookup table area, and the bitstream is stored in the data area. The arrangement of the lookup table area and the data area facilitates skipping data blocks that do not need to be read. The decompression unit is connected to the lookup table signal. After reading the lookup table and the corresponding bitstream in the memory, the decompression unit decompresses one or more bitstreams and outputs image data.

[0023] In one possible implementation, the device further includes a bandwidth control circuit connected to the decompression unit. The bandwidth control circuit is used to control the decompression unit to read or skip the bitstream of corresponding position blocks during decompression. It can control the decompression unit to read or skip all blocks in the entire image, or control the decompression unit to dynamically select to read or skip blocks at certain positions in the image.

[0024] In one possible implementation, the decompression unit synthesizes output data by decompressing one or more of the bitstreams; the decompression unit synthesizes lossless output data by decompressing all of the bitstreams; the decompression unit synthesizes lossy output data by decompressing a portion of the bitstreams; the output data is transmitted to the display output control unit or the graphics processing unit; the video signal receiving unit is used to receive video image signals from outside the system; and the display output control unit is used to convert the received output data into a signal suitable for the display device to receive.

[0025] Compared with existing technologies, the advantages of this invention are as follows: This invention generates multiple bitstreams through lossless compression after filtering and decomposing the original image data, and records the compression and location information of each bitstream to create a lookup table, enabling the same frame buffer format to adapt to different application scenarios. It eliminates the need to store multiple frame buffers separately for different applications such as recognition, detection, and display output, effectively saving memory space and reducing bandwidth consumption during data transmission and storage, thus alleviating the overall bandwidth burden of the system.

[0026] In the decompression stage, this invention uses a bandwidth control module combined with a lookup table to determine the target bitstream. It can select to decompress the entire bitstream to obtain lossless output image data according to application requirements, thus meeting the data accuracy requirements of recognition and detection applications; or it can select to decompress a portion of the bitstream to obtain visually lossless output image data, adapting to the image quality requirements of display applications.

[0027] This invention stores multiple bitstreams and lookup tables in a specific format in a memory divided into a lookup table area and a data area. Through a clear partitioned storage method, combined with the bitstream location information recorded in the lookup table, the target bitstream can be quickly located and read, reducing the location time during data reading, improving operational convenience, and avoiding the reading of unnecessary data blocks, further helping to reduce bandwidth consumption. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A multimedia system architecture diagram illustrating the application of the frame buffer compression and decompression method provided in this embodiment of the invention;

[0030] Figure 2 A flowchart of the compression encoding process of the frame buffer compression unit provided in an embodiment of the present invention;

[0031] Figure 3 This is a flowchart of the decompression process of the frame buffer decompression unit provided in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram showing the distribution of multiple bitstreams after image region compression, as provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0034] Example:

[0035] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0036] This implementation method focuses on the compression and decompression processing of the frame buffer, in conjunction with the appendix. Figure 1 A multimedia system contains multiple functional modules and multiple compression and decompression units. The video signal receiving module 101 receives video signals from the video interface outside the system. After acquiring the video signal, it transmits it to the compression unit 106, which performs lossless compression on the video signal. After compression, the compressed video data is transmitted to the memory 113 for storage via the memory bus interface 112 for subsequent system retrieval.

[0037] When the image and video encoder 102 needs to process video data, it first performs lossless decompression on the video data stored in the memory 113 through the decompression unit 107. After the data is restored to its original precision, the decompressed video data is then processed by image and video encoding to ensure that the data does not deviate in subsequent encoding processing.

[0038] Before performing identification and detection, the identification and detection module 103 reads the compressed video data from the memory 113 through the memory bus interface 112, and then transmits the read compressed data to the decompression unit 108. The decompression unit 108 performs lossless decompression to obtain the original precision video data before performing the identification and detection function, ensuring that the identification and detection results are not affected by the data precision.

[0039] When the video image processor 104 processes video data, it first performs lossless decompression on the video data in the memory 113 through the decompression unit 109, obtains the decompressed video data, and performs internal calculations. After the calculations are completed, the processed video data is transmitted to the compression unit 110, which performs lossless compression on it. Finally, the compressed video data is written back to the memory 113 through the memory bus interface 112.

[0040] When the display output controller 105 needs to output an image, it decompresses the video data in the memory 113 through the decompression unit 111. Operators can configure parameters to allow the decompression unit 111 to perform either lossless or lossy decompression. After the video data is restored, it is output to the external display device. Through the operation of the above modules, different modules can use the same compression format frame buffer. Lossless compression of the video source data reduces the data bandwidth written to the memory 113, while in display processing-related operations, the output signal can be decompressed using either lossy or lossless methods, meeting the needs of different applications while minimizing bus bandwidth consumption.

[0041] This implementation describes the compression encoding process of the frame buffer, in conjunction with the appendix. Figure 2 The specific operation process is as follows.

[0042] First, the video signal receiving module receives externally transmitted data from the system's video interface, obtaining the original image 201. After acquiring the original image 201, dynamic compression and filtering are performed on it. This processing adjusts the format and details of the image data, generating image 203. Subsequently, image 203 is input into the lossless compression unit for lossless compression. After compression, bitstream 0 is generated, corresponding to data 205. After compression, bitstream 0 is transmitted to the system's memory for storage, for later retrieval.

[0043] While obtaining image 203, the difference between the original image 201 and image 203 is calculated to obtain the image data difference between the two. For this difference, dynamic compression and filtering 1 are performed to optimize the structure of the difference data and generate image 207. Next, image 207 is losslessly compressed to generate bitstream 1, which corresponds to data 209. Consistent with the storage method of bitstream 0, bitstream 1 is also written into the aforementioned memory to achieve centralized storage of multiple bitstreams.

[0044] Next, the difference between the original image 201 and image 203 is calculated first, and then this difference is used to perform a difference operation with image 207. Data 210 is obtained through these two difference calculations. Lossless compression processing is performed on data 210 to generate bitstream 2, which corresponds to data 212. Bitstream 2 is also transferred to the memory for storage, and together with bitstream 0 and bitstream 1, it constitutes the multi-bitstream data corresponding to the original image 201.

[0045] During the entire process of generating bitstreams 0, 1, and 2, the system synchronously records compression information such as compression parameters and data size for each bitstream. Based on this recorded compression information and the image region location information corresponding to each bitstream, a lookup table 213 is generated. Through all the above steps, an image framebuffer is finally compressed into multiple bitstreams, bitstream 0, bitstream 1, and bitstream 2, which, together with the generated lookup table 213, complete the compression encoding process of the framebuffer.

[0046] This implementation describes the decompression process of the frame buffer, in conjunction with the appendix. Figure 3 The specific operation process is as follows.

[0047] Appendix Figure 3 In the table, 301 to 303 correspond to bitstreams 0 to 2 generated during the compression encoding process, respectively. 304 is the lookup table 213 corresponding to these bitstreams. Regardless of whether lossy or lossless decompression is performed subsequently, the contents of lookup table 304 must be read first. After reading lookup table 304, bitstream 0 is read and decompressed using the information in lookup table 304 to obtain image 306; simultaneously, bitstream 1 is read and decompressed using the information in lookup table 304 to obtain image 309; and bitstream 2 is read and decompressed using the information in lookup table 304 to obtain image 311. The bandwidth control module 307 first obtains the bandwidth limitation parameters, then combines them with the compression information recorded in lookup table 304 to generate and output a control signal. This control signal is transmitted to the decoding module corresponding to the bitstream, i.e., the decoding module responsible for processing images 306, 309, and 311. The control signal controls whether each decoding module needs to read data from specific image positions in the corresponding bitstream.

[0048] Combined with appendix Figure 4As shown, a certain region of an image is processed into multiple bitstreams during compression. Due to the differences in content between different image regions, the sizes of the bitstreams generated after compression vary. When using bitstream data, the various application modules in the system can control the bandwidth by configuring parameters on the decoding unit. If at least one bitstream of the image block at the current position is prevented from being read and decoded, the image data obtained after decompression of that bitstream will not participate in the synthesis of the output image. If all bitstreams are read and decoded, the image data after decompression of all bitstreams will participate in the synthesis of the output image, thus adapting to the different application requirements for data precision and bandwidth.

[0049] In the process of processing video data in a multimedia system, the compression and decompression operations of the frame buffer need to be adapted to different scenarios according to the needs of different functional modules. The specific processing methods are as follows.

[0050] After receiving external video data, the video signal receiving module 101 transmits the data to the compression unit 106. The compression unit 106 performs lossless compression on the received data. After compression, the compressed data is stored in the memory 113 via the memory bus interface 112, providing a data basis for subsequent calls to various functional modules. For the image and video encoder 102, the video image processor 104, and the recognition and detection module 103, these modules all need to obtain image data with the original precision when processing video data. Therefore, they read the compressed data from the memory 113 through their respective decompression units (decompression unit 107 for image and video encoder 102, decompression unit 109 for video image processor 104, and decompression unit 108 for recognition and detection module 103), and perform lossless decoding on the compressed data. This operation restores the original data to meet the data precision requirements of encoding processing, pre-display image processing, and recognition and detection functions.

[0051] The display output controller 105 is mainly used to output image data to an external display device. In display application scenarios, image quality can tolerate a certain degree of loss, and display applications usually require a high display frame rate. In this case, the display output controller 105 will perform lossy decompression on the compressed data in the memory 113 through the corresponding decompression unit 111. By appropriately reducing the precision of the image data, more system bandwidth is obtained, ensuring the smoothness of high frame rate display. For example, when displaying 4K@60Hz YUV420 format video data, the original data bandwidth of the video data is 712MB / s. After lossless compression processing, the data bandwidth can be reduced to 400MB / s. If lossy compression processing is used, the data bandwidth can be further reduced to 250MB / s, significantly reducing the system bandwidth burden.

[0052] It should be noted that the appendix Figure 2 The corresponding 200 embodiments (compression encoding process) and appendices Figure 3 In the corresponding 300 implementation (decompression process), the scheme of decomposing the original image data into 3 bitstreams is only a specific example. In practical applications, the original image data can be decomposed into 2 or more bitstreams according to the different requirements of the system for compression efficiency, data accuracy and bandwidth consumption. The compression processing method of each bitstream after decompression and the synthesis logic during decompression are consistent with the principle when decomposing into 3 bitstreams. Both can achieve lossless compression and flexible decompression.

[0053] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0054] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A frame buffer compression and decompression method, characterized in that, Includes the following steps: Obtain the raw image data; The original image data is filtered and decomposed to obtain multiple sub-images; The multiple sub-images are each subjected to lossless compression to obtain multiple bitstreams; Record the compression and location information of the multiple bitstreams and generate a lookup table; After generating the lookup table, the multiple bitstreams and the lookup table are stored in a memory in a specific format, and the memory is divided into a lookup table area and a data area. When reading the target bitstream according to the lookup table, the bandwidth control circuit, in conjunction with the information in the lookup table, determines the bitstream and corresponding block data that need to be read or skipped. The target bitstream is read according to the lookup table, the target bitstream is decompressed, and the output image data is synthesized. Decompress all the bitstreams and perform a synthesis operation to obtain lossless output image data; decompress a portion of the bitstreams and perform a synthesis operation to obtain visually lossless output image data; After synthesizing the output image data, the output image data is transmitted to the corresponding application module to adapt to different application requirements such as recognition, detection, or display output.

2. The frame buffer compression and decompression method according to claim 1, characterized in that, When filtering and decomposing the original image data, the original image data is gradually decomposed into multiple sub-images according to the characteristics of human vision; when performing lossless compression on the multiple sub-images, lossless compression is performed on each sub-image in blocks of a specific size and in a certain order to obtain the corresponding bitstream.

3. The frame buffer compression and decompression method according to claim 1, characterized in that, When determining the bitstream and corresponding block data that need to be read or skipped, it is possible to select to read or skip all blocks of the entire image, or dynamically select to read or skip blocks at certain locations in the image; when decompressing the target bitstream, one or more target bitstreams are decompressed, and then the output image data is obtained through a synthesis operation.

4. A frame buffer compression and decompression apparatus, comprising a video signal receiving unit, an image / video encoder, a graphics / image processing unit, a display output control unit, a data bus port, a compression unit, a decompression unit, and a memory; wherein the video signal receiving unit is signal-connected to the memory via the data bus port, and the video signal receiving unit is also signal-connected to the compression unit; the compression unit is signal-connected to the memory via the data bus port; the decompression unit is signal-connected to the memory via the data bus port; the image / video encoder is signal-connected to the memory and the compression unit via the data bus port; the graphics / image processing unit is signal-connected to the memory, the compression unit, and the decompression unit via the data bus port; and the display output control unit is signal-connected to the decompression unit via the data bus port. The memory is divided into a lookup table area and a data area. The lookup table is stored in the lookup table area, and the bitstream is stored in the data area. The arrangement of the lookup table area and the data area facilitates skipping data blocks that do not need to be read. The decompression unit is connected to the lookup table signal. After reading the lookup table and the corresponding bitstream in the memory, the decompression unit decompresses one or more bitstreams and outputs image data. The decompression unit synthesizes output data by decompressing one or more of the bitstreams; the decompression unit synthesizes lossless output data by decompressing all of the bitstreams; the decompression unit synthesizes lossy output data by decompressing a portion of the bitstreams; the output data is transmitted to the display output control unit or the graphics processing unit; the video signal receiving unit is used to receive video image signals from outside the system; and the display output control unit is used to convert the received output data into a signal suitable for the display device to receive.

5. The frame buffer compression and decompression apparatus according to claim 4, characterized in that, The compression unit is used to compress the original data into blocks of a specific size and in a certain order, and output one or more bitstreams. The compression unit also decomposes the original image into different sub-images in a way that conforms to the characteristics of human vision, compresses each sub-image separately to generate the bitstream, and generates a lookup table containing the position information of each block in the corresponding bitstream when outputting multiple bitstreams. Both the bitstream and the lookup table are transmitted to the memory for storage.

6. The frame buffer compression and decompression apparatus according to claim 4, characterized in that, The device further includes a bandwidth control circuit, which is signal-connected to the decompression unit. The bandwidth control circuit is used to control the decompression unit to read or skip the bitstream of corresponding position blocks during decompression. It can control the decompression unit to read or skip all blocks in the entire image, or control the decompression unit to dynamically select to read or skip blocks at certain positions in the image.

Images