Method for using frame buffer in decoding process

Abstract

A method of using a frame buffer in a decoding process is provided. The method is performed on a dedicated hardware circuit. The method includes storing data representative of first frame data using a frame buffer. The data representative of the first frame data is used in subsequent decoding of the second frame data. The frame buffer is stored in a memory external to the dedicated hardware circuit. The data representing the first frame data is a set of transform elements indicating a degree of spatial correlation of the first frame data. The method compresses the set of transform elements using a lossless compression technique and sends the compressed set of transform elements to a frame buffer for extraction when decoding the second frame data.

Description

Technical Field

[0001] The present application relates to the use of a frame buffer by a decoder during a decoding process. In particular (but not limited to), the decoder is configured to decode a data signal containing frame data. In particular (but not limited to), the data signal is related to video data. In particular (but not limited to), the decoder uses low complexity video coding (LCEVC) technology. In particular (but not limited to), the decoder is implemented on a dedicated hardware circuit, and the dedicated hardware circuit uses an external memory where the frame buffer is located to store data used in the decoding process. Background Art

[0002] Data is often transferred from one place to another in order to be used. For example, video or image data may be transferred from a server or storage medium to a client device for display. To facilitate transmission and storage, the data is often encoded. Once the client device receives the data, it must decode the encoded data to reconstruct the original signal or an approximation of it.

[0003] In some embodiments, when decoding subsequent data, the decoder can reuse data generated previously in the decoding process. Previously generated data is stored in a memory (particularly a "frame buffer"), and the decoder can access this data when needed, such as when decoding subsequent data frames. It is important to ensure that the data stored in the frame buffer is accessed quickly enough to achieve real-time decoding, otherwise the decoding process may encounter unacceptable bottlenecks and the decoded data will not be presented in a timely manner. For example, individual frames of video data must be decoded in a timely manner so that the video frames can be presented at the appropriate time to maintain the frame rate. This challenge is greater for relatively high frame resolutions (such as the current 8K), and further increased for relatively high frame rates (such as the current 60FPS), and vice versa.

[0004] Typically, it is preferred to implement the decoder on a dedicated hardware circuit, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). Typically, the frame buffer is located in main memory external to the dedicated hardware circuit (i.e., "off-chip memory"), because the cost of setting up memory on a dedicated hardware circuit is relatively high. This arrangement introduces a potential bottleneck to the decoding process because accessing off-chip memory is relatively slow compared to accessing on-chip memory. In some applications, especially those involving large amounts of data that push the limits of current hardware processing and storage technology (e.g., 8K60FPS video data), the process of reading and writing data to off-chip memory may be so slow that it interrupts real-time decoding.

[0005] Therefore, there is a need for a technique for managing data at a decoder to prevent unnecessary delays in the decoding process due to memory access speed limitations. There is also a need for a technique for managing data at a decoder to prevent unnecessary delays in the decoding process due to memory access speed limitations in the special case where the decoder is located on a dedicated hardware circuit and the frame buffer is located on a memory external to the dedicated hardware circuit. There is also a need for a technique that can perform real-time decoding for the above-mentioned situation. The invention described in this application attempts to provide a solution that at least partially satisfies one or more of the above-mentioned needs. Summary of the invention

[0006] According to a first aspect of the present invention, a method for using a frame buffer in a decoding process is provided. The method of this aspect is usually performed on a dedicated hardware circuit, such as an ASIC, FPGA, etc. However, the method can also be used in other implementation schemes where there is a read / write bottleneck when accessing the memory. The method includes using a frame buffer to store data representing a first frame of data. The data representing the first frame of data is used when processing a second frame of data. In a general embodiment, the frame buffer is stored in a memory outside the dedicated hardware circuit. The data representing the first frame of data is a set of transform elements indicating the degree of spatial correlation in the first frame of data. The method compresses the transform element set using a lossless compression technique, and sends the compressed transform element set to the frame buffer to be taken out when processing the second frame of data.

[0007] Thus, the data representing the first frame can be compressed by a relatively large factor, for example, by a factor of 100, whereas conventional frame buffer compression techniques may only achieve a factor of 2-3. Data indicating the degree of spatial correlation in the frame data is relatively sparse compared to the frame data itself, and can be highly compressed when compressed using lossless compression techniques. As a result, the time required for the decoder to write the data representing the first frame to and retrieve the data from an external memory can be significantly reduced. As a result, there is less likelihood of unnecessary delays or interruptions in real-time decoding due to the slow speed at which devices such as off-chip memory can read and write data. In addition, the use of lossless compression techniques can reduce artifacts in the decoded data.

[0008] Preferably, retrieving the transform element set from the frame buffer comprises: performing an inverse lossless compression technique on the compressed transform element set. In this way, the compressed data stored in the external memory can be returned to an uncompressed format that can be used by the decoder during decoding.

[0009] Preferably, the first frame data includes a first residual element set.

[0010] Preferably, the first set of residual elements is based on differences between a first representation of a first frame associated with the first frame data at a first quality level in a hierarchical structure having a plurality of quality levels and a second representation of the first frame at the first quality level.

[0011] Preferably, the transform element sets indicate the degree of spatial correlation between the residual element sets, so that at least one of the transform element sets indicates at least one of the average, horizontal, vertical and diagonal (AHVD) relationship between adjacent residual elements in the residual element set. In this way, by using data related to the AHVD relationship between adjacent residual elements in the residual element set, a higher compression rate can be achieved because such data is sparse and can be significantly compressed. In addition, AVHD data can be processed in parallel, allowing for fast compression and faster reading and writing of data to and from memory.

[0012] Preferably, the method comprises receiving first input data indicating a degree of temporal correlation between the set of transformed elements and a second set of transformed elements.

[0013] Preferably, the second transform element set indicates a degree of spatial correlation of the second residual element set.

[0014] Preferably, said second set of residual elements is used for reconstructing presentation of the second frame associated with said second frame data at said first quality level by using data based on presentation of the second frame at said second quality level.

[0015] Preferably, said second set of residual elements is based on differences between a first representation of said second frame at said first quality level and a second representation of said second frame at said first quality level in a hierarchical hierarchy having a plurality of quality levels.

[0016] Preferably, the second transform element set indicates a degree of spatial correlation between multiple residual elements in the second residual element set associated with the second frame, so that at least one of the second transform element sets indicates at least one of an average, horizontal, vertical and diagonal relationship between adjacent residual elements in the second residual element set.

[0017] Preferably, the method comprises combining the first input data with the set of transformed elements to generate the second set of transformed elements.

[0018] Preferably, the method comprises performing an inverse transform operation on the second set of transformed elements to generate the second set of residual elements.

[0019] Preferably, the method comprises receiving second input data. In one example, the second input data is at a second quality level in the hierarchical structure. Optionally, the second quality level is lower than the first quality level.

[0020] Preferably, the method comprises performing an upsampling operation on said second input data to generate a second representation of said second frame at said first quality level.

[0021] Preferably, the method comprises combining a second representation of said second frame with said second set of residual elements to reconstruct said second frame.

[0022] Preferably, the first input data comprises a quantized version of a difference result between the first set of transformed elements and the second set of transformed elements.

[0023] Preferably, the set of transform elements is associated with an array of signal elements in the first frame, and the second set of transform elements is associated with an array of signal elements in the second frame at the same spatial position as the array of signal elements in the first frame.

[0024] Preferably, the lossless compression technique comprises two different lossless compression techniques. However, in some embodiments, It may be useful to use one lossless compression technique or two or more lossless compression techniques.

[0025] Preferably, the lossless compression technique comprises at least one of run length encoding and Huffman encoding. However, in some embodiments, it may be useful to use other types of lossless compression techniques such as range encoding.

[0026] Preferably, the lossless compression technique comprises run length encoding followed by Huffman encoding.

[0027] In one example, the decoding process is configured to decode a video signal. In a more specific example, the video signal is at least an 8K, 60FPS video signal.

[0028] According to a second aspect of the present invention, there is provided a decoder device implemented on a dedicated hardware circuit, wherein the decoder device comprises a data communication link for communicating with an external memory. The decoder device is configured to perform any of the aforementioned methods.

[0029] According to a second aspect of the present invention, there is provided a computer program comprising instructions, which, when executed, The decoder device is caused to perform any of the aforementioned methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention will now be described, by way of example only, with reference to the accompanying drawings:

[0031] Figure 1 is a block diagram showing an example of a signal processing system including hardware modules;

[0032] Figure 2 To display in more detail Figure 1 A schematic diagram of a hardware module of FIG. 1 and a flowchart according to an embodiment of the present invention are also shown;

[0033] Figure 3 A schematic diagram of a decoding process according to an embodiment of the present invention;

[0034] Figure 4 A flowchart describing a method of storing and retrieving a frame buffer according to an embodiment of the present invention;

[0035] Figure 5 A timing diagram for describing the timing process of a portion of the serial enhancement architecture for decoding input data; and

[0036] Figure 6 A timing diagram describing the parallel architecture timing process of decoding input data. DETAILED DESCRIPTION

[0037] Figure 1 1 is a block diagram of an example of a signal processing system 100 for understanding the context of the present disclosure. The signal processing system 100 is used to process signals. The types of signals include, but are not limited to, video signals, image signals, audio signals, volumetric signals (such as signals used for medical, scientific or holographic imaging), or other multi-dimensional signals.

[0038] The signal processing system 100 includes a first device 102 and a second device 104. The first device 102 and the second device 104 may have a client-server relationship, wherein the first device 102 performs the function of a server device and the second device 104 performs the function of a client device. The first device 102 and / or the second device 104 may include one or more components. These components may be implemented by hardware and / or software. In the signal processing system 100, one or more components may be located in the same location or may be remote from each other. The types of devices include, for example, but are not limited to, computer equipment, routers, workstations, handheld or laptop computers, tablet computers, mobile devices, game consoles, smart TVs, set-top boxes, augmented and / or virtual reality helmets, etc.

[0039] The first device 102 is connected to the second device 104 through a data communication network 106. The data communication network 106 includes, for example, but is not limited to, the Internet, a local area network (LAN), and a wide area network (WAN). The first and / or second device 102, 104 can be connected to the data communication network 106 via wired and / or wireless connections.

[0040] The first device 102 includes an encoder device 108. The encoder device 108 is configured to encode the signal by encoding the signal data in the signal. In addition to encoding the signal data, the encoder device 108 can also perform one or more other functions. The encoder device 108 can be embodied in various different ways. For example, the encoder device 108 can be embodied in hardware and / or software.

[0041] The second device 104 includes a hardware module 110 and an external memory 112 (i.e., an off-chip memory) outside the hardware module 110. In this example, the hardware module 110 is an application-specific integrated circuit (ASIC), but it can also be other types of hardware circuits or modules. The hardware module 110 includes other types of dedicated hardware circuits, such as a field programmable gate array (FPGA). The hardware module 110 includes a decoder device 114.

[0042] The encoder device 108 encodes the signal data and transmits the encoded signal data to the decoder device 114 via the data communication network 106. The decoder device 114 decodes the received encoded signal data and generates decoded signal data. The decoder device 114 is configured to use or output the decoded signal data, or data generated by the decoded signal data. For example, the decoder device 114 can output such data for display on one or more display devices associated with the second device 104. When decoding the encoded signal data, the decoder device 114 is configured to use the external memory 112 to store the frame buffer 116. In addition to decoding the encoded signal data, the decoder device 114 can also perform one or more other functions.

[0043] In some examples, the encoder device 108 transmits to the decoder device 114 a signal presentation of a given quality level and information that the decoder device 114 can use to reconstruct the signal presentation of one or more higher quality levels. The signal presentation of a given quality level can be considered as a representation, version, or description of the data contained in the signal of the given quality level. The difference between the signal presentation of a given quality level (where the signal presentation has been sampled from a lower quality level) and the original signal of the given quality level is called a residual. The term residual data is well known to those skilled in the art, see, for example, WO2018046940A1. The information used by the decoder device 114 to reconstruct one or more higher quality level signals may be represented by the residual data.

[0044] The above concept is known as scalable video coding. An example of scalable video coding is Low Complexity Video Coding (LCEVC), which allows a relatively small amount of information to be used for such reconstruction. This can reduce the amount of data transmitted over the data communication network 106. This savings may be particularly important when the signal data corresponds to high-quality video data.

[0045] Alternatively, the encoded signal data may be stored in a storage medium accessible to the second device 104 and / or the decoder device 114 and may not be transmitted over a network.

[0046] In some embodiments, the decoder device 114 can reuse signal data previously generated during the decoding process when decoding subsequent signal data, such as when there is a temporal connection between signal data elements. The previously generated signal data is stored in the frame buffer 116, and the decoder device 114 can access this data when needed, for example, when the signal data is arranged in frames (such as a video signal), the decoder device 114 can access this data when decoding subsequent frame signal data. It is important to ensure that access to the data stored in the frame buffer 116 is fast enough to achieve real-time decoding, otherwise an unacceptable bottleneck may occur in the decoding pipeline and the decoded data will not be presented in a timely manner. For example, each frame of video data must be decoded in a timely manner so that a frame of video is presented at the appropriate time to maintain the frame rate. This challenge is greater for relatively high frame resolutions (such as the current 8K), and further increased for relatively high frame rates (such as the current 60FPS), and vice versa.

[0047] Figure 2 To display in more detail Figure 1 1, and also illustrates the process of storing and retrieving relevant parts of signal data into and from the frame buffer 116 according to an embodiment of the present invention. The hardware module 110 includes a lossless compression module 210, a storage controller 212, and an inverse lossless compression module 214 in addition to the decoder device 114. Figure 2 Also shown is an external memory 112 having stored thereon a frame buffer 116. The external memory 112 responds to access requests, as is well known in the art and will be understood by those skilled in the art.

[0048] In a specific example, during the decoding process, the decoder device 114 receives the encoded frame data (frame n) 202 as part of the encoded signal data, typically from the first device 102, but may also come from other sources, such as computer memory (not shown), and the set of transform elements (frame n-1) 206 from the frame buffer 116. The decoder device 114 decodes the signal as needed and outputs a reconstructed frame (frame n) 216 using the encoded frame data (frame n) 202 and the set of transform elements (frame n-1) 206. The decoder device 114 also outputs a new or updated set of transform elements (frame n) 208 to be stored in the frame buffer 116 for use in subsequent frame decoding processes. It will be apparent that "n" refers to the frame number in the frame sequence, frame n refers to the current frame being processed, and frame n-1 refers to the previous frame that has been processed. Frame n is the subsequent frame of frame n-1.

[0049] In this example, the encoded frame data (frame n) 202, the transform element set (frame n-1) 206, the transform element set (frame n) Both the reconstructed frame n 208 and the reconstructed frame n 216 are associated with a video signal. However, other signals may also be processed in this manner.

[0050] The transform element set (frame n-1) 206 indicates the degree of spatial correlation in the corresponding frame data, for example, the correlation in the frame data corresponding to frame n-1. The decoder device 114 uses the encoded frame data (frame n) 202 and the transform element set (frame n-1) 206 to reconstruct frame n, and in the process generate a new set of transform elements (frame n) for reconstructing the subsequent frame (frame n+1).

[0051] The decoder device 114 sends the generated transform element set (frame n) 208 to the lossless compression module 210 for lossless compression operation. Then, the compressed transform element set (frame n) is sent to the storage controller 212 to be forwarded to the frame buffer 116 in the external memory 112. The generated transform element set (frame n) 208 overwrites the transform element set (frame n-1) 206 previously stored in the frame buffer 116.

[0052] When the decoder device 114 is decoding the current frame (frame n), the memory controller 212 will read the current frame from the external memory 112. The compressed transform element set (frame n-1) is taken out from the external memory 112. As described above, the transform element set is stored in a compressed format in the external memory 112. The storage controller 212 is configured to send the taken out compressed transform element set (frame n-1) to the inverse lossless compression module 214 to generate the transform element set (frame n-1) 206 in an uncompressed format, which is then sent to the decoder device 114 to reconstruct the current frame.

[0053] The decoder device 114 repeats the above process for the subsequent frame (frame n+1) and retrieves the information in the transform element set (frame n) 208 from the frame buffer 116 .

[0054] The disclosed technique requires storage of the previous frame data in order to generate future frame data. The disclosed technique stores the previous frame data in the form of data indicating the degree of spatial correlation in the previous frame data rather than the original data itself. The data indicating the spatial correlation in the frame data is sparse and highly compressible compared to the original frame data itself. Therefore, the compressed data can be sent to an external memory and restored from the external memory relatively quickly, so that real-time decoding can be performed uninterruptedly.

[0055] In this example implementation, the compression module 210 and the inverse compression module 214 are located outside the decoder device 114. However, The compression module may also be located inside the decoder device 114 .

[0056] In this example embodiment, the transform element set (frame n-1) 206 represents at least one of the average, horizontal, vertical and diagonal (AHVD) relationships between adjacent signal elements in the previous frame data, or any combination thereof. Thus, by using data related to the AHVD data, higher compressibility can be achieved because the data is sparse and can be compressed significantly. In addition, the AVHD data can be processed in parallel, thereby achieving fast compression and increasing the speed of reading and writing data to the memory.

[0057] In this example embodiment, the encoded frame data (frame n) 202 indicates the degree of temporal correlation between the transform element set (frame n−1) 206 and the transform element set (frame n) 208 .

[0058] In this example embodiment, the transform element set (frame n) 208 represents at least one of an average, horizontal, vertical, and diagonal relationship between adjacent signal elements in the current frame data, or any combination thereof.

[0059] In this example embodiment, the encoded frame data (frame n) 202 includes a quantized version of the difference between the transformed element set (frame n−1) 206 and the transformed element set (frame n) 208 .

[0060] In this example embodiment, transform element set (frame n-1) 206 is associated with the array of signal elements in the previous frame, The transform element set (frame n) 208 is associated with the array of signal elements in the current frame, and the spatial position of the transform element set (frame n) 208 is the same as the array of signal elements in the previous frame.

[0061] In this embodiment, the lossless compression technique includes two different lossless compression techniques. Alternatively, the lossless compression technique includes at least one of run-length encoding and Huffman encoding. Alternatively, the lossless compression technique includes run-length encoding followed by Huffman encoding, or Huffman encoding followed by run-length encoding.

[0062] Figure 3 A more specific decoding flow chart according to an embodiment of the present invention. Figure 3 The decoding process shown is suitable for Figure 2 The illustrated embodiment is implemented on a hardware module 110 , and for ease of reference, like reference numerals denote like components and signals. Figure 3 The process shown is applicable to a particular type of scalable video coding, although the present disclosure Figure 2 The decoder device 114 receives the first input data 302 and the second input data 304 , and outputs a reconstructed frame 316 .

[0063] In this example embodiment, the first input data 302 and the second input data 304 are related to video signals. The first input data 302 and the second input data 304 may also be related to other signals. In this example, the first input data 302 and the second input data 304 are received from an encoder device via the network 106 or a storage medium. In this example embodiment, the first input data is at a first quality level in the hierarchical structure, and the second input data 304 is at a second quality level in the hierarchical structure, and the second quality level is lower than the first quality level. Therefore, the first input data 302 and the second input data 304 are received from an encoder device via the network 106 or a storage medium. Figure 3 The second input data 304 corresponds to the higher level in Figure 3 corresponds to the lower level in .

[0064] The decoder device 114 may reconstruct the signal at the first quality level using the first input data 302. When the decoder device 114 uses the second input data 304 without the first input data 202, the decoder device 114 may use the second input data 304 to reconstruct the signal at the second quality level.

[0065] Figure 3 An example of a decoding scheme with two quality levels is shown. However, the concepts disclosed in this application are also applicable to alternative schemes with a single quality level or more than two quality levels.

[0066] When decoding frame n, the decoder device 114 will retrieve the transform element set (frame n-1) 206 stored in the frame buffer 116, such as Figure 2 In addition, Figure 2As described above, the transform element set (frame n-1) 206 is subjected to a lossless compression operation 210 before being stored in the frame buffer 116. Figure 2 In the example shown in FIG. 1 , the frame buffer 116 is located outside the hardware module where the decoder device 114 is located.

[0067] Retrieving the transform element set (frame n-1) 206 from the frame buffer 116 also includes performing an inverse lossless compression technique 214 (eg, Figure 2 ). The inverse lossless compression technique is the reverse of the lossless compression technique originally used to store the frame n-1 transform elements into the frame buffer, or otherwise allows the set of transform elements to be recovered uncompressed.

[0068] Using data (i.e., a set of transform elements) that indicates the degree of spatial correlation of corresponding frame data, for example, the correlation of frame data corresponding to frame n-1, a relatively sparse data plane is generated compared to the previous frame data itself. When the relatively sparse data is compressed using lossless compression techniques, a very high compression ratio can be achieved. In this way, the decoder device 114 can store and retrieve the contents of the frame buffer 116 relatively quickly. Therefore, unnecessary delays or interruptions in real-time decoding are less likely to occur. In addition, the use of lossless compression techniques can reduce artifacts in the stored data.

[0069] In this example embodiment, the lossless compression technique includes two different lossless compression techniques. Alternatively, the lossless compression technique includes at least one of run-length encoding and Huffman encoding. Alternatively, the lossless compression technique includes run-length encoding followed by Huffman encoding, or Huffman encoding followed by run-length encoding.

[0070] about Figure 3 In more detail of the example, the first input data 302 represents a relationship between the transform element set (frame n-1) 206 and the transform element set (frame n) 208, such that when the decoder device 114 combines the transform element set (frame n-1) 206 with the first input data 302, the transform element set (frame n) 208 is generated. The generated transform element set (frame n) 208 Follow Figure 2 , to cover the transform element set (frame n-1) 206.

[0071] In this example embodiment, the first input data 302 indicates the degree of temporal correlation between the transform element set (frame n−1) 206 and the frame n transform elements 308 .

[0072] In this example embodiment, the first input data 312 includes a quantized version of the difference result between the transform element set (frame n−1) 206 and the frame n transform elements 308 .

[0073] In the inverse transform module 310, the transform element set (frame n) 308 is subjected to an inverse transform operation, such as a direct discrete inverse transform. Residual data 312 is generated and used to create a reconstructed frame 316 .

[0074] In this example embodiment, the residual data 312 is based on the difference between a first presentation of frame n at a first quality level and a second presentation of frame n at the first quality level in a hierarchical structure having a plurality of quality levels.

[0075] The second input data 304 is upsampled to generate upsampled second input data 314. The residual data 312 and the upsampled second input data 314 are combined to generate a reconstructed frame 316. Such a process of upsampling data and combining the data with the residual data is generally known to the skilled person, for example, see WO2018046940A1. In this example embodiment, the upsampling operation on the second input data 304 generates a second presentation of the first quality level of the video signal frame n.

[0076] It is obvious to the reader that Figure 3 The transform element sets in represent residual data having residual elements for modifying the first input data 302. The residual elements are based on differences between a first presentation of a particular frame at a first quality level and a second presentation of the particular frame at the first quality level in a hierarchical structure having a plurality of quality levels. The transform element sets indicate a degree of spatial correlation between residual element sets of the residual data, such that the transform element sets indicate at least one of an average, horizontal, vertical, and diagonal (AHVD) relationship between adjacent residual elements, or any combination thereof.

[0077] Thus, by using data related to the AHVD relationship between adjacent residual elements in the transform element set (frame n-1) 206, higher compressibility can be achieved, because this data is sparse and can be compressed significantly. AVHD data can be processed in parallel, allowing for rapid compression and faster reading and writing of data to memory.

[0078] In this example implementation, transform element set (frame n-1) 306 is associated with the signal element array of frame n-1, transform element set (frame n) 308 is associated with the signal element array of frame n, and the spatial position of transform element set (frame n) 308 is the same as the signal element array of frame n-1.

[0079] Figure 3 An example is the LCEVC implementation of the inventive concept. Alternatively, in scalable video coding (such as LCEVC) In other cases, the transform element set does not necessarily need to represent residual data. Instead, the transform element set can represent the main frame or original frame data of the accurate or true original signal.

[0080] Figure 4 Flowchart of a method for storing and retrieving a frame buffer according to an embodiment of the present invention. In step 402, The method includes compressing a set of transform elements using lossless compression, wherein the set of transform elements indicates a degree of spatial correlation in the first frame of data. At step 404, the method includes storing the compressed set of transform elements in an external memory. At step 406, the method includes receiving a second frame of data. At step 408, the method includes retrieving the compressed set of transform elements from the external memory.

[0081] Figure 4 Not shown in Figure 2 and Figure 3 All features discussed in can optionally be added to Figure 4 in the method steps.

[0082] Figure 5 A timing diagram to describe the partial serial enhancement architecture timing process for decoding an input data stream, in this example, the input data stream is a base and LCEVC bitstream comprising a base layer and one or more LCEVC enhancement layers of frame image information, which represent frames of image information. Figure 5 A decoding timing flow with three cycles and three frames is shown, namely, frame X, frame X+1, and frame X+2. The first cycle includes time blocks 502, 504, and 506, which are used to decode the base layer of frame X, and also for decoding any enhancement layers of frame X-1 (not shown). The second cycle includes time blocks 508, 510, and 512, which are used to decode the base layer of frame X+1, and also for decoding any enhancement layers of frame X. The third cycle includes time blocks 514, 516, and 518, which are used to decode the base layer of frame X+2, and also for decoding any enhancement layers of frame X+1 (not shown). These three cycles will produce decoded frames for frame X-1, frame X, and frame X+1, and produce a basic decoded version of frame X+2.

[0083] The completion time of each cycle is 1 / fps, where fps is the frame rate in frames per second.

[0084] In more detail for the first cycle, at block 502, the base and LCEVC bitstreams corresponding to frame X are read. At the second time fast 504, the base layer of frame X is decoded to create a base decoded frame X. At block 506, the HW- DMA (Hardware Direct Memory Access) writes the base decoded frame X to memory. In this example, the resolution of the frame in the base layer is one quarter of that in the LCEVC enhancement layer, but the frame rate is the same as the corresponding frame in the LCEVC enhancement layer.

[0085] In the second cycle, the above process is repeated for frame X+1. Specifically, in block 508, the base layer is read from the input data stream corresponding to frame X+1. In block 510, the base layer of frame X+1 is decoded. In block 512, HW- DMA writes basic decoded frame X+1 to memory.

[0086] In the third cycle, the above process is repeated for frame X+2. Specifically, in block 514, the base layer is read from the input data corresponding to frame X+2. In block 516, the base layer of frame X+2 is decoded. In block 518, HW- DMA writes basic decoded frame X+2 to memory.

[0087] In all three cycles described above, during block 502 and equivalent blocks 508 and 514, the LCEVC enhancement layer data required for each cycle is read as described in the following paragraphs. For example, block 508 reads the enhancement layer data for frame X and block 514 reads the enhancement layer data for frame X-1.

[0088] The second cycle shows how each cycle works in more detail. While blocks 508 and 510 perform the basic decoding operations for frame X+1, the necessary enhancement data in the LCEVC enhancement layer is decoded and applied in parallel to the low-quality data to create a fully decoded frame X. At block 520, the HW-DMA reads the basic reconstruction of frame X stored in the first cycle of block 506 from the memory. At block 522, the basic reconstruction of frame X is upsampled to the first quality level LoQ1. At block 524, the LCEVC enhancement layer data of the first quality level LoQ1 of frame X is decoded and applied to the basic reconstruction of frame X to generate a reconstruction of the first quality level LoQ1. At block 526, the reconstruction of frame X at the first quality level LoQ1 is written to the memory using HW-DMA. Blocks 520, 522, 524, and 526 operate in parallel, so once information related to the processable data elements or pixel blocks in frame X is obtained, the next block begins to perform its operations, while the current block continues to operate on the remaining processable data elements or pixel blocks.

[0089] After processing is completed at block 526, a new set of parallel blocks begins at blocks 528 and 530. Specifically, in block 528, HW-DMA reads the reconstruction of frame X at the first quality level LoQ1 from the memory. In block 530, HW-DMA reads the timing preset data from the memory, which is used to save the record of the previous enhancement data, thereby reducing the amount of signal data in the enhancement layer. In execution block 532, frame X at the first quality level LoQ1 is upsampled and reconstructed to generate a version of frame X at the second quality level LoQ0. In execution block 534, the LCEVC enhancement layer data of frame X at the LoQ0 level is decoded and applied to the version of frame X at the second quality level LoQ0 to generate a reconstruction of the second quality level. In block 536, using HW-DMA, the temporal prediction data is written to the memory for reuse in the next cycle. In block 538, using HW-DMA, the reconstructed frame X at the quality level LoQ0 is written to the memory as part of the decoded video output of the decoding process. At this point, the decoded output of frame X of the decoding process can be streamed out. At block 540, the frame X with the reconstruction level of quality LoQ0 is written to the host memory. This step 540 may or may not be included in the system design, depending on the specific requirements of the application. The video data may be stored in a double data rate (DDR) memory.

[0090] from Figure 5 As can be seen in FIG. 5 , in this example, the time between reading the base reconstruction of frame X from memory at block 520 and writing frame X at a reconstruction level of quality LoQ0 at block 538 is 3.51 milliseconds. Thus, it takes 3.51 milliseconds from the time the base reconstruction of frame X is read until frame X is available for streaming output. This time is at least partially due to the fact that block 528 does not begin processing data until block 526 is complete, i.e., writing the reconstruction of frame X at the first quality level LoQ1 to memory is complete. Thus, Figure 5 The process is called the partial serial enhanced architecture timing process.

[0091] Figure 6 The following is a timing diagram that describes the parallel architecture timing process of decoding input data. For ease of reference, Figure 5 and Figure 6 The same reference numerals in the drawings represent the same blocks and are not described again herein.

[0092] Figure 6 Blocks 526 and 528 are not used in . Therefore, the time from reading the basic reconstruction of frame X from the memory at block 520 to writing the video data at block 538 is reduced to 1.76 milliseconds, which is Figure 5 So, from the time we start reading the basic reconstruction of frame X to the time we stream frame X out, it takes 1.76 milliseconds.

[0093] Figure 6 Also shown is the process of preparing the frame X+1 stream output. Figure 6 Blocks 620-640 are shown, which are the same as blocks 520-540, but in the context of frame X+1.

[0094] By using HW-DMA writes, the encoder or decoder is allowed to bypass the CPU, thereby speeding up the encoding or decoding of video data for improved performance.

[0095] The techniques described herein can be implemented in software or hardware, or in a combination of software and hardware. Software can be a computer program consisting of instructions that, when executed by a device, perform the techniques described herein.

[0096] The above embodiments are to be understood as examples. Further embodiments are contemplated. It should be understood that any feature associated with any one embodiment may be used alone or in combination with the other features described, or in combination with one or more features of any other embodiment, or in combination with any combination of any other embodiment. In addition, equivalents and modifications not described above may also be employed without departing from the scope of the invention.

Claims

1. A method for using a frame buffer in a decoding process, wherein: The method is performed on a dedicated hardware circuit, and the method comprises: using a frame buffer to store data representing the first frame of data, wherein the data representing the first frame of data is used in processing the second frame of data; in: The frame buffer is stored in a memory external to the dedicated hardware circuit; The data representing the first frame of data is a set of transform elements indicating a degree of spatial correlation in the first frame of data; The method compresses the transform element set using a lossless compression technique, and sends the compressed transform element set to the frame buffer to be retrieved when processing the second frame data.

2. The method according to claim 1, wherein: Fetching the transform element set from the frame buffer includes: performing an inverse lossless compression technique on the compressed transform element set.

3. The method according to claim 2, wherein: The first frame data includes a first residual element set.

4. The method according to claim 3, wherein: The first set of residual elements is based on differences between a first representation of a first frame associated with the first frame data at a first quality level in a hierarchical hierarchy having a plurality of quality levels and a second representation of the first frame at the first quality level.

5. The method according to claim 4, wherein: The transform element set indicates a degree of spatial correlation between the first residual element sets, so that the transform element set indicates at least one of an average, horizontal, vertical and diagonal relationship between adjacent residual elements in the residual element set.

6. The method according to claim 4 or 5, wherein: The method comprises receiving first input data, wherein the first input data indicates a degree of temporal correlation between the set of transformed elements and a second set of transformed elements.

7. The method according to claim 6, wherein: The second transform element set indicates a degree of spatial correlation of the second residual element set.

8. The method according to claim 7, wherein: The second set of residual elements is used to reconstruct a presentation of the second frame associated with the second frame data at the first quality level by using data based on the presentation of the second frame at the second quality level.

9. The method according to claim 7 or 8, wherein: The second set of residual elements is based on differences between a first representation of the second frame at the first quality level and a second representation of the second frame at the first quality level in a hierarchical hierarchy having a plurality of quality levels.

10. The method according to any one of claims 7 to 9, wherein: The second transform element set indicates a degree of spatial correlation between multiple residual elements in the second residual element set associated with the second frame, so that the second transform element set indicates at least one of an average, horizontal, vertical and diagonal relationship between adjacent residual elements in the second residual element set.

11. The method according to claims 6-10, wherein: The method comprises combining the first input data with the set of transformed elements to generate the second set of transformed elements.

12. The method according to claim 11, wherein: The method includes performing an inverse transform operation on the second set of transformed elements to generate the second set of residual elements.

13. The method according to claim 12, wherein: The method includes receiving second input data, wherein the second input data is at a second quality level in the hierarchical hierarchy, the second quality level being lower than the first quality level.

14. The method according to claim 13, wherein: The method includes performing an upsampling operation on the second input data to generate a second representation of the second frame at the first quality level.

15. The method according to claim 14, wherein: The method includes combining a second representation of the second frame with the second set of residual elements to reconstruct the second frame.

16. The method according to any one of claims 6 to 15, wherein: The first input data includes a quantized version of a difference result between the set of transformed elements and the second set of transformed elements.

17. The method according to any one of claims 6 to 16, wherein: The set of transform elements is associated with an array of signal elements in the first frame, and the second set of transform elements is associated with an array of signal elements in the second frame at the same spatial locations as the array of signal elements in the first frame.

18. A method according to any preceding claim, wherein: The lossless compression technology includes two different lossless compression technologies.

19. The method according to any one of claims 1 to 17, wherein: The lossless compression technique includes at least one of run-length coding and Huffman coding.

20. The method according to any one of claims 1 to 17, wherein: The lossless compression technique includes run-length encoding followed by Huffman encoding.

21. A method according to any one of the preceding claims, wherein: The decoding process is configured to decode a video signal.

22. The method according to claim 21, wherein: The video signal is at least an 8K, 60FPS video signal.

23. A decoder device implemented on a dedicated hardware circuit, wherein: The decoder device comprises a data communication link for communicating with an external memory, wherein the decoder device is configured to perform the method of any of the preceding claims.

24. A computer program comprising instructions which, when executed, cause the decoder device of claim 23 to perform the method of any one of claims 1-22.

Images