Caching picture for video coding

Abstract

A method includes generating a plurality of future reference picture lists associated with a plurality of future pictures in a picture set, where the picture set includes a current picture and the plurality of future pictures, and the plurality of future pictures follow the current picture in coding order; determining whether to write the current picture to the dedicated chip memory or to write the current picture to the non-dedicated system memory based on information derived from a plurality of future reference picture lists associated with the plurality of future pictures; and writing the current picture in the dedicated chip memory or the non-dedicated system memory based on a determination of whether to write the current picture in the dedicated chip memory or the non-dedicated system memory.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 337,109, filed June 19, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to video encoding and video decoding. Background Technology

[0003] Digital video capabilities can be incorporated into a wide variety of devices, including digital televisions, digital live broadcast systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones (so-called "smartphones"), video conferencing equipment, video streaming devices, and more. Digital video devices implement video decoding technologies, such as those defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 (Part 10, Advanced Video Decoding (AVC)), ITU-T H.265 / High Efficiency Video Decoding (HEVC), ITU-T H.266 / Variety Video Decoding (VVC) and extensions to these standards, as well as proprietary video codecs / formats such as AOMedia Video1 (AV1) developed by the Open Media Alliance. By implementing such video decoding technologies, video devices can more efficiently send, receive, encode, decode, and / or store digital video information.

[0004] Video decoding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video decoding, video slices (e.g., video pictures or portions of video pictures) can be divided into video blocks, which may also be referred to as decoding tree units (CTUs), decoding units (CUs), and / or decoding nodes. Video blocks in a slice after intra-frame decoding (I) of a picture are encoded using spatial prediction relative to reference samples in adjacent blocks within the same picture. Video blocks in a slice after inter-frame decoding (P or B) of a picture can use spatial prediction relative to reference samples in adjacent blocks within the same picture or temporal prediction relative to reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame. Summary of the Invention

[0005] Generally, this disclosure describes techniques for determining whether to write the current image to dedicated on-chip memory (e.g., cache) or non-dedicated system memory (e.g., random access memory (RAM)). A video encoder or decoder may generate a list of future reference images associated with future images (e.g., images following the current image in decoding order), and based on information derived from the list of future reference images, the video encoder or decoder may determine whether to write the current image to dedicated on-chip memory or non-dedicated system memory. If the current image is used for inter-frame prediction of another image, the video encoder and decoder may access the current image from either dedicated on-chip memory or non-dedicated system memory.

[0006] Accessing images from dedicated-chip memory may require less bandwidth and power compared to accessing images from non-dedicated system memory. By utilizing information derived from a list of future reference images, a video encoder or decoder can determine an estimate of the frequency or likelihood that a current image will be accessed in the future as a reference image for inter-frame prediction of another image. By determining which images are estimated to be accessed more frequently as reference images and writing them into dedicated-chip memory, the example technique can reduce power and bandwidth consumption due to more frequent access to dedicated-chip memory and reduced access to non-dedicated system memory.

[0007] In one example, this disclosure describes a method for processing video data, the method comprising: generating a list of multiple future reference images associated with a plurality of future images in a set of images, wherein the set of images includes a current image and the plurality of future images, and the plurality of future images are arranged in a decoding order following the current image; determining, based on information derived from the list of multiple future reference images associated with the plurality of future images, whether to write the current image to a dedicated chip memory or to a non-dedicated system memory; and writing the current image to either the dedicated chip memory or the non-dedicated system memory based on the determination of whether to write the current image to the dedicated chip memory or to the non-dedicated system memory.

[0008] In one example, this disclosure describes an apparatus for processing video data, the apparatus comprising: a dedicated-chip memory; and processing circuitry coupled to the dedicated-chip memory and configured to: generate a list of multiple future reference images associated with a plurality of future images in a set of images, wherein the set of images includes a current image and the plurality of future images, and the plurality of future images are arranged in a decoding order following the current image; determine, based on information derived from the list of multiple future reference images associated with the plurality of future images, whether to write the current image to the dedicated-chip memory or to a non-dedicated system memory; and write the current image to the dedicated-chip memory or the non-dedicated system memory based on the determination of whether to write the current image to the dedicated-chip memory or to the non-dedicated system memory.

[0009] In one example, this disclosure describes one or more computer-readable storage media storing instructions that, when executed, cause one or more processors to: generate a list of multiple future reference images associated with a plurality of future images in a set of images, wherein the set of images includes a current image and the plurality of future images, and the plurality of future images are arranged in a decoding order following the current image; determine, based on information derived from the list of multiple future reference images associated with the plurality of future images, whether to write the current image to dedicated chip memory or to non-dedicated system memory; and write the current image to dedicated chip memory or non-dedicated system memory based on the determination of whether to write the current image to dedicated chip memory or to non-dedicated system memory.

[0010] In one example, this disclosure describes an apparatus for processing video data, the apparatus comprising: means for generating a list of multiple future reference images associated with a plurality of future images in a set of images, wherein the set of images includes a current image and the plurality of future images, and the plurality of future images are arranged in a decoding order following the current image; means for determining, based on information derived from the list of multiple future reference images associated with the plurality of future images, whether to write the current image into a dedicated chip memory or into a non-dedicated system memory; and means for writing the current image into the dedicated chip memory or the non-dedicated system memory based on the determination of whether to write the current image into the dedicated chip memory or into the non-dedicated system memory.

[0011] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating an example video encoding and decoding system that can perform the techniques of this disclosure.

[0013] Figure 2 This is a block diagram illustrating an example video encoder that can perform the techniques of this disclosure.

[0014] Figure 3 This is a block diagram illustrating an example video decoder that can perform the techniques of this disclosure.

[0015] Figure 4A and Figure 4B This is a conceptual diagram illustrating the dependencies of images used for inter-frame prediction.

[0016] Figure 5 This is a flowchart illustrating an example method for determining whether a current image is written to a dedicated chip memory or a non-dedicated system memory according to the technology of this disclosure.

[0017] Figure 6 This is a flowchart illustrating another example method for determining whether a current image is written to a dedicated chip memory or a non-dedicated system memory, according to the technology of this disclosure.

[0018] Figure 7 This is a flowchart illustrating another example method for determining whether a current image is written to a dedicated chip memory or a non-dedicated system memory, according to the technology of this disclosure.

[0019] Figure 8 This is a flowchart illustrating another example method for determining whether a current image is written to a dedicated chip memory or a non-dedicated system memory, according to the technology of this disclosure.

[0020] Figure 9 This is a flowchart illustrating another example method for determining whether a current image is written to a dedicated chip memory or a non-dedicated system memory, according to the technology of this disclosure.

[0021] Figure 10 This is a flowchart illustrating another example method for determining whether a current image is written to a dedicated chip memory or a non-dedicated system memory, according to the technology of this disclosure.

[0022] Figure 11 This is a flowchart illustrating another example method for determining whether a current image is written to a dedicated chip memory or a non-dedicated system memory, according to the technology of this disclosure. Detailed Implementation

[0023] Inter-frame prediction is an example video coding or decoding technique used to encode or decode a current block in a current image. In inter-frame prediction, a video encoder or video decoder determines a reference block in a reference image. The video encoder or video decoder can generate a prediction block from the reference block (e.g., through filtering, etc.). In some examples, the prediction block and the reference block are the same block. The video encoder determines residual information indicating the difference between samples of the prediction block and the current block, and signals the residual information. The video decoder receives the residual information and adds it to the prediction block to reconstruct the current block. The video decoder can perform a similar operation to reconstruct the current image.

[0024] The video encoder and decoder write the current image to memory. The current image can then serve as a reference image for encoding or decoding one or more future images. A future image is an image that follows the current image in the decoding order (i.e., is encoded or decoded after the current image).

[0025] For a video encoder or decoder, there may be different types of memory available for writing the current image. For example, an integrated circuit (IC) chip that includes a video encoder and / or decoder may have dedicated chip memory, which can be on-chip or off-chip and accessible via a dedicated bus. Dedicated chip memory may be shared by components of the IC chip, but may not be accessible by other chips. An example of dedicated chip memory is cache memory. Dedicated chip memory is an example of the memory that a video encoder or decoder can write the current image to.

[0026] Furthermore, systems that include IC chips may include non-dedicated system memory. For example, an IC chip may be part of a larger system (e.g., a phone, tablet, laptop, etc.) and may have system memory shared by system components that does not require dedicated space for the IC chip. Random access memory (RAM) is an example of non-dedicated system memory. Another example of non-dedicated system memory is the memory that a video encoder or decoder can write the current image to.

[0027] Generally speaking, writing and reading from dedicated-chip memory requires less power and access time compared to writing and reading from non-dedicated system memory. Therefore, writing the current image to dedicated-chip memory can be beneficial, allowing for faster and lower power retrieval of the current image if it is used as a reference image, compared to storing the current image in non-dedicated system memory. However, dedicated-chip memory may have limitations in size, and it is typically smaller than non-dedicated system memory. The size of dedicated-chip memory may not be sufficient to store a large number of images. In some cases, dedicated-chip memory may be able to store only one image.

[0028] This disclosure describes example techniques for determining whether to write the current image to dedicated chip memory or non-dedicated system memory. For example, a video encoder and a video decoder may determine metrics indicating the frequency at which the current image is predicted to be used as a reference image for future images and the frequency at which cached images (i.e., images already stored in dedicated chip memory) are predicted to be used as reference images for future images. Based on the determined metrics, the video encoder and the video decoder may determine whether to write the current image to dedicated chip memory or non-dedicated system memory.

[0029] In one or more examples, to determine a metric of how frequently an image is predicted to be used as a reference image, the video encoder and video decoder may determine a list of reference images for future images. The list of reference images includes a list of candidate reference images constructed by the video encoder and video decoder using similar techniques. One or more lists of reference images may exist. In some examples, the video encoder and video decoder may each construct two lists of reference images, referred to as list 0 and list 1. For ease of description only, this disclosure describes an example technique with list 0 and list 1. However, the example technique is not limited to having two lists of reference images. The example technique is applicable to having one or more lists of reference images.

[0030] The video encoder signals the index value of one or both of lists 0 and 1 to identify a reference picture (e.g., for unidirectional prediction) or two reference pictures (e.g., for bidirectional prediction). The video decoder receives the index value to determine the reference picture for the current picture.

[0031] The construction of the reference image list may not occur until the current image is encoded or decoded. In other words, while the video decoder is reconstructing the current image, it may not have information from the video encoder about how to construct the reference image list following the current image in decoding order.

[0032] In one or more examples, the video encoder and video decoder can generate a list of multiple future reference images associated with multiple future images in a set of images. This set of images includes the current image and multiple future images, which are ordered after the current image in the decoding sequence.

[0033] As an example, the image set could be a second image set. The video decoder can generate a list of multiple future reference images associated with multiple future images based on a first image set decoded prior to the second image set. The second image set includes the current image and multiple future images, which are decoded after the current image in the order they were decoded. Both the first and second image sets can include a predefined number of images.

[0034] As an example, the first image set may be referred to as the first group of images (GOP), and the second image set may be referred to as the second GOP. As another example, the first and second image sets may be defined such that their reference images are images within the respective image sets. For example, the reference image for an image in the second image set must be within the second image set. As yet another example, the first instantaneous decoder refresh (IDR) image and the second IDR image may define the end of the first and second image sets, and the first image set may be the first half of the images between the first and second IDR images, and the second image set may be the second half of the images between the first and second IDR images.

[0035] In some examples, there is a relatively high probability that there are certain similarities between the list of reference images for images in the first image set and the list of reference images for images in the second image set. For example, there is a possibility that a video sequence has a repeating set of images with a fixed period. As an example, the relative distance between an image in the first image set and a corresponding image in the list of reference images for that image in the first image set may be the same as the distance between a co-located image in the second image set and a corresponding image in the list of reference images for that image in the second image set.

[0036] As an example, suppose the first entry in list 0 of the sixth image in the first image set, ordered by decoding, identifies the image that is two images earlier in the first image set, ordered by decoding. In this example, the first entry in list 0 of the sixth image in the second image set, ordered by decoding, might identify the image that is two images earlier in the second image set, ordered by decoding. It is possible, but less likely, that the first entry in list 0 of the sixth image in the second image set is not the image that is two images earlier in the second image set, ordered by decoding. This similarity can be applied to other entries in list 0 and list 1 of the sixth image, as well as to other images in the first and second image sets.

[0037] In this example, assume the current image is the second image in the second image set. Even if the sixth image in the second image set has not yet been decoded (i.e., the sixth image is a future image), the video decoder can still generate a reference image list for the sixth image in the second image set based on the reference image list for the sixth image in the first image set. Because the video decoder has already decoded the images in the first image set, the reference image list for the sixth image in the first image set is available to the video decoder.

[0038] Therefore, in one or more examples, to generate multiple lists of future reference images, the video decoder may determine a corresponding previous image in a first image set (e.g., the sixth image in the first image set in the example above), which is located in the first image set at the same relative position as the multiple future images in a second image set (e.g., the sixth image in the second image set in the example above). The video decoder may determine a corresponding list of reference images for the corresponding previous image (e.g., the first entry of list 0 of the sixth image in the first image set in the example above). The video decoder may generate corresponding multiple lists of future reference images for each of the multiple future images based on the determined corresponding list of reference images for the corresponding previous image (e.g., generating list 0 of the sixth image in the second image set based on list 0 of the sixth image in the first image set).

[0039] A video encoder can use techniques similar to those described above for a video decoder to generate a list of multiple future reference images associated with multiple future images. However, because the video encoder has the task of determining which images are included in the list of reference images, it may not need to generate a list of multiple future reference images in a second set of images based on a list of reference images from previous images in a first set of images. In other words, the video encoder can use techniques similar to or different from those used by the video decoder to generate a list of multiple future reference images associated with multiple future images in a set of images.

[0040] From the perspective of the video decoder, the list of future reference images can be a prediction of the actual list of future reference images, because it is possible that the actual list of future reference images will ultimately differ from the list determined by the video decoder. From the perspective of the video encoder, the list of future reference images can be determined and is known, but is not necessarily a prediction. Therefore, in one or more examples, multiple lists of future reference images can be (e.g., for the video decoder) predicted or (e.g., for the video encoder) determined lists of future reference images whose identifiers can be used as reference images for multiple future images.

[0041] Using a list of future reference images, the video encoder and decoder can derive information (e.g., one or more metrics) from the list. For example, the video encoder and decoder can determine which images are identified at different entries in lists 0 and 1 of future images. In some examples, the video encoder and decoder can determine which reference images are identified in the first entry (e.g., index 0) and the second entry (e.g., index 1) of lists 0 and 1 of future images. In one or more examples, the reference images identified in the first entry of lists 0 and 1 are most likely to be used as reference images, and the reference images identified in the second entry of lists 0 and 1 are less likely to be used as reference images.

[0042] As an example, a video encoder and decoder can determine different metrics based on the number of times a specific reference image is identified in the first entry of list 0 of future images, the first entry of list 1 of future images, the second entry of list 0 of future images, and the second entry of list 1 of future images. The video encoder and decoder can use these metrics to determine whether to write the current image to dedicated on-chip memory or non-dedicated system memory.

[0043] For example, a video encoder and video decoder can determine a first metric (referred to as a first reference gain value) for the current image and a second metric (referred to as a second reference gain value) for the cached image (i.e., the image currently stored in dedicated chip memory). If the first reference gain value is greater than the second reference gain value, the video encoder and video decoder can clear the cached image from the dedicated chip memory, write the current image to the dedicated chip memory, and write the cached image to non-dedicated system memory. If the first reference gain value is less than the second reference gain value, the video encoder and video decoder can write the current image to non-dedicated system memory. Generally, the reference gain value can be a prediction of how often the current image or the cached image will be used as a reference image.

[0044] For ease of description, the above example describes the use of reference images identified in the first and second entries of lists 0 and 1 of future images. However, the example technique is not limited to this. In some examples, the video encoder and video decoder may determine the reference gain value based solely on list 0 or list 1 and / or only one entry from list 0 or list 1. In some examples, the video encoder and video decoder may determine the reference gain value based on more entries than the first and second entries, or based on a larger list of reference images.

[0045] The number of entries in the list of future reference images being evaluated and / or the total number of future reference images can be based on a balance of complexity. For example, evaluating more entries than the first and second entries in lists 0 and 1 of future images may provide a more accurate measure of the reference effectiveness compared to evaluating only the first and second entries in lists 0 and 1 of future images. However, evaluating more entries than the first and second entries in lists 0 and 1 of future images may require additional processing time. It may be beneficial to determine whether to write the current image to dedicated chip memory or non-dedicated system memory based on the list of future reference images associated with the future image, and the number of entries in the list of future reference images being evaluated can be based on a desired level of complexity, and examples are consistent with the techniques described in this disclosure.

[0046] Furthermore, the above example techniques are described with respect to a dedicated chip memory storing a single image. However, the example techniques are not limited to this. In some examples, the dedicated chip memory can store multiple images. In such examples, the video encoder and video decoder can compare a reference gain value of the current image with reference gain values ​​of one or more cached images. If the reference gain value of the current image is greater than the reference gain value of the cached image with the minimum reference gain value, the video encoder and video decoder can remove the cached image with the minimum reference gain value from the dedicated chip memory, write the current image into the dedicated chip memory, and write the cached image into non-dedicated system memory.

[0047] Some other techniques rely on evaluating motion vectors from a future image to determine whether to write the current image to dedicated on-chip memory or non-dedicated system memory. The motion vectors point to the locations in a reference image from which prediction blocks are generated. In these other techniques, the video decoder receives information about the motion vectors of the future image before decoding it. The video decoder can then parse the motion vector information to determine whether to write the image to dedicated on-chip memory or non-dedicated system memory before decoding the future image.

[0048] However, the availability of motion vector information for future images cannot be guaranteed. Furthermore, a video decoder may need to expend resources parsing information from the bitstream regarding future images that are useless to the current image. Using the example techniques described in this disclosure, a video decoder can be able to determine a list of future reference images based on already decoded images, and the video decoder does not need to rely on parsing potentially unusable information or expend resources parsing unnecessary information.

[0049] Several other techniques also rely on a reference image set (RPS). A reference image set differs from a reference image list. A reference image set includes information about which images may be part of the reference image list, but not all images in the reference image set may necessarily be part of the reference image list. These other techniques that rely on a reference image set can be flawed because they may determine that an image is written to dedicated chip memory, but the probability that that image is actually used as a reference image is low. Utilizing the example techniques described in this disclosure, by relying on a future reference image list as a substitute for or supplement to the reference image set, and determining information (e.g., various metrics) based on the future reference image list, the image written to dedicated chip memory may be more likely to be accessed more frequently than other images.

[0050] Figure 1 This is a block diagram illustrating an example video encoding and decoding system 100 that can perform the techniques of this disclosure. The techniques of this disclosure generally involve decoding (encoding and / or decoding) video data. Typically, video data includes any data used for processing video. Therefore, video data may include unencoded raw video, encoded video, decoded (e.g., reconstructed) video, and video metadata, such as signaling data.

[0051] like Figure 1 As shown, in this example, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. Specifically, source device 102 provides the video data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can be or may include any of a wide range of devices, such as desktop computers, laptop computers, mobile devices, tablet computers, set-top boxes, mobile phones such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and destination device 116 may be configured for wireless communication and are therefore referred to as wireless communication devices.

[0052] exist Figure 1In the example, source device 102 includes a video source 104, memory 106, video encoder 200, and output interface 108. Destination device 116 includes an input interface 122, video decoder 300, memory 120, and display device 118. According to this disclosure, the video encoder 200 of source device 102 and the video decoder 300 of destination device 116 can be configured to apply techniques for determining whether to write images to dedicated chip memory or non-dedicated system memory. Thus, source device 102 represents an example of a video encoding device, while destination device 116 represents an example of a video decoding device. In other examples, the source device and destination device may include other components or arrangements. For example, source device 102 may receive video data from an external video source such as an external camera. Similarly, destination device 116 may interface with an external display device instead of including an integrated display device.

[0053] like Figure 1 The system 100 shown is merely an example. Generally, any digital video encoding and / or decoding device can perform techniques for determining whether to write an image to dedicated on-chip memory or non-dedicated system memory. Source device 102 and destination device 116 are merely examples of such decoding devices, where source device 102 generates decoded video data for transmission to destination device 116. This disclosure refers to a “decoding” device as a device that performs the decoding (e.g., encoding and / or decoding) of data. Thus, video encoder 200 and video decoder 300 represent examples of decoding devices, specifically, video encoder and video decoder, respectively. In some examples, source device 102 and destination device 116 may operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes video encoding and decoding components. Therefore, system 100 may support one-way or two-way video transmission between source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0054] Typically, video source 104 represents the source of video data (i.e., unencoded raw video data) and provides a sequential series of pictures (also referred to as "frames") of the video data to video encoder 200, which encodes the data for the pictures. Video source 104 of source device 102 may include video capture devices such as cameras, video archives containing previously captured raw video, and / or video feed interfaces for receiving video from video content providers. Alternatively, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes the captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the pictures from the received order (sometimes referred to as "display order") to a decoding order for decoding. Video encoder 200 may generate a bitstream comprising the encoded video data. Then, the source device 102 can output the encoded video data to the computer-readable medium 110 via the output interface 108 for reception and / or retrieval by, for example, the input interface 122 of the destination device 116.

[0055] The memory 106 of source device 102 and the memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw video data, such as raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106 and 120 may store software instructions executable by, for example, video encoder 200 and video decoder 300. Although memories 106 and 120 are shown separately from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store encoded video data, such as output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106 and 120 may be allocated as one or more video buffers, for example, to store raw decoded and / or encoded video data.

[0056] Computer-readable medium 110 may represent any type of medium or device capable of transmitting encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to directly transmit encoded video data to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. According to a communication standard such as a wireless communication protocol, output interface 108 may modulate the transmitted signal including the encoded video data, and input interface 122 may demodulate the received transmitted signal. The communication medium may include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network (such as the Internet). The communication medium may include a router, switch, base station, or any other equipment that may be useful for facilitating communication from source device 102 to destination device 116.

[0057] In some examples, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any data storage medium of various distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0058] In some examples, source device 102 can output encoded video data to file server 114 or another intermediate storage device that can store the encoded video data generated by source device 102. Destination device 116 can access the stored video data from file server 114 via streaming or download.

[0059] File server 114 can be any type of server device capable of storing encoded video data and sending the encoded video data to destination device 116. File server 114 may represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or FLUTE-based file delivery protocol), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a Network Attached Storage (NAS) device. File server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as HTTP-based Dynamic Adaptive Streaming (DASH), HTTP Live Streaming (HLS), Real-Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.

[0060] Destination device 116 can access encoded video data from file server 114 via any standard data connection, including an internet connection. This may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on file server 114. Input interface 122 can be configured to operate according to any or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0061] Output interface 108 and input interface 122 can represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 can be configured to transmit data such as encoded video data according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc. In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 can be configured according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee), etc. ™ ),Bluetooth ™Standards are used to transmit data such as encoded video data. In some examples, source device 102 and / or destination device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include an SoC device for performing functions belonging to video encoder 200 and / or output interface 108, and destination device 116 may include an SoC device for performing functions belonging to video decoder 300 and / or input interface 122.

[0062] The technology disclosed herein can be applied to video decoding to support any multimedia application in a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission (such as HTTP-based Dynamic Adaptive Streaming (DASH)), digital video encoded onto data storage media, decoding of digital video stored on data storage media, or other applications.

[0063] The input interface 122 of the destination device 116 receives an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200 and also used by the video decoder 300, such as syntax elements having values ​​describing the characteristics and / or processing of video blocks or other decoded units (e.g., slices, pictures, picture groups, sequences, etc.). The display device 118 displays decoded images of the decoded video data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0064] Despite Figure 1Not shown, but in some examples, both the video encoder 200 and the video decoder 300 may be integrated with the audio encoder and / or audio decoder (e.g., audio codec), and may include appropriate MUX-DEMUX units or other hardware and / or software to process multiplexed streams that include both audio and video in a common data stream. Example audio codecs may include AAC, AC-3, AC-4, ALAC, ALS, AMBE, AMR, AMR-WB (G.722.2), AMR-WB+, aptX (various versions), ATRAC, BroadVoice (BV16, BV32), CELT, Enhanced AC-3 (E-AC-3), EVS, FLAC, G.711, G.722, G.722.1, G.722.2 (AMR-WB), G.723.1, G.726, G.728, G.729, G.729.1, GSM-FR, HE-AAC, iLBC, iSAC, LA Lyra, Monkey's Audio, MP1, MP2 (MPEG-1, 2 Audio Layer II), MP3, Musepack, Nellymoser Asao, OptimFROG, Opus, Sac, Satin, SBC, SILK, Siren 7, Speex, SVOPC, True Audio (TTA), TwinVQ, USAC, Vorbis (Ogg), WavPack and Windows Media Aud.

[0065] Both the video encoder 200 and the video decoder 300 can be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the technology of this disclosure. Each of the video encoder 200 and the video decoder 300 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder / decoder (CODEC) in the respective device. Devices including the video encoder 200 and / or the video decoder 300 may implement the video encoder 200 and / or the video decoder 300 in processing circuitry such as integrated circuits and / or microprocessors. Such devices may be wireless communication devices (such as cellular phones) or any other type of device described herein.

[0066] The video encoder 200 and video decoder 300 may operate according to a video decoding standard such as ITU-T H.265 (also known as High Efficiency Video Decoding (HEVC)) or extensions thereof such as Multi-View and / or Scalable Video Decoding Extensions. Alternatively, the video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards such as ITU-T H.266 (also known as Multi-Functional Video Decoding (VVC)). In other examples, the video encoder 200 and video decoder 300 may operate according to proprietary video codecs / formats such as AOMedia Video 1 (AV1), extensions to AV1, and / or subsequent versions of AV1 (e.g., AV2)). In other examples, the video encoder 200 and video decoder 300 may operate according to other proprietary formats or industry standards. However, the techniques disclosed herein are not limited to any particular decoding standard or format. In general, the video encoder 200 and video decoder 300 may be configured to perform the techniques of this disclosure in conjunction with any video decoding technique using inter-frame predictive decoding.

[0067] Typically, video encoder 200 and video decoder 300 perform block-based decoding of images. The term "block" generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used during encoding and / or decoding). For example, a block may include a two-dimensional matrix of samples of luminance and / or chrominance data. Typically, video encoder 200 and video decoder 300 decode video data represented in YUV (e.g., Y, Cb, Cr) format. That is, instead of decoding the red, green, and blue (RGB) data used for images, video encoder 200 and video decoder 300 decode the luminance and chrominance components, where the chrominance components may include both red hue chrominance components and blue hue chrominance components. In some examples, video encoder 200 converts the received RGB format data to a YUV representation before encoding, and video decoder 300 converts the YUV representation to RGB format. Alternatively, preprocessing and postprocessing units (not shown) may perform these conversions.

[0068] This disclosure generally relates to the decoding (e.g., encoding and decoding) of images to include processes of encoding or decoding data of the image. Similarly, this disclosure may relate to the decoding of blocks of images to include processes of encoding or decoding data for the blocks (e.g., prediction and / or residual decoding). Encoded video bitstreams typically include a series of values ​​for syntax elements representing decoding decisions (e.g., decoding modes) and the partitioning of images into blocks. Therefore, references to the decoding of images or blocks should generally be understood as the decoded values ​​of the syntax elements that form the images or blocks.

[0069] HEVC defines various blocks, including decoding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video decoder (such as a video encoder 200) divides the decoding tree unit (CTU) into CUs based on a quadtree structure. That is, the video decoder divides the CTU and CU into four equal, non-overlapping squares, and each node of the quadtree has zero or four child nodes. Nodes without child nodes are called "leaf nodes," and the CU of such leaf nodes may include one or more PUs and / or one or more TUs. The video decoder may further divide the PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents the partitioning of the TU. In HEVC, the PU represents inter-frame prediction data, while the TU represents residual data. The CU after intra-frame prediction includes intra-frame prediction information, such as intra-frame mode indication.

[0070] As another example, video encoder 200 and video decoder 300 can be configured to operate according to VVC. According to VVC, the video decoder (such as video encoder 200) partitions the image into multiple CTUs. Video encoder 200 can partition the CTUs according to a tree structure (such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure removes the concept of multiple partition types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels: a first level partitioned according to quadtree partitioning, and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to CUs.

[0071] In the MTT partitioning structure, blocks can be divided using quadtree (QT) partitioning, binary tree (BT) partitioning, and one or more types of ternary tree (TT) partitioning (also known as triplet tree (TT)). A ternary tree or triplet tree partition is a partition in which a block is divided into three sub-blocks. In some examples, a ternary tree or triplet tree partition divides a block into three sub-blocks without dividing the original block through the center. Partition types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0072] When operating according to the AV1 codec, the video encoder 200 and video decoder 300 can be configured to decode video data in blocks. In AV1, the largest decoded block that can be processed is called a superblock. In AV1, a superblock can be 128x128 luma samples or 64x64 luma samples. However, in subsequent video decoding formats (e.g., AV2), superblocks can be defined by different (e.g., larger) luma sample sizes. In some examples, the superblock is the top level of a block quadtree. The video encoder 200 can further divide the superblock into smaller decoded blocks. The video encoder 200 can use square or non-square partitions to divide the superblock and other decoded blocks into smaller blocks. Non-square blocks can include N / 2xN blocks, NxN / 2 blocks, N / 4xN blocks, and NxN / 4 blocks. The video encoder 200 and video decoder 300 can perform separate prediction and transform processing for each decoded block.

[0073] AV1 also defines video data tiles. A tile is a rectangular array of superblocks that can be decoded independently of other tiles. That is, the video encoder 200 and video decoder 300 can encode and decode the decoding blocks within a tile separately without using video data from other tiles. However, the video encoder 200 and video decoder 300 can perform filtering across tile boundaries. The tile size can be uniform or non-uniform. Tile-based decoding enables parallel processing and / or multithreading in the encoder and decoder implementations.

[0074] In some examples, the video encoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video encoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luma component and another QTBT / MTT structure for the two chroma components (or two QTBT / MTT structures for the respective chroma components).

[0075] The video encoder 200 and the video decoder 300 can be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning or other partitioning structures.

[0076] In some examples, a CTU includes a decoded tree block (CTB) of luminance samples, two corresponding CTBs of chrominance samples of an image with three sample arrays, or a CTB of samples of an image decoded using three separate color planes and a syntax structure for decoding the samples. A CTB can be an NxN sample block of some value N, such that a partitioning method divides the components into CTBs. A component is an array or a single sample from one of the three arrays (luminance and two chrominance) constituting a 4:2:0, 4:2:2, or 4:4:4 color format image, or an array or a single sample constituting an array or array constituting a monochrome format image. In some examples, a decoded block is an MxN sample block of values ​​M and N, such that a partitioning method divides the CTB into decoded blocks.

[0077] Blocks (e.g., CTUs or CUs) can be grouped in various ways within an image. As an example, a brick can refer to a rectangular area of ​​a row of CTUs within a specific tile in an image. A tile can be a rectangular area of ​​CTUs within a specific tile column and a specific tile row in an image. A tile column refers to a rectangular area of ​​a CTU having a height equal to the height of the image and a width specified by syntax elements (e.g., such as in an image parameter set). A tile row refers to a rectangular area of ​​a CTU having a height specified by syntax elements (e.g., such as in an image parameter set) and a width equal to the width of the image.

[0078] In some examples, a tile can be divided into multiple bricks, each brick comprising one or more CTU rows within the tile. A tile that is not divided into multiple bricks can also be called a brick. However, bricks that are a true subset of a tile cannot be called a tile. Bricks in an image can also be arranged in slices. A slice can be an integer number of bricks in an image that can be uniquely contained within a single Network Abstraction Layer (NAL) unit. In some examples, a slice comprises multiple complete tiles or a consecutive sequence of complete bricks comprising only one tile.

[0079] This disclosure uses "N×N" and "N multiplied by N" interchangeably to refer to the sample size of a block (such as a CU or other video block) in the vertical and horizontal dimensions, for example, 16×16 samples or 16 by 16 samples. Typically, a 16×16 CU will have 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Similarly, an N×NCU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU may be arranged in rows and columns. Furthermore, a CU does not necessarily need to have the same number of samples in the horizontal direction as it does in the vertical direction. For example, a CU may include N×M samples, where M is not necessarily equal to N.

[0080] The video encoder 200 encodes video data representing prediction and / or residual information, as well as other information, for use in the control unit (CU). The prediction information indicates how the CU should be predicted to form a prediction block for the CU. The residual information typically represents the sample-by-sample difference between a sample of the CU before encoding and the prediction block.

[0081] To predict the Cubic Frame (CU), the video encoder 200 typically forms prediction blocks for the CU through inter-frame prediction or intra-frame prediction. Inter-frame prediction typically refers to predicting the CU from data in a previously decoded image, while intra-frame prediction typically refers to predicting the CU from data in a previously decoded image within the same frame. To perform inter-frame prediction, the video encoder 200 can use one or more motion vectors to generate prediction blocks. The video encoder 200 can typically perform a motion search to identify reference blocks that closely match the CU, for example, based on the differences between the CU and a reference block. The video encoder 200 can use sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to compute difference metrics to determine whether a reference block closely matches the current CU. In some examples, the video encoder 200 can use unidirectional or bidirectional prediction to predict the current CU.

[0082] The video encoder 200 encodes data representing the prediction mode of the current block. For example, for inter-frame prediction modes, the video encoder 200 may encode data indicating which of the various available inter-frame prediction modes is used, as well as the motion information for the corresponding mode. For example, for unidirectional or bidirectional inter-frame prediction, the video encoder 200 may encode motion vectors using Advanced Motion Vector Prediction (AMVP) or merging modes. The video encoder 200 may use similar modes to encode motion vectors used for affine motion compensation modes.

[0083] AV1 includes two common techniques for encoding and decoding blocks of video data. These two common techniques are intra-frame prediction (e.g., intra-frame prediction or spatial prediction) and inter-frame prediction (e.g., inter-frame prediction or temporal prediction). In the context of AV1, when using intra-frame prediction modes to predict blocks of video data for the current frame, the video encoder 200 and video decoder 300 do not use video data from other frames of the video data. For most intra-frame prediction modes, the video encoder 200 encodes blocks of the current frame based on the difference between sample values ​​in the current block and predicted values ​​generated from reference samples in the same frame. The video encoder 200 determines the predicted values ​​generated from the reference samples based on the intra-frame prediction mode.

[0084] After prediction (such as intra-frame or inter-frame prediction for a block), the video encoder 200 can compute residual data for the block. The residual data (such as a residual block) represents the sample-by-sample difference between the block and the prediction block used to form the block, which is formed using the corresponding prediction mode. The video encoder 200 can apply one or more transforms to the residual block to produce transformed data in the transform domain rather than the sample domain. For example, the video encoder 200 can apply a Discrete Cosine Transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. Additionally, the video encoder 200 can apply a secondary transform after the first transform, such as the Mode Correlated Inseparable Secondary Transform (MDNSST), the Signal Correlation Transform, the Karhunen-Loeve Transform (KLT), etc. The video encoder 200 produces transform coefficients after applying one or more transforms.

[0085] As noted above, after any transform that produces the transform coefficients, the video encoder 200 may perform quantization on the transform coefficients. Quantization generally refers to a process in which the transform coefficients are quantized to reduce the amount of data used to represent them, thereby providing further compression. By performing the quantization process, the video encoder 200 may reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 may, during quantization, quantize the transform coefficients... n Bit value rounded down to m Bit value, where n Greater than m In some examples, in order to perform quantization, the video encoder 200 may perform a bitwise right shift on the value to be quantized.

[0086] After quantization, the video encoder 200 can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix containing the quantized transform coefficients. The scan can be designed to place higher-energy (and therefore lower-frequency) transform coefficients before the vector and lower-energy (and therefore higher-frequency) transform coefficients after the vector. In some examples, the video encoder 200 can utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector, and then entropy-encode the quantized transform coefficients of that vector. In other examples, the video encoder 200 can perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 200 can entropy-encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic decoding (CABAC). The video encoder 200 can also entropy-encode the values ​​of syntax elements describing metadata associated with the encoded video data, which is used by the video decoder 300 when decoding the video data.

[0087] To perform CABAC, the video encoder 200 can assign context within a context model to the symbols to be transmitted. Context may involve, for example, whether the neighboring values ​​of a symbol are zero. Probability determination can be based on the context assigned to the symbols.

[0088] The video encoder 200 may further generate syntax data for the video decoder 300, such as block-based syntax data, image-based syntax data, and sequence-based syntax data, for example, in image headers, block headers, and slice headers, or generate other syntax data such as sequence parameter sets (SPS), image parameter sets (PPS), or video parameter sets (VPS). The video decoder 300 may also decode such syntax data to determine how to decode the corresponding video data.

[0089] In this way, the video encoder 200 can generate a bitstream that includes encoded video data, such as syntax elements describing the partitioning of images into blocks (e.g., CUs) and prediction and / or residual information for the blocks. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0090] Typically, the video decoder 300 performs the reverse process of the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 can use CABAC to decode the values ​​of syntax elements used for the bitstream in a manner substantially similar to but reversed by the CABAC encoding process of the video encoder 200. Syntax elements can define partitioning information for dividing a picture into CTUs and defining the CUs of each CTU according to a corresponding partitioning structure such as a QTBT structure. Syntax elements can further define prediction and residual information for video data blocks (e.g., CUs).

[0091] The residual information can be represented, for example, by quantized transform coefficients. The video decoder 300 can inversely quantize and inverse transform the quantized transform coefficients of the block to reconstruct the residual block for the block. The video decoder 300 uses a signaling prediction mode (intra-frame prediction or inter-frame prediction) and associated prediction information (e.g., motion information for inter-frame prediction) to form a prediction block for the block. The video decoder 300 can then combine the prediction block and the residual block (on a sample-by-sample basis) to reconstruct the original block. The video decoder 300 can perform additional processing, such as performing a deblocking process to reduce visual artifacts along the block boundaries.

[0092] This disclosure may generally relate to "signaling" certain information (such as syntax elements). The term "signaling" can generally refer to the communication of values ​​and / or other data of syntax elements used to decode encoded video data. That is, video encoder 200 may signal the values ​​of syntax elements in the bitstream. Generally, signaling refers to generating values ​​in the bitstream. As noted above, source device 102 may transmit the bitstream to destination device 116 substantially in real time or not in real time (such as when syntax elements are stored in storage device 112 for later retrieval by destination device 116).

[0093] According to the technology disclosed herein, the video encoder 200 and video decoder 300 can determine whether to write the current image to dedicated on-chip memory or non-dedicated system memory. Dedicated on-chip memory can be a memory that the video encoder 200 and video decoder 300 can access quickly with relatively low power compared to non-dedicated system memory. Examples of dedicated on-chip memory include on-chip cache memory of an integrated circuit (IC) chip that includes the video encoder 200 or video decoder 300. Dedicated on-chip memory can be shared by components within the IC chip but cannot be shared by components outside the IC chip. Examples of non-dedicated system memory include off-chip memory, such as random access memory (RAM).

[0094] As described above, one type of video decoding technology is inter-frame prediction, in which video encoder 200 and video decoder 300 can access a reference image (e.g., a previously decoded image) to encode or decode the current block of the current image. That is, video encoder 200 and video decoder 300 can generate predicted blocks for encoding or decoding from samples of the reference image. The amount of time and power required to access the reference image can be based on whether the reference image is stored in dedicated on-chip memory or non-dedicated system memory, with accessing the reference image from dedicated on-chip memory requiring less time and power.

[0095] In one or more examples, the video encoder 200 and video decoder 300 may determine whether the current image should be stored in dedicated chip memory or non-dedicated system memory. In some examples, the video encoder 200 and video decoder 300 may even determine whether the current image should be stored in dedicated chip memory or non-dedicated system memory before encoding or decoding the current image. However, it is possible to determine whether the current image should be stored in dedicated chip memory or non-dedicated system memory after or during the encoding or decoding of the current image.

[0096] The video encoder 200 and video decoder 300 can determine whether to write the current image to dedicated chip memory or non-dedicated system memory based on information derived from a list of future reference images. A future image refers to an image that follows the current image in decoding order. The list of reference images includes candidate reference images that can be used as reference images for the current block being encoded or decoded. As described above, two lists of reference images may exist: list 0 and list 1, but the example technology is not limited to lists 0 and list 1. Each of lists 0 and list 1 may include multiple entries (e.g., 5 to 16 entries) that each identifies a different reference image. A reference image may be included in both lists 0 and list 1.

[0097] The video encoder 200 can signal the video decoder 300 with information about how to construct the list of reference images for the current image. However, for future images, the video decoder 300 may not have information about how to construct the list of reference images.

[0098] In one or more examples, the video decoder 300 may generate a list of multiple future reference images associated with multiple future images based on previously decoded images. For example, multiple sets of images may exist in a video sequence being decoded. An image set is an example of a decoding structure with a set of features. For example, multiple consecutive image sets may be of the same size. As another example, a first instantaneous decoder refresh (IDR) image and a second IDR image may define the end of the first image set and the second image set, and the first image set may be the first half of the images between the first IDR image and the second IDR image, and the second image set may be the second half of the images between the first IDR image and the second IDR image. In some examples, each image set may begin with an IDR image. In some examples, images in one image set may not be used for inter-frame prediction without reference images in another image set. An image set may be an example of a group of pictures (GOP) structure, such as the group of pictures structure in H.264 or AV1, but the example techniques are not limited to the GOP of H.264 or AV1.

[0099] In some examples, the structural profile of the first image set may be the same as that of the second image set. For instance, there may be some similarity between the list of reference images for the images in the first image set and the list of reference images for the images in the second image set, where the second image set follows the first image set in decoding order. The structural profile may refer to information about which images were predicted from which other images, the prediction mode of the images, the temporal layer identifier of the images, etc.

[0100] For example, suppose the second set of images includes the current image and future images (e.g., images decoded after the current image), and the first set of images includes previous images (e.g., previously decoded images). Because the video decoder 300 has reconstructed the previous images in the first set of images, the list of reference images for the first set of images is available to the video decoder 300.

[0101] In some cases, the difference between a previous image in the first image set and a reference image in the reference image list of that previous image in the first image set can be the same as the difference between an associated image in the second image set and a reference image in the reference image list of that associated image in the second image set. Associated images in the second image set can be in the same decoding order as previous images in the first image set. For example, a first image in the first image set in decoding order is associated with a first image in the second image set in decoding order. A second image in the first image set in decoding order is associated with a second image in the second image set in decoding order, and so on.

[0102] For example, suppose the first entry in list 0 of the Nth image in the first image set, ordered by decoding, identifies the first image set as NP in the decoding order. i The image. In this example, the first entry in list 0 of the Nth image in the second image set, ordered by decoding, might identify the second image set as NP in the second image set, ordered by decoding. i The image.

[0103] In this way, even if the images in the second image set have not yet been decoded, the video decoder 300 can determine a list of reference images for the images in the second image set. For example, the second image set may include current images and future images. The video decoder 300 can determine the list of future reference images for future images in the second image set based on associated previous images in the first image set (e.g., based on previous images in the first image set that are in the same relative decoding order position as the corresponding future images in the second image set).

[0104] In other words, the video decoder 300 can generate a list of multiple future reference images associated with multiple future images based on a first set of images decoded prior to a second set of images. The second set of images includes the current image and multiple future images, which are decoded after the current image in the order they were decoded. As an example, the video decoder 300 can determine a corresponding previous image in the first set of images that is located at the same relative position in the first set of images (e.g., in the order of decoding) as the multiple future images are located in the second set of images (e.g., in the order of decoding). The video decoder 300 can determine a corresponding list of reference images for the corresponding previous image and generate a corresponding list of multiple future reference images for each of the multiple future images based on the determined list of corresponding reference images for the corresponding previous image.

[0105] The examples above describe an example of generating multiple lists of future reference images. However, the generation of multiple lists of future reference images should not be interpreted as requiring the generation of the entire list of future reference images. For example, the list of future reference images may include multiple entries. In some examples, the video decoder 300 may generate a portion of the list of future reference images (e.g., the first two entries of the list of future reference images for each future image in the future images). Therefore, in this disclosure, the description of generating multiple lists of future reference images includes examples in which the video decoder 300 generates the entire list of future reference images or a portion of the list of future reference images.

[0106] Video encoder 200 can be configured to use a similar process to generate multiple lists of future reference images. However, in some examples, because video encoder 200 determines which images belong to the list of reference images, video encoder 200 may not need to rely on the list of reference images of previous images in the first set of images to determine the list of reference images for images in the second set of images.

[0107] In other words, the video encoder 200 and the video decoder 300 can generate a list of multiple future reference images associated with multiple future images in a set of images. The set of images includes the current image and multiple future images, and the multiple future images are ordered after the current image in the decoding order. For the video decoder 300, this set of images can be a second set of images. To generate the list of multiple future reference images, the video decoder 300 can generate the list of multiple future reference images associated with multiple future images based on a first set of images that were decoded before the second set of images.

[0108] From the perspective of video decoder 300, the list of future reference images can be a prediction of the actual list of future reference images, because it is possible that the actual list of future reference images will ultimately differ from the list of future reference images determined by the video decoder. From the perspective of video encoder 200, the list of future reference images can be determined and is known, but is not necessarily a prediction. Therefore, in one or more examples, multiple lists of future reference images can be (e.g., for video decoder 300) predicted or (e.g., for video encoder 200) determined lists of future reference images whose identifiers can be used as reference images for multiple future images.

[0109] According to one or more examples described in this disclosure, video encoder 200 and video decoder 300 may determine whether to write the current image to dedicated chip memory or non-dedicated chip memory based on information derived from multiple lists of future reference images associated with multiple future images. As an example, video encoder 200 and video decoder 300 may select between dedicated chip memory and non-dedicated system memory to which the current image is to be written based on information derived from multiple lists of future reference images. The current image refers to an image that is about to be encoded or is being encoded or decoded. In some examples, video encoder 200 and video decoder 300 may determine whether to write the current image to dedicated chip memory or non-dedicated chip memory before, during, or after encoding or decoding the current image.

[0110] In one or more examples, the future images evaluated by the video encoder 200 and video decoder 300 for their future reference image list may be a subset of all images in a second image set. That is, the second image set may include the current image, multiple future images, and one or more additional images. In some examples, the video encoder 200 and video decoder 300 may be configured with a defined search range. The search range defines the number of future images that the video encoder 200 and video decoder 300 derive for their evaluation of the future reference image list information used to determine whether to write the current image to dedicated chip memory or non-dedicated chip memory. For example, multiple future images may form the images of the search range, and one or more additional images may be images other than the future images forming the search range but within the image set.

[0111] In some examples, the search range is five future images. In other examples, the search range can be based on the number of images in the image set. For example, the search range can be equal to log2(the number of images in the image set). If log2(the number of images in the image set) + 1 < 2, then the search range can be set to two.

[0112] Video encoder 200 and video decoder 300 can derive example metrics from multiple lists of future reference images associated with multiple future images. For example, as described, the multiple lists of future reference images include multiple entries referenced by corresponding index values ​​(e.g., index 0 of list 0 is the first entry in list 0, index 1 of list 0 is the second entry in list 0, and so on). Video encoder 200 and video decoder 300 can determine which reference image is identified at the index of the future reference image list for each of the multiple future images.

[0113] As an example, suppose there are five future images in the search range: future images 1 to 5. Video encoder 200 and video decoder 300 can determine which reference image is identified at index 0 in the future reference image list 0 for future image 1, which reference image is identified at index 0 in the future reference image list 0 for future image 2, and so on. Video encoder 200 and video decoder 300 can determine which reference image is identified at index 0 in the future reference image list 1 for future image 1, which reference image is identified at index 1 in the future reference image list 0 for future image 2, and so on. Video encoder 200 can determine which reference image is identified at index 1 in the future reference image list 1 for future image 1, which reference image is identified at index 1 in the future reference image list 1 for future image 2, and so on.

[0114] The video encoder 200 and video decoder 300 may repeat these operations for all entries in the future reference image list of future images 1 to 5, or for a subset of the entries in the future reference image list of future images 1 to 5. In some examples, the video encoder 200 and video decoder 300 may determine which reference image is identified at one or more indices in the future reference image list (and not other reference image lists).

[0115] The size of the search scope, the number of future reference images evaluated, and the number of entries in the evaluated future reference image list can be based on a balance between the complexity and accuracy of determining whether to write the current image to dedicated chip memory or non-dedicated system memory. In some cases, determining which reference image is identified at a larger number of entries in the future reference image list may, but not necessarily, provide better information for determining whether to write the current image to dedicated chip memory or non-dedicated system memory, compared to identifying a smaller number of entries in the future reference image list. However, determining which reference image is identified at a larger number of entries in the future reference image list may require additional processing time and memory requirements.

[0116] Similarly, increasing the size of the search range (e.g., the number of future images to which the future reference image list is evaluated) or increasing the number of future reference image lists can, but not necessarily, produce better information for determining whether to write the current image to dedicated on-chip memory or non-dedicated system memory compared to identifying a smaller list of future reference images or having a smaller search range. However, increasing the size of the search range or evaluating two or more future reference image lists may require additional processing time and memory compared to a smaller search range or evaluating a smaller list of future reference images.

[0117] The video encoder 200 and video decoder 300 may determine one or more metrics based on determining which reference image is identified at an index in the future reference image list for each of the multiple future images. An example of a metric is called a future reference count value. A future reference count value indicates the number of times a particular reference image is identified at certain indices (including all indices) in the future reference image list. As an example, suppose index 0 of the future reference image list 0 for future images 1, 2, and 5 identifies reference image X, and suppose index 0 of the future reference image list 0 for future images 3 and 4 identifies reference image Y. In this example, the future reference count value for reference image X is three, and the future reference count value for reference image Y is two.

[0118] In the examples above, video encoder 200 and video decoder 300 consider index 0 of the future reference image list 0 for future images 1 to 5 to determine the future reference count value. In some examples, video encoder 200 and video decoder 300 may also consider index 0 of the future reference image list 1 for future images 1 to 5.

[0119] The video encoder 200 and video decoder 300 can determine whether the current image should be written to dedicated chip memory or non-dedicated chip memory based on future reference count values. For example, suppose the reference image X in the example above is the current image. Again, because reference image X is a reference image identified in the list of future reference images, it is possible that the current image is reference image X, since the current image can be a reference image for future images. Furthermore, suppose the reference image Y is a reference image currently stored in dedicated chip memory.

[0120] In this example, because the future reference count value of the current image (i.e., reference image X) is greater than the future reference count value of reference image Y, the current image is more likely to be accessed more frequently than reference image Y. Therefore, the video encoder 200 and the video decoder 300 can write the current image into a dedicated chip memory and replace the reference image Y in the dedicated chip memory. For example, the video encoder 200 and the video decoder 300 can write the reference image Y into a non-dedicated system memory.

[0121] In the examples above, the video encoder 200 and video decoder 300 use a future reference count (i.e., the number of times a reference image is identified at an index in a list of future reference images) to determine whether to write the current image to dedicated chip memory or non-dedicated system memory. In some examples, the video encoder 200 and video decoder 300 may use the future reference count to further refine the determination of whether to write the current image to dedicated chip memory or non-dedicated system memory.

[0122] As an example, another metric that the video encoder 200 and video decoder 300 can derive from a list of multiple future reference images associated with multiple future images is called a reference gain value, where the reference gain value is derived from a future reference count value. For example, the video encoder 200 and video decoder 300 can consider in which entry of the future reference image the reference image is identified.

[0123] Reference images identified in the first entry of List 0 or List 1 (e.g., index 0) tend to be accessed more frequently than reference images identified in other entries of List 0 or List 1. As an example, for illustrative purposes only, reference images used for inter-frame predictive coding or decoding of the current block in the current image are identified in index 0 of List 0 or List 1 90% of the time. Reference images used for inter-frame predictive coding or decoding of the current block in the current image are identified in index 1 of List 0 or List 1 approximately 5% to 9% of the time.

[0124] Video encoder 200 and video decoder 300 may determine a reference gain value for the current image based on a weighted future reference count value, whereby the future reference count value of the current image is weighted according to the entries in the future reference image list where the current image is identified. For example, video encoder 200 and video decoder 300 may determine a first reference count value for the current image based on the number of times the current image is identified in index 0 of each future reference image list associated with (e.g., within the search range) future images. Video encoder 200 and video decoder 300 may scale the first reference count value by a first weighting factor to generate a first scaled value. Video encoder 200 and video decoder 300 may determine a second reference count value for the current image based on the number of times the current image is identified in index 1 of each future reference image list associated with (e.g., within the search range) future images. Video encoder 200 and video decoder 300 may scale the second reference count value by a second weighting factor to generate a second scaled value.

[0125] The video encoder 200 and video decoder 300 can determine a reference gain value for the current image based on a first scaling value and a second scaling value. For example, the video encoder 200 and video decoder 300 can add the first scaling value and the second scaling value together to determine the reference gain value for the current image.

[0126] The video encoder 200 and video decoder 300 can perform similar operations to determine a reference gain value for the cached image (i.e., the image currently stored in dedicated chip memory). If the reference gain value of the current image is greater than the reference gain value of the cached image, the video encoder 200 and video decoder 300 can write the current image to the dedicated chip memory. Otherwise, the video encoder 200 and video decoder 300 can write the current image to non-dedicated system memory.

[0127] In this way, video encoder 200 and video decoder 300 can determine whether to write the current image to dedicated chip memory or non-dedicated system memory based on information derived from multiple lists of future reference images associated with multiple future images, and write the current image to dedicated chip memory or non-dedicated system memory based on the determination of whether to write the current image to dedicated chip memory or non-dedicated system memory. For example, video encoder 200 and video decoder 300 can derive information based on the number of times the current image is identified in multiple lists of future reference images (e.g., at index 0 or index 1 of future reference image lists 0 and 1). Video encoder 200 and video decoder 300 can also determine the number of times a cached image is identified in multiple lists of future reference images (e.g., at index 0 or index 1 of future reference image lists 0 and 1). Video encoder 200 and video decoder 300 can determine whether to write the current image to dedicated chip memory or non-dedicated system memory based on whether the current image or the cached image is identified more often in multiple lists of future reference images.

[0128] As a further refinement, the video encoder 200 and video decoder 300 can determine various metrics in addition to a metric referred to as a future reference count value. For example, information derived from a list of future reference images could be a first metric derived from the number of times the current image is identified in multiple lists of future reference images associated with multiple future images (e.g., a current image reference gain value based on the future reference count value of the current image). The video encoder 200 and video decoder 300 can derive a second metric associated with cached images stored in dedicated chip memory based on multiple lists of future reference images associated with multiple future images (e.g., a current image reference gain value based on the future reference count value of the current image). The video encoder 200 and video decoder 300 can compare the first metric with the second metric and, based on the comparison, determine whether to write the current image to dedicated chip memory or non-dedicated system memory.

[0129] Figure 2 This is a block diagram illustrating an example video encoder 200 that can perform the techniques of this disclosure. Figure 2 This disclosure is provided for illustrative purposes and should not be construed as a limitation on the techniques extensively illustrated and described herein. For illustrative purposes, this disclosure describes the video encoder 200 in accordance with the techniques of VVC and HEVC. However, the techniques of this disclosure can be performed by video encoding devices configured for other video decoding standards and video decoding formats, such as AV1 and subsequent formats of AV1 video decoding.

[0130] exist Figure 2In the example, the video encoder 200 includes a video data memory 232, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a storage circuit 218, a decoded image buffer (DPB) 228, and an entropy encoding unit 220. Any or all of the video data memory 232, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, storage circuit 218, DPB 228, and entropy encoding unit 220 can be implemented in one or more processors or in processing circuitry. For example, the units of the video encoder 200 can be implemented as one or more circuit or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. In addition, the video encoder 200 may include additional or alternative processors or processing circuitry to perform these and other functions.

[0131] exist Figure 2 In the example, DPB 228 includes dedicated chip memory 230 and non-dedicated system memory 240. In some examples, dedicated chip memory 230 (such as cache memory) may be internal to an integrated circuit (IC) chip including video encoder 200. Non-dedicated system memory 240 may be external to the IC chip and may be shared by components other than those of the IC chip including video encoder 200. As described in more detail, DPB 228 stores images that can be used as reference images. According to one or more examples, storage circuitry 218 may determine whether to write the current image being encoded to dedicated chip memory 230 or non-dedicated system memory 240 based on a list of future reference images for future images. Storage circuitry 218 may determine whether to write the current image being encoded to dedicated chip memory 230 or non-dedicated system memory 240 before, during, or after encoding the current image.

[0132] The video data storage device 232 can store video data to be encoded by components of the video encoder 200. The video encoder 200 can receive video data from, for example, a video source 104 (…). Figure 1The video data is received in video data memory 232. DPB 228 can act as a reference picture memory, storing reference video data for use when the video encoder 200 predicts subsequent video data. The non-dedicated system memory 240 of video data memory 232 and DPB 228 can be formed of any memory device from a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Dedicated on-chip memory 230 can be formed as a cache memory (e.g., L1 or L2 cache, or last-level cache). In various examples, DPB 228 may be at least partially on-chip with other components of the video encoder 200 (as shown), or at least partially off-chip relative to those components (as shown).

[0133] In this disclosure, references to video data memory 232 should not be construed as limited to memory inside video encoder 200 (unless specifically described as such) or memory outside video encoder 200 (unless specifically described as such). Rather, references to video data memory 232 should be understood as a reference memory that stores video data received by video encoder 200 for encoding (e.g., video data for the current block to be encoded). Figure 1 The memory 106 can also provide temporary storage for the outputs from various units of the video encoder 200.

[0134] Examples Figure 2 Various units help understand the operations performed by the video encoder 200. Units can be implemented as fixed-function circuits, programmable circuits, or combinations thereof. Fixed-function circuits are circuits that provide specific functionality and are pre-defined for the operations that can be performed. Programmable circuits are circuits that can be programmed to perform various tasks and provide flexible functionality for the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more units in the unit may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units in the unit may be integrated circuits.

[0135] The video encoder 200 may include an arithmetic logic unit (ALU), an essential function unit (EFU), digital circuitry, analog circuitry, and / or a programmable core, all formed by programmable circuitry. In an example where the operation of the video encoder 200 is performed using software executed by programmable circuitry, memory 106 ( Figure 1 The video encoder 200 may store instructions (e.g., target code) of the software received and executed by the video encoder 200, or another memory (not shown) within the video encoder 200 may store such instructions.

[0136] The video data storage unit 232 is configured to store the received video data. The video encoder 200 can retrieve images of the video data from the video data storage unit 232 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data storage unit 232 can be the raw video data to be encoded.

[0137] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra-frame prediction unit 226. The mode selection unit 202 may include additional functional units for performing video prediction based on other prediction modes. As an example, the mode selection unit 202 may include a palette unit, an intra-frame block copying unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0138] Mode selection unit 202 typically coordinates multiple coding channels to test combinations of coding parameters and the resulting rate-distortion values ​​for such combinations. Coding parameters may include the CTU-CU partitioning, the prediction mode for the CU, the transformation type of the residual data for the CU, the quantization parameters of the residual data for the CU, etc. Mode selection unit 202 can ultimately select a combination of coding parameters that has a better rate-distortion value compared to other tested combinations.

[0139] The video encoder 200 can divide a picture retrieved from the video data storage 232 into a series of CTUs and encapsulate one or more CTUs within a slice. The mode selection unit 202 can divide the CTUs of the picture according to a tree structure (such as an MTT structure, a QTBT structure, a superblock structure, or a quadtree structure described above). As described above, the video encoder 200 can form one or more CUs by dividing the CTUs according to a tree structure. Such CUs are also commonly referred to as "video blocks" or "blocks".

[0140] Typically, mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-prediction unit 226) to generate prediction blocks for the current block (e.g., the current CU, or, in HEVC, the overlapping portion of PU and TU). To perform inter-frame prediction for the current block, motion estimation unit 222 may perform a motion search to identify one or more closely matching reference blocks in one or more reference images (e.g., one or more previously decoded images stored in DPB 228). Specifically, motion estimation unit 222 may calculate values ​​representing the similarity between a potential reference block and the current block, for example, based on sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), etc. Motion estimation unit 222 may typically perform these calculations using sample-by-sample differences between the current block and the reference blocks under consideration. Motion estimation unit 222 may identify reference blocks with the lowest values ​​produced by these calculations to indicate the reference block that best matches the current block.

[0141] Motion estimation unit 222 can generate one or more motion vectors (MVs) that define the location of a reference block in a reference image relative to the location of the current block in the current image. Motion estimation unit 222 can then provide the motion vectors to motion compensation unit 224. For example, for unidirectional inter-frame prediction, motion estimation unit 222 can provide a single motion vector, while for bidirectional inter-frame prediction, motion estimation unit 222 can provide two motion vectors. Motion compensation unit 224 can then use the motion vectors to generate a prediction block. For example, motion compensation unit 224 can use the motion vectors to retrieve data for the reference block. As another example, where the motion vectors have fractional sample precision, motion compensation unit 224 can interpolate the values ​​of the prediction block according to one or more interpolation filters. Furthermore, for bidirectional inter-frame prediction, motion compensation unit 224 can retrieve data for two reference blocks identified by corresponding motion vectors and combine the retrieved data, for example, by per-sample averaging or weighted averaging.

[0142] When operating according to the AV1 video decoding format, the motion estimation unit 222 and the motion compensation unit 224 can be configured to encode the decoded blocks of video data (e.g., both luma and chroma decoded blocks) using translational motion compensation, affine motion compensation, overlap block motion compensation (OBMC), and / or composite inter-intra-frame prediction.

[0143] As another example, for intra-prediction or intra-prediction decoding, intra-prediction unit 226 may generate a prediction block from samples adjacent to the current block. For example, for directional mode, intra-prediction unit 226 may typically mathematically combine the values ​​of adjacent samples and fill these calculated values ​​across the current block in a defined direction to produce a prediction block. As another example, for DC mode, intra-prediction unit 226 may calculate the average of the adjacent samples of the current block and generate a prediction block to include the resulting average for each sample of the prediction block.

[0144] When operating according to the AV1 video decoding format, the intra-frame prediction unit 226 can be configured to encode decoded blocks of video data (e.g., both luma and chroma decoded blocks) using directional intra-frame prediction, non-directional intra-frame prediction, recursive filter intra-frame prediction, luma-chroma (CFL) prediction, intra-block copying (IBC), and / or palette modes. The mode selection unit 202 may include additional functional units for performing video prediction based on other prediction modes.

[0145] Mode selection unit 202 provides a prediction block to residual generation unit 204. Residual generation unit 204 receives an uncoded raw version of the current block from video data memory 232 and a prediction block from mode selection unit 202. Residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block for the current block. In some examples, residual generation unit 204 may also determine the differences between sample values ​​in the residual block to generate the residual block using residual differential pulse decoding modulation (RDPCM). In some examples, residual generation unit 204 may be formed using one or more subtractor circuits performing binary subtraction.

[0146] In the example where mode selection unit 202 divides a CU into PUs, each PU can be associated with a luma prediction unit and a corresponding chroma prediction unit. Video encoder 200 and video decoder 300 can support PUs of various sizes. As noted above, the size of a CU can refer to the size of the luma decoding block of the CU, while the size of a PU can refer to the size of the luma prediction unit of the PU. Assuming a particular CU size is 2N×2N, video encoder 200 can support PU sizes of 2N×2N or N×N for intra-frame prediction, and symmetric PU sizes of 2N×2N, 2N×N, N×2N, N×N, or similar for inter-frame prediction. Video encoder 200 and video decoder 300 can also support asymmetric partitioning for PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0147] In an example where mode selection unit 202 does not further divide the CU into PUs, each CU can be associated with a luminance decoding block and a corresponding chrominance decoding block. As mentioned above, the size of the CU can refer to the size of the luminance decoding block of the CU. The video encoder 200 and the video decoder 300 can support CU sizes of 2N×2N, 2N×N, or N×2N.

[0148] For other video decoding techniques, such as intra-block copy mode decoding, affine mode decoding, and linear model (LM) mode decoding, as some examples, mode selection unit 202 generates a prediction block for the current block being encoded via a corresponding unit associated with the decoding technique. In some examples (such as palette mode decoding), mode selection unit 202 may not generate a prediction block, but instead generate syntax elements indicating how the block is reconstructed based on a selected palette. In such modes, mode selection unit 202 may provide these syntax elements to entropy coding unit 220 for encoding.

[0149] As described above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. Then, the residual generation unit 204 generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0150] Transform processing unit 206 applies one or more transformations to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transformations to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply a discrete cosine transform (DCT), direction transform, Karhunen-Loeve transform (KLT), or conceptually similar transformations to the residual block. In some examples, transform processing unit 206 may perform multiple transformations on the residual block, such as primary and secondary transformations (e.g., rotation transformations). In some examples, transform processing unit 206 does not apply any transformations to the residual block.

[0151] When operating according to AV1, transform processing unit 206 may apply one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply a combination of horizontal / vertical transforms, which may include the Discrete Cosine Transform (DCT), the Asymmetric Discrete Sine Transform (ADST), the Reversed ADST (e.g., ADST in reverse order), and the Identity Transform (IDTX). When using the Identity Transform, the transform is skipped in either the vertical or horizontal direction. In some examples, transform processing may be skipped entirely.

[0152] Quantization unit 208 quantizes the transform coefficients in a transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 quantizes the transform coefficients of the transform coefficient block according to the quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode selection unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may cause information loss, and therefore, the quantized transform coefficients may have lower accuracy compared to the original transform coefficients produced by transform processing unit 206.

[0153] The video encoder 200 includes a reconstruction loop in which the encoded current image is reconstructed and stored in the DPB 228. Thus, the current image can be used as a reference image for future images. The reconstruction loop includes an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, and an optional filter unit 216.

[0154] The inverse quantization unit 210 and the inverse transform processing unit 212 can apply inverse quantization and inverse transform, respectively, to the quantized transform coefficient block to reconstruct the residual block based on the transform coefficient block. The reconstruction unit 214 can generate a reconstructed block corresponding to the current block (although potentially with some degree of distortion) based on the reconstructed residual block and the prediction block generated by the mode selection unit 202. For example, the reconstruction unit 214 can add samples of the reconstructed residual block to corresponding samples of the prediction block generated by the mode selection unit 202 to generate the reconstructed block.

[0155] Filter unit 216 may perform one or more filtering operations on the reconstructed block. For example, filter unit 216 may perform a deblocking operation to reduce block artifacts along the edges of the CU. In some examples, the operation of filter unit 216 may be skipped.

[0156] When operating according to AV1, filter unit 216 may perform one or more filtering operations on the reconstructed block. For example, filter unit 216 may perform a deblocking operation to reduce block artifacts along the edges of the CU. In other examples, filter unit 216 may apply a constrained direction enhancement filter (CDEF) after deblocking and may include the application of a non-separable, nonlinear, low-pass directional filter based on the estimated edge direction. Filter unit 216 may also include a loop recovery filter applied after CDEF and may include a separable symmetric normalized Wiener filter or a dual-guided filter.

[0157] The video encoder 200 stores reconstructed blocks in the DPB 228. For example, in an example where the filter unit 216 is not operated, the reconstruction unit 214 may store reconstructed blocks in the DPB 228. In an example where the filter unit 216 is operated, the filter unit 216 may store filtered reconstructed blocks in the DPB 228. The motion estimation unit 222 and the motion compensation unit 224 may retrieve reference images formed by the reconstructed (and potentially filtered) blocks from the DPB 228 to perform inter-frame prediction of blocks in subsequent encoded images. Additionally, the intra-frame prediction unit 226 may use the reconstructed blocks of the current image in the DPB 228 to perform intra-frame prediction of other blocks in the current image.

[0158] In one or more examples described in this disclosure, storage circuitry 218 may determine whether a reconstructed block of the current image should be stored in dedicated chip memory 230 or non-dedicated system memory 240. For example, mode selection unit 202 may be configured to generate a list of reference images (e.g., list 0 and list 1) from which mode selection unit 202 can determine reference images for the current image. In some examples, mode selection unit 202 may determine a list of reference images for the current image, and a list of future reference images associated with future images following the current image in coded order. The current image, future images, and possible additional images may form an image set. For example, future images may form a search range, but the image set may contain more images (e.g., additional images in the image set besides the current image and the images forming the search range).

[0159] Storage circuit 218 can determine whether to write the current image to dedicated chip memory 230 or non-dedicated system memory 240 based on information derived from multiple lists of future reference images associated with multiple future images. In other words, storage circuit 218 can select between dedicated chip memory 230 and non-dedicated system memory 240 to write the current image based on information derived from multiple lists of future reference images. For example, storage circuit 218 can derive information based on the number of times the current image is identified in the multiple lists of future reference images. As an example, storage circuit 218 can determine the number of times the current image is identified in the first entry (e.g., index 0) of the future reference image list 0 for each of the multiple future images, the number of times the current image is identified in the first entry (e.g., index 0) of the future reference image list 1 for each of the multiple future images, the number of times the current image is identified in the first entry (e.g., index 0) and second entry (e.g., index 1) of the future reference image list 0 for each of the multiple future images, the number of times the current image is identified in the first entry (e.g., index 0) and second entry (e.g., index 1) of the future reference image list 1 for each of the multiple future images, or any combination thereof. Furthermore, the example technique should not be considered limited to indices 0 and 1, and other entries are possible.

[0160] Similarly, storage circuitry 218 can determine (e.g., the number of times a cached image is identified in a plurality of future reference image lists while the current image is being encoded) the number of times a cached image is identified in the plurality of future reference image lists. Storage circuitry 218 can evaluate first and / or second entries of future reference image lists 0 and / or 1, as described above, to determine the number of times a cached image is identified in the plurality of future reference image lists. Storage circuitry 218 can determine whether to write the current image to the plurality of future reference image lists based on the number of times the current image is identified in the plurality of future reference image lists and the number of times the cached image is identified in the plurality of future reference image lists.

[0161] As another example, storage circuitry 218 may derive a first metric from the number of times the current image is identified in a list of multiple future reference images associated with multiple future images, and derive a second metric associated with cached images stored in dedicated chip memory based on the list of multiple future reference images associated with multiple future images. Storage circuitry 218 may compare the first metric with the second metric and determine, based on the comparison, whether to write the current image to dedicated chip memory 230 or non-dedicated system memory 240. If the first metric is less than the second metric, storage circuitry 218 may write a block of the current image to non-dedicated system memory 240. If the first metric is greater than the second metric, storage circuitry 218 may write a block of the current image to dedicated chip memory 230.

[0162] Typically, entropy coding unit 220 can entropy-encode syntax elements received from other functional components of video encoder 200. For example, entropy coding unit 220 can entropy-encode quantized transform coefficient blocks from quantization unit 208. As another example, entropy coding unit 220 can entropy-encode predictive syntax elements (e.g., motion information for inter-frame prediction or intra-frame mode information for intra-frame prediction) from mode selection unit 202. Entropy coding unit 220 can perform one or more entropy coding operations on syntax elements (another example of video data) to generate entropy-coded data. For example, entropy coding unit 220 can perform context-adaptive variable-length decoding (CAVLC), CABAC, variable-to-variable (V2V) length decoding, syntax-based context-adaptive binary arithmetic decoding (SBAC), probability interval partitioning entropy (PIPE) decoding, exponential Golomb coding, or another type of entropy coding operation on the data. In some examples, entropy coding unit 220 can operate in a bypass mode where syntax elements are not entropy-encoded.

[0163] The video encoder 200 can output a bitstream that includes the entropy coding syntax elements required to reconstruct slices or blocks of images. Specifically, the entropy coding unit 220 can output a bitstream.

[0164] According to AV1, entropy coding unit 220 can be configured as a symbol-to-symbol adaptive multi-symbol arithmetic decoder. The syntax elements in AV1 consist of an N-element alphabet, and the context (e.g., a probability model) consists of a set of N probabilities. Entropy coding unit 220 can store the probabilities as an n-bit (e.g., 15-bit) cumulative distribution function (CDF). Entropy coding unit 220 can perform recursive scaling to update the context using an update factor based on the alphabet size.

[0165] The operations described above are relative to blocks. This description should be understood as operations applied to luma decoding blocks and / or chroma decoding blocks. As described above, in some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the CU. In some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the PU.

[0166] In some examples, it is not necessary to repeat the operations performed relative to the luma decoder for the chroma decoder block. As an example, the operations for identifying the motion vector (MV) and reference image of the luma decoder block do not need to repeat the MV and reference image used to identify the chroma block. Instead, the MV used for the luma decoder block can be scaled to determine the MV used for the chroma block, and the reference image can be the same. As another example, the intra-frame prediction process can be the same for both the luma and chroma decoders.

[0167] Figure 3 This is a block diagram illustrating an example video decoder 300 that can perform the techniques of this disclosure. Figure 3 This disclosure is provided for illustrative purposes and not for limiting the techniques extensively illustrated and described herein. For illustrative purposes, the video decoder 300 is described in accordance with VVC and HEVC techniques. However, the techniques of this disclosure can be implemented by video decoding devices configured for other video decoding standards.

[0168] exist Figure 3 In the example, the video decoder 300 includes a decoded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, a storage circuit 314, and a DPB 322. Any or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, storage circuit 314, and DPB 322 may be implemented in one or more processors or in processing circuitry. For example, the units of the video decoder 300 may be implemented as one or more circuit or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuitry to perform these and other functions.

[0169] exist Figure 3In the example, DPB 322 includes dedicated chip memory 324 and non-dedicated system memory 326. In some examples, dedicated chip memory 324 (such as cache memory) may be internal to an integrated circuit (IC) chip including video decoder 300. Non-dedicated system memory 326 may be external to the IC chip and may be shared by components other than those of the IC chip including video decoder 300. As described in more detail, DPB 322 stores images that can be used as reference images. According to one or more examples, storage circuitry 314 may determine whether to write the current image being decoded to dedicated chip memory 324 or non-dedicated system memory 326 based on a list of future reference images for future images. Storage circuitry 314 may determine whether to write the current image being decoded to dedicated chip memory 324 or non-dedicated system memory 326 before, during, or after decoding the current image.

[0170] The prediction processing unit 304 includes a motion compensation unit 316 and an intra-prediction unit 318. The prediction processing unit 304 may include additional units that perform predictions based on other prediction modes. As an example, the prediction processing unit 304 may include a palette unit, an intra-block copying unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components.

[0171] When operating according to AV1, motion compensation unit 316 can be configured to decode video data blocks (e.g., both luma and chroma blocks) using translational motion compensation, affine motion compensation, OBMC, and / or composite inter-intra-frame prediction, as described above. Intra-frame prediction unit 318 can be configured to decode video data blocks (e.g., both luma and chroma blocks) using directional intra-frame prediction, non-directional intra-frame prediction, recursive filter intra-frame prediction, CFL, IBC, and / or palette mode, as described above.

[0172] CPB memory 320 can store video data to be decoded by components of video decoder 300, such as encoded video bitstreams. For example, it can be retrieved from computer-readable medium 110 ( Figure 1The video data stored in the CPB memory 320 is obtained. The CPB memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. Furthermore, the CPB memory 320 may store video data other than the syntax elements of the decoded picture, such as temporary data representing the output of various units from the video decoder 300. The DPB 322 typically stores a decoded picture, which the video decoder 300 may output, and / or uses as reference video data when decoding subsequent data or pictures of the encoded video bitstream. The non-dedicated system memory 326 of the CPB memory 320 and DPB 322 may be formed by any memory device from various memory devices, such as DRAM (including SDRAM), MRAM, RRAM, or other types of memory devices. The dedicated chip memory 324 may be formed as a cache memory (e.g., L1 or L2 cache, or last-level cache). In various examples, the DPB 322 may be at least partially on-chip with other components of the video decoder 300 (as shown), or at least partially off-chip relative to those components (as shown).

[0173] Additionally or alternatively, in some examples, the video decoder 300 may be from the memory 120 ( Figure 1 The decoded video data can be retrieved from the memory. In other words, memory 120 can utilize CPB memory 320 to store data as discussed above. Similarly, when some or all of the functionality of video decoder 300 is implemented in software to be executed by the processing circuitry of video decoder 300, memory 120 can store instructions to be executed by video decoder 300.

[0174] Examples Figure 3 The various units shown help to understand the operations performed by the video decoder 300. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Similar to... Figure 2 Fixed-function circuits are circuits that provide specific functionality and are predefined for the operations they can perform. Programmable circuits are circuits that can be programmed to perform various tasks and provide flexible functionality for the operations they can perform. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is usually immutable. In some examples, one or more units in a cell can be different circuit blocks (fixed-function or programmable), and in some examples, one or more units in a cell can be integrated circuits.

[0175] The video decoder 300 may include an ALU, EFU, digital circuitry, analog circuitry, and / or a programmable core formed by programmable circuitry. In an example where the operation of the video decoder 300 is performed by software executed on programmable circuitry, on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0176] The entropy decoding unit 302 can receive encoded video data from the CPB and perform entropy decoding on the video data to reproduce the syntax elements. The prediction processing unit 304, the inverse quantization unit 306, the inverse transform processing unit 308, the reconstruction unit 310, and the filter unit 312 can generate decoded video data based on the syntax elements extracted from the bitstream.

[0177] Typically, the video decoder 300 reconstructs the image block by block. The video decoder 300 can perform the reconstruction operation on each block individually (where the block currently being reconstructed (i.e., decoded) can be referred to as the "current block").

[0178] Entropy decoding unit 302 can entropy decode the syntax elements of the quantized transform coefficients defining the quantized transform coefficient block, as well as transform information (such as quantization parameters (QP) and / or transform mode indications). Inverse quantization unit 306 can use the QP associated with the quantized transform coefficient block to determine the degree of quantization, and similarly, determine the degree of inverse quantization to be applied by inverse quantization unit 306. Inverse quantization unit 306 can, for example, perform a bit-by-bit left shift operation to inverse quantize the quantized transform coefficients. Inverse quantization unit 306 can thereby form a transform coefficient block including the transform coefficients.

[0179] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 may apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotation transform, an inverse direction transform, or another inverse transform to the transform coefficient block.

[0180] Furthermore, the prediction processing unit 304 generates a prediction block based on the prediction information syntax elements entropy-decoded by the entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is an inter-frame prediction, the motion compensation unit 316 can generate the prediction block. In this case, the prediction information syntax elements may indicate a reference image in the DPB 322 from which the reference block is retrieved, and a motion vector identifying the position of the reference block in the reference image relative to the current block position in the current image. The motion compensation unit 316 can typically be configured according to the parameters relative to the motion compensation unit 224 ( Figure 2 The method described is essentially the same as the method used to perform the inter-frame prediction process.

[0181] As another example, when the prediction information syntax element indicates that the current block is intra-predictive, the intra-predictive unit 318 can generate a prediction block according to the intra-predictive mode indicated by the prediction information syntax element. Similarly, the intra-predictive unit 318 can generally be associated with the intra-predictive unit 226 ( Figure 2 The intra-prediction process is performed in a manner substantially similar to that described above. The intra-prediction unit 318 can retrieve data of neighboring samples of the current block from the DPB 322.

[0182] Reconstruction unit 310 can use the prediction block and the residual block to reconstruct the current block. For example, reconstruction unit 310 can add samples from the residual block to the corresponding samples from the prediction block to reconstruct the current block.

[0183] Filter unit 312 can perform one or more filtering operations on the reconstructed block. For example, filter unit 312 can perform a deblocking operation to reduce block artifacts along the edges of the reconstructed block. The operation of filter unit 312 is not necessarily performed in all examples.

[0184] The video decoder 300 can store reconstructed blocks in the DPB 322. For example, in an example where the operation of the filter unit 312 is not performed, the reconstruction unit 310 can store reconstructed blocks in the DPB 322. In an example where the operation of the filter unit 312 is performed, the filter unit 312 can store filtered reconstructed blocks in the DPB 322. As discussed above, the DPB 322 can provide reference information (such as samples of the current image for intra-frame prediction and samples of previously decoded images for subsequent motion compensation) to the prediction processing unit 304. Furthermore, the video decoder 300 can output decoded images (e.g., decoded video) from the DPB 322 for subsequent processing, such as... Figure 1 The display is presented on the display device 118.

[0185] In one or more examples described in this disclosure, storage circuitry 314 may determine whether a reconstructed block of the current image should be stored in dedicated chip memory 324 or non-dedicated system memory 326. In one or more examples, to determine whether a block of the current image should be stored in dedicated chip memory 324 or non-dedicated system memory 326, storage circuitry 314 may generate a list of multiple future reference images associated with multiple future images based on a first set of images decoded prior to a second set of images. The second set of images includes the current image and multiple future images, which are decoded after the current image in the order they were decoded.

[0186] As an example, to generate a list of multiple future reference images, storage circuit 314 can determine corresponding previous images in a first image set, which are located at the same relative positions in the first image set as the multiple future images are located in a second image set. Storage circuit 314 can determine a corresponding list of reference images for each corresponding previous image, and based on the determined corresponding list of reference images for each of the multiple future images, generate a corresponding list of multiple future reference images for each of the multiple future images. The example method for generating multiple lists of future reference images is as follows: Figure 4A and Figure 4B It has been described.

[0187] Storage circuitry 314 can determine whether to write the current image to dedicated chip memory 324 or non-dedicated system memory 326 based on information derived from multiple lists of future reference images associated with multiple future images. For example, storage circuitry 314 can select between dedicated chip memory 324 and non-dedicated system memory 326 to write the current image based on information derived from multiple lists of future reference images. For example, storage circuitry 314 can derive information based on the number of times the current image is identified in the multiple lists of future reference images. As an example, storage circuit 314 can determine the number of times the current image is identified in the first entry (e.g., index 0) of the future reference image list 0 for each of the multiple future images, the number of times the current image is identified in the first entry (e.g., index 0) of the future reference image list 1 for each of the multiple future images, the number of times the current image is identified in the first entry (e.g., index 0) and second entry (e.g., index 1) of the future reference image list 0 for each of the multiple future images, the number of times the current image is identified in the first entry (e.g., index 0) and second entry (e.g., index 1) of the future reference image list 1 for each of the multiple future images, or any combination thereof. Furthermore, the example technique should not be considered limited to indices 0 and 1, and other entries are possible.

[0188] Similarly, storage circuitry 314 can determine (e.g., the number of times a cached image is identified in a plurality of future reference image lists while the current image is being decoded) the number of times the cached image is identified in the plurality of future reference image lists. Storage circuitry 314 can evaluate first and / or second entries of future reference image lists 0 and / or 1, as described above, to determine the number of times the cached image is identified in the plurality of future reference image lists. Storage circuitry 314 can determine whether to write the current image to the plurality of future reference image lists based on the number of times the current image is identified in the plurality of future reference image lists and the number of times the cached image is identified in the plurality of future reference image lists.

[0189] As another example, storage circuitry 314 may derive a first metric from the number of times the current image is identified in a list of multiple future reference images associated with multiple future images, and derive a second metric associated with cached images stored in dedicated chip memory based on the list of multiple future reference images associated with multiple future images. Storage circuitry 314 may compare the first metric with the second metric and determine, based on the comparison, whether to write the current image to dedicated chip memory 324 or non-dedicated system memory 326. If the first metric is less than the second metric, storage circuitry 314 may write a block of the current image to non-dedicated system memory 326. If the first metric is greater than the second metric, storage circuitry 314 may write a block of the current image to dedicated chip memory 324.

[0190] Figure 4A and Figure 4B This is a concept graph illustrating the dependencies of images used for inter-frame prediction. For example, Figure 4A Example of the first image set 400A, and Figure 4B Example of the second image set 400B. In Figure 4A and Figure 4B In the example, the first image set 400A may be previously decoded images (e.g., including previous images) and includes images I0 to B16. The second image set may include the current image and one or more future images and includes images B17 to I32. I0 and I32 may be IDR images, and in this example, the first image set 400A may include the first half of the images between I0 and I32, and the second image set 400B may include the second half of the images between I0 and I32.

[0191] Figure 4A and Figure 4B The arrows in the diagram indicate which images are reference images to a given image. For example, image B1 has arrows pointing towards image I0 and towards image B2. Therefore, images I0 and B2 are reference images to image B1.

[0192] exist Figure 4A and Figure 4B In this text, image names B1, B2, etc., do not indicate the decoding order. The decoding order refers to the order in which the video encoder 200 or video decoder 300 encodes or decodes the images. For illustrative purposes, in... Figure 4A In the example, the decoding order could be as follows: B16, B8, B4, B2, B1, B3, B6, B5, B7, B12, B10, B9, B11, B14, B13, and B15. Therefore, image B6 is decoded (e.g., encoded or decoded) before B5. I0 of the first image set 400 is an intra-frame image and is not predicted using a reference image.

[0193] The dependencies in the second image set 400B can be based on the predicted dependencies of the first image set 400A. That is, it can be assumed that the second image set 400B includes images decoded in the same order as the first image set 400A, and the reference image list of the images in the second image set 400B can be associated with the reference image list of the images in the first image set 400A.

[0194] The assumption that the decoding order of the images in the first image set 400A and the second image set 400B is correlated with the reference image list generally holds true throughout the video sequence. For example, there may be epochs in the video sequence, where an epoch includes multiple image sets, each associated with a previous image set in terms of image decoding order and reference images. At some point, the correlation may no longer hold. However, within an epoch that may include hundreds of image sets, the assumed correlation may hold. Example techniques can utilize this correlation to predict future reference image lists, and based on these future reference image lists, determine whether to store the current image in dedicated chip memory 230 or 324 or non-dedicated system memory 240 or 326.

[0195] therefore, Figure 4B The decoding order in the image set can be assumed to be as follows: B16, B24, B20, B18, B17, B19, B22, B21, B23, B28, B26, B25, B27, B30, B29, and B31. The I32 frames in the second image set 400B are intra-frame images and were not predicted using the reference image. Figure 4A and Figure 4B In the example, image B8 in the first image set 400A is located in the same relative position in the first image set 400A according to the decoding order as image B24 in the second image set 400B (e.g., the second image in the second image set 400B according to the decoding order) (e.g., the second image in the first image set 400A according to the decoding order). Similarly, image B4 in the first image set 400A is located in the same relative position in the first image set 400A according to the decoding order as image B20 in the second image set 400B (e.g., the third image in the second image set 400B according to the decoding order) (e.g., the third image in the first image set 400A according to the decoding order), and so on.

[0196] As described, there may be a correlation between the list of reference images for images in the first image set 400A and the list of reference images for images in the second image set 400B. For example, suppose that index 0 of list 0 for image B5 identifies image B6. In this example, the decoding order distance (e.g., the difference in decoding order values) is one because B5 follows B6 in decoding order. The decoding order value can indicate the ascending decoding order of the images. For example, in the first image set 400A, image B6 is the seventh image in decoding order, and image B5 is the eighth image in decoding order. Eight minus seven equals one; therefore, the decoding order distance between images B5 and B6 is one.

[0197] The decoding order position of image B21 in the second image set 400B is the same as the decoding order position of image B5 in the first image set 400A. The decoding order position of image B22 in the second image set 400B is the same as the decoding order position of image B6 in the first image set 400A.

[0198] In one or more examples, storage circuit 314 can determine that, for index 0 of list 0 of image B21, the decoding order distance between the images to be identified at index 0 is one, because the decoding order distance between the images to be identified at index 0 of list 0 of image B5 is one, and B5 and B21 are in the same relative decoding order position in the first image set 400A and the second image set 400B. In this example, storage circuit 314 can determine that image B22 should be identified at index 0 of list 0 of image B21.

[0199] For example, suppose image B18 is the current image being decoded. Therefore, video decoder 300 may not yet have received information indicating which images belong to the reference image list of image B21. However, because video decoder 300 has previously constructed a reference image list for image B5, and image B5's decoding order position in the first image set 400A is the same as image B21's decoding order position in the second image set 400B, storage circuit 314 can predict that image B22 is at index 0 of list 0 of image B21, since image B6 is at index 0 of list 0 of image B5.

[0200] In other words, in order to generate a list of future reference images for image B21 in the second image set 400B based on the first image set 400A that was decoded before the second image set 400B, the storage circuit 314 can determine the decoding order position of image B21 in the second image set 400B (e.g., the eighth image in decoding order). The storage circuit 314 can determine a previous image in the first image set 400A that is at the same decoding order position as image B21 in the second image set 400B. For example, image B5 (e.g., a previous image) is the eighth image in the first image set 400A in decoding order and is at the same decoding order position as image B21 in the second image set 400B.

[0201] Storage circuit 314 can determine the decoding order difference between each entry in lists 0 and 1 of the previous image B5. For example, the decoding order difference of the image identified at index 0 of list 0 of the previous image B5 (e.g., image B6) is one. Storage circuit 314 can use the decoding order difference to determine the entries in lists 0 and 1 of image B21. For example, the decoding order difference of the image identified at index 0 of list 0 of image B21 should also be one. In this example, image B22 has a decoding order difference of one relative to image B21. Therefore, storage circuit 314 can determine that image B22 should be identified at index 0 of list 0 of image B21.

[0202] In this way, the storage circuit 314 can be configured to generate multiple lists of future reference images. For example, suppose image B18 is the current image. The storage circuit 314 can perform the above example technique to generate lists 0 and / or 1 of future reference images for image B17 based on lists 0 and / or 1 of the previous image B1, because images B17 and B1 are in the same decoding order position in the corresponding second image set 400B and first image set 400A. Similarly, the storage circuit 314 performs the above example technique to generate lists 0 and / or 1 of future reference images for image B19 based on lists 0 and / or 1 of the previous image B3, because images B19 and B3 are in the same decoding order position in the corresponding second image set 400B and first image set 400A, and so on.

[0203] The storage circuit 314 may not need to generate the entire list of future reference images 0 and list 1 for future images in the second image set 400B. Instead, the storage circuit 314 may generate a portion of the list of future reference images 0 and list 1 for future images in the second image set 400B (e.g., the first two entries of list 0 and list 1). Furthermore, the storage circuit 314 may not need to generate both the list of future reference images 0 and list 1 for future images in the second image set 400B, and may generate only the list of future reference images 0 or list 1 for future images in the second image set 400B.

[0204] As described above, the number of entries in future reference image list 0 or list 1, and whether storage circuit 314 determines the number of entries in both future reference image list 0 and list 1, can be a balance of complexity. For example, determining more entries in future reference image list 0 and / or list 1 by storage circuit 314 compared to determining fewer entries can, but does not necessarily, produce a more accurate determination of whether to store the current image in dedicated chip memory 324 or non-dedicated system memory 326. However, the complexity and storage requirements for determining more entries in future reference image list 0 and / or list 1 may be greater than determining fewer entries in future reference image list 0 and / or list 1.

[0205] Therefore, in one or more examples, to generate a list of multiple future reference images, the storage circuit 314 may be configured to determine corresponding previous images in a first image set 400A, which are located in the same relative decoding order position as the multiple future images are located in a second image set 400B. For example, in the first image set 400A and the second image set 400B, images B5 and B21 are located in the same relative decoding order position, B6 and B22 are located in the same relative decoding order position, and so on.

[0206] Storage circuit 314 can determine a corresponding list of reference images for a given previous image. For example, storage circuit 314 can determine reference image list 0 and list 1 for image B5 (e.g., where index 0 of list 0 for image B5 identifies image B6). Storage circuit 314 can generate corresponding multiple lists of future reference images for each of a plurality of future images based on the determined corresponding list of reference images for the given previous images. For example, for image B21, storage circuit 314 can determine that image B22 should be in index 0 of list 0 for image B21. This is because the decoding order difference between image B5 and the image identified in index 0 of list 0 for image B5 is one, and the decoding order difference between image B21 and image B22 is one.

[0207] Figure 5This is a flowchart illustrating an example method for determining whether a current image is written to a dedicated chip memory or a non-dedicated system memory according to the technology of this disclosure. For simplicity, the video decoder 300 is described below. Figure 5 Examples.

[0208] Video decoder 300 (e.g., via storage circuitry 314) can generate a list (500) of multiple future reference images associated with multiple future images based on a first set of images decoded prior to a second set of images. The second set of images includes the current image and multiple future images, which are decoded after the current image in the order they were decoded. For example, as per [reference to...] Figure 4A and Figure 4B As described, in order to generate a list of multiple future reference images, the video decoder 300 can determine corresponding previous images in a first image set 400A, which are located in the same relative decoding order position in the first image set 400A as the multiple future images are located in a second image set 400B. The video decoder 300 can determine the corresponding reference image list for the corresponding previous images, and generate a corresponding multiple future reference image list for each of the multiple future images based on the determined corresponding reference image list for the corresponding previous images.

[0209] The video decoder 300 may determine, based on information derived from a list of multiple future reference images associated with multiple future images, whether to write the current image to dedicated chip memory 324 or to non-dedicated system memory 326 (502). As an example, the video decoder 300 may select between dedicated chip memory and non-dedicated system memory to write the current image based on information derived from a list of multiple future reference images. In some examples, the video decoder 300 may determine whether to write the current image to dedicated chip memory 324 or non-dedicated system memory 326 before decoding the current image.

[0210] The video decoder 300 can derive information based on the number of times the current image is identified in multiple lists of future reference images. The video decoder 300 can also determine the number of times a cached image (e.g., an image stored in dedicated chip memory when the current image is about to be decoded) is identified in multiple lists of future reference images. The video decoder 300 can determine whether to write the current image to dedicated chip memory 324 or non-dedicated system memory 326 based on the number of times the current image and cached image are identified in multiple lists of future reference images, such as the number of times the current image and cached image are identified in index 0 of list 0 of future images, the number of times the current image and cached image are identified in index 0 of list 1 of future images, the number of times the current image and cached image are identified in index 1 of list 0 of future images, the number of times the current image and cached image are identified in index 1 of list 1 of future images, or any combination thereof, or any other index of list 0 or list 1 of future images (based on the programmed complexity level).

[0211] For example, video decoder 300 can derive a first metric from the number of times the current image is identified in a list of multiple future reference images associated with multiple future images, and derive a second metric associated with cached images stored in dedicated chip memory 324 based on the list of multiple future reference images associated with multiple future images. Video decoder 300 can compare the first metric with the second metric and determine, based on the comparison, whether to write the current image to dedicated chip memory 324 or non-dedicated system memory 326.

[0212] The video decoder 300 may write the current image to either the dedicated chip memory 324 or the non-dedicated system memory 326 based on a determination of whether to write the current image to the dedicated chip memory 324 or the non-dedicated system memory 326 (504). For example, the video decoder 300 may write the current image to either the dedicated chip memory 324 or the non-dedicated system memory 326. For example, if the video decoder 300 determines to write the current image to the dedicated chip memory 324, the video decoder 300 may clear the cached image and store the cached image in the non-dedicated system memory 326. The video decoder 300 may then write the current image to the dedicated chip memory 324. If the video decoder 300 determines to write the current image to the non-dedicated system memory 326, the video decoder 300 may write the current image to the non-dedicated system memory 326 and leave the cached image in the dedicated chip memory 324.

[0213] As an example, such as regarding Figure 3As described, the video decoder 300 can reconstruct the current image. To write the current image, the video decoder 300 can write the current image after or during its reconstruction. For example, the storage circuit 314 can determine that the current image needs to be written to the dedicated chip memory 324 before the current image is decoded. In this example, the storage circuit 314 can write blocks of the current image to the dedicated chip memory 324 while blocks of the current image are being reconstructed and output by the filter unit 312 or the reconstruction unit 310. That is, the video decoder 300 can write the current image after or during inter-frame predictive decoding of the current image (e.g., reconstruction of the current image based on a reference image).

[0214] Assume the current image is written to dedicated chip memory 324. In some cases, prediction processing unit 304 can determine that the current image is a reference image for a future image within a future image. Video decoder 300 can reconstruct the future image based on accessing the current image from dedicated chip memory 324.

[0215] Figure 6 This is a flowchart illustrating another example method for determining whether a current image is written to a dedicated chip memory or a non-dedicated system memory according to the technology of this disclosure. For simplicity, the video encoder 200 is described below. Figure 6 Examples.

[0216] Video encoder 200 (e.g., via storage circuitry 218) can generate a list of multiple future reference images (600) associated with multiple future images. Video encoder 200 can perform operations similar to those used by video decoder 300 to generate the list of multiple future reference images. However, the example techniques are not limited thereto. Because, unlike video decoder 300, video encoder 200 determines which images should be included in the reference image list based on quality and compression calculations, video encoder 200 may have already determined a list of reference images for the future reference image list. At runtime, video encoder 200 may rely on the list of future reference images already generated for future images for use with the example techniques described in this disclosure. Therefore, in one or more examples, the list of multiple future reference images may be a list of future reference images predicted (e.g., by video decoder 300) or determined (e.g., by video encoder 200) that identifies images that can be used as reference images for the multiple future images.

[0217] The video encoder 200 may determine, based on information derived from a list of multiple future reference images associated with multiple future images, whether to write the current image to dedicated chip memory 230 or to non-dedicated system memory 240 (602). The video encoder 200 may select between dedicated chip memory 230 and non-dedicated system memory 240 to write the current image based on information derived from the list of multiple future reference images. In some examples, the video encoder 200 may determine whether to write the current image to dedicated chip memory 230 or non-dedicated system memory 240 before encoding the current image.

[0218] The video encoder 200 can derive information based on the number of times the current image is identified in multiple lists of future reference images. The video encoder 200 can also determine the number of times a cached image (e.g., an image stored in dedicated chip memory when the current image is about to be decoded) is identified in multiple lists of future reference images. The video encoder 200 can determine whether to write the current image to dedicated chip memory 230 or non-dedicated system memory 240 based on the number of times the current image and cached image are identified in multiple lists of future reference images, such as the number of times the current image and cached image are identified in index 0 of list 0 of future images, the number of times the current image and cached image are identified in index 0 of list 1 of future images, the number of times the current image and cached image are identified in index 1 of list 0 of future images, the number of times the current image and cached image are identified in index 1 of list 1 of future images, or any combination thereof, or any other index of list 0 or list 1 of future images (based on the programmed complexity level).

[0219] For example, video encoder 200 can derive a first metric from the number of times the current image is identified in a list of multiple future reference images associated with multiple future images, and derive a second metric associated with cached images stored in dedicated chip memory 230 based on the list of multiple future reference images associated with multiple future images. Video decoder 300 can compare the first metric with the second metric and determine, based on the comparison, whether to write the current image to dedicated chip memory 230 or non-dedicated system memory 240.

[0220] The video encoder 200 may write the current image to either the dedicated chip memory 230 or the non-dedicated system memory 240 based on a determination of whether to write the current image to the dedicated chip memory 230 or to the non-dedicated system memory 240 (604). For example, the video encoder 200 may write the current image to either the dedicated chip memory 230 or the non-dedicated system memory 240. For example, if the video encoder 200 determines to write the current image to the dedicated chip memory 230, the video encoder 200 may clear the cached image and store the cached image in the non-dedicated system memory 240. The video encoder 200 may then write the current image to the dedicated chip memory 230. If the video encoder 200 determines to write the current image to the non-dedicated system memory 240, the video encoder 200 may write the current image to the non-dedicated system memory 240 and leave the cached image in the dedicated chip memory 230.

[0221] As an example, such as regarding Figure 2 As described, the video encoder 200 includes a reconstruction loop in which the encoded current image is also reconstructed (e.g., via inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, and filter unit 216 (optionally)). To write the current image, the video encoder 200 may write the current image after or during its reconstruction. For example, storage circuitry 218 may determine that the current image is to be written to dedicated chip memory 230 before it is encoded. In this example, storage circuitry 218 may write blocks of the current image to dedicated chip memory 230 while blocks of the current image are being reconstructed and output by filter unit 216 or reconstruction unit 214 in the reconstruction loop.

[0222] Assume the current image is written to dedicated chip memory 230. In some cases, mode selection unit 202 can determine that the current image is a reference image for a future image within a future image. Video encoder 200 can encode future images based on accessing the current image from dedicated chip memory 230.

[0223] Figure 7 This is a flowchart illustrating another example method for determining whether a current image is written to dedicated chip memory or non-dedicated system memory according to the technology of this disclosure. For simplicity, the video decoder 300 and video encoder 200 are described below. Figure 7 .

[0224] For example, video encoder 200 and video decoder 300 can generate multiple lists of future reference images (e.g., future reference image list 0 and / or list 1) associated with multiple future images in an image set (700). The image set includes the current image and multiple future images, and the multiple future images are ordered after the current image in decoding order. For example, video decoder 300 can generate the list of future reference images based on a previous image set. Video decoder 300 can utilize information about... Figure 4A , Figure 4B and Figure 5 The described technique is used to generate a list of future reference images. The video encoder 200 can generate the list of future reference images similarly to the video decoder 300, or it can generate the list of future reference images based on quality and compression calculations before runtime.

[0225] Video encoder 200 and video decoder 300 can determine future reference counts (702). Various examples of future reference count values ​​exist for the current image and cached images. Video encoder 200 and video decoder 300 can determine the future reference count of the current image based on the number of times the current image is identified in one or more entries of one or more future reference image lists, and determine the future reference count of the cached image based on the number of times the cached image is identified in one or more entries of one or more future reference image lists. Video encoder 200 and video decoder 300 can evaluate that the number of future reference image list entries used to determine the future reference count of the current image or cached image can be based on various factors.

[0226] As an example, the video encoder 200 and the video decoder 300 may determine a first reference count indicating the number of times the current image is identified in index N (e.g., index 0) of a list of future images, or determine a first reference count indicating the number of times the current image is identified in index N of a list of future images, or determine a first reference count indicating the number of times the current image is identified in both lists of future images 0 and 1.

[0227] In some examples, the video encoder 200 and the video decoder 300 may determine a second reference count indicating the number of times the current image is identified in index X (e.g., index 1) of list 0 of future images, or determine a second reference count indicating the number of times the current image is identified in index X of list 1 of future images, or determine a second reference count indicating the number of times the current image is identified in index X of both list 0 and list 1 of future images.

[0228] The video encoder 200 and video decoder 300 may determine one of the example first reference counts instead of one of the example second reference counts, or vice versa. The video encoder 200 and video decoder 300 may determine any combination of the first and second reference counts. The video encoder 200 and video decoder 300 may also determine additional reference counts indicating the number of times the current image is identified in a list of future images, list 0, list 1, or both lists 0 and list 1, at indices greater than N and X. All of these examples are contemplated in this disclosure.

[0229] Similarly, the video encoder 200 and the video decoder 300 may determine a third reference count indicating the number of times a cached image is identified in index N (e.g., index 0) of a list of future images, or determine a third reference count indicating the number of times a cached image is identified in index N of a list of future images, or determine a third reference count indicating the number of times a cached image is identified in index N of both lists of future images, 0 and 1.

[0230] In some examples, video encoder 200 and video decoder 300 may determine a fourth reference count indicating the number of times a cached image is identified in index X (e.g., index 1) of a list of future images, or determine a fourth reference count indicating the number of times a cached image is identified in index X of a list of future images, or determine a fourth reference count indicating the number of times a cached image is identified in index X of both lists of future images 0 and 1.

[0231] The video encoder 200 and video decoder 300 may determine one of the example third reference counts instead of one of the example fourth reference counts, or vice versa. The video encoder 200 and video decoder 300 may determine any combination of the third and fourth reference counts. The video encoder 200 and video decoder 300 may also determine additional reference counts indicating the number of times a cached image is identified in a list of future images, list 0, list 1, or both lists 0 and list 1, at indices greater than N and X. All of these examples are contemplated in this disclosure.

[0232] The video encoder 200 and video decoder 300 may determine a reference gain (704). In some examples, the reference gain value for the current image may be either a first reference count or a second reference count, or some combination of the first and second reference counts, or some combination of the first, second, or additional reference counts. The reference gain value for a cached image may be either a third or a fourth reference count, or some combination of the third and fourth reference counts, or some combination of the third, fourth, or additional reference counts.

[0233] For example, video encoder 200 and video decoder 300 may apply a first weight to a first reference count and a second weight to a second reference count to generate a reference gain value for the current image. Video encoder 200 and video decoder 300 may also apply a first weight to a third reference count and a second weight to a fourth reference count to generate a reference gain value for the cached image.

[0234] The video encoder 200 and video decoder 300 determine whether the reference gain value of the current image is greater than or less than the reference gain value of the cached image (706). If the reference gain value of the current image is greater than the reference gain value of the cached image ("Yes" in 706), the video encoder 200 and video decoder 300 may write the current image to dedicated chip memory 230 or 324 (708). If the reference gain value of the current image is less than the reference gain value of the cached image ("No" in 706), the video encoder 200 and video decoder 300 may write the current image to non-dedicated system memory 240 or 326 (710).

[0235] Figures 8 to 11 This is a flowchart illustrating an example method for determining whether a current image is written to a dedicated chip memory or a non-dedicated system memory according to the technology of this disclosure. For simplicity, example values ​​from the table below are used for description. Figures 8 to 11 In the table below, it is assumed that image B4 is the current image being encoded or decoded. Furthermore, it is assumed that image I0 is stored in dedicated chip memory.

[0236] The future images include images B2, B1, B3, and B6 (e.g., the search scope is four images) and are in the same image set as the current image B4. Additional images may also exist in the image set (e.g., in addition to the images forming the search scope). Assume that storage circuits 218 and 314, as described above, determine future reference image lists 0 and 1 for images B2, B1, B3, and B6, and the following table includes the images identified in the future reference image lists.

[0237] Table 1. List of Future Reference Images

[0238] about Figure 8The video encoder 200 and video decoder 300 can determine which reference image is identified at an index (e.g., entry) in the list of future reference images for each of the multiple future images (800). For example, the video encoder 200 and video decoder 300 can determine that image I0 is identified at index 0 (e.g., first entry) in list 0 of future image B2, image I0 is identified at index 0 (e.g., first entry) in list 0 of future image B1, image B2 is identified at index 0 (e.g., first entry) in list 0 of future image B3, and image B4 is identified at index 0 (e.g., first entry) in list 0 of future image B6. The video encoder 200 and video decoder 300 can similarly determine which reference image is identified at index 0, index 1 of list 0, and index 1 of list 1 in future images B2, B1, B3, and B6.

[0239] The video encoder 200 and video decoder 300 can determine the future reference count values ​​of one or more reference images based on which reference image is identified at the index (802). As an example, the video encoder 200 and video decoder 300 can determine the future reference count values ​​of the current image B4 and the cached image I0.

[0240] For example, one example of the future reference count value for the current image B4 is one, because the current image B4 is identified once in index 0 of list 0 of images B2, B1, B3, and B6 (e.g., identified in index 0 of list 0 of future image B6). Another example of the future reference count value for the current image B4 is two, because the current image B4 is identified twice in index 0 of list 1 of images B2, B1, B3, and B6 (e.g., identified in index 0 of list 1 of future images B2 and B3). Another example of the future reference count value for the current image B4 is three, because the current image B4 is identified three times in index 0 of list 0 or list 1 of images B2, B1, B3, and B6 (e.g., identified in index 0 of list 0 of future image B6 and in index 0 of list 1 of future images B2 and B3).

[0241] An example of a future reference count value for cached image I0 is two, because cached image I0 is identified twice in index 0 list 0 of images B2, B1, B3, and B6 (e.g., identified in index 0 list 0 of future images B2 and B1). Another example of a future reference count value for cached image I0 is zero, because cached image I0 is not identified in index 0 list 1 of images B2, B1, B3, and B6. Another example of a future reference count value for cached image I0 is two, because cached image I0 is identified twice in index 0 of either list 0 or list 1 of images B2, B1, B3, and B6 (e.g., identified in index 0 list 0 of future images B2 and B1).

[0242] The video encoder 200 and video decoder 300 can determine whether to write the current image to dedicated chip memory or non-dedicated system memory based on a future reference count value (804). As an example, the video encoder 200 and video decoder 300 can compare the future reference count values ​​of the current image B4 and the cached image I0, and determine whether to write the current image to dedicated chip memory or non-dedicated system memory based on the comparison.

[0243] As another example, the video encoder 200 and video decoder 300 may determine another metric, such as a reference gain value, based on a reference count value, and determine whether to write the current image to dedicated chip memory or non-dedicated system memory based on the reference gain value. As an example, the video encoder 200 and video decoder 300 may determine the number of times the current image B4 and cached image I0 are identified in index 1 list 0 or list 1 of future images B2, B1, B3, and B6. For example, the current image B4 is identified twice in index 1 list 0 or list 1 of future images B2, B1, B3, and B6 (e.g., index 1 list 0 of future image B2 and index 1 list 1 of future image B1). The cached image I0 is identified twice in index 1 list 0 or list 1 of future images B2, B1, B3, and B6 (e.g., index 1 list 0 of future images B3 and B6).

[0244] An example of the equation for the reference benefit of the current image B4 or the cached image I0 could be... In this equation, "i" refers to the index (e.g., the entry), L0 refers to list 0, and L1 refers to list 1.

[0245] In the above equation, the current image B4... This represents the number of times the current image B4 is identified in index i of the list of future images B2, B1, B3, and B6. The cached image I0... This is the number of times cached image I0 is identified in index i of the list of future images B2, B1, B3, and B6. In this example, the current image B4 is identified at index i=0. It is one, and the cached image I0 for i=0 It is two. The current image B4 for i=1... It is one, and the cached image I0 for i=1 It is two.

[0246] The current image B4 represents the number of times it is identified in index i of list 1 of future images B2, B1, B3, and B6. The cached image I0... This is the number of times cached image I0 is identified in index i of list 1 for future images B2, B1, B3, and B6. In this example, the current image B4 is identified for i=0. It is two, and the cached image I0 for i=0 It is zero. The current image B4 for i=1... It is one, and the cached image I0 for i=1 It is zero.

[0247] Equal to applied The weights. In some examples, For i=0, it equals 10, and for i=1, it equals 1. Equal to applied The weights. In some examples, For i=0, it equals 10, and for i=1, it equals 1. and These values ​​may have different values, and the values ​​above are just examples.

[0248] Therefore, for the current image B4, For i=0 equals And for i=1 equals Therefore, the current image B4... It's 11. For the current image B4, For i=0 equals And for i=1 equals Therefore, the current image B4... It is 21. Therefore, the reference benefit value for the current image B4 is 21. That is, 11 + 21 = 32.

[0249] In the above equation, the current image B4... This represents the number of times the current image B4 is identified in index i of the list of future images B2, B1, B3, and B6. The cached image I0... This is the number of times cached image I0 is identified in index i of the list of future images B2, B1, B3, and B6. In this example, the current image B4 is identified at index i=0. It is one, and the cached image I0 for i=0 It is two. The current image B4 for i=1... It is one, and the cached image I0 for i=1 It is two.

[0250] The current image B4 represents the number of times it is identified in index i of list 1 of future images B2, B1, B3, and B6. The cached image I0... This is the number of times cached image I0 is identified in index i of list 1 for future images B2, B1, B3, and B6. In this example, the current image B4 is identified for i=0. It is two, and the cached image I0 for i=0 It is zero. The current image B4 for i=1... It is one, and the cached image I0 for i=1 It is zero.

[0251] Equal to applied The weights. In some examples, For i=0, it equals 10, and for i=1, it equals 1. Equal to applied The weights. In some examples, For i=0, it equals 10, and for i=1, it equals 1. and These values ​​may have different values, and the values ​​above are just examples.

[0252] Therefore, for the current image B4, For i=0 equals And for i=1 equals Therefore, the current image B4... It's 11. For the current image B4, For i=0 equals And for i=1 equals Therefore, the current image B4... It is 21. Therefore, the reference benefit value for the current image B4 is 21. That is, 11 + 21 = 32.

[0253] Therefore, for cached image I0, For i=0 equals And for i=1 equals Therefore, cache image I0 It's 22. For cached image I0, For i=0 equals And for i=1 equals Therefore, cache image I0 It is 0. Therefore, the reference benefit value for cached image I0 is 0. That is, 22 + 0 = 22.

[0254] Based on the above, the reference gain value for the current image B4 is 32, and the reference gain value for the cached image I0 is 22. Because the reference gain value for the current image B4 is greater than the reference gain value for the cached image I0, the video encoder 200 and the video decoder 300 can determine to write the current image B4 into the dedicated chip memory instead of the non-dedicated system memory.

[0255] The equation above used to calculate the reference return value is an example, and other examples may exist. For instance, as some non-limiting examples, examples of the equation for the reference return value could be the following: Reference return value = Reference return value = Reference return value = Reference return value = Reference return value = In the example above, the video encoder 200 and the video decoder 300 consider reference images from both list 0 and list 1 of future images. However, the example technique is not limited to this. Figures 9 to 11 The examples described build upon each other. For example, Figure 9 The example uses only the entries in index 0 of list 0 to determine whether to write the current image B4 to dedicated chip memory or non-dedicated system memory. Figure 10 Based on Figure 9 Up, and Figure 10 The example uses entries in index 0, list 0, and list 1 to determine whether to write the current image B4 to dedicated chip memory or non-dedicated system memory. Figure 11 Based on Figure 10 Up, and Figure 11The example uses entries in indices 0 and 1 of lists 0 and 1 to determine whether to write the current image B4 to dedicated chip memory or non-dedicated system memory.

[0256] The video encoder 200 and video decoder 300 can be configured to perform Figures 9 to 11 Example techniques of either of them. Furthermore, although... Figure 9 Use list 0, but Figure 9 List 1 may be used.

[0257] refer to Figure 9 The video encoder 200 and video decoder 300 can generate a portion (900) of a corresponding future reference image list for each of a plurality of future images. For example, as shown in Table 1, the video encoder 200 and video decoder 300 can generate the first two entries of future reference image list 0 for future images B2, B1, B3, and B6. Therefore, in this disclosure, generating a future reference image list includes examples of generating a partial or complete future reference image list.

[0258] The video encoder 200 and the video decoder 300 may determine a first future reference count value (902) based on the number of times the current image B4 is identified in a portion of the list of future reference images. For example, considering only the index 0 list 0 of future images B2, B1, B3, and B6, the video encoder 200 and the video decoder 300 may determine that the first future reference count value of the current image B4 is equal to one.

[0259] The video encoder 200 and the video decoder 300 may determine a second future reference count value (904) based on the number of times the cached image I0 is identified in a portion of the future reference image list. For example, considering only the index 0 list 0 of future images B2, B1, B3, and B6, the video encoder 200 and the video decoder 300 may determine that the second future reference count value of the cached image I0 is equal to two.

[0260] The video encoder 200 and video decoder 300 can determine whether to write the current image B4 into dedicated chip memory or non-dedicated system memory based on a first reference count value and a second reference count value (906). For example, if the reference gain values ​​of the current image B4 and the cached image I0 are set to be equal to the first reference count and the second reference count, respectively, the video encoder 200 and video decoder 300 can determine to write the current image B4 into dedicated chip memory if the reference count value of the current image B4 is greater than the reference count value of the cached image I0. In some examples, the video encoder 200 and video decoder 300 can use either of the reference gain value equations described above to determine the reference gain value of the current image B4 or the cached image I0.

[0261] The video encoder 200 and video decoder 300 may write the current image B4 to dedicated chip memory or non-dedicated system memory based on a determination of whether to write the current image B4 to dedicated chip memory or non-dedicated system memory (908). For example, if it is determined to write to dedicated chip memory, the video encoder 200 and video decoder 300 may move the cached image I0 from dedicated chip memory to non-dedicated system memory and write the current image B4 to dedicated chip memory. Otherwise, the video encoder 200 and video decoder 300 may write the current image B4 to non-dedicated system memory.

[0262] exist Figure 9 In the example, only list 0 of future images B2, B1, B3, and B6 is considered. However, the example technique is not limited to this, and only list 1 of future images B2, B1, B3, and B6 may be considered. In some examples, the video encoder 200 and the video decoder 300 may consider both list 0 and list 1.

[0263] For example, Figure 10 Examples are built on Figure 9 The example above. The video encoder 200 and video decoder 300 can generate a portion (1000) of a second corresponding future reference image list for each of a plurality of future images. For example, as shown in Table 1, in addition to list 0, the video encoder 200 and video decoder 300 can also generate the first two entries of future reference image list 1 for future images B2, B1, B3, and B6. Therefore, in this disclosure, generating a future reference image list includes examples of generating a partial or complete future reference image list.

[0264] The video encoder 200 and video decoder 300 may determine a third future reference count value (1002) based on the number of times the current image B4 is identified in a portion of the second corresponding future reference image list. For example, the video encoder 200 and video decoder 300 may determine that the third future reference count value of the current image B4 is equal to two if only index 0 list 1 of future images B2, B1, B3, and B6 is considered, or equal to three if index 0 list 0 and list 1 of future images B2, B1, B3, and B6 are considered.

[0265] The video encoder 200 and the video decoder 300 may determine a fourth future reference count value (1004) based on the number of times the cached image I0 is identified in a portion of the second corresponding future reference image list. For example, the video encoder 200 and the video decoder 300 may determine that the fourth future reference count value of the cached image I0 is equal to zero if only index 0 list 1 of future images B2, B1, B3, and B6 is considered, or equal to two if index 0 list 0 and list 1 of future images B2, B1, B3, and B6 are considered.

[0266] The video encoder 200 and video decoder 300 can determine whether to write the current image B4 to dedicated chip memory or non-dedicated system memory based on a first reference count value, a second reference count value, a third reference count value, and a fourth reference count value (1006). For example, the video encoder 200 and video decoder 300 can determine a reference gain value for the current image B4 based on the first and third reference count values, and determine a reference gain value for the cached image I0 based on the second and fourth reference count values. The video encoder 200 and video decoder 300 can determine whether to write the current image B4 to dedicated chip memory or non-dedicated system memory based on the reference gain values ​​of the current image B4 and the cached image I0. In some examples, the video encoder 200 and video decoder 300 can use either of the reference gain value equations described above to determine the reference gain value of the current image B4 or the cached image I0.

[0267] The video encoder 200 and video decoder 300 can write the current image B4 to dedicated chip memory or non-dedicated system memory based on a determination of whether to write the current image B4 to dedicated chip memory or non-dedicated system memory (1008). For example, if it is determined to write to dedicated chip memory, the video encoder 200 and video decoder 300 can move the cached image I0 from dedicated chip memory to non-dedicated system memory and write the current image B4 to dedicated chip memory. Otherwise, the video encoder 200 and video decoder 300 can write the current image B4 to non-dedicated system memory.

[0268] refer to Figure 11 The video encoder 200 and video decoder 300 can determine the first future count value (1100) of the current image B4 based on the number of times the current image B4 is identified in index 0 (e.g., the first entry) of multiple first future reference image lists (e.g., list 0) or second future reference image lists (e.g., list 1) of multiple future images. For example, the first future reference count value of the current image B4 is three because the current image B4 is identified three times in index 0 list 0 and index 0 list 1 in Table 1.

[0269] The video encoder 200 and video decoder 300 may determine a second future count value (1102) of the current image B4 based on the number of times the current image B4 is identified in index 1 (e.g., second entry) of multiple first future reference image lists (e.g., list 0) or second future reference image lists (e.g., list 1) of multiple future images. For example, the second future reference count value of the current image B4 is two because the current image B4 is identified twice in index 1 list 0 and index 1 list 1 in Table 1.

[0270] The video encoder 200 and video decoder 300 can determine a first reference gain value (1104) for the current image B4 based on a weighted average of a first future count value and a second future count value. For example, suppose the first weight of the first future count value is 10, and the second weight of the second future count value is 1. In this example, the video encoder 200 and video decoder 300 can multiply the first weight by the first reference count value of the current image B4 (e.g., The second weight is multiplied by the second reference count value of the current image B4 (e.g., ), and add the results together (for example, 30 + 2 = 32).

[0271] The video encoder 200 and the video decoder 300 may determine a third future count value (1106) of cached image I0 based on the number of times it is identified in index 0 (e.g., the first entry) of multiple first future reference image lists (e.g., list 0) or second future reference image lists (e.g., list 1) of multiple future images. For example, the third future reference count value of cached image I0 is two because cached image I0 is identified twice in index 0 list 0 and index 0 list 1 in Table 1.

[0272] The video encoder 200 and the video decoder 300 may determine a fourth future count value (1108) of cached image I0 based on the number of times it is identified in index 1 (e.g., second entry) of multiple first future reference image lists (e.g., list 0) or second future reference image lists (e.g., list 1) of multiple future images. For example, the fourth future reference count value of cached image I0 is two because cached image I0 is identified twice in index 1 list 0 and index 1 list 1 of Table 1.

[0273] The video encoder 200 and video decoder 300 can determine a second reference gain value (1110) for the cached image I0 based on a corresponding weighted average of a third future count value and a fourth future count value. For example, suppose the third weight of the third future count value is 10, and the fourth weight of the fourth future count value is 1. In this example, the video encoder 200 and video decoder 300 can multiply the third weighted average by the third reference count value of the cached image I0 (e.g., ), multiply the fourth weighted value by the fourth reference count value of the cached image I0 (e.g., ), and add the results together (for example, 20+2=22).

[0274] The video encoder 200 and video decoder 300 can determine whether to write the current image B4 into dedicated chip memory or non-dedicated system memory based on a first reference gain value and a second reference gain value (1112). In this example, because the first reference gain value of the current image B4 is greater than the second reference gain value of the cached image I0, the video encoder 200 and video decoder 300 can determine to write the current image B4 into dedicated chip memory.

[0275] The following example techniques can be used alone or in any combination.

[0276] Clause 1A. A method for processing video data, the method comprising: generating a list of multiple future reference images associated with a plurality of future images in a set of images, wherein the set of images includes a current image and the plurality of future images, and the plurality of future images are arranged in a decoding order following the current image; determining, based on information derived from the list of multiple future reference images associated with the plurality of future images, whether to write the current image into a dedicated chip memory or into a non-dedicated system memory; and writing the current image into the dedicated chip memory or the non-dedicated system memory based on the determination of whether to write the current image into the dedicated chip memory or into the non-dedicated system memory.

[0277] Clause 2A. The method according to Clause 1A, wherein the set of pictures includes a second set of pictures, wherein generating the plurality of future reference picture lists comprises: generating the plurality of future reference picture lists associated with the plurality of future pictures based on a first set of pictures that has been decoded prior to the second set of pictures, and wherein writing the current picture comprises: writing the current picture after or during inter-frame prediction decoding of the current picture.

[0278] Clause 3A. The method according to Clause 2A, wherein generating the plurality of future reference image lists comprises: determining a corresponding previous image in the first image set, the corresponding previous image being located in the first image set at a relative decoding order position that is the same as the relative decoding order position of the plurality of future images in the second image set; determining a corresponding reference image list for the corresponding previous image; and generating a corresponding plurality of future reference image lists for each of the plurality of future images based on the determined corresponding reference image lists for the corresponding previous images.

[0279] Clause 4A. The method according to any one of Clauses 1A to 3A, wherein the set of images includes the current image, the plurality of future images, and one or more additional images.

[0280] Clause 5A. The method according to any one of Clauses 1A to 4A, wherein determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory comprises: determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory before encoding or decoding the current image.

[0281] Clause 6A. The method according to any one of Clauses 1A to 5A, the method further comprising: deriving the information based on the number of times the current image is identified in the plurality of future reference image lists.

[0282] Clause 7A. The method of any one of Clauses 1A to 6A, wherein the information includes a first metric derived from the number of times the current image is identified in a plurality of future reference images associated with the plurality of future images, the method further comprising: deriving a second metric associated with a cached image stored in the dedicated chip memory based on the plurality of future reference images associated with the plurality of future images; and comparing the first metric with the second metric, wherein determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory comprises: determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory based on the comparison of the first metric with the second metric.

[0283] Clause 8A. The method according to Clause 7A, wherein determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory comprises: determining to write the current image to the dedicated chip memory based on the first metric being greater than the second metric.

[0284] Clause 9A. The method according to any one of Clauses 1A to 8A, the method further comprising: reconstructing the current image, wherein writing the current image includes writing the current image after or during the reconstruction of the current image.

[0285] Clause 10A. The method according to Clause 9A, wherein reconstructing the current image comprises: reconstructing the current image as part of a reconstruction loop of video coding.

[0286] Clause 11A. The method according to any one of Clauses 1A to 10A, wherein determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory includes determining that the current image is written to the dedicated chip memory, the method further comprising: determining that the current image is a reference image of a future image in a future image; and reconstructing the future image based on accessing the current image from the dedicated chip memory.

[0287] Clause 12A. An apparatus for processing video data, the apparatus comprising: a dedicated-chip memory; and processing circuitry coupled to the dedicated-chip memory and configured to: generate a list of a plurality of future reference images associated with a plurality of future images in a set of images, wherein the set of images includes a current image and the plurality of future images, and the plurality of future images are arranged in a decoding order following the current image; determine, based on information derived from the list of a plurality of future reference images associated with the plurality of future images, whether to write the current image into the dedicated-chip memory or into a non-dedicated system memory; and write the current image into the dedicated-chip memory or into the non-dedicated system memory based on the determination of whether to write the current image into the dedicated-chip memory or into the non-dedicated system memory.

[0288] Clause 13A. The apparatus according to Clause 12A, wherein the image set includes a second image set, wherein, in order to generate the plurality of future reference image lists, the processing circuitry is configured to generate the plurality of future reference image lists associated with the plurality of future images based on a first image set decoded prior to the second image set, and wherein, in order to write the current image, the processing circuitry is configured to write the current image after or during inter-frame prediction decoding of the current image.

[0289] Clause 14A. The apparatus according to Clause 13A, wherein, in order to generate the plurality of future reference image lists, the processing circuitry is configured to: determine a corresponding previous image in the first image set, the corresponding previous image being located in the first image set at a relative decoding order position that is the same as the relative decoding order position of the plurality of future images in the second image set; determine a corresponding reference image list for the corresponding previous image; and generate a corresponding plurality of future reference image lists for each of the plurality of future images based on the determined corresponding reference image lists for the corresponding previous images.

[0290] Clause 15A. The device pursuant to any one of Clauses 12A to 14A, wherein the image set includes the current image, the plurality of future images, and one or more additional images.

[0291] Clause 16A. The device according to any one of Clauses 12A to 15A, wherein, in order to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory, the processing circuitry is configured to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory before encoding or decoding the current image.

[0292] Clause 17A. The device according to any one of Clauses 12A to 16A, wherein the processing circuitry is configured to derive the information based on the number of times the current image is identified in the plurality of future reference image lists.

[0293] Clause 18A. The device according to any one of Clauses 12A to 17A, wherein the information includes a first metric derived from the number of times the current image is identified in a plurality of future reference images associated with the plurality of future images, and wherein the processing circuitry is configured to: derive a second metric associated with a cached image stored in the dedicated chip memory based on the plurality of future reference images associated with the plurality of future images; and compare the first metric with the second metric, wherein, in order to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory, the processing circuitry is configured to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory based on the comparison of the first metric with the second metric.

[0294] Clause 19A. The device according to Clause 18A, wherein, in order to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory, the processing circuitry is configured to determine whether to write the current image to the dedicated chip memory based on the first metric being greater than the second metric.

[0295] Clause 20A. The device according to any one of Clauses 12A to 19A, wherein the processing circuitry is configured to: reconstruct the current image, wherein, in order to write the current image, the processing circuitry is configured to write the current image after or during the reconstruction of the current image.

[0296] Clause 21A. The apparatus according to Clause 20A, wherein, in order to reconstruct the current image, the processing circuitry is configured to reconstruct the current image as part of a reconstruction loop of video encoding.

[0297] Clause 22A. The device according to any one of Clauses 12A to 21A, wherein, in order to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory, the processing circuitry is configured to determine that the current image should be written to the dedicated chip memory, and wherein the processing circuitry is configured to: determine that the current image is a reference image of a future image in a future image; and reconstruct the future image based on accessing the current image from the dedicated chip memory.

[0298] Clause 23A. One or more computer-readable storage media storing instructions that, when executed, cause one or more processors to: generate a list of multiple future reference images associated with a plurality of future images in a set of images, wherein the set of images includes a current image and the plurality of future images, and the plurality of future images are arranged in a decoding order following the current image; determine, based on information derived from the list of multiple future reference images associated with the plurality of future images, whether to write the current image to a dedicated chip memory or to a non-dedicated system memory; and, based on the determination of whether to write the current image to the dedicated chip memory or to the non-dedicated system memory, write the current image to the dedicated chip memory or to the non-dedicated system memory.

[0299] Clause 24A. One or more computer-readable storage media as described in Clause 23A, wherein the set of pictures includes a second set of pictures, wherein the instructions for causing the one or more processors to generate the plurality of future reference picture lists include instructions for causing the one or more processors to generate the plurality of future reference picture lists associated with the plurality of future pictures based on a first set of pictures decoded prior to the second set of pictures, and wherein the instructions for causing the one or more processors to write the current picture include instructions for causing the one or more processors to write the current picture after or during inter-frame prediction decoding of the current picture.

[0300] Clause 25A. One or more computer-readable storage media as described in Clause 24A, wherein the instructions for causing the one or more processors to generate the plurality of future reference image lists include instructions for causing the one or more processors to: determine a corresponding previous image in the first image set, the corresponding previous image being located in the first image set at a relative decoding order position that is the same as the relative decoding order position of the plurality of future images in the second image set; determine a corresponding reference image list for the corresponding previous image; and generate a corresponding plurality of future reference image lists for each of the plurality of future images based on the determined corresponding reference image lists for the corresponding previous images.

[0301] Clause 26A. One or more computer-readable storage media according to any one of Clauses 23A to 25A, wherein the instructions causing the one or more processors to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory include instructions causing the one or more processors to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory before encoding or decoding the current image.

[0302] Clause 27A. One or more computer-readable storage media according to any one of Clauses 23A to 26A, the one or more computer-readable storage media further comprising instructions to cause the one or more processors to derive the information based on the number of times the current image is identified in the plurality of future reference image lists.

[0303] Clause 28A. One or more computer-readable storage media according to any one of Clauses 23A to 27A, wherein the information includes a first metric derived from the number of times the current image is identified in a plurality of future reference images associated with the plurality of future images, and wherein the instructions further include instructions to cause the one or more processors to: derive a second metric associated with a cached image stored in the dedicated-chip memory based on the plurality of future reference images associated with the plurality of future images; and compare the first metric with the second metric, wherein the instructions to cause the one or more processors to determine whether to write the current image to the dedicated-chip memory or the non-dedicated-system memory include instructions to cause the one or more processors to: determine whether to write the current image to the dedicated-chip memory or the non-dedicated-system memory based on the comparison of the first metric with the second metric.

[0304] Clause 29A. One or more computer-readable storage media storing instructions that, when executed, cause one or more processors to perform the method according to any one of Clauses 1A to 11A.

[0305] Clause 30A. An apparatus for processing video data, the apparatus comprising: means for generating a list of a plurality of future reference images associated with a plurality of future images in a set of images, wherein the set of images includes a current image and the plurality of future images, and the plurality of future images are arranged in a decoding order following the current image; means for determining, based on information derived from the list of the plurality of future reference images associated with the plurality of future images, whether to write the current image into a dedicated chip memory or into a non-dedicated system memory; and means for writing the current image into the dedicated chip memory or the non-dedicated system memory based on the determination of whether to write the current image into the dedicated chip memory or into the non-dedicated system memory.

[0306] Clause 31A. The apparatus according to Clause 30A, the apparatus further comprising components for performing the method according to any one of Clauses 1A to 11A.

[0307] Clause 1B. A method for processing video data, the method comprising: generating a portion of a corresponding reference image list for each of a plurality of future images, the plurality of future images being ordered after a current image in a decoding sequence; determining a first future reference count value based on the number of times the current image is identified in the portion of the corresponding reference image list of the plurality of future images; determining a second future reference count value based on the number of times a cached image stored in a dedicated chip memory is identified in the portion of the corresponding reference image list of each of the plurality of future images; determining, at least based on the first future reference count value and the second future reference count value, whether to write the current image to the dedicated chip memory or to a non-dedicated system memory; and writing the current image to the dedicated chip memory or the non-dedicated system memory based on the determination of whether to write the current image to the dedicated chip memory or to the non-dedicated system memory.

[0308] Clause 2B. The method according to Clause 1B, wherein the current image and the plurality of future images are part of a current image set having a defined number of images, the method further comprising: determining a corresponding previous image in a set of previous images, the corresponding previous image in the set of previous images being located at a relative decoding order position that is the same as the relative decoding order position of each of the plurality of future images in the current image set, wherein the set of previous images includes the defined number of images, and wherein the images in the set of previous images are encoded or decoded prior to the images in the current image set; determining said portion of a corresponding reference image list of the corresponding previous image in the set of previous images; and generating said portion of the corresponding reference image list of each of the plurality of future images based on the determined portion of the corresponding reference image list of the corresponding previous image in the set of previous images.

[0309] Clause 3B. The method according to any one of Clauses 1B and 2B, wherein the portion of the corresponding reference image list includes a corresponding reference image identified at a first index in the corresponding reference image list, the method further comprising: determining which reference image is identified at the first index in the corresponding reference image list for each of the plurality of future images; determining a first future reference count value based on the number of times the current image is identified at the first index in the corresponding reference image list for each of the plurality of future images; and determining a second future reference count value based on the number of times the cached image is identified at the first index in the corresponding reference image list for each of the plurality of future images.

[0310] Clause 4B. The method according to any one of Clauses 1B to 3B, the method further comprising: determining a first reference gain value for the current image based on a first future reference count value, the first reference gain value indicating a prediction frequency at which the current image will be used for inter-frame prediction; determining a second reference gain value for the cached image based on a second future reference count value, the second reference gain value indicating a prediction frequency at which the cached image will be used for inter-frame prediction; wherein determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory comprises: determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory based on the first reference gain value or the second reference gain value.

[0311] Clause 5B. The method according to any one of Clauses 1B to 4B, wherein the corresponding reference image list is a first corresponding reference image list, the method further comprising: generating a portion of a second corresponding reference image list for each of the plurality of future images; determining a third future reference count value based on the number of times the current image is identified in the portion of the second corresponding reference image list for each of the plurality of future images; determining a fourth future reference count value based on the number of times the cached image is identified in the portion of the second corresponding reference image list for each of the plurality of future images; and determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory based at least on the first future reference count value, the second future reference count value, the third future reference count value, and the fourth future reference count value.

[0312] Clause 6B. The method according to any one of Clauses 1B to 5B, wherein the portion of the corresponding reference image list includes a corresponding reference image identified at a first index in the corresponding reference image list and a corresponding reference image identified at a second index in the corresponding reference image list, the method further comprising: determining a first future reference count value based on the number of times the current image is identified at the first index in the corresponding reference image list for each of the plurality of future images; determining a second future reference count value based on the number of times the cached image is identified at the first index in the corresponding reference image list for each of the plurality of future images; determining a third future reference count value based on the number of times the current image is identified at the second index in the corresponding reference image list for each of the plurality of future images; determining a fourth future reference count value based on the number of times the cached image is identified at the second index in the corresponding reference image list for each of the plurality of future images; and determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory based at least on the first future reference count value, the second future reference count value, the third future reference count value, and the fourth future reference count value.

[0313] Clause 7B. The method according to Clause 6B, the method further comprising: generating a portion of a second corresponding reference image list for each of the plurality of future images, said portion of the second corresponding reference image list including a corresponding reference image identified at a first index in the second corresponding reference image list and a corresponding reference image identified at a second index in the second corresponding reference image list; determining a fifth future reference count value based on the number of times the current image is identified at the first index in the second corresponding reference image list; and determining a sixth future reference count value based on the number of times the cached image is identified at the first index in the second corresponding reference image list. A seventh future reference count value is determined based on the number of times the current image is identified at the second index in the second corresponding reference image list; an eighth future reference count value is determined based on the number of times the cached image is identified at the second index in the second corresponding reference image list; and a determination is made, based at least on the first future reference count value, the second future reference count value, the third future reference count value, the fourth reference count value, the fifth future reference count value, the sixth future reference count value, the seventh future reference count value, and the eighth future reference count value, whether to write the current image to the dedicated chip memory or the non-dedicated system memory.

[0314] Clause 8B. The method according to Clause 7B, the method further comprising: applying a corresponding weight to each of the first future reference count value, the third future reference count value, the fifth future reference count value, and the seventh future reference count value to determine a first reference gain value for the current image, the first reference gain value indicating the predicted frequency of accessing the current image from the dedicated chip memory; applying a corresponding weight to each of the second future reference count value, the fourth future reference count value, the sixth future reference count value, and the eighth future reference count value to determine a second reference gain value for the cached image, the second reference gain value indicating the predicted frequency of accessing the cached image from the dedicated chip memory; and determining, at least based on the first future reference count value and the second future reference count value, whether to write the current image to the dedicated chip memory or the non-dedicated system memory.

[0315] Clause 9B. The method according to any one of Clauses 1B to 8B, the method further comprising: storing the current image in the dedicated chip memory; determining that the current image stored in the dedicated chip memory is used for inter-frame predictive coding or decoding of future images among the plurality of future images; retrieving the current image from the dedicated chip memory; and performing inter-frame predictive coding or decoding of the future images based on the retrieved current image.

[0316] Clause 10B. An apparatus including a memory and processing circuitry, said apparatus being configured to perform a method according to any one of Clauses 1B to 9B.

[0317] Clause 11B. A computer-readable storage medium having instructions thereon stored thereon, which, when executed, cause one or more processors to perform the method according to any one of Clauses 1B to 9B.

[0318] It should be recognized that, based on the examples, certain actions or events of any technique described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary for implementing the technique). Furthermore, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt handling, or multiple processors) rather than sequentially.

[0319] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0320] By way of example, and not limitation, such computer-readable storage media may include one or more of RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (such as infrared, radio, and microwave) is included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. Combinations of these should also be included within the scope of computer-readable media.

[0321] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0322] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Rather, as described above, various units may be combined in a codec hardware unit, or various units may be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0323] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for processing video data, the method comprising: Generate a list of multiple future reference images associated with multiple future images in an image set, wherein the image set includes the current image and the multiple future images, and the multiple future images are ordered after the current image in the decoding order; It is determined whether to write the current image into a dedicated chip memory or into a non-dedicated system memory based on information derived from a list of multiple future reference images associated with the multiple future images; as well as Based on the determination of whether to write the current image to the dedicated chip memory or to the non-dedicated system memory, the current image is written to either the dedicated chip memory or the non-dedicated system memory.

2. The method according to claim 1, wherein the image set includes a second image set, and generating the plurality of future reference image lists includes: The plurality of future reference images are generated based on a first set of images that have been decoded prior to the second set of images, and the writing of the current image includes writing the current image after or during the inter-frame prediction decoding of the current image.

3. The method according to claim 2, wherein generating the plurality of future reference image lists comprises: Determine a corresponding previous image in the first image set, wherein the corresponding previous image is located in the first image set at the same relative decoding order position as the plurality of future images are located in the second image set; Determine the corresponding reference image list for the aforementioned previous image; as well as Based on the determined list of corresponding reference images of the corresponding previous images, a list of corresponding future reference images is generated for each of the plurality of future images.

4. The method according to claim 1, wherein the image set includes the current image, the plurality of future images, and one or more additional images.

5. The method of claim 1, wherein determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory comprises: Before encoding or decoding the current image, it is determined whether to write the current image into the dedicated chip memory or the non-dedicated system memory.

6. The method according to claim 1, further comprising: The information is derived based on the number of times the current image is identified in the list of multiple future reference images.

7. The method of claim 1, wherein the information includes a first metric derived from the number of times the current image is identified in a plurality of future reference images associated with the plurality of future images, the method further comprising: A second metric associated with cached images stored in the dedicated chip memory is derived based on the list of multiple future reference images associated with the multiple future images; as well as Compare the first metric with the second metric. Determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory includes: determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory based on the comparison of the first metric and the second metric.

8. The method of claim 7, wherein determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory comprises: Based on the fact that the first metric is greater than the second metric, it is determined that the current image will be written into the dedicated chip memory.

9. The method according to claim 1, further comprising: Reconstruct the current image. Writing the current image includes writing the current image after or during the reconstruction of the current image.

10. The method of claim 9, wherein reconstructing the current image comprises: The current image is reconstructed as part of the video encoding reconstruction loop.

11. The method of claim 1, wherein determining whether to write the current image to the dedicated chip memory or the non-dedicated system memory includes determining to write the current image to the dedicated chip memory, the method further comprising: The current image is determined to be a reference image for a future image within a future image; as well as The future image is reconstructed based on accessing the current image from the dedicated chip memory.

12. An apparatus for processing video data, the apparatus comprising: Dedicated chip memory; and Processing circuitry, coupled to the dedicated chip memory and configured to: Generate a list of multiple future reference images associated with multiple future images in an image set, wherein the image set includes the current image and the multiple future images, and the multiple future images are ordered after the current image in the decoding order; It is determined whether to write the current image into the dedicated chip memory or into the non-dedicated system memory based on information derived from the list of multiple future reference images associated with the multiple future images; as well as Based on the determination of whether to write the current image to the dedicated chip memory or to the non-dedicated system memory, the current image is written to either the dedicated chip memory or the non-dedicated system memory.

13. The device of claim 12, wherein the image set includes a second image set, wherein, in order to generate the plurality of future reference image lists, the processing circuitry is configured to generate the plurality of future reference image lists associated with the plurality of future images based on a first image set decoded prior to the second image set, and wherein, in order to write the current image, the processing circuitry is configured to write the current image after or during inter-frame prediction decoding of the current image.

14. The device of claim 13, wherein, in order to generate the plurality of future reference image lists, the processing circuitry is configured to: Determine a corresponding previous image in the first image set, wherein the corresponding previous image is located in the first image set at the same relative decoding order position as the plurality of future images are located in the second image set; Determine the corresponding reference image list for the aforementioned previous image; as well as Based on the determined list of corresponding reference images of the corresponding previous images, a list of corresponding future reference images is generated for each of the plurality of future images.

15. The device of claim 12, wherein the image set includes the current image, the plurality of future images, and one or more additional images.

16. The device of claim 12, wherein, in order to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory, the processing circuitry is configured to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory before encoding or decoding the current image.

17. The device of claim 12, wherein the processing circuitry is configured to derive the information based on the number of times the current image is identified in the plurality of future reference image lists.

18. The device of claim 12, wherein the information includes a first metric derived from the number of times the current image is identified in the plurality of future reference images associated with the plurality of future images, and wherein the processing circuitry is configured to: A second metric associated with cached images stored in the dedicated chip memory is derived based on the list of multiple future reference images associated with the multiple future images; and Compare the first metric with the second metric. In order to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory, the processing circuit is configured to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory based on the comparison of the first metric and the second metric.

19. The device of claim 18, wherein, in order to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory, the processing circuitry is configured to determine to write the current image to the dedicated chip memory based on the first metric being greater than the second metric.

20. The apparatus of claim 12, wherein the processing circuit is configured to: Reconstruct the current image. In order to write the current image, the processing circuit is configured to write the current image after or during the reconstruction of the current image.

21. The apparatus of claim 20, wherein, in order to reconstruct the current image, the processing circuitry is configured to reconstruct the current image as part of a reconstruction loop of video encoding.

22. The device of claim 12, wherein, in order to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory, the processing circuitry is configured to determine that the current image is written to the dedicated chip memory, and wherein the processing circuitry is configured to: Determine that the current image is a reference image for a future image within a future image; and The future image is reconstructed based on accessing the current image from the dedicated chip memory.

23. One or more computer-readable storage media storing instructions that, when executed, cause one or more processors to: Generate a list of multiple future reference images associated with multiple future images in an image set, wherein the image set includes the current image and the multiple future images, and the multiple future images are ordered after the current image in the decoding order; It is determined whether to write the current image into a dedicated chip memory or into a non-dedicated system memory based on information derived from a list of multiple future reference images associated with the multiple future images; as well as Based on the determination of whether to write the current image to the dedicated chip memory or to the non-dedicated system memory, the current image is written to either the dedicated chip memory or the non-dedicated system memory.

24. The one or more computer-readable storage media of claim 23, wherein the set of images includes a second set of images, wherein the instructions for causing the one or more processors to generate the plurality of future reference image lists include instructions for causing the one or more processors to: generate the plurality of future reference image lists associated with the plurality of future images based on a first set of images decoded prior to the second set of images, and wherein the instructions for causing the one or more processors to write the current image include instructions for causing the one or more processors to: write the current image after or during inter-frame prediction decoding of the current image.

25. The one or more computer-readable storage media of claim 24, wherein the instructions for causing the one or more processors to generate the plurality of future reference picture lists include instructions for causing the one or more processors to perform the following operations: Determine a corresponding previous image in the first image set, wherein the corresponding previous image is located in the first image set at the same relative decoding order position as the plurality of future images are located in the second image set; Determine the corresponding reference image list for the aforementioned previous image; as well as Based on the determined list of corresponding reference images of the corresponding previous images, a list of corresponding future reference images is generated for each of the plurality of future images.

26. The one or more computer-readable storage media of claim 23, wherein the instructions for causing the one or more processors to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory include instructions for causing the one or more processors to: determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory before encoding or decoding the current image.

27. The one or more computer-readable storage media of claim 23, further comprising instructions for causing the one or more processors to derive the information based on the number of times the current image is identified in the plurality of future reference image lists.

28. One or more computer-readable storage media of claim 23, wherein the information includes a first metric derived from the number of times the current image is identified in a plurality of future reference images associated with the plurality of future images, and wherein the instructions further include instructions to cause the one or more processors to perform the following operations: A second metric associated with cached images stored in the dedicated chip memory is derived based on the list of multiple future reference images associated with the multiple future images; and Compare the first metric with the second metric. The instructions that cause the one or more processors to determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory include instructions that cause the one or more processors to perform the following operation: determine whether to write the current image to the dedicated chip memory or the non-dedicated system memory based on the comparison of the first metric and the second metric.

Images