Variable intra (I-frame) time interval and picture group (GOP) length for video coding

Abstract

Systems and techniques for processing video data are provided. For example, a process may include obtaining a frame of video data associated with a display of a computing device, where the frame of video data includes one or more layers. Layer information associated with the one or more layers included in the frame of video data may be compared to layer information associated with one or more layers included in a previous frame of video data. Based on determining a frame geometry change associated with the video data frame, an inter prediction frame may be generated using the video data frame. An updated group of pictures (GOP) length may be determined based on the layer information associated with the one or more layers included in the frame of video data.

Description

Technical Field

[0001] The present disclosure generally relates to video coding (eg, including encoding and / or decoding of video data). For example, aspects of the present disclosure relate to improving video coding techniques associated with variable intra-frame time intervals and / or group of pictures (GOP) lengths. Background Art

[0002] Digital video capabilities can be incorporated into a variety of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones (so-called "smart phones"), video teleconferencing devices, video streaming devices, etc. Such devices allow video data to be processed and output for consumption. Digital video data includes a large amount of data to meet the needs of consumers and video providers. For example, consumers of video data expect the highest quality video, with high fidelity, high resolution, high frame rate, etc. As a result, the large amount of video data required to meet these needs burdens the communication networks and devices that process and store video data.

[0003] Digital video devices can implement video decoding techniques to compress video data. Video decoding is performed according to one or more video decoding standards or formats. For example, video decoding standards or formats include universal video coding (VVC), high-efficiency video coding (HEVC), advanced video coding (AVC), MPEG-2 Part 2 decoding (MPEG stands for Moving Picture Experts Group), etc., and proprietary video codecs / formats, such as AOMedia Video 1 (AV1) developed by the Open Media Alliance. Video decoding generally utilizes prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize redundancy present in video images or sequences. The goal of video decoding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. As evolving video services become available, decoding techniques with better decoding efficiency are needed. Summary of the invention

[0004] In some examples, systems and techniques for performing video decoding using variable intra-frame (I-frame) time intervals and / or variable-length group of pictures (GOP) lengths are described. For example, the systems and techniques may perform video decoding (e.g., encoding and / or decoding) using variable I-frame intervals and / or variable GOP lengths determined based on information such as video frame layer information, video frame layer geometry, and the like. According to at least one illustrative example, a method for processing video data is provided. The method includes: obtaining a video data frame associated with a display of a computing device, wherein the video data frame includes one or more layers; comparing layer information associated with one or more layers included in the video data frame with layer information associated with one or more layers included in a previous video data frame; generating an inter-frame prediction frame using the video data frame based on determining a frame geometry change associated with the video data frame; and determining an updated group of pictures (GOP) length based on layer information associated with one or more layers included in the video data frame.

[0005] In another example, an apparatus is provided, the apparatus comprising at least one memory (e.g., configured to store data) and at least one processor (e.g., implemented in a circuit), the at least one processor coupled to the at least one memory. The at least one processor is configured and may: obtain a video data frame associated with a display of a computing device, wherein the video data frame includes one or more layers; compare layer information associated with one or more layers included in the video data frame with layer information associated with one or more layers included in a previous video data frame; generate an inter-frame prediction frame using the video data frame based on determining a frame geometry change associated with the video data frame; and determine an updated group of pictures (GOP) length based on the layer information associated with one or more layers included in the video data frame.

[0006] In another example, a non-transitory computer-readable medium is provided, having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: obtain a video data frame associated with a display of a computing device, wherein the video data frame includes one or more layers; compare layer information associated with one or more layers included in the video data frame with layer information associated with one or more layers included in a previous video data frame; generate an inter-frame prediction frame using the video data frame based on determining a frame geometry change associated with the video data frame; and determine an updated group of pictures (GOP) length based on the layer information associated with one or more layers included in the video data frame.

[0007] In another example, an apparatus is provided that includes: a component for obtaining a video data frame associated with a display of a computing device, wherein the video data frame includes one or more layers; a component for comparing layer information associated with one or more layers included in the video data frame and layer information associated with one or more layers included in a previous video data frame; a component for generating an inter-frame prediction frame using the video data frame based on determining a frame geometry change associated with the video data frame; and a component for determining an updated group of pictures (GOP) length based on the layer information associated with one or more layers included in the video data frame.

[0008] In some aspects, one or more of the devices described herein are, are part of, and / or include a mobile device or wireless communication device (e.g., a mobile phone or other mobile device), an extended reality (XR) device or system (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a wearable device (e.g., a network-connected watch or other wearable device), a camera, a personal computer, a laptop computer, a vehicle or a computing device or component of a vehicle, a server computer or server device, another device, or a combination thereof. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the above-mentioned devices may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 is a block diagram illustrating examples of encoding devices and decoding devices according to some examples of the present disclosure;

[0013] Figure 2A is a diagram illustrating an example of an angular prediction mode according to some examples;

[0014] Figure 2B is a diagram illustrating an example of a directional intra prediction mode in Versatile Video Coding (VVC) according to some examples;

[0015] Figure 3 is a diagram illustrating examples of group of pictures (GOP) length and inter-frame (I-frame) time interval according to some examples;

[0016] Figure 4A is a diagram illustrating an example of a frame layer associated with a frame of captured video display data according to some examples;

[0017] Figure 4B is a diagram illustrating an example of a video frame layer stack associated with a captured frame of video display data according to some examples;

[0018] Figure 4C is a diagram illustrating an example table including layer information associated with a plurality of layers included in a given frame of captured video display data according to some examples;

[0019] Figure 5 is a flow chart illustrating an example of a process for performing video decoding using variable I-frame time intervals and variable GOP lengths according to some examples;

[0020] Figure 6 is a flow chart illustrating another example of a process for performing video decoding using variable I-frame time intervals and variable GOP lengths according to some examples;

[0021] Figure 7 is a block diagram illustrating an example video encoding device according to some examples; and

[0022] Figure 8 is a block diagram illustrating an example video decoding device according to some examples. DETAILED DESCRIPTION

[0023] Some aspects and aspects of the present disclosure are provided below. Some of these aspects and aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is obvious that various aspects can be implemented without these specific details. Each drawing and description are not intended to be restrictive.

[0024] The following description provides only exemplary aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the exemplary aspects will provide a description that can be used to implement the exemplary aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0025] Digital video data may include large amounts of data, especially as the demand for high-quality video data continues to grow. For example, consumers of video data generally expect increasingly high-quality video, with high fidelity, high resolution, high frame rate, etc. However, the large amounts of video data required to meet such demands may place a significant burden on communication networks and devices that process and store video data.

[0026] A video decoding device (e.g., an encoding device, a decoding device, or a combined encoding-decoding device) implements video compression techniques to efficiently decode (e.g., encode and / or decode) video data. Video compression techniques may include applying different prediction modes, including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques for reducing or removing redundancy inherent in video sequences. The video encoder may divide each picture of the original video sequence into a plurality of rectangular areas, which are referred to as video blocks or decoding units (described in more detail below). These video blocks may be encoded using a specific prediction mode.

[0027] Video blocks may be divided into one or more groups of smaller blocks in one or more ways. Blocks may include coding tree blocks, prediction blocks, transform blocks, or other suitable blocks. Unless otherwise specified, references to "blocks" may generally refer to such video blocks (e.g., coding tree blocks, coding blocks, prediction blocks, transform blocks, or other suitable blocks or sub-blocks, as will be understood by those of ordinary skill in the art). In addition, each of these blocks may also be referred to interchangeably herein as a "unit" (e.g., coding tree unit (CTU), coding unit, prediction unit (PU), transform unit (TU), etc.). In some cases, a unit may indicate a decoding logic unit encoded in a bitstream, and a block may indicate a portion of a process in a video frame buffer for which a process is directed.

[0028] For inter-prediction mode, the video encoder may search for a block similar to the block being encoded in a frame (or picture) located at another temporal position (which is called a reference frame or reference picture). The video encoder may limit the search to a certain spatial displacement from the block to be encoded. A two-dimensional (2D) motion vector including a horizontal displacement component and a vertical displacement component may be used to locate the best match. For intra-prediction mode, the video encoder may use spatial prediction techniques to form a prediction block based on data from previously encoded neighboring blocks within the same picture.

[0029] The video encoder may determine a prediction error. For example, the prediction may be determined as the difference between the pixel values ​​in the block being encoded and the predicted block. The prediction error may also be referred to as a residual. The video encoder may also apply a transform to the prediction error (e.g., a discrete cosine transform (DCT) or other suitable transform) to generate transform coefficients. After the transform, the video encoder may quantize the transform coefficients. The quantized transform coefficients and motion vectors may be represented using syntax elements and, together with control information, form a representation of the decoding of the video sequence. In some instances, the video encoder may entropy encode the quantized transform coefficients and / or syntax elements to further reduce the number of bits required for their representation.

[0030] After entropy decoding and dequantization of the received bitstream, the video decoder can use the syntax elements and control information discussed above to construct prediction data (e.g., a prediction block) for decoding the current frame. For example, the video decoder can add the prediction block and the compressed prediction error. The video decoder can determine the compressed prediction error by weighting the transform basis function using the quantized coefficients. The difference between the reconstructed frame and the original frame is called the reconstruction error.

[0031] Video coding may be performed according to a specific video coding standard. Examples of video coding standards include, but are not limited to, ITU-T H.261, ISO / IEC MPEG-1 video, ITU-T H.262 or ISO / IEC MPEG-2 video, ITU-T H.263, ISO / IEC MPEG-4 video, Advanced Video Coding (AVC) or ITU-T H.264 (including its Scalable Video Coding (SVC) and Multi-view Video Coding (MVC) extensions), High Efficiency Video Coding (HEVC) or ITU-T H.265 (including its Range and Screen Content Coding, 3D Video Coding (3D-HEVC), Multi-view (MV-HEVC), and Scalable (SHVC) extensions), Versatile Video Coding (VVC) or ITU-T H.266 and its extensions, VP9, ​​Alliance for Open Media (AOMedia) Video 1 (AV1), Essential Video Coding (EVC), and the like.

[0032] As noted above, the video encoder may divide each picture of the original video sequence into one or more smaller blocks or rectangular areas, which may then be encoded using, for example, intra-prediction or intra-frame prediction to remove spatial redundancy inherent in the original video sequence. If the block is encoded in intra-prediction mode, a prediction block is formed based on a previously encoded and reconstructed block (e.g., included in the same video data frame), which may be used in both the video encoder and the video decoder to form a prediction reference. For example, the pixel values ​​of the previously encoded adjacent blocks associated with the same video data frame may be used to determine the spatial prediction of the pixel values ​​inside the current block (e.g., currently encoded or currently decoded). These pixel values ​​are used as reference pixels. Reference pixels may be organized into one or more reference pixel lines and / or reference pixel groups. In some examples, intra-prediction may be applied to both the luminance component and the chrominance component of the block.

[0033] Multiple different intra-frame prediction modes can be used to provide different spatial prediction techniques to form a prediction reference or prediction block based on data from previously encoded neighboring blocks (e.g., from reference pixels) within the same picture. Intra-frame prediction modes may include plane and DC modes and / or directional intra-frame prediction modes (also referred to as "regular intra-frame prediction modes"). In some examples, a single plane intra-frame prediction and a single DC intra-frame prediction mode as well as multiple directional intra-frame prediction modes may be used. The intra-frame prediction mode describes different variants or methods for calculating pixel values ​​in the area being decoded based on reference pixel values. In an illustrative example, the HEVC standard provides 33 directional intra-frame prediction modes. In another illustrative example, VVC and / or VVC Test Model 5 (VTM5) extend the HEVC directional intra-frame prediction mode to provide a total of 93 directional intra-frame prediction modes.

[0034] At the video decoder, the intra-prediction mode selection for each block (e.g., corresponding to the intra-prediction mode selection made by the video encoder when generating the coded block) can be determined by the decoder (e.g., derived) or can be signaled to the video decoder, such as in the syntax of the bitstream. For example, in some cases, the intra-prediction modes between adjacent blocks can be correlated (e.g., if intra-prediction mode 2 is used to predict two previously encoded adjacent blocks, then the best intra-prediction mode for the current block may also be intra-prediction mode 2). In some examples, for each current block, the video encoder and the video decoder can calculate the most likely intra-prediction mode. The video encoder can also signal the intra-prediction mode to the video decoder (e.g., using a flag, a mode parameter, a mode selector, etc.).

[0035] In the current VVC standard, 93 directional intra prediction modes are provided, as previously noted. Each intra prediction mode is associated with a different angular direction, so that the intra prediction modes are unique and non-overlapping. Directional intra prediction modes can be classified as integer angle modes or fractional (non-integer angle) modes. For a given block of video data, the integer angle intra prediction mode has reference pixels at integer positions, for example, the integer angle intra prediction mode has a slope through the position of reference pixels located at the periphery of the current decoding block. In contrast, the fractional intra prediction mode does not have a reference pixel at an integer position, but has a slope that passes through a point somewhere between two adjacent reference pixels (for example, the slope of a pixel at a fractional position i+f (i: integer part, f: fractional part) passes through pixel i and pixel i+1).

[0036] In some aspects, video coding may be performed using a combination of intra-predicted frames (I frames), predicted frames (P frames), and / or bidirectional frames (B frames). For example, an I frame may include only video data blocks using intra-prediction. In some aspects, an I frame may be used as a key frame based on the fact that the I frame includes only video data blocks that reference other video data blocks within the same I frame. For example, an I frame may be used as a key frame relative to one or more P frames and / or B frames that are sequentially located before or after the I frame. In some aspects, one or more P frames may be generated to reference previously encoded I frames (e.g., to act as key frames) and / or previously encoded P frames. In some examples, one or more B frames may be generated to reference both previously encoded and subsequently decoded (e.g., future) frames, both of which may include an I frame (e.g., to act as a key frame).

[0037] In some aspects, the distance between two key frames (e.g., I-frames) may be referred to as a group of pictures (GOP) or GOP length. The GOP length may be measured in terms of the number of frames between two I-frame key frames or the amount of time between two I-frame key frames. For example, if for every second of a 30-frame-per-second video, an I-frame is inserted and used as a key frame, the GOP length is 30 frames or one second.

[0038] In some examples, video coding may be performed using variable I-frame time intervals (e.g., also referred to as "variable GOP lengths"). For example, variable I-frame time intervals or GOP lengths may be used to improve the efficiency of video coding. In some aspects, optimal video coding efficiency for different types of video content may be associated with different GOP lengths. For example, relatively static video content may be coded using a relatively large GOP length (e.g., based on the video content changing relatively little over time), while video content with fast-moving or many moving objects may be coded using a relatively small GOP length (e.g., based on the video content changing relatively quickly over time).

[0039] In some aspects, variable length GOP video coding (e.g., encoding and / or decoding) may be based on video content analysis and / or motion analysis to determine an amount of change over time associated with video content. Such techniques may achieve improved video coding performance, but are typically associated with high computational complexity and power consumption (e.g., associated with performing video content analysis and / or motion analysis separately from video coding). In some cases, fixed length GOP video coding (e.g., fixed I-frame time interval video coding) may be utilized based on computational constraints, power constraints, and / or coding time constraints associated with a given video coding device.

[0040] For example, WiFi display technologies (e.g., such as ) can be used to wirelessly share video content between WiFi devices by capturing and encoding the display content of a source device (e.g., a smartphone) and sending the encoded video data to a receiving device (e.g., a television). Based on the computing resources and / or power and energy resources available at the source device (e.g., a smartphone), WiFi display technologies such as ) typically utilize a fixed-length GOP to encode (and subsequently decode) video data for display at a source device. In some aspects, based at least in part on the source device (e.g., smartphone) video display data varying in source, type, content, etc., using a fixed-length GOP can reduce the need for WiFi display technologies (such as ) is performed. Systems and techniques are needed that can be used to perform variable length GOP video decoding (e.g., variable I-frame time interval video decoding) without using computationally intensive techniques (such as video content analysis or motion analysis). There is also a need for systems and techniques that can be used to implement WiFi display sharing (e.g., ) in a power-efficient manner to perform variable length GOP video decoding (e.g., variable I-frame time interval video decoding).

[0041] As described in greater detail herein, systems, apparatus, methods, and computer-readable media (collectively, "systems and techniques") are described herein for performing video coding using variable intra-frame (I-frame) time intervals and / or variable length group of pictures (GOP) lengths. For example, the systems and techniques may perform video coding (e.g., encoding and / or decoding) using variable I-frame intervals and / or variable GOP lengths determined based on information such as video frame layer information, video frame layer geometry, and the like. For example, the systems and techniques may obtain and analyze layer information associated with one or more layers included in a plurality of frames to be encoded. In some aspects, the plurality of frames to be encoded may be used to perform wireless display sharing (e.g., )’s smart phone or other mobile computing device.

[0042] In some aspects, one or more layers may be layer primitive types that represent compositing work and interaction with display hardware (e.g., a display or other display hardware associated with an encoding device or other computing device). In some examples, a layer may also be referred to as a compositing unit. A layer may be a combination of a surface and an instance of a SurfaceControl. Each layer may have a set of properties that define how the layer interacts with other layers. For example, layer properties may include, but are not limited to, one or more of the layer properties described below.

[0043] The "position" layer property may indicate where the layer appears on its corresponding display (e.g., a display is another type of primitive that, in combination with a layer, can represent compositing work and interaction with display hardware). The "position" layer property may include information such as the location of the edge of the layer and the z-order of the layer relative to other layers (e.g., whether the layer is in front of or behind other layers).

[0044] "Content" layer properties may indicate how content displayed on the layer should be rendered within the bounds of the layer (e.g., given by the position property). "Content" layer properties may include information such as cropping information (e.g., expanding a portion of the content to fill the bounds of the layer) and transformation information (e.g., showing rotated or flipped content).

[0045] "Compositing" layer properties may indicate how this layer should be composited with other layers, and may include information such as blending modes and layer-wide alpha values ​​for alpha compositing.

[0046] An "optimized" layer characteristic may indicate or otherwise include information that may not be used directly for compositing the layer, but may be used by a hardware compositor (HWC) to optimize its compositing performance. For example, an "optimized" layer characteristic may include information such as the visible area of ​​the layer and which portion(s) of the layer have been updated since the previous frame.

[0047] In some aspects, the layer information may include the geometry of individual layers, where each frame of captured display data associated with the source device includes one or more layers. In some cases, the layer information may include coordinate information, format information, etc. associated with individual layers of the frames of captured display data. In some aspects, the systems and techniques may use the layer information to determine adaptive and / or variable length GOPs (e.g., adaptive and / or variable I-frame time intervals). For example, the layer information may include rich scene information that can be analyzed more efficiently than pixel-based content or motion analysis. Based on the rich scene information determined from the layer information, the systems and techniques may implement scene change analysis, notifications, and / or I-frame triggering.

[0048] For example, layer information of a current frame of captured display data from a source device may be analyzed and compared to layer information associated with one or more previous frames of captured display data from the source device. In some aspects, based on a determination that some (or all) of the layers represented in the layer information have changed by greater than a threshold amount or percentage (e.g., based on a sufficient change in the frame geometry of the captured display data), I-frame decoding may be triggered and a new (e.g., variable or adaptive) GOP length may be applied to the newly generated I-frame. In some aspects, the systems and techniques described herein may perform adaptive and / or variable length GOP video decoding (e.g., adaptive and / or variable I-frame time interval video decoding) based on the layer information as described above and further based on a display idle determination.

[0049] For example, the systems and techniques may determine that video content represented in a frame of captured display data from a source device has transitioned to an idle state. In some aspects, the idle state may be associated with no content changes detected and / or no additional rendering or refreshing of a frame of captured display data detected within a predetermined time period. In some examples, detecting or determining a display idle state may cause the systems and techniques to apply a new relatively longer GOP length associated with the display idle state. For example, the display idle GOP length may be 300 frames, although a greater or lesser number of frames may also be used for the display idle GOP length.

[0050] In some examples, each type of layer that may be included in a given frame of captured display data from a source device may be associated with its own GOP length (e.g., in terms of the number of frames or time intervals between I-frame keyframes). In some aspects, each layer type may be associated with a different GOP length (e.g., in terms of the number of frames or time intervals between I-frame keyframes). In some cases, each layer type may be associated with its own GOP length, wherein one or more of the layer types are associated with the same GOP length value (e.g., in terms of the number of frames or time intervals between I-frame keyframes). In some examples, one or more (or all) of the layer type GOP lengths may be predetermined. For example, different layer type GOP lengths may be predetermined based on the type of video content and / or the type of motion associated with each respective layer of the different layer types. In some aspects, the GOP length may be determined for frames included in multiple frames of captured display data based on the primary content layer or primary content layer type associated with each given frame. For example, the frame GOP length may be adaptively or variably determined using a GOP length associated with a primary content layer determined for each given frame of a plurality of frames of captured display data.

[0051] Additional details regarding the systems and techniques will be described with respect to the accompanying figures.

[0052] Figure 1 1 is a block diagram showing an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or the receiving device may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the source device and the receiving device may include one or more wireless transceivers for wireless communication. The decoding techniques described herein are applicable to video decoding in various multimedia applications, including streaming video transmission (e.g., via the Internet), television broadcasting or transmission, encoding digital video for storage on a data storage medium, decoding digital video stored on a data storage medium, or other applications. As used herein, the term decoding may refer to encoding and / or decoding. In some examples, system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0053] The encoding device 104 (or encoder) can be used to encode video data using a video coding standard, format, codec or protocol to generate a coded video bitstream. Examples of video coding standards and formats, codecs include ITU-T H.261, ISO / IEC MPEG-1 video, ITU-T H.262 or ISO / IEC MPEG-2 video, ITU-T H.263, ISO / IEC MPEG-4 video, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC) (including its scalable video coding (SVC) and multi-view video coding (MVC) extensions), high efficiency video coding (HEVC) or ITU-T H.265 and universal video coding (VVC) or ITU-TH.266. There are various extensions to HEVC to handle multi-layer video coding, including range and screen content coding extensions, 3D video coding (3D-HEVC) and multi-view extensions (MV-HEVC) and scalable extensions (SHVC). HEVC and its extensions have been developed by the Joint Collaborative Team on Video Coding (JCT-VC) and the Joint Collaborative Team on 3D Video Coding Extensions Development (JCT-3V) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). VP9, ​​AOMedia Video 1 (AV1) developed by the Alliance for Open Media (AOMedia), and Essential Video Coding (EVC) are other video coding standards to which the techniques described herein may be applied.

[0054] VVC is a state-of-the-art video coding standard developed by the Joint Video Experts Team (JVET) of ITU-T and ISO / IEC to achieve, at least in part, high compression capabilities beyond HEVC for a wide range of applications. The VVC specification was completed in July 2020 and published by both ITU-T and ISO / IEC. The VVC specification specifies standard bitstream and picture formats, high-level syntax (HLS) and coding unit-level syntax, parsing process, decoding process, etc. VVC also specifies profile / layer / level (PTL) restrictions, byte stream format, assumed reference decoder, and supplemental enhancement information (SEI) in annexes.

[0055] The systems and techniques described herein may be applied to any of the existing video codecs (e.g., VVC, HEVC, AVC, or other suitable existing video codecs), and / or may be effective decoding tools for any video decoding standards being developed and / or future video decoding standards. For example, video codecs such as VVC, HEVC, AVC, and / or their extensions may be used to perform the examples described herein. However, the techniques and systems described herein may also be applicable to other decoding standards, codecs, or formats, such as MPEG, JPEG (or other decoding standards for still images), VP9, ​​AV1, their extensions, or other suitable decoding standards that are already available or not yet available or not yet developed. For example, in some examples, the encoding device 104 and / or the decoding device 112 may operate according to a proprietary video codec / format (such as AV1, an extension of AVI, and / or a successor version of AV1 (e.g., AV2)) or other proprietary formats or industry standards. Therefore, although the techniques and systems described herein may be described with reference to a specific video decoding standard, it will be understood by those of ordinary skill in the art that the description should not be interpreted as being applicable only to that specific standard.

[0056] refer to Figure 1 , video source 102 may provide video data to encoding device 104. Video source 102 may be part of a source device, or may be part of a device other than a source device. Video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive including stored video, a video server or content provider providing video data, a video feed interface receiving video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or any other suitable video source.

[0057] The video data from the video source 102 may include one or more input pictures or frames. A picture or frame is a static image, which in some cases is part of a video. In some examples, the data from the video source 102 may be a static image that is not part of a video. In HEVC, VVC and other video coding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays, which are represented as SL, SCb and SCr. SL is a two-dimensional array of luma samples, SCb is a two-dimensional array of Cb chroma samples, and SCr is a two-dimensional array of Cr chroma samples. Chroma samples may also be referred to as "chroma" samples in this article. A pixel may refer to all three components (luminance samples and chroma samples) at a given position in an array of pictures. In other cases, a picture may be monochrome and may include only a luma sample array, in which case the terms pixel and sample may be used interchangeably. With respect to the example techniques described herein that refer to individual samples for illustrative purposes, the same techniques may be applied to pixels (e.g., all three sample components at a given position in an array of pictures). With respect to the example techniques described herein that refer to pixels (eg, all three sample components of a given position in an array of a picture) for illustrative purposes, the same techniques may be applied to individual samples.

[0058] The encoder engine 106 (or encoder) of the encoding device 104 encodes the video data to generate an encoded video bitstream. In some examples, the encoded video bitstream (or "video bitstream" or "bitstream") is a series of one or more decoded video sequences. The decoded video sequence (CVS) includes a series of access units (AUs) starting from an AU having a random access point picture in the base layer and having certain attributes, until the next AU having a random access point picture in the base layer and having certain attributes and not including the next AU. For example, certain attributes of the random access point picture that starts the CVS may include a RASL marker equal to 1 (e.g., NoRaslOutputFlag). Otherwise, the random access point picture (with a RASL marker equal to 0) does not start the CVS. The access unit (AU) includes one or more decoded pictures and control information corresponding to the decoded pictures sharing the same output time. The decoded slices of the picture are encapsulated as data units at the bitstream level, which are called network abstraction layer (NAL) units. For example, a HEVC video bitstream may include one or more CVSs, which include NAL units. Each NAL unit in the NAL unit has a NAL unit header. In one example, the header is one byte for H.264 / AVC (except for multi-layer extensions) and two bytes for HEVC. The syntax elements in the NAL unit header use designated bits and are therefore visible to all kinds of systems and transport layers (such as transport streams, real-time transport (RTP) protocols, file formats, etc.).

[0059] There are two types of NAL units in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units. The VCL NAL unit includes a slice or slice segment of decoded picture data (described below), and the non-VCL NAL unit includes control information related to one or more decoded pictures. In some cases, the NAL unit may be referred to as a packet. The HEVC AU includes a VCL NAL unit containing decoded picture data and a non-VCL NAL unit corresponding to the decoded picture data (if any).

[0060] The NAL unit may include a bit sequence (e.g., a coded video bitstream, a CVS of a bitstream, etc.) that forms a representation of the decoding of video data, such as a representation of the decoding of a picture in a video. The encoder engine 106 generates a representation of the decoding of a picture by partitioning each picture into multiple slices. A slice is independent of other slices so that the information in the slice can be decoded without relying on data from other slices in the same picture. A slice includes one or more slice segments, which include independent slice segments and (if present) one or more dependent slice segments that depend on the previous slice segment. The slice is partitioned into coding tree blocks (CTBs) of luminance samples and chrominance samples. The CTB of the luminance sample and one or more CTBs of the chrominance sample together with the syntax for the sample are referred to as a coding tree unit (CTU). A CTU may also be referred to as a "tree block" or a "maximum coding unit" (LCU). A CTU is a basic processing unit for HEVC encoding. A CTU may be split into multiple coding units (CUs) of different sizes. A CU contains a luminance sample array and a chrominance sample array called a coding block (CB).

[0061] Luma CB and chroma CB can also be split into prediction blocks (PB). A PB is a sample block of a luma component or a chroma component that uses the same motion parameters for inter prediction or intra block copy prediction (when available or enabled). The luma PB and one or more chroma PBs together with the associated syntax form a prediction unit (PU). For inter prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) are signaled in the bitstream for each PU, as well as inter prediction for luma PB and one or more chroma PBs. Motion parameters may also be referred to as motion information. The CB may also be partitioned into one or more transform blocks (TB). A TB represents a square block of samples of a color component, to which a residual transform (e.g., in some cases, the same two-dimensional transform) is applied to decode the prediction residual signal. A transform unit (TU) represents a TB of luma samples and chroma samples and corresponding syntax elements.

[0062] The size of the CU corresponds to the size of the decoding mode and can be a square shape. For example, the size of the CU can be 8x8 samples, 16x16 samples, 32x32 samples, 64x64 samples, or any other appropriate size up to the size of the corresponding CTU. The phrase "NxN" is used herein to refer to the pixel size of the video block in the vertical and horizontal dimensions (e.g., 8 pixels x8 pixels). The pixels in the block can be arranged in rows and columns. In some examples, the block may not have the same number of pixels in the horizontal direction as in the vertical direction. The syntax data associated with the CU can describe, for example, the partitioning of the CU into one or more PUs. The partitioning mode may vary depending on whether the CU is encoded in an intra-frame prediction mode or an inter-frame prediction mode. The PU can be partitioned into a non-square shape. The syntax data associated with the CU can also, for example, describe the partitioning of the CU into one or more TUs according to the CTU. The TU can be square or non-square in shape.

[0063] According to the HEVC standard, a transform unit (TU) may be used to perform the transform. The TU may vary for different CUs. The size of the TU may be set based on the size of the PU within a given CU. The TU may have the same size as the PU or be smaller than the PU. In some examples, a quadtree structure called a residual quadtree (RQT) may be used to subdivide the residual samples corresponding to the CU into smaller units. The leaf nodes of the RQT may correspond to the TU. The pixel difference values ​​associated with the TU may be transformed to generate transform coefficients. The transform coefficients may be quantized by the encoder engine 106.

[0064] Once the picture of the video data is partitioned into CUs, the encoder engine 106 predicts each PU using a prediction mode. The prediction unit or prediction block is subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be signaled in the bitstream using syntax data. The prediction mode may include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction exploits the correlation between spatially adjacent samples within a picture. For example, using intra-frame prediction, each PU is predicted from adjacent image data in the same picture using, for example, DC prediction to find an average value for the PU, using plane prediction to adapt the plane surface to the PU, using directional prediction to infer from adjacent data, or using any other suitable prediction type. Inter-frame prediction uses temporal correlation between pictures to derive motion compensated prediction for blocks of image samples. For example, using inter-frame prediction, each PU is predicted from image data in one or more reference pictures (before or after the current picture in output order) using motion compensated prediction. For example, a decision can be made at the CU level whether to use inter-picture prediction or intra-picture prediction to decode a picture area.

[0065] The encoder engine 106 and the decoder engine 116 (described in more detail below) may be configured to operate according to VVC. According to VVC, a video decoder (such as the encoder engine 106 and / or the decoder engine 116) partitions a picture into a plurality of coding tree units (CTUs) (wherein the CTB of a luminance sample and one or more CTBs of a chrominance sample together with the syntax for the sample are referred to as CTUs). The video decoder may partition the CTU according to a tree structure such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure removes the concept of multiple partition types, such as the separation between CU, PU, ​​and TU of HEVC. The QTBT structure includes two levels, including a first level partitioned according to quadtree partitioning, and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to the coding units (CUs).

[0066] In the MTT segmentation structure, blocks can be segmented using quadtree segmentation, binary tree segmentation, and one or more types of ternary tree segmentation. A ternary tree segmentation is a segmentation in which a block is split into three sub-blocks. In some examples, a ternary tree segmentation divides a block into three sub-blocks without dividing the original block through the center. The segmentation types in MTT (e.g., quadtree, binary tree, and ternary tree) can be symmetric or asymmetric.

[0067] When operating according to the AV1 codec, the encoding device 104 and the decoding device 112 may be configured to decode the video data in units of blocks. In AV1, the largest decoding block that can be processed is called a super block. In AV1, a super block may be 128×128 luma samples or 64×64 luma samples. However, in subsequent video coding formats (e.g., AV2), super blocks may be defined by different (e.g., larger) luma sample sizes. In some examples, the super block is the top layer of the block quadtree. The encoding device 104 may further divide the super block into smaller decoding blocks. The encoding device 104 may divide the super block and other decoding blocks into smaller blocks using square or non-square partitioning. Non-square blocks may include N / 2×N blocks, N×N / 2 blocks, N / 4×N blocks, and N×N / 4 blocks. The encoding device 104 and the decoding device 112 may perform separate prediction and transform processing on each decoding block.

[0068] AV1 also defines tiles of video data. A tile is a rectangular array of super blocks that can be decoded independently of other tiles. That is, the encoding device 104 and the decoding device 112 can encode and decode coded blocks within a tile, respectively, without using video data from other tiles. However, the encoding device 104 and the decoding device 112 can perform filtering across tile boundaries. The size of the tiles can be uniform or non-uniform. Tile-based decoding can enable parallel processing and / or multi-threading specifically implemented by the encoder and decoder.

[0069] In some examples, the encoding device 104 and the decoding device 112 may use a single QTBT or MTT structure to represent each of the luma component and the chroma components, while in other examples, the video decoder may use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luma component and another QTBT or MTT structure for the two chroma components (or two QTBT and / or MTT structures for the respective chroma components).

[0070] The encoding device 104 and the decoding device 112 may be configured to use quadtree partitioning per HEVC, QTBT partitioning, MTT partitioning, or other partitioning structures.

[0071] In some examples, a slice type is assigned to one or more slices of a picture. Slice types include I slices, P slices, and B slices. An I slice (intra, independently decodable) is a slice of a picture that is coded only by intra prediction, and is therefore independently decodable because an I slice only requires data within the frame to predict any prediction unit or prediction block of the slice. A P slice (unidirectionally predicted frame) is a slice of a picture that can be coded with intra prediction and with unidirectional inter prediction. Each prediction unit or prediction block within a P slice is coded with intra prediction or inter prediction. When inter prediction is applied, the prediction unit or prediction block is predicted by only one reference picture, and therefore the reference sample comes from only one reference region of a frame. A B slice (bidirectionally predictive frame) is a slice of a picture that can be coded with intra prediction and inter prediction (e.g., either bidirectional prediction or unidirectional prediction). The prediction unit or prediction block of a B slice may be predicted bidirectionally from two reference pictures, where each picture contributes to one reference region and the sample sets of the two reference regions are weighted (e.g., with equal weights or with different weights) to produce a prediction signal for the bidirectional prediction block. As described above, the slices of a picture are coded independently. In some cases, a picture may be coded as only one slice.

[0072] As noted above, intra-picture prediction exploits the correlation between spatially adjacent samples within a picture. There are multiple intra-frame prediction modes (also referred to as "intra-frame modes"). In some examples, intra-frame prediction of luma blocks includes 35 modes, including a planar mode, a DC mode, and 33 angular modes (e.g., a diagonal intra-frame prediction mode and an angular mode adjacent to the diagonal intra-frame prediction mode). The encoding device 104 and / or the decoding device 112 may select a prediction mode for each block that minimizes the residual between the prediction block and the block to be encoded (e.g., based on the sum of absolute errors (SAE), the sum of absolute differences (SAD), the sum of absolute transform differences (SATD), or other measures of similarity). For example, the SAE may be calculated by obtaining the absolute difference between each pixel (or sample) in the block to be encoded and the corresponding pixel (or sample) in the prediction block for comparison. The differences of the pixels (or samples) are summed to produce a measure of block similarity, such as the L1 norm of the difference image, the Manhattan distance between two image blocks, or other calculations. Using SAE as an example, the SAE for each prediction using each of the intra prediction modes indicates the magnitude of the prediction error. The intra prediction mode having the best match with the actual current block is given by the intra prediction mode giving the smallest SAE.

[0073] The 35 modes of intra prediction are indexed as shown in Table 1 below. In other examples, more intra modes may be defined, including prediction angles that may not yet be represented by the 33 angle modes. In other examples, the prediction angles associated with the angle modes may be different from those used in HEVC.

[0074] Intra prediction mode Associated name 0 INTRA_PLANAR 1 INTRA_DC 2..34 INTRA_ANGULAR2..INTRA_ANGULAR34

[0075] Table 1 - Specification of intra prediction modes and associated names

[0076] To perform planar prediction on an N×N block, for each sample p at (x, y) xy , the predicted sample value can be calculated by applying a bilinear filter to four specific adjacent reconstructed samples (used as reference samples for intra prediction). The four reference samples include the upper right reconstructed sample TR, the lower left reconstructed sample BL, and the same column (r x,-1 ) and row(r -1,y ). The plane pattern can be expressed by the following formula:

[0077] p xy =((N-x1)*·L+(N-y1)*·T+x1*·R+y1*·B) / (2*N),

[0078] Where x1=x+1, y1=y+1, R=TR and B=BL.

[0079] For DC mode, the prediction block is filled with the average of neighboring reconstructed samples. Generally speaking, both planar mode and DC mode are used to model smoothly varying and constant image regions.

[0080] For the angular intra prediction modes in HEVC (which include 33 different prediction directions), the intra prediction process can be described as follows. For each given angular intra prediction mode, the intra prediction direction can be identified accordingly; for example, intra mode 18 corresponds to a purely horizontal prediction direction, and intra mode 26 corresponds to a purely vertical prediction direction. Figure 2A The angular prediction modes are shown in example diagram 200a of . In some codecs, different numbers of intra prediction modes may be used. For example, in addition to the planar mode and DC mode, 93 angular modes may be defined, where mode 2 indicates a prediction direction of -135°, mode 34 indicates a prediction direction of -45°, and mode 66 indicates a prediction direction of 45°. In some codecs (e.g., VVC), angles greater than -135° (less than -135°) and greater than 45° (greater than 45°) may also be defined; these may be referred to as wide-angle intra modes. Although the description herein is about an intra mode design in HEVC (e.g., with 35 modes), the disclosed technology may also be applied to more intra modes (e.g., intra modes defined by VVC or other codecs).

[0081] The coordinates (x, y) of each sample of the prediction block are projected along a specific intra prediction direction (e.g., one of the angular intra prediction modes). For example, given a specific intra prediction direction, the coordinates (x, y) of the samples of the prediction block are first projected along the intra prediction direction to the rows / columns of the adjacent reconstructed samples. In the case where (x, y) is projected to a fractional position α between two adjacent reconstructed samples L and R; a two-tap bilinear interpolation filter can be used to calculate the predicted value of (x, y), which is expressed as follows:

[0082] p xy =(1-a)·L+a·R

[0083] To avoid floating point operations, in HEVC, the above calculation can be estimated using integer arithmetic as:

[0084] p xy =((32-a')·L+a'·R+16)>>5,

[0085] Where a' is an integer equal to 32*a.

[0086] In some examples, before intra prediction, adjacent reference samples are filtered using a 2-tap bilinear or 3-tap (1,2,1) / 4 filter, which may be referred to as intra reference smoothing or mode-dependent intra smoothing (MDIS). When performing intra prediction, given an intra prediction mode index (predModeIntra) and a block size (nTbS), it is decided whether to perform a reference smoothing process and which smoothing filter to use. The intra prediction mode index is an index indicating an intra prediction mode.

[0087] Inter-picture prediction uses temporal correlations between pictures in order to derive motion compensated predictions for blocks of image samples. Using a translational motion model, the position of a block in a previously decoded picture (reference picture) is indicated by a motion vector (Δx, Δy), where Δx specifies the horizontal displacement of the reference block relative to the position of the current block, and Δy specifies the vertical displacement of the reference block relative to the position of the current block. In some cases, the motion vector (Δx, Δy) may have integer sample accuracy (also known as integer accuracy), in which case the motion vector points to an integer pixel grid (or integer pixel sampling grid) of a reference frame. In some cases, the motion vector (Δx, Δy) may have fractional sample accuracy (also known as fractional pixel accuracy or non-integer accuracy) to more accurately capture the movement of underlying objects, without being limited to an integer pixel grid of a reference frame. The accuracy of a motion vector may be represented by the quantization level of the motion vector. For example, the quantization level may be integer accuracy (e.g., 1 pixel) or fractional pixel accuracy (e.g., 1 / 4 pixel, 1 / 2 pixel, or other sub-pixel value). When the corresponding motion vector has fractional sampling accuracy, interpolation is applied to the reference picture to derive the prediction signal. For example, the samples available at integer positions may be filtered (e.g., using one or more interpolation filters) to estimate the values ​​at fractional positions. The previously decoded reference picture is indicated by a reference index (refIdx) of a reference picture list. The motion vector and the reference index may be referred to as motion parameters. Two types of inter-picture prediction may be performed, including unidirectional prediction and bidirectional prediction.

[0088] In the case of inter-frame prediction using bidirectional prediction (also referred to as bidirectional inter-frame prediction), two sets of motion parameters (Δx0, y0, refIdx0 and Δx1, y1, refIdx1) are used to generate two motion compensated predictions (from the same reference picture or possibly from different reference pictures). For example, in the case of bidirectional prediction, each prediction block uses two motion compensated prediction signals and generates B prediction units. The two motion compensated predictions are combined to obtain the final motion compensated prediction. For example, the two motion compensated predictions can be combined by averaging. In another example, weighted prediction can be used, in which case different weights can be applied to each motion compensated prediction. The reference pictures that can be used in the bidirectional prediction are stored in two separate lists, represented as list 0 and list 1 respectively. The motion parameters can be derived using a motion estimation process at the encoding device 104.

[0089] In the case of inter prediction using unidirectional prediction (also referred to as unidirectional inter prediction), a set of motion parameters (Δx0, y0, refIdx0) is used to generate motion compensated prediction from a reference picture. For example, in the case of unidirectional prediction, each prediction block uses at most one motion compensated prediction signal and generates P prediction units.

[0090] The PU may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when the PU is encoded using intra prediction, the PU may include data describing the intra prediction mode for the PU. As another example, when the PU is encoded using inter prediction, the PU may include data defining a motion vector for the PU. The data defining the motion vector for the PU may describe, for example, a horizontal component of the motion vector (Δx), a vertical component of the motion vector (Δy), a resolution for the motion vector (e.g., integer precision, quarter-pixel precision, or eighth-pixel precision), a reference picture to which the motion vector points, a reference index, a reference picture list for the motion vector (e.g., list 0, list 1, or list C), or any combination thereof.

[0091] AV1 includes two general techniques for encoding and decoding coded blocks of video data. The two general techniques are intra-frame prediction (e.g., intra-frame prediction or spatial prediction) and inter-frame prediction (e.g., inter-frame prediction or temporal prediction). In the content of AV1, when an intra-frame prediction mode is used to predict a block of a current frame of video data, the encoding device 104 and the decoding device 112 do not use video data from other frames of the video data. For most intra-frame prediction modes, the video encoding device 104 encodes a block of the current frame based on the difference between the sample value in the current block and the prediction value generated from the reference sample in the same frame. The video encoding device 104 determines the prediction value generated from the reference sample based on the intra-frame prediction mode.

[0092] After performing prediction using intra prediction and / or inter prediction, the encoding device 104 may perform transformation and quantization. For example, after prediction, the encoder engine 106 may calculate a residual value corresponding to the PU. The residual value may include a pixel difference value between a current pixel block (PU) being decoded and a prediction block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a prediction block (e.g., using inter prediction or intra prediction), the encoder engine 106 may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantizes the difference between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block may be represented by a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of the pixel values.

[0093] Any residual data that may remain after prediction is performed is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of size 32×32, 16×16, 8×8, 4×4, or other suitable sizes) may be applied to the residual data in each CU. In some examples, TUs may be used for the transform and quantization process implemented by the encoder engine 106. A given CU with one or more PUs may also include one or more TUs. As described in further detail below, the residual values ​​may be transformed into transform coefficients using a block transform, and may be quantized and scanned using a TU to produce serialized transform coefficients for entropy coding.

[0094] In some examples, after intra-prediction or inter-prediction decoding using a PU of a CU, the encoder engine 106 may calculate residual data for a TU of the CU. A PU may include pixel data in a spatial domain (or pixel domain). A TU may include coefficients in a transform domain after applying a block transform. As previously noted, the residual data may correspond to a pixel difference between a pixel of an uncoded picture and a predicted value corresponding to the PU. The encoder engine 106 may form a TU including the residual data for the CU, and may transform the TU to generate transform coefficients for the CU.

[0095] The encoder engine 106 may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

[0096] Once quantization is performed, the decoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction mode, motion vectors, block vectors, etc.), segmentation information, and any other suitable data (such as other syntax data). Different elements of the decoded video bitstream can be entropy encoded by the encoder engine 106. In some examples, the encoder engine 106 can scan the quantized transform coefficients using a predefined scanning order to produce a serialized vector that can be entropy encoded. In some examples, the encoder engine 106 can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 106 can entropy encode the vector. For example, the encoder engine 106 can use context adaptive variable length coding, context adaptive binary arithmetic coding, grammar-based context adaptive binary arithmetic coding, probability interval segmentation entropy coding, or another suitable entropy coding technology.

[0097] The output 110 of the encoding device 104 may transmit the NAL units constituting the encoded video bitstream data to a decoding device 112 of a receiving device over a communication link 120. The input 114 of the decoding device 112 may receive the NAL units. The communication link 120 may include a channel provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, a WiFi network, or a wireless network). TM , Radio Frequency (RF), UWB, WiFi Direct, Cellular, Long Term Evolution (LTE), WiMax TM The wired network may include any wired interface (e.g., optical fiber, Ethernet, power line Ethernet, Ethernet based on coaxial cable, digital signal line (DSL), etc.). The wired network and / or wireless network may be implemented using various equipment (such as base stations, routers, access points, bridges, gateways, switches, etc.). The encoded video bitstream data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to a receiving device.

[0098] In some examples, the encoding device 104 may store the encoded video bitstream data in a storage device 108. The output end 110 may retrieve the encoded video bitstream data from the encoder engine 106 or from the storage device 108. The storage device 108 may include any of a variety of distributed or locally accessed data storage media. For example, the storage device 108 may include a hard drive, a storage disk, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing the encoded video data. The storage device 108 may also include a decoded picture buffer (DPB) for storing reference pictures for use in inter-frame prediction. In a further example, the storage device 108 may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. In such a case, a receiving device including a decoding device 112 can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server capable of storing encoded video data and sending the encoded video data to a receiving device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The receiving device may access the encoded video data through any standard data connection (including an Internet connection), and may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both suitable for accessing the encoded video data stored on a file server. The transmission of the encoded video data from the storage device 108 may be a streaming transmission, a download transmission, or a combination thereof.

[0099] The input end 114 of the decoding device 112 receives the encoded video bitstream data, and the video bitstream data can be provided to the decoder engine 116, or provided to the storage device 118 for later use by the decoder engine 116. For example, the storage device 118 may include a DPB for storing reference pictures used in inter-frame prediction. The receiving device including the decoding device 112 can receive the encoded video data to be decoded via the storage device 108. The encoded video data can be modulated according to a communication standard (such as a wireless communication protocol) and sent to the receiving device. The communication medium for sending the encoded video data may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a part of a packet-based network such as a local area network, a wide area network, or a global network (such as the Internet). The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from a source device to a receiving device.

[0100] The decoder engine 116 can decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more encoded video sequences that constitute the encoded video data. The decoder engine 116 can rescale the encoded video bitstream data and perform an inverse transform on it. The residual data is passed to the prediction stage of the decoder engine 116. The decoder engine 116 predicts a pixel block (e.g., a PU). In some examples, the prediction is added to the output of the inverse transform (residual data).

[0101] The decoding device 112 may output the decoded video to a video destination device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some aspects, the video destination device 122 may be part of a receiving device that includes the decoding device 112. In some aspects, the video destination device 122 may be part of a separate device distinct from the receiving device.

[0102] In some examples, the video encoding device 104 and / or the video decoding device 112 may be integrated with the audio encoding device and the audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the decoding techniques described above, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. The video encoding device 104 and the video decoding device 112 may be integrated as part of a combined encoder / decoder (codec) in the corresponding device. Reference is made to Figure 7 An example of specific details of the encoding device 104 is described below. Figure 8 An example of specific details of the decoding device 112 is described.

[0103] exist Figure 1The example system shown in is an illustrative example that can be used in this article. The technology for processing video data using the technology described in this article can be performed by any digital video encoding and / or decoding device. Although generally speaking, the technology of the present disclosure is performed by a video encoding device or a video decoding device, these technologies can also be performed by a combined video encoder-decoder (commonly referred to as a "codec"). In addition, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the receiving device are merely examples of such decoding devices, wherein the source device generates decoded video data for sending to the receiving device. In some examples, the source device and the receiving device can operate in a substantially symmetrical manner so that each of these devices includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0104] As previously described, the HEVC bitstream includes a group of NAL units, including VCL NAL units and non-VCL NAL units. The VCL NAL unit includes decoded picture data that forms a decoded video bitstream. For example, a bit sequence that forms a decoded video bitstream is present in a VCL NAL unit. In addition to other information, a non-VCL NAL unit may also contain a parameter set with advanced information related to the encoded video bitstream. For example, a parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). Examples of the goals of a parameter set include bit rate efficiency, fault tolerance, and providing a system layer interface. Each slice references a single active PPS, SPS, and VPS to access information that a decoding device 112 can use to decode the slice. An identifier (ID) may be decoded for each parameter set, including a VPS ID, an SPS ID, and a PPS ID. The SPS includes an SPS ID and a VPS ID. The PPS includes a PPS ID and an SPS ID. Each slice header includes a PPS ID. Using the ID, a valid parameter set may be identified for a given slice.

[0105] The PPS includes information applied to all slices in a given picture. In some examples, all slices in a picture reference the same PPS. Slices in different pictures may also reference the same PPS. The SPS includes information applied to all pictures in the same decoded video sequence (CVS) or bitstream. As previously described, a decoded video sequence is a series of access units (AUs) starting from a random access point picture (e.g., an instantaneous decoding reference (IDR) picture or a broken link access (BLA) picture, or other appropriate random access point pictures) in the base layer and having certain properties (described above) until the next AU (or the end of the bitstream) with a random access point picture in the base layer and having certain properties and not including the next AU. The information in the SPS may not change with pictures within the decoded video sequence. Pictures in a decoded video sequence may use the same SPS. The VPS includes information applied to all layers within a decoded video sequence or bitstream. The VPS includes a syntax structure having syntax elements applied to the entire decoded video sequence. In some examples, the VPS, SPS, or PPS may be sent in-band with the coded bitstream. In some examples, the VPS, SPS, or PPS may be sent out-of-band in a separate transmission from the NAL units that include the coded video data.

[0106] The present disclosure may generally involve "signaling" certain information (such as syntax elements). The term "signaling" may generally refer to the communication of values ​​of syntax elements and / or other data used to decode encoded video data. For example, video encoding device 104 may signal the values ​​of syntax elements in a bitstream. Generally, signaling refers to generating values ​​in a bitstream. As noted above, video source 102 may transmit the bitstream to video destination device 122 in substantially real time or in non-real time (such as may occur when storing syntax elements to storage device 108 for later retrieval by video destination device 122).

[0107] The video bitstream may also include a supplemental enhancement information (SEI) message. For example, a SEI NAL unit may be part of a video bitstream. In some cases, an SEI message may contain information that is not required for the decoding process. For example, the information in the SEI message may not be necessary for a decoder to decode a video picture of the bitstream, but the decoder may use the information to improve the display or processing of the picture (e.g., the decoded output). The information in the SEI message may be embedded metadata. In an illustrative example, the information in the SEI message may be used by a decoder-side entity to improve the visibility of the content. In some instances, certain application standards may indicate the presence of such SEI messages in the bitstream, so that quality improvements can be brought to all devices that comply with the application standards (e.g., carrying a frame encapsulation SEI message for a frame-compatible planar stereoscopic 3DTV video format, where an SEI message is carried for each frame of the video; processing recovery point SEI messages; using a pull-scan rectangle SEI message in DVB; and many other examples).

[0108] As noted above, the encoding device 104 may encode one or more blocks or rectangular areas of a picture of an original video sequence using intra-prediction and / or intra-frame prediction to remove spatial redundancy. The decoding device 112 may decode the encoded block using the same intra-prediction mode used by the encoding device 104. The intra-prediction mode describes different variants or methods for calculating pixel values ​​in the area being decoded based on reference pixel values. In the VVC standard, one or more smoothing filters and interpolation filters may be selected based on the intra-prediction mode and then applied to the intra-prediction of the reference pixel and / or the current block. In this method, the same selection between the smoothing filter and the interpolation filter for intra-prediction is applied to all block sizes, for example, a fixed degree of smoothing is applied to all possible block sizes. Different directional intra-prediction modes are provided in the VVC standard.

[0109] Figure 2B An example diagram 200b illustrates directional intra prediction modes (also referred to as "angular intra prediction modes") in VVC. In some examples, planar mode and DC mode remain the same in VVC as in HEVC. As illustrated, intra prediction modes with even indices between 2 and 66 may be equivalent to 33 HEVC intra prediction modes, where Figure 2BThe remaining intra prediction modes represent the newly added intra prediction modes in VVC. As an illustrative example, in order to better capture arbitrary edge directions presented in natural videos, the number of directional intra prediction modes in VTM5 (VVC Test Model 5) is increased from 33 HEVC directions to a total of 93 directions. The intra prediction modes are described in more detail in B. Bross, J. Chen, S. Liu, "Versatile Video Coding (Draft 10)", 19th JVET Conference, Remote Conference, July 2020, JVET-S2001, which is hereby incorporated by reference in its entirety for all purposes. In some examples, the denser directional intra prediction modes introduced in the VVC standard can be applied to all block sizes and to both luma intra prediction and chroma intra prediction. In some cases, these directional intra prediction modes can be used in combination with multiple reference lines (MRL) and / or with intra sub-division mode (ISP). Further details are described in J. Chen, Y. Ye, S. Kim, “Algorithm description for Versatile Video Coding and Test Model 10 (VTM10)”, 19th JVET Conference, Remote Conference, July 2020, JVET-S2002, which is hereby incorporated by reference in its entirety for all purposes.

[0110] As previously described, video coding may be performed using a combination of intra-predicted frames (I frames), predicted frames (P frames), and / or bidirectional frames (B frames). In some aspects, I frames may be used as key frames for coding (e.g., encoding or decoding) multiple frames of video data. For example, I frames may be used as key frames for encoding or decoding multiple frames of captured video display data, as will be described in more depth below.

[0111] Figure 3 is a diagram illustrating an example of a group of pictures (GOP) length and an inter-frame (I-frame) time interval that may be used to code (eg, encode and / or decode) multiple frames of video data. Figure 3 A plurality of frames 300 of video data including a first I frame 310 and a second I frame 320 are depicted. In some aspects, the I frames 310 and 320 can be used as key frames for decoding one or more B frames and / or one or more P frames that are sequentially located before or after the I frame. For example, the I frames 310 and 320 can be used as key frames for a plurality of P frames and B frames that are also included in the plurality of frames 300 of video data. In some aspects, the I frames 310 and 320 can be used as key frames based on the I frames 310 and 320 only including video data blocks that reference other video data blocks within the same I frame.

[0112] In some aspects, a P frame may be generated to reference a previously encoded I frame key frame. For example, a P frame may be generated. Figure 3 300 to reference the first I frame 310 as a key frame (e.g., based on generating or encoding the first I frame 310 before any of the four P frames included in the plurality of frames 300 of video data). In some examples, the first I frame 310 may be generated. Figure 3 One or more (or all) of the ten B frames depicted in FIG. 3 may generate a first I frame 310 as a key frame. Figure 3 One or more (or all) of the ten B frames depicted in FIG. 3 may reference the second I frame 320 as a key frame, or may generate Figure 3 One or more (or all) of the ten B frames depicted in FIG. 3 may reference both the first I frame 310 and the second I frame 320 as key frames.

[0113] The distance between two key frames (e.g., between two I frames 310, 320) may be referred to as a group of pictures (GOP) length and / or an I frame time interval. For example, the plurality of frames 300 includes a first I frame 310, fourteen intermediate frames (e.g., each intermediate frame is a P frame or a B frame), and a second I frame 320. In some aspects, an open GOP length associated with the plurality of frames 300 is 15 frames. For example, an open GOP length of 15 frames may be determined as fourteen P frames and B frames plus the first I frame 310 (e.g., the first I frame 310 and the fourteen P frames and B frames include a group of pictures (GOP) of length 1+14=15 frames). In some aspects, an open GOP length of 15 frames may be determined as fourteen P frames and B frames plus the next I frame (e.g., the second I frame 320). For example, an open GOP length of 15 frames may be used to indicate that the next I frame (e.g., a key frame) after the first I frame 310 (e.g., the first key frame) will be located 15 frames after the first I frame 310. In some cases, the closed GOP length may be determined as the number of intermediate frames (e.g., P frames and / or B frames) located between two I frames, where the closed GOP length is determined by excluding the frames occupied by the corresponding I frame associated with the calculation. For example, the plurality of frames 300 may be associated with a closed GOP length of 14 frames.

[0114] As mentioned previously, the GOP length may measure the distance between two key frames (e.g., I-frames) in a series of video frames (e.g., such as the plurality of video frames 300). In one illustrative example, the GOP length may also be referred to herein as an I-frame time interval. The GOP length (e.g., or I-frame interval) may be measured in terms of the number or amount of frames between I-frames (e.g., as described above) and / or may be measured as the amount of time between I-frames. For example, in Figure 3In the illustrated case of the plurality of video frames 300, an I frame may be inserted and used as a key frame for every 15 frames of video. If the plurality of video frames 300 is associated with a playback speed of 30 frames per second (fps), the GOP length may also be measured as 0.5 seconds (e.g., two I frame key frames per second of video).

[0115] In some examples, efficiency and / or performance of video coding may be improved based on using variable I-frame time intervals (e.g., "variable GOP lengths"). In some aspects, improved video coding efficiency for different types of video content may be associated with different GOP lengths. For example, relatively static video content may be coded using a relatively large GOP length (e.g., based on the video content changing relatively little over time), while video content with fast-moving or many moving objects may be coded using a relatively small GOP length (e.g., based on the video content changing relatively quickly over time).

[0116] In some examples, variable length GOP video coding may be performed based on video content analysis and / or motion analysis. For example, video content analysis and / or motion analysis may be performed to determine an amount of change associated with video content (e.g., an amount of change over time). Such techniques may achieve improved video coding performance, which may generally be associated with high computational complexity and power consumption. For example, computational complexity and power consumption may be increased based on performing video content analysis and / or motion analysis separately from video coding.

[0117] In some cases, fixed-length GOP video coding (e.g., fixed I-frame time interval video coding) may be utilized based on computational constraints, power constraints, and / or coding time constraints associated with a given video coding device. For example, wireless display sharing and / or WiFi display technologies (e.g., such as ) can be used to wirelessly share video content between WiFi devices by capturing and encoding the display content of a source device (e.g., a smartphone) and sending the encoded video data to a receiving device (e.g., a television). Based on the computing resources and / or power and energy resources available at the source device (e.g., a smartphone), WiFi display technologies such as ) typically utilize a fixed-length GOP to encode (and subsequently decode) video data for display at a source device. In some aspects, based at least in part on the source device (e.g., smartphone) video display data varying in source, type, content, etc., using a fixed-length GOP can reduce the need for WiFi display technologies (such as ) is performed. Systems and techniques are needed that can be used to perform variable length GOP video decoding (e.g., variable I-frame time interval video decoding) without using computationally intensive techniques (such as video content analysis or motion analysis). There is also a need for systems and techniques that can be used to implement WiFi display sharing (e.g., ) in a power-efficient manner to perform variable length GOP video decoding (e.g., variable I-frame time interval video decoding).

[0118] As described in more detail below, systems and techniques are provided for performing video coding using variable intra-frame (I-frame) time intervals and / or variable length group of pictures (GOP) lengths. For example, the systems and techniques may perform video coding (e.g., encoding and / or decoding) using variable I-frame intervals and / or variable GOP lengths determined based on information such as video frame layer information, video frame layer geometry, and the like. For example, the systems and techniques may obtain and analyze layer information associated with one or more layers included in a plurality of frames to be encoded. In some aspects, the plurality of frames to be encoded may be used to perform wireless display sharing (e.g., )’s smart phone or other mobile computing device.

[0119] For example, Figure 4A FIG. 4 is a diagram illustrating an example of a frame layer that may be associated with a frame of captured video display data. In one illustrative example, a frame 410 of captured video display data may be associated with a wireless display sharing (e.g., WiFi display sharing, The captured frames 410 of video display data may be associated with layer information including at least a layer index 420 and a layer type 430. For example, each frame of captured video display data (e.g., such as captured frame 410 of video display data) may include one or more layers, wherein separate layers are associated with different icons, applications, UI elements, etc. displayed in the captured frame 410 of video display data.

[0120] For example, a first layer may be associated with a background displayed or included in frame 410 of the captured video display data; a second layer may be associated with a status bar system user interface (UI) element (e.g., displaying system information such as the current time, notifications, connected networks, notifications, battery percentage, etc.); a third layer may be associated with application icons rendered on top of the background display layer; a fourth layer may be associated with a picture-in-picture (PIP) or floating window UI element that displays a mobile application on top of the other layers; and so on.

[0121] In some examples, the systems and techniques may utilize frames of captured video display data that are Frames or other frames associated with wireless display sharing. In some aspects, A frame may include or be associated with layer information for each layer included in a given frame. In some aspects, each frame of captured video display data (e.g., such as captured frame 410 of video display data) may include one or more layers. For example, frame 410 may include 1 to 15 layers that make up a final rendered view of frame 410 that is provided as a or other wireless display sharing frames to capture and share.

[0122] For example, the frames of captured video display data may be generated or otherwise obtained by a source device (e.g., a smartphone) performing wireless display sharing. The layer information for each respective frame may be determined by the source device (e.g., populated by the source device) during the wireless display sharing process. Frames or other frames that include the geometry of the individual layers included in each corresponding frame, the coordinates of each layer, the format or layer type of each layer, etc.

[0123] For example, Figure 4A As illustrated, the captured frame 410 of video display data may include layers with indices 0, 1, 6, 7, and 8 and a layer type (e.g., composition type 430) "SDE" (e.g., Snapdragon Display Engine or other hardware display processing unit). For example, the layer type or composition type "SDE" may indicate that the corresponding layer is to be composed by an SDE or other hardware display processing unit. The frame 410 may additionally include a layer with index 11 and a layer type (e.g., composition type 430) "GPU_TARGET". In some aspects, layer information such as layer index 420 and layer composition type 430 may be provided by a source device associated with the frame 410 of captured video display data.

[0124] Figure 4B 4 is a diagram illustrating an example layer stack 450 associated with a frame of captured video display data according to some examples. For example, layer stack 450 may be an application layer stack in which applications running on (e.g., currently rendering on a display of) a smartphone or other mobile computing device are each associated with one or more layers in layer stack 450.

[0125] The layer stack 450 may include a plurality of different layers ordered (e.g., "stacked") in a z-order, where the layer on the top of the stack 450 is the top-most layer visible when the layer stack 450 is rendered on a display of a smartphone or other computing device. For example, the layer associated with "Application 4" may be the top-most layer visible in a frame of captured video display data associated with the example layer stack 450, while the layer associated with "Application 0" may be the bottom-most layer (e.g., of the layers included in the layer stack 450) rendered below the rest of the layers of the layer stack 450.

[0126] As previously mentioned, each frame of captured video display data may include multiple layers (e.g., also referred to as "sub-layers"). In some aspects, the number and / or type of layers between consecutive frames of captured video display data may change. Each layer may be associated with layer information and / or layer parameters. For example, each layer may be associated with a layer name, a z-order, a format, a layer category, one or more transforms, one or more tags, metadata, frame region coordinates, etc. In an illustrative example, the layer information utilized by the systems and techniques described herein may be the same or similar to information used by a source device (e.g., a source device participating in wireless display sharing) to determine frame composition, geometry rendering, tone mapping, and / or various other display post-processing operations.

[0127] In some aspects, the layer information may determine the geometry of the resulting frame. For example, the geometry of a frame including five layers may be determined based on the layer information associated with each of the five layers. Based on the layer information that determines the geometry of the resulting frame, the layer information may also determine the final content pixels rendered (or otherwise visible) in a final frame generated based on the layer information.

[0128] The layer name may be specified by an application (e.g., running on a source device) and / or an operating system (OS) framework (e.g., an OS framework of a source device). In some aspects, the layer name may also be referred to as a layer readable ID or layer identifier. Based on a layer name parameter specified at least in part by an application running on a source device, layers associated with different applications may have different names and naming conventions. In an illustrative example, the layer name associated with a given layer may be used to determine a use or use case associated with the given layer, one or more content classifications associated with the given layer, and / or application information associated with the given layer. For example, the system and technology may use a layer name parameter to determine whether a given layer is a gaming layer or a non-gaming layer (e.g., because a gaming layer may be associated with rapidly changing content and may benefit from a shorter GOP length than a non-gaming layer).

[0129] The layer information may also include a z-order of some (or all) of the layers associated with a given frame of captured video display data. The z-order of the layers may indicate the relative positioning of the layers along the z-axis (e.g., relative to other layers of the same frame). For example, the z-order of the layers may be used to determine which layers are on top of other layers, which layers are below other layers, and so on.

[0130] The layer information associated with a given frame may include a frame layer number or a frame layer quantity parameter that indicates the number of layers included in a given frame of captured video display data. For example, the frame layer number information may indicate that a given frame of captured video display data includes a total of eight layers. In some aspects, the frame layer number information may additionally identify a subset of layers associated with a particular application or source. For example, the frame layer number information may indicate that a given frame of captured video display data includes a total of eight layers, where four layers are associated with a first application, two layers are associated with a second application, and two layers are associated with a source device OS.

[0131] In some aspects, the layer information may include transformation information. For example, the transformation information may indicate one or more transformations applied to the layer content. In some aspects, the layer transformation information may indicate how a given layer is rotated or flipped (e.g., if at all). For example, the layer transformation information may indicate that the layer undergoes no transformation, is rotated 90 degrees clockwise, is rotated 90 degrees counterclockwise, and so on.

[0132] In some aspects, the layer information may include display frame information. For example, the display frame information may indicate where a given layer is located within its associated frame (e.g., a frame of captured video display data that includes the given layer). In an illustrative example, the display frame information may include two-dimensional (2D) coordinates that indicate the positioning of the given layer within its associated frame and the extent (e.g., size) of the given layer within its associated frame. For example, the display frame information may include (x, y) coordinates indicating the location of one or more corners of a given layer included in a frame.

[0133] In some aspects, the layer information may include metadata information. For example, the metadata information may be information associated with rendering some (or all) of the visual content or pixels of a given layer. In an illustrative example, the metadata information may include one or more of layer color, layer brightness, high dynamic range (HDR) information, electro-optical transfer function (EOTF) information, and / or optoelectronic transfer function (OETF) information, etc.

[0134] In some aspects, layer information may include a composition type (e.g., also referred to as a "comptype") The composition type may indicate whether a given layer will be composed by a data processing unit (DPU), a graphics processing unit (GPU), or other hardware processor.

[0135] In some aspects, the layer information may include one or more flags indicating specific processing required for a given layer. For example, a layer may be associated with a flag indicating that the layer requires security processing, external display-only processing, etc.

[0136] In some aspects, the layer information may include a layer format. The layer format may include RGB or YUV, RGB1010102 or FP16, etc.

[0137] In some examples, layer information may be obtained for each layer included in a given frame of captured video display data. In one illustrative example, layer information for layers included in a given frame may be obtained in a combined list of layer information. For example, Figure 4C An example table 400c including an example listing of layer information obtainable for a captured frame of video display data is depicted.

[0138] In some aspects, a conventional video stream (e.g., captured by a camera or a camera included in a smartphone or other mobile device) may not include any layer information because such video streams are captured as a single layer recording. In an illustrative example, the systems and techniques described herein may use layer information included in or otherwise associated with a frame of captured video display data to determine scene information for setting an adaptive or variable length GOP (e.g., an adaptive or variable I-frame time interval). In some cases, the layer information may be associated with a frame of captured video display data based on the layer information used to initially render the video display data on a display of a source device participating in wireless display sharing. In some examples, the layer information associated with a frame of captured video display data may include rich scene information that can be analyzed more effectively than pixel-based content or motion analysis. Based on the rich scene information determined from the layer information, the systems and techniques may implement scene change analysis, notifications, and / or I-frame triggering, as described in more depth below.

[0139] Figure 5 is a flow diagram illustrating an example of a process 500 that may be implemented by the systems and techniques described herein to perform video decoding using a variable length GOP (e.g., a variable I-frame time interval) that is adaptively determined based on layer information associated with one or more frames of video display data.

[0140] In one illustrative example, one or more frames of video display data may be from a user participating in wireless display sharing (e.g., such as ). For example, the plurality of frames 510 of captured video display data may include a current frame 504 (e.g., associated with time t), a previous frame 502 (e.g., associated with time t-1), and n additional frames until a final frame associated with time t+n.

[0141] The plurality of frames 510 of captured video display data may be encoded with a variable GOP length and a variable I-frame time interval based on monitoring and analyzing layer information of some (or all) of the corresponding frames included in the plurality of frames 510. For example, at block 550, the systems and techniques may monitor and analyze layer information obtained for or otherwise associated with a current frame 504 (e.g., a frame associated with a current time t). In some aspects, monitoring and analyzing the layer information of the current frame 504 at block 550 may include obtaining the layer information of the current frame 504 prior to performing the analysis.

[0142] At block 552, the systems and techniques may compare the current frame 504 to one or more previous frames (e.g., such as a previous frame 502 associated with time t-1) to determine whether a scene change has occurred and / or to determine whether a frame geometry change has occurred. For example, layer information associated with the previous frame 502 may be compared to layer information associated with the current frame 504 (e.g., layer information determined and analyzed at block 550). In some aspects, the layer information associated with the previous frame 502 may be stored in a previous time step in which the frame 502 was the current frame provided to block 550 for analysis.

[0143] In one illustrative example, a scene change or frame geometry change may be detected at block 552 based on one or more parameters included in the layer information of the previous frame 502 and / or included in the layer information of the current frame 504 changing by an amount greater than a predetermined threshold. For example, a scene or frame geometry change may be detected at block 552 based on a change in the number of layers (e.g., increasing from four layers in frame 502 to six layers in frame 504) and / or based on one or more layer size changes (e.g., a layer size increases by at least 25% from frame 502 to frame 504). In some cases, a scene or frame geometry change may be detected at block 552 based on one or more layer position changes (e.g., moving up / down, left / right by an amount greater than 50% of the corresponding size of the layer) and / or based on one or more layer format changes (e.g., changing from HDR YUV 10-bit in frame 502 to SDR YUV 8-bit in frame 504).

[0144] In some aspects, a scene change or frame geometry change may additionally or alternatively be detected at block 552 based on identifying a change in the primary layer from a previous frame 502 to a current frame 504. For example, the primary layer of a given frame 510 of captured video display data may be the highest positioned layer that is large enough (e.g., in pixels). Figure 4B4. The z-order of the illustrated layer stack 450. In some cases, the primary layer for a given frame may be the highest positioned layer that exceeds a predetermined size threshold (e.g., at or near the top of the given frame). For example, the predetermined size threshold may be 30% of the pixel area of ​​the given frame.

[0145] Based on detecting a scene change or frame geometry change at block 552 (e.g., Figure 5 ), the systems and techniques can automatically trigger I frame encoding at block 554. For example, a new I frame can be encoded based on the current frame 504 and used as a key frame for the encoded video data generated from the plurality of frames 510 of captured video display data.

[0146] A new GOP length may additionally be determined and applied at block 558 (e.g., in conjunction with a new I-frame encoding triggered at block 554 in response to detecting a significant scene change or frame geometry change at block 552). For example, the new GOP length may indicate the number of frames (e.g., the number of the plurality of frames 510 of captured video display data) that are to be encoded as B-frames or P-frames before another I-frame keyframe is to be encoded. In one illustrative example, the new GOP length determined and applied at block 558 may be based on a GOP length determined for a primary layer identified for the currently encoded frame 504. As previously mentioned, the primary layer of a given frame 510 of captured video display data may be the highest positioned layer that is large enough in size (e.g., with a size of 100). Figure 4B 4.) In some aspects, the GOP length may be based on the type of primary layer and / or may be based on layer information associated with the primary layer.

[0147] For example, the GOP length may be determined based on one or more of a layer name, a layer format, a layer metadata, and / or a layer size associated with an identified primary layer of a currently encoded frame (e.g., current frame 504) of the captured video display data. In some aspects, a mapping of a GOP length value to a combination of layer information may be predetermined. In some examples, a machine learning (ML) network or classifier may be trained to generate a GOP length value based on layer information associated with the primary layer. For example, a neural network or a deep neural network (DNN) may be trained as a classifier on layer information such as layer name, layer format, layer metadata, layer size, etc. In some examples, a neural network classifier may be trained to output a GOP length value based on receiving layer information of the primary layer as input. In some examples, a neural network classifier may be trained to output a semantic classification indicating a primary layer type based on receiving layer information of the primary layer as input (e.g., and the semantic classification may then be mapped to a predetermined GOP length value of the corresponding semantic classification).

[0148] Different GOP lengths may be determined or utilized for different types of primary layers based at least in part on the content type of the primary layer and / or the expected frequency of frame geometry changes for the content type of the primary layer. For example, a primary layer that is a gaming layer may be associated with a relatively short GOP length (e.g., 60 frames), while a primary layer that is an email client / application layer may be associated with a relatively long GOP length (e.g., 120 frames). In some examples, a video playback layer and / or a YUV color format layer may be associated with a shorter GOP length than the gaming layer GOP length (e.g., a video playback layer and / or a YUV color format layer may be associated with a GOP length of 30 frames). In some aspects, a primary layer that includes relatively static content (such as content from a text-based chat or messaging application) may have a relatively long GOP length (e.g., 300 frames). Similarly, a primary layer that includes relatively static content (such as content from a slideshow application) may also have a relatively long GOP length (e.g., 300 frames).

[0149] After triggering encoding of a new I-frame (e.g., a keyframe) for the currently encoded frame 504 at block 554 and applying a new GOP length based on the layer information or layer type of the primary layer of the currently encoded frame 504 (e.g., at block 558), the systems and techniques may proceed to the next frame at block 566. For example, after encoding the current frame 504 (e.g., associated with time t), the systems and techniques may proceed from block 566 to analyzing layer information for the next frame (e.g., associated with time t+1) by returning to block 550.

[0150] Returning to the discussion of encoding the current frame 504, if a scene change or frame geometry change is not detected at block 552 (e.g., a "No" option), the systems and techniques may proceed to block 556, which determines whether a display idle state has occurred or is occurring. For example, at block 556, the systems and techniques may determine that the video content represented in a frame of captured display data from a source device has transitioned to an idle state. In some aspects, the idle state may be associated with no content change detected within a predetermined period of time and / or no additional rendering or refreshing of a frame of captured display data detected. For example, the display idle state may be detected or triggered at block 556 based on no scene or frame geometry change being detected within a predetermined number of consecutive frames at block 552.

[0151] In some examples, detecting or determining a display idle state may cause the systems and techniques to apply a new relatively longer GOP length associated with the display idle state (e.g., also referred to as a "display idle GOP length"). For example, the display idle GOP length may be 300 frames, although a greater or lesser number of frames may also be used for the display idle GOP length. For example, if a display idle state is detected at block 556 (e.g., a "yes" option), the systems and techniques may apply the display idle GOP length at block 560. If a display idle state is not detected at block 556 (e.g., a "no" option), the systems and techniques may apply (e.g., maintain) the current GOP length at block 562. In one illustrative example, the current GOP length that may be applied or maintained at block 562 (e.g., when display idle is not detected) may be the same as the most recently applied new GOP length associated with the most recent I-frame (e.g., the most recently applied new GOP length since the last time block 558 was reached).

[0152] After applying the display idle GOP length at block 560 or maintaining the current GOP length at block 562, the systems and techniques may encode the currently encoded frame 504 as a P frame or a B frame (e.g., not an I frame, since the current frame 504 is encoded as an I frame only when block 554 is reached based on detecting a scene or frame geometry change at block 552). After encoding the currently encoded frame 504 as a P frame or a B frame at block 564, the systems and techniques may return process 500 to block 550 based on block 566 to advance to the next frame (e.g., a frame associated with time t+1).

[0153] In an illustrative example, if a previous (e.g., associated with time t-1) frame 502 includes four layers and a current (e.g., associated with time t) frame 504 includes six layers, a scene change and / or frame geometry change may be detected at box 552, new I frame encoding may be triggered at box 554, and a new GOP length may be applied at box 558 (e.g., where the new GOP length is based on a GOP length associated with a primary layer of the currently encoded frame 504).

[0154] In another illustrative example, if the previous frame 502 included a layer having layer size information given by 0 0 1080 2400 and the current frame 504 includes the same layer having layer size information now given by 220 180 540 1200, a frame geometry change may be detected at box 552, new I frame encoding may be triggered at box 554, and a new GOP length may be applied at box 558.

[0155] In another illustrative example, if the previous frame 502 included a layer having metadata information and / or format information indicating that the layer was in HDR YUV 10-bit format, and the current frame 504 includes the same layer having metadata and / or format information now indicating that the layer is an SDR YUV 8-bit layer, a scene change and / or frame geometry change may be detected at box 552, new I frame encoding may be triggered at box 554, and a new GOP length may be applied at box 558.

[0156] In another example, if the layer name changes between the previous frame 502 and the current frame 504, a scene change may be detected at block 552. For example, a scene change may be detected based on determining that the layer name from the previous frame 502 is not included in the current frame 504. In some aspects, the layer name that is present in the previous frame 502 but not in the current frame 504 may indicate that one or more new layers are included in the current frame 504. In some cases, the layer name that is present in the previous frame 502 but not in the current frame 504 may indicate that a new primary layer will be active for or included in the current frame 504. In either case, a scene change may be detected at block 552, new I-frame encoding may be triggered at block 554, and a new GOP length may be applied at block 558.

[0157] In some aspects, the plurality of frames 510 of captured video display data may be consecutive frames that are captured or obtained and then encoded in real time. In some aspects, the plurality of frames 510 of captured video display data may include a plurality of frames associated with a wireless display sharing (e.g., Miracast) session, in which a source device (e.g., a smartphone or other computing device) mirrors its display content to a target device (e.g., a television). For example, frame 502 may be the first frame captured for a wireless display sharing session, and frame 508 may represent the last frame captured for the same wireless display sharing session.

[0158] As previously mentioned, the GOP length and I-frame interval used to decode the plurality of frames 510 of captured video display data may vary over time. For example, if frame 502 includes eight layers and frame 504 includes four layers, a scene change is detected at block 552, thereby triggering a new I-frame (e.g., keyframe) encoding at block 554 and applying a new GOP length (e.g., specific to the primary layer of frame 504) at block 558. If the primary layer of the currently encoded frame 504 is a YUV format layer having a 4K resolution, the new GOP length applied at block 558 may be a target GOP length associated with the YUV 4K layer. For example, if the YUV 4K layer is associated with a target GOP length of 30 frames, the new target GOP length applied at block 558 may be 30 frames.

[0159] If no scene change or frame geometry change exceeding a predetermined threshold is subsequently detected within one minute (e.g., the determination at block 552 is "no" for frame 510, which is one minute's worth of frame after frame 504), the target GOP length will remain at 30 frames and a new I frame will be triggered every 30 frames. If frame 510 is associated with a playback speed of 30 fps, a new I frame will be triggered once per second, for a total of 60 I frames within a one minute period in which no scene change or frame geometry change is detected at block 552.

[0160] If the number of frame layers currently being encoded increases from four to six at the one minute mark, a scene change is detected at box 552, a new I frame is immediately triggered at box 554 (e.g., regardless of whether the target GOP length of 30 frames has been met), and the new target GOP length is applied at box 558.

[0161] If no further scene changes or frame geometry changes are detected within the next 20 minutes at block 552, the I-frame triggering will continue for a 20 minute period based on the current target GOP length. Initially, the target GOP length may be the GOP length determined at block 558 when the frame layer count increases from four to six. After a predetermined number of consecutive frames are analyzed at block 552 without detecting a scene change or frame geometry change, the target GOP length is then updated to the display idle GOP length (e.g., at block 560). For example, the display idle GOP length may be a relatively long GOP length (e.g., such as 300 frames or 10 seconds for 30 fps video).

[0162] If at the 20 minute mark, the number of frame layers remains the same (e.g., remains at six layers) but one or more of the layer size and / or layer coordinates change, then a scene change will be detected at block 552 and new I-frame encoding will be immediately triggered at block 554 (e.g., even if the display idle GOP length has not been reached). A new GOP length may then be determined based on the primary layer of the frame associated with the layer size change and / or layer coordinate change, and may be applied at block 558 to replace the display idle GOP length.

[0163] In one illustrative example, the systems and techniques described herein may be used to perform real-time or "online" video decoding. For example, in the context of wireless display sharing (e.g., Miracast) from a source device to a target device, frames of captured video display data are obtained at the source device, encoded, and sent for mirrored display on the target device in substantially real time. Based on detecting scene changes and / or frame geometry changes without performing content analysis or motion analysis based on analyzing pixel-level information, the systems and techniques may utilize variable GOP lengths and / or I-frame intervals to more efficiently provide real-time video encoding and decoding.

[0164] Figure 6 6 is a flowchart illustrating an example of a process 600 for encoding or decoding (decoding) video data according to various aspects described herein. At block 602, process 600 includes obtaining a video data frame associated with a display of a computing device, wherein the video data frame includes one or more layers. For example, the video data frame may be obtained from Figure 1 The illustrated video source 102 associated with a display of an encoding device 104 obtains a video data frame. In some examples, the video data frame may be associated with Figure 4A The illustrated frame 410 (which includes Figure 4B In some cases, the video data frame may be the same as or similar to one or more layers 450 illustrated. Figure 5 One or more of the illustrated frames 510 (eg, one or more of frames 502, 504, 508, etc.) are the same or similar.

[0165] In some cases, the video data frame includes video display data displayed on a display of the computing device. In some examples, the video data frame can be a frame of captured video display data associated with wireless display sharing (e.g., Miracast) from the computing device to a second computing device. For example, the video data frame can be a frame of captured video display data associated with wireless display sharing (e.g., Miracast) from the computing device to a second computing device. Figure 1 The illustrated encoding device 104 is Figure 1 The illustrated wireless display of the decoding device 112 shares the associated captured frames of video display data.

[0166] At block 604, process 600 includes comparing layer information associated with one or more layers included in a video data frame with layer information associated with one or more layers included in a previous video data frame. For example, the layer information may include at least Figure 4A The illustrated layer index value 420 and the synthesis type 430 information. In some cases, the layer information may be associated with one or more layers (such as Figure 4B The layer information may include one or more layers included in the illustrated application layer stack 450. In some cases, the layer information may include Figure 4CSome (or all) of the layer information illustrated in Table 400c depicted in FIG.

[0167] In some cases, the layer information may include, for each respective layer included in the one or more layers, at least one of a layer name associated with each respective layer, a layer format associated with each respective layer, and one or more coordinates associated with each respective layer. In some examples, the layer information may include at least one of a layer number or a frame layer number (e.g., such as Figure 4A In some cases, the video data frame and the previous video data frame may be consecutive frames of captured video display data. Figure 5 The illustrated captured frame 504 of display data is the same or similar, and the previous video data frame may be the same as Figure 5 The illustrated captured display data frame 502 is the same or similar.

[0168] In some examples, the video data frame and the previous video data frame may be multiple frames of captured video display data (e.g., such as Figure 5 The encoded video data may include consecutive frames included in the illustrated plurality of frames 510). In some examples, at least a portion of the plurality of frames of captured video display data may be encoded using inter-frame prediction frames, as described in more depth below. In some cases, the encoded video data may be sent to a second device associated with the same wireless display sharing session as the computing device. The encoded video may include inter-frame prediction frames and at least one of unidirectional prediction frames or bidirectional prediction frames generated based on encoding at least the portion of the plurality of frames of captured video display data.

[0169] At block 606, process 600 includes generating an inter-prediction frame using the video data frame based on determining a frame geometry change associated with the video data frame. For example, determining the frame geometry change associated with the video data frame may be based on comparing layer information associated with one or more layers included in the video data frame and layer information associated with one or more layers included in a previous video data frame. In some cases, the frame geometry change may be determined based on determining that layer information associated with one or more layers included in the video data frame has changed by greater than a threshold amount compared to layer information associated with one or more layers included in a previous video data frame.

[0170] At block 608, process 600 includes determining an updated group of pictures (GOP) length based on layer information associated with one or more layers included in the video data frame. For example, the updated GOP length may be determined based on layer information associated with a primary layer included in the video data frame. In some cases, the primary layer included in the video data frame may be rendered using a z-order that is greater than a corresponding z-order associated with one or more additional layers included in the video data frame.

[0171] In some examples, process 600 may include determining that a frame geometry change is not associated with a video data frame based on layer information associated with one or more layers included in the video data frame changing less than a threshold amount compared to layer information associated with one or more layers included in a previous video data frame. In some cases, a display idle state associated with a video data frame may be detected based on the video data frame and a predetermined number of previous video data frames not being associated with a frame geometry change. In some examples, a display idle GOP length may be applied, wherein the display idle GOP length is greater than an updated GOP length. In some cases, a video data frame may be encoded as a predicted frame (P frame) or a bidirectional frame (B frame) based on the video data frame not being associated with a frame geometry change.

[0172] In some cases, process 600 may be performed by a decoding device (e.g., Figure 1 and Figure 8 In some cases, process 600 may be performed by an encoding device (e.g., Figure 1 and Figure 7 For example, process 600 may include generating an encoded video bitstream including information associated with a block of video data. In some examples, process 600 may include storing the encoded video bitstream (e.g., in at least one memory of the device). In some examples, process 600 may include transmitting the encoded video bitstream (e.g., using a transmitter of the device).

[0173] In some implementations, the processes (or methods) described herein may be performed by a computing device or apparatus (such as Figure 1 For example, the system 100 shown in FIG. Figure 1 and Figure 7 The encoding device 104 shown in the figure is provided by another video source side device or a video sending device. Figure 1 and Figure 8The decoding device 112 shown and / or these processes are performed by another client-side device (such as a player device, a display, or any other client-side device). In some cases, the computing device or apparatus may include a processor, a microprocessor, a microcomputer, or other components of a device configured to perform the steps of the process described herein. In some examples, the computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. In some examples, the camera or other capture device that captures the video data is separated from the computing device, in which case the computing device receives or obtains the captured video data. The computing device may also include a network interface configured to communicate video data. The network interface may be configured to communicate data based on an Internet Protocol (IP) or other types of data. In some examples, the computing device or apparatus may include a display for displaying output video content (such as a sample of a picture of a video bitstream).

[0174] The processes may be described relative to logical flow diagrams whose operations represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, and data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the processes.

[0175] In addition, these processes can be performed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, or a combination thereof. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0176] The decoding techniques discussed herein can be implemented in an example video encoding and decoding system (e.g., system 100). In some examples, a system includes a source device that provides encoded video data that will be decoded later by a destination device. Specifically, the source device provides video data to the destination device via a computer-readable medium. The source device and the destination device may include any of a variety of devices, including desktop computers, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets (such as so-called "smart" phones, so-called "smart" tablets), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, the source device and the destination device may be equipped for wireless communication.

[0177] The destination device may receive the encoded video data to be decoded via a computer-readable medium. The computer-readable medium may include any type of medium or device capable of moving the encoded video data from the source device to the destination device. In one example, the computer-readable medium may include a communication medium that enables the source device to directly send the encoded video data to the destination device in real time. The encoded video data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to the destination device. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a part of a packet-based network such as a local area network, a wide area network, or a global network (such as the Internet). The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from the source device to the destination device.

[0178] In some examples, the encoded data can be output from the output interface to the storage device. Similarly, the encoded data can be accessed from the storage device through the input interface. The storage device may include any data storage medium in a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. The destination device can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server that can store the encoded video data and send the encoded video data to the destination device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The destination device can access the encoded video data through any standard data connection (including an Internet connection). This can include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a DSL, a cable modem, etc.) suitable for accessing the encoded video data stored on the file server, or a combination of the two. The transmission of the encoded video data from the storage device may be streaming transmission, download transmission, or a combination thereof.

[0179] The techniques of the present disclosure are not necessarily limited to wireless applications or settings. These techniques can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions (such as Dynamic Adaptive Streaming over HTTP (DASH)), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, the system can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0180] In one example, the source device includes a video source, a video encoder, and an output interface. The destination device may include an input interface, a video decoder, and a display device. The video encoder of the source device may be configured to apply the technology disclosed herein. In other examples, the source device and the destination device may include other components or arrangements. For example, the source device may receive video data from an external video source such as an external camera. Similarly, the destination device may be docked with an external display device rather than including an integrated display device.

[0181] The above example system is just an example. The technology for processing video data simultaneously can be performed by any digital video encoding and / or decoding device. Although generally speaking the technology of the present disclosure is performed by a video encoding device, these technologies can also be performed by a video encoder / decoder (commonly referred to as a "codec"). In addition, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the destination device are just examples of such decoding devices, wherein the source device generates decoded video data to send to the destination device. In some examples, the source device and the destination device can operate in a substantially symmetrical manner so that each of these devices includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting or video telephony.

[0182] The video source may include a video capture device, such as a video camera, a video archiving unit including previously captured video, and / or a video feed interface for receiving video from a video content provider. As another alternative, the video source may generate computer graphics-based data as the source video, or generate a combination of real-time video, archived video, and computer-generated video. In some cases, if the video source is a video camera, the source device and the destination device may form a so-called camera phone or video phone. However, as mentioned above, the technology described in the present disclosure is generally applicable to video decoding, and may be applicable to wireless and / or wired applications. In each case, the captured, pre-captured or computer-generated video may be encoded by a video encoder. The encoded video information may then be output to a computer-readable medium by an output interface.

[0183] As noted above, computer-readable media may include transient media (such as wireless broadcasts or wired network transmissions) or storage media (i.e., non-transitory storage media), such as hard disks, flash drives, compact discs, digital video discs, Blu-ray discs, or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from a source device and provide the encoded video data to a destination device, for example, via a network transmission. Similarly, a computing device of a media production facility (such as a disc stamping facility) may receive encoded video data from a source device and produce a disc containing the encoded video data. Therefore, in various examples, computer-readable media may be understood to include one or more computer-readable media in various forms.

[0184] An input interface of the destination device receives information from a computer-readable medium. The information of the computer-readable medium may include syntax information defined by a video encoder, which is also used by a video decoder, and includes syntax elements that describe characteristics and / or processing of blocks and other units of decoding (e.g., groups of pictures (GOPs)). A display device displays the decoded video data to a user and may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device. Various examples of the present application have been described.

[0185] The details of the encoding device 104 and the decoding device 112 are respectively described in Figure 7 and Figure 8 Shown in. Figure 7 1 is a block diagram illustrating an example encoding device 104 that can implement one or more of the techniques described in the present disclosure. For example, the encoding device 104 can generate the syntax structures described herein (e.g., syntax structures of VPS, SPS, PPS, or other syntax elements). The encoding device 104 can perform intra-frame prediction decoding and inter-frame prediction decoding of video blocks within a video slice. As previously described, intra-frame decoding relies at least in part on spatial prediction to reduce or remove spatial redundancy within a given video frame or picture. Inter-frame decoding relies at least in part on temporal prediction to reduce or remove temporal redundancy within adjacent or surrounding frames of a video sequence. Intra-frame mode (I mode) may refer to any of several spatial-based compression modes. Inter-frame mode (such as unidirectional prediction (P mode) or bidirectional prediction (B mode)) may refer to any of several temporal-based compression modes.

[0186] The encoding device 104 includes a segmentation unit 35, a prediction processing unit 41, a filter unit 63, a picture memory 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 includes a motion estimation unit 42, a motion compensation unit 44, and an intra-frame prediction processing unit 46. For video block reconstruction, the encoding device 104 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and a summer 62. The filter unit 63 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although the filter unit 63 is not limited to a single-loop filter, the filter unit 63 is not limited to a single-loop filter, and the filter unit 63 is not limited to a single-loop filter. Figure 8 6 is shown as an in-loop filter, but in other configurations, filter unit 63 may be implemented as a post-loop filter. Post-processing device 57 may perform additional processing on the encoded video data generated by encoding device 104. In some examples, the techniques of this disclosure may be implemented by encoding device 104. However, in other cases, one or more of the techniques of this disclosure may be implemented by post-processing device 57.

[0187] like Figure 7 As shown, the encoding device 104 receives video data, and the segmentation unit 35 segments the data into video blocks. Segmentation may also include segmentation into slices, slice segments, tiles or other larger units and video block segmentation (e.g., according to a quadtree structure of LCUs and CUs). The encoding device 104 generally shows components for encoding video blocks within a video slice to be encoded. A slice may be divided into multiple video blocks (and may be divided into sets of video blocks called tiles). The prediction processing unit 41 may select one of a plurality of possible decoding modes for the current video block based on error results (e.g., decoding rate and distortion level, etc.), such as one of one or more inter-frame prediction decoding modes in a plurality of intra-frame prediction decoding modes. The prediction processing unit 41 may provide the resulting intra-frame decoded block or inter-frame decoded block to the summer 50 to generate residual block data and to the summer 62 to reconstruct the encoded block for use as a reference picture.

[0188] Intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-predictive coding of the current video block relative to one or more neighboring blocks in the same frame or slice as the current block to be coded to provide spatial compression. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-predictive coding of the current video block relative to one or more predictive blocks in one or more reference pictures to provide temporal compression.

[0189] Motion estimation unit 42 may be configured to determine an inter-prediction mode for a video slice according to a predetermined pattern for a video sequence. The predetermined pattern may designate a video slice in a sequence as a P slice, a B slice, or a GPB slice. Motion estimation unit 42 and motion compensation unit 44 may be highly integrated but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors, which estimate the motion of a video block. For example, a motion vector may indicate the displacement of a prediction unit (PU) of a video block within a current video frame or picture relative to a predictive block within a reference picture.

[0190] A predictive block is a block that is found to closely match a PU of a video block to be coded in terms of pixel difference, which may be determined by a sum of absolute difference (SAD), a sum of squared difference (SSD), or other difference metrics. In some examples, encoding device 104 may calculate values ​​for sub-integer pixel positions of a reference picture stored in picture memory 64. For example, encoding device 104 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform a motion search relative to full pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

[0191] Motion estimation unit 42 calculates a motion vector for a PU by comparing the position of a PU of a video block in an inter-coded slice to the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (List 0) or a second reference picture list (List 1), each of which identifies one or more reference pictures stored in picture memory 64. Motion estimation unit 42 communicates the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

[0192] Motion compensation performed by motion compensation unit 44 may involve extracting or generating a predictive block based on a motion vector determined by motion estimation, possibly performing interpolation to sub-pixel precision. Upon receiving the motion vector of the PU of the current video block, motion compensation unit 44 may locate the predictive block pointed to by the motion vector in the reference picture list. The encoding device 104 forms a residual video block by subtracting the pixel values ​​of the predictive block from the pixel values ​​of the current video block being decoded, thereby forming a pixel difference value. The pixel difference value forms the residual data of the block and may include both a luminance difference component and a chrominance difference component. Summer 50 represents one or more components that perform the subtraction operation. Motion compensation unit 44 may also generate syntax elements associated with video blocks and video slices for use by the decoding device 112 when decoding video blocks of the video slices.

[0193] Intra-prediction processing unit 46 may perform intra-prediction on the current block as an alternative to inter-prediction performed by motion estimation unit 42 and motion compensation unit 44, as described above. Specifically, intra-prediction processing unit 46 may determine an intra-prediction mode for encoding the current block. In some examples, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during a separate encoding process, and intra-prediction processing unit 46 may select an appropriate intra-prediction mode to use from the tested modes. For example, intra-prediction processing unit 46 may calculate rate-distortion values ​​using rate-distortion analysis for various tested intra-prediction modes, and may select an intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis typically determines the amount of distortion (or error) between an encoded block and an original unencoded block that is encoded to produce the encoded block, as well as the bit rate (i.e., the number of bits) used to produce the encoded block. Intra-prediction processing unit 46 may calculate ratios from the distortions and rates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0194] In any case, after selecting an intra-prediction mode for a block, intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode. Encoding device 104 may include in the transmitted bitstream configuration data the definition of the coding contexts for the various blocks and the most probable intra-prediction mode, the intra-prediction mode index table, and the indication of the modified intra-prediction mode index table for each of the contexts. The bitstream configuration data may include a plurality of intra-prediction mode index tables and a plurality of modified intra-prediction mode index tables (also referred to as codeword mapping tables).

[0195] After the prediction processing unit 41 generates a predictive block for the current video block via inter-frame prediction or intra-frame prediction, the encoding device 104 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block may be included in one or more TUs and applied to the transform processing unit 52. The transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (such as a discrete cosine transform (DCT) or a conceptually similar transform). The transform processing unit 52 may convert the residual video data from a pixel domain to a transform domain, such as a frequency domain.

[0196] Transform processing unit 52 may pass the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scan.

[0197] After quantization, entropy encoding unit 56 entropy encodes the quantized transform coefficients. For example, entropy encoding unit 56 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding technique. After entropy encoding by entropy encoding unit 56, the encoded bitstream may be sent to decoding device 112, or archived for later sending or retrieval by decoding device 112. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video slice being coded.

[0198] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain for later use as a reference block of a reference picture. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the reference pictures in a reference picture list. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for use in motion estimation. Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reference block for storage in picture memory 64. The reference block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block for inter-prediction of a block in a subsequent video frame or picture.

[0199] In this way, Figure 7 The encoding device 104 of the present disclosure represents an example of a video encoder configured to perform the techniques described herein. For example, the encoding device 104 can perform any of the techniques described herein, including the processes described herein. In some cases, some of the techniques of the present disclosure may also be implemented by the post-processing device 57.

[0200] Figure 8 1 is a block diagram illustrating an example decoding device 112. The decoding device 112 includes an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, a summer 90, a filter unit 91, and a picture memory 92. The prediction processing unit 81 includes a motion compensation unit 82 and an intra-frame prediction processing unit 84. In some examples, the decoding device 112 may perform substantially the same Figure 7 The encoding pass described by the encoding device 104 is a decoding pass that is the inverse of the encoding pass.

[0201] During the decoding process, the decoding device 112 receives an encoded video bitstream representing video blocks and associated syntax elements of an encoded video slice transmitted by the encoding device 104. In some examples, the decoding device 112 may receive the encoded video bitstream from the encoding device 104. In some examples, the decoding device 112 may receive the encoded video bitstream from a network entity 79 such as a server, a media-aware network element (MANE), a video editor / splicer, or other such device configured to implement one or more of the above-mentioned techniques. The network entity 79 may or may not include the encoding device 104. Some of the techniques described in the present disclosure may be implemented by the network entity 79 before the network entity 79 sends the encoded video bitstream to the decoding device 112. In some video decoding systems, the network entity 79 and the decoding device 112 may be parts of separate devices, while in other instances, the functionality described with respect to the network entity 79 may be performed by the same device including the decoding device 112.

[0202] The entropy decoding unit 80 of the decoding device 112 entropy decodes the bitstream to generate quantized coefficients, motion vectors and other syntax elements. The entropy decoding unit 80 forwards the motion vectors and other syntax elements to the prediction processing unit 81. The decoding device 112 may receive syntax elements at the video slice level and / or the video block level. The entropy decoding unit 80 may process and parse both fixed length syntax elements and variable length syntax elements in one or more parameter sets (such as VPS, SPS, and PPS).

[0203] When the video slice is decoded as an intra-coded (I) slice, the intra-prediction processing unit 84 of the prediction processing unit 81 may generate prediction data for the video block of the current video slice based on the signaled intra-prediction mode and data from a previously decoded block of the current frame or picture. When the video frame is decoded as an inter-coded (i.e., B, P, or GPB) slice, the motion compensation unit 82 of the prediction processing unit 81 generates a predictive block for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 80. The predictive block may be generated from one of the reference pictures in the reference picture list. The decoding device 112 may construct the reference frame lists (list 0 and list 1) using a default construction technique based on the reference pictures stored in the picture memory 92.

[0204] Motion compensation unit 82 determines prediction information for a video block of the current video slice by parsing the motion vectors and other syntax elements, and uses the prediction information to generate a predictive block for the current video block being decoded. For example, motion compensation unit 82 may use one or more syntax elements in a parameter set to determine a prediction mode (e.g., intra prediction or inter prediction) for coding a video block of a video slice, an inter-prediction slice type (e.g., a B slice, a P slice, or a GPB slice), construction information for one or more reference picture lists for the slice, a motion vector for each inter-coded video block of the slice, an inter-prediction state for each inter-coded video block of the slice, and other information for decoding a video block in the current video slice.

[0205] The motion compensation unit 82 may also perform interpolation based on interpolation filters. The motion compensation unit 82 may calculate interpolated values ​​for sub-integer pixels of the reference block using interpolation filters as used by the encoding device 104 during encoding of the video block. In this case, the motion compensation unit 82 may determine the interpolation filters used by the encoding device 104 from the received syntax elements and may use the interpolation filters to produce the prediction block.

[0206] Inverse quantization unit 86 inverse quantizes or dequantizes the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 80. The inverse quantization process may include determining the degree of quantization using quantization parameters calculated by encoding device 104 for each video block in the video slice, and likewise determining the degree of inverse quantization that should be applied. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT or other suitable inverse transform), an inverse integer transform, or a conceptually similar inverse transform process to the transform coefficients to produce residual blocks in the pixel domain.

[0207] After motion compensation unit 82 generates a predictive block for the current video block based on the motion vector and other syntax elements, decoding device 112 forms a decoded video block by adding the residual block from inverse transform processing unit 88 to the corresponding predictive block generated by motion compensation unit 82. Summer 90 represents one or more components that perform this summing operation. If desired, a loop filter (in the decoding loop or after the decoding loop) may also be used to smooth pixel transitions or otherwise improve video quality. Filter unit 91 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although filter unit 91 is Figure 8 91 is shown as an in-loop filter, but in other configurations, filter unit 91 may be implemented as a post-loop filter. The decoded video blocks in a given frame or picture are then stored in picture memory 92, which stores reference pictures for subsequent motion compensation. Picture memory 92 also stores the decoded video for later viewing on a display device such as Figure 1 The video is presented on the video destination device 122 shown.

[0208] In this way, Figure 8 The decoding device 112 of represents an example of a video decoder configured to perform the techniques described herein. For example, the decoding device 112 can perform any of the techniques described herein, including the processes described herein.

[0209] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions that may represent any combination of a process, function, subroutine, program, routine, subroutine, module, software package, category, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0210] In some examples, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0211] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be appreciated by those skilled in the art that examples can be implemented without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in methods implemented with a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these examples and aspects in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing each example and aspect.

[0212] Individual examples and aspects may be described above as processes or methods, which are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flow charts may describe operations as sequential processes, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0213] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0214] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be implemented in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0215] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0216] In the foregoing description, various aspects of the present application are described with reference to the specific examples of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Although the illustrative examples and aspects of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, each example and aspect can be utilized in any number of environments and applications beyond those described herein without departing from the broader essence and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order.

[0217] One of ordinary skill in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.

[0218] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0219] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0220] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0221] Various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples and aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0222] The technology described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technology can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these technologies can be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions, which execute one or more of the above methods when executed. Computer-readable data storage media can form part of a computer program product, which may include packaging materials. Computer-readable media may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0223] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0224] Exemplary embodiments of the present disclosure include:

[0225] Aspect 1: A device for processing video data, the device comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor being configured to: obtain a video data frame associated with a display of a computing device, wherein the video data frame comprises one or more layers; compare layer information associated with the one or more layers included in the video data frame with layer information associated with one or more layers included in a previous video data frame; generate an inter-frame prediction frame using the video data frame based on determining a frame geometry change associated with the video data frame; and determine an updated group of pictures (GOP) length based on the layer information associated with the one or more layers included in the video data frame.

[0226] Aspect 2: The apparatus according to aspect 1, wherein the at least one processor is configured to determine the updated GOP length based on layer information associated with a primary layer included in the video data frame.

[0227] Aspect 3: The apparatus of aspect 2, wherein the primary layer included in the video data frame is rendered using a z-order greater than a corresponding z-order associated with one or more additional layers included in the video data frame.

[0228] Aspect 4: An apparatus according to any one of Aspects 1 to 3, wherein the layer information includes, for each corresponding layer included in the one or more layers, at least one of a layer name associated with each corresponding layer, a layer format associated with each corresponding layer, and one or more coordinates associated with each corresponding layer.

[0229] Aspect 5: The apparatus according to any one of aspects 1 to 4, wherein the layer information comprises at least one of a layer number or a frame layer number.

[0230] Aspect 6: An apparatus according to any one of Aspects 1 to 5, wherein the at least one processor is configured to determine the frame geometry change associated with the video data frame based on comparing the layer information associated with the one or more layers included in the video data frame and the layer information associated with the one or more layers included in the previous video data frame.

[0231] Aspect 7: An apparatus according to Aspect 6, wherein, in order to determine the frame geometry change associated with the video data frame, the at least one processor is configured to: determine that the layer information associated with the one or more layers included in the video data frame has changed by more than a threshold amount compared to the layer information associated with the one or more layers included in the previous video data frame.

[0232] Aspect 8: An apparatus according to any one of Aspects 1 to 7, wherein the at least one processor is further configured to determine that a frame geometry change is not associated with the video data frame based on the layer information associated with the one or more layers included in the video data frame changing by less than a threshold amount compared to the layer information associated with the one or more layers included in the previous video data frame.

[0233] Aspect 9: An apparatus according to Aspect 8, wherein the at least one processor is further configured to: detect a display idle state associated with the video data frame based on the video data frame and a predetermined number of previous video data frames not being associated with a frame geometry change; and apply a display idle GOP length, wherein the display idle GOP length is greater than the updated GOP length.

[0234] Aspect 10: An apparatus according to any one of Aspects 8 to 9, wherein the at least one processor is further configured to: encode the video data frame as a predicted frame (P frame) or a bidirectional frame (B frame) based on the fact that the video data frame is not associated with a frame geometry change.

[0235] Aspect 11: The apparatus of any one of aspects 1 to 10, wherein the video data frame comprises video display data displayed on a display of the computing device.

[0236] Aspect 12: The apparatus of any one of Aspects 1 to 11, wherein the frame of video data is a frame of captured video display data associated with wireless display sharing from the computing device to a second computing device.

[0237] Aspect 13: The apparatus of aspect 12, wherein the video data frame and the previous video data frame are consecutive frames included in a plurality of frames of captured video display data.

[0238] Aspect 14: The apparatus of aspect 13, wherein the at least one processor is further configured to: encode at least a portion of the plurality of frames of the captured video display data using the inter-frame prediction frame.

[0239] Aspect 15: An apparatus according to Aspect 14, wherein the at least one processor is further configured to: send the encoded video data to a second device associated with the same wireless display sharing session as the computing device; wherein the encoded video data includes the inter-frame prediction frame and at least one of a unidirectional prediction frame or a bidirectional prediction frame generated based on encoding at least the portion of the multiple frames of the captured video display data.

[0240] Aspect 16: A method for processing video data, the method comprising: obtaining a video data frame associated with a display of a computing device, wherein the video data frame comprises one or more layers; comparing layer information associated with the one or more layers included in the video data frame with layer information associated with one or more layers included in a previous video data frame; generating an inter-frame prediction frame using the video data frame based on determining a frame geometry change associated with the video data frame; and determining an updated group of pictures (GOP) length based on the layer information associated with the one or more layers included in the video data frame.

[0241] Aspect 17: The method according to aspect 16, wherein the updated GOP length is determined based on layer information associated with a main layer included in the video data frame.

[0242] Aspect 18: The method of aspect 17, wherein the primary layer included in the video data frame is rendered using a z-order greater than a corresponding z-order associated with one or more additional layers included in the video data frame.

[0243] Aspect 19: A method according to any one of Aspects 16 to 18, wherein the layer information includes, for each corresponding layer included in the one or more layers, at least one of a layer name associated with each corresponding layer, a layer format associated with each corresponding layer, and one or more coordinates associated with each corresponding layer.

[0244] Aspect 20: The method according to any one of aspects 16 to 19, wherein the layer information comprises at least one of a layer number or a frame layer number.

[0245] Aspect 21: A method according to any one of Aspects 16 to 20, wherein determining the frame geometry change associated with the video data frame is based on comparing the layer information associated with the one or more layers included in the video data frame and the layer information associated with the one or more layers included in the previous video data frame.

[0246] Aspect 22: A method according to Aspect 21, wherein determining the frame geometry change associated with the video data frame includes: determining that the layer information associated with the one or more layers included in the video data frame has changed by more than a threshold amount compared to the layer information associated with the one or more layers included in the previous video data frame.

[0247] Aspect 23: According to the method described in any one of Aspects 16 to 22, the method further includes: determining that a frame geometry change is not associated with the video data frame based on that the layer information associated with the one or more layers included in the video data frame changes less than a threshold amount compared to the layer information associated with the one or more layers included in the previous video data frame.

[0248] Aspect 24: According to the method described in Aspect 23, the method also includes: detecting a display idle state associated with the video data frame based on the video data frame and a predetermined number of previous video data frames not being associated with frame geometry changes; and applying a display idle GOP length, wherein the display idle GOP length is greater than the updated GOP length.

[0249] Aspect 25: The method according to any one of Aspects 23 to 24, further comprising: encoding the video data frame as a predicted frame (P frame) or a bidirectional frame (B frame) based on the video data frame not being associated with a frame geometry change.

[0250] Aspect 26: The method of any one of Aspects 16 to 25, wherein the frame of video data comprises video display data displayed on a display of the computing device.

[0251] Aspect 27: The method of any one of aspects 16 to 26, wherein the frame of video data is a frame of captured video display data associated with wireless display sharing from the computing device to a second computing device.

[0252] Aspect 28: The method of aspect 27, wherein the video data frame and the previous video data frame are consecutive frames included in a plurality of frames of captured video display data.

[0253] Aspect 29: The method according to aspect 28, the method further comprising: encoding at least a portion of the plurality of frames of the captured video display data using the inter-frame prediction frame.

[0254] Aspect 30: According to the method of Aspect 29, the method also includes: sending the encoded video data to a second device associated with the same wireless display sharing session as the computing device; wherein the encoded video data includes the inter-frame prediction frame and at least one of the unidirectional prediction frame or the bidirectional prediction frame generated based on encoding at least the portion of the multiple frames of the captured video display data.

[0255] Aspect 31: An apparatus, comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, wherein the at least one processor is configured to perform operations according to any one of aspects 1 to 30.

[0256] Aspect 32: An apparatus comprising means for performing the operations according to any one of aspects 1 to 30.

[0257] Aspect 33: A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform the operations according to any one of aspects 1 to 30.

Claims

1. A device for processing video data, the device comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: obtaining a frame of video data associated with a display of a computing device, wherein the frame of video data comprises one or more layers; comparing layer information associated with the one or more layers included in the frame of video data with layer information associated with one or more layers included in a previous frame of video data; generating an inter-prediction frame using the frame of video data based on determining a frame geometry change associated with the frame of video data; as well as An updated group of pictures (GOP) length is determined based on the layer information associated with the one or more layers included in the video data frame.

2. The apparatus of claim 1, wherein the at least one processor is configured to determine the updated GOP length based on layer information associated with a primary layer included in the video data frame.

3. The apparatus of claim 2, wherein the primary layer included in the video data frame is rendered using a z-order greater than a corresponding z-order associated with one or more additional layers included in the video data frame.

4. A device according to claim 1, wherein the layer information includes at least one of a layer name associated with each corresponding layer, a layer format associated with each corresponding layer, and one or more coordinates associated with each corresponding layer for each corresponding layer included in the one or more layers. The apparatus of claim 1 , wherein the layer information comprises at least one of a layer number or a frame layer number.

6. An apparatus according to claim 1, wherein the at least one processor is configured to determine the frame geometry change associated with the video data frame based on comparing the layer information associated with the one or more layers included in the video data frame and the layer information associated with the one or more layers included in the previous video data frame.

7. The apparatus of claim 6, wherein to determine the frame geometry change associated with the frame of video data, the at least one processor is configured to: It is determined that the layer information associated with the one or more layers included in the frame of video data has changed by greater than a threshold amount compared to the layer information associated with the one or more layers included in the previous frame of video data.

8. The apparatus of claim 1, wherein the at least one processor is further configured to: Based on the layer information associated with the one or more layers included in the video data frame changing less than a threshold amount compared to the layer information associated with the one or more layers included in the previous video data frame, it is determined that a frame geometry change is not associated with the video data frame.

9. The apparatus of claim 8, wherein the at least one processor is further configured to: detecting a display idle state associated with the frame of video data based on the frame of video data and a predetermined number of previous frames of video data not being associated with a frame geometry change; and A display idle GOP length is applied, wherein the display idle GOP length is greater than the updated GOP length.

10. The apparatus of claim 8, wherein the at least one processor is further configured to: The video data frame is encoded as a predicted frame (P frame) or a bidirectional frame (B frame) based on the video data frame not being associated with a frame geometry change.

11. The apparatus of claim 1, wherein the frame of video data comprises video display data displayed on a display of the computing device.

12. The apparatus of claim 1, wherein the frame of video data is a frame of captured video display data associated with wireless display sharing from the computing device to a second computing device.

13. The apparatus of claim 12, wherein the video data frame and the previous video data frame are consecutive frames included in a plurality of frames of captured video display data.

14. The apparatus of claim 13, wherein the at least one processor is further configured to: At least a portion of the plurality of frames of captured video display data are encoded using the inter-prediction frame.

15. The apparatus of claim 14, wherein the at least one processor is further configured to: transmitting the encoded video data to a second device associated with the same wireless display sharing session as the computing device; The encoded video data includes the inter-predicted frame and at least one of a uni-directionally predicted frame or a bi-directionally predicted frame generated based on encoding at least the portion of the plurality of frames of captured video display data.

16. A method for processing video data, the method comprising: obtaining a frame of video data associated with a display of a computing device, wherein the frame of video data comprises one or more layers; comparing layer information associated with the one or more layers included in the frame of video data with layer information associated with one or more layers included in a previous frame of video data; generating an inter-prediction frame using the frame of video data based on determining a frame geometry change associated with the frame of video data; as well as An updated group of pictures (GOP) length is determined based on the layer information associated with the one or more layers included in the video data frame.

17. The method of claim 16, wherein the updated GOP length is determined based on layer information associated with a primary layer included in the video data frame.

18. The method of claim 17, wherein the primary layer included in the video data frame is rendered using a z-order greater than a corresponding z-order associated with one or more additional layers included in the video data frame.

19. A method according to claim 16, wherein the layer information includes at least one of a layer name associated with each corresponding layer, a layer format associated with each corresponding layer, and one or more coordinates associated with each corresponding layer for each corresponding layer included in the one or more layers.

20. The method of claim 16, wherein the layer information comprises at least one of a layer number or a frame layer number.

21. The method of claim 16, wherein determining the frame geometry change associated with the video data frame is based on comparing the layer information associated with the one or more layers included in the video data frame and the layer information associated with the one or more layers included in the previous video data frame.

22. The method of claim 21 , wherein determining the frame geometry change associated with the frame of video data comprises: It is determined that the layer information associated with the one or more layers included in the frame of video data has changed by greater than a threshold amount compared to the layer information associated with the one or more layers included in the previous frame of video data.

23. The method according to claim 16, further comprising: Based on the layer information associated with the one or more layers included in the video data frame changing less than a threshold amount compared to the layer information associated with the one or more layers included in the previous video data frame, it is determined that a frame geometry change is not associated with the video data frame.

24. The method according to claim 23, further comprising: detecting a display idle state associated with the frame of video data based on the frame of video data and a predetermined number of previous frames of video data not being associated with a frame geometry change; as well as A display idle GOP length is applied, wherein the display idle GOP length is greater than the updated GOP length.

25. The method according to claim 23, further comprising: The video data frame is encoded as a predicted frame (P frame) or a bidirectional frame (B frame) based on the video data frame not being associated with a frame geometry change.

26. The method of claim 16, wherein the frame of video data comprises video display data displayed on a display of the computing device.

27. The method of claim 16, wherein the frame of video data is a frame of captured video display data associated with wireless display sharing from the computing device to a second computing device.

28. The method of claim 27, wherein the frame of video data and the previous frame of video data are consecutive frames included in a plurality of frames of captured video display data.

29. The method according to claim 28, further comprising: At least a portion of the plurality of frames of captured video display data are encoded using the inter-prediction frame.

30. The method according to claim 29, further comprising: transmitting the encoded video data to a second device associated with the same wireless display sharing session as the computing device; The encoded video data includes the inter-predicted frame and at least one of a uni-directionally predicted frame or a bi-directionally predicted frame generated based on encoding at least the portion of the plurality of frames of captured video display data.

Images