Orientation-aware coding for higher video quality

Abstract

The subject of the invention is orientation-aware coding for higher video quality. The encoder may be tuned to handle video in which a majority of motion is horizontal. Retuning an encoder to content with non-horizontal motion is a difficult process. Instead of a retuning process, a solution for encoding video content may include rotating a video prior to encoding the video and rotating it back after decoding its compressed bitstream. This solution may allow the encoder to take into account the appropriate orientation of each lens and apply different encoding parameters based on whether the lens needs to rotate. Similarly, the solution may allow a decoder to display the video correctly by rotating the decoded video data to the original orientation. The rotation information may be written into the encoded video bitstream, such as with a sequence header open bitstream unit.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This non-provisional application claims priority to and / or receives the benefit of a provisional application entitled “ORIENTATION-AWARE ENCODING FOR HIGHER VIDEOQUALITY,” serial number 63 / 591498, filed on October 19, 2023. The provisional application is hereby incorporated by reference in its entirety. Background Art

[0002] Video compression is a technique used to make video files smaller and easier to transmit over the Internet. There are different methods and algorithms for video compression with different performance and tradeoffs. Video compression involves encoding and decoding. Encoding is the process of converting (uncompressed) video data into a compressed format. Decoding is the process of recovering video data from a compressed format. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, the same reference numerals denote the same structural elements. In the figures of the accompanying drawings, the embodiments are illustrated by way of example and not by way of limitation.

[0004] FIG. 1 illustrates an encoding system and multiple decoding systems according to some embodiments of the present disclosure.

[0005] Figure 2 An exemplary encoder for encoding video frames and outputting an encoded bitstream according to some embodiments of the present disclosure is illustrated.

[0006] Figure 3 An exemplary decoder for decoding an encoded bitstream and outputting a decoded video according to some embodiments of the present disclosure is illustrated.

[0007] Figure 4 An example of a shot-based encoding pipeline according to some embodiments of the present disclosure is illustrated.

[0008] Figure 5 Exemplary rotation options for a lens according to some embodiments of the present disclosure are illustrated.

[0009] Figure 6 Examples of syntax elements according to some embodiments of the present disclosure are illustrated.

[0010] Figure 7 An example of a shot-based encoding pipeline with orientation detection and rotation options according to some embodiments of the present disclosure is illustrated.

[0011] Figure 8Selecting a rotation option for a lens according to some embodiments of the present disclosure is described.

[0012] Fig. 9 Selecting a rotation option for a lens according to some embodiments of the present disclosure is described.

[0013] Fig.10 Depicted is a flow chart of an exemplary method for orientation-aware decoding according to some embodiments of the present disclosure.

[0014] Fig.11 Depicted is a flow chart of an exemplary method for orientation-aware decoding according to some embodiments of the present disclosure.

[0015] Fig.12 Depicted is a flow chart of an exemplary method for orientation-aware decoding according to some embodiments of the present disclosure.

[0016] Fig.13 A block diagram of an exemplary computing device is depicted in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION Overview

[0017] Video coding is the process of compressing video data for storage, transmission and playback. Video coding may involve obtaining a large amount of raw video data and applying one or more compression techniques to reduce the amount of data required to represent the video while maintaining an acceptable level of visual quality. Some widely used video coding standards include AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding) and AV1 (Open Media Alliance Video 1). Video coding standards can be used in a wide range of applications, including, for example, video streaming, video conferencing, broadcasting, etc. AV1 is a video coding format (or codec) designed for video transmission over the Internet. The "AV1 Bitstream and Decoding Process Specification" version 1.1.1 (with errata) was last modified on January 18, 2019. HEVC, also known as "H.265ITU-T H.265 (V9) (09 / 2023)", was approved on September 13, 2023. VVC, also known as "ITU-TH.266 (V3) (09 / 2023)", was approved on September 29, 2023.

[0018] The encoding process may include dividing the video into a series of shots and applying one or more compression techniques to each shot individually. One of the challenges in video encoding may include achieving a high level of compression while maintaining visual quality. A high level of compression may be achieved through a combination of techniques such as motion estimation, prediction, quantization, and entropy coding.

[0019] However, some encoders may be tuned or optimized to handle natural video, where most of the motion is horizontal due to the natural way humans move. When encoding non-natural content, tuning to handle horizontal motion may result in reduced efficiency and quality. Non-natural content (such as screen sharing content, game content, virtual reality content, mixed reality content, and augmented reality content) may have different motion modes that are mostly not horizontal. When encoding natural content with different motion modes that are mostly not horizontal, tuning to handle horizontal motion may result in reduced efficiency and quality. In some cases, natural content captured on a mobile device may have different motion modes (e.g., due to the orientation of a handheld camera). In some cases, natural content captured by a drone or satellite may have different motion modes (e.g., because the drone moves in a specific direction overhead).

[0020] The encoder may be tuned or optimized to handle videos where most of the motion is horizontal. Constraints may be that motion vectors may be limited to having more horizontal dominant motion vectors. Another constraint may be that computing resources are dedicated to searching for both vertical dominant motion vectors and horizontal dominant motion vectors, and by design, more computing resources are allocated to identify horizontal dominant motion vectors. Another constraint may be that entropy coding or other coding operations may start with a raster scan sequence from the upper left block of a frame. Due to the fact that most of the motion patterns are not horizontal, context changes may be more frequent, resulting in lower entropy coding efficiency.

[0021] Re-tuning or re-optimizing an encoder to handle non-natural content or other natural content with different motion patterns can be an expensive and time-consuming process. In some cases, re-tuning or re-optimizing an encoder can include extensive retraining of non-natural data sets or natural data sets and making significant adjustments to encoding algorithms and tools. Therefore, it can be beneficial and / or advantageous to implement methods that can more efficiently handle non-natural content or natural content without requiring extensive re-tuning or re-optimization efforts.

[0022] One retuning approach that has been used to address the problem of encoding video content with non-horizontal motion involves adjusting the encoding parameters to better suit the content. For example, by increasing the search range for motion estimation in the vertical direction or adjusting the block size used for prediction. In another retuning approach to address this problem, alternative scanning orders for blocks are proposed, such as diagonal or spiral scanning orders. These scanning orders can help the encoder better capture non-horizontal motion in the video and reduce the amount of blocking artifacts and compression errors.

[0023] Instead of retuning or reoptimizing the encoding process, a solution for encoding unnatural and natural video content (e.g., video content with non-horizontal motion) may include rotating the video before encoding it, and rotating it back after decoding its compressed bitstream. This solution may allow the encoder to consider the appropriate orientation of each shot and apply different encoding parameters based on whether the shot needs to be rotated. Similarly, the solution may allow the decoder to display the video correctly by rotating the decoded video data to the original orientation.

[0024] In some cases, it may be useful to incorporate and / or write the rotation information into the encoded video bitstream itself to ensure that the decoder has the necessary information to correctly display the video. One approach may include adding the rotation information to a sequence header, which may signal: (1) whether the decoded bitstream needs to be rotated back, and (2) if applicable, the degree of rotation.

[0025] This solution can be useful in a range of video processing workflows, including shot-based encoding and video on demand (VOD) applications. By incorporating rotation information into the encoded video bitstream, the encoding process can help improve the overall quality of the output video, especially if the original video contains shots captured at different angles or has non-horizontal motion patterns.

[0026] The technique of rotating the video before encoding and rotating it back after decoding, and incorporating the rotation information into the video bitstream, is a useful technique for optimizing the video encoding and decoding process and improving the output video quality.

[0027] The technology can be used in video coding standards such as AV1 (Alliance for Open Media Video 1) and higher versions of AV1. Video coding standards that include this technology can improve video coding efficiency and quality by integrating rotation information directly into the encoded video bitstream. The technology can provide significant benefits for natural and non-natural content with non-horizontal motion, such as games and screen content.

[0028] Furthermore, this technique can be highly appreciated by content streaming platforms as it complements shot-based encoding methodologies. A shot may include portions or segments of a video that may be captured by the same camera. Shot-based encoding methodologies may apply different encoding parameters and / or encoding techniques independently to different shots to optimize the quality and size / bitrate of the encoded bitstream.

[0029] By rotating the source video (e.g., one or more shots of a video) before encoding and then returning the decoded bitstream to its original orientation after decoding, the encoder can capture more motion (or motion type) and improve its prediction capabilities, resulting in superior coding efficiency and video quality. Different rotation parameters can be used or selected independently for each shot of the video. This technology may be particularly useful for content streaming platforms because it can enhance their ability to provide high-quality video content to their users and add value to their users.

[0030] In addition, this technique has the advantage of being relatively elegant to implement. The technique may involve minimal modifications to video encoding and decoding algorithms. In addition, the technique may be modified to incorporate optimization into the rotation process. The technique may be modified to incorporate different techniques for determining rotation information. The technique may be modified to incorporate artificial intelligence or machine learning techniques to determine optimal orientation and / or rotation information.

[0031] The orientation-aware shot-based coding technique is different from applying different rotations to individual blocks within a frame or applying different rotations to individual frames within a shot. By operating at the shot level rather than at the block or frame level, the technique can take advantage of the fact that shots have uniform spatiotemporal properties. The technique can avoid the computational resources and signaling overhead required to implement frame-level and block-level rotations while achieving similar coding performance. In addition, operating at the block or frame level may significantly increase the complexity of the optimization problem. In addition, operating at the block or frame level may result in rotations that do not improve compression performance, and, worse, result in overall worse compression performance.

[0032] In some embodiments, orientation-aware coding techniques may include incorporating rotation information into the video bitstream to ensure that the decoder has the information to display the video correctly. The technique may be implemented by adding a rotation direction parameter to the bitstream (e.g., in a sequence header). The rotation direction parameter may signal whether the decoded frame needs to be rotated back, and optionally the degree of rotation. Video Compression

[0033] Figure 1 The encoding system 130 and one or more decoding systems 150 according to some embodiments of the present disclosure are illustrated. 1…D .

[0034] The encoding system 130 may be Fig.13The encoding system 130 may be implemented on a computing device 1300 of a computer system. The encoding system 130 may be implemented in the cloud or in a data center. The encoding system 130 may be implemented on a device for capturing video. The encoding system 130 may be implemented on an independent computing system. The encoding system 130 may perform an encoding process in video compression. The encoding system 130 may receive a video (e.g., uncompressed video, raw video, unprocessed video, etc.) including a sequence of video frames 104. The video frames 104 may include image frames or images that constitute the video. The video may have a frame rate or frames per second (FPS), which defines the number of frames per second of the video. The higher the FPS, the more realistic and smoother the video looks. Typically, for a natural, realistic viewing experience for human viewers, the FPS is greater than 24 frames per second. Examples of videos may include television episodes, movies, short films, short videos (e.g., less than 15 seconds long), video capture game experiences, computer screen content, video conferencing content, live event broadcast content, sports content, surveillance video, videos shot using a mobile computing device (e.g., a smart phone), and the like. In some cases, a video may include a mix or combination of different types of videos.

[0035] The encoding system 130 may include an encoder 102 that receives video frames 104 and encodes the video frames 104 into an encoded bitstream 180 . Figure 2 An exemplary implementation of encoder 102 is illustrated in .

[0036] The coded bitstream 180 may be compressed, which means that the coded bitstream 180 may be smaller in size than the video frame 104. The coded bitstream 180 may include a series of bits, for example, with 0s and 1s. The coded bitstream 180 may have header information, payload information, and footer information that may be encoded as bits in the bitstream. The header information may provide information about one or more of the following: the format of the coded bitstream 180, the encoding process implemented in the encoder 102, the parameters of the encoder 102, and the metadata of the coded bitstream 180. For example, the header information may include one or more of the following: resolution information, frame rate, aspect ratio, color space, etc. The payload information may include data representing the content of the video frame 104, such as sample frames, symbols, syntax elements, etc. For example, the payload information may include bits encoding one or more of the motion predictor, transform coefficients, prediction mode, and quantization level of the video frame 104. The footer information may indicate the end of the coded bitstream 180. The footer information may include other information, including one or more of the following: checksums, error correction codes, and signatures. The format of the encoded bitstream 180 may vary depending on the specifications of the encoding and decoding processes (ie, codecs).

[0037] The encoded bitstream 180 may be transmitted via the network 140 to one or more decoding systems 150 1…D The network 140 may be the Internet. The network 140 may include one or more of the following: a cellular data network, a wireless data network, a wired data network, a cable Internet network, a fiber optic network, a satellite Internet network, and the like.

[0038] The number D of decoding systems 150 1…D Decoding system 150 1…D At least one of the Fig.13 The system 150 is implemented on the computing device 1300. 1…D Examples of may include personal computers, mobile computing devices, gaming devices, augmented reality devices, mixed reality devices, virtual reality devices, televisions, etc. Decoding system 150 1…D Each of the decoding systems 150 can perform the decoding process in video compression. 1…D Each of the decoders may include decoders (eg, decoders 1 ... D162 1…D ) and one or more display devices (eg, display devices 1 ... D 164 1…D ). An exemplary implementation of a decoder (e.g., decoder 1 1621) is Figure 3 is described in.

[0039] For example, decoding system 1 1501 may include decoder 1 1621 and display device 1 1641. Decoder 1 1621 may implement a decoding process for video compression. Decoder 1 1621 may receive encoded bitstream 180 and generate decoded video 1681. Decoded video 1681 may include a series of video frames, which may be a version or a reconstructed version of video frame 104 encoded by encoding system 130. Display device 1 1641 may output decoded video 1681 for display to one or more human viewers or users of decoding system 1 1501.

[0040] For example, decoding system 2 1502 may include decoder 2 1622 and display device 2 1642. Decoder 2 1622 may implement a decoding process for video compression. Decoder 2 1622 may receive encoded bitstream 180 and generate decoded video 1682. Decoded video 1682 may include a series of video frames, which may be a version or a reconstructed version of video frame 104 encoded by encoding system 130. Display device 2 1642 may output decoded video 1682 for display to one or more human viewers or users of decoding system 2 1502.

[0041] For example, decoding system D150 D A decoder D 162 may be included Dand display device D 164 D . Decoder D 162 D It can realize the decoding process of video compression. Decoder D162 D The encoded bitstream 180 may be received and the decoded video 168 may be generated D . Decode video 168 D A series of video frames may be included, which may be versions or reconstructed versions of video frames 104 encoded by encoding system 130. Display device D 164 D Can output decoded video 168 D For display to the decoding system D 150 D One or more human viewers or users.

[0042] As discussed herein, encoder 102 may be modified to implement Figure 4 and 6 -10. Such as decoder 1 ... D 162 1…D The decoder can be modified to implement Figure 4 , 6 -9 and the operations described in 11-12. Video Encoder

[0043] Figure 2 An encoder 102 for encoding a video frame and outputting an encoded bitstream according to some embodiments of the present disclosure is illustrated. The encoder 102 may include one or more of the following: signal processing operations and data processing operations, including inter-frame and intra-frame prediction, transforms, quantization, in-loop filtering, and entropy coding. The encoder 102 may include a reconstruction loop involving inverse quantization and inverse transforms to ensure that the decoder will see the same reference blocks and frames. The encoder 102 may receive a video frame 104 and encode the video frame 104 into an encoded bitstream 180. The encoder 102 may include one or more of partitioning 206, transform and quantization 214, inverse transform and inverse quantization 218, in-loop filter 228, motion estimation 234, inter-frame prediction 236, intra-frame prediction 238, and entropy coding 216.

[0044] The partitioning 206 may partition the frame in the video frame 104 into blocks of pixels. Different codecs may allow different variable ranges of block sizes. In one codec, the frame may be partitioned by partitioning 206 into super blocks of size 128×128 or 64×64 pixels. The partitioning 206 may further use a multi-way partitioning tree structure to split each super block. In some cases, the partitioning of the super block may be further recursively split by partitioning 206 using a multi-way partitioning tree structure (e.g., down to blocks of size 4×4). In another codec, the frame may be partitioned by partitioning 206 into coding tree units of size 128×128 pixels. The partitioning 206 may use a quadtree partitioning structure to split the coding tree unit into four coding units. The partitioning 206 may further use a quadtree partitioning structure to recursively split the coding unit. The partitioning 206 may (further) use a multi-type tree structure (e.g., a quadtree, a binary tree, or a ternary tree structure) to subdivide the coding unit. The minimum coding unit may have a size of 4×4. In some codecs, a coding unit for luma pixels may be subdivided into smaller coding units (e.g., performing more tree structure subdivisions) than a coding unit for chroma pixels (e.g., stopping the tree structure subdivision earlier). The partitioning 206 may output raw samples 208, for example, as a pixel block.

[0045] Intra-frame prediction 238 can predict the sample of the block according to the reconstructed prediction sample of the previously encoded spatial neighboring block of the same frame. Intra-frame prediction 238 can receive the reconstructed prediction sample 226 (of the previously encoded spatial neighboring block of the same frame). The reconstructed prediction sample 226 can be generated by the adder 222 according to the reconstructed prediction residual 224 and the prediction sample 212. Intra-frame prediction 238 can determine the appropriate predictor for predicting the sample according to the reconstructed prediction sample of the previously encoded spatial neighboring block of the same frame. Intra-frame prediction 238 can generate the prediction sample 212 generated using the appropriate predictor. Intra-frame prediction 238 can output or identify the neighboring blocks and predictors used in generating the prediction sample 212. The identified neighboring blocks and predictors can be encoded in the coded bitstream 180 so that the decoder can reconstruct the block using the same neighboring blocks and predictors. In a codec, intra-frame prediction 238 can support multiple diverse predictors, such as 56 different predictors. Some predictors (e.g., directional predictors) can capture different spatial redundancies in directional textures. A directional predictor may be used in intra prediction 238 to predict pixel values ​​of a block by inferring pixel values ​​of neighboring blocks along a certain direction. Intra prediction 238 of different codecs may support different sets of predictors to exploit different spatial patterns within the same frame. Examples of predictors may include direct current (DC), planar, Paeth, smooth, smooth vertical, smooth horizontal, recursive-based filter mode, chroma-from-luma, intra-block copy, palette, multiple reference lines, intra-subdivision, matrix-based intra prediction (matrix coefficients may be defined by offline training using a neural network), wide-angle prediction, cross-component linear model, template matching, etc.

[0046] Motion estimation 234 and inter prediction 236 may predict samples of a block based on samples of a previously encoded frame (e.g., a reference frame in a decoded picture buffer 232). Motion estimation 234 may receive raw samples 208 from partition 206. Motion estimation 234 may receive samples (e.g., samples of a previously encoded frame or reference frame) from a decoded picture buffer 232. Motion estimation 234 may use multiple reference frames to determine one or more suitable motion predictors. Motion predictors may include motion vectors that capture the movement of blocks between frames in a video. Motion estimation 234 may output or identify one or more reference frames and one or more suitable motion predictors. Inter prediction 236 may apply one or more suitable motion predictors determined in motion estimation 234 and one or more reference frames to generate prediction samples 212. The identified reference frame(s) and motion predictor(s) may be encoded in the encoded bitstream 180 to enable a decoder to reconstruct the block using the same reference frame(s) and motion predictor(s). In one codec, motion estimation 234 may implement a single reference frame prediction mode, where a single reference frame with a corresponding motion predictor is used for inter prediction 236. Motion estimation 234 may implement a composite reference frame prediction mode, where two reference frames with two corresponding motion predictors are used for inter prediction 236. In one codec, motion estimation 234 may implement techniques for searching and identifying good reference frames (one or more) that can produce the most efficient motion predictor. The techniques in motion estimation 234 may include searching for good reference frame candidates spatially (within the same frame) and temporally (in previously encoded frames). The techniques in motion estimation 234 may include searching deep spatial neighborhoods to find a pool of spatial candidates. The techniques in motion estimation 234 may include utilizing a temporal motion field estimation mechanism to generate a pool of temporal candidates. The techniques in motion estimation 234 may use a motion field estimation process. Afterwards, the temporal and spatial candidates may be sorted, and a suitable motion predictor may be determined. In one codec, inter prediction 236 may support multiple diverse motion predictors. Examples of predictors may include geometric motion vectors (complex non-linear motion), warp motion compensation (affine transformations that capture non-translational object movement), overlapped block motion compensation, advanced composite prediction (compound wedge prediction, differential modulation mask prediction, composite prediction based on frame distance, and composite inter-intra prediction), dynamic spatial and temporal motion vector references, affine motion compensation (captures high-order motion such as rotation, scaling, and shearing), adaptive motion vector resolution mode, geometric partitioning mode, bidirectional optical flow, prediction refinement using optical flow, bidirectional prediction using weights, extended merge prediction, and the like.

[0047] Mode selection 230 may be informed by components such as motion estimation 234 to determine whether inter prediction 236 or intra prediction 238 may be more efficient for encoding a block.

[0048] The transform and quantization 214 may receive the prediction residual 210. The prediction residual 210 may be generated by a subtractor 220 that takes the original sample 208 and subtracts the prediction sample 212 to output the prediction residual 210. The prediction residual 210 may be referred to as a prediction error (e.g., an error between the original sample and the prediction sample 212) for intra prediction 238 and inter prediction 236. The prediction error has a smaller range of values ​​than the original sample and may be encoded with fewer bits in the coded bitstream 180. The transform and quantization 214 may include one or more of a transform and a quantization. The transform may include converting the prediction residual 210 from a spatial domain to a frequency domain. The transform may include applying one or more transform kernels. Examples of transform kernels may include discrete cosine transforms (DCTs) in horizontal and vertical forms, asymmetric discrete sine transforms (ADSTs), flipped ADSTs and identity transforms (IDTXs), multiple transform selections, low-frequency inseparable transforms, sub-block transforms, non-square transforms, DCT-VIII, discrete sine transforms VII (DST-VII), discrete wavelet transforms (DWTs), and the like. The transform may convert the prediction residual 210 into transform coefficients. Quantization may quantize the transform coefficients, for example, by reducing the precision of the transform coefficients. Quantization may include using a quantization matrix (e.g., a linear and nonlinear quantization matrix). The elements in the quantization matrix may be larger for higher frequency bands and smaller for lower frequency bands, meaning that higher frequency coefficients are more coarsely quantized and lower frequency coefficients are more finely quantized. Quantization may include dividing each transform coefficient by the corresponding element in the quantization matrix and rounding to the nearest integer. Effectively, the quantization matrix may implement different quantization parameters (QPs) for different frequency bands and chroma planes, and spatial prediction may be used. A suitable quantization matrix may be selected and signaled for each frame, and the suitable quantization matrix may be encoded in the encoded bitstream 180. The transform and quantization 214 may output quantized transform coefficients and syntax elements 278 , which indicate the encoding mode and parameters used in the encoding process implemented in the encoder 102 .

[0049] The inverse transform and inverse quantization 218 may apply the inverse operations performed in the transform and quantization 214 to produce a reconstructed prediction residual 224 as part of a reconstruction path to produce a decoded picture buffer 232 for the encoder 102. The inverse transform and inverse quantization 218 may receive quantized transform coefficients and syntax elements 278. The inverse transform and inverse quantization 218 may perform one or more inverse quantization operations, such as applying an inverse quantization matrix, to obtain unquantized / original transform coefficients. The inverse transform and inverse quantization 218 may perform one or more inverse transform operations, such as an inverse transform (e.g., inverse DCT, inverse DWT, etc.) to obtain the reconstructed prediction residual 224. A reconstruction path is provided in the encoder 102 to generate reference blocks and frames, which are stored in the decoded picture buffer 232. The reference blocks and frames may match the blocks and frames to be generated in the decoder. The reference blocks and frames are used as reference blocks and frames by motion estimation 234, inter-frame prediction 236, and intra-frame prediction 238.

[0050] The in-loop filter 228 may implement a filter to smooth artifacts introduced by the encoding process in the encoder 102 (e.g., the processing performed by the partitioning 206 and the transform and quantization 214). The in-loop filter 228 may receive the reconstructed prediction samples 226 from the adder 222 and output the frame to the decoded picture buffer 232. Examples of filters may include a constrained low-pass filter, a directional de-ringing filter, an edge-directional conditional replacement filter, a loop restoration filter, a Wiener filter, a self-guided restoration filter, a constrained directional enhancement filter, a luma map with chroma scaling, a sample adaptive offset filter, an adaptive loop filter, a cross-component adaptive loop filter, and the like.

[0051] Entropy coding 216 may receive quantized transform coefficients and syntax elements 278 (e.g., referred to herein as symbols) and perform entropy coding. Entropy coding 216 may generate and output a coded bitstream 180. Entropy coding 216 may utilize statistical redundancy and apply a lossless algorithm to encode symbols and generate a compressed bitstream (e.g., coded bitstream 180). Entropy coding 216 may implement a certain version of arithmetic coding. Different versions may have different advantages and disadvantages. In one codec, entropy coding 216 may implement (symbol to symbol) adaptive multi-symbol arithmetic coding. In another codec, entropy coding 216 may implement context-based adaptive binary arithmetic coder (CABAC). Binary arithmetic coding is different from multi-symbol arithmetic coding. Binary arithmetic coding encodes only one bit at a time, for example, either with a binary value of 0 or with a binary value of 1. Binary arithmetic coding may first convert each symbol into a binary representation (e.g., each symbol uses a fixed number of bits). Handling only binary values ​​0 or 1 may simplify calculations and reduce complexity. Binary arithmetic coding may assign a probability to each binary value (e.g., the likelihood of a bit having a binary value of 0 and the likelihood of a bit having a binary value of 1). Multi-symbol arithmetic coding performs encoding on an alphabet having at least two or three symbol values ​​and assigns a probability to each symbol value in the alphabet. Multi-symbol arithmetic coding may encode more bits at a time, which may result in a smaller number of operations for encoding the same amount of data. Multi-symbol arithmetic coding may require more computation and storage (because the probability estimates may be updated for each element in the alphabet). Maintaining and updating the probability (e.g., cumulative probability estimates) for each possible symbol value in multi-symbol arithmetic coding may be more complex (e.g., the complexity grows with the size of the alphabet). Multi-symbol arithmetic coding is not to be confused with binary arithmetic coding, because these two different entropy coding processes are implemented differently and may result in different encoded bit streams for the same set of quantized transform coefficients and syntax elements 278.

[0052] As discussed herein, encoder 102 may be modified to implement Figure 4 and 6 -10. The encoder 102 may include orientation detection. The encoder 102 may include rotating the uncompressed video frame. The encoder 102 may write the rotation information into the encoded bitstream. Video Decoder

[0053] Figure 3Decoder 1 1621 for decoding a coded bitstream and outputting a decoded video according to some embodiments of the present disclosure is described. Decoder 1 1621 may include one or more of the following: signal processing operations and data processing operations, including entropy decoding, inverse transform, inverse quantization, inter-frame and intra-frame prediction, in-loop filtering, etc. Decoder 1 1621 may have signal and data processing operations that reflect the operations performed in the encoder. Decoder 1 1621 may apply the signal and data processing operations signaled in the coded bitstream 180 to reconstruct the video. Decoder 1 1621 may receive the coded bitstream 180 and generate and output a decoded video 1681 having multiple video frames. The decoded video 1681 may be provided to one or more display devices for display to one or more human viewers. Decoder 1 1621 may include one or more of entropy decoding 302, inverse transform and inverse quantization 218, in-loop filter 228, inter-frame prediction 236, and intra-frame prediction 238. The previously described and in encoders (such as Figure 2 Some of the functionalities are used in the encoder 102).

[0054] The entropy decoding 302 may decode the coded bitstream 180 and output symbols encoded in the coded bitstream 180. The symbols may include quantized transform coefficients and syntax elements 278. The entropy decoding 302 may reconstruct the symbols from the coded bitstream 180. The inverse transform and inverse quantization 218 may receive the quantized transform coefficients and syntax elements 278 and perform operations performed in the encoder. The inverse transform and inverse quantization 218 may output a reconstructed prediction residual 224. The adder 222 may receive the reconstructed prediction residual 224 and the prediction sample 212 and generate a reconstructed prediction sample 226. Depending on the prediction mode signaled in the coded bitstream 180 (e.g., as a syntax element in the quantized transform coefficients and syntax elements 278), intra prediction 238 or inter prediction 236 may be applied to generate the prediction sample 212. The in-loop filter 228 may receive the reconstructed prediction sample and output a decoded video 1681.

[0055] As discussed herein, such as decoder 1 ... D 162 1…D The decoder can be modified to implement Figure 4 , 6 -9 and 11-12. The decoder can read the rotation information from the encoded bit stream. The decoder can include the rotation after decoding. Example use case: shot-based encoding

[0056] Figure 4An example of a shot-based encoding pipeline according to some embodiments of the present disclosure is illustrated. A video 404 comprising a sequence of uncompressed video frames may be provided as an input to an encoding side 402. The encoding side 402 may perform encoding operations and generate an encoded bitstream of the video 404. The encoding side 402 may transmit the encoded bitstream data in a transport format (e.g., in the form of packets) over a network (e.g., the Internet 486). A decoding side 408 may receive the packets and perform decoding operations and reconstruct the uncompressed video frames of the video 404.

[0057] Video 404 may have a length or duration of more than one minute. Video 404 may have a length or duration of 2 minutes, representing a short video clip posted on a social media platform. Video 404 may have a length or duration of 5 minutes, representing a music video on a video streaming platform. Video 404 may have a length or duration of 45 minutes, representing an episode of a television series. Video 404 may have a length or duration of 3 hours, representing a documentary or movie. Video 404 may have a length or duration of 5 hours, representing screen content from a webcast event.

[0058] like Figure 4 As depicted in , shot-based encoding may include shot detection 462, which may include a process of identifying boundaries between shots 480 in video 404. Each of the one or more shots 480 may be encoded separately, which may help improve the overall quality of the video while maintaining a reasonable size / bitrate. A shot (such as one or more shots 480) may be defined as a continuous sequence of video frames captured by a single camera without any interruption or change in camera angle. A shot may include a portion or segment of video 404 having uniform spatiotemporal characteristics across a particular shot or the duration of a particular shot's video frame. A shot may include different scenes of video 404 separated by scene changes. A shot may include a portion or segment of video 404 having a uniform motion pattern across a shot or the duration of a shot's video frame.

[0059] In some embodiments, shot detection 462 may implement one or more of the following: digital signal processing algorithms, computer vision algorithms, machine learning models, and deep learning models. Shot detection 462 may process video 404 and output one or more shots 480 of video 404. The video may include many shots with corresponding subsets or sequences of uncompressed video frames. Preferably, shot detection 462 may identify one or more points where the spatiotemporal characteristics of the video change significantly. Shot detection 462 may analyze the content and / or signals in the uncompressed video frames to identify one or more points in video 404. One or more points may be considered as camera interruptions or video edits. Camera interruptions may include cuts. Camera interruptions may include gradual transitions (e.g., wipes and dissolves). Shot detection 462 may detect one or more points in video 404 where there are significant changes in visual content, such as one or more changes in camera angle, lighting, and / or motion. Shot detection 462 may perform pixel-based comparisons between adjacent uncompressed frames to detect significant changes. Shot detection 462 may perform color-based comparisons between adjacent uncompressed frames to detect significant changes. Shot detection 462 can use a machine learning model (e.g., a deep learning model) for adjacent uncompressed frames to extract and compare feature embeddings to detect significant changes. Shot detection 462 can perform feature-based comparisons (e.g., edges, background, detected objects, etc.) between adjacent uncompressed frames to detect significant changes. Shot detection 462 can perform texture-based comparisons (e.g., edges, background, detected objects, etc.) between adjacent uncompressed frames to detect significant changes. Shot detection 462 can use one or more points to separate or segment video 404 into one or more shots 480.

[0060] Shot-based encoding may include applying different encoding techniques and / or parameters to different shots. In some cases, shot-based encoding may apply one or more encoding stages for a given shot, such as a first stage 410 of shot encoding and a second stage 416 of shot encoding. The encoding stages may be used as multiple passes encoding, where the first stage is used to collect some information about the best way to encode the shot, and the second stage is used to use this information to more efficiently and optimally encode the shot.

[0061] Once the video 404 is sliced ​​or divided into one or more shots 480, the shots in the one or more shots 480 can be encoded using one or more QPs and one or more resolutions in a first stage 410 of shot encoding. The first stage 410 of shot encoding can use, for example, a very fast preset, a fast encoder, or even a hardware encoder for faster processing. The first stage 410 of shot encoding can be a coarse encoder, a crude encoder, a light encoder, or a first pass encoder. The first stage 410 of shot encoding can be implemented using a first set of encoding parameters. Figure 1 The operations described in encoder 102.

[0062] QP may include parameters that control the amount of compression applied to video data during the encoding process. Higher QP values ​​may result in more compression and smaller file sizes, but also lower quality. On the other hand, lower QP values ​​may result in less compression, larger file sizes, but higher quality.

[0063] In order to select a suitable subset of QP(s) and resolution(s) for a shot in the one or more shots 480, a method involving finding a convex hull 412 may be used. This method may find a convex hull 412 that represents the minimum number of points (e.g., QPs and resolutions, or (Q, R) pairs) that enclose the possible combinations of QPs and resolutions for a given shot while minimizing the distortion between the original video and the encoded video. The convex hull 412 may be used to select a suitable subset of QP(s) and resolution(s) from the best (Q, R) pairs selected for each shot 414. The suitable subset of possible QPs and resolutions (e.g., one or more (Q, R) pairs) may be used to encode a particular shot in the one or more shots 480 of the video 404 at a next stage (e.g., in a second stage 416 of shot encoding). In addition to the convex hull method, other suitable optimization methods may be used to determine a suitable subset of QP(s) and resolution(s).

[0064] In the second stage 416 of shot encoding, the shots of the video 404 may be encoded (e.g., a second time) at the selected QP and resolution from the selected best (Q, R) pair(s) for each shot 414. The second stage 416 of shot encoding may use a more sophisticated encoder than the first stage 410 of shot encoding to achieve the highest quality possible. The second stage 416 of shot encoding may be a fine, thorough, heavyweight, or second pass encoder. The second stage 416 of shot encoding may be implemented using a second set of encoding parameters. Figure 2The second set of encoding parameters optionally has a suitable set of selected QP(s) and resolution(s) from the selected best (Q, R) pair(s) for each lens 414, as described in the operation of the encoder 102 of FIG.

[0065] The convex hull method may again be used to determine the convex hull 418. The convex hull 418 may be used to select one or more best / optimal / appropriate QP and resolution pairs for a particular shot in the one or more shots 480 of the video 404. The convex hull method may ensure that the encoded video maintains a high level of quality while minimizing file size.

[0066] One or more encoded bitstreams corresponding to one or more corresponding (Q, R) pairs may be generated in encoding side 402. In some cases, multiple (Q, R) pairs may be used to generate multiple encoded bitstreams for shots in one or more shots 480 having different resolutions and bitrates, which may form a bitrate ladder. The bitrate ladder may make specific shots in one or more shots 480 of video 404 available within a range of bandwidths / data rates and resolutions for display on various devices (e.g., device 1 432, device 2 436, and device 3 440) having different connection speeds.

[0067] The one or more encoded bitstreams generated by the encoding side 402 may be sent, for example, over the Internet 486 to be displayed to one or more users using one or more devices (e.g., device 1 432, device 2 436, device 3 440, etc.). The one or more users may refer to the intended viewers of the encoded video streams. Depending on the use case, a user may be anyone who needs to view video content, such as a viewer watching a streaming service on their laptop, tablet, or smartphone.

[0068] One or more encoded bitstreams may be decoded using corresponding decoders (e.g., decode (R1, Q1) 430, decode (R2, Q2) 434, decode (R3, Q3) 438, etc.). The decoder may select an encoded bitstream having a particular resolution and / or bitrate (e.g., (R1, Q1), (R2, Q2), (R3, Q3), etc.) with a bitrate ladder. R1, R2, and R3 may correspond to a first resolution, a second resolution, and a third resolution, respectively. Q1, Q2, and Q3 may correspond to a first bitrate, a second bitrate, and a third bitrate. A particular bitrate and / or resolution may be best suited for a display resolution and a (current) connection speed of the device. The decoder may implement Figure 3 The operations described in decoder 1 1621 in .

[0069] To optimize the viewing experience for each device, the bitstream quality (e.g., bitrate and / or resolution) may be statically and / or dynamically adjusted or selected based on the capabilities of the device. For example, a laptop computer with a larger display and more processing power may be able to display higher quality video at a higher resolution (higher bitrate). On the other hand, a smartphone with a smaller screen and lower processing power may only be able to display lower quality video at a lower resolution (lower bitrate).

[0070] In order to adjust the bitstream quality of different devices, one or more coded bitstreams can be transmitted or transferred in a transport format, for example, using an adaptive streaming protocol such as Hypertext Transfer Protocol (HTTP) Real-time Streaming (HLS) or Dynamic Adaptive Streaming (DASH) over HTTP over the Internet 486. These protocols can allow the video to be delivered in small pieces or fragments of different qualities, thereby allowing the device to select the appropriate quality level based on its capabilities and network conditions. For example, one or more coded bitstreams can have multiple versions, each with different bitrates and resolutions. When a user who is using a device requests to watch a video on his device, the client software of the device selects the appropriate version of one or more coded bitstreams based on the available network bandwidth and the capabilities of the device. This process can ensure that the video is displayed with the highest possible quality while avoiding buffering or other playback problems.

[0071] The encoding side 402 (such as the algorithms in the first stage 410 of shot encoding and the second stage 416 of shot encoding) can be optimized or tuned to encode one or more shots 480 that primarily include horizontal motion. To address this issue, the encoding side 402 and the decoding side 408 can be enhanced to be orientation-aware. Figure 5-12 As described in , the encoding side 402 and the decoding side 408 may include rotation of the lens before encoding and after decoding.

[0072] Figure 5An exemplary rotation option 504 of a lens according to some embodiments of the present disclosure is illustrated. The video to be encoded may include uncompressed video frames. The video may be segmented into one or more lenses. The original lens 502 (an example of one or more lenses) may include a subset of uncompressed video frames of the video. The subset of uncompressed video frames of the video may have one or more uniform spatiotemporal characteristics, such as camera angle, camera motion, and object motion. A different lens (not depicted) of the video may have one or more uniform spatiotemporal characteristics different from one or more spatiotemporal characteristics of the original lens 502. In this example, the original lens 502 includes a sequence of uncompressed video frames capturing a cat on a sofa from a first camera angle. A different lens (not depicted) of the video may include a different sequence of uncompressed video frames. An example of a different lens of the video may include a different sequence of uncompressed video frames capturing a robot vacuum cleaner on the floor from a second camera angle. Camera cutting may separate the original lens 502 and the different lenses of the video.

[0073] The rotation option 504 may not include rotation. The orientation of the original shot 502 is preserved or not changed. The (unrotated) uncompressed video frame is illustrated as unrotated shot 510.

[0074] Exemplary rotation options 504 may include a 90 degree clockwise rotation, or equivalently, a 270 degree counterclockwise rotation. The uncompressed video frame of the original lens 502 may be rotated 90 degrees clockwise, or equivalently 270 degrees counterclockwise. The rotated uncompressed video frame is illustrated as rotated lens 520.

[0075] Exemplary rotation options 504 may include a 180 degree clockwise rotation, or equivalently, a 180 degree counterclockwise rotation. The uncompressed video frame of the original lens 502 may be rotated 180 degrees clockwise, or equivalently, 180 degrees counterclockwise. The rotated uncompressed video frame is illustrated as rotated lens 530.

[0076] Exemplary rotation options 504 may include a 270 degree clockwise rotation, or equivalently, a 90 degree counterclockwise rotation. The uncompressed video frame of the original lens 502 may be rotated 180 degrees clockwise, or equivalently 90 degrees counterclockwise. The rotated uncompressed video frame is illustrated as rotated lens 540.

[0077] One or more rotation options in rotation options 504 may correspond to a set of one or more sequence rotation index values. The set of one or more sequence rotation index values ​​may depend on implementation and / or application. The set of one or more sequence rotation index values ​​may be determined or defined based on whether rotation options are allowed and which rotation options are allowed.

[0078] Taking the AV1 coded bitstream as an example, information applicable to a shot may be incorporated into a sequence header. A sequence header provides information about how to decode a particular sequence of video frames, which may follow the sequence header in the coded bitstream. A sequence of video frames may constitute at least a portion of a shot. Information applicable to a shot or a sequence of video frames constituting a shot (such as, for example, rotation information as indicated by a sequence rotation index value) may be appropriately included in a sequence header. A sequence header may include (header) signaling information to be applied to a sequence of video frames (such as a sequence of video frames corresponding to a shot such as the original shot 502). A sequence header may include (header) signaling information to be applied to a sequence of coded video frames. Other types of headers (e.g., frame headers) may be used to include (header) signaling information to be applied to other types of data units (e.g., frames).

[0079] When the coded bitstream is in transport format, the coded bitstream has a series of packets. The sequence header can be put into a sequence header packet. AV1 has a structure called "sequence header OBU" (OBU stands for open bitstream unit), which can be used as a sequence header packet to carry sequence header information in the coded bitstream. The sequence header information in the sequence header packet can be extracted and used by the decoder to correctly decode the shot or the sequence of video frames of the shot. In other words, the sequence header information is carried in the sequence header OBU, which has information to be applied to a specific shot or a sequence of video frames of a specific shot. Specifically, the sequence header can be transmitted as a packet in OBU format or a sequence header OBU in AV1, and is part of the coded bitstream. The packet in OBU format can include a header and a payload.

[0080] The header of an OBU packet may include one or more of the following: One or more bits to indicate the type of OBU packet, A bit used to mark whether the extension is used, A bit indicating whether the packet has a size field, One or more bits to indicate a time identifier, and • One or more bits used to indicate a space identifier.

[0081] The payload of the OBU packet may include one or more of the following: One or more bits to indicate the size of the packet, and • One or more bits used to encode data (eg, syntax elements, quantized transform coefficients, etc.).

[0082] A sequence header packet may include a header having one or more bits to indicate that the packet is a sequence header packet. The header may indicate that the packet is a sequence header packet (type).

[0083] The sequence header packet may include a payload having one or more bits for encoding one or more syntax elements. The one or more syntax elements may be extracted, used, and applied by a decoder to decode a sequence of video frames, such as a sequence of video frames that constitute at least a portion of a shot.

[0084] Figure 6 Examples of syntax elements according to some embodiments of the present disclosure are described. Syntax elements may be synonymous with parameters or variables. Examples of syntax elements may be included in the payload of a sequence header packet. One way to incorporate rotation information is to add a syntax element to the sequence header. This syntax element may be named "rotation_info" (or the like). The syntax element may indicate one or more rotation parameters for each shot in the video. For example, if a video has a shot that needs to be rotated 90 degrees clockwise, the syntax element for the shot in the sequence header may be as follows: Figure 6 , where "rotation_info" = value 0x1. The "rotation_info" syntax element may be a custom syntax element added to the sequence header to indicate rotation parameters for a video shot. A value of "0x1" (or a suitable value) in the "rotation_info" element may indicate that the shot is to be rotated 90 degrees clockwise (indicated by "1"). In some embodiments, four rotation options may be indicated by values ​​"0x0", "0x1", "0x2", and "0x3", signaling Figure 5 Exemplary rotation options 504 in .

[0085] Exemplary syntax elements for a sequence header may include one or more of a sequence profile syntax element 610 ("seq_profile"), a sequence level index syntax element 620 ("seq_level_idx"), a sequence tier syntax element 630 ("seq_tier"), and a sequence rotation syntax element 640 ("rotation_info").

[0086] In this example, one or more of the syntax elements (e.g., a sequence profile syntax element 610 ("seq_profile"), a sequence level index syntax element 620 ("seq_level_idx"), and a sequence layer syntax element 630 ("seq_tier") may be part of a payload of a sequence header packet. The sequence profile syntax element 610 ("seq_profile"), the sequence level index syntax element 620 ("seq_level_idx"), and the sequence layer syntax element 630 ("seq_tier") may be used to specify a profile, level, and tier, respectively, of a video shot.

[0087] For the sequence profile syntax element 610, the sequence profile index value may be set to indicate a sequence profile having one or more of the following: bit depth, monochrome support, chroma subsampling. For the sequence level index syntax element 620, the sequence level index value may be set to indicate a sequence level having one or more of the following: maximum picture size, maximum H size, maximum V size, maximum display rate, maximum decoding rate, maximum header rate, primary megabits per second, high megabits per second, primary compression ratio, high compression ratio, maximum tiles, and maximum tile columns. For the sequence layer syntax element 630, the sequence layer index value may be set to indicate a (sequence) layer within a sequence level (e.g., high, normal, low, etc.).

[0088] One or more syntax elements (e.g., sequence rotation syntax element 640 ("rotation_info")) may be used to signal rotation information, such as signaling which of the rotation options is applied in the encoder side. Possible rotation options are Figure 5 For example, a sequence rotation index value may be set for a sequence rotation syntax element 640 ("rotation_info") to indicate rotation information, e.g., how a sequence or shot of video frames is rotated before encoding, if performed. A decoder that extracts the sequence rotation index value may use the sequence rotation index value to rotate a sequence of decoded frames after decoding.

[0089] In some examples, the sequence rotation index value may be encoded as one bit and may include an index value selected from two possible index values. The two possible index values ​​may be 0 or 1. In some embodiments, the two possible index values ​​may include indicating a 90 degree clockwise rotation (by Figure 5 In some embodiments, the two possible index values ​​may include a first index value indicating a 90 degree counterclockwise rotation (illustrated by rotating lens 520) and a second index value indicating a 90 degree counterclockwise rotation (illustrated by rotating lens 540). Figure 5 The sequence rotation syntax element may indicate a first index value indicating a clockwise rotation of 180 degrees (illustrated by rotation lens 520), and a second index value indicating a clockwise rotation of 180 degrees (illustrated by rotation lens 530). The absence of the sequence rotation syntax element may indicate no rotation. The absence of the sequence rotation syntax element may indicate the same rotation as the previously signaled rotation option.

[0090] In some examples, the sequence rotation index value may be encoded as two bits and may include an index value selected from four possible index values. The four possible index values ​​may be 0 or 1. In some embodiments, the four possible index values ​​may include a first index value indicating no rotation, a second index value indicating a clockwise rotation of 90 degrees, a third index value indicating a clockwise rotation of 180 degrees, and a fourth index value indicating a clockwise rotation of 270 degrees.

[0091] By incorporating the rotation information into the sequence header, a decoder can, for example, display the video correctly without requiring any additional metadata or processing. A decoder can read the "rotation_info" syntax element from the sequence header, e.g. Figure 6 , and during the decoding process (eg, after the shots have been decoded and turned into a sequence of decoded / reconstructed video frames) the rotation information is used to rotate each shot as needed.

[0092] As an application example, this approach can be particularly beneficial in the context of shot-based encoding, where each shot can have its own unique rotation information and each shot can be encoded separately, allowing for more efficient and accurate encoding of each shot. Figure 4 As discussed, shot-based coding is a technique used in video coding that breaks down a video into shots, which are groups of frames that share similar spatiotemporal characteristics, such as camera angle, camera motion, and object motion. This allows for more efficient encoding of each shot, as the encoder can adjust its encoding parameters to better suit the specific characteristics of that shot. Figure 6 Incorporating rotation information into the video bitstream itself in the manner described in can help ensure that the decoder has the information to display the video correctly, especially when dealing with games and on-screen content.

[0093] Figure 7-9 Different encoder side 402 and decoder side 408 implementations that incorporate per-shot rotation are illustrated. The encoder side 402 may implement suitable techniques for determining rotation information. The decoder side 408 may implement extraction of the rotation information and application of the rotation based on the rotation information. The decoder side 408 may be agnostic to the particular technique used to determine the rotation information. Figure 7-8 As depicted in , a shot-based encoding pipeline (eg, encoding side 402 ) and a decoding pipeline (eg, decoding side 408 ) may incorporate rotation information into the encoded bitstream.

[0094] Figure 7An example of a shot-based encoding pipeline with orientation detection and rotation options according to some embodiments of the present disclosure is illustrated. The encoding side 402 may receive an uncompressed video 404 having one or more shots. The shot detection in the shot detection + orientation detection 702 may segment the video 404 into one or more shots, for example, including a first shot, a second shot, a third shot, etc. The shot detection may determine one or more shots of the video 404 based on the spatiotemporal characteristics of the uncompressed video frames of the video 404. The first shot may include a first subset of uncompressed video frames having one or more uniform spatiotemporal characteristics.

[0095] In addition to lens detection (with Figure 4In addition to the above (which may be performed in the manner discussed in the above), the encoding side 402 may include orientation detection in shot detection + orientation detection 702. A process such as orientation detection or motion direction estimation may be added on top of the shot detection process in the encoding side 402. The process may be added to the video encoding pipeline during or after the shot detection stage. This additional process may involve selecting the best orientation (or degree of rotation) for each shot based on the spatiotemporal characteristics of the shot (e.g., the motion within the shot). The orientation detection in shot detection + orientation detection 702 may detect a first orientation of the movement of a first shot in one or more shots. The orientation detection in shot detection + orientation detection 702 may detect the orientation of the movement of one or more shots on a per-shot basis. In some cases, the orientation detection may determine a feature or characteristic in the shot that may indicate the direction and / or orientation of the motion or movement in the shot. For a shot, the direction and / or orientation of the motion or movement may be uniform across a sequence of video frames forming the shot. The direction and / or orientation of the motion or movement may be measured relative to an angle with a horizontal axis. The degree of rotation may depend on the amount of rotation that will cause the orientation of the motion or movement to be (mostly) horizontal after the rotation of the shot. The degree of rotation may depend on the amount of rotation (selected from the available rotation options), which will result in the rotated lens having more horizontal motion. The direction and / or orientation of the motion or movement may be measured as a ratio of vertical movement to horizontal movement. The direction and / or orientation of the motion or movement may be measured based on whether there is an amount or ratio or magnitude of vertical movement greater than horizontal movement in the lens or across the video frame of the lens. Vertical movement may include upward and downward movement along the vertical axis. Horizontal movement may include left and right movement along the horizontal axis. The direction and / or orientation of the motion or movement may include a vector having vertical and horizontal components. Lens detection + orientation detection 702 may output one or more lenses and orientation information corresponding to the one or more lenses as one or more lenses + orientation information 704. In some cases, lens detection + orientation detection 702 may determine the degree of rotation based on the orientation information, and may output one or more lenses and the degree of rotation (or an indication of a rotation option) corresponding to the one or more lenses as one or more lenses + orientation information 704.

[0096] Once the orientation and / or appropriate degree of rotation has been determined in shot detection + orientation detection 702, the input video 404 may be rotated accordingly in rotation 706 before the first stage 708 of shot encoding and / or the second stage 710 of shot encoding, if appropriate. Each video frame in the video 404 is rotated by the same amount in rotation 706. The orientation and / or appropriate degree of rotation may be provided to rotation 706 from shot detection + orientation detection 702. The orientation and / or appropriate degree of rotation may be provided to the encoder from shot detection + orientation detection 702. In some cases, if the appropriate degree of rotation is different from zero, the rotation is (only) applied by rotation 706. If the degree of rotation is different from zero (e.g., 90 degrees, 180 degrees, and 270 degrees), the rotation is applied by rotation 706. Rotation 706 may rotate the first uncompressed video frame of the first shot based on the first orientation of the move to obtain a first rotated uncompressed video frame. The rotation applied by rotation 706 (e.g., rotation information) may be provided to the encoder. In some cases, rotation 706 may select a first sequence of rotation index values ​​based on the first orientation of the movement and / or the rotation applied in 706. Rotation 706 may provide the first sequence of rotation index values ​​to the encoder. In some cases, the encoder may determine the first sequence of rotation index values ​​based on the rotation information provided by lens detection + orientation detection 702 and / or rotation 706 and / or the first orientation of the movement.

[0097] In the example shown for shot-based encoding, the encoding pipeline in the encoding side 402 may include a first stage 708 of shot encoding and / or a second stage 710 of shot encoding. Not all shot-based encoding techniques have multiple stages. In some cases, shot-based encoding may be performed in one stage. The encoder pipeline may receive a first rotated uncompressed video frame of a first shot and transform the first rotated uncompressed video frame into an encoded bitstream with rotation information. The first stage 708 of shot encoding and / or the second stage 710 of shot encoding of the encoder side 402 may encode the first rotated uncompressed video frame to obtain first encoded video data. The first rotated uncompressed video frame may be encoded in the same manner as the rotation information. Figure 4 The first stage 708 of shot encoding may be implemented in a similar manner to the first stage 410 of shot encoding. The second stage 710 of shot encoding may be implemented in a similar manner to the second stage 416 of shot encoding.

[0098] In some embodiments, the first stage 708 of shot encoding and / or the second stage 710 of shot encoding at the encoder side 402 may (separately) apply the operations described for the encoder 102. The first stage 708 of shot encoding and / or the second stage 710 of shot encoding at the encoder side 402 may apply partitioning of the first rotated uncompressed video frame to obtain a first block (e.g., described as Figure 2The first stage 708 of shot coding and / or the second stage 710 of shot coding at the encoder side 402 may apply intra prediction or inter prediction to the first block to obtain a prediction residual (e.g., illustrated as Figure 2 The first stage 708 of shot coding and / or the second stage 710 of shot coding at the encoder side 402 may apply transform and quantization to the prediction residual to obtain quantized transform coefficients and syntax elements (e.g., illustrated as Figure 2 The first stage 708 of shot encoding and / or the second stage 710 of shot encoding at the encoder side 402 may apply entropy coding to quantized transform coefficients and syntax elements (e.g., illustrated as Figure 2 entropy coding 216).

[0099] In some embodiments, the rotated uncompressed video frame can be encoded using a two-stage process, such as using a first stage 708 of shot encoding and / or a second stage 710 of shot encoding at the encoder side 402. The first stage 708 of shot encoding can perform a first stage coarse encoding of the first rotated uncompressed video frame to obtain first coarsely encoded video data. A convex hull method (e.g., involving the convex hull 412 and the selected best (Q, R) pair(s) for each shot 414) can be used to determine one or more parameters that reduce or minimize distortion in the first coarsely encoded video data. The second stage 710 of shot encoding can use the one or more parameters to perform a second stage fine encoding of the first rotated uncompressed video frame to obtain first encoded video data.

[0100] During encoding, rotation information (e.g., sequence rotation index values) may be determined by the first stage 708 of shot encoding and / or the second stage 710 of shot encoding together with the encoded video data and included in the encoded bitstream. The rotation information may be included in a sequence header. For example, in the first stage 708 of shot encoding, the sequence rotation index value may be written into a sequence rotation syntax element rotation_info of a sequence header (e.g., a sequence header packet) of the encoded bitstream. In the second stage 710 of shot encoding, the sequence rotation index value may be written into a sequence rotation syntax element rotation_info of a sequence header (e.g., a sequence header packet) of the encoded bitstream. The sequence header may be used to signal information for decoding and rotating a particular shot.

[0101] The first stage 708 of shot encoding and / or the second stage 710 of shot encoding may write a first sequence of header packets having one or more first syntax elements in a first payload of a first sequence of header packets into the encoded bitstream. The one or more first syntax elements are to be applied by a decoder (a decoder in the decoding side 408) to decode the first shot. The one or more first syntax elements may include a first sequence of rotation index values ​​corresponding to a first orientation of the movement, such as Figure 6 The first stage 708 of shot encoding and / or the second stage 710 of shot encoding may write the first encoded video data as frame packets into an encoded bitstream.

[0102] The sequence rotation syntax element rotation_info may be used as a signal to the decoder in the decoding side 408 that specifies the degree of rotation to correctly display the video (e.g., to rotate the lens back to its original orientation to reverse the rotation operation performed in the encoding side 402). By including the rotation information as a syntax element in the encoded bitstream (e.g., in the sequence header), the decoder in the decoding side 408 may adjust the orientation of the lens according to the rotation information after the lens has been decoded. Figure 7 As described in , the decoding side 408 may include one or more of the following: decoding (R1, Q1) 720, decoding (R2, Q2) 730, and decoding (R3, Q3) 740. Decoding (R1, Q1) 720, decoding (R2, Q2) 730, and decoding (R3, Q3) 740 may be similar to Figure 4 The decoding side 408 may include one or more of the following: rotation 722, rotation 732, and rotation 742. The decoding (R1, Q1) 720, decoding (R2, Q2) 730, and decoding (R3, Q3) 740 may include providing the rotation information or its derivative to rotation 722, rotation 732, and rotation 742, respectively, for a particular lens, so that rotation 722, rotation 732, and rotation 742 may appropriately reverse the rotation operation performed in rotation 706. During the decoding process on the decoding side 408, the rotation information may be read and the rotation may be applied to the decoded / reconstructed video frame based on the rotation information after the decoding process. Device 1 432, device 2 436, and device 3 440 may display or output the rotated decoded / reconstructed video frame to a user.

[0103] In some embodiments, a decoder (e.g., decode (R1, Q1) 720, decode (R2, Q2) 730, and decode (R3, Q3) 740) may receive a coded bitstream, for example, via the Internet 486. The coded bitstream may be generated by the encoding side 402. The coded bitstream may include a first packet and a frame packet having a first sequence of coded video frames. The first sequence of coded video frames may correspond to a first shot of the video 404. The first packet includes a first header and a first payload. The first packet may be a sequence header packet. The first packet may be in an OBU format. The first payload may include one or more first syntax elements to be applied by the decoder to the first sequence of coded video frames. The one or more first syntax elements may include a first sequence rotation index value. The decoder may decode the first sequence of coded video frames according to the one or more first syntax elements to obtain a first sequence of decoded video frames. The decoder may implement functions such as Figure 3 The decoder or rotation component may determine the degree of rotation based on the first sequence rotation index value to reverse the rotation applied in rotation 706. The decoder or rotation operator may use the degree of rotation to rotate the first sequence of decoded video frames. The rotation operator may generate a rotated version of the sequence of decoded video frames, and the device (e.g., device 1 432, device 2 436, and device 3 440) may display the rotated version to the user.

[0104] Figure 8 Selecting a rotation option for a shot according to some embodiments of the present disclosure is described. In some cases, the encoding side 402 may process the shot in one or more shots 480 and not utilize a rotation operator or utilize a rotation operator to apply a different rotation option (e.g., such as Figure 5 The process may select the coded bitstream of the shot with the best rate and / or quality (e.g., resolution) to use and transmit as the coded bitstream.

[0105] The encoding side 402 may include one or more rotation operators capable of performing rotations according to one or more rotation options. The encoding side 402 may have a path without a rotation operator to effectively implement a "no rotation" rotation option. For example, rotation 802 may perform a rotation according to a first rotation option. Rotation 804 may perform a rotation according to a second rotation option. Another rotation operator (not shown) may be included to perform a rotation according to a third rotation option. The rotation operator may rotate the uncompressed video frame of the shot according to a particular rotation option.

[0106] The (parallel) encoder may include encoder 880, encoder 810, and encoder 820. Encoder 880 may receive an uncompressed video frame of the lens (without rotation applied) and encode the uncompressed video frame. Encoder 810 may receive a rotated uncompressed video frame of the lens from rotation 802 and encode it. Encoder 820 may receive a rotated uncompressed video frame of the lens from rotation 804 and encode it. The encoder may implement the following: Figure 2 The encoder 102 of FIG. 100 may perform the operations described in the encoder 102 of FIG. 101 to generate a corresponding encoded bit stream. In some cases, the encoder may write the corresponding rotation information into a sequence header packet in the corresponding encoded bit stream.

[0107] For example, encoder 880 may encode a first uncompressed video frame of a first shot of one or more shots to obtain first encoded video data or a first encoded bitstream having the first encoded video data. Rotation 802 may rotate the first uncompressed video frame of the first shot (e.g., rotate a first amount) to obtain a first rotated uncompressed video frame of the first shot. Encoder 810 may encode the first rotated uncompressed video frame of the first shot to obtain second encoded video data or a second encoded bitstream having the second encoded video data.

[0108] The selected coded shot with best rate + quality 830 can analyze the corresponding coded bitstreams produced by the (parallel) encoders. The selected coded shot with best rate + quality 830 can compare the quality and / or size of the first coded video data and the second coded video data. The selected coded shot with best rate + quality 830 can determine which rotation option may have resulted in the smallest size or the lowest bitrate for the same resolution. The selected coded shot with best rate + quality 830 can determine which rotation option may have resulted in the smallest amount of distortion or degradation in visual quality. The selected coded shot with best rate + quality 830 can select one of the coded bitstreams (e.g., the first coded bitstream and the second coded bitstream) based on the comparison and / or analysis. The selected one of the coded bitstreams (e.g., the selected one of the first coded bitstream and the second coded bitstream) can be transmitted, for example, via the Internet 486 to a decoder in the decoding side 408 for processing (the details of which are described in detail in the text of the decoder). Figure 7 ). The selected one of the coded bitstreams may have a first sequence header packet having one or more first syntax elements in a first payload of the first sequence header packet. The one or more first syntax elements are to be applied by a decoder to decode a first shot. The one or more first syntax elements may include a first sequence rotation index value indicating an amount of rotation applied prior to encoding. For example, if the first coded bitstream is selected for transmission, the first sequence rotation index value may correspond to no rotation or indicate that no rotation is applied prior to encoding. If the second coded bitstream is selected for transmission, the first sequence rotation index value may correspond to or indicate that a first amount of rotation is applied prior to encoding.

[0109] Fig. 9 The following describes some embodiments of the present disclosure in which a rotation option is selected for a lens. The implementation may be similar to Figure 8 The encoder side 402 includes one or more rotation operators, such as Figure 8 There is no rotation operator (e.g., direct path) or a suitable rotation operator can produce a rotation with different rotation options (e.g., such as Figure 5 The different versions of the lens may be processed using (in parallel) first stage encoders (e.g., first stage encoder 910, first stage encoder 920, and first stage encoder 930) to generate different encoded bitstreams. Fig. 9 The first stage encoder in can be similar to Figure 7 The first stage 708 of the shot encoding of the encoded bitstream may be implemented. The encoded bitstream may have corresponding rotation information written into a sequence header of the encoded bitstream. The process may select the best rotation option and optionally one or more parameters based on the different encoded bitstreams. The shot with the selected best rotation option and optionally one or more parameters may be encoded into a final / best encoded bitstream using a second stage encoder (e.g., second stage encoder 940). The second stage encoder 940 may be similar to Figure 7 The final / optimal encoded bitstream may have an appropriate sequence rotation index value corresponding to the optimal rotation option (eg, the amount of rotation applied before encoding) in the syntax element of the sequence header packet.

[0110] In some embodiments, determine optimal rotation and one or more parameters 990 may determine one or more parameters based on different encoded bitstreams produced by the (parallel) first stage encoders. A convex hull method may be applied to determine one or more parameters that may optimize the encoding result of the lens (in terms of Figure 4In a similar manner as discussed in ). In some embodiments, optimal rotation and one or more parameters 990 may determine which rotation option applied prior to encoding results in a better encoding result for the lens.

[0111] Based on the best rotation option, an uncompressed video frame with the best rotation option (shown as a lens 980 with the best rotation) may be provided as input to a second stage encoder 940. The second stage encoder 940 may utilize one or more parameters determined by determining the best rotation and one or more parameters 990 during the encoding process to obtain a final / best encoded bitstream. The second stage encoder 940 may write a sequence rotation index value corresponding to the best rotation option (e.g., the amount of rotation applied to obtain the lens 980 with the best rotation) into a syntax element of a sequence header packet during the encoding process. Method for orientation-aware decoding and method for orientation-aware encoding

[0112] Fig.10 Depicted is a flow chart of an exemplary method 1000 for orientation-aware decoding according to some embodiments of the present disclosure. The method 1000 may be implemented in a decoder side, such as the decoder side 408 of the figure.

[0113] In 1002, a coded bitstream may be received. The coded bitstream may include a first packet and a frame packet having a first sequence of coded video frames. The first packet may include a first header and a first payload. The first payload may include one or more first syntax elements to be applied to the first sequence of coded video frames. The one or more first syntax elements may include a first sequence rotation index value. The first sequence rotation index value may indicate rotation information. The first sequence rotation index value may indicate an amount of rotation before encoding. The first sequence rotation index value may indicate whether a rotation is to be applied after decoding and how much rotation is to be applied.

[0114] At 1004, a first sequence of encoded video frames may be decoded according to one or more first syntax elements to obtain a first sequence of decoded video frames.

[0115] The first sequence of decoded video frames may be rotated according to the first sequence rotation index values ​​at 1006. In some cases, if the first sequence rotation index values ​​indicate that no rotation was applied prior to encoding, no rotation is applied.

[0116] Method 1000 can be applied on a shot-by-shot basis. The coded bitstream may include a second packet and another frame packet having a second sequence of coded video frames. The second packet may include a second header and a second payload. The second payload may include one or more second syntax elements to be applied to the second sequence of coded video frames. The one or more second syntax elements may include a second sequence rotation index value that may or may not be different from the first sequence rotation index value. The second sequence of coded video frames may be decoded according to the one or more second syntax elements to obtain a second sequence of decoded video frames. The second sequence of decoded video frames may be rotated according to the second sequence rotation index value.

[0117] Fig.11 A flow chart of an exemplary method 1100 for orientation-aware encoding according to some embodiments of the present disclosure is depicted. The method 1100 may be implemented in an encoder side, such as the encoder side 402 of the figure.

[0118] At 1102, uncompressed video having one or more shots may be received.

[0119] At 1104 , a first orientation of movement of a first lens of the one or more lenses may be detected.

[0120] At 1106, the first uncompressed video frame of the first shot may be rotated based on the first orientation of the movement to obtain a first rotated uncompressed video frame. In some cases, if it is determined that the first orientation does not warrant rotation, the first compressed video frame may not be rotated.

[0121] At 1108, the first rotated uncompressed video frame may be encoded to obtain first encoded video data.

[0122] In 1110, a first sequence header packet may be written into a coded bitstream. The first sequence header packet may have one or more first syntax elements in a first payload of the first sequence header packet. The one or more first syntax elements are to be applied by a decoder to decode a first shot. The one or more first syntax elements may include a first sequence rotation index value corresponding to a first orientation of a move.

[0123] At 1112, first encoded video data may be written as frame groups into an encoded bitstream.

[0124] Method 1100 may be applied on a shot-by-shot basis. A second orientation of movement of a second shot in one or more shots may be detected. The second orientation of movement may be different from the first orientation of movement. A second uncompressed video frame of the second shot may be rotated based on the second orientation of movement to obtain a second rotated uncompressed video frame. The second rotated uncompressed video frame may be encoded to obtain second encoded video data. A second sequence header packet may be written into an encoded bitstream. The second sequence header packet may have one or more second syntax elements in a second payload of the second sequence header packet. The one or more second syntax elements are to be applied by a decoder to the second encoded video data. The one or more second syntax elements may include a second sequence rotation index value corresponding to the second orientation of movement. The second encoded video data may be written into the encoded bitstream as an additional frame packet.

[0125] Fig.12 A flow chart of an exemplary method 1200 for orientation-aware encoding according to some embodiments of the present disclosure is depicted. The method 1200 may be implemented in an encoder side, such as the encoder side 402 of the figure.

[0126] At 1202 , a first uncompressed video frame of a first shot of one or more shots of a video may be encoded to obtain a first encoded bitstream.

[0127] At 1204, a first uncompressed video frame of a first shot may be rotated by a first amount to obtain a first rotated uncompressed video frame of the first shot.

[0128] At 1206, the first rotated uncompressed video frame of the first shot may be encoded to obtain a second encoded bitstream.

[0129] At 1208, the sizes (and / or qualities) of the first encoded bitstream and the second encoded bitstream may be compared.

[0130] In 1210, a coded bitstream from among the first coded bitstream and the second coded bitstream may be selected based on the comparison.

[0131] In 1212, the coded bitstream selected in 1210 may be transmitted to, for example, a decoder or a device. The coded bitstream selected in 1210 may include a first sequence header packet having one or more first syntax elements in a first payload of the first sequence header packet. The one or more first syntax elements are to be applied by a decoder to decode a first shot. The one or more first syntax elements may include a first sequence rotation index value indicating an amount of rotation applied prior to encoding.

[0132] Method 1200 may be applied on a shot-by-shot basis. A second uncompressed video frame of a second shot of one or more shots of a video may be encoded to obtain a third encoded bitstream. The second uncompressed video frame of the second shot may be rotated a first amount to obtain a third rotated uncompressed video frame of the second shot. The third rotated uncompressed video frame of the first shot may be encoded to obtain a fourth encoded bitstream. The size (and / or quality) of the third encoded bitstream and the fourth encoded bitstream may be compared. Exemplary Computing Devices

[0133] Fig.13 is a block diagram of an apparatus or system (e.g., an exemplary computing device 1300) according to some embodiments of the present disclosure. One or more computing devices 1300 may be used to implement the functionality described in the figures and herein. Multiple components illustrated in the figures may be included in computing device 1300, but any one or more of these components may be omitted or duplicated as appropriate for the application. In some embodiments, some or all of the components included in computing device 1300 may be attached to one or more motherboards. In some embodiments, one or more of these components are manufactured onto a single system-on-chip (SoC) die. Additionally, in various embodiments, computing device 1300 may not include Figure 4 , and the computing device 1300 may include an interface circuit module for coupling to the one or more components. For example, the computing device 1300 may not include the display device 1306, and may include a display device interface circuit module (e.g., a connector and a driver circuit module), and the display device 1306 may be coupled to the display device interface circuit module. In another example set, the computing device 1300 may not include the audio input device 1318 or the audio output device 1308, and may include an audio input or output device interface circuit module (e.g., a connector and a supporting circuit module), and the audio input device 1318 or the audio output device 1308 may be coupled to the audio input or output device interface circuit module.

[0134] The computing device 1300 may include a processing device 1302 (e.g., one or more processing devices, one or more of the same type of processing devices, one or more of different types of processing devices). The processing device 1302 may include an electronic circuit module that processes electronic data from a data storage element (e.g., a register, a memory, a resistor, a capacitor, a qubit unit) to transform the electronic data into other electronic data that can be stored in the register and / or the memory. Examples of the processing device 1302 may include a central processing unit (CPU), a graphics processing unit (GPU), a quantum processor, a machine learning processor, an artificial intelligence processor, a neural network processor, an artificial intelligence accelerator, an application specific integrated circuit (ASIC), an analog signal processor, an analog computer, a microprocessor, a digital signal processor, a field programmable gate array (FPGA), a tensor processing unit (TPU), a data processing unit (DPU), etc.

[0135] The computing device 1300 may include a memory 1304, which may itself include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid-state memory, and / or a hard drive. The memory 1304 includes one or more non-transitory computer-readable storage media. In some embodiments, the memory 1304 may include a memory that shares a die with the processing device 1302. In some embodiments, the memory 1304 includes one or more non-transitory computer-readable media storing instructions that are executable to perform the operations described herein, such as Figure 1-4and 7-12, the operations described in method 1000, method 1100, and method 1200. The memory 1304 may include one or more non-transitory computer-readable media storing instructions, the instructions being executable to perform operations associated with orientation-aware encoding (e.g., operations performed in the encoder side 402) and / or orientation-aware decoding (e.g., operations performed in the decoder side 408). The memory 1304 may include one or more non-transitory computer-readable media storing instructions, the instructions being executable to perform operations associated with determining rotation information and / or rotation parameters of a lens. The memory 1304 may include one or more non-transitory computer-readable media storing instructions, the instructions being executable to perform operations associated with lens-based encoding with a rotation option. The memory 1304 may include one or more non-transitory computer-readable media storing one or more of the following: input frames to the encoder, intermediate data structures calculated by the encoder, bitstreams generated by the encoder, bitstreams received by the decoder, intermediate data structures calculated by the decoder, and reconstructed frames generated by the decoder. Instructions stored in one or more non-transitory computer-readable media may be executed by processing device 1302. In some embodiments, memory 1304 may store data, such as data structures, binary data, bits, metadata, files, binary large objects (blobs), etc., as described using the figures and herein. Exemplary data that may be stored in memory 1304 is depicted. Memory 1304 may store one or more data as depicted.

[0136] In some embodiments, the computing device 1300 may include a communication device 1312 (e.g., one or more communication devices). For example, the communication device 1312 may be configured to manage wired and / or wireless communications for transmitting data to and from the computing device 1300. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. that can transfer data through the use of modulated electromagnetic radiation passing through a non-solid medium. The term does not imply that the associated device does not contain any wires, although in some embodiments they may not. The communication device 1312 may implement any of a plurality of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including: Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE802.16-2005 amendments), Long Term Evolution (LTE) projects together with any amendments, updates and / or revisions (e.g., Advanced LTE projects, Ultra Mobile Broadband (UMB) projects (also known as "3GPP2"), etc.). Broadband wireless access (BWA) networks compatible with IEEE 802.16 are generally referred to as WiMAX networks (an acronym for Worldwide Interoperability for Microwave Access), which is a certification mark for products that have passed the conformance and interoperability tests of the IEEE802.16 standard. The communication device 1312 may operate according to a Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication device 1312 may operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication device 1312 may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution Data Optimized (EV-DO), and its derivatives, as well as any other wireless protocols designated as 3G, 4G, 5G, and higher. In other embodiments, the communication device 1312 may operate according to other wireless protocols. The computing device 1300 may include an antenna 1322 to facilitate wireless communications and / or receive other wireless communications (such as radio frequency transmissions). The computing device 1300 may include receiver circuitry and / or transmitter circuitry. In some embodiments, the communication device 1312 may manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication device 1312 may include multiple communication chips.For example, the first communication device 1312 may be dedicated to short-range wireless communication such as Wi-Fi or Bluetooth, and the second communication device 1312 may be dedicated to long-range wireless communication such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, etc. In some embodiments, the first communication device 1312 may be dedicated to wireless communication, and the second communication device 1312 may be dedicated to wired communication.

[0137] Computing device 1300 may include power supply / power circuit module 1314. Power supply / power circuit module 1314 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuit modules for coupling components of computing device 1300 to an energy source (e.g., DC power, AC power, etc.) separate from computing device 1000.

[0138] Computing device 1300 may include a display device 1306 (or a corresponding interface circuit module, as discussed above). Display device 1306 may include any visual indicator, such as, for example, a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.

[0139] Computing device 1300 may include audio output device 1308 (or corresponding interface circuit modules, as discussed above). Audio output device 1308 may include any device that generates audible indicators, such as, for example, speakers, headphones, or earphones.

[0140] The computing device 1300 may include an audio input device 1318 (or a corresponding interface circuit module, as discussed above). The audio input device 1318 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital instrument (e.g., an instrument with a Musical Instrument Digital Interface (MIDI) output).

[0141] Computing device 1300 may include GPS device 1316 (or a corresponding interface circuit module, as discussed above). GPS device 1316 may communicate with a satellite-based system and may receive the location of computing device 1300, as is known in the art.

[0142] Computing device 1300 may include sensor 1330 (or one or more sensors). Computing device 1300 may include corresponding interface circuit modules, as discussed above). Sensor 1330 may sense physical phenomena and convert the physical phenomena into electrical signals that can be processed by, for example, processing device 1302. Examples of sensor 1330 may include: capacitive sensors, inductive sensors, resistive sensors, electromagnetic field sensors, light sensors, cameras, imagers, microphones, pressure sensors, temperature sensors, vibration sensors, accelerometers, gyroscopes, strain sensors, moisture sensors, humidity sensors, distance sensors, range sensors, time of flight sensors, pH sensors, particle sensors, air quality sensors, chemical sensors, gas sensors, biosensors, ultrasonic sensors, scanners, etc.

[0143] The computing device 1300 may include another output device 1310 (or a corresponding interface circuit module, as discussed above). Examples of other output devices 1310 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, a tactile output device, a gas output device, a vibration output device, a lighting output device, a home automation controller, or an additional storage device.

[0144] The computing device 1300 may include another input device 1320 (or a corresponding interface circuit module, as discussed above). Examples of other input devices 1320 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a barcode reader, a quick response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0145] The computing device 1300 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, a personal digital assistant (PDA), an ultra-mobile personal computer, a remote control, a wearable device, a head-mounted device, glasses, footwear, electronic clothing, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, an Internet of Things device, or a wearable computer system. In some embodiments, the computing device 1300 may be any other electronic device that processes data. Featured Examples

[0146] Example 1 provides a method, comprising receiving a coded bitstream, wherein: the coded bitstream comprises a first packet and a frame group having a first sequence of coded video frames; the first packet comprises a first header and a first payload; the first payload comprises one or more first syntax elements to be applied to the first sequence of coded video frames; and the one or more first syntax elements comprise a first sequence rotation index value; decoding the first sequence of coded video frames according to the one or more first syntax elements to obtain a first sequence of decoded video frames; and rotating the first sequence of decoded video frames according to the first sequence rotation index value.

[0147] Example 2 provides the method of Example 1, wherein the first packet and the frame packet are in an open bitstream unit format.

[0148] Example 3 provides the method of Example 1 or 2, wherein: the first sequence of encoded video frames forms a first shot of the video; and the first shot includes a subset of video frames of the video having one or more uniform spatiotemporal characteristics.

[0149] Example 4 provides the method of any of Examples 1-3, wherein the first header indicates that the first packet is a sequence header packet.

[0150] Example 5 provides the method of any of Examples 1-4, wherein the first sequence of rotation index values ​​includes index values ​​selected from two possible index values.

[0151] Example 6 provides the method of Example 5, wherein the two possible index values ​​include a first index value indicating a clockwise rotation of 90 degrees; and a second index value indicating a counterclockwise rotation of 90 degrees.

[0152] Example 7 provides the method of any of Examples 1-4, wherein the first sequence of rotation index values ​​includes index values ​​selected from four possible index values.

[0153] Example 8 provides the method of Example 7, wherein the four possible index values ​​include a first index value indicating no rotation; a second index value indicating a clockwise rotation of 90 degrees; a third index value indicating a clockwise rotation of 180 degrees; and a fourth index value indicating a clockwise rotation of 270 degrees.

[0154] Example 9 provides the method of any one of Examples 1-8, wherein the one or more first syntax elements further include a sequence profile index value indicating a sequence profile having one or more of: bit depth, monochrome support, chroma subsampling.

[0155] Example 10 provides a method of any of Examples 1-9, wherein the one or more first syntax elements further include a sequence level index value indicating a sequence level having one or more of the following: maximum picture size, maximum H size, maximum V size, maximum display rate, maximum decoding rate, maximum header rate, primary megabits per second, high megabits per second, primary compression ratio, high compression ratio, maximum tile, and maximum tile column.

[0156] Example 11 provides the method of Example 10, wherein the one or more first syntax elements further include a sequence layer index value indicating a sequence layer within the sequence level.

[0157] Example 12 provides a method of any of Examples 1-11, wherein: the encoded bitstream includes a second group and another frame group having a second sequence of encoded video frames; the second group includes a second header and a second payload; the second payload includes one or more second syntax elements to be applied to the second sequence of encoded video frames; the one or more second syntax elements include a second sequence rotation index value different from the first sequence rotation index value; the second sequence of encoded video frames is decoded according to the one or more second syntax elements to obtain a second sequence of decoded video frames; and the second sequence of decoded video frames is rotated according to the second sequence rotation index value.

[0158] Example 13 provides a method, comprising: receiving an uncompressed video having one or more shots; detecting a first orientation of movement of a first shot in the one or more shots; rotating a first uncompressed video frame of the first shot based on the first orientation of movement to obtain a first rotated uncompressed video frame; encoding the first rotated uncompressed video frame to obtain first encoded video data; writing a first sequence of header packets having one or more first syntax elements in a first payload of the first sequence of header packets into a coded bitstream, wherein the one or more first syntax elements are to be applied by a decoder to decode the first shot, and the one or more first syntax elements include a first sequence rotation index value corresponding to the first orientation of movement; and writing the first encoded video data into the coded bitstream as a frame packet.

[0159] Example 14 provides the method of Example 13, further comprising determining the one or more shots of the uncompressed video based on spatiotemporal characteristics of uncompressed video frames of the uncompressed video.

[0160] Example 15 provides the method of Example 13 or 14, wherein the first shot comprises a first subset of uncompressed video frames of the uncompressed video having one or more uniform spatiotemporal characteristics.

[0161] Example 16 provides a method of any one of Examples 13-15, wherein encoding a first rotated uncompressed video frame includes: applying partitioning of the first rotated uncompressed video frame to obtain a first block; applying intra-frame prediction or inter-frame prediction to the first block to obtain a prediction residual; applying transformation and quantization to the prediction residual to obtain quantized transform coefficients and syntax elements; and applying entropy coding to the quantized transform coefficients and syntax elements.

[0162] Example 17 provides the method of any of Examples 13-16, further comprising selecting a first sequence of rotation index values ​​based on a first orientation of the movement.

[0163] Example 18 provides a method of any of Examples 13-17, wherein encoding a first rotated uncompressed video frame includes: performing a first stage coarse encoding of the first rotated uncompressed video frame to obtain first coarsely encoded video data; determining one or more parameters for reducing distortion in the first coarsely encoded video data; and performing a second stage fine encoding of the first rotated uncompressed video frame using the one or more parameters to obtain first encoded video data.

[0164] Example 19 provides a method of any one of Examples 13-18, further comprising detecting a second orientation of movement of a second lens in the one or more lenses, wherein the second orientation of the movement is different from the first orientation of the movement; rotating a second uncompressed video frame of the second lens based on the second orientation of the movement to obtain a second rotated uncompressed video frame; encoding the second rotated uncompressed video frame to obtain second encoded video data; writing a second sequence header packet having one or more second syntax elements in a second payload of the second sequence header packet into a coded bitstream, wherein the one or more second syntax elements are to be applied by a decoder to the second coded video data, and the one or more second syntax elements include a second sequence rotation index value corresponding to the second orientation of the movement; and writing the second encoded video data into the coded bitstream as an additional frame packet.

[0165] Example 20 provides a method, comprising encoding a first uncompressed video frame of a first shot of one or more shots of a video to obtain a first encoded bitstream; rotating the first uncompressed video frame of the first shot by a first amount to obtain a first rotated uncompressed video frame of the first shot; encoding the first rotated uncompressed video frame of the first shot to obtain a second encoded bitstream; comparing the sizes of the first encoded bitstream and the second encoded bitstream; selecting an encoded bitstream from the first encoded bitstream and the second encoded bitstream based on the comparison; and transmitting the selected encoded bitstream from the first encoded bitstream and the second encoded bitstream, wherein: the selected encoded bitstream has a first sequence header packet, the first sequence header packet has one or more first syntax elements in a first payload of the first sequence header packet; the one or more first syntax elements are to be applied by a decoder to decode the first shot; and the one or more first syntax elements include a first sequence rotation index value indicating the amount of rotation applied before encoding.

[0166] Example 21 provides the method of Example 20, further comprising rotating a first uncompressed video frame of a first shot by a second amount to obtain a second rotated uncompressed video frame of the first shot; encoding the first rotated uncompressed video frame to obtain a third encoded bitstream; and comparing the sizes of the first encoded bitstream, the second encoded bitstream, and the third encoded bitstream.

[0167] Example 22 provides the method of Example 20 or 21, wherein: the first sequence of rotation index values ​​corresponds to no rotation.

[0168] Example 23 provides the method of Example 20 or 21, wherein: the first sequence of rotation index values ​​corresponds to a rotation of a first amount.

[0169] Example 24 provides the method of any one of Examples 20-23, further comprising encoding a second uncompressed video frame of a second shot of the one or more shots of the video to obtain a third encoded bitstream; rotating the second uncompressed video frame of the second shot by a first amount to obtain a third rotated uncompressed video frame of the second shot; encoding the third rotated uncompressed video frame of the first shot to obtain a fourth encoded bitstream; and comparing the sizes of the third encoded bitstream and the fourth encoded bitstream.

[0170] Example 25 provides one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: receive a coded bitstream, wherein: the coded bitstream includes a first packet and a frame packet having a first sequence of coded video frames; the first packet includes a first header and a first payload; the first payload includes one or more first syntax elements to be applied to the first sequence of coded video frames; and the one or more first syntax elements include a first sequence rotation index value; decode the first sequence of coded video frames according to the one or more first syntax elements to obtain a first sequence of decoded video frames; and rotate the first sequence of decoded video frames according to the first sequence rotation index value.

[0171] Example 26 provides the one or more non-transitory computer-readable media of Example 25, wherein the first packet and the frame packet are in an open bitstream unit format.

[0172] Example 27 provides one or more non-transitory computer-readable media of Example 25 or 26, wherein: a first sequence of encoded video frames forms a first shot of the video; and the first shot includes a subset of video frames of the video having one or more uniform spatiotemporal characteristics.

[0173] Example 28 provides the one or more non-transitory computer-readable media of any of Examples 25-27, the first header indicating that the first packet is a sequence header packet.

[0174] Example 29 provides the one or more non-transitory computer-readable media of any of Examples 25-28, wherein the first sequence of rotation index values ​​includes index values ​​selected from two possible index values.

[0175] Example 30 provides the one or more non-transitory computer-readable media of Example 29, wherein the two possible index values ​​include a first index value indicating a clockwise rotation of 90 degrees; and a second index value indicating a counterclockwise rotation of 90 degrees.

[0176] Example 31 provides the one or more non-transitory computer-readable media of any of Examples 25-28, wherein the first sequence of rotation index values ​​includes index values ​​selected from four possible index values.

[0177] Example 32 provides one or more non-transitory computer-readable media of Example 31, wherein the four possible index values ​​include a first index value indicating no rotation; a second index value indicating a clockwise rotation of 90 degrees; a third index value indicating a clockwise rotation of 180 degrees; and a fourth index value indicating a clockwise rotation of 270 degrees.

[0178] Example 33 provides one or more non-transitory computer-readable media of any of Examples 25-32, wherein the one or more first syntax elements further include a sequence profile index value indicating a sequence profile having one or more of the following: bit depth, monochrome support, chroma subsampling.

[0179] Example 34 provides one or more non-transitory computer-readable media of any of Examples 25-33, wherein the one or more first syntax elements further include a sequence level index value indicating a sequence level having one or more of the following: maximum picture size, maximum H size, maximum V size, maximum display rate, maximum decoding rate, maximum header rate, primary megabits per second, high megabits per second, primary compression ratio, high compression ratio, maximum tile, and maximum tile column.

[0180] Example 35 provides the one or more non-transitory computer-readable media of Example 34, wherein the one or more first syntax elements further include a sequence layer index value indicating a sequence layer within a sequence level.

[0181] Example 36 provides one or more non-transitory computer-readable media of any of Examples 25-35, wherein: the encoded bitstream includes a second group and additional frame groups having a second sequence of encoded video frames; the second group includes a second header and a second payload; the second payload includes one or more second syntax elements to be applied to the second sequence of encoded video frames; the one or more second syntax elements include a second sequence rotation index value different from the first sequence rotation index value; the second sequence of encoded video frames is decoded according to the one or more second syntax elements to obtain a second sequence of decoded video frames; and the second sequence of decoded video frames is rotated according to the second sequence rotation index value.

[0182] Example 37 provides one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: receive an uncompressed video having one or more shots; detect a first orientation of movement of a first shot in the one or more shots; rotate a first uncompressed video frame of the first shot based on the first orientation of the movement to obtain a first rotated uncompressed video frame; encode the first rotated uncompressed video frame to obtain first encoded video data; write a first sequence of header packets having one or more first syntax elements in a first payload of the first sequence of header packets into a coded bitstream, wherein the one or more first syntax elements are to be applied by a decoder to decode the first shot, and the one or more first syntax elements include a first sequence rotation index value corresponding to the first orientation of the movement; and write the first encoded video data into the coded bitstream as a frame packet.

[0183] Example 38 provides one or more non-transitory computer-readable media of Example 37, wherein the instructions cause the one or more processors to further: determine the one or more shots of the uncompressed video based on spatiotemporal characteristics of uncompressed video frames of the uncompressed video.

[0184] Example 39 provides the one or more non-transitory computer-readable media of Example 37 or 38, wherein the first shot comprises a first subset of uncompressed video frames of the uncompressed video having one or more uniform spatiotemporal characteristics.

[0185] Example 40 provides one or more non-transitory computer-readable media of any of Examples 37-39, wherein encoding a first rotated uncompressed video frame comprises: applying partitioning of the first rotated uncompressed video frame to obtain a first block; applying intra-frame prediction or inter-frame prediction to the first block to obtain a prediction residual; applying a transform and quantization to the prediction residual to obtain quantized transform coefficients and syntax elements; and applying entropy coding to the quantized transform coefficients and syntax elements.

[0186] Example 41 provides the one or more non-transitory computer-readable media of any of Examples 37-40, wherein the instructions cause the one or more processors to further: select a first sequence of rotation index values ​​based on the first orientation of the movement.

[0187] Example 42 provides one or more non-transitory computer-readable media of any of Examples 37-41, wherein encoding a first rotated uncompressed video frame comprises: performing a first stage coarse encoding of the first rotated uncompressed video frame to obtain first coarsely encoded video data; determining one or more parameters that reduce distortion in the first coarsely encoded video data; and performing a second stage fine encoding of the first rotated uncompressed video frame using the one or more parameters to obtain first encoded video data.

[0188] Example 43 provides one or more non-transitory computer-readable media of any of Examples 37-42, wherein instructions cause the one or more processors to further: detect a second orientation of movement of a second lens in the one or more lenses, wherein the second orientation of movement is different from the first orientation of movement; rotate a second uncompressed video frame of the second lens based on the second orientation of movement to obtain a second rotated uncompressed video frame; encode the second rotated uncompressed video frame to obtain second encoded video data; write a second sequence header packet having one or more second syntax elements in a second payload of the second sequence header packet into a coded bitstream, wherein the one or more second syntax elements are to be applied by a decoder to the second coded video data, and the one or more second syntax elements include a second sequence rotation index value corresponding to the second orientation of movement; and write the second encoded video data into the coded bitstream as additional frame packets.

[0189] Example 44 provides one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: encode a first uncompressed video frame of a first shot of one or more shots of a video to obtain a first encoded bitstream; rotate the first uncompressed video frame of the first shot by a first amount to obtain a first rotated uncompressed video frame of the first shot; encode the first rotated uncompressed video frame of the first shot to obtain a second encoded bitstream; compare the sizes of the first encoded bitstream and the second encoded bitstream; select an encoded bitstream from the first encoded bitstream and the second encoded bitstream based on the comparison; and transmit the selected encoded bitstream from the first encoded bitstream and the second encoded bitstream, wherein: the selected encoded bitstream has a first sequence header packet, the first sequence header packet having one or more first syntax elements in a first payload of the first sequence header packet; the one or more first syntax elements are to be applied by a decoder to decode the first shot; and the one or more first syntax elements include a first sequence rotation index value indicating an amount of rotation applied prior to encoding.

[0190] Example 45 provides one or more non-transitory computer-readable media of Example 44, wherein the instructions cause the one or more processors to further: rotate a first uncompressed video frame of a first shot by a second amount to obtain a second rotated uncompressed video frame of the first shot; encode the first rotated uncompressed video frame to obtain a third encoded bitstream; and compare the sizes of the first encoded bitstream, the second encoded bitstream, and the third encoded bitstream.

[0191] Example 46 provides the one or more non-transitory computer-readable media of Example 44 or 45, wherein: the first sequence of rotation index values ​​corresponds to no rotation.

[0192] Example 47 provides the one or more non-transitory computer-readable media of Example 44 or 45, wherein: the first sequence of rotation index values ​​corresponds to a first amount of rotation.

[0193] Example 48 provides one or more non-transitory computer-readable media of any of Examples 44-47, wherein the instructions cause the one or more processors to further: encode a second uncompressed video frame of a second shot of the one or more shots of the video to obtain a third encoded bitstream; rotate the second uncompressed video frame of the second shot by a first amount to obtain a third rotated uncompressed video frame of the second shot; encode the third rotated uncompressed video frame of the first shot to obtain a fourth encoded bitstream; and compare the sizes of the third encoded bitstream and the fourth encoded bitstream.

[0194] Example A provides a device comprising means for performing or for performing any of the computer-implemented methods provided in Examples 1-24.

[0195] Example B provides a computer-implemented system comprising one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods provided in Examples 1-24.

[0196] Example C provides a device comprising: one or more processors for executing instructions, and one or more non-transitory computer-readable media according to one or more of the non-transitory computer-readable media of Examples 25-48.

[0197] Example D provides an encoder or one or more encoder-side components to generate an orientation-aware encoded bitstream using the operations described herein.

[0198] Example E provides a decoder or one or more decoder-side components to decode an orientation-aware encoded bitstream using the operations described herein. Variants and other notes

[0199] although Figure 1-4 7-12 and references Figure 1-4 The operations of the example methods described in 7-12 are illustrated as occurring once each and in a particular order, but it will be appreciated that the operations may be performed in any suitable order and repeated as desired. In addition, one or more operations may be performed in parallel. In addition, Figure 1-4The operations described in 7-12 may be combined or may include more or less details than described.

[0200] The above description of the illustrated implementation of the present disclosure, including the content described in the abstract, is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Although specific implementations and examples of the present disclosure are described herein for illustrative purposes, as will be appreciated by those skilled in the relevant art, various equivalent modifications are possible within the scope of the present disclosure. These modifications may be made to the present disclosure based on the above detailed description.

[0201] For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without the specific details and / or may be practiced with only some of the described aspects. In other instances, well-known features are omitted or simplified in order not to obscure the illustrative implementations.

[0202] In addition, reference is made to the accompanying drawings forming a part of this article, and embodiments that can be put into practice are shown therein by way of illustration. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be considered as a restrictive meaning.

[0203] The various operations may be described in sequence as multiple discrete actions or operations in a manner that is most helpful for understanding the disclosed subject matter. However, the order of description should not be interpreted as implying that these operations must rely on the order. In particular, these operations may not be performed in the order presented. The described operations may be performed in an order different from the described embodiments. In additional embodiments, various additional operations may be performed or the described operations may be omitted.

[0204] For purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B or C" or the phrase "A, B and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). When used with reference to a measurement range, the term "between" includes the endpoints of the measurement range.

[0205] The description uses the phrase "in one (an) embodiment" or "in an embodiment", which can each refer to one or more embodiments in the same or different embodiments. The terms "including", "comprising", "having", etc. used with respect to the embodiments of the present disclosure are synonymous. The present disclosure may use perspective-based descriptions such as "above", "below", "top", "bottom", and "side" to explain the various features of the drawings, but these terms are merely for ease of discussion and do not imply a desired or required orientation. The drawings are not necessarily drawn to scale. Unless otherwise specified, the use of ordinal adjectives "first", "second", and "third", etc. to describe common objects only indicates that different instances of the same object are being referenced, and is not intended to imply that the objects described so far must be in a given sequence either in time, space, order, or in any other way.

[0206] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0207] The terms "substantially," "close," "approximately," "near," and "about" generally refer to within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating the orientation of various elements, such as "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements, generally refer to within + / - 5-20% of a target value as described herein or as known in the art.

[0208] Furthermore, the terms "comprise," "comprising," "include," "including," "have," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, or apparatus. Furthermore, the term "or" refers to an inclusive "or" rather than an exclusive "or."

[0209] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for all of the desirable attributes disclosed herein.The details of one or more implementations of the subject matter described in this specification are set forth in the specification and the drawings.

Claims

1. A method comprising: Receives a coded bit stream where: The encoded bitstream includes a first packet and a frame packet having a first sequence of encoded video frames; The first packet includes a first header and a first payload; The first payload comprises one or more first syntax elements to be applied to the first sequence of encoded video frames; and The one or more first syntax elements include a first sequence rotation index value; Decoding the first sequence of encoded video frames according to the one or more first syntax elements to obtain a first sequence of decoded video frames; and The first sequence of decoded video frames is rotated according to the first sequence rotation index value.

2. The method according to claim 1, wherein: The first packet and the frame packet adopt an open bitstream unit format.

3. The method according to claim 1, wherein: The first sequence of encoded video frames forms a first shot of a video; and The first shot includes a subset of video frames of the video having one or more uniform spatiotemporal characteristics.

4. The method according to any one of claims 1 to 3, wherein: The first header indicates that the first packet is a sequence header packet.

5. The method according to any one of claims 1 to 3, wherein: The first sequence of rotation index values ​​includes index values ​​selected from two possible index values.

6. The method according to claim 5, wherein: The two possible index values ​​include: a first index value indicating a clockwise rotation of 90 degrees; and A second index value indicating a counter-clockwise rotation of 90 degrees.

7. The method according to any one of claims 1 to 3, wherein: The first sequence of rotation index values ​​includes index values ​​selected from four possible index values.

8. The method according to claim 7, wherein: The four possible index values ​​include: a first index value indicating no rotation; A second index value indicating a clockwise rotation of 90 degrees; a third index value indicating a clockwise rotation of 180 degrees; and A fourth index value indicating a clockwise rotation of 270 degrees.

9. The method according to any one of claims 1 to 3, wherein: The one or more first syntax elements further include: A sequence profile index value indicating a sequence profile having one or more of: bit depth, monochrome support, chroma subsampling.

10. The method according to any one of claims 1 to 3, wherein: The one or more first syntax elements further include: A sequence level index value indicating a sequence level having one or more of: maximum picture size, maximum H size, maximum V size, maximum display rate, maximum decoding rate, maximum header rate, primary megabits per second, high megabits per second, primary compression ratio, high compression ratio, maximum tile, and maximum tile column.

11. The method according to claim 10, wherein: The one or more first syntax elements further include: A sequence layer index value indicating a sequence layer within the sequence level.

12. The method according to any one of claims 1 to 3, wherein: The encoded bitstream comprises a second packet and further frame packets having a second sequence of encoded video frames; The second packet includes a second header and a second payload; The second payload comprises one or more second syntax elements to be applied to a second sequence of the encoded video frames; The one or more second syntax elements include a second sequence rotation index value different from the first sequence rotation index value; decoding the second sequence of encoded video frames according to the one or more second syntax elements to obtain a second sequence of decoded video frames; and The second sequence of decoded video frames is rotated according to the second sequence rotation index value.

13. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: Receives a coded bit stream where: The encoded bitstream includes a first packet and a frame packet having a first sequence of encoded video frames; The first packet includes a first header and a first payload; The first payload comprises one or more first syntax elements to be applied to the first sequence of encoded video frames; as well as The one or more first syntax elements include a first sequence rotation index value; Decoding the first sequence of encoded video frames according to the one or more first syntax elements to obtain a first sequence of decoded video frames; and The first sequence of decoded video frames is rotated according to the first sequence rotation index value.

14. The one or more non-transitory computer-readable media of claim 13, wherein the first packet and the frame packet are in an open bitstream unit format.

15. The one or more non-transitory computer-readable media of claim 13, wherein: The first sequence of encoded video frames forms a first shot of a video; and The first shot includes a subset of video frames of the video having one or more uniform spatiotemporal characteristics.

16. The one or more non-transitory computer-readable media of any of claims 13-15, the first header indicating that the first packet is a sequence header packet.

17. One or more non-transitory computer-readable media according to any one of claims 13-15, wherein: The first sequence of rotation index values ​​includes index values ​​selected from two possible index values.

18. The one or more non-transitory computer-readable media of claim 17, wherein: The two possible index values ​​include: a first index value indicating a clockwise rotation of 90 degrees; and A second index value indicating a counter-clockwise rotation of 90 degrees.

19. One or more non-transitory computer-readable media according to any one of claims 13-15, wherein: The first sequence of rotation index values ​​includes index values ​​selected from four possible index values.

20. The one or more non-transitory computer-readable media of claim 19, wherein: The four possible index values ​​include: a first index value indicating no rotation; A second index value indicating a clockwise rotation of 90 degrees; a third index value indicating a clockwise rotation of 180 degrees; and A fourth index value indicating a clockwise rotation of 270 degrees.

21. One or more non-transitory computer-readable media according to any one of claims 13-15, wherein: The one or more first syntax elements further include: A sequence profile index value indicating a sequence profile having one or more of: bit depth, monochrome support, chroma subsampling.

22. One or more non-transitory computer-readable media according to any one of claims 13-15, wherein: The one or more first syntax elements further include: A sequence level index value indicating a sequence level having one or more of: maximum picture size, maximum H size, maximum V size, maximum display rate, maximum decoding rate, maximum header rate, primary megabits per second, high megabits per second, primary compression ratio, high compression ratio, maximum tile, and maximum tile column.

23. The one or more non-transitory computer-readable media of claim 22, wherein: The one or more first syntax elements further include: A sequence layer index value indicating a sequence layer within the sequence level.

24. One or more non-transitory computer-readable media according to any one of claims 13-15, wherein: The encoded bitstream comprises a second packet and further frame packets having a second sequence of encoded video frames; The second packet includes a second header and a second payload; The second payload comprises one or more second syntax elements to be applied to a second sequence of the encoded video frames; The one or more second syntax elements include a second sequence rotation index value different from the first sequence rotation index value; decoding the second sequence of encoded video frames according to the one or more second syntax elements to obtain a second sequence of decoded video frames; and The second sequence of decoded video frames is rotated according to the second sequence rotation index value.

25. A computer-implemented system comprising: one or more processors; as well as One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods 1-3.

Images