How to annotate VVC subpictures in DASH

Abstract

The method and apparatus include computer code configured to cause one or more processors to perform the steps of obtaining a dynamic adaptive streaming over HTTP (DASH) video bitstream including one or more subpictures; determining whether the one or more subpictures include a Versatile Video Coding (VVC) compliant subpicture; annotating the one or more subpictures based on one or more flags in response to the DASH video bitstream including a VVC compliant subpicture; and manipulating the DASH video stream based on the annotated one or more subpictures.

Description

[Technical field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. provisional application US63 / 298,924, filed January 12, 2022, and U.S. patent application US18 / 073,984, filed December 2, 2022, the entire contents of which are incorporated by reference into this application.

[0002] [Technical field] This disclosure is directed to signaling for dynamic adaptive streaming over HTTP (DASH). [Background technology]

[0003] MPEG DASH provides a standard for streaming multimedia content over IP networks. The ISO / IEC 23009-1 DASH standard enables streaming of multi-rate content. A DASH manifest (MPD) can describe various media content. Although the DASH standard provides a way to describe various content and their relationships, it does not provide an interoperable solution for annotating VVC subpictures used for picture-in-picture applications. Summary of the Invention

[0004] To address one or more different technical problems, the present disclosure provides a technical solution for reducing network overhead and server computation overhead, including a method and an apparatus including a memory configured to store computer program code and one or more processors configured to access the computer program code and operate as instructed by the computer program code. The computer program code includes an acquisition code configured to cause at least one processor to acquire a dynamic adaptive streaming over HTTP (DASH) video bitstream including one or more sub-pictures, a determination code configured to cause the at least one processor to determine whether the one or more sub-pictures include Versatile Video Coding (VVC) compliant sub-pictures, annotating code configured to cause the at least one processor to annotate the one or more sub-pictures based on one or more flags in response to the DASH video bitstream including a VVC compliant sub-picture, and an operation code configured to cause the at least one processor to operate the DASH video stream based on the annotated one or more sub-pictures.

[0005] According to an exemplary embodiment, the video data is dynamic adaptive streaming over HTTP (DASH) and the client is a DASH client.

[0006] According to an example embodiment, determining whether to annotate one or more of the first sub-picture and the second sub-picture includes determining whether to annotate one or more of the first sub-picture and the second sub-picture in a DASH media picture description (MPD).

[0007] According to an example embodiment, the other of the first stream and the second stream comprises a picture-in-picture (PIP) stream.

[0008] According to an exemplary embodiment, controlling the client to replace at least a portion of one of the first stream and the second stream includes replacing an area represented by multiple sub-pictures of the first sub-picture and the second sub-picture with a PIP stream.

[0009] According to an exemplary embodiment, one of the first stream and the second stream represents a primary video, and the other of the first stream and the second stream represents a stream separately coded as an auxiliary video to the primary video.

[0010] According to an exemplary embodiment, controlling the client to replace at least a portion of one of the first stream and the second stream includes determining whether a user has requested to view the auxiliary video.

[0011] According to an exemplary embodiment, the primary and auxiliary videos each include versatile video coding (VVC) sub-pictures.

[0012] According to an example embodiment, controlling the client to replace at least a portion of one of the first stream and the second stream includes merging versatile video coding (VVC) sub-picture properties such that a single merged stream from both the first stream and the second stream is provided to the decoder. [Brief description of the drawings]

[0013] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1] FIG. 2 is a simplified schematic diagram according to an embodiment. [Diagram 2] FIG. 2 is a simplified schematic diagram according to an embodiment. [Diagram 3] FIG. 2 is a simplified block diagram of a decoder according to an embodiment. [Figure 4] FIG. 2 is a simplified block diagram of an encoder according to an embodiment. [Diagram 5] FIG. 2 is a simplified block diagram according to an embodiment. [Figure 6] FIG. 2 is a simplified diagram according to an embodiment. [Figure 7] FIG. 2 is a simplified diagram according to an embodiment. [Figure 8] FIG. 2 is a simplified flow diagram according to an embodiment. [Figure 9] FIG. 2 is a simplified diagram according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The proposed features discussed below may be used separately or combined in any order. Furthermore, the embodiments may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0015] 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 includes at least two terminals 102 and 103 interconnected via a network 105. For one-way transmission of data, a first terminal 103 may encode video data at a local location for transmission to the other terminal 102 via the network 105. A second terminal 102 may receive the encoded video data of the other terminal from the network 105, decode the encoded data, and display the restored video data. One-way data transmission may be common in media provisioning applications, etc.

[0016] 1 shows a second pair of terminals 101 and 104 provided to support bidirectional transmission of encoded video that may occur, for example, during a video conference. For the bidirectional transmission of data, each terminal 101 and 104 may encode video data captured at a local location for transmission to the other terminal over network 105. Each terminal 101 and 104 may also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the reconstructed video data on a local display device.

[0017] In FIG. 1, terminals 101, 102, 103, and 104 may be depicted as servers, personal computers, and smartphones, although the principles of the present disclosure are not limited thereto. The embodiments of the present disclosure may be applied to laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. Network 105 represents any number of networks that convey encoded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 105 is not important to the operation of the present disclosure, unless otherwise described herein below.

[0018] 2 shows the arrangement of video encoders and decoders in a streaming environment as an example of an application of the disclosed subject matter, which is equally applicable to other video-enabled applications including, for example, videoconferencing, digital TV, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.), etc.

[0019] The streaming system can include a capture subsystem 203, which can include, for example, a digital camera generating a stream of uncompressed video pictures 213. The sample stream 213 can be emphasized as a high amount of data when compared to an encoded video bitstream and can be processed by an encoder 202 coupled to the camera 201. The encoder 202 can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data 204 can be emphasized as a lower amount of data when compared to the sample stream and can be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 can access the streaming server 205 to obtain copies 208 and 206 of the encoded video data 204. The client 212 can include a video decoder 211 that decodes an input copy of the encoded video bitstream 208 and generates an output video sample stream 210 that can be rendered on a display 209 or other rendering device (not shown). In some streaming systems, the video bitstreams 204, 206, and 208 may be encoded according to a particular video encoding / compression standard, examples of which are described above and further herein.

[0020] FIG. 3 may be a functional block diagram of a video decoder 300 according to an embodiment of the present invention.

[0021] The receiver 302 may receive one or more coded video sequences to be decoded by the decoder 300, or in the same or other embodiments, one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel 301, which may be a hardware / software link to a storage device that stores the coded video data. The receiver 302 may receive the coded video data together with other data (e.g., coded audio data and / or auxiliary data streams), which may be forwarded to a respective using entity (not shown). The receiver 302 may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory 302 may be coupled between the receiver 302 and an entropy decoder / parser 304 (hereinafter referred to as the "parser"). If the receiver 302 is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isochronous network, the buffer 303 may not be necessary or may be small. For use in a best effort packet network such as the Internet, buffer 303 may be required and may be relatively large, and may advantageously be adaptively sized.

[0022] The video decoder 300 may include a parser 304 for recovering symbols 313 from the entropy coded video sequence. These categories of symbols include information used to manage the operation of the decoder 303, and potentially information for controlling a rendering device such as a display 312 that is not an integral part of the decoder but may be coupled to the decoder. The rendering device control information may be in the form of a Supplementary Enhancement Information (SEI) (SEI message) or a Video Usability Information (VUI) parameter set fragment (not shown). The parser 304 may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be according to a video coding technique or standard and may be according to principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 304 may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transformation unit (TU), a prediction unit (PU), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantization parameter values, motion vectors, etc. from the coded video sequence.

[0023] The parser 304 may perform an entropy decoding / parsing operation on the video sequence received from the buffer 303 to generate symbols 313. The parser 304 may receive the encoded data and selectively decode particular symbols 313. Additionally, the parser 304 may determine whether the particular symbol 313 should be provided to the motion compensated prediction unit 306, the scaler / inverse transform unit 305, the intra prediction unit 307, or the loop filter 311.

[0024] The recovery of symbols 313 may involve several different units, depending on the type of coded video picture or portion thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by parser 304. The flow of such subgroup control information between parser 304 and the following units is not shown for clarity.

[0025] In addition to the functional blocks described above, the decoder 300 can be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0026] The first unit is a scalar / inverse transform unit 305. The scalar / inverse transform unit 305 receives the quantized transform coefficients from the parser 304 as symbols 313, along with control information (including which transform to use, block size, quantization factor, quantization scaling matrix, etc.). The scalar / inverse transform unit can output blocks containing sample values ​​that can be input to an aggregator 310.

[0027] In some cases, the output samples of the scalar / inverse transform 305 may relate to intra-coded blocks, i.e. blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 uses surrounding already reconstructed information retrieved from the (partially reconstructed) current picture 308 to generate blocks of the same size and shape of the block being reconstructed. In some cases, the aggregator 310 adds, sample by sample, the prediction information generated by the intra-prediction unit 307 to the output sample information provided by the scalar / inverse transform unit 305.

[0028] In other cases, the output samples of the scalar / inverse transform unit 305 may relate to an inter-coded and potentially motion-compensated block. In such a case, the motion compensation prediction unit 306 may access the reference picture memory 308 to retrieve samples used for prediction. After motion compensating the retrieved samples according to the symbols 313 related to the block, these samples may be added by the aggregator 310 to the output of the scalar / inverse transform unit (in this case called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory available to the motion compensation unit from which the motion compensation unit retrieves prediction samples may be controlled by a motion vector, for example in the form of the symbols 313 that may have X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0029] The output samples of the aggregator 310 may be subjected to various loop filtering techniques in a loop filter unit 311. Video compression techniques may include in-loop filter techniques, controlled by parameters contained in the coded video bitstream and made available to the loop filter unit 311 as symbols 313 from the parser 304, which may be responsive to meta-information obtained during decoding of previous parts of the coded pictures or coded video sequences (in decoding order), as well as to previously reconstructed loop filtered sample values.

[0030] The output of the loop filter unit 311 may be a sample stream that may be output to the rendering device 312 as well as stored in the reference picture memory 557 for use in future inter-picture prediction.

[0031] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 304), the current reference picture 309 can become part of the reference picture buffer 308, and a new current picture memory can be reallocated before commencing reconstruction of the subsequent coded picture.

[0032] The video decoder 300 may perform decoding operations according to a given video compression technique, which may be documented in a standard such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to the syntax of the video compression technique or standard as specified in the video compression technique or standard, and in particular as specified in its profile document. Also required for compliance is that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level can be further limited through a Hypothetical Reference Decoder (HRD) specification and metadata about HRD buffer management conveyed in the coded video sequence.

[0033] In one embodiment, the receiver 302 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder 300 to properly decode the data and / or to more accurately recover the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0034] FIG. 4 may be a functional block diagram of a video encoder 400 according to one embodiment of the present disclosure.

[0035] The encoder 400 may receive video samples from a video source 401 (not part of the encoder), which may capture video images to be encoded by the encoder 400 .

[0036] The video source 401 may provide a source video sequence to be encoded by the encoder 303 in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media presentation system, the video source 401 may be a storage device that stores pre-prepared videos. In a video conferencing system, the video source 401 may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples, depending on the sampling structure, color space, etc. in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0037] According to one embodiment, the encoder 400 may encode and compress pictures of a source video sequence into an encoded video sequence 410 in real-time or under any other time constraint required by an application. Achieving an appropriate encoding rate is one function of the controller 402. The controller controls and is operatively coupled to other functional units, as described below. Coupling is not shown for clarity. Parameters set by the controller may include rate control related parameters (picture skip, quantization, lambda value for rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. One skilled in the art may readily recognize other functions of the controller 402 that may be relevant to a video encoder 400 optimized for a particular system design.

[0038] Some video encoders operate in what those skilled in the art will easily recognize as a "coding loop". As a very simplified description, the coding loop can consist of the coding part of the encoder 402 (herein referred to as the "source coder") (responsible for generating symbols based on the input picture to be coded and the reference pictures) and a (local) decoder 406 embedded in the encoder 400. The decoder 406 reconstructs the symbols to generate sample data similar to what the (remote) decoder would generate (so that any compression between the symbols and the coded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). That reconstructed sample stream is input to the reference picture memory 405. Since the decoding of the symbol stream results in bit-wise accurate results independent of the location of the decoder (local or remote), the contents of the reference picture buffer are also bit-wise accurate between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronization (including the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is well known to those skilled in the art.

[0039] The operation of the "local" decoder 406 may be the same as the "remote" decoder 300, which has already been described in detail above in relation to Figure 3. However, with brief reference to Figure 4, the entropy decoding part of the decoder 300, including the channel 301, the receiver 302, the buffer memory 303 and the parser 304, may not be fully implemented in the local decoder 406, since symbols are available and the encoding / decoding of the symbols into the coded video sequence by the entropy coder 408 and the parser 304 may be lossless.

[0040] An observation that can be made at this point is that any decoder techniques, other than analysis / entropy decoding, present in the decoder must necessarily be present in substantially the same functional form in the corresponding encoder. A description of the encoder techniques can be omitted, as they are the inverse of the decoder techniques, which are described generically. Only in certain areas are more detailed descriptions necessary, and are provided below.

[0041] As part of its operation, the source coder 403 may perform motion compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence designated as “reference frames.” In this manner, the coding engine 407 codes differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as predictive references for the input frame.

[0042] The local video decoder 406 may decode the encoded video data of frames that may be designated as reference frames based on the symbols generated by the source coder 403. The operation of the encoding engine 407 may advantageously be a lossy process. If the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The local video decoder 406 may replicate the decoding process that may be performed by the video decoder on the reference frames and store the reconstructed reference frames in the reference picture cache 405. In this way, the encoder 400 may locally store copies of reconstructed reference frames that have common content as the reconstructed reference frames (without transmission errors) obtained by the far-end video decoder.

[0043] The predictor 404 may perform a prediction search for the coding engine 407. That is, for a new frame to be coded, the predictor 404 may search the reference picture memory 405 for sample data (as candidate reference pixel blocks) or specific metadata (reference picture motion vectors, block shapes, etc.). These may serve as suitable prediction references for the new picture. The predictor 404 may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, the input picture determined by the search results obtained by the predictor 404 may have prediction references drawn from multiple reference pictures stored in the reference picture memory 405.

[0044] The controller 402 may manage the encoding operations of the video coder 403, including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0045] The output of all the above functional units may be subjected to entropy coding in an entropy coder 408. The entropy coder converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as, for example, Huffman coding, variable length coding, arithmetic coding, etc.

[0046] The transmitter 409 may buffer the encoded video sequence produced by the entropy coder 408 and prepare it for transmission over a communication channel 411, which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter 409 may merge the encoded video data from the video coder 403 with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (not shown).

[0047] A controller 402 may manage the operation of the encoder 400. During encoding, the controller 405 may assign each coded picture a particular coding picture type, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following frame types:

[0048] An intra picture (I-picture) may be one that can be coded and decoded without using other pictures in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh pictures. Those skilled in the art will recognize these variations of I-pictures and their respective uses and characteristics.

[0049] A predictive picture (P-picture) may be encoded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0050] Bidirectionally predicted pictures (B-pictures) may be encoded and decoded using intra- or inter-prediction, using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0051] In general, a source picture may be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8 or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture of the blocks. For example, blocks of an I-picture may be non-predictively coded or may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture may be non-predictively coded via spatial or temporal prediction with reference to the reference picture coded one step before. Blocks of a B-picture may be non-predictively coded via spatial or temporal prediction with reference to the reference picture coded one or two steps before.

[0052] Video coder 400 may perform encoding operations in accordance with a given video encoding technique or standard, such as ITU-T Rec. H.265. In its operation, video coder 400 may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0053] In one embodiment, the transmitter 409 may transmit additional data along with the coded video. The source coder 403 may include such data as part of the coded video sequence. The additional data may include other types of redundant data, such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0054] 5 shows a sample DASH client processing model 500, such as a client sample architecture for processing DASH and Common Media Application Format (CMAF) events, where a client request for a media segment may be based on a description address in the manifest that also describes a metadata track where the client may access segments of the metadata track, parse them, and send them to an application. Additionally, according to an exemplary embodiment, the DASH manifest may provide addresses for media segments as described below, addressed for index segments. Each index segment may provide information about one segment duration and size, and a representation index may provide index information for all segments of a given representation.

[0055] 6 shows an example picture-in-picture use case 600, where the main picture may occupy the entire screen, such as a windowed display or an augmented reality view, while the overlay picture, picture-in-picture, occupies a smaller area of ​​the screen covering the corresponding area of ​​the main picture. The coordinates of a picture-in-picture (pip) are given by x, y, height and width, which correspondingly define the position and size of the pip with respect to the main picture coordinates.

[0056] Looking at the example 700 in FIG. 7 and the flowchart 800 in FIG. 8, it can be understood that in the case of streaming, in S801, the main video and the pip video may be delivered as two separate streams, for example from different servers or at least from different sources such as video conferencing, according to an exemplary embodiment. Also, these streams may be delivered as independent streams to be decoded by separate decoders in S804 after the determination in S802 is determined to be No, or may be delivered directly from S801 and then combined for rendering. However, according to an exemplary embodiment, when the video codec used supports stream merging, in S802, for example, by a flag included with one or more of the video streams and / or as its metadata, it may be determined to have respective sub-pictures in S801, and in S803, the pip video stream may be combined with the main video stream, possibly replacing the streaming representing the coverage area of ​​the main video with the pip video, and then in S804, a single stream is sent to the decoder for decoding and subsequent rendering. This reduces the technical burden on the decoder, since a single merged stream may be delivered for decoding, rather than the initial independent streams being decoded separately and then merged.

[0057] According to exemplary embodiments such as examples 700 and 800, VVC sub-pictures can be used for picture-in-picture services by using both the extraction and merging properties of VVC sub-pictures. For such services, the primary video is coded using several sub-pictures, one of which is the same size as the auxiliary video, placed at the exact location where the auxiliary video is intended to be composited into the primary video, and coded independently to enable extraction. If the user selects to watch a version of the service that includes the auxiliary video, such as in S802, the sub-picture corresponding to the picture-in-picture area of ​​the primary video is extracted from the primary video bitstream, and the auxiliary video bitstream is merged with the primary video bitstream instead, as shown in examples 700 and 800, although for the purposes of such exemplary embodiments with respect to coding, decoding is replaced by coding as understood.

[0058] Also, the above annotations to one or more sub-picture properties of VVC in DASH, such as S802 and S803, may use MPD ContentComponent elements to describe properties of various sub-pictures of a VVC stream, according to an exemplary embodiment. For example, the use of such elements can be understood from Table 1 below. [Table 1] According to an exemplary embodiment, any VVC subpicture, ContentComponent element can annotate the subpicture and add it to an adaptation set or representation, and according to an exemplary embodiment, a DASH client can provide annotations to a bitstream manipulator to replace the desired subpicture stream with a picture-in-picture video stream, and then feed the manipulated VVC bitstream to a VVC decoder.

[0059] Moreover, the embodiments herein extend to other codecs such that the embodiments herein can be used for other video streams composed of such sub-pictures, and the same methods described above can be used for any audio or media stream composed of multiple independently decodable sub-streams, i.e., each sub-stream can be annotated using any one or more of the methods described above, thereby at least reducing the coding burden.

[0060] The techniques described above may be implemented as computer software using computer readable instructions, physically stored on one or more computer readable media, or implemented by one or more specially configured hardware processors. For example, Figure 9 illustrates a computer system 900 suitable for implementing certain embodiments of the disclosed subject matter.

[0061] Computer software may be encoded using any suitable machine code or computer language, which may be assembled, compiled, linked, or similar mechanisms to generate code including instructions that may be executed by a computer central processing unit (CPU), graphics processing unit (GPU), or the like, either directly or through an interpreter, microcode execution, or the like.

[0062] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0063] 9 for computer system 900 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 900.

[0064] The computer system 900 may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, through tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (scanned images, photographic images obtained from a still camera, etc.), and video (2D video, 3D video including stereoscopic pictures, etc.).

[0065] The input human interface devices may include one or more of a keyboard 901 , a mouse 902 , a trackpad 903 , a touch screen 910 , a joystick 905 , a microphone 906 , a scanner 908 , and a camera 907 .

[0066] The computer system 900 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 910 or joystick 905, although there may also be haptic feedback devices that do not function as input devices), audio output devices (speakers 909, headphones (not shown), etc.), visual output devices (screens 910 including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have touch screen input capability, each of which may or may not have haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or three or more dimensional output through such means as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0067] The computer system 900 may also include human accessible storage devices and associated media, such as optical media, including CD / DVD 911 or CD / DVD ROM / RW 920 with similar media, thumb drives 922, removable hard drives or solid state drives 923, legacy magnetic media such as tapes and floppy disks (not shown), specialized ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0068] Additionally, those skilled in the art should understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other non-transitory signals.

[0069] The computer system 900 may also include an interface 999 to one or more communication networks 998. The network 998 may be, for example, wireless, wired, optical. The network 998 may be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks 998 include Ethernet, wireless LAN, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), TV wired or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial broadcast TV), vehicular and industrial (including CANBus), etc. A particular network 998 typically requires an external network interface adapter (e.g., a USB port of the computer system 900, etc.) that is attached to a particular general-purpose data port or peripheral bus (950 and 951), while other network interface adapters are typically integrated into the core of the computer system 900 by being attached to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network to a smartphone computer system) as described below. Using any of these networks 998, computer system 900 can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used in each of these networks and network interfaces.

[0070] The above-mentioned human interface devices, human accessible storage devices and network interfaces may be attached to a core 940 of the computer system 900 .

[0071] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, graphics adapters 917, specialized programmable processing units in the form of field programmable gate arrays (FPGAs) 943, hardware accelerators for specific tasks 944, etc. These devices may be connected through a system bus 948, along with read only memory (ROM) 945, random access memory 946, and internal mass storage (internal non-user accessible hard drive, SSD, etc.) 947. In some computer systems, the system bus 948 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 948 or through a peripheral bus 951. Peripheral bus architectures include PCI, USB, etc.

[0072] The CPU 941, GPU 942, FPGA 943, and accelerator 944 may execute certain instructions, which in combination may constitute the above computer code. The computer code may be stored in ROM 845 or RAM 946. Also, temporary data may be stored in RAM 946, and persistent data may be stored, for example, in internal mass storage device 947. The use of cache memory, which may be closely associated with one or more of the CPU 941, GPU 942, mass storage device 947, ROM 945, RAM 946, etc., allows for fast storage and retrieval in any of the memory devices.

[0073] The computer readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0074] By way of example and not limitation, the architecture 900 corresponding to the computer system 900, and in particular the core 940, may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage devices as described above, as well as specific storage of the core 940 of a non-transitory nature, such as mass storage device 947 internal to the core or ROM 945. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core 940. The computer-readable media may include one or more memory devices or chips according to specific needs. The software may cause the core 940, and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain operations or certain portions of certain operations described herein, including defining data structures stored in RAM 946 and modifying such data structures according to operations defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator 944) that may operate in place of or in conjunction with software to perform particular operations or portions of particular operations described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure includes any appropriate combination of hardware and software.

[0075] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be recognized that those skilled in the art may devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. 1. A method of video coding, comprising: obtaining a dynamic adaptive streaming over HTTP (DASH) video bitstream including one or more subpictures; determining whether the one or more sub-pictures include a Versatile Video Coding (VVC) compliant sub-picture; in response to the DASH video bitstream including the VVC-compliant sub-picture, annotating the one or more sub-pictures as picture-in-picture based on one or more flags; manipulating the DASH video bitstream based on the annotated one or more subpictures; The method includes:

2. the one or more sub-pictures are a sub-picture of a first stream and a sub-picture of a second stream; The method includes controlling a client to replace at least a portion of one of the first stream and the second stream with another of the first stream and the second stream based on the annotated one or more sub-pictures; The video data includes Dynamic Adaptive Streaming over HTTP (DASH) video data, The method of claim 1 , wherein the client is a DASH client.

3. 3. The method of claim 2, further comprising: determining whether to annotate one or more of the sub-pictures of the first stream and the sub-pictures of the second stream, the determining step comprising determining whether to annotate the one or more of the sub-pictures of the first stream and the sub-pictures of the second stream in a DASH media picture description (MPD).

4. The method of claim 2 , wherein the other of the first stream and the second stream comprises a picture-in-picture (PIP) stream.

5. 5. The method of claim 4, wherein controlling the client to replace the at least a portion of the one of the first stream and the second stream comprises replacing an area represented by a plurality of sub-pictures of a sub-picture of the first stream and a sub-picture of the second stream with the PIP stream.

6. 6. The method of claim 5, wherein the one of the first stream and the second stream represents a primary video, and the other of the first stream and the second stream represents a stream separately coded as an auxiliary video to the primary video.

7. 7. The method of claim 6, wherein controlling the client to replace the at least a portion of the one of the first stream and the second stream includes determining whether a user has requested to view the auxiliary video.

8. The method of claim 7 , wherein the primary video and the auxiliary video each include a versatile video coding (VCC) sub-picture.

9. 3. The method of claim 2, wherein controlling the client to replace the at least a portion of the one of the first stream and the second stream comprises merging VVC subpicture properties such that a single merged stream from both the first stream and the second stream is provided to a decoder.

10. The method of claim 2 , wherein acquiring the video data comprises acquiring the first stream separately from the second stream.

11. An apparatus for video streaming, comprising: at least one memory configured to store computer program code; at least one processor configured to access the computer program code and to operate as instructed by the computer program code; Including, The computer program code comprises: Apparatus configured to cause said at least one processor to perform the method of any one of claims 1 to 10.

12. A computer program product causing a computer to carry out the method according to any one of claims 1 to 10.

Images