Video encoding method and video transmission apparatus

Abstract

A method and apparatus comprising computer code configured to cause one or more processors to implement: obtaining a dynamic adaptive streaming over HTTP (DASH) video bitstream comprising one or more sub-pictures; determining whether the one or more sub-pictures comprise VVC compatible sub-pictures; responsive to the DASH video bitstream comprising VVC compatible sub-pictures, annotating the one or more sub-pictures based on one or more flags; and controlling the DASH video stream based on the annotated one or more sub-pictures.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 298,924, filed January 12, 2022, and U.S. Application No. 18 / 073,984, filed December 2, 2022, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the transmission of dynamic adaptive streaming over HTTP (DASH) signals, and particularly to video encoding methods, video transmission apparatus, non-transitory computer-readable media, and computer devices. Background Technology

[0004] MPEG DASH provides a standard for streaming multimedia content over IP networks. The ISO / IEC 23009-1 DASH standard allows for streaming of multi-rate content. DASH manifests and MPDs can describe various media content. While the DASH standard provides methods for describing various content and their relationships, it does not provide an interoperable solution for annotating VVC sub-pictures for picture-in-picture applications.

[0005] Therefore, the technical problem to be solved by this disclosure is how to annotate VVC sub-screens for use in picture-in-picture applications. Summary of the Invention

[0006] To address one or more different technical problems, this disclosure provides technical solutions for reducing network overhead and server computational overhead, and includes methods and apparatus. The apparatus includes a memory configured to store computer program code and one or more processors configured to access and operate as instructed by the computer program code. The computer program code includes: acquisition code configured to cause at least one processor to acquire an HTTP-based Dynamic Adaptive Streaming (DASH) video bitstream comprising one or more sub-frames; determination code configured to cause at least one processor to determine whether one or more sub-frames include Versatile Video Coding (VVC) compliant sub-frames; annotation code configured to cause at least one processor to annotate one or more sub-frames based on one or more flags in response to the DASH video bitstream including VVC compliant sub-frames; and control code configured to cause at least one processor to control the DASH video stream based on the annotated one or more sub-frames.

[0007] According to an exemplary implementation, the video data is in an HTTP-based Dynamic Adaptive Stream (DASH), and the client is a DASH client.

[0008] According to an exemplary implementation, determining whether to annotate one or more of the sub-pictures corresponding to the first stream and the second stream includes determining whether to annotate one or more of the sub-pictures corresponding to the first stream and the second stream with a DASH media picture description (MPD).

[0009] According to an exemplary implementation, the other of the first and second streams includes a picture-in-picture (PIP) stream.

[0010] According to an exemplary implementation, controlling the client to replace at least a portion of one of the first stream and the second stream includes: replacing the area represented by a plurality of sub-screens in a sub-screen corresponding to the first stream or the sub-screen corresponding to the second stream with a PIP stream.

[0011] According to an exemplary implementation, one of the first stream and the second stream represents the main video, and the other of the first stream and the second stream represents a separately encoded stream that serves as a supplementary video to the main video.

[0012] According to an exemplary implementation, controlling the client to replace at least a portion of one of the first and second streams includes: determining whether the user requests to watch supplementary video.

[0013] According to an exemplary implementation, the main video and the supplementary video each include a Universal Video Coding (VVC) sub-picture.

[0014] According to an exemplary implementation, controlling the client to replace at least a portion of one of the first and second streams includes: merging the properties of generic video coding (VVC) sub-pictures such that the decoder is provided with a single merged stream merged from both the first and second streams.

[0015] According to an exemplary embodiment, a video encoding method is provided, comprising: acquiring an HTTP-based Dynamic Adaptive Streaming (DASH) video bitstream, the DASH video bitstream comprising one or more sub-frames; determining whether the one or more sub-frames comprise a Universal Video Coding (VVC) compatible sub-frame; if the DASH video bitstream comprises the VVC compatible sub-frame, annotating the one or more sub-frames based on one or more flags; and controlling the DASH video bitstream based on the annotated one or more sub-frames.

[0016] According to an exemplary embodiment, a video transmission method is provided, comprising: an acquisition unit configured to acquire a Dynamic Adaptive Streaming (DASH) video bitstream based on HTTP, the DASH video bitstream comprising one or more sub-frames; a determination unit configured to determine whether the one or more sub-frames include a Universal Video Coding (VVC) compatible sub-frame; an annotation unit configured to annotate the one or more sub-frames based on one or more flags if the DASH video bitstream includes the VVC compatible sub-frame; and a control unit configured to control the DASH video stream based on the annotated one or more sub-frames.

[0017] According to an exemplary embodiment, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium storing a program that causes a computer to perform: acquiring an HTTP-based Dynamic Adaptive Streaming (DASH) video bitstream, the DASH video bitstream comprising one or more sub-frames; determining whether the one or more sub-frames include a Universal Video Coding (VVC) compatible sub-frame; if the DASH video bitstream includes the VVC compatible sub-frame, annotating the one or more sub-frames based on one or more flags; and controlling the DASH video bitstream based on the annotated one or more sub-frames.

[0018] According to an exemplary embodiment, a computer device is provided, including a processor and a memory. The memory is used to store program code and transmit the program code to the processor; the processor is used to execute, according to instructions in the program code: acquiring an HTTP-based Dynamic Adaptive Streaming (DASH) video bitstream, the DASH video bitstream including one or more sub-frames; determining whether the one or more sub-frames include Universal Video Coding (VVC) compatible sub-frames; if the DASH video bitstream includes the VVC compatible sub-frames, annotating the one or more sub-frames based on one or more flags; and controlling the DASH video bitstream based on the annotated one or more sub-frames.

[0019] According to the video encoding method and video transmission apparatus of this disclosure, a DASH video bitstream is obtained, the DASH video bitstream comprising one or more sub-frames; it is determined whether the one or more sub-frames include VVC-compatible sub-frames; if the DASH video bitstream includes the VVC-compatible sub-frames, the one or more sub-frames are annotated based on one or more flags; and the DASH video bitstream is controlled based on the annotated one or more sub-frames. The method and apparatus of this application can be used for any audio or media stream composed of multiple sub-streams that can be independently decoded. That is, any one or more of the above methods can be used to annotate each sub-stream, thereby at least reducing the encoding burden. Attached Figure Description

[0020] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0021] Figure 1 This is a simplified schematic diagram based on the implementation method.

[0022] Figure 2 This is a simplified schematic diagram based on the implementation method.

[0023] Figure 3 This is a simplified block diagram of the decoder according to an implementation method.

[0024] Figure 4 This is a simplified block diagram of the encoder according to an implementation method.

[0025] Figure 5 This is a simplified block diagram based on the implementation method.

[0026] Figure 6 This is a simplified diagram based on the implementation method.

[0027] Figure 7 This is a simplified diagram based on the implementation method.

[0028] Figure 8 This is a simplified flowchart based on the implementation method.

[0029] Figure 9 This is a schematic diagram based on the implementation method. Detailed Implementation

[0030] The proposed features discussed below can be used individually or in any order. Furthermore, implementations can be carried out using a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium.

[0031] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present disclosure is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may encode video data at a local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission is common in media service applications, etc.

[0032] Figure 1 A second pair of terminals 101 and 104 is shown, configured to support bidirectional transmission of encoded video, for example, during video conferencing. For bidirectional data transmission, each terminal 101 and 104 can encode video data captured at a local location for transmission to the other terminal via network 105. Each terminal 101 and 104 can also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0033] exist Figure 1 In this disclosure, terminals 101, 102, 103, and 104 may be shown as servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 refers to any number of networks transmitting encoded video data between terminals 101, 102, 103, and 104, including, for example, wired communication networks and / or wireless communication networks. Communication network 105 may exchange data in line-switched channels and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, unless otherwise stated below, the architecture and topology of network 105 may be irrelevant to the operation of this disclosure.

[0034] Figure 2 The placement of a video encoder and video decoder in a streaming environment is illustrated as an example of an application to the disclosed subject matter. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0035] The streaming system may include a capture subsystem 203, which may include a video source 201, such as a digital camera device, that creates, for example, an uncompressed video sample stream 213. The sample stream 213 may be characterized by a high data volume when compared to an encoded video bitstream and may be processed by an encoder 202 coupled to the camera device 201. The encoder 202 may include hardware, software, or a combination thereof to implement or enforce various aspects of the disclosed subject matter as described in more detail below. An encoded video bitstream 204, characterized by a lower data volume when compared to the sample stream, may be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve copies 208 and 206 of the encoded video bitstream 204. Client 212 may include video decoder 211, which decodes an incoming copy of the encoded video bitstream 208 and creates an outgoing video sample stream 210 that can be presented on display 209 or other presentation device (not depicted). In some streaming systems, video bitstreams 204, 206, and 208 may be encoded according to certain video encoding / compression standards. Examples of these standards have been mentioned above and are further described herein.

[0036] Figure 3 This can be a functional block diagram of a video decoder 300 according to an embodiment of the present invention.

[0037] Receiver 302 can receive one or more codec video sequences to be decoded by decoder 300; in the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Encoded video sequences can be received from channel 301, which can be a hardware / software link to a storage device storing the encoded video data. Receiver 302 can receive encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective user entities (not depicted). Receiver 302 can separate the encoded video sequences from other data. To prevent network jitter, buffer memory 303 can be coupled between receiver 302 and entropy decoder / parser 304 (hereinafter referred to as "parser"). Buffer 303 may not be required or may be small when receiver 302 is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network. For use in best-effort packet networks such as the Internet, a buffer 303 may be required. The buffer 303 can be relatively large and can advantageously have an adaptive size.

[0038] Video decoder 300 may include parser 304 to reconstruct symbols 313 from an entropy-encoded video sequence. These symbols may include information for managing the operation of decoder 300, and potentially information for controlling a presentation device such as display 312, which is not part of the decoder but may be coupled to it. Control information for the presentation device may be in the form of supplementary enhancement information (SEI) messages or fragments of video availability information parameter sets (not depicted). Parser 304 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 304 may extract a subgroup parameter set from at least one subgroup of pixels in the encoded video sequence for the video decoder based on at least one parameter corresponding to a group. Subgroups can include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, Coding Units (CU), blocks, Transform Units (TU), Prediction Units (PU), etc. The entropy decoder / parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0039] The parser 304 can perform entropy decoding / parsing operations on the video sequence received from the buffer 303 to create symbol 313. The parser 304 can receive encoded data and selectively decode specific symbols 313. Furthermore, the parser 304 can determine whether to provide specific symbols 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra-frame prediction unit 307, or the loop filter 311.

[0040] Depending on the type of the encoded video picture or a portion thereof (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 313 may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 304. For the sake of brevity, the flow of such subgroup control information between parser 304 and the following units is not described.

[0041] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into multiple functional units as described below. In actual implementation under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the following functional units.

[0042] The first unit is the scaler / inverse transform unit 305. The scaler / inverse transform unit 305 receives quantization transform coefficients as symbols 313 from the parser 304, along with control information including the type of transform used, block size, quantization factor, and quantization scaling matrix. The scaler / inverse transform unit 305 can output blocks containing sample values, which can be input to the aggregator 310.

[0043] In some cases, the output samples of the scaler / inverse transform 305 may belong to an intra-coded block; that is, this block does not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information can be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 uses surrounding reconstructed information extracted from the current (partially reconstructed) image 309 to generate a block of the same size and shape as the reconstructed block. In some cases, the aggregator 310 adds the predictive information already generated by the intra-picture prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305 based on each sample.

[0044] In other cases, the output samples of the scaler / inverse transform unit 305 may belong to inter-frame encoded and possibly motion-compensated blocks. In this case, the motion compensation prediction unit 306 can access the reference image buffer 308 to extract samples for prediction. After motion compensation of the extracted samples according to the symbol 313 belonging to the block, these samples can be added by the aggregator 310 to the output of the scaler / inverse transform unit (referred to in this case as residual samples or residual signals) to generate output sample information. The address in the reference image memory from which the motion compensation unit extracts its predicted samples can be controlled by motion vectors, which are provided to the motion compensation unit in the form of symbols 313, which may have, for example, X components, Y components, and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0045] The output samples of aggregator 310 can undergo various loop filtering techniques in loop filter unit 311. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 311 as symbols 313 from parser 304. However, video compression techniques may also respond to metadata acquired during decoding of previous (in decoding order) portions of encoded images or encoded video sequences, and to previously reconstructed and loop-filtered sample values.

[0046] The output of the loop filter unit 311 can be a sample stream, which can be output to the presentation device 312 and stored in the reference image memory 557 for future inter-frame image prediction.

[0047] Once fully reconstructed, certain encoded images can be used as reference images for future predictions. Once an encoded image has been fully reconstructed and has been identified as a reference image (e.g., by parser 304), the current reference image 309 can become part of the reference image buffer 308, and new current image memory can be reallocated before reconstructing subsequent encoded images begins.

[0048] The video decoder 300 can perform decoding operations according to predetermined video compression techniques that can be recorded in standards such as ITU-T Rec. H.265. In the sense that the encoded video sequence conforms to the syntax of a video compression technique or standard, the encoded video sequence can conform to the syntax specified by the video compression technique or standard being used, such as the syntax specified in the video compression technique document or standard and explicitly in the brief document therein. For compliance, the complexity of the encoded video sequence is also required to be within the limits defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffers that are signaled in the encoded video sequence.

[0049] In this implementation, receiver 302 can receive supplemental (redundant) data as well as encoded video. The supplemental data can be included as part of one or more encoded video sequences. The supplemental data can be used by video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The supplemental data can take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0050] Figure 4 This may be a functional block diagram of a video encoder 400 according to an embodiment of this disclosure.

[0051] The encoder 400 can receive video samples from a video source 401 (which is not part of the encoder), which can capture video images to be encoded by the encoder 400.

[0052] Video source 401 can provide a source video sequence in the form of a digital video sample stream to be encoded by encoder (303). This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, video source 401 can be a storage device storing previously prepared video. In a video conferencing system, video source 401 can be a camera device capturing local image information as a video sequence. Video data can be provided as multiple individual pictures that are given motion when viewed in sequence. The pictures themselves can be organized as spatial pixel arrays, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0053] According to the implementation, encoder 400 can encode and compress images of the source video sequence into an encoded video sequence 410 in real time or according to any other time constraints required by the application. Performing an appropriate encoding rate is a function of controller 402. The controller controls and is functionally coupled to other functional units described below. For simplicity, the coupling is not depicted. Parameters set by the controller may include: rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of images (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can readily identify other functions of controller 402, as these functions may belong to the video encoder 400 optimized for a particular system design.

[0054] Some video encoders operate with an “encoding loop” as readily understood by those skilled in the art. As an oversimplification, the encoding loop may include: the encoding portion of encoder 402 (hereinafter referred to as the “source encoder”) (responsible for creating symbols based on the input picture and reference picture to be encoded) and a (local) decoder 406 embedded in encoder 400, which reconstructs the symbols to create sample data that the (remote) decoder will also create (since any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). This reconstructed sample stream is input to a reference picture memory 405. Since decoding the symbol stream results in bit-precise results independent of the decoder location (local or remote), the contents of the reference picture buffer are also bit-precise between the local and remote encoders. In other words, the reference picture samples “seen” by the encoder’s prediction portion are exactly the same sample values ​​that the decoder will “see” when using prediction during decoding. The basic principles of this reference picture synchronization (and the resulting drift if synchronization cannot be maintained, for example, due to channel errors) are well known to those skilled in the art.

[0055] The operation of the "local" decoder 406 can be combined with what has already been mentioned above. Figure 3 The operation is the same as that of the "remote" decoder 300 described in detail. However, a brief reference is also provided. Figure 4 When symbols are available and the entropy encoder 408 and the parser 304 can losslessly encode / decode the symbols into an encoded video sequence, the entropy decoding portion of the decoder 300, which includes the channel 301, receiver 302, buffer 303, and parser 304, may not be fully implemented in the local decoder 406.

[0056] It can then be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also exist in the corresponding encoder in essentially the same functional form. Since encoder techniques are inverses of the fully described decoder techniques, the description of encoder techniques can be simplified. More detailed descriptions are only needed in certain areas and are provided below.

[0057] As part of its operation, the source encoder 403 can perform motion-compensated predictive coding, which predictively codes the input frame with reference to one or more previously encoded frames in the video sequence designated as "reference frames". In this way, the encoding engine 407 encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which can be selected as the prediction reference for the input frame.

[0058] The local video decoder 406 can decode encoded video data of frames that can be designated as reference frames based on symbols created by the source encoder 403. The operation of the encoding engine 407 can advantageously be lossy. When encoded video data can be decoded by the video decoder (… Figure 4 When decoded at (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 406 copies the decoding processing performed on the reference frame by the video decoder, and the reconstructed reference frame can be stored in the reference image buffer 405. In this way, the encoder 400 can locally store a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame that will be acquired by the remote video decoder.

[0059] Predictor 404 can perform a prediction search against encoding engine 407. That is, for a new image to be encoded, predictor 404 can search the reference image memory 405 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can be used as appropriate prediction references for the new image. Predictor 404 can operate pixel-by-pixel based on the sample blocks to find appropriate prediction references. In some cases, as determined by the search results obtained by predictor 404, the input image may have prediction references extracted from multiple reference images stored in reference image memory 405.

[0060] The controller 402 can manage the encoding operations of the video encoder 403, including, for example, setting parameters and subgroup parameters for encoding video data.

[0061] The outputs of all the functional units mentioned above can undergo entropy encoding in the entropy encoder 408. The entropy encoder converts the symbols generated by the various functional units into an encoded video sequence by performing lossless compression of the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, arithmetic coding, etc.).

[0062] Transmitter 409 may buffer one or more encoded video sequences created by entropy encoder 408 in preparation for transmission via communication channel 411, which may be a hardware / software link to a storage device storing the encoded video data. Transmitter 409 may combine encoded video data from video encoder 403 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0063] Controller 402 can manage the operation of encoder 400. During encoding, controller 405 can assign a specific encoded image type to each encoded image, which may affect the encoding technique that can be applied to the corresponding image. For example, one of the following frame types can typically be assigned to an image:

[0064] An intra-frame picture (I-picture) can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, independent decoder refresh pictures. Those skilled in the art will understand these variations of I-pictures and their corresponding applications and characteristics.

[0065] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0066] Bidirectional predictive images (B-images) can be images that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multi-predictive images can be reconstructed from a single block using more than two reference images and associated metadata.

[0067] Source images are typically spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples) and encoded block-by-block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignment of the corresponding images applied to the blocks. For example, blocks of an I-image can be nonpredictively coded, or the blocks can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be nonpredictively coded with reference to a previously coded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial prediction or temporal prediction.

[0068] The video encoder 400 can perform encoding operations according to a predetermined video coding technology or standard, such as ITU-T Rec H.265. In its operation, the video encoder 400 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technology or standard being used.

[0069] In this implementation, transmitter 409 may transmit additional data as well as encoded video. Source encoder 403 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant images and slices, Supplementary Enhancement Information (SEI) messages, Video Usability Information (VUI) parameter set fragments, etc.

[0070] Figure 5 A sample DASH client processing model 500 is shown, for example, for handling DASH events and Common Media Application Format (CMAF) events, where client requests for media segments can be based on addresses described in a manifest that also describes metadata tracks from which the client can access segments, parse segments, and send segments to the application. Furthermore, according to an exemplary implementation, the DASH manifest can provide addressing for index segments regarding the addresses of media segments as described below. Each index segment can provide information about the duration and size of a segment, and the index can provide index information for all segments of a given representation.

[0071] Figure 6 Example 600 illustrates a picture-in-picture use case where the main screen can occupy the entire screen, such as a window display or augmented reality view, while the overlay, or picture-in-picture, occupies a small area of ​​the screen, covering the corresponding area of ​​the main screen. The coordinates of the picture-in-picture (pip) are indicated by x, y, height, and width, where these parameters define the position and size of the pip relative to the coordinates of the main screen.

[0072] Check Figure 7 Example 700 and Figure 8As will be understood from flowchart 800, in the case of streaming, the main video and pip video can be transmitted as two separate streams at step 801, for example from different servers or at least different sources, such as in a video conference. These streams can be transmitted as additional independent streams decoded by a separate decoder at step 804 after a determination of no or no regarding the determination at step 802, or transmitted directly from step 801 and then combined for presentation. However, in the case of the exemplary embodiment, when the video codec used supports merged streams, the corresponding sub-picture at step 801 can be determined at step 802, for example by a flag included in one or more video streams in the video stream and / or as their metadata. At step 803, the pip video stream can be combined with the main video stream, replacing the stream representing the coverage area of ​​the main video with the pip video. Then, at step 804, a single stream is sent to the decoder for decoding and then presentation. This reduces the technical burden on the decoder, as a single merged stream can be transmitted for decoding, rather than decoding and merging the initial independent streams separately.

[0073] According to exemplary embodiments, such as Examples 700 and 800, VVC sub-pictures can be used for picture-in-picture services by using both extraction and merging attributes of the VVC sub-pictures. For such a service, the main video is encoded using several sub-pictures, one of which is the same size as the supplementary video, located at the exact position where the supplementary video is intended to be composited into the main video, and is independently encoded for extraction. If, for example, a user selects to watch a version of the service that includes the supplementary video at step 802, the sub-picture corresponding to the picture-in-picture region of the main video is extracted from the main video bitstream, and the supplementary video bitstream is merged with the main video bitstream, as shown in Examples 700 and 800, but encoding is used instead of the decoding described above; this will be understood as the purpose of such an exemplary embodiment regarding encoding.

[0074] Furthermore, according to exemplary embodiments, such as at steps 802 and 803, the aforementioned annotations of the attributes of one or more sub-screens of the VVC in the DASH can be described using MPD content component elements to describe the attributes of various sub-screens of the VVC stream. For example, the use of such elements can be understood from Table 1 below:

[0075] Table 1 - Semantics of Content Components for VVC Sub-screen Annotations according to Exemplary Implementations

[0076]

[0077] Furthermore, according to an exemplary implementation, any VVC sub-picture or content component can be added to the adapter set or representation, and the sub-picture can be annotated. Also according to an exemplary implementation, the DASH client can provide annotations to the bitstream controller to replace the desired sub-picture stream with a picture-in-picture video stream, and then feed the controlled VVC bitstream to the VVC decoder.

[0078] Furthermore, the embodiments described herein are extended to other codecs, enabling them to be used for other video streams composed of such sub-pictures, and the same methods described above can be used for any audio or media stream composed of multiple sub-streams that can be decoded independently. That is, any one or more of the methods described above can be used to annotate each sub-stream, thereby at least reducing the coding burden.

[0079] According to an exemplary embodiment, this disclosure provides a video encoding method, including:

[0080] Acquire an HTTP-based Dynamic Adaptive Streaming (DASH) video bitstream, wherein the DASH video bitstream comprises one or more sub-frames;

[0081] Determine whether the one or more sub-frames include a Universal Video Coding (VVC) compatible sub-frame;

[0082] If the DASH video bitstream includes the VVC-compatible sub-frame, annotate the one or more sub-frames based on one or more flags; and

[0083] The DASH video bitstream is controlled based on one or more annotated sub-pictures.

[0084] In one example, the one or more sub-screens are sub-screens corresponding to a first stream and sub-screens corresponding to a second stream; wherein, the method includes: controlling the client to replace at least a portion of one of the first stream and the second stream with another of the first stream and the second stream based on the annotated one or more sub-screens.

[0085] The video data includes HTTP-based Dynamic Adaptive Streaming (DASH) video data, and

[0086] The client in question is a DASH client.

[0087] In one example, the method further includes: determining whether to annotate one or more of the sub-screens corresponding to the first stream and the second stream.

[0088] The step of determining whether to annotate one or more of the sub-screens corresponding to the first stream and the second stream includes: determining whether to annotate one or more of the sub-screens corresponding to the first stream and the second stream using DASH Media Screen Description (MPD).

[0089] In one example, the other of the first and second streams includes a picture-in-picture (PIP) stream.

[0090] In one example, controlling the client to replace at least a portion of one of the first stream and the second stream includes: replacing the area represented by a plurality of sub-pictures in a sub-picture corresponding to the first stream or the sub-picture corresponding to the second stream with the PIP stream.

[0091] In one example, one of the first stream and the second stream represents the main video, and the other of the first stream and the second stream represents a separately encoded stream that serves as a supplementary video to the main video.

[0092] In one example, controlling the client to replace at least a portion of one of the first and second streams includes determining whether the user requests to watch the supplementary video.

[0093] In one example, the main video and the supplementary video each include a Universal Video Coding (VVC) sub-picture.

[0094] In one example, controlling the client to replace at least a portion of one of the first stream and the second stream includes: merging the attributes of the VVC sub-screen such that the decoder is provided with a single merged stream that is merged from both the first stream and the second stream.

[0095] In one example, acquiring video data includes acquiring the first stream separately from the second stream.

[0096] According to an exemplary embodiment, this disclosure provides a video transmission apparatus, characterized in that the apparatus includes: an acquisition unit configured to acquire a Dynamic Adaptive Streaming (DASH) video bitstream based on HTTP, the DASH video bitstream including one or more sub-frames; a determination unit configured to determine whether the one or more sub-frames include a Universal Video Coding (VVC) compatible sub-frame; an annotation unit configured to annotate the one or more sub-frames based on one or more flags if the DASH video bitstream includes the VVC compatible sub-frame; and a control unit configured to control the DASH video stream based on the annotated one or more sub-frames.

[0097] The above-described technologies can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media, or they can be implemented using one or more specially configured hardware processors. For example, Figure 9 A computer system 900 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0098] Computer software can be encoded using any suitable machine code or computer language. Machine code or computer language can be subjected to mechanisms such as assembly, compilation, and linking to create code that includes instructions. These instructions can be executed directly by the computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, etc.

[0099] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0100] Figure 9 The components shown for computer system 900 are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any one or a combination of components shown in the exemplary embodiments of computer system 900.

[0101] Computer system 900 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), and olfactory input (not depicted). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images acquired from still image capturing devices), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0102] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard 901, mouse 902, touchpad 903, touch screen 910, joystick 905, microphone 906, scanner 908, and camera device 907.

[0103] The computer system 900 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such HMI output devices may include: tactile output devices (e.g., tactile feedback from a touchscreen 910 or joystick 905, but tactile feedback devices that are not used as input devices may also exist); audio output devices (e.g., speakers 909, headphones (not depicted)); visual output devices (e.g., screens 910 including CRT screens, LCD screens, plasma screens, OLED screens, each screen with or without touchscreen input functionality, each screen with or without tactile feedback capability—some of which may be able to output two-dimensional or more than three-dimensional visual outputs through means such as stereoscopic output; virtual reality glasses (not depicted); holographic displays and smoke boxes (not depicted)); and printers (not depicted).

[0104] The computer system 900 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 920 or similar media including CD / DVD 911, thumb drives 922, removable hard disk drives or solid-state drives 923, conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD, such as security dongles (not depicted), etc.

[0105] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0106] Computer system 900 may also include interfaces 999 to one or more communication networks 998. Network 998 may be, for example, wireless, wired, or optical. Network 998 may also be local, wide area, metropolitan area, vehicle and industrial, real-time, latency-tolerant, etc. Examples of network 998 include: local area networks (e.g., Ethernet, wireless LAN); cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; cable television connections or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television; vehicle and industrial networks including CANBus, etc. Some networks 998 typically require external network interface adapters that are attached to certain general-purpose data ports or peripheral buses (950 and 951) (such as, for example, the USB port of computer system 900); other networks are typically integrated into the core of computer system 900 by attaching to system buses as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks 998, computer system 900 can communicate with other entities. Such communication can be one-way receive-only (e.g., broadcasting TV), one-way send-only (e.g., CANbus to certain CANbus devices), or bidirectional, such as using a local area or wide area digital network to other computer systems. Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0107] The aforementioned human-machine interface device, human-accessible storage device, and network interface can be attached to the core 940 of the computer system 900.

[0108] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, graphics adapters 917, dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) 943, hardware accelerators 944 for certain tasks, etc. These devices, along with read-only memory (ROM) 945, random access memory 946, and internal mass storage devices such as internal non-user-accessible hard disk drives (SDs) 947, can be connected via a system bus 948. In some computer systems, the system bus 948 may be accessed as one or more physical connectors to allow for expansion by adding CPUs, GPUs, etc. Peripheral devices can be attached directly or via a peripheral bus 951 to the core's system bus 948. Peripheral bus architectures include PCI, USB, etc.

[0109] The CPU 941, GPU 942, FPGA 943, and accelerator 944 can execute specific instructions, and combinations of these instructions can constitute the aforementioned computer code. This computer code can be stored in ROM 945 or RAM 946. Transient data can also be stored in RAM 946, while permanent data can be stored, for example, in an internal mass storage device 947. Any fast storage and retrieval of the memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs 941, GPUs 942, mass storage devices 947, ROMs 945, RAMs 946, etc.

[0110] Computer-readable media may have computer code thereon for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type known and available to those skilled in the art of computer software.

[0111] By way of example and in a non-limiting manner, corresponding to the architecture of computer system 900, and particularly core 940, functionality can be provided by processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, as well as certain storage devices of core 940 with non-transitory characteristics, such as internal mass storage device 947 or ROM 945. Software implementing various embodiments of this disclosure can be stored in such devices and executed by core 940. Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause core 940, and particularly its processors (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 946 and modifying such data structures according to software-defined processes. Alternatively or as an alternative, the computer system may provide functionality through logic hardwired or otherwise implemented in circuitry (e.g., accelerator 944), which may replace or operate with software to perform the specific processing or a specific portion of the specific processing described herein. Where appropriate, references to software may include logic, and conversely, references to logic may include software. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry implementing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0112] Although several exemplary embodiments have been described in this disclosure, there are changes, substitutions, and various alternative equivalents that fall within the scope of this disclosure. It will therefore be appreciated that those skilled in the art will be able to conceive of many systems and methods that, while not expressly shown or described herein, implement the principles of this disclosure and are therefore within its spirit and scope.

Claims

1. A video encoding method, characterized in that, The method includes: Obtain a dynamically adaptive streaming DASH video bitstream based on HTTP, wherein the DASH video bitstream includes one or more sub-frames; Determine whether the one or more sub-frames include VVC-compatible sub-frames; If the DASH video bitstream includes the VVC-compatible sub-frame, annotate the one or more sub-frames based on one or more flags; and The DASH video bitstream is controlled based on one or more annotated sub-pictures.

2. The method according to claim 1, characterized in that, The one or more sub-screens are sub-screens corresponding to the first stream and sub-screens corresponding to the second stream; wherein, the method includes: controlling the client to replace at least a portion of one of the first stream and the second stream with the other of the first stream and the second stream based on the annotated one or more sub-screens. The video data includes HTTP-based dynamically adaptive streaming DASH video data, and The client in question is a DASH client.

3. The method according to claim 1 or 2, characterized in that, The method further includes: determining whether to annotate one or more of the sub-screens corresponding to the first stream and the second stream. The step of determining whether to annotate one or more of the sub-screens corresponding to the first stream and the second stream includes: determining whether to annotate one or more of the sub-screens corresponding to the first stream and the second stream using DASH Media Screen Description (MPD).

4. The method according to claim 2, characterized in that, Another of the first and second streams includes a picture-in-picture (PIP) stream.

5. The method according to claim 4, characterized in that, Controlling the client to replace at least a portion of one of the first stream and the second stream includes: replacing the area represented by a plurality of sub-pictures in a sub-picture corresponding to the first stream or the sub-picture corresponding to the second stream with the PIP stream.

6. The method according to claim 4 or 5, characterized in that, One of the first stream and the second stream represents the main video, and the other of the first stream and the second stream represents a separately encoded stream that serves as a supplementary video to the main video.

7. The method according to claim 6, characterized in that, Controlling the client to replace at least a portion of one of the first stream and the second stream includes: determining whether the user requests to watch the supplementary video.

8. The method according to claim 7, characterized in that, The main video and the supplementary video each include a VVC sub-screen.

9. The method according to claim 2, characterized in that, Controlling the client to replace at least a portion of one of the first stream and the second stream includes: merging the attributes of VVC sub-screens such that the decoder is provided with a single merged stream that is merged from both the first stream and the second stream.

10. The method according to claim 2, characterized in that, The acquisition of video data includes acquiring the first stream separately from the second stream.

11. A video transmission device, characterized in that, The device includes: Memory, which stores instructions; and A processor that communicates with the memory, wherein, when the processor executes the instructions, the processor is configured to cause the device to perform the method according to any one of claims 1 to 10.

12. A non-transitory computer-readable medium storing a program that causes a computer to perform the method of any one of claims 1-10.

13. A computer device, comprising a processor and a memory, characterized in that, The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method of any one of claims 1-10 according to the instructions in the program code.

14. A method for storing a bit stream, characterized in that, Generate a bit stream by performing the method of any one of claims 1-10; and store the bit stream.

15. A method for transmitting a bit stream, characterized in that, Generate a bit stream by performing the method of any one of claims 1-10; and transmit the bit stream.

16. A computer-readable storage medium storing a computer program / instructions and a bit stream thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-10 to generate the bit stream.

Images