AV1 codec for real-time video communication

Abstract

Techniques for encapsulating AV1 encoded video data within NAL units are described. For example, the NAL units can be H.264 or HEVC NAL units. Encapsulation can include using a reserved NAL unit type. For example, an open bitstream unit including AV1 encoded video data can be encapsulated within a NAL unit using a reserved NAL unit type. The NAL units can be grouped for delivery to another computing device via a computer network.

Description

Background Art

[0001] AOMedia Video 1 (AV1) is an open, royalty-free video coding format designed for video transmission. Compared to the H.264 codec, it is able to achieve up to 50% higher data compression rates, which allows better quality video to be delivered in limited bandwidth networks and enables 4K or high resolution. However, the AV1 format does not provide specifications for streaming video over a network connection. Summary of the invention

[0002] This Summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Techniques for encapsulating AV1 coded video data within NAL units are described. For example, AV1 coded video data can be encapsulated as OBUs within H.264 or HEVC NAL units.

[0004] For example, AV1 encoded video data can be encapsulated in a NAL unit by receiving an OBU including video content encoded according to the AV1 video coding specification, encapsulating the OBU in a NAL unit using a reserved NAL unit type, grouping the NAL unit into one or more network packets, and transmitting the one or more network packets to another computing device. The NAL unit can be an H.264 or HEVC NAL unit.

[0005] As another example, encapsulated AV1 encoded video data encapsulated in a NAL unit can be received by receiving one or more network packets, depacketizing the one or more network packets to obtain a NAL unit, obtaining a NAL unit type value from a NAL unit header of the NAL unit, and when the NAL unit type value is set to a reserved value indicating that the NAL unit encapsulates an OBU, extracting the OBU (including video content encoded according to the AV1 video coding specification) from the NAL unit and decoding the OBU. The NAL unit may be an H.264 or HEVC NAL unit.

[0006] As described herein, a variety of additional features and advantages can be incorporated into the technology as desired. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a block diagram depicting an exemplary packing of AV1 frames within NAL units.

[0008] Figure 2 is a diagram depicting an exemplary environment for sending and receiving network packets with encapsulated AV1 frames.

[0009] Figure 3 is a diagram depicting an exemplary process for establishing a video communication session, wherein AV1 frames are encapsulated within NAL units.

[0010] Figure 4 is a flow chart of an example method for encapsulating encoded video data within NAL units.

[0011] Figure 5 is a flow chart of an exemplary method for encapsulating AV1 frames within NAL units.

[0012] Figure 6 is a flow chart of an exemplary method for receiving and decoding AV1 frames.

[0013] Figure 7 is a diagram of an exemplary computing system in which some described embodiments can be implemented.

[0014] Figure 8 is an exemplary cloud support environment that can be used in conjunction with the techniques described herein. DETAILED DESCRIPTION

[0015] As described herein, various techniques and solutions can be applied to encapsulate AV1 encoded video data in H.264 or HEVC network abstraction layer (NAL) units. For example, an open bitstream unit (OBU) can be received, wherein the OBU includes video content encoded according to the AV1 video coding specification. The OBU can be encapsulated in a NAL unit, which uses a reserved NAL unit type in the NAL unit header. The NAL unit can be grouped into one or more network packets. The one or more network packets can be sent for reception by a receiver device.

[0016] One or more network packets can be received (e.g., from a transmitter device). The one or more network packets can be depacketized to obtain a NAL unit. The NAL unit can be inspected to obtain a NAL unit type value from a NAL unit header of the NAL unit. The payload of the NAL unit can be decoded according to the obtained NAL unit type value. For example, when the NAL unit type value is set to a reserved value indicating that the NAL unit encapsulates an OBU, the OBU (containing AV1 encoded video data) can be extracted and decoded by an AV1 decoder. When the NAL unit type value is not set to the reserved value, depending on whether the NAL unit is an H.264 NAL unit or an HEVC NAL unit, the NAL unit can be decoded normally using an H.264 or HEVC decoder.

[0017] AOMedia Video 1 (AV1) is a video codec and associated video coding specification provided by the Alliance for Open Media (AOMedia; https: / / aomedia.org).

[0018] NAL units

[0019] In the technology described herein, AV1 coded video data (generally referred to herein as AV1 frames) is encapsulated in NAL units. Specifically, in some implementations, the OBU containing the AV1 coded video data is encapsulated in NAL units. NAL units are defined by H.264 and HEVC standards. NAL units are used to construct coded video data, which is used for transmission through a packet-oriented or bitstream-oriented transmission system. For example, NAL units can be used to organize coded video data, which is used for transmission via a wired or wireless network connection.

[0020] In some implementations, NAL units are packetized for network transmission according to RFC 6184 ("RTP Payload Format for H.264 Video"), which also uses the Real-time Transport Protocol (RTP) defined by RFC 3550 ("RTP: A Transport Protocol for Real-Time Applications"). For example, NAL units can be packetized (e.g., can be segmented and delivered in multiple network packets) based at least in part on a maximum transmission unit (MTU) of the network protocol being used.

[0021] The techniques described herein mainly describe encapsulating AV1 coded video data within H.264 NAL units. However, AV1 coded video data can be encapsulated within HEVC NAL units in the same manner. For example, encapsulating AV1 coded video data within HEVC NAL units can use similar reserved values ​​for NAL unified types.

[0022] The techniques described herein primarily describe encapsulation of AV1 encoded video data. However, encoded video data generated by other video coding standards (in addition to H.264 or HEVC) can also be encapsulated in H.264 or HEVC NAL units. For example, a video bitstream representing video data encoded by other video coding standards can be divided into bitstream units and encapsulated in H.264 or HEVC NAL units using reserved values ​​for the NAL unit type.

[0023] Encapsulation using NAL units

[0024] In the techniques described herein, AV1 frames are encapsulated in NAL units. For example, AV1 frames are encapsulated in NAL units using a reserved type of NAL unit header.

[0025] Figure 1 is a block diagram 100 depicting an exemplary encapsulation of an AV1 frame within a NAL unit. Block diagram 100 depicts a visual representation of a NAL unit 105. For example, NAL unit 105 may be an H.264 NAL unit or a HEVC NAL unit.

[0026] The NAL unit 105 includes a NAL unit header 110. In an implementation using H.264 NAL units, the NAL unit header 110 is a single-byte header that signals three pieces of information. The first piece of information is a 1-bit value set to zero (also referred to as the forbidden_zero_bit syntax element). The second piece of information is the reference index field, which is a 2-bit value indicating the reference picture status of the NAL unit (also referred to as nal_ref_idc). The third piece of information is a 5-bit value that signals the NAL unit type (also referred to as nal-unit_type).

[0027] In an implementation using HEVC NAL units, the NAL unit header 110 is a two-byte header that signals, among other values, a 1-bit value set to zero (also referred to as a forbidden_zero-bit syntax element) and a 6-bit value (also referred to as nal-unit-type) that signals the NAL unit type.

[0028] In the techniques described herein, the NAL unit type value in the NAL unit header 110 is set to a reserved type indicating that the NAL unit encapsulates an AV1 frame. In some implementations, the NAL unit type is set to a reserved value of 22 or 23 (e.g., values ​​22 and 23 for the NAL unit type are specified as reserved in the H.264 standard). In other implementations, the NAL unit type is set to another reserved or unspecified value (e.g., a value not used by the H.264 or HEVC standards). For example, for HEVC NAL units, the NAL unit type can be set to a reserved value of 41 to 47.

[0029] The video encoding and decoding processes can be configured to recognize the reserved NAL unit type (e.g., for H.264 NAL units, the NAL unit type with a value of 22) and change the behavior accordingly. For example, on the video encoding side, the software and / or hardware performing the video encoding can be pre-configured to use a specific reserved NAL unit type to indicate that the AV1 encoded video data is encapsulated within the NAL unit. On the video decoding side, the software and / or hardware performing the video decoding can be pre-configured to use the same specific reserved NAL unit type to indicate that the AV1 encoded video data is encapsulated within the NAL unit.

[0030] The NAL unit 105 also includes an open bitstream unit (OBU) 115. In the AV1 specification, the encoded video bitstream is grouped into OBUs. The OBU 115 includes an OBU header 120 and OBU data 125. The OBU header 120 identifies the type of the OBU 115. Table 1 depicts exemplary OBU types for the current version of the AV1 specification.

[0031] OBU Type name 0 reserve 1 QBU_SEQUENCE_HEADER 2 OBU_TEMPORAL_DELIMITER 3 OBU_FRAME_HEADER 4 OBU_TILE-GROUP 5 OBU_METADATA 6 OBU_FRAME 7 OBU_REDUNDANT_FRAME_HEADER 8 OBU_TILE_LIST 9-14 reserve 15 OBU_PADDING

[0032] Table 1

[0033] The OBU data 125 contains the payload of the OBU 115. For example, the OBU data 125 can contain data for an AV1 encoded video frame (e.g., having an OBU type of 6). In general, the OBU can contain any type of AV1 encoded video data. For example, a real-time video communication session can be performed by encapsulating the OBU containing the AV1 encoded video bitstream within a NAL unit for transmission to another computing device.

[0034] NAL Reference Index Field

[0035] In some implementations, encapsulating an AV1 frame within a NAL unit (e.g., within an H.264 NAL unit) includes setting the reference index field in the NAL unit header. Specifically, the 2-bit reference index field (e.g., nal_ref_idc) is set to one of the following four values ​​described in Table 2 below. A value of 0 (AV1_NRI_NON_REF) indicates that the frame is a non-reference frame. A value of 1 (AV1_NRI_REG_PFRAME) indicates that the frame is a regular predicted frame (P frame) that can be used as a short-term reference frame. A value of 2 (AV1_NRI_LTR_PFRAME) indicates that the frame is a long-term P frame that can be used as a long-term reference frame. A value of 3 (AV1_NRI_KEY_FRAME) indicates that the frame is a key frame, which is an intra frame that resets the decoding process when shown.

[0036] value Example Name 0 AV1_NRI_NON_REF 1 AV1_NRI_REF_PFRAME 2 AV1_NRI_LTR_PFRAME 3 AV1_NRI_KEY_FRAME

[0037] Table 2

[0038] The reference index field values ​​of Table 2 have different meanings from those used by the H.264 standard. Specifically, the meanings of values ​​1 to 3 in Table 2 are not defined in the H.264 standard.

[0039] In some implementations, the Reference Index field values ​​of Table 2 are used for loss recovery. For example, the Reference Index field values ​​can be used to perform loss recovery of AV1 frames without decoding or using information within the AV1 OBU itself. For example, the Reference Index field values ​​can be used when implementing loss recovery according to U.S. Patent No. 10,313,685.

[0040] In some implementations, encapsulating the AV1 frame within a NAL unit (e.g., within an H.264 or HEVC NAL unit) includes including a separate syntax element (e.g., replacing the 2-bit reference index field). For example, the separate syntax element can be included after the NAL unit header (e.g., after the NAL unit header 110) and before the OBU (e.g., before the OBU 115). The separate syntax element is not part of the H.264 or HEVC standard. The separate syntax element can use the same value of Table 2, or it can use a different value (e.g., it can be a value of more than 2 bits and can therefore indicate more than four values).

[0041] Environment for applying encapsulated AV1 frames

[0042] Figure 2is a diagram depicting an exemplary environment 200 for sending and receiving network packets containing AV1 frames encapsulated within NAL units. For example, the NAL units may be H.264 NAL units or HEVC NAL units.

[0043] Exemplary environment 200 depicts a sender device 210. Sender device 210 can be any type of computing device (e.g., a server computer, virtual machine, desktop computer, laptop computer, tablet computer, phone, or other type of computing device) that encapsulates OBUs in NAL units for transmission in network packets.

[0044] The transmitter device 210 performs a number of operations involving encapsulating the OBU within a NAL unit. As depicted at 212, the transmitter device 210 receives an OBU containing AV1 encoded video data. For example, the transmitter device 210 can receive the OBU from an AV1 encoder (e.g., running on the transmitter device 210) that receives video content (e.g., from a video camera of the transmitter device 210) and encodes the video content using the AV1 encoder to generate the OBU.

[0045] At 214, the transmitter device 210 encapsulates the OBU within a NAL unit using a reserved NAL unit type (eg, by setting the NAL unit type in the NAL unit header to a value of 22 or another reserved or unused value).

[0046] At 216, the sender device 210 groups the NAL units into network packets. At 218, the sender device 210 transmits the network packets to a receiver device 220 via a network 230 (e.g., the Internet and / or another type of public or private data network). The receiver device 220 can be any type of computing device (e.g., a server computer, a virtual machine, a desktop computer, a laptop computer, a tablet computer, a phone, or other type of computing device) that receives and decodes the OBUs encapsulated within the NAL units.

[0047] At 222, the receiver device 220 receives and depacketizes the network packet from the sender device 210. At 224, the receiver device 220 obtains the NAL unit from the network packet.

[0048] At 226, the receiver device 220 extracts and decodes the encapsulated OBU. For example, the receiver device 220 can identify that the NAL unit contains an OBU (e.g., rather than H.264 or HEVC encoded video content) by inspecting the NAL unit type of the NAL header and determining that it is a reserved value indicating that the NAL unit contains an OBU. The receiver device 220 can provide the OBU to an AV1 decoder (e.g., for decoding and display at the receiver device 220).

[0049] For example, as part of a real-time communication session (e.g., a real-time video call), the transmitter device 210 can perform operations for each of the encapsulated multiple OBUs. Similarly, as part of a real-time communication session, the receiver device 220 can perform operations for each of the multiple OBUs it receives and decodes.

[0050] For ease of illustration, the two computing devices depicted in the exemplary environment 200 are labeled as a transmitter device 210 and a receiver device 220. However, any one (or both) of the devices can be a "transmitter" that encapsulates an OBU and sends it to another device. Similarly, any one (or both) of the devices can be a "receiver" that receives the encapsulated OBU. In addition, any one (or both) of the devices can communicate with additional transmitter and / or receiver devices (not depicted).

[0051] SDP Negotiation

[0052] In some implementations, AV1 encoded video data (e.g., in OBU format) is encapsulated within NAL units as part of a streaming video session (e.g., as part of a real-time video communication session) using a Session Description Protocol (SDP). For example, the streaming video session may be Skype or Teams communication session.

[0053] In some implementations, a new media line is used to support the AV1 codec in the SDP negotiation between the sender and the receiver. The new media line is:

[0054] a=rtpmap:119X-MSAV1 / 90000

[0055] a=rtpmap:100rtx / 90000

[0056] Even if both clients (e.g., a sender and a receiver) support AV1, the client still chooses to use H.264 or another codec. For example, H.264 may be selected based on available computing resources (e.g., an H.264 encoder may use fewer computing resources than an AV1 encoder).

[0057] VSR Message

[0058] In some implementations, AV1 encoded video data (e.g., in the format of an OBU) is encapsulated within NAL units as part of a streaming video session (e.g., as part of a real-time video communication session) using a Video Source Request (VSR) message defined by the Microsoft Real-time Transport Protocol (RTP) extension specification. For example, the streaming video session may be Skype or Teams communication session.

[0059] In some implementations, VSR messages are exchanged between the sender and the receiver after performing SDP negotiation.

[0060] Once the SDP negotiation is complete, the receiver generates a VSR with an AV1 entry added to indicate receiver side preferences and capabilities. In some implementations, the following entry specifies details for the AV1 VSR:

[0061] Payload Type (1 byte): The encoding type of the requested video (e.g., use 119 for AV1).

[0062] Flags (1 byte): Video flags. Reserved. The sender should set this to 0. The receiver must ignore it.

[0063] Required Instance Number (2 bytes): This entry MUST be 0 when both AV1 and H.264 are supported.

[0064] Possible Instance Number (2 bytes): When both AV1 and H.264 are supported, this entry contains positive values ​​for both H.264 and AV1.

[0065] The remainder of the VSR field is defined according to the Microsoft Real-time Transport Protocol (RTP) extension specification.

[0066] Figure 3 is a diagram depicting an exemplary process 300 for establishing a video communication session using the techniques described herein. In the exemplary process 300, a first computing device 310 communicates with a second computing device 320 over a network. For example, the first computing device 310 can be a sender device (e.g., sender device 210) and the second computing device 320 can be a receiver device (e.g., receiver device 220).

[0067] To establish the video communication session, as part of the SDP negotiation phase 330, the first computing device 310 and the second computing device 320 exchange SDP messages. For example, the first computing device 310 can send an SDP message to the second computing device 320 that includes information describing codecs supported by the first computing device 310 (e.g., indicating that the first computing device 310 supports the MSAV1 codec or otherwise indicates that the first computing device 310 supports AV1 data encapsulated within NAL units). In response to the SDP message from the first computing device 310, the second computing device 320 can send an SDP message to the first computing device 310 that includes information describing codecs supported by the second computing device 320 (e.g., indicating that the second computing device 320 supports the MSAV1 codec or otherwise indicates that the second computing device 320 supports AV1 data encapsulated within NAL units). After the SDP messages have been exchanged, the first computing device 310 and the second computing device 320 can determine which codec(s) to use. For example, if both computing devices support encapsulation of AV1 encoded video data within NAL units (e.g., both support the MSAV1 codec), then both computing devices can configure themselves to send and receive streaming video content using H.264 RTP encapsulating AV1 (e.g., using a predetermined reserved NAL unit type for the NAL units, such as value 22).

[0068] After the SDP negotiation phase 330, the first computing device 310 and the second computing device 320 exchange VSR messages as part of the VSR negotiation phase 340. The VSR messages can be exchanged in one or both directions. For example, the VSR messages can be used to inform other devices of video parameters that are supported and / or being used (e.g., supported resolutions, profiles and bit rates, codecs being used, etc.). For example, the second computing device 320 can send a VSR message to the first computing device 310 indicating that the second computing device 320 will encapsulate AV1 encoded video data (e.g., using the MSAV1 codec) within a NAL unit. In some implementations, the VSR payload type field is used to specify AV1 encoded video data (e.g., the MSAV1 codec). After the VSR negotiation phase 340, the video communication session will be configured between the first computing device 310 and the second computing device 320.

[0069] After the SDP negotiation phase 330 and the VSR negotiation phase 340 have completed, the first computing device 310 and the second computing device 320 are able to perform video communications (e.g., as part of a real-time video communications session) using the encapsulated AV1 encoded video data, as depicted at 350.

[0070] Method for encapsulating AV1 encoded video data within NAL units

[0071] In any examples herein, a method for encapsulating encoded video data (e.g., AV1 encoded video data) within a NAL unit can be provided. For example, the method can be performed by any number of computing devices (e.g., computing devices that send and / or receive the encoded video data). For example, two or more computing devices can participate in a video communication session (e.g., directly and / or through a server or an intermediate computing device).

[0072] Figure 4 is a flow chart of an exemplary method 400 for encapsulating encoded video data within a NAL unit. The exemplary method 400 can be performed by a computing device, such as the transmitter device 210 (eg, via software and / or hardware resources of the computing device), for example.

[0073] At 410, a bitstream unit is received. The bitstream unit includes video content encoded according to a video coding specification (e.g., the AV1 video coding specification or another video coding specification that is not H.264 or HEVC). For example, the bitstream unit can be received from a video encoding process (e.g., using an AV1 encoder) that receives video data (e.g., from a video camera or other video source), encodes the received video data according to the video coding specification, and generates a bitstream unit containing the encoded video data.

[0074] At 420, the bitstream unit is encapsulated within a NAL unit using a reserved NAL unit type. The reserved NAL unit type uses a value that has been defined as a reserved value (e.g., defined as reserved by the H.264 or HEVC standard). For example, a predetermined reserved value (e.g., value 22 or 23 for H.264 NAL units) can be used.

[0075] At 430, the NAL units are grouped into one or more network packets.

[0076] At 440, the one or more network packets are transmitted for receipt by a receiver computing device. For example, the one or more network packets can be transmitted directly to the receiver computing device. The one or more network packets can also be transmitted to another device (e.g., a server or a centralized communication service) for delivery to the receiver computing device. The receiver computing device can receive and decode the bitstream unit. For example, exemplary method 400 can be performed to encapsulate and transmit multiple bitstream units as part of a real-time video communication session.

[0077] In some implementations, multiple video codecs are supported for encapsulation within an H.264 or HEVC NAL unit. The multiple video codecs are different from H.264 or HEVC. For example, a first reserved NAL unit type value (e.g., value 22) can be used to encapsulate a first video codec, a second reserved NAL unit type value (e.g., value 23) can be used to encapsulate a second video codec (different from the first video codec), and so on.

[0078] Figure 5 is a flow chart of an exemplary method 500 for encapsulating AV1 encoded video data within a NAL unit. For example, the exemplary method 500 can be performed by a computing device (eg, via software and / or hardware resources of the computing device), such as the transmitter device 210.

[0079] At 510, an OBU is received. The OBU includes video content encoded according to the AV1 video coding specification. For example, the OBU can be received from a video encoding process (e.g., using an AV1 encoder) that receives video data (e.g., from a video camera or other video source), encodes the received video data according to the AV1 video coding specification, and generates an OBU containing the AV1 encoded video data.

[0080] At 520, the OBU is encapsulated within a NAL unit using a reserved NAL unit type. The reserved NAL unit type uses a value that has been defined as a reserved value (e.g., defined as reserved by the H.264 or HEVC standard). For example, a predetermined reserved value (e.g., value 22 for H.264 NAL units) can be used.

[0081] At 530, the NAL units are grouped into one or more network packets.

[0082] At 540, the one or more network packets are transmitted for receipt by a receiver computing device. For example, the one or more network packets can be transmitted directly to the receiver computing device. The one or more network packets can also be transmitted to another device (e.g., a server or centralized communication service) for delivery to the receiver computing device. The receiver computing device can receive and decode the OBU.

[0083] For example, exemplary method 500 can be performed to encapsulate and transmit multiple OBUs as part of a real-time video communication session.

[0084] Figure 66 is a flow chart of an exemplary method 600 for receiving encapsulated AV1 encoded video data within a NAL unit. For example, the exemplary method 600 can be performed by a computing device (eg, via software and / or hardware resources of the computing device), such as the receiver device 220.

[0085] At 610, one or more network packets are received. The one or more network packets contain NAL units.

[0086] At 620, the one or more network packets are depacketized to obtain the NAL units.

[0087] At 630, a NAL unit type is obtained from a NAL unit header of the NAL unit.

[0088] At 640, when the NAL unit type value is set to a reserved value indicating that the NAL unit encapsulates an OBU (i.e., the NAL unit encapsulates AV1 encoded video data), the OBU is extracted from the NAL unit and decoded. For example, the OBU is provided to an AV1 decoder for decoding and playback. When the NAL unit type value is set to a value different from the reserved value, the NAL unit can be decoded using a different codec. For example, if the NAL unit is an H.264 NAL unit whose NAL unit type is not 22 or 23 (e.g., a NAL unit type value of 1-21), the NAL unit can be decoded using an H.264 decoder.

[0089] Computing System

[0090] Figure 7 A generalized example of a suitable computing system 700 is depicted in which the described techniques may be implemented. Computing system 700 is not intended to suggest any limitation as to scope of use or functionality, as the described techniques may be implemented in various general-purpose or special-purpose computing systems.

[0091] refer to Figure 7 , the computing system 700 includes one or more processing units 710, 715 and memories 720, 725. Figure 7 , the basic configuration 730 is contained within the dashed line. Processing units 710, 715 run computer executable instructions. The processing unit can be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. The processing unit can also include multiple processors. In a multi-processing system, multiple processing units run computer executable instructions to increase processing power. For example, Figure 7A central processing unit 710 and a graphics processing unit or co-processing unit 715 are shown. Tangible memory 720, 725 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two, which is accessible by the processing unit(s). The memory 720, 725 stores software 780 implementing one or more techniques described herein in the form of computer-executable instructions suitable for execution by the processing unit(s).

[0092] The computing system may have additional features. For example, the computing system 700 includes storage 740, one or more input devices 750, one or more output devices 760, and one or more communication connections 770. An interconnection mechanism (not shown) (such as a bus, controller, or network) interconnects the components of the computing system 700. Typically, operating system software (not shown) provides an operating environment for other software running in the computing system 700 and coordinates the activities of the components of the computing system 700.

[0093] Tangible storage 740 may be removable or non-removable and include disks, magnetic tapes or cartridges, CD-ROMs, DVDs, or any other medium that can be used to store information and accessed within computing system 700. Storage 740 stores instructions for software 780 implementing one or more techniques described herein.

[0094] Input device(s) 750 may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing system 700. For video encoding, input device(s) 750 may be a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video samples into the computing system 700. Output device(s) 760 may be a display, printer, speaker, CD burner, or another device that provides output from the computing system 700.

[0095] Communication connection(s) 770 enable communication with another computing entity via a communication medium. The communication medium transmits information, such as computer executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that sets or changes one or more of its characteristics in a manner that encodes the information in the signal. By way of example and not limitation, the communication medium can use electrical, optical, RF or other carriers.

[0096] The techniques can be described in the general context of computer executable instructions, such as those included in program modules, which are executed in a computing system on a target real or virtual processor. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or split between program modules as needed. Computer executable instructions for program modules can be executed within a local or distributed computing system.

[0097] The terms "system" and "device" are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. In general, a computing system or computing device can be local or distributed, and can include any combination of special-purpose hardware and / or general-purpose hardware and software that implements the functionality described herein.

[0098] For purposes of presentation, the detailed description uses terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer and should not be confused with actions performed by a human. The actual computer operations corresponding to these terms vary from implementation to implementation.

[0099] Cloud-powered environments

[0100] Figure 8 A generalized example of a suitable cloud support environment 800 in which the described embodiments, techniques, and technologies can be implemented is illustrated. In the exemplary environment 800, various types of services (e.g., computing services) are provided by a cloud 810. For example, the cloud 810 can include a collection of computing devices, which can be centrally located or distributed, that provide cloud-based services to various types of users and devices connected via a network such as the Internet. The implementation environment 800 can be used in different ways to complete computing tasks. For example, some tasks (e.g., processing user input and presenting a user interface) can be performed on a local computing device (e.g., connected devices 830, 840, 850), while other tasks (e.g., storing data to be used in subsequent processing) can be performed in the cloud 810.

[0101] In the exemplary environment 800, the cloud 810 provides services for connected devices 830, 840, and 850 with various screen capabilities. The connected device 830 represents a device with a computer screen 835 (e.g., a medium-sized screen). For example, the connected device 830 can be a personal computer, such as a desktop computer, a laptop computer, a notebook, a netbook, etc. The connected device 840 represents a device with a mobile device screen 845 (e.g., a small-sized screen). For example, the connected device 840 can be a mobile phone, a smart phone, a personal digital assistant, a tablet computer, etc. The connected device 850 represents a device with a large screen 855. For example, the connected device 850 can be a TV screen (e.g., a smart TV) or another device connected to the TV (e.g., a set-top box or a game console), etc. One or more of the connected devices 830, 840, and 850 can include touch screen capabilities. The touch screen can accept input in different ways. For example, a capacitive touch screen detects touch input when an object (e.g., a fingertip or a stylus) distorts or interrupts the current running across the surface. As another example, a touch screen can use an optical sensor to detect touch input when a light beam from the optical sensor is interrupted. Some touch screens can detect input without physical contact with the screen surface. Devices without screen capabilities can also be used in exemplary environment 800. For example, cloud 810 can provide services for one or more computers (e.g., server computers) without a display.

[0102] Services can be provided by cloud 810 through service provider 820 or through other providers of online services (not depicted). For example, cloud services can be customized for the screen size, display capabilities, and / or touch screen capabilities of specific connected devices (e.g., connected devices 830, 840, 850).

[0103] In exemplary environment 800, cloud 810 provides the techniques and solutions described herein to various connected devices 830, 840, 850, at least in part using service provider 820. For example, service provider 820 can provide a centralized solution for various cloud-based services. Service provider 820 can manage service subscriptions for users and / or devices (e.g., for connected devices 830, 840, 850 and / or their respective users).

[0104] Exemplary Implementations

[0105] Although the operations of some of the disclosed methods are described in a particular sequential order for ease of presentation, it should be understood that such description includes rearrangement unless the specific language set forth below requires a particular order. For example, the operations described in sequence may be rearranged or performed simultaneously in some cases. In addition, for simplicity, the accompanying drawings may not illustrate the various ways in which the disclosed methods may be used in conjunction with other methods.

[0106] Any of the disclosed methods can be implemented as computer-executable instructions or computer program products, stored on one or more computer-readable storage media and run on a computing device (i.e., any available computing device, including a smartphone or other mobile device that includes computing hardware). A computer-readable storage medium is a tangible medium (one or more optical disks, such as DVDs or CDs, volatile memory (such as DRAM or SRAM), or non-volatile memory (such as flash memory or a hard drive)) that can be accessed within a computing environment. By way of example and with reference to Figure 7 , computer-readable storage media include memories 720 and 725 and storage device 740. The term "computer-readable storage medium" does not include signals and carrier waves. In addition, the term "computer-readable storage medium" does not include communication connections, such as 770.

[0107] Any computer executable instructions for implementing the disclosed technology and any data created and used during the implementation of the disclosed embodiments can be stored on one or more computer-readable storage media. The computer executable instructions can be, for example, a dedicated software application or a part of a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software can be, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network) or other such networks) using one or more network computers to run.

[0108] For the sake of clarity, only certain selected aspects of software-based implementations are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any specific computer language or program. For example, the disclosed technology can be implemented by software written in C++, Java, Perl, or any other suitable programming language. Similarly, the disclosed technology is not limited to any specific computer or hardware type. The specific details of suitable computers and hardware are known and do not need to be elaborated in this disclosure.

[0109] In addition, any software-based embodiments (e.g., including computer-executable instructions for causing a computer to perform any disclosed method) can be uploaded, downloaded, or remotely accessed via a suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communications means.

[0110] The disclosed methods, apparatuses, and systems should not be construed as limiting in any way. Instead, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, alone and in various combinations and sub-combinations with each other. The disclosed methods, apparatuses, and systems are not limited to any particular aspect or feature or combination thereof, nor do the disclosed embodiments require the presence of any one or more specific advantages or problems to be solved.

[0111] Techniques from any example may be combined with techniques described in any one or more other examples. In view of the many possible embodiments to which the principles of the disclosed technology may be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology.

Claims

1. A computing device, comprising: processor; Network interface; as well as Memory; The computing device is configured to perform operations for encapsulating frames of encoded video data within one or more Network Abstraction Layer (NAL) units defined in a video coding standard, the operations comprising: Receiving a bitstream unit including video content encoded according to a video coding specification, wherein the bitstream unit is an open bitstream unit (OBU) that has been encoded according to the video coding specification, and the video coding specification is the AV1 video coding specification; Encapsulating the OBU encoded according to the AV1 video coding specification within a NAL unit, wherein the OBU is encapsulated in a first field of a NAL unit header of the NAL unit using a reserved NAL unit type, wherein encapsulating the OBU within the NAL unit comprises setting a second field of the NAL unit header to a value indicating a reference frame status of the OBU, and wherein the second field supports at least the following values: a first value indicating that the frame is a non-reference frame; a second value indicating that the frame is a conventional predicted frame P frame that can be used as a short-term reference frame; a third value indicating that the frame is a long-term P frame that can be used as a long-term reference frame; and a fourth value indicating that the frame is a key frame; grouping the NAL units into one or more network packets; and The one or more network packets are transmitted for receipt by a receiver computing device.

2. The computing device of claim 1, wherein: Encapsulating the OBU in the NAL unit comprises: determining a reserved value for the reserved NAL unit type; and The first field of the NAL unit header is set to the reserved value.

3. The computing device of claim 1, wherein: The reserved NAL unit type uses the reserved value 22 or 23.

4. The computing device of claim 1, wherein: The OBU includes: OBU header; and OBU data.

5. The computing device of claim 1, wherein: The computing device performs the operations to package and transmit a plurality of OBUs for receipt by the receiver computing device as part of a real-time communication session.

6. The computing device of claim 1 , further comprising: receiving a second bitstream unit, the second bitstream unit comprising video content encoded according to a second video coding specification different from the AV1 video coding specification; encapsulating the second bitstream unit in a second NAL unit, wherein the second bitstream unit is encapsulated using a second reserved NAL unit type different from the reserved NAL unit type; grouping the second NAL unit into one or more network packets; as well as The one or more network packets containing the second NAL unit are transmitted for receipt by the receiver computing device.

7. A method implemented by a computing device for encapsulating AV1 encoded video data within one or more Network Abstraction Layer (NAL) units defined in a video coding standard, the method comprising: Receiving an open bitstream unit (OBU), wherein the OBU includes frames of video content encoded according to a video coding specification different from the video coding standard, the video coding specification being the AV1 video coding specification; Encapsulating the OBU encoded according to the AV1 video coding specification in a NAL unit, including: determining a reserved value corresponding to a reserved NAL unit type; and Setting a first field of a NAL unit header of the NAL unit to the reserved value; Setting the second field of the NAL unit header to a value indicating a reference frame status of the OBU; The second field supports at least the following values: a first value indicating that the frame is a non-reference frame; a second value indicating that the frame is a conventional predicted frame P frame that can be used as a short-term reference frame; a third value indicating that the frame is a long-term P frame that can be used as a long-term reference frame; and a fourth value indicating that the frame is a key frame; Grouping the NAL units into one or more network packets; and The one or more network packets are transmitted for receipt by a second computing device.

8. A method implemented by a computing device for receiving encapsulated AV1 encoded video data within one or more network abstraction layer (NAL) units, the method comprising: receiving one or more network packets; Depacketizing the one or more network packets to obtain NAL units defined in a video coding standard; Obtain a NAL unit type value from a first field of a NAL unit header of the NAL unit; as well as When the NAL unit type value is set to a reserved value, it indicates that the NAL unit encapsulates an open bitstream unit (OBU) encoded according to the AV1 video coding specification: Extracting the OBU from the NAL unit, wherein the OBU includes frames of video content encoded according to a video coding specification different from the video coding standard, the video coding specification being an AV1 video coding specification; Extract a value indicating a reference frame status of the OBU from a second field of the NAL unit header, wherein the second field supports at least the following values: a first value indicating that the frame is a non-reference frame; a second value indicating that the frame is a conventional predicted frame P frame that can be used as a short-term reference frame; a third value indicating that the frame is a long-term P frame that can be used as a long-term reference frame; and a fourth value indicating that the frame is a key frame; and The OBU is decoded.

Images