Bitstream Structure for Immersive Teleconferencing and Telepresence for Remote Terminals

Abstract

To provide a method, a computer program, and a computer system for coding a viewport bitstream.SOLUTION: A first coded video bitstream having a segment size defined for a viewport is encoded on the basis of a threshold corresponding to a user's head movement. A second coded video bitstream having a lower resolution is encoded. The second coded video bitstream may correspond to a background having a certain segment duration or size. The first coded video bitstream and the second coded bitstream are used to create a streaming bitstream for decoding or rendering.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority based on U.S. Provisional Patent Application No. 63 / 111,425, filed on November 9, 2020, and U.S. Patent Application No. 17 / 362,068, filed on June 29, 2021, the entire contents of which are incorporated herein by reference.

[0002] This disclosure generally relates to the field of data processing, and more particularly to video streaming.

Background Art

[0003] Immersive video streaming involves the transmission of a "world" or "omnidirectional" view from a transmitter to a receiver, and the receiver will render only a portion of the received worldview, for example, based on a viewport. The viewport can be selected based on the direction of head movement when wearing virtual reality goggles. Viewport - dependent video streaming can relate to a technique in which only a portion of the view is transmitted and rendered to the user based on the viewport selected by the user from a scene recorded covering the "world" view.

Summary of the Invention

Means for Solving the Problems

[0004] Embodiments relate to a method, system, and computer-readable medium for splitting a viewport bitstream. According to one aspect, a method for splitting a viewport bitstream is provided. The method may include encoding a first coded video bitstream having a segment size defined for a viewport based on a threshold corresponding to a user's head movement. A second coded video bitstream having a low resolution is encoded. The second coded video bitstream may correspond to a background having a constant segment duration or size. Using the first coded video bitstream and the second coded bitstream, a streaming bitstream for decoding or rendering is created.

[0005] According to another aspect, a computer system for splitting a viewport bitstream is provided. The computer system may include one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored in at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, whereby the computer system is capable of executing the method. The method may include encoding a first coded video bitstream having a segment size defined for a viewport based on a threshold corresponding to a user's head movement. A second coded video bitstream having a low resolution is encoded. The second coded video bitstream may correspond to a background having a constant segment duration or size. Using the first coded video bitstream and the second coded bitstream, a streaming bitstream for decoding or rendering is created.

[0006] According to yet another aspect, a computer-readable medium for splitting a viewport bitstream is provided. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored in at least one of the one or more tangible storage devices, the program instructions being executable by a processor. The program instructions are executable by the processor to perform a method that may optionally include encoding a first coded video bitstream having a segment size defined for a viewport based on a threshold corresponding to movement of a user's head. A second coded video bitstream having a low resolution is encoded. The second coded video bitstream may correspond to a background having a constant segment duration or size. A streaming bitstream for decoding or rendering is created using the first coded video bitstream and the second coded bitstream.

[0007] These and other objects, features, and advantages will become apparent from the following detailed description of exemplary embodiments, which should be read in conjunction with the accompanying drawings. The figures are provided to clarify for ease of understanding by those skilled in the art in conjunction with the detailed description, and various features of the drawings are not to scale.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

[0009] Although detailed embodiments of the claimed structures and methods are disclosed herein, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that may be embodied in various forms. However, these structures and methods may be embodied in many different forms and should not be construed as limited to the exemplary embodiments described herein. On the contrary, these exemplary embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope to those skilled in the art. In this description, well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0010] Embodiments generally relate to the field of data processing, and more particularly to video streaming. The exemplary embodiments described below provide, among other things, a system, method, and computer program for viewport-based video streaming. Thus, some embodiments have the ability to improve the field of computing by enabling the splitting of viewport bitstreams into smaller fragmented files or DASH segments and providing fallback bitstream fragments or segments to support fast bitstream random access in playback.

[0011] As described above, immersive video streaming involves the transmission of a "world" or "omnidirectional" view from a transmitter to a receiver, and the receiver will render only a portion of the received worldview, for example, based on a viewport. The viewport can be selected based on the direction of head movement when wearing virtual reality goggles. Viewport-dependent video streaming can relate to a technique in which only a portion of the view is transmitted and rendered to the user based on the viewport selected by the user from a scene recorded covering the "world" view.

[0012] However, as the speed at which the user moves his / her head increases, the demand for new tiles also increases accordingly, thereby increasing the M2HQ latency. Additionally, much of the data downloaded is not viewed by the user and is thus wasted. Therefore, it can be advantageous to reduce the latency experienced when the viewport is changed in immersive video by reconfiguring the bitstream.

[0013] In this specification, aspects are described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0014] The exemplary embodiments described below relate to viewport-based video streaming, and more particularly, to a viewport bitstream structure for immersive teleconferencing and telepresence for remote terminals when a user's viewport changes by using variable bitstream segment sizes, and provide a system, method, and computer program.

[0015] FIG. 1 shows a block diagram of an ecosystem 100 for streaming immersive video. 360 video (101) passes through an encoder (102), and the entire picture is streamed via a content delivery network (CDN) (103). The user decodes and reconstructs the received content (104) with a VR player (105). Since the user's field of view (FoV) (106) at any given time only covers a certain degree of inclusion, the problem of transmitting the entire picture is a waste of bandwidth and rendering resources. The bitstream outside the user's current FoV cannot be viewed, yet is received and rendered by the client. Viewport selection can be based on, for example, the direction of head movement when wearing virtual reality goggles. Viewport-dependent video streaming can relate to a technique in which only a portion of the view is transmitted and rendered to the user based on the viewport selected by the user from a scene recorded covering the "world" view, thereby eliminating the need to transmit the entire world view.

[0016] Using tile (or sub - picture) - based immersive video streaming technology can reduce bandwidth requirements and improve video quality for video playback. Referring to Figure 2, a block diagram of a 360 - viewport - dependent video streaming system 200 is depicted. After the video scene is properly stitched together and projected onto a planar video, for example, using an orthographic cylindrical or cube - map projection, the 360 - video scene (201) is encoded by using a planar video encoder (202).

[0017] To use a DASH - coded video bitstream, the sequence can be put into a file having a format that splits the bitstream into smaller HTTP - based video bitstream segments. Those video files can have different bitrates and durations. They can be transmitted over an IP network and can be independently decodable on the client side.

[0018] The encoded immersive video can be fragmented by a DASH packager (203) as described above. The fragmented content can be stored on a content delivery server (not depicted) and transmitted by a CDN (204) to a compatible player (205) for rendering (206). Here, the transmitted video (209) consists only of the high - resolution FoV (208) instead of sending the whole picture (207). On the receiver side, the 4K picture is decoded and reconstructed by the player.

[0019] Tile - based streaming spatially divides 360 - video frames into tiles or blocks. Here, panoramic videos are encoded and then split into tiles after compression. The user then requests only the tiles that fully or partially fall within the user's field of view (FOV). By dividing a large immersive video bit - stream picture into smaller fragments or tiles and transmitting the fragments or tiles that fall within the user's FOV, network and rendering - side resources can be saved.

[0020] Large immersive content (after projection, i.e., planar video stream) can be spatially subdivided into tiles of, for example, the same resolution. For example, a source picture of a 4k×2k video sequence can be divided into tiles of the same size of 512×156 samples, resulting in 64 tiles. Each tile can be encoded and packaged at different bit - rates and quality levels (as is common with DASH) and can be requested at a different quality than its adjacent tiles. Tiles within the user's viewport can be prioritized and streamed more favorably at a higher quality than tiles outside the viewport. In some cases, a particular tile can be completely omitted from transmission. As a fallback, for example, a special layer with a lower resolution / quality / bit - rate that covers the entire panorama can be used. Assuming appropriate player design, doing so can prevent visual artifacts, such as black areas, when the FoV changes but new tiles are not immediately available due to network / streaming - server latency.

[0021] The resolution of the tiles can be changed when the user moves his / her head, but only, for example, at a Random Access Point (RAP). The RAP can be an access unit where the receiver can start to correctly decode the tile or video. Picture frames can be grouped together with different GOP (Group of Pictures) sizes. A P frame, which can contain a coded representation of the changes of the previous frames, can follow an I frame. Thus, a P frame depends on the I frame and the previous P frames. The GOP structure is used in a typical encoder that makes each I frame a random access point so that decoding can start at the I frame. Thus, the response time required for the tiles to be changed depends on the tile granularity and the RAP distance. When the user's orientation changes, the tiles in the current viewport may need to be replaced (at least partially) by different tiles. These new tiles can only be switched at the next available RAP, resulting in a delay in the response to the user input.

[0022] Referring now to FIG. 3, a block diagram 300 of a fragmented immersive video bitstream having an embodiment of a frame boundary is depicted. For example, 8k (301 - 303) and 2k (304 - 306) resolution video bitstreams can be included within the representation of the same projection scene. The low resolution (304 - 306) can be used as a fallback when changing the FoV and can be streamed continuously. The high resolution (301 - 303) can be a tiled representation and can be used for viewport-independent streaming as described above. Both streams can have an equal number of frames and can be divided into equal fixed frame fragments, which can consist of a 30-frame random access point period. Only specific tiles with 8K resolution are delivered as needed to cover the FoV, while the entire frame with 2K resolution is delivered as a fallback. The RAP can occur once per second for each bitstream, assuming a fixed 30fps frame rate.

[0023] Such a configuration can enable a visually comfortable high-speed response by rendering using a reconfigured fallback bitstream when the FoV changes, but there is a problem that, since high-resolution tiles may not be rendered due to a change in the FoV, on average 15 frames of useless high-resolution tiles are sent for each FoV change.

[0024] Referring to FIG. 4, a block diagram 400 of viewport updates during immersive bitstream playback having a frame boundary embodiment is depicted. Block diagram 400 illustrates emulating an FoV update while playback is being performed. When the viewport changes to 402, since the I-frame (401) is located at the beginning of the frame, even if the viewport is at the center of the frame, in addition to 404 and 405, the entire 403 frame is downloaded and decoded. If the current network bandwidth is not very ideal, downloading a new fragment with 8K resolution may cause additional delay. A way to overcome this latency is to have a reduced bitstream segment size for the viewport.

[0025] Referring now to FIG. 5, a block diagram 500 of 8K and 2k immersive bitstream fragmentation having a frame boundary embodiment is depicted. The random access point period can be shortened to, for example, 10 frames (501 - 508) for high-resolution segments within the viewport. The first picture is encoded as an I-frame and the rest as P-frames, and for low-resolution fallback, either only the first picture is encoded as an I-frame and subsequent pictures as P-frames, or a fast random access with an unequal random access period should be defined.

[0026] When the random access period is small, when the viewport changes, the client does not have to download all 30 frames, but only 10 frames. Therefore, the random access point period is shortened, and more random access points will be available in the future, so the delay between the request for a new viewport and the rendering of the new viewport is reduced. A fallback bitstream 509 with a lower bitstream resolution can be provided for less than ideal network situations. Of course, this efficiency improvement is achieved by the coding overhead of additional random access pictures.

[0027] As described above, downloading an intra-frame coded picture may require more network bandwidth than necessary for the user's FoV playback and rendering.

[0028] When streaming high-resolution video, segmentation can cause delays in bitstream download and rendering when network resources are not ideal. Each segment consists of one or more encoded frames. Changes in the viewer's FoV can similarly cause (additional) delays and degrade the user's quality of experience (QoE).

[0029] Using segments with shorter durations is effective in reducing the M2HQ delay for the viewport. Of course, since each segment has at least one random access frame, this efficiency improvement is achieved by the coding overhead of additional random access pictures.

[0030] In one embodiment, as the head-mounted display (HMD) moves, new tiles are requested. As the HMD speed increases, the M2HQ latency similarly increases. This is because many segments that are downloaded and decoded are not rendered, so the viewport changes that contribute to latency are very frequent. Therefore, the bitstream structure can be defined for the viewport based on the movement of the user's head.

[0031] When the user's viewport does not change, i.e., when the HMD is not moving, the bitstream segment duration (i.e., the number of encoded frames within a segment) can be the same for the low-resolution background and high-resolution tiles within the viewport.

[0032] By using segments with longer durations (and thus larger segment sizes), it becomes possible to optimally compress the video, thereby reducing the required bandwidth. However, when the bandwidth is not limited, segments with reduced duration and size can be used.

[0033] Here, as the movement of the user's head, a new HQ tile is requested. The maximum latency of M2HQ can be defined by the segment duration. If the user's orientation does not change or the head moves at a very reduced rate, the bitstream for the viewport can be of a longer duration.

[0034] A head speed threshold (H TH ) can be defined based on factors such as the available bandwidth. When the bandwidth is not limited, the segment duration / size can be reduced to decrease the M2HQ latency. The bitstream segment duration / size can be further reduced as the HMD speed increases.

[0035] Thus, the server can encode segments with variable duration / size based on Real-Time Transport Protocol (RTP) Control Protocol (RTCP) feedback (which will include bandwidth information and HMD speed).

[0036] Alternatively, the server can have multiple versions of a bitstream with variable segment duration / size, and based on RTCP feedback (which will include bandwidth information and HMD speed), the bitstream segment size can be defined. Thus, when the server receives a bitstream request from a user, it can send bitstream segments based on the HMD speed and available bandwidth.

[0037] Referring now to FIG. 6, a high-quality (8K) viewport bitstream can have multiple bitstreams of variable duration / size and one fallback low-resolution (2K) for the background with a fixed segment duration / size.

[0038] The transmitter can define a lower bound (D min ) on the segment duration due to bandwidth constraints. Even if the HMD speed increases beyond the corresponding HMD maximum threshold (H MAX ), the server does not reduce the segment duration below its lower bound.

[0039] The server varies the segment duration in this way, but it maintains it within the codec profile / level compliance requirements of the notified codec profile and level, so that the decoder at the receiver can decode segments without re-initialization.

[0040] To minimize M2HQ latency and improve the user experience, when sufficient bandwidth is available, the receiver can request a high-quality and additional margin around the viewport. The margin can be of the same quality as the viewport, but not necessarily.

[0041] In the same or another embodiment, if the movement of the HMD is within the margin, the new tiles required (to update the margin) may be segments of longer duration and may not necessarily require a lower segment duration / size. This is applicable when the resolution of the margin is the same as that of the viewport (i.e., high-resolution tiles).

[0042] If the resolution of the tiles within the margin is lower compared to the tiles within the viewport, shorter high-resolution segments may be required as the HMD moves even within the margin.

[0043] However, if the viewport moves beyond the margin, new shorter-duration segments may be required to reduce M2HQ. Optionally, the size of the bitstream segments within the viewport may be downloaded with a reduced segment duration compared to the segments for the margin.

[0044] The transmitter may optionally use one or more of the following parameters, namely, the head velocity threshold (H TH ), the maximum threshold (H MAX ), and the minimum segment duration (D min ) to notify the receiver of the use of variable-duration segments in session setup using SDP. Thus, the receiver optimizes its segment requests based not only on including the head velocity in its RTCP report but also on optional parameters.

[0045] Figure 7 illustrates the system design as invented. After the content is captured by the camera, it is stitched together into a panoramic representation (701), and then encoded (702) and passed through a low-latency packetizer (703) that structures the bitstream with a reduced random access point period. The transmitted video (709) includes high-resolution content of the user's FoV (708) and reduced quality for the rest of the picture (707). This content is transmitted to the VR player (705) for rendering (706) via the CDN (704).

[0046] The techniques for the bitstream structure for the above immersive viewport-based video streaming can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, FIG. 8 shows a computer system 800 suitable for implementing a particular embodiment of the disclosed subject matter.

[0047] The computer software can be coded using any suitable machine code or computer language that can be subject to mechanisms such as assembly, compilation, linking, etc. to create code that includes instructions that can be executed directly by a computer central processing unit (CPU), a graphics processing unit (GPU), etc., or through interpretation, execution of microcode, etc.

[0048] The instructions can be executed in various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, Internet of Things devices, etc.

[0049] The components shown in FIG. 8 with respect to computer system 800 are essentially illustrative and are not intended to imply any limitation as to the use or functionality of the computer software implementing the embodiments of the present disclosure. Also, the configuration of the components should not be construed as having any dependencies or requirements related to any one or combination of the components shown in the exemplary embodiments of computer system 800.

[0050] Computer system 800 may include a particular human interface input device. Such a human interface input device may respond to input by one or more human users via, for example, tactile input (keystrokes, swipes, movement of a data glove, etc.), voice input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not depicted). Using a human interface device, it is also possible to capture certain media that is not necessarily directly related to conscious human input, such as voice (utterances, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0051] The input human interface device may include one or more (each depicted only once) of keyboard 801, mouse 802, trackpad 803, touch screen 810, data glove (not depicted), joystick 805, microphone 806, scanner 807, camera 808.

[0052] Computer system 800 may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices include tactile output devices (e.g., tactile feedback by touch screen 810, data glove (not depicted), or joystick 805, although there may also be tactile feedback devices that do not function as input devices), audio output devices (such as speaker 809, headphones (not depicted), etc.), visual output devices (such as screens 810 including CRT screens, LCD screens, plasma screens, OLED screens, regardless of whether each has a touch screen input function and regardless of whether each has a tactile feedback function, some of which may be capable of outputting two-dimensional visual output or three-dimensional or higher output through means such as stereographic output, virtual reality glasses (not depicted), holographic displays, and smoke tanks (not depicted)), and may also include printers (not depicted).

[0053] Computer system 800 can also include human-accessible memory devices and associated media, for example, optical media including CD / DVD ROM / RW 820 having CD / DVD or similar media 821, thumb drive 822, removable hard drive or solid state drive 823, legacy magnetic media such as tapes and floppy disks (not depicted), special ROM / ASIC / PLD-based devices such as security dongles (not depicted), etc.

[0054] One of ordinary skill in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0055] Computer system 800 can also include an interface to one or more communication networks. The network can be, for example, wireless, wired, or optical. The network can further be local, wide area, metropolitan, vehicle and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, etc., cellular networks including GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide area digital networks for TV including cable TV, satellite TV, and terrestrial broadcast TV, vehicle and industrial including CANBus, etc. Certain networks typically require an external network interface adapter attached to a certain general-purpose data port or peripheral bus (849) (such as a USB port of computer system 800), and other networks are typically integrated into the core of computer system 800 by attaching to the system bus as described later (such as an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 800 can communicate with other entities. Such communication can be only unidirectional reception (such as broadcast TV), only unidirectional transmission (such as CANbus to a certain CANbus device), or bidirectional with other computer systems using, for example, local or wide area digital networks. Certain protocols and protocol stacks can be used for each of those networks and network interfaces as described above.

[0056] The aforementioned human interface device, human-accessible storage device, and network interface can be connected to the core 840 of computer system 800.

[0057] The core 840 can include special programmable processing devices in the form of one or more central processing units (CPUs) 841, graphics processing units (GPUs) 842, field programmable gate arrays (FPGAs) 843, hardware accelerators 844 for certain tasks, etc. These devices can be connected via a system bus 848 together with a read-only memory (ROM) 845, a random access memory 846, and internal mass storage devices 847 such as internal hard drives and SSDs that are not accessible to the user. In some computer systems, access to the system bus 848 can be in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices can be directly connected to the core's system bus 848 or via a peripheral bus 849. The architecture of the peripheral bus includes PCI, USB, etc.

[0058] The CPU 841, GPU 842, FPGA 843, and accelerator 844 can execute certain instructions that can, in combination, constitute the aforementioned computer code. The computer code can be stored in the ROM 845 or RAM 846. Migration data can also be stored in the RAM 846, while persistent data can be stored, for example, in the internal mass storage device 847. Fast storage and retrieval of any memory device can be enabled through the use of cache memory that can be closely associated with one or more CPUs 841, GPUs 842, mass storage device 847, ROM 845, RAM 846, etc.

[0059] A computer-readable medium can have computer code for performing various computer-implemented operations. The medium and the computer code may be specially designed and configured for the purposes of the present disclosure or may be of the types well-known and available to those skilled in the computer software arts.

[0060] As an example, without limitation, a computer system having an architecture 800, specifically a core 840, can provide functions as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software incorporated in one or more tangible computer-readable media. Such computer-readable media can be associated with the user-accessible mass storage device introduced above, as well as media associated with specific storage devices of the core 840 having a non-transitory nature, such as the on-core mass storage device 847 and the ROM 845. The software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core 840. The computer-readable media can include one or more memory devices or chips according to specific requirements. The software can cause the core 840, particularly the processor (including a CPU, GPU, FPGA, etc.) therein, to define data structures stored in the RAM 846 and modify such data structures according to processes defined by the software, thereby executing specific processes described herein or specific parts of specific processes. Additionally, or alternatively, the computer system can provide functions as a result of being logically wired to, or otherwise embodied as, a circuit (e.g., an accelerator 844) that operates instead of or together with the software to execute specific processes described herein or specific parts of specific processes. Optionally, references to software can include logic, and vice versa. Optionally, references to computer-readable media can include circuits (such as integrated circuits (ICs)) that store software for execution, circuits that embody logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0061] Referring now to FIG. 9, an operational flowchart showing the steps of a method 900 executed by a program for splitting a viewport bitstream is depicted.

[0062] At 902, method 900 may include encoding a first coded video bitstream having a segment size defined for a viewport based on a threshold corresponding to a user's head movement.

[0063] At 904, method 900 may include encoding a second coded video bitstream having a low resolution corresponding to a background having a constant segment duration or size.

[0064] At 906, method 900 may include creating a streaming bitstream for decoding or rendering using the first coded video bitstream and the second coded bitstream.

[0065] It can be understood that FIG. 9 provides only an illustration of one implementation and does not imply any limitation as to how different embodiments may be implemented. Many modifications to the described environment may be made based on design and implementation requirements.

[0066] Some embodiments may relate to systems, methods, and / or computer-readable media in any possible technical detail level of integration. The computer-readable media may include a computer-readable non-transitory storage medium having computer-readable program instructions for causing a processor to execute operations.

[0067] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing, but is not limited thereto. A non-exhaustive list of more specific examples of computer-readable storage media includes the following, namely, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile discs (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves in which instructions are recorded, and any suitable combination of the foregoing. A computer-readable storage medium as used herein should not be construed to be a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through a wire.

[0068] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or can be downloaded from an external computer or an external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within each respective computing / processing device.

[0069] The computer-readable program code / instructions for performing the operations may be in any combination of one or more programming languages, including assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or source code or object code written in an object-oriented programming language such as Smalltalk or C++, and a procedural programming language such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), may execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit to perform the aspects or operations.

[0070] These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which are executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in the flowchart and / or block diagram blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that includes instructions for causing a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium includes a product that includes instructions for implementing the functions / operations specified in the flowchart and / or block diagram blocks.

[0071] The computer-readable program instructions may also be loaded onto a computer, other programmable apparatus, or other device, such that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, whereby the instructions which are executed on the computer, other programmable apparatus, or other device implement the functions / operations specified in the flowchart and / or block diagram blocks.

[0072] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of one or more executable instructions for implementing the specified logical function. The methods, computer systems, and computer-readable media may include additional blocks, fewer blocks, different blocks, or differently arranged blocks compared to those depicted in the figures. In some alternative implementations, the functions described in the blocks may be executed in a different order than that shown in the figures. For example, two blocks shown in succession may actually be executed simultaneously or substantially simultaneously, or the blocks may be executed in the reverse order depending on the related functionality, in some cases. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or that realizes a combination of dedicated hardware and computer instructions.

[0073] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it is understood that software and hardware can be designed based on the description herein to implement the systems and / or methods.

[0074] Elements, operations, or instructions used herein should not be construed as essential or mandatory unless explicitly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more". Further, as used herein, the term "set" includes one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more". When only one item is intended, the term "one" or similar words are used. Also, as used herein, terms such as "has", "have", "having", etc. are intended to be open-ended terms. Further, the phrase "based on" means "at least partially based on" unless otherwise specified.

[0075] The descriptions of various aspects and embodiments are presented for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Combinations of features are listed in the claims and / or disclosed herein, but these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically listed in the claims and / or not disclosed herein. Each of the dependent claims listed below may depend directly on only one claim, but the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, actual use, or technological improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Description of Reference Numerals

[0076] 100 Ecosystem 101 360 Video 102 Encoder 103 Content Delivery Network 104 Received Content 105 VR Player 106 User's Field of View 200 360 Viewport-Dependent Video Streaming System 201 360 Video Scene 202 Planar Video Encoder 203 DASH Packager 204 CDN 205 Compatibility Player 206 Rendering 207 Picture 208 High-Resolution FoV 209 Transmitted Video 300 Block Diagram 301 - 303 8k 304 - 306 2k 400 Block Diagram 401 I-Frame 402 Viewport 403 - 405 Frames 500 Block Diagram 501 - 508 Frames 509 Fallback Bitstream 701 Panoramic Representation 702 Encoded 703 Low-Latency Packager 704 CDN 705 VR Player 706 Rendering 707 Picture 708 FoV 709 Transmitted Video 800 Computer System 801 Keyboard 802 Mouse 803 Trackpad 805 Joystick 806 Microphone 807 Scanner 808 Camera 809 Speaker 810 Touch Screen 820 CD / DVD ROM / RW 821 CD / DVD or Similar Medium 822 Thumb Drive 823 Solid State Drive 840 Core 841 CPU 842 GPU 843 FPGA 844 Accelerator 845 ROM 846 RAM 847 Internal Mass Storage Device 848 System Bus 849 Peripheral Bus 900 Method

Claims

1. 1. A method executable by a processor for encoding a viewport bitstream, comprising: receiving information from a video decoder regarding (i) a speed of a user's head movement as the user watches video content via a wearable device and (ii) available bandwidth, the video content being decoded by the video decoder and received via a content delivery network (CDN); encoding a first coded video bitstream having a first resolution and a segment size defined for a viewport based on the available bandwidth and a head velocity threshold corresponding to a rate of head movement of the user, such that the segment size is inversely correlated to a rate of head movement of the user; encoding a second coded video bitstream having a second resolution lower than the first resolution, the second resolution corresponding to a background having a constant segment duration or size; using the first coded video bitstream and the second coded video bitstream to create a streaming bitstream for decoding or rendering; transmitting the streaming bitstream to the video decoder via the CDN; The method includes:

2. The method of claim 1 , wherein the first coded video bitstream has multiple segment sizes for the viewport.

3. The method of claim 1 , wherein the head velocity threshold depends on a rate of the user's head movement and available bandwidth.

4. The method of claim 3 , wherein receiving the information regarding the rate of the user's head movement and the available bandwidth is performed via Real-time Transport Protocol (RTP) Control Protocol (RTCP) feedback.

5. The method of claim 1 , wherein a margin around the viewport is required with a segment size equal to or longer than the viewport.

6. The method of claim 5 , wherein new tiles having longer segments are requested to update the margin based on the user's head movement being within the margin.

7. The method of claim 5 , wherein shorter segments having the first resolution are requested based on tiles in the margin having a lower resolution than tiles in the viewport.

8. The method of claim 7 , wherein the shorter segment is requested regardless of whether the user's head movement is within the margin.

9. The method of claim 5 , wherein shorter segments are requested based on the viewport moving beyond the margin.

10. A computer system configured to carry out the method of any one of claims 1 to 9.

11. A program for causing a computer to execute the method according to any one of claims 1 to 9.

12. 1. A method executable by a processor for decoding a viewport bitstream, comprising: receiving a first video content through a content delivery network (CDN); decoding the first video content; sending to a video encoder information about (i) a speed of the user's head movement as the user watches the first video content via a wearable device, and (ii) available bandwidth; receiving from the video encoder, based on the transmitted information, a streaming bitstream including: (i) a first coded video bitstream having a first resolution defined for a viewport based on available bandwidth, a minimum segment duration, and a head velocity threshold corresponding to a velocity of the user's head movement such that a segment size is inversely proportional to the velocity of the user's head movement and is no shorter than the minimum segment duration; and (ii) a second coded video bitstream having a second resolution lower than the first resolution, corresponding to a background having a constant segment duration or size; and rendering a second video content using the first coded video bitstream and the second coded video bitstream.

13. The method of claim 12 , wherein the first coded video bitstream has multiple segment sizes for the viewport.

14. The method of claim 12 , wherein the head velocity threshold depends on the user's head movement and the available bandwidth.

15. 15. The method of claim 14, wherein transmitting information regarding the speed of the user's head movement and the available bandwidth is via Real-time Transport Protocol (RTP) Control Protocol (RTCP) feedback.

16. The method of claim 12 , wherein a margin around the viewport is required with a segment size equal to or longer than the viewport.

17. The method of claim 16 , wherein new tiles having longer segments are requested to update the margin based on the user's head movement being within the margin.

18. The method of claim 16 , wherein shorter segments having the first resolution are requested based on tiles in the margin having a lower resolution than tiles in the viewport.

19. The method of claim 18 , wherein the shorter segment is requested regardless of whether the user's head movement is within the margin.

20. The method of claim 16 , wherein shorter segments are requested based on the viewport moving beyond the margin.

21. A decoder configured to perform the method according to any one of claims 12 to 20.

22. A program for causing a computer to carry out the method according to any one of claims 12 to 20.

Images