Method and apparatus for encoding viewport bitstreams

Abstract

A method, computer program, and computer system for encoding viewport bitstreams are provided. A first coded video bitstream containing segment sizes defined for the viewport is encoded based on a threshold corresponding to user head movement. A second coded video bitstream with low resolution is encoded. The second coded video bitstream may correspond to a background containing segments of constant duration or size. A streaming bitstream for decoding or rendering is created using the first and second coded video bitstreams.

Description

[0001] Cross-references to related applications

[0002] This application claims priority based on U.S. Provisional Application No. 63 / 111,425 (filed November 9, 2020) and U.S. Application No. 17 / 362,068 (filed June 29, 2021), the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the field of data processing, and more specifically to a method and apparatus for encoding viewport bitstreams. Background Technology

[0004] Immersive video streaming involves transmitting a “world” or “omnidirectional” view from a transmitter to a receiver, where the receiver renders only a portion of the received world view based on, for example, a viewport. The viewport can be selected based on the direction of head movement while wearing virtual reality goggles. Viewport-dependent video streaming can involve techniques where, in a recorded scene covering the “world” view, only a portion of that view is being transmitted and presented to the user based on a viewport already selected by the user. This is because at any given time, a user’s field-of-view (FoV) only includes a certain range; bitstreams outside the user’s current FoV are not viewed, and transmitting the entire image would result in wasted bandwidth and rendering resources. By segmenting large immersive video bitmaps into smaller segments or tiles and transmitting only those falling within the user’s FOV, network and rendering resources are saved. Additionally, when streaming high-resolution video, in situations with limited network resources, the transmission of segments can cause delays in bitstream download and rendering because each segment comprises one or more encoded frames. Changes in the viewer's FoV can also lead to latency and reduce the user's quality experience.

[0005] Therefore, the problem that this invention needs to solve is how to reduce latency and save resources when transmitting viewport bit streams. Summary of the Invention

[0006] Implementations relate to methods, systems, and computer-readable media for segmenting viewport bitstreams. According to one aspect, a method for encoding a viewport bitstream is provided. The method may include encoding a first coded video bitstream comprising segment sizes defined for the viewport based on a threshold corresponding to a user's head movement. A second coded video bitstream having low resolution is then encoded. The second coded video bitstream may correspond to a background comprising segments of constant duration or size. A streaming bitstream for decoding or rendering is created using the first and second coded video bitstreams.

[0007] According to another aspect, a computer system for segmenting viewport bitstreams is provided. The computer system may include one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the storage devices via at least one of the one or more memories, the program instructions being executable by at least one of the one or more processors, thereby enabling the computer system to perform a method. The method may include encoding a first coded video bitstream comprising segment sizes defined for the viewport based on a threshold corresponding to a user's head movement. Encoding a second coded video bitstream having a low resolution. The second coded video bitstream may correspond to a background comprising segments of constant duration or size. A streaming bitstream for decoding or rendering is created using the first and second coded video bitstreams.

[0008] According to another aspect, a computer-readable medium is provided for segmenting viewport bitstreams. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, the program instructions being executable by a processor. The program instructions are executable by the processor to perform a method, which may accordingly include encoding a first coded video bitstream comprising segment sizes defined for the viewport based on a threshold corresponding to a user's head movement; encoding a second coded video bitstream having a low resolution; the second coded video bitstream may correspond to a background comprising segments of constant duration or size; and creating a streaming bitstream for decoding or rendering using the first and second coded video bitstreams.

[0009] According to another aspect, an apparatus is provided for encoding a viewport bitstream, the apparatus comprising: a first encoding unit configured to encode a first coded video bitstream comprising a segment size defined for the viewport based on a threshold corresponding to a user's head movement; a second encoding unit configured to encode a second coded video bitstream having a low resolution, the second coded video bitstream corresponding to a background comprising a constant segment duration or size; and a creation unit configured to create a streaming bitstream for decoding or rendering using the first coded video bitstream and the second coded video bitstream.

[0010] According to another aspect, a computer device is provided, the computer device including a processor and a memory for storing program code and transmitting the program code to the processor; the processor is configured to execute, according to instructions in the program code: encoding a first coded video bitstream comprising a segment size defined for a viewport based on a threshold corresponding to a user's head movement; encoding a second coded video bitstream having a low resolution, the second coded video bitstream corresponding to a background comprising a constant segment duration or size; and creating a streaming bitstream for decoding or rendering using the first coded video bitstream and the second coded video bitstream.

[0011] According to this disclosure, a method and apparatus for encoding a viewport bitstream are provided. A first coded video bitstream containing segment sizes defined for the viewport is encoded based on a threshold corresponding to a user's head movement. A second coded video bitstream with low resolution is encoded, which may correspond to a background containing segments of constant duration or size. A streaming bitstream for decoding or rendering is created using the first and second coded video bitstreams. This application defines the segment size based on the viewport and encodes the first video bitstream containing the aforementioned segment size based on the speed of the user's head movement, and also encodes the second video bitstream corresponding to the background, thereby creating a streaming bitstream for decoding or rendering using the first and second coded video bitstreams. Therefore, when transmitting the viewport bitstream, transmission resources are saved because only a portion of the bitstream defined by the viewport is transmitted. Furthermore, the low resolution of the background-corresponding video bitstream reduces network latency for transmission and rendering. Attached Figure Description

[0012] These and other objects, features, and advantages will become apparent from the following detailed description of illustrative embodiments, which will be read in conjunction with the accompanying drawings. The various features in the drawings are not to scale, as the illustrations are provided for clarity by those skilled in the art in conjunction with the detailed description. In the drawings:

[0013] Figure 1 This is a schematic diagram of an ecosystem for streaming immersive video according to at least one embodiment;

[0014] Figure 2 This is a schematic diagram of a viewport-based streaming workflow for immersive video according to at least one embodiment.

[0015] Figure 3 This is a schematic diagram of a sliced ​​immersive video bitstream including frame boundaries according to at least one embodiment.

[0016] Figure 4 This is a schematic diagram of viewport updates during immersive bitstream playback including frame boundaries, according to at least one embodiment.

[0017] Figure 5 This is a schematic diagram of 8K and 2k immersive bitstream fragmentation including frame boundaries according to at least one embodiment.

[0018] Figure 6 This is a schematic diagram of the proposed immersive bitstream fragmentation method that includes frame boundary implementation.

[0019] Figure 7 It is a block diagram of a system for viewport bitstream segmentation according to at least one embodiment;

[0020] Figure 8 It is a schematic diagram of a computer system according to at least one embodiment; and

[0021] Figure 9 This is an operational flowchart illustrating the steps for viewport bitstream segmentation performed by a program according to at least one embodiment. Detailed Implementation

[0022] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that can be implemented in various forms. These structures and methods can be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the scope to those skilled in the art. Details of well-known features and techniques may be omitted in the description to avoid unnecessarily obscuring the presented embodiments.

[0023] The implementations generally relate to the field of data processing, and more specifically to video streaming. The exemplary implementations described below provide systems, methods, and computer programs for viewport-based video streaming, etc. Therefore, some implementations have the capability to improve computational efficiency by allowing the viewport bitstream to be segmented into smaller fragment files or DASH segments, and to provide backup bitstream fragments or segments that support fast random access to the bitstream during playback.

[0024] As previously mentioned, immersive video streaming involves transmitting a "world" or "omnidirectional" view from a transmitter to a receiver, whereby the receiver presents only a portion of the received world view based on, for example, a viewport. The viewport can be selected based on the direction of head movement while wearing virtual reality goggles. Viewport-dependent video streaming can involve techniques where, in a recorded scene covering the "world" view, only a portion of that view is being transmitted and presented to the user based on a viewport already selected by the user.

[0025] However, as the rate at which the user moves his / her head increases, the request for new tiles also increases, thus increasing M2HQ latency. Furthermore, much of the downloaded data is not viewed by the user and is therefore wasted. Therefore, reducing the latency incurred when changing the viewport by reconstructing the bitstream in immersive video can be beneficial.

[0026] Various aspects are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0027] The exemplary embodiments described below provide systems, methods, and computer programs relating to viewport-based video streaming, and more specifically to bitstream structures of viewports for immersive teleconference and telepresentation for remote terminals when the user's viewport changes by using variable bitstream segment sizes.

[0028] Figure 1 A block diagram of an ecosystem 100 for streaming immersive video is shown. 360 video (101) is transmitted via an encoder (102), and the entire image is streamed via a Content Delivery Network (CDN) (103). The user decodes and reconstructs the received content (104) in a VR player (105). The disadvantage of sending the entire image is the waste of bandwidth and rendering resources, because at any given time, the user's field-of-view (FoV) (106) only falls within the coverage area. Bitstreams outside the user's current FoV may not be viewed, but are received and rendered by the client anyway. For example, viewport selection can be based on the direction of head movement when wearing virtual reality goggles. Viewport-dependent video streaming can involve techniques in which only a portion of the recorded scene covering the "world" view is being transmitted and presented to the user based on the viewport already selected by the user, thus eliminating the need to transmit the entire world view.

[0029] Tile-based (or sub-picture) immersive video streaming technology can be used to reduce bandwidth requirements and improve video playback quality. (See also...) Figure 2 A block diagram of a 360-viewport-related video streaming system 200 is depicted. After the video scene has been properly stitched and projected onto a planar video using, for example, equirectangular projection and cubemap projection, the 360-viewport video scene (201) is encoded using a planar video encoder (202).

[0030] To use encoded video bitstreams on DASH, the sequence can be placed into a file with a format that divides the bitstream into a series of smaller, HTTP-based video bitstream segments. These video files can contain different bitrates and durations. These video files can be transmitted over IP networks and decoded independently on the client side.

[0031] The encoded immersive video can be fragmented using the DASH packer (203) as described above. The fragmented content can be stored in a content delivery server (not depicted) and delivered by a CDN (204) to a compatible player (205) for presentation (206). Here, the transmitted video (209) consists only of high-resolution FoV (208), rather than sending the entire image (207). At the receiver side, the 4K image is decoded and reconstructed in the player.

[0032] Tile-based streaming spatially divides 360-degree video frames into multiple tiles or blocks. Here, the panoramic video is encoded and then, after compression, divided into tiles. The user then requests only the tiles that fall entirely or partially within their field of view (FOV). By segmenting large immersive video bitmaps into smaller pieces or tiles, and transmitting only those pieces or tiles that fall within the user's FOV, network and presentation-side resources are saved.

[0033] Large immersive content (after projection, i.e., a planar video stream) can be spatially subdivided into tiles of equal resolution, for example. For instance, the source image of a 4k×2k video sequence can be divided into tiles of equal size, each a sample of 512×156, resulting in 64 tiles. Each tile can be encoded and packaged at a different bitrate and quality level (as is typically the quality level in DASH) and can be required to have a quality different from its neighboring tiles. Tiles within the user's viewport can be prioritized and advantageously streamed at a higher quality than tiles outside the viewport. In some cases, certain tiles can be omitted entirely from the transmission. As a fallback, an additional layer with, for example, lower resolution / quality / bitrate can be used to cover the entire panorama. Assuming proper player design, this prevents visual artifacts, such as black areas when the FoV changes but new tiles are not immediately available due to network / streaming server latency.

[0034] The resolution of a tile can change as the user moves their head, but this change can occur, for example, only at a Random-Access Point (RAP). A RAP can be an access unit where the receiver can successfully begin decoding the tile or video. Picture frames can be grouped together with different Group of Pictures (GOP) sizes. An I-frame may be followed by a P-frame, which can contain a coded representation of changes from previous frames. Therefore, a P-frame depends on both the I-frame and earlier P-frames. The GOP structure, used in typical encoders, makes each I-frame a random access point, allowing decoding to begin on the I-frame. Therefore, the response time required to change a tile depends on the tile granularity and the RAP distance. When the user's orientation changes, the tile currently in the viewport may need to be (at least partially) replaced by a different tile. These new tiles can only be switched at the next available RAP, resulting in a delayed response to user input.

[0035] Now refer to Figure 3A block diagram 300 depicts a tiled immersive video bitstream including a frame boundary implementation. For example, 8k (301 to 303) and 2k (304 to 306) resolution video bitstreams can be contained within the same representation of a projected scene. When the FoV changes, the lower resolution (304 to 306) can be used as a backup and can be streamed continuously. The higher resolution (301 to 303) as described above can be a tiled representation and can be used for viewport-independent streaming. The two streams can have equal frame counts and can be divided into equal fixed-frame chunks, which can include a random access point period of 30 frames. Only certain tiles of 8K resolution are delivered as needed to cover the FoV, while the entire frame with 2K resolution is delivered as a backup. Assuming a fixed frame rate of 30fps, the RAP can exist once per second in each bitstream.

[0036] While such a configuration allows for visually pleasing and rapid responses by using a reconstructed backup bitstream in the event of a FoV change, the following problem remains: an average of 15 frames of useless high-resolution tiles are sent with each FoV change, as these high-resolution tiles may not be able to be rendered due to the changed FoV.

[0037] Reference Figure 4 Block diagram 400 depicts viewport updates during immersive bitstream playback in an implementation that includes frame boundaries. Block diagram 400 illustrates a simulated FoV update when playback is in place. When the viewport becomes 402, the entire 403 frame, except for 404 and 405, will be downloaded and decoded, even though the viewport is in the middle of the frame because the I-frame (401) is at the beginning of the frame. Downloading new fragments with 8K resolution can cause additional latency if the current network bandwidth is not ideal. A way to overcome this latency is to include a reduced bitstream fragment size for the viewport.

[0038] Now refer to Figure 5 A block diagram 500 depicts 8K and 2K immersive bitstream fragmentation including frame boundary implementations. For high-resolution segments in the viewport, the random access point period can be reduced to, for example, 10 frames (501 to 508). The first image is encoded as an I-frame and the remaining images are encoded as P-frames, and for low-resolution backups, either only the first image is encoded as an I-frame and subsequent images are encoded as P-frames, or fast random access including unequal random access periods should be limited.

[0039] When the random access period is short, the client doesn't need to download all 30 frames when the viewport changes, but only the current 10 frames. Therefore, the latency between requesting a new viewport and rendering it is reduced because the random access point period is shorter and more random access points are now available. For less-than-ideal network conditions, a fallback bitstream of 509 with a lower bitstream resolution can be provided. Of course, this improved efficiency is achieved through the encoding overhead of the additional random access images.

[0040] As mentioned above, downloading intra-frame encoded images may consume more network bandwidth than the user's FoV playback and rendering.

[0041] When streaming high-resolution video, segments can cause delays in bitstream download and rendering when network resources are not ideal. Each segment consists of one or more encoded frames. Variations in the viewer's FoV can also cause (additional) delays and degrade the user's Quality of Experience (QoE).

[0042] Using segments with shorter durations can effectively reduce viewport M2HQ latency. Of course, since each segment contains at least one random access frame, this improved efficiency is achieved through the coding overhead of the additional random access images.

[0043] In this implementation, new tiles are requested as the head-mounted display (HMD) moves. As the speed of the HMD increases, the M2HQ latency also increases. This is because viewport changes are very frequent, increasing latency as many downloaded and decoded segments are not rendered. Therefore, the bitstream structure can be tailored to the viewport based on the user's head movement.

[0044] When the user's viewport does not change, i.e. the HMD does not move, the duration of a bitstream segment (i.e., the number of coded frames in the segment) can be the same as a segment of low-resolution background and high-resolution tiles in the viewport.

[0045] Using longer duration segments (and therefore larger segment sizes) allows for optimal video compression, reducing the required bandwidth. However, when bandwidth is not a limiting factor, segments with reduced duration and size can be used.

[0046] Now, as the user's head moves, a new HQ tile is requested. The maximum latency in M2HQ can be limited by the duration of the fragment. If the user's orientation does not change or the head moves at a very low rate, the viewport's bitstream can contain a longer duration.

[0047] It may be possible to limit the head velocity threshold (H) based on factors such as available bandwidth.TH If bandwidth is not a limiting factor, the segment duration / size can be reduced to decrease M2HQ latency. As HMD speeds increase, the bitstream segment duration / size can be further reduced.

[0048] Therefore, the server can encode segments containing variable durations / sizes based on feedback from the Real-time Transport Protocol (RTP) Control Protocol (RTCP), which includes bandwidth information and HMD speed.

[0049] Alternatively, the server can include multiple versions of bitstreams with variable segment durations / sizes, and can limit the bitstream segment size based on RTCP feedback (which includes bandwidth information and HMD speed). Therefore, when the server receives a bitstream request from a user, it can send bitstream segments based on the HMD speed and available bandwidth.

[0050] Now refer to Figure 6 The high-quality (8K) viewport bitstream can have multiple bitstreams with variable durations / sizes, and a backup low-resolution (2K) background with constant segment durations / sizes.

[0051] Due to bandwidth limitations, the transmitter can limit the lower bound of the segment duration (D). min Even if the HMD speed increases beyond the corresponding maximum HMD threshold (H). MAX The server will not reduce the fragment duration from that lower limit.

[0052] When the server changes the segment duration in this scheme, the server maintains consistency with the codec profile / level required by the signaled codec profile / level, so the receiver's decoder can decode the segment without reinitialization.

[0053] To minimize M2HQ latency and enhance user experience, receivers can add a margin around the viewport with high quality requests when sufficient bandwidth is available. This margin can have the same quality as the viewport, but it is not required.

[0054] In the same or another implementation, if the HMD movement is within the margin, the requested new tile (for updating the margin) may contain a longer duration segment, and a lower segment duration / size may not necessarily be required. This applies when the resolution of the margin is the same as the viewport resolution, i.e., a high-resolution tile.

[0055] If the resolution of tiles in the margin is lower than that of tiles in the viewport, shorter, higher-resolution segments can be requested even when the HMD moves within the margin.

[0056] However, if the viewport moves beyond the margin, a new, shorter duration segment can be requested to reduce the M2HQ. Optionally, the size of the bitstream segment in the viewport can be downloaded with a reduced segment duration compared to the segment used for the margin.

[0057] The transmitter can signal to the receiver during a session using SDP settings that it is using a variable duration segment, optionally with one or more of the following parameters: head velocity threshold (H TH ), maximum threshold (H) MAX ) and minimum segment duration (D min This allows the receiver to include the header velocity in its RTCP report and optimize its fragment requests based on optional parameters.

[0058] Figure 7 The system design of the invention is illustrated. After content is captured by a camera device, it is stitched into a panoramic representation (701), and then the stitched content is encoded (702) and passed through a low-latency packetizer (703), in which the bitstream is constructed to include reduced random access point periods. The transmitted video (709) contains high-resolution content of the user's FoV (708) and reduced quality for the rest of the images (707). The content is delivered to a VR player (705) for presentation (706) via a CDN (704).

[0059] The aforementioned bitstream structure for immersive viewport-based video streaming can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 8 A computer system 800 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0060] Computer software can be coded using any suitable machine code or computer language, which can be subjected to mechanisms such as assembly, compilation, and linking to create code including instructions. These instructions can be executed directly by the computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, etc.

[0061] The instructions can be executed on various types of computers or components, including personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0062] Figure 8 The components shown for computer system 800 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any component or combination of components shown in the exemplary embodiments of computer system 800.

[0063] Computer system 800 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., speech, clapping), visual input (e.g., gestures), and olfactory input (not depicted). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image capturing devices), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0064] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard 801, mouse 802, touchpad 803, touch screen 810, data glove (not depicted), joystick 805, microphone 806, scanner 807, and camera device 808.

[0065] The computer system 800 may also include certain human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: tactile output devices (e.g., tactile feedback via touchscreen 810, data gloves (not depicted), or joystick 805, but tactile feedback devices that are not used as input devices may also exist); audio output devices (e.g., speakers 809, headphones (not depicted)); visual output devices (e.g., screens 810, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without tactile feedback capability—some of which may be able to output two-dimensional or more than three-dimensional visual output in a manner such as stereoscopic image output; virtual reality glasses (not depicted); holographic displays and smoke boxes (not depicted)); and printers (not depicted).

[0066] The computer system 800 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 820 with media such as CD / DVD 821, thumb drives 822, removable hard disk drives or solid-state drives 823, conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD such as security dongles (not depicted), etc.

[0067] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0068] Computer system 800 may also include interfaces to one or more communication networks. For example, the network may be wireless, wired, or optical. The network may also be a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), vehicular and industrial network, real-time network, latency-tolerant network, etc. Examples of networks include LANs such as Ethernet, wireless LAN, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital television networks (including cable television, satellite television, and terrestrial broadcast television), vehicular and industrial networks (including CANbus), etc. Some networks typically require external network interface adapters (such as, for example, a USB port of computer system 800) to be attached to some general-purpose data port or peripheral bus (849); other networks are typically integrated into the core of computer system 800 by attaching to the system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 800 can communicate with other entities. Such communication can be one-way (e.g., broadcast television), one-way (e.g., to a CANbus device), or bidirectional (e.g., to other computer systems using local or wide area digital networks). Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0069] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces can be attached to the core 840 of the computer system 800.

[0070] The core 840 may include one or more central processing units (CPUs) 841, graphics processing units (GPUs) 842, dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) 843, hardware accelerators 844 for certain tasks, etc. These devices, along with read-only memory (ROM) 845, random access memory 846, and internal mass storage devices 847 such as internal non-user-accessible hard disk drives (SDs), can be connected via a system bus 748. In some computer systems, the system bus 748 can be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus 748 or via a peripheral bus 849. Peripheral bus architectures include PCI, USB, etc.

[0071] The CPU 841, GPU 842, FPGA 843, and accelerator 844 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 845 or RAM 846. Transient data can also be stored in RAM 846, while permanent data can be stored, for example, in an internal mass storage device 847. Fast storage and retrieval of any storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs 841, GPUs 842, mass storage devices 847, ROM 845, RAM 846, etc.

[0072] Computer-readable media may have computer code thereon for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

[0073] By way of example and not limitation, a computer system 800 with an architecture, and particularly a core 840, can be functionalized by a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software implemented in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, and certain storage devices of the core 840 with non-transitory characteristics, such as internal mass storage device 847 or ROM 845. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core 840. Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause the core 840, and particularly its processors (including CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 846 and modifying such data structures according to the software-defined processes. Alternatively or as an alternative, the computer system may provide functionality by means of logic hardwired or otherwise implemented in circuitry (e.g., accelerator 844), which may replace or operate with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and conversely, references to logic may include software. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry implementing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0074] Now refer to Figure 9 The flowchart depicts the steps of a method 900 for segmenting viewport bitstreams, executed by a program.

[0075] At 902, method 900 may include encoding a first coded video bitstream containing a segment size defined for the viewport based on a threshold corresponding to the user’s head movement.

[0076] At 904, method 900 may include encoding a second coded video bitstream with low resolution, the second coded video bitstream corresponding to a background containing constant segment duration or size.

[0077] At 906, method 900 may include creating a streaming bitstream for decoding or presentation using a first encoded video bitstream and a second encoded bitstream.

[0078] It is worth noting that, Figure 9This illustration provides only one possible implementation and does not imply any limitations on how different implementations can be achieved. Many modifications can be made to the depicted environment based on design and implementation requirements.

[0079] This disclosure also provides an apparatus for encoding a viewport bitstream, the apparatus comprising: a first encoding unit configured to encode a first coded video bitstream comprising a segment size defined for the viewport based on a threshold corresponding to a user's head movement; a second encoding unit configured to encode a second coded video bitstream having a low resolution, the second coded video bitstream corresponding to a background comprising a constant segment duration or size; and a creation unit configured to create a streaming bitstream for decoding or rendering using the first coded video bitstream and the second coded video bitstream.

[0080] In some examples, the first encoded video bitstream contains multiple segment sizes for the viewport.

[0081] In some examples, the threshold depends on the user's head movement and available bandwidth.

[0082] In some examples, the user can transmit feedback information about the head movement and the available bandwidth via the Real-time Transport Protocol (RTP) Control Protocol (RTCP).

[0083] In some examples, margin around the viewport is requested in the form of a fragment size that is the same as or longer than the viewport.

[0084] In some examples, based on the user's head movement within the margin, a new tile containing a longer segment is requested to update the margin. In some examples,

[0085] In some examples, shorter, high-resolution segments are requested because the resolution of tiles in the margin is lower than that of tiles in the viewport.

[0086] In some examples, the shorter, high-resolution segment is requested regardless of whether the user's head movement is within the margin.

[0087] In some examples, a shorter segment is requested based on the viewport movement exceeding the margin.

[0088] This disclosure also provides a computer device, characterized in that the device includes a processor and a memory, the memory being used to store program code and transmit the program code to the processor; the processor being used to execute, according to instructions in the program code: encoding a first coded video bitstream containing segment sizes defined for a viewport based on a threshold corresponding to a user's head movement; encoding a second coded video bitstream having low resolution, the second coded video bitstream corresponding to a background containing constant segment durations or sizes; and creating a streaming bitstream for decoding or rendering using the first coded video bitstream and the second coded video bitstream.

[0089] Some implementations may relate to systems, methods, and / or computer-readable media at any possible level of integration technical detail. Computer-readable media may include computer-readable non-transitory storage media (or media) having computer-readable program instructions on them for causing a processor to perform operations.

[0090] A computer-readable storage medium can be a tangible device capable of retaining and storing instructions for use by an instruction execution device. For example, a computer-readable storage medium can be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital multifunction discs (DVDs), memory sticks, floppy disks, mechanical encoding devices such as punched cards or raised structures in recesses on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses transmitted through fiber optic cables), or electrical signals transmitted through wires.

[0091] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network such as the Internet, local area network, wide area network, and / or wireless network to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0092] Computer-readable program code / instructions used to perform operations can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, configuration data of an integrated circuit system, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.) and procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of execution entirely on a remote computer or server, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the use of the Internet provided by an Internet service provider). In some implementations, electronic circuit systems, including, for example, programmable logic circuit systems, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be specialized by using state information of computer-readable program instructions to execute computer-readable program instructions in order to perform various aspects or operations.

[0093] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0094] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, implement the function / action specified in one or more blocks of a flowchart and / or block diagram.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a portion of a module, segment, or instruction, which includes one or more executable instructions for implementing a specified logical function. The method, computer system, and computer-readable medium may include additional blocks, fewer blocks, different blocks, or blocks arranged differently compared to those depicted in the drawings. In some alternative implementations, the functions indicated in a block may not occur in the order shown in the drawings. For example, two blocks shown consecutively may actually be executed simultaneously or substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0096] It will be apparent that the systems and / or methods described herein can be implemented in various forms, including hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited in its implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the descriptions herein.

[0097] Unless explicitly stated otherwise, no element, action, or instruction used herein should be construed as critical or necessary. Furthermore, as used herein, the article “a” or “one” is intended to include one or more items and may be used interchangeably with “one or more.” Additionally, as used herein, the term “group” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with “one or more.” The term “one” or similar language is used when referring to only one item. Furthermore, as used herein, the terms “has,” “have,” “having,” etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.”

[0098] Descriptions of various aspects and implementations have been presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed implementations. Even combinations of features recited in the claims and / or disclosed in the specification are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or not disclosed in the specification. Although each listed dependent claim may directly refer to only one claim, the disclosure of possible implementations includes combinations of each dependent claim with each other claim in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described implementations. The terminology used herein has been chosen to best illustrate the principles of the implementations, their practical application, or technical improvements to technologies found in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method for encoding a viewport bitstream, the method comprising: A first coded video bitstream containing segment sizes defined for the viewport is encoded based on a threshold corresponding to the user’s head movement speed, the threshold depending on information received from the decoder regarding the user’s head movement speed and available bandwidth when viewing video content through a wearable device; The first encoded video bitstream has a first resolution; Encode a second coded video bitstream with low resolution, the second coded video bitstream having a background with constant segment duration or size, the second coded video bitstream having a second resolution, the second resolution being lower than the first resolution; as well as A streaming bitstream for decoding or rendering is created using the first and second encoded video bitstreams.

2. The method according to claim 1, characterized in that, The first encoded video bitstream contains multiple segment sizes for the viewport.

3. The method according to claim 1, characterized in that, The user transmits information about the head movement speed and the available bandwidth via Real-time Transmission Control Protocol (RTCP).

4. The method according to claim 1 or 2, characterized in that, Request margin around the viewport with a segment size that is the same as or longer than the viewport.

5. The method according to claim 4, characterized in that, Based on the user's head movement within the margin, a new tile containing a longer segment is requested to update the margin.

6. The method according to claim 4, characterized in that, Since the resolution of the tiles in the margin is lower than that of the tiles in the viewport, a shorter, higher-resolution fragment is requested.

7. The method according to claim 6, characterized in that, The shorter, higher-resolution segment is requested regardless of whether the user's head movement is within the margin.

8. The method according to claim 4, characterized in that, Based on the viewport movement exceeding the margin, a shorter segment is requested.

9. An apparatus for encoding a viewport bitstream, the apparatus comprising: A first encoding unit is configured to encode a first encoded video bitstream containing a segment size defined for a viewport based on a threshold corresponding to the user's head movement speed, the threshold depending on information received from a decoder regarding the user's head movement speed and available bandwidth when viewing video content through a wearable device, the first encoded video bitstream having a first resolution; A second encoding unit is configured to encode a second encoded video bitstream having a low resolution, the second encoded video bitstream having a background with a constant segment duration or size, and the second encoded video bitstream having a second resolution lower than the first resolution; as well as A creation unit is configured to create a streaming bitstream for decoding or presentation using the first encoded video bitstream and the second encoded video bitstream.

10. The apparatus according to claim 9, characterized in that, The first encoded video bitstream contains multiple segment sizes for the viewport.

11. The apparatus according to claim 9, characterized in that, The user transmits information about the head movement speed and the available bandwidth via Real-time Transmission Control Protocol (RTCP).

12. The apparatus according to claim 10 or 11, characterized in that, Request margin around the viewport with a segment size that is the same as or longer than the viewport.

13. The apparatus according to claim 12, characterized in that, Based on the user's head movement within the margin, a new tile containing a longer segment is requested to update the margin.

14. The apparatus according to claim 12, characterized in that, Since the resolution of the tiles in the margin is lower than that of the tiles in the viewport, a shorter, higher-resolution fragment is requested.

15. The apparatus according to claim 14, characterized in that, The shorter, higher-resolution segment is requested regardless of whether the user's head movement is within the margin.

16. The apparatus according to claim 12, characterized in that, Based on the viewport movement exceeding the margin, a shorter segment is requested.

17. A method for decoding a viewport bitstream, the method comprising: A first encoded video bitstream containing segment sizes defined for the viewport is decoded based on a threshold corresponding to the user’s head movement speed, the threshold depending on information received from the decoder regarding the user’s head movement speed and available bandwidth when viewing video content through a wearable device, the first encoded video bitstream having a first resolution; Decoding a second coded video bitstream with low resolution, the second coded video bitstream having a background with constant segment duration or size, the second coded video bitstream having a second resolution, the second resolution being lower than the first resolution; as well as A streaming bitstream for presentation is created using the decoding results of the first and second encoded video bitstreams.

18. A non-transitory computer-readable medium having stored thereon a computer program for segmenting a viewport bitstream, said computer program being configured to cause one or more computer processors to perform the method according to any one of claims 1-8.

19. A computer device, characterized in that, The device includes a processor and a memory. The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method according to any one of claims 1-8 according to the instructions in the program code.

20. A method for transmitting a video bitstream, characterized in that, The video bitstream is decoded according to the method of claim 17, or the video bitstream is generated according to any one of claims 1 to 8.

Images