Method and apparatus for reducing latency during viewport switching in immersive video

Abstract

Methods, apparatuses, systems, and articles of manufacture are disclosed to reduce latency during viewport switching in immersive video. An example apparatus includes at least one memory, instructions in the apparatus, and a processor circuit to execute the instructions to: obtain a first bitstream having a first encoded frame and a second encoded frame, the second encoded frame encoded at a higher resolution than the first encoded frame and having a coding dependency on the first encoded frame; rewrite the first bitstream to a second bitstream based on field of view information, the second bitstream including a third encoded frame indicating a portion of the second encoded frame corresponding to the field of view information, and including the first encoded frame; and transmit the second bitstream to a client device for decoding and rendering of the portion of the second encoded frame.

Description

Technical Field

[0001] This disclosure generally relates to immersive video, and more specifically to methods and apparatus for reducing latency during viewport switching in immersive video. Background Technology

[0002] In video compression / decompression (codec) systems, compression efficiency and video quality are important performance metrics. For example, in many video applications, visual quality is a crucial aspect of the user experience. Compression efficiency affects the storage space required to store video files and / or the bandwidth required to transmit and / or stream video content. Video encoders typically compress video information to enable the transmission of more information over a given bandwidth or the storage of more information within a given storage space, and so on. The compressed signal or data is then decoded by the decoder, which decodes or decompresses the signal or data to display it to the user. In most examples, higher visual quality is desirable in conjunction with greater compression. Attached Figure Description

[0003] Figure 1 This is a block diagram of an example video decoding process and a viewport-dependent distribution process for delivering immersive video in a time-sensitive manner.

[0004] Figure 2 It is by Figure 1 The example video decoding structure is included in the example distribution server implementation.

[0005] Figure 3 It is used to rewrite a bitstream for a given FOV. Figure 1 A block diagram of an example distribution server.

[0006] Figure 4 This is a block diagram of an example video decoding system for reconfiguring bitstreams for a given FOV.

[0007] Figure 5 This is a flowchart representing example machine-readable instructions that can be executed by example processor circuitry. Figure 1 and Figure 3 The distribution server generates instructions for configuring the encoding circuitry to encode the video in a specific manner.

[0008] Figure 6 and Figure 7 This is a flowchart representing example machine-readable instructions that can be executed by example processor circuitry. Figure 1 and Figure 3 The distribution server thus rewrites the bitstream for streaming that depends on the viewport.

[0009] Figure 8This is a block diagram of an example processing platform including processor circuitry configured to execute... Figure 5-7 Example machine-readable instructions to implement Figure 1 and Figure 3 The distribution server.

[0010] Figure 9 yes Figure 8 A block diagram of an example implementation of the processor circuit.

[0011] Figure 10 yes Figure 8 A block diagram of another example implementation of the processor circuit.

[0012] These figures are not drawn to scale. Generally, all (one or more) figures and accompanying written descriptions will use the same reference numerals to refer to the same or similar parts throughout.

[0013] Unless otherwise specifically indicated, descriptors such as “first,” “second,” “third,” etc., used herein do not in any way indicate any meaning of priority, physical order, arrangement in a list, and / or sorting, but are merely used to distinguish elements for ease of understanding of the labels and / or arbitrary names of the disclosed examples. In some examples, the descriptor “first” may be used in the detailed description to refer to an element, while different descriptors such as “second” or “third” may be used in the claims to refer to the same element. In such cases, it should be understood that such descriptors are only used to clearly identify those elements that may, for example, originally share the same name. As used herein, the phrase “in communication” (including variations thereof) covers direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or continuous communication, but additionally includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals, and / or one-off events. As used herein, “processor circuitry” is defined to include: (i) one or more special-purpose electrical circuits configured to perform one or more specific operations and comprising one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors); and / or (ii) one or more general-purpose semiconductor-based electrical circuits programmed with instructions to perform specific operations and comprising one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuitry include programmable microprocessors, field-programmable gate arrays (FPGAs) that can be instantiated with instructions, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), XPUs, or microcontrollers, as well as integrated circuits such as application-specific integrated circuits (ASICs). For example, an XPU may be implemented by a heterogeneous computing system that includes multiple types of processor circuitry (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc., and / or combinations thereof) and one or more application programming interfaces (APIs) that can assign one or more computing tasks to whichever of the multiple types of processing circuitry is best suited to perform the one or more computing tasks. Detailed Implementation

[0014] Immersive video decoding, such as high-resolution virtual reality (VR) and 360-degree video streaming (e.g., 8K or 12K flat video, 360-degree video, VR, AR, etc.), has become prevalent with the increase in internet capacity. In some examples, immersive video decoding utilizes high bandwidth. Immersive video is video that supports multiple views per frame, where multiple views correspond to different positions, orientations, etc. When displaying immersive video, many different views corresponding to the immersive video are provided to the display. For example, in 360-degree video, the rendering system (e.g., the client) can obtain a bitstream corresponding to the 360-degree video and output a view corresponding to the orientation of the display (e.g., as the display corresponding to a 90-degree window rotates, the view corresponding to the 90-degree window is displayed based on the rotation, allowing the rendering system to view the entire 360-degree video in a 90-degree window by rotating the device around the 360-degree landscape). In some examples, instead of providing the client with all views in the immersive video, only the content within a specific field of view (FOV) (e.g., the viewport) is provided and displayed. Streaming specific content associated with the FOV is called viewport-dependent streaming. Bandwidth can be saved by using viewport-dependent streaming (e.g., by streaming only the views that are to be displayed and not the views that are not displayed).

[0015] The Moving Picture Experts Group (MPEG) developed the Omnidirectional Media Format (OMAF), which defines a video format and packing method for viewport-dependent streaming. This format enables omnidirectional media applications that focus on 360-degree video, images, and audio, along with associated timed text, supporting three degrees of freedom. One method used in OMAF viewport-dependent streaming is slice-based viewport-dependent streaming. Slice-based viewport-dependent streaming encodes the projected picture into several slices. In OMAF, the projected picture is the result of generating a 2-D picture from a stitched spherical signal by applying the omnidirectional projection format on the content production side. OMAF utilizes two types of projection formats: equirectangular projection (ERP) and cubemap projection (CMP). In CMP, the spherical signal is projected linearly onto six square faces, which are then unfolded to form rectangles with a 3:2 aspect ratio, some of which are rotated to maximize the continuity of the edges across the faces. Region-based packing (RWP) is performed on the projected picture prior to slice-based viewport-dependent streaming encoding. In RWP, the projected image is resized, repositioned, rotated, and / or vertically / horizontally rotated before encoding. In some examples, the FOV is highlighted, selected, and labeled on the projection when the projected image is generated. In some examples, the FOV is not uniform on the projected image. For example, the FOV may be scattered across multiple unconnected areas on the projected image. In such examples, viewport encoding / decoding can be challenging for the encoder and / or decoder for viewport-dependent streaming.

[0016] During encoding, the projected image is segmented into pieces, which can be merged into a single bitstream and decoded using a single decoder instance. In some examples, a two-dimensional (2-D) video codec such as High Efficiency Video Decoding (HEVC) is used to define the piece size and to develop spatial and temporal redundancy for the geometric and texture components of these pieces and / or the projected image. As used herein, a piece is an isolated region that depends only on co-located isolated regions within the projected image and not on any other image regions. In some examples, several versions of the piece are encoded at different bitrates and / or resolutions. For example, the projected image can be encoded in two separate layers, each with the same pieces, but corresponding to different bitrates and / or resolutions. The decoded piece sequence, along with metadata describing the position of the pieces on the omnidirectional video, is made available for streaming.

[0017] Viewport-dependent streaming operates based on interaction between the client and server. For example, the client sends a viewport to the server, and the server responds to the client with content for that given viewport. This client-server interaction must occur within a short timeframe to provide a smooth viewport switching and / or content switching experience. If the interaction is not short-lived, the client's user may see blurry, misaligned, and / or outdated content. As used herein, the content can be any type of media, such as streaming media, broadcast media, 360-degree video content, etc.

[0018] In some examples, time dependencies in video encoding can affect the speed and / or smoothness of interaction between the client and server. For instance, when a new viewport is received on the server side, the server must package the new content and transmit it to the client. For such operations to be performed quickly, encoding (e.g., packaging) is done on keyframes within the content, where keyframes have little time dependency on previous frames. In HEVC, time dependencies are typically developed and utilized for efficient encoding, and therefore, the server uses more time to encode new content, which increases the latency of interaction between the client and server. Furthermore, packaging operations are applied to specific parts, segments, or sections of content (e.g., for viewport-dependent streaming). Therefore, due to time dependencies and the specificity of which content is being encoded, timely encoding and transmission of new content can become challenging for the server.

[0019] Several methods have been developed to overcome the latency issues of viewport-dependent streaming. One approach involves identifying the boundaries around the content of the FOV. A drawback of this boundary-based approach is that it only covers slow viewport changes. In some examples, methods involve implementing smaller Groups of Pictures (GOPs) during encoding. A GOP is a group of pictures that are independent of any other frames outside the group and can be decoded independently. A disadvantage of the smaller GOP approach is that smaller GOPs can lead to increased bitrate even when the viewport is not changing. Some methods involve implementing multiple streams, such as a low-resolution stream for the full view and a high-resolution stream for the content within the FOV. The disadvantages of the multi-stream approach include the requirement for region-by-region packing (RWP) for the high-resolution stream and encoding before it is sent to the client. Some methods involve implementing multiple streams (e.g., a low-resolution stream and multiple high-resolution streams for the content within the FOV). High-resolution streams can have different GOP sizes. For example, a high-resolution stream can utilize a long GOP size when the viewport is not changing, and a short GOP size when the viewport is changing. The drawback of this approach is that encoding these multiple streams requires more computation and complex switching logic.

[0020] Conventional methods for overcoming viewport-dependent streaming latency issues reduce time dependence, but at the cost of increased bandwidth, redundant data, etc. For example, in the method mentioned above, the server must wait for the Group of Optical Blocks (GOPs) to begin for viewport switching so that the content can be decoded.

[0021] The examples disclosed herein provide a video decoding architecture that minimizes temporal dependencies and provides a low-latency viewport switching solution for immersive video streaming that depends on the viewport. The examples disclosed herein simplify the delivery architecture of the MPEG-I OMAF standard by encoding frames as having a spatially dependent layer and a base layer, where frames in the spatially dependent layer are spatially dependent on the base layer but temporally independent of previous frames in the spatially dependent layer. In the examples disclosed herein, encoders are configured to implement uniform slice sizes / regions across frames (e.g., projected images) and encode these slices at different resolutions. For example, some methods, devices, and systems disclosed herein segment frames into equal, isolated regions and encode these regions at two or more different resolutions, generating frames in the base layer and frames in the spatially dependent layer.

[0022] The examples disclosed in this paper obtain encoded frames and identify the portion of a frame to be extracted for a specific field of view (FOV) based on information from the client. The examples disclosed in this paper extract frames from the spatial dependency layer based on the FOV and repackage the frames. Encoded frames in the base layer are not extracted for a specific FOV. The examples disclosed in this paper associate the extracted and repackaged frames in the spatial dependency layer with the corresponding frames in the base layer and transmit the associated frames as a bitstream to the client. The examples disclosed in this paper improve latency issues in viewport-dependent streaming by removing temporal dependencies between frames in the spatial dependency layer, while still facilitating viewport switching.

[0023] Figure 1 This is a block diagram of an example video decoding process 100 and a viewport-dependent distribution process for delivering immersive video in a time-sensitive (e.g., low-latency) manner. The example video decoding process 100 includes an example media provider processing side 102, an example server processing side 104, an example network delivery side 106, and an example client processing side 108. The example media provider processing side 102 implements an example encoding circuit 110. The example server processing side 104 implements an example distribution server 112. The example client processing side 108 implements an example client device 114.

[0024] exist Figure 1In this example, the media provider processing side 102 implements the first stage of the video decoding process 100 to encode media such as the example immersive video requested by the example client device 114. In this example, the media provider processing side 102 appears at a media provider, such as a specific media provider server that contains and / or stores multiple media. In some examples, the media may be immersive video. Additionally and / or alternatively, the media may be any type of audio and / or video requested by the client device 114.

[0025] Media provider processing side 102 implements encoding circuitry 110 to encode immersive video into a first bitstream 116 and provides the first bitstream 116 to example server processing side 104. In this example, the first bitstream 116 is a bit sequence generated by encoding (e.g., projection, packing, and encoding) a frame sequence corresponding to the entire 360-degree view. Example encoding circuitry 110 implements the MPEG-I OMAF standard for encoding media such as immersive video. Additionally and / or alternatively, encoding circuitry 110 may utilize any type of video decoding standard to encode media. In some examples, encoding circuitry 110 receives configuration instructions from server processing side 104 indicating how to encode immersive video. For example, encoding circuitry 110 is configured to encode immersive video such that the encoded immersive video (e.g., the first bitstream 116) can be efficiently processed and rewritten by server processing side 104, which is implementing low-latency viewport-dependent streaming. The following is in conjunction with... Figure 1 and Figure 5 The example configuration directives are described in further detail.

[0026] exist Figure 1 In the example, server processing side 104 implements the second stage of video decoding process 100 to rewrite the bitstream from media provider processing side 102 (e.g., bitstream 116). In some examples, the second stage of video decoding process 100 is a viewport-dependent streaming stage. For example, server processing side 104 rewrites the bitstream from media provider processing side 102 into a new bitstream indicating the FOV of the immersive video.

[0027] Example server processing side 104 includes example distribution server 112 to rewrite a bitstream (e.g., bitstream 116) from media provider processing side 102 into a new bitstream indicating the field of view (FOV). Example distribution server 112 generates configuration instructions for example encoding circuitry 110, specifying the particular encoding tasks to be performed by example encoding circuitry 110 during the generation of the first bitstream 116. In some examples, the configuration instructions enable distribution server 112 to rewrite the first bitstream 116 into a second bitstream 118, which indicates the FOV and can be efficiently decoded regardless of viewport switching. In some examples, distribution server 112 obtains example FOV parameter 120 from client processing side 108. Example distribution server 112 may utilize FOV parameter 120 to identify segments (e.g., slices, isolated regions, etc.) in the encoded media to be extracted and encapsulated into the second bitstream 118. In this example, the second bitstream 118 is a bit sequence generated by encoding (e.g., projection, packing, and encoding) a frame sequence corresponding to a specific segment and / or view in the entire 360-degree view. The following is in conjunction with... Figure 3 Example distribution server 112 is described in further detail.

[0028] In some examples, the distribution server 112 includes components for rewriting the bitstream. For example, the rewriting components may be implemented via server circuitry 112. In some examples, server circuitry 112 may be implemented via machine-executable instructions, such as at least those executed by processor circuitry. Figure 5 Boxes 502, 504, 506, and 508 Figure 6 Boxes 602, 604, 606 and 608, and / or Figure 7 The processor circuitry can be implemented using frames 702, 704, 706, 708, 710, 712, and / or 714. Figure 8 Example processor circuit 812 Figure 9 Example processor circuit 900, and / or Figure 10 The example is implemented using a field-programmable gate array (FPGA) circuit 1000. In other examples, the server circuit 112 is implemented using other hardware logic circuits, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the server circuit 112 can be implemented using at least one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGA, application-specific integrated circuit (ASIC), comparator, operational amplifier (op-amp), logic circuitry, etc.) configured to perform the corresponding operation without executing software or firmware, but other structures are equally suitable.

[0029] exist Figure 1In this example, the network transmission side 106 implements the third stage of the video decoding process 100 to transfer information between the distribution server 112 and the client device 114. In some examples, the network transmission side 106 is implemented via the Internet. For example, the distribution server 112 and the client device 114 may be connected via a network such as the Internet. Additionally and / or alternatively, the example network transmission side 106 may be implemented via a hardwired connection, a short-range wireless connection (e.g., Bluetooth®), etc. The example network transmission side 106 transmits FOV parameters 120 to the example distribution server 112 and transmits the FOV bitstream (e.g., a second bitstream 118) to the example client device 114.

[0030] exist Figure 1 In this example, the example client processing side 108 implements the fourth stage of the video decoding process 100 to decode the bitstream from the server processing side 104. The example client processing side 108 includes a client device 114 to decode a bitstream (e.g., a second bitstream 118) from the distribution server 112. In this example, the client device 114 is an extended reality headset. Additionally and / or alternatively, the example client device 114 can be a television, smartphone, computer (e.g., laptop, desktop, tablet, etc.), and / or any type of computing device capable of streaming and / or displaying immersive video. In some examples, the FOV parameter 120 is modified based on the client device 114. In some examples, a viewport switch occurs when the FOV parameter changes. In some examples, the viewport switch occurs on the client device 114 instead of the distribution server 112. For example, the second bitstream 118 includes information that enables the client device 114 to decode a new FOV that was not initially used to generate the second bitstream 118. The following is in conjunction with... Figure 2 , Figure 3 and Figure 7 The information in the second bitstream 118 is described in further detail.

[0031] exist Figure 1In this example, the example client device 114 can generate FOV parameters 120, which are values ​​(e.g., angles) representing a region and / or position in a sphere representing 360-degree video projected onto a two-dimensional plane (e.g., 2D ERP, CubeMap projection). FOV parameters 120 include yaw, pitch, and roll angles. Yaw, pitch, and roll angles are angles in the sphere about the X-axis, Y-axis, and Z-axis, respectively. In some examples, the client device 114 determines the yaw, pitch, and roll angles based on one or more sensors monitoring the movement of the client device 114. The example distribution server 112 uses the yaw, pitch, and roll angles to determine a region in the two-dimensional plane representing immersive video to generate a second bitstream 118. The example client device 114 can use the FOV parameters 120 to subsequently identify the FOV to be decoded from the second bitstream 118.

[0032] Reducing the interaction latency between server processing side 104 and client processing side 108 reduces the likelihood that a delayed or misaligned view on client processing side 108 is detectable by the human eye. In some examples, to avoid detecting a delayed view and to provide the user with a fully immersive experience, the interaction latency between server 112 and client 114 should be maintained at approximately 25 milliseconds (ms). To determine the total latency between server 112 and client 114, the total latency calculation is broken down into three distinct parts: server processing time, network transmission time (in both directions), and client processing time.

[0033] In some examples, the network transmission time occurring on the network transmission side 106 is determined by the network architecture and the distance between server 112 and client 114. Adopting modern network infrastructure such as 5G wireless technology standards and edge computing will reduce network transmission time.

[0034] In some examples, the client processing time occurring on the client processing side 108 is determined by the time spent by the client device 114 decoding and rendering the immersive video. In some examples, this amount of time depends on the type of hardware implemented by the client device 114 and the different hardware configurations available to the client device 114. In some examples, using improved processors such as multi-core CPUs, GPUs, and other accelerators will reduce client processing time.

[0035] In some examples, the server processing time occurring on the server processing side 104 is determined based on the Group of Pictures (GOP) size. In this example, the GOP can be interpreted as follows: Using a previous keyframe as a reference, at regular intervals, frames such as I-frames and / or P-frames (described below) are processed. Figure 2(Further details will be provided below.) Key picture decoding is performed in a hierarchical manner. Images between two key pictures are predicted hierarchically and, together with subsequent key pictures, are referred to as a Group of Pictures (GOP). The sequence of key pictures represents the group of images decoded at the lowest temporal and spatial (e.g., temporal base layer) level, which can be augmented by partitioning non-key pictures into one or more augmentation layers (e.g., those layers with a different resolution than the base layer). The following is combined with... Figure 2 These layers in the video decoding structure will be described in further detail.

[0036] The larger the GOP, the longer the server processing time. For example, when server 112 receives a new viewport and / or viewport change from client 114, server 112 waits until it receives the keyframe (e.g., I-frame) from media provider processing side 102 before generating the FOV for immersive video. When the GOP size is 30 in a 30 frames per second (FPS) video stream, the average latency for the keyframe is approximately 500ms, which is far higher than the latency required for a fully immersive experience for the user. For example, the average latency can be determined by dividing 1 second by 2, resulting in 500ms. The average latency can be considered as a uniform distribution within (0, GOP / FPS). Therefore, the example distribution server 112 implements a method to reduce the GOP size in the decoded video by removing temporal redundancy in the enhancement layer of the video decoding structure.

[0037] In an example operation of the video decoding process 100, an example encoding circuit 110 located on the media provider processing side 102 receives a request and configuration instructions from an example distribution server 112 located on the server processing side 104. The request instructs that media (e.g., immersive video) be streamed to the client device 114, and the configuration instructions instruct how the media (e.g., immersive video) to be streamed to the client device 114 be encoded. The example encoding circuit 110 encodes the media (e.g., immersive video) and encapsulates the encoded media into a first bitstream 116. The first bitstream 116 represents a complete 360-degree frame in the immersive video. The example encoding circuit 110 transmits the first bitstream 116 to the example distribution server 112 located on the server processing side 104.

[0038] Example distribution server 112 rewrites the first bitstream 116 into a second bitstream 118 indicating the viewport and / or field of view (FOV) of a complete 360-degree image. Example distribution server 112 generates the second bitstream 118 based on FOV parameters 120 obtained from client device 114 located on client processing side 108. For example, distribution server 112 extracts segments (e.g., slices, isolated regions, etc.) indicating FOV parameters 120 from the high-resolution encoded image contained in the first bitstream 116. In this example, distribution server 112 associates the low-resolution encoded image contained in the first bitstream 116 and representing a complete 360-degree image with the extracted segments of the high-resolution encoded image contained in the first bitstream 116 to generate the second bitstream 118. Example distribution server 112 transmits the second bitstream 118 to example client device 114 located on client processing side 108 using network transport side 106.

[0039] Example client device 114 acquires and decodes the second bitstream 118. In some examples, client device 114 renders the images and / or video in the decoded second bitstream 118. When example client device 114 detects a viewport switch, it uses the low-resolution encoded image in the second bitstream 118 to decode the new FOV. In this way, client device 114 does not need to wait on distribution server 112 for a new bitstream with the new FOV to be encoded and transmitted.

[0040] Figure 2 It is by Figure 1 Example distribution server 112 implements example video decoding structure 200. Example video decoding structure 200 represents a viewport-dependent bitstream generated by example distribution server 112 (e.g., Figure 1 The second bitstream 118). The video decoding structure 200 includes an example first layer 202 and an example second layer 204. The example first layer 202 includes example first frames 206A, 206B, 206C and 206D (collectively referred to as first frame 206), while the example second layer 204 includes example second frames 208A, 208B, 208C and 208D (collectively referred to as second frame 208).

[0041] exist Figure 2 In the example, the first frame 206 and the second frame 208 are... Figure 1The example encoding circuit 110 provides the encoded picture. In some examples, the first frame 206A in the first frame is an I-frame. An I-frame, also known as an internally decoded picture, is a complete image like a JPEG and / or BMP image file. An I-frame does not require decoding additional information from other video frames. In some examples, the second frame 206B in the first frame is a P-frame. A P-frame, also known as a prediction picture, is a frame that contains image changes relative to the previous frame (e.g., I-frame 206A). P-frames can be decompressed using data from previous frames, and P-frames are more compressible than I-frames. In some examples, the first frame 208A in the second frame is a high-resolution version of the first frame 206A in the first frame. For example, to generate the first frame 208A in the second frame, the encoding circuit 110 encodes the first frame 206A in the first frame at a higher resolution. Similarly, in some examples, the second frame 208B in the second frame is a high-resolution version of the second frame 206B in the first frame. For example, encoding circuit 110 generates second frame 208B in the second frame by encoding second frame 206B in the first frame at a higher resolution.

[0042] exist Figure 2 In the example, layer 202 is the base layer, and layer 204 is the enhancement layer. For example, as mentioned above, layer 202 includes a first frame 206 encoded at a resolution lower than that of the second frame 208 in layer 204. In some examples, the video decoding structure 200 may include a third, fourth, fifth, etc., and is not limited to two layers.

[0043] exist Figure 2 In the example, the first layer 202 includes time dependencies 210A, 210B, and 210C, collectively referred to as time dependency 210. For example, the first frame 206 in the first layer 202 is temporally dependent on each other. For example, the second frame 206B in the first frame is temporally dependent on the first frame 206A in the first frame (indicated by the first time dependency 210A), the third frame 206C in the first frame is temporally dependent on the second frame 206B in the first frame (indicated by the second time dependency 210B), and the fourth frame 206D in the first frame is temporally dependent on the third frame 206C in the first frame (indicated by the third time dependency 210C). Time dependency, also known as temporal redundancy, occurs when pixels in two video frames have the same value at the same location. For example, a pixel in the second frame 206B in the first frame may have the same value and location as a pixel in the first frame 206A in the first frame. In some examples, the encoding circuit 110 is configured to construct a time-dependent first layer 202 using configuration instructions from the distribution server 112.

[0044] exist Figure 2In this context, the second layer 204 includes spatial dependencies 212A, 212B, 212C, and 212D, collectively referred to as spatial dependency 212. For example, the second frame 208 is spatially dependent on the first frame 206. Specifically, the first frame 208A in the second frame is spatially dependent on the first frame 206A in the first frame (indicated by the first spatial dependency 212A), the second frame 208B in the second frame is spatially dependent on the second frame 206B in the first frame (indicated by the second spatial dependency 212B), the third frame 208C in the second frame is spatially dependent on the third frame 206C in the first frame (indicated by the third spatial dependency 212C), and the fourth frame 208D in the second frame is spatially dependent on the fourth frame 206D in the first frame (indicated by the fourth spatial dependency 212D). Spatial dependency, also known as spatial redundancy, occurs when elements (such as pixels in a still image and bit patterns in a file) are repeated within a structure. For example, pixels from the first frame 206A of the first frame are repeated in the first frame 208A of the second frame. Therefore, the first frame 208A of the second frame is a copy of the first frame 206A of the first frame, but encoded at a higher resolution. In some examples, using configuration instructions from the distribution server 112, the encoding circuit 110 is configured to construct a second layer 204 with only spatial dependencies from the first layer 202. The encoding circuit 110 does not construct a second layer 204 with temporal dependencies.

[0045] exist Figure 2 In this example, the first frame 206 and the second frame 208 are encoded into a slice. For example, the encoding circuit 110 identifies dimensions and segments the projected image (e.g., a video frame) into slices. Figure 2 In the example, the second frame 208 includes slices T1, T2, T3, T4, T5, T6, T7, T8, and T9. The example first frame 206 also includes slices T1, T2, T3, T4, T5, T6, T7, T8, and T9 (not shown). The slices in the second frame 208 are encoded at a higher resolution than the slices in the first frame 206. In some examples, the distribution server 112 provides configuration instructions to the encoding circuit 110 to segment the frame into slices and encode the slices at different resolutions.

[0046] exist Figure 2 In the second frame 208, slices T1, T2, T3, T4, T5, T6, T7, T8, and T9 represent complete flat (e.g., projected) frames in the immersive video. For example, slices T1, T2, T3, T4, T5, T6, T7, T8, and T9 may include pixels constituting a spherical signal that has been linearly projected onto six square faces and unfolded to form a rectangle with an aspect ratio of 3:2, where some faces are rotated to maximize the continuity of the edges across the faces.

[0047] exist Figure 2 In the second frame, the first frame 208A contains shadow tiles T1, T2, T4, and T5. These shadow tiles indicate a viewport. For example, among all tiles T1, T2, T3, T4, T5, T6, T7, T8, and T9 that define a complete video frame, shadow tiles T1, T2, T4, and T5 define the FOV that the user of client device 114 may currently be viewing. In the second frame 208B, the viewport has changed relative to the viewport of the first frame 208A. For example, the second frame 208B contains shadow tiles T5, T6, T8, and T9 that define the new FOV. The third frame 208C contains a viewport different from that of the first and second frames 208A and 208B. For example, the third frame 208C contains shadow tiles T4, T7, T6, and T9 that define a different FOV. The fourth frame 208D in the second frame contains a viewport different from the first frame 208A, the second frame 208B, and the third frame 208C in the second frame. For example, the fourth frame 208D in the second frame contains shadow tiles T4, T7, and T9 that define different FOVs.

[0048] Example distribution server 112 generates second bitstream 118 by extracting the slices corresponding to the FOV from second frame 208 and repackaging them into new frames. Figure 2 The new frames will be grouped together in the second bitstream 118. The example distribution server 112 does not extract the FOV slice from the first frame 206. The example distribution server 112 associates the new frames with the corresponding first frames to generate the final bitstream (e.g., the second bitstream 118) to be sent to the client device 114.

[0049] Advantageously, the video decoding structure 200 saves 106 (units) on the network transmission side by ensuring that only high-quality segments within the FOV are transmitted. Figure 1 The bandwidth in the second layer 204. Advantageously, by eliminating temporal redundancy / dependency in the second layer 204, the video decoding structure 200 makes it possible for the client device 114 to switch to any slice in the second layer 204 (e.g., the enhancement layer) at any time. Advantageously, the video decoding structure 200 enables the decoding of non-adjacent regions in a frame for a given FOV (e.g., non-adjacent slices, such as those FOV slices in the third frame 208C and the fourth frame 208D in the second frame).

[0050] Figure 3 It is used to rewrite a bitstream for a given FOV. Figure 1A block diagram of the example distribution server 112. The example distribution server 112 includes example configuration circuitry 302, example interface 304, example field of view (FOV) data storage component (datastore) 306, example frame data storage component 308, example chip selection circuitry 310, example repackaging circuitry 312, and example bitstream configuration circuitry 314.

[0051] exist Figure 3 In the example, configuration circuit 302 generates information for the encoding circuit (e.g., Figure 1 The encoding circuit 110 provides instructions, such as configuration instructions, on how to process and encode the immersive video. Example configuration circuit 302 generates first instructions for configuring the encoding circuit to encode the projected image into slices, the first instructions defining at least two resolutions for the slices. For example, the first instructions could cause the encoding circuit to slice the projected image at a first resolution and a second resolution, where the first resolution is lower than the second resolution. The first instructions cause the encoding circuit to segment the projected image encoded at the first resolution into slices with the same number of rows and columns as the slices segmented at the second resolution. In some examples, the size of the slices encoded at the first resolution is proportional to the size of the slices encoded at the second resolution. For example, the first projected image can be encoded at 8K resolution (e.g., 8000 pixels), and the second projected image can be encoded at 4K resolution (e.g., 4000 pixels). The first and second projected images are segmented into equal numbers of rows and columns, and therefore, the number of pixels in each slice of the first projected image will be four times the number of pixels in each slice of the second projected image (e.g., a slice ratio of 1:4).

[0052] Example configuration circuit 302 generates a second instruction for configuring the encoding circuit to construct a second layer of chip groups that are time-independent and correspond to at least one of two resolutions. For example, configuration circuit 302 causes the encoding circuit to eliminate inter-chip groups encoded at higher resolutions (e.g., Figure 2 The temporal dependence between the second frame (208). As used in this paper, a slice group is a frame of immersive video that has been encoded into segments. In some examples, a slice group may be referred to as a frame. For example, Figure 2 The first frame 208A in the second frame shows frame 208A consisting of a group of pieces.

[0053] In some examples, configuration circuitry 302 provides instructions to encoding circuitry via interface 304. For example, configuration circuitry 302 is communicatively coupled to interface 304 via example bus 316. Therefore, example configuration circuitry 302 can instruct example interface 304 to encapsulate instructions (e.g., generate data packets, network packets, etc.) and transmit them to encoding circuitry. In some examples, configuration circuitry 302 responds to streaming requests dependent on the viewport by transmitting first and second instructions. For example, interface 304 initializes configuration circuitry 302 to generate responses from client devices (e.g., ...). Figure 1 The client device 114) is a user's instruction to stream immersive video.

[0054] In some examples, configuration circuitry 302 includes components for generating configuration instructions. For example, the components for generating instructions can be implemented via configuration circuitry 302. In some examples, configuration circuitry 302 can be implemented via machine-executable instructions, such as at least those executed by processor circuitry. Figure 5 The processor circuitry can be implemented using boxes 502 and 504. Figure 8 Example processor circuit 812 Figure 9 Example processor circuit 900, and / or Figure 10 Example Field Programmable Gate Array (FPGA) circuit 1000 is used. In other examples, configuration circuit 302 is implemented using other hardware logic circuits, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, configuration circuit 302 can be implemented using at least one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGA, application-specific integrated circuit (ASIC), comparator, operational amplifier (op-amp), logic circuitry, etc.) configured to perform the corresponding operation without executing software or firmware, but other structures are equally suitable.

[0055] exist Figure 3 In the example interface 304, data is passed from the example encoding circuit (e.g., Figure 1 The encoding circuit 110) and / or example client device (e.g., Figure 1Information from the client device 114. Example interface 304 can be implemented using hardware conforming to any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB) interface, a Bluetooth® interface, a Near Field Communication (NFC) interface, a PCI interface, and / or a PCIe interface. Example interface 304 can receive requests for streaming depending on the viewport, transmit configuration commands, obtain bitstreams, transmit bitstreams, obtain FOV parameters, etc. In some examples, interface 304 stores communication information in appropriate data storage components (e.g., field-of-view data storage component 306 and / or frame data storage component 308).

[0056] In some examples, interface 304 includes components for communication, components for transmission, and / or components for acquisition. For example, the components for communication may be implemented via interface circuitry 304. In some examples, interface circuitry 304 may be implemented via machine-executable instructions, such as at least those executed by processor circuitry. Figure 5 Boxes 506 and 508 and / or Figure 6 The processor circuitry can be implemented using boxes 602, 604, and 608. Figure 8 Example processor circuit 812 Figure 9 Example processor circuit 900, and / or Figure 10 The example is implemented using a field-programmable gate array (FPGA) circuit 1000. In other examples, the interface circuit 304 is implemented using other hardware logic circuits, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the interface circuit 304 can be implemented using at least one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGA, application-specific integrated circuit (ASIC), comparator, operational amplifier (op-amp), logic circuitry, etc.) configured to perform the corresponding operation without executing software or firmware, but other structures are equally suitable.

[0057] In some examples, the components used for determination include those for use with the client device (e.g., Figure 1 The client device 114) is a component that obtains field of view information.

[0058] exist Figure 3 In the example, field-of-view data storage component 306 stores and / or records data from one or more client devices (e.g., ...). Figure 1The client device 114) and / or example interface 304 data. The example field-of-view data storage component 306 includes field-of-view parameters. The field-of-view data storage component 306 of this example may be implemented using volatile memory (e.g., synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS dynamic random access memory (RDRAM), etc.) and / or non-volatile memory (e.g., flash memory). The field-of-view data storage component 306 may additionally or alternatively be implemented using one or more double data rate (DDR) memories (such as DDR, DDR2, DDR3, DDR4, mobile DDR (mDDR), etc.). The field-of-view data storage component 306 may additionally or alternatively be implemented using one or more mass storage devices such as: (one or more) hard disk drives, (one or more) compact disc (CD) drives, (one or more) digital versatile disc (DVD) drives, (one or more) solid-state drives, etc. Although the field-of-view data storage component 306 is shown as a single data storage component in the example shown, the field-of-view data storage component 306 can be implemented by any number and / or (one or more) types of data storage components. Furthermore, the data stored in the field-of-view data storage component 306 can take any data format, such as, for example, binary data, comma-delimited data, tab-delimited data, Structured Query Language (SQL) structures, etc.

[0059] exist Figure 3 In the example, the frame data storage component 308 stores and / or records data from the encoding circuitry (e.g., Figure 1The example frame data storage component 308 includes frame data, chip group data, and / or bit stream data. The example frame data storage component 308 can be implemented using volatile memory (e.g., Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM), etc.) and / or non-volatile memory (e.g., flash memory). The frame data storage component 308 can be additionally or alternatively implemented using one or more Double Data Rate (DDR) memories (such as DDR, DDR2, DDR3, DDR4, Mobile DDR (mDDR), etc.). The frame data storage component 308 can be additionally or alternatively implemented using one or more mass storage devices such as: (one or more) hard disk drives, (one or more) compact disc (CD) drives, (one or more) digital versatile disc (DVD) drives, (one or more) solid-state drives, etc. Although the frame data storage component 308 is shown as a single data storage component in the example shown, the frame data storage component 308 can be implemented by any number and / or (one or more) types of data storage components. Furthermore, the data stored in the frame data storage component 308 can take any data format, such as, for example, binary data, comma-delimited data, tab-delimited data, Structured Query Language (SQL) structures, etc.

[0060] exist Figure 3 In the example chip selection circuit 310, the chip selection circuit 310 needs to identify the chip in the chip group that corresponds to the FOV. The example chip selection circuit 310 uses the FOV parameter (e.g., ... Figure 1The FOV parameter 120 is used to identify which slices to extract and to generate a second bitstream. For example, slice selection circuit 310 identifies a mapping from the FOV parameter to a list of slices within the viewer's FOV. In some examples, such as 360-degree video, the FOV parameter indicates yaw, pitch, and roll angles. In such examples, slice selection circuit 310 maps these angles to a two-dimensional plane (e.g., a 2D ERP, a CubeMap plane, etc.). The yaw, pitch, and roll angles are used to determine a region in a sphere representing the 360-degree video (e.g., a slice's region), which is then projected onto a two-dimensional plane (e.g., a 2D ERP, a CubeMap plane, etc.). Example slice selection circuit 310 selects slices in this region that indicate the FOV. The selected slices are slices in a first encoded frame, which corresponds to a first resolution. In some examples, the first resolution is a high resolution relative to a second encoded frame encoded at a lower resolution. Therefore, example slice selection circuit 310 selects slices from frames encoded at a higher resolution. As used herein, "resolution" is a term used to describe the number of pixels used when constructing a frame. Frames with higher resolution consist of a larger number of pixels compared to those with lower resolution.

[0061] Example slice selection circuit 310 extracts selected slices from the first coded frame. In some examples, slice selection circuit 310 utilizes the syntax of a video decoding structure (e.g., AV1 Scalable Video Decoding (SVC)) to extract the selected slices. In some examples, depending on the video decoding structure, slice selection circuit 310 separates slices identified as being in the FOV from other slices in the slice group corresponding to the entire frame (e.g., the entire 360-degree video frame). For example, slice selection circuit 310 may label and / or mark slices not in the FOV as “skipped” slices, meaning these slices will not be decoded. Additionally and / or alternatively, repackaging circuit 312 and / or bitstream configuration circuit 314 label unselected slices as slices to be “skipped”. Example slice selection circuit 310 provides the selected slices to example repackaging circuit 312.

[0062] In some examples, components for selecting a chip and / or for retrieving a chip are included. For example, the components for selecting a chip and / or for retrieving a chip may be implemented by chip selection circuitry 310. In some examples, chip selection circuitry 310 may be implemented by machine-executable instructions, such as at least those executed by processor circuitry. Figure 7 The 704 and 706 blocks can be used to implement the processor circuitry. Figure 8 Example processor circuit 812 Figure 9 Example processor circuit 900, and / or Figure 10The example is implemented using a field-programmable gate array (FPGA) circuit 1000. In other examples, the chip select circuit 310 is implemented using other hardware logic circuits, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the chip select circuit 310 can be implemented using at least one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGA, application-specific integrated circuit (ASIC), comparator, operational amplifier (op-amp), logic circuitry, etc.) configured to perform the corresponding operation without executing software or firmware, but other structures are equally suitable.

[0063] In some examples, the components used for determination include components for selecting those slices in the slices that correspond to the field of view information to generate coded frames.

[0064] exist Figure 3 In this example, the repackaging circuit 312 repackages the extracted slices into a third frame, which is encoded at a higher resolution and contains only the selected slices representing the field of view (FOV). For example, the repackaging circuit 312 generates a new frame encoded at a higher resolution that contains only the FOV. In some examples, the repackaging circuit 312 implements region-based packing (RWP) to repackage the slices selected by the slice selection circuit 310. Additionally and / or alternatively, the example repackaging circuit 312 may implement any type of algorithm and / or packing criterion to repackage the selected slices.

[0065] In some examples, the repackaging circuit 312 includes components for repackaging the die. For example, the components for repackaging can be implemented by the repackaging circuit 312. In some examples, the repackaging circuit 312 can be implemented by machine-executable instructions, such as at least those executed by processor circuitry. Figure 7 The 708 block is used to implement this, and the processor circuit can be implemented through... Figure 8 Example processor circuit 812 Figure 9 Example processor circuit 900, and / or Figure 10 The example is implemented using a Field Programmable Gate Array (FPGA) circuit 1000. In other examples, the repackaged circuit 312 is implemented using other hardware logic circuits, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the repackaged circuit 312 can be implemented using at least one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGA, Application-Specific Integrated Circuit (ASIC), comparator, operational amplifier, logic circuitry, etc.) configured to perform the corresponding operation without executing software or firmware, but other structures are equally suitable.

[0066] exist Figure 3 In this example, the example bitstream configuration circuit 314 generates a bitstream corresponding to the field of view (FOV). The example bitstream configuration circuit 314 uses frames repackaged by the example repackaging circuit 312 to generate the bitstream. In some examples, the bitstream configuration circuit 314 implements a video decoding structure such as AV1 SVC to generate the bitstream. In the example video decoding structure, slices and / or frames are packetized into Open Bitstream Units (OBUs). Each OBU contains a header that provides information identifying the payload of the bitstream (e.g., the type of information transmitted in the bitstream, such as encoded frames). Examples of OBU types that may exist in the bitstream include sequence header OBUs, frame header OBUs, metadata OBUs, time delimiter OBUs, and slice group OBUs. A frame OBU includes a frame header and a slice group OBU packed into one OBU.

[0067] In some examples, when the chip selection circuit 310 selects a chip, the repackaging circuit 312 encapsulates the chip into a chip group OBU. In such examples, the repackaging circuit 312 identifies the frame OBU corresponding to the chip group OBU and associates the chip group OBU with the frame OBU. After all chips in the FOV have been encapsulated in the various chip group OBUs, the repackaging circuit 312 associates the chip group OBU with its corresponding frame OBU. In some examples, there are multiple frame OBUs in the bitstream. Therefore, the example bitstream configuration circuit 314 sorts these frame OBUs in a certain order, in which they will be rendered and / or displayed and / or decoded. In some examples, the chip group OBU includes chip group (tg) start information and tg end information. The tg start information and tg end information can be numerical values ​​indicating the position of these chips in the chip group. In some examples, the tg start and tg end information can be used to calculate and / or determine the position of the chip. In some examples, the position value of the chip is a numerical value corresponding to the position of the total number of chips in the chip group.

[0068] In some examples, two frame OBUs correspond to each other. For example, a frame can be encoded at a first resolution and a second resolution, resulting in two frames. Example bitstream configuration circuit 314 associates these two frames in the bitstream based on time delimiter OBUs. A time delimiter OBU is an indication that any subsequent OBU will have a different rendering / decoding timestamp than the frame preceding the time delimiter. The time delimiter OBU associates frames in the first layer with frames in the second layer by indicating that those frame OBUs following the first time delimiter OBU and ending at the second time delimiter OBU have the same decoding timestamp.

[0069] Example bitstream configuration circuitry 314 includes metadata OBUs in the bitstream to map slices to positions within frames. For example, metadata OBUs can be defined to calculate and / or determine slice positions based on tg start and tg end information. Example bitstream configuration circuitry 314 generates metadata to determine the positions of one or more slices in frames encoded at lower resolution that are not defined for a specific FOV. Example bitstream configuration circuitry 314 utilizes these slice positions to enable a decoder (e.g., client device 114) to: 1) skip slices that do not need to be decoded for a specific FOV, and 2) use the skipped slices when the viewport changes. In some examples, bitstream configuration circuitry 314 configures a new bitstream (e.g., an FOV bitstream, a second bitstream 118, etc.) to have skippable slices. Configuring a new bitstream to have skippable slices may involve setting and / or removing flags corresponding to slice skipping in the bitstream. For example, if the bitstream configuration circuit 314 implements the AV1 SVC standard, it disables cumulative distribution function (CDF) updates at the frame level and sets the global motion vector (MV) flag to zero. An example bitstream configuration circuit 314 implementing the AV1 SVC standard sets the transform skip parameter to true at the slice level and sets the "use global MV" flag to true. In different video decoding structures, the example bitstream configuration circuit 314 sets and disables different flags to ensure that slices in lower-resolution frames are skippable.

[0070] Example bitstream configuration circuit 314 packets the encoded frames corresponding to the FOV (e.g., frame group OBUs containing only slices in the FOV), the encoded frames corresponding to the lower resolution (e.g., frame group OBUs containing all slices), time delimiters, and metadata into a new bitstream (e.g., a second bitstream 118). Example bitstream configuration circuit 314 then sends the new bitstream to a client (e.g., client device 114) for decoding.

[0071] In some examples, the bitstream configuration circuit 314 includes components for configuring the bitstream. For example, the components for configuring the bitstream can be implemented by the bitstream configuration circuit 314. In some examples, the bitstream configuration circuit 314 can be implemented by machine-executable instructions, such as at least those executed by processor circuitry. Figure 7 The processor circuitry can be implemented using frames 710, 712, and 714. Figure 8 Example processor circuit 812 Figure 9 Example processor circuit 900, and / or Figure 10The example is implemented using a field-programmable gate array (FPGA) circuit 1000. In other examples, the bitstream configuration circuit 314 is implemented using other hardware logic circuits, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the bitstream configuration circuit 314 can be implemented using at least one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGA, application-specific integrated circuit (ASIC), comparator, operational amplifier (op-amp), logic circuitry, etc.) configured to perform the corresponding operation without executing software or firmware, but other structures are equally suitable.

[0072] In some examples, the components used for determination include metadata for generating indication of the slice location so that the client device (e.g., Figure 1 The client device 114) is capable of decoding components with different fields of view from the encoded frame.

[0073] Figure 4 This is a block diagram of an example video decoding system 400 for implementing bitstream reconfiguration. The example video decoding system 400 includes an example encoding circuit 402, an example distribution server 404, an example client device 406, and an example decoding circuit 408. In some examples, the encoding circuit 402 can be... Figure 1 The encoding circuit 110 is used to implement this, and the example distribution server 404 can be accessed via... Figure 1 and Figure 3 Example distribution server 112 is used to implement this, while example client device 406 can be implemented through... Figure 1 The example client device 114 is used for implementation.

[0074] exist Figure 4 In this example, the example decoding circuit 408 is to decode the bitstream reconfigured by the example distribution server 404. In some examples, the decoding circuit 408 is included in the example client device 406. For example, the decoding circuit 408 may be used internally by the client device 406. Additionally and / or alternatively, the example decoding circuit 408 may be external to the client device 406. It may be available in, for example... Figure 1 Example of a client-side implementation, such as client-side processing side 108, is decoding circuit 408.

[0075] In an example operation of the video decoding system 400, encoding circuitry 402 encodes a first frame 410 at a first resolution and encodes a second frame 412 at a second resolution greater than the first resolution. The first frame 410 constitutes a first layer 414 (e.g., a base layer), and the second frame 412 constitutes a second layer 416 (e.g., an enhancement layer). The first frame 410 and the second frame 412 represent a 360-degree picture in the video (e.g., a complete view of the frames in a 360-degree video). The example encoding circuitry 402 segments the first frame 410 and the second frame 412 into pieces. The second frame 412 includes pieces encoded at a higher resolution than the pieces encoded in the first frame 410. The example encoding circuitry 402 encapsulates frames 410 and 412 in a first bitstream and sends the first bitstream to an example distribution server 404.

[0076] In the example operation of the video decoding system 400, the example distribution server 404 obtains a first bitstream. The example distribution server 404 obtains FOV information from the example client device 406. The example distribution server 404 reconfigures the first bitstream based on the FOV information.

[0077] For example, distribution server 404 selects the slice in the second frame 412 that corresponds to the FOV information. In some examples, distribution server 404 identifies the yaw, pitch, and roll angles in a sphere projected onto a two-dimensional plane. Example distribution server 404 extracts the selected slice from the second frame 412 to generate a third frame 418 corresponding to the FOV. Example distribution server 404 repackages the extracted slice. For example, distribution server 404 repackages these slices into separate slice group OBUs. Example distribution server 404 associates the separate slice group OBUs corresponding to the slices in the third frame 418 with the frame OBU.

[0078] In some examples, distribution server 404 generates metadata to identify the locations of one or more slices in the first frame 410. For example, distribution server 404 maps slices in the first frame 410 to locations within the first frame 410. This mapping can be included in the metadata. In some examples, distribution server 404 makes slices in the first frame 410 "skipable" to inform client device 406 that these slices do not need to be decoded for that particular FOV. Example distribution server 404 sends a second bitstream to client device 406.

[0079] In an example operation of the video decoding system 400, the client device 406 may send a second bitstream to the decoding circuit 408 for decoding. In some examples, the decoding circuit 408 may store information indicating "skippable" segments in the first frame 410. In some examples, the decoding circuit 408 stores the second bitstream for later use when the FOV changes.

[0080] For example, client device 406 determines a change in FOV. In some examples, client device 406 queries decoding circuitry 408 for the new FOV. In some examples, the new FOV may correspond to a slice in second frame 412 that was not selected and used to generate third frame 418. In such examples, decoding circuitry 408 uses the slice mapping in the metadata of the second bitstream to identify and decode the slice that was not encoded in third frame 418. Decoding circuitry 408 provides the new FOV to client device 406 for rendering. Advantageously, client device 406 does not need to wait indefinitely for distribution server 404 to encode a new frame corresponding to the new FOV.

[0081] Example distribution server 404 wants to use the third frame 418 to generate a second bitstream. For example, distribution server 404 wants to... Although Figure 3 and Figure 4 The implementation is shown in the figure. Figure 1 The example method for distribution server 112, but Figure 3 and Figure 4 One or more of the elements, processes, and / or devices shown may be combined, split, rearranged, omitted, eliminated, and / or implemented in any other way. Furthermore, example configuration circuit 302, example interface 304, example field of view (FOV) data storage component 306, example frame data storage component 308, example chip select circuit 310, example repackaging circuit 312, example bitstream configuration circuit 314, and / or (more generally) Figure 1The example distribution server 112 can be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, any of the example configuration circuitry 302, example interface 304, example field-of-view (FOV) data storage component 306, example frame data storage component 308, example chip select circuitry 310, example repackaging circuitry 312, example bitstream configuration circuitry 314, and / or (more generally) the example distribution server 112 can be implemented by processor circuitry, analog circuitry (one or more), digital circuitry (one or more), logic circuitry (one or more), programmable processors (one or more), programmable microcontrollers (one or more), graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field-programmable gate arrays (FPGAs) such as field-programmable logic devices (FPLDs). When reading any of the device or system claims of this patent that cover pure software and / or firmware implementations, at least one of the example configuration circuit 302, example interface 304, example field of view (FOV) data storage component 306, example frame data storage component 308, example chip selection circuit 310, example repackaging circuit 312, and / or example bitstream configuration circuit 314 is hereby expressly defined as including a non-transitory computer-readable storage device or disk containing software and / or firmware, such as a memory, digital versatile disc (DVD), compact disc (CD), Blu-ray disc, etc. Furthermore, Figure 1 Example distribution server 112 may include supplementary or alternative methods. Figure 3 and Figure 4 One or more elements, processes and / or devices of those items shown, and / or may include more than one of any or all of the elements, processes and devices shown.

[0082] Figure 5-7 The diagram shows the representation used for implementation. Figure 1 , Figure 3 and Figure 4 The distribution server 112 includes an example hardware logic circuit, machine-readable instructions, a hardware-implemented state machine, and / or any combination of these items in its flowchart. Machine-readable instructions may be for processor circuitry (such as those combined below). Figure 8 The processor circuitry 812 shown in the example processor platform 800 discussed herein and / or in conjunction with the following Figure 9 and / or Figure 10The example processor circuitry discussed herein executes one or more executable programs or portions thereof. The program(s) may be implemented using software stored on one or more non-transitory computer-readable storage media (such as CDs, floppy disks, hard disk drives (HDDs), DVDs, Blu-ray discs, volatile memory (e.g., any type of random access memory (RAM) etc.) or non-volatile memory (e.g., FLASH memory, HDDs, etc.) associated with the processor circuitry located in one or more hardware devices, but the entire program and / or portions thereof may alternatively be executed by one or more hardware devices other than the processor circuitry, and / or implemented using firmware or dedicated hardware. Machine-readable instructions may be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., server and client hardware devices). For example, the client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that facilitates communication between the server and the endpoint client hardware device). Similarly, non-transitory computer-readable storage media may include one or more media located in one or more hardware devices. Furthermore, although references... Figure 5-7 The flowcharts shown describe one or more example programs, but many other methods for implementing example distribution server 112 may be used alternatively. For example, the execution order of the boxes may be changed, and / or some of the boxes described may be changed, eliminated, or combined. Additionally or alternatively, any or all boxes may be implemented by one or more hardware circuits (e.g., processor circuitry, discrete and / or integrated analog and / or digital circuitry, FPGA, ASIC, comparator, operational amplifier, logic circuitry, etc.) configured to perform the corresponding operation without executing software or firmware. Processor circuitry may be distributed across different network locations and / or within one or more hardware devices in a single machine (e.g., a single-core processor (e.g., a single-core central processing unit (CPU)), a multi-core processor (e.g., a multi-core CPU), etc.), multiple processors distributed across multiple servers in a server rack, multiple processors distributed across one or more server racks, CPUs and / or FPGAs located in the same package (e.g., in the same integrated circuit (IC) package, or in two or more separate enclosures, etc.).

[0083] The machine-readable instructions described herein may be stored in one or more formats, such as compressed formats, encrypted formats, fragmented formats, compiled formats, executable formats, and encapsulated formats. Machine-readable instructions as described herein may be stored as data or data structures (e.g., as instruction parts, code, code representations, etc.) that can be used to create, manufacture, and / or produce machine-executable instructions. For example, machine-readable instructions may be fragmented and stored on one or more storage devices and / or computing devices (e.g., servers) located in the same or different locations (e.g., in the cloud, at the edge, etc.) within a network or network aggregate. Machine-readable instructions may require one or more of the following to be installed, modified, adapted, updated, combined, supplemented, configured, decrypted, decompressed, unpacked, distributed, reassigned, compiled, etc., so that they can be directly read, interpreted, and / or executed by computing devices and / or other machines. For example, machine-readable instructions may be stored in multiple parts, each of which may be compressed, encrypted, and / or stored on separate computing devices, wherein these parts, when decrypted, decompressed, and / or combined, form a set of machine-executable instructions that perform one or more operations, which together may form a program, such as the program described herein.

[0084] In another example, machine-readable instructions may be stored in a state in which they can be read by processor circuitry, but require the addition of libraries (e.g., dynamic link libraries (DLLs)), software development kits (SDKs), application programming interfaces (APIs), etc., to execute these machine-readable instructions on a specific computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., storage settings, input data, recording network addresses, etc.) before they can be executed in whole or in part. Therefore, as used herein, machine-readable media may include machine-readable instructions and / or / one or more programs, regardless of the specific format or state in which the machine-readable instructions and / or / one or more programs are stored or otherwise quiescent or transported.

[0085] The machine-readable instructions described in this article can be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, machine-readable instructions can be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0086] As mentioned above, Figure 5-7Example operations can be implemented using executable instructions (e.g., computer and / or machine-readable instructions) stored on one or more non-transitory computer and / or machine-readable media, such as optical storage devices, magnetic storage devices, HDDs, flash memory, read-only memory (ROM), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage devices or disks, wherein information is stored for any duration (e.g., extended storage periods, permanent storage, for brief cases, for temporary buffering, and / or for caching information). As used herein, the terms non-transitory computer-readable media and non-transitory computer-readable storage media are explicitly defined to include any type of computer-readable storage device and / or disk, while excluding propagating signals and transmission media.

[0087] "Comprising" and "including" (and all their forms and tenses) are used herein as open-ended terms. Therefore, whenever a claim uses "comprising" or "including" in any form (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or in any kind of claim recitation, it is understood that additional elements, terms, etc., may be present without falling outside the scope of the corresponding claim or recitation. As used herein, the phrase "at least" is open-ended when used as a transitional term in, for example, a claim preamble, just as the terms "comprising" and "including" are open-ended. The term "and / or" when used, for example, in forms such as A, B, and / or C, refers to any combination or subset of A, B, C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, or (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A and B” is intended to refer to an implementation that includes any one of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to refer to an implementation that includes any one of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in the context of describing the conduct or execution of processes, instructions, actions, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to an implementation that includes any one of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing the conduct or execution of processes, instructions, actions, activities and / or steps, the phrase “at least one of A or B” is intended to refer to an implementation including any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0088] As used herein, singular references (e.g., “a,” “a,” “first,” “second,” etc.) do not exclude plurals. As used herein, the term “a” or “a” refers to one or more of that object. The terms “a” (or “a”), “one or more,” and “at least one” are used interchangeably herein. Furthermore, although each is listed separately, multiple parts, elements, or method actions can be implemented by, for example, the same entity or object. Additionally, although individual features may be included in different examples or claims, these may be combined, and inclusion in different examples or claims does not imply that a combination of features is not feasible and / or advantageous.

[0089] Figure 5 This is a flowchart representing example machine-readable instructions and / or example operations 500, which can be executed and / or instantiated by processor circuitry to generate instructions for configuring encoding circuitry to encode video in a specific manner. Figure 5 Machine-readable instructions and / or operations 500 begin at block 502, where distribution server 112 generates first instructions to configure encoding circuitry to encode projected images into slice groups, the first instructions defining at least two resolutions for the slice groups. For example, configuration circuitry 302 generates configuration instructions that cause encoding circuitry (e.g., ...) to... Figure 1 The encoding circuit 110 and / or Figure 4 The encoding circuit 402) segments the projected image at a first resolution and a second resolution, wherein the first resolution is lower than the second resolution.

[0090] Example distribution server 112 generates second instructions to configure the encoding circuitry to construct a second layer (box 504) of a chip set that is time-independent and corresponds to at least one of two resolutions. For example, configuration circuitry 302 causes the encoding circuitry (e.g., Figure 1 The encoding circuit 110 and / or Figure 4 The encoding circuit 402) eliminates the gaps between chip groups encoded at higher resolution (e.g., Figure 2 The time dependency between the second frame (208).

[0091] Example distribution server 112 waits until it receives a request to utilize a viewport-dependent stream (box 506). For example, configuration circuit 302 waits for a request to be sent to encoding circuit (e.g., Figure 1 The encoding circuit 110 and / or Figure 4 The encoding circuit 402) sends first and second instructions until the client device (e.g., Figure 1 Client device 114 and / or Figure 4 The client device (406) requests a streaming service that depends on the viewport.

[0092] When example distribution server 112 receives a request to utilize viewport-dependent streaming (e.g., box 506 returns a "Yes" value), example distribution server 112 transmits first and second instructions to encoding circuitry (box 508). For example, configuration circuitry 302 transmits first and second instructions in response to a request to utilize viewport-dependent streaming. In some examples, interface 304 initializes configuration circuitry 302 to generate a response from a client device (e.g., Figure 1 Client device 114 and / or Figure 4 The client device (406) is a user's instruction to stream immersive video.

[0093] Example operation 500 ends when example distribution server 112 transmits the first and second instructions. In some examples, operation 500 may be repeated when different encoding circuits (e.g., encoding circuits not yet configured via the first and second configuration instructions) are used during viewport-dependent streaming.

[0094] Figure 6 This is a flowchart representing example machine-readable instructions and / or example operations 600, which can be executed and / or instantiated by processor circuitry to rewrite the bitstream for viewport-dependent streaming. Figure 6 Machine-readable instructions and / or operations 600 begin at box 602, where distribution server 112 obtains a first bitstream having one or more encoded frames. For example, interface 304 receives instructions and / or operations from encoding circuitry (e.g., Figure 1 The encoding circuit 110 and / or Figure 4 The encoding circuit 402) obtains a first bitstream, which corresponds to a sequence of frames indicating a full 360-degree view of the immersive video. The frame sequence is encoded at a first resolution and a second resolution, where the second resolution is higher than the first resolution. Each frame encoded at the second resolution does not contain any temporal dependence on any corresponding subsequent or preceding frame.

[0095] Example distribution server 112 needs to obtain field of view (FOV) information (box 604). For example, interface 304 needs to obtain information from a client device (e.g., Figure 1 Client device 114 and / or Figure 4 The client device 406 obtains the yaw, pitch and roll rotation angles corresponding to the FOV.

[0096] Example distribution server 112 rewrites a first bitstream into a second bitstream based on field of view (FOV) information (box 606). For example, chip selection circuit 310, repackaging circuit 312, and bitstream configuration circuit 314 generate bit sequences by encoding (e.g., projection, packing, and encoding) frame sequences corresponding to specific segments and / or views in the entire 360-degree view.

[0097] Example distribution server 112 is to send a second bitstream to a client device (box 608). For example, interface 304 is to send the second bitstream to the client device for decoding and rendering. In some examples, the second bitstream may be utilized by the client device in subsequent and / or different views of the 360-degree view.

[0098] Example operation 600 ends when example distribution server 112 sends the second bitstream to the client device. In some examples, operation 600 is repeated when the encoding circuitry provides a new bitstream of encoded immersive video and / or when the client device provides new FOV information.

[0099] Figure 7 This is a flowchart representing example machine-readable instructions and / or example operations 606, which can be executed and / or instantiated by processor circuitry to rewrite the first bitstream into a second bitstream based on FOV information. Figure 7 Machine-readable instructions and / or operations 606 begin at block 702, where the distribution server 112 selects slices corresponding to the field of view in a first encoded frame, which corresponds to a first resolution. For example, slice selection circuitry 310 uses FOV information to identify which slices to extract and uses it to generate a second bitstream.

[0100] Example distribution server 112 extracts selected slices from the first encoded frame (box 704). For example, slice selection circuitry 310 uses the syntax of a video decoding structure (e.g., AV1 Scalable Video Decoding (SVC)) to extract the selected slices. In some examples, depending on the video decoding structure, slice selection circuitry 310 separates slices identified as being in the FOV from other slices in the slice group corresponding to the entire frame (e.g., the entire 360-degree video frame).

[0101] Example distribution server 112 repackages the extracted slices into a second encoded frame, the second encoded frame being encoded at a first resolution and containing only the selected slices representing the field of view (box 706). For example, repackaging circuitry 312 generates a new frame encoded at a higher resolution, containing only the FOV. In some examples, repackaging circuitry 312 encapsulates a slice into a slice group (e.g., slice group OBU). Once the selected and extracted slices have been encapsulated into their respective slice groups, example repackaging circuitry 312 encapsulates the slice group into a frame group (e.g., frame group OBU).

[0102] Example distribution server 112 inserts a time delimiter before a second and a third encoded frame to associate these frames, with the third encoded frame corresponding to a second resolution lower than the first resolution (box 708). For example, bitstream configuration circuitry 314 associates a frame (e.g., the second encoded frame) in a first layer (e.g., an enhancement layer) with a frame (e.g., the third encoded frame) in a second layer (e.g., a base layer) by indicating that the frame OBU following the first time delimiter OBU has the same decoding timestamp. The second and third encoded frames belong to the same frame. For example, the content in the second and third encoded frames is the same, but that content is encoded at a different resolution. For example, go to... Figure 2As depicted on the timeline, the first frame 206A in the first frame and the first frame 208A in the second frame belong to frame 0. Similarly, the second and third coded frames belong to the same frame. The time delimiter inserted before the second and third coded frames indicates that the second and third coded frames belong to the same frame.

[0103] Example distribution server 112 generates metadata (box 710) to identify the location of one or more slices in a third encoded frame. For example, bitstream configuration circuitry 314 determines a slice group start value and a slice group end value for each slice in the third encoded frame, which are used to determine the slice's location within the third encoded frame. In some examples, bitstream configuration circuitry 314 includes information in the metadata informing client devices and / or decoding circuitry that slices in the third encoded frame can be skipped. For example, bitstream configuration circuitry 314 sets and / or removes flags in the bitstream corresponding to slice skipping.

[0104] Example distribution server 112 packets a second encoded frame, a third encoded frame, a time delimiter, and metadata into a second bitstream (box 712). For example, bitstream configuration circuitry 314 combines information (e.g., encoded frames, time delimiters OBU, metadata, etc.) into bitstream packets that can be transmitted over a network (e.g., the Internet).

[0105] Example distribution server 112 needs to determine whether another frame to be encoded exists (box 714). For example, the first bitstream may include the one to be streamed to a client device (e.g., Figure 1 The client device 114) receives multiple encoded frames. In such examples, each frame is split and rewritten into a second bitstream based on the field of view (FOV). In some examples, if the distribution server 112 determines that another frame exists (e.g., box 714 returns a value "yes"), control returns to box 702.

[0106] In some examples, if distribution server 112 determines that another frame does not exist (e.g., box 714 returns a value of "No"), then when example distribution server 112 packets information (e.g., second encoded frame, third encoded frame, time delimiter, and metadata) into a second bitstream, control returns to operation 600, specifically to box 608.

[0107] Figure 8 This is a block diagram of an example processor platform 800, which is configured to execute and / or instantiate... Figure 5-7 Machine-readable instructions and / or operations to achieve Figure 1 and Figure 3The distribution server 112. The processor platform 800 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smartphone, a tablet such as an iPad™), an internet appliance, a headset (e.g., an augmented reality (AR) headset, a virtual reality (VR) headset, etc.) or other wearable device, or any other type of computing device.

[0108] The processor platform 800 shown in the example includes processor circuitry 812. The processor circuitry 812 shown in the example is hardware. For example, processor circuitry 812 can be implemented using one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired family or manufacturer. Processor circuitry 812 can be implemented using one or more semiconductor-based (e.g., silicon-based) devices. In this example, processor circuitry 812 implements example configuration circuitry 302, example chip select circuitry 310, example repackaging circuitry 312, and example bitstream configuration circuitry 314.

[0109] The processor circuitry 812 of the illustrated example includes local memory 813 (e.g., cache, registers, etc.). The processor circuitry 812 of the illustrated example communicates via bus 818 with main memory, which includes volatile memory 814 and non-volatile memory 816. Volatile memory 814 may be implemented using synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS® dynamic random access memory (RDRAM®), and / or any other type of RAM device. Non-volatile memory 816 may be implemented using flash memory and / or any other desired type of memory device. Access to the main memory 814, 816 of the illustrated example is controlled by memory controller 817.

[0110] The processor platform 800 shown in the example also includes interface circuitry 820. Interface circuitry 820 can be implemented using hardware conforming to any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB) interface, a Bluetooth® interface, a Near Field Communication (NFC) interface, a PCI interface, and / or a PCIe interface. In this example, interface circuitry 820 implements example interface 304.

[0111] In the example shown, one or more input devices 822 are connected to interface circuitry 820. The input devices 822 allow a user to input data and / or commands to processor circuitry 812. The input devices 822 may be implemented, for example, an audio sensor, microphone, camera (still or video), keyboard, buttons, mouse, touchscreen, and / or voice recognition system.

[0112] One or more output devices 824 are also connected to the interface circuitry 820 of the illustrated example. The output devices 824 may be implemented, for example, via display devices (e.g., light-emitting diode (LED), organic light-emitting diode (OLED), liquid crystal display (LCD), cathode ray tube (CRT) display, in-situ switch (IPS) display, touchscreen, etc.) and / or haptic output devices. Therefore, the interface circuitry 820 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or graphics processor circuitry (such as a GPU).

[0113] The interface circuit 820 of the example shown also includes communication devices such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate the exchange of data with external machines (e.g., any kind of computing device) via network 826. Communication can be carried out via, for example, Ethernet connections, digital subscriber line (DSL) connections, telephone line connections, coaxial cable systems, satellite systems, line-to-line wireless systems, cellular telephone systems, optical connections, etc.

[0114] The processor platform 800 shown in the example also includes one or more mass storage devices 828 for storing software and / or data. Examples of such mass storage devices 828 include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, CDs, Blu-ray disc drives, redundant array of independent disks (RAID) systems, solid-state storage devices (such as flash memory devices), and DVD drives. In this example, the mass storage device 828 may implement the example field-of-view data storage component 306 and frame data storage component 308.

[0115] can be Figure 5-7 The machine-executable instructions 832 implemented by the machine-readable instructions can be stored in a mass storage device 828, a volatile memory 814, a non-volatile memory 816, and / or on a removable, non-transitory computer-readable storage medium (such as a CD or DVD).

[0116] Figure 9 yes Figure 8 A block diagram of an example implementation of the processor circuit 812. In this example, Figure 8The processor circuitry 812 is implemented via a microprocessor 900. For example, the microprocessor 900 may implement multi-core hardware circuitry, such as a CPU, DSP, GPU, XPU, etc. While it may include any number of example cores 902 (e.g., one core), this example microprocessor 900 is a multi-core semiconductor device comprising N cores. The cores 902 of the microprocessor 900 may operate independently or collaboratively to execute machine-readable instructions. For example, machine code corresponding to firmware, embedded software, or software programs may be executed by one of these cores 902, or by multiple cores 902 at the same or different times. In some examples, the machine code corresponding to firmware, embedded software, or software programs is split into threads and executed in parallel by two or more cores 902. The software program may correspond to... Figure 5-7 The flowchart represents part or all of the machine-readable instructions and / or operations.

[0117] Core 902 can communicate via example bus 904. In some examples, bus 904 may implement a communication bus to enable communication with one or more of cores 902. For example, bus 904 may implement at least one of an Inter-Integrated Circuit (I2C) bus, a Serial Peripheral Interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, bus 904 may implement any other type of computing or electrical bus. Core 902 may obtain data, instructions, and / or signals from one or more external devices via example interface circuitry 906. Core 902 may output data, instructions, and / or signals to one or more external devices via interface circuitry 906. While core 902 in this example includes example local memory 920 (e.g., a Level 1 (L1) cache, which may be split into an L1 data cache and an L1 instruction cache), microprocessor 900 also includes example shared memory 910 (e.g., a Level 2 (L2) cache) that may be shared by these cores for high-speed access to data and / or instructions. Data and / or instructions can be transferred (e.g., shared) by writing to and / or reading from shared memory 910. The local memory 920 and shared memory 910 of each of these cores 902 can be a combination of multi-level cache memory and main memory (e.g., ...). Figure 8 The cache is part of a hierarchy of storage devices (main memory 814, 816). Typically, higher-level memories in the hierarchy exhibit shorter access times and have smaller storage capacities than lower-level memories. Changes at each level of the cache hierarchy are managed (e.g., coordinated) through cache coherence strategies.

[0118] Each core 902 may be referred to as a CPU, DSP, GPU, or any other type of hardware circuitry. Each core 902 includes a control unit circuitry 914, an arithmetic and logic (AL) circuitry (sometimes called an ALU) 916, multiple registers 918, an L1 cache 920, and an example bus 922. Other structures may also be present. For example, each core 902 may include vector unit circuitry, single instruction multiple data (SIMD) unit circuitry, load / store unit (LSU) circuitry, branch / jump unit circuitry, floating-point unit (FPU) circuitry, etc. The control unit circuitry 914 includes semiconductor-based circuitry configured to control (e.g., coordinate) the movement of data within the corresponding core 902. The AL circuitry 916 includes semiconductor-based circuitry configured to perform one or more mathematical and / or logical operations on data within the corresponding core 902. In some examples, the AL circuitry 916 performs integer-based operations. In other examples, the AL circuitry 916 also performs floating-point operations. In some other examples, AL circuit 916 may include a first AL circuit performing integer-based operations and a second AL circuit performing floating-point operations. In some examples, AL circuit 916 may be referred to as an Arithmetic Logic Unit (ALU). Register 918 is a semiconductor-based structure for storing data and / or instructions, such as the results of one or more operations performed by AL circuit 916 corresponding to core 902. For example, register 918 may include one or more vector registers, one or more SIMD registers, one or more general-purpose registers, one or more flag registers, one or more segment registers, one or more machine-specific registers, one or more instruction pointer registers, one or more control registers, one or more debug registers, one or more memory management registers, one or more machine check registers, etc. Register 918 may be arranged in a manner such as Figure 9 As shown in the diagram. Alternatively, register 918 may be organized in any other arrangement, format, or structure (including distribution throughout core 902 to reduce access time). Bus 904 may implement at least one of I2C bus, SPI bus, PCI bus, or PCIe bus.

[0119] Each core 902 and / or (more generally) microprocessor 900 may include additional and / or alternative structures of those shown and described above. For example, one or more clock circuits, one or more power supplies, one or more power gating systems, one or more cache home agents (CHAs), one or more converged / common mesh stops (CMSs), one or more shifters (e.g., one or more barrel shifters), and / or other circuitry may be present. The microprocessor 900 is a semiconductor device in one or more integrated circuits (ICs) contained in one or more packages, fabricated to include a number of transistors interconnected to implement the structures described above. The processor circuitry may include one or more accelerators and / or cooperate with one or more accelerators. In some examples, accelerators are implemented by logic circuitry to perform certain tasks faster and / or more efficiently than a general-purpose processor is capable of. Examples of accelerators include ASICs and FPGAs such as those discussed herein. GPUs or other programmable devices may also be accelerators. Accelerators may be onboard to the processor circuit board, in the same chip package as the processor circuitry, and / or in one or more packages separate from the processor circuitry.

[0120] Figure 10 yes Figure 8 A block diagram of another example implementation of the processor circuit 812 is shown. In this example, the processor circuit 812 is implemented by an FPGA circuit 1000. The FPGA circuit 1000 can be used, for example, to perform operations that would otherwise be conducted via... Figure 9 The example microprocessor 900 executes the corresponding machine-readable instructions. However, once configured, the FPGA circuitry 1000 instantiates the machine-readable instructions in hardware, and thus can often execute these operations much faster than could be achieved by executing the corresponding software via a general-purpose microprocessor.

[0121] More specifically, as described above Figure 9 The microprocessor 900 (which is programmable to execute) Figure 5-7 Compared to a general-purpose device whose flowcharts represent some or all of the machine-readable instructions, but whose interconnections and logic circuitry are fixed once manufactured, Figure 10 Example FPGA circuit 1000 includes components that can be configured and / or interconnected in different ways after manufacturing to, for example... Figure 5-7The flowchart represents some or all of the machine-readable instructions instantiated in the interconnects and logic circuits. Specifically, the FPGA 1000 can be viewed as an array of logic gates, interconnects, and switches. Switches can be programmed to change the way logic gates are interconnected via interconnects, effectively forming one or more dedicated logic circuits (unless and until the FPGA circuit 1000 is reprogrammed). The configured logic circuits enable logic gates to cooperate in different ways, thereby performing different operations on the data received by the input circuits. Those operations can correspond to... Figure 5-7 The flowchart represents some or all of the software. Therefore, the FPGA circuit 1000 can be configured to... Figure 5-7 The flowchart effectively instantiates some or all of the machine-readable instructions into dedicated logic circuits to perform the operations corresponding to those software instructions in a dedicated manner similar to that of an ASIC. Therefore, the FPGA circuit 1000 can perform operations comparable to those of a general-purpose microprocessor. Figure 5-7 The same operation can be performed faster than some or all of the machine-readable instructions.

[0122] exist Figure 10 In the example, the FPGA circuit 1000 is configured to be programmed (and / or reprogrammed once or multiple times) by the end user using a hardware description language (HDL) such as Verilog. Figure 10 The FPGA circuit 1000 includes example input / output (I / O) circuitry 1002 to obtain data from example configuration circuitry 1004 and / or external hardware (e.g., external hardware circuitry) 1006 and / or output data to example configuration circuitry 1004 and / or external hardware (e.g., external hardware circuitry) 1006. For example, configuration circuitry 1004 may implement interface circuitry that obtains machine-readable instructions for configuring FPGA circuitry 1000 or portions thereof. In some such examples, configuration circuitry 1004 may obtain machine-readable instructions from a user, a machine (e.g., hardware circuitry (e.g., programming or dedicated circuitry)) that implements an artificial intelligence / machine learning (AI / ML) model to generate instructions. In some examples, external hardware 1006 may implement… Figure 9 The microprocessor 900. The FPGA circuit 1000 also includes an array of example logic gates 1008, a plurality of example configurable interconnects 1010, and example memory circuits 1012. The logic gates 1008 and interconnects 1010 are configurable to correspond to Figure 5-7 Instantiate one or more operations and / or other required operations of at least some machine-readable instructions. Figure 10The logic gates 1008 shown are manufactured in groups or blocks. Each block includes semiconductor-based electrical structures that can be configured into logic circuits. In some examples, the electrical structures include logic gates (e.g., AND gates, OR gates, NOR gates, etc.) that provide basic building blocks for the logic circuits. Electrically controlled switches (e.g., transistors) are present within each logic gate 1008 to enable the configuration of the electrical structures and / or logic gates to form circuits that perform desired operations. The logic gates 1008 may include other electrical structures such as lookup tables (LUTs), registers (e.g., flip-flops or latches), multiplexers, etc.

[0123] The interconnect 1010 in the example shown is a conductive path, trace, via, or the like that may include an electronically controlled switch (e.g., a transistor), the state of which can be changed (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more of the logic gates 1008, thereby programming the desired logic circuit.

[0124] The storage circuit 1012 shown in the example is configured to store the results of one or more operations performed by the corresponding logic gates. The storage circuit 1012 can be implemented using registers, etc. In the example shown, the storage circuit 1012 is distributed within the logic gate circuit 1008 to facilitate access and improve execution speed.

[0125] Figure 10 The example FPGA circuit 1000 also includes example dedicated operation circuitry 1014. In this example, dedicated operation circuitry 1014 includes special-purpose circuitry 1016, which can be invoked to implement common functions, thus avoiding the need for field programming of those functions. Examples of such special-purpose circuitry 1016 include memory (e.g., DRAM) controller circuitry, PCIe controller circuitry, clock circuitry, transceiver circuitry, memory, and multiplier-accumulator circuitry. Other types of special-purpose circuitry may be present. In some examples, FPGA circuitry 1000 may also include example general-purpose programmable circuitry 1018, such as example CPU 1020 and / or example DSP 1022. Other general-purpose programmable circuitry 1018, such as GPUs, XPUs, etc., may be additionally or alternatively present and programmed to perform other operations.

[0126] Although Figure 9 and Figure 10 Show Figure 8 Two example implementations of the processor circuit 812 are presented, but many other approaches have also been conceived. For example, as mentioned above, modern FPGA circuits may include an onboard CPU, such as... Figure 10 One or more of the example CPUs 1020. Therefore, Figure 8 The processor circuit 812 can be additionally combined Figure 9 Example microprocessor 900 and Figure 10 Example FPGA circuit 1000 is used for implementation. In some such hybrid examples, Figure 5-7 The first part of the machine-readable instructions represented by the flowchart can be derived from... Figure 9 It is executed by one or more cores in the 902 core, and Figure 5-7 The second part of the machine-readable instructions represented by the flowchart can be derived from... Figure 10 The FPGA circuit 1000 is used to execute this.

[0127] In some examples, Figure 8 The processor circuitry 812 can be housed in one or more packages. For example, Figure 9 The processor circuit 900 and / or Figure 10 The FPGA circuitry 1000 can be housed in one or more packages. In some examples, the XPU can be accessed via... Figure 8 The processor circuitry 812 is used to implement this, and it can be housed in one or more packages. For example, the XPU may include a CPU in one package, a DSP in another package, a GPU in yet another package, and an FPGA in yet another package.

[0128] As will be apparent from the foregoing, example systems, methods, apparatuses, and articles of art have been disclosed to improve and / or reduce latency during viewport switching. The examples disclosed herein save bandwidth between the client device and the distribution server distributing immersive video to the client device by ensuring that only high-quality segments within the FOV are transmitted. The examples disclosed herein eliminate temporal redundancy and / or dependencies between encoded frames in enhancement layers (e.g., high-quality layers) to allow the client device to perform viewport switching on any segment in the enhancement layer at any time. Advantageously, the examples disclosed herein enable decoding of non-adjacent regions (e.g., non-adjacent segments, such as segments whose positions are not adjacent but are within the FOV) within a frame for a given FOV, without requiring the distribution server to provide new video data (e.g., new frames). The disclosed systems, methods, apparatuses, and articles of art improve the efficiency of using computing devices by reducing latency between client and server interactions when the viewport changes during immersive video streaming. Therefore, the disclosed systems, methods, apparatuses, and articles of art represent one or more improvements in the operation of machines such as computers or other electronic and / or mechanical devices.

[0129] This document discloses example methods, devices, systems, and artifacts for reducing latency during viewport switching in immersive video. Further examples and combinations thereof include the following: Example 1 includes an apparatus comprising at least one memory, instructions within the apparatus, and processor circuitry for executing the instructions to: obtain a first bitstream having a first coded frame and a second coded frame, the second coded frame being encoded at a higher resolution than the first coded frame and having a decoding dependency on the first coded frame; rewrite the first bitstream into a second bitstream based on field-of-view information, the second bitstream including a third coded frame indicating a portion of the second coded frame corresponding to the field-of-view information, and including the first coded frame; and transmit the second bitstream to a client device for decoding and rendering said portion of the second coded frame.

[0130] Example 2 includes the device of Example 1, wherein a first coded frame is divided into a first piece and a second coded frame is divided into a second piece, the first piece being proportional to the second piece.

[0131] Example 3 includes the device of Example 2, wherein the processing circuitry is used to select a second slice from the second slice that corresponds to the field of view information to generate a third coded frame.

[0132] Example 4 includes the device of Example 1, wherein the decoding dependency of the second encoded frame is a spatial decoding dependency, and the second encoded frame has no temporal dependency on any subsequent or previous encoded frames contained in the first bitstream and encoded at a higher resolution.

[0133] Example 5 includes the device of Example 1, wherein the first bitstream indicates a full 360-degree video frame in the immersive video, and the second bitstream indicates a portion of the full 360-degree video frame in the immersive video.

[0134] Example 6 includes the device of Example 1, wherein the processing circuitry is used to obtain field of view information from the client device, the field of view information including yaw, pitch and roll angles.

[0135] Example 7 includes the device of Example 1, wherein a first coded frame is segmented into a first piece and a second coded frame is segmented into a second piece, and processing circuitry is configured to: select a second piece from the second piece that corresponds to the field of view information to generate a third coded frame; and generate metadata indicating the position of the first piece so that a client device can decode different fields of view from the first coded frame, the different fields of view not corresponding to the second piece from the second piece that corresponds to the field of view information and not included in the third coded frame.

[0136] Example 8 includes a non-transitory computer-readable storage medium containing instructions that, when executed, cause one or more processors to at least: obtain a first bitstream having a first coded frame and a second coded frame, the second coded frame being encoded at a higher resolution than the first coded frame and having a decoding dependency on the first coded frame; rewrite the first bitstream into a second bitstream based on field-of-view information, the second bitstream including a third coded frame indicating a portion of the second coded frame corresponding to the field-of-view information, and including the first coded frame; and transmit the second bitstream to a client device for decoding and rendering said portion of the second coded frame.

[0137] Example 9 includes the non-transitory computer-readable storage medium of Example 8, wherein a first coded frame is divided into a first piece and a second coded frame is divided into a second piece, the first piece being proportional to the second piece.

[0138] Example 10 includes the non-transitory computer-readable storage medium of Example 9, wherein, when executed, the instructions cause one or more processors to select a second slice of the second slice corresponding to the field of view information to generate a third coded frame.

[0139] Example 11 includes the non-transitory computer-readable storage medium of Example 8, wherein the decoding dependency of the second encoded frame is a spatial decoding dependency, and the second encoded frame has no temporal dependency on any subsequent or previous encoded frames contained in the first bitstream and encoded at a higher resolution.

[0140] Example 12 includes the non-transitory computer-readable storage medium of Example 8, wherein a first bitstream indicates a complete 360-degree video frame in an immersive video, and a second bitstream indicates a portion of a complete 360-degree video frame in an immersive video.

[0141] Example 13 includes the non-transitory computer-readable storage medium of Example 8, wherein, when executed, the instructions cause one or more processors to obtain field-of-view information from a client device, the field-of-view information including yaw, pitch, and roll angles.

[0142] Example 14 includes the non-transitory computer-readable storage medium of Example 8, wherein a first coded frame is segmented into a first piece, and a second coded frame is segmented into a second piece, and instructions, when executed, cause one or more processors to: select a second piece in the second piece corresponding to the field of view information to generate a third coded frame; and generate metadata indicating the location of the first piece so that a client device can decode different fields of view from the first coded frame, the different fields of view not corresponding to the second piece in the second piece corresponding to the field of view information and not included in the third coded frame.

[0143] Example 15 includes a method comprising: obtaining a first bitstream having a first coded frame and a second coded frame, the second coded frame being encoded at a higher resolution than the first coded frame and having a decoding dependency on the first coded frame; rewriting the first bitstream into a second bitstream based on field-of-view information, the second bitstream including a third coded frame indicating a portion of the second coded frame corresponding to the field-of-view information, and including the first coded frame; and transmitting the second bitstream to a client device for decoding and rendering said portion of the second coded frame.

[0144] Example 16 includes the method of Example 15, wherein a first coded frame is divided into a first piece and a second coded frame is divided into a second piece, the first piece being proportional to the second piece.

[0145] Example 17 includes the method of Example 16, and further includes: selecting a second slice in the second slice that corresponds to the field of view information to generate a third coded frame.

[0146] Example 18 includes the method of Example 15, wherein the decoding dependency of the second encoded frame is a spatial decoding dependency, and the second encoded frame has no temporal dependency on any subsequent or previous encoded frames contained in the first bitstream and encoded at a higher resolution.

[0147] Example 19 includes the method of Example 15, further including: obtaining field of view information from a client device, the field of view information including yaw, pitch and roll angles.

[0148] Example 20 includes the method of Example 15, wherein a first coded frame is segmented into a first piece and a second coded frame is segmented into a second piece, the method further comprising: selecting a second piece in the second piece that corresponds to field-of-view information to generate a third coded frame; and generating metadata indicating the position of the first piece to enable a client device to decode different fields of view from the first coded frame, the different fields of view not corresponding to the second piece in the second piece that corresponds to the field-of-view information and not included in the third coded frame.

[0149] Example 21 includes the method of Example 15, wherein a first bitstream indicates a full 360-degree video frame in an immersive video, and a second bitstream indicates a portion of a full 360-degree video frame in an immersive video.

[0150] Example 22 includes the method of Example 15, further including: obtaining field of view information from a client device, the field of view information including yaw, pitch and roll angles.

[0151] While certain example systems, methods, apparatuses, and articles of art have been disclosed herein, the scope of this patent is not limited thereto. Rather, this patent covers all systems, methods, apparatuses, and articles of art that fall entirely within the scope of the claims of this patent.

[0152] The following claims are hereby incorporated by reference into this detailed description, wherein each claim independently represents a separate embodiment of the present disclosure.

Claims

1. A server, comprising: Interface circuit; Machine-readable instructions; and At least one processor circuit, which is programmed based on the machine-readable instructions, to: Obtain first encoded data and second encoded data, wherein the first encoded data includes a first slice and the second encoded data includes a second slice, the first slice and the second slice correspond to the same complete video frame, the first slice has a first resolution and the second slice has a second resolution, the second resolution being higher than the first resolution; Obtain viewport data from the client device; Extract the subset of the second piece corresponding to the viewport based on the viewport data; The subsets of the first slice and the second slice corresponding to the viewport corresponding to the complete video frame are merged into the bitstream; as well as The bit stream is transmitted to the client device.

2. The server as claimed in claim 1, wherein the viewport data includes yaw information, pitch information, and roll information.

3. The server of claim 1, wherein one or more of the at least one processor circuitry are configured to select the subset of the second piece based on the mapping of the viewport to the two-dimensional projection plane.

4. The server of claim 1, wherein one or more of the at least one processor circuitry are configured to include metadata in the bitstream, the metadata being configured to allow selection of a slice in the bitstream corresponding to the viewport.

5. The server of claim 1, wherein the second slice is a slice that is time-independent of previous video frames.

6. The server of claim 1, wherein the complete video frame is associated with omnidirectional video.

7. The server of claim 1, wherein one or more of the at least one processor circuitry are configured to incorporate the subset of the first and second slices into the bitstream based on instructions for encoding immersive video.

8. A calculation method, comprising: Obtain first encoded data and second encoded data, wherein the first encoded data includes a first slice and the second encoded data includes a second slice, the first slice and the second slice correspond to the same complete video frame, the first slice has a first resolution and the second slice has a second resolution, the second resolution being higher than the first resolution; Obtain viewport data from the client device; Extract the subset of the second piece corresponding to the viewport based on the viewport data; The subsets of the first slice and the second slice corresponding to the viewport corresponding to the complete video frame are merged into the bitstream; as well as The bit stream is transmitted to the client device.

9. The calculation method as described in claim 8, wherein the viewport data includes yaw information, pitch information, and roll information.

10. The calculation method of claim 8, further comprising selecting the subset of the second piece based on the mapping from the viewport to the two-dimensional projection plane.

11. The calculation method of claim 8, further comprising including metadata in the bitstream, the metadata being used to allow selection of a slice in the bitstream corresponding to the viewport.

12. The calculation method of claim 8, wherein the second slice is time-independent of the previous video frame.

13. The calculation method of claim 8, wherein the complete video frame is associated with omnidirectional video.

14. The calculation method of claim 8, further comprising incorporating the subsets of the first slice and the second slice into the bitstream based on instructions for encoding immersive video.

15. At least one non-transitory computer-readable medium comprising computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the computation method as described in any one of claims 8-14.

16. A computing system, comprising: Memory, which stores instructions; and One or more processors, The instructions thereon are executed by the one or more processors to cause the one or more processors to perform the computation method as described in any one of claims 8-14.

17. A computing device comprising means for performing the computing method as described in any one of claims 8-14.

18. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the computation method as described in any one of claims 8-14.

Images