Method and apparatus for reducing 360-degree viewport adaptive streaming latency

Abstract

A client device adaptively streams 360-degree video. A first segment is displayed based on a first viewing direction at a first time, where the first viewing direction is associated with a first viewport. The client requests a first base buffer segment based on the first viewport. The first base buffer segment has a presentation time after the first segment. At a second time, the viewing direction changes to a second viewing direction associated with a second viewport. The client requests a first viewport buffer segment having the same presentation time based on the second viewport before the presentation time. The client device displays a second segment at the presentation time, where the second segment is the first viewport buffer segment or the first base buffer segment. The client provides a report on a viewport switch delay and a most requested segment.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 201780066997.2, filed on September 8, 2017, entitled "Method and Apparatus for Reducing Adaptive Streaming Latency in 360-Degree View Zone".

[0002] Cross-references to related applications

[0003] This application is a non-provisional application of U.S. Provisional Application 62 / 385,733, filed on September 9, 2016, entitled “Method and apparatus for reducing 360-degree view-adaptive streaming latency”, and claims the benefit of it under 35 U.S.SC §119(e), the entire contents of which are incorporated herein by reference. Background Technology

[0004] 360° (or 360-degree) video is a rapidly emerging new format in the media industry. It is enabled by the increasing availability of VR devices and offers viewers a very new sense of presence. Compared to traditional linear video (2D or 3D), 360° video presents a series of new and challenging engineering challenges for video processing and transmission. Achieving a comfortable and immersive user experience requires extremely high video quality and extremely low latency, but large video sizes can be an obstacle to delivering high-quality, large-scale 360° video.

[0005] When watching a 360° video, a portion of the video is displayed to the user. This portion changes based on feedback provided by the head-mounted display (HMD) or other type of user interface (e.g., a smartphone) when the user looks around or zooms in. The entire spatial area of ​​the 360° video that can be presented to the user in whole or in part and can have a different quality than the rest of the 360° video is called the viewport.

[0006] To enable users to select any viewpoint, 360° video is typically captured and / or rendered on a spherical surface. This spherical video format cannot usually be directly transmitted using conventional video codecs. Instead, a two-step process is typically used to compress 360° or spherical video: (1) projecting the spherical video onto a 2D plane using some projection method; and (2) encoding the projected 2D video using a conventional video codec. An example of a commonly used projection method is called equirectangular projection, which uses the following equation to represent coordinates (θ, θ) on the sphere. Point P is mapped to point P with coordinates (u, v) in the 2D plane, such as... Figure 1 As shown.

[0007]

[0008]

[0009] Currently, most 360° video applications and services encode the entire 360° video into a standards-compliant stream for progressive download or adaptive streaming. Serving the entire 360° video to the client allows for low-latency rendering (the client has access to the entire 360° video content and can choose to render the portion the user wants to see without further restrictions). From the server's perspective, the same stream can support multiple users with potentially different viewpoints. However, on the downside, because the entire 360° video must be encoded in high quality (e.g., 60fps per eye in 4K, or 90fps per eye in 6K), the video size will be very large, resulting in high transmission bandwidth during video delivery. On the client side, much of this high bandwidth consumption during transmission will be wasted because the user only views a small portion of the entire image (the target viewpoint).

[0010] To reduce bandwidth requirements, alternative projection methods can be used to convert 360° video into 2D planar video. For example, instead of the typical equal-rectangular projection, other geometric projection methods such as pyramid projection, cube projection, and offset cube projection can be used to represent spherical video with less data. For instance, cube projection can typically save up to 20% of pixels compared to equal-rectangular projection because it introduces less warping. Other projection methods, such as pyramid projection, can further resample pixels that are unlikely to be viewed by the viewer to reduce the size of the projected 2D video.

[0011] An alternative approach to storing and transmitting full 360° video between servers and clients is to use view-zone-specific representations. Some projection methods, such as cubemap or pyramid projections, can provide non-uniform quality representations for different view zones (i.e., some view zones can be represented with higher quality than others). In this case, to support all view zones of a spherical video, multiple versions of the same video with different target view zones can be generated and stored on the server side. For example, in Facebook's VR video delivery implementation, an offset cubemap format is used. Offset cubemaps provide the highest resolution (and therefore the highest quality) for the front view zone, the lowest resolution (and therefore the lowest quality) for the rear view zone, and an intermediate resolution (and therefore the middle quality) for the side view zones. To accommodate client requests for different view zones of the same content, the server would store multiple versions of the same content, for example, a total of 150 different versions of the same content (30 view zones, 5 resolutions per view zone). Then, during delivery in a streaming session, the client can request a specific version corresponding to its current view zone, which will be delivered by the server. While this viewport-adaptive streaming method saves bandwidth, its drawbacks include increased storage requirements on the server and potential increased latency if / when the client changes its viewport. The latency issue can be particularly severe when these viewport changes are frequent. Summary of the Invention

[0012] This section describes systems and methods related to monitoring and reducing latency in view-adaptive streaming of 360-degree video.

[0013] In an exemplary method of operating a head-mounted 360-degree video client device, the device displays a first viewport representation of viewport-adaptive 360-degree video to a user of the client device. The first representation is associated with a first viewing direction. While displaying the first viewport representation, the device detects that the user's head orientation has changed to a second viewing direction associated with a second viewport. In response to the detected change in head orientation, the device retrieves a second viewport representation of the 360-degree video associated with the second viewport. The device begins displaying the second viewport representation to the user. The device also measures a viewport switching delay between the detected change in head orientation and the start of the display of the second viewport representation. The device sends a delay report to a server based at least in part on the measured delay.

[0014] In another exemplary method of operating a head-mounted 360-degree video client device, the device tracks a series of user head orientations, each corresponding to one of multiple corresponding viewport representations of viewport-adaptive 360-degree video. The device adaptively requests the viewport representation corresponding to the tracked head orientation. The device receives and displays the requested viewport representation to the user. The device also sends a viewport request report to a metrics server, wherein the viewport request report is based on the number of requests made for each viewport representation.

[0015] In some implementations, the viewport-adaptive streaming method employs a dual-buffered segmented scheduling. In one such implementation, a method is provided that includes initiating viewport-adaptive streaming of 360-degree video from a server to a head-mounted display (HMD). The method further includes displaying a first segment at least partially based on a first orientation of the HMD at the first time, wherein the first orientation is associated with a first viewport. The method also includes requesting, at least partially based on the first viewport, that at least a first base buffer segment be included in a base buffer, wherein the first base buffer segment has a rendering time following the first segment. The method further includes determining, at a second time following the first time, a change in the HMD's orientation from the first orientation to a second orientation, wherein the second orientation is associated with a second viewport. The method also includes requesting a first viewport buffer segment based on the second viewport before the rendering time of the first base buffer segment, wherein the first viewport buffer segment has the same rendering time as the first base buffer segment. The method further includes displaying a second segment at the rendering time, wherein the second segment is either the first viewport buffer segment or the first base buffer segment.

[0016] In one implementation, a method is provided that includes initiating a 360-degree video stream from a server to a user's head-mounted device (HMD). The method further includes requesting that at least a first segment be included in a base buffer. The method also includes requesting that at least a second segment be included in a viewport buffer. The method further includes displaying at least one segment from the base buffer or the viewport buffer based at least in part on the current orientation of the HMD.

[0017] In one implementation, a method is provided that includes estimating the bandwidth (BW) of a 360-degree video stream from a server to a user's head-mounted device (HMD). The method also includes detecting target view segmentation. The method further includes determining that no underlying buffer underflow exists. The method also includes, in response to determining... The target view segment is downloaded to the view buffer in segments, where S is the size of the target view segment, t is the time between the current time and the next rendering time, and Δ is a reserved bandwidth margin to avoid potential mismatches. The method also includes downloading the target view segment to the base buffer in other ways.

[0018] In one implementation, a method for dual-buffer scheduling of block-based viewport-adaptive streaming of 360-degree video is provided, wherein a base buffer is filled with at least one low-quality block representation, and a viewport buffer is filled with at least one high-quality viewport representation at least in part based on the detected orientation of the head-mounted display.

[0019] In one implementation, a method for reducing switching latency for streaming 360-degree video to a head-mounted display is provided, including partial download scheduling.

[0020] In one implementation, a method is provided that includes initiating view-zone-adaptive streaming of 360-degree video from a server to a head-mounted display (HMD). The method further includes displaying a first segment at least partially based on a first orientation of the HMD at the first time, wherein the first orientation is associated with a first view zone. The method also includes requesting and initiating the download of at least a first download segment and a second download segment, wherein the first download segment is associated with the first view zone, and the second download segment is associated with a second view zone, wherein the second view zone is determined at least partially based on the first view zone, and wherein the first and second download segments have a rendering time following the first segment. The method further includes determining a change in the orientation of the HMD from the first orientation to the second orientation at a second time after the first time and before the rendering time, wherein the second orientation is associated with at least a second view zone. The method also includes canceling the download of at least the first download segment at a previously determined time before rendering. The method further includes rendering the second download segment to the HMD.

[0021] In another exemplary embodiment, a method for operating a 360-degree video client device, such as an HMD, tablet, smartphone, or other display device, is disclosed. In this method, a first viewport representation of a viewport-adaptive 360-degree video is displayed to a user of the client device. The first representation is associated with a first viewing direction. While displaying the first viewport representation, the client detects that the user's viewing direction has changed to a second viewing direction associated with a second viewport. In an embodiment using an HMD, the change in viewing direction can be detected by detecting a change in head orientation (e.g., using the HMD's accelerometer, gyroscope, and / or camera components). In an embodiment using other client devices such as tablets or smartphones, the change in viewing direction can be detected based on a change in the device's orientation and / or from the user's scrolling input (e.g., the user swiping on a touchscreen surface to change the viewing direction). In response to the detected change in viewing direction, the client retrieves a second viewport representation of the 360-degree video associated with the second viewport and displays the second viewport representation to the user. The client also measures a viewport switching delay between the detected change in viewing direction and the display of the second viewport representation. The client sends a delay report to a server based at least in part on the measured delay.

[0022] In some implementations, multiple viewport switching delays are measured, and the delay report includes the maximum viewport switching delay among the multiple viewport switching delays. In some implementations, multiple viewport switching delays are measured, and the delay report includes the average (or optionally, the median) viewport switching delay among the multiple viewport switching delays. The delay report can be sent to the DASH-aware network element (DANE) in a server and network-assisted DASH (SAND) message. In some implementations, the viewport representation is retrieved from the same server that sent the delay report.

[0023] In another exemplary embodiment, a method for operating a 360-degree video client device, such as an HMD, tablet computer, smartphone, or other display device, is disclosed. The client device tracks a series of viewing directions of a user, each viewing direction corresponding to one of multiple corresponding viewport representations of viewport-adaptive 360-degree video. The client device adaptively requests the viewport representation corresponding to the tracked viewing direction, and the client receives and displays the requested viewport representation to the user. The client also sends a viewport request report to a metrics server, wherein the viewport request report is based on the number of requests made for each viewport representation.

[0024] In some such implementations, a viewport representation is requested from at least one DASH-aware network element that is different from the measurement server.

[0025] In some such implementations, for each of a plurality of viewport representations, the viewport request report indicates the number of requests made to the corresponding representation. In some implementations, the viewport request report identifies the most frequently viewed viewport representation for at least one time interval.

[0026] In some embodiments, a system is provided that includes a processor and a non-transitory storage medium for storing instructions that, when executed on the processor, are operable to perform functions such as those described herein. Attached Figure Description

[0027] A more detailed understanding can be obtained by referring to the following description, presented in conjunction with the accompanying drawings as examples, wherein:

[0028] Figure 1 An example of an equal rectangular projection from a sphere to a 2D plane is depicted.

[0029] Figure 2 An example of a hierarchical data model for MPD is described.

[0030] Figure 3 The diagram illustrates four types of SAND messages exchanged between network components and clients.

[0031] Figure 4 An implementation of a frame-based viewport frame is shown, in which the central region is encoded with high quality, but other regions are encoded with lower quality.

[0032] Figure 5 An example of a frame-based viewport adaptive streaming method is shown.

[0033] Figure 6 An example of a block-based viewport adaptive streaming media partitioning method is shown.

[0034] Figure 7 An example of block-based viewport adaptive streaming is shown.

[0035] Figure 8 An example of layer-based viewport overlay combination is shown.

[0036] Figure 9 A layer-based viewport adaptive streaming example is shown.

[0037] Figure 10 An example of viewport-adaptive streaming and switching latency is shown.

[0038] Figure 11 An example of a viewport-adaptive streaming segmentation request is shown.

[0039] Figure 12 An example of minimum switching latency is shown.

[0040] Figure 13 An implementation of a double-buffer scheduling method for frame-based viewport adaptive streaming is shown.

[0041] Figure 14 An example of the three degrees of freedom of an HMD is shown.

[0042] Figure 15 A flowchart of one implementation of double-buffered scheduling is shown.

[0043] Figure 16 A diagram illustrating one implementation of segmented request timing is shown.

[0044] Figure 17 A diagram illustrating one implementation of double-buffered scheduling for block-based streams is shown.

[0045] Figure 18 A diagram illustrating one implementation of double-buffered scheduling for layer-based streams is shown.

[0046] Figure 19 A diagram illustrating one implementation of a partial download scheduling method is shown.

[0047] Figure 20 One implementation of gaze tracking analysis is shown.

[0048] Figure 21 An example of one implementation of viewport representation adjustment is shown.

[0049] Figure 22 A flowchart illustrating one implementation of a partial download scheduling method is shown.

[0050] Figure 23 An implementation of the communication flow for SAND messages used in viewport-adaptive streaming is shown.

[0051] Figure 24 A message flow diagram is shown for one implementation of DASH SAND messages for viewport-adaptive streaming.

[0052] Figure 25 An exemplary wireless transmit / receive unit (WTRU) is shown that can be used as a client device or HMD in some implementations.

[0053] Figure 26 Exemplary network entities that may be employed in some implementations are shown. Detailed Implementation

[0054] A detailed description of illustrative embodiments will now be provided with reference to the accompanying drawings. While this description provides detailed examples of possible implementations, it should be noted that the details provided are intended as examples and in no way limit the scope of this application.

[0055] Dynamic Adaptive Streaming Media Based on HTTP (DASH)

[0056] HTTP streaming has become a primary method in commercial deployments. For example, streaming platforms such as Apple's HTTP Live Streaming (HLS), Microsoft's Smooth Streaming (SS), and Adobe's HTTP Dynamic Streaming (HDS) all use HTTP streaming as their underlying delivery method. Standards for HTTP streaming of multimedia content will allow standards-based clients to stream content from any standards-based server, enabling interoperability between servers and clients from different vendors. MPEG Dynamic Adaptive HTTP Streaming (MPEG-DASH) is a delivery format that provides end-users with a high-quality video experience by dynamically adapting to changing network conditions. DASH is built on top of the ubiquitous HTTP / TCP / IP stack. It defines a manifest format called Media Presentation Description (MPD) and defines the ISO Basic Media File Format and the segmentation format for MPEG-2 transport streams.

[0057] Dynamic HTTP streaming requires a variety of bitrate alternatives for the multimedia content to be available at the server. Furthermore, multimedia content can consist of several media components (e.g., audio, video, text), each with different characteristics. In MPEG-DASH, these characteristics are described by a Media Presentation Description (MPD).

[0058] MPD is an XML document that contains metadata used by the DASH client to construct appropriate HTTP-URLs for adaptive access to video segments (described below) during a streaming session. Figure 2 An example of a hierarchical data model for MPD is shown. MPD describes a sequence of time periods in which a consistent set of encoded versions of media content components does not change over a period of time. Each time period has a start time and duration, and consists of one or more adaptive sets.

[0059] An adaptive set represents a set of encoded versions of one or more media content components that share the same attributes, such as language, media type, image aspect ratio, character, accessibility, viewpoint, and rating attributes. For example, an adaptive set could contain different bitrates of video components of the same multimedia content. Another adaptive set could contain different bitrates of audio components of the same multimedia content (e.g., lower-quality stereo and higher-quality surround sound). Each adaptive set typically includes multiple representations.

[0060] A representation describes a deliverable encoded version of one or more media components, distinguished from other representations by bitrate, resolution, number of channels, or other characteristics. Each representation consists of one or more segments. Attributes of the representation element (such as @id, @bandwidth, @qualityRanking, and @dependencyId) are used to specify properties associated with the representation.

[0061] A segment is the largest unit of data that can be retrieved using a single HTTP request. Each segment has a URL that identifies an addressable location on the server. The segment can be downloaded using an HTTP GET request with a byte range.

[0062] To use this data model, the DASH client parses the MPD XML document and selects a set of adaptive sets suitable for its environment based on the information provided in each adaptive set element. Within each adaptive set, the client typically selects a representation based on the value of the @bandwidth attribute, but also considers the client's decoding and rendering capabilities. The client downloads the initial segments of the selected representation and then accesses the content by requesting the entire segment or byte range of a segment. Once a representation begins, the client continues to consume media content by continuously requesting media segments or portions of media segments and playing the content according to the media rendering timeline. The client may switch representations based on updated information from its environment. The client should play content continuously across time slots. Once the client has consumed the media contained in a segment and is nearing the end of the media announced in the representation, it can terminate media rendering, start a new time slot, or re-fetch the MPD.

[0063] Part 5 of the DASH standard, Server and Network Assisted DASH (SAND), specifies messages between DASH clients and network elements or between various network elements by providing information about the real-time operational characteristics of networks, servers, proxies, caches, CDNs, as well as the performance and status of DASH clients, to improve the efficiency of streaming media sessions.

[0064] Figure 3 The diagram illustrates four types of SAND messages exchanged between network components and clients.

[0065] - Parameter Enhancement Delivery (PED) messages are exchanged between DASH-aware network elements (DANEs).

[0066] - Parameter Enhancement Receive (PER) messages are sent from DANE to the DASH client.

[0067] Status messages are sent from the DASH client to the DANE. Status messages provide real-time feedback from the client to the DANE to support real-time operation.

[0068] Metric messages are sent from the DASH client to the metrics server. Metric messages provide a session summary or longer time intervals.

[0069] Viewport Adaptive Streaming

[0070] View-region adaptive streaming methods deliver a high-quality view region and a lower-quality region to reduce bandwidth. Several adaptive view-region streaming methods exist, including frame-based, block-based, and layer-based methods.

[0071] Frame-based view-region adaptive streaming methods encode the entire 360 ​​video into a single bitstream, where the view region within the bitstream is encoded in high quality, while other regions of the same bitstream are encoded in low quality. This approach can be achieved through preprocessing or rate control. The aforementioned projection methods, such as pyramid and offset cube mapping, represent a form of preprocessing. Similar methods, such as applying low-pass filtering to non-view regions before encoding, can also reduce the visual quality of non-view regions. The same result can be achieved by allocating more bits to the view region and fewer bits to other regions during encoding through rate control. Figure 4 An implementation of frame-based viewport frames is illustrated, where the central region is encoded in high quality, while other regions are encoded in lower quality. The streaming server stores multiple representations of the same content with different target viewport regions at different bitrates, and the client requests a specific representation based on head orientation and available bandwidth. The client can switch between different representation segments as the head orientation changes.

[0072] Figure 5 An example of this frame-based viewport adaptive streaming method is shown. The server has six representations, each including all viewports: front viewport (F), right viewport (R), back viewport (B), left viewport (L), top viewport (T), and bottom viewport (O), with a specific viewport of high quality ( Figure 5 (High-quality viewport is highlighted in bold). When the user is viewing the front viewport, the segment with the high-quality front viewport is transmitted to the user. When the user moves to the right viewport, the segment with the high-quality right viewport is transmitted to the user, and so on.

[0073] The block-based viewport adaptive streaming method divides the 360° video projection frame into multiple blocks, such as... Figure 6As shown, each block can be independently encoded into multiple quality level representations. The client can request different quality level representations for each block based on header orientation and available bandwidth, and then combine all blocks together for rendering. Compared to frame-based methods, block-based methods have lower storage costs, but they require additional combination processing on the client side.

[0074] Figure 7 This illustrates a block-based viewport-adaptive streaming example. Based on head orientation, the client can request a high-quality corresponding viewport block representation (...). Figure 7 (Bolded) and low-quality remaining blocks. All blocks can be combined into a complete 360° video projection frame to be rendered to the user.

[0075] The layer-based approach encodes the entire 360° video as a low-quality representation as a base, and encodes each individual viewport as a different quality level. The client always requests the base representation and the specific viewport representation based on the head orientation, and then overlays the high-quality viewport on top of the base frame for rendering. Figure 8 It is an instance based on the superposition and combination of layers.

[0076] Figure 9 This illustrates an example of layer-based viewport-adaptive streaming. A low-quality base representation comprises all viewports, and each viewport (T, O, F, R, B, L) is encoded separately as a high-quality representation. Based on head orientation, the low-quality base representation and a specific viewport representation are delivered to the user to compose a 360° video for rendering. Compared to block-based methods, layer-based methods can use fewer copies of the representation stored on the server side. The viewport representation can be encoded independently of the base representation, or it can be encoded as a scalable enhancement layer, such as a scalable extension of HEVC (SHVC), which depends on the base representation, thus reducing the viewport representation size.

[0077] Viewport-adaptive streaming reduces transmission bandwidth but requires more storage space and introduces latency when switching between different viewports. Since only the target viewport is of high quality and the rest of the video is of poor quality, the latency between viewport changes and the rendering of the corresponding viewport representation can severely impact the user experience. Figure 10 This is an example of view-zone adaptive streaming. The time it takes for a user to change their viewing direction from one viewzone to another is denoted as t. c The time interval for a user to request segments from different viewports is represented as t. r And the time segmentation of different viewpoints is represented as t. p . Figure 10 This shows the user at time t while a forward-looking segment (F0) is being rendered. cSwitch from the front view (F) to the right view (R). The download buffer contains three front view segments (F1, F2, and F3), and one segment (F4) is being downloaded. As a result, in t r Request the first right view segment R5, and the user may not be able to see until t p Only then did I view the right visual segment R5. c and t p The time interval between these segments is the switching latency. A negative user experience can be addressed using shorter segment lengths, but this can also reduce encoding efficiency. Another way to reduce this switching latency is to reduce the download buffer size to keep fewer segments; however, a smaller download buffer size can also affect streaming quality adaptation, for example, by causing buffer underflows to occur more frequently.

[0078] Switching latency issues exist in all three of the aforementioned viewport-adaptive streaming methods. This disclosure describes systems and methods to address switching latency and other problems.

[0079] Exemplary Double-Buffer Segmented Scheduling Method

[0080] Switching latency depends on factors such as segment length, download buffer size, orientation detection frequency, and segment request time. Because the viewport can change constantly, each segment stored in the buffer may not match the current viewport at render time. Pre-downloading all these segments may not be worth the bandwidth investment, but a sufficient buffer fill level should be maintained to prevent buffer underflow. Figure 11 Is with Figure 10 Compared to the simplified example, the view segment is defined as S(i,j) of the j-th segment of the i-th view region. The length of each segment is fixed at T. s The download buffer can hold N segments. When segment S(p, 0) is playing, the header is facing the segment at t. c The client changes from the p-th viewport to the q-th viewport. r Request S(q, N+1)a, where the rendering time of S(q, N+1) is t. p . t c and t p The time interval between them is the switching delay T. L , where (N)*T s ≤T L ≤(N+1)*T s .

[0081] Streaming segment requests typically occur sequentially after the previous segment has finished downloading. In this way, segments can be appropriately selected based on the most recent bandwidth estimate. For viewport-adaptive streaming, the ideal segment request time is t. r Should be with t pThe same, so that the downloaded view segments can be rendered at the same time (t). p Matching the header orientation. However, this requires very high bandwidth. Reducing t r and t p The time interval between them helps reduce switching latency. For example... Figure 12 As shown, by setting the buffer size to a segment, the switching latency can be reduced to less than or equal to 2*T. s The segmentation request occurs at the end of the playback of each segment.

[0082] exist Figure 12 In the middle, when the head orientation occurs at the start of rendering segment S(p, 0), the switching delay is equal to 2*T. s And when the head faces the end of the rendering of segment S(p, 0), the switching delay is equal to T. s However, this small buffer size requires sufficient network bandwidth (BW) to fully download high-quality view segments based on the following constraints to avoid buffer underflow.

[0083] BW≥(S(q,j+1) / T s Equation 1

[0084] Where S(q, j+1) is the size of the (j+1)th segment of the qth view region, and T s This refers to the segment length. The estimated bandwidth (BW) is... est This may not match the actual available bandwidth (BW). When the estimated bandwidth is higher than BW, a buffer underflow may occur when bandwidth unexpectedly drops. Furthermore, because segmentation requests only occur when a segment download is complete, the bandwidth estimate may not be up-to-date if the last segment download completes before the next segment request. Additionally, bandwidth utilization may not be efficient during periods when no segments are being acquired.

[0085] To prevent buffer underflow and reduce switching latency, this paper proposes a dual-buffer segmented scheduling method for view-adaptive streaming media.

[0086] Figure 13 This is a diagram of one implementation of a dual-buffer scheduling method for frame-based viewport-adaptive streaming scenarios. There are two buffers: a base buffer and a viewport buffer. The base buffer, similar to a regular streaming download buffer, is used to adapt to bandwidth fluctuations and maintain an appropriate buffer fullness level to prevent underflow or overflow. The viewport buffer stores segments rendered immediately after the current segment to minimize switching latency.

[0087] Because each segment in the download buffer can be extracted before the most recent head orientation detection, the viewport of each segment may not match the current head orientation during rendering. Therefore, unless the head orientation prediction is reliable, requesting a high-quality, high-bitrate representation may not be worthwhile. The client can collect the user's head orientation data during playback and analyze orientation consistency to determine which quality level of segment should be requested for the download buffer. When the head moves slowly or presents a specific high-priority viewport (e.g., a specific viewport might be marked as high-priority by a film director or content provider for a specific duration), a high-quality predicted viewport representation can be requested. Otherwise, a low-quality viewport representation is requested to maintain the buffer fullness level and conserve bandwidth.

[0088] The head orientation at time t is denoted as V(t), with a horizontal orientation ranging from -180° to +180° and a vertical orientation ranging from -90° to 90°. The quality level of a particular viewport representation in the basic buffer can be selected based on factors such as the intensity of head movement, for example, by measuring three degrees of freedom (pitch, yaw, and roll) over a period of time (e.g., 2 seconds). Figure 14 An example of a 3-DOF HMD is shown. When the head moves around the X (e.g., The rotation orientation of the Y-axis (e.g., θ) changes by less than a predetermined threshold. At times, due to slow head movement, the DASH client may request medium or high-quality view segmentation. Otherwise, it may request the lowest rate representation segmentation. Another alternative implementation is to determine the most frequently viewed orientation from viewing history data analysis and always request the segmentation of the most frequently viewed view area.

[0089] The viewport buffer stores the target viewport segment to be rendered immediately after the current rendering segment. This segmentation minimizes switching latency because it is requested based on the most recently detected head orientation. Figure 15 The requested timer t is shown. r This is critical for switching latency. r and t p (The smaller the interval between j, the more accurate the orientation detection. However, small intervals may require high bandwidth to complete the download of the entire viewport target segment in a timely manner. If the request occurs too early, the latest head orientation change may not be captured, but time will be available to download the segment. If the request occurs too late, the requested viewport may match the viewport orientation, but the corresponding segment download may not be completed in time.)

[0090] Let the presentation time of the j-th segment be t. p (j), and for simplicity, assume t p(j) The j-th segment is the same for all view regions. The bandwidth is BW, and the head orientation change time is t. c And the request S(i,j) for the j-th segment of the i-th view region occurs at t r To obtain the entire S(q, j), the following equation must be satisfied.

[0091] BW≥S(i,j) / (t) p (j)-t r Equation 2

[0092] t r ≤t p Equation 3 (j)-(S(i,j) / BW) states that since there are two view segmentation candidates, S(p,j) and S(q,j), in the basic buffer and view buffer, the client can base its view segmentation on t. p The head at (j) is oriented towards one of the segments to be presented and the other is removed from the buffer.

[0093] Figure 15 A flowchart of the double-buffered scheduling process is shown. S is the size of the target view segment, and t is the time distance between the current time and the next rendering time. Δ is the reserved bandwidth margin to avoid potential mismatches between the estimated bandwidth (BW) and the actual available network bandwidth.

[0094] When each view segment has multiple quality levels, the j-th segment of the i-th view segment of the k-th quality level can be represented as S(i,j,k), where

[0095] S(i, j, 0) < S(i, j, 1) < ... < S(i, j, k) < ... < S(q, j, M), assuming the i-th viewport represents M+1 quality levels. Users can determine the requested quality level based on available bandwidth and changes in head orientation. Given the same available bandwidth BW, a client might prefer to select the medium quality segment S(i, j, k), where k < M, such that if head orientation changes frequently, the request time t... r It can get closer to t p (j). If the head orientation is consistent over time, the client can select the highest quality segment S(i,j,M) at an earlier time. Figure 16 This illustrates how different quality levels of the j-th segment in the q-th viewport are mapped to different segment request times. So that at presentation time t p (j) Example of a previously completed download.

[0096] In the event that network bandwidth drops rapidly and the corresponding target view segment S(q,j,k) cannot be fully downloaded, the streaming client can retrieve the j-th segment from the basic buffer for rendering without stopping playback.

[0097] When the head is facing t r When a view segment in the basic buffer is matched, the client can determine not to request a segment of the same view segment again, or the client can determine to request a higher-quality segment of the same view segment. Once the target view segment has been downloaded, the client can continue to request subsequent segments to fill the basic buffer.

[0098] The proposed double-buffer segmented scheduling method is illustrated above for frame-based view-region adaptive streaming. However, the same method can also be used for block-based and layer-based view-region adaptive streaming.

[0099] For block-based viewport adaptive streaming, the base buffer holds all lower-quality block representations, while the viewport buffer stores specific high-quality viewport representations, depending on t. r The latest head orientation detection. Figure 17 An exemplary double-buffer scheduling method for block-based viewport-adaptive streaming is shown. For simplicity, only two quality levels are presented: S L (j, k) represents the low-quality j-th segment representation of the k-th block, and S H (j, p) represents the high-quality j-th segment representation of the p-th block. The client requests low-quality blocks to fill the base buffer and in t r The request requests high-quality blocks, and the number of high-quality blocks requested depends on the available bandwidth and the number of blocks in the target view area. Figure 17 Two high-quality blocks are shown in response to the viewport q request.

[0100] Because block segment sizes are smaller than frame-based segmentation, the bandwidth required to download high-quality blocks can be significantly lower than that required for frame-based viewport-adaptive streaming. Therefore, the request time t r It can get closer to t p (j) makes head orientation detection more accurate. Assuming the total number of high-quality blocks to be requested is K, the required bandwidth (BW) or segmented request time t can be derived from the following equation. r .

[0101]

[0102]

[0103] In another implementation, it is not necessary to request all low-quality blocks for the base buffer. The download priority of each block can be determined by its distance from the previously detected target view area. Previously detected view area blocks and surrounding blocks that are likely to be viewed are downloaded to the base buffer first, while blocks far from the target view area can be downloaded later or not at all, depending on available bandwidth and head motion analysis. The priority of each block (within or outside the view area) can also be determined based on its importance, where the importance of a block can be measured in various ways, such as by the total amount of motion in the block, the contrast of the block, or other quality-related features of the block. More important blocks can be downloaded to the base buffer first, while less important blocks can be downloaded later or not at all, depending on available bandwidth and head motion analysis. As a result, the bandwidth saved by downloading partial blocks can be used to download higher-quality block segments. In the worst case, if the user is viewing an unavailable block, the corresponding previously downloaded block can be used to fill that gap.

[0104] In another implementation, there can be multiple blocks within the viewport. Not all of these blocks need to be of high quality or have the same quality requirements. More important blocks can be requested at the same or higher quality and / or before less important blocks, which will then be requested at the same or lower quality and / or after more important blocks. The importance of a block can be measured in various ways, such as by the total amount of motion within the block, the block's contrast, or other quality-related characteristics of the block. Therefore, a better experience can be achieved by using the same or less amount of resources (e.g., bandwidth).

[0105] For layer-based viewport adaptive streaming methods, it is preferable to download the basic representation first to avoid buffer underflow. Figure 18 An example of double-buffered scheduling for layer-based viewport-adaptive streaming is shown. The j-th segment of the basic representation is denoted as S. L (j), the high-quality j-th segment of the q-th view region is represented as S. H (j, q). Similar to block-based streaming scheduling, one or more view segments can be downloaded to overlay on the base frame, depending on the field of view and available bandwidth. Equations (4) and (5) also apply to layer-based streaming methods. If high-quality view segments cannot be downloaded in time, the base representation is used as a backup video for presentation.

[0106] With layer-based scalable 360° video representations available, clients can also use a double-buffered scheduling method to request independent viewport segments for enhancement layer segments instead of viewport buffers. Enhancement layers can cover part or all of the 360° video's viewport, and if enhancement layer segments cover more viewports, the scheduling scheme can be implemented at an earlier time (t...). rThe request time is t. Additional signaling (e.g., viewportId) can be used to identify the viewports included in the enhancement layer. If the enhancement layer covers all viewports with the same quality, the request time is t. r It may depend solely on bandwidth, as it is no longer viewport-adaptive streaming.

[0107] Example download scheduling method

[0108] In another implementation, an alternative method to reduce switching latency could be as follows: Figure 19 The download schedule is shown in the diagram. In reality, it's a segmented download for multiple recognized view regions. For example, matching in t... p The method detects the orientation of the view region at position (j-1), such as S(p,j), and its neighboring view regions, such as S(p+1,j), S(p+1,j), ..., S(q,j). This method assumes that the user's motion is moderate and t p The head orientation at (j) can be matched with a visual region next to S(p,j). The selection of the additional visual region can be based on data analysis of head angular velocity and body motion to estimate the orientation of the next target visual region.

[0109] Existing streaming protocols allow clients to specify the number of bytes in a segment to download, rather than downloading the entire segment. At some point t... d The final matched view segment S(q, j) is determined for continuous download, while downloading segments from other view segments is terminated. As a result, the switching delay can be less than or equal to a segment length T. s .

[0110] This partial download method improves view area detection accuracy and minimizes switching latency, but may require additional bandwidth to partially download segments of all potential view areas.

[0111] Let S represent the average size of each view segment, and α represent the percentage of data partially downloaded. The number of view segments initially requested is denoted by N. The value of N depends on the user's gaze tracking analysis and the estimated bandwidth. Motion analysis can determine the value of N using the total number of view segments the user views over a period of time. For example, if the user's viewing range traverses a 3x2 view segment in the past 2 seconds, such as... Figure 20 As shown, the value of N can be set to 6, and six segments of the current viewport and its surrounding viewports can be requested. Another approach is to calculate how many viewport segments the available bandwidth can accommodate. For example, if the average segment size is 1MB and the segment length is 2 seconds, then α is set to 0.4, and each segment will download 400KB during the first second; then the value of N would be 10 when the available bandwidth is 4Mbps. A combination of gaze tracking analysis and bandwidth estimation can also be used to determine the value of N.

[0112] The time required to identify the specific view segment to be completed is denoted as t. d Choose parameters that satisfy the following equation to avoid buffer underflow. By utilizing more bandwidth (BW), the client can request more potential viewports (N), or increase the download percentage α to postpone viewport orientation decisions.

[0113] BW≥((N*S*α)+S*(1-α)) / T s Equation 6

[0114] The time to identify the final head orientation is t d This can be derived from Equation 7. In t d It can continuously download the remaining data of the identified view segment and discard some segments of the remaining view areas. A higher value of α allows decisions to be made closer to presentation time, meaning head orientation detection can be more accurate. On the other hand, more bandwidth may be spent on mismatched view segmentation.

[0115] t d =((N*S*α) / (BW*T) s ))+t p (j-1) Equation 7

[0116] Based on user motion analysis and available bandwidth, the client can determine appropriate values ​​for N and α to substantially minimize handover latency in order to enhance the user experience.

[0117] With different quality level representations for each viewport, the client can request the most likely quality level for t. p (j) High-quality representation of those view areas viewed from point t, and for those in t p The low-quality representation of the view region that is unlikely to be viewed at (j). Equations (6) and (7) can be expressed as follows, where N is the total number of view region segments initially downloaded:

[0118]

[0119]

[0120] The balance between BW, N, and α can be dynamically adjusted to avoid buffer underflow. For example, when BW decreases, if the head orientation is consistent, the client can reduce N or minimize α as much as possible to determine the target view area. Figure 21 As shown in the example diagram, when bandwidth decreases, the number of segments downloaded in a partial download decreases.

[0121] Figure 22This is a flowchart of a partial download scheduling method according to an implementation. S is the size of the target view segment, and t is the time distance between the current time and the next rendering time. Δ is the reserved bandwidth margin to avoid potential mismatches between the estimated bandwidth (BW) and the actual available network bandwidth. For the basic buffer, the entire segment is downloaded in the event of a basic buffer underflow. Otherwise, partial segments can be downloaded for the basic buffer to increase the frequency of updating N and α.

[0122] This partial download scheduling method can be applied to viewport buffers for both block-based and layer-based viewport adaptive streaming. The client can begin downloading partial data segments of multiple identified high-quality viewports to the viewport buffer, and the client can then... d The final target viewports are determined to complete the download. The number of target viewports can be less than or equal to the number of viewports partially downloaded.

[0123] With layer-based scalable 360° video representations available, clients can also partially use a partial download scheduling method to request enhancement layer segments instead of individual viewport segments. Enhancement layers can cover part or all of the 360° video viewport. If a particular enhancement layer segment covers more viewports, the scheduling scheme can reduce the total number of segment candidates N. Additional signaling (e.g., viewportId) can be used to identify the viewports included in the enhancement layer. When the enhancement layer covers all viewports of equal quality, time t is determined. d It may depend solely on bandwidth, as it is no longer viewport-adaptive streaming.

[0124] Exemplary implementation using SAND messages

[0125] In some implementations, to improve the efficiency of viewport buffer downloads, DASH SAND messages can be used to support real-time operation of double-buffer scheduling and partial download scheduling methods. For example, a client can use status messages such as Anticipated Requests (Table 1) to notify the server which set of viewport segments it is interested in by setting the value of @sourceUrl to the URL of these potential viewport segments. The client can use AbsoluteDeadline (Table 2) to indicate the absolute deadline of the wall clock time and when the requested segment needs to be fully received. As a result, the server can pre-cache the corresponding segment to the client's nearest network element, such as a CDN or edge node. The server can also use PER messages, ResourceStatus (Table 3), or DaneResourceStatus (Table 4) to inform the client of the cache status of the corresponding viewport segment so that the client can select the appropriate viewport segment to request.

[0126] Table 1 - Expected Request Parameters

[0127]

[0128]

[0129] Table 2 - Absolute Deadline Parameters

[0130]

[0131] Table 3 - Resource Status (with baseUrl parameters)

[0132]

[0133] Table 4 - Dane Resource Status Parameters

[0134]

[0135]

[0136] PER message throughput (Throughput), with guaranteed throughput measured in bits per second (as shown in Table 5), can also be used to improve the performance of the proposed scheduling method. Clients can have prior information about the download time for a specific view segment and a request time t for the double-buffered scheduling method. r Or it can be determined that t is used for partial download scheduling methods. d .

[0137] Table 5 - Throughput (with baseUrl) Parameters

[0138]

[0139] In an exemplary embodiment, in addition to the existing messages, two operational features of viewport adaptive streaming are added to the SAND message: ViewportRequests and ViewportSwitchLatency.

[0140] ViewportRequests messages can include the number of requests for each viewport representation or the most-viewed viewport representation for a specific time interval. DASH servers can collect this information on the server side for a specific client, but this may not cover the entire streaming session if clients request segments of the same viewport from different servers. ViewportRequests messages can be used as metric messages to provide a summary of a specific 360° video streaming session, such as the most-requested viewport representation for each time interval. DANE can use such statistics to allocate more resources to the most-requested viewport representations to improve user experience and remove the least-requested viewports to save storage space. Table 6 shows exemplary ViewportRequests parameters, with each viewport representation identified by a viewportId (viewportId).

[0141] Table 6 - Viewport Request Parameters

[0142]

[0143] In some implementations, the ViewportSwitchLatency message serves as a status message, providing real-time latency feedback from the client to the DANE to support real-time operation. If the handover latency exceeds expectations, the DANE can allocate more bandwidth to a specific client link. Table 7 shows an exemplary parameter format for ViewportSwitchLatency. In some implementations, the latency parameter specified in Table 7 is the maximum viewport-oriented handover latency. In other implementations, the latency parameter can be the average or median of the handover latency over a given time period, or another latency measure.

[0144] Table 7 - View Segmentation Delay Parameters

[0145]

[0146]

[0147] Figure 23The SAND message used in viewport-adaptive streaming in an exemplary implementation is shown to improve user experience and real-time operation.

[0148] Figure 24 An example of exchanging SAND messages between DANE, DASH clients, and metrics servers to improve real-time operation and metrics collection is illustrated. DASH clients can send status messages such as AnticipateRequests and AbsoluteDeadline to the origin server to indicate which viewport it can request based on the current header orientation prediction, and the absolute deadline when the requested segment needs to be fully received. The origin server can also send Parameter Enhancement Receive (PER) messages to the DASH client, such as ResourceStatus, DaneResourceStatus, and Throughput, to inform the DASH client of the availability of the current time segment and the throughput characteristics from the origin server to the DASH client. The same status messages can also be exchanged between DANE CDN / caching and DASH clients when the corresponding data is hosted by a CDN / caching server.

[0149] The DASH client requests a new viewport segment based on the header orientation from the origin server, and the origin server passes the segment to the client. The client can calculate the switching latency and send ViewportSwitchLatency back to the origin server to inform the server of the real-time 360° video streaming status. If performance is lower than expected, the server can reallocate resources for the session. For example, the origin server can distribute the segment to a cache closer to the client to reduce latency. The DANE cache can also notify the client of the availability of the corresponding segment on the cache, allowing the client to request the next segment directly from the cache. The client can continue to send ViewportSwitchLatency status to the DANE origin server or the DANE cache to improve real-time operation performance. At the end of the viewing session, the client can collect viewport statistics and send a metric message ViewportRequests to the metrics server to indicate which viewport was viewed most frequently. Statistics on the most-watched viewports can be used to improve caching strategies and marketing analytics.

[0150] Figure 25An exemplary wireless transmit / receive unit (WTRU) 102 is shown that can be used as a client device or HMD in some embodiments. WTRU 102 includes a communication interface 119 with a transceiver 120 and an antenna 122 for wireless communication 116. WTRU 102 also includes a processor 118 connected to a speaker / microphone 124, a keyboard 126, a display / touchpad 128, non-removable memory 130, removable memory 132, peripherals 138, a GPS chipset 136, and a power supply 134.

[0151] Figure 26 An exemplary network entity 190 that may be employed in some implementations is shown. Network entity 190 includes a communication interface 192, a processor 194, and a data memory 196 having instructions 197.

[0152] Note that the various hardware elements in one or more of the described embodiments are referred to as “modules”, which perform (i.e., implement, carry out, etc.) the various functions described herein in conjunction with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that are deemed suitable for a given implementation by one of those skilled in the art. Each described module may also include instructions executable to implement one or more functions performed by the respective module, and it should be noted that these instructions may take or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or similar instructions, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0153] Although the features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (e.g., internal hard disks and removable hard disks), magneto-optical media, and optical media (e.g., CD-ROM disks and digital universal disks (DVDs)). A processor associated with the software can be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Render the first segment S(p, j-1) of the viewport-adaptive 360-degree video. The first segment is associated with the first target viewport p and the first time period j-1. The first segment includes all viewports of the video and has only the high-quality viewport p. The second segment S(p, j) of the video is retrieved via the network. The second segment is associated with the first target view area p and the second time period j following the first time period. The second segment includes all view areas of the video and only has the high-quality view area p. The user's viewing direction has been detected to have changed from the direction associated with the first target view area p to the direction associated with the second target view area q. In response to the detected change in viewing direction, a third segment S(q,j) of the video is retrieved via the network. The third segment is associated with the second target view area q and the second time period j. The third segment includes all view areas of the video and has only a high-quality view area p. as well as The second segment or the third segment is rendered during the second time period.

2. The method according to claim 1, further comprising: Based on the user's start time t in the second time period p (j) determines whether the viewing direction is to render the second segment or the third segment.

3. The method according to claim 1, further comprising: Estimate the bandwidth on the network, wherein only in response to determining that the estimated bandwidth is large enough to allow at the start time t of the second time period. p (j) The third segment was retrieved previously, and the retrieval of the third segment was performed.

4. The method according to any one of claims 1-3, wherein the first segment and the second segment are stored in a base buffer, and wherein the third segment is retrieved only in response to determining that the base buffer is not predicted to underflow.

5. The method according to any one of claims 1-3, wherein the third segment is stored in the viewport buffer.

6. The method according to any one of claims 1-3, further comprising: The quality level for the second segment is determined at least in part based on the level of user head movement, wherein a higher quality level is selected for the lower quality level of head movement. The second segment is retrieved according to the determined quality level.

7. The method of claim 2, wherein the first segment and the second segment are stored in a base buffer, the third segment is stored in a viewport buffer, and wherein, In response to determining that the third segment should be rendered, the second segment is removed from the base buffer.

8. An apparatus comprising at least one processor, said at least one processor being configured to perform: Render the first segment S(p, j-1) of the viewport-adaptive 360-degree video. The first segment is associated with the first target viewport p and the first time period j-1. The first segment includes all viewports of the video and has only the high-quality viewport p. The second segment S(p, j) of the video is retrieved via the network. The second segment is associated with the first target view area p and the second time period j following the first time period. The second segment includes all view areas of the video and only has the high-quality view area p. The user's viewing direction has been detected to have changed from the direction associated with the first target view area p to the direction associated with the second target view area q. In response to the detected change in viewing direction, a third segment S(q,j) of the video is retrieved via the network. The third segment is associated with the second target view area q and the second time period j. The third segment includes all view areas of the video and has only a high-quality view area p. as well as The second segment or the third segment is rendered during the second time period.

9. The apparatus of claim 8, further configured to: based on the start time t of the user in the second time period p (j) determines whether the viewing direction is to render the second segment or the third segment.

10. The apparatus of claim 8, further configured to: estimate bandwidth on the network, wherein only in response to determining that the estimated bandwidth is large enough to allow at the start time t of the second time period p (j) The third segment was retrieved previously, and the retrieval of the third segment was performed.

11. The apparatus according to any one of claims 8-10, wherein the first segment and the second segment are stored in a base buffer, and wherein the third segment is retrieved only in response to determining that the base buffer is not predicted to underflow.

12. The apparatus according to any one of claims 8-10, wherein the third segment is stored in a view buffer.

13. The apparatus according to any one of claims 8-10, further configured to perform: The quality level for the second segment is determined at least in part based on the level of user head movement, wherein a higher quality level is selected for the lower quality level of head movement. The second segment is retrieved according to the determined quality level.

14. The apparatus of claim 9, wherein the first segment and the second segment are stored in a basic buffer, the third segment is stored in a viewport buffer, and wherein, In response to determining that the third segment should be rendered, the second segment is removed from the base buffer.

Images