360-degree video slice super-resolution rendering compensation method based on short-time viewport prediction

Abstract

The invention discloses a 360-degree video slice super-resolution rendering compensation method based on short-time viewport prediction, and belongs to the technical field of virtual reality, video streaming media transmission and image processing. Aiming at the problem of image quality loss caused by long-time prediction deviation, the method comprises the following steps: collecting real-time head movement data of a user at a client, and calculating an accurate viewport position by using a short-time prediction model; comparing the cache queue, and screening out a slice which is located in the precise viewport and only has low resolution as a compensation object; and performing real-time image enhancement on the slices by using a local super-resolution model, and splicing and rendering the slices with the original high-resolution slices. According to the method, the high accuracy of short-time prediction is fully utilized, and on the premise that the network transmission burden is not increased, the image blurring caused by long-time prediction errors is immediately repaired through the terminal computing power, so that the immersive experience of a user is remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of virtual reality, video streaming transmission and image processing technology, and more specifically, to a 360-degree video slice rendering compensation method based on short-time viewport prediction. Background Technology

[0002] In recent years, with the popularization of 5G networks and virtual reality devices, 360-degree panoramic video applications have become increasingly widespread. Unlike traditional planar video, 360-degree video contains image data covering the entire sphere, resulting in huge bandwidth consumption. However, due to the limitations of human visual range and the characteristics of head-mounted display devices, viewers can only see a limited area within the viewport at any given time, while most of the remaining spherical image remains invisible.

[0003] Based on this characteristic, to provide a high-quality experience with limited bandwidth, current mainstream technologies typically employ a viewport-adaptive slice transmission scheme. This scheme first unfolds the 360-degree video into a two-dimensional planar video using equidistant cylindrical projection, and then divides its space into several independent rectangular slices on this projection plane. Subsequently, the system transmits only the slices within the user's current viewport at a high bitrate, while slices outside the viewport are transmitted at a low bitrate. While this mechanism significantly reduces bandwidth pressure, it also heavily relies on the accuracy of viewport prediction. The system needs to predict the user's gaze area on the ERP plane 1 to 2 seconds in advance and request the corresponding high-definition slices accordingly.

[0004] However, due to the highly random and sudden nature of user head movements, the accuracy of such long-term viewport predictions, which can last 1 to 2 seconds, is often low. When long-term predictions deviate, the predicted viewport cannot be perfectly aligned with the actual viewport viewed by the user. This results in the area the user is actually looking at (especially the edge of the viewport or a new area after a rapid head turn) having only low-bitrate slices in the local cache. Due to the round-trip time (RTT) of network transmission, the client is unable to promptly request and download the corresponding high-resolution slices from the server to correct this deviation, thus causing the user to see a blurry image or obvious pixelation.

[0005] This invention proposes a 360-degree video slice super-resolution rendering compensation method based on short-time viewport prediction. While retaining the existing long-time prediction download mechanism, this invention introduces a local compensation mechanism on the client side: Within a very short time before image rendering, the precise viewport center on the ERP projection plane is calculated using high-accuracy short-time viewport prediction, and the short-time prediction viewport coverage area is delineated accordingly. By filtering out prediction deviation slices within this area that were not downloaded in high-definition due to long-time prediction errors, real-time image reconstruction and image quality enhancement are performed using a local super-resolution model. This invention cleverly utilizes the accuracy of short-time prediction and the terminal's computing power to effectively solve the image quality degradation problem caused by viewport prediction errors without increasing the network transmission burden. Summary of the Invention

[0006] This invention provides a 360-degree video slice super-resolution rendering compensation method based on short-time viewport prediction. By combining the accurate viewport predicted by the client in a short time, the low-quality slices located within the viewport are subjected to super-resolution processing, which can fix the problems of mismatch between the predicted viewport and the real viewport, blurred image quality at the viewport edge, or missing high-resolution slices caused by long-time viewport prediction errors.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] A method for super-resolution rendering compensation of 360-degree video slices based on short-time viewport prediction includes the following steps:

[0009] 1) During 360-degree video playback, the client records the user's head posture data in real time and maintains a long-term viewport prediction result record for bitrate allocation;

[0010] 2) Within a preset time window before video frame rendering, based on the latest head pose data recorded in 1), the precise viewport center for future rendering moments is calculated using a short-time viewport prediction model.

[0011] 3) Based on the accurate viewpoint center predicted in 3), delineate the short-term predicted viewpoint coverage area;

[0012] 4) Map and compare the short-term prediction viewport coverage area defined in 4) with the current cache queue to filter out the prediction deviation slice set;

[0013] 5) Call the client-preset super-resolution model to perform real-time image enhancement on the slices in the prediction bias slice set to generate high-resolution compensation slices;

[0014] 6) Perform spherical projection stitching between the compensation slice generated in step 5) and the original high-resolution slice in the cache to complete the image rendering.

[0015] Preferably, in step 1), during 360-degree video playback, the client records the user's head pose data in real time and maintains a long-term viewport prediction result record for bitrate allocation, including:

[0016] The client uses the device's built-in inertial measurement unit to collect the user's head posture data in real time at a fixed sampling frequency. This posture data is represented as Euler angles. The client maintains a historical posture queue to store head movement trajectories over a recent period (e.g., the past 2 seconds).

[0017] Meanwhile, the client maintains a "long-term viewport prediction result record," also known as a TileQuality Map. Due to round-trip time (RTT) in network transmission, the server-side bitrate allocation is typically based on long-term prediction results from 1-2 seconds prior. Therefore, the quality of video tiles stored in the client's download buffer is inconsistent: areas predicted to be viewed by the user in long-term prediction are designated as high quality, while areas predicted not to be viewed are designated as low quality. This record is updated in real-time for each tile index in the current buffer. The corresponding resolution level (e.g., 4K, 1080P or 720P).

[0018] Preferably, step 2), within a preset time window before video frame rendering, calculates the precise viewport center for future rendering moments using a short-time viewport prediction model based on the latest head pose data, including:

[0019] Since user head pose data is usually represented by yaw and pitch in spherical coordinates, there are periodic boundary abrupt changes (e.g., from 359 degrees to 1 degree), the historical viewport coordinates need to be made continuous first.

[0020] No. Continuous yaw angle at any moment The calculation formula is:

[0021]

[0022] in, This represents the time sampling index within the historical observation window, with a value range of [value range missing]. , For window length, Indicates the first The raw yaw angle observations collected at each moment, with a range of values ​​of: , Indicates the first The cumulative period offset of the yaw angle at any given time, with an initial value of 0. This represents the algebraic difference between the original yaw angles at two adjacent sampling times. Indicates the first The raw yaw angle observations collected at each moment.

[0023] No. Continuous pitch angle at any moment The calculation formula is:

[0024]

[0025] in, This represents the time sampling index within the historical observation window, with a value range of [value range missing]. , For window length, Indicates the first The raw pitch angle observations collected at each moment, with a range of values ​​of: , Indicates the first The cumulative period offset of the pitch angle at any given time, initially set to 0. This represents the algebraic difference between the original yaw angles at two adjacent sampling times. Indicates the first The raw yaw angle observations collected at each moment.

[0026] Obtaining continuous coordinates and Then, a linear regression model was established using the least squares (OLS) method:

[0027]

[0028] in, This represents the predicted value calculated based on the linear model. This represents the index of the future time to be predicted. This represents the slope of the linear regression model. This represents the intercept of the linear regression model. and This represents the sum of products and sum of squares of deviations used to calculate the slope. and This represents the average of the time index and historical coordinates.

[0029] , , and The calculation formula is:

[0030]

[0031] The predicted values ​​obtained from linear regression are remapped back to the effective range of the standard spherical coordinate system to correct the periodic offset caused by the continuous mapping process. The calculation formula is as follows:

[0032]

[0033] in, This represents the normalized, precise viewport center coordinates of the final output. This represents the unnormalized predicted value calculated in the first part. Represents the complete period constant of the viewport coordinates for the yaw angle. Take 360 ​​degrees for the pitch angle. Take 180 degrees.

[0034] Preferably, step 3), which involves defining the short-term predicted viewport coverage area based on the precise viewport center, includes:

[0035] Based on the precise viewport center coordinates calculated in step 2) Based on preset field of view parameters, a rectangular coverage area is delineated on the equidistant cylindrical projection (ERP) plane. .

[0036]

[0037] in, This represents the rectangular coverage area of ​​the short-term prediction viewport on a two-dimensional plane. This indicates the center of the viewport yaw angle obtained from short-time prediction. This indicates the center of the viewport pitch angle obtained from short-time prediction. This indicates the horizontal field of view (FOV) of the display device. This indicates the vertical field of view (FOV) of the display device.

[0038] Preferably, step 4) involves mapping and comparing the short-term predicted viewport coverage area with the current cache queue to filter out a set of prediction deviation slices, including:

[0039] The system determines whether any slice is within the filtering range based on whether the slice's position on the spherical ERP projection plane spatially overlaps with the short-term predicted viewport coverage area and whether the slice's resolution is lower than a preset threshold.

[0040]

[0041] This represents the final set of tile indices selected from those within the viewport that require image quality compensation. A two-dimensional index (row number, column number) representing a video slice. Indicates the first A slice is a fixed rectangular region on the projection plane. This represents the short-term predicted viewport coverage area calculated in step 3. This indicates the current cache quality level of the slice. This indicates the quality threshold that triggers super-resolution compensation.

[0042] Preferably, step 5), which involves calling a pre-set super-resolution model on the client to perform real-time image enhancement on the slices in the prediction bias slice set and generate high-resolution compensation slices, includes:

[0043] For each low-quality slice selected in step 4), its image data is input into the client's preset super-resolution model to map the low-resolution image to a high-resolution image. The processing can be represented as follows:

[0044]

[0045] in, This represents a low-resolution video slice as input. This represents the output high-resolution compensated slice. This indicates the super-resolution processing function running on the client side, which is the extremely fast ESPCN algorithm.

[0046] Preferably, step 6) involves spherically projecting and stitching the compensation slice generated in step 5) with the original high-resolution slice in the cache to complete the image rendering, including:

[0047] The system is based on the index of the video slices. The high-resolution compensated slice generated in step 5 is logically combined with the original high-resolution slice already in the local cache. Then, using the graphics rendering engine, based on the current user head pose matrix, the combined image is mapped onto a 3D sphere to generate the final viewport rendering screen. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly explained below. The accompanying drawings described herein are merely some embodiments of the present invention.

[0049] Figure 1 This is a step diagram of the present invention. Detailed Implementation

[0050] To clarify the technical issues, technical solutions, implementation process, and performance demonstration, the present invention will be further described in detail below with reference to embodiments. Note. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The following will refer to the accompanying drawings. Various exemplary embodiments, features, and aspects of this disclosure are described in detail. The same reference numerals in the accompanying drawings denote elements that have the same or similar functions.

[0051] Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0052] This invention provides a 360-degree video slice super-resolution rendering compensation method based on short-time viewport prediction. By combining the accurate viewport predicted by the client in a short time, the low-quality slices located within the viewport are subjected to super-resolution processing, which can fix the problems of mismatch between the predicted viewport and the real viewport, blurred image quality at the viewport edge, or missing high-resolution slices caused by long-time viewport prediction errors.

[0053] To achieve the above objectives, the present invention adopts the following technical solution.

[0054] A method for super-resolution rendering compensation of 360-degree video slices based on short-time viewport prediction includes the following steps:

[0055] 1) During 360-degree video playback, the client records the user's head posture data in real time and maintains a long-term viewport prediction result record for bitrate allocation;

[0056] 2) Within a preset time window before video frame rendering, based on the latest head pose data recorded in 1), the precise viewport center for future rendering moments is calculated using a short-time viewport prediction model.

[0057] 3) Based on the accurate viewpoint center predicted in 3), delineate the short-term predicted viewpoint coverage area;

[0058] 4) Map and compare the short-term prediction viewport coverage area defined in 4) with the current cache queue to filter out the prediction deviation slice set;

[0059] 5) Call the client-preset super-resolution model to perform real-time image enhancement on the slices in the prediction bias slice set to generate high-resolution compensation slices;

[0060] 6) Perform spherical projection stitching between the compensation slice generated in step 5) and the original high-resolution slice in the cache to complete the image rendering.

[0061] Preferably, in step 1), during 360-degree video playback, the client records the user's head pose data in real time and maintains a long-term viewport prediction result record for bitrate allocation, including:

[0062] The client uses the device's built-in inertial measurement unit to collect the user's head posture data in real time at a fixed sampling frequency. This posture data is represented as Euler angles. The client maintains a historical posture queue to store head movement trajectories over a recent period (e.g., the past 2 seconds).

[0063] Meanwhile, the client maintains a "long-term viewport prediction result record," also known as a TileQuality Map. Due to round-trip time (RTT) in network transmission, the server-side bitrate allocation is typically based on long-term prediction results from 1-2 seconds prior. Therefore, the quality of video tiles stored in the client's download buffer is inconsistent: areas predicted to be viewed by the user in long-term prediction are designated as high quality, while areas predicted not to be viewed are designated as low quality. This record is updated in real-time for each tile index in the current buffer. The corresponding resolution level (e.g., 4K, 1080P or 720P).

[0064] Preferably, step 2), within a preset time window before video frame rendering, calculates the precise viewport center for future rendering moments using a short-time viewport prediction model based on the latest head pose data, including:

[0065] Since user head pose data is usually represented by yaw and pitch in spherical coordinates, there are periodic boundary abrupt changes (e.g., from 359 degrees to 1 degree), the historical viewport coordinates need to be made continuous first.

[0066] No. Continuous yaw angle at any moment The calculation formula is:

[0067]

[0068] in, This represents the time sampling index within the historical observation window, with a value range of [value range missing]. , For window length, Indicates the first The raw pitch angle observations collected at each moment, with a range of values ​​of: , Indicates the first The cumulative period offset of the yaw angle at any given time, with an initial value of 0. This represents the algebraic difference between the original yaw angles at two adjacent sampling times. Indicates the first The raw yaw angle observations collected at each moment.

[0069] No. Continuous pitch angle at any moment The calculation formula is:

[0070]

[0071] in, This represents the time sampling index within the historical observation window, with a value range of [value range missing]. , For window length, Indicates the first The raw pitch angle observations collected at each moment, with a range of values ​​of: , Indicates the first The cumulative period offset of the yaw angle at any given time, with an initial value of 0. This represents the algebraic difference between the original yaw angles at two adjacent sampling times. Indicates the first The raw yaw angle observations collected at each moment.

[0072] Obtaining continuous coordinates and Then, a linear regression model was established using the least squares (OLS) method:

[0073]

[0074] in, This represents the predicted value calculated based on the linear model. This represents the index of the future time to be predicted. This represents the slope of the linear regression model. This represents the intercept of the linear regression model. and This represents the sum of products and sum of squares of deviations used to calculate the slope. and This represents the average of the time index and historical coordinates.

[0075] , , and The calculation formula is:

[0076]

[0077] The predicted values ​​obtained from linear regression are remapped back to the effective range of the standard spherical coordinate system to correct the periodic offset caused by the continuous mapping process. The calculation formula is as follows:

[0078]

[0079] in, This represents the normalized, precise viewport center coordinates of the final output. This represents the unnormalized predicted value calculated in the first part. Represents the complete period constant of the viewport coordinates for the yaw angle. Take 360 ​​degrees for the pitch angle. Take 180 degrees.

[0080] Preferably, step 3), which involves defining the short-term predicted viewport coverage area based on the precise viewport center, includes:

[0081] Based on the precise viewport center coordinates calculated in step 2 Based on preset field of view parameters, a rectangular coverage area is delineated on the equidistant cylindrical projection (ERP) plane. .

[0082]

[0083] in, This represents the rectangular coverage area of ​​the short-term prediction viewport on a two-dimensional plane. This indicates the center of the viewport yaw angle obtained from short-time prediction. This indicates the center of the viewport pitch angle obtained from short-time prediction. This indicates the horizontal field of view (FOV) of the display device. This indicates the vertical field of view (FOV) of the display device.

[0084] Preferably, step 4) involves mapping and comparing the short-term predicted viewport coverage area with the current cache queue to filter out a set of prediction deviation slices, including:

[0085] The system determines whether any slice is within the filtering range if the slice's position on the spherical ERP projection plane has geometric overlap with the short-term predicted viewport coverage area, and the slice's current cache resolution is lower than a preset threshold.

[0086]

[0087] This represents the final set of tile indices selected from those within the viewport that require image quality compensation. A two-dimensional index (row number, column number) representing a video slice. Indicates the first A slice is a fixed rectangular region on the projection plane. This represents the short-term predicted viewport coverage area calculated in step 3. This indicates the current cache quality level of the slice. This indicates the quality threshold that triggers super-resolution compensation.

[0088] Preferably, step 5), which involves calling a pre-set super-resolution model on the client to perform real-time image enhancement on the slices in the prediction bias slice set and generate high-resolution compensation slices, includes:

[0089] For each low-quality slice selected in step 4), its image data is input into the client's preset super-resolution model to map the low-resolution image to a high-resolution image. The processing can be represented as follows:

[0090]

[0091] in, This represents a low-resolution video slice as input. This represents the output high-resolution compensated slice. This indicates the super-resolution processing function running on the client side, which is the extremely fast ESPCN algorithm.

[0092] Preferably, step 6) involves spherically projecting and stitching the compensation slice generated in step 5) with the original high-resolution slice in the cache to complete the image rendering, including:

[0093] The system is based on the index of the video slices. The high-resolution compensated slice generated in step 5 is logically combined with the original high-resolution slice already in the local cache. Then, using the graphics rendering engine, based on the current user head pose matrix, the combined image is mapped onto a 3D sphere to generate the final viewport rendering screen.

Claims

1. A method for super-resolution rendering compensation of 360-degree video slices based on short-time viewport prediction, characterized in that, Includes the following steps: 1) During 360-degree video playback, the client records the user's head posture data in real time and maintains a long-term viewport prediction result record for bitrate allocation; 2) Within a preset time window before video frame rendering, the precise viewport center for future rendering moments is calculated using a short-time viewport prediction model based on the latest head pose data. 3) Based on the precise viewport center, delineate the short-term predicted viewport coverage area; 4) Map and compare the short-term predicted viewport coverage area with the current cache queue to filter out the prediction deviation slice set; 5) Call the client-preset super-resolution model to perform real-time image enhancement on the slices in the prediction bias slice set to generate high-resolution compensation slices; 6) Perform spherical projection stitching between the compensation slice generated in step 5) and the original high-resolution slice in the cache to complete the image rendering.

2. The 360-degree video slice super-resolution rendering compensation method based on short-time viewport prediction according to claim 1, characterized in that, The long-term viewport prediction has a time span greater than the network round-trip time (RTT) and is used to guide the server-side bitrate allocation and slice download; the short-term viewport prediction has a time span less than the refresh interval of a preset short-term threshold and is used to guide the client-side local image post-processing.

3. The 360-degree video slice super-resolution rendering compensation method based on short-time viewport prediction according to claim 1, characterized in that, The aforementioned calculation of the precise viewport center at the future rendering moment based on the latest head pose data within a preset time window before video frame rendering includes: the short-time viewport prediction model uses linear regression, with the input being the Euler angle coordinates of the past N sampling points and the output being the viewport center coordinates after a future time interval of Δt.

4. The 360-degree video slice super-resolution rendering compensation method based on short-time viewport prediction according to claim 1, characterized in that, The method of defining the short-term predicted viewport coverage area based on the precise viewport center includes: using the precise viewport center as a geometric reference, calculating the coordinate span in combination with preset viewport size parameters, thereby defining a rectangular short-term predicted viewport coverage area on the 360-degree video.

5. The 360-degree video slice super-resolution rendering compensation method based on short-time viewport prediction according to claim 1, characterized in that, The method of mapping and comparing the short-term predicted viewport coverage area with the current cache queue to filter out the prediction deviation slice set includes: the specific method for filtering out the prediction deviation slice set is to traverse all video slices on the ERP plane, filter out video slices that have geometric overlap with the short-term predicted viewport coverage area in spatial coordinates and do not reach the preset high resolution threshold in the client's local cache, and classify such slices into the prediction deviation slice set.

6. The 360-degree video slice super-resolution rendering compensation method based on short-time viewport prediction according to claim 1, characterized in that, The method of calling the client-preset super-resolution model to perform real-time image enhancement on the slices in the prediction bias slice set and generate high-resolution compensation slices includes: the super-resolution enhancement processing adopts a lightweight convolutional neural network (CNN), which is deployed on the client device, takes low-resolution video slices as input, and outputs corresponding high-resolution video slices to compensate for the image blur caused by long-term viewport prediction errors and network bandwidth limitations.

Images