Systems and methods for controlling video quality

Abstract

A method performed in a system comprising a decoder and a display device. The method includes obtaining a bitstream comprising a sequence of encoded pictures. The method also includes decoding, by the decoder, the encoded pictures to produce a sequence of decoded pictures. The method also includes using the sequence of decoded pictures to display video on the display device. The method also includes, while the video is being displayed on the display device, determining whether a viewer of the display device is attentive to the display device. In one embodiment, the method includes decreasing a quality of the video to be displayed on the display device as a result of determining that the viewer is not attentive to the display device. In another embodiment, the method includes increasing a quality of the video to be displayed on the display device as a result of determining that the viewer is attentive to the display device.

Description

SYSTEMS AND METHODS FOR CONTROLLING VIDEO QUALITYTECHNICAL FIELD

[0001] Disclosed are embodiments related to systems and method for controlling video quality.BACKGROUND

[0002] Versatile Video Coding (VVC)

[0003] Versatile Video Coding (VVC) and its predecessor, High Efficiency Video Coding (HEVC), are block-based video codecs standardized and developed jointly by ITU-T and MPEG. The codecs utilize both temporal and spatial prediction. Spatial prediction is achieved using intra (I) prediction from within the current picture. Temporal prediction is achieved using uni-directional (P) or bi-directional inter (B) prediction on the block level from previously decoded reference pictures.

[0004] In the encoder, the difference between the original sample data and the predicted sample data, referred to as the residual, is transformed into the frequency domain, quantized, and then entropy coded before transmitted together with necessary prediction parameters such as prediction mode and motion vectors, also entropy coded. The decoder performs entropy decoding, inverse quantization, and inverse transformation to obtain the residual, and then adds the residual to an intra or inter prediction to reconstruct a picture.

[0005] The VVC version 1 specification was published as Rec. ITU-T H.266 | ISO / IEC 23090-3, “Versatile Video Coding,” in 2020. MPEG and ITU-T are working together within the Joint Video Exploratory Team (JVET) on updated versions of HEVC and VVC as well as the successor to VVC, i.e., the next generation video codec.

[0006] Components

[0007] A video sequence consists of a series of pictures where each picture consists of one or more components. A picture in a video sequence is sometimes denoted ‘image’ or ‘frame’. Each component in a picture can be described as a two-dimensional rectangular array of sample values (or “samples” for short). It is common that a picture in a video sequenceconsists of three components; one luma component Y where the sample values are luma values and two chroma components Cb and Cr, where the sample values are chroma values. Other common representations include ICtCb, IPT, constant-luminance YCbCr, YCoCg and others. It is also common that the dimensions of the chroma components are smaller than the luma components by a factor of two in each dimension. For example, the size of the luma component of an HD picture would be 1920x1080 and the chroma components would each have the dimension of 960x540. Components are sometimes referred to as ‘color components’, and other times as ‘channels’.

[0008] Coding Units and Coding Blocks

[0009] In many video coding standards, such as HEVC and VVC, each component of a picture is split into blocks and the coded video bitstream consists of a series of coded blocks. A block is a two-dimensional array of samples. It is common in video coding that the picture is split into units that cover a specific area of the picture.

[0010] Each unit consists of all blocks from all components that make up that specific area and each block belongs fully to one unit. The macroblock in H.264 and the Coding unit (CU) in HEVC and VVC are examples of units. In VVC the CUs may be split recursively to smaller CUs. The CU at the top level is referred to as the coding tree unit (CTU). A CU usually contains three coding blocks, i.e. one coding block for luma and two coding blocks for chroma. A block to which a transform used in coding is applied is referred to as a “transform block.” And a block to which a prediction mode is applied is referred to as a “prediction blocks.”

[0011] Loop Filtering

[0012] In VVC, the decoding of a picture is carried out in two stages: Reconstruction and loop filtering. In the reconstruction decoding stage, the samples of the components (Y, Cb and Cr) are partitioned into rectangular blocks. As an example, one block may be of size 4^8 samples, whereas another block may be of size 64x64 samples. The decoder obtains instructions for how to reconstruct each block. This involves deriving a prediction for each block, for instance by deriving samples (e.g., copying samples) from a previously decoded picture (an example of temporal prediction (inter prediction)) or deriving samples (e.g., copying samples) from already decoded parts of the current picture (an example of intra prediction), or a combination thereof. Previously decoded pictures are stored in what is knownas the decoded picture buffer (DPB). Pictures that are inter predicted thus fetches samples from the DPB for the prediction. In addition to deriving a prediction for each block, the decoder may also obtain a residual, often encoded using transform coding such as discrete sine or cosine transform, (DST or DCT). This residual is added to the prediction, and the decoder can proceed to decode the next block.

[0013] The output from the reconstruction decoding stage is the three components Y, Cb and Cr. At this stage, the samples are often called reconstructed samples, so we denote them with YREC, CT>REC and CTREC. It is possible to further improve the quality of these components, and this is done in the loop filtering stage. The loop filtering stage in VVC consists of three sub-stages; a deblocking filter sub-stage, a sample adaptive offset filter (SAO) sub-stage, and an adaptive loop filter (ALF) sub-stage.

[0014] In the deblocking filter sub-stage, the decoder changes YREC, CT>REC and CIKEC by smoothing edges near block boundaries when certain conditions are met. The output of this stage is denoted YDBL, CbuBL and CrDBL. In the SAO sub-stage, the decoder adds or subtracts a signaled value to samples that meet certain conditions, such as being in a certain value range (band offset SAO) or having a specific neighborhood (edge offset SAO). This can reduce ringing noise since such noise often aggregate in certain value range or in specific neighborhoods (e.g., in local maxima). The output of the SAO sub-stage is denoted YSAO, CbsAo and CrsAo. The third sub-stage of the loop filtering stage is adaptive loop filtering (ALF) sub-stage. The basic idea behind adaptive loop filtering is that the fidelity of the picture components YSAO CbsAo and CrsAo can often be improved by filtering the picture using a linear filter that is signaled from the encoder to the decoder. The output of the ALF sub-stage is denoted YALF, CbALF and CFALF. The final output YALF, CbALF, CFALF is then stored in the decoded picture buffer (DPB). From here, it can be output to a display or postfilter and also be used for temporal prediction of future pictures. Since deblocking, SAO and ALF in this way affect the prediction of future pictures, it is essential that all three sub-stages are done exactly according to the specification, otherwise the decoder will experience drift. Put another way, the deblocking filter, SAO and ALF are inside the coding loop. This is the reason why they are called loop filters. It should be noted that deblocking, SAO and ALF are just examples of loop filters. There are other loop filters, such as the bilateral filter which is currently part of ECM. ECM is an enhanced compression model with compression capabilities beyond VVC.

[0015] Postfilters

[0016] A postfilter (a.k.a., post-filter) is a filter that can be applied to the picture before it is displayed or otherwise further processed. A postfilter does not affect the contents of the decoded picture buffer (DPB), i.e., it does not affect the samples that future pictures are predicted from. Instead, it may take samples from the DPB (or from the reconstruction stage or form one of the loop filter sub-stages) and filter them before they are being displayed or further processed. As an example, such further processing can involve scaling the picture to allow it to be rendered in full-screen mode, reencoding the picture (this is known to a person skilled in the art as ‘transcoding’), using machine vision algorithms to extract information from the picture etc. Since a postfilter does not affect the prediction, doing postfilters a bit differently in every decoder does not give rise to drift. Hence it is often not necessary to standardize postfilters. In some codecs, the postfilter may be considered to be part of the decoder, and the samples output from the decoder are the samples output from the postfilter. In other codecs, the postfilter may be considered to be outside the decoder, and the samples output from the decoder are the samples that are inputted to the postfilter. In this document we are covering both cases.

[0017] Neural Network Technologies

[0018] Neural network (NN) technologies are being investigated in JVET to improve the compression efficiency of video compression. Neural networks are incorporated into the video decoding and / or video handling process in order to allow the compression of video to a lower bit rate at preserved video quality, compared to not using neural networks. In JVET, the current exploration activities have looked at neural networks to a) improve intra prediction, b) improve inter prediction, c) improve image upscaling (superresolution), d) improve loop filtering, e) generate reference pictures, and f) improve postfiltering. Common for all neural network-based approaches so far studied in JVET is that they have a very high computational cost measured in number of multiplications needed per pixel. Whereas many non-neural network postfilter require tens or hundreds of multiplications per pixel, some of the neural network-based postfilters require 300,000 multiplications per pixel. Even if this number would come down to, say, 1000 multiplications per pixel, it is still an order of magnitude more costly than non-NN-based postfilters.SUMMARY

[0019] Certain challenges presently exist. For instance, the high computational complexity of some postfilters can cause problems. For example, with respect to mobile devices that rely on a battery for power, a postfilter can cause battery drain and also cause the device to overheat. Even for non-mobile devices a postfilter can become a problem in case CPU-only decoding is needed, because even a multicore computer may not have the computational power to process the required number of multiplications per second on the CPU. On simple solution is deactivate the postfilter, but doing this may give a drop in picture quality, which may not be acceptable to the end user.

[0020] Accordingly, in one aspect there is provided a method performed in a system comprising a decoder and a display device. The method includes obtaining a bitstream comprising a sequence of encoded pictures. The method also includes decoding, by the decoder, the encoded pictures to produce a sequence of decoded pictures. The method also includes using the sequence of decoded pictures to display video on the display device. The method also includes, while the video is being displayed on the display device, determining whether a viewer of the display device is attentive to the display device. In one embodiment, the method includes decreasing a quality of the video to be displayed on the display device as a result of determining that the viewer is not attentive to the display device. In another embodiment, the method includes increasing a quality of the video to be displayed on the display device as a result of determining that the viewer is attentive to the display device.

[0021] In another aspect there is provided a device that is configured to perform the methods disclosed herein. The device may include memory and processing circuitry coupled to the memory.

[0022] In some aspects, there is provided a computer program comprising instructions which when executed by processing circuitry of a device causes the device to perform any of the methods disclosed herein. In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0023] An advantage of embodiments disclosed herein is that they reduce power consumption while having negligible impact on the viewer’s viewing experience.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0025] FIG. 1 illustrates a system according to an embodiment.

[0026] FIG. 2 is a schematic block diagram of an encoder according to an embodiment.

[0027] FIG. 3 is a schematic block diagram of a decoder according to an embodiment.

[0028] FIG. 4A illustrates an XR system according to an embodiment.

[0029] FIG. 4B illustrates an XR device according to an embodiment.

[0030] FIG. 5 is a flowchart illustrating a process according to an embodiment.

[0031] FIG. 6 is a hardware block diagram of a device according to an embodiment.DETAILED DESCRIPTION

[0032] FIG. 1 illustrates a system 100 according to an embodiment. System 100 includes an encoder 102 and a decoder 104. In some embodiments, encoder 102 is in communication with decoder 104 via a network 110 (e.g., the Internet or other network).Encoder 102 encodes a source video sequence 101 into a bitstream comprising an encoded video sequence and may transmit the bitstream to decoder 104 via network 110. In some embodiments, encoder 102 is not in communication with decoder 104, and, in such an embodiment, rather than transmitting bitstream to decoder 104, the bitstream is stored in a data storage unit 190 and decoder 104 retrieves from the data storage unit 190 the bitstream containing the encoded video sequence. Data storage unit 190 may be co-located with encoder 102 or may be remote from encoder 102.

[0033] Decoder 104 decodes the pictures included in the encoded video sequence to produce video data (e.g., reconstructed pictures) for display and / or postfiltering. Accordingly, decoder 104 may be part of a device 103 having a display device 106 (or display 106 for short) and a postfilter 105, such as a neural network postfilter (NNPF), that that may receive reconstructed pictures from decoder 104 and that may process the reconstructed pictures using a specified postfiltering algorithm. In the embodiment shown, postfilter 105 is separate from decoder 104, but in other embodiments, postfilter 105 may be a component of decoder 104.

[0034] FIG. 2 illustrates functional components of encoder 102 according to some embodiments. It should be noted that encoders may be implemented differently so implementation other than this specific example can be used. Encoder 102 employs a subtractor 241 to produce a residual block which is the difference in sample values between an input block and a prediction block (i.e., the output of a selector 251, which is either an inter prediction block output by an inter predictor 250 (a.k.a., motion compensator) or an intra prediction block output by an intra predictor 249). Then a forward transform 242 is performed on the residual block to produce a transformed block comprising transform coefficients. A quantization unit 243 quantizes the transform coefficients based on a quantization parameter (QP) value (e.g., a QP value obtained based on a picture QP value for the picture in which the input block is a part and a block specific QP offset value for the input block), thereby producing quantized transform coefficients which are then encoded into the bitstream by encoder 244 (e.g., an entropy encoder) and the bitstream with the encoded transform coefficients is output from encoder 102. Next, encoder 102 uses the quantized transform coefficients to produce a reconstructed block. This is done by first applying inverse quantization 245 and inverse transform 246 to the transform coefficients to produce a reconstructed residual block and using an adder 247 to add the prediction block to the reconstructed residual block, thereby producing the reconstructed block, which is stored in the reconstructed picture buffer (RPB) 266. Loop filtering by a loop filter (LF) stage 267 is applied and the final decoded picture is stored in a decoded picture buffer (DPB) 268, where it can then be used by the inter predictor 250 to produce an inter prediction block for the next picture to be processed. LF stage 267 may include three sub-stages: i) a deblocking filter, ii) a sample adaptive offset (SAO) filter, and iii) an Adaptive Loop Filter (ALF).

[0035] FIG. 3 illustrates functional components of decoder 104 according to some embodiments. It should be noted that decoder 104 may be implemented differently so implementations other than this specific example can be used. Decoder 104 includes a decoder module 361 (e.g., an entropy decoder) that decodes from the bitstream quantized transform coefficient values of a block. Decoder 104 also includes a reconstruction stage 398 in which the quantized transform coefficient values are subject to an inverse quantization process 362 and inverse transform process 363 to produce a residual block. This residual block is input to adder 364 that adds the residual block and a prediction block output from selector 390 to form a reconstructed block. Selector 390 either selects to output an inter prediction block or an intraprediction block. The reconstructed block is stored in a RPB 365. The inter prediction block is generated by the inter prediction module 350 and the intra prediction block is generated by the intra prediction module 369. Following the reconstruction stage 398, a loop filter stage 367 applies one or more filters to the reconstructed blocks and the final decoded picture may be stored in a decoded picture buffer (DPB) 368 and output to postfilter 105 for post processing and / or display 106. Pictures are stored in the DPB for two primary reasons: 1) to wait for picture output and 2) if the picture is a reference picture, to be used for reference when decoding future pictures. In some embodiments, postfilter 105 may receive the reconstructed picture before all loop filters are applied.

[0036] The device 103 may be a mobile device, a set-top device, a head-mounted display, or any other device, and may be a component of an extended reality (XR) system.

[0037] FIG. 4A illustrates such an XR system 400 in which the embodiments disclosed herein may be applied. XR system 400 includes speakers 404 and 405 (which may be speakers of headphones worn by the user) and device 103, which, in this example, is an XR device that is configured to be worn by the viewer. In the illustrated embodiment, XR device 102 is designed to be worn on the user‘s head and is commonly referred to as a head-mounted display (HMD).

[0038] FIG. 4B illustrates XR device 103 according to an embodiment. As shown in FIG. 4B, XR device 103 may comprise one or more sensors 490, which may include an orientation sensing function (OSF) 491, a position sensing function (PSF) 492, and an eyetracking function (ETF) 493. XR device also includes a controller 403 that receives output from sensors 490 and controls an audio Tenderer 451, decoder 104, and / or postfilter 105.

[0039] Orientation sensing function 491 is configured to detect a change in the orientation of the listener and provides information regarding the detected change to controller 403. In some embodiments, controller 403 determines the absolute orientation (in relation to some coordinate system) given the detected change in orientation detected by orientation sensing function 491. There could also be different systems for determination of orientation and position, e.g., a system using lighthouse trackers (LIDAR). In one embodiment, orientation sensing function 491 may determine the absolute orientation (in relation to some coordinate system) given the detected change in orientation. In this case the controller 403 may simplymultiplex the absolute orientation data from orientation sensing function 491 and positional data from position sensing function 492. In some embodiments, orientation sensing function 491 may comprise one or more accelerometers and / or one or more gyroscopes.

[0040] Audio Tenderer 451 can be configured to produce audio output signals based on input audio signals and metadata regarding the XR scene the listener is experiencing, and information about the location and orientation of the listener.

[0041] Lately, some commercially available XR head mounted displays such as the Varjo XR-3 and the Apple Vision Pro come with an eye-tracking function. In the Varjo XR-3 head mounted display, there are two low-resolution displays, one for each eye, that cover the entire visual field. In addition to this, there are two smaller high-resolution displays (also one for each eye) that can be moved so that they are centered where the eye-tracking software has determined that the user is looking. This way, the user can get a very sharp picture quality in the spot where the user is focusing, while at the same time get a big field of view.

[0042] In the Apple Vision Pro, the eye-tracking function is used as part of the user interface: As an example, by looking at an icon and touching your thumb / index finger, you can select that icon. The Apple Vision Pro also uses the eye-tracking function for what is referred to as foveated rendering. As an example, in order to reduce staircase-like rendering artifacts called aliasing, it is possible to use supersampling. With supersampling, the entire image is rendered at twice the resolution in both the x- and y- dimensions and then the average of each nonoverlapping 2x2 pixel area is calculated to form a pixel in a picture of original resolution size. Supersampling is very computationally expensive, taking about four times as much resources as regular rendering, but it reduces aliasing. With foveated rendering, the Apple Vision Pro can use supersampling only in the region where the viewer is looking, greatly reducing the computational burden but giving much the same quality experience as supersampling everywhere.

[0043] As described above, a challenge presently exists because the high computational complexity of some postfilters can cause problems. As one example, with respect to mobile devices that rely on a battery for power, a postfilter can cause battery drain and also cause the device to overheat. One simple solution is to deactivate the postfilter, but doing this may give a drop in picture quality, which may not be acceptable to the end user.

[0044] Accordingly, this disclosure takes advantage of eye-tracking capabilities which know where the user is looking, such as, for example, the ETF 493 of XR device 103, but this disclosure is not limited to XR devices or systems. Nevertheless, embodiments will be described with reference to XR device 103 to illustrate the various features. In one embodiment, the postfilter is turned on by default, and when the user is not looking at the video output, controller 402 may turn off postfilter 105 to save power and computing resources and when the user returns his or her gaze to the video, the postfilter is turned on again. In another embodiment, the postfilter is turned off by default to save power and computing resources, and when the user starts looking at the video output, controller 402 may turn on postfilter 105, and when the user turns his or her gaze away from the video output, controller 402 may turn off postfilter 105.

[0045] In one embodiment, the eye-tracking function is running continuously, and the postfilter is turned off as soon as the controller detects that the user is no longer watching the decoded video, and turned back on again as soon as the user is again watching the decoded video.

[0046] In one example, the eye-tracking function is repetitively providing gaze direction information that comprises information identifying a location (e.g., one or more coordinate values) the user is looking at. This location information may be compared with the position or coordinates of the rendered video to determine whether the user is watching the video or not. In one variant, gaze prediction is used such that a predicted future gaze is used rather than the measured gaze direction at that time instant. Running the eye-tracking function also costs power, but in some cases, such as in the Varjo XR-3 headset and the Apple Visio Pro headset, the eyetracking function may be running in any case, as part of the user interface and / or rendering system. In such cases the embodiments provide an energy saving even if the postfiltering is only turned off occasionally.

[0047] In another embodiment, the eye-tracking function is turned on at regular intervals, for instance, once per second, while the user is watching the video. Once it has been detected that the user is no longer watching the video, the eye-tracking function is put into continuous mode, so that the controller will be able to turn postfiltering back on as soon as the user’s gaze goes back to the video. In an alternative embodiment, the eye-tracking function will not go to continuous mode when the user stops watching the video, but instead the eye-tracking function isactivated more frequently. For instance, the eye-tracking function can run twice a second when the user is looking away from the video instead of running once a second as when the user is looking at the video. This way, the power consumed by the eye-tracking function can be saved also when the user is looking away from the video, but still have a relatively quick recovery (i.e., turning the postfiltering back on) as soon as the user’s gaze is back. In an alternative embodiment, the eye-tracking function continues with the same frequency, for instance, once per second. Compared to continuous or more frequent eye-tracking, this may degrade the speed with which the postfiltering will be turned on, but in return it may save some power on the eyetracking function.

[0048] Most postfilters will only change subtle things in the image quality. This means, that if you turn the postfilter on and off while the user has the video in his / her peripheral vision, it will be very difficult for the user to notice. However, if the postfilter is very strong, or if it is affecting the average intensity (brightness) of the video, it may be possible for the user to notice when the postfilter is turned off or turned on. This can be very distracting for the user because, as soon as he / she looks away, the video will change, capturing again the attention of the user. Now when the user again looks at the video, it will change back. This can be frustrating to the user.

[0049] Therefore, in one embodiment, the postfiltering will be gradually removed when the user’s gaze has left the video. Let the N-th picture of the sequence be denoted IN, and the postfiltered version of that picture be denoted P(JN)- Further assume that the user watches the video from frame N=0 up to N=120 and then looks away. In other words, assuming a frame rate of 60 frames per second, the user watches the first two seconds of the video and then looks away. Then, in this embodiment, the first two seconds will show the postfiltered version, then a gradual blending from the postfiltered to the non-postfiltered picture will take place over the next 60 frames (one second), and after that the non-postfiltered picture will be shown. Mathematically, this can be expressed as the following, where ON represents the image that is displayed: P(INf for N = 0, ..., 120

[0050] 121, ..., 180( IN,for N > 180

[0051] This will prevent the user from noticing any sudden stoppage of postfiltering.Assume that the user returns to watching the video at frame 800. Then, in one embodiment, the postfiltering will gradually be blended back according to: r N = 180, ..., 800

[0052] (A — 800)) / w) for IV = 801, ... , 860N),for N > 860

[0053] This blending may appear more computationally intensive than just displaying the postfiltered version because now an additional step of averaging must take place. For many postfilters, however, the way they are actually implemented is according to:

[0054] P( / w) = IN+ d( / w),

[0055] where d( / w) is a postfilter offset calculated from the input IN. This means that the blending equation can be written as

[0057] rather than N — 120) / w+ (60 — (A — 120))P( / w) Apart from multiplying with a scalar, this is no more complex than applying the postfilter.

[0058] In another embodiment, the postfiltering is applied to a portion of the picture and the postfilter is turned off for that portion of the picture when it is detected that the user is no longer watching that portion of the decoded picture or video, and turned back on again when the user is again watching that portion of the decoded picture or video. In a variant of this embodiment different postfilters are applied to different portions of the picture.

[0059] In another embodiment, means other than the eye-tracking function are used to detect if the decoded video is being watched or not. In one example another type of sensor such as a motion sensor detects if the decoded video is being watched or not and the postfilter is turned off when it is detected that the user is no longer watching the decoded video. In another example a sensor that detects if the eyelids are closed can be used - if the eyes are shut for long enough, postfiltering can be turned off. In another example a combination of sensor inputs is used to turn off the postfiltering when it is concluded that the user is not watching the decoded video. In another example if the user is engaged in another activity that takes up user’s attentionit is concluded that the user is not watching the decoded picture or video and so the postfiltering is turned off. In another embodiment, the system detects that part of the video is outside the displayable area (field of view) of the display device. As an example, the user may put the video displaying window so that only the left part is visible, and the right half is “outside the screen”. In this situation, the postfilter is turned off for the entire area (or parts of the entire area) that is not visible. In another example, the user may put the video right in front so that all of it is visible, but later the user turns its head so that part or all of the video is invisible, i.e., outside the screen’s field of view, such as behind the user. The postfilter can then be turned off for the invisible part.

[0060] In one embodiment, the energy saving method is used with scalable video rather than with a postfilter. In this embodiment, the decoder is receiving at least two layers, a lower layer A and a high layer B. If the system detects that the user is not looking at the video, the decoder only decodes layer A, resulting in a video of lower quality, but also resulting in few computing resources being used which translates into an energy savings. If at a later point in time the controller detects that the user starts looking at the video again, the controller instructs the decoder to decode both layer A and layer B, resulting in a video of higher quality. Layer B may for example here be an enhancement layer of one of quality scalability, spatial scalability, temporal scalability, or a combination thereof.

[0061] As an example, layer A may convey a 1080p HD video and layers A and B convey a 2160p UHD video. In one variant of this embodiment, the video bitstream contains non-reference pictures. These are pictures that are not used as reference for any other picture in the bitstream. When the user is looking at the video, the non-reference pictures are decoded by the decoder, but when the user is not looking, they are not. In this variant, in order to avoid uneven temporal display of pictures, it is beneficial to encode the non-reference pictures in regular picture intervals. For example, every other picture in the bitstream may be coded as a non-reference picture, which means that the rendered video is of half the original framerate when the user is not looking at the video. Any one or more of the various embodiments may be used in combination with one or more of the other embodiments.

[0062] One of the changes that is most noticeable for peripheral vision is the change of brightness, also known as intensity, luminance or luma. If, when activated, the energy savingsmode (e.g., postfilter is turned off, not all layers are decoded, etc.) substantially changes the luma, it may be very noticeable, unless blending is used. Therefore, in another embodiment, the encoder signals side information on how much the energy savings mode changes the luma for each frame. When the end user device detects that the user no longer is looking at the screen, it can activate energy savings mode (e.g., deactivate postfiltering) but instead add the average luma change to each pixel of the frame. This is substantially cheaper than running the postfilter, needing only three additions per pixel rather than, say, 1000 multiplications per pixel. The upside compared to the blending case is that the power saving can start instantly instead of being blended over the course of, say, one second.

[0063] In another embodiment, the postfilter that is used in parts of the blending process may be different from the postfilter that is used in other stages of the blending. In one example the blending process is happening from no-postfiltered image IN at N=0 to fully postfiltered image P(IN) at N=60 in the course of 1 second and a lower computationally expensive postfilter is used for the first half of the second, N=l, ..., 30, and then a more computationally complex postfilter for the second half of the second, N=31, . . . , 60. This embodiment can take advantage of the lower sensitivity of the human vision to certain aspects of the image quality at the point of a scene cut to design lower complexity postfilters for parts of the blending process.

[0064] In yet another embodiment, the end user device may discover that the end user is looking at only the left part of the screen, or at a point located immediately to the left of the video screen. In this case, it may be necessary to run the postfilter on the left part of the video to obtain a seamless experience of high quality, but it may not be necessary to use the postfilter on the right part of the screen. Therefore, in this embodiment, the postfilter is only applied to the left part of the screen. It may again be necessary to fade out the application of the postfilter in the right part of the image over time. It may also be desirable to fade the strength of the postfiltering from left to right, in order not to have a sharp line in the middle of the screen between postfiltered- and not postfiltered data. For instance, if the width of the screen is 2100 pixels, the left most third of the image, i.e., columns 0 through 699 may be fully postfiltered (i.e., using P(JNy). Likewise, the rightmost third of the image, columns 1400 through 2099, may be not postfiltered at all (i.e., using / w). Finally, for the middle third of the image, columns 700 through 1399 (i.e., where x is the index of the column and x goes from 700 to 1399) using:

[0066] This produces a smooth change from using the postfilter fully in the left third of the image towards not using it at all in the rightmost third of the image. In another example, the postfilter may be intended to be applied to all CTUs of a picture. It may be determined from the eye-tracking system that the user is looking at a particular CTU, i.e., the CTU that contains the pixel coordinates obtained from the eye tracking. According to one embodiment, the postfilter is then only applied to the CTU the user is looking at. According to another embodiment, the postfilter is applied to the CTU the user is looking at as well as the surrounding neighboring CTUs, but not to other CTUs. According to yet another embodiment, the postfilter is applied to the CTU the user is looking at, and in the surrounding (neighboring) CTUs, a blend between non-postfiltered and postfiltered output is used. This blend can be influenced by the pixel distance to the CTU or region the user is looking at so that the postfiltered output is given a higher weight closer to that region.

[0067] In one embodiment, the user may have the video being displayed in part of the field of view of the head mounted display. As an example, perhaps the user has the video of a sporting event in the right half of the field of view, and statistics about the players of the game in the left half of the field of view. The method can then decide that the user is no longer looking at the video when the eye-tracking function determines that the viewer is no longer looking in the right-hand half of the field of view.

[0068] In another embodiment, the user may still be deemed to be watching the video if the distance (in pixels or degree of viewing angle) to the running video is smaller than a set threshold. This will prevent the postfiltering to be turned off if the user’s gaze hovers close to the edge of the video.

[0069] In yet another embodiment, the user may still be deemed to be watching the video if the time that has passed since he / she last watched the video is below a certain threshold, for instance 0.5 seconds. This will prevent the postfiltering to be turned off if the user’s gaze has momentarily strayed from the video or the user is blinking.

[0070] In one embodiment, the user may have the video being displayed in most of the active field of view of the display. As an example, perhaps the full width of the displayable areaof the head mounted display is used, but there are regions above and below the displayed video that are still within the active field of view of the display. In such a case, the user may be deemed to be not watching the video when he / she is gazing above or below the video.

[0071] In some embodiments, the eye-tracking function may be capable of tracking the eyes in areas outside of the field of view that can be used to display content. In such a case, the entire field of view may be occupied by the video, but the system can decide that the user is no longer watching the video if the gaze is outside the displayable area.

[0072] In one embodiment, the postfilter is turned off if it is determined that the video is too small. As an example, the postfilter may be turned on if the video is taking up most of the field-of-view of the device, but if the user minimizes the video so that only a thumbnail is visible, the postfilter is turned off.

[0073] FIG. 5 is a flowchart illustrating a process 500 according to an embodiment. Process 500 may begin in step s502.

[0074] Step s502 comprises obtaining a bitstream comprising a sequence of encoded pictures.

[0075] Step s504 comprises decoding, by the decoder, the encoded pictures to produce a sequence of decoded pictures.

[0076] Step s506 comprises using the sequence of decoded pictures to display video on the display device.

[0077] Step s508 comprises, while the video is being displayed on the display device, determining whether a viewer of the display device is attentive to the display device.

[0078] After step s508 either step s510 or step s512 is performed.

[0079] Step s510 comprises decreasing a quality of the video to be displayed on the display device as a result of determining that the viewer is not attentive to the display device. In one embodiment, decreasing the quality of the video comprises deactivating or adjusting a first postfilter used for filtering the decoded pictures produced by the decoder. In another embodiment, the encoded pictures include encoded pictures belonging to an enhancement layer and decreasing the quality of the video comprises refraining from decoding one or more of the encoded pictures belonging to the enhancement layer. In another embodiment the encodedpictures include encoded non-reference pictures and decreasing the quality of the video comprises refraining from decoding one or more of the encoded non-reference pictures.

[0080] Step s512 comprises increasing a quality of the video to be displayed on the display device as a result of determining that the viewer is attentive to the display device, wherein increasing the quality of the video comprises activating or adjusting the first postfilter used for filtering decoded pictures produced by the decoder, decoding one or more encoded pictures belonging to an enhancement layer, and / or decoding one or more encoded non-reference pictures.

[0081] In one embodiment, process 500 comprises decreasing the quality of the video as a result of determining that the viewer is not attentive to the display device, and process 500 further comprises: after decreasing the quality of the video, determining that the viewer is attentive to the display device; and as a result of determining that the viewer is attentive to the display device, increasing a quality of the video to be displayed on the display device, wherein increasing the quality of the video comprises: reactivating or readjusting the first postfilter, and / or decoding one or more of the encoded pictures belonging to the enhancement layer, and / or decoding one or more of the encoded non-reference pictures.

[0082] In one embodiment, process 500 comprises decreasing the quality of the video as a result of determining that the viewer is not attentive to the display device, and decreasing the quality of the video comprises deactivating or adjusting the first postfilter.

[0083] In one embodiment, decreasing the quality of the video comprises adjusting the first postfilter.

[0084] In one embodiment, adjusting the first postfilter comprises gradually adjusting the first postfilter.

[0085] In one embodiment, gradually adjusting the first postfilter comprises: obtaining a first input picture, II, obtaining a first post filter offset, DI, calculating a first weight, Wl; outputting a first output picture, 01, where O1 = I1 + (W1XD1), obtaining a second input picture, 12, wherein 12 immediately follows II, obtaining a second post filter offset, D2, calculating a second weight, W2; outputting a second output picture, 02, where 02 = 12 + (W2 x D2), where W2 < W1.

[0086] In one embodiment, W1 = (FR - 1) / FR, W2 = (FR - 2) / FR, and FR is the frame rate of the video.

[0087] In one embodiment, the encoded pictures include the encoded pictures belonging to the enhancement layer, and process 500 further comprises refraining from decoding one or more of the encoded pictures belonging to the enhancement layer as a result of determining that the viewer is not attentive to the display device.

[0088] In one embodiment, the encoded pictures include the encoded non-reference pictures, and process 500 further comprises refraining from decoding one or more of the encoded non-reference pictures as a result of determining that the viewer is not attentive to the display device.

[0089] In one embodiment, the sequence of decoded pictures comprises a first decoded picture followed by a second decoded picture, using the decoded pictures to display video on the display device comprises applying the first postfilter to a first portion of the first decoded picture to produce a postfiltered first portion of the first decoded picture and displaying the postfiltered first portion of the first decoded picture using a first portion of the display device; determining whether a viewer of the display device is attentive to the display device comprises determining whether the viewer is attentive to the first portion of the display device that is being used to display the postfiltered first portion of the first decoded picture; as a result of determining that the viewer is not attentive to the first portion of the display device, deactivating or adjusting the first postfilter so that the first postfilter is not applied to any portion of the second decoded picture; and displaying the second decoded picture.

[0090] In one embodiment, using the decoded pictures to display video on the display device further comprises applying a second postfilter to a second portion of the first decoded picture to produce a postfiltered second portion of the first decoded picture and displaying the postfiltered second portion of the first decoded picture using a second portion of the display device, and the second postfilter is different than the first postfilter.

[0091] In one embodiment, the sequence of decoded pictures comprises a first decoded picture followed by a second decoded picture, using the decoded pictures to display video on the display device comprises applying the first postfilter to the first decoded picture to produce a first postfiltered picture and displaying the first postfiltered picture, process 500 furthercomprises determining that the viewer is attentive to a first portion of the display device or a point adjacent to the first portion of the display device, and is not attentive to a second portion of the display device, as a result of determining that the viewer is attentive to the first portion of the display device or the adjacent point, but not the second portion, adjusting the first postfilter such that the first postfilter is applied to a first portion of the seconded decoded picture but not applied to, or not fully applied to, a second portion of the seconded decoded picture; displaying the postfiltered first portion of the seconded decoded picture using the first portion of the display device; and displaying the second portion of the seconded decoded picture using the second portion of the display device.

[0092] In one embodiment, determining whether the viewer is attentive to the display device comprises determining an amount of time that the viewer is looking at something other than the display device, and determining whether the amount of time exceeds a threshold.

[0093] In one embodiment, the display device is attentive to the display device comprises: determining a direction in which the viewer is looking; based on the determined direction, determining a set of one or more coordinates; and comparing at least one of the determined coordinates with a coordinate of the video

[0094] In one embodiment, the display device is attentive to the display device comprises: predicting a direction in which the user may look; based on the predicted direction, determining a set of one or more coordinates; and comparing at least one of the determined coordinates with a coordinate of the video.

[0095] In one embodiment, determining whether the viewer of the display device is attentive to the display device comprises: not more than once every si seconds, obtaining data from a sensor and using the obtained data to determine whether the viewer of the display device is attentive to the display device, and process 500 further comprises, after determining that the viewer of the display device is not attentive to the display device, then, at least once every s2 seconds, obtaining data from the sensor and using the obtained data to determine whether the viewer of the display device is attentive to the display device, wherein si > s2.

[0096] In one embodiment, process 500 further comprises obtaining side luma information indicating an amount by which the first postfilter changes the luma for each picture,and decreasing the quality of the video comprises deactivating or adjusting the first postfilter and using the luma information to adjust pixels of decoded pictures.

[0097] In one embodiment, the enhancement layer is: a spatial scalability enhancement layer, a quality scalability enhancement layer, and / or a temporal scalability enhancement layer.

[0098] In one embodiment, increasing the quality of the video comprises: activating or adjusting the first postfilter used for filtering decoded pictures produced by the decoder; decoding one or more encoded pictures belonging to an enhancement layer; and / or decoding one or more encoded non-reference pictures.

[0099] In one embodiment, process 500 comprises decreasing the quality of the video as a result of determining that the viewer is not attentive to the display device, and decreasing the quality of the video comprises: deactivating or adjusting a first postfilter used for filtering the decoded pictures produced by the decoder, and / or refraining from decoding one or more encoded pictures belonging to an enhancement layer, and / or refraining from decoding one or more encoded non-reference pictures.[000100] FIG. 6 is a hardware block diagram of device 103, according to some embodiments. As shown in FIG. 6, device 103 may comprise: processing circuitry (PC) 602, which includes one or more processors (P) 655 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., encoder device 103 may be a distributed computing apparatus comprising multiple computers or may be a monolithic device comprising a single computer); at least one network interface 648 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 645 and a receiver (Rx) 647 for enabling device 103 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 648 is connected (physically or wirelessly) (e.g., network interface 648 may be coupled to an antenna arrangement comprising one or more antennas for enabling encoder device 103 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 608, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 602 includes a programmable processor, a computer readablestorage medium (CRSM) 642 may be provided. CRSM 642 may store a computer program (CP) 643 comprising computer readable instructions (CRI) 644. CRSM 642 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 644 of computer program 643 is configured such that when executed by PC 602, the CRI causes encoder device 103 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, encoder device 103 may be configured to perform steps described herein without the need for code. That is, for example, PC 602 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.[000101] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.[000102] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS1. A method (500) in a system comprising a decoder (104) and a display device (106), the method comprising: obtaining (s502) a bitstream comprising a sequence of encoded pictures; decoding (s504), by the decoder, the encoded pictures to produce a sequence of decoded pictures; using (s506) the sequence of decoded pictures to display video on the display device; while the video is being displayed on the display device, determining (s508) whether a viewer of the display device is attentive to the display device; and as a result of determining that the viewer is not attentive to the display device, decreasing (s510) a quality of the video to be displayed on the display device, or as a result of determining that the viewer is attentive to the display device, increasing (s512) a quality of the video to be displayed on the display device.

2. The method of claim 1, wherein the method comprises decreasing the quality of the video as a result of determining that the viewer is not attentive to the display device, and the method further comprises: after decreasing the quality of the video, determining that the viewer is attentive to the display device; and as a result of determining that the viewer is attentive to the display device, increasing a quality of the video to be displayed on the display device, wherein increasing the quality of the video comprises: reactivating or readjusting the first postfilter, and / or decoding one or more of the encoded pictures belonging to the enhancement layer, and / or decoding one or more of the encoded non-reference pictures.

3. The method of claim 1 or 2, whereinthe method comprises decreasing the quality of the video as a result of determining that the viewer is not attentive to the display device, and decreasing the quality of the video comprises deactivating or adjusting the first postfilter.

4. The method of claim 3, wherein decreasing the quality of the video comprises adjusting the first postfilter.

5. The method of claim 4, wherein adjusting the first postfilter comprises gradually adjusting the first postfilter.

6. The method of claim 5, wherein gradually adjusting the first postfilter comprises: obtaining a first input picture, II, obtaining a first post filter offset, DI, calculating a first weight, Wl; outputting a first output picture, 01, where 01 = 11 + (WlxDI), obtaining a second input picture, 12, wherein 12 immediately follows II, obtaining a second post filter offset, D2, calculating a second weight, W2; outputting a second output picture, 02, where 02 = 12 + (W2 x D2), where W2 < Wl .

7. The method of claim 6, whereinWl = (FR - 1) / FR,W2 = (FR - 2) / FR, andFR is the frame rate of the video.

8. The method of any one of claims 3-7, wherein the encoded pictures include the encoded pictures belonging to the enhancement layer, and the method further comprises refraining from decoding one or more of the encoded pictures belonging to the enhancement layer as a result of determining that the viewer is not attentive to the display device.

9. The method of any one of claims 3-8, wherein the encoded pictures include the encoded non-reference pictures, and the method further comprises refraining from decoding one or more of the encoded nonreference pictures as a result of determining that the viewer is not attentive to the display device.

10. The method of any one of claims 1-9, wherein the sequence of decoded pictures comprises a first decoded picture followed by a second decoded picture, using the decoded pictures to display video on the display device comprises applying the first postfilter to a first portion of the first decoded picture to produce a postfiltered first portion of the first decoded picture and displaying the postfiltered first portion of the first decoded picture using a first portion of the display device; determining whether a viewer of the display device is attentive to the display device comprises determining whether the viewer is attentive to the first portion of the display device that is being used to display the postfiltered first portion of the first decoded picture; as a result of determining that the viewer is not attentive to the first portion of the display device, deactivating or adjusting the first postfilter so that the first postfilter is not applied to any portion of the second decoded picture; and displaying the second decoded picture.

11. The method of claim 10, wherein using the decoded pictures to display video on the display device further comprises applying a second postfilter to a second portion of the first decoded picture to produce a postfiltered second portion of the first decoded picture and displaying the postfiltered second portion of the first decoded picture using a second portion of the display device, and the second postfilter is different than the first postfilter.

12. The method of any one of claims 1-11, wherein the sequence of decoded pictures comprises a first decoded picture followed by a second decoded picture,using the decoded pictures to display video on the display device comprises applying the first postfilter to the first decoded picture to produce a first postfiltered picture and displaying the first postfiltered picture, the method further comprises determining that the viewer is attentive to a first portion of the display device or a point adjacent to the first portion of the display device, and is not attentive to a second portion of the display device, as a result of determining that the viewer is attentive to the first portion of the display device or the adjacent point, but not the second portion, adjusting the first postfilter such that the first postfilter is applied to a first portion of the seconded decoded picture but not applied to, or not fully applied to, a second portion of the seconded decoded picture; displaying the postfiltered first portion of the seconded decoded picture using the first portion of the display device; and displaying the second portion of the seconded decoded picture using the second portion of the display device.

13. The method of any one of claims 1-12, wherein determining whether the viewer is attentive to the display device comprises determining an amount of time that the viewer is looking at something other than the display device, and determining whether the amount of time exceeds a threshold.

14. The method of any one of claims 1-13, wherein determining whether the viewer of the display device is attentive to the display device comprises: determining a direction in which the viewer is looking; based on the determined direction, determining a set of one or more coordinates; and comparing at least one of the determined coordinates with a coordinate of the video15. The method of any one of claims 1-14, wherein determining whether the viewer of the display device is attentive to the display device comprises: predicting a direction in which the user may look; based on the predicted direction, determining a set of one or more coordinates; and comparing at least one of the determined coordinates with a coordinate of the video.

16. The method of any one of claims 1-15, wherein determining whether the viewer of the display device is attentive to the display device comprises: not more than once every si seconds, obtaining data from a sensor and using the obtained data to determine whether the viewer of the display device is attentive to the display device, and the method further comprises, after determining that the viewer of the display device is not attentive to the display device, then, at least once every s2 seconds, obtaining data from the sensor and using the obtained data to determine whether the viewer of the display device is attentive to the display device, wherein si > s2.

17. The method of any one of claims 1-16, wherein the method further comprises obtaining side luma information indicating an amount by which the first postfilter changes the luma for each picture, and decreasing the quality of the video comprises deactivating or adjusting the first postfilter and using the luma information to adjust pixels of decoded pictures.

18. The method of any one of claims 1-17, wherein the enhancement layer is: a spatial scalability enhancement layer, a quality scalability enhancement layer, and / or a temporal scalability enhancement layer.

19. The method of any one of claims 1-18, wherein increasing the quality of the video comprises: activating or adjusting the first postfilter used for filtering decoded pictures produced by the decoder; decoding one or more encoded pictures belonging to an enhancement layer; and / or decoding one or more encoded non-reference pictures.

20. The method of any one of claims 1-19, whereinthe method comprises decreasing the quality of the video as a result of determining that the viewer is not attentive to the display device, and decreasing the quality of the video comprises: deactivating or adjusting a first postfilter used for filtering the decoded pictures produced by the decoder, and / or refraining from decoding one or more encoded pictures belonging to an enhancement layer, and / or refraining from decoding one or more encoded non-reference pictures.

21. A computer program (643) comprising instructions (644) which when executed by processing circuitry (602) of a device (103) causes the apparatus to perform the method of any one of claims 1-19.

22. A carrier containing the computer program of embodiment 21, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (642).

23. A device (103), the device being configured to perform a method comprising: obtaining a bitstream comprising a sequence of encoded pictures; decoding the encoded pictures to produce a sequence of decoded pictures; using the sequence of decoded pictures to display video on a display device; while the video is being displayed on the display device, determining whether a viewer of the display device is attentive to the display device; and as a result of determining that the viewer is not attentive to the display device, decreasing a quality of the video to be displayed on the display device, or as a result of determining that the viewer is attentive to the display device, increasing a quality of the video to be displayed on the display device.

24. The device of claim 23, wherein the device is further configured to perform the method of any one of claims 2-20.

Images