Augmented reality coding

Abstract

The present disclosure relates to a system for displaying a sequence of image frames, the system comprising an image source and a display device wherein the image source is configured to: generate an image (3); obtaining a predicted display window position (2a); and transmitting the generated image (3) and the predicted display window position (2a) to the display device; and wherein the display device is configured to: receive the generated image (3) and the predicted display window position (2a) from the image source; and adjusting the generated image to obtain a first adjusted image (1a) corresponding to the predicted display window position (2a).

Description

Technical Field

[0001] The following disclosure relates to a system for generating and displaying an image on a display device, such as in a VR or XR application. The image may, for example, virtually represent a 3D space. The display device may, for example, be a user headset. The image is typically generated remotely from the display device, and there are typically limitations on the speed and capacity of communication between the image source and the display device. For example, the image source and the display device may be connected via a network, and the image source may, for example, be located on a server or in the cloud. Background Art

[0002] Graphics for current-generation consoles require hundreds of watts of power, cooling systems, discrete graphics cards, and more, all of which can make for a unit that’s too big and heavy to be used as a headset.

[0003] To provide a practical headset that supports similar graphics (e.g., in VR, XR, or Metaverse applications), it may be necessary to render the graphics on a separate device (e.g., on a local device or a remote device such as a cloud system) and then transmit the rendered image (or another form of visual data such as a point cloud) to a display device.

[0004] Due to the limitations of communication speed and capacity between devices and the processing speed and processing resources required for image generation, it is desirable to reduce factors such as frame rate and amount of transmitted data as much as possible without affecting the user experience.

[0005] For example, in a system including a cloud server and a VR headset, the communication speed cannot be higher than 50Mbps. This is not enough to support stereoscopic 4K at a high frame rate (e.g., 72fps). Therefore, it may be necessary to limit the quality of the VR display, or implement compression techniques, preferably lossless compression techniques, so that the final display does not show compression artifacts.

[0006] Additionally, due to the time required to generate an image and transmit the image to the display device, there may be a difference between the display required at the time the image is generated and the display required at the time the image is displayed on the display device. Rendering, encoding, transmission, and decoding all take time, so the time to display the image may be at least 30ms after the time the image is generated. For example, when the display device is a user headset, the user may intentionally or unintentionally move their head so that the user is facing a different direction when the image is generated than when the image is displayed. This difference may cause discomfort to the user and is preferably minimized.

[0007] One known technique for dealing with finite frame rates and delays between image generation and image display is called warping or time warping.

[0008] The preprocessor used in the comparative warping method is configured to preprocess (e.g., render, encode, etc.) a video containing the user's field of view at the time of preprocessing (e.g., the time before rendering starts / before it starts) and the (predicted) field of view at the (predicted) time when the preprocessed frame is displayed. For example, instead of just transmitting a video of the user's actual field of view at time T=0ms (i.e., when rendering starts), the user's head position is predicted to be XYZ (T=30ms) based on the user's head movement, and a slightly larger window centered on XYZ (T=30ms) is rendered and transmitted starting at time T=0ms. Then, when the user's headphones receive the larger window, the transmitted window is adjusted based on the user's actual field of view at time T=30ms based on the user's most recent head movement, thereby providing an overall impression of zero latency. Therefore, comparative warping methods generally take into account predicted or potential changes in the field of view between the preprocessing time and the display time of the frame.

[0009] The described embodiments take into account changes in the field of view between the preprocessing time and multiple (subsequent) frames after the preprocessing time of a frame. In other words, the described embodiments generate images that contain a sufficiently large field of view to take into account predicted head movements that may be made during the next (several) frames after the preprocessing time for a given frame.

[0010] Thus, in general, the field of view considered in the images generated by preprocessing in the described embodiments is a larger field of view than the field of view generated by the comparative warping method. Thus, in general, the images generated by preprocessing in the described embodiments are (spatially) larger images than the images generated by the comparative warping method. For the sake of completeness, in general, the field of view considered in the images generated in the preprocessing in the described embodiments is therefore a larger field of view than the field of view considered in the images generated by the comparative non-warping method. Summary of the invention

[0011] According to a first aspect, the present disclosure provides a system for displaying a sequence of image frames, the system comprising an image source and a display device, wherein the image source is configured to: generate an image; obtain a predicted display window position; and transmit the generated image and the predicted display window position to the display device; and wherein the display device is configured to: receive the generated image and the predicted display window position from the image source; and adjust the generated image to obtain a first adjusted image corresponding to the predicted display window position.

[0012] An image frame sequence is a series of image data frames. The image data may be suitable for display by a VR display device.

[0013] The display device may be, for example, a VR viewing device such as an Oculus Quest or HTC Vive Cosmos Elite.

[0014] By generating an image that is larger than the predicted display window position, and including an indication of the predicted display window position with information transmitted to the display device, the boundaries of the image can be determined independently of the predicted display window position and can be static for multiple frames. Making the image boundaries static can improve the compressibility of the transmitted image.

[0015] Optionally, the generated image has a field of view with an extra-large field of view.

[0016] Optionally, the image source is configured to generate an image based on the predicted display window position. In other words, the image is generated to include at least a field of view including the predicted display window position.

[0017] Optionally, the image source is configured to generate an image based on at least two predicted display window positions, wherein the predicted display window positions correspond to different frames of the image frame sequence. In other words, the image is generated to include a field of view that includes at least the predicted display window position at the first frame and the predicted display window position at the second frame.

[0018] Optionally, the display device is further configured to: obtain a final display window position; and adjust the first adjustment image to obtain a second adjustment image corresponding to the final display window position. Optionally, the display device is further configured to: display the second adjustment image. In other words, in addition to the processing of the first aspect, a distortion may also be performed.

[0019] The final display window position may be the user's field of view in the 3D rendering space at the time the image is displayed.

[0020] Adjusting the first adjustment image may include cropping, translating, or rotating the first adjustment image.

[0021] The predicted display window position may be a prediction of the user's field of view in the 3D rendering space at the time the image frame is displayed, wherein the prediction is performed while the image frame is being rendered.

[0022] Adjusting the generated image may include cropping, translating, or rotating the generated image.

[0023] Optionally, coordinate values ​​are used to indicate the predicted display window position.

[0024] Optionally, the display device is further configured to: receive a second predicted display window position from an image source; and adjust the generated image to obtain a third adjusted image corresponding to the second predicted display window position. In other words, the display device is configured to use a single generated image to obtain adjusted images corresponding to multiple predicted display window positions.

[0025] Optionally, the display device is further configured to: calculate a second predicted display window position; and adjust the generated image to obtain a third adjusted image corresponding to the second predicted display window position. In other words, the display device can use a single generated image and a single predicted display window position to obtain an adjusted image corresponding to the display window position. For example, the display device can use a motion vector associated with the first predicted display window position to calculate the second display window position without further communication from the image source.

[0026] Optionally, the display device is configured to store the generated image of the previous frame, and the image source is configured to transmit the difference between the generated image of the current frame and the generated image of the previous frame. When the boundary of the generated image is static for multiple frames, the difference between images of different frames is reduced, and the image can be compressed more efficiently by transmitting the difference with the previous image.

[0027] Optionally, the image source includes an encoder and the display device includes a decoder, the image source is configured to: encode the generated image and the predicted display window position into a coded frame; and transmit the coded frame to the display device, and the display device is configured to: receive and decode the coded frame to obtain the generated image and the predicted display window position.

[0028] Optionally, the image encoder is an LCEVC encoder and the image decoder is an LCEVC decoder.

[0029] LCEVC compression performance is particularly effective for scenes including static components and high motion components. However, in the case of small or random motion, LCEVC efficiency may be lower. As described above, according to the present invention, the boundaries of the transmitted image can be determined independently of the predicted display window position and can be static for multiple frames, which means that LCEVC can provide effective compression in combination with the method of the first aspect.

[0030] Optionally, the system further comprises a network, wherein the image source is configured to stream the sequence of image frames to the display device via the network.

[0031] According to a second aspect, the present invention provides a transmission method for a system for displaying an image frame sequence, wherein the method comprises: obtaining a generated image from an image source; obtaining a predicted display window position; and transmitting the generated image and the predicted display window position to a display device.

[0032] According to a third aspect, the present invention provides an encoding method for a system for displaying a sequence of image frames, wherein the method comprises: obtaining a generated image from an image source; obtaining a predicted display window position; and encoding the generated image and the predicted display window position into an encoded frame.

[0033] According to a fourth aspect, the present disclosure provides a receiving method of a system for displaying an image frame sequence, wherein the method comprises: receiving a generated image and a predicted display window position from an image source; and adjusting the generated image to obtain a first adjusted image corresponding to the predicted display window position.

[0034] According to a fifth aspect, the present invention provides a decoding method for a system for displaying an image frame sequence, wherein the method comprises: decoding a coded frame to obtain a generated image and a predicted display window position; adjusting the generated image to obtain a first adjusted image corresponding to the predicted display window position; and outputting the first adjusted image for use by a display device.

[0035] According to a sixth aspect, we describe a method for generating data suitable for constructing a series of image data frames. The construction can be performed by a viewing device, in particular a VR viewing device such as Oculus Quest or HTC Vive Cosmos Elite. The image data can be suitable for display by a VR display device. The method may include determining an extra-large field of view. The extra-large field of view may include and / or contain and / or be greater than a first field of view. The first field of view may correspond to an expected field of view. The expected field of view may be a predicted / expected field of view that will (and / or is expected to) be displayed on the VR display device at the display (and / or generation) time of the first frame (in a series of frames).

[0036] The extra-large field of view may include and / or contain and / or be larger than the second field of view. The second field of view may correspond to a second expected field of view. The second expected field of view may be a predicted / expected field of view that will (and / or is expected to) be displayed on the VR display device at the display (and / or generation) time of the second frame (in the series of frames).

[0037] The method may include generating an extra-large image having the determined extra-large field of view. In other words, the extra-large field of view includes a field of view that is large enough to include: a first field of view corresponding to the generation and / or display time of a first image data frame; a second field of view corresponding to the generation and / or display time of a second image data frame at a later time. Generating an extra-large image may include processing input data. The input data may include multiple images to be rendered, data indicating a viewer's field of view, viewer data, volume data, technical information indicating the resolution of a VR display device, and information fed to a renderer. Generating an extra-large image may implicitly include the previously described step of determining an extra-large field of view. In general, the extra-large image will be processed by a display module of a VR display device to generate a VR scene for a viewer, wherein the viewer has a field of view that may depend at least on the position or movement of the viewer's head and eyes.

[0038] According to a seventh aspect, we describe a related method for generating a sequence of frames for display on a VR display device, preferably performed by a VR display device. The method may further include displaying the sequence of frames generated for display on a VR display device. The method for generating a sequence of frames for display on a VR display device may include obtaining an ultra-large image (such as the ultra-large image described above) with an ultra-large field of view, wherein the ultra-large field of view may include: a first field of view corresponding to a (known or predicted) display time of a first frame; and a second field of view corresponding to a (known or predicted) display time of a later second frame. The method may include obtaining first position data of the first frame, wherein the position data is suitable for combining with the ultra-large image to generate the first frame. The method may include combining the ultra-large image with the first position data to generate a first frame at a first display time (i.e., for display on a VR display device). The method may include obtaining position data of the second frame, wherein the position data is suitable for combining with rendering of the ultra-large image to generate a second frame. The method may include combining the rendering of the ultra-large image with the second position data to generate a second frame at a second display time (i.e., for display on a VR display device). The second display time may be after the first display time. Thus, generally speaking, the method may include applying a stabilization method to (a rendering of) the oversized image to generate a first frame. At a later time, the method may include applying a stabilization method to (a rendering of) the oversized image to generate a second frame. The method may further include warping the first frame to produce a warped first frame. The warping may include adjusting the first frame based on data received from a sensor at (or shortly before) an expected display time of the first frame.

[0039] The method may include the VR display device obtaining positioning data, the positioning data being suitable for combination (at the VR display device) with the oversized image to create a reduced image. The reduced image may be suitable for providing a field of view at a viewing / display time. The method may include the VR display device obtaining subsequent position data at a later time, wherein the subsequent position data is suitable for combination with a rendering of the generated oversized image to create another reduced image, wherein the reduced image represents (or is suitable for providing) a viewer's field of view at a later time (i.e., a time associated with the display and / or generation of the other image). The position data may be associated with a frame or frame time or viewing time, and the subsequent position data may be associated with a subsequent frame or a subsequent frame time or a subsequent viewing time.

[0040] According to an eighth aspect, we describe another related method preferably performed by a VR display device. The method may include adjusting the super large image by using the first position data received and / or generated at a first time to select a field of view included in the super large image. The first time may be the time when the super large image is obtained (for example, by the VR display device) and / or the time when the selected field of view is displayed. The method may include updating the rendering of the super large image (i.e., the super large image initially generated in conjunction with the previous frame (such as the first frame)) at a later time by using the second position data received and / or generated to select a field of view included in the rendering of the super large image, and the selected field of view may correspond to the view of the viewer at the time of the update and / or the predicted time of displaying the updated image. The rendering of the super large image may be the result of enhancement data (for example, such as residuals, specifically, residuals obtained from an LCEVC stream) applied to the super large image. The method may include generating (for example, by a VR viewing device) the rendering of the super large image by enhancing the super large image using the enhancement data. The enhancement data may be derived from the LCEVC enhancement layer.

[0041] By field of view, we generally mean the view that a viewer of a VR viewing device can see, for example, at a given time. The field of view may depend on a number of factors, such as the viewer's eye position and / or motion, such as the viewer's head position and / or motion, the resolution of the VR display device, the size of the VR display device. The field of view may also be referred to as a scene, as this is what a viewer of a VR display device sees at a given time.

[0042] In other words, the super-large field of view may include a field of view that is larger than: the field of view at the time of said generation of the super-large image; and / or the predicted and / or expected field of view at the expected display time of the first frame in a series of frames; and the predicted and / or expected field of view corresponding to the time when a future (e.g., second, third, another, etc.) frame will be displayed (by the VR display device). The expected field of view may be considered to be the field of view that includes the widest range of predicted fields of view. In other words, the expected field of view may include a field of view that corresponds to an extreme (but realistic) head movement of a viewer of the VR display device.

[0043] Position data may be data that may be combined with a super large image to produce an image that accounts for movement of a viewer's head and / or eyes relative to a previous time, such as the time the super large image was generated.

[0044] A VR display device may be referred to as a VR viewing device. References to VR may also be references to AR, or more generally, to XR.

[0045] In comparative methods, such as warping methods, an image may be generated (e.g. rendered) that is larger than the final displayed image. However, the generated image is only created large enough to capture the (potential) change in the field of view between the time the image is generated and the time the generated (and subsequently warped) image is displayed. In contrast, embodiments of the described methods generate an "oversized" image that is large enough to capture any potential change in the field of view between the time the (first) image is generated and the time the second, third, fourth, fifth image is generated or displayed. Thus, in general, the oversized image of the described method is larger than the image generated in the comparative warping method. It is counterintuitive to generate such a large image as part of the comparative warping method, because the comparative warping method periodically sends (entire) update frames (and before the next frame needs to be displayed). Therefore, a skilled person would consider that modifying the comparative warping method to send such an "oversized" image would be an inefficient use of bandwidth between the pre-processing device (e.g. renderer) and the VR display device.

[0046] In the described embodiments, the super large image may be encoded using a layered codec, in particular using LCEVC. By sending a single super large image including the field of view of multiple frames, the LCEVC temporal approach may be utilized. This is because LCEVC temporal data (i.e., "residual delta") may be utilized, rather than non-temporal signals (e.g., actual residual values), because the super large image may be adjusted (and / or enhanced) by the LCEVC temporal data, because the super large image includes the field of view of multiple frames. In other words, by combining the super large image with positional data (e.g., such as the view of the viewer within the super large image at a given time / frame) for each frame, the super large image may be advantageously (re)used for multiple frames (i.e., frames displayed on a VR display device). Thus, in general, a single super large image and multiple sets of positional data are used. This is in contrast to a more distorted approach by sending a "fresh" frame for each frame displayed by the VR display device (i.e., from "renderer to VR display device").

[0047] According to a ninth aspect, we describe a related method, in particular a method for preprocessing data for encoding (e.g., volume data, position data, etc.). Preprocessing may generally include processing data to generate a rendering of image data, such as a frame of image data, multiple planes of image data, an encoded rendering of image data. The method may include generating an image for encoding (e.g., the aforementioned super-large image), wherein the rendering of the image is suitable for use by a VR display device to create a VR display for a viewer of the VR display device. The rendering may be a decoded version of the encoded rendering of the image. The image may include a field of view that is larger than and / or includes the field of view of the viewer at the (e.g., predicted and / or reasonable and / or expected) generation time of the image. The image may include a field of view that is larger than and / or includes the field of view of the viewer (e.g., predicted and / or reasonable and / or expected) at the display time of the generated image. The image may include a field of view that is larger than and / or includes the field of view of the viewer (e.g., predicted and / or reasonable and / or expected) at the display time of another image. The generated image and the other image may form an image sequence suitable for viewing by a viewer, and the other image is displayed later (i.e., at a later time) in the sequence.

[0048] The pre-processing method may include encoding the image, in particular using the LCEVC encoding method. The pre-processing method may include sending the encoded image to the VR display device via a wired or wireless connection.

[0049] The method of preprocessing may include capturing position data via a sensor to obtain a user's field of view. The position data may indicate a viewer's head and / or eye position and / or gaze. The method of preprocessing may include generating an image by processing the position data associated with the time of the generation.

[0050] According to a tenth aspect, the present disclosure provides a device configured to perform the method of any one of the second to ninth aspects. The device may include a processor and a memory storing instructions, and when the instructions are executed by the processor, the processor performs the method. Alternatively, the device may include a circuit in which all or part of the method is hard-coded. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 schematically illustrates a sequence of image frames according to the present invention;

[0052] FIG2 schematically illustrates a sequence of image frames according to a known warping technique;

[0053] Figure 3 A system for displaying a sequence of image frames is schematically illustrated;

[0054] Figure 4A and 4B Schematically showing the features of LCEVC encoding and decoding related to the present invention;

[0055] Figure 5 is an illustrative example of layered decoding;

[0056] Figure 6 is a schematic example of a residual plot;

[0057] Figure 7 is a flow chart schematically showing a method performed by an image source device;

[0058] Figure 8 is a flow chart schematically showing a method performed by a display device;

[0059] Fig. 9 is a flow chart schematically illustrating another method performed by a display device. DETAILED DESCRIPTION

[0060] Figure 1 A sequence of image frames according to the invention is schematically illustrated.

[0061] like Figure 1 As shown, the display image sequence 1a, 1b, 1c displayed by the display device corresponds to different parts of the entire image. The entire image can be a projection of a large 2D or 3D area, and the display device can only display a part of it at any given time. This is particularly suitable for VR, XR or AR displays, in which the user has a variable position and viewing direction in the virtual 3D space. The viewing direction can be associated with a position in the real space - for example, the orientation of the headset. In addition or alternatively, the viewing direction can change according to virtual events in the virtual space or user input such as controller input.

[0062] Each of the displayed images 1a, 1b, 1c is a portion of a region of a warpable image 2a, 2b, 2c which is suitable for warping before displaying the image based on the user's most recent viewing direction.

[0063] Each of the warpable images 2a, 2b, 2c is itself part of a static image region 3. In other words, the same static image region 3 applies to multiple frames - each frame of the sequence of image frames. For example, the static image region 3 may also change when the user changes their position in the virtual environment, or when the user's viewing direction changes by more than a threshold.

[0064] Conventionally, a distortable image 2a, 2b, 2c is transmitted to a display device for each frame, as shown in FIG2. Figure 1 The invention shown provides the entire static image area 3 to the display device for each frame.

[0065] Here, "providing" may refer to transmitting the static image region freshly from the image source to the display device for each frame. However, preferably, the display device updates the stored version of the static image region 3 based on the data received from the image source for each frame. By updating the stored version of the static image region, the amount of data transmitted from the image source to the display device for each frame can be reduced.

[0066] For example, the static image area 3 may be encoded using a coding technique that performs compression based on the difference between the current frame and the previous frame before being transmitted from the image source to the display device. Among these techniques, LCEVC is particularly effective for compressing the static image area 3.

[0067] Providing a static image area 3 may alternatively be described as performing some kind of inverse stabilization of the extended field of view transmitted to the display device.

[0068] For example, instead of assuming that the center of the field of view (at the display time, e.g. XYZ(T=30)) is always located at the center of the transmitted video, the transmitted video is stabilized as much as possible, and the XYZ coordinates of the predicted head position are additionally transmitted relative to the transmitted video. If the user moves their head slightly, they will receive a stabilized video, with a moving XYZ reference. This can then be further corrected at the last moment by a warping process, as described above.

[0069] The difference between this "inverse stabilization" and conventional stabilization is that in conventional stabilization, moving video frames are cropped, leaving smaller frames in the stabilization window. On the other hand, this inverse stabilization method benefits from the fact that the rendering device can process the entire scene. A larger window of image 3 is stably defined while still being able to show the original motion between frames 1a, 2a, 3a. In other words, the advantages of stabilization can be obtained by extending the extended field of view used for warping to double the extended field of view video.

[0070] Figure 3 A system for displaying a sequence of image frames is schematically shown.

[0071] Reference Figure 3 The system includes an image generator 31, an encoder 32, a transmitter 33, a network 34, a receiver 35, a decoder 36 and a display device 37.

[0072] The image generator 31 may, for example, include a rendering engine for initially rendering a virtual environment such as a game or a virtual conference room. The image generator 31 is configured to generate a sequence of images 3 to be displayed. The image may be based on the state of the virtual environment, the position of the user, or the viewing direction of the user. Here, the position and viewing direction may be physical attributes of the user in the real world, or the position and viewing direction may also be purely virtual, such as controlled using a handheld controller. The image generator 31 may, for example, obtain information indicating the position, viewing direction, or motion of the user from a display device 37. In other cases, the generated image may be independent of the user's position and viewing direction. The generation of such images generally requires a large amount of computer resources and may be implemented in a cloud service or on a local but powerful computer. Here, "rendering" refers at least to the initial stage of rendering to generate an image. Further rendering may be performed at the display device 37 based on the generated image to produce the final image displayed.

[0073] The encoder 32 is configured to encode frames to be transmitted to the display device 37. The encoder 32 may be implemented using executable software or may be implemented on specific hardware such as an ASIC. Each frame includes all or part of an image generated by the image generator 31.

[0074] In addition, each frame includes a predicted display window position. The predicted display window position is the position of the portion of the generated image 2a, 2b, 2c that may be displayed by the display device 37. The predicted display window position may be based on the viewing direction of the user obtained from the display device 37. The predicted display window position may be defined using one or more coordinates. For example, referring to Figure 1, the predicted display window position can be defined using the coordinates of the corners or center of the predicted display window, and the predicted display window position can be defined using the size of the predicted display window. The predicted display window position can be encoded as part of the metadata included in the frame.

[0075] The encoder may apply inter-frame or intra-frame compression based on the current encoded frame and optionally one or more previously encoded frames. For example, encoder 32 may be such as Figure 4A The LCEVC encoder shown. The described method is expected to improve the compression of all codecs, especially LCEVC.

[0076] The transmitter 33 may be any known type of transmitter for wired or wireless communication, including an Ethernet transmitter or a Bluetooth transmitter.

[0077] The network 34 is used for communication between the transmitter 33 and the receiver 35 and can be any known type of network, such as a WAN or LAN or a wireless Wi-Fi or Bluetooth network. The network 34 can also be a combination of multiple different types of networks.

[0078] Receiver 35 may be any known type of receiver for wired or wireless communication, including an Ethernet transmitter or a Bluetooth transmitter.

[0079] The decoder 36 is configured to receive the coded frame and decode the coded frame to obtain the generated image 3 and the predicted display window position. The decoder is further configured to adjust the generated image 3 to obtain the first adjusted image 2a, 2b, 2c corresponding to the predicted display window position. The decoder 36 can be implemented using executable software or can be implemented on specific hardware such as ASIC.

[0080] The display device 37 may be, for example, a television screen or a VR headset. The display device 37 is configured to receive the first adjusted images 2a, 2b, 2c and display the corresponding images 1a, 1b, 1c to the user. The timing of the display may be associated with the configured frame rate so that the display device 37 may wait before displaying the image. The display device 37 may be configured to perform distortion, i.e., to obtain the final display window position, adjust the first adjusted images 2a, 2b, 2c to obtain the second adjusted images 1a, 1b, 1c corresponding to the final display window position, and display the second adjusted image. Since the first adjusted image corresponds to images 2a, 2b, and 2c transmitted to some known display devices (see FIG. 2 ), this distortion may be a pre-existing technology and the present invention may be implemented without modifying the display device 37.

[0081] The image generator 31, encoder 32 and transmitter 33 may be combined into a single device, or may be separated into two or more devices. The image generator 31, encoder 32 and transmitter 33 may be collectively referred to as the "image source" portion of the system.

[0082] The receiver 35, the decoder 36, and the display device 37 may be combined into a single device, or may be separated into two or more devices. For example, some VR headset systems include a base unit and a headset unit that communicate with each other. The receiver 35 and the decoder 36 may be incorporated into such a base unit.

[0083] In some embodiments, network 34 may be omitted. For example, a home display system may include a base unit configured as an image source and a portable display unit including display device 37.

[0084] In a preferred embodiment, the encoder or decoder is part of a layered coding scheme or format based on a layer. Examples of layered coding schemes based on layers include LCEVC: MPEG-5 Part 2 LCEVC ("Low Complexity Enhanced Video Coding") and VC-6: SMPTE VC-6ST-2117, the former described in PCT / GB2020 / 050695, published as WO 2020 / 188273 (and associated standard documents), and the latter described in PCT / GB2018 / 053552, published as WO 2019 / 111010 (and associated standard documents), all of which are incorporated herein by reference. However, the concepts shown herein are not necessarily limited to these specific layered coding schemes.

[0085] Another example is described in WO2018 / 046940, which is incorporated herein by reference. In this example, a set of residuals is encoded relative to the residuals stored in the time buffer.

[0086] LCEVC (Low Complexity Enhancement Video Coding) is a standardized coding method proposed in standard specification documents, including ISO / IEC 23094-2 Version 1 Low Complexity Enhancement Video Coding, released in November 2021, which is incorporated herein by reference.

[0087] Figure 4A and Figure 4BSelected features of an LCEVC encoder 402 and an LCEVC decoder 404 are schematically shown, which show how LCEVC can be used to efficiently encode and decode static image regions 3. Further implementation details of these types of encoders and decoders are set out in earlier published patent applications GB1615265.4 and WO2020188273, each of which is incorporated herein by reference.

[0088] In each of the encoder 402 and the decoder 404, items are shown on two logical layers. The two layers are separated by a dotted line. The items in the first highest layer relate to data of a higher quality layer. The items in the second lowest layer relate to data of a lower quality layer. The higher and lower quality layers relate to a hierarchical structure having multiple quality layers. In some examples, the hierarchical structure includes more than two quality layers. In such examples, the encoder 402 and the decoder 404 may include more than two different layers. Figure 4A and 4B There may be one or more other layers above and / or below the layers depicted in FIG.

[0089] Reference Figure 4A , the encoder 402 obtains input data 406 at a higher quality layer. The input data 406 includes a first rendering of a first time sample t1 of a signal at the higher quality layer. The input data 406 may, for example, include an image generated by the image generator 31. The encoder 402 uses the input data 406, for example, by performing a downsampling operation on the input data 406, to derive downsampled data 412 at a lower quality layer. In the case of processing the downsampled data 412 at the lower quality layer, such processing generates processed data 413 at the lower quality layer.

[0090] In some examples, generating processed data 413 involves encoding downsampled data 412. Encoding downsampled data 412 produces a lower quality layer encoded signal. Encoder 402 may output, for example, an encoded signal for transmission to decoder 404. The encoded signal may be generated by an encoding device separate from encoder 402, rather than being generated in encoder 402. The encoded signal may be an H.264 encoded signal. H.264 encoding may involve arranging a sequence of images into groups of pictures (GOPs). Each image in a GOP represents a different time sample of a signal. In a process known as "inter-frame prediction", a given image in a GOP may be encoded using one or more reference images associated with earlier and / or later time samples from the same GOP.

[0091] Generating processed data 413 at a lower quality layer may further involve decoding the encoded signal at the lower quality layer. The decoding operation may be performed to simulate the decoding operation at the decoder 404, as will become apparent below. Decoding the encoded signal produces a decoded signal at the lower quality layer. In some examples, the encoder 402 decodes the encoded signal at the lower quality layer to produce a decoded signal at the lower quality layer. In other examples, the encoder 402 receives the decoded signal at the lower quality layer, for example, from an encoding and / or decoding device separate from the encoder 402. The encoded signal may be decoded using an H.264 decoder. H.264 decoding produces a sequence of images at the lower quality layer (i.e., a sequence of time samples of the signal). After the H.264 decoding process is completed, no individual image indicates the temporal correlation between different images in the sequence. Therefore, any use of the temporal correlation between sequential images employed by H.264 encoding is eliminated during H.264 decoding because the sequential images are decoupled from each other. Therefore, subsequent processing is performed on an image-by-image basis, where the encoder 402 processes the video signal data.

[0092] In an example, generating processed data 413 at a lower quality layer further involves obtaining correction data based on a comparison between the downsampled data 412 and a decoded signal obtained by the encoder 402, for example, based on a difference between the downsampled data 412 and the decoded signal. The correction data may be used to correct errors introduced when encoding and decoding the downsampled data 412. In some examples, the encoder 402 outputs the correction data, for example, for transmission to the decoder 404, along with the encoded signal. This allows a recipient to correct errors introduced when encoding and decoding the downsampled data 412.

[0093] In some examples, generating processed data 413 at the lower quality layer further involves correcting the decoded signal using the correction data. In other examples, the encoder 402 uses the downsampled data 412 instead of correcting the decoded signal using the correction data.

[0094] In some examples, generating processed data 413 involves performing one or more operations in addition to the encoding, decoding, obtaining, and correcting actions described above.

[0095] However, in some examples, no processing is performed on the downsampled data 412 .

[0096] The data at the lower quality layer is used to derive upsampled data 414 at the higher quality layer, for example, by performing an upsampling operation on the data at the lower quality layer. The upsampled data 414 includes a second rendering of a first time sample of the signal at the higher quality layer. The encoder 402 obtains a set of residual elements 416 that can be used to reconstruct the input data 406 using the upsampled data 414. The set of residual elements 416 is associated with the first time sample t1 of the signal. The set of residual elements 416 is obtained by comparing the input data 406 with the upsampled data 414.

[0097] In this example, the encoder 402 generates a set of time correlation elements 426. The term "time correlation element" is used herein to represent a correlation element indicating the degree of time correlation. The time correlation element can also be a spatiotemporal correlation element indicating the degree of spatial correlation between residual elements. In this example, the set of time correlation elements 426 is associated with a first time sample t1 of the signal and a second time sample t0 of the signal. In the example described herein, the second time sample t0 is an earlier time sample relative to the first time sample. However, in other examples, the second time sample t0 is a later time sample relative to the first time sample t1. In some examples where the input data 406 includes a sequence of time samples, the earlier time sample refers to a time sample in the input data before the first time sample t1. In the case where the first time sample t1 and the earlier time sample are arranged in a presentation order, the earlier time sample is before the first time sample t1.

[0098] The second time sample t0 may be a time sample immediately preceding the first time sample t1. In some examples, the second time sample t0 is a previous time sample relative to the first time sample t1, rather than a previous time sample relative to the first time sample t1.

[0099] In this example, the set of temporal correlation elements 426 indicates a degree of spatial correlation between a plurality of residual elements in the set of residual elements 416. The set of temporal correlation elements 426 also indicates a degree of temporal correlation between first reference data based on the input data 406 and second reference data based on a rendering of, for example, a second time sample t0 of the signal at a higher quality layer. Thus, the first reference data is associated with the first time sample t1 of the signal, and the second reference data is associated with the second time sample t0 of the signal. The first reference data and the second reference data are used as a reference or comparison for determining a degree of temporal correlation associated with the first time sample t1 of the signal and the second time sample t2 of the signal. The first reference data and / or the second reference data may be at a higher quality layer.

[0100] In some examples, the first reference data and the second reference data include a first set of spatial correlation elements and a second set of spatial correlation elements, respectively, the first set of spatial correlation elements being associated with a first time sample t1 of the signal and the second set of spatial correlation elements being associated with a second time sample t0 of the signal.

[0101] In other examples, the first reference data and the second reference data include first and second renderings, respectively, of the signal, the first rendering being associated with a first time sample t1 of the signal and the second rendering being associated with a second time sample t0 of the signal.

[0102] The set of time correlation elements 426 will be referred to hereinafter as “Δ t Correlation element” because the temporal correlation is exploited using data from different time samples to generate Δ t Correlation element 426.

[0103] In this example, encoder 402 transmits the group Δ t Correlation element 426. Since the group Δ t The correlation element 426 exploits temporal redundancy at higher residual layers, so in the presence of strong temporal correlation, the set Δ t The correlation elements 426 may be smaller and may in some cases include more correlation elements with zero values. Thus, when applied to a static image region 3 (which is static and only changes internally), the set of Δ t Correlation element 426.

[0104] Now turn to Figure 4B , the decoder 404 receives the data 420 based on the downsampled data 412 and receives the group Δ t Correlation element 426.

[0105] Where the encoder 402 has processed the downsampled data 412 to generate processed data 413, the decoder 404 processes the received data 420 to generate processed data 422. The processing may include decoding the encoded signal to produce a decoded signal at a lower quality layer. In some examples, the decoder 404 does not perform such processing on the received data 420. The data at the lower quality layer, such as the received data 420 or the processed data 422, is used to derive the upsampled data 414. The upsampled data 414 may be derived by performing an upsampling operation on the data at the lower quality layer.

[0106] Decoder 404 is based at least in part on the set of Δ tThe correlation element 426 is used to obtain the set of residual elements 416. The set of residual elements 416 can be used to reconstruct the input data 406 using the upsampled data 414.

[0107] The present disclosure describes implementations for integrating hybrid backward-compatible coding techniques with existing decoders, optionally via software updates. In a non-limiting example, the present disclosure relates to implementations and integrations of MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC). LCEVC is a hybrid backward-compatible coding technique that is a flexible, adaptable, efficient, and computationally inexpensive coding format that combines different video coding formats, base codecs (i.e., encoder-decoder pairs such as AVC / H.264, HEVC / H.265, or any other current or future codecs, as well as non-standard algorithms such as VP9, ​​AV1, etc.) with coded data of one or more enhancement layers.

[0108] An example hybrid backward compatible coding technique uses a downsampled source signal encoded using a base codec to form a base stream. An enhanced stream is formed using a set of encoded residuals that correct or enhance the base stream, for example, by increasing the resolution or by increasing the frame rate. There may be multiple levels of enhanced data in the hierarchical structure. In some arrangements, the base stream may be decoded by a hardware decoder, while the enhanced stream may be suitable for processing using a software implementation. Therefore, the stream is considered to be a base stream and one or more enhanced streams, wherein there may typically be two enhanced streams but one enhanced stream is often used. It is worth noting that typically the base stream may be decoded by a hardware decoder, while the enhanced stream may be suitable for software processing implementations with suitable power consumption. The stream may also be considered to be a layer.

[0109] The combined intermediate image is then upsampled again to give a preliminary output image of the highest resolution. The second enhancement sublayer is combined with the preliminary output image to give a combined output image.

[0110] The second enhancement sublayer may be derived in part from a temporal buffer, which is a storage for the second enhancement sublayer of a previous frame. An indication of whether the temporal buffer may be used for the current frame or which parts of the temporal buffer may be used for the current frame may be included in the coded frame. The use of a temporal buffer reduces the amount of data that needs to be included in the coded frame. The temporal buffer may also be used for the first enhancement sublayer.

[0111] Compared to the block-based approach used in the MPEG family of algorithms, video frames are encoded in a hierarchical manner. Encoding frames in a hierarchical manner includes generating a residual for a full frame, and then generating a residual for a reduced or decimated frame, etc. In the examples described herein, the residual can be considered as an error or difference at a specific quality level or resolution.

[0112] For context purposes only, as the detailed structure of the LCEVC is known and is set out in the approved draft standard specification, Figure 1 How LCEVC operates on the decoding side is explained in a logical flow, assuming H.264 as the base codec. Those skilled in the art will understand how the examples described herein are also applicable to other multi-layer coding schemes (e.g., those using base layers and enhancement layers) based on the general description of LCEVC. The LCEVC decoder operates on a separate video frame layer. It takes the decoded low-resolution pictures and LCEVC enhancement data from the base (H.264 or other) video decoder as input to produce decoded full-resolution pictures ready to be displayed on the display view. LCEVC enhancement data is usually received in the supplemental enhancement information (SEI) of the H.264 network abstraction layer (NAL) or in the additional data packet identifier (PID) and separated from the base coded video by a demultiplexer. Therefore, the base video decoder receives the demultiplexed encoded base stream and the LCEVC decoder receives the demultiplexed encoded enhancement stream, which is decoded by the LCEVC decoder to generate a set of residuals for combining with the decoded low-resolution pictures from the base video decoder.

[0113] LCEVC can be quickly implemented in existing decoders with software updates and is inherently backward compatible since devices that have not yet been updated to decode LCEVC can play video using the basic base codec, further simplifying deployment.

[0114] In this context, a decoder implementation is proposed herein to integrate decoding and presentation with existing systems and devices that perform basic decoding. The integration is easy to deploy. This also enables support for a wide range of encoding and player vendors, and can be easily updated to support future systems. Embodiments of the present invention specifically relate to how to implement LCEVC in a way that provides decoding of protected content in a secure manner.

[0115] The proposed decoder implementation can be provided through an optimized software library for decoding MPEG-5 LCEVC enhancement streams, thereby providing a simple and powerful control interface or API. This allows developers to flexibly deploy LCEVC at any level of the software stack, such as from low-level command line tools to integration with common open source encoders and players. Specifically, embodiments of the present invention generally relate to driver-level implementations and system-on-chip (SoC)-level implementations.

[0116] The terms LCEVC and enhancement may be used interchangeably herein, for example, an enhancement layer may include one or more enhancement streams, ie, residual data of the LCEVC enhancement data.

[0117] Figure 5is a schematic diagram showing the process flow of LCEVC. In the first step, the base decoder decodes the base layer to obtain a low-resolution frame (i.e., the base layer). In the next step, the initial enhancement (a sublayer of the enhancement layer) corrects artifacts in the base. As another step, the final frame (for output) is reconstructed at the target resolution by applying another (e.g., other residual details) sublayer of the enhancement layer. This shows that LCEVC improves quality and reduces the overall computational requirements for encoding by making the best use of existing codecs and enhanced features. Embodiments of the present invention provide ways to achieve this in a secure manner (e.g., when processing protected content).

[0118] Figure 6 The enhancement layers are shown. As can be seen, the enhancement layers include sparse, highly detailed information that would not be interesting (or valuable to the viewer) without the base video.

[0119] In the context of the currently addressed problem, subtle motion in the rendered XR view will increase the information stored in this enhancement layer.

[0120] Figure 7 is a flow chart schematically illustrating an example method performed by the image generator 31 .

[0121] Reference Figure 7 , in step S710, the image generator 31 obtains the predicted display window position of the first frame.

[0122] The predicted display window position is a portion of a 2D or 3D area. The predicted display window position can be represented by coordinates, such as representing a viewing position (i.e., where the user is "located" in the 2D or 3D area), a viewing orientation (i.e., the direction in which the user is viewing in the 2D or 3D area), and / or a viewing range (angle and / or distance) in 3D space. The "predicted display window position" may alternatively be referred to as a "predicted field of view." The predicted display window position may be predicted, for example, based on a state or event in a virtual environment (such as a dynamically generated game or 3D video recording).

[0123] Additionally, the predicted display window position may be predicted based on feedback from the display device 37. For example, where the display device 37 is a VR headset, the predicted display window position may move with the actual motion of the display device 37. For example, the feedback may include a position vector and / or a motion vector of the display device 37.

[0124] The first frame may be the next frame to be displayed by the display device 31. In other words, the predicted display window position of the first frame is the predicted display window position for a short time in the future, where the short time corresponds to the minimum delay between rendering and display (eg, 30 ms).

[0125] In step S720, the image generator 31 obtains the predicted display window position of the second frame.

[0126] The second frame may be immediately after the first frame, or may be after two or more frames after the first frame. In other words, the predicted display window position of the second frame may be the predicted display window position at a time of an integer multiple of the frame spacing after displaying the first frame.

[0127] In step S730, the image generator 31 generates image 3. Image 3 represents a super-portion of a 2D or 3D area, including at least the predicted display window position of the first frame and the predicted display window position of the second frame. In other words, when it is predicted that the viewing orientation or viewing range of the user will change between the first frame and the second frame, the generated image includes enough information to display the two frames. The generated image can then be cropped, translated or rotated at the display device 37 to generate the first frame and the second frame. In some embodiments, the generated image can support changes in the viewing position (or even changes in the position of objects within the virtual environment), however, this will require some further rendering (or at least coordinate transformation) at the display device 37 before the frame can be displayed. Therefore, the time between the first frame and the second frame is preferably selected so that any change in the viewing position (and the movement of objects within the virtual environment) is minimized or zero.

[0128] The image may be generated based on further information such as the resolution of the display device 37 .

[0129] In an alternative, step S720 may be omitted, and the generated image may simply represent an oversized portion of the 2D or 3D area including the predicted display window position of the first frame. The degree of oversizing of the generated image may, for example, be a fixed predetermined oversizing, or may be based on the speed at which the viewing orientation or viewing range changes.

[0130] In another alternative, steps S710 and S720 may be omitted, and the generated image may represent a fixed portion of the 2D or 3D area, or the entire 2D or 3D area. For example, the image generator 31 may generate an image of the complete virtual environment at a first frame rate, and the display device 37 may display a frame including a portion of the virtual environment at a second frame rate that is higher than the first frame rate.

[0131] In step S740, the image generator 31 transmits the generated image and the predicted display window position of the first frame to the display device 37, so that the display device 37 can display the first frame.

[0132] The transmission of the generated image may involve compressing and / or encoding the image. For example, the image may be encoded using LCEVC or VC-6 encoding.

[0133] In step S750, the image generator 31 transmits the predicted display window position of the second frame to the display device 37 so that the display device 37 can display the second frame using the previously received generated image.

[0134] Alternatively, step S750 may be omitted. For example, if the display device 37 obtains feedback about the user's motion, the display device 37 may calculate the display window position for the second frame without further data from the image generator 31 .

[0135] Figure 8 is a flow chart schematically showing a method performed by a display device. The method corresponds to a method performed by an image generator Figure 7 Specifically, step S810 corresponds to step S740, and step S840 corresponds to step S750.

[0136] Reference Figure 8 In step S810, the display device 37 receives the generated image 3 and the predicted display window position of the first frame from the image source 31. The display device 37 may store the generated image in a memory.

[0137] In step S820, the display device 37 adjusts the generated image 3 to obtain an adjusted image 2a or 1a corresponding to the predicted display window position of the first frame. For example, the display device 37 can crop, translate or rotate the generated image based on the predicted display window position to obtain the adjusted image of the first frame.

[0138] In step S830 , the display device 37 displays the adjusted image 2 a or 1 a of the first frame.

[0139] In step S840, display device 37 receives the predicted display window position of the second frame from image source 31. Alternatively, display device 37 may calculate the display window position of the second frame (as described below) instead of receiving the predicted display window position of the second frame.

[0140] Then, although the generated image 3 received in step S810 is used again to display the adjusted image 2b or 1b of the second frame without another image to be received in step S840, steps S850 and S860 for the second frame are similar to steps S820 and S830 for the first frame.

[0141] Fig. 9 is a flow chart schematically illustrating another method performed by a display device. Fig. 9 Methods and Figure 8The difference is that the time warping is applied before displaying the frame. In other words, steps S910 and S920 correspond to steps S810 and S820 or steps S840 and S850.

[0142] Reference Fig. 9 At step S930, the display device 37 obtains the final display window position of the frame. The final display window position is different from the predicted display window position due to any changes in viewing position, viewing orientation, and / or viewing range that occurred between rendering the generated image at the image source 31 and displaying the frame at the display device 37 and were not anticipated in the predicted display window position.

[0143] In step S940, the display device 37 adjusts the first adjusted image 2a (obtained in step S920) to obtain a second adjusted image 1a corresponding to the final display window position. This is a similar process to step S920 and may involve cropping, rotating or translating the first adjusted image.

[0144] In step S950 , the display device 37 displays the second adjustment image 1 a (instead of displaying the first adjustment image 2 a ).

[0145] As Fig. 9 As an alternative to the method, the final display window position may be obtained before adjusting the generated image (step S930), and steps S920 and S940 may be combined into a single adjustment step for adjusting the generated image to obtain a second adjusted image corresponding to the final display window position.

[0146] As another alternative, the transmission of the predicted display window position may be omitted entirely. In other words, the image source 31 may perform Figure 7 , and the method described above, but without transmitting any predicted display position. In this case, the display device 37 may: receive a generated image from the image source (similar to step S910); obtain a final display window position (as in step S930); adjust the generated image to obtain a second adjusted image corresponding to the final display window position; and display the second adjusted image. This alternative is particularly relevant if the display window position is determined based on movement of the headset display device 37, in which case the image source 31 cannot effectively provide supplemental information about the display window position to the display device 37.

[0147] According to one example, a method for generating an extended reality field of view is provided for effectively encoding subtle motion in a video sequence, including: identifying an initial field of view to be presented to a user; generating an enlarged field of view relative to the initial field of view; generating a set of coordinates in the enlarged field of view, the coordinates identifying a starting position of a display window; instructing an enhancement encoder to encode the enlarged field of view and the set of coordinates. The enhancement encoder includes: one or more corresponding base encoders for implementing a base encoding layer for encoding a video signal; and an enhancement encoder for implementing an enhancement decoding layer, the enhancement decoder being configured to: encode the enhancement signal to generate one or more residual data layers, the one or more residual data layers being generated based on a comparison of data derived from the decoded video signal with data derived from the original input video signal,

[0148] According to another example, a method for generating an extended reality user display from a received field of view is provided, comprising: receiving a magnified field of view decoded by an enhanced decoder; receiving a set of coordinates decoded by an enhanced encoder; creating a display window in the magnified field of view using the set of coordinates; and presenting the display window to a user.

Claims

1. A system for displaying a sequence of image frames, the system comprising an image source and a display device, The image source is configured as: Generate images; obtaining a predicted display window position; and transmitting the generated image and the predicted display window position to the display device; and The display device is configured as follows: receiving the generated image and the predicted display window position from the image source; and The generated image is adjusted to obtain a first adjusted image corresponding to the predicted display window position.

2. The system according to claim 1, wherein the display device is further configured to: Get the final display window position; and The first adjustment image is adjusted to obtain a second adjustment image corresponding to the final display window position.

3. The system according to claim 2, wherein the display device is further configured to: The second adjusted image is displayed.

4. The system according to any one of claims 2 to 3, wherein the final display window position is the user's field of view in the 3D rendering space at the time when the image is displayed. 5 . The system of claim 2 , wherein adjusting the first adjustment image comprises cropping, translating, or rotating the first adjustment image.

6. A system according to any preceding claim, wherein the predicted display window position is a prediction of a user's field of view in a 3D rendering space at the time the image frame is displayed, wherein the prediction is made while the image frame is being rendered.

7. The system of any preceding claim, wherein adjusting the generated image comprises cropping, translating or rotating the generated image.

8. A system according to any preceding claim, wherein the predicted display window position is indicated using coordinate values.

9. A system according to any preceding claim, wherein the display device is configured to store a generated image of a previous frame, and the image source is configured to transmit the difference between the generated image of the current frame and the generated image of the previous frame.

10. A system according to any preceding claim, wherein: The image source comprises an encoder, and the display device comprises a decoder, The image source is configured as: encoding the generated image and the predicted display window position into a coded frame; and transmitting the encoded frame to the display device, and The display device is configured to: The encoded frame is received and decoded to obtain the generated image and the predicted display window position.

11. The system of claim 10, wherein the image encoder is an LCEVC encoder and the image decoder is an LCEVC decoder.

12. The system of any preceding claim, further comprising a network, wherein the image source is configured to stream the sequence of image frames to the display device over the network.

13. A transmission method for a system for displaying a sequence of image frames, wherein the method comprises: Obtaining a generated image from an image source; Get the predicted display window position; and The generated image and the predicted display window position are transmitted to a display device.

14. A method for encoding a system for displaying a sequence of image frames, wherein the method comprises: Obtaining a generated image from an image source; Get the predicted display window position; and The generated image and the predicted display window position are encoded into a coded frame.

15. A receiving method of a system for displaying a sequence of image frames, wherein the method comprises: receiving a generated image and a predicted display window position from an image source; and The generated image is adjusted to obtain a first adjusted image corresponding to the predicted display window position.

16. A decoding method for a system for displaying a sequence of image frames, wherein the method comprises: Decoding the coded frame to obtain the generated image and the predicted display window position; and Adjusting the generated image to obtain a first adjusted image corresponding to the predicted display window position; and The first adjusted image is outputted for use by a display device.

17. An apparatus configured to perform the method according to any one of claims 13 to 16.

18. A method for generating data suitable for constructing a series of image data frames by a VR display device, the method comprising: Determine an ultra-large field of view, wherein the ultra-large field of view includes: A first field of view corresponding to an expected field of view at the VR display device at the time of the first image data frame; and a second field of view corresponding to an expected field of view at the VR display device at a later second image data frame; An ultra-large image including the determined ultra-large field of view is generated.

19. A method for generating a frame sequence for display on a VR display device, optionally wherein the method is performed by the VR display device, the method comprising: Obtaining an ultra-large image with an ultra-large field of view, wherein the ultra-large field of view includes: a first field of view corresponding to a display time of a first frame; and a second field of view corresponding to a display time of a later second frame; obtaining first position data of the first frame, wherein the position data can be combined with the super large image to generate the first frame; combining the super large image with the first position data to generate the first frame for display on the VR display device at a first display time; obtaining position data of the second frame, wherein the position data can be combined with rendering of the super large image to generate the second frame; The rendering of the super large image is combined with the second position data to generate the second frame for display on the VR display device at a second display time.

20. The method of claim 19, wherein the method further comprises: Receiving the encoded rendering of the super large image; and The encoded rendering of the super large image is decoded.

21. The method of claim 20, wherein the encoded rendering of the super large image is a rendering of the super large image encoded by an LCEVC encoding method, and the method further comprises decoding the encoded rendering of the super large image comprises decoding according to an LCEVC decoding method.

22. A method for preprocessing data for encoding, the method comprising: generating an image for encoding, wherein rendering of the image is used on a VR display device to create a VR display for a viewer, Wherein the image includes a field of view: is larger than and includes the field of view of the viewer at the time of generation of the super-large image; is larger than and includes the field of view of the viewer at the display time of the generated image; and is greater than and includes the field of view of the viewer at the time of display of the other image; The generated image and the other image form an image sequence suitable for viewing by a viewer, and the other image is displayed later than the generated image in the sequence.

23. The method of claim 22, further comprising: Encoding the image, in particular using the LCEVC encoding method; Sending the encoded image to the VR display device via a wired or wireless connection; Position data is obtained, the position data being used to obtain a user's field of view, wherein the position data is indicative of head / eye position and / or movement of the viewer, wherein generating the image further comprises processing the position data in association with a time of the generation of the image.

24. A method, preferably performed by a VR viewing device, comprising: adjusting the super large image using the received or generated first position data to select a field of view to be included in the super large image at a first time, the first time being a time when the super large image is acquired or a time when the selected field of view will be displayed; and The rendering of the super-large image is updated at a later time by using the received or generated second position data to select a field of view for inclusion within the super-large image, wherein the selected field of view corresponds to a viewer's field of view at the time of the update or a predicted time when the updated image is displayed.

25. The method of claim 24, further comprising: The rendering of the very large image is generated by enhancing the very large image using enhancement data, in particular wherein the enhancement data is derived from an LCEVC enhancement layer.

Images