Virtual reality device, method and computer program

Abstract

A decoding device (110) receives data usable to generate data for a data signal representing a first quality level. The decoding device (110) receives enhancement data usable to generate data for a data signal representing a second, higher quality level based on the data for the data signal representing the first quality level. The decoding device (110) generates data for a target region of a data signal having a target quality level using selected portions of the received enhancement data. The selected portions are associated with the target region. The target quality level is higher than the first quality level. The decoding device (110) generates data for other regions of a data signal representing quality levels lower than the target quality level.

Description

[0001] The present application is a divisional application of the patent application entering Chinese national phase from a PCT international application (application date: July 20, 2017, application no: 201780055704.0, invention name: Decoding apparatus, method and computer program). TECHNICAL FIELD

[0002] The present application relates to a decoding apparatus, method and computer program. BACKGROUND

[0003] In some situations, it is desirable to display an image or video of a high level of quality, such as high resolution, to a viewer. This can be desirable in situations where the screen displaying the image or video is very close to the viewer. Otherwise, if the image or video quality is low, then the viewer can notice imperfections in the image or video. This can detract from the viewer's enjoyment and experience of the image or video.

[0004] If the viewer's attention is focused on a relatively high quality area of the image, then the viewer can not notice, or at least can not be distracted by, a relatively low quality area of the image. Foveated imaging is a digital image processing technique in which the amount of detail on an image varies based on where in the image the viewer is looking. The amount of detail is greatest in the area of the image that the viewer is looking at.

[0005] There are problems and challenges in allowing a viewer to perceive an image or video in this way when faced with various limiting factors. SUMMARY

[0006] According to a first aspect of the application, there is provided a decoding apparatus configured to: receive data, the data being usable to generate a data signal representing a first quality level; receive enhancement data, the enhancement data being usable to generate data for a data signal representing a second quality level based on a representation of the data signal of the first quality level, the second quality level being higher than the first quality level; generate data using a selected portion of the received enhancement data, the generated data being for a target region of the data signal having a target quality level, the selected portion of the received enhancement data being associated with the target region of the data signal, the target quality level being higher than the first quality level; and generate data for other regions of the data signal representing a quality level lower than the target quality level.

[0007] According to a second aspect of the application, there is provided a method comprising, at a decoding device: receiving data, the data being usable to generate data for a data signal representing a first quality level; receiving enhancement data, the enhancement data being usable to generate data for a data signal representing a second quality level based on the representation of the data signal of the first quality level, the second quality level being higher than the first quality level; using a selected portion of the received enhancement data, the selected portion of the received enhancement data being associated with a target region of the data signal, to generate data for the target region of the data signal representing a target quality level, the target quality level being higher than the first quality level; and generating data for other regions of the data signal representing quality levels lower than the target quality level.

[0008] According to a third aspect of the application, there is provided a computer program comprising instructions which, when executed on a decoding device, cause the decoding device to perform a method comprising: receiving data, the data being usable to generate data for a data signal representing a first quality level; receiving enhancement data, the enhancement data being usable to generate data for a data signal representing a second quality level based on the representation of the data signal of the first quality level, the second quality level being higher than the first quality level; using a selected portion of the received enhancement data, the selected portion of the received enhancement data being associated with a target region of the data signal, to generate data for the target region of the data signal representing a target quality level, the target quality level being higher than the first quality level; and generating data for other regions of the data signal representing quality levels lower than the target quality level.

[0009] Other features and advantages will become apparent from the following description of preferred embodiments, taken in conjunction with the accompanying drawings. The preferred embodiments merely illustrate examples of the application. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 A block schematic diagram illustrating an example of a data signal processing system according to an embodiment of the application is shown.

[0011] Figure 2 A schematic diagram illustrating an example of a hierarchical data signal processing structure according to an embodiment of the application is shown;

[0012] Figure 3 A schematic diagram illustrating another example of a hierarchical data signal processing structure according to an embodiment of the application is shown;

[0013] Figure 4 A schematic diagram illustrating an example of an image according to an embodiment of the application is shown;

[0014] Figure 5 A schematic diagram illustrating another example of an image according to an embodiment of the application is shown;

[0015] Figure 6 Fig. 1 shows a schematic diagram illustrating an example of data available in a data signal processing system according to an embodiment of the application;

[0016] Figure 7 Fig. 2 shows a schematic diagram illustrating another example of data available in a data signal processing system according to an embodiment of the application;

[0017] Figure 8 Fig. 3 shows a schematic diagram illustrating another example of data available in a data signal processing system according to an embodiment of the application;

[0018] Figure 9 Fig. 4 shows a schematic diagram illustrating another example of data available in a data signal processing system according to an embodiment of the application;

[0019] Figure 10 Fig. 5 shows a schematic diagram illustrating an example of a plurality of images according to an embodiment of the application; and

[0020] Figure 11 Fig. 6 shows a block schematic diagram illustrating an example of an apparatus according to an embodiment of the application. DETAILED DESCRIPTION

[0021] Reference Figure 1 Fig. 1 shows an example of a data signal processing system 100. The data signal processing system 100 is configured to process a data signal. A data signal is a signal carrying and / or representing data. For convenience and brevity, in the specific examples described in more detail below, the data signal is typically image data and / or video data, it being understood that the type of data signal can be different. For example, the data signal can be an audio signal. An audio signal carries audio data. The data signal can be an ultrasound signal associated with a medical ultrasound device. Other examples of data signals include, but are not limited to, multi-view video signals (e.g. three-dimensional video), volumetric signals used in, for example, medical, scientific or holographic imaging, or other multi-dimensional signals.

[0022] The data signal processing system 100 comprises a first apparatus 102 and a second apparatus 104. The first apparatus 102 and the second apparatus 104 can have a client-server relationship, with the first apparatus 102 performing the functions of a server device and the second apparatus 104 performing the functions of a client device. The data signal processing system 100 can comprise at least one additional apparatus. The first apparatus 102 and / or the second apparatus 104 can comprise one or more components, which can be implemented in hardware and / or software. In the data signal processing system 100, one or more components can be located at the same place, or can be located at different places remote from each other in a distributed arrangement.

[0023] In some examples, the second device 104 comprises or is comprised in a virtual reality device. Examples of virtual reality devices include, but are not limited to, virtual reality headsets and virtual reality glasses. Virtual reality is also known as augmented reality or immersive multimedia. Virtual reality is a computer technology that generates, augments or replicates an environment and simulates a user's physical presence in this environment.

[0024] In some examples, the second device 104 comprises or is comprised in a medical imaging device. Medical imaging is used to generate visual representations of the whole body or a part of the body to facilitate clinical analysis and treatment, and can include generating images of specific organs or cells of the body. Examples of medical imaging devices include, but are not limited to, ultrasound devices, magnetic resonance imaging (MRI) devices, X-ray computed tomography (CT) devices, positron emission tomography (PET) devices and functional near-infrared spectroscopy (FNIR) devices.

[0025] In some examples, the second device 104 comprises or is comprised in a machine vision device. Machine vision devices allow machines to obtain visual information related to the environment surrounding the machine. One example of a machine vision device is a smart camera, another example of a machine vision device is a robotic device.

[0026] In some examples, the second device 104 comprises or is comprised in a mobile communication device. Mobile communication devices are devices for mobile communication, e.g. using cellular and / or wireless networks. Examples of mobile communication devices include, but are not limited to, smartphones and tablet computing devices.

[0027] The first and / or second device 102, 104 can be powered by an internal battery. The internal battery can be a rechargeable battery. In these cases, considerations for the use of the battery can be relevant. Alternatively or additionally, the first and / or second device 102, 104 can be powered by an external power source.

[0028] The first device 102 is communicatively coupled to the second device 104 by one or more data communication networks 106. Examples of data communication networks 106 include, but are not limited to, the Internet, a local area network (LAN) and a wide area network (WAN). The first and / or second device 102, 104 can have wired and / or wireless connections to the data communication network 106.

[0029] The first device 102 comprises an encoding device 108. The encoding device 108 is configured to encode a data signal, e.g. image data and / or video data. In addition to encoding a data signal, the encoding device 108 can perform one or more other functions. The encoding device 108 can be implemented in various different ways. For example, the encoding device 108 can be implemented as hardware and / or software.

[0030] The second device 104 includes a decoding apparatus 110. The decoding apparatus 110 is configured to decode a data signal, such as image data and / or video data. In addition to decoding the data signal, the decoding apparatus 110 can perform one or more other functions. The decoding apparatus 110 can be implemented in a variety of different forms. For example, the decoding apparatus 110 can be implemented as hardware and / or software.

[0031] The encoding apparatus 108 encodes a data signal and transmits the encoded data signal to the decoding apparatus 110 over the data communication network 106. The decoding apparatus 110 decodes the received encoded data signal and generates a decoded data signal. The decoding apparatus 110 can output the decoded data signal or output data derived using the decoded data signal. For example, the decoding apparatus 110 can output this data for display on one or more display devices associated with the second device 104. The one or more display devices can form part of the second device 104 or can otherwise be associated with the second device 104. Examples of display devices include, but are not limited to, a head-mounted display (HMD), an optical head-mounted display (OHMD), smart glasses, a virtual retinal display (VRD), a mobile phone or tablet display, a computer screen, a video monitor, an oscilloscope screen, etc.

[0032] The decoding apparatus 110 can output data that allows an image having a plurality of different quality levels to be displayed to a viewer. The image can have one or more relatively high quality regions and one or more relatively low quality regions. The image can have one or more medium quality regions. The relatively high quality regions can correspond to regions of the image that the viewer is viewing or is likely to be viewing. The relatively low quality regions can be regions of the image that the viewer is not viewing or is unlikely to be viewing.

[0033] There are a variety of ways in which it can be determined which region or regions of an image a viewer is viewing or is likely to be viewing. For example, the viewer can manually indicate one or more regions of interest using, for example, a pointing device. An eye tracker can be used to determine one or more points of view of the viewer. The point of view can be determined based on the center of the retina of the viewer's eye. The first and / or second devices 102, 104 can be able to determine regions of an image that are likely to be of interest. For example, the first and / or second devices 102, 104 can be able to predict regions of interest based on historical regions of interest and / or based on knowledge of the data signal.

[0034] In one possible arrangement of the data signal processing system 100, the decoding device is configured to provide feedback to the encoding device in respect of a region of interest in the image. For example, the region of interest in the image can correspond to the field of view of the viewer and / or one or more viewing points of the viewer. The encoding device can then generate and transmit to the decoding device an encoded version of the image in which the region of interest has a relatively high quality level and other regions of the image have one or more lower quality levels. The decoding device can decode the received encoded data and output the decoded data so that this image with the characteristics set by the encoding device can be displayed to the viewer. In this case, the size of the encoded data transmitted by the encoding device to the decoding device can be less than the size of a corresponding version of the image in which all of the image is at the same high quality level. This means that less data can be transmitted from the encoding device to the decoding device over the data communications network.

[0035] While it can be desirable in some circumstances to reduce the amount of data transmitted over the data communications network, the above possible arrangement involves the decoding device feeding back information to the encoding device in respect of the region of interest to allow the encoding device to encode the various regions of the image at the required quality levels. This can involve, for example, establishing and maintaining a dedicated feedback channel and relies on a data connection between the encoding device and the decoding device in order to work. In addition, when the decoding device provides feedback information to the encoding device, the encoding device encodes the image based on the feedback, the encoding device transmits the encoded image to the decoding device and the decoding device decodes the image for display to the viewer, the feedback of data in the above manner can increase the time required to display the image. In some circumstances, for example where substantially real-time image processing is required, this delay can be unacceptable or undesirable. This can be relevant in respect of virtual reality applications where a noticeable delay or lag can reduce the user experience.

[0036] Furthermore, in the above possible arrangement, the decoding device has limited control over which regions of the image are to be displayed at different quality levels. When the decoding device receives the encoded image, the quality levels of the various regions of the image have already been set by the encoding device based on feedback from the decoding device, limiting the extent to which the encoding device has control over how the image is displayed. This can impact the decoding device and / or the second device, for example so that the decoding device and / or the second device have limited capacity, battery life, processing power, etc.

[0037] In contrast, as described herein, the decoding device 110 is to receive data usable to generate data representative of a data signal, e.g., an image data signal and / or a video data signal, at a first quality level. The data used to represent the data signal is data usable to represent the data signal. The decoding device 110 is to receive enhancement data usable to generate data representative of the data signal at a second or higher quality level based on the representation of the data signal at the first quality level.

[0038] The decoding device 110 is to generate data using the selected portion of the enhancement data, the data being used to represent a target region of the data signal at a target quality level. The target region of the data signal can correspond to a location at which a viewer is or can be viewing.

[0039] Only some, but not all, of the enhancement data is selected. The selected portion of the enhancement data is associated with the target region of the data signal. The decoding device 110 can select the portion of the enhancement data itself and / or can determine the selected portion of the enhancement data based on receiving data including an identification of the selected portion of the enhancement data. The data identifying the selected portion of the enhancement data can be received from one or more other entities. The one or more other entities can include, for example, the encoding device 108. The decoding device 110 can determine the selected portion of the enhancement data to use based on the received data.

[0040] The target quality level is higher than the first quality level. The target quality level can be the same as the second quality level. The target quality level can also be between the first quality level and the second quality level.

[0041] The decoding device 110 can identify the target region of the data signal itself and / or can determine the target region of the data signal based on receiving data including an identification of the target region. The data identifying the target region can be received from one or more other entities. The one or more other entities can include, for example, the encoding device 108.

[0042] The decoding device 110 is to generate data representative of other regions of the data signal, the quality level of the data signal being lower than the target quality level. In the case that the data signal is an image data signal and / or a video data signal, the other regions can correspond to regions of the image and / or video that are not being viewed or are not likely to be viewed by a viewer.

[0043] The quality level associated with the other regions can be the first quality level. Alternatively, the quality level associated with the other regions can be between the first quality level and the target quality level.

[0044] In some examples, where the quality level associated with the other region is between the first quality level and the target quality level, the decoding device 110 is to use another portion of the selected enhancement data to generate data representative of the other region. The decoding device 110 can select the other portion of the enhancement data itself and / or can determine the selected other portion of the enhancement data based on received data including an identification of the selected other portion of the enhancement data.

[0045] In this way, the decoding device 110 receives a fully encoded data signal; for example, from the encoding device 108. The data signal is fully encoded in that the decoding device 110 can use the fully encoded data signal to generate a version of the data signal in which all regions have a high quality level. However, instead, the decoding device 110 decodes a portion of the fully encoded data signal at a first quality level and another portion of the fully encoded data signal at a second or higher quality level. In some examples, the decoding device 110 generates data representative of the entire data signal at the first quality level and data representative of only the regions of the data signal at the higher quality level. The decoding device 110 can use the portion of the data to represent the entire data signal at the first quality level to generate the data representative of only the regions of the data signal at the higher quality level.

[0046] These examples differ from the aforementioned possible arrangement in which the encoding device does not provide a fully encoded image and / or video data signal, but instead selects which portions of the image and / or video stream to set to different quality levels based on feedback from the decoding device. Instead, in the examples described here, the decoding device 110 has more control over which regions of the image and / or video to decode to a particular quality level. Rather than feeding such information back to the encoding device 108 over the data communications network 106, the decoding device 110 is able to use the feedback information to determine the target region and the other region based on the position in the image and / or video that the viewer is watching. Such feedback information can include data identifying the target region and / or the other region and / or can include data that can be used by the decoding device 110 to determine the target region and / or the other region itself.

[0047] Furthermore, the decoding device 110 can selectively optimise resources for decoding. This can save battery usage. Additionally, the decoding device 110 can not need to provide any feedback to the encoding device 108. Furthermore, if the encoding device 108 is to transmit a fully encoded data signal, there is no need to modify any functionality at the encoding device 108.

[0048] One approach that can be traded off against the encoding device 108 sending a fully encoded data signal is that more data can be sent between the encoding device 108 and the decoding device 110 over the data communication network 106 than would be the case if the encoding device 108 selectively encoded the data signal at a predetermined different quality level based on feedback from the decoding device 110.

[0049] Referring to Figure 2 a schematic diagram illustrating an example of a layered data signal processing structure 200 is shown.

[0050] The layered data signal processing structure 200 represents a plurality of different quality levels. The quality levels can relate to different data quality levels associated with the data signal. One factor that can be used to determine the quality of image and / or video data is resolution. Higher resolution corresponds to higher quality levels. Resolution can be spatial and / or temporal. Other factors that can be used to determine the quality of image and / or video data include, but are not limited to, quantization level of the data, frequency filtering level of the data, peak signal-to-noise ratio of the data, structural similarity (SSIM) index, etc.

[0051] In this example, the layered data signal processing structure 200 has three different layers (or "levels"), namely a first layer 202, a second layer 204 and a third layer 206. However, the layered data signal processing structure can have a different number of layers. The first layer 202 can be considered a base layer as it represents a basic quality level, and the second and third layers 204, 206 can be considered enhancement layers as they represent enhancements in quality relative to the base layer. The first layer 202 corresponds to a first quality level L0Q1. The second layer 204 corresponds to a second quality level L0Q2. The second quality level L0Q2 is higher than the first quality level L0Q1. The third layer 206 corresponds to a third quality level L0Q2. The third quality level L0Q3 is higher than the second quality level L0Q2. The second quality level L0Q2 is between (or "intermediate") the first quality level L0Q1 and the third quality level L0Q3.

[0052] In some examples, the layered data signal processing structure 200 represents a plurality of different levels of quality of video data. For example, the first layer 202 can correspond to standard definition (SD) quality video, the second layer 204 can correspond to high definition (HD) quality video, and the third layer 206 can correspond to ultra high definition (UHD) video.

[0053] In this example, each of the layers 202, 204, 206 is associated with respective enhancement data. The enhancement data can be used to generate data, such as image and / or video data, having the quality levels associated with the respective layers, as will be described in more detail below.

[0054] Referring to Figure 3The diagram shows an example of a hierarchical data signal processing structure 300.

[0055] Figure 3 The layered data signal processing structure 300 shown is... Figure 2 The hierarchical data signal processing structure 200 shown is similar, comprising three layers 302, 304, and 306. In this example, each of layers 302, 304, and 306 comprises a set of sublayers (or “sub-levels”). In this specific example, each layer comprises four sublayers.

[0056] The first layer 302 is associated with a first quality level LOQ1. Each sublayer of the first layer 302 is associated with a corresponding quality level. The first sublayer of the first layer 302 is associated with quality level LOQ11, the second sublayer of the first layer 302 is associated with quality level LOQ12, the third sublayer of the first layer 302 is associated with quality level LOQ13, and the fourth sublayer of the first layer 302 is associated with quality level LOQ14. Similarly, the second layer 304 is associated with a second quality level LOQ2, the third layer 306 is associated with a third quality level LOQ3, and the sublayers of the second layer 304 and the third layer 306 are associated with corresponding incremental quality levels. In the hierarchical data signal processing structure 300, the quality levels associated with higher layers and sublayers are higher than the quality levels associated with lower layers and sublayers. Therefore, in the hierarchical data signal processing structure 300, the quality levels increase from bottom to top.

[0057] Some or all of layers 302, 304, and 306 may have multiple sublayers in addition to the four sublayers. Some layers 302, 304, and 306 may have a different number of sublayers than each other. Some layers 302, 304, and 306 may not have any sublayers.

[0058] In this example, each sublayer is associated with corresponding augmented data. The augmented data can be used to generate data with a quality level associated with the corresponding sublayer, such as image and / or video data, which will be described in more detail below.

[0059] Accordingly, the layered data signal processing structure 300 comprises a first layer having a first set of sub-layers and a second layer having a second set of sub-layers. Each sub-layer is associated with a respective quality level and is associated with respective enhancement data. Using a layered data signal processing structure, such as the layered data signal processing structure 300, some devices can be permitted to reconstruct at a particular layer (e.g. LOQ3) but using only the first two sub-layers of that layer, LOQ31 and LOQ32. This can be to improve efficiency, save battery or limit capacity. Using only some sub-layers can be considered partial reconstruction. Other devices can use all four sub-layers in LOQ3, i.e. LOQ31, LOQ32, LOQ33 and LOQ34, and fully reconstruct the signal. Using all sub-layers can be considered full reconstruction. The reader is directed to UK patent application number GB1603727.7 for a more detailed description of the layered arrangement. The entire contents of GB1603727.7 are incorporated herein by reference.

[0060] Referring to Figure 4 , an example of an image 400 is shown. The image 400 can be obtained, for example, from video data. The image can depict a scene. For example, the image can depict a concert.

[0061] The image 400 comprises a target region 402. The target region 402 is a region of interest in relation to the image 400. The target region 402 can correspond to a region of the image 400 that a viewer is currently viewing and / or is likely to view. The target region 402 is a region of interest in relation to the image 400 because the viewer is likely to be particularly sensitive to the quality level of the image in the target region 402. In the example of the image 400 depicting a concert, the target region 402 can correspond to a region of the image 400 that includes a singer. Although the target region 402 is depicted as a rectangle in Figure 4 , it can take a different form. For example, it can correspond to an outline of an item of interest in the image, such as an outline of a singer.

[0062] The image can comprise one or more target regions. For example, the image can comprise a plurality of target regions associated with respective different viewers, where the plurality of viewers are viewing the image simultaneously.

[0063] The image 400 comprises other regions 404. The other regions 404 can correspond to regions of the image 400 that are not currently being viewed and / or are unlikely to be viewed by the viewer. The viewer can be less sensitive to the quality level of the image in the other regions 404. In the example of the image 400 depicting a concert, the other regions 404 can correspond to regions of the image 400 that include an audience. The other regions 404 surround the target region 402. One image can comprise one or more such other regions 404.

[0064] Referring toFigure 5 An example of an image 500 is shown.

[0065] The image 500 comprises a target region 502, a first other region 504 and a second other region 506. In this example, the target region 502 is associated with a high quality level, the first other region 504 is associated with a low level of quality and the second other region 506 is associated with an intermediate quality level. The second other region 506 surrounds the target region 502. The first other region 504 partially surrounds the target region 502 and the second other region 506.

[0066] Referring back to Figure 6 An example of data 600 associated with the data signal processing system 100 is shown. In this example, the data 600 is obtained and / or generated by the encoding device 108. In this example, the data 600 is related to data for representing images of different quality levels.

[0067] The data 600 comprises first data 602. In this example, the first data 602 corresponds to data for representing images of different quality levels (from a highest quality level LOQ3 to a lowest quality level LOQ1). For convenience, example values have been included in the data for representing images of different levels in LOQ3, LOQ2 and LOQ1. It will be appreciated that other values can be used. For example, the values in the data for representing images can correspond to pixel values. Figure 6

[0068] In this example, the data for representing images of LOQ3 in the first data 602 corresponds to an original version of the image. That is, in this example, the data for representing images of LOQ3 in the first data 602 is the same as the original version of the image. The resolution of the data for representing images of LOQ3 is 4x4. It will be appreciated that in practice, the resolution of the data for representing images can be much higher than 4x4. For example, the resolution can be tens, hundreds or thousands of image elements by tens, hundreds or thousands of image elements. For example, the resolution can be 1920x540.

[0069] The data for representing images of LOQ3 in the first data 602 is downsampled to generate data for representing images of LOQ2 in the first data 602. The resolution of the data for representing images of LOQ2 in the first data 602 is 2x2. In this example, the downsample operation calculates an average of four adjacent values in the data for representing images of LOQ3 in the first data 602 and rounds the average to the nearest integer. For example, with reference to Figure 6 ​The lower left value in LOQ3 (i.e., 7, 3, 2, 2) has been averaged to produce the lower left value in LOQ2 (i.e., 4), and the values in LOQ2 (i.e., 3, 2, 4, 1) have been averaged to produce the values in LOQ1 (i.e., 3). It will be appreciated, however, that other techniques can be used to generate the lower quality representations of the images. In particular, in some examples, the data used to represent the images of the lower quality levels has the same resolution as the data used to represent the images of the higher quality levels. In such examples, the difference in quality levels can relate to the amount of enhancement or correction applied to the data used to represent the images.

[0070] The data in the first data 602 used to represent the image of LOQ2 is downsampled to generate the data in the first data 602 used to represent the image of LOQ1. In this example, the downsample operation calculates the average of the four values in the data in the first data 602 used to represent the image of LOQ2 and rounds the average to the nearest integer. The resolution of the data in the first data 602 used to represent the image of LOQ1 is 1 x 1. Although in this example the resolution of the data in the first data 602 used to represent the image of LOQ1 is 1 x 1, in practice the downsample operation can not reduce the resolution to 1 x 1. For example, the resolution of the data in the first data 602 used to represent the image of LOQ1 can be tens, hundreds or thousands of image elements by tens, hundreds or thousands of image elements.

[0071] The data 600 also includes second data 604. In this example, the second data 604 corresponds to different representations of the images of the quality levels LOQ1, LOQ2 and LOQ3 (from the lowest quality level LOQ1 to the highest quality level LOQ3).

[0072] In this example, the data in the second data 604 used to represent the image of LOQ1 is the same as the data in the first data 602 used to represent the image of LOQ1. In other examples, the data in the second data 604 used to represent the image of LOQ1 is different to the data in the first data 602 used to represent the image of LOQ1. The resolution of the data in the second data 604 used to represent the image of LOQ1 is 1 x 1.

[0073] The data in the second data 604 used to represent the image of LOQ1 is upsampled to generate the data in the second data 604 used to represent the image of LOQ2. In this example, the upsample operation comprises a nearest neighbour operation, such as nearest neighbour interpolation. It will be appreciated that other techniques can be used to generate the higher quality representations of the images. The resolution of the data in the second data 604 used to represent the image of LOQ2 is 2 x 2.

[0074] The data in the second data 604 for representing the image at LOQ2 is upsampled to generate the data in the second data 604 for representing the image at LOQ3. In this example, the upsample operation comprises a nearest neighbor operation, such as nearest neighbor interpolation. The resolution of the data in the second data 604 for representing the image at LOQ3 is 4x4.

[0075] The upsample and downsample operations result in a difference between the values of the data in the first data 602 and the second data 604 for representing the image at a given quality level. This is because the upsample and downsample operations are asymmetric. For example, the top right value in the data in the first data 602 for representing the image at LOQ2 is "2", whereas the top right value in the data in the second data 604 for representing the image at LOQ2 is "3".

[0076] In this example, the enhancement data 606 is generated by subtracting the values in the data in the second data 604 for representing the image at a given quality level from the corresponding values in the data in the first data 602 for representing the image at the given quality level. For example, the top right value in the enhancement data 606 for LOQ2 is "-1", which is obtained by subtracting the top right value in the data in the second data 604 for representing the image at LOQ2, which is "3", from the top right value in the data in the first data 602 for representing the image at LOQ2, which is "2". In other examples, the enhancement data 606 is generated based on another relationship between the values in the data in the second data 604 for representing the image at a given quality level and the corresponding values in the data in the first data 602 for representing the image at the given quality level.

[0077] In this example, the encoding device 108 transmits the data in the second data 604 for representing the image at LOQ1 and all of the enhancement data 606 to the decoding device 110. The decoding device 110 is then able to recover all of the representations of the image in the first data 602.

[0078] Referring to Figure 7 FIG. 7 shows example data 700 associated with the data signal processing system 100. In this example, the data 700 is obtained by the decoding device 110. In this example, the data 700 is for representing images at different quality levels.

[0079] The example data 700 comprises second data 702, enhancement data 704 and first data 706. The second data 702 is the same as the second data 604 described above with reference to Figure 6 The enhancement data 704 is the same as the enhancement data 606 described above with reference to Figure 6 The first data 706 is the same as the first data 602 described above with reference to Figure 6 ​

[0080] from Figure 7 As can be seen from the above reference, Figure 6 The inverse operation performed by the described encoding device 108 involves using the data in the second data 702 representing the image of LOQ1 and the enhancement data 704 to recover the first data 706. Specifically, the decoding device 110 uses the data in the second data 702 representing the image of LOQ1 and the enhancement data 704 of LOQ1 to generate the data in the first data 706 representing the image of LOQ1. Then, the decoding device 110 upsamples the data in the first data 706 representing the image of LOQ1 to generate the data in the second data 702 representing the image of LOQ2, and uses the enhancement data 704 of LOQ2 and the data in the second data 702 representing the image of LOQ2 to generate the data in the first data 706 representing the image of LOQ3. Then, the decoding device 110 upsamples the data in the first data 706 used to represent the image of LOQ3 to generate the data in the second data 702 used to represent the image of LOQ3, and uses the enhanced data 704 of LOQ3 and the data in the second data 702 used to represent the image of LOQ3 to generate the data in the first data 706 used to represent the image of LOQ3. The data in the first data 706 used to represent the image of LOQ3 is consistent with the above reference. Figure 6 The data used to represent the image of LOQ3 in the first data 602 described is the same.

[0081] In this way, the encoding device 108 fully encodes the original image, enabling the decoding device 110 to recover the original image.

[0082] Therefore, the decoding device 110 receives data that can be used to generate data representing an image of a first quality level (e.g., LOQ1 or LOQ2). The decoding device 110 also receives enhancement data based on the data representing the image of the first quality level (LOQ1 or LOQ2), which can be used to generate data representing an image of a second quality level (e.g., LOQ2 or LOQ3).

[0083] refer to Figure 8 Example data 800 associated with the data signal processing system 100 is shown. In this example, data 800 is obtained by the decoding device 110. In this example, data 800 is associated with data used to represent images of different quality levels.

[0084] Example data 800 includes second data 802, enhanced data 804, and first data 806.

[0085] In this example, decoding device 110 receives data from encoding device 108 as described above.Figure 7 The data described is the same data. However, the decoding device 110 uses only a portion of the enhancement data 804 corresponding to the target region of the image. In this example, the portion of the data used to represent the image corresponding to the target region and the portion of the enhancement data 804 is shown using bolded border lines.

[0086] In this example, the decoding device 110 uses the data in the second data 802 used to represent the image of LOQ1 and the enhancement data 804 for LOQ1 to generate the data in the first data 806 used to represent the image of LOQ1. The decoding device 110 then upsamples the data in the first data 806 used to represent the image of LOQ1 to generate the data in the second data 802 used to represent the image of LOQ2. Since only a portion of the enhancement data 804 for LOQ2 is associated with the region of interest in the data in the second data 802 used to represent the image of LOQ2, this portion of the enhancement data 804 for LOQ2 is selected and used with the data in the second data 802 used to represent the image of LOQ2 to generate the data in the first data 806 used to represent the image of LOQ2. Since the decoding device 110 used only the selected portion of the enhancement data 804 for LOQ2 to generate the data in the first data 806 used to represent the image of LOQ2, only one element in the data in the second data 802 used to represent the image of LOQ2 is enhanced using the selected portion of the enhancement data 804 for LOQ2. In particular, the value of the bottom right element of the data in the first data 806 used to represent the image of LOQ2 is “0” and the enhanced (or “corrected”) value of this element is “1”. In this example, the decoding device 110 only upsamples the region of interest in the data in the first data 806 used to represent the image of LOQ2 to generate the data in the second data 802 used to represent the image of LOQ3 for LOQ3. In this example, the decoding device 110 does not upsample any region of the data in the first data 806 used to represent the image of LOQ2 other than the region of interest. In other examples, the decoding device 110 can upsample one or more regions of the data in the first data 806 used to represent the image of LOQ2 other than the region of interest.

[0087] Since only a portion of the enhancement data 804 for LOQ3 is associated with the region of interest in the data in the second data 802 used to represent the image of LOQ3, this portion of the enhancement data 804 for LOQ3 is selected and used with the values in the region of interest in the data in the second data 802 used to represent the image of LOQ3 to generate the values in the region of interest in the first data 806 used to represent the image of LOQ3. The values in the region of interest in the first data 806 used to represent the image of LOQ3 are the same as the values in the region of interest in the second data 802 used to represent the image of LOQ3 referenced above.Figure 6 The values in the corresponding region in the data in the first data 602 used to represent the image of LOQ3 are the same.

[0088] In this example, the region of interest in the data in the first data 806 used to represent the image of LOQ3 is the data used to represent the target region of the image of the target quality level. In this example, the region of the data in the first data 806 used to represent the image of LOQ2 other than the region of interest is the data used to represent other regions of the image of a quality level lower than the target quality level.

[0089] Thus, the decoding device 110 receives data that can be used to generate data used to represent a data signal of a first quality level (e.g., LOQ1). The decoding device 110 also receives enhancement data that can be used to generate data used to represent a data signal of a second quality level (e.g., LOQ3) based on the representation of the data signal of the first quality level (LOQ1). The decoding device 110 uses selected portions of the received enhancement data to generate data used to represent a target region of a data signal of a target quality level (e.g., LOQ2 or LOQ3). The selected portions of the received enhancement data are associated with the target region of the data signal. The decoding device 110 generates data used to represent other regions of the data signal of a quality level lower than the target quality level (e.g., LOQ1 or LOQ2).

[0090] Since, in this example, the resolution of the data used to represent the image of LOQ3 is different from the resolution of the data used to represent the image of LOQ2, some or all of the data output by the decoding device 110 can need to be upsampled and / or otherwise modified for display to a viewer.

[0091] For ease of explanation, assume that the display resolution is 4x4, then the generated values (including the four values) in the region of interest in the data in the first data 806 used to represent the image of LOQ3 can be used for display. In addition, the region of the data in the first data 806 used to represent the image of LOQ2 (other than the region of interest) including the three values can be upsampled to generate twelve values so that a total of sixteen values are available for display. In practice, the display resolution can be much higher than 4x4 and can be of a different size.

[0092] Assuming the display resolution is 8x8, the region of interest (comprising four values) in the data in the first data 806 representing the image for LOQ3 can be upsampled once to produce sixteen values. The region of the data in the first data 806 representing the image for LOQ2 comprising four values (other than the region of interest) can be upsampled twice to generate 48 values, thus making a total of 64 values available for display. In the same way, the display resolution can in practice be much higher than 8x8, and can be of a different size.

[0093] As mentioned above, in practice the display resolution can be much higher than the 4x4 and 8x8 resolutions provided in the above examples. For example, the display resolution can be 1920x540. Furthermore, although examples are provided above in which the display resolutions have the same size (e.g. 4x4 and 8x8), the display resolutions can have unequal sizes, and thus can be non-square, e.g. the display resolution for a widescreen display.

[0094] Referring to Figure 9 , an example data 900 associated with the data signal processing system 100 is shown. In this example, the data 900 is obtained by the decoding device 110. In this example, the data 900 relates to data representing images at different quality levels.

[0095] The example data 900 comprises second data 902, enhancement data 904 and first data 906.

[0096] In this example, the decoding device 110 receives the same data from the encoding device 108 as the data described above with reference to Figure 7 and 8 However, the decoding device 110 uses only a portion of the enhancement data 904 corresponding to the target region of the image.

[0097] In this particular example, the decoding device 110 does not use all of the enhancement data 904 for LOQ3, the enhancement data 904 for LOQ3 can be used to enhance a region of interest of the data in the second data 902 used to represent the image for LOQ3. In this example, the portion of the enhancement data for LOQ3 selected by the decoding device 110 is associated with a sub-level of LOQ3, such as sub-level LOQ33. In particular, the decoding device 110 does not select enhancement data that can be used to enhance the value "1" in the lower left corner of the data in the second data 902 used to represent the image for LOQ3. This is true even though such enhancement data is available to the decoding device 110, and such enhancement data is within the portion of the enhancement data associated with the region of interest of the data in the second data 902 used to represent the image for LOQ3. Based on the value of such enhancement data, the decoding device 110 can select which enhancement data associated with the region of interest to use. For example, enhancement data having a value of zero or close to zero can not be selected because such enhancement data can not be as important as enhancement data having a value that differs more from zero for changing the data in the second data 902 used to represent the image for LOQ3.

[0098] Although in this example, the accuracy of the values in the region of interest in the data in the first data 906 used to represent the image for LOQ3 is lower than the accuracy of the values in the region of interest in the data in the first data 806 used to represent the image for LOQ3 shown in FIG. 8, the decoding device 110 weighs such accuracy against one or more other considerations. For example, the decoding device 110 can sacrifice accuracy due to entering a power saving mode. Figure 8

[0099] As such, the layers in the hierarchical data signal processing structure 200, 300 can be subdivided into a plurality of sub-levels, each sub-level containing data that allows for the enhancement or correction of an increased number of data signal elements using enhancement data associated with the sub-level.

[0100] The decoding device 110 can be configured to use some or all of the enhancement data associated with a sub-level to generate data used to represent a target region of a target quality level based on one or more operational modes of the decoding device. The decoding device 110 and / or the second device 104 can have several different operational modes. In some modes, such as a power saving mode, only selected layers and / or sub-levels are decoded and used.

[0101] The decoding device 110 can not use a portion of the enhancement data associated with at least one sub-level in a first operational mode of the decoding device 110. The first operational mode can be a power saving mode. For example, all sub-levels of a lower quality level can be used and none of the sub-levels of a higher quality level can be used. Other combinations of sub-levels of different levels can be used. ​

[0102] The decoding device 110 can be configured to use a portion of the enhancement data that is associated with all sub-layers in the second operating mode of the decoding device 110. The decoding device 110 can be configured to use a portion of the enhancement data that is associated with at least one sub-layer of the first and second layers in the third operating mode of the decoding device 110. For example, a single sub-layer can be used for each quality level.

[0103] In a specific example, if a first layer associated with a lower definition has three sub-layers and a second layer associated with a higher definition also has three sub-layers, the decoding device 110 can select to decode only the lowest sub-layer in the first and second layers in the power saving mode and to decode all three sub-layers of the first and second layers in the full power mode. Image processing in this manner provides lower image quality than fully decoding a fully encoded image, but can represent a defined and acceptable "trade-off", for example for energy saving gains.

[0104] Referring to Figure 10 , an example of a plurality of images 1000 is shown. The plurality of images 1000 includes a first image 1002 associated with a first time tl and a second image 1004 associated with a second time t2. The first and second images 1002, 1004 may, for example, be included in a video sequence.

[0105] The first image 1002 has a first target region 1006 and the second image 1004 has a second target region 1008. The position of the first target region 1006 in the first image 1002 is different to the position of the second target region 1008 in the second image 1004. Thus, in this example, the position of the target region is dynamic rather than static.

[0106] In some examples, the position of the target region changes based on the position in the image or sequence of images that the viewer is watching. For example, the position of the target region can change based on the field of view of the viewer and / or one or more gaze positions of the viewer in the image. The field of view of the viewer is the extent of the image or environment that is visible to the viewer at a given moment. The field of view can be a property of an external device, such as a virtual reality head mounted display (HMD) or a human eye. In the example of a human eye, the field of view can be defined as the angle of view in degrees during a steady fixation of the eyeball. The movement of the eyeball does not change the field of view. In contrast, the gaze or fixation position can depend on the movement of the viewer's eyeball around the image. The gaze position is the position in the image that the viewer is directing their gaze at a given moment. Thus, the gaze position is within the field of view of the viewer. The gaze position can be at the centre of the retina of the viewer's eye. In addition to the gaze position, the field of view also includes a peripheral field of view around the gaze position at a given moment.

[0107] The decoding device 110 can be configured to receive data associated with a field of view, the field of view being associated with a viewer; and the decoding device 110 uses the data associated with the field of view to identify a target region. The decoding device 110 can be configured to select the target region so as to be in the field of view. In this way, the decoding device 110 can align the target region with a location that the viewer is looking at or is likely to be looking at to allow a high quality region of the image to be displayed in the region that the viewer is looking at. The decoding device 110 can be configured to select at least a portion of other regions so as to be in the field of view. In the event that at least a portion of the other regions are in the field of view of the user, the decoding device 110 can generate data for representing a quality level of at least a portion of the other regions between a highest and a lowest quality level as it is visible to the viewer and having a lowest quality level can degrade the user experience. The decoding device 110 can be configured to receive data associated with one or more gaze locations, the one or more gaze locations being associated with a user; and the decoding device 110 uses the data associated with the one or more gaze locations to identify a target region. Thus, the decoding device 110 can align the target region with a location that the user is looking at and / or is likely to be looking at.

[0108] For example, for applications in virtual reality, the target regions 1006, 1008 appear as a sliding (or "moving") window in which the decoding device 110 decodes a higher quality image or video in a sliding window fashion. The sliding window can contain one or more gaze points of the viewer.

[0109] In this example, the images 1002, 1004 include a plurality of tiles. The enhancement data is associated with the tiles. The enhancement data can be used to generate data for representing one or more regions of the images 1002, 1004 at a relatively high quality level. For example, the enhancement data can be used to generate data for representing target regions 1006, 1008 at a relatively high quality level compared to the quality level associated with other regions associated with tiles outside of the target regions 1006, 1008. The target regions 1006, 1008 can correspond to the visible (or "viewable") portions of the images 1006, 1008. For example, in the context of virtual reality, the target regions move in a moving (or "sliding") window fashion as the viewer moves. In some examples, one or more tiles are added to and / or removed from the target regions when the amount of movement or deviation of the viewer's view exceeds a threshold amount. Thus, the moving window moves tile-by-tile as the viewer moves. The region 1010 in the first image 1002 corresponds to a set of tiles that are not in the target region 1006 of the first image 1002 but are in the target region 1008 of the second image 1004. The decoding device 110 can predict which regions and corresponding tiles will be added to the target region in subsequent images, and generate data for representing the regions at the desired quality level once such a prediction is made. Using this prediction method can reduce the amount of time needed to generate data for representing the target region in subsequent images. The decoding device 110 can not need to generate data for representing any overlapping regions of the target region between subsequent images, as data for representing the overlapping regions from a previous image can already be available to the decoding device 110. For example, such overlapping data can be cached for a given number of subsequent images, and retrieved from the cache if needed.

[0110] In this way, even if high quality image or video data is available for the entire image or video to the decoding device 110, the decoding device 110 decodes only the viewable portions to a relatively high quality level, e.g., up to the highest quality level, thereby saving power and other resources at the decoding device 110 and / or the second apparatus 104. With scalable video encoding techniques similar to those described above, a high quality layer can be defined in each tile, and the decoding device 110 obtains data related to the base layer and data related to the high quality layer for the tile at the focus, thereby saving bandwidth, power, and processing power compared to obtaining and processing such data for all of the tiles in the image. The decoding device 110 can determine the location of the target region in a subsequent image based on the field of view and / or one or more gaze locations of the viewer at the point in time associated with the subsequent image.

[0111] Furthermore, in the context of virtual reality applications of the above techniques, a viewer can be physically very close to the display screen. It can therefore be desirable to have a high resolution image or video, particularly where the display screen has a high display resolution, so that the viewer does not notice imperfections that can be apparent in lower quality or resolution levels in the image or video. This can limit the hardware or software that can be used in these applications, as the hardware and / or software and / or other conditions need to be able to cope with such high levels of data. For example, it can be desirable to send a UHD or 4K video stream to the decoding device 110 and / or the second apparatus 104. The decoding device 110 and / or the second apparatus 104 would need to be able to download or stream the video and would need to be able to decode and display the video stream at 4K. There can be limited demand for such devices. In the above example, the decoding device 110 can not need a 4K decoder to handle such video data. Instead, the decoding device 110 can decode video data at a lower quality level (e.g. HD) and use upscaled and enhanced data to generate a 4K quality level video representation. This can also result in more efficient bandwidth usage by using scalable encoding, as there is no need to send a 4K video stream over the network, even for particular tiles. In some examples, the decoding device 110 decodes a relatively low quality 4K video stream and uses enhancement data to generate a relatively high quality 4K video stream representation. As mentioned above, the relatively low quality 4K stream and the relatively high quality 4K representation can be associated with different sub-layers of a layered data processing apparatus.

[0112] Reference is made to Figure 11 FIG. 1 100 shows a block schematic diagram of an example apparatus 1 100.

[0113] In an example, the apparatus 1 100 comprises a decoding device.

[0114] Examples of the apparatus 1 100 include, but are not limited to, a mobile computer, a personal computer system, a wireless device, a base station, a telephone device, a tablet computer, a laptop computer, a notebook computer, a network appliance, a server, a storage device, a consumer electronics device, such as a camera, a camcorder, a mobile device, a video game console, a handheld video game device, a peripheral device such as a switch, a modem, a router, etc., or any other type of computing or electronic device.

[0115] In this example, the apparatus 1100 includes one or more processors 1101 for processing information and / or instructions. The one or more processors 1101 can include a central processing unit (CPU). The one or more processors 1101 are coupled with a bus 1102. The operations performed by the one or more processors 1101 can be performed by hardware and / or software. The one or more processors 1101 can include multiple co-located processors or multiple processors located at different locations.

[0116] In this example, the apparatus 1100 includes computer-usable volatile memory 1103 for storing information and / or instructions for the one or more processors 1101. The computer-usable volatile memory 1103 is coupled with the bus 1102. The computer-usable volatile memory 1103 can include random access memory (RAM).

[0117] In this example, the apparatus 1100 includes computer-usable non-volatile memory 1104 for storing information and / or instructions for the one or more processors 1101. The computer-usable non-volatile memory 1104 is coupled with the bus 1102. The computer-usable non-volatile memory 1104 can include read only memory (ROM).

[0118] In this example, the apparatus 1100 includes one or more data storage units 1105 for storing information and / or instructions. The one or more data storage units 1105 are coupled with the bus 1102. The one or more data storage units 1105 can include, for example, a magnetic or optical disk and disk drive or a solid state drive (SSD).

[0119] In this example, the apparatus 1100 includes one or more input / output (I / O) devices 1106 for communicating information to and / or from the one or more processors 1101. The one or more I / O devices 1106 are coupled with the bus 1102. The one or more I / O devices 1106 can include at least one network interface. The at least one network interface can enable the apparatus 1100 to communicate via one or more data communication networks. Examples of data communication networks include, but are not limited to, the Internet and local area networks (LANs). The one or more I / O devices 1106 can enable a user to provide input to the apparatus 1100 via one or more input devices (not shown). The one or more input devices can include, for example, a remote control, one or more physical buttons, etc. The one or more I / O devices 1106 can enable information to be provided to a user through one or more output devices (not shown). The one or more output devices can include, for example, a display screen.

[0120] Various other entities are depicted with respect to the apparatus 1100. For example, when an operating system 1107, a data signal processing module 1108, one or more other modules 1109, and data 1110 are present, they are shown as being present in one or a combination of computer-usable volatile memory 1103, computer-usable non-volatile memory 1104, and one or more data storage units 1105. The data signal processing module 1108 can be implemented by computer program code stored in memory locations in the computer-usable non-volatile memory 1104, the one or more data storage units 1105, and / or other tangible computer-readable storage media. Examples of tangible computer-readable storage media include, but are not limited to, optical media (e.g., CD-ROMs, DVD-ROMs, or Blu-ray discs), flash memory cards, soft disks or hard disks, or any other medium capable of storing computer-readable instructions. Computer-readable instructions include, for example, firmware or microcode stored in at least one ROM or RAM or programmable ROM (PROM) chip, or, for example, an application-specific integrated circuit (ASIC).

[0121] Accordingly, the apparatus 1100 can include a data signal processing module 1108, which can be executed by the one or more processors 1101. The data signal processing module 1108 can include instructions for implementing at least some of the operations described herein. During operation, the one or more processors 1101 activate, run, execute, interpret or otherwise process the instructions in the data signal processing module 1108.

[0122] While at least some aspects of the examples described herein with reference to the drawings comprise computer processes performed in processing systems or processors, examples described herein also extend to computer programs, e.g., computer programs on or in computer-readable media, adapted to perform any of the methods described herein, e.g. when run on the one or more processors.

[0123] It is to be understood that the apparatus 1100 can include Figure 11 more, less and / or different components than those depicted in FIG. 11.

[0124] The apparatus 1100 can be located in a single location, or can be distributed over a

[0125] The techniques described herein can be implemented in software or hardware, or can be implemented using a combination of software and hardware, and can include configuring an apparatus to perform and / or support any or all of the techniques described herein.

[0126] Various measures (e.g. a decoding device, a method and a computer program) are provided. Data is received, the data being usable to generate data for a data signal representing a first quality level. Based on the data for the data signal representing the first quality level, enhancement data is received, the enhancement data being usable to generate data for a data signal representing a second quality level. The second quality level is higher than the first quality level. A selected portion of the received enhancement data is used to generate data for a target region of the data signal representing a target quality level. The selected portion of the received enhancement data is associated with the target region of the data signal. The target quality level is higher than the first quality level. Data is produced for other regions of the data signal representing a quality level lower than the target quality level.

[0127] The target quality level can be the second quality level.

[0128] The target quality level can be between the first quality level and the second quality level.

[0129] The quality level of the other regions can be between the first quality level and the target quality level.

[0130] The decoding device can be operable to generate data for the other regions of the data signal using a further selected portion of the enhancement data. The further selected portion of the enhancement data can be associated with the other regions of the data signal.

[0131] The quality level of the other regions can be the first quality level.

[0132] The other regions can at least partially surround the target region.

[0133] The data associated with the field of view and / or the data associated with the one or more gaze positions can be used to identify the target region of the data signal.

[0134] The target region of the data signal can be selected so as to be in the field of view.

[0135] At least a portion of the other regions of the data signal can be selected so as to be in the field of view.

[0136] The field of view and / or the one or more gaze positions can be monitored at a plurality of points in time. The position of the target region in a subsequent data signal can be determined from the field of view and / or the one or more gaze positions at a point in time associated with the subsequent data signal.

[0137] The target region of the data signal can be associated with one or more data signal blocks and the other regions of the data signal can be associated with one or more data signal blocks. At least one data signal block associated with the other regions can be within the target region in a subsequent data signal.

[0138] Data for a target region of the data signal indicative of the first quality level can be generated. The generated data for the target region of the data signal indicative of the first quality level can be used to generate data for a target region of the data signal indicative of a target quality level.

[0139] The operations can be based on a hierarchical data signal processing structure. The hierarchical data signal processing structure can include at least one layer having a set of sub-layers. Each sub-layer can be associated with a respective quality level.

[0140] The enhancement data associated with at least one sub-layer can not be used in a first operational mode of the decoding device.

[0141] The enhancement data associated with all sub-layers can be used to generate data for a target region of the data signal indicative of a target quality level in a second operational mode of the decoding device.

[0142] The operations can be based on a hierarchical data signal processing structure. The hierarchical data signal processing structure can include a first layer having a first set of sub-layers and a second layer having a set of sub-layers. Each sub-layer can be associated with a respective quality level.

[0143] The enhancement data associated with at least one sub-layer of the first and second layers can be used to generate data for a target region of the data signal indicative of a target quality level in a third operational mode of the decoding device.

[0144] The first quality level can correspond to a quality level associated with a lowest sub-layer in the hierarchical data signal processing structure.

[0145] The second quality level can correspond to a quality level associated with a highest sub-layer in the hierarchical data signal processing structure.

[0146] The target quality level can correspond to a quality level associated with a sub-layer between the highest sub-layer and the lowest sub-layer in the hierarchical data signal processing structure.

[0147] The data signal can include image data.

[0148] The data signal can include video data.

[0149] A target region of the data signal can be identified.

[0150] A portion of the enhancement data associated with the target region of the data signal can be selected.

[0151] The decoding device can be included in a virtual reality device, a medical imaging device, a machine vision device, and / or a mobile communication device.

[0152] The virtual reality device can include the decoding device.

[0153] The medical imaging device can comprise the decoding device.

[0154] The machine vision device can comprise the decoding device.

[0155] The mobile communication device can comprise the decoding device.

[0156] The above embodiments are to be understood as illustrative examples. Further other embodiments are envisaged.

[0157] In some of the above examples, the decoding device 110 receives the fully encoded data from the encoding device 108 via a data communication network 106. In other examples, the decoding device 110 obtains such data from local storage, for example non-volatile memory. In other examples, the decoding device 110 obtains such data from removable storage media. For example, removable storage media includes compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray, etc.

[0158] In some of the above examples, the data signal processing system 100 processes image and / or video data. In other examples, the data signal processing system 100 processes other types of data signal, for example audio and / or volumetric data. In these systems, it can be desirable to enhance the quality level of a target region of the data processed by the system. In the case of audio data, the target region can comprise a particular frequency and / or time range of the audio data. In the case of volumetric data, for example “three-dimensional” medical imaging data, the target region can be a region in the volumetric data corresponding to a particular physical region of interest, for example an organ, a foetus or a tumour.

[0159] It will be appreciated that any feature described in relation to any one embodiment can be used alone, or in combination with other features described, and that the means for effecting amongst other things the enhancement of the quality level of a target region of data processed by the system can be implemented by one or more of the elements of any other embodiment, or any combination of any other embodiments. Furthermore, equivalents and modifications not described expressly herein will also be encompassed by the scope of the appended claims.

Claims

1. A virtual reality device, comprising a decoding device, the decoding device being used for: Receive data, which can be used to generate a data signal representing a first quality level; Receive enhanced data, which can be used to generate data representing a second quality level based on data representing a first quality level, wherein the second quality level is higher than the first quality level; Data representing a target region of a data signal having a target quality level is generated using a selected portion of the received enhanced data, the selected portion of the received enhanced data being associated with the target region of the data signal, the target quality level being higher than the first quality level; and Data for other regions is generated to represent data signals for quality levels lower than the target quality level; The enhanced data is used to correct the difference between one or more values ​​of one or more image elements in the original version of the data representing the first quality level and one or more values ​​of one or more corresponding image elements in another version of the data representing the first quality level, the difference being caused by upsampling and downsampling operations performed on the data representing the first quality level to generate the data representing the first quality level.

2. The virtual reality device according to claim 1, wherein the target quality level is the second quality level, or the target quality level is between the first quality level and the second quality level.

3. The virtual reality device according to claim 1 or 2, wherein the quality level of other regions of the data signal is between the first quality level and the target quality level.

4. The virtual reality device of claim 1 or 2, wherein the decoding device is configured to use a selected portion of the augmented data to generate data representing other regions of the data signal, the selected portion of the augmented data being associated with other regions of the data signal.

5. The virtual reality device according to claim 1 or 2, wherein the quality level of other regions of the data signal is the first quality level.

6. The virtual reality device according to claim 1 or 2, wherein the decoding device is configured to select a target region of the data signal to be within the field of view, or the decoding device is configured to select at least a portion of other regions of the data signal to be within the field of view.

7. The virtual reality device according to claim 6, wherein the decoding device is used for: Monitor the field of view and / or one or more gaze positions at multiple time points; and The location of the target region in the subsequent data signal is determined based on the field of view and / or the one or more gaze positions at a time point associated with the subsequent data signal, or the target region of the data signal is associated with one or more data signal blocks, and other regions of the data signal are associated with one or more data signal blocks, wherein... At least one data signal block associated with other areas of the data signal is located within the target area of ​​the subsequent data signal.

8. The virtual reality device according to claim 1 or 2, wherein the decoding device is used for: Generate data for the target region to represent the data signal of the first quality level; The generated data, representing the target region of the data signal for the first quality level, is used to generate data representing the target region of the data signal for the target quality level.

9. The virtual reality device according to claim 1 or 2, wherein the decoding device is configured to operate according to a hierarchical data signal processing structure, the hierarchical data signal processing structure comprising at least one layer having a set of sublayers, each sublayer being associated with a corresponding quality level.

10. The virtual reality device of claim 9, wherein the decoding device is not used in a first operating mode of the decoding device to use augmented data associated with at least one of the sublayers, or the decoding device is used to use augmented data associated with all of the sublayers to generate data for a target region of a data signal representing a target quality level in a second operating mode of the decoding device.

11. The virtual reality device according to claim 1 or 2, wherein the decoding device is configured to operate according to a hierarchical data signal processing structure, the hierarchical data signal processing structure comprising a first layer having a first set of sublayers and a second layer having a set of sublayers, each sublayer being associated with a corresponding quality level.

12. The virtual reality device of claim 11, wherein the decoding device is configured to use enhanced data associated with at least one sublayer of the first and second layers to generate data for a target region of a data signal representing the target quality level in a third operating mode of the decoding device.

13. The virtual reality device of claim 9, wherein the first quality level corresponds to the quality level associated with the lowest sublayer in the hierarchical data signal processing structure.

14. The virtual reality device of claim 9, wherein the second quality level corresponds to the quality level associated with the highest sublayer in the hierarchical data signal processing structure.

15. The virtual reality device of claim 9, wherein the target quality level corresponds to a quality level associated with a sublayer between the highest and lowest sublayers in the hierarchical data signal processing structure.

16. The virtual reality device according to claim 1 or 2, wherein the decoding device is used to identify the target area of ​​the data signal.

17. The virtual reality device of claim 1 or 2, wherein the decoding device is configured to select a portion of augmented data associated with a target region of the data signal.

18. A decoding method, comprising, in a decoding device: Receive data, which can be used to generate a data signal representing a first quality level; Receive enhanced data, which can be used to generate data representing a second quality level based on data representing a first quality level, wherein the second quality level is higher than the first quality level; Using a selected portion of the received enhanced data, data representing a target region of a data signal is generated to represent a target quality level, wherein the selected portion of the received enhanced data is associated with the target region of the data signal, and the target quality level is higher than the first quality level. and Data for other regions is generated to represent data signals for quality levels lower than the target quality level; The enhanced data is used to correct the difference between one or more values ​​of one or more image elements in the original version of the data representing the first quality level and one or more values ​​of one or more corresponding image elements in another version of the data representing the first quality level, the difference being caused by upsampling and downsampling operations performed on the data representing the first quality level to generate the data representing the first quality level.

19. A computer-readable medium comprising a computer program, the computer program including instructions that, when executed, cause a decoding device to perform the decoding method according to claim 18.

Images