Use upsampling regulator to process signal data

By introducing an upsampling adjuster in the encoder and decoder, operating in parallel with the upsampler, the inefficiency problem caused by data differences in signal transmission is solved, and efficient signal reconstruction and resource conservation is achieved, especially in high quality The data volume is significantly saved when video transmission.

CN111699692BActive Publication Date: 2024-11-15V NOVA INT LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201880088547.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-12-06
Filing Date
2018-12-06
Publication Date
2024-11-15
Estimated Expiration
2038-12-06

AI Technical Summary

Technical Problem

During the signal compression and decompression process, existing systems may cause data differences when using upsamplers, resulting in low efficiency when the decoder reconstructs the original signal, especially when high-quality video transmission is transmitted, resulting in a large amount of data, resulting in waste of resources.

Method used

The upsampling adjuster is introduced in the encoder and decoder, which operates in parallel with the upsampler. By adjusting the data, the difference after upsampling is reduced, a higher quality signal representation is generated, and the reference representation is reconstructed using residual data to reduce the value of the reference representation. The amount of data transmitted.

Benefits of technology

Through the use of the upsampling adjuster, the amount of data transmitted is reduced, the efficiency of signal reconstruction is improved, and resource consumption is reduced, especially when high-quality video transmission is saved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111699692B_ABST
    Figure CN111699692B_ABST
Patent Text Reader

Abstract

Signal data is processed at an encoder device (108). A representation of the signal at a first quality level is obtained. The representation at the first level is upsampled using an upsampler in the encoder device (108) to derive a preliminary representation of the signal at a second, higher quality level. The representation at the first level is used as an input to an upsampling adjuster arranged in parallel with the upsampler in the encoder device (108) to derive adjustment data. The adjustment data can be used by the encoder device (108) to adjust the preliminary representation. The preliminary representation and the adjustment data are used to derive an adjusted representation of the signal at a second level. The adjusted representation is used to derive residual data that can be used by a decoder device (110) to reconstruct a reference representation of the signal at a second level using the adjusted representation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to processing signal data using an up-sampling regulator. Background Art

[0002] Compression and decompression of signals is an important consideration in many known systems.

[0003] Many types of signals, such as video, image, audio or volumetric signals, can be compressed and encoded for transmission, for example, over a data communication network. Other signals can be stored in compressed form, for example on a storage medium such as a digital versatile disc (DVD). When decoding such a signal, it may be desirable to improve the quality level of the signal and / or to recover as much information as possible contained in the original signal.

[0004] Some known systems utilize scalable coding techniques. Scalable coding involves encoding a signal as well as information to allow reconstruction of the signal at different quality levels depending on the capabilities of the decoder and the available bandwidth. However, relatively large amounts of information may need to be stored and / or transmitted, especially as the use of higher quality, higher definition video becomes more prevalent.

[0005] Some known scalable coding systems use an upsampler to upsample the signal data. However, there may be differences between the upsampled data and the original signal. The inefficiency may be related to the encoded data that the decoder uses to reconstruct the original signal. Summary of the invention

[0006] According to a first aspect of the present invention, there is provided a method for processing signal data on an encoder device, the method comprising:

[0007] obtaining a representation of the signal at a first quality level;

[0008] upsampling the representation at the first quality level using an upsampler in the encoder device to derive a preliminary representation of the signal at a second, higher quality level;

[0009] using the representation at the first quality level as input to an upsampling adjuster arranged in parallel with the upsampler in the encoder device to derive adjustment data, the adjustment data being usable by the encoder device to adjust the preliminary representation;

[0010] deriving an adjusted representation of the signal at the second quality level using the preliminary representation and the adjustment data; and

[0011] Residual data is derived using the adjusted representation, the residual data being usable by a decoder device to reconstruct a reference representation of the signal at the second quality level using the adjusted representation.

[0012] According to a second aspect of the present invention, there is provided a method for processing signal data on a decoder device, the method comprising:

[0013] obtaining a representation of the signal at a first quality level;

[0014] upsampling the representation at the first quality level using an upsampler in the decoder device to derive a preliminary representation of the signal at a second, higher quality level;

[0015] using the representation at the first quality level as input to an upsampling adjuster arranged in parallel with the upsampler in the decoder device to derive adjustment data, the adjustment data being usable by the decoder device to adjust the preliminary representation;

[0016] deriving an adjusted representation of the signal at the second quality level using the preliminary representation and the adjustment data;

[0017] receiving data usable by the decoder device to derive residual data usable by the decoder device to reconstruct a reference representation of the signal at the second quality level using the adjusted representation;

[0018] deriving the residual data using the received data; and

[0019] The reference representation is reconstructed using the adjusted representation and the residual data.

[0020] According to a third aspect of the present invention, there is provided a method for processing signal data on an encoder device, the method comprising:

[0021] obtaining a representation of the signal at a first quality level;

[0022] upsampling the representation at the first quality level using a first upsampler in the encoder device to derive a first preliminary representation of the signal at a second higher quality level;

[0023] using the representation at the first quality level as input to a first upsampling adjuster arranged in parallel with the first upsampler in the encoder device to derive first adjustment data, the first adjustment data being usable by the encoder device to adjust the first preliminary representation;

[0024] deriving a first adjusted representation of the signal at the second quality level using the first preliminary representation and the first adjustment data;

[0025] deriving first residual data using the first adjusted representation, the first residual data being usable by a decoder device to reconstruct a reference representation of the signal at the second quality level using the first adjusted representation;

[0026] reconstructing the reference representation at the second quality level using the first adjusted representation and the first residual data;

[0027] upsampling the reference representation at the second quality level using a second upsampler in the encoder device to derive a second preliminary representation of the signal at a third quality level higher than the second quality level;

[0028] using the reference representation at the second quality level and data output from the first upsampling adjuster as input to a second upsampling adjuster arranged in parallel with the second upsampler in the encoder device to derive second adjusted data, the second adjusted data being usable by the encoder device to adjust the second preliminary representation;

[0029] deriving a second adjusted representation of the signal at the third quality level using the second preliminary representation and the second adjustment data; and

[0030] Second residual data is derived using the second adjusted representation, the second residual data being usable by a decoder device to reconstruct a reference representation of the signal at the third quality level using the second adjusted representation.

[0031] According to a fourth aspect of the present invention, there is provided a method for processing signal data on a decoder device, the method comprising:

[0032] obtaining a representation of the signal at a first quality level;

[0033] upsampling the representation at the first quality level using a first upsampler in the decoder device to derive a first preliminary representation of the signal at a second higher quality level;

[0034] using the representation at the first quality level as input to a first upsampling adjuster arranged in parallel with the first upsampler in the decoder device to derive first adjustment data, the first adjustment data being usable by the decoder device to adjust the first preliminary representation;

[0035] deriving a first adjusted representation of the signal at the second quality level using the first preliminary representation and the first adjustment data;

[0036] receiving first data usable by the decoder device to derive first residual data usable by the decoder device to reconstruct a reference representation of the signal at a second, higher quality level using the first adjusted representation;

[0037] deriving the first residual data using the received first data;

[0038] reconstructing the reference representation at the second quality level using the first adjusted representation and the first residual data;

[0039] upsampling the reference representation at the second quality level using a second upsampler in the decoder device to derive a second preliminary representation of the signal at a third quality level higher than the second quality level;

[0040] using the reference representation at the second quality level and data output from the first upsampling adjuster as input to a second upsampling adjuster arranged in parallel with the second upsampler in the decoder device to derive second adjustment data, the second adjustment data being usable by the decoder device to adjust the second preliminary representation;

[0041] deriving a second adjusted representation of the signal at the third quality level using the second preliminary representation and the second adjustment data;

[0042] receiving second data usable by the decoder device to derive second residual data, the second residual data being usable by the decoder device to reconstruct a reference representation of the signal at the third quality level using the second adjusted representation;

[0043] deriving the second residual data using the received second data; and

[0044] The reference representation at the third level of quality is reconstructed using the second adjusted representation and the second residual data.

[0045] According to a fifth aspect of the present invention, there is provided an apparatus arranged to perform a method according to any one of the first, second, third or fourth aspects of the present invention.

[0046] According to a sixth aspect of the present invention, there is provided a computer program comprising instructions which, when executed, cause a device to perform a method according to any one of the first, second, third or fourth aspects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic block diagram showing an example of a signal processing system according to an embodiment of the present invention;

[0048] Figure 2 A schematic diagram showing an example of a signal processing technique according to an embodiment of the present invention;

[0049] Figure 3 A schematic diagram showing another example of a signal processing technique according to an embodiment of the present invention;

[0050] Figure 4 A schematic diagram showing an example of a convolutional neural network;

[0051] Figure 5 A schematic diagram showing another example of a signal processing technique according to an embodiment of the present invention; and

[0052] Figure 6 A schematic block diagram of an example of an apparatus according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0053] refer to Figure 1 , shows an example of a signal processing system 100. The signal processing system 100 is used to process signals. Examples of signal types include, but are not limited to, video signals, image signals, audio signals, volumetric signals (such as those used in medical, scientific, or holographic imaging), or other multi-dimensional signals.

[0054] The signal processing system 100 includes a first device 102 and a second device 104. The first device 102 and the second device 104 may have a client-server relationship, wherein the first device 102 performs the function of a server device and the second device 104 performs the function of a client device. The signal processing system 100 may include at least one additional device. The first device 102 and / or the second device 104 may include one or more components. The components may be implemented in hardware and / or software. One or more components may be located in the same location in the signal processing system 100 or may be remote from each other. Examples of device types include, but are not limited to, computerized devices, routers, workstations, handheld or laptop computers, tablet computers, mobile devices, game consoles, smart TVs, set-top boxes, etc.

[0055] The first device 102 is communicatively coupled to the second device 104 via a data communication network 106. Examples of the data communication network 106 include, but are not limited to, the Internet, a local area network (LAN), and a wide area network (WAN). The first and / or second devices 102, 104 may have a wired and / or wireless connection to the data communication network 106.

[0056] The first device 102 includes an encoder device 108. The encoder device 108 is configured to encode signal data. The encoder device 108 can perform one or more other functions in addition to encoding signal data. The encoder device 108 can be embodied in various different ways. For example, the encoder device 108 can be embodied as hardware and / or software.

[0057] The second device 104 includes a decoder device 110. The decoder device 110 is configured to decode signal data. The decoder device 110 can perform one or more other functions in addition to decoding signal data. The decoder device 110 can be embodied in a variety of different ways. For example, the decoder device 110 can be embodied as hardware and / or software.

[0058] The encoder device 108 encodes the signal data and transmits the encoded signal data to the decoder device 110 via the data communication network 106. The decoder device 110 decodes the received encoded signal data and generates decoded signal data. The decoder device 110 can output the decoded signal data or data derived using the decoded signal data. For example, the decoder device 110 can output such data for display on one or more display devices associated with the second device 104.

[0059] In some examples described herein, the encoder device 108 transmits information that the decoder device 110 can use to reconstruct a representation of the signal at one or more higher quality levels. Such information can be referred to as "reconstructed data". In some examples, the "reconstruction" of the representation involves obtaining a representation that is not an exact copy of the original representation. The extent to which the representation is the same as the original representation may depend on various factors, including but not limited to the quantization level. The representation of the signal at a given quality level can be considered to be a reproduction, version or depiction of the data included in the signal at a given quality level. In some examples, the reconstruction data is included in the signal data encoded by the encoder device 108 and transmitted to the decoder device 110. For example, the reconstruction data can be in the form of metadata. In some examples, the reconstruction data is encoded and transmitted separately from the signal data. The reconstruction data is an example of compressed data. The compressed data may include data associated with the compressed data stream.

[0060] The information used by the decoder device 110 to reconstruct a representation of the signal at one or more higher quality levels may include residual data, as described in more detail below. Residual data is an example of reconstructed data. The reader is referred to international patent application PCT / EP2013 / 059847, which was published as WO2013 / 171173. PCT / EP2013 / 059847 describes the use of residual data in a hierarchical hierarchy. The entire contents of PCT / EP2013 / 059847 are incorporated herein by reference. In addition, for more details on hierarchical methods of encoding and decoding consistent with the use contemplated by the present invention, the reader is referred to U.S. Patents 8,977,065, 8,948,248, 8,711,943, 9,129,411, 8,531,321, 9,510,018, 9,509,990, which are incorporated herein by reference.

[0061] Compared to some known techniques, the examples described herein involve transmitting a relatively small amount of information for such reconstruction. Compared to some known techniques, this reduces the amount of data transmitted via the data communications network 106. The savings may be particularly significant in situations where the signal data corresponds to high-quality video data, where the amount of information transmitted in known systems may be particularly high.

[0062] In the examples described herein, measures are provided to allow the residual data to have one or more desired attributes. For example, it may be desirable to reduce the size of the residual data to below a target size and / or configure the residual data for efficient encoding. Thus, the examples described herein allow for controlling the residual data and / or the encoding of the residual data in a flexible manner.

[0063] refer to Figure 2 , schematically illustrates an example of a signal processing technique 200. The signal processing technique 200 may be performed by the encoder device 108 and / or the decoder device 110.

[0064] Input data 210 is obtained. The input data 210 may be derived by a device (e.g., encoder device 108 or decoder device 110) performing the signal processing technique 200, or the input data 210 may be received from one or more other entities. For example, when the signal processing technique 200 is performed by the decoder device 110, the decoder device 110 may generate the input data as a representation of the signal at a lower quality level and use it as the input data 210 in the process 200. Alternatively, the decoder device 110 may receive the input data 210 from the encoder device 108 and / or from another entity (such as an intermediate device that relays the input data 210 to the decoder device 110). The input data 210 may be received via a compressed data stream.

[0065] The input data 210 comprises a representation of a signal at a first quality level. For example, the input data 210 may comprise image data having a given resolution. The first quality level may be a given quality level in a hierarchical structure comprising a plurality of different quality levels. The first quality level may be the lowest quality level in the hierarchical structure, or may be an intermediate quality level between the highest quality level and the lowest quality level in the hierarchical structure. The reader is referred to international patent application PCT / EP2013 / 059847, which describes a hierarchical structure of quality levels.

[0066] The obtained input data 210 may be derived using a reference representation of the signal at a second quality level. The input data 210 may be derived by a device performing the signal processing technique 200, or the input data may be derived by one or more other entities. The second quality level is higher than the first quality level. For example, the second quality level may be a separate higher quality level in the hierarchical structure relative to the first quality level. The second quality level may be the highest quality level in the hierarchical structure, or may be an intermediate quality level between the highest quality level and the lowest quality level in the hierarchical structure. The input data 210 may be derived by performing a downsampling operation on the reference representation at the second quality level.

[0067] The input data 210 is upsampled using an upsampler 220. The upsampler 220 may be a physical or logical component of a device that performs the signal processing technique 200. The upsampler 220 may be configured with one or more predetermined upsampling functions. Examples of upsampling functions include, but are not limited to, nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation.

[0068] The upsampler 220 upsamples the input data 210 to derive upsampled data 225. The upsampled data 225 is a preliminary representation of the signal at a second quality level.

[0069] The input data 210 is also used as an input to an up-sampling adjuster 230. In this example, the up-sampling adjuster 230 is arranged in parallel with the up-sampler 220. The term "in parallel" is used herein to indicate that the up-sampling adjuster 230 and the up-sampler 220 receive a common input, namely the input data 210. The up-sampling adjuster 230 is used to derive adjustment data 235. The adjustment data 235 can be used to adjust the up-sampled data 225 derived by the up-sampler 220.

[0070] In some examples, the upsampling adjuster 230 includes an upsampling compensator. The upsampling compensator allows compensation for upsampling properties of the upsampler 220. Such upsampling properties may be undesirable properties of the upsampler 220 and / or the upsampled data 225 derived by the upsampler 220. For example, the upsampling properties may involve unwanted artifacts in the upsampled data 225. The upsampling properties may be compensated by adjusting the upsampled data 225 using the adjustment data 235.

[0071] In some examples, the upsampling adjuster 230 includes an upsampling enhancer. The upsampling enhancer allows the output of the upsampler 220 to be enhanced. "Enhancement" as described herein can be distinguished from "compensation" because enhancement involves improving a specific upsampling property, while compensation involves compensating for a specific upsampling property, such as an undesirable property. For example, the visual quality of the upsampled data 220 can be improved by using the upsampling adjuster 230. The output from the upsampler 220 can be enhanced by adjusting the upsampled data 225 using the adjustment data 235.

[0072] The up-sampling adjuster 230 may be a physical or logical component of a device that performs the signal processing technique 200. In this example, the up-sampling adjuster 230 and the up-sampler 220 are separate components arranged in parallel. In other examples, the up-sampling adjuster 230 and the up-sampler 220 are included in a single component.

[0073] In some examples, the upsampling adjuster 230 uses machine learning to derive the adjustment data 235. For example, the upsampling adjuster 230 can be trained using one or more training data sets in an "offline" mode. The upsampling adjuster 230 can then use the knowledge gained during training to derive the adjustment data 235. Using machine learning techniques to derive the adjustment data 235 for adjusting the upsampling data 225 improves the accuracy of the adjusted upsampling data 240 compared to when machine learning techniques are not used. The accuracy of the adjusted upsampling data 240 can relate to the degree of similarity of the adjusted upsampling data 240 to the reference representation. By improving the accuracy of the adjusted upsampling data 240, the encoding efficiency of the residual data can be improved. For example, the size of the residual data and / or the entropy associated with encoding the residual data can be reduced.

[0074] In some examples, the upsampling adjuster 230 includes a trained artificial neural network. An example of such a trained artificial neural network is a convolutional neural network. A convolutional neural network is a type of deep feedforward artificial neural network. A convolutional neural network may include a convolution layer and a pooling layer. A convolutional neural network may provide higher accuracy than other types of neural networks, especially in image processing applications. However, there may be a trade-off between accuracy and processing resources. For example, some other types of neural networks may use fewer processing resources than a convolutional neural network (e.g., a single-layer or multi-layer perceptron network), but may sacrifice accuracy.

[0075] An example of an artificial neural network designed to process images can be a neural network with the shape F(x,θ i )=y, where x is the input image, θ i are the parameters optimized by training the network, and y is the output. Function F can be linear or nonlinear. The input image x can have dimensions h×w. In the example, the output y has dimensions 2h×2w, it should be understood that in other examples, y can have other dimensions. In this way, function F can be considered an upsampler function because the output is an image with at least one dimension larger than the input image.

[0076] A training dataset including multiple training images can be used to train the network. For N training images, the training dataset includes where x t is a training image, and y t is the expected or “ground truth” output. t , the original image at a higher quality level can be used, such as a reference representation at a second quality level. t Yes t The un-upsampled version of t The network attempts to reconstruct y t Training involves calculating the loss and updating the network's parameters accordingly. The loss function is used to find the error E = || F(x t ,θ i )-y t ||Minimize the parameter θ i In other words, the network output for a training image is compared to the expected output for that image. The network also uses the i Previously set hyperparameters. Hyperparameters can specify the structure of the network itself and / or how the network is trained and / or used. Hyperparameters can be fixed during network training.

[0077] In some examples, instead of or in addition to an artificial neural network, upsampling adjuster 230 includes a lookup table. The lookup table may include one or more predetermined adjustment data sets associated with different upsampling and / or image attributes. The lookup table may be easier to implement than an artificial neural network because it does not require training. However, the lookup table may not be as accurate and / or reliable as an artificial neural network in adjusting upsampled data 225.

[0078] The upsampled data 225 and the adjustment data 235 are used to derive the adjusted upsampled data 240. The adjusted upsampled data 240 includes an adjusted representation of the signal at the second quality level. In some examples, the adjustment data 235 is used to adjust the upsampled data 225 to derive the adjusted upsampled data 240. For example, the adjustment data 235 may include additional corrections to be applied to the upsampled data 240. In other examples, the upsampled data 225 is applied to the adjustment data 235 to derive the adjusted upsampled data 240.

[0079] In some examples, for example, when the signal processing technique 200 is performed by the encoder device 108, the adjusted representation 240 is used to derive residual data. The residual data may be used by the decoder device 110 to reconstruct a reference representation using the adjusted representation 240. The residual data may be derived by comparing the adjusted representation 240 to a reference representation. In some examples, the residual data is derived by calculating a difference between the adjusted representation 240 and the reference representation. The comparison made between the reference representation and the adjusted representation 240 to determine the residual data may be reversed, for example, at the decoder device 110, so that the reference representation may be obtained by comparing the adjusted representation 240 to the residual data.

[0080] The output data may be generated by the encoder device 108. The output data may be used by the decoder device 110 to derive the input data 210. The data that may be used to derive the input data 210 may include the input data 210 itself. In some examples, the data that may be used to derive the input data 210 includes data from one or more quality levels that are lower than the first quality level, which may be used by the decoder device 110 to obtain the input data 210. The output data may also be used by the decoder device 110 to derive residual data. The output data may be encoded for transmission to the decoder device 110 to allow the decoder device 110 to reconstruct the reference representation. For example, the output data may be transmitted as a compressed data stream via the data communication network 106. The output data may be encoded before transmission to reduce the amount of data involved in transmitting the output data to the decoder device 110.

[0081] In some examples, for example, when the signal processing technique 200 is performed by the decoder device 110, data that can be used to derive residual data is received. The data that can be used to derive the residual data includes residual data or converted residual data (collectively referred to herein as "residual data"), encoded residual data, or other data that can be used by the decoder device 110 to derive residual data. The data that can be used to derive the residual data can be received from the encoder device 108. The data that can be used to derive the residual data can be received as part of a compressed data stream. The decoder device 110 can receive the data that can be used to derive the residual data and the data that can be used to obtain the input data 210. The decoder device 110 can use the residual data and the adjusted representation 240 to reconstruct the reference representation.

[0082] Using an upsampling adjuster 230 (e.g., which may be a convolutional neural network) to adjust the upsampled data 225 may allow for greater control or influence over the residual data used for signal reconstruction than when the upsampled adjuster 230 is not used. The adjusted representation 240 may be more similar to the reference representation than the upsampled data 225. Therefore, the difference between the adjusted representation 240 and the reference representation may be less than the difference between the upsampled data 225 and the reference representation. In this way, adjusting the upsampled data 225 may reduce the size of the resulting residual data that may be used to reconstruct the reference representation. Additionally or alternatively, adjusting the upsampled data 225 using the upsampled adjuster 230 may enable the residual data to be more efficiently encoded. For example, adjusting the upsampled data 225 using the upsampled adjuster 230 may reduce the coding entropy of the residual data. In some examples, fewer unique symbols may be used to represent the residual data than when the upsampled adjuster 230 is not used, thereby improving coding efficiency.

[0083] In some examples, the adjustment data derived at the encoder device 108 is not output to the decoder device 110. There may be a known relationship between the configuration of the encoder device 108 and the configuration of the decoder device 110. For example, the decoder device 110 may include an upsampling adjuster configured in a similar manner to the upsampling adjuster at the encoder device 108. Thus, the upsampling adjuster at the decoder device 110 may independently derive the adjustment data without requiring the decoder device 110 to receive the adjustment data from the encoder device 108.

[0084] In some examples, the up-sampling adjuster 230 is arranged to achieve a target coding property of the residual data. The target coding property may include an entropy such as Shannon entropy. In some examples, the target coding property is related to the value of the residual data. For example, it may be desirable to reduce or minimize these values, and / or it may be desirable to reduce the difference between the values ​​of different residual data elements.

[0085] In this example, the upsampling adjuster 230 is arranged in parallel with the upsampler 220. This design is particularly effective because the upsampling adjuster 230 can be used to supplement the existing upsampler by effectively providing enhanced functionality to the upsampler 220. The upsampling adjuster 230 can be retroactively incorporated into a system including the upsampler 220, thereby reducing the need to construct a new upsampler including the functionality of the upsampling adjuster 230 and increasing the service life of the upsampler 220. In addition, since different parts of the system can be independently controlled, enabled and / or disabled, applying the upsampling adjuster to the existing upsampler improves the flexibility of the signal processing system. In addition, in the example where the upsampling adjuster 230 includes an artificial neural network, the existing upsampler 220 provides guidance during the training of the artificial neural network. For example, the existing upsampler 220 can provide a "starting point" for training the artificial neural network, thereby promoting the improvement of the efficiency and / or accuracy of the artificial neural network.

[0086] refer to Figure 3 , schematically illustrates an example of a signal processing technique 300. The signal processing technique 300 may be performed by the encoder device 108 and / or the decoder device 110. Figure 3 Some of the items shown are Figure 2 The items shown are identical or similar. Therefore, identical or similar items are given corresponding reference numerals (increased by 100).

[0087] In this example, the upsampling adjuster 330 receives as a first input input data 310, ie a representation of a signal at a first quality level. The upsampling adjuster 330 also receives as a second input further data 350. The further data 350 may be associated with the first quality level.

[0088] In this example, the additional data 350 includes data derived using an additional up-sampling adjuster (not shown). The additional up-sampling adjuster can be arranged in a device that performs the signal processing technique 300. The additional up-sampling adjuster is associated with the additional up-sampler (not shown). The additional up-sampler is used to up-sample the representation of the signal at a quality level lower than the first quality level. The data derived using the additional up-sampling adjuster can be used to guide or direct the up-sampling adjuster 330 when deriving the adjustment data 335.

[0089] In some examples, the further data 350 is received from a further upsampling adjuster. In other examples, the further data 350 is retrieved from a memory.

[0090] The additional up-sampling adjuster may use machine learning. For example, the additional up-sampling adjuster may include an artificial neural network, such as a convolutional neural network. The additional up-sampling adjuster may be separate from the up-sampling adjuster 330. The additional up-sampling adjuster may be operable to derive adjustment data that may be used to adjust a preliminary representation of the signal at a first quality level, the preliminary representation being derived by up-sampling a representation of the signal at a quality level lower than the first quality level.

[0091] Thus, for a given quality level, Figure 3 The illustrated arrangement of upsampler 320 and upsample adjuster 330 may also be provided at one or more other quality levels. For example, the corresponding upsampler and upsample adjuster may be provided at a lower quality level.

[0092] In some examples, the up-sampling adjuster 330 and the additional up-sampling adjuster include an artificial neural network that is arranged to operate on the representation of the signal at different quality levels. The signal processing technique 300 may therefore involve linking artificial neural networks associated with different quality levels. Therefore, information can be shared between artificial neural networks associated with different quality levels. Sharing information between artificial neural networks can improve the accuracy and / or processing efficiency of one or two artificial neural networks. In addition to the effect obtained by using the up-sampling adjuster 330 itself, this effect is additional, that is, it is possible to perform additional control on the residual data. In some examples, such as when the up-sampling adjuster 330 and the additional up-sampling adjuster include artificial neural networks, the additional data 350 includes data output from the additional up-sampling adjuster. In an example, the additional data 350 includes intermediate data used by the additional up-sampling adjuster in the process of deriving the adjustment data. For example, the additional data 350 may include one or more kernels, filters, parameters, weights, etc. used by the additional up-sampling adjuster.

[0093] In examples where the upsampling regulator 330 and the further upsampling regulator comprise convolutional neural networks, the convolution filter used by the further upsampling regulator may be proportionally larger than the convolution filter used by the upsampling regulator 330. This is due to the further upsampling regulator operating on a lower quality level representation of the signal than the upsampling regulator 330 operates on.

[0094] Chaining neural networks is particularly advantageous. Since the "deeper" neural networks are, the better they perform (i.e., the more levels of neural networks there are, the better they perform), it may be desirable to try to maximize the depth. However, the deeper the neural networks are, the more computationally expensive they are, thus making them impractical for real-time processing (such as that required for live video) or for devices that do not have the necessary computational power. As such, conventional neural networks may be difficult to use in applications such as video processing. However, the design proposed in the present invention of chaining multiple independent neural networks together means that the neural networks working at higher levels are actually as deep as the stack of chained neural networks, thereby effectively utilizing the knowledge generated by the neural networks at lower levels. For example, if the neural network working at the first level in the hierarchy has three edge levels, and the neural network working at the second level in the hierarchy has three edge levels, and the two neural networks are chained, the neural network working at the second level in the hierarchy actually has six edge levels, so its depth is doubled without additional computational cost. In this way, by chaining the neural network at the second quality level with the neural network at the first quality level, the effective depth of the neural network at the second quality level can be enhanced. Thus, by linking neural networks associated with different hierarchical quality levels, the hierarchical nature of the signal processing arrangement can be exploited to share information between neural networks. In some examples, linking neural networks reduces the impact of undesirable transition-related artifacts in the reconstructed signal.

[0095] Therefore, neural networks can be used to perform upsampling as part of the encoding / decoding process. This use of neural networks is different from the use of neural networks for obtaining super-resolution images. In other words, artificial neural networks are used in the context of the encoding / decoding pipeline of this paper.

[0096] In some examples, the additional data 350 includes upsampler property data. The upsampler property data indicates properties of the upsampler 320. For example, the additional data 350 may include data indicating a type of upsampling and / or one or more upsampling parameters used by the upsampler 320.

[0097] In some examples, the additional data 350 includes signal attribute data. The signal attribute data indicates attributes of the signal and / or input data 310. For example, where the signal includes an image, the additional data 350 may include data indicating one or more image parameters or characteristics that may affect the upsampling adjuster 330.

[0098] In this example, if there is a second quality level that also uses an upsampling adjuster, the upsampling adjuster 330 can output additional data 360. The additional data 360 can be used by an upsampling adjuster associated with an upsampling of the representation of the signal at the second quality level. In this way, the upsampling adjuster 330 can generate data to be fed to another upsampling adjuster associated with a higher quality level. The additional data 360 can be sent to the upsampling adjuster associated with the higher quality level, or the additional data 360 can be stored in a memory for subsequent use by the upsampling adjuster. In the case where the upsampling adjuster 330 includes an artificial neural network, the additional data 360 can include additional outputs of the artificial neural network, which can be used to guide another artificial neural network associated with a higher quality level.

[0099] In some examples, the additional data 360 output from the upsampling adjuster 330 is information derived from upsampling the representation at the first quality level using the upsampling adjuster 330. Due to the sharing of information between upsampling adjusters associated with different quality levels, the additional data 360 can be used by an upsampling adjuster associated with a higher quality level to improve the accuracy of the upsampling adjuster associated with the higher quality level.

[0100] In some examples, an upsampler and a corresponding upsample adjuster are provided at each quality level in the hierarchical structure. In such examples, an upsample adjuster at a given quality level is linked to an upsample adjuster at the next lower quality level and an upsample adjuster at the next higher quality level. In this way, information can be shared between upsample adjusters at different hierarchical levels to improve the accuracy of upsample adjustments and / or improve the efficiency of processing using upsample adjusters.

[0101] In some examples, such as where a given upsampling regulator includes an artificial neural network, the hierarchical nature of the signal processing arrangement is exploited by providing interconnected neural networks at multiple quality levels in a hierarchy. The neural networks (which can be viewed together as an entire neural network architecture) can have an overall objective function that is optimized, which corresponds to the sum of the errors at each quality level in the hierarchy, rather than just the error at the highest quality level. Such a neural network architecture can provide a "step-by-step" or "level-by-level" criterion that facilitates efficient and accurate training of the neural networks, because the neural networks at different quality levels are able to share information and therefore guide each other in the optimization process. This "step-by-step" criterion can provide more efficient training than the case where the neural networks are not linked or the case where only the highest quality level in the hierarchy has an artificial neural network.

[0102] refer to Figure 4, a schematic diagram of an example of a convolutional neural network (CNN) 400 is shown. CNN 400 can be used according to the signal processing method described herein. In some examples, different convolutional neural networks operate at different signal quality levels in a hierarchical structure with multiple quality levels. In this example, the convolutional neural network 400 includes four layers, namely an input layer, two hidden layers, and an output layer. Each layer includes one or more nodes, which can be called "neurons". In this example, the input layer contains 5 neurons and the output layer contains 1 neuron. The neurons in the convolutional neural network 400 can be arranged in three dimensions (width, height, depth). For image input, for example, the depth dimension can be the number of color channels of the image. Each neuron may have one or more learnable weights or "biases" that can be adjusted during training. A set of one or more weights of a given neuron can be referred to as a "filter" or "kernel", which is convolved with the input image to produce an output. CNN 400 will also typically include a patch extraction and representation component for processing input data at a lower input quality level, and a reconstruction component for generating output data at a higher output quality level.

[0103] refer to Figure 5 , schematically illustrates an example of a signal processing technique 500. The signal processing technique 500 may be performed by the encoder device 108 and / or the decoder device 110. Figure 5 Some of the items shown are Figure 3 Items shown are the same or similar. Therefore, the same or similar items have corresponding reference numerals (increased by 200). The signal processing technique 500 may include a CNN used as an upsampling regulator, such as Figure 4 The CNN described.

[0104] like Figure 5 As shown, in this example, an upsampler 520 is provided N , CNN 530 N and residual processor 545 N , to generate samples at a given quality level (LOQ N ) signal. This arrangement of components can also be provided at one or more lower quality levels and / or one or more higher quality levels. For example, the corresponding upsampler (not shown), CNN (not shown) and residual processor (not shown) can be provided at a lower quality level (LOQ N-1 ) is provided. Alternatively or additionally, the corresponding upsampler 520 N+1 , CNN 530 N+1 and residual processor 545 N+1 Can be used at a higher quality level (LOQ N+1 )supply.

[0105] In this example, CNN 530 N is arranged to operate as an up-sampling regulator, which receives input data 510 N-1 (i.e., the signal is at the first quality level (LOQ N-1 ) is taken as the first input. Upsampling adjuster CNN530 N Also receiving additional input data 550 N-1 As the second input. Additional input data 550 N-1 May be associated with a first quality level. Upsampling Adjuster CNN 530 N Generate a second quality level (LOQ N ) under the adjustment data 535 N , the second quality level may be, for example, twice the resolution of the first quality level. Upsampling adjuster CNN 530 N Additional adjusted output data 550 is also generated N As the second input. Additional output data 550 N A second quality level may be associated with it.

[0106] In the example, the additional input data 550 N-1 Includes use of first quality level (LOQ N-1 ) is used as an up-sampler at a lower level CNN (not shown). The lower level CNN is associated with a lower level up-sampler (not shown). The lower level up-sampler is used to up-sample the representation of the signal at a quality level lower than the first quality level. Additional input data 550 derived using the lower level CNN N-1 Can be used to guide or direct CNN530 N Export Adjustment Data 535 N And additional output data 550 N .

[0107] Alternatively, if the first quality level is the lowest quality level in the signal processing technique 500, no additional input data 550 may be provided. N-1 .

[0108] In the example, the additional output data 550 N Including the use of the second quality level (LOQ N ) used as an upsampling regulator CNN 530 N Exported data. CNN 530 N With upsampler 520 N Upsampler 520 NUsed to upsample the representation of the signal at the first quality level. Using CNN 530 N Exported additional output data 550 N Can be used to guide or direct higher-level CNNs 530 N+1 Exporting higher level adjustment data 535 N+1 and higher level additional output data 550 N+1 .

[0109] Alternatively, if CNN 530 N Operating at the highest quality level, no additional output data may be generated. N .

[0110] In some examples, the additional input data 550 N-1 The latent representation data indicates the first input data 510 N-1 For example, where the signal comprises an image, the additional input data 550 N-1 May include indications that may affect CNN 530 N The data of one or more image characteristics are derived using a lower level CNN (not shown). Additional input data 550 may be provided N-1 As one or more channels of latent representation data from a lower level CNN. Similarly, additional output data 550 may be provided N As CNN N In some examples of the invention, the additional input data 550 N-1 The number of channels is greater than the additional output data 550 N This allows for greater efficiency because the amount of processing associated with higher quality levels in the CNN can be reduced accordingly.

[0111] Figure 5 The arrangement shown can be used to perform upsampling as part of the encoding / decoding process. When implemented in an encoder, the residual processor 545 N Input signal 540 N The quality level (LOQ) of the residual processor operation N ) is compared with a reference image associated with the image, and a residual data signal (not shown) is generated to be transmitted to a decoder, and a residual processor 545 N The residual data signal is applied as a correction to the input signal to generate its output signal 510 N When implemented in a decoder, the residual processor 545 N Using the quality level (LOQN ) is associated with the received residual data signal (not shown) to correct its input signal 540 N , and generates its output signal 510 N , the output signal can optionally be output for display.

[0112] refer to Figure 6 , a schematic block diagram of an example of a device 600 is shown.

[0113] In an example, the apparatus 600 comprises an encoder device. In another example, the apparatus 600 comprises a decoder device.

[0114] Other examples of apparatus 600 include, but are not limited to, a mobile computer, a personal computer system, a wireless device, a base station, a telephone device, a desktop computer, a laptop computer, a netbook, a notebook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, an application server, a storage device, a consumer electronic device such as a camera, a camcorder, a mobile device, a video game console, a handheld video game device, or generally any type of computing or electronic device.

[0115] In this example, device 600 includes one or more processors 601 configured to process information and / or instructions. One or more processors 601 may include a central processing unit (CPU). One or more processors 601 are coupled to bus 602. The operations performed by one or more processors 601 may be performed by hardware and / or software. One or more processors 601 may include multiple processors located at the same location or multiple processors located at different locations.

[0116] In this example, device 600 includes computer usable volatile memory 603 configured to store information and / or instructions for one or more processors 601. Computer usable volatile memory 603 is coupled to bus 602. Computer usable volatile memory 603 may include random access memory (RAM).

[0117] In this example, device 600 includes computer usable non-volatile memory 604 configured to store information and / or instructions for one or more processors 601. Computer usable non-volatile memory 604 is coupled to bus 602. Computer usable non-volatile memory 604 may include read only memory (ROM).

[0118] In this example, device 600 includes one or more data storage units 605 configured to store information and / or instructions. One or more data storage units 605 are coupled to bus 602. One or more data storage units 605 may include, for example, magnetic or optical disks and disk drives or solid state drives (SSDs).

[0119] In this example, the device 600 includes one or more input / output (I / O) devices 606 configured to transmit information to and / or from one or more processors 601. The one or more I / O devices 606 are coupled to the bus 602. The one or more I / O devices 606 may include at least one network interface. The at least one network interface may enable the device 600 to communicate via one or more data communication networks. Examples of data communication networks include, but are not limited to, the Internet and a local area network (LAN). The one or more I / O devices 606 may enable a user to provide input to the device 600 via one or more input devices (not shown). The one or more input devices may include, for example, a remote control, one or more physical buttons, etc. The one or more I / O devices 606 may enable information to be provided to a user via one or more output devices (not shown). The one or more output devices may, for example, include a display screen.

[0120] Various other entities of the device 600 are depicted. For example, when present, an operating system 607, a signal processing module 608, one or more additional modules 609, and data 610 are shown as residing in one of the computer usable volatile memory 603, the computer usable non-volatile memory 604, and the one or more data storage units 605, or a combination thereof. The signal processing module 608 may be implemented by computer program code stored in a memory location within the computer usable non-volatile memory 604, a computer readable storage medium within the one or more data storage units 605, and / or other tangible computer readable storage media. Examples of tangible computer readable storage media include, but are not limited to, optical media (e.g., CD-ROM, DVD-ROM, or Blu-ray), flash memory cards, floppy disks, or hard disks, or any other medium capable of storing computer readable instructions, such as firmware or microcode in at least one ROM or RAM or programmable ROM (PROM) chip, or as an application specific integrated circuit (ASIC).

[0121] The device 600 may thus include a signal processing module 608, which may be executed by one or more processors 601. The signal processing module 608 may be configured to include instructions to implement at least some of the operations described herein. During operation, the one or more processors 601 initiate, run, execute, interpret or otherwise perform the instructions in the signal processing module 608.

[0122] Although at least some aspects of the examples described herein with reference to the drawings include computer processes executed in a processing system or processor, the examples described herein also extend to computer programs, such as computer programs on or in a carrier suitable for putting the examples into practice. The carrier may be any entity or device capable of carrying the program.

[0123] It should be understood that the device 600 may include Figure 6 More, fewer and / or different components than those depicted.

[0124] Device 600 may be located at a single location or may be distributed across multiple locations. Such locations may be local or remote.

[0125] The techniques described herein may be implemented in software or hardware, or may be implemented using a combination of software and hardware. The techniques may include configuring a device to perform and / or support any or all of the techniques described herein.

[0126] Various measures (e.g., apparatus, methods, computer programs, and computer-readable media) are provided, wherein an encoder device obtains a representation of a signal at a first quality level. The representation at the first quality level is upsampled using an upsampler in the encoder device to derive a preliminary representation of the signal at a second, higher quality level. The representation at the first quality level is used as an input to an upsampling adjuster arranged in parallel with the upsampler in the encoder device to derive adjustment data, which can be used by the encoder device to adjust the preliminary representation. An adjusted representation of the signal at the second quality level is derived using the preliminary representation and the adjustment data. Residual data is derived using the adjusted representation, which can be used by a decoder device to reconstruct a reference representation of the signal at the second quality level using the adjusted representation.

[0127] In some examples, the encoder device outputs data that can be used by the decoder device to derive the representation at the first quality level and data that can be used by the decoder device to derive residual data.

[0128] Various measures (e.g., apparatus, methods, computer programs, and computer-readable media) are provided, wherein a decoder device obtains a representation of a signal at a first quality level. The representation at the first quality level is upsampled using an upsampler in the decoder device to derive a preliminary representation of the signal at a second, higher quality level. The representation at the first quality level is used as an input to an upsampling adjuster arranged in parallel with the upsampler in the decoder device to derive adjustment data, which can be used by the decoder device to adjust the preliminary representation. The preliminary representation and the adjustment data are used to derive an adjusted representation of the signal at the second quality level. The decoder device receives data that can be used to derive residual data. The residual data can be used by the decoder device to reconstruct a reference representation of the signal at the second quality level using the adjusted representation. The decoder device uses the received data to derive the residual data. The decoder device uses the adjusted representation and the residual data to reconstruct the reference representation.

[0129] In some examples, deriving the adjusted data includes using additional data as additional input to the upsampling adjuster.

[0130] In some examples, the further data includes data derived using a further upsampling adjuster. The further upsampling adjuster is associated with a further upsampler for upsampling a representation of the signal at a quality level lower than the first quality level.

[0131] In some examples, the upsampling adjuster uses machine learning to derive the adjusted data.

[0132] In some examples, the upsampling adjuster includes a trained artificial neural network.

[0133] In some examples, the upsampling regulator includes a convolutional neural network.

[0134] In some examples, the upsampling adjuster is arranged to achieve target encoding properties of the residual data.

[0135] In some examples, the upsampling adjuster includes an upsampling compensator.

[0136] In some examples, using the upsampling adjuster includes outputting data usable by an upsampling adjuster associated with an upsampler for upsampling a representation of the signal at the second quality level.

[0137] Various measures (e.g., apparatus, methods, computer programs, and computer-readable media) are provided, wherein an encoder device obtains a representation of a signal at a first quality level. The encoder device uses an upsampler in the encoder device to upsample the representation at the first quality level to derive a preliminary representation of the signal at a second, higher quality level. The encoder device derives adjustment data, which can be used by the encoder device to adjust the preliminary representation. The representation at the first quality level is used as a first input to an upsampler adjuster in the encoder device to derive the adjustment data. Further data is used as a second input to the upsampler adjuster to derive the adjustment data. The further data includes data derived using a further upsampler adjuster in the encoder device. The further upsampler adjuster is associated with a further upsampler, which is used to upsample the representation of the signal at a quality level lower than the first quality level. The encoder device uses the preliminary representation and the adjustment data to derive an adjusted representation of the signal at the second quality level. The encoder device uses the adjusted representation to derive residual data, which can be used by a decoder device to reconstruct a reference representation at the second quality level using the adjusted representation. The encoder device outputs data that can be used by the decoder device to derive a representation at the first quality level and data that can be used by the decoder device to derive residual data.

[0138] Various measures (e.g., apparatus, methods, computer programs, and computer-readable media) are provided, wherein a decoder device obtains a representation of a signal at a first quality level. The decoder device uses an upsampler in the decoder device to upsample the representation at the first quality level to derive a preliminary representation of the signal at a higher second quality level. The decoder device derives adjustment data, which can be used by the decoder device to adjust the preliminary representation. The representation at the first quality level is used as a first input to the upsample adjuster in the decoder device to derive the adjustment data. Further data is used as a second input to the upsample adjuster to derive the adjustment data. The further data includes data derived using a further upsample adjuster in the decoder device. The further upsample adjuster is associated with a further upsampler, which is used to upsample the representation of the signal at a quality level lower than the first quality level. The decoder device uses the preliminary representation and the adjustment data to derive an adjusted representation of the signal at the second quality level. The decoder device receives data that can be used to derive residual data. The residual data can be used by the decoder device to reconstruct a reference representation of the signal at the second quality level. The decoder device uses the received data to derive the residual data. A decoder device reconstructs the reference representation using the adjusted representation and the residual data.

[0139] In some examples, the additional data includes upsampler property data indicating properties of the upsampler.

[0140] In some examples, the further data includes signal property data indicative of a property of the representation of a signal at the first quality level.

[0141] Various measures (e.g., apparatus, methods, computer programs, and computer-readable media) are provided, wherein an encoder device obtains a representation of a signal at a first quality level. The encoder device uses a first upsampler in the encoder device to upsample the representation at the first quality level to derive a first preliminary representation of the signal at a higher second quality level. The encoder device uses the representation at the first quality level as an input to a first upsampling adjuster arranged in parallel with the first upsampler in the encoder device to derive first adjustment data. The first adjustment data can be used by the encoder device to adjust the preliminary representation. The encoder device uses the first preliminary representation and the first adjustment data to derive a first adjusted representation of the signal at the second quality level. The encoder device uses the first adjusted representation to derive first residual data, which can be used by a decoder device to reconstruct a reference representation of the signal at the second quality level using the first adjusted representation. The encoder device uses the first adjusted representation and the first residual data to reconstruct the reference representation at the second quality level. The encoder device then uses the second upsampler in the encoder device to upsample the reference representation at the second quality level to derive a second preliminary representation of the signal at a third quality level. The third quality level is higher than the second quality level. The encoder device uses the reference representation at the second quality level and the data output from the first upsampling adjuster as input to a second upsampling adjuster arranged in parallel with the second upsampling adjuster in the encoder device to derive second adjustment data. The second adjustment data can be used by the encoder device to adjust the second preliminary representation. The encoder device uses the second preliminary representation and the second adjustment data to derive a second adjusted representation of the signal at the third quality level. The encoder device uses the second adjusted representation to derive second residual data. The second residual data can be used by the decoder device to reconstruct the reference representation of the signal at the third quality level using the second adjusted representation.

[0142] In some examples, the encoder device outputs data usable by the decoder device to derive a representation at a first quality level, data usable by the decoder device to derive first residual data, and data usable by the decoder device to derive second residual data.

[0143] Various measures (e.g., apparatus, methods, computer programs, and computer-readable media) are provided, wherein a decoder device obtains a representation of a signal at a first quality level. The decoder device uses a first upsampler in the decoder device to upsample the representation at the first quality level to derive a first preliminary representation of the signal at a higher second quality level. The decoder device uses the representation at the first quality level as an input to a first upsampling adjuster arranged in parallel with the first upsampler in the decoder device to derive first adjustment data. The first adjustment data can be used by the decoder device to adjust the preliminary representation. The decoder device uses the first preliminary representation and the first adjustment data to derive a first adjusted representation of the signal at the second quality level. The decoder device receives first data that can be used by the decoder device to derive first residual data, which can be used by the decoder device to reconstruct a reference representation of the signal at a second quality level using the first adjusted representation. The decoder device derives the first residual data using the received first data. The decoder device reconstructs the reference representation at the second quality level using the first adjusted representation and the first residual data. Then, the decoder device uses a second upsampler in the decoder device to upsample the reference representation at the second quality level to derive a second preliminary representation of the signal at a third quality level. The third quality level is higher than the second quality level. The decoder device uses the reference representation at the second quality level and the data output from the first upsample adjuster as input to a second upsample adjuster arranged in parallel with the second upsampler in the decoder device to derive second adjustment data. The second adjustment data can be used by the decoder device to adjust the second preliminary representation. The decoder device uses the second preliminary representation and the second adjustment data to derive a second adjusted representation of the signal at the third quality level. The decoder device receives second data that can be used by the decoder device to derive second residual data, which can be used by the decoder device to reconstruct the reference representation of the signal at the third quality level using the second adjusted representation. The decoder device derives second residual data using the received second data. The decoder device reconstructs the reference representation at the third quality level using the second adjusted representation and the second residual data.

[0144] In some examples, the data output from the first upsampling adjuster is information derived from upsampling the representation at the first quality level using the first upsampling adjuster and can be used by the second upsampling adjuster to improve the accuracy of the second upsampling adjuster.

[0145] In some examples, the second upsampling adjuster includes a neural network, and the data output from the first upsampling adjuster is used as a data input to the neural network.

[0146] In some examples, the first upsampling adjuster includes a neural network.

[0147] In some examples, the first upsampling adjuster and / or the second upsampling adjuster includes a convolutional neural network.

[0148] Various measures (e.g., apparatus, methods, computer programs, and computer-readable media) are provided, wherein an encoder device obtains a representation of a signal at a first quality level, which representation at the first quality level has been derived using a reference representation of the signal at a higher second quality level. The representation at the first quality level is upsampled using an upsampler in the encoder device to derive a preliminary representation of the signal at a second quality level. The representation at the first quality level is used as an input to an upsample adjuster in the encoder device to derive adjustment data. The upsample adjuster and the upsampler are independent elements of the encoder device. The adjustment data can be used by the encoder device to adjust the preliminary representation. The preliminary representation and the adjustment data are used to derive an adjusted representation of the signal at the second quality level. Residual data is derived using the adjusted representation, which can be used by a decoder device to reconstruct a reference representation using the adjusted representation. Data that can be used by a decoder device to derive the representation at the first quality level and the residual data is output. The use of the adjuster and the adjustment data facilitates the control of the residual data.

[0149] The above-described embodiments are to be understood as illustrative examples. Other embodiments are contemplated.

[0150] In some examples described herein, the upsampler and the upsampling adjuster are arranged in parallel with the encoder device or the decoder device. In other examples, the upsampler and the upsampling adjuster are arranged in different configurations. For example, the upsampler and the upsampling adjuster can be arranged in series. In this way, the upsampling adjuster can receive data derived using the upsampler as input. In other examples, the upsampler and the upsampling adjuster are combined. In this way, the upsampler and the upsampling adjuster can be included in a single physical or logical component. For example, such a combined component can include a trained artificial neural network that performs both upsampling and upsampling adjustment. In some cases, compared with the situation of using a separate upsampler and an upsampling adjuster, the total amount of processing power, logic, storage space used, complexity and / or cost can be reduced using a combined upsampler and an upsampling adjuster.

[0151] In some examples, the adjustment data derived by the upsampling adjuster is combined with the input data to derive the adjusted input data. The adjusted input data can then be input to the upsampler. For example, the input data can be modified based on knowledge about how the upsampler will process the input data to generate the upsampled data. Such knowledge can be acquired through training, for example, in the case where the upsampling adjuster includes an artificial neural network. In such an example, when the adjustment data is applied to the input data instead of the high-resolution upsampled data, the resolution of the adjustment data may be lower than when the adjustment data is applied to the upsampled data. Therefore, compared with the case where the higher-resolution adjustment data and the higher-resolution upsampled data are combined together, the processing efficiency can be improved.

[0152] It should be understood that any feature described with respect to any embodiment can be used alone or in combination with other features described, and can also be used in combination with one or more features of any other embodiment, or in combination with any combination of any other embodiment. In addition, equivalents and modifications not described above may also be adopted without departing from the scope of the invention as defined in the appended claims.

Claims

1. A method for processing signal data on an encoder device, the method comprising: obtaining a representation of the signal at a first quality level; upsampling the representation at the first quality level using an upsampler in the encoder device to derive a preliminary representation of the signal at a second, higher quality level; using the representation at the first quality level as input to an upsampling adjuster arranged in parallel with the upsampler in the encoder device to derive adjustment data, the adjustment data being usable by the encoder device to adjust the preliminary representation; deriving an adjusted representation of the signal at the second quality level using the preliminary representation and the adjustment data; as well as deriving residual data using the adjusted representation, the residual data being usable by a decoder device to reconstruct a reference representation of the signal at the second quality level using the adjusted representation; wherein said deriving of said adjusted data comprises using further data as a further input to said upsampling adjuster; and wherein the further data comprises data derived using a further upsampling adjuster associated with a further upsampler for upsampling a representation of the signal at a quality level lower than the first quality level, and the further data further comprises upsampler attribute data indicating at least one of: a type of upsampling performed using the upsampler, or at least one upsampling parameter used by the upsampler.

2. The method according to claim 1, comprising: output data, which can be used by the decoder device to derive: said representation at said first quality level; and The residual data.

3. A method according to claim 1, wherein the further data comprises signal property data indicative of properties of the representation at the first quality level. The method of claim 1 , wherein the upsampling adjuster uses machine learning to derive the adjustment data. The method of claim 1 , wherein the upsampling regulator comprises a trained artificial neural network.

6. The method of claim 1, wherein the upsampling regulator comprises a convolutional neural network.

7. The method of claim 1, wherein the upsampling regulator is arranged to achieve a target coding property of the residual data. The method of claim 1 , wherein the upsampling adjuster comprises an upsampling compensator.

9. The method of claim 1, wherein using the upsampling adjuster comprises outputting data usable by an upsampling adjuster associated with an upsampler for upsampling a representation of the signal at the second quality level.

10. A method of processing signal data at a decoder device, the method comprising: obtaining a representation of the signal at a first quality level; upsampling the representation at the first quality level using an upsampler in the decoder device to derive a preliminary representation of the signal at a second, higher quality level; using the representation at the first quality level as input to an upsampling adjuster arranged in parallel with the upsampler in the decoder device to derive adjustment data, the adjustment data being usable by the decoder device to adjust the preliminary representation; deriving an adjusted representation of the signal at the second quality level using the preliminary representation and the adjustment data; receiving data usable by the decoder device to derive residual data usable by the decoder device to reconstruct a reference representation of the signal at the second quality level using the adjusted representation; deriving the residual data using the received data; as well as reconstructing the reference representation using the adjusted representation and the residual data; wherein said deriving of said adjusted data comprises using further data as a further input to said upsampling adjuster; and wherein the further data comprises data derived using a further upsampling adjuster associated with a further upsampler for upsampling a representation of the signal at a quality level lower than the first quality level, and the further data further comprises upsampler attribute data indicating at least one of: a type of upsampling performed using the upsampler, or at least one upsampling parameter used by the upsampler.

11. A method of processing signal data on an encoder device, the method comprising: obtaining a representation of the signal at a first quality level; upsampling the representation at the first quality level using a first upsampler in the encoder device to derive a first preliminary representation of the signal at a second higher quality level; using the representation at the first quality level as an input to a first upsampling adjuster arranged in parallel with the first upsampler in the encoder device to derive first adjustment data, the first adjustment data being usable by the encoder device for adjusting the first preliminary representation, wherein the deriving the first adjustment data comprises using further data as a further input to the first upsampling adjuster, the further data comprising upsampler property data, the upsampler property data indicating at least one of: a type of upsampling performed by using the first upsampler, or at least one upsampling parameter used by the first upsampler; deriving a first adjusted representation of the signal at the second quality level using the first preliminary representation and the first adjustment data; deriving first residual data using the first adjusted representation, the first residual data being usable by a decoder device to reconstruct a reference representation of the signal at the second quality level using the first adjusted representation; reconstructing the reference representation at the second quality level using the first adjusted representation and the first residual data; upsampling the reference representation at the second quality level using a second upsampler in the encoder device to derive a second preliminary representation of the signal at a third quality level higher than the second quality level; using the reference representation at the second quality level and data output from the first upsampling adjuster as input to a second upsampling adjuster arranged in parallel with the second upsampler in the encoder device to derive second adjusted data, the second adjusted data being usable by the encoder device to adjust the second preliminary representation; deriving a second adjusted representation of the signal at the third quality level using the second preliminary representation and the second adjustment data; and Second residual data is derived using the second adjusted representation, the second residual data being usable by a decoder device to reconstruct a reference representation of the signal at the third quality level using the second adjusted representation.

12. The method according to claim 11, comprising: output data, which can be used by the decoder device to derive: said representation at said first quality level; the first residual data; and The second residual data.

13. The method of claim 11, wherein the data output from the first upsampling adjuster is information derived from upsampling the representation at the first quality level using the first upsampling adjuster and can be used by the second upsampling adjuster to improve the accuracy of the second upsampling adjuster.

14. The method of claim 11, wherein the second up-sampling regulator comprises a neural network, and the data output from the first up-sampling regulator is used as data input to the neural network.

15. The method of claim 11, wherein the first upsampling regulator comprises a neural network.

16. The method of claim 11, wherein the first upsampling adjuster and / or the second upsampling adjuster comprises a convolutional neural network.

17. A method of processing signal data at a decoder device, the method comprising: obtaining a representation of the signal at a first quality level; upsampling the representation at the first quality level using a first upsampler in the decoder device to derive a first preliminary representation of the signal at a second higher quality level; using the representation at the first quality level as an input to a first upsampling adjuster arranged in parallel with the first upsampler in the decoder device to derive first adjustment data, the first adjustment data being usable by the decoder device for adjusting the first preliminary representation, wherein said deriving the first adjustment data comprises using further data as further input to the first upsampling adjuster, the further data comprising upsampler property data, the upsampler property data indicating at least one of: a type of upsampling performed by using the first upsampler, or at least one upsampling parameter used by the first upsampler; deriving a first adjusted representation of the signal at the second quality level using the first preliminary representation and the first adjustment data; receiving first data usable by the decoder device to derive first residual data usable by the decoder device to reconstruct a reference representation of the signal at a second, higher quality level using the first adjusted representation; deriving the first residual data using the received first data; reconstructing the reference representation at the second quality level using the first adjusted representation and the first residual data; upsampling the reference representation at the second quality level using a second upsampler in the decoder device to derive a second preliminary representation of the signal at a third quality level higher than the second quality level; using the reference representation at the second quality level and data output from the first upsampling adjuster as input to a second upsampling adjuster arranged in parallel with the second upsampler in the decoder device to derive second adjustment data, the second adjustment data being usable by the decoder device to adjust the second preliminary representation; deriving a second adjusted representation of the signal at the third quality level using the second preliminary representation and the second adjustment data; receiving second data usable by the decoder device to derive second residual data usable by the decoder device to reconstruct a reference representation of the signal at the third quality level using the second adjusted representation; deriving the second residual data using the received second data; and The reference representation at the third level of quality is reconstructed using the second adjusted representation and the second residual data.

18. A device arranged to include a processor and a memory storing instructions which, when executed by the processor, cause the device to perform the method of any one of claims 1-9, 10, 11-16 or 17.

19. A computer program product comprising instructions which, when executed by a processor, cause an apparatus to perform the method of any one of claims 1-9, 10, 11-16 or 17.

20. A computer readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program causes a device to perform the method of any one of claims 1-9, 10, 11-16 or 17.