Method and device for decoding data representative of sound or visual content, method and device for encoding such data, and associated data stream

Abstract

A method for decoding data representative of audio or visual content comprises the following steps: - decoding first data (B1) to obtain a signal (IDSdec) at a first resolution; - decoding second data (B2) to obtain a plurality of weights (Wo); - upsampling the signal (IDSdec) at the first resolution to a signal (IUS) at a second resolution higher than the first resolution; - filtering the signal (IUS) at the second resolution, the filtering comprising at least one convolution using a convolution matrix, at least some of whose coefficients are respectively the weights of the plurality of weights (Wo). A decoding method, associated devices, and data streams are also described. Figure for the abstract: Figure 2

Description

Title of the invention: Method and device for decoding data representing sound or visual content, method and device for encoding such data, and associated data stream Technical field of the invention

[0001] The present invention relates to the technical field of video or audio coding.

[0002] It relates in particular to a method and a device for decoding data representative of sound or visual content, a method and a device for encoding such data, and an associated data stream. State of the art

[0003] It has been proposed in the prior art to use artificial neural networks to improve the reconstruction quality of oversampled images.

[0004] Reference may be made, for example, to the article "Enhanced Deep Residual Networks for Single Image Super-Resolution" by Bee Lim et al., published at the "Computer Vision and Pattern Recognition 2017" conference.

[0005] These solutions allow the processing of all types of images, but have, on the other hand, a relatively high computational cost. Furthermore, the artificial neural network is optimized for processing subsampled images using a given subsampling process and is therefore not suitable for processing subsampled images using a different subsampling process. Presentation of the invention

[0006] In this context, a method for decoding data representative of audio or visual content is proposed, comprising the following steps:

[0007] - decoding initial data so as to obtain a signal at a first resolution ;

[0008] - decoding second data in such a way as to obtain a plurality of weights;

[0009] - oversampling of the signal at the first resolution into a signal at one second resolution higher than the first resolution;

[0010] - filtering of the signal at the second resolution, the filtering comprising at least one convolution by means of a convolution matrix where at least some of the coefficients are respectively the weights of the plurality of weights.

[0011] The filtering, defined in an adaptive way thanks to the decoding of the second data, makes it possible to improve the quality of the oversampled signal (for example by making it approach an original signal that one wishes to reproduce).

[0012] The method may further include a step of decoding third data indicative of a location of said weights within the convolution matrix.

[0013] These third data elements may include fourth data elements defining the shape of a pattern at which said weights are placed within the convolution matrix. These fourth data elements thus include, for example, an identifier that identifies said shape among a plurality of predetermined shapes.

[0014] Some at least of the third data may also define an extent of said pattern within the convolution matrix.

[0015] The aforementioned filtering may include a plurality of convolutions implemented respectively (and successively) by means of a plurality of convolution matrices, each defined at least in part by weights obtained by decoding a part of the second data.

[0016] The third data can then include, for each convolution matrix of the plurality of convolution matrices, data indicating a location of the weights within the convolution matrix concerned.

[0017] The third data may also include a number of convolutions for which the third data include indicative data of a location of the weights.

[0018] Alternatively, the number of convolutions for which the third data include data indicative of a location of the weights can be predetermined.

[0019] Convolutions for which the third data include data indicative of a location of the weights are, for example, the first convolutions (in the order of application of the convolutions).

[0020] The aforementioned filtering may include at least one convolution implemented by means of a predetermined convolution matrix, or several convolutions implemented respectively by means of a plurality of predetermined convolution matrices (some of which may possibly be distinct from each other).

[0021] Said at least one convolution can be implemented in practice by a layer of an artificial neural network.

[0022] The process can then include a data decoding step (forming part of the third data in the example described later and) indicating a number of layers of the artificial neural network for which weights are coded among the second data.

[0023] In some embodiments, the second resolution can be double the first resolution in each of the signal dimensions.

[0024] When the sound or visual content is an image, the first resolution and the second resolution can be spatial resolutions.

[0025] When the image is defined by several components, the decoding of the first data can be followed by a step of conversion from a first color representation system to a second color representation system.

[0026] Also proposed is a method for encoding data representative of sound or visual content, comprising the following steps:

[0027] - subsampling, into a signal at a first resolution, of a signal at a second resolution higher than the first resolution;

[0028] - encoding the signal at the first resolution so as to obtain first data ;

[0029] - obtaining an intermediate signal by decoding the first data and oversampling at the second resolution;

[0030] - determination of a plurality of weights which minimize a criterion involving a distance between the signal at the second resolution, transformed by colorimetric conversion or not, and a signal produced by filtering the intermediate signal using at least one convolution by means of a convolution matrix of which at least some of the coefficients are respectively the weights of the plurality of weights;

[0031] - coding of the determined weights in order to obtain second data.

[0032] This method may include, for each of a plurality of weight configurations within the convolution matrix, a step of determining a set of weights which minimizes a criterion involving a distance between the signal at the second resolution and a signal produced by filtering the intermediate signal using at least one convolution by means of a convolution matrix having the configuration concerned and defined by this set of weights, the coded weights being the weights of the set of weights for which the produced signal satisfies a predetermined criterion.

[0033] The method may include a step of encoding third data indicative of the location of the weights within the convolution matrix in the configuration for which the produced signal satisfies the predetermined criterion.

[0034] When the aforementioned content is a video sequence, the steps of encoding and determining a plurality of weights (with the associated location) can be carried out for each of the images of the video sequence.

[0035] For decoding, the decoding device will then receive first data, second data, and third data as defined above for each frame of the video sequence. Each frame of the video sequence can thus be decoded (by the decoding, upsampling, and filtering steps) in accordance with what has been described above.

[0036] Also proposed is a device for decoding data representative of audio or visual content, comprising:

[0037] - a decoding unit configured to decode initial data in a to obtain a signal at a first resolution and second data in such a way as to obtain a plurality of weights;

[0038] - an oversampling unit configured to oversample the signal to the first resolution into a signal with a second resolution higher than the first resolution;

[0039] - a filtering unit configured to filter the signal at the second resolution, the unit filtering being configured to apply at least one convolution by means of a convolution matrix of which at least some of the coefficients are respectively the weights of the plurality of weights.

[0040] A device for encoding data representative of audio or visual content is also proposed, comprising:

[0041] - a subsampling unit configured to subsample, into a signal to a first resolution, a signal at a second resolution higher than the first resolution;

[0042] - a coding unit configured to encode the signal at the first resolution of in order to obtain initial data;

[0043] - a decoding unit configured to obtain a decoded signal on the first resolution by decoding the initial data;

[0044] - an oversampling unit configured to oversample the decoded signal, respectively transformed by colorimetric conversion or not, so as to obtain an intermediate signal at the second resolution;

[0045] - a learning unit configured to determine a plurality of weights which minimize a criterion involving a distance between the signal at the second resolution, respectively transformed by colorimetric conversion or not, and a signal produced by filtering the intermediate signal using at least one convolution by means of a convolution matrix of which at least some of the coefficients are respectively the weights of the plurality of weights;

[0046] in which the coding unit is configured to encode the determined weights so as to obtain second data.

[0047] Finally, a data stream representing sound or visual content is proposed, comprising first data representing a signal at a first resolution and second data representing weights usable as coefficients of a convolution matrix useful for filtering a signal at a second resolution obtained by oversampling the signal at the first resolution.

[0048] Such a data stream may also include third data indicative of a location of said weights within the convolution matrix.

[0049] Of course, the various features, variants, and embodiments of the invention can be combined with one another in various ways, provided they are not incompatible or mutually exclusive. Detailed description of the invention

[0050] In addition, various other features of the invention become apparent from the attached description made with reference to the drawings which illustrate non-limiting embodiments of the invention and where:

[0051] [Fig-1] represents the main elements of a device for encoding data representative of an image;

[0052] [Fig.2] represents the main elements of a data decoding device representative of an image;

[0053] [Fig.3] represents a first example of a convolution matrix usable in these devices;

[0054] [Fig.4] represents a second example of a convolution matrix usable in these devices;

[0055] [Fig.5] represents a third example of a convolution matrix usable in these devices;

[0056] [Fig.6] represents a fourth example of a convolution matrix usable in these devices; and

[0057] [Fig.7] represents a fifth example of a convolution matrix usable in these devices.

[0058] The present contribution lies in the field of encoding and decoding data representative of sound or visual content.

[0059] The following description presents embodiments in which this content is an image. However, the proposed solution can be applied without difficulty to other sound or visual content, for example a video sequence (in which case the solution applies, for example, to the different images of the video sequence) or audio content (in which case the notion of spatial resolution used in the following description is replaced by the notion of temporal resolution of the sound signal concerned).

[0060] Fig. 1 represents the main elements of an electronic device for encoding representative data of an original IO image having an initial spatial resolution.

[0061] This coding device therefore implements a coding process whose steps are outlined in the following description.

[0062] The initial spatial resolution is, for example, a resolution of more than 3000 pixels in the horizontal direction (i.e., an original IO image comprising more than 3000 columns of pixels) and / or a resolution of more than 1800 pixels in the vertical direction (i.e., an original IO image comprising more than 1800 lines of pixels), such as a resolution of 3840 x 2160 pixels (generally referred to as "4K format").

[0063] In the example described here, the original IO image is in YUV format, meaning that the original IO image comprises one luminance component and two chrominance components. Alternatively, other formats with one luminance component and two chrominance components can be used, for example, the YCrCb format. According to yet another alternative, which is mentioned in places below, the original IO image could be in RGB format.

[0064] In these different examples, the original image comprises three components (together defining a color image), each component having the aforementioned initial spatial resolution.

[0065] The electronic coding device 10 comprises a first colorimetric conversion unit 11, a downsampling unit 12, a coding unit 14, a decoding unit 16, a second colorimetric conversion unit 17, an upsampling unit 18 and a learning unit 20.

[0066] Each of these units can be implemented in practice due to the execution, by a processor of the electronic coding device 10, of dedicated computer program instructions enabling the functionalities described below to be performed for the unit concerned, when these instructions are executed by the processor.

[0067] Alternatively, however, one or more of these units could be implemented by a dedicated integrated circuit (different from the aforementioned processor), for example an application-specific integrated circuit.

[0068] The first color conversion unit 11 is designed to convert the original image from a first colorimetric representation format (or system) (here, the YUV format) to a second colorimetric representation format (for example, a display format), here, the RGB format. The converted original image thus obtained is denoted IR in the following. The converted original image IR is therefore defined at the initial resolution.

[0069] The first colorimetric conversion unit 11 performs, for example, the colorimetric conversion by multiplying, for each pixel of the original image IO, a vector formed from the values ​​of the different components of the original image IO for that pixel by a predefined conversion matrix, in order to obtain a vector comprising the values ​​of the different components of that pixel in the converted original image IR. The number (here three) of components in the original image IO is equal to the number of components in the converted original image IR. Alternatively However, these numbers could be different from each other, for example in the case of a conversion from RGB format to CMYK format (Cyan, Magenta, Yellow, Black, format used in the technical field of printing).

[0070] In certain embodiments (for example when the original image 10 is already in RGB format), the first color conversion unit 11 can be omitted.

[0071] The subsampling unit 12 is configured to subsample the original IO image into an IDS image having a spatial resolution lower than the initial spatial resolution.

[0072] This lower spatial resolution is for example a spatial resolution of less than 3000 pixels in the horizontal direction (i.e. the IDS subsampled image comprises less than 3000 columns of pixels) and / or a resolution of less than 1800 pixels in the vertical direction (i.e. an IDS subsampled image comprising less than 1800 rows of pixels), such as a resolution of 1920 x 1080 pixels or a resolution of 1280 x 720 pixels.

[0073] In the case where the initial spatial resolution is 3840 x 2160 pixels and the lower spatial resolution is 1920 x 1080 pixels, the initial spatial resolution is thus double the lower spatial resolution in the horizontal dimension of the image and in the vertical dimension of the image.

[0074] The subsampling unit 12 performs the aforementioned subsampling, for example, by Lanczos filtering, or, alternatively, by phase extraction, or even by bicubic filtering. According to yet another alternative, the aforementioned subsampling unit 12 can perform the aforementioned subsampling by means of an artificial neural network.

[0075] Such subsampling is applied here separately to each component forming the original image IO.

[0076] The coding unit 14 includes a first coding module 141 designed to encode the subsampled IDS image so as to obtain first Bl data. This first coding module 141 may be an intra-image encoder of the HEVC or VVC type, or a JPEG-type encoder. The first coding module 141 may, however, alternatively perform another type of lossy coding.

[0077] The decoding unit 16 is designed to perform reverse decoding of the encoding carried out by the first encoding module 141. Thus, the decoding unit 16 produces, by decoding the first data Bl, an IDSdec decoded image having the aforementioned lower resolution. Because the encoding used by the first encoding module 141 is lossy, the IDSdec decoded image is generally not strictly identical to the IDS undersampled image.

[0078] The second colorimetric conversion unit 17 is designed to convert the IDSdec decoded image from the first colorimetric representation format (or system) (here The YUV format used for the original image (10), for the IDS-downsampled image, and therefore for the IDSdec-decoded image, is converted to a second colorimetric representation format (for example, a display format), here the RGB format. We will denote the converted decoded image thus obtained as A in the following.

[0079] The second colorimetric conversion unit 17 performs, for example, the colorimetric conversion by multiplying, for each pixel of the decoded IDSdec image, a vector formed of the values ​​of the different components of the decoded IDSdec image for that pixel by a predefined conversion matrix, in order to obtain a vector comprising the values ​​of the different components of that pixel in the converted decoded image A. The number (here three) of components in the decoded IDSdec image is here equal to the number of components in the converted decoded image A. Alternatively, however, these numbers could be different from each other.

[0080] In certain embodiments (such as for example in the aforementioned variant where the original IO image is in RGB format, or in the case of processing an audio signal), the second colorimetric conversion unit 17 can be omitted.

[0081] The oversampling unit 18 is configured to oversample the decoded image (here after colorimetric conversion, i.e. the converted decoded image A) so as to obtain an intermediate image B at the initial spatial resolution.

[0082] The oversampling carried out by the oversampling unit 18 is for example achieved by means of a filtering associated with the filtering used by the undersampling unit 12.

[0083] According to a first conceivable approach, the oversampling unit 18 can use a plurality of distinct filters each producing a phase (having the resolution of the decoded image, i.e. here the lower resolution mentioned above) from the decoded (here converted) image A, and multiplex the different phases in order to obtain the intermediate image B.

[0084] For example, when the initial resolution is double the lower resolution in both dimensions of the image, the upsampling unit 18 uses 4 separate filters producing 4 phases respectively from the decoded (here converted) image A and multiplexes these 4 phases in order to obtain the intermediate image B.

[0085] According to a second conceivable approach, the oversampling unit 18 can insert rows and / or columns of zeros into the decoded (here converted) image A in order to obtain an image having the initial resolution, then apply to this image a convolution filter (for example a bilinear filter or a bicubic filter or a Lanczos filter) in order to obtain the intermediate image B.

[0086] When the initial resolution is twice the lower resolution in both dimensions of the image, the upsampling unit 18 inserts in this case a line a column of zero-value pixels under each row of pixels of the decoded (here converted) image A and a column of zero-value pixels after each column of pixels of the decoded (here converted) image, then applies the convolution filter to this image in order to obtain the intermediate image B.

[0087] Regardless of the approach used, when the oversampling unit 18 performs oversampling by means of a filter, it is possible in certain embodiments to modify the parameters of the filter during a learning phase described later (which makes it possible, in particular, if necessary, to adapt the oversampling carried out to the undersampling carried out by the undersampling unit 12).

[0088] The learning unit 20 includes a filtering module 22 and an optimization module 24.

[0089] The filtering module 22 receives as input the intermediate image B and is designed to apply to this intermediate image B a convolution by means of a convolution matrix or, in certain embodiments such as those presented below, a plurality of convolutions each carried out by means of a convolution matrix.

[0090] The filtering module thus produces an image C (having the same spatial resolution as the intermediate image B, i.e. the initial spatial resolution).

[0091] The filtering module 22 can in certain embodiments implement an artificial neural network, each of the aforementioned convolutions then being able to be carried out by means of a layer of the artificial neural network.

[0092] The coefficients of the convolution matrix defining a given convolution performed by the filtering module 22 are then respectively the weights associated with the neurons of the layer corresponding to this given convolution in the artificial neural network.

[0093] Each convolution matrix (also called a "convolution kernel") has a number of elements (or coefficients) significantly less than the number of pixels in the intermediate image B, for example a number of elements less than one ten-thousandth of the number of pixels in the intermediate image B (i.e. in the initial resolution).

[0094] The number of elements in each convolution matrix can in practice be less than 256.

[0095] In the examples described later with reference to Figures 3 to 7, the convolution matrices are matrices with a maximum of 5 rows and 5 columns (5 x 5 matrices) and therefore comprise 25 elements (or coefficients).

[0096] Thus, the filtering module 22 applies the convolution (or series of convolutions) successively to blocks of pixels extracted from the intermediate image B (here for each of the three components of the intermediate image B), these blocks having the same dimensions that the convolution matrix(s), so as to produce, for each extracted pixel block, a value of one pixel (of one component) of the image C.

[0097] In embodiments using a neural network (as already mentioned), each layer of the artificial neural network applies a given convolution to the set of pixel values ​​received as input to the layer concerned (by applying the convolution matrix successively to the different blocks of pixels received as input, these blocks of pixels being of the same dimensions as the convolution matrix) so as to produce at the output of the layer concerned a set of pixel values ​​(or latent values) of the same dimensions as the intermediate image C or, for the last layer, the set of pixel values ​​of the image C.

[0098] As is usual in an artificial neural network, the pixel values ​​(or latent values) produced by a given layer are applied as input to the next layer.

[0099] Each layer of the artificial neural network can apply, in addition to the aforementioned convolution, at least one other function, such as a linear function (or activation function), for example a ReLU (or rectifier) ​​type function. In this case, the activation function is, for example, applied to each pixel value produced by the convolution associated with the layer in question, and each value produced by the activation function forms a pixel value (latent value) to be applied as input to the next layer.

[0100] The coefficients of the convolution matrices, i.e. in this case the weights defining the artificial neural network, are determined during a learning phase described below.

[0101] The optimization module 24 receives as input the image C produced at the output of the filtering module 22 and the original IR converted image produced by the first colorimetric conversion unit 11 and determines a distance between these two images, for example a distortion measurement between these two images.

[0102] The optimization module 24 is configured to test, for at least one convolution (i.e., for at least one layer of the artificial neural network), a plurality of predefined locations of the coefficients within the relevant convolution matrix and, each time, to determine the coefficients (i.e., the weights of the relevant layer in the artificial neural network) that minimize the aforementioned distance between the C image and the IR image, or, alternatively, a rate-distortion cost involving not only a measure of distortion between the C image and the IR image but also a measure of the amount of information required to encode the IO image.

[0103] As already indicated, the parameters optimized to minimize the aforementioned distance (or the aforementioned rate-distortion cost) may include, in addition to the coefficients (or weights) of the convolution matrix(s), the parameters of the filter used by the oversampling unit 18.

[0104] In the example described here, each predefined location of the coefficients (or weights) within the convolution matrix is ​​defined by the form and extent of a pattern at the level of which the coefficients (or weights) are placed within the convolution matrix concerned (the coefficients of this convolution matrix located outside this pattern being systematically harmed).

[0105] In this context, the optimization module 24 may optionally also test certain locations of the coefficients, each defined by superposition of several patterns, each defined by a shape and an extent, as explained below with reference in particular to [Fig.7].

[0106] The aforementioned localization of the coefficients among a plurality of predefined localizations can be tested separately for several convolutions used (i.e. for several layers of the artificial neural network), the number of these convolutions being able to vary.

[0107] For example, a set of possible configurations is defined as follows:

[0108] - a first group of configurations defined by considering all locations envisaged for the first envisaged convolution (i.e. for the first layer of the artificial neural network), the subsequent convolutions (i.e. the subsequent layers of the artificial neural network) being predetermined, i.e. carried out by means of predetermined convolution matrices;

[0109] - a second group of configurations defined by considering all locations envisaged for the first envisaged convolution (i.e. for the first layer of the artificial neural network) and all locations envisaged for the second envisaged convolution (i.e. for the second layer of the artificial neural network) according to all possible combinations, the subsequent convolutions (i.e. the subsequent layers of the artificial neural network) being predetermined, i.e. carried out by means of predetermined convolution matrices;

[0110] - a third group of configurations defined by considering all locations considered for the first envisaged convolution (i.e., for the first layer of the artificial neural network), all locations envisaged for the second envisaged convolution (i.e., for the second layer of the artificial neural network) and all locations envisaged for the third envisaged convolution (i.e., for the third layer of the artificial neural network) according to all possible combinations, the subsequent convolutions (i.e., the subsequent layers of the artificial neural network) being predetermined, i.e., carried out by means of predetermined convolution matrices;

[0111] and so on until a group of configurations is reached in which all the locations considered are considered for all convolutions (i.e. for all layers of the artificial neural network) in all possible combinations, without predetermined convolution.

[0112] According to one conceivable variant, the number of convolutions (i.e. the number of layers of the artificial neural network) for which the location of the coefficients is variable among several possible locations is predetermined, which makes it possible to reduce the number of configurations to be tested.

[0113] During a learning phase, for each of the conceivable configurations defined above, the optimization module 22 determines (for example by a least squares method or by gradient descent) the coefficients (or weights) of the convolution matrix(ies), located at the locations specified by the configuration concerned, and possibly the parameters of the filter of the oversampling unit 18, which minimize the criterion used (for example as already indicated the distortion measurement between the IR image and the C image, or, alternatively, a rate-distortion cost) and stores, in association with the current configuration, the minimum value of the criterion used thus obtained for this configuration.

[0114] When all configurations have been tested, the optimization module 22 selects the configuration for which the stored value of the criterion used is optimal (here minimal); this is for example the configuration for which the stored distortion measurement is minimal.

[0115] According to a possible variant, instead of using predetermined convolutions for the last layers as proposed below, one can use convolutions whose location of the coefficients is predetermined (for example extended over the whole convolution matrix), but whose value of the coefficients is determined during the learning phase.

[0116] The optimization module 22 thus produces, for at least one convolution (i.e., one layer of the artificial neural network) for which several locations of the coefficients have been tested, a set of coefficients (or weights) and a location of these (corresponding to the selected configuration) which is part of the plurality of locations tested.

[0117] Specifically, in the example described here, the optimization module 22 produces the following output:

[0118] - the NNL number of convolutions (i.e., layers of neural networks) artificial) defined by a pattern and weights (as explained below) in the selected configuration, with subsequent convolutions being predetermined;

[0119] - for each of these NNL convolutions (i.e., for each of the NNL first layers of the artificial neural network), the localization of the weights (or coefficients) within the convolution matrix among the plurality of tested locations and the weights (or coefficients) W to be used at the locations defined (within the convolution matrix) by this location.

[0120] In the variant mentioned above where only the location relative to the subsequent convolutions is predetermined (but not the value of the weights or coefficients defining these subsequent convolutions), the optimization module 22 also produces as output the weights to be used (at the predetermined locations) for the subsequent convolutions.

[0121] In some embodiments, the optimization module 24 can also output the optimized parameters of the filter used by the oversampling unit 18.

[0122] The coding unit 14 includes a second coding module 142 designed to code (for each convolution for which such weights are determined by the learning process described above, i.e. here for NNL convolutions) the weights W so as to obtain second data B2.

[0123] The second coding module 142 can perform lossy coding or lossless coding.

[0124] According to a first possible embodiment, the second coding module 142 quantifies the weights W with a determined quantization step, then applies a known entropy coding algorithm, such as arithmetic coding or Huffman coding.

[0125] According to a second possible embodiment, the second coding module 142 encodes the weights W (which define, as indicated above, layers of an artificial neural network) in accordance with the MPEG-7 part 17 standard (used to encode the parameters of an artificial neural network).

[0126] The coding unit 14 also includes a third coding module 143 designed to encode (for each convolution for which weights are defined, i.e. here for NNL convolutions) the location L of the weights W within the relevant convolution matrix so as to form third data B3. Since the volume of third data B3 is relatively small, the third coding module 143 uses a lossless coding technique, for example by juxtaposing the data indicated below (number of NNL, flag or identifier(s) and / or parameter(s) for each of the NNL convolutions).

[0127] In the example described here, the third data B3 define at least one pattern at the level of which the weights W are placed within the relevant convolution matrix.

[0128] To do this, these third data B3 include:

[0129] - of the fourth data defining the shape of the aforementioned motif, and which for example, they may include an identifier that identifies this shape among a plurality of predetermined shapes;

[0130] - optionally, data defining an extent of this pattern.

[0131] According to one conceivable embodiment, the third data B3 comprises:

[0132] - the number NNL of convolutions (i.e., here the number of layers of the network of artificial neurons) for which localization information L is available such as the following;

[0133] - for each of these NNL convolutions (that is, here for each of these NNL layers of the artificial neural network), a use_default_loc flag indicating whether a default convolution pattern (e.g., a 3x3 convolution matrix) is used (case where use_default_loc is 1);

[0134] - a loc_type identifier (part of the fourth data mentioned above) identifying a pattern shape from a plurality of predetermined shapes and / or a loc_scale parameter defining the extent of the pattern when, for a convolution (i.e., for a layer of the artificial neural network), the use_default_loc flag is 0 (the pattern defined by the loc_type identifier and the loc_scale parameter then indicating, as already mentioned, the positions of the weights W encoded by some of the B2 seconds within the relevant convolution matrix).

[0135] As indicated above, the NNL layers of the artificial neural network concerned here are the first NNL layers of this artificial neural network, the location information thus being able to be coded from the first layer to the NNL order layer.

[0136] In the variant mentioned above where the number of convolutions (i.e. here the number of layers of the artificial neural network) for which the location of the coefficients is variable is predetermined (the subsequent layers using a predefined location of the coefficients in each convolution matrix concerned), the number NNL can be omitted from the third data B3.

[0137] Some examples of usable predetermined pattern shapes are described later with reference to Figures 4 and 6.

[0138] The coding unit 14 can also be configured to encode the optimized parameters of the filter used by the oversampling unit 18.

[0139] The coded data Bl, B2, B3 can be stored within the coding device 10 for later use, or transmitted to a decoding device (for example such as that described below with reference to [Fig.2]) by means of a communication unit (not shown) of the coding device 10.

[0140] When the coded data Bl, B2, B3 (representative of the original IO image) are transmitted in this way, the transmitted data stream then comprises:

[0141] - the first B1 data representative of the image at the lower resolution (IDS image);

[0142] - the second data B2 representing the weights W usable as coefficients of a useful convolution matrix (as explained later) for filtering an image at the initial resolution obtained by oversampling the IDS image at the lower resolution;

[0143] - the third B3 data indicative of a location of these weights within the convolution matrix.

[0144] This data stream may optionally also include the optimized parameters of a filter usable for the aforementioned oversampling (which corresponds to the filter used by the oversampling unit 18).

[0145] In the already mentioned case where the sound or visual content is a video sequence, it can be expected that the learning process described above will apply to each image of the video sequence, thus making it possible to obtain location information L and weights W (as well as possibly optimized parameters of an oversampling filter) for each of the images of the video sequence.

[0146] In this case, the data stream representing the video sequence comprises, for each frame of the video sequence:

[0147] - of the first representative data of the image concerned at the resolution inferior;

[0148] - second data representing weights usable as coefficients of a convolution matrix useful for filtering an image at the initial resolution obtained by oversampling the image in question to the lower resolution;

[0149] - third indicative data for the location of these weights within the convolution matrix;

[0150] - possibly, optimized parameters of a filter usable for this oversampling.

[0151] Fig. 2 represents the main elements of an electronic device for decoding such data representative of an image.

[0152] This decoding device therefore implements a decoding process whose steps are outlined in the following description.

[0153] The electronic decoding device 30 includes a decoding unit 32, a colorimetric conversion unit 34, an oversampling unit 36 ​​and a filtering unit 38.

[0154] Each of these units can be implemented in practice due to the execution, by a processor of the electronic decoding device 30, of dedicated computer program instructions enabling the performance of the functionalities described below for the unit in question, when these instructions are executed by the processor.

[0155] Alternatively, however, one or more of these units could be implemented by a dedicated integrated circuit (different from the aforementioned processor), for example an application-specific integrated circuit.

[0156] The data Bl, B2, B3 processed by the electronic decoding device 30 as explained below are for example received by a receiving unit (not shown) of the electronic decoding device 30.

[0157] Alternatively, the electronic encoding device 10 described with reference to [Fig.1] and the electronic decoding device 30 can have access to the same memory (not shown), in particular when the encoding device 10 and the decoding device 30 are the same electronic device, and the data Bl1, B2, B3 (stored in this memory by the encoding device 10 as described above) can be read from this memory.

[0158] The decoding unit 32 includes a first decoding module 321 designed to decode the first B1 data so as to obtain an IDSdec decoded image at a first resolution (here the lower resolution mentioned above).

[0159] The first decoding module 321 is of the same type as the decoding unit 16 used by the encoding device 10 and therefore one can refer to the explanations given above concerning the decoding unit 16.

[0160] The decoding unit 32 includes a second decoding module 322 designed to decode the second data B2 so as to obtain a plurality of weights Wo. Herein, Wo denotes the decoded weights thus obtained; indeed, because the coding technique used to encode the second data B2 may be a lossy coding technique, the decoded weights Wo may not be strictly identical to the weights W obtained in the context of the coding process described above with reference to [Fig. 1].

[0161] The decoding unit 32 includes a third decoding module 323 designed to decode the third indicative data B3, for each convolution matrix for which weights are coded by the second data B2, of a location L of the weights within the convolution matrix concerned.

[0162] As mentioned above, these third B3 data include here:

[0163] - an NNL number of convolutions (i.e., as explained below, of layers of an artificial neural network) for which weights are coded among the second data B2;

[0164] - for each of these NNL convolutions, a use_default_loc flag indicating whether a The default convolution pattern is used (within a convolution matrix) for the convolution in question;

[0165] - for each convolution for which the default convolution pattern is not used, a loc_type identifier (fourth data) identifying a convolution pattern shape (within the relevant convolution matrix) from among a plurality of predetermined shapes and / or a loc_scale parameter defining the extent of the convolution pattern.

[0166] For each convolution for which weights are coded among the second data B2, the third data B3 thus define a pattern representing, within the convolution matrix concerned, the locations where the decoded weights will be used as coefficients of the convolution matrix (the other coefficients of the convolution matrix being nus).

[0167] The color conversion unit 34 is designed to convert the IDSdec decoded image from a first color representation format (here the YUV format used for the original image 10 and for the IDSdec decoded image) to a second color representation format (for example, a display format), here the RGB format. This color conversion unit 34 is of the same type as the second color conversion unit 17.

[0168] The colorimetric conversion unit 34 performs, for example, the colorimetric conversion by multiplying, for each pixel of the IDSdec decoded image, a vector formed from the values ​​of the different components of the IDSdec decoded image for that pixel by a predefined conversion matrix, in order to obtain a vector comprising the values ​​of the different components of that pixel in the converted decoded image. The number (here three) of components in the IDSdec decoded image is equal to the number of components in the converted decoded image. Alternatively, however, these numbers could be different.

[0169] In certain embodiments (such as when the decoded IDSdec image is in RGB format, or in the case of processing an audio signal), the color conversion unit 34 may be omitted.

[0170] The upsampling unit 36 ​​is configured to upsample the decoded IDSdec image (here converted by the color conversion unit 34), which is at the first resolution, into an IUS image at a second resolution higher than the first resolution (this second resolution being here the initial resolution, i.e. the resolution of the original IO image).

[0171] The oversampling performed by the oversampling unit 36 ​​is, for example, carried out by means of a filter associated with the filtering used by the undersampling unit 12. In embodiments where optimized parameters relating to oversampling are transmitted as indicated above, this filtering can be defined by these parameters. In other cases, the oversampling unit 36 ​​uses, for example, a predefined filter.

[0172] According to a first conceivable approach, the oversampling unit 36 ​​can use a plurality of distinct filters each producing a phase (having the resolution of the decoded IDSdec image, i.e. here the first resolution) from the decoded IDSdec image (here converted by the colorimetric conversion unit 34), and multiplex the different phases in order to obtain the IUS image at the second resolution.

[0173] For example, when the second resolution is double the first resolution in both dimensions of the image, the upsampling unit 36 ​​uses 4 separate filters producing 4 phases respectively from the decoded IDSdec image (here converted by the color conversion unit 34) and multiplexes these 4 phases in order to obtain the IUS image at the second resolution.

[0174] According to a second conceivable approach, the oversampling unit 36 ​​can insert rows and / or columns of zeros into the decoded IDSdec image (here converted by the colorimetric conversion unit 34), and then apply a convolution filter (for example a bilinear filter or a bicubic filter or a Lanczos filter) to this image in order to obtain the IUS image at the second resolution.

[0175] When the second resolution is double the first resolution in both dimensions of the image, the upsampling unit 36 ​​inserts a row of zero-value pixels under each row of pixels of the IDSdec decoded image (here converted by the color conversion unit 34) and a column of zero-value pixels after each column of pixels of the IDSdec decoded image (here converted by the color conversion unit 34), and then applies a convolution filter to this image.

[0176] The filtering unit 38 is configured to apply filtering to the IUS image at the second resolution in order to obtain a final IF image, this filtering comprising one or more successive convolutions by means of one or more convolution matrix(s) defined respectively by the second data B2 and the third data B3.

[0177] In the example described here, the filtering unit 38 implements an artificial neural network whose successive layers respectively implement the convolutions used to perform the filtering applied by the filtering unit 38 as mentioned above.

[0178] The filtering unit 38 is of the same type as the filtering module 22 described above. However, the coefficients of the convolution matrices used in the filtering unit 38 are determined as a function of the coded data B2, B3 received by the decoding device (whereas the coefficients of the convolution matrices used in the filtering module 22 vary during the learning phase described above).

[0179] The filtering unit 38 uses the weights Wo (obtained from the coded data B2) and the location information L of the weights (obtained from the coded data B3) to configure the different convolutions used (i.e., to define, for at least some of the convolutions used, the associated convolution matrix (in other words, here the weights of the layer of the artificial neural network implementing this convolution): for each convolution defined by some of the data B2, B3, the filtering unit 38 determines, on the basis of the location information L relating to this convolution, locations within the convolution matrix defining this convolution, and sets the coefficients of the convolution matrix located at these locations (taken in a predefined order) to the respective values ​​of the weights Wo relating to this convolution (or, in other words, uses, as coefficients of the convolution matrix located at these locations, respectively and in a predefined order, the weights Wo relating to this convolution).

[0180] In the example described here, the filtering unit 38 performs the following steps to achieve this:

[0181] - for each of the first NNL convolutions (i.e., here for each of the first layers of the artificial neural network), reading the use_default_loc flag indicating whether a default convolution pattern is used (within a convolution matrix) for the convolution in question;

[0182] - if the use_default_loc flag indicates that a default convolution pattern is used, configuration of the relevant convolution matrix (i.e. here of the relevant artificial neural network layer) using, for the coefficients located at the locations of the default convolution pattern, respectively the weights Wo obtained from the B1 data for this convolution (the coefficients located outside the locations of the default convolution pattern being set to zero);

[0183] - if the use_default_loc flag indicates the default convolution pattern is not used, determination of the target locations (within the relevant convolution matrix) based on the convolution pattern shape identified by the identifier loc_type and the extent of this pattern defined by the parameter loc_scale, and configuration of the relevant convolution matrix (i.e. here the relevant artificial neural network layer) using, for the coefficients located at the target locations, respectively the weights Wo obtained from the B1 data for this convolution (the coefficients located outside the target locations being set to zero);

[0184] - for each of the possible convolutions (or neural network layers) artificial) subsequent to the first NNL convolutions (layers), use of a predetermined convolution matrix (separate predetermined convolution matrices may possibly be used for these different subsequent convolutions), i.e. configuration of the relevant artificial neural network layer using this predetermined convolution matrix.

[0185] To determine the targeted locations on the basis of the pattern shape and the pattern extent, the filtering unit 38 applies, for example, a homothety to the locations defined by the pattern shape, a homothety centered on the center of the convolution matrix (or kernel) and with a ratio equal to the pattern extent.

[0186] Furthermore, in some embodiments, several pairs of identifier loc_type and parameter loc_scale can be associated with the same convolution: a location is then a target location as soon as it is present in at least one of the patterns defined by one of the identifier loc_type - parameter loc_scale pairs associated with this convolution.

[0187] Figures 4 to 7 give examples of defining target locations in a convolution matrix on the basis of at least one identifying part loc_type - parameter loc_scale.

[0188] Thanks to the processing of the IUS image at the second resolution by a filtering (carried out by the filtering unit 38) adapted to this image (the coefficients, or weights, of the convolution matrices and their locations as coded in the data B2, B4 being specifically associated with this image), the final IF image is closer to the original IO image than the IUS image.

[0189] The IF image can then, for example, be displayed on a display device (for example at the second resolution).

[0190] Figures 3 to 7 present examples of convolution matrix (or kernel) usable within the filtering module 22 and the filtering unit 38.

[0191] In all these examples, we denote below x(i,j) the value of a pixel in row i and column j in the image (or more generally in the set of values) to which (to which) the convolution defined by this convolution matrix is ​​applied, and we denote y(i,j) the value of the pixel in row i and column j in the image (or more generally in the set of values) obtained by application of the convolution.

[0192] Fig. 3 represents a first example of a convolution matrix.

[0193] In the example described here, the location of the coefficients in this convolution matrix of [Fig.3] is defined by the default pattern; it is therefore the convolution matrix used when the use_default_loc flag is 1 for a given convolution.

[0194] The processing performed by this convolution is written:

[0195] y(i,j) = al.x(il,jl) + a2.x(il, j) + a3.x(il,j+l) + a4.x(i,jl) + a5.x(i,j)

[0196] + a6.x(i,j+l) + a7.x(i+l,jl) + a8.x(i+l,j) + a9.x(i+l,j+l).

[0197] Two predefined pattern forms are used in the following as examples:

[0198] - an X-shaped pattern, the shape of which is defined here by an ID1 identifier;

[0199] - a cross pattern, the shape of which is here defined by an ID2 identifier.

[0200] Other predefined pattern forms can of course be used in practice.

[0201] For the convolution matrix examples given below with reference to Figures 4 to 7, the use_default_loc flag is 0 in the example described (since, in these examples, the weights used are not positioned in the convolution matrix according to the default pattern used for [Fig.3]).

[0202] Fig. 4 represents a second example of a convolution matrix.

[0203] In the example described here, the location of the coefficients in this convolution matrix is ​​defined by the identifier ID1 (associated here with the pattern shape in X) and by a scope parameter equal to 1 (meaning that the pattern defined by the identifier ID1 is used as such or in other words, by applying a homothety with a ratio of 1).

[0204] The processing performed by this convolution is written:

[0205] y(ij) = al.x(il,jl) + a2.x(il,j+l) + a3.x(i,j) + a4.x(i+l,jl) + a5.x(i+l,j+l).

[0206] Fig. 5 represents a third example of a convolution matrix.

[0207] In the example described here, the location of the coefficients in this convolution matrix is ​​defined by the identifier ID1 (associated here with the pattern shape in X) and by a range parameter equal to 2 (meaning that the pattern defined by the identifier ID1 is used transformed by applying a homothety centered on the central coefficient, here a3, and with a ratio of 2).

[0208] The processing performed by this convolution is written:

[0209] y(i,j) = al.x(i-2,j-2) + a2.x(i-2,j+2) + a3.x(i,j) + a4.x(i+2,j-2) + a5.x(i+2,j+2).

[0210] Fig. 6 represents a fourth example of a convolution matrix.

[0211] In the example described here, the location of the coefficients in this convolution matrix is ​​defined by the identifier ID2 (associated here with the cross pattern shape) and by a range parameter equal to 1 (meaning that the pattern defined by the identifier ID2 is used as such or in other words, by applying a homothety with a ratio of 1).

[0212] The processing performed by this convolution is written:

[0213] y(i,j) = al.x(il,j) + a2.x(i,jl) + a3.x(i,j) + a4.x(i,j+l) + a5.x(i+l,j).

[0214] The [Fig.7] represents a fifth example of a convolution matrix.

[0215] In the example described here, the location of the coefficients in this convolution matrix is ​​defined by two identifier-parameter pairs (and therefore by superposition of a first motif and a second motif):

[0216] - the identifier ID1 (associated here with the X-shaped pattern) and a scope parameter equal to 1, meaning that the first pattern is the pattern defined by the ID1 identifier used as is;

[0217] - the identifier ID1 (associated here with the X-shaped pattern) and a scope parameter equal to 2, meaning that the second defined pattern is the pattern defined by the identifier ID1 transformed by application of a homothety centered on the central coefficient, here a5, and with a ratio of 2.

[0218] The processing performed by this convolution is written:

[0219] y(i,j) = al.x(i-2,j-2) + a2.x(i-2, j+2) + a3.x(il,jl) + a4.x(il,j+l) + a5.x(i,j)

[0220] + a6.x(i+l,jl) + a7.x(i+l,j+l) + a8.x(i+2,j-2) + a9.x(i+2,j+2).

[0221] As can be seen from the examples given above, only the location of the coefficients is defined by the location information (coded by the B3 data). The values ​​of the coefficients (denoted ak above, with k between 1 and 9) are given by the weights W, Wo (coded by the B2 data).

[0222] As already indicated, the weights Wo (denoted above as al, a2, etc.) are assigned to the coefficients defined by the location information L in a predefined order, for example by increasing row index (denoted i above) and, in each row, by increasing column index (denoted j above), as is the case in the examples in Figures 3 to 7.

Claims

Demands

1. A method for decoding data representative of audio or visual content, comprising the following steps: - decoding first data (B1) so as to obtain a signal (IDSdec) at a first resolution; - decoding second data (B2) so as to obtain a plurality of weights (Wo); - upsampling the signal (IDSdec) at the first resolution into a signal (IUS) at a second resolution higher than the first resolution; - filtering the signal (IUS) at the second resolution, the filtering comprising at least one convolution by means of a convolution matrix of which at least some of the coefficients are respectively the weights of the plurality of weights (Wo).

2. Method according to claim 1, comprising a step of decoding third data (B3) indicative of a location (L) of said weights within the convolution matrix.

3. A method according to claim 2, wherein the third data (B3) comprise fourth data defining the shape of a pattern at the level of which said weights are placed within the convolution matrix.

4. A method according to claim 3, wherein the fourth data includes an identifier that identifies said shape among a plurality of predetermined shapes.

5. A method according to claim 3 or 4, wherein at least some of the third data define an extent of said pattern within the convolution matrix.

6. A method according to any one of claims 1 to 5, wherein said filtering comprises a plurality of convolutions implemented respectively by means of a plurality of convolution matrices, each defined at least in part by weights obtained by decoding a portion of the second data.

7. A method according to claim 6 taken in dependence on any one of claims 2 to 5, wherein the third data includes, for each convolution matrix of the plurality of convolution matrices, data indicative of a location of the weights within the convolution matrix concerned.

8. A method according to claim 6 taken in dependence on any one of claims 2 to 5, wherein the third data comprise a number of convolutions for which the third data comprise data indicative of a location of the weights.

9. A method according to claim 6 taken in dependence on any one of claims 2 to 5, wherein the number of convolutions for which the third data includes data indicative of a weight location is predetermined.

10. A method according to claim 8 or 9, wherein the convolutions for which the third data include data indicative of a weight location are the first convolutions.

11. A method according to any one of claims 1 to 10, wherein said filtering comprises at least one convolution implemented by means of a predetermined convolution matrix.

12. A method according to any one of claims 1 to 11, wherein said at least one convolution is implemented by a layer of an artificial neural network.

13. A method according to any one of claims 1 to 12, comprising a data decoding step indicating a number of layers of the artificial neural network for which weights are coded among the second data.

14. A method according to any one of claims 1 to 13, wherein the second resolution is double the first resolution in each of the signal dimensions.

15. A method according to any one of claims 1 to 14, wherein the sound or visual content is an image, and wherein the first resolution and the second resolution are spatial resolutions.

16. A method according to claim 15, wherein the image is defined by several components, and wherein the decoding of the first data is followed by a step of conversion from a first color representation system to a second color representation system.

17. A method for encoding data representative of sound or visual content, comprising the following steps: - subsampling, into a signal (IDS) at a first resolution, of a signal (10) at a second resolution higher than the first resolution; - encoding the signal (IDS) at the first resolution so as to obtain first data (B1); - obtaining an intermediate signal (B) by decoding the first data (B1) and oversampling at the second resolution; - determining a plurality of weights (W) which minimize a criterion involving a distance between the signal at the second resolution (10), transformed by colorimetric conversion or not, and a signal (C) produced by filtering the intermediate signal (B) using at least one convolution by means of a convolution matrix of which at least some of the coefficients are respectively the weights of the plurality of weights (W); - encoding the determined weights (W) so as to obtain second data (B2).

18. A coding method according to claim 17, comprising, for each of a plurality of configurations of the weights within the convolution matrix, a step of determining a set of weights which minimizes a criterion involving a distance between the signal at the second resolution (10) and a signal produced by filtering the intermediate signal (B) using at least one convolution by means of a convolution matrix having the configuration concerned and defined by this set of weights, the coded weights being the weights of the set of weights for which the produced signal satisfies a predetermined criterion.

19. A coding method according to claim 18, comprising a third data coding step (B3) indicative of the location (L) of the weights within the convolution matrix in the configuration for which the produced signal satisfies the predetermined criterion.

20. Decoding device (30) for data representative of audio or visual content, comprising: - a decoding unit (32) configured to decode first data (Bl) so as to obtain a signal (IDSdec) at a first resolution and second data (B2) so as to obtain a plurality of weights (Wo); - an oversampling unit (36) configured to oversample the signal (IDSdec) at the first resolution into a signal (IUS) at a second resolution higher than the first resolution; - a filtering unit (38) configured to filter the signal (IUS) at the second resolution, the filtering unit (38) being configured to apply at least one convolution by means of a convolution matrix of which at least some of the coefficients are respectively the weights of the plurality of weights (Wo).

21. Encoding device (10) for data representative of audio or visual content, comprising: - a subsampling unit (12) configured to subsample, into a signal (IDS) at a first resolution, a signal (10) at a second resolution higher than the first resolution; - an encoding unit (14) configured to encode the signal (IDS) at the first resolution so as to obtain first data (Bl); - a decoding unit (16) configured to obtain a decoded signal (IDSdec) at the first resolution by decoding the first data (Bl); - an upsampling unit (18) configured to upsample the decoded signal (IDSdec), respectively transformed by colorimetric conversion or not, so as to obtain an intermediate signal (B) at the second resolution;- a learning unit configured to determine a plurality of weights (W) that minimize a criterion involving a distance between the signal at the second resolution (10), respectively transformed by colorimetric conversion or not, and a signal (C) produced by filtering the intermediate signal (B) using at least one convolution by means of a convolution matrix of which at least some of the coefficients are respectively the weights of the plurality of weights; wherein the coding unit (14) is configured to encode the determined weights (W) so as to obtain second data (B2).

22. A data stream representing audio or visual content, comprising first data (B1) representing a signal at a first resolution and second data (B2) representing

23. Weights usable as coefficients in a convolution matrix for filtering a second-resolution signal obtained by oversampling the first-resolution signal. Data stream according to claim 22, comprising third data (B3) indicating the location of said weights within the convolution matrix.

Images