Weighted image prediction and image encoding and decoding using such weighted prediction - Patents.com

Abstract

The present invention provides at least one set of current pixels (B c ) is predicted on the basis of at least one set of reference pixels (BR0;BR0,BR1) with the aid of a pixel prediction weighting function, c ), the method includes predicting at least one set of current pixels (B c ) is associated with at least one weighting value (P2-P3) calculated on the basis of an analysis (P1) of at least one set of reference pixels (BR0;BR0,BR1).

Description

[Technical field]

[0001] The present invention relates generally to the field of image processing, and more particularly to the encoding and decoding of digital images and sequences of digital images.

[0002] The digital image encoding / decoding applies in particular to images from at least one video sequence, including: - Images from the same camera and consecutive in time (2D encoding / decoding), - Images from different cameras oriented in different fields of view (3D encoding / decoding), - Supports texture and depth components (3D encoding / decoding), - and so on.

[0003] The invention applies equally to the encoding / decoding of 2D or 3D images.

[0004] The present invention may be particularly, but not exclusively, applied to video encoding and corresponding decoding implemented in current AVC, HEVC and VVC video encoders and their extensions (MVC, 3D-AVC, MV-HEVC, 3D-HEVC, etc.). [Background technology]

[0005] Current video encoders (MPEG, AVC, HEVC, VVC, AV1, etc.) use a block-based representation of a video sequence. An image is divided into blocks that can be divided again recursively. Then, each block is coded by intra- or inter-image prediction. Thus, some images are coded by spatial prediction (intra-prediction, IBC (Intra Block Copy) prediction) by using motion compensation well known to those skilled in the art, and other images are also coded by temporal prediction (inter-prediction) with respect to one or more coded-decoded reference images.

[0006] The prediction block BP associated with the currently coded block is determined by at least one reference block BR ​​of the image to which the currently coded block belongs (or of an image already decoded, conventionally called the reference image). 0 The reference block BR 0 In order to match the reference block BR 0 is displaced by the spatial location (x,y) of its pixel. Next, the motion-compensated reference block BC 0 Then, the predicted block BP and the motion compensated reference block BC 0 The relationship between is expressed as follows: BP(x,y)=(1-w)*BC 0 (x,y) where w is a prediction weighting parameter (most likely 0 but can be adjustable) described below.

[0007] For example, if the block currently being coded is a block of two reference blocks BR belonging to one or two already decoded reference images, 0 , B.R. 1 When the prediction is performed for two reference blocks BR 0 , B.R. 1 is motion compensated to produce two motion compensated reference blocks BC 0 , B.C. 1 Each pixel of the prediction block BP is divided into two motion compensated reference blocks BC 0 , B.C. 1 More precisely, if the prediction is performed row-wise and left-to-right, for example: The first pixel in the upper left corner of the prediction block BP is the pixel of the motion compensated reference block BC 0 and the first pixel in the upper left corner of the motion compensated reference block BC 1 is the weighted result of the first pixel in the top left of The second pixel adjacent to the first pixel on the first row of the prediction block BP is the second pixel of the motion compensated reference block BC 0A second pixel adjacent to the first pixel on the first row of the motion compensated reference block BC 1 is the result of weighting the second pixel adjacent to the first pixel on the first row of - and so on. The last pixel to the bottom right of the prediction block BP is the motion compensated reference block BC 0 the last pixel at the bottom right of 1 is the result of weighting the last pixel in the bottom right of

[0008] The most common weighting is the half-sum. To this end, the prediction block BP is calculated according to the following relationship: BP(x,y)=0.5*BC 0 (x,y)+0.5*BC 1 (x,y)

[0009] More elaborate weightings are possible.

[0010] In the HEVC standard, a linear weighting is applied uniformly to the currently coded picture. A weighting parameter w for each sub-picture or slice of the currently coded "picture" is fixed therein and signaled to the decoder. In the default setting, in case of bi-prediction of the currently coded block, a balanced weighting (0.5 / 0.5) is applied unless explicitly indicated in the PPS ("Picture Parameter Set") information.

[0011] In the VVC standard, the prediction is weighted block-wise by using the BCW ("bi-prediction with CU level weights") tool. The predicted block BP is calculated according to the following relationship: BP(x,y)=(1-w)*BC 0 (x,y)+w*BC 1 (x,y) Here, the weighting parameter w can take five values: 0.5, 0.625, 0.375, 1.25, -0.25. The optimal value of the weighting parameter w to be applied is determined in the encoder and signaled to the decoder for each block. The optimal value is the value used w=0.5 (i.e., the weighting parameter w for the motion compensated reference block BC 0 , B.C. 1 otherwise, the weighting is signaled on two bits to indicate one of the four remaining values.

[0012] This principle is adopted in AV1 technology.

[0013] It should be noted that "regardless of the video standard used, the weighting parameter w is associated with a relatively small number of values, which leads to a lack of precision in the weighting prediction applied." Furthermore, an encoder according to the above mentioned standards needs to systematically code and transmit the values ​​of the selected weighting parameter w to the decoder, which increases the signaling cost. Summary of the Invention [Problem to be solved by the invention]

[0014] One object of the present invention is to remedy the above-mentioned shortcomings of the prior art, advantageously by reducing the cost of signaling information involved in weighted prediction and improving the accuracy of this prediction over the prior art. [Means for solving the problem]

[0015] To this end, one subject of the invention relates to a method for predicting at least one current pixel set, implemented by a prediction device, said at least one current pixel set being predicted on the basis of at least one reference pixel set by using a pixel prediction weighting function, the method being characterized in that the pixel prediction weighting function for said at least one current pixel set is associated to at least one weighting value calculated on the basis of an analysis of the at least one reference pixel set.

[0016] Such a prediction method according to the invention advantageously makes it possible to rely only on one or more reference pixel sets (in other words, one or more sets of pixels already decoded at the time of prediction) to estimate the prediction weights of the current pixel set. Since this or these reference pixel sets are available at the time of prediction of the current pixel set, the estimation of the prediction weights is improved as it is more spatially accurate than that performed in the prior art, which requires approximating or quantizing the prediction weight value or values.

[0017] According to one particular embodiment, the prediction weighting function is modified by using at least one modification parameter resulting from an analysis of said at least one current set of pixels.

[0018] Such an embodiment advantageously makes it possible to apply a correction to the calculated prediction weighting function when the current pixel set contains elements / predictable elements that were not present in one or more reference pixel sets.

[0019] The present invention also relates to a device for predicting at least one current pixel set, the device including a processor configured to predict the at least one current pixel set on the basis of at least one reference pixel set by using a pixel prediction weighting function.

[0020] Such a prediction device is characterized in that a pixel prediction weighting function of said at least one current pixel set is associated to at least one weighting value calculated on the basis of an analysis of at least one reference pixel set.

[0021] In one particular embodiment, the prediction device is a neural network.

[0022] The use of a neural network advantageously makes it possible to optimize the quality of the weighted prediction.

[0023] Such a prediction device is in particular capable of implementing the prediction method described above.

[0024] The invention also relates to a method for decoding at least one current set of pixels, the method being implemented by an encoding device, the method comprising: - calculating a signal representative of a difference between said at least one current set of pixels and a predicted set of pixels resulting from a prediction of said at least one current set of pixels; - Encoding this signal.

[0025] Such a coding method is characterized in that the set of predicted pixels is obtained by using the above-described prediction method according to the invention.

[0026] Such an encoding method is advantageous in that it does not require the encoding of one or more prediction weighting values ​​of the prediction weighting function. This means that this or these prediction weighting values ​​do not need to be transmitted by the encoder to the decoder of the current set of pixels, thereby making it possible to reduce the cost of signaling information transmitted between the encoder and the decoder in favor of a better quality of the image related to an improved accuracy of the prediction. Furthermore, any weighting values ​​associated with the prediction weighting function do not need to be approximated or quantized for the purpose of being transmitted to the decoder, thereby making it possible to make these weighting values ​​continuous as a set of pixels is predicted.

[0027] According to one particular embodiment, the encoding method comprises: - encoding at least one modification parameter of a prediction weighting function; - transmitting said at least one modification parameter to an image decoder.

[0028] The invention also relates to an encoding device or encoder for decoding at least one current pixel set, comprising a processor configured to: - calculating a signal representative of a difference between said at least one current set of pixels and a predicted set of pixels resulting from a prediction of said at least one current set of pixels; - encoding said signal.

[0029] Such a coding device is characterized in that the set of predicted pixels is obtained by using the above-mentioned prediction device according to the invention.

[0030] Such a coding device is in particular capable of implementing the coding method described above.

[0031] The present invention also relates to a method for decoding at least one current pixel set implemented by a decoding device, the method comprising: - determining in a data signal a data representation of a signal representative of a difference between said at least one current set of pixels and a predicted set of pixels resulting from a prediction of said at least one current set of pixels; - reconstructing said at least one current pixel set from the determined data and a predicted pixel set.

[0032] Such a decoding method is characterized in that the set of predicted pixels is obtained by using the above-described prediction method according to the invention.

[0033] The advantage of such a decoding method lies in the fact that the prediction weighting function is calculated autonomously by the decoder on the basis of one or more available reference pixel sets, without the decoder having to read any specific information from the data signal received from the encoder. Moreover, as already explained above, at least one weighting value of the prediction weighting function is not coded or transmitted in the data signal, as is the case in the prior art, and can therefore be made continuous without having to be approximated or quantized.

[0034] In one particular embodiment, such a decoding method further comprises: - determining at least one modification parameter of the prediction weighting function within the data signal or within a further data signal; - calculating a modified prediction weighting function based on an analysis of said at least one reference pixel set and said at least one determined modification parameter.

[0035] The invention also relates to a decoding device or decoder for decoding at least one current pixel set, comprising a processor, the processor being configured to: - determining in a data signal a data representation of a signal representative of a difference between said at least one current set of pixels and a predicted set of pixels resulting from a prediction of said at least one current set of pixels; - reconstructing said at least one current pixel set from the determined data and a predicted pixel set.

[0036] Such a decoding device is characterized in that the set of predicted pixels is obtained by using the above-mentioned prediction device according to the invention.

[0037] Such a decoding device is in particular capable of implementing the above mentioned decoding method.

[0038] The invention also relates to a method implemented by a video data processing device for constructing at least one set of pixels from at least one reference pixel set.

[0039] Such a construction method is characterized in that a set of pixels is constructed by using a pixel prediction weighting function (such as the prediction function used in the above-mentioned prediction method of the invention).

[0040] The prediction weighting function of the present invention is therefore not limited to the context of image prediction, either generating or not generating a prediction residual, but may also be advantageously used in the case of interpolation or image synthesis based on one or more already decoded reference images.

[0041] The invention also relates to a computer program comprising instructions for implementing the prediction method according to the invention, and also to an encoding or decoding method incorporating the prediction method according to the invention, or the above-mentioned construction method according to any one of the particular embodiments described above, when said program is executed by a processor.

[0042] Such instructions may be permanently stored in a non-transitory storage medium of an encoder implementing the above-mentioned encoding method, a decoder implementing the above-mentioned decoding method, a video processing device implementing the above-mentioned construction method, or a prediction device implementing the above-mentioned prediction method.

[0043] This program may use any programming language and may be in the form of source code, object code, or an intermediate code between source code and object code (such as a partially compiled form), or any other desired form.

[0044] The invention is also directed to a computer readable recording medium or information medium containing instructions for the above-mentioned computer program.

[0045] The recording medium may be any entity or device capable of storing a program. For example, the medium may include storage means such as ROM (e.g. CD-ROM, DVD-ROM, synthetic DNA (deoxyribonucleic acid), etc., or microelectronic circuit ROM), or else magnetic recording means (e.g. USB key, hard disk).

[0046] Furthermore, the recording medium may be a transmittable medium, such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, or by radio or by other means. The program according to the invention may in particular be downloaded from a network, such as the Internet.

[0047] Alternatively, the recording medium may be an integrated circuit in which the program is incorporated, the integrated circuit being designed to perform or be used in the performance of the prediction method, encoding method, decoding method or construction method described above.

[0048] Other features and advantages become apparent from a reading of particular embodiments of the invention given as illustrative and non-limiting examples and in the accompanying drawings, in which: [Brief description of the drawings]

[0049] [Figure 1] 2 illustrates the main steps of the image prediction method according to the invention. [Figure 2A] FIG. 2 shows a prediction of the type used in the prediction method of FIG. 1 in a first particular embodiment of the invention. [Figure 2B] 2 shows a prediction of the type used in the prediction method of FIG. 1 in a second particular embodiment of the invention; [Figure 3A] 2 shows a prediction device for implementing the prediction method of FIG. 1 in a first embodiment. [Figure 3B] 2 shows a prediction device for implementing the prediction method of FIG. 1 in a second embodiment; [Figure 4] 3B shows in more detail some steps of the prediction method implemented by the prediction device of FIG. 3A. [Figure 5A] 4 illustrates a first example displacement of a predicted version of a current pixel set relative to two reference pixel sets. [Figure 5B] 13 illustrates a second example displacement of a predicted version of a current pixel set relative to two reference pixel sets. [Figure 5C] 13 illustrates a third example displacement of a predicted version of a current pixel set relative to two reference pixel sets. [Figure 5D] 5B illustrates motion compensation performed for the type of displacement of FIG. 5A in one particular embodiment of the invention. [Figure 5E] 5D illustrates weighted prediction performed at the end of the motion compensation of FIG. 5D in one particular embodiment of the present invention. [Figure 5F] 5C illustrates the total weighting performed at the end of the weighted prediction of FIG. 5E in one particular embodiment of the present invention. [Figure 5G] 5C illustrates a compensation weighting performed at the end of the total weighting of FIG. 5F in one particular embodiment of the present invention. [Figure 5H] 5C illustrates obtaining a predicted version of the current pixel set from the compensated weighting shown in FIG. 5G in one particular embodiment of the present invention. [Figure 6] 2 shows the main steps of an image coding method implementing the prediction method of FIG. 1 in one particular embodiment of the invention; [Figure 7A] 7 shows an encoder for implementing the encoding method of FIG. 6 in the first embodiment. [Figure 7B] 7 shows an encoder for implementing the encoding method of FIG. 6 in a second embodiment. [Figure 8] 2 shows the main steps of an image decoding method implementing the prediction method of FIG. 1 in one particular embodiment of the invention; [Figure 9A] 9 shows a decoder for implementing the decoding method of FIG. 8 in the first embodiment. [Figure 9B] 9 shows a decoder implementing the decoding method of FIG. 8 in a second embodiment. [Figure 10] 2 shows the steps of an image coding method implementing a modification of the prediction method of FIG. 1 in one particular embodiment of the invention; [Figure 11] 11 shows an encoder implementing the encoding method of FIG. 10 in one particular embodiment of the invention. [Figure 12] 2 shows the steps of an image decoding method implementing a modification of the prediction method of FIG. 1 in one particular embodiment of the invention. [Figure 13] 13 shows a decoder implementing the decoding method of FIG. 12 in one particular embodiment of the invention. [Figure 14] 11 shows an encoder implementing the encoding method of FIG. 10 in accordance with another particular embodiment of the present invention. [Figure 15] 13 shows a decoder implementing the decoding method of FIG. 12 in another particular embodiment of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0050] Exemplary implementation of the image prediction method General principles of the invention Image weighted prediction method Below is given a description of a 2D or 3D image prediction method that can be implemented in any type of video encoder or decoder, e.g. compliant with the AVC, HEVC, VVC standards and their extensions (MVC, 3D-AVC, MV-HEVC, 3D-HEVC, etc.), or e.g. in a convolutional neural network (or CNN).

[0051] Referring to FIG. 1, the prediction method according to the present invention uses at least one reference pixel set BR 0 (i.e., a reference pixel set that has already been coded and decoded and is therefore available at the time of prediction). One feature of the prediction method according to the invention is that the current pixel set B c is not taken into account for the prediction because it is not available at the time of the prediction.

[0052] Within the meaning of the present invention, the current pixel set B c is understood to mean: - original current image; - a part or region of the original current image, and - the blocks of the current image resulting from the division of this image according to what is done in a standardized AVC, HEVC or VVC encoder.

[0053] According to the present invention, as shown in FIG. 2A, a reference pixel set BR 0 is the current pixel set B c Current image I including i In this case, the current pixel set B c is a reference pixel set BR by using intra prediction (e.g., intra or IBC ("intrablock replication")). 0 is predicted for

[0054] Of course, the current pixel set B c is the reference pixel set BR 0 For and current image I i, the pixel values ​​may be predicted relative to one or more other reference pixel sets belonging to

[0055] According to the present invention, as shown in FIG. 2B, a reference pixel set BR 0 is the current image in time, I i In this case, the current pixel set B c By using inter prediction, the reference pixel set BR 0 In the example shown, the current pixel set B c By using type P unidirectional inter prediction, the reference pixel set BR 0 and a reference pixel set BR 0 For example, the previous image IR i-1 , but of course it belongs to another reference image (e.g. image IR i-2 or image IR in coding order i-2 Of course, one or more other reference pixel sets may belong to the current pixel set B. c A reference pixel set BR 0 may be used in conjunction with

[0056] In the example shown, the current pixel set B c Also, the current image I i A reference pixel set BR located in the reference image preceding 0 For and current image I i at least one other set of reference pixels BR located in the reference image 1 For the reference pixel set BR 0 is the reference image IR i-2 The reference pixel set BR 1 is the reference image IR i+1 Still within the context of such type B inter prediction, as shown in FIG. 2B, c is the current image I iTwo reference pixel sets BR 0 , B.R. 1 In the example shown, the pixel set BR 0 is the reference image IR i-2 The reference pixel set BR 1 is the reference image IR i-1 is placed within.

[0057] Of course, one or more other reference pixel sets may be used to predict the current predicted pixel set BP c The reference pixel set BR is used to calculate 0 , B.R. 1 may be used in conjunction with

[0058] In the embodiments presented below, one or more reference pixel sets BR 0 ,BR 1 etc. is the predicted current pixel set B c Of course, depending on the prediction context, the current pixel set B to be predicted c It is also possible to oversize this reference pixel set to cover an area greater than or equal to the area of ​​interest.

[0059] Referring again to FIG. 1, such a prediction method according to the present invention includes:

[0060] P1, the at least one reference pixel set BR 0 Such an analysis is carried out by using the at least one previously displaced reference pixel set BR 0 and the current pixel set B that is not available at the time of prediction. c Prediction version of BP c The motion estimation includes estimating pixel shifts between the pixel BRC and the pixel BRC. This motion estimation includes conventional motion compensation, at the end of which a set of motion compensated pixels BRC is obtained. 0 During this analysis, the BRC 0 BR in the direction of 0 Displacement vector V describing the displacement of0 Displacement information such as the above is obtained.

[0061] In P2, a set of motion compensated pixels BRC 0 The weighting value w of each pixel of 0 is calculated depending on the results of the analysis performed in P1.

[0062] In P3, the motion compensated reference pixel set BRC 0 Consider the current pixel set B for each coordinate (x,y) of the pixel c Prediction version of BP c is calculated according to the following function: BP c (x,y)=w 0 (x,y)*BRC 0 (x,y)

[0063] Two reference pixel sets BR 0 ,BR 1 are taken into account during the analysis P1, which results in two motion compensated pairs of pixels BRC 0 , B.R.C. 1 and the corresponding displacement vector V 0 (V 1 ) is generated, a set of motion compensated pixel BRCs 0 and a set of motion compensated pixels BRC 1 The two weighting values ​​w 0 , w 1 are calculated at P2 respectively. Next, the current predicted pixel set BP c is calculated in P3 for each pixel coordinate (x,y) taking into account the motion compensated set of pixels according to the following function: BP c (x,y)=w 0 (x,y)*BRC 0 (x,y)+w 1 (x,y)*BRC 1 (x,y)

[0064] The prediction method just described above can then be performed for each current set of pixels to be predicted (which are considered unavailable at the time of prediction).

[0065] Exemplary implementation of a weighted prediction device FIG. 3A shows a prediction device PRED1 suitable for implementing the prediction method shown in FIG. 1 according to a first embodiment of the invention.

[0066] According to this first embodiment, the actions performed by the prediction method are implemented by computer program instructions. To this end, the prediction device PRED1 has a conventional architecture of a computer and includes in particular a memory MEM_P1 and a processing unit UT_P1, for example equipped with a processor PROC_P1 and driven by a computer program PG_P1 stored in the memory MEM_P1. The computer program PG_P1 includes instructions for implementing the actions of the prediction method as described above when executed by the processor PROC_P1.

[0067] At initialization, the code instructions of the computer program PG_P1 are for example loaded into a RAM memory (not shown) before being executed by the processor PROC_P1. The processor PROC_P1 of the processing unit UT_P1 performs in particular the acts of the prediction method described above according to the instructions of the computer program PG_P1.

[0068] The prediction device uses one or more reference pixel sets BR 0 ,BR 1 etc. at input E_P1, and one and / or multiple corresponding weighting values ​​w 0 , w 1 etc. together with one and / or multiple corresponding displacement vectors V 0 , V 1 and calculate the predicted pixel set BP c at output S_P1.

[0069] FIG. 3B shows a prediction device PRED2 suitable for implementing the prediction method shown in FIG. 1 according to a second embodiment of the invention.

[0070] According to this second embodiment, the prediction device PRED2 is a neural network (denoted RNC1), for example a convolutional neural network, a multilayer perceptron, an LSTM ("Long Short Term Memory"), which predicts at its input one or more reference pixel sets BR 0 ,BR 1 Then, steps P1 to P2 of the weighted prediction method in FIG. 1 are carried out: one or more respective motion vectors V 0 , V 1 etc., and - one or more respective weighting values ​​w 0 , w 1 etc.

[0071] The computer CAL predicts the pixel set BP according to the above relationship. c We receive this information at input to calculate: BP c (x,y)=w 0 (x,y)*BRC 0 (x,y)+w 1 (x,y)*BRC 1 (x,y)

[0072] In a manner known per se, the convolutional neural network RNC1 performs a series of layers of filtering, nonlinear and scaling operations. Each filter used is parameterized by a convolution kernel, the nonlinearity of which is parameterized (ReLU, leaky ReLU, GDN (generalized divisive normalization), etc.). The neural network RNC1 is for example of the type described in the non-patent document D. Sun, et al., “PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume”, CVPR 2018.

[0073] In this case, the neural network RNC1 may be trained to: - BR to the current image being coded to make a prediction 0 , B.R. 1 To interpolate the motion from V 0 , V 1 To estimate, etc.; and - weighting value w 0 , w 1 To estimate, etc.

[0074] To this end, in a preliminary phase, a network RNC1 is trained to perform an operation P1. For example, the network RNC1 is trained to perform an operation P1 on the image I i and the weighted prediction results in Fig. 1 BP c At the end of this process, V 0 , V 1 etc. and w 0 , w 1 A latent space U representing the above is obtained.

[0075] Network RNC1 uses the current pixel set B c Together with a number of associated reference pixel sets BR 0 ,BR 1 By presenting, etc., and B c and the result BP calculated according to the above relationship c During the training phase, the network is trained by varying the weights (eg, by using a gradient descent algorithm) to minimize the mean squared error between (x,y).

[0076] At the end of this preliminary training phase, the network RNC1 is fixed and suitable for use in the prediction device PRED2.

[0077] The network RNC1 is therefore optimised to implement the weighted prediction steps P1, P2 of FIG.

[0078] An embodiment of the weighted prediction method implemented by the prediction device PRED1 Next, a description of an embodiment is given with reference to Fig. 4 and Figs. 5A to 5H, in which the prediction weighting of the current pixel set is implemented in the prediction device PRED1 of Fig. 3A.

[0079] In the example shown, there are two reference pixel sets BR 0 ,BR 1 are considered for the prediction.

[0080] To this end, as shown in FIG. 4, the analysis P1 of at least one reference pixel set comprises: At P10, BR 0 and B.R. 1 A motion estimate between is calculated. Such a process can be performed via a conventional motion search process (e.g., estimating displacement vectors, etc.).

[0081] 5A-5C show two reference pixel sets BR that may be encountered during this step P10. 0 ,BR 1 Current pixel set B for c Prediction version of BP c 5A-5C, three different exemplary displacements of the reference pixel set BR 0 and B.R. 1 The displacement of an element E (symbolized by a circle) between 01 and shown as a dotted line), whereas in the example shown, BR 0 From BR 1 (The motion with respect to other parts of the image is assumed to be zero.) However, as in the case of optical flow motion estimation, for example, the reference pixel set BR 0 ~BR 1 It goes without saying that there are as many motion vectors as there are pixels representing BR. 1 From BR 0Vector V describing the (opposite) movement towards 10 may be calculated.

[0082] Vector V obtained at P10 and P11 (Figure 4) 01 Or V 10 BR 0 and B.R. 1 The current pixel set B predicted for c This estimation is illustrated in Figures 5A-5C, where the displacement of element E is calculated based on the current pixel set B c BR, BR are the times when 1 is estimated at a time other than the time at which it was placed. 01 Or V 10 By using the same notation for the computation of: - Single Vector V 0 (BR 0 From B c ) to the predicted position of the vector V 01 It is calculated from -BR 1 From B c A single vector V describing the movement of 1 is a vector V 01 It is calculated from.

[0083] Current pixel set B c BR 0 and B.R. 1 In the example of FIG. 5A, the displacement of element E at the current time is 0 and B.R. 1 1 / 2 of the displacement between (i.e., vector V 01 Or V 10 Such a displacement configuration is estimated to correspond to a 1 / 2 of the BR 0 is the reference image IR i-1 Belongs to BR 1 is the reference image IR i+1 This is often encountered when adopting a

[0084] Current pixel set B cBR 1 From BR 0 In the example of FIG. 5B, the displacement of element E at the current time is BR 0 and B.R. 1 It is estimated that the distance between the BR and the 0 is the reference image IR i-1 Belongs to BR 1 is the reference image IR i+2 If the element E belongs to BR, the displacement of the element E at the current time is 0 and B.R. 1 1 / 3 of the displacement between 01 Or V 10 It is estimated that this corresponds to 1 / 3 of the

[0085] Current pixel set B c In terms of time, BR 0 (Reference image IR i-2 belongs to), then BR 1 (Reference image IR i-2 In the example of FIG. 5C, the displacement of element E at the current time is BR 0 and B.R. 1 twice the displacement between 01 Or V 10 It is estimated that the number of

[0086] Referring to FIG. 4 and FIG. 5D, in P12, BR 0 and B.R. 1 are B c Two forecast versions of (BRC 0 and B.R.C. 1 To generate the vector V 0 , V 1 The motion is compensated by using

[0087] As shown in FIG. 5D, the vector V 0 , V 1 is the displacement of element E at the current time BR 0 and B.R. 1 1 / 2 of the displacement between (i.e., vector V 01 Or V 10The motion configuration shown in FIG. 5A is assumed to correspond to a time period of 1 / 2 of the time period.

[0088] FIG. 5D shows: - The interpolated position of element E is the vector V 0 By BR 0 A set of pixels ERC resulting from the motion compensation of element E of 0 A set of right motion compensated pixels BRC including 0 , and - The interpolated position of element E is the vector V 1 By BR 1 A set of pixels ERC resulting from the motion compensation of element E of 1 A set of left motion compensated pixels BRC including 1 .

[0089] In contrast, ERC 0 Part Z of 0 and ERC 1 Part Z of 1 BR 0 Elements E and BR 1 However, as can be seen in FIG. 5D, the part Z 0 is ERC 1 is defined in the part Z 1 is ERC 0 It is defined within.

[0090] Next, the predicted weight value w 0 , w 1 The calculation of P2 includes: 4 and 5E, the motion compensated reference pixel set BRC according to the present invention is 0 , B.R.C. 1 In order to achieve an efficient combination of 0 and B.R.C. 1 The intermediate weighting w int 0 (x,y) and w int 1 This involves calculating (x,y) for each pixel.0 and B.R.C. 1 BRs that are not defined in 0 Part and BR 1 This allows the separation of the parts.

[0091] To achieve this goal, as shown in FIG. 5E, it is proposed that: - Part Z shown in white 0 and Z 1 To assign the weighting value of each pixel of w int 0 (x,y)=0 and w int 1 (x,y)=0), - a motion-compensated reference pixel set BRC corresponding to the grey in FIG. 5E; 0 and B.R.C. 1 Assigning a weighting value for each defined pixel of to a constant K (i.e., w int 0 (x,y)=K and w int 1 (x,y)=K (for example, K=0.5).

[0092] At the end of this operation, what is obtained is a motion compensated reference pixel set BRC 0 and B.R.C. 1 and the white content is w int 0 (x,y)=0 and w int 1 Corresponding to (x,y)=0, the gray content w int 0 (x,y)=0.5 and w int 1 Corresponds to (x,y)=0.5.

[0093] Referring to FIG. 4, P21 is the intermediate weight w int 0 (x,y) and w int 1 (x,y) and the sum w s This involves computing (x,y), where

number

[0094] Intermediate weighting w int 0 (x,y) and w int 1 The calculation of the sum with (x,y) is shown in FIG. 5F, where the black content is w s Corresponding to (x,y)=1, the grey content is w s Corresponds to (x,y)=0.5.

[0095] Next, calculating the prediction weighting function P3 includes: Referring to FIG. 4, during operation P30, a motion compensated predictive BRC 0 and B.R.C. 1 are the weights of each 0 and w 1 Next, the weighted compensated predictive BRCW 0 and BRCW 1 is obtained.

[0096] To achieve this goal, the following compensation weights w 0 (x,y) and w 1 (x,y) is the motion compensated reference pixel set BRC 0 , B.R.C. 1 Each is calculated separately:

number

number

[0097] Such compensation weighting is shown in FIG. 5G. White content is represented by w 0 (x,y)=0 and w 1 Undefined region Z where (x,y)=0 0 and Z 1 The black content corresponds to w 0 (x,y)=1 and w 1The gray content corresponds to the fully defined region where (x,y)=1. 0 (x,y)=0.5 and w 1 This corresponds to the background and element E where (x,y)=0.5.

[0098] Next, the predictive BRC 0 and B.R.C. 1 are motion compensated and weighted by w 0 and w 1 Next, the weighted compensated predictive BRCW 0 and BRCW 1 is obtained.

[0099] 4 and 5H, during operation P31, the weighted compensated predicted BRCW 0 and BRCW 1 are summed together to produce a motion compensated predicted pixel set BP c to provide.

[0100] Therefore, this is done only at the time of prediction (i.e., the reference pixel set BR 0 and B.R. 1 We construct a motion compensation scheme that includes compensation weights determined by the elements present (only above). One particular advantage of such compensation weights compared to currently standardized solutions is that: 0 and B.R. 1 The present invention uses only the fully known BR 0 and B.R. 1 Therefore, the above-mentioned occluded region Z 0 and Z 1 can be recovered with very good spatial accuracy, and it is possible to effectively handle disocclusion during prediction as shown in Figures 5D-5H.

[0101] The prediction weights may be expressed in other forms. For example, a single weight w may be used. Then, w is the motion-compensated reference pixel set BRC 0 where (1-w) is the motion-compensated reference pixel set BRC 1Weight the following:

[0102] w is the pre-calculated weighting value w 0 and w 1 Based on the above embodiment starting from, it can be calculated by performing the following formula:

number

[0103] Although a prediction method has been described above, such a method may also be used to predict one or more reference pixel sets BR by using the weighted prediction function just described. 0 ,BR 1 And so on, a set of pixels B c For example, a set of pixels B corresponding to a set of missing pixels or a set of pixels not captured by the camera (360° video) may be constructed or synthesized. c The prediction method of may be called a construction or synthesis method. In this case, strictly speaking, the prediction devices PRED1 and PRED2 would rather be considered as construction or synthesis devices. The prediction device PRED1 may thus implement a so-called view "synthesis" algorithm. For example, the VSRS ("View Synthesis Reference") software, the VVS ("Versatile View Synthesizer") algorithm may be used as a view synthesis algorithm. The construction or synthesis device PRED2 itself may be a neural network, as mentioned above, for example a convolutional neural network, a multi-layer perceptron, a LSTM ("Long Short-Term Memory"), etc.

[0104] Image encoding method general principle A description of an image coding method implementing the weighted prediction described with reference to FIG. 1 is given below with reference to FIG.

[0105] Such encoding methods include: In C1, the weighted prediction of steps P1 to P3 shown in FIG. 1 is performed, and the current predicted pixel set BP c Generate.

[0106] The following encoding process is conventional and conforms to AVC, HEVC, VVC encoding, etc. Thus: - In C2, the current pixel set B c and the current predicted pixel set BP obtained in C1 c The BE of the signal that represents the difference between c is calculated; - In C3, this signal BE c is quantized or coded when optimizing the coding against conventional coding performance criteria (e.g., minimizing distortion / data rate cost, etc., or otherwise choosing the best efficiency / complexity tradeoff, criteria well known to those skilled in the art).

[0107] At the end of this operation, the quantized or coded difference signal BE c cod is obtained.

[0108] During the encoding C3, multiple encoding possibilities may be explored, e.g., multiple prediction weighting values ​​w 0 , w 1 etc. may be investigated to find the best data rate / distortion or efficiency / complexity tradeoff.

[0109] The encoder may place the weighting values ​​calculated according to the prediction method described above in competition with the weighting values ​​it may choose to transmit. To this end, the encoder may use the predicted BP obtained from the prediction method. c The quality of the prediction error can be evaluated, for example by using the root mean square error to measure the prediction error. The prediction error is determined by a set of pairwise predefined weighting values ​​(w 0 ,w 1) may be compared to the prediction error resulting from a 1:1 CU level prediction. This set may be restricted, for example, to (0.5,0.5) in the HEVC (High Efficiency Video Coding) standard, or may include other values, such as those used in the BCW ("Bi-prediction with CU level Weights") tool of the VVC ("Versatile Video Coding") standard. A flag will instruct the decoder whether to use the prediction methods described above, or whether to apply the VVC BCW tool, or whether to apply the HEVC balanced prediction.

[0110] Prediction weight value w 0 , w 1 etc. has the advantage of optimizing the accuracy of the weighted prediction compared to the default prediction weighting implemented in prediction devices from the prior art. In fact, the prediction weighting of the present invention produces more accurate weighting values, but may lead to higher distortion if the signal is not predictable. Conventional prediction weighting is less spatially accurate, but may lead to lower distortion at the expense of the transmitted data rate.

[0111] In C4, the quantized or coded difference signal BE c cod This data is written into a transport stream F that can be transmitted to a decoder (described later in this specification).

[0112] In contrast, according to the present invention, the weighting w 0 and / or weighting w 1 is advantageously not encoded or transmitted to the decoder.

[0113] Of course, the quantized or coded difference signal BE c If is zero, which may be the case in SKIP encoding mode, steps C2 and C4 described above are not performed.

[0114] Encoder Example Implementation Figure 7A shows an encoder COD1 suitable for implementing the encoding method according to a first embodiment of the invention and shown in Figure 6. The encoder COD1 comprises a prediction device PRED1.

[0115] According to this first embodiment, the actions performed by the encoding method are implemented by computer program instructions. To that end, the encoding device COD1 has a conventional architecture of a computer and includes in particular a memory MEM_C1 and a processing unit UT_C1, for example with a processor PROC_C1, which is computer driven by a program PG_C1 stored in the memory MEM_C1. The computer program PG_C1 includes instructions for implementing the actions of the above-mentioned encoding method when the program is executed by the processor PROC_C1.

[0116] At initialization, the code instructions of the computer program PG_C1 are for example loaded into a RAM memory (not shown) before being executed by the processor PROC_C1, which in particular performs the acts of the coding method described above according to the instructions of the computer program PG_C1.

[0117] Encoder COD1 takes the current pixel set B c at its input E_C1 and delivers at its output S_C1 a transport stream F which is transmitted to the decoder by using a suitable communication interface (not shown).

[0118] Fig. 7B shows an encoder COD2 suitable for implementing the encoding method shown in Fig. 6 according to a second embodiment of the invention. The encoder COD2 comprises the above-mentioned prediction device PRED2 followed by a convolutional neural network RNC2, which predicts for example a difference signal BE c and then encoding it, or the predicted pixel set BP generated by the prediction device PRED2 c along with the current pixel set Bc Such a network RNC2 is for example of the type described in the non-patent document Ladune “Optical Flow and Mode Selection for Learning-based Video Coding”, IEEE MMSP 2020.

[0119] Image Decoding Method general principle A description of an image decoding method implementing the weighted prediction described with reference to FIG. 1 is given below with reference to FIG.

[0120] Such a decoding method implements an image decoding corresponding to the image coding of figure 6. In particular, apart from weighted prediction, the decoding method implements a conventional decoding step, such as in accordance with AVC, HEVC, VVC decoding.

[0121] The decoding method includes: At D1, the encoded difference signal BE c cod The data is extracted from the received transport stream F in a conventional manner.

[0122] In D2, BE c cod is decoded in the conventional manner. At the end of this operation, the decoded difference signal BE c dec is obtained.

[0123] In D3, the weighted prediction according to the present invention in steps P1 to P3 shown in FIG. 1 is performed, and the current predicted pixel set BP c Generate.

[0124] In D4, the reconstructed current pixel set BD c is the decoded difference signal BE obtained at D2 c dec and the predicted pixel set BP obtained in D3 c It is calculated by combining

[0125] In a manner known per se, the reconstructed current pixel set BD c may possibly be subjected to filtering by a loop filter (well known to those skilled in the art) performed on the reconstructed signal.

[0126] Of course, the difference signal BE calculated during the encoding method described above c If is zero, which may be the case in the SKIP encoding mode, steps D1 and D2 described above are not performed.

[0127] Decoder Example Implementation Fig. 9A shows a decoder DEC1 suitable for implementing the decoding method shown in Fig. 8 according to a first embodiment of the invention. The decoder DEC1 comprises a prediction device PRED1.

[0128] According to this first embodiment, the actions performed by the decoding method are implemented by computer program instructions. To this end, the decoder DEC1 has a conventional architecture of a computer and includes in particular a memory MEM_D1 and a processing unit UT_D1, for example with a processor PROD_D1, which is driven by a computer with a program PG_D1 stored in the memory MEM_D1. The computer program PG_D1 includes instructions for implementing the actions of the above-mentioned decoding method when the program is executed by the processor PROC_D1.

[0129] At initialization, the code instructions of the computer program PG_D1 may for example be loaded into a RAM memory (not shown) before being executed by the processor PROC_D1, which in particular performs the acts of the decoding method described above in connection with FIG.

[0130] The decoder DEC1 receives at an input E_D1 the transport stream F transmitted by the encoder COD1 of FIG. 7A and outputs a current decoded set of pixels BD c is sent at output S_D1.

[0131] Fig. 9B shows a decoder DEC2 suitable for implementing the decoding method shown in Fig. 8 according to a second embodiment of the invention. The decoder DEC2 comprises the above-mentioned prediction device PRED2 followed by a convolutional neural network RNC3, for example to predict the encoded difference signal BE c cod Decode or decode the difference signal BE c cod The predicted pixel set BP generated by the prediction device PRED2 c Such a network RNC3 is for example of the type described in the non-patent document Ladune "Optical Flow and Mode Selection for Learning-based Video Coding", IEEE MMSP 2020.

[0132] Variations of the Image Weighted Prediction Method A description of a variant of the weighted prediction method shown in FIG. 1 implemented in the encoder COD3 is now given with reference to FIGS.

[0133] Such a variant aims to improve the weighted prediction method of FIG. 1 in cases where the accuracy / quality of the predictions obtained is not satisfactory.

[0134] To this end, on the encoder side, the at least one reference pixel set BR 0 is the current pixel set B c For example, two reference pixel sets BR 0 and B.R. 1 B c In the example shown, BR 0 is B in time. cIt is placed before BR 1 is B in terms of time c is placed after.

[0135] As shown in FIG. 11, the analysis C′1 is performed by dividing the two reference pixel sets BR 0 and B.R. 1 From and current pixel set B c This is implemented by using a convolutional neural network RNC4 that generates a transformation from a certain number of layers, such as layers implementing convolutional filters (CNN), followed by layers implementing nonlinearity and decimation.

[0136] At the end of step C'1, a set of latent variables is obtained in the form of a signal U'.

[0137] The signal U' is quantized in C'2 by a quantizer QUANT (e.g., a uniform or vector quantizer controlled by a quantization parameter). The quantized signal U' is then q is obtained.

[0138] In C'3, the quantized signal U' q is encoded by using an entropy encoder CE (e.g. of arithmetic type) with the determined statistics, which are for example parameterized by the probability of the statistics (e.g. by modeling the variance and mean (σ,μ) of the Laplace law, or else by considering a hyperprior as in the non-patent document "Variational image compression with a scale" by Balle at the ICLR 2018 conference). The encoded quantized signal U' q cod is obtained.

[0139] In C'4, the encoded quantized signal U' qcod is written into a transport stream F' which is sent to the decoder DEC3 (shown in FIG. 13).

[0140] Next, a description of a variant of the weighted prediction method shown in FIG. 1 implemented in the decoder DEC3 is given with reference to FIGS.

[0141] To this end, at the decoder side, in D′1, at least one reference pixel set BR 0 are analyzed (in the example shown, two sets of reference pixels BR 0 and B.R. 1 ). Such an analysis is identical to that carried out in step P1 of FIG. 1 by using the neural network RNC1. At the end of this step, V 0 , V 1 etc. and w 0 , w 1 A latent space U representing the above is obtained.

[0142] Following reception of the stream F', in D'2, an entropy decoding is performed using an entropy decoder DE corresponding to the entropy encoder CE of FIG. 11 to obtain an encoded quantized signal U' (having the same determined statistics (such as modeling the variance and mean (σ, μ) of the Laplace method) q cod The decoded quantized signal U' q is obtained at the end of this operation.

[0143] In D'3, the decoded quantized signal U' q is the latent space U obtained by the neural network RNC1 of FIG. 13 and the reference pixel set BR 0 and B.R. 1 It is concatenated with the latent space U, which represents the analysis of only

[0144] Next, the neural network RNC1 calculates the corresponding weights w 0 , w 1 Along with the above, movement information V0 , V 1 , etc., we process this connection through the various layers in a manner similar to step P2 of FIG.

[0145] In a manner corresponding to FIG. 3B, the computer CAL calculates the predicted pixel set BP according to the above-mentioned relationship. c This information is received at input to calculate: This step is identical to step P3 in FIG.

[0146] In the embodiment disclosed above with reference to FIG. 3A et seq., two reference pixel sets BR 0 ,BR 1 is used in the weighted prediction method.

[0147] These embodiments can be extended to more than two reference pixel sets. To this end, the neural network RNC1 described with reference to FIG. c Three or more reference pixel sets BR to obtain (x,y) 0 , B.R. 1 , B.R. 2 They will be trained from the following:

[0148] Of course, a low-level weighted prediction mode is possible, for example when only one reference coordinate system is used for prediction (type P prediction in video coding modes). Such a low-level mode is illustrated with reference to Figs. 14 and 15. The example of Fig. 14 shows an encoder COD3' similar to the encoder COD3 of Fig. 11. The encoder COD3' uses a single reference pixel set (BR in Fig. 14). 0 ) is different from the encoder COD3 through its neural network RNC4' which is trained from a single reference pixel set BR 0 and the current pixel set B c As already explained above in relation to Figures 10 and 11, the signal U' is converted into a quantized signal U' qThe quantized signal U' is quantized to obtain q is entropically coded. Then, the coded quantized signal U' q cod The encoded quantized signal U' is obtained. q cod are written into a transport stream F' which is sent to a decoder DEC3' (shown in FIG. 15).

[0149] The decoder DEC3' uses a single reference pixel set BR 0 13 through its neural network RNC1', which analyzes V. At the end of this analysis, 0 and w 0 As already explained with reference to FIG. 12, following reception of the stream F′, a latent space U is obtained which represents the encoded quantized signal U′ q cod is entropically decoded and the decoded quantized signal U' q The decoded quantized signal U' is generated. q is concatenated with the latent space U obtained by the neural network RNC1′. Then, the neural network RNC1′ obtains a single corresponding weight value w 0 Together with a single motion information V 0 The computer CAL processes this concatenation to estimate BR 1 was not taken into account and therefore was not motion compensated, so BRC 1 = 0 according to the above relationship with the predicted pixel set BP c We receive this information at input to calculate w in the conventional manner. 1 (x,y)=0.

[0150] This is the predicted pixel set BP obtained at the end of step P3 of FIG. 1 and delivered at the output of the computer CAL. c This means that: BP c (x,y)=w 0 (x,y)*BRC 0(x,y)+0*0=w 0 (x,y)*BRC 0 (x,y) [Explanation of symbols]

[0151] BR 0 , B.R. 1 , B.R. 2 Reference Pixel Set V 0 ,V 1 Corresponding Displacement Vector

Claims

1. At least one current pixel set (B ｃ ) based on said at least one current pixel set (B ｃ ) is weighted by using a pixel prediction weighting function to estimate at least one reference pixel set (BR) belonging to a reference image already decoded. ０ ;BR ０ , B.R. １ ), in which the at least one current pixel set (B ｃ The pixel prediction weighting function for at least one reference pixel set (BR ０ ;BR ０ , B.R. １ ) based on an analysis (P1) of the at least one weighting value (P2-P3).

2. At least one reference pixel set (BR ０ ;BR ０ , B.R. １ 2. The method of claim 1, wherein the analysis of (a) performs motion estimation or filtering of the at least one reference pixel set.

3. The prediction method of claim 2 , wherein the motion estimation is optical flow motion estimation.

4. The pixel prediction weighting function includes at least one modified parameter (U′) resulting from a joint analysis of the at least one current pixel set and at least one reference pixel set. ｑ 2. The method of claim 1, wherein the prediction is modified by using

5. A device for predicting at least one current pixel set, comprising a processor configured to predict the at least one current pixel set based on at least one reference pixel set belonging to an already decoded reference image by using a pixel prediction weighting function, the at least one current pixel set (B ｃ 4. A prediction device according to claim 1, wherein said pixel prediction weighting function for a pixel prediction step comprises at least one weighting value calculated based on an analysis of at least one reference pixel set.

6. A prediction device according to claim 5, characterized in that it uses a neural network (PRED2).

7. A computer program comprising program code instructions for implementing the prediction method according to any one of claims 1 to 4 when said computer program is executed on a computer.

8. A computer readable information medium comprising instructions for the computer program of claim 7.

9. A method for decoding at least one current set of pixels, implemented by a coding device (COD1; COD2; COD3; COD3'), comprising: - calculating (C2) a signal representative of the difference between said at least one current set of pixels and a predicted set of pixels resulting from a prediction of said at least one current set of pixels; - encoding (C3) said signal, In a method comprising: Coding method, characterized in that the set of predicted pixels is obtained (C1) by using a prediction method according to any one of claims 1 to 4.

10. - encoding (C'3) at least one modification parameter of said pixel prediction weighting function, - transmitting (C'4) said at least one coded modification parameter to an image decoder. The encoding method of claim 9, comprising:

11. 1. A device for decoding at least one current pixel set, comprising: - calculating a signal representative of the difference between said at least one current set of pixels and a predicted set of pixels resulting from a prediction of said at least one current set of pixels; - encoding said signal; 1. A device comprising a processor configured to implement A device, characterized in that the set of predicted pixels is obtained by using a prediction device according to claim 5 or 6.

12. 1. A method for decoding at least one current set of pixels, performed by a decoding device, comprising: determining (D1) in a data signal a data representation of a signal representative of a difference between said at least one current set of pixels and a predicted set of pixels resulting from a prediction of said at least one current set of pixels; - reconstructing (D4) said at least one current pixel set from said determined data and said predicted pixel set; In a method comprising:

5. A method of decoding, characterized in that the set of predicted pixels is obtained (D3) by using a prediction method according to any one of claims 1 to 4.

13. determining (D'2) at least one coding parameter in said data signal or in another data signal, said coding parameter being a modification parameter of said pixel prediction weighting function, - decoding said encoded modification parameters; - calculating a modified pixel prediction weighting function based on an analysis of said at least one reference pixel set and said at least one decoded modification parameter (D'3, P2 to P3); The method of claim 12 further comprising:

14. 1. A device for decoding at least one current set of pixels, comprising: determining, in a data signal, a data representation of a signal representative of a difference between said at least one current set of pixels and a predicted set of pixels resulting from a prediction of said at least one current set of pixels; - reconstructing said at least one current pixel set from said determined data and said predicted pixel set.

1. A device, comprising: a processor configured to implement A device, characterized in that the set of predicted pixels is obtained by using a prediction device according to claim 5 or 6.

15. A computer program comprising program code instructions for carrying out the encoding method according to claim 9 or 10 or the decoding method according to claim 12 or 13, when said computer program is executed on a computer.

16. A computer readable information medium comprising instructions for a computer program according to claim 15.

Images