Image and video coding method using continuous training of machine learning model for prediction

Abstract

The present application relates to the field of computer vision, in particular to the subject matter of video processing and video encoding, and more particularly to a method, decoder, encoder and computer readable medium for video encoding using a machine learning model. According to a first aspect, a method of processing image and / or video data by a decoder is provided. The method includes continuously decoding a plurality of segments of an image or video. For one or more of the plurality of segments, the decoding comprises: decoding a corresponding encoded residual segment to obtain a decoded residual segment; generating predicted segments from respective reference segments using a machine learning model, wherein the respective reference segments are selected from one or more reference segments stored in a set of reference segments; generating a reconstructed fragment based on the prediction fragment and the decoded residual fragment; storing the reconstructed fragment in the reference fragment set; and updating parameters of the machine learning model based on the corresponding reference segments and the reconstructed segments.

Description

Image and video coding methods that use machine learning models for continuous training and prediction Technical Field

[0001] This application relates to the field of computer vision, specifically to the topics of video / image processing and video / image coding, and more specifically to methods, decoders, encoders, and computer-readable media for video / image coding using machine learning models. Background Technology

[0002] Image / video compression typically employs image prediction and encoding of only the residual signal (prediction error). The encoding error (residual) can be encoded losslessly or lossily. Prediction is performed based on already encoded / transmitted segments of the image / video available to both the encoder and decoder. Generally, video / image coding techniques allow for the selection of prediction modes, which are signaled from the encoder to the decoder.

[0003] Multiple prediction modes are available for the encoder, allowing the encoder to select a better prediction mode and adapt it to the content being encoded. This limits the amount of information required to encode the residual signal, but it can also increase the amount of information required to encode the selected prediction mode.

[0004] Typically, a prediction scheme operates by using some prediction tools (T) to predict (P) new segments of the original image (O) based on previously encoded and reconstructed content (A). A prediction error E (also called a "residual signal" or "residual segment") is determined based on the original image segment O and the predicted segment P, for example, as: E = OP. The prediction error E is encoded (losslessly or lossily) to obtain E′. For example, the encoded prediction error is available to both the encoder and decoder after being transmitted over the network to the decoder. Information about the prediction tools T is also transmitted to the decoder (if more than one prediction tool exists). The considered image segment O is reconstructed (losslessly or lossily) to give a reconstructed segment R, for example: R = P + E′. The reconstructed segment R is included in the set of encoded content that can be used for further prediction.

[0005] The challenge is to achieve the best possible prediction quality, making the prediction error negligible, while also limiting the amount of information about the prediction pattern that needs to be encoded. Summary of the Invention

[0006] Embodiments of this application provide a video encoding method, decoder, encoder, and computer-readable medium that uses a machine learning model for prediction.

[0007] According to a first aspect, a method for processing image and / or video data by a decoder is provided. The method includes sequentially decoding multiple segments of an image or video. For one or more of the multiple segments, decoding includes: decoding a corresponding encoded residual segment to obtain a decoded residual segment; generating a predicted segment from a corresponding reference segment using a machine learning model, wherein the corresponding reference segment is selected from one or more reference segments stored in a set of reference segments; generating a reconstructed segment based on the predicted segment and the decoded residual segment; storing the reconstructed segment in the set of reference segments; and updating the parameters of the machine learning model based on the corresponding reference segment and the reconstructed segment.

[0008] In some implementations, the set of reference fragments includes one or more previously reconstructed fragments.

[0009] In some embodiments, using a machine learning model to predict a predicted segment from a corresponding reference segment may include: inputting the corresponding reference segment into a machine learning model; the machine learning model processing the corresponding reference segment based on the current values ​​of the parameters of the machine learning model to generate the predicted segment; and outputting the predicted segment from the machine learning model.

[0010] In some embodiments, updating the parameters of the machine learning model based on the corresponding reference fragment and reconstructed fragment may include: comparing the predicted fragment generated from the corresponding reference fragment with the reconstructed fragment; and updating the parameters of the machine learning model to reduce the difference between the predicted fragment and the reconstructed fragment from the corresponding reference fragment. Comparing the predicted fragment from the corresponding reference fragment with the reconstructed fragment may include: determining the value of an objective function based on the predicted fragment and the reconstructed fragment. The objective function may include one or more of the following: L1 loss, L2 loss, structural similarity index, and / or multi-scale structural similarity index. Updating the parameters of the machine learning model to reduce the difference between the predicted fragment and the reconstructed fragment from the corresponding reference fragment may include: applying an optimization routine to the objective function.

[0011] In some embodiments, generating a reconstructed fragment based on a predicted fragment and a decoded residual fragment may include summing the predicted fragment and the decoded residual fragment. Alternatively, generating a reconstructed fragment based on a predicted fragment and a decoded residual fragment may include multiplying the predicted fragment and the decoded residual fragment.

[0012] In some implementations, the corresponding reference segment corresponds to an adjacent segment of the reconstructed segment.

[0013] In some embodiments, the machine learning model includes a neural network. The neural network may include one or more of the following: one or more fully connected layers, one or more convolutional layers, one or more recurrent layers, one or more skip connections, and / or one or more attention layers.

[0014] Alternatively, machine learning models may include decision trees, support vector machines, and / or Gaussian processes.

[0015] In some embodiments, the method further includes receiving a bitstream from a network comprising encoded residual segments. The bitstream may also include one or more of the following: data indicating that a machine learning model will be used to decode the encoded residual segments; and / or data indicating the identifier of a reference segment.

[0016] In some implementations, the encoded residual segments are lossy encoded. Alternatively, in some implementations, the encoded residual segments are lossless encoded.

[0017] In some embodiments, a fragment is an image pixel or an image block.

[0018] According to a second aspect, a computer-readable medium is provided. The computer-readable medium includes computer-executable instructions stored thereon, which, when executed by a computing device, cause the computing device to perform any of the methods discussed in the first aspect.

[0019] According to a third aspect, a decoder is provided. The decoder includes: one or more processors and a computer-readable medium including computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform any of the methods discussed in the first aspect.

[0020] According to a fourth aspect, a method for processing image and / or video data performed by an encoder is provided. The method includes sequentially encoding multiple segments of an input image or video. For one or more of the multiple segments, the encoding includes: generating a predicted segment from a corresponding reference segment using a machine learning model, wherein the predicted segment corresponds to a corresponding input segment, and the corresponding reference segment is selected from one or more reference segments stored in a set of reference segments; determining a residual segment based on the predicted segment and the corresponding input segment; encoding the residual segment to obtain an encoded residual segment; decoding the encoded residual segment to obtain a decoded residual segment; generating a reconstructed segment corresponding to the corresponding input segment based on the predicted segment and the decoded residual segment; storing the reconstructed segment in the set of reference segments; and updating the parameters of the machine learning model based on the corresponding reference segment and the reconstructed segment.

[0021] In some implementations, the set of reference segments includes one or more previously encoded segments.

[0022] In some embodiments, predicting a predicted segment from a corresponding reference segment using a machine learning model includes: inputting the corresponding reference segment into a machine learning model; processing the corresponding reference segment based on the current values ​​of the parameters of the machine learning model to generate the predicted segment; and outputting the predicted segment from the machine learning model.

[0023] In some embodiments, updating the parameters of the machine learning model based on the corresponding reference fragment and reconstructed fragment includes: comparing the predicted fragment from the corresponding reference fragment with the reconstructed fragment; and updating the parameters of the machine learning model to reduce the difference between the predicted fragment from the corresponding reference fragment and the reconstructed fragment. In some embodiments, comparing the predicted fragment from the corresponding reference fragment with the reconstructed fragment includes: determining the value of an objective function based on the predicted fragment and the reconstructed fragment. In some embodiments, the objective function includes one or more of the following: L1 loss, L2 loss, structural similarity index, and / or multi-scale structural similarity index. In some embodiments, updating the parameters of the machine learning model to reduce the difference between the predicted fragment from the corresponding reference fragment and the reconstructed fragment includes: applying an optimization routine to the objective function.

[0024] In some embodiments, determining a residual segment based on the predicted segment and the corresponding input segment includes: subtracting the predicted segment from the corresponding input segment; and generating the reconstructed segment based on the predicted segment and the decoded residual segment includes: summing the predicted segment and the decoded residual segment.

[0025] In some embodiments, determining a residual segment based on the predicted segment and the corresponding input segment includes: dividing the corresponding input segment by the predicted segment; and generating the reconstructed segment based on the predicted segment and the decoded residual segment includes: multiplying the predicted segment and the decoded residual segment.

[0026] In some implementations, the corresponding reference segment corresponds to an adjacent segment of the reconstructed segment.

[0027] In some embodiments, the machine learning model includes a neural network. In some embodiments, the neural network includes one or more of the following: one or more fully connected layers, one or more convolutional layers, one or more recurrent layers, one or more skip connections, and / or one or more attention layers.

[0028] In some embodiments, the machine learning model includes: decision tree, support vector machine, and / or Gaussian process.

[0029] In some embodiments, encoding the residual segment to obtain an encoded residual segment includes using a lossy encoding scheme. In some embodiments, encoding the residual segment to obtain an encoded residual segment includes using a lossless encoding scheme.

[0030] In some embodiments, a fragment is an image pixel or an image block.

[0031] In some embodiments, the method further includes sending encoded residual segments from the bitstream to a decoder. In some embodiments, the bitstream also includes an indication of an identifier for a corresponding reference segment for each encoded residual segment.

[0032] In some implementations, the method further includes: encoding the plurality of segments with one or more additional encoding schemes; for one or more of the plurality of segments: comparing the size of a residual segment generated by each of the one or more additional encoding schemes with the size of a residual segment generated using a predicted segment from the machine learning model; selecting an encoding scheme for the segment based on the comparison result; and encoding the segment into a bitstream using the selected encoding scheme along with an indication of the selected encoding scheme.

[0033] According to a fifth aspect, a computer-readable medium is provided. The computer-readable medium includes computer-executable instructions stored thereon, which, when executed by a computing device, cause the computing device to perform any of the methods discussed in the fourth aspect.

[0034] According to a sixth aspect, an encoder is provided. The encoder includes one or more processors and a computer-readable medium including computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform any of the methods discussed in the fourth aspect.

[0035] According to the seventh aspect, a system is provided. The system includes an encoder according to the sixth aspect and a decoder according to the third aspect.

[0036] Because the machine learning model used for prediction is trained continuously on each reconstructed image segment during encoding and decoding, the prediction is continuously adapted to the characteristics of the encoded content. This can lead to more efficient encoding of image segments, for example, a reduction in the bit size of the encoded image segments. Furthermore, since the machine learning model used for prediction is trained on the same data in both the encoder and decoder, there is no need to send or predefine information about updates to the machine learning model (e.g., updated weight / parameter values ​​for an artificial neural network).

[0037] These and other aspects of this application may become more readily apparent from the following description of the embodiments. Attached Figure Description

[0038] Embodiments will now be described by way of example only with reference to the accompanying drawings, in which: FIG1 shows a flowchart of the operation of a decoder according to an embodiment; FIG2 shows a flowchart of the operation of an encoder according to the embodiment; FIG3 shows an overview of a method performed by a decoder according to an embodiment; FIG4 shows an overview of a method performed by an encoder according to an embodiment; FIG5 shows a schematic diagram of a decoder according to an embodiment; and FIG6 shows a schematic diagram of an encoder according to an embodiment. Detailed Implementation

[0039] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0040] These technical solutions can be applied to H.265 / HEVC or H.266 / VVC video coding systems, particularly for predictive coding stages. However, it should be understood that these technical solutions can be applied to any other image or video coding system involving predictive coding, regardless of whether it falls within the scope of current standards. Furthermore, while these principles are primarily illustrated with reference to video processing, they are also applicable to other data formats, including image processing and even audio processing.

[0041] In this embodiment, "video" refers to one or more images. In other words, a video may include one or more images. An image may also be referred to as a "picture".

[0042] An encoder is a device that encodes data into a bitstream, while a decoder is a device that decodes a bitstream to obtain the encoded data or an approximation of the encoded data. A bitstream consists of a sequence of bits.

[0043] Intra-frame prediction and inter-frame prediction are two prediction operations that can be used within HEVC and VVC frameworks or any other hybrid video coding technique for the decoder to process the received bitstream to obtain the original signal. In this embodiment, "original signal" or "original video" refers to the data prior to encoding at encoder 20. Reference samples in this embodiment may refer to spatially and / or temporally spaced image data used to predict images (or regions of images). Intra-frame and inter-frame prediction operations are also used at the encoder to make rate-distortion determinations.

[0044] More specifically, intra-frame prediction involves predicting data spatially within a single image, without referencing other images (at different time intervals). In other words, data used for a first region of an image is used to predict data for another region of the same image, but without depending on another image that is temporally spaced from another frame. In this context, the data for the first region of the image is considered a "reference sample."

[0045] Inter-frame prediction involves predicting data across images spanning multiple time intervals. In other words, data from a first region of a first image is used to predict data from a second region of a second image. The first and second regions may or may not be spatially separated. In this context, the data from the first region of the first image is considered a "reference sample." Further note that inter-frame prediction can sometimes use multiple reference regions from different images simultaneously, i.e., for a single prediction operation.

[0046] In the embodiments, "residual" may refer to the value obtained based on the original value of a region of the image and the predicted value of a region of the image (e.g., the difference between the original value and the predicted value).

[0047] In this embodiment, a "block" can refer to a portion of an image that includes multiple pixels. For example, an image can be divided into two or more blocks. However, this is just an example. If the image is not divided, then a "block" can refer to the entire image.

[0048] In this embodiment, "fragment" can refer to a portion of an image. For example, a fragment can be a block of an image. Alternatively, a fragment can be a single pixel of an image. If the image is not segmented, then "fragment" can refer to the entire image.

[0049] A "machine learning model" (ML model) is a model that has been trained to predict one or more output data from one or more sets of input data. Examples of machine learning models include neural networks, decision trees, random forests, support vector machines, linear regression models, Gaussian processes, and so on.

[0050] A “neural network” is an example of a machine learning model that includes multiple “nodes” arranged in one or more “layers”. Each node takes one or more outputs from nodes in the previous layer of the neural network or one or more initial inputs to the neural network as input and generates an output based on a set of “weights” associated with that node. For example, a node can determine a weighted sum of its inputs and apply a nonlinear function to that sum to generate the node’s output. The nonlinear function (also referred to as an “activation function” in this document) can be, for example, a sigmoid function, a tanh function, a ReLU function, etc. As used herein, the term “weights” can encompass the weights and / or biases of a neural network.

[0051] The layers of a neural network can include one or more of the following: one or more fully connected layers, one or more convolutional layers, one or more recurrent layers, one or more Long Short-Term Memory (LSTM) layers, one or more attention layers, etc. A neural network can include one or more skip connections between layers. Neural networks can, for example, have transformer architectures, convolutional architectures, residual network architectures, recursive architectures, autoencoder architectures, etc.

[0052] Figure 1 shows a flowchart of operation 100 of decoder 50 according to an embodiment. Figure 2 shows a flowchart of operation 200 of encoder 60 according to the same embodiment.

[0053] Turning to Figure 1, the operations of Figure 1 are performed sequentially on a sequence of encoded residual segments of an image or video to reconstruct the corresponding sequence of image or video segments encoded at the encoder (e.g., with respect to Figure 2).

[0054] At operation 102, the decoder decodes (e.g., decompresses) the encoded residual segment E' of the image / video to obtain the decoded residual segment E'". The encoded residual segment may have already been encoded by the encoder using a lossy or lossless compression scheme. The decoder applies an inverse transform to decompress / decode the encoded residual segment to obtain the decoded residual segment. For example, the decoder may apply inverse quantization and / or scaling to the encoded residual segment.

[0055] The encoded residual fragments can be extracted, for example, from a bitstream received by a decoder, such as via a network. The bitstream may also include an indication of the prediction mode used to encode the encoded residual fragments, such as whether ML-based prediction or some other prediction mode (e.g., coding based on discrete cosine transform (DCT)).

[0056] At operation 104, the decoder uses a machine learning model to generate a predicted image / video segment P from the corresponding reference segment A. The machine learning model can be, for example, a neural network (also known as an "artificial neural network," ANN). Alternatively, the machine learning model can be a decision tree, support vector machine, and / or Gaussian process.

[0057] The corresponding reference segment is taken from a set of one or more reference segments stored at the decoder. The corresponding reference segment can come from the same image as the current image segment being decoded (i.e., using intra-frame prediction). Alternatively, the reference segment can come from a different image than the current image segment being decoded (i.e., using inter-frame prediction), such as different images in a sequence of images that make up the video.

[0058] The reference segment set may include one or more previously decoded image / video segments, for example, segments from the same segment sequence as the current segment being decoded. The reference segment set may be stored in the decoder's memory, for example, in a reference segment buffer.

[0059] Reference segments are fed into a machine learning model. The machine learning model processes the input reference segments based on the current values ​​of its parameters to generate predicted segments and outputs the predicted segments.

[0060] At operation 106, the decoder generates a reconstructed segment R of the image / video based on the predicted segment and the decoded residual segment. The predicted segment and the decoded residual segment can be combined by addition or multiplication to generate the reconstructed segment.

[0061] At operation 108, the decoder stores the reconstructed segment of the image / video in this set of reference segments as a new reference segment. Therefore, the reconstructed segment can be used to reconstruct other image / video segments in the sequence.

[0062] At operation 110, the decoder updates the parameters of the machine learning model based on the corresponding reference and reconstructed segments of the image / video. Any machine learning procedure used to update the machine learning model can be used to update the parameters.

[0063] For example, a predicted segment generated by a machine learning model from a corresponding reference segment can be compared with a reconstructed segment. This comparison can be performed using one or more loss / objective functions, such as L1 and L2 loss, structural similarity indices, and / or multi-scale structural similarity indices. Based on the comparison results, an update for the machine learning model is determined. The update is then determined to reduce the difference between the predicted segment and the corresponding reconstructed segment (e.g., as measured by the loss / objective function). For example, an optimization routine such as gradient descent can be applied to the loss / objective function to determine parameter updates for the machine learning model.

[0064] In some examples, the decoder updates the machine learning model immediately after reconstructing a segment; that is, it updates the machine learning model before reconstructing the next segment. The updated machine learning model is then used to reconstruct the next segment.

[0065] Alternatively, the reconstructed segments can be stored in a set of training data for later updates to the machine learning model. For example, the decoder can use the same machine learning model to reconstruct multiple segments, and then update the machine learning model based on the training dataset that includes the multiple reconstructed segments. The next multiple segments can then be reconstructed using the updated machine learning model.

[0066] Alternatively, the same machine learning model can be used to reconstruct all segments of an image / video, i.e., without performing an update process.

[0067] After reconstructing the image / video segment and (if) updating the ML model parameters, the method returns to operation 102 and reconstructs the next segment in the sequence using the (possibly updated) ML model, at which point the currently reconstructed segment can be used as a reference segment.

[0068] Turning to Figure 2, the operations of Figure 2 are performed sequentially on a sequence of input image / video segments from one or more input images / videos to encode the segments into multiple encoded residual segments.

[0069] At operation 202, the encoder uses a machine learning model to generate a predicted segment P of the input image / video from the corresponding reference segment A. The machine learning model can be, for example, a neural network (also known as an "artificial neural network," ANN). Alternatively, the machine learning model can be a decision tree, support vector machine, and / or Gaussian process.

[0070] The corresponding reference segment is taken from a set of one or more reference segments stored at the encoder, which corresponds to a previously encoded image / video segment. This set of reference segments can be stored in the encoder's memory, for example, in a reference segment buffer.

[0071] Reference clips are fed into a machine learning model. The machine learning model processes the input reference clips based on the current values ​​of its parameters to generate predicted clips of the image / video and outputs the predicted clips.

[0072] At operation 204, the encoder determines the residual segment E (also known as the "error segment") of the input image / video segment based on the predicted segment and the corresponding input segment. For example, the encoder can subtract the predicted segment from the input segment to determine the residual segment. Alternatively, the encoder can divide the input segment by the predicted segment to obtain the residual segment.

[0073] At operation 206, the encoder encodes the residual segment to obtain the encoded residual segment E'. In some examples, the encoder uses a lossless compression scheme to encode the residual segment. In other examples, the encoder uses a lossy compression scheme to encode the residual segment. For example, the encoder may apply quantization and / or binarization to the residual segment.

[0074] In some embodiments, the encoder also determines one or more additional residual segments, each using a different prediction scheme. The properties of the residual segments (i.e., the ML-generated residual segments and one or more additional residual segments) are compared, and a prediction scheme for encoding the segment is selected based on the comparison results. For example, the encoder may select a prediction scheme that produces the smallest encoded residual segment (e.g., the encoded residual segment with the fewest bits).

[0075] The encoder can encode residual fragments into a bitstream using a selected prediction mode. An indication of which prediction mode has been selected can also be encoded into the bitstream. The bitstream (also called the "encoded bitstream") can be transmitted over a network to the decoder and / or stored in memory.

[0076] At operation 208, the encoder decodes the encoded residual segment to obtain the decoded residual segment E". For example, the encoder may apply inverse quantization and / or scaling to the encoded residual segment. At operation 210, the encoder generates a reconstructed segment of the input image / video based on the predicted segment and the decoded residual segment. At operation 212, the encoder stores the reconstructed segment in a set of reference segments. At operation 214, the encoder updates the parameters of the machine learning model based on the reconstructed segment of the input image / video and the corresponding reference segments. For example, the machine learning model update may be performed immediately after reconstructing the image segment, i.e., before reconstructing the next image segment. Alternatively, the reconstructed image segment may be stored in a set of training data for later updates to the machine learning model. Operations 208-214 correspond to the encoder-side implementation of operations 102-110 of the decoder, and their descriptions are omitted here for brevity.

[0077] In examples where a non-ML-based prediction mode is chosen, operations 208-214 can be omitted; that is, the machine learning model is not updated. Instead, the encoder decompresses the encoded residual segments and reconstructs the image / video segments using the inverse operation corresponding to the prediction mode used to encode the residual segments. The reconstructed segments are then stored in a set of reconstructed segments. In some examples, the machine learning model is updated even when a non-ML-based prediction mode is used. For example, the machine learning model update can be performed immediately after reconstructing an image segment, i.e., before reconstructing the next image segment. Alternatively, the reconstructed image segments can be stored in a set of training data for later updates to the machine learning model.

[0078] The encoder then returns to operation 202 and encodes the next segment of the input image / video. However, the reconstructed segment of the current segment is now available for the encoder to use in predictions. In examples using ML-based predictions, the machine learning model can also have updated parameters.

[0079] Figure 3 illustrates a schematic overview of method 300 performed by decoder 50 according to an embodiment. Method 300 may correspond to method 100 of Figure 1.

[0080] The decoder receives a sequence 302, or E′, of encoded residual segments. The encoded residual segments 302 are encoded in a bitstream. Each encoded residual segment 302 corresponds to a corresponding segment of one or more images / videos that have already been encoded (e.g., encoded using the methods of Figures 2 and / or 4). The decoder's goal is to reconstruct these one or more images / videos from the encoded residual segments 302 and the previously reconstructed segment 304, or A.

[0081] The encoded residual segment can be received from the encoder or storage system, for example, via a network; for instance, the encoded segment 302 is sent from the source system to the decoder. Alternatively, the decoder can receive the encoded residual segment 302 from its local memory, for example, by retrieving the encoded residual segment 302 from the decoder's memory.

[0082] Each encoded residual segment 302 can be encoded / compressed into a bitstream using a lossy or lossless compression scheme. When decoding a segment, the encoder therefore decodes the encoded residual segment 302 corresponding to 306, for example, using an inverse / decompression scheme to obtain the decoded residual segment 308, i.e., E". In some examples, the bitstream may contain an indication of the encoding / compression scheme used by the encoder. Alternatively, the decompression scheme may be known in advance to the decoder, for example, it is fixed.

[0083] The decoder uses a machine learning model 312 to generate a predicted image segment 310, or P, from previously reconstructed segments 304 (also referred to as "reference segments") stored at the decoder. The previously reconstructed segment 304 may correspond to a previously decoded segment of the same image as the segment currently being decoded (i.e., the decoder is using intra-frame prediction). Alternatively or additionally, the previously reconstructed segment 304 may correspond to a previously decoded segment of a different image (i.e., the decoder is using inter-frame prediction). In some embodiments, for each encoded residual segment 302, the identifier of the corresponding previously reconstructed segment 304 may be signaled in the bitstream. Alternatively, one or more predefined schemes may be used to determine the identifier of the corresponding previously reconstructed segment 304 to be used with each encoded residual segment 302. For example, the previously reconstructed segment 304 to be used with each encoded residual segment 302 may correspond to an adjacent segment of the image / video being decoded, such as a segment to the left, upper left, above, upper right, etc., of the current segment being decoded. In some examples, the scheme adopted for each segment may be signaled in the bitstream, for example, on a segment-by-segment basis or for a set of segments.

[0084] In any case, the selected previously reconstructed segment 304 is fed into the machine learning model 312, which processes the previously reconstructed segment 304 based on the current values ​​of the parameters of the machine learning model 312 (e.g., weights and / or biases) to generate a predicted segment 310, and outputs the predicted segment 310 from the model 312.

[0085] Then, in the reconstruction process 314, the predicted segment 310 is combined with the decoded residual segment 308 to obtain the reconstructed segment 316, i.e., R. For example, the reconstructed segment 316 can be obtained by summing the predicted segment 310 and the decoded residual segment 308, i.e., R = E” + P. In some alternative embodiments, the reconstructed segment 316 can be obtained by multiplying the predicted segment 310 with the decoded residual segment 308, i.e., R = E” * P.

[0086] The reconstructed segment 316 is output for reconstructing the original image / video (not shown). Additionally, the reconstructed segment 316 is output to a set of previously reconstructed segments 318 for potential use in future segments of the decoding sequence (e.g., in a buffer of a reference segment stored at the decoder).

[0087] The reconstructed fragment 316 is also output to the machine learning training process 320. The ML training process 320 uses the reconstructed fragment 316 and the previous reconstructed fragment 304 (or the predicted fragment 310 generated from it) to update the parameters of the machine learning model 312, with the training objective of reducing the difference between the reconstructed fragment 316 and the previous reconstructed fragment 304.

[0088] For example, a loss / objective function L can be used to compare the predicted fragment 310 output by the machine learning model 312 based on the previously reconstructed fragment 304 with the currently reconstructed fragment 316. The objective function can be constructed based on the difference between the predicted fragment 310 and the currently reconstructed fragment 316, i.e., L=1(RP), such as L1 loss (e.g., L=|RP|) or L2 loss (e.g., L=|RP|). 2 ).

[0089] An optimization routine, such as gradient descent, can be applied to the loss / objective function to determine updates to the parameters of the machine learning model 312. In some embodiments, the optimization routine can be an iterative routine. At each iteration, the machine learning model generates candidate predicted segments from the previously reconstructed segment 304 based on the values ​​of the parameters of the machine learning model 312 used for the iteration. The candidate predicted segments are compared with the reconstructed segment 316 using the loss / objective function, and the gradient of the loss / objective function relative to the parameters of the machine learning model is determined, for example, using backpropagation. An update rule (e.g., based on gradient descent) is then used to determine a set of machine learning model parameters for the next iteration based on the gradient of the loss / objective function and the parameter values ​​of the current iteration.

[0090] An optimization routine can be applied until a threshold condition is met. The threshold condition can be, for example, the number of threshold iterations in the optimization routine, such as 1, 2, 4, 8, or 16. Alternatively, the threshold condition can be that the loss / objective function value of the candidate predicted segment is below a threshold.

[0091] Once the optimization routine is complete, the resulting machine learning model parameters are stored for use when decoding the next segment of the image / video sequence.

[0092] After this, the same process is used to decode the next segment of the image / video in the segment sequence, but at this time the reconstructed image segment 316 of the image segment is available for decoding, and the parameters of the machine learning model 312 have been updated (if the machine learning has been updated).

[0093] Figure 4 shows a schematic overview of method 400 performed by encoder 60 according to an embodiment. Method 400 may correspond to method 200 of Figure 2.

[0094] The encoder receives an input segment 402, or O, of an input image I (a single image or an image acquired from a video) and obtains a reference segment 404, or A, from a set of previously encoded segments 406 for encoding the input segment 402. The set of previously encoded segments 406 may include segments from the input image and / or segments from one or more other images (e.g., images in a sequence of images forming a video).

[0095] The encoder may select a reference segment 404 from a set of previously encoded segments 406 based on its similarity to the input segment 402, for example, selecting the most similar reference segment. Alternatively, the reference segment 404 may correspond to a segment in an image that has a predefined spatial location relative to the input segment 402. For example, the reference segment 404 may be a neighboring segment from the input image, such as a segment to the left, upper left, above, or upper right. Alternatively, the reference segment 404 may be a segment from another input image, for example, a segment at the same or adjacent location in another image (e.g., in the preceding / following images of a sequence of images forming a video).

[0096] The encoder generates a predicted fragment 408, or P, using the reference fragment 404 by applying the machine learning model 410 to the reference fragment 404. The reference fragment 404 is input into the machine learning model 410, which processes the reference fragment 404 based on the current values ​​of the machine learning model's parameters (e.g., weights and / or biases) to generate the predicted fragment 408.

[0097] The encoder uses a residual calculator 412 to generate a residual / error segment 414, or E, from the input segment 402 and the prediction segment 408. For example, the encoder can subtract the prediction segment 408 from the input segment 402 to obtain the residual segment 414, i.e., E=OP. Alternatively, the encoder can divide the input segment 402 by the prediction segment 408 to obtain the residual segment 414, i.e., E=O / P.

[0098] The encoder then encodes (e.g., compresses) the residual to obtain an encoded (compressed) residual segment 418, or E'. In some examples, the encoder uses a lossy compression scheme to encode the residual segment. In some examples, the encoder uses a lossless compression scheme to encode the residual segment. The encoded residual segment 418 may be included in the bitstream along with auxiliary information indicating, for example, the prediction mode used, the identifier of the reference segment, etc.

[0099] In some implementations, the encoder then compares the encoded residual segment 418 with one or more other encoded residual segments of the same input segment 402, each encoded residual segment determined using a different prediction scheme (e.g., a non-ML-based prediction scheme). The encoder then selects a prediction scheme based on the comparison results and encodes the residual segments generated from the selected scheme into a bitstream (not shown).

[0100] The encoder then decodes / decompresses the encoded residual segment 418 of 420 (i.e., using the inverse operation of encoding process 416) to obtain the decoded residual segment 422, i.e., E”. In the example where encoding process 416 is a lossless compression scheme, the decoded residual segment 422 is the same as the residual / error segment 414, i.e., E” = E. In the example where encoding process 416 is a lossy compression scheme, the decoded residual segment 422 is an approximation of the residual / error segment 414, i.e., E” ≈ E.

[0101] The encoder generates a reconstructed segment 426, or R, using a reconstruction process 424 that is the reverse of the residual calculation process 412. For example, in an embodiment where the residual calculation process 412 subtracts the prediction segment 408 from the input segment 402, the reconstruction process 424 adds the decoded residual segment 422 to the prediction segment 408, i.e., R = E” + P. In an embodiment where the residual calculation process 412 divides the input segment 402 by the prediction segment 408, the reconstruction process 424 multiplies the decoded residual segment 422 by the prediction segment 408, i.e., R = E” * P.

[0102] The reconstructed segment 426 is output to a set of previously encoded segments 406 so that it may be potentially used when encoding image segments in the sequence, for example, image segments in the sequence stored in a buffer of a reference segment at the encoder.

[0103] The reconstructed fragment 426 is also output to the machine learning training process 428. The machine learning process 428 uses the same process as the decoder machine learning process 320 described with respect to Figure 3, and will not be repeated here for brevity. Using the same ML training process on the same training data (i.e., the same reference fragments 304, 404 and the same reconstructed fragments 316, 426) at both the encoder and decoder ensures that the encoder and decoder maintain identical copies of the machine learning models 320, 428, without having to pass parameter update values ​​between them.

[0104] After this, the same process is used to encode the next image / video segment in the segment sequence, but at this time the reconstructed segment 426 of that image segment can be used for decoding, and if the machine learning model has been updated, the updated parameters of the machine learning model 410 are used.

[0105] An example of this encoding process is the improved JPEG compression technique. The image is encoded in macroblocks of 16×16 pixels. In standard JPEG, the only prediction tool is a prediction pattern of the discrete cosine (DC) coefficients of the DC transform. In the improved JPEG compression technique, machine learning-based prediction patterns, as described herein, are also available, for example, prediction patterns employing artificial neural networks (ANNs).

[0106] During encoding, the currently encoded macroblock (O) is predicted (given a sample P) based on the macroblock immediately following the one encoded previously (e.g., the macroblock to the left). The encoder has two prediction modes: a DCT-based prediction mode (P...). DC ) and machine learning-based prediction patterns (P ANN ).

[0107] For both prediction modes, the prediction error (i.e., the residual signal) is calculated in the encoder as E=OP, respectively for P DC and P ANN Given E DC and E ANN Prediction error E DC and E ANN It is speculatively encoded (e.g., quantized and binary) to check which mode results in a smaller bitstream. The encoder selects the better encoding mode and sends E respectively. DC or E ANN And information about the selected prediction mode, which allows the prediction error E' to be reconstructed (e.g., dequantized, scaled) at both the encoder and decoder.

[0108] The macroblock of image O is reconstructed by summing the prediction error and the sample P, i.e., R = E' + P, where P is the sum of the prediction error and the sample P. DC or P ANN , which is suitable for the selected prediction mode, and R is the reconstructed macroblock.

[0109] The reconstructed macroblock R is complete and can be used for further prediction. If P is selected... ANN Then, the new training example is used to train the ANN: A→R.

[0110] Figure 5 illustrates a schematic diagram of a decoder 50 according to an embodiment. Specifically, Figure 5 illustrates a schematic diagram of a decoder 50 configured to perform any of the decoder methods discussed herein. For the sake of brevity, these detailed descriptions are omitted here.

[0111] As shown in Figure 5, the decoder 50 includes a processor 51 and a computer-readable medium 52. The processor 51 and the computer-readable medium 52 can be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or code. The processor 51 is used to execute the programs, instructions, or code in the computer-readable medium 52 to perform the operations described in the decoder method embodiments of this paper.

[0112] Therefore, in this embodiment, the computer-readable medium 52 is configured to store a computer program that can run in the processor 51, and the processor 51 is configured to run the computer program to perform the steps in any decoder method discussed herein.

[0113] Figure 6 illustrates a schematic diagram of an encoder 60 according to an embodiment. Specifically, Figure 6 illustrates a schematic diagram of an encoder 60 configured to perform any of the encoder methods discussed herein. For the sake of brevity, these detailed descriptions are omitted herein.

[0114] As shown in Figure 6, the encoder 60 includes a processor 61 and a computer-readable medium 62. The processor 61 and the computer-readable medium 62 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or code. The processor 61 is used to execute the programs, instructions, or code in the computer-readable medium 62 to perform the operations described in the decoder method embodiments of this document.

[0115] Therefore, in this embodiment, the computer-readable medium 62 is configured to store a computer program that can run in the processor 61, and the processor 61 is configured to run the computer program to perform the steps in any encoder method discussed herein.

[0116] Embodiments of the present invention may also provide a computer-readable medium having computer-executable instructions to cause one or more processors of a computing device to perform the methods of any embodiment of the present invention.

[0117] Examples of computer-readable media include volatile and non-volatile media, removable and non-removable media, and include, but are not limited to: solid-state storage, removable disks, hard disk drives, magnetic media, and optical disks. Generally, computer-readable media includes any type of media suitable for storing, encoding, or carrying a series of instructions executable by one or more computers to perform any or more of the processes and features described herein.

[0118] It should be understood that the functionality of each of the components discussed can be combined in various ways other than those discussed in the foregoing description. For example, in some embodiments, more than one function of the devices discussed can be incorporated into a single device. In other embodiments, the functionality of at least one of the devices discussed can be split into multiple separate (or distributed) devices.

[0119] Conditional language such as “may” is often used to indicate the use of features / steps in a particular embodiment, but alternative embodiments may include alternative features or omit these features entirely.

[0120] Furthermore, the method steps are not limited to the specific order described, and it should be understood that these steps can be combined in any other suitable order. In some embodiments, this can result in the parallel execution of some method steps. Additionally, in some embodiments, specific method steps may be omitted entirely.

[0121] While certain embodiments have been discussed, it should be understood that these embodiments are intended to illustrate the general teachings of the invention, and various modifications may be made without departing from the scope of the invention. The scope of the invention should be interpreted in accordance with the appended claims and any equivalents thereof.

[0122] Many further variations and modifications will occur to those skilled in the art when referring to the foregoing illustrative embodiments, which are given by way of example only and are not intended to limit the scope of the invention as defined by the appended claims.

Claims

1. A method for processing image and / or video data, the method being performed by a decoder and comprising sequentially decoding a plurality of segments of an image or video, wherein for one or more of the plurality of segments, the decoding comprises: Decode the corresponding encoded residual fragment to obtain the decoded residual fragment; A machine learning model is used to generate a predicted fragment from a corresponding reference fragment, wherein the corresponding reference fragment is selected from one or more reference fragments stored in a set of reference fragments; a reconstructed fragment is generated based on the predicted fragment and the decoded residual fragment; the reconstructed fragment is stored in the set of reference fragments; and the parameters of the machine learning model are updated based on the corresponding reference fragment and the reconstructed fragment.

2. The method according to claim 1, wherein, The reference fragment set includes one or more previously reconstructed fragments.

3. The method according to claim 1 or 2, wherein, Predicting the predicted segment from a corresponding reference segment using the machine learning model includes: inputting the corresponding reference segment into the machine learning model; processing the corresponding reference segment based on the current values ​​of the parameters of the machine learning model to generate the predicted segment; and outputting the predicted segment from the machine learning model.

4. The method according to any one of the preceding claims, wherein, Updating the parameters of the machine learning model based on the corresponding reference fragment and the reconstructed fragment includes: comparing the predicted fragment from the corresponding reference fragment with the reconstructed fragment; and updating the parameters of the machine learning model to reduce the difference between the predicted fragment from the corresponding reference fragment and the reconstructed fragment.

5. The method according to claim 4, wherein, Comparing the predicted segment from the corresponding reference segment with the reconstructed segment includes determining the objective function value based on the predicted segment and the reconstructed segment of the image.

6. The method according to claim 5, wherein, The objective function includes one or more of the following: L1 loss, L2 loss, structural similarity index, and / or multi-scale structural similarity index.

7. The method according to claim 5 or 6, wherein, Updating the parameters of the machine learning model to reduce the difference between the predicted fragment and the reconstructed fragment from the corresponding reference fragment includes applying an optimization routine to the objective function.

8. The method according to any one of the preceding claims, wherein, Generating the reconstructed segment based on the predicted segment and the decoded residual segment includes summing the predicted segment and the decoded residual segment.

9. The method according to any one of claims 1 to 7, wherein, Generating the reconstructed segment based on the predicted segment and the decoded residual segment includes multiplying the predicted segment with the decoded residual segment.

10. The method according to any one of the preceding claims, wherein, The corresponding reference segment corresponds to the adjacent segment of the reconstructed segment.

11. The method according to any one of the preceding claims, wherein, The machine learning model includes neural networks.

12. The method according to claim 11, wherein, The neural network includes one or more of the following: one or more fully connected layers, one or more convolutional layers, one or more recurrent layers, one or more skip connections, and / or one or more attention layers.

13. The method according to any one of claims 1 to 10, wherein, The machine learning models include: decision trees, support vector machines, and / or Gaussian processes.

14. The method according to any one of the preceding claims, further comprising: Receive a bit stream from the network that includes the encoded residual fragment.

15. The method according to claim 14, wherein, The bitstream also includes one or more of the following: data indicating that the machine learning model will be used to decode the encoded residual segment; and / or data indicating the identifier of a reference segment.

16. The method according to any one of the preceding claims, wherein, The encoded residual fragments are lossy encoded.

17. The method according to any one of claims 1 to 15, wherein, The encoded residual fragments are lossless encoded.

18. The method according to any one of the preceding claims, wherein, The fragment is an image pixel or an image block.

19. A computer-readable medium comprising computer-executable instructions stored thereon, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 18.

20. A decoder, comprising: One or more processors; and a computer-readable medium, including computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 18.

21. A method for processing image and / or video data, the method being performed by an encoder and comprising sequentially encoding a plurality of segments of an input image or video, wherein for one or more of the plurality of segments, the encoding comprises: A machine learning model is used to generate predicted segments from corresponding reference segments, wherein the predicted segments correspond to corresponding input segments, and the corresponding reference segments are selected from one or more reference segments stored in a set of reference segments; residual segments are determined based on the corresponding input segments and the predicted segments of the input image; the residual segments are encoded to obtain encoded residual segments; the encoded residual segments are decoded to obtain decoded residual segments; a reconstructed segment corresponding to the corresponding input segment is generated based on the predicted segments and the decoded residual segments; the reconstructed segments are stored in the set of reference segments; and the parameters of the machine learning model are updated based on the corresponding reference segments and the reconstructed segments.

22. The method according to claim 21, wherein, The reference fragment set includes one or more previously encoded fragments.

23. The method according to claim 21 or 22, wherein, Predicting the predicted segment from a corresponding reference segment using the machine learning model includes: inputting the corresponding reference segment into the machine learning model; processing the corresponding reference segment based on the current values ​​of the parameters of the machine learning model to generate the predicted segment; and outputting the predicted segment from the machine learning model.

24. The method according to any one of claims 21 to 23, wherein, Updating the parameters of the machine learning model based on the corresponding reference fragment and the reconstructed fragment includes: comparing the predicted fragment from the corresponding reference fragment with the reconstructed fragment; and updating the parameters of the machine learning model to reduce the difference between the predicted fragment from the corresponding reference fragment and the reconstructed fragment.

25. The method according to claim 24, wherein, Comparing the predicted segment from the corresponding reference segment with the reconstructed segment includes determining the value of the objective function based on the predicted segment and the reconstructed segment.

26. The method of claim 25, wherein, The objective function includes one or more of the following: L1 loss, L2 loss, structural similarity index, and / or multi-scale structural similarity index.

27. The method according to claim 25 or 26, wherein, Updating the parameters of the machine learning model to reduce the difference between the predicted fragment and the reconstructed fragment from the corresponding reference fragment includes applying an optimization routine to the objective function.

28. The method according to any one of claims 21 to 27, wherein, Determining a residual segment based on the predicted segment and the corresponding input segment includes: subtracting the predicted segment from the corresponding input segment; and wherein generating the reconstructed segment based on the predicted segment and the decoded residual segment includes: summing the predicted segment and the decoded residual segment.

29. The method according to any one of claims 21 to 27, wherein, Determining a residual segment based on the predicted segment and the corresponding input segment includes: dividing the corresponding input segment by the predicted segment; and wherein generating the reconstructed segment based on the predicted segment and the decoded residual segment includes: multiplying the predicted segment by the decoded residual segment.

30. The method according to any one of claims 21 to 29, wherein, The corresponding reference segment corresponds to the adjacent segment of the reconstructed segment.

31. The method according to any one of claims 21 to 27, wherein, The machine learning model includes neural networks.

32. The method according to claim 31, wherein, The neural network includes one or more of the following: one or more fully connected layers, one or more convolutional layers, one or more recurrent layers, one or more skip connections, and / or one or more attention layers.

33. The method according to any one of claims 21 to 30, wherein, The machine learning models include: decision trees, support vector machines, and / or Gaussian processes.

34. The method according to any one of claims 21 to 33, wherein, Encoding the residual fragment to obtain the encoded residual fragment includes using a lossy coding scheme.

35. The method according to any one of claims 21 to 33, wherein, Encoding the residual fragment to obtain the encoded residual fragment includes using a lossless coding scheme.

36. The method according to any one of claims 21 to 35, wherein, The fragment is an image pixel or an image block.

37. The method according to any one of claims 21 to 36, the method further comprising: The encoded residual fragment in the bitstream is sent to the decoder.

38. The method according to claim 37, wherein, The bitstream further includes an indication of the identifier of the corresponding reference segment in each of the encoded residual segments.

39. The method according to any one of claims 21 to 36, further comprising: The plurality of segments are encoded using one or more other encoding schemes; For one or more of the plurality of segments: compare the size of the residual segment generated by each of the one or more other encoding schemes with the size of the residual segment generated using the predicted segment from the machine learning model; select an encoding scheme for the segment based on the comparison result; and encode the segment into a bitstream using the selected encoding scheme along with an indication of the selected encoding scheme.

40. A computer-readable medium comprising computer-executable instructions stored thereon, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 21 to 39.

41. An encoder, comprising: One or more processors; and a computer-readable medium, including computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 21 to 39.

Images