Context-based image coding

Abstract

According to the implementation of this disclosure, a context-based image encoding and decoding scheme is provided. According to this scheme, a reference image of a target image is obtained. A context feature representation of the reference image is extracted. The context feature representation characterizes context information associated with the target image. Conditional encoding or decoding of the target image is performed based on the context feature representation. This achieves performance improvements in reconstruction quality and compression efficiency.

Description

Background Technology

[0001] In this article, "coding" can include encoding and / or decoding. Generally, video frames are encoded by an encoder at the transmitting end to compress the video for transmission over a network. Encoding of a given frame can be performed with reference to another frame in the video. The resulting encoded representation is then used to transmit the corresponding bitstream to the receiving end. At the receiving end, the corresponding decoder can decode the received bitstream to retrieve the given video frame for output to the receiving end's screen. During encoding and decoding, frame reconstruction quality and compression efficiency are always important considerations. Summary of the Invention

[0002] Based on the implementation of this disclosure, a context-based image encoding and decoding scheme is proposed. In this scheme, a reference image of the target image is obtained. A contextual feature representation of the reference image is extracted. The contextual feature representation characterizes the contextual information associated with the target image. Conditional encoding or decoding of the target image is performed based on the contextual feature representation. This achieves performance improvements in reconstruction quality and compression efficiency.

[0003] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0004] Figure 1 A schematic block diagram of a conventional residual-based video encoding and decoding system is shown.

[0005] Figure 2 A schematic block diagram of a context-based video encoding / decoding system according to some implementations of this disclosure is shown;

[0006] Figure 3 Examples of the ability of context feature representations to represent contextual information are shown in some implementations of this disclosure;

[0007] Figure 4 Some implementations of this disclosure are shown in Figure 2 A block diagram of an example structure of a context generator in a system;

[0008] Figure 5 Some implementations of this disclosure are shown in Figure 2 A block diagram of an example structure of an entropy model in a system;

[0009] Figure 6A comparison of some implementations of context-based video coding and decoding schemes according to this disclosure with conventional video coding and decoding schemes is shown;

[0010] Figure 7 Flowcharts illustrating processes for video encoding and decoding according to some implementations of this disclosure are shown; and

[0011] Figure 8 A block diagram of a computing device capable of implementing multiple implementations of the present disclosure is shown.

[0012] In these accompanying figures, the same or similar reference symbols are used to indicate the same or similar elements. Detailed Implementation

[0013] This disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement this disclosure, and not to imply any limitation on the scope of this disclosure.

[0014] As used herein, the term "comprising" and its variations are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "at least partially based on". The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation". The term "another implementation" is to be interpreted as "at least one other implementation". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0015] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0016] A neural network is a machine learning network based on deep learning. A neural network can process inputs and provide corresponding outputs. It typically consists of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the inputs to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the layer above.

[0017] Machine learning typically comprises three phases: training, testing, and usage (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the usage phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0018] In this article, "frame" or "video frame" refers to the individual images in a video clip. "Image" and "frame" are used interchangeably in this article. Multiple consecutive images can form a dynamic video clip, where each image is considered a frame.

[0019] Currently, with the development of machine learning technology, its application in video encoding and decoding processes has been proposed. However, limited by conventional encoding and decoding workflows, the reconstruction quality and compression efficiency of video frames still need improvement.

[0020] Traditional video encoding and decoding based on residuals

[0021] Conventional video codec schemes, including the H.261 video codec standard developed in 1988 and the H.266 video codec standard released in 2020, widely adopt residual-based codec schemes. This scheme is based on the predictive codec paradigm, which generates a reference image for the current image and encodes and decodes the residual between the current image and the reference image. Figure 1A schematic block diagram of a conventional residual-based video encoding / decoding system 100 is shown. System 100 includes an encoder 110, a decoder 120, an image predictor 130, a residual generation module 140, and a residual addition module 150. In the residual-based encoding / decoding process, the encoder 110 is referred to as a residual encoder, and the decoder 120 is referred to as a residual decoder.

[0022] Suppose that the image 102 to be encoded is the image at time t in the video clip. The image predictor 130 is configured to generate a predicted image for image 102 based on the reference image 170. 132. Reference image 170 may include the decoded image at time t-1 before t in the video segment. The residual generation module 140 calculates the image 102. With predicted image The residual between 132. Encoder 110 encodes the residual to generate image 102. The encoded representation. The corresponding bitstream 112 is transmitted to the decoding side.

[0023] On the decoding side, decoder 120 receives bitstream 112 and decodes the decoded image from bitstream 112. Residual addition module 150 combines the decoded image provided by decoder 120 with the predicted image generated by image predictor 130. Adding 132 together yields the decoded image 160 at time t. .

[0024] Residual-based video encoding and decoding can be represented as follows:

[0025]

[0026] In the above formula (1), This indicates the encoding processing of encoder 110. This indicates the decoding process of decoder 120. This represents the prediction processing of the image predictor 130, and This indicates a quantization operation. In machine learning-based applications, encoder 110 can utilize a machine learning model to implement residual encoding, and correspondingly, decoder 120 can utilize a machine learning model to implement residual decoding.

[0027] Working principle and example system

[0028] Given the strong temporal correlation between frames in a video, residual coding has historically been considered a simple and effective method for video compression. However, the inventors of this application have discovered through research that, given a predicted image... Encoding the current image under the circumstances Residual encoding and decoding is not optimal because it always uses simple subtraction to remove redundancy between images. The entropy of residual encoding and decoding is greater than or equal to the entropy of conditional encoding and decoding. ,in This represents the Shannon entropy. Theoretically, the current image... A single pixel is associated with all pixels in the decoded image from the previous time step, and these pixels are in the image. The image has already been decoded. For traditional codecs, it's difficult to explicitly characterize all the correlations between the decoded image from a previous time step and the current image using manually defined rules. Therefore, residual-based encoding and decoding utilizes the assumption that pixels in the current image are only related to their corresponding predicted pixels in the predicted image, thus simplifying the encoding and decoding process. However, such encoding and decoding schemes are not sufficiently optimized in terms of reconstruction quality and compression ratio.

[0029] Based on an example implementation of this disclosure, a context-based encoding / decoding scheme is proposed. Unlike traditional schemes that require generating a predicted image of the target image and encoding the residual between the target and predicted images, the example implementation of this disclosure extracts contextual feature representations from a reference image to perform conditional encoding / decoding of the target image. This scheme uses contextual information as a condition in the feature domain to guide adaptive encoding of the target image. Such a scheme can achieve a higher compression ratio at the same bit rate. Furthermore, since contextual information relating to various aspects of the current image can be represented in a higher dimension in the feature domain, context-based image encoding / decoding can achieve higher reconstruction quality. Thus, performance improvements are achieved in both reconstruction quality and compression efficiency.

[0030] Some example implementations of this disclosure will be described in more detail below with reference to the accompanying drawings.

[0031] First refer to Figure 2 The diagram illustrates a schematic block diagram of a context-based video encoding / decoding system 200 according to some implementations of the present disclosure. System 200 includes an encoder 210, a decoder 220, and a context generator 230.

[0032] Encoder 210 is configured to generate the image to be encoded. Encoded representation of 202 (referred to as the target image in this paper) Also known as latent coding. Target image 202 may include a frame at time t in a video segment. In some implementations, system 200 may also include an entropy model 250, which is configured to perform entropy encoding (on the encoding side) or entropy decoding (on the decoding side). On the encoding side, entropy model 250 modifies the encoded representation. Quantization is performed to obtain the quantized encoding representation. And from quantization encoding representation Determine the bitstream 214 of the target image 202.

[0033] On the decoding side, the bitstream 214 corresponding to the target image 202 can be received, and a quantized encoded representation can be generated from the bitstream 214. Decoder 220 is configured to generate the target image. Decoded image corresponding to 202 222. Decoder 220 can handle quantized encoded representations. Decode the image to determine the decoded image 222.

[0034] In some implementations, encoder 210 and decoder 220 may reside in the same or different devices. When located in different devices, each device may also include a context generator 230 and an entropy model 250.

[0035] According to the example implementation of this disclosure, the target image The encoding and decoding of 202 are based on its reference image 240. Reference image 240 may include the decoded image at time t-1 prior to t in the video segment. On the decoding side, the decoded image can be obtained directly. As a reference image 240, the decoded image can be generated on the encoding side by performing corresponding operations on the decoding side. Used as a reference image 240. In other implementations, it is also possible to select an image considered to be the target image. 202 Other images with temporal correlation are used as reference images 240, for example, decoded images at one or more other times before or after t can be selected as reference images.

[0036] Context generator 230 is configured to extract reference images. 240 contextual features are represented as 232 (represented as) (In reference image) 240 and target image Under the assumption of temporal correlation, Contextual Feature Representation 202 232 can represent the target image in the feature domain. 202 Related contextual information.

[0037] In this paper, "feature representation" refers to the representation of corresponding feature information (in this case, context information) in the form of a vector, which may have multiple dimensions. "Feature representation" is sometimes also referred to as "vectorized representation," "eigenvector," or "feature," and these terms are used interchangeably in this paper.

[0038] In some implementations, the context generator 230 can utilize machine learning models to extract context feature representations. 232. Some example implementations of context feature extraction will be referenced below. Figure 4 Let's discuss this in more detail.

[0039] During the encoding process, contextual feature representation 232 is provided to encoder 210. Encoder 210 is configured to use context-based feature representation. 232 to encode the target image 202. Contextual Feature Representation 232 was provided as the target image for encoding. Condition 202 is used to help with better encoding. Encoder 210 is configured to represent features within a given context. Under condition 232, perform the operation on the target image. The 202 encoding yields the encoded representation. Such encoding is also called conditional encoding, and encoder 210 can be a context encoder. In this paper, conditional encoding and decoding refers to using arbitrary information as conditions to aid in the encoding and decoding of an image.

[0040] Accordingly, during the decoding process, contextual feature representation 232 is provided to decoder 220. Decoder 220 is configured to be based on context feature representation. 232 is used to decode and obtain the target image. The decoded image 222 corresponds to 202. Decoder 220 is configured to represent features within a given context. Under condition 232, perform the operation on the target image. Conditional decoding of 202. The decoding side also includes a context generator 230. In some implementations, the bitstream 214 is received on the decoding side and based on context feature representations. 232 is used to decode the decoded image 222 from the bitstream 214.

[0041] Starting from conventional residual-based encoding and decoding schemes, when it is desired to obtain certain conditions to guide encoding and decoding, a direct approach might be to use the current target image... Predicted image As a condition, such conditional encoding and decoding can be represented as:

[0042]

[0043] In the above formula (2), Indicates that in a given predicted image Under the condition of target image The encoding, and Indicates that in a given predicted image Decoding the encoded result is performed under these conditions. However, such conditions are still limited by the pixel domain of the image, where each pixel can only be represented by a limited number of channel dimensions (e.g., values ​​in the three dimensions of RGB). Such conditions will limit the representation of contextual information.

[0044] In the implementation of this disclosure, by means of a reference image 240 uses higher-dimensional contextual feature representations in the feature domain to characterize richer and more relevant contextual information for encoding the target image. Furthermore, because the feature representations have higher-dimensional information representation capabilities, different channels in the contextual feature representation 232 can extract different types of contextual information with greater freedom, including color information, texture information, high-frequency component information, object edge information, and so on.

[0045] In some implementations, context-based image encoding and decoding can be represented as follows:

[0046]

[0047] In the above formula (3), This indicates the encoding process of encoder 210. This indicates the decoding process of decoder 220. This indicates the processing operation of the context generator 230, and This indicates quantization achieved through a rounding operation.

[0048] Based on an example implementation of this disclosure, a context-based image encoding / decoding scheme is proposed, particularly a machine learning-based context image encoding / decoding. In the feature domain, a higher-dimensional contextual feature representation is used to characterize richer and more relevant contextual information for encoding the target image. By extracting various contextual features from the contextual feature representation, context-based image encoding / decoding achieves higher reconstruction quality, especially for images with complex textures and more high-frequency content.

[0049] Figure 3 This demonstrates the ability of the context feature representation 232 extracted by the context generator 230 to represent contextual information. For example... Figure 3 As shown, a target image 310 and a reference image 312 are provided. Feature map 320 includes feature maps 321, 322, 323, and 324 from four different channels of the contextual feature representation extracted from the target image 310. These four channels have different emphases.

[0050] Feature map 321 focuses on extracting motion information because the basketball player in motion shown therein has higher intensity, corresponding to the motion vector between target image 310 and reference image 312. The high-intensity regions in the visual representation 314. By comparing with the visual representation 330 of the high-frequency content in the target image 310, it can be seen that feature map 323 focuses more on the high-frequency content to characterize the feature information related to the high-frequency content. In contrast, feature maps 322 and 324 focus more on color information, with feature map 322 focusing on green and feature map 324 focusing more on red.

[0051] Figure 3 Figure 340 illustrates the reduction in reconstruction error achieved by the context coding / decoding scheme implemented according to the examples of this disclosure, compared to a conventional residual coding / decoding scheme. As can be seen from Figure 340, the context coding / decoding scheme implemented according to the examples of this disclosure achieves a significant reduction in error, particularly in high-frequency regions in the foreground and background. Such high-frequency regions are considered difficult to compress for many conventional codecs.

[0052] In some implementations, encoder 210 can be configured to perform conditional coding using an encoding model. Context feature representation. 232 and target image 202 is provided as input to the coding model, which processes the image and outputs a coded representation corresponding to the target image 202.

[0053] In some implementations, decoder 220 can also be configured to perform conditional decoding using a decoding model. Context feature representation 232 and target image The encoding representation of 202, such as quantization encoding. It is provided as input to the decoding model, so that the decoding model processes and outputs the decoded image 222 corresponding to the target image 202.

[0054] The encoding and decoding models can be implemented based on various machine learning or deep learning techniques. For example, the encoding and decoding models can be based on neural networks (NNs), where each model has multiple network layers, which may include, for example, one or more convolutional layers, generalized normalization (GDN) layers (for the encoding model), inverse GDN (IGND) layers (for the decoding model), residual block layers, etc. In the implementation of this disclosure, the configuration of the encoding and decoding models is unrestricted.

[0055] By utilizing machine learning techniques, the encoding model can automatically learn the target image. 202 and contextual feature representation The correlation between 232 is used to reduce the encoding of redundant information, rather than removing redundancy through fixed subtraction operations as in conventional residual coding and decoding schemes.

[0056] On the other hand, the encoding model can also adaptively learn how to use contextual feature representations. 232. For example, due to motion in the video, new content may always appear in the edge region of an object. In this case, since residual-based encoding and decoding schemes always require encoding the residual, and for newly appearing content, the residual is very large, inter-frame coding performed via subtraction may be less efficient than intra-frame coding. Conversely, the context encoding and decoding proposed in the implementation of this disclosure can adaptively utilize contextual features to represent this condition. For newly appearing content, the coding model can adaptively learn to perform intra-frame coding, thereby significantly improving compression efficiency. Figure 3 As shown in Figure 340, the reconstruction error is significantly reduced for new content appearing in the target image 310. This demonstrates that the context encoding / decoding proposed in this disclosure can effectively encode new content caused by motion and significantly reduce reconstruction error.

[0057] In addition to being used for encoding and decoding the target image 202 in encoder 210 and decoder 220, in some implementations, contextual feature representation... 232 can also be used in entropy model 250 for entropy encoding of the coded representation generated from target image 202 to obtain bitstream 214, or for entropy decoding of bitstream 214 to generate a corresponding quantized coded representation for decoding by decoder 220. Example processing of entropy model 250 will be referenced below. Figure 5 Let's discuss this in more detail.

[0058] Extraction of contextual feature representations

[0059] In some implementations, the machine learning model utilized by the context generator 230 can be a reference image. 240 as input, from the reference image 240 Extracting contextual feature representations 232.

[0060] In some implementations, considering that video clips often contain various types of content and may contain many complex movements, motion-related information can be used to help extract better contextual feature representations. 232. For example, for the target image One of the locations in 202, reference image Among 240, the same positions may have little correlation. In this case, contextual features represent... The same location in the feature map of 232 is also the target image. The location in 202 has low relevance, and less relevant contextual information may not be helpful in understanding the target image. The 202 compression encoding is used. Based on this, some implementations propose using motion-related information, such as motion vector (MV) information, to extract contextual feature representations. 232.

[0061] Figure 4 Some implementations of this disclosure are shown in Figure 2 A block diagram of an example structure of the context generator 230 in the system. Figure 4 In the example implementation, the context generator 230 includes a feature extractor 410, which is configured to extract features from a reference image. 240 Extracting initial contextual feature representations The feature extractor 410 can be implemented by a machine learning model to transform the reference image 240 from the pixel domain to the feature domain.

[0062] Context generator 230 also includes methods for determining reference images. 240 and target image The component containing motion vector information between 202. Figure 4 The context generator 230 is shown to include a motion estimator 420, an MV encoder 430, and an MV decoder 440, for estimating motion vector information.

[0063] The motion estimator 420 is configured to be based on the target image 202, Generate motion vector information between time t-1 and time t. In some examples, the motion estimator 420 can utilize an optical flow estimation model to determine the optical flow between time t-1 and time t as motion vector information. Optical flow refers to the instantaneous velocity of pixels moving on the imaging plane of a spatially moving object. Therefore, after training, an optical flow estimation model can utilize the temporal changes of pixels in an image sequence and the correlation between adjacent images to find the correspondence between the previous and current moments, thereby calculating the motion information of objects between adjacent images. Any existing or future motion vector estimation technique can be used to determine the motion vector information. Implementation of this disclosure is not limited in this respect.

[0064] The MV encoder 430 is configured to process motion vector information. Encoding is performed to obtain the encoded representation of the motion vector information (represented as...) ). with target image The processing of the encoded representation of 202 is similar; the encoded representation can be entropy encoded using an entropy model to obtain the 432 bitstream. The bitstream corresponding to the motion vector information can be correlated with the target image. The 202 bitstream is transmitted together to the decoder. Therefore, on the decoder side, the motion estimator 420 and MV encoder 430 are absent. The MV decoder 440 is configured to process the motion vector information. The bitstream 432 generates a quantized encoded representation. and quantization encoding representation Decode the motion vector information to obtain the decoded motion vector information. The MV encoder 430 and MV decoder 440 can also be implemented based on machine learning models.

[0065] Context generator 230 also includes sampling module 450, which is configured to sample based on decoded motion vector information. To adjust the initial contextual feature representation extracted by feature extractor 410 This is done to extract contextual information that is more relevant to the target image 202. In some implementations, the sampling module 450 is configured to transform the initial contextual feature representation through a warping operation. To obtain the intermediate context feature representation The processing of the sampling module 450 can be represented as follows: Where warp() represents the warping operation performed by sampling module 450. Decoding motion vector information. It can be used to guide the initial context feature representation. Interpolation sampling is performed on the values ​​of each element in the dataset.

[0066] Intermediate context feature representation It can be considered to be able to represent context information relatively coarsely because the warping operation can introduce some spatial discontinuities. The context generator 230 may also include a context fine-tuning module 460, which is configured to refine the intermediate context feature representations. Generate the final context feature representation 232, of which The context fine-tuning module 460 can also utilize machine learning models to fine-tune feature representations. These machine learning models may include, for example, multiple network layers, such as one or more convolutional layers, residual block layers, etc.

[0067] In some implementations, context-based image encoding and decoding can be represented as follows:

[0068]

[0069] In the above formula (4), ( ) indicates the feature extraction process of feature extractor 410, and warp ( ) indicates the warping operation performed by sampling module 450. This indicates the context fine-tuning module 460.

[0070] The above text combined Figure 4 An example implementation of extracting contextual feature representations based on motion vector information is described. It should be understood that other methods, such as configuring various other types of machine learning models, can also be used to extract contextual feature representations from a reference image to facilitate the encoding and decoding of the target image. The implementations disclosed herein are not limited in this respect.

[0071] Example implementation of the entropy model

[0072] As briefly mentioned above, in some implementations, contextual feature representation 232 can also be used in entropy model 250 to perform entropy encoding or entropy decoding on image 202. The entropy model is a commonly used quantization coding model in image encoding and decoding. On the encoding side, entropy model 250 can obtain the encoded representation output from encoder 210. A bitstream 214 is generated. On the decoding side, the entropy model 250 is able to determine the quantized encoded representation of the target image 202 from the bitstream 214. , so that it can be further decoded by decoder 220.

[0073] The entropy model primarily considers estimating the probability distribution and quantization encoding representation. The cross-entropy between the distributions, which is the lower bound of the actual bitrate, can be expressed as:

[0074]

[0075] in and These represent quantization encoding respectively. The estimated probability mass distribution and the actual probability mass function; Indicates the actual bit rate. This represents the cross-entropy.

[0076] In fact, arithmetic encoding and decoding can encode quantized representations at almost the same code rate as cross-entropy. However, the actual bitrate The difference between cross-entropy and cross-entropy still exists. Therefore, in some implementations of this disclosure, contextual feature representations are introduced. 232 enables the entropy model 250 to more accurately estimate the probability distribution of the hidden code. .

[0077] Figure 5 Some implementations of this disclosure are shown in Figure 2 A block diagram of an example structure of the entropy model 250 in the system. Figure 5 In the entropy model 250, a temporal correlation component 510 is configured to be based on contextual feature representations. 232 to determine the target image 202 and reference image Temporal correlation information between 240. The temporal correlation component 510 can be derived from contextual feature representation using a temporal prior coding model 512. 232. Determine temporal correlation information. Temporal correlation information provides prior temporal information so that the temporal correlation between processed implicit codes can be taken into account.

[0078] Apart from the temporal correlation component 510, the entropy model 250 includes a typical side information extraction component 520, used to extract information from the encoded representation. Extract edge information and spatial correlation part 530, used from the encoded representation Extract spatial correlation information. Side information provides hierarchical prior information in the target image 202, while spatial correlation information provides spatial prior information. The side information extraction part 520 and the spatial correlation part 530 can be implemented using the modules in the conventional entropy model that extract these two types of information. Figure 5 Only example implementations of these two parts are shown.

[0079] like Figure 5 As shown, the edge information extraction part 520 includes a hyper-prior encoder (HPE) 521, used to extract the encoded representation. Encoding to obtain intermediate encoded representation The quantization (Q) unit 522 is used to represent intermediate codes. Quantization is performed to obtain the quantized encoding representation. The arithmetic encoder (AE) 523 is used to quantize the quantized encoded representation to obtain the bitstream 524 corresponding to the edge information; the arithmetic decoder (AD) 525 is used to decode the bitstream 524 corresponding to the edge information to obtain the quantized encoded representation. ; and a hyper-prior decoder (HPD) 526, used for quantizing the arithmetic decoder representation. Decoding is performed to obtain the side information. The corresponding bitstream 524 can be transmitted to the decoding side.

[0080] Entropy model 250 also includes a quantizer (Q) 550 for encoding representations. Quantization is performed to obtain the quantized encoding representation. The quantization encoding output by quantizer 550 represents... The spatial correlation component 530 is provided. The spatial correlation component 530 can utilize an auto-regressive model 532 to perform the transformation from the quantized encoded representation. Obtain spatial correlation information of target image 202.

[0081] In some implementations, temporal correlation information, edge information, and spatial correlation information are provided to the prior fusion module 560. The prior fusion module 560 is configured to fuse the temporal correlation information, edge information, and spatial correlation information to determine the mean of the probability distribution at time t. and variance Mean and variance It can be provided to the AE 552. The AE 552 is configured to be based on the mean. and variance The quantized encoding representation output by quantizer 550 Arithmetic coding is performed to obtain the bitstream 554 corresponding to the target image 202. The arithmetic-coded representation 554 is provided to the AD 556, which is configured to be based on the mean. and variance Quantized encoding representation is decoded from bitstream 554. .

[0082] In some implementations, HPE 521, quantizers 522 and AE 523, and quantizers 550 and AE 552 in the side information extraction section 520 are only included on the encoding side and may not be needed on the decoding side. The bitstream 524 of the side information extracted by the side information extraction section 520 can be transmitted to the decoding side for use during decoding. During decoding, the quantized encoding representation can be determined based on the bitstream 554 corresponding to the target image 202 via AD 556. In this process, the mean is still provided by the prior fusion module 560. and variance Information. Quantization encoding representation. It is provided to decoder 220 to generate a decoded image.

[0083] In some implementations, this is achieved through entropy model 250 processing. The determination of can be expressed as follows:

[0084]

[0085] In equation (6) above, index i represents the spatial location in the image, assuming It follows a Laplace distribution. Of course, we can also assume... It conforms to other parts, such as Gaussian parts, Gaussian mixture parts, etc. In the above equation (6), This indicates the processing of HPD 526; This indicates the processing of the autoregressive model 532; This indicates the processing of the temporal prior coding model 512. This indicates the processing of the prior fusion module 560.

[0086] It should be understood that Figure 5 The example given is one for determining edge information and time-related information. Other techniques can be used to determine edge information and time-related information in other examples. Alternatively or additionally, other information can be determined to be used together with the time-related information given by the context feature representation for entropy encoding or entropy decoding of the encoded representation.

[0087] Typically, extracting spatial correlations takes a relatively long time. In some implementations, the spatial correlation component 530 can be ignored from the entropy model 250. For example, the spatial correlation component 530 can be bypassed using a switch module 534. The prior fusion module 560 and subsequent modules generate the bitstream 214 based on temporal correlation information and edge information. Through numerous experiments, the inventors found that omitting spatial correlation information has a very small impact on reconstruction quality, but can significantly improve processing efficiency.

[0088] Example implementation of model training

[0089] In the above description, many components of system 200 can be implemented by machine learning models, therefore the parameters of these machine learning models need to be determined through a training process. Various appropriate model training techniques can be used to train the various machine learning models in system 200. In some implementations, the training loss function can be configured based on the distortion and bitrate overhead of the decoded image. For example, the loss function can be determined as follows:

[0090]

[0091] Where parameters It can be a preset value used to control the degree of distortion. and bitrate overhead The previous trade-offs. In some examples, distortion depends on different application requirements. It can be represented by mean squared error (MSE) or multi-scale structural similarity (MS-SSIM). During training, It can be determined as the cross-entropy between the true probability distribution and the estimated probability distribution of the quantized encoding representation.

[0092] Example performance comparison

[0093] Figure 6 The presentation compares the proposed contextual video compression scheme (DCVC) with four conventional codec schemes in terms of performance metrics: reconstruction quality (represented by PSNR, where PSNR refers to peak signal-to-noise ratio) and bit rate overhead (BPP, bits per pixel). The four conventional codec schemes are denoted as DVC (deep video compression), DVCPro, x264, and x265 (selected at the "very slow" configuration level).

[0094] Graphs 610, 620, 630, 640, 650, and 660 show the performance metrics of the five schemes on two video datasets: MCL-JCV, UVG, HEVC Class B, HEVC Class C, HEVC Class D, and HEVC Class E, respectively. These graphs show that, at the same BPP (Bounds Per Pixel), the proposed context-coding / decoding scheme DCVC achieves higher reconstruction quality, i.e., PSNR. Conversely, at the same PSNR, the proposed context-coding / decoding scheme DCVC achieves lower BPP.

[0095] Example process

[0096] Figure 7 A flowchart of an image encoding / decoding process 700 according to some implementations of this disclosure is shown. This process 700 can be implemented in… Figure 2 The system has 200 locations.

[0097] In box 700, a reference image for the target image is obtained. In box 720, a contextual feature representation of the reference image is extracted. The contextual feature representation characterizes the contextual information associated with the target image. In box 730, conditional encoding or decoding of the target image is performed based on the contextual feature representation.

[0098] In some implementations, performing conditional encoding of the target image includes generating an encoded representation of the target image by applying the context feature representation and the target image as input to an encoding model, wherein the encoding model is configured to perform conditional encoding; in some implementations, decoding the target image includes generating a decoded image corresponding to the target image by applying the context feature representation and the encoded representation of the target image as input to a decoding model, wherein the decoding model is configured to perform conditional decoding.

[0099] In some implementations, extracting the contextual feature representation of the reference image includes: extracting an initial contextual feature representation from the reference image; determining motion vector information between the reference image and the target image; and adjusting the initial contextual feature representation based on the motion vector information to obtain the contextual feature representation.

[0100] In some implementations, performing conditional encoding or decoding of the target image further includes: determining temporal correlation information between the target image and the reference image based on the context feature representation; and performing entropy encoding or entropy decoding of the target image based at least on the temporal correlation information.

[0101] In some implementations, performing entropy encoding or entropy decoding on the target image includes: acquiring side information of the target image; and performing entropy encoding or entropy decoding on the target image based at least on the temporal related information and the side information.

[0102] In some implementations, performing entropy encoding or entropy decoding on the target image includes: obtaining spatial correlation information of the target image from the encoded representation; and performing entropy encoding or entropy decoding on the target image based at least on the temporal correlation information and the spatial correlation information.

[0103] In some implementations, performing the entropy encoding includes: acquiring an encoded representation of the target image, and generating a bitstream of the target image from the encoded representation of the target image, at least based on the temporal related information. In some implementations, performing the entropy decoding includes: acquiring the bitstream of the target image, determining an encoded representation of the target image from the bitstream, at least based on the temporal related information, and determining the decoded image from the encoded representation of the target image.

[0104] Example device

[0105] Figure 8 A block diagram of a computing device 800 capable of implementing multiple implementations of the present disclosure is shown. It should be understood that... Figure 8The computing device 800 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. The computing device 800 can be used to implement image encoding and / or image decoding processes according to the implementations herein.

[0106] like Figure 8 As shown, the computing device 800 includes a computing device in the form of a general-purpose computing device. Components of the computing device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage devices 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860.

[0107] In some implementations, the computing device 800 can be implemented as various user terminals or service terminals with computing capabilities. Service terminals can be servers, large computing devices, etc., provided by various service providers. User terminals can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, sites, units, devices, multimedia computers, multimedia tablets, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that the computing device 800 can support any type of user-facing interface (such as "wearable" circuitry).

[0108] Processing unit 810 can be a physical or virtual processor and is capable of performing various processes according to the program stored in memory 820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 800. Processing unit 810 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0109] Computing device 800 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Memory 820 may include image encoding / decoding modules 822, which are configured to perform the functions of the various implementations described herein. Image encoding / decoding modules 822 can be accessed and run by processing unit 810 to implement the corresponding functions.

[0110] Storage device 830 may be a removable or non-removable medium and may include machine-readable media capable of storing information and / or data and accessible within computing device 800. Computing device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 8 As shown, disk drives for reading from or writing to removable, non-volatile disks and optical disc drives for reading from or writing to removable, non-volatile optical discs can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces.

[0111] The communication unit 840 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 800 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 800 can operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or other general network nodes.

[0112] Input device 850 can be one or more various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 860 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 800 can also communicate as needed with one or more external devices (not shown) via communication unit 840. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 800, or with any device that enables computing device 800 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0113] In some implementations, in addition to being integrated into a single device, some or all of the components of computing device 800 may be configured in the form of a cloud computing architecture. In a cloud computing architecture, these components can be remotely deployed and can work together to achieve the functionality described herein. In some implementations, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing provides services over a wide area network (WAN), such as the Internet, using appropriate protocols. For example, cloud computing providers offer applications over a WAN, and these applications can be accessed via a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated at remote data center locations or they may be distributed. Cloud computing infrastructure can provide services through shared data centers, even if they appear as a single access point for users. Therefore, the components and functionality described herein can be provided from service providers at remote locations using a cloud computing architecture. Alternatively, they may also be provided from conventional servers, or they may be installed directly or otherwise on client devices.

[0114] The computing device 800 can be used to implement context image encoding and decoding in various implementations of this disclosure. The computing device 800, such as memory 820, includes an image encoding / decoding module 822. When implementing image encoding, the image encoding / decoding module 822 can be configured to implement the image encoding-related functions described above. When implementing image decoding, the image encoding / decoding module 822 can be configured to implement the image decoding-related functions described above.

[0115] The computing device 800 can receive input 870 via input device 850 or communication unit 840. During encoding, input 870 includes the target image to be encoded. During decoding, input 870 includes the bitstream to be decoded. Input 870 is provided to image encoding / decoding module 822 to perform image encoding / decoding operations. During encoding, image encoding / decoding module 822 generates the bitstream of the target image as output 880. During decoding, image encoding / decoding module 822 generates the decoded image of the target image as output 880. In some implementations, output 800 can be output by output device 860, or it can be transmitted to other devices by communication unit 840.

[0116] Example implementation

[0117] The following are some example implementations of this disclosure.

[0118] In one aspect, this disclosure provides a computer-implemented method. The method includes: acquiring a reference image of a target image; extracting a contextual feature representation of the reference image, the contextual feature representation characterizing contextual information associated with the target image; and performing conditional encoding or decoding of the target image based on the contextual feature representation.

[0119] In some implementations, performing conditional encoding of the target image includes generating an encoded representation of the target image by applying the context feature representation and the target image as input to an encoding model, wherein the encoding model is configured to perform the conditional encoding; in some implementations, performing conditional decoding of the target image includes generating a decoded image corresponding to the target image by applying the context feature representation and the encoded representation of the target image as input to a decoding model, wherein the decoding model is configured to perform the conditional decoding.

[0120] In some implementations, extracting the context feature representation of the reference image includes: extracting an initial context feature representation from the reference image; determining motion vector information between the reference image and the target image; and adjusting the initial context feature representation based on the motion vector information to obtain the context feature representation.

[0121] In some implementations, performing conditional encoding or decoding of the target image further includes: determining temporal correlation information between the target image and the reference image based on the context feature representation; and performing entropy encoding or entropy decoding of the target image based at least on the temporal correlation information.

[0122] In some implementations, performing entropy encoding or entropy decoding on the target image includes: acquiring side information of the target image; and performing entropy encoding or entropy decoding on the target image based at least on the temporal related information and the side information.

[0123] In some implementations, performing entropy encoding or entropy decoding on the target image includes: obtaining spatial correlation information of the target image from the encoded representation; and performing entropy encoding or entropy decoding on the target image based at least on the temporal correlation information and the spatial correlation information.

[0124] In some implementations, performing the entropy encoding includes: acquiring an encoded representation of the target image, and generating a bitstream of the target image from the encoded representation of the target image, at least based on the temporal related information. In some implementations, performing the entropy decoding includes: acquiring the bitstream of the target image, determining an encoded representation of the target image from the bitstream, at least based on the temporal related information, and determining the decoded image from the encoded representation of the target image.

[0125] In another aspect, this disclosure provides an electronic device. The electronic device includes: a processor; and a memory coupled to the processor and containing instructions stored thereon, the instructions, when executed by the processor, causing the device to perform the following actions: acquiring a reference image of a target image; extracting a contextual feature representation of the reference image, the contextual feature representation characterizing contextual information associated with the target image; and performing conditional encoding or decoding of the target image based on the contextual feature representation.

[0126] In some implementations, performing conditional encoding of the target image includes generating an encoded representation of the target image by applying the context feature representation and the target image as input to an encoding model, wherein the encoding model is configured to perform the conditional encoding; in some implementations, performing conditional decoding of the target image includes generating a decoded image corresponding to the target image by applying the context feature representation and the encoded representation of the target image as input to a decoding model, wherein the decoding model is configured to perform the conditional decoding.

[0127] In some implementations, extracting the context feature representation of the reference image includes: extracting an initial context feature representation from the reference image; determining motion vector information between the reference image and the target image; and adjusting the initial context feature representation based on the motion vector information to obtain the context feature representation.

[0128] In some implementations, performing conditional encoding or decoding of the target image further includes: determining temporal correlation information between the target image and the reference image based on the context feature representation; and performing entropy encoding or entropy decoding of the target image based at least on the temporal correlation information.

[0129] In some implementations, performing entropy encoding or entropy decoding on the target image includes: acquiring side information of the target image; and performing entropy encoding or entropy decoding on the target image based at least on the temporal related information and the side information.

[0130] In some implementations, performing entropy encoding or entropy decoding on the target image includes: obtaining spatial correlation information of the target image from the encoded representation; and performing entropy encoding or entropy decoding on the target image based at least on the temporal correlation information and the spatial correlation information.

[0131] In some implementations, performing the entropy encoding includes: acquiring an encoded representation of the target image, and generating a bitstream of the target image from the encoded representation of the target image, at least based on the temporal related information. In some implementations, performing the entropy decoding includes: acquiring the bitstream of the target image, determining an encoded representation of the target image from the bitstream, at least based on the temporal related information, and determining the decoded image from the encoded representation of the target image.

[0132] In another aspect, this disclosure provides a computer program product tangibly stored in a computer storage medium and including computer-executable instructions that, when executed by a device, cause the device to perform the following actions: acquiring a reference image of a target image; extracting a contextual feature representation of the reference image, the contextual feature representation characterizing contextual information associated with the target image; and performing conditional encoding or decoding of the target image based on the contextual feature representation.

[0133] In some implementations, performing conditional encoding of the target image includes generating an encoded representation of the target image by applying the context feature representation and the target image as input to an encoding model, wherein the encoding model is configured to perform the conditional encoding; in some implementations, performing conditional decoding of the target image includes generating a decoded image corresponding to the target image by applying the context feature representation and the encoded representation of the target image as input to a decoding model, wherein the decoding model is configured to perform the conditional decoding.

[0134] In some implementations, extracting the context feature representation of the reference image includes: extracting an initial context feature representation from the reference image; determining motion vector information between the reference image and the target image; and adjusting the initial context feature representation based on the motion vector information to obtain the context feature representation.

[0135] In some implementations, performing conditional encoding or decoding of the target image further includes: determining temporal correlation information between the target image and the reference image based on the context feature representation; and performing entropy encoding or entropy decoding of the target image based at least on the temporal correlation information.

[0136] In some implementations, performing entropy encoding or entropy decoding on the target image includes: acquiring side information of the target image; and performing entropy encoding or entropy decoding on the target image based at least on the temporal related information and the side information.

[0137] In some implementations, performing entropy encoding or entropy decoding on the target image includes: obtaining spatial correlation information of the target image from the encoded representation; and performing entropy encoding or entropy decoding on the target image based at least on the temporal correlation information and the spatial correlation information.

[0138] In some implementations, performing the entropy encoding includes: acquiring an encoded representation of the target image, and generating a bitstream of the target image from the encoded representation of the target image, at least based on the temporal related information. In some implementations, performing the entropy decoding includes: acquiring the bitstream of the target image, determining an encoded representation of the target image from the bitstream, at least based on the temporal related information, and determining the decoded image from the encoded representation of the target image.

[0139] In another aspect, this disclosure provides a computer-readable medium having computer-executable instructions stored thereon, which, when executed by a device, cause the device to perform the methods described above.

[0140] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, example types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Load Programmable Logic Devices (CPLDs), and so on.

[0141] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of a single implementation may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0144] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image encoding / decoding method, comprising: Obtain a reference image for the target image; The initial contextual feature representation of the reference image is extracted by using a machine learning model to transform the reference image from the pixel domain to the feature domain; Determine the motion vector information between the reference image and the target image; Based on the motion vector information, the initial context feature representation is transformed into an intermediate context feature representation; The intermediate context feature representation is fine-tuned using a machine learning model to generate a final context feature representation that represents the contextual information associated with the target image. as well as Conditional encoding or decoding of the target image is performed based on the contextual feature representation.

2. The method according to claim 1, The conditional encoding of the target image includes: An encoded representation of the target image is generated by applying the contextual feature representation and the target image as input to an encoding model, the encoding model being configured to perform the conditional encoding; or The conditional decoding of the target image includes: A decoded image corresponding to the target image is generated by applying the context feature representation and the encoded representation of the target image as input to a decoding model, wherein the decoding model is configured to perform the conditional decoding.

3. The method according to claim 1, wherein performing the conditional encoding or decoding of the target image further comprises: The temporal correlation information between the target image and the reference image is determined based on the contextual feature representation; as well as Entropy encoding or entropy decoding of the target image is performed at least based on the temporal correlation information.

4. The method of claim 3, wherein performing entropy encoding or entropy decoding of the target image comprises: Obtain the edge information of the target image; as well as Entropy encoding or entropy decoding of the target image is performed based at least on the temporal correlation information and the edge information.

5. The method of claim 3, wherein performing entropy encoding or entropy decoding of the target image comprises: Spatial correlation information of the target image is obtained from the encoded representation; as well as Entropy encoding or entropy decoding of the target image is performed based at least on the temporal correlation information and the spatial correlation information.

6. The method according to claim 3, The entropy encoding process includes: Obtain the encoded representation of the target image, and Based at least on the temporal correlation information, a bitstream of the target image is generated from the coded representation of the target image, and The entropy decoding process includes: Obtain the bitstream of the target image. Based at least on the temporal correlation information, the encoded representation of the target image is determined from the bitstream, and The decoded image is determined from the encoded representation of the target image.

7. An electronic device, comprising: processor; as well as A memory, coupled to the processor and containing instructions stored thereon, which, when executed by the processor, cause the device to perform the following actions: Obtain a reference image for the target image; The initial contextual feature representation of the reference image is extracted by using a machine learning model to transform the reference image from the pixel domain to the feature domain; Determine the motion vector information between the reference image and the target image; Based on the motion vector information, the initial context feature representation is transformed into an intermediate context feature representation; The intermediate contextual feature representation is fine-tuned using a machine learning model to generate a final contextual feature representation, which characterizes the contextual information associated with the target image; and Conditional encoding or decoding of the target image is performed based on the contextual feature representation.

8. The electronic device according to claim 7, The conditional encoding of the target image includes: An encoded representation of the target image is generated by applying the contextual feature representation and the target image as input to an encoding model, the encoding model being configured to perform the conditional encoding; or The conditional decoding of the target image includes: A decoded image corresponding to the target image is generated by applying the context feature representation and the encoded representation of the target image as input to a decoding model, wherein the decoding model is configured to perform the conditional decoding.

9. The electronic device of claim 7, wherein performing the conditional encoding or decoding of the target image further comprises: The temporal correlation information between the target image and the reference image is determined based on the contextual feature representation; as well as Entropy encoding or entropy decoding of the target image is performed at least based on the temporal correlation information.

10. The electronic device of claim 9, wherein performing entropy encoding or entropy decoding of the target image comprises: Obtain the edge information of the target image; as well as Entropy encoding or entropy decoding of the target image is performed based at least on the temporal correlation information and the edge information.

11. The electronic device of claim 9, wherein performing entropy encoding or entropy decoding of the target image comprises: Spatial correlation information of the target image is obtained from the encoded representation; as well as Entropy encoding or entropy decoding of the target image is performed based at least on the temporal correlation information and the spatial correlation information.

12. The electronic device of claim 9, wherein performing the entropy encoding comprises: Obtain the encoded representation of the target image, and Based at least on the temporal correlation information, a bitstream of the target image is generated from the coded representation of the target image, and The entropy decoding process includes: Obtain the bitstream of the target image. Based at least on the temporal correlation information, the encoded representation of the target image is determined from the bitstream, and The decoded image is determined from the encoded representation of the target image.

13. A computer program product, said computer program product being tangibly stored in a computer storage medium and including computer-executable instructions, which, when executed by a device, cause the device to perform the following actions, said actions including: Obtain a reference image for the target image; The initial contextual feature representation of the reference image is extracted by using a machine learning model to transform the reference image from the pixel domain to the feature domain; Determine the motion vector information between the reference image and the target image; Based on the motion vector information, the initial context feature representation is transformed into an intermediate context feature representation; The intermediate context feature representation is fine-tuned using a machine learning model to generate a final context feature representation that represents the contextual information associated with the target image. as well as Conditional encoding or decoding of the target image is performed based on the contextual feature representation.

14. The computer program product according to claim 13, The conditional encoding of the target image includes: An encoded representation of the target image is generated by applying the contextual feature representation and the target image as input to an encoding model, the encoding model being configured to perform the conditional encoding; or The conditional decoding of the target image includes: A decoded image corresponding to the target image is generated by applying the context feature representation and the encoded representation of the target image as input to a decoding model, wherein the decoding model is configured to perform the conditional decoding.

15. The computer program product of claim 13, wherein performing the conditional encoding or decoding of the target image further comprises: The temporal correlation information between the target image and the reference image is determined based on the contextual feature representation; as well as Entropy encoding or entropy decoding of the target image is performed at least based on the temporal correlation information.

16. The computer program product of claim 15, wherein performing entropy encoding or entropy decoding of the target image comprises: Obtain the edge information of the target image; as well as Entropy encoding or entropy decoding of the target image is performed based at least on the temporal correlation information and the edge information.

17. The computer program product of claim 15, wherein performing entropy encoding or entropy decoding of the target image comprises: Spatial correlation information of the target image is obtained from the encoded representation; as well as Entropy encoding or entropy decoding of the target image is performed based at least on the temporal correlation information and the spatial correlation information.

18. The computer program product according to claim 15, The entropy encoding process includes: Obtain the encoded representation of the target image, and Based at least on the temporal correlation information, a bitstream of the target image is generated from the coded representation of the target image, and The entropy decoding process includes: Obtain the bitstream of the target image. Based at least on the temporal correlation information, the encoded representation of the target image is determined from the bitstream, and The decoded image is determined from the encoded representation of the target image.

Images