Video coding and decoding method and video coding device based on neural network

Abstract

A method for neural network-based video coding is provided. The method includes estimating a motion vector between an input image and a reference image based on a temporal layer of the input image, transforming the motion vector into a potential representation, scaling the potential representation of the motion vector based on the temporal layer of the input image, and outputting the scaled potential representation. And obtaining a temporal context of the input image based on the scaled potential representation of the motion vector and the reference image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2023-0102601 filed in the Korean Intellectual Property Office on August 7, 2023, and Korean Patent Application No. 10-2023-0160501 filed in the Korean Intellectual Property Office on November 20, 2023, the disclosures of which are incorporated herein in their entirety for all purposes by reference. Technical Field

[0003] The present disclosure relates to methods and apparatus for encoding and decoding data, and more particularly, to methods and apparatus for neural network-based video encoding and decoding. Background Art

[0004] In recent years, the Internet video market has continued to grow. However, since the type and / or quality of services may be limited by network bandwidth, advanced video coding techniques are required. To this end, neural network-based video coding techniques have been proposed, which can be trained end-to-end by replacing components of ordinary video coding techniques with neural networks. Summary of the invention

[0005] According to one or more aspects of the present disclosure, an apparatus and method for neural network-based video encoding using bidirectional prediction and temporal layer information are provided.

[0006] According to one aspect of the present disclosure, a method for neural network-based video encoding is provided, the method comprising: estimating a motion vector between an input image and a reference image based on a temporal layer of the input image; transforming the motion vector into a potential representation; scaling the potential representation of the motion vector based on the temporal layer of the input image; and obtaining a temporal context of the input image based on the scaled potential representation of the motion vector and the reference image.

[0007] The reference images may include a first bidirectional reference image temporally preceding the input image, and a second bidirectional reference image temporally following the input image.

[0008] Temporal layer based scaling may include scaling the latent representation of the motion vectors by using a smaller quantization step size for higher temporal layers.

[0009] The temporal layer based scaling may include scaling the latent representation of the motion vector by dividing the latent representation of the motion vector by a quantization step size determination parameter defined for the temporal layer.

[0010] The method may also include: performing entropy encoding and entropy decoding on a latent representation of the motion vector; rescaling the scaled latent representation of the motion vector based on a temporal layer; and reconstructing the motion vector based on the rescaled latent representation of the motion vector, wherein obtaining the temporal context may include: obtaining the temporal context based on the reconstructed motion vector and a reference image.

[0011] The temporal layer based rescaling may include multiplying the scaled latent representation of the motion vector by a quantization step size determination parameter defined for the temporal layer.

[0012] The method may further include: quantizing a latent representation of the scaled motion vector before performing entropy encoding and entropy decoding; and estimating a probability distribution of the latent representation of the motion vector.

[0013] Obtaining the temporal context may include: outputting a reference feature map by inputting a reference image into a feature extraction neural network, performing bilinear warping on the reference feature map based on a reconstructed motion vector to output a warped reference feature map, inputting the warped reference feature map into a post-processing neural network, and inputting the output of the post-processing neural network into a context fusion network to output the temporal context.

[0014] According to another aspect of the present disclosure, a method for neural network-based video encoding is provided, which may include: estimating a motion vector between an input image and a reference image based on a temporal layer of the input image; transforming the motion vector into a potential representation; obtaining a temporal context of the input image based on the potential representation of the motion vector and a reference image; and performing a smoothing operation on the temporal context based on a smoothing object, the smoothing object including a reference image, an input image, a motion vector, an input in obtaining the temporal context, an output in obtaining the temporal context, or at least one of an input or an output of a sub-process in obtaining the temporal context.

[0015] The reference images may include a first bidirectional reference image temporally preceding the input image, and a second bidirectional reference image temporally following the input image.

[0016] The performing of the smoothing may include: obtaining a smoothing level of a smoothing object with respect to a temporal layer of an input image; generating a smoothing filter based on the smoothing level; and performing smoothing on the smoothing object based on the smoothing filter.

[0017] The obtaining of the smoothness level of the smooth object may include obtaining a lower smoothness level for a lower temporal layer.

[0018] Obtaining the smoothness level of the smooth object may include: obtaining different smoothness levels according to spatial positions.

[0019] According to another aspect of the present disclosure, a method for neural network-based video decoding is provided, the method comprising: reconstructing a potential representation of a motion vector based on a first bitstream for the motion vector; rescaling the potential representation of the reconstructed motion vector based on a temporal layer; reconstructing the motion vector based on the rescaled potential representation of the motion vector; obtaining a temporal context based on the reconstructed motion vector and a reference image; and reconstructing an input image based on a second bitstream for the input image and the temporal context.

[0020] The reference images may include a first bidirectional reference image temporally preceding the input image, and a second bidirectional reference image temporally following the input image.

[0021] The temporal layer based rescaling may include multiplying the latent representation of the motion vector by a quantization step size determination parameter defined for the temporal layer.

[0022] Reconstructing the input image may include: reconstructing a latent representation of the input image based on a second bitstream for the input image; rescaling the latent representation of the input image based on a temporal layer; and reconstructing the input image based on the rescaled latent representation of the input image and a temporal context.

[0023] According to another aspect of the present disclosure, a method for neural network-based video decoding is provided, the method comprising: reconstructing a potential representation of a motion vector based on a first bit stream for the motion vector; obtaining a temporal context by using the motion vector and a reference image; performing smoothing by using the reference image, the motion vector, an input in obtaining the temporal context, an output in obtaining the temporal context, and at least one of an input and an output of a sub-process in obtaining the temporal context as a smoothing object; and reconstructing an input image based on a second bit stream for the input image and the temporal context.

[0024] The performing of the smoothing may include: obtaining different smoothing levels based on at least one of a temporal layer and a spatial position of the input image; generating a smoothing filter based on the smoothing level; and performing smoothing on the smoothed object based on the smoothing filter.

[0025] According to another aspect of the present disclosure, an electronic device is provided, including: a memory configured to store one or more instructions and a reference image; and a processor configured to execute one or more instructions to: estimate a motion vector between an input image and a reference image based on the reference image, the input image, and a temporal layer of the input image; transform the motion vector into a potential representation; scale the potential representation of the motion vector based on the temporal layer of the input image; and obtain a temporal context of the input image based on the scaled potential representation of the motion vector and the reference image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1A is a flowchart showing a method for neural network-based video encoding according to an embodiment of the present disclosure.

[0027] Figure 1B is a flowchart showing a method for neural network-based video decoding according to an embodiment of the present disclosure.

[0028] Figure 2A is a flowchart showing a method for neural network-based video encoding according to another embodiment of the present disclosure.

[0029] Figure 2B is a flowchart illustrating the operation of creating a bidirectional temporal context according to an embodiment of the present disclosure.

[0030] Figure 3A and Figure 3B is a diagram explaining an example of scaling and rescaling according to time layers.

[0031] Figure 4 is a flow chart illustrating the operation of smoothing bidirectional temporal context according to an embodiment of the present disclosure.

[0032] Figure 5 is a diagram explaining an example of smoothing bidirectional temporal context according to temporal layers.

[0033] Fig. 6A is a flowchart illustrating an operation of encoding an input image according to an embodiment of the present disclosure.

[0034] Figure 6B is a flowchart illustrating an operation of decoding an input image according to an embodiment of the present disclosure.

[0035] Figure 7 is a block diagram showing an electronic device according to an embodiment of the present disclosure.

[0036] FIG. 8A to FIG. 8I is a diagram for explaining in detail a video encoding process according to an embodiment of the present disclosure.

[0037] Fig. 9A is a diagram illustrating an example of a context fusion network.

[0038] Fig. 9B is a diagram showing an example of an entropy model for estimating probability distribution.

[0039] Fig.10 is a block diagram illustrating an electronic device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Details of other embodiments are included in the following detailed description and drawings. Advantages and features of the present invention and methods of implementing the same will be more clearly understood according to the embodiments described in detail below with reference to the drawings. Throughout the drawings and detailed description, unless otherwise described, the same reference numerals will be understood to refer to the same elements, features and structures.

[0041] It will be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish elements from each other. Unless otherwise explicitly stated, any reference to the singular may include the plural. In addition, unless explicitly described to the contrary, expressions such as "including" or "comprising" will be understood to mean including the elements set forth, but not excluding any other elements.

[0042] The embodiments herein can be described and illustrated in accordance with the blocks that perform the one or more functions described. These blocks, which may be referred to herein as units, modules, etc., can be physically implemented by analog and / or digital circuits (e.g., logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.), and can be optionally driven by firmware. For example, the circuit can be implemented in one or more semiconductor chips, or on substrate supports such as printed circuit boards, etc. The circuits constituting the blocks can be implemented by dedicated hardware or by processors (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware for performing some functions of the block and processors for performing other functions of the block. Without departing from the scope of the present disclosure, each block of the embodiment can be physically divided into two or more interactive and discrete blocks. Similarly, without departing from the scope of the present disclosure, the blocks of the embodiment can be physically combined into more complex blocks. However, the present disclosure is not limited thereto, and therefore, the blocks, which may be referred to herein as units, modules, etc., can be software modules implemented by software code, program code, software instructions, etc. Software blocks can be executed on one or more processors. According to the embodiment, terms such as “block”, “unit”, “module”, etc. should be understood as a unit that performs at least one function or operation and may be embodied as hardware, software, or a combination thereof.

[0043] Expressions such as “at least one of…” (for example, the expression “at least one of a, b, and c”) should be interpreted as including only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0044] Figure 1A is a flowchart showing a method for neural network-based video encoding according to an embodiment of the present disclosure.

[0045] refer to Figure 1A , an example of a method for video encoding using bidirectional prediction and / or temporal layer information will be described below. According to an embodiment, the method can be performed by a video encoding device based on a neural network. By using bidirectional prediction, higher encoding performance can be achieved than unidirectional prediction. First, the flow of the method for video encoding using bidirectional prediction will be described, and then the method using temporal layer information will be described.

[0046] According to an embodiment, in operation 111, the method may include estimating a bidirectional motion vector. For example, a video encoding apparatus based on bidirectional prediction may use a bidirectional reference image And the input image x t As input, estimate the bidirectional motion vector m f , m b . Input image x t Refers to an image to be encoded at the current time t in the image sequence constituting the video. According to an embodiment, the reference image may refer to an image that has been decoded (eg, a previous image) and / or an input image x for encoding or decoding. t The bidirectional reference image refers to the input image x t For example, the image before time t from the input image can be called the forward reference image. And the image after time t from the input image can be called the backward reference image There may be one or more forward reference images and backward reference images, but for ease of explanation, the following description will be given using an example in which there is one forward reference image and one backward reference image.

[0047] According to an embodiment, in operation 112, the method may include transforming the bidirectional motion vector into a latent representation. For example, the bidirectional motion vector m may be transformed into a latent representation by using a neural network. f , m b Transformed into the potential representation of the bidirectional motion vector g t The neural network may include, but is not limited to, a convolutional neural network (CNN), etc. Here, the latent representation is the output of a neural network using an input image or motion information as an input, and is collectively referred to as a latent feature, a latent vector, etc.

[0048] According to an embodiment, in operation 113, the method may include performing entropy encoding or entropy decoding on a latent representation of the bidirectional motion vector. For example, the latent representation of the bidirectional motion vector g t can be quantized and entropy coded to be represented as a bitstream of bidirectional motion vectors b m , and is sent to the decoder, and through the bit stream b of the motion vectorm Entropy decoding can reconstruct the potential representation of the quantized motion vector

[0049] According to an embodiment, in operation 114, the method may include reconstructing the bidirectional motion vector. For example, by using the latent representation of the quantized motion vector Bidirectional motion vectors can be reconstructed

[0050] According to an embodiment, in operation 115, the method may include creating or generating a temporal context based on a latent representation of the bidirectional vector. For example, by using the reconstructed bidirectional motion vector and bidirectional reference images A bidirectional temporal context C can be created t The bidirectional temporal context C t It can be a feature map with a single resolution or a feature map with two or more resolutions, and can be the same as the input image x t The prediction information is used to encode and decode the input image x t Used in.

[0051] According to an embodiment, in operation 116, the method may include: transforming the input image into a latent representation based on the bidirectional temporal context. For example, the input image may be transformed into a latent representation based on the bidirectional temporal context C t Transform the input image into a latent representation, and transform the input image x t Transformed into the latent representation y of the input image t .

[0052] According to an embodiment, in operation 117, the method may include performing entropy encoding or entropy decoding on the latent representation of the input image. t can be entropy encoded to be represented as a bit stream b of the input image x , and is sent to the decoder. According to an embodiment, by x Entropy decoding can be performed to reconstruct the potential representation of the quantized input image

[0053] According to an embodiment, in operation 118, the method may include reconstructing the input image based on the bidirectional temporal context. For example, by using the latent representation of the quantized input image The input image can be reconstructed In this case, the bidirectional temporal context C t Can be used as additional input. Reconstructed Image Can be used as a reference image for subsequent images.

[0054] According to an embodiment, the method for video encoding using bidirectional prediction may also use temporal layer l t The method of using the time layer may include: a method of scaling and rescaling according to the time layer, and / or a method of smoothing according to the time layer.

[0055] refer to Figure 1A , you can The potential representation g of the bidirectional motion vector t Perform time-based scaling and / or The potential representation y of the input image t Performs scaling based on time layers and can be The potential representation of the bidirectional motion vector Perform rescaling, and / or The potential representation of the input image Perform rescaling.

[0056] In comparative video encoding methods, different encoding methods and neural network models may be used for temporal layers, and such methods may require storing various neural network models for temporal layers, making the video compression method inefficient in terms of complexity such as memory efficiency. According to an embodiment, by using a single neural network model for the temporal layer, scaling and rescaling may be performed according to the temporal layer. In this case, the latent representation may be scaled differently according to the temporal layer. In scaling according to the temporal layer, before performing uniform quantization on the latent representation, scaling is performed by dividing the latent representation by a scaling parameter determined differently for the temporal layer, and after performing entropy encoding and decoding, rescaling is performed by multiplying the scaled latent representation by the previously determined scaling parameter.

[0057] refer to Figure 1A , which can be in all or some locations Smoothing based on the time layer is performed at Figure 1A An example of performing scaling and smoothing according to a time layer is shown, but this is only exemplary and the present disclosure is not limited thereto, and scaling and smoothing may be selectively performed. Smoothing may be performed on various smoothing objects according to a location where smoothing is performed.

[0058] In the example case where smoothing is performed between the operation 111 of estimating the bidirectional motion vector and the operation 112 of transforming the latent representation of the bidirectional motion vector, the bidirectional motion vector m f , m b is a smoothing object. However, smoothing is not limited to this, and Figure 1AIn addition to the smoothed positions shown, smoothing according to temporal layers may also be performed between the various neural networks used in operation 115 of creating a bidirectional temporal context based on a latent representation of motion vectors.

[0059] Typically, a neural network used to create a bidirectional temporal context is trained to create a bidirectional temporal context that can improve the average coding performance for the input images to be learned. However, in the example case where motion vectors are not estimated correctly, or in the example case where the video contains complex motion, it is possible that the neural network trained to create a bidirectional temporal context may have an adverse effect on the coding performance of each individual input image. For example, in hierarchical B-frame coding, more distant reference images are used compared to unidirectional prediction structures, resulting in a significant decrease in coding performance due to the use of bidirectional temporal contexts that are not optimized for each temporal layer.

[0060] On the other hand, according to an embodiment of the present disclosure, as a method for adjusting an incorrect time context caused by using a bidirectional time context that is not optimized for each time layer, smoothing is performed differently on the time context or information for creating the time context for the time layer, thereby creating a time context that is smoothed differently for the time layer. A higher smoothing level can be used for a higher time layer, and a lower smoothing level can be used for a lower time layer. In this way, the incorrect time context can be adjusted, thereby achieving higher coding performance compared to ordinary techniques.

[0061] Figure 1B is a flowchart showing a method for neural network-based video decoding according to an embodiment of the present disclosure.

[0062] refer to Figure 1B , an example of a method for video decoding using bidirectional prediction and / or temporal layer information performed by a neural network-based video decoding device will be described below. For operational details of the method for video decoding, reference may be made to its corresponding operations described throughout the specification.

[0063] According to an embodiment, in operation 121, the method may include reconstructing a potential representation of the bidirectional motion vector. For example, the video decoding device may reconstruct the potential representation of the bidirectional motion vector by using the bit stream b for the bidirectional motion vector. m To reconstruct the latent representation of the bidirectional motion vector The video encoding device may generate a bit stream b for the bidirectional motion vector by quantization and / or entropy encoding. m , as in the example reference above Figure 1A By using the bitstream b for the motion vector mTo perform entropy decoding, a potential representation of the motion vector can be reconstructed. In this case, entropy decoding can be performed using the mean and standard deviation used in the entropy encoding process. Entropy decoding can be arithmetic decoding.

[0064] According to an embodiment, in operation 122, the method may include reconstructing a bidirectional motion vector. For example, the video decoding device may reconstruct a bidirectional motion vector by using a potential representation of the bidirectional motion vector. To reconstruct the bidirectional motion vector

[0065] According to an embodiment, in operation 123, the method may include creating or generating a bidirectional temporal context based on a latent representation of a bidirectional motion vector. For example, a video decoding device may generate a bidirectional temporal context by using a latent representation of a bidirectional motion vector. To create a temporal context. In this case, the video decoding device can use a bidirectional reference image To create a bidirectional temporal context C t .

[0066] According to an embodiment, in operation 124, the method may include reconstructing a latent representation of the input image. For example, the video decoding apparatus may reconstruct a latent representation of the input image by using a bit stream b of the input image. x Takes as input the latent representation of the input image In this case, the bitstream b of the input image x It can be generated by quantization and / or entropy coding in the video coding process. x To perform entropy decoding, a potential representation of the input image can be reconstructed. In this case, entropy decoding can be performed using the mean and standard deviation used in the entropy encoding process. Entropy decoding can be arithmetic decoding.

[0067] According to an embodiment, in operation 125, the method may include reconstructing the input image based on the bidirectional temporal context. For example, the video decoding apparatus may reconstruct the input image based on the bidirectional temporal context by using the latent representation of the input image and the temporal context Ct. Reconstructing the image Can be used as a reference image for subsequent images.

[0068] According to an embodiment, the method for video decoding using bidirectional prediction may also use a temporal layer. The method using the temporal layer may include a scaling and rescaling method according to the temporal layer, and / or a smoothing method according to the temporal layer, wherein the two methods may be performed during the video decoding process, or one of the two methods may be selectively performed.

[0069] refer to Figure 1B , during video encoding ( Figure 1A Position in In the example case where scaling is performed at a position , the potential representation of the bidirectional motion vector and / or the potential representation of the input image may be scaled at a position The rescaling may be performed by multiplying the latent representation scaled after entropy decoding by a scaling parameter (quantization step size determination parameter) used in the video encoding process.

[0070] According to an embodiment, by using a reference image At least one of a motion vector, an input when creating a temporal context in operation 123, an output when creating a temporal context in operation 123, and an input or output of a subprocess for creating a temporal context in operation 123 is used as a smoothing object to perform smoothing. For example, smoothing may be performed according to a temporal layer. For example, referring to Figure 1B , which can be in all or some locations Smoothing according to the time layer is performed at the position where the smoothing is performed. Smoothing can be performed on various smoothing objects according to the position where the smoothing is performed.

[0071] Figure 2A 1 is a flowchart showing a method for neural network-based video encoding according to another embodiment of the present disclosure. The method may be performed by a device configured to perform video encoding and / or decoding based on a neural network. Figure 2A The process shows that by using Figure 1A The method for video encoding is performed by an example of performing smoothing at an output position in operation 115 of creating a bidirectional temporal context based on a potential representation of a motion vector. Although the smoothed position is shown as the output position in operation 115 of creating a bidirectional temporal context based on a potential representation of a motion vector, the smoothed position is not limited thereto and can be easily applied to other smoothed positions. That is, smoothing can be performed on a reference image, an input image, a motion vector, an input in an operation of creating a temporal context, an output in an operation of creating a temporal context, and an input or output in a detailed operation of creating a temporal context.

[0072] Although the following description will focus on video encoding, the present disclosure is not limited thereto, and thus, the description may also be applied to decoding.

[0073] According to an embodiment, in operation 200, the method may include creating and generating a bidirectional temporal context. For example, in operation 200, the video encoding apparatus may generate a bidirectional temporal context by using a bidirectional reference image. and the temporal layer l of the input image t To output the bidirectional temporal context C of the input image t and / or a bitstream b for bidirectional motion vectors m .

[0074] Input image xt Refers to the image to be encoded at the current time t in the image sequence constituting the video. Reference image refers to the decoded image, as well as the input image x for encoding or decoding. t The bidirectional reference image refers to the input image x t The images before and after time t are called forward reference images. and the backward reference image There may be one or more forward reference images and backward reference images, but for ease of explanation, the following description will be given using an example in which there is one forward reference image and one backward reference image. t It can be a feature map with a single resolution, or a feature map with two or more resolutions. t Refers to each layer in an example case where images in a video are temporally layered so that higher layer images can be encoded even without decoding lower layer images.

[0075] According to an embodiment, in operation 400, the method may include performing a smoothing operation on the bidirectional temporal context. For example, in operation 400, the video compression apparatus may use the bidirectional temporal context C t and the temporal layer l of the input image t , according to the time layer l t For the bidirectional temporal context C t Perform smoothing and output the smoothed bidirectional temporal context In this case, the video compression device may apply different smoothness levels to the bidirectional temporal context according to the temporal layer and / or spatial position so as to adjust the incorrect temporal context caused by using a bidirectional temporal context that is not optimized for each layer. According to an embodiment, the video compression device may apply a first smoothness level to the bidirectional temporal context corresponding to the first temporal layer, and apply a second smoothness level to the bidirectional temporal context corresponding to the second temporal layer. Here, the first smoothness level may be different from the second smoothness level. According to an embodiment, the video compression device may apply a first smoothness level to the bidirectional temporal context corresponding to the first spatial position, and apply a second smoothness level to the bidirectional temporal context corresponding to the second spatial position. Here, the first spatial position may be different from the second spatial position. For example, the spatial position may be a spatial zone or a spatial region.

[0076] According to an embodiment, in operation 600, the method may include: performing an encoding or decoding operation on the input image based on the bidirectional temporal context. For example, in operation 600, by using the smoothed bidirectional temporal context And the input image x tAs input, the video compression device can encode the input image to output a bit stream b x , and if necessary, based on the smoothed bidirectional temporal context and bitstream b x To perform decoding to output the reconstructed image Reconstructing the image Can be used as a reference picture for encoding subsequent input pictures.

[0077] Figure 2B is a flow chart illustrating operation 200 of creating a bidirectional temporal context according to an embodiment of the present disclosure. Figure 3A and Figure 3B is a diagram explaining an example of scaling and rescaling according to time layers. Figure 2B , Figure 3A and Figure 3B An example of the operation 200 of creating a bidirectional temporal context is described below. As described above, scaling and rescaling may be selectively omitted.

[0078] According to an embodiment, in operation 600, the method may include estimating a bidirectional motion vector. For example, in operation 210, by using the input image x t and bidirectional reference images The video encoding device can estimate the input image x t and each reference image The bidirectional motion information between the input image x t and bidirectional reference images is input to the neural network, and the motion vector m f , m b And / or pixel-by-pixel optical flow can be output as motion information

[0079] According to an embodiment, in operation 220, the method may include transforming the bidirectional motion vector into a potential representation. For example, the video compression device may transform the bidirectional motion vector m f , m b Transformed into the potential representation of the bidirectional motion vector g t . Bidirectional motion vector m f , m b can be input into a neural network to be transformed into a latent representation g t For example, the neural network may include, but is not limited to, a convolutional neural network (CNN). Here, the latent representation is the output of a neural network using an input image or motion information as input, and is collectively referred to as a latent feature, a latent vector, etc.

[0080] According to an embodiment, in operation 230, the method may include scaling the potential representation of the bidirectional motion vector. For example, by usingt The bidirectional motion vector m f , m b The potential representation of g t and the temporal layer l of the input image t As input, the video compression device can be based on the temporal layer l t To scale the latent representation, the scaled bidirectional motion vector latent representation can be output

[0081] The same single neural network model can be used for each bidirectional motion vector m f , m b , and scaling may be performed differently for each temporal layer. According to an embodiment, the potential representation may be scaled by using a single neural network, thereby improving memory efficiency compared to a typical method of using various neural network models according to temporal layers.

[0082] For example, the latent representation of the bidirectional motion vector g t Scaling is performed by dividing by a predefined scaling parameter according to the temporal layer. Scaling the potential representation according to the temporal layer has the same or equivalent effect as that produced by adjusting the quantization step size, so that the term "scaling parameter" in the following description will be referred to as "quantization step size determination parameter". The quantization step size determination parameter can be learned in advance to determine the optimal quantization step size for each temporal layer. In the case of an image in a higher temporal layer, as the temporal layer value decreases, the image is used as a reference image in many instances, and therefore, the image can be reconstructed at a higher picture quality level than an image in a lower temporal layer. For example, in the case of an image in a higher temporal layer, the distance between the input image and the reference image is longer than when the image is in a temporal layer lower than the higher temporal layer. To this end, the quantization step size determination parameter can be learned to use a smaller quantization step size for a higher temporal layer and a larger quantization step size for a lower temporal layer.

[0083] Assume there is a latent representation of the input image or a bidirectional motion vector (Here, R: real number, C: number of channels in the latent representation represented as feature map, and H, W: spatial height and width of the latent representation), then for the temporal layer l of the input image t The quantization step size determines the parameter set Q temp and quantization step size to determine the parameters It can be defined as in the following Equation 1. Here, L represents the total number of time layers.

[0084] [Equation 1]

[0085]

[0086] Figure 3A is a diagram showing the bidirectional prediction structure in which the group of pictures (GOP) size is 12 for four temporal layers (l t =0, 1, 2, 3) is a diagram of an example of determining parameters using different quantization step sizes. For ease of explanation, the highest temporal layer contains only I frames, that is, images to which a still image encoding method is to be applied, but is not limited thereto. Figure 3A , I represents an I frame, and B represents a B frame.

[0087] Return to reference Figure 2B According to an embodiment, in operation 240, the method may include quantizing the latent representation of the scaled bidirectional motion vector. For example, by using the latent representation of the scaled bidirectional motion vector As input, quantization is performed on the scaled latent representation to output a quantized and scaled latent representation of the bidirectional motion vector In this case, quantization may be performed using uniform quantization (ie, a rounding operation), but is not limited thereto.

[0088] According to an embodiment, in operation 250, the method may include estimating a probability distribution of a latent representation of the quantized and scaled bidirectional motion vector. For example, by using the latent representation of the quantized and scaled bidirectional motion vector As input, the latent representation can be estimated The probability distribution of Entropy coding is performed. An entropy model can be used, and based on the assumption that the latent representation follows a distribution such as a Laplace distribution or a Gaussian distribution, the entropy model can output the mean of the distribution and standard deviation

[0089] According to an embodiment, in operation 260, the method may include performing entropy encoding or decoding on the latent representation of the quantized and scaled bidirectional motion vector. For example, the latent representation of the quantized and scaled bidirectional motion vector may be entropy encoded or decoded. Perform entropy encoding and decoding. By using the latent representation of quantized and scaled motion vectors and the mean estimated in operation 250 and standard deviation As input, the latent representation Perform entropy coding to output a bit stream b m In addition, by using entropy coded bit stream b m As input, we can use the mean value used in the entropy coding process and standard deviation To perform entropy decoding to reconstruct the latent representation The entropy coding process is usually lossless coding, so that the reconstructed latent representation The input of the entropy encoder Entropy encoding and decoding may be performed using common arithmetic encoding and arithmetic decoding, etc., but are not limited thereto.

[0090] According to an embodiment, in operation 270, the method may include performing a rescaling operation on the latent representation of the quantized and scaled bidirectional motion vector. For example, by using the temporal layer l of the input image t and a potential representation of quantized and scaled bidirectional motion vectors As input, a rescaling is performed to return the latent representation to the original scale, and the latent representation of the rescaled bidirectional motion vector is output

[0091] Figure 3B This is an explanation for the first t Diagram of an example of scaling and rescaling the latent representation by temporal layers.

[0092] Referring to Equation 1 above and Figure 3B , the parameter can be determined by dividing the number C of channel maps by the quantization step size Scaling is performed in operation 230 using C parameters of , which can be represented by the following Equation 2.

[0093] [Equation 2]

[0094]

[0095] Here, C indicates the number of channels in the latent representation denoted as a feature map, and H, W indicate the spatial height and width of the latent representation, respectively.

[0096] The scaled latent representation Quantization (240) and entropy encoding (261) and decoding (262) are performed, and the quantization step size determination parameter that is the same as the quantization step size determination parameter used in scaling can be used. Multiply by the quantized and scaled latent representation To perform rescaling (270). As a result, the quantized and rescaled latent representation is output The rescaling can be expressed by the following equation 3.

[0097] [Equation 3]

[0098]

[0099] By using the same neural network model, different quantization step sizes can be applied to images at different temporal layers, thereby encoding at different image quality levels for each temporal layer. In addition, only a small number of neural network parameters are stored and used in memory, thereby improving memory efficiency.

[0100] Return to reference Figure 2B According to an embodiment, in operation 280, the method may include reconstructing the bidirectional motion vector. For example, by using the latent representation of the rescaled bidirectional motion vector As input, bidirectional motion vectors can be reconstructed and output In this case, a convolutional neural network may be used, but the neural network is not limited thereto.

[0101] According to an embodiment, in operation 290, the method may include creating or generating a bidirectional temporal context based on a latent representation of the bidirectional motion vector. For example, by using the reconstructed motion vector and reference image As input, a bidirectional temporal context C for encoding the input image can be created and output t In this case, motion compensation techniques and convolutional neural networks can be used. The motion compensation technique can use bilinear warping, but is not limited to this.

[0102] Figure 4 is a flow chart illustrating the operation 400 of smoothing bidirectional temporal context according to an embodiment of the present disclosure. Figure 5 is a diagram explaining an example of smoothing bidirectional temporal context according to temporal layers.

[0103] Figure 4 The operation 400 of smoothing the bidirectional temporal context may include: determining for the temporal layer l t The smoothing level s t Operation 410, creating a smoothing filter F t operation 420 , and a smoothing operation 430 .

[0104] The smoothing filter may be a predefined filter (e.g., a Gaussian filter or a bilateral filter based on a Gaussian filter), or may be a trainable filter (e.g., a convolutional neural network), but is not limited thereto. In the example case of using a trainable filter as the smoothing filter, the smoothing filter may be trained for the time layer l. t To train multiple trainable filters differently, and can be based on the time layer l t to select one of the trainable filters for use. In the example case of using a Gaussian filter as the smoothing filter, the smoothing level s t It can be the standard deviation, and if in the example case where the smoothing filter is a trainable filter, the filter can be the index of the filter that is trained differently for the temporal layer.

[0105] The operation 400 of smoothing the bidirectional temporal context will be described below based on an example of using a predefined smoothing filter as a smoothing filter.

[0106] According to an embodiment, in operation 410, the method may include determining a smoothing level for a temporal layer. For example, by using the temporal layer l of the input image t As input, the video encoding device can determine and output a smooth bidirectional temporal context C t The smoothing level.

[0107] For example, a Gaussian filter can be used, and the standard deviation for creating the Gaussian filter can be output as a smoothing level. In this case, by determining a constant value according to the time layer, the constant value can be used as the standard deviation as it is, or the standard deviation can be used by learning different parameters according to the time layer. A higher smoothing level (large standard deviation) can be output for a higher time layer, and a lower smoothing level (small standard deviation) can be output for a lower time layer. Even in the same time layer, different smoothing levels for a bidirectional temporal context can be determined according to the spatial position. In this case, the information output during decoding in operation 260 can be used as spatial position information for smoothing. During decoding in operation 260, smoothing information for the spatial position can also be output.

[0108] According to an embodiment, in operation 420, the method may include generating a smoothing filter. For example, by using the smoothing level s determined in operation 410 t As input to determine the parameters of the predefined filter, the smoothing filter F to be applied for the actual smoothing can be generated t The predefined filter may include, but is not limited to, a Gaussian filter or a bilateral filter using a Gaussian kernel, etc.

[0109] According to an embodiment, in operation 430, the method may include performing a smoothing operation on the bidirectional temporal context. For example, by using the smoothing filter generated in operation 420 as an input to perform smoothing on the bidirectional temporal context, the smoothed bidirectional temporal context may then be output.

[0110] Figure 5 is a diagram showing an example of a bidirectional temporal context smoothed by using different smoothing levels for different temporal layers or different spatial locations. Figure 5 In the figure, arrow ① shows the result of applying a lower smoothing level to a lower temporal layer, i.e., as the temporal layer l t In addition, Figure 5 The arrow ② in FIG. 1 shows the result of using different smoothing levels for spatial locations even in the same time layer, where for the convenience of visualization, C t and Represents the average value calculated for all channels.

[0111] In general, incorrect temporal contexts are likely to be created for higher temporal layers where the distance between the input image and the reference image is longer. According to an embodiment, by smoothing the temporal context differently for the temporal layers, incorrect temporal contexts may be adjusted, thereby achieving higher encoding performance compared to conventional techniques.

[0112] Fig. 6A and Figure 6B is a flow chart illustrating operation 600 of encoding and / or decoding an input image according to an embodiment of the present disclosure.

[0113] refer to Fig. 6A According to an embodiment, in operation 610, the method may include transforming the input image into a latent representation based on the bidirectional temporal context. For example, by using the input image x t and the smoothed bidirectional temporal context As input, the video encoding device can take the input image x t Transformed into latent representation y t , and outputs the latent representation. A convolutional neural network can be used, and instead of using the temporal context created from a unidirectional reference image, a smoothed bidirectional temporal context can be used.

[0114] According to an embodiment, in operation 620, the method may include performing scaling on the latent representation of the input image. For example, by using the temporal layer l of the input image t and the latent representation y of the input image t As input, the video compression device can generate a latent representation y of the input image according to the temporal layer. t Perform scaling and can output the latent representation of the scaled input image As shown in operation 230, the latent representation y of the input image can be t Scaling is performed by dividing by a quantization step determination parameter defined for the temporal layer.

[0115] According to an embodiment, in operation 630, the method may include performing quantization on the latent representation of the scaled input image. For example, the video compression device may quantize the latent representation of the scaled input image. Perform quantization and output the quantized and scaled latent representation of the input image The quantization may be performed using uniform quantization (ie, a typical rounding operation), but is not limited thereto.

[0116] According to an embodiment, in operation 640, the method may include estimating a probability distribution of a latent representation of the quantized and scaled input image. For example, by using the latent representation of the quantized and scaled input image As input, the video compression device can estimate the latent representation The probability distribution of In this case, an entropy model can be used, and based on the assumption that the latent representation follows a distribution such as a Laplace distribution or a Gaussian distribution, the entropy model can output the mean of the distribution and standard deviation

[0117] According to an embodiment, in operation 650, the method may include performing entropy encoding on a latent representation of the quantized and scaled input image. For example, by using the latent representation of the quantized and scaled input image and the mean and standard deviation As input, the video compression device can represent the latent Perform entropy coding to output a bit stream b x The entropy coding may be arithmetic coding.

[0118] refer to Figure 6B , processing may be performed based on the result of the entropy encoding in operation 650.

[0119] According to an embodiment, in operation 660, the method may include performing entropy decoding on the latent representation of the quantized and scaled input image. For example, by using the entropy coded bit stream b used in the entropy coding process x , and the mean and standard deviation As input, the video compression device can perform entropy decoding to reconstruct the latent representation The entropy encoding process in operation 650 is generally lossless encoding, so that the latent representation reconstructed in operation 660 can be used as the latent representation of the input for entropy coding The entropy decoding may be arithmetic decoding, but is not limited thereto.

[0120] According to an embodiment, in operation 670, the method may include performing a rescaling operation on the latent representation of the quantized and scaled input image. For example, by using the temporal layer l of the input image t , and the latent representation of the quantized and scaled input image As input, the video compression device can perform rescaling to return the latent representation of the scaled input image to the original scale to output the latent representation of the rescaled input image The rescaling may be performed in the same manner as operation 270 described above.

[0121] According to an embodiment, in operation 680, the method may include reconstructing the input image based on the bidirectional temporal context. For example, by using the latent representation of the rescaled input image and the smoothed bidirectional temporal context As input, the video compression device can reconstruct and output the input image In this case, a convolutional neural network can be used, and instead of using the temporal context created from a unidirectional reference image, a smoothed bidirectional temporal context can be used. The reconstructed image can be used as a reference image for subsequent images.

[0122] Figure 7 is a block diagram showing an electronic device according to an embodiment of the present disclosure.

[0123] refer to Figure 7 , the electronic device 700 may include a processor 710 and a processor 720 .

[0124] According to an embodiment, the electronic device 700 may include a neural network-based video encoding device or a neural network-based video decoding device, which is configured to perform various examples of the above-mentioned methods for video encoding or video decoding. According to an embodiment, the electronic device 700 may be a device configured to perform both video encoding and video decoding. The electronic device may include, but is not limited to, devices such as edge devices that require applications such as video encoding in an environment with limited computing resources, various image sending / receiving devices (e.g., mobile phones, TVs, monitors, Internet of Things (IoT) devices), and cloud computing devices, etc.

[0125] The memory 710 may store various instructions for image processing (e.g., video encoding, decoding, etc.), and data associated with image processing (neural network, weight, quantization step size determination parameter, reference image, reconstructed image, bit stream generated by encoding, etc.). The memory 710 may include a random access memory (RAM) (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.), a read-only memory (ROM), a flash memory, a cache memory, a virtual memory, etc., but is not limited thereto.

[0126] The processor 720 may execute instructions stored in the memory 710 and perform various video processing operations by referring to various data stored in the memory 710. These instructions may include one or more instructions configured to perform various embodiments of the aforementioned method for video encoding. For example, the instructions may include: creating a bidirectional temporal context of an input image by using an input image, a bidirectional reference image, and a temporal layer of the input image as input, smoothing the bidirectional temporal context by using the bidirectional temporal context and the temporal layer as input, and encoding and decoding the input image by using the smoothed bidirectional temporal context and the input image as input.

[0127] The processor 720 according to one or more embodiments may perform overall control operations of the electronic device 700 .

[0128] The processor 720 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a multi-integrated core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. The processor 720 may execute at least one program or instruction stored in the memory. For example, the processor 720 may execute the method according to one or more embodiments by executing at least one instruction stored in the memory.

[0129] According to one or more embodiments of the present disclosure, one or more operations may be performed by one processor, or by multiple processors. For example, when the first operation, the second operation, and the third operation are performed by the method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor), and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor). According to an embodiment, at least one of the first processor, the second processor, and the third processor may be outside the electronic device 700.

[0130] The processor 720 may be implemented as a single-core processor including one core, or may be implemented as at least one multi-core processor including multiple cores (e.g., homogeneous multi-cores or heterogeneous multi-cores). In the case where the processor 720 is implemented as a multi-core processor, the multiple cores included in the multi-core processor may respectively include a memory inside the processor (e.g., a cache memory and an on-chip memory), and the multi-core processor may include a common cache shared by the multiple cores.

[0131] According to one or more embodiments of the present disclosure, the processor may refer to a system on chip (SoC), a single-core processor, a multi-core processor, or a single-core processor on which at least one processor and other electronic components are integrated, or a core included in a multi-core processor, and the core described herein may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, machine learning accelerator, etc., but the embodiments of the present disclosure are not limited thereto.

[0132] FIG. 8A to FIG. 8I is a diagram for explaining in detail a video encoding and / or decoding process according to an embodiment of the present disclosure. Fig. 9A is a diagram illustrating an example of a context fusion network. Fig. 9B is a diagram showing an example of an entropy model for estimating probability distribution.

[0133] refer to Fig. 8A , the overall video encoding operation performed by the electronic device 700 will be described below. The electronic device 700 may include a video encoding device 800 and / or a video decoding device 800a. As shown herein, the video encoding device 800 and / or the video decoding device 800a may share components to use bidirectional prediction and temporal layer information. The video decoding device 800a may be configured as a hardware chip and / or software separate from the video encoding device 800.

[0134] The video encoding apparatus 800 may convert the input image x into t And bidirectional reference images and A single motion vector estimation network 811 is input and the forward reference image can be estimated And the input image x t The motion vector m between f , and the backward reference image And the input image x t The motion vector m between b In this case, various optical flow estimation neural networks can be used as motion vector estimation networks.

[0135] Then, in order to obtain the two motion vectors m output from the motion vector estimation network 811 f and m b Transformed into a potential representation of a bidirectional motion vector g t , the two motion vectors are cascaded, and the cascaded result can be input into the motion vector encoder (MV Enc) 812. MV Enc can be based on a convolutional neural network and use a bidirectional motion vector as input, so that the number of channels of the convolution layer can be doubled.

[0136] Then, according to the temporal layer l of the input image t From the quantization step parameter set Select the quantization step size parameter in And by converting the potential representation g of the bidirectional motion vector output from MV Enc 812 in 813 t Dividing by the chosen quantization step size parameter, the latent representation of the bidirectional motion vector can be scaled according to the temporal layer.

[0137] Next, at 814, the latent representation of the scaled bidirectional motion vector Quantization is performed, and an entropy model 815 can be used to estimate the potential representation of the quantized and scaled bidirectional motion vector The entropy model may be an entropy model using a hyper-prior and an autoregressive context, etc., but is not limited thereto.

[0138] By using the probability distribution estimated by the entropy model 815 (e.g., mean and standard deviation ), a potential representation of a quantized and scaled bidirectional motion vector It is encoded in AE 8161 and decoded in AD 8162. As a result of the encoding, a bit stream b for the potential representation of the bidirectional motion vector is output. m , and use bitstream b m and probability distribution to perform decoding.

[0139] Then, by using the temporal layer l of the input image t From the quantization step parameter set Select the quantization step size parameter in And rescaling according to the temporal layer may be performed by multiplying the selected quantization step size parameter by the latent representation of the quantized and scaled bidirectional motion vector in 817 .

[0140] Subsequently, at 818, the rescaled latent representation of the bidirectional motion vector Input to the motion vector decoder (MV Dec), and can reconstruct the forward motion vector and the backward motion vector In this case, the MV decoder is based on a convolutional neural network, and the number of channels of the convolution layer is doubled in order to reconstruct all bidirectional motion vectors. In addition, in order to perform smoothing differently according to the spatial position, the MV decoder can increase the number of channels and output smoothing information associated with the spatial position.

[0141] Next, by using the reconstructed bidirectional motion vector and bidirectional reference images As input, the bidirectional temporal context C is output in 819 t .

[0142] For example, each bidirectional reference image is input to the feature extractor 8191 to output a forward reference feature map and the backward reference feature map In this case, the feature extractor 8191 may be a convolutional neural network. Subsequently, based on the forward motion vector Forward reference feature map Perform a typical bilinear warp 8192 to output a warped forward reference feature map Similarly, the backward motion vector and the backward reference feature map Perform warping to output a warped backward reference feature map The warped forward reference feature map and the warped backward reference feature map may be subjected to context refinement 8193 for post-processing and may be output as the forward temporal context C f and the backward temporal context C b In this case, the neural network used for post-processing can be a convolutional neural network. Finally, the forward temporal context C f and the backward temporal context C b Input context fusion network 8194 to output bidirectional temporal context C t .

[0143] refer to Fig. 9A , the context fusion network 8194 may include a cascade 911, convolutional layers 912 and 914, and a nonlinear layer such as ReLU (e.g., LeakyReLU) 913. In the convolutional layer Conv(k, c in , c out , s) 912 and 914, k represents the horizontal and vertical dimensions of the kernel, c in represents the number of input channels, c out represents the number of output channels, and s represents the stride of the convolution. Fig. 9A In , N represents the number of channels in the forward temporal context and the bidirectional temporal context.

[0144] Return to reference Fig. 8A , can be obtained by using the temporal layer l of the input image t As input, smoothing is performed. By using the temporal layer l of the input image t As input, a smoothing level s for generating a smoothing filter may be output in 821 t .

[0145] As a smoothing method, Gaussian smoothing or a bilateral filter based on a Gaussian kernel can be used, in which case the standard deviation used to generate the smoothing filter can be output. In this case, in order to use a larger level of smoothing for higher temporal layers, the temporal layer l can be t Use a monotonically decreasing smoothing level, such as s 1 =10,s 2 =5 and s 3 = 2.5. The spatial position difference can be taken into account by Gaussian smoothing, and a bilateral filter can be used to take into account not only the spatial position difference but also the difference in the value input to the smoothed object.

[0146] Then, a smoothing filter for use in a smoothed bidirectional temporal context may be generated in 822. Assuming that Gaussian smoothing is performed, the smoothing filter may be generated by using a smoothing level s according to the temporal layer. t (i.e., standard deviation) as input to generate a smoothing filter. By defining the smoothing level as s t , the Gaussian kernel size is defined as K×K, and the value of K can be calculated according to, for example, Equation 4.

[0147] [Equation 4]

[0148]

[0149] here, Represents a typical rounding operation.

[0150] In the example case of a given value of K, the value based on a given smoothing level s can be calculated as in Equation 5 t The Gaussian kernel F t In this equation, (k x , k y ) represents the spatial position of the nucleus.

[0151] [Equation 5]

[0152]

[0153]

[0154] Then, by using the bidirectional temporal context C t and a smoothing filter (e.g., a Gaussian kernel) as input, and outputting a smoothed bidirectional temporal context in 823 In this case, we can use the bidirectional temporal context C t At each spatial location of , the kernel is multiplied by the stride one.

[0155] Then, by using the smoothed bidirectional temporal context And the input image x tAs input, the latent representation y of the input image t It can be output through inter-channel cascading and context encoder (Enc) 831.

[0156] Next, by using the temporal layer l of the input image t As input, the quantization step size parameter set Select the quantization step size parameter in By transforming the latent representation y of the input image into t Dividing by the chosen quantization step size parameter, the latent representation of the bidirectional motion vector can be scaled according to the temporal layer.

[0157] The latent representation of the scaled bidirectional motion vector is then quantized in 833, and the latent representation of the quantized and scaled input image can be estimated by using an entropy model 834 The probability distribution of and standard deviation ).

[0158] refer to Fig. 9B , the entropy model 834 may include a common hyper-prior model 921, an autoregressive context model 923, a temporal prior encoder 924, and a prior fusion network 925. The potential representation of the quantized and scaled input image is input into the super prior model 921 and the autoregressive context model 923 to output the super prior information and the autoregressive context respectively. In addition, the temporal prior encoder 924 can be used to smooth the bidirectional temporal context The prior fusion network 925 can output a latent representation of the quantized and scaled input image by using the temporal prior information, the hyper-prior information, and the autoregressive context as input. The mean and standard deviation

[0159] Return to reference Fig. 8A , by using the mean estimated by the entropy model 834 and standard deviation Latent representation of the quantized and scaled input image in 8351 Decode and output the bit stream b of the input image x As a result.

[0160] Then, by using the output bit stream b x , and the mean and standard deviation to be decoded in 8352, and then in 836 by converting the temporal layer l according to the input image t The quantization step size parameter Multiply the latent representation of the quantized and scaled input image to rescale it according to the temporal layer, and then add the smoothed bidirectional temporal context And the input image x t Input to the context decoder 837 to output a reconstructed image of the input image Reconstructing the image Stored in memory for use as a reference image for encoding of subsequent input images.

[0161] As described above, the bidirectional temporal context C can be t Smoothing is performed according to the temporal layer, but may be similarly performed at various other smoothing locations.

[0162] exist Fig. 8A , ①, ②, ③, ④, ⑤, ⑥, ⑦, ⑧, ⑨ indicate various smoothing positions. In the example case where smoothing is performed, the smoothing object at each position is as shown in Table 1 below. In the example case where there are two smoothing objects, as indicated in the "Remarks" column in the table, smoothing can be performed on each smoothing object, or smoothing can be performed by cascading the two objects into one.

[0163] FIG. 8B to FIG. 8I is a diagram showing an example of performing smoothing at smoothing positions ② to ⑨. As shown in this article, by t The smoothing object is input into the smoothing module SM to perform smoothing. Fig. 8A In the example case of ⑧, the smoothing object is a bidirectional motion vector m as the output of the motion vector estimation network 811 f , m b , and by using a smoothing module SM for each bidirectional motion vector m f , m b Perform smoothing to obtain smoothed bidirectional motion vectors

[0164] [Table 1]

[0165]

[0166] Return to reference Fig. 8A The video decoding device 800a, which is a neural network-based device, can reconstruct the motion vector and generate the bidirectional motion vector by using the bit stream b of the potential representation generated by the video decoding device 800a. m and bidirectional reference pictures to create a bidirectional temporal context and can be achieved by using the bitstream b for the input picture xand bidirectional temporal context to reconstruct the input image. In this case, rescaling and / or smoothing can be performed on the latent representation by using the temporal layer. The video encoding device is described in detail above, so its redundant description will be omitted.

[0167] Fig.10 is a block diagram illustrating an electronic device according to another embodiment of the present disclosure.

[0168] refer to Fig.10 , the electronic device 1000 may include a memory 100, a processor 1020, an image capturing device 1030, a communication device 1040, and an output device 1050. The memory 100 and the processor 1020 are described above in detail, and thus a description thereof will be omitted.

[0169] The image capture device 1030 may include a device (e.g., a camera, etc.) for capturing still images or moving images, etc., and may store the captured image in the memory 1030 and send the image to the processor 1020. The image capture device 1030 may include a lens assembly having one or more lenses, an image sensor, a charge coupled device (CCD sensor), an image signal processor, and / or a flash. The lens assembly included in the camera module may collect light emitted from an object to be imaged.

[0170] The communication device 1040 can support the establishment of a direct communication channel and / or a wireless communication channel between the electronic device 1000 and other electronic devices, servers or sensor devices within the network environment by using various communication technologies, and perform communication via the established communication channel. For example, the direct communication channel can be a wired communication between the electronic device 1000 and other electronic devices. The communication device 1040 can send the image captured by the image capture device 1030 and / or the data processed by the processor 1020 (e.g., encoded video, etc.) to other electronic devices. In addition, the communication device 1040 can receive a video to be processed from a cloud device or other electronic device, store the received video in the memory 100, and send the video to the processor 1020 so that the processor 1020 can process the video.

[0171] The output device 1050 may visually / non-visually output the image captured by the image capture device 1030 and / or the data processed by the processor 1020. The output device 1050 may include a sound output device, a display device (e.g., a display), an audio module, and / or a haptic module. The output device 1050 may display the video processed by the processor 1020, etc., on the display, thereby improving the user's video experience.

[0172] In addition, the electronic device 1000 may also include sensor devices configured to detect various data (for example, an accelerometer, a gyroscope, a magnetic field sensor, a proximity sensor, an illumination sensor, a fingerprint sensor, etc.), input devices configured to receive instructions and / or data to be used from a user (for example, a microphone, a mouse, a keyboard and / or a digital pen (for example, a stylus, etc.)), etc.

[0173] The above one or more embodiments of the present disclosure may be implemented as a computer-readable code written on a computer-readable recording medium. A computer-readable recording medium may be any type of recording device that stores data in a computer-readable manner. For example, the method for encoding and / or decoding an image shown in accordance with the above various embodiments may be implemented as a computer-readable code that may be executed on one or more processors. Although the accompanying drawings specifically illustrate the operation of the method according to one or more embodiments of the present disclosure, the present disclosure is not limited thereto, and therefore, according to various other embodiments, the order of operations may vary, one or more operations may be added, or one or more operations may be omitted.

[0174] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and carrier wave (e.g., data transmission via the Internet). Computer-readable recording media can be distributed on multiple computer systems connected to a network so that computer-readable codes are written therein and executed therefrom in a decentralized manner. Those skilled in the art to which the present invention belongs can easily infer the functional programs, codes, and code segments required to implement the present invention.

[0175] The present disclosure has been described above with reference to preferred embodiments. However, it is obvious to those skilled in the art that various changes and modifications can be made without changing the technical concept and features of the present disclosure. Therefore, it is clear that the above embodiments are illustrative in all aspects and are not intended to limit the present disclosure.

Claims

1. A method for neural network-based video encoding, the method comprising: estimating a motion vector between the input image and a reference image based on a temporal layer of the input image; transforming the motion vector into a latent representation; scaling the latent representation of the motion vector based on the temporal layers of the input image; as well as A temporal context of the input image is obtained based on a latent representation of the scaled motion vector and the reference image.

2. The method according to claim 1, wherein: The reference images include a first bidirectional reference image temporally preceding the input image, and a second bidirectional reference image temporally following the input image.

3. The method according to claim 1, wherein: The temporal layer-based scaling includes scaling the latent representation of the motion vector by using a smaller quantization step size for a higher temporal layer.

4. The method according to claim 1, wherein: The temporal layer-based scaling includes scaling the latent representation of the motion vector by dividing the latent representation of the motion vector by a quantization step size determination parameter defined for the temporal layer.

5. The method according to claim 1, further comprising: performing entropy encoding and entropy decoding on the latent representation of the motion vector; rescaling the scaled latent representation of the motion vector based on the temporal layer; as well as Reconstruct the motion vector based on the latent representation of the rescaled motion vector, The obtaining of the temporal context includes: obtaining the temporal context based on the reconstructed motion vector and the reference image.

6. The method according to claim 5, wherein: The rescaling based on the temporal layer comprises multiplying the latent representation of the scaled motion vector by a quantization step size determination parameter defined for the temporal layer.

7. The method according to claim 5, further comprising: quantizing a latent representation of the scaled motion vector prior to performing entropy encoding and entropy decoding; as well as A probability distribution of the latent representation of the motion vector is estimated.

8. The method according to claim 5, wherein: The obtaining of the time context includes: Outputting a reference feature map by inputting the reference image into a feature extraction neural network, performing bilinear warping on the reference feature map based on the reconstructed motion vector to output a warped reference feature map, inputting the distorted reference feature map into a post-processing neural network, and The output of the post-processing neural network is input into a context fusion network to output the temporal context.

9. A method for neural network-based video encoding, the method comprising: estimating a motion vector between the input image and a reference image based on a temporal layer of the input image; transforming the motion vector into a latent representation; obtaining a temporal context of the input image based on the latent representation of the motion vector and the reference image; as well as A smoothing operation is performed on the temporal context based on a smoothing object, wherein the smoothing object includes at least one of: the reference image, the input image, the motion vector, an input in obtaining the temporal context, an output in obtaining the temporal context, or an input or output of a sub-process in obtaining the temporal context.

10. The method according to claim 9, wherein: The reference images include a first bidirectional reference image temporally preceding the input image, and a second bidirectional reference image temporally following the input image.

11. The method according to claim 9, wherein: The execution of the smoothing operation includes: For the temporal layer of the input image, obtaining a smoothness level of the smoothed object; generating a smoothing filter based on the smoothing level; and The smoothing operation is performed on the smoothing object based on the smoothing filter.

12. The method according to claim 11, wherein: The obtaining of the smoothness level of the smooth object includes obtaining a lower smoothness level for a lower temporal layer.

13. The method according to claim 11, wherein: The obtaining of the smoothness level of the smooth object includes: obtaining different smoothness levels according to spatial positions.

14. A method for neural network-based video decoding, the method comprising: reconstructing a latent representation of the motion vectors based on a first bitstream for the motion vectors; rescaling the latent representation of the reconstructed motion vector based on the temporal layer; reconstructing the motion vector based on a latent representation of the rescaled motion vector; Obtaining temporal context based on the reconstructed motion vector and the reference image; as well as An input image is reconstructed based on a second bitstream for the input image and the temporal context.

15. The method according to claim 14, wherein: The reference images include a first bidirectional reference image temporally preceding the input image, and a second bidirectional reference image temporally following the input image.

16. The method according to claim 14, wherein: The rescaling based on the temporal layer comprises multiplying the latent representation of the motion vector by a quantization step size determination parameter defined for the temporal layer.

17. The method according to claim 14, wherein: The reconstruction of the input image comprises: reconstructing a latent representation of the input image based on the second bitstream for the input image; rescaling the latent representation of the input image based on the temporal layer; and The input image is reconstructed based on a latent representation of the rescaled input image and the temporal context.

18. A method for neural network-based video decoding, the method comprising: reconstructing a latent representation of the motion vectors based on a first bitstream for the motion vectors; obtaining a temporal context by using the motion vector and a reference image; performing smoothing by using the reference image, the motion vector, an input in obtaining the temporal context, an output in obtaining the temporal context, and at least one of an input and an output of a subprocess in obtaining the temporal context as a smoothing object; as well as An input image is reconstructed based on a second bitstream for the input image and the temporal context.

19. The method according to claim 18, wherein: The smooth execution includes: obtaining different smoothing levels based on at least one of a temporal layer and a spatial position of the input image; generating a smoothing filter based on at least one of the smoothing levels; and The smoothing is performed on the smoothed object based on the smoothing filter.

20. An electronic device, comprising: a memory configured to store one or more instructions and a reference image; as well as a processor configured to execute the one or more instructions to: estimate a motion vector between the input image and the reference image based on the reference image, the input image, and a temporal layer of the input image; transforming the motion vector into a latent representation; scaling the latent representation of the motion vector based on the temporal layers of the input image; as well as A temporal context of the input image is obtained based on a latent representation of the scaled motion vector and the reference image.

Images