Neural network-based video compression method using motion vector field compression

Abstract

A neural network-based image processing method and apparatus according to an embodiment of the present invention may generate a motion vector field by using motion information used in motion prediction of a processing unit included in a current picture, and generating a tensor of the motion vector field by performing compression on the motion vector field based on a neural network including a plurality of neural network layers.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for compressing a motion vector field, and more particularly, to a method and apparatus for compressing / reconstructing a motion vector field in a technique for compressing motion vectors for temporal motion vector prediction for video coding. Background Art

[0002] The video image is compressed and encoded by removing spatial-temporal redundancy and inter-view redundancy, and can be transmitted through a communication line or stored on a storage medium in an appropriate form. Summary of the invention

[0003] [Technical issues]

[0004] The present invention proposes a method and apparatus for compressing / reconstructing a motion vector field for temporal motion vector prediction using a neural network.

[0005] [Technical solution]

[0006] In order to solve the above problems, a method and apparatus for compressing / reconstructing a motion vector field using a neural network are provided.

[0007] According to an embodiment of the present invention, a neural network-based image processing method and device can generate a motion vector field by using motion information for motion prediction of a processing unit included in a current picture, and generate a tensor of the motion vector field by performing compression on the motion vector field based on a neural network including multiple neural network layers.

[0008] In a neural network-based image processing method and apparatus according to an embodiment of the present invention, the motion information may include at least one of a prediction direction flag, a reference index, or a motion vector.

[0009] In a neural network-based image processing method and apparatus according to an embodiment of the present invention, the plurality of neural network layers may include at least one convolutional layer.

[0010] According to an embodiment of the present invention, a neural network-based image processing method and apparatus can perform spatial sampling on a motion vector field based on at least one convolutional layer.

[0011] According to an embodiment of the present invention, a neural network-based image processing method and apparatus can perform normalization on a motion vector field based on a picture order count (POC) difference between a reference picture specified by a reference index of a processing unit and a current picture.

[0012] In the neural network-based image processing method and apparatus according to one embodiment of the present invention, a tensor may be generated by compressing a motion vector field on which normalization has been performed.

[0013] According to an embodiment of the present invention, a neural network-based image processing method and apparatus can obtain a motion vector with a unit POC difference by scaling a motion vector used for motion prediction of a processing unit by the POC difference, and can use the motion vector with a unit POC difference to modify a motion vector field.

[0014] According to an embodiment of the present invention, a neural network-based image processing method and apparatus can generate a quantized tensor by performing quantization on a tensor, and store the quantized tensor in a memory.

[0015] In a neural network-based image processing method and apparatus according to an embodiment of the present invention, the stored quantized tensor can be used for motion prediction of a processing unit in a subsequent picture of the current picture.

[0016] In a neural network-based image processing method and apparatus according to an embodiment of the present invention, a neural network may be learned by a loss function defined based on the sum of distortion and bit rate.

[0017] In the neural network-based image processing method and apparatus according to one embodiment of the present invention, the distortion may represent the difference between the original motion vector field and the reconstructed motion vector field.

[0018] In the neural network-based image processing method and apparatus according to one embodiment of the present invention, the difference may be calculated using MSE (mean square error) or SAD (sum of absolute differences).

[0019] In a neural network-based image processing method and apparatus according to an embodiment of the present invention, a potential tensor may be used to predict a bit rate.

[0020] In the neural network-based image processing method and apparatus according to one embodiment of the present invention, the bit rate may be predicted using a probability value obtained based on the neural network.

[0021] In a neural network-based image processing method and apparatus according to an embodiment of the present invention, a loss function may be defined by additionally considering the distortion between a motion vector field estimated by a teacher network and a motion vector field reconstructed by a student network.

[0022] In a neural network-based image processing method and apparatus according to an embodiment of the present invention, the teacher network may be a flow network that predicts the optical flow between a previous image and a subsequent image based on a current image.

[0023] [Technical Effect]

[0024] The motion vector field compression method and device according to the present invention can improve the video signal encoding efficiency.

[0025] In addition, by using the motion vector field compression method using a neural network proposed in the present invention, the encoding efficiency can be improved.

[0026] In addition, since the motion vector field compression method using a neural network proposed in the present invention can perform spatial sampling together, the motion can be expressed more accurately compared to the existing technology that performs sampling at a specific position, and therefore the more accurately expressed motion information can be used for subsequent temporal motion vector prediction, thereby improving video encoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 An example of an encoder / decoder using motion vector field compression according to an embodiment of the present disclosure is shown.

[0028] Figure 2 is a diagram illustrating a method for sampling a motion vector field according to one embodiment of the present disclosure.

[0029] Figure 3 An example of a coding unit encoder / decoder for neural network-based motion vector field compression using the present disclosure is shown.

[0030] Figure 4 An example of a conceptual graph for compressing and reconstructing neural network learning according to one embodiment of the present disclosure is shown.

[0031] Figure 5 A conceptual diagram showing compression and reconstruction neural network learning using knowledge distillation according to one embodiment of the present disclosure.

[0032] Figure 6 is a flow chart illustrating a neural network-based motion vector field compression method according to one embodiment of the present disclosure.

[0033] Figure 7 is a flowchart illustrating a neural network-based motion vector field reconstruction and motion prediction method according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The embodiments of the present disclosure will be described in detail so that those skilled in the art can easily implement the embodiments by referring to the drawings attached to this specification. However, the present disclosure can be implemented in different forms, and the present disclosure is not limited to the embodiments described herein. Moreover, in order to clearly describe the present disclosure in the figures, parts that are not related to the description are omitted, and throughout this specification, similar reference numerals are attached to similar parts.

[0035] Throughout this specification, when a part is referred to as being “connected” to another part, this may include an electrical connection as well as a direct connection with other elements present therebetween.

[0036] In addition, throughout this specification, when a part is referred to as “comprising” a component, this means that other components may also be included without excluding other components, unless otherwise stated.

[0037] In addition, terms such as first, second, etc. may be used to describe various components, but the components should not be limited by these terms. The terms are only used to distinguish one component from other components.

[0038] In addition, for the embodiments of the device and method described in this specification, some configurations of the device or some steps of the method may be omitted. In addition, the order of some configurations of the device or some steps of the method may be changed. In addition, other configurations or other steps may be inserted into some configurations of the device or some steps of the method.

[0039] In addition, some configurations or some steps of the first embodiment of the present disclosure may be added to the second embodiment of the present disclosure, or may replace some configurations or some steps of the second embodiment.

[0040] In addition, the construction units shown in the embodiments of the present disclosure are shown separately to represent different characteristic functions, and they do not mean that each construction unit is configured with separate hardware or a software construction unit. In other words, for ease of description, each construction unit is described by being listed as each construction unit, and at least two construction units in each construction unit can be combined to form a construction unit, or a construction unit can be divided into multiple construction units to perform functions. The integrated implementation and separation implementation of each construction unit are also included in the scope of the rights of the present disclosure, unless they deviate from the spirit of the present disclosure.

[0041] First, the terms used in this application can be briefly described as follows.

[0042] The decoding device (video decoding equipment) to be described later may be a device included in a server terminal (for example, a civilian security camera device, a civilian security system, a military security camera device, a military security system, a personal computer (PC), a laptop computer, a portable multimedia player (PMP), a wireless communication terminal, a smart phone, a TV application server, and a service server, etc.), and it may refer to various devices equipped with a user terminal, including various equipment, a communication device including a communication modem for communicating with a wired / wireless communication network, a memory for storing various programs and data for decoding images or performing intra-frame prediction or inter-frame prediction for decoding, a microprocessor for executing programs and performing operations and controls, and others.

[0043] In addition, the image encoded as a bit stream by the encoder can be transmitted to the image decoding device in real time or non-real time through various communication interfaces (e.g., cables, universal serial bus (USB), etc.) or through wired or wireless communication networks (e.g., the Internet, wireless local area network, wireless LAN network, Wi-Bro network, mobile communication network, etc.), decoded and reconstructed and reproduced as an image. Alternatively, the bit stream generated by the encoder can be stored in a memory. The memory can include both volatile memory and non-volatile memory. In this specification, the memory can be represented as a recording medium storing a bit stream.

[0044] Generally, a video may be configured with a series of pictures, and each picture may be divided into coding units such as blocks. In addition, a person skilled in the art to which the present embodiment belongs may understand that the term picture input below may be used by replacing it with other terms having the same meaning as image, frame, etc. Also, a person skilled in the art to which the present embodiment belongs may understand that the term coding unit may be used by replacing it with other terms having the same meaning as unit block, block, etc.

[0045] Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure are described in more detail. In describing the present disclosure, repeated descriptions of the same components are omitted.

[0046] Figure 1 An example of an encoder / decoder using motion vector field compression according to an embodiment of the present disclosure is shown.

[0047] According to an embodiment of the present disclosure, a motion vector field may be compressed / reconstructed and used for encoding / decoding of an image, specifically, for motion estimation / compensation (or motion prediction). In the present disclosure, a motion vector field may include motion information of a previously decoded image (or a lower processing unit), and a motion vector field may also be referred to as a motion information field, a motion information list, a motion vector list, a motion information table, a motion vector table, a motion information storage, a motion vector storage, a motion vector set, a motion vector group, a motion information set, a motion information group, etc.

[0048] Reference Figure 1 , an encoder / decoder using motion vector field compression may include a motion vector field reconstruction unit (100), a motion vector field scaling unit (110), a coding unit encoder / decoder (120), a motion vector field sampling unit (130), a motion vector field compression unit (140) and a storage unit (150). The present disclosure is an example, and when an encoder / decoder is implemented using motion vector field compression, other components may be added except Figure 1 Components other than those shown in the figure and may be omitted Figure 1 Some of the components shown in .

[0049] Here, a coding unit may represent a coding / decoding unit. In the present disclosure, a coding unit may be referred to as a processing unit or a processing unit. As an example, a coding unit may be one of a frame (or picture), a tile, a slice, a coding tree unit, and a coding unit (or block) of a video.

[0050] The motion vector field reconstruction unit (100) can reconstruct the motion vector field according to the data stored in the storage unit (150). The data stored in the storage unit (150) may include one or more motion information obtained from one of the previously reconstructed pictures. The motion vector field may include the motion information. In this case, the motion information may include two prediction flags (or prediction direction flags), a reference picture index (or reference index) and a compressed motion vector.

[0051] Here, one of the two prediction flags may be a flag indicating inter-frame prediction using a picture included in reference picture list (RPL) 0, and the other may be a flag indicating inter-frame prediction using a picture included in RPL1. That is, when both flags are 1, it may refer to bidirectional inter-frame prediction using pictures respectively included in RPL0 and RPL1. In addition, here, the reference picture index may refer to an index of a picture used for inter-frame prediction among pictures included in the RPL.

[0052] As an example, a compressed motion vector may be compressed and represented with a bit depth that is less than the bit depth represented by the original motion vector. In the present disclosure, bit depth may also be referred to as resolution or precision. For example, if an element of the original motion vector has a bit depth of -2 17 To +2 17 -1, the value of the motion vector can be represented by fixed-point 18 bits. In this case, the motion vector can be compressed by fixed-point 10 bits and stored in the memory (or storage unit (150)) for use in temporal motion prediction during encoding / decoding of the next frame. In this case, 4 of the 10 bits can represent the exponent and 6 bits can represent the signed mantissa.

[0053] The motion vector field reconstruction unit (100) may reconstruct the compressed motion vector to generate a reconstructed motion vector. For example, the motion vector field reconstruction unit (100) may reconstruct the fixed-point 10-bit represented as the exponent and the mantissa into a fixed-point 18-bit as in the following equation 1.

[0054] [Formula 1]

[0055] mvLXCol=mantissa<<(exponent-1)

[0056] In Formula 1, << represents a left shift operation, mantissa is a variable representing a mantissa, and exponent is a variable representing an exponent. The motion vector field reconstruction unit (100) may transmit motion information including the reconstructed motion vector to the motion vector field scaling unit (110).

[0057] The motion vector field scaling unit (110) can scale the motion vector in the input motion information. According to an embodiment of the present disclosure, the motion vector used in the previous picture may have a different reference picture and thus may have a different degree of motion scaling. Therefore, the motion vector field scaling unit (110) can perform scaling according to the reference picture so that the scaling of the motion vector of the motion vector field is the same.

[0058] In one embodiment, each of the motion vectors reconstructed by the motion vector field reconstruction unit (100) may be scaled based on the following Equation 2.

[0059] [Formula 2]

[0060] tx=(16384+(Abs(td)>>1)) / td

[0061] td=Clip3(-128, 127, colPocDiff)

[0062] tb=Clip3(-128, 127, currPocDiff)

[0063] distScaleFactor=Clip3(-4096, 4095, (tb*tx+32)>>6)

[0064] mvLXCol=Clip3(-131072, 131071, (distScaleFactor*mvCol+128-(distScaleFactor*mCol>=0)>>8)

[0065] Referring to Formula 2, the motion vector can be scaled by the scaling factor variable distScaleFactor calculated by colPocDiff and currlPocDiff. Here, mvCol is a variable representing the motion vector before scaling, and mvLXCol is a variable representing the motion vector after scaling. colPocDiff is a variable representing the difference between the POC (Picture Order Count) of the reference picture (RefColPic) of the collocated picture (ColPic) and the POC of ColPic. In addition, currlPocDiff is a variable representing the difference between the POC of currPic and the POC of currRefPic when the current picture and the reference picture of the current picture are currPic and currRefPic, respectively.

[0066] The motion information including the scaled motion vectors may be transmitted to the coding unit encoder / decoder (120).

[0067] The coding unit encoder / decoder (120) may perform encoding / decoding of a current coding unit using the input motion information.

[0068] For example, the input motion information can be used for inter prediction of the current coding unit. For example, in inter prediction, it can be used as a temporal motion vector prediction candidate in merge mode. Alternatively, for example, it can be used to obtain SbTMVP (sub-block based temporal merge candidate) in sub-block merge candidates of inter prediction. Alternatively, for example, it can be used to obtain a constructed affine control point motion vector in affine mode of inter prediction.

[0069] The motion information used for motion prediction in the encoding encoder / decoder (120) may be transmitted to the motion vector field sampling unit (130).

[0070] The motion vector field sampling unit (130) may generate a spatially sampled motion vector field based on the input motion information and transmit it to the motion vector field compression unit (140). For example, the motion vector of the motion vector field used in the encoding unit encoder / decoder (120) may exist in units of 4×4 pixels. That is, the motion vector field sampling unit (130) may perform sampling in units of 4×4 pixels.

[0071] Alternatively, as an embodiment, the motion vector may be sampled in units greater than 4×4 to efficiently utilize the memory. As an example, the unit for sampling the motion vector field may be predefined. For example, the motion vector field sampling unit (130) may perform sampling in units of 8×8 pixels. Alternatively, the motion vector field sampling unit (130) may perform sampling in units of 16×16 pixels. Alternatively, for example, the unit for sampling the motion vector field may be variably determined based on encoding information. In this case, the encoding information may include at least one of the size of the block, the width / height of the block, the width / height ratio of the block, the inter-frame prediction mode, or whether it is located at the boundary of the image (or slice, tile, coding tree unit).

[0072] In addition, the position of the sample where sampling is performed may be defined as one of the upper left, upper right, lower left, lower right, and center positions of a predefined (or predetermined) unit. In this case, if sampling is performed in units of 8×8 pixels, the upper left corner of the 8×8 pixels may be the sampling position. Alternatively, the center of the 8×8 pixels may be the sampling position.

[0073] Figure 2 is a diagram illustrating a method for sampling a motion vector field according to one embodiment of the present disclosure.

[0074] Figure 2 Shown in W mi ×H mi An example of sampling at the upper left position in pixels. Figure 2 As shown, the motion vector at a specific position within a specific size unit can be sampled.

[0075] Refer again Figure 1 The motion information including the sampled motion vector field may be transmitted to the motion vector field compression unit (140). The motion vector field compression unit (140) may compress the motion vector using the motion information including the transmitted sampled motion vector field.

[0076] As mentioned above, the motion vector can be compressed into a fixed-point 10-bit number and stored in memory for use in temporal motion prediction when encoding / decoding the next frame. In this case, 4 of the 10 bits can represent the exponent and 6 bits can represent the signed mantissa.

[0077] The motion information including the compressed motion vector field may be transmitted to a storage unit (150).

[0078] The storage unit (150) can store and manage the received motion information including the compressed motion vector field in a memory.

[0079] Figure 3 An example of a coding unit encoder / decoder for neural network-based motion vector field compression using the present disclosure is shown.

[0080] The present disclosure is an example, and when implementing an encoder / decoder using neural network based motion vector field compression, other than Figure 3 Components other than those shown in the figure and may be omitted Figure 3 Some of the components shown in .

[0081] According to an embodiment of the present disclosure, a neural network can be used for compression / reconstruction of a motion vector field. Figure 3 , an encoder / decoder using a neural network-based motion vector field compression may include a reconstruction neural network (300), a motion vector field scaling unit (310), a coding unit encoder / decoder (320), a motion vector field normalization unit (330), a compression neural network (340), a quantization unit (350) and a storage unit (360). As an example, the above Figure 1 The embodiments of the present invention may also be similarly applied to the present embodiment, and any repeated description thereof will be omitted.

[0082] As an embodiment, the neural network used for the reconstruction neural network (300) and the compression neural network (340) may include one or more neural network layers. The neural network layer may include at least one of a convolution layer, a deconvolution layer, a transposed convolution layer, a dilated convolution layer, a grouped convolution layer, a graph convolution layer, an average pooling layer, a maximum pooling layer, an upsampling layer, a downsampling layer, a pixel shuffle layer, a channel shuffle layer, a batch normalization layer, a weight normalization layer, or a generalized normalization layer.

[0083] The input / output data of each neural network layer may be transmitted in the form of a tensor as three-dimensional data. As an example, the input / output data may be a feature tensor, a symbol tensor, an input tensor, an output tensor, or a feature map. In addition, the reconstruction neural network (300) and the compression neural network (340) may be neural networks that have been learned through a learning process.

[0084] The reconstruction neural network may receive the compressed motion vector field stored in the storage unit. In this case, the compressed motion vector field may be in the form of a tensor. The compressed motion vector field may be reconstructed into a motion vector field by the reconstruction neural network. The reconstructed motion vector field may be transmitted to the motion vector field scaling unit.

[0085] The motion vector field scaling unit (310) can scale the input motion vector field.

[0086] As an example, the original motion vector field compressed by the compression neural network (340) may be a motion vector field having a unit POC difference of 1. Therefore, scaling by the amount of the POC difference between the current POC and the POC of the reference picture may be required.

[0087] For example, each motion vector may be scaled by the distScaleFactor calculated by currlPocDiff, as shown in Equation 3 below.

[0088] [Formula 3]

[0089] distScaleFactor=Clip3(-128, 127, currPocDiff)

[0090] mvLXCol=Clip3(-131072, 131071, (distScaleFactor*mvCol-(distScaleFactor*mvCol>=0)))

[0091] Here, mvCol is a variable representing a motion vector before scaling, and mvLXCol is a variable representing a motion vector after scaling. currlPocDiff may represent a difference between the POC of currPic and the POC of currRefPic when the current picture and the reference picture of the current picture are currPic and currRefPic, respectively.

[0092] The motion information including the scaled motion vector may be transmitted to the coding unit encoder / decoder (320).

[0093] The coding unit encoder / decoder ( 320 ) may perform encoding / decoding operations of a current coding unit using the input motion information.

[0094] For example, the input motion information can be used for inter prediction of the current coding unit. In inter prediction, it can be used as a temporal motion vector prediction candidate in merge mode. Alternatively, as an example, it can be used to obtain a sub-block based temporal merge candidate (SbTMVP) in a sub-block merge candidate of inter prediction. Alternatively, as an example, it can be used to obtain a constructed affine control point motion vector in an affine mode of inter prediction.

[0095] Motion information including the motion vector field used in the coding unit encoder / decoder (320) may be transmitted to the motion vector field normalization unit (330).

[0096] The motion vector field normalization unit (330) may normalize the motion vector field using motion information including the input motion vector field, and transmit the normalized motion vector field to the compression neural network (340).

[0097] According to one embodiment of the present disclosure, the motion vectors of the input motion vector field may refer to different reference pictures. Therefore, the scaling of the motion vectors may be different from each other, and data with spatially different scaling cannot be processed by the neural network, so it is necessary to normalize the scaling of all motion vectors of the motion vector field to be the same. In this case, the motion vectors may be scaled based on a POC difference of 1, as in the following equation 4. In other words, the motion vector field normalization unit (330) may scale the motion vectors included in the motion vector field so that they have a unit POC difference of 1.

[0098] [Formula 4]

[0099] td=Clip3(-128, 127, currPocDiff)

[0100] tx=(16384+(Abs(td)>>1)) / td

[0101] distScaleFactor=Clip3(-4096, 4095, (tx+32)>>6)

[0102] mvLXCol=Clip3(-131072, 131071, (distScaleFactor*mvCol+128-(distScaleFactor*mvCol>=0)>>8)

[0103] The compression neural network (340) can generate a compressed tensor by performing compression on the input normalized motion vector field using multiple neural network layers. As an example, the tensor compressed by the compression neural network (340) can have a lower spatial resolution than the input motion vector field.

[0104] Since the compression neural network (340) can perform spatial sampling together using a convolution filter (or convolution layer), it can express motion more accurately than the existing technology that performs sampling for a specific position. Therefore, video coding efficiency can be improved by using the motion information that is more accurately expressed for subsequent temporal motion vector prediction. The compressed tensor can be transmitted to the quantization unit (350).

[0105] The quantization unit (350) may quantize the input compressed tensor to generate a quantized tensor. The quantized tensor may be transmitted to the storage unit (360).

[0106] The storage unit (360) can store the received quantized tensor in a memory for encoding / decoding of subsequent frames.

[0107] Figure 4 An example of a conceptual graph for compressing and reconstructing neural network learning according to one embodiment of the present disclosure is shown.

[0108] Reference Figure 4 , the original motion vector field ( Figure 4 The original MVF in can be input to the compression neural network by merging the motion vector fields of the two prediction directions into the channel axis, such as Figure 4 As an example, Figure 4 The compression neural network and reconstruction neural network can be respectively Figure 3 The compression neural network (340) and the reconstruction neural network (300) of Figure 4 In , MVF represents a motion vector field, and each MVF may be a motion vector field for L0 and L1. In addition, as an example, each MVF may be a motion vector field normalized to a unit POC difference by motion information.

[0109] As an example, Figure 4 The quantization in may mean a general ROUND operation. Alternatively, it may be a quantization performed by a specific quantization parameter.

[0110] The motion vector field reconstructed from the reconstruction neural network can have the same spatial resolution and number of channels as the original motion vector field.

[0111] In addition, as an example, the sum of distortion and bit rate can be used as a loss function for training the compression neural network and the reconstruction neural network. As an example, the loss function can be defined as in the following equation 5.

[0112] [Formula 5]

[0113] Loss=Bitrate+λ×Distortion

[0114] Here, the distortion may be derived as the difference between the original motion vector field and the reconstructed motion vector field. As an example, the mean square error (MSE) or the sum of absolute differences (SAD) may be used to calculate the difference between the original motion vector field and the reconstructed motion vector field.

[0115] In addition, as an example, since it is difficult to measure the actual bit generation amount, the potential tensor can be used to predict the bit rate. In this case, the predicted value of the bit rate can be the entropy of the distribution of the usage value. Alternatively, it can be a predicted value based on a probability value obtained by a neural network.

[0116] As described above, the reconstructed motion vector field can be used for temporal motion vector prediction. Therefore, a neural network can be used to generate motion vectors that more accurately express motion than existing motion vector fields.

[0117] In addition, according to one embodiment of the present disclosure, a knowledge distillation-based learning method may be used to train a compression neural network and a reconstruction neural network for more accurate motion expression.

[0118] Knowledge distillation is a learning method that performs network learning by using the results of a previously learned network that performs the same task as the network to be learned. Knowledge distillation can be used when the network to be learned is relatively small or when learning needs to be done on new data. Knowledge distillation can be used with the previously used loss function by adding the sum of the differences between the results obtained from the previously learned network and the results obtained from the network being learned to the loss function.

[0119] Figure 5 A conceptual diagram showing compression and reconstruction neural network learning using knowledge distillation according to one embodiment of the present disclosure.

[0120] Reference Figure 5 , the compression and reconstruction neural network can be trained by learning based on knowledge distillation to train the neural network so that the reconstructed motion vector field can express more accurate motion. In this case, the trained network can be called a teacher network, and the compression and reconstruction neural network can be called a student network.

[0121] In one embodiment, the teacher network can be a flow network (or optical flow network), which is one of the neural networks that predicts optical flow. The flow network can take two images as input and predict the motion between them. The two images can be the previous frame and the next frame of the current frame.

[0122] That is, the optical flow can be predicted by inputting the previous / following frames of the current frame into the flow network. By changing the optical flow into a form similar to the motion vector field, the distortion relative to the reconstructed motion vector field can be measured. The measured distortion can be used as a distillation loss. In this case, MSE, SAD, etc. can be used to calculate the distortion. As an example, the loss function can be defined as in the following equation 6.

[0123] [Formula 6]

[0124] Loss=Bitrate+λ×Distortion+Distillation loss

[0125] Referring to Formula 6, the loss may be defined by using the same distortion and bit rate as previously used and additionally adding a distillation loss, and the compression and reconstruction neural network may be trained to minimize the loss. As an example, the training may be repeated a predefined number of times, or the training may be repeated such that the loss is reduced below a predefined threshold.

[0126] In other words, a compression neural network can be trained based on the above-mentioned flow network, and the trained compression neural network can be used to compress and store motion vectors (or motion information) for time / space prediction. Alternatively, a reconstruction neural network can be trained based on the above-mentioned flow network, and the trained reconstruction neural network can be used to reconstruct motion vectors (or motion information) for time / space prediction.

[0127] Figure 6 is a flow chart illustrating a neural network-based motion vector field compression method according to one embodiment of the present disclosure.

[0128] Above Figures 1 to 5 The embodiments described in the above can be similarly applied to the motion vector field compression process according to the present disclosure. The repeated description in this regard is omitted.

[0129] Reference Figure 6 , a motion vector field may be generated using motion information for motion prediction of a processing unit included in the current picture (S600). Here, the motion information may include at least one of a prediction direction flag, a reference index, or a motion vector. The processing unit may be the above-mentioned encoding unit.

[0130] A tensor of a motion vector field may be generated by performing compression on the motion vector field based on a neural network including a plurality of neural network layers (S610). In this case, as an example, the plurality of neural network layers may include at least one convolutional layer.

[0131] Additionally, as described above, the motion vector field may be spatially sampled based on at least one convolutional layer.

[0132] As described above, as an example, the motion vector field may be normalized based on the POC (Picture Order Count) difference between the reference picture specified by the reference index of the processing unit and the current picture. In this case, the tensor may be generated by performing compression on the motion vector field on which normalization has been performed. In addition, the above-mentioned normalization may be performed by scaling the motion vector used for motion prediction of the processing unit by the POC difference to obtain a motion vector having a unit POC difference, and modifying the motion vector field using the motion vector having the unit POC difference.

[0133] As described above, as an example, a quantized tensor may be generated by performing quantization on a tensor, and the generated quantized tensor may be stored in a memory. In this case, the stored quantized tensor may be used for motion prediction of a processing unit in a subsequent picture of the current picture.

[0134] As described above, as an example, the neural network can be trained by a loss function defined based on the sum of distortion and bit rate. In this case, the distortion can represent the difference between the original motion vector field and the reconstructed motion vector field. And, the difference can be calculated using MSE (mean square error) or SAD (sum of absolute differences). In addition, as an example, the bit rate can be predicted using the tensor of the motion vector field generated in step S610. Alternatively, the bit rate can be predicted using a potential tensor. Alternatively, the bit rate can be predicted using a probability value obtained based on a neural network.

[0135] As described above, as an example, the loss function can be defined by additionally considering the distortion between the motion vector field estimated by the teacher network and the motion vector field reconstructed by the student network. In this case, the teacher network can be a flow network that predicts the optical flow between the previous picture and the next picture based on the current picture.

[0136] Figure 7 is a flowchart illustrating a neural network-based motion vector field reconstruction and motion prediction method according to one embodiment of the present disclosure.

[0137] Above Figures 1 to 6 The embodiments described in the above can be applied to the motion vector field reconstruction and motion prediction process according to the present disclosure in a substantially identical manner. That is, the motion vector field reconstruction can be a process corresponding to the motion vector field compression. The repeated description in this regard will be omitted.

[0138] The motion vector field compressed and stored in the memory may be reconstructed for motion prediction (S700). In this case, the motion vector field may be reconstructed using Figures 3 to 5 The reconstruction neural network described in .

[0139] And, the reconstructed motion vector field may be scaled (S710). As described above, the motion vector used in the previous picture may have a different reference picture, and thus the scaling of the motion degree may be different. Therefore, scaling may be performed according to the reference picture to make the scaling of the motion vector in the motion vector field the same.

[0140] In addition, the reconstructed motion vector field may be a motion vector field having a unit POC difference of 1. Therefore, scaling may be performed as much as the POC difference between the current POC and the POC of the reference picture.

[0141] The encoding / decoding of the current processing unit may be performed using the scaled motion vector field (S720). In other words, the motion prediction of the current processing unit may be performed using the motion vector field.

[0142] For example, the input motion information can be used for inter prediction of the current coding unit. In inter prediction, it can be used as a temporal motion vector prediction candidate in merge mode. Alternatively, as an example, it can be used to obtain a sub-block based temporal merge candidate (SbTMVP) in a sub-block merge candidate of inter prediction. Alternatively, as an example, it can be used to obtain a constructed affine control point motion vector in an affine mode of inter prediction.

[0143] The above-mentioned embodiments may be a combination of components and features of the present disclosure in a predetermined form. Unless otherwise expressly stated, each component or feature should be considered to be selective. Each component or feature may be implemented in a form not combined with other components or features. In addition, some components and / or features may be combined to configure the embodiments of the present disclosure. The order of the operations described in the embodiments of the present disclosure may be changed. Some configurations or features of the embodiments may be included in other embodiments, or may be replaced with configurations or features corresponding to other embodiments. It is obvious that claims that do not have a clear reference relationship in the scope of the claims may be combined to configure the embodiments, or may be included as new claims by modification after the application.

[0144] The embodiments according to the present disclosure may be implemented in various ways, for example, hardware, firmware, software, or a combination thereof, etc. For implementation by hardware, the embodiments of the present disclosure may be implemented by one or more ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, etc.

[0145] In addition, for the implementation by firmware or software, the embodiments of the present disclosure can be implemented in the form of modules, programs, functions, etc. that perform the above functions or operations, and can be recorded in a readable recording medium by various computer means. Here, the recording medium may include program instructions, data files, data structures, etc., alone or in combination. The program instructions recorded in the recording medium may be those program instructions specially designed and configured for the present disclosure, or those program instructions that are available by notifying computer software technicians. For example, the recording medium includes: magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as CD-ROMs (compact disk read-only memory) and DVDs (digital video disks); magneto-optical media, such as optical floppy disks; and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, flash memories, etc. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter, etc., and machine language codes such as those generated by a compiler. Such a hardware device may be configured to operate as at least one software device for performing the operations of the present disclosure, and vice versa.

[0146] In addition, the device or terminal according to the present disclosure can be driven by a command that causes at least one processor to perform the above functions and processes. For example, such a command may include, for example, an interpreted command, such as a script command such as a JavaScript or ECMAScript command, or other commands stored in a computer-readable medium readable or executable code. In addition, the device according to the present disclosure can be implemented in a distributed manner across a network such as a server group, or can be implemented in a single computer device.

[0147] In addition, a computer program (also referred to as a program, software, software application, script or code) provided together with an apparatus according to the present disclosure and executing a method according to the present disclosure can be written in any form of programming language including a compiled or interpreted language or a priori or procedural language, and can be deployed in any form including independent programs, modules, components or subroutines or other units suitable for use in a computer environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a single file provided for a requested program, or in a plurality of interactive files (e.g., a file storing a portion of at least one module, subroutine or code), or in a portion of a file having other programs or data (e.g., at least one script stored in a markup language document). A computer program may be located in a site or distributed across multiple sites, and may be deployed to execute on a computer or on multiple computers interconnected by a communication network.

[0148] It is obvious to those skilled in the art that the present disclosure may be implemented in other specific forms without departing from the basic features of the present disclosure. Therefore, the detailed description described above should not be interpreted restrictively in all aspects, but should be regarded as illustrative. The scope of the present disclosure should be determined by a reasonable interpretation of the attached claims, and all changes within the equivalent range of the present disclosure are included within the scope of the present disclosure.

Claims

1. A neural network-based image processing method, comprising: generating a motion vector field using motion information for motion prediction of a processing unit included in a current picture, the motion information including at least one of a prediction direction flag, a reference index, or a motion vector; and A tensor of the motion vector field is generated by performing compression on the motion vector field based on a neural network including a plurality of neural network layers.

2. The method according to claim 1, wherein: The plurality of neural network layers include at least one convolutional layer.

3. The method according to claim 2, wherein: Performing the compression on the motion vector field comprises: The motion vector field is spatially sampled based on the at least one convolutional layer.

4. The method according to claim 1, further comprising: performing normalization on the motion vector field based on a picture order count (POC) difference between a reference picture specified by a reference index of the processing unit and the current picture, The tensor is generated by performing compression on the motion vector field after normalization.

5. The method according to claim 4, wherein: Performing the normalization includes: Obtaining a motion vector having a unit POC difference by scaling a motion vector for motion prediction of the processing unit by the POC difference; and The motion vector field is modified using the motion vector having the unit POC difference.

6. The method according to claim 1, further comprising: generating a quantized tensor by performing quantization on the tensor; as well as storing the quantized tensor in a memory, The stored quantized tensor is used for motion prediction of a processing unit in a subsequent picture of the current picture.

7. The method according to claim 1, wherein: The neural network is learned using a loss function defined as the sum of distortion and bit rate, wherein the distortion represents the difference between the original motion vector field and the reconstructed motion vector field, and Here, the difference is calculated using MSE (mean square error) or SAD (sum of absolute differences).

8. The method according to claim 7, wherein: The bitrate is predicted using a latent tensor.

9. The method according to claim 7, wherein: The bit rate is predicted using the probability value obtained based on the neural network.

10. The method according to claim 7, wherein: The loss function is defined by additionally considering the distortion between the motion vector field estimated by the teacher network and the motion vector field reconstructed by the student network.

11. The method according to claim 10, wherein: The teacher network is a flow network that predicts the optical flow between a previous picture and a subsequent picture based on the current picture.

12. An image processing device based on a neural network, comprising: a processor, the processor controlling the image processing device; as well as a memory coupled to the processor and storing data, Wherein, the processor is configured to: generating a motion vector field using motion information for motion prediction of a processing unit included in a current picture, the motion information including at least one of a prediction direction flag, a reference index, or a motion vector, and A tensor of the motion vector field is generated by performing compression on the motion vector field based on a neural network including a plurality of neural network layers.

Images