AI-based image encoding apparatus and image decoding apparatus, and image encoding and decoding method using image encoding apparatus and image decoding apparatus

Abstract

An artificial intelligence (AI)-based image encoding method includes: obtaining motion vector feature data of a current frame by inputting the current frame and a reference frame to a motion vector encoder and applying a plurality of resolution scales to the current frame and the reference frame via the motion vector encoder; obtaining a current motion vector by inputting the motion vector feature data to a motion vector decoder; obtaining a current predicted image based on the current motion vector and the reference frame; and generating a bitstream based on the motion vector feature data and the current predicted image.

Description

Technical Field

[0001] The present disclosure relates to image encoding and decoding. More specifically, the present disclosure relates to a technique for encoding and decoding images by using artificial intelligence (AI) (eg, neural network). Background Art

[0002] Codecs such as H.264 Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC) can divide an image into blocks and predictively encode and decode each block through inter-frame prediction or intra-frame prediction.

[0003] Intra-frame prediction is a method of compressing an image by removing spatial redundancy in the image, and inter-frame prediction is a method of compressing an image by removing temporal redundancy between images.

[0004] A representative example of inter-frame prediction is motion estimation coding. Motion estimation coding predicts a block of a current image by using a reference image. By using a predetermined evaluation function, a reference block that is most similar to the current block can be found within a predetermined search range. The current block is predicted based on the reference block, and the prediction block generated as a prediction result is subtracted from the current block to generate a residual block. The residual block is then encoded.

[0005] In order to derive a motion vector indicating a reference block in a reference image, a motion vector of a previously encoded block may be used as a motion vector predictor of a current block. A differential motion vector signaling corresponding to a difference between a motion vector of a current block and a motion vector predictor of a current block is notified to a decoder side by a predetermined method.

[0006] Recently, a technology for encoding / decoding an image by using artificial intelligence (AI) has been proposed, and a method for efficiently encoding / decoding an image using AI (eg, a neural network) has been required. Summary of the invention

[0007] Solution to the problem

[0008] According to an embodiment of the present disclosure, an artificial intelligence (AI)-based image encoding method may include: obtaining motion vector feature data of a current frame by inputting the current frame and a reference frame into a motion vector encoder and applying multiple resolution scales to the current frame and the reference frame via the motion vector encoder; obtaining a current motion vector by inputting the motion vector feature data into a motion vector decoder; obtaining a current predicted image based on the current motion vector and the reference frame; and generating a bitstream based on the motion vector feature data and the current predicted image.

[0009] According to an embodiment of the present disclosure, an AI-based image encoding device may include a prediction encoder, a prediction decoder, and a generator. The prediction encoder may be configured to obtain motion vector feature data of the current frame by inputting the current frame and the reference frame into a motion vector encoder and applying multiple resolution scales to the current frame and the reference frame via the motion vector encoder. The prediction decoder may be configured to obtain the current motion vector by inputting the motion vector feature data into a motion vector decoder. The prediction decoder may be configured to obtain a current predicted image based on the current motion vector and the reference frame. The generator may be configured to generate a bitstream based on the motion vector feature data and the current predicted image.

[0010] According to an embodiment of the present disclosure, an AI-based image decoding method may include: obtaining motion vector feature data of a current frame of multiple resolutions from a bitstream; obtaining a current motion vector by applying the motion vector feature data to a motion vector decoder; and obtaining a current predicted image by using the current motion vector and a reference frame. The multiple resolutions may include an original resolution of the current frame and a resolution reduced from the original resolution of the current frame.

[0011] According to an embodiment of the present disclosure, an AI-based image decoding device may include an obtainer and a prediction decoder. The obtainer may be configured to obtain motion vector feature data of a current frame of multiple resolutions from a bitstream. The prediction decoder may be configured to obtain a current motion vector by applying the motion vector feature data to a motion vector decoder. The prediction decoder may be configured to obtain a current predicted image by using the current motion vector and a reference frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram illustrating an image encoding and decoding process based on artificial intelligence (AI) according to an embodiment of the present disclosure.

[0013] Figure 2 is a view for explaining a motion search range according to the resolution of an image.

[0014] Figure 3 is a diagram for explaining the receptive field of a neural network according to a convolution operation.

[0015] Figure 4 is a block diagram of a neural network structure of a motion vector encoder according to an embodiment of the present disclosure.

[0016] Figure 5 is a block diagram of a neural network structure of a motion vector encoder according to an embodiment of the present disclosure.

[0017] Figure 6 is a block diagram of a neural network structure of a motion vector decoder according to an embodiment of the present disclosure.

[0018] Figure 7 is a block diagram of a neural network structure of a motion vector decoder according to an embodiment of the present disclosure.

[0019] Figure 8 is a block diagram of a neural network structure of a motion vector encoder according to an embodiment of the present disclosure.

[0020] Fig. 9 is a block diagram of a neural network structure of a motion vector decoder according to an embodiment of the present disclosure.

[0021] Fig.10 is a view used to explain the method of performing pooling.

[0022] Fig.11 is a view for explaining an example of a method of performing dilated convolution.

[0023] Fig.12 is a flowchart of an image encoding method according to an embodiment of the present disclosure.

[0024] Fig.13 is a block diagram showing a structure of an image encoding device according to an embodiment of the present disclosure.

[0025] Fig.14 is a flowchart of an image decoding method according to an embodiment of the present disclosure.

[0026] Fig.15 is a flowchart of an image decoding method according to an embodiment of the present disclosure.

[0027] Fig.16 is a diagram for describing a method of training a motion vector encoder, a motion vector decoder, a residual encoder, and a residual decoder according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Since the present disclosure allows various changes and many embodiments, specific embodiments will be shown in the drawings and described in detail in the written description. However, this is not intended to limit the present disclosure to a specific mode of practice, and it should be understood that all changes, equivalents and substitutes that do not depart from the spirit and technical scope of the present disclosure are included in the present disclosure.

[0029] In the description of the embodiments of the present disclosure, when it is considered that some detailed explanations of the related art may unnecessarily obscure the essence of the present disclosure, these detailed explanations are omitted. When terms such as "first", "second", etc. can be used to describe various components, such components should not be limited to the above terms. The above terms are only used to distinguish one component from another component.

[0030] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0031] When one element (eg, a first element) is “coupled” or “connected” to another element (eg, a second element), the first element may be directly coupled or connected to the second element, or, unless otherwise described, a third element may be present therebetween.

[0032] Regarding the components represented as "part (unit)" or "module" used in this document, two or more components may be combined into one component, or one component may be divided into two or more components according to the subdivided functions. In addition, each component described below may additionally perform some or all of the functions performed by another component in addition to its own main functions, and some main functions of each component may be completely performed by another component.

[0033] As used herein, an “image” may refer to a still image, a picture, a frame, a moving picture composed of a plurality of consecutive still images, or a video.

[0034] The "neural network" used in this article is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model that uses a specific algorithm. A neural network can also be called a deep neural network.

[0035] The "parameter" used herein (which is a value used in the calculation process of each layer included in the neural network) can be used, for example, when applying an input value to a predetermined calculation formula. The parameter whose value is set as a result of training can be updated by separate training data as needed.

[0036] As used herein, "feature data" refers to data obtained by processing input data by a neural network-based encoder. Feature data may be one-dimensional or two-dimensional (1D or 2D) data including a plurality of samples. Feature data may also be referred to as a potential representation. Feature data may represent potential features of data output by a decoder described below.

[0037] The "current image" used herein refers to the image currently being processed, and the "previous image" used herein refers to the image to be processed before the current image. The "current motion vector" refers to the motion vector obtained for processing the current image.

[0038] As used herein, a "sample" (which is data assigned to a sampling position in an image, feature map, or feature data) refers to data to be processed. For example, a sample may be a pixel in a 2D image.

[0039] Figure 1is a diagram illustrating an image encoding and decoding process based on artificial intelligence (AI) according to an embodiment of the present disclosure.

[0040] refer to Figure 1 , the motion vector encoder 110, the residual encoder 130, the motion vector decoder 150 and the residual decoder 170 can be used for AI-based inter-frame prediction.

[0041] The motion vector encoder 110, the residual encoder 130, the motion vector decoder 150, and the residual decoder 170 may be implemented as neural networks.

[0042] The motion vector encoder 110 and the motion vector decoder 150 can be understood as neural networks, which are used to extract motion vector feature data w of the current image (eg, current frame) 10 of multiple resolutions from the current image 10 and the previously reconstructed image (eg, reference frame) 30. i , and from the motion vector feature data w i Extract the current motion vector g i The current frame and the reference frame may be part of the same video sequence.

[0043] The residual encoder 130 and the residual decoder 170 may be used to extract the received residual image r i A neural network that takes feature data and reconstructs an image from the feature data.

[0044] Inter prediction is a process of encoding and decoding a current image 10 by using temporal redundancy between the current image 10 and a previously reconstructed image 30 .

[0045] The position differences (or motion vectors) between blocks or samples in the current image 10 and reference blocks or reference samples in the previously reconstructed image 30 are used for encoding and decoding the current image 10. These position differences may also be referred to as optical flows.

[0046] The optical flow (specifically, the current optical flow) may indicate how the positions of samples in the previous reconstructed image 30 are changed in the current image 10 , or where samples identical to or similar to samples in the current image 10 are located in the previous reconstructed image 30 .

[0047] For example, when the sample point that is identical or most similar to the sample point located at (1, 1) in the current image 10 is located at (2, 1) in the previous reconstructed image 30, the optical flow or motion vector of the sample point may be derived as (1 (=2-1), 0 (=1-1)).

[0048] In the image encoding and decoding process using AI, the motion vector encoder 110 and the motion vector decoder 150 can be used to obtain the current optical flow g of the current image 10. i .

[0049] In detail, the previous reconstructed image 30 and the current image 10 may be input to the motion vector encoder 110. The motion vector encoder 110 may output motion vector feature data w of the current image 10 of multiple resolutions by processing the current image 10 and the previous reconstructed image 30 according to the parameters set as a training result. i .

[0050] Motion vector feature data w i may be input to the motion vector decoder 150. The motion vector decoder 150 may process the motion vector feature data w according to the parameters set as a training result. i To output the current motion vector g i .

[0051] Based on the current motion vector g i Motion compensation 190 may be performed on the previously reconstructed image 30 and the current predicted image x' i It may be obtained as a result of motion compensation 190 .

[0052] Motion compensation 190 is a process of obtaining a current predicted image of the current image 10 using pixels indicated by a motion vector in a previously reconstructed image 30 .

[0053] The current predicted image x' is similar to the current image 10 i The current motion vector g representing the relative position relationship between the sample points in the previous reconstructed image 30 and the sample points in the current image 10 can be used to i This is obtained by applying motion compensation 190 (or warping) to the previously reconstructed image 30 .

[0054] For example, when a sample located at (1, 1) in the previous reconstructed image 30 is most similar to a sample located at (2, 1) in the current image 10, the position of the sample located at (1, 1) in the previous reconstructed image 30 may be changed to (2, 1) through motion compensation 190.

[0055] Since the current predicted image x' is generated from the previous reconstructed image 30 i It is not the current image 10 itself, so we can get the corresponding current predicted image x' i The residual image r of the difference between the current image 10 i For example, the current predicted image x' can be obtained by subtracting the sample value in the current image 10 from the sample value in the current image 10. i The residual image r is obtained by i .

[0056] Residual image r imay be input to the residual encoder 130. The residual encoder 130 may process the residual image r by using the parameters set as a result of the training. i To output the residual image r i The characteristic data v i .

[0057] Residual image r i The characteristic data v i can be input to the residual decoder 170. The residual decoder 170 can process the input feature data v by the parameters set as the training result. i To output the reconstructed residual image r' i .

[0058] The current reconstructed image 50 can be obtained by converting the current predicted image x′ generated by motion compensation 190 with respect to the previous reconstructed image 30. i And the reconstructed residual image data r' i Combine to obtain.

[0059] when Figure 1 When the image encoding and decoding process shown is implemented by an encoding device and a decoding device, the encoding device can quantize the motion vector feature data w obtained by encoding the current image 10. i and the residual image r i The characteristic data v i , generating a bit stream including quantized feature data, and sending the generated bit stream to a decoding device. The decoding device can obtain motion vector feature data w by dequantizing the quantized feature data extracted from the bit stream. i and the residual image r i The characteristic data v i The decoding apparatus may process the motion vector feature data w by using the motion vector decoder 150 and the residual decoder 170. i and the residual image r i The characteristic data v i To obtain the current reconstructed image 50.

[0060] Figure 2 is a view for explaining a motion search range according to the resolution of an image.

[0061] Motion estimation is performed by comparing the pixels of the reference frame with the pixels of the current frame. The searchable size of the motion vector is determined by the motion search range. When the motion search range is small, small motion vectors can be accurately estimated, and when the motion search range is large, large motion vectors and small motion vectors can be accurately estimated. In order to improve the encoding and decoding performance of large motion sequences, a large motion search range is required.

[0062] Within a limited motion search range, the smaller the resolution, the more suitable the resolution is for motion search of large motion vectors.

[0063] refer to Figure 2 , when the motion vector of the current pixel is large, the position indicated by the motion vector of the current frame 215 of the original resolution corresponding to the high resolution relative to the reference frame 210 is not within the limited motion search range, so the motion search is not appropriate. In other words, at the original resolution, the large movement of the current pixel may not be covered.

[0064] In the case of a motion vector of the current frame 225 having a resolution reduced by 1 / 2 relative to the reference frame 220, the position indicated by the motion vector may be located within the limited motion search range and may be located at the boundary of the limited motion search range, and thus may not be suitable for motion search. In other words, at the resolution reduced by 1 / 2, a large movement of the current pixel may be barely covered.

[0065] In the case of a motion vector of the current frame 235 having a resolution reduced by 1 / 4 relative to the reference frame 230, the position indicated by the motion vector can be located within the limited motion search range and can therefore be suitable for motion search. In other words, at a resolution reduced by 1 / 4, a large movement of the current pixel can be fully covered.

[0066] Figure 3 is a diagram for explaining the receptive field of a neural network according to a convolution operation.

[0067] refer to Figure 3 , when a convolution operation is performed by applying a 3×3 filter kernel to the first layer 310 of 5×5 sample data, 3×3 sample data is obtained as a part of the second layer 320 corresponding to the 5×5 sample data. When a convolution operation is performed by applying a 3×3 filter kernel to 3×3 sample data corresponding to the 5×5 sample data in the first layer 310 among the sample data of the second layer 330, 1×1 final sample data is obtained from the final layer 330. Because the sample data of the first layer 310 used to obtain the 1×1 final sample data is 5×5 sample data, the receptive field of the neural network is 5×5. In other words, the range of input pixels that contribute to the final output pixel is the receptive field.

[0068] For example, when there are 5 layers before the final layer and the filter kernel is 3×3, the receptive field is 11×11. When there are 10 layers before the final layer and the filter kernel is 3×3, the receptive field is 21×21. When there are 20 layers before the final layer and the filter kernel is 3×3, the receptive field is 41×41. When there are 40 layers before the final layer and the filter kernel is 3×3, the receptive field is 81×81.

[0069] Not every input pixel in the receptive field contributes significantly to the final output pixel. Input pixels in the center part of the receptive field may contribute significantly to the final output pixel, and input pixels in the edge part of the receptive field may contribute moderately to the final output pixel. Therefore, the effective receptive field may be smaller than the receptive field.

[0070] Convolutional neural networks are provided with a small motion search range for motion estimation. Therefore, Figure 2 As shown, by performing motion estimation at a resolution obtained by reducing the resolution of an image, motion estimation can be performed on a large motion vector within an appropriate motion search range.

[0071] By having a neural network structure that can obtain feature data at both the original resolution and the resolution reduced from the original resolution, the effective receptive field can be extended, so both large and small motion vectors can be covered within a limited motion search range.

[0072] Now refer to Figures 4 to 9 An example of a neural network structure of a motion vector decoder and a motion vector encoder is described, which obtains motion vector feature data of multiple resolutions and obtains a current motion vector by using the motion vector feature data.

[0073] Figure 4 is a block diagram of a neural network structure of a motion vector encoder according to an embodiment of the present disclosure.

[0074] refer to Figure 4 , the motion vector encoder 110 may include a plurality of neural network layers, namely, a first neural network layer 405, a second neural network layer 420, and a third neural network layer 440. In detail, all pixel data of an input image 400 obtained by concatenating a current image (e.g., a current frame) with a previous reconstructed image (e.g., a previous frame or a reference frame) is input to each of a first convolution layer 401 and a pooling layer 402 in the first neural network layer 405 in the motion vector encoder 110. A first feature map 410 is obtained from the first convolution layer 401 in the first neural network layer 405, and a second feature map 415 is obtained from the pooling layer 402 and the second convolution layer 403 in the first neural network layer 405.

[0075] The first convolution layer 401 may be a convolution layer with a stride of N, the pooling layer 402 may be a pooling layer that reduces the size of the feature map by M, and the second convolution layer 403 may be a convolution layer with a stride of L, where L may be N / M. Because the pooling layer reduces the size of the input data, the second feature map 415 output by the pooling layer 402 and the second convolution layer 403 may be a feature map of a resolution obtained by reducing the original resolution of the input image by 1 / M.

[0076] The following will refer to Fig.10 Describes how to perform "pooling".

[0077] For example, when N is 2 and M is 2, L may be 1. In this case, the first feature map may be a feature map of the original resolution of the input image, and the second feature map may be a feature map of a resolution obtained by reducing the original resolution of the input image by 1 / 2.

[0078] The number of channels of the first feature map 410 and the number of channels of the second feature map 415 need not be the same as each other.

[0079] The first feature map 410 and the second feature map 415 are concatenated, and then the entire concatenated feature map data (the first feature map 410 and the second feature map 415) are input to each of the first convolution layer 421 and the pooling layer 422 in the second neural network layer 420. The third feature map 425 is obtained from the first convolution layer 421 in the second neural network layer 420, and the fourth feature map 430 is obtained from the pooling layer 422 and the second convolution layer 423 in the second neural network layer 420.

[0080] The second neural network layer 420 may be the same as the first neural network layer 405. The first convolution layer 421 may be a convolution layer with a stride of N, the pooling layer 422 may be a pooling layer that reduces the size of the feature map by M, and the second convolution layer 423 may be a convolution layer with a stride of L, where L may be N / M. When N is 2 and M is 2, and therefore L is 1, the third feature map 425 may be a feature map of the original resolution of the input image and a resolution obtained by reducing the original resolution of the input image by 1 / 2, and the fourth feature map 430 may be a feature map of a resolution obtained by reducing the original resolution of the input image by 1 / 2 and a resolution obtained by reducing the original resolution of the input image by 1 / 4.

[0081] The number of channels of the third feature map 425 and the number of channels of the fourth feature map 430 need not be identical to each other.

[0082] The third feature map 425 and the fourth feature map 430 are concatenated, and then the entire concatenated feature map data (the third feature map 425 and the fourth feature map 430) are input to each of the first convolution layer 441 and the pooling layer 442 in the third neural network layer 440. The fifth feature map 450 is obtained from the first convolution layer 441 in the third neural network layer 440, and the sixth feature map 450 is obtained from the pooling layer 442 and the second convolution layer 423 in the third neural network layer 440.

[0083] The third neural network layer 440 may be the same as the first neural network layer 405. The first convolution layer 441 may be a convolution layer with a stride of N, the pooling layer 442 may be a pooling layer that reduces the size of the feature map by M, and the second convolution layer 443 may be a convolution layer with a stride of L, where L may be N / M. When N is 2 and M is 2, and therefore L is 1, the fifth feature map 445 may be a feature map of the original resolution of the input image, a resolution obtained by reducing the original resolution of the input image by 1 / 2, and a resolution obtained by reducing the original resolution of the input image by 1 / 4, and the sixth feature map 450 may be a feature map of a resolution obtained by reducing the original resolution of the input image by 1 / 2, a resolution obtained by reducing the original resolution of the input image by 1 / 4, and a resolution obtained by reducing the original resolution of the input image by 1 / 8.

[0084] Because the effective receptive field is very large in the case of a feature map with a resolution obtained by reducing the original resolution of the input image by 1 / 8 compared to a feature map with a relatively large scale resolution, a large motion vector can be effectively estimated in the case of a feature map with a resolution obtained by reducing the original resolution of the input image by 1 / 8. Since only feature maps of the same resolution are obtained in a general neural network structure, the effective receptive field is limited, and in Figure 4 In the motion vector encoder of , since pooling is performed before the convolution operation, the effective receptive field is expanded for the layers that output feature maps with reduced resolution.

[0085] The number of channels of the fifth feature map 445 and the number of channels of the sixth feature map 450 need not be the same as each other.

[0086] The motion vector feature data of the current image at multiple resolutions are obtained using the fifth feature map 445 and the sixth feature map 450 .

[0087] The motion vector encoder comprises at least one layer of reduced feature maps.

[0088] Because the output of each layer from the motion vector encoder is a mixture of feature maps of multiple resolutions, the final output data from the motion vector encoder is the motion vector feature data of the current image at multiple resolutions. The final output motion vector feature data of the current image at multiple resolutions is compressed to the size of the feature map by the motion vector encoder, thereby reducing the bit rate.

[0089] Figure 5 is a block diagram of a neural network structure of a motion vector encoder according to an embodiment of the present disclosure.

[0090] refer to Figure 5, the motion vector encoder 110 may include a plurality of neural network layers, namely, a first neural network layer 505, a second neural network layer 520, and a third neural network layer 540. In detail, all pixel data of the input image 500 obtained by concatenating the current image with the previous reconstructed image is input to each of the first convolution layer 501 and the pooling layer 503 in the first neural network layer 505 of the motion vector encoder 110. The first feature map 510 is obtained from the first convolution layer 501 and the pooling layer 502 in the first neural network layer 505, and the second feature map 515 is obtained from the pooling layer 503 and the second convolution layer 504 in the first neural network layer 505.

[0091] The first convolution layer 501 and the second convolution layer 504 may be convolution layers with a stride of N, and the pooling layer 502 and the pooling layer 503 may be pooling layers that reduce the size of the feature map by M. Because the pooling layer 503 reduces the size of the input data, the second feature map 515 output by the pooling layer 503 and the second convolution layer 504 may be a feature map of a resolution obtained by reducing the original resolution of the input image by 1 / M. Because the first feature map 510 output by the first convolution layer 501 and the pooling layer 502 is merely the result of first obtaining an intermediate feature map through the first convolution layer 501 and then reducing the size of the intermediate feature map, the first feature map 510 may be a feature map of the original resolution of the input image.

[0092] For example, when N is 1 and M is 2, the first feature map may be a feature map of the original resolution of the input image, and the second feature map may be a feature map of a resolution obtained by reducing the original resolution of the input image by 1 / 2.

[0093] The number of channels of the first feature map 510 and the number of channels of the second feature map 515 need not be the same as each other.

[0094] The first feature map 510 and the second feature map 515 are concatenated, and then the entire concatenated feature map data (the first feature map 510 and the second feature map 515) are input to each of the first convolution layer 521 and the pooling layer 523 of the second neural network layer 520. The third feature map 525 is obtained from the first convolution layer 521 and the pooling layer 522 in the second neural network layer 520, and the fourth feature map 530 is obtained from the pooling layer 523 and the second convolution layer 524 in the second neural network layer 520.

[0095] The second neural network layer 520 may be the same as the first neural network layer 505. When N is 1 and M is 2, the third feature map 525 may be a feature map of the original resolution of the input image and a resolution obtained by reducing the original resolution of the input image by 1 / 2, and the fourth feature map 530 may be a feature map of a resolution obtained by reducing the original resolution of the input image by 1 / 2 and a resolution obtained by reducing the original resolution of the input image by 1 / 4.

[0096] The number of channels of the third feature map 525 and the number of channels of the fourth feature map 530 need not be identical to each other.

[0097] The third feature map 525 and the fourth feature map 530 are concatenated, and then the entire concatenated feature map data (the third feature map 525 and the fourth feature map 530) are input to each of the first convolution layer 541 and the pooling layer 543 of the third neural network layer 540. The fifth feature map 545 is obtained from the first convolution layer 541 and the pooling layer 522 in the third neural network layer 540, and the sixth feature map 450 is obtained from the pooling layer 543 and the second convolution layer 524 in the third neural network layer 440.

[0098] The third neural network layer 540 may be the same as the first neural network layer 505. When N is 1 and M is 2, the fifth feature map 545 may be a feature map of the original resolution of the input image, a resolution obtained by reducing the original resolution of the input image by 1 / 2, and a resolution obtained by reducing the original resolution of the input image by 1 / 4, and the sixth feature map 550 may be a feature map of a resolution obtained by reducing the original resolution of the input image by 1 / 2, a resolution obtained by reducing the original resolution of the input image by 1 / 4, and a resolution obtained by reducing the original resolution of the input image by 1 / 8.

[0099] Because the effective receptive field is very large in the case of a feature map with a resolution obtained by reducing the original resolution of the input image by 1 / 8 compared to a feature map with a relatively large resolution, a large motion vector can be effectively estimated in the case of a feature map with a resolution obtained by reducing the original resolution of the input image by 1 / 8. Since only feature maps of the same resolution are obtained in a general neural network structure, the effective receptive field is limited, and in Figure 5 In the motion vector encoder of , since pooling is performed before the convolution operation, the effective receptive field is expanded for the layers that output feature maps with reduced resolution.

[0100] The number of channels of the fifth feature map 545 and the number of channels of the sixth feature map 550 need not be the same as each other.

[0101] Motion vector feature data of the current image at multiple resolutions are obtained using the fifth feature map 545 and the sixth feature map 550 .

[0102] When the step size of the first convolution layer 501 is N1, the step size of the second convolution layer 504 is N2, the reduction ratio of the pooling layer 502 is M1, and the reduction ratio of the pooling layer 503 is M2, the neural network structure of the motion vector encoder can be determined under the condition of N1*M1=N2*M2.

[0103] When the step size of the first convolution layer 521 is N3, the step size of the second convolution layer 524 is N4, the reduction ratio of the pooling layer 522 is M3, and the reduction ratio of the pooling layer 523 is M4, the neural network structure of the motion vector encoder can be determined under the condition of N3*M3=N4*M4.

[0104] When the step size of the first convolution layer 541 is N5, the step size of the second convolution layer 544 is N6, the reduction ratio of the pooling layer 542 is M5, and the reduction ratio of the pooling layer 543 is M6, the neural network structure of the motion vector encoder can be determined under the condition of N5*M5=N6*M6. The motion vector encoder includes at least one layer of reduced feature maps.

[0105] Because the output of each layer from the motion vector encoder is a mixture of feature maps of multiple resolutions, the final output data from the motion vector encoder is the motion vector feature data of the current image at multiple resolutions. The final output motion vector feature data of the current image at multiple resolutions is compressed to the size of the feature map by the motion vector encoder, thereby reducing the bit rate.

[0106] Figure 6 is a block diagram of a neural network structure of a motion vector decoder according to an embodiment of the present disclosure.

[0107] refer to Figure 6 , the motion vector decoder 150 may include a plurality of neural network layers, namely, a first neural network layer 605, a second neural network layer 620, and a third neural network layer 640. In detail, the motion vector feature data 600 of the current image 10 of a plurality of resolutions are input to each of the first upper convolution layer 601 and the second upper convolution layer 602 in the first neural network layer 605 in the motion vector decoder 150. A first feature map 610 is obtained from the first upper convolution layer 601 in the first neural network layer 605, and a second feature map 615 is obtained from the second upper convolution layer 602 and the multiplication layer 603 in the first neural network layer 605.

[0108] The first upper convolution layer 601 and the second upper convolution layer 602 are convolution layers that perform upsampling and convolution. The upper convolution layer may also be referred to as a transposed convolution layer. The upsampling rate is a size corresponding to the reduction ratio of the motion vector encoder 110. The upper convolution may be, for example, an operation in which deconvolution, convolution operation, and pixel shuffling are performed together. The multiplication layer 603 multiplies the input by a size corresponding to the reduction ratio of the motion vector encoder 110.

[0109] The first feature map 610 and the second feature map 615 are concatenated, and then the entire concatenated feature map data (the first feature map 610 and the second feature map 615) are input to each of the first upper convolution layer 621 and the second upper convolution layer 622 in the second neural network layer 620. The third feature map 625 is obtained from the first upper convolution layer 621 in the second neural network layer 620, and the fourth feature map 630 is obtained from the second upper convolution layer 622 and the multiplication layer 603 in the second neural network layer 620.

[0110] The third feature map 625 and the fourth feature map 630 are concatenated, and then the entire concatenated feature map data (the third feature map 625 and the fourth feature map 630) are input to each of the first upper convolution layer 641 and the second upper convolution layer 642 in the third neural network layer 640. The fifth feature map 645 is obtained from the first upper convolution layer 641 in the third neural network layer 640, and the sixth feature map 650 is obtained from the second upper convolution layer 642 and the multiplication layer 643 in the third neural network layer 640.

[0111] The current motion vector is obtained using the fifth feature map 645 and the sixth feature map 650 .

[0112] When the reduction ratio in the motion vector encoder is N, the neural network structure of the motion vector decoder can be determined under the condition that the step size of the first upper convolution layer 601 and the second upper convolution layer 602 is N and the multiplication layer 603 is multiplied by N.

[0113] When the reduction ratio in the motion vector encoder is N, the neural network structure of the motion vector decoder can be determined under the condition that the step size of the first upper convolution layer 621 and the second upper convolution layer 622 is N and the multiplication layer 623 is multiplied by N.

[0114] When the reduction ratio in the motion vector encoder is N, the neural network structure of the motion vector decoder can be determined under the condition that the step size of the first upper convolution layer 641 and the second upper convolution layer 642 is N and the multiplication layer 643 is multiplied by N.

[0115] Figure 6 The neural network structure of the motion vector decoder corresponds to the above reference Figure 4 or Figure 5The neural network structure of the motion vector encoder described in the present invention is used to receive the small-size feature data output by the motion vector encoder and restore the compressed data to output the current motion vector. Since the output current motion vector reflects the result of motion estimation at multiple resolutions, the image can be accurately restored even when the size of the original motion vector is large and when the size of the original motion vector is small.

[0116] Figure 7 is a block diagram of a neural network structure of a motion vector decoder according to an embodiment of the present disclosure.

[0117] refer to Figure 7 , the motion vector decoder 150 may include a plurality of neural network layers, namely, a first neural network layer 710, a second neural network layer 730, and a third neural network layer 750. In detail, the motion vector feature data 700 of the current image 10 of a plurality of resolutions are input to each of the first convolution layer 701 and the second upsampling layer 703 in the first neural network layer 710 in the motion vector decoder 150. A first feature map 715 is obtained from the first convolution layer 701 and the first upsampling layer 702 in the first neural network layer 710, and a second feature map 615 is obtained from the second upsampling layer 703, the second convolution layer 704, and the multiplication layer 705 in the first neural network layer 710.

[0118] The first upsampling layer 702 and the second upsampling layer 703 are layers that perform upsampling. The upsampling rate is a size corresponding to a reduction ratio of the motion vector encoder 110.

[0119] The multiplication layer 705 multiplies the input by a size corresponding to the downscaling ratio of the motion vector encoder 110 .

[0120] The first feature map 715 and the second feature map 720 are concatenated, and then the entire concatenated feature map data (the first feature map 715 and the second feature map 720) are input to each of the first convolution layer 721 and the second upsampling layer 723 in the second neural network layer 730. The third feature map 735 is obtained from the first convolution layer 721 and the first upsampling layer 722 in the second neural network layer 730, and the fourth feature map 740 is obtained from the second upsampling layer 723, the second convolution layer 724, and the multiplication layer 725 in the second neural network layer 730.

[0121] The third feature map 735 and the fourth feature map 740 are concatenated, and then the entire concatenated feature map data (the third feature map 735 and the fourth feature map 740) are input to each of the first convolution layer 741 and the second upsampling layer 743 in the third neural network layer 750. The fifth feature map 755 is obtained from the first convolution layer 741 and the first upsampling layer 742 in the third neural network layer 750, and the sixth feature map 760 is obtained from the second upsampling layer 743, the second convolution layer 744, and the multiplication layer 745 in the third neural network layer 750.

[0122] The current motion vector is obtained using the fifth feature map 755 and the sixth feature map 760 .

[0123] When the step size of the first convolution layer 701 is N1, the step size of the second convolution layer 704 is N2, the upsampling rate of the first upsampling layer 702 is M1, and the upsampling rate of the second upsampling layer 703 is M2, the neural network structure of the motion vector decoder can be determined under the condition of N1*M1=N2*M2.

[0124] When the step size of the first convolution layer 721 is N3, the step size of the second convolution layer 724 is N4, the upsampling rate of the first upsampling layer 722 is M3, and the upsampling rate of the second upsampling layer 723 is M4, the neural network structure of the motion vector decoder can be determined under the condition of N3*M3=N4*M4.

[0125] When the step size of the first convolution layer 741 is N5, the step size of the second convolution layer 744 is N6, the upsampling rate of the first upsampling layer 742 is M5, and the upsampling rate of the second upsampling layer 743 is M6, the neural network structure of the motion vector decoder can be determined under the condition of N5*M5=N6*M6.

[0126] Figure 7 The neural network structure of the motion vector decoder corresponds to the above reference Figure 4 or Figure 5 The neural network structure of the motion vector encoder described in the present invention is used to receive the small-size feature data output by the motion vector encoder and restore the compressed data to output the current motion vector. Since the output current motion vector reflects the result of motion estimation at multiple resolutions, the image can be accurately restored even when the size of the original motion vector is large and when the size of the original motion vector is small.

[0127] Figure 8 is a block diagram of a neural network structure of a motion vector encoder according to an embodiment of the present disclosure.

[0128] refer to Figure 8, the motion vector encoder 110 may include a plurality of neural network layers, namely, a first neural network layer 805, a second neural network layer 825, and a third neural network layer 845. In detail, all pixel data of an input image 800 obtained by concatenating a current image with a previously reconstructed image is input to each of a first dilated convolution layer 801, a second dilated convolution layer 802, and a third dilated convolution layer 803 in the first neural network layer 805 of the motion vector encoder 110. A first feature map 810 is obtained from the first dilated convolution layer 801 in the first neural network layer 805, a second feature map 815 is obtained from the second dilated convolution layer 802 in the first neural network layer 805, and a third feature map 820 is obtained from the third dilated convolution layer 803 in the first neural network layer 805.

[0129] The first dilated convolution layer 801 may be a layer on which a dilated convolution operation with a dilation rate of 1 is performed, the second dilated convolution layer 802 may be a layer on which a dilated convolution operation with a dilation rate of d2 is performed, and the third dilated convolution layer 803 may be a layer on which a dilated convolution operation with a dilation rate of d3 is performed.

[0130] The first dilated convolution layer 801, the second dilated convolution layer 802, and the third dilated convolution layer 803 are layers in which convolution operations with different dilation rates and the same step lengths are performed.

[0131] Will refer to it later Fig.11 A detailed description of "Dilated Convolution".

[0132] Because the dilated convolution operation can reduce the size of the input data by the dilation rate, the first feature map 810 obtained from the first dilated convolution layer 801 with the dilation rate 1 can be a feature map of the original resolution of the input image, the second feature map 815 obtained from the second dilated convolution layer 802 with the dilation rate d2 can be a feature map of the resolution reduced from the original resolution of the input image based on d2, and the third feature map 820 obtained from the third dilated convolution layer 803 with the dilation rate d3 can be a feature map of the resolution reduced from the original resolution of the input image based on d3. The value of d2 can be 2, and the value of d3 can be 3.

[0133] The first feature map 810, the second feature map 815, and the third feature map 820 are concatenated, and then the entire concatenated feature map data (the first feature map 810, the second feature map 815, and the third feature map 820) are input to each of the first dilated convolution layer 821, the second dilated convolution layer 822, and the third dilated convolution layer 823 in the second neural network layer 825. The fourth feature map 830 is obtained from the first dilated convolution layer 821 in the second neural network layer 825, the fifth feature map 835 is obtained from the second dilated convolution layer 822 in the second neural network layer 825, and the sixth feature map 840 is obtained from the third dilated convolution layer 823 in the second neural network layer 845.

[0134] The first dilated convolution layer 821 may be the same as the first dilated convolution layer 801, the second dilated convolution layer 822 may be the same as the second dilated convolution layer 802, and the third dilated convolution layer 823 may be the same as the third dilated convolution layer 803. Therefore, the fourth feature map 830 may be a feature map of the original resolution of the input image, a resolution reduced from the original resolution of the input image based on d2, and a resolution reduced from the original resolution of the input image based on d3, the fifth feature map 835 may be a feature map of a resolution reduced from the original resolution of the input image based on d2, a resolution reduced twice from the original resolution of the input image based on d2, and a resolution reduced from the original resolution of the input image based on d3 and d2, and the sixth feature map 840 may be a feature map for a resolution reduced from the original resolution of the input image based on d3, a resolution reduced from the original resolution of the input image based on d2 and d3, and a resolution reduced twice from the original resolution of the input image based on d3.

[0135] The first dilated convolution layer 821, the second dilated convolution layer 822, and the third dilated convolution layer 823 are layers in which convolution operations with different dilation rates and the same step lengths are performed.

[0136] The fourth feature map 830, the fifth feature map 835, and the sixth feature map 840 are concatenated, and then the entire concatenated feature map data (the fourth feature map 830, the fifth feature map 835, and the sixth feature map 840) are input to each of the first dilated convolution layer 841, the second dilated convolution layer 842, and the third dilated convolution layer 843 in the third neural network layer 845. The seventh feature map 850 is obtained from the first dilated convolution layer 841 in the third neural network layer 845, the eighth feature map 855 is obtained from the second dilated convolution layer 842 in the third neural network layer 845, and the ninth feature map 860 is obtained from the third dilated convolution layer 843 in the third neural network layer 845.

[0137] The first dilated convolution layer 841 may be the same as the first dilated convolution layer 801, the second dilated convolution layer 842 may be the same as the second dilated convolution layer 802, and the third dilated convolution layer 843 may be the same as the third dilated convolution layer 803. Therefore, the seventh feature map 850 may be a feature map of the original resolution of the input image, a resolution reduced from the original resolution of the input image based on d2, a resolution reduced from the original resolution of the input image based on d3, a resolution reduced twice from the original resolution of the input image based on d2, a resolution reduced from the original resolution of the input image based on d3 and d2, and a resolution reduced twice from the original resolution of the input image based on d3, and the eighth feature map 855 may be a feature map of the original resolution of the input image based on d2, a resolution reduced twice from the original resolution of the input image based on d2, a resolution reduced from the original resolution of the input image based on d3 and d2, and a resolution reduced three times from the original resolution of the input image based on d2. The ninth feature map 860 may be a feature map of a resolution reduced from the original resolution of the input image based on d3, a resolution reduced twice from the original resolution of the input image based on d2 and d3, a resolution reduced twice from the original resolution of the input image based on d3, a resolution reduced twice from the original resolution of the input image based on d2 and d3, a resolution reduced twice from the original resolution of the input image based on d3, a resolution reduced twice from the original resolution of the input image based on d2 and once based on d3, a resolution reduced twice from the original resolution of the input image based on d3 and once based on d2, and a resolution reduced three times from the original resolution of the input image based on d3.

[0138] The first dilated convolution layer 841, the second dilated convolution layer 842, and the third dilated convolution layer 843 are layers in which convolution operations with different dilation rates and the same step lengths are performed.

[0139] Because the effective receptive field is very large in the case of a feature map with a resolution obtained by reducing the original resolution of the input image three times based on d3, compared with a feature map with a relatively large scale resolution, a large motion vector can be effectively estimated in the case of a feature map with a resolution obtained by reducing the original resolution of the input image three times based on d3. Because only feature maps of the same resolution are obtained in a general neural network structure, the effective receptive field is limited, and in Figure 8 In the motion vector encoder of , due to the dilated convolution operation performed, the effective receptive field is expanded for the layers that output feature maps with reduced resolution.

[0140] The motion vector feature data of the current image at multiple resolutions are obtained using the seventh feature map 850 , the eighth feature map 855 , and the ninth feature map 860 .

[0141] The motion vector encoder comprises at least one layer of reduced feature maps.

[0142] Because the output of each layer from the motion vector encoder is a mixture of feature maps of multiple resolutions, the final output data from the motion vector encoder is the motion vector feature data of the current image at multiple resolutions. The final output motion vector feature data of the current image at multiple resolutions is compressed to the size of the feature map by the motion vector encoder, thereby reducing the bit rate.

[0143] Fig. 9 is a block diagram of a neural network structure of a motion vector decoder according to an embodiment of the present disclosure.

[0144] refer to Fig. 9 , the motion vector decoder 150 may include a plurality of neural network layers, namely, a first neural network layer 905, a second neural network layer 925, and a third neural network layer 945. In detail, the motion vector feature data 900 of the current image of the plurality of resolutions are input to each of the first inverse operation layer 901, the second inverse operation layer 902, and the third inverse operation layer 903 in the first neural network layer 905 in the motion vector decoder 150. For the first dilated convolution in the first neural network layer 905, a first feature map 910 is obtained from the first inverse operation layer 901, for the second dilated convolution in the first neural network layer 905, a second feature map 915 is obtained from the second inverse operation layer 902, and for the third dilated convolution in the first neural network layer 905, a third feature map 915 is obtained from the third inverse operation layer 903.

[0145] The first inverse operation layer 901 performs the corresponding Figure 8 The inverse operation of the first dilated convolutional layer 801 is performed by the second inverse operation layer 902 corresponding to Figure 8 The third inverse operation layer 903 performs the inverse operation of the second dilated convolution layer 802 corresponding to Figure 8 The inverse operation of the third expanded convolutional layer 803.

[0146] The first inverse operation layer 901, the second inverse operation layer 902, and the third inverse operation layer 903 perform inverse operations of dilated convolutions with different dilation rates and the same step length.

[0147] The first feature map 910, the second feature map 915, and the third feature map 920 are concatenated, and then the entire concatenated feature map data (the first feature map 910, the second feature map 915, and the third feature map 920) are input to each of the first inverse operation layer 921, the second inverse operation layer 922, and the third inverse operation layer 923 in the second neural network layer 925. For the first dilated convolution in the second neural network layer 925, the fourth feature map 930 is obtained from the first inverse operation layer 921, for the second dilated convolution in the second neural network layer 925, the fifth feature map 935 is obtained from the second inverse operation layer 922, and for the third dilated convolution in the second neural network layer 925, the sixth feature map 930 is obtained from the third inverse operation layer 923.

[0148] The first inverse operation layer 921 , the second inverse operation layer 922 , and the third inverse operation layer 923 perform inverse operations of dilated convolutions having different dilation rates and the same step length.

[0149] The fourth feature map 930, the fifth feature map 935, and the sixth feature map 940 are concatenated, and then the entire concatenated feature map data (the fourth feature map 930, the fifth feature map 935, and the sixth feature map 940) are input to each of the first inverse operation layer 941, the second inverse operation layer 942, and the third inverse operation layer 943 in the third neural network layer 945. For the first dilated convolution in the third neural network layer 945, the seventh feature map 950 is obtained from the first inverse operation layer 941, for the second dilated convolution in the third neural network layer 945, the eighth feature map 955 is obtained from the second inverse operation layer 942, and for the third dilated convolution in the third neural network layer 945, the ninth feature map 960 is obtained from the third inverse operation layer 943.

[0150] The first inverse operation layer 941, the second inverse operation layer 942, and the third inverse operation layer 943 perform inverse operations of dilated convolutions with different dilation rates and the same step length.

[0151] The current motion vector is obtained using the seventh feature map 950 , the eighth feature map 955 , and the ninth feature map 960 .

[0152] Fig. 9 The neural network structure of the motion vector decoder corresponds to the above reference Figure 8 The neural network structure of the motion vector encoder described in the present invention is used to receive the small-size feature data output by the motion vector encoder and restore the compressed data to output the current motion vector. Since the output current motion vector reflects the result of motion estimation at multiple resolutions, the image can be accurately restored even when the size of the original motion vector is large and when the size of the original motion vector is small.

[0153] Fig.10is a view used to explain the method of performing pooling.

[0154] refer to Fig.10 , maximum pooling may be applied to the first feature map 1000 having a size of 6×6 to halve the dimension of the first feature map 1000, resulting in the generation of a second feature map 1010 having a size of 3×3. During this maximum pooling operation, a value of 9, which is the value of the maximum pixel among the four 2×2 pixels 1001 in the upper left portion of the first feature map 1000, becomes the pixel value of the upper left pixel 1011 within the second feature map 1010. A value of 7, which is the maximum pixel value among the four 2×2 pixels 1002 in the left portion of the first feature map 1000, becomes the pixel value of the left pixel 1012 within the second feature map 1010.

[0155] Average pooling that reduces the dimension to 1 / 2 can be performed on the third feature map 1020 having a size of 6×6, and thus a fourth feature map 1030 having a size of 3×3 can be obtained. In average pooling, the pixel value of the reduced feature map can be obtained by calculating the average value of the pixel or rounding up the average value. In this case, the value 5, which is a rounded value of 4.5 obtained by calculating the average value of the four 2×2 pixels 1021 in the upper left portion of the third feature map 1020, becomes the pixel value of the upper left pixel 1031 within the fourth feature map 1030. The value 4, which is the average value of the four 2×2 pixels 1002 in the left portion of the third feature map 1020, becomes the pixel value of the left pixel 1032 within the fourth feature map 1030.

[0156] In addition, the pixel value of the minimum value among the pixels can be used as the minimum pooling to reduce the pixel value of the feature map. The above pooling method is presented as an example, and the embodiments of the present disclosure are not limited thereto.

[0157] Pooling can be applied to create a reduced-scale feature map or frame from the current input feature map or the current input frame, thereby effectively reducing the resolution of the current input feature map or frame. Therefore, the effective receptive field can be expanded, thereby enabling large motions to be estimated at low resolution with a limited search range.

[0158] Fig.11 is a view for explaining an example of a method of performing dilated convolution.

[0159] refer to Fig.11 , in the normal convolution, the convolution operation is performed by applying the 3×3 convolution filter 1110 to the feature map 1120 to be convolved without change. This corresponds to the case where the dilation rate is 1.

[0160] However, in the case of a dilated convolution using a 3×3 convolution filter 1130 and a dilation rate of 2, values ​​with zero weight are filled one by one between each filter weight of the 3×3 convolution filter 1130, and thus the convolution operation is performed in the same manner as when a 5×5 convolution filter is applied to the feature map 1140 to be convolved.

[0161] In the case of dilated convolution using a 3×3 convolution filter and a dilation rate of 3, two values ​​with zero weight are padded between each filter weight of the 3×3 convolution filter 1130, and thus the convolution operation can be performed in the same manner as when a 7×7 convolution filter is applied to the feature map to be convolved.

[0162] When a dilated convolution with a large dilation rate is performed, a large receptive field corresponding to a small scale can be obtained, and when a dilated convolution with a small dilation rate is performed, a small receptive field corresponding to a relatively large scale can be obtained.

[0163] In the dilated convolution, the current input feature map or the current input frame can be used to generate a feature map or frame of a small scale (ie, small resolution), and feature maps or frames of various scales can be generated by using dilated convolutions with various dilation rates. Therefore, the effective receptive field can be expanded to estimate motion vectors of large sizes.

[0164] Fig.12 is a flowchart of an image encoding method according to an embodiment of the present disclosure.

[0165] In operation S1210, the image encoding device 1300 obtains motion vector feature data of a current image of multiple resolutions by applying the current image and a previous reconstructed image to a motion vector encoder.

[0166] According to an embodiment of the present disclosure, the plurality of resolutions may include an original resolution of the current image and a resolution reduced from the original resolution of the current image.

[0167] According to an embodiment of the present disclosure, a motion vector encoder may include multiple first encoding neural network layers, each of which may include a first encoding layer and a second encoding layer operating in parallel, the first encoding layer may include a first convolutional layer, and the second encoding layer may include a pooling layer and a second convolutional layer. The first encoding layer and the second encoding layer may operate simultaneously to process two separate data streams in parallel.

[0168] According to an embodiment of the present disclosure, the motion vector encoder may include multiple second encoding neural network layers, each of the second encoding neural network layers may include a third encoding layer and a fourth encoding layer operating in parallel, the third encoding layer may include a third convolutional layer and a first pooling layer, and the fourth encoding layer may include a second pooling layer and a fourth convolutional layer.

[0169] According to an embodiment of the present disclosure, the motion vector encoder may include a plurality of third encoding neural network layers, and each of the third encoding neural network layers may include a first dilated convolution layer, a second dilated convolution layer, and a third dilated convolution layer operating in parallel. The first dilated convolution layer, the second dilated convolution layer, and the third dilated convolution layer may operate simultaneously to process separate data streams in parallel.

[0170] According to an embodiment of the present disclosure, the current image may be feature data output by applying pixel data of the current image to a convolutional neural network, and the previous reconstructed image may be feature data output by applying pixel data of the previous reconstructed image to a convolutional neural network. In other words, the current image and the previous reconstructed image may be feature map domains.

[0171] According to an embodiment of the present disclosure, the current image may be pixel data of the current image, and the previous reconstructed image may be pixel data of the previous reconstructed image. In other words, the current image and the previous reconstructed image may be pixel domains.

[0172] In operation S1230, the image encoding device 1300 obtains a current motion vector by applying motion vector characteristic data to a motion vector decoder.

[0173] According to an embodiment of the present disclosure, a motion vector decoder may include multiple first decoding neural network layers, each of the first decoding neural network layers may include a first decoding layer and a second decoding layer operating in parallel, the first decoding layer may include a first upper convolution layer, the second decoding layer may include a second upper convolution layer and a first multiplication layer for multiplying the input by a magnification ratio corresponding to the reduction ratio.

[0174] According to an embodiment of the present disclosure, a motion vector decoder may include multiple second decoding neural network layers, each of the second decoding neural network layers may include a third decoding layer and a fourth decoding layer operating in parallel, the third decoding layer may include a first convolutional layer and a first upsampling layer, and the fourth decoding layer may include a second convolutional layer and a first multiplication layer for multiplying the input by a magnification ratio corresponding to the reduction ratio.

[0175] According to an embodiment of the present disclosure, the motion vector decoder may include a plurality of third decoding neural network layers, and each of the third decoding neural network layers may include a first inverse operation layer for dilated convolution, a second inverse operation layer for dilated convolution, and a third inverse operation layer for dilated convolution operating in parallel. The first inverse operation layer, the second inverse operation layer, and the third inverse operation layer may operate simultaneously to process separate data streams in parallel.

[0176] In operation S1250, the image encoding device 1300 obtains a current predicted image by using a current motion vector and a previous reconstructed image.

[0177] In operation S1270, the image encoding device 1300 generates a bitstream including motion vector feature data.

[0178] According to an embodiment of the present disclosure, the image encoding device 1300 may obtain a current residual image corresponding to a difference between a current predicted image and a current image, and may obtain feature data of the current residual image by applying the current residual image to a residual encoder.

[0179] According to an embodiment of the present disclosure, the bitstream may further include feature data of the current residual image.

[0180] Fig.13 is a block diagram showing a structure of an image encoding device according to an embodiment of the present disclosure.

[0181] refer to Fig.13 , the image encoding device 1300 according to an embodiment of the present disclosure may include a prediction encoder 1310 , a generator 1320 , an obtainer 1330 , and a prediction decoder 1340 .

[0182] The prediction encoder 1310, the generator 1320, the obtainer 1330, and the prediction decoder 1340 may be implemented as a processor. The prediction encoder 1310, the generator 1320, the obtainer 1330, and the prediction decoder 1340 may operate according to instructions stored in a memory.

[0183] Despite Fig.13 13, the prediction encoder 1310, the generator 1320, the obtainer 1330, and the prediction decoder 1340 are shown separately, but the prediction encoder 1310, the generator 1320, the obtainer 1330, and the prediction decoder 1340 may be implemented as one processor. In this case, the prediction encoder 1310, the generator 1320, the obtainer 1330, and the prediction decoder 1340 may be implemented as a dedicated processor, or may be implemented by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). The dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processing unit for using an external memory.

[0184] The prediction encoder 1310, the generator 1320, the acquirer 1330, and the prediction decoder 1340 may be implemented as a plurality of processors. In this case, the prediction encoder 1310, the generator 1320, the acquirer 1330, and the prediction decoder 1340 may be implemented as a combination of dedicated processors, or may be implemented by a combination of software and a plurality of general-purpose processors (such as an AP, a CPU, or a GPU). The processor may include a dedicated artificial intelligence (AI) processor. As another example, the dedicated AI processor may be configured as a chip separate from the processor.

[0185] The prediction encoder 1310 includes a motion vector encoder 1315. The prediction encoder 1310 obtains motion vector feature data of a current image of multiple resolutions by applying the current image and a previous reconstructed image to the motion vector encoder 1315.

[0186] The generator 1320 generates a bitstream including the motion vector feature data obtained by the prediction encoder 1310 .

[0187] The obtainer 1330 obtains motion vector feature data from the bitstream generated by the generator 1320 .

[0188] The prediction encoder 1310 may directly transmit the motion vector feature data of the current image of multiple resolutions to the prediction decoder 1340 without using the generator 1320 and the obtainer 1330 .

[0189] The prediction decoder 1340 includes a motion compensator 1341 and a motion vector decoder 1342. The prediction decoder 1340 obtains a current motion vector by applying motion vector feature data to the motion vector decoder 1342. The prediction decoder 1340 sends the current motion vector to the motion compensator 1341. The prediction decoder 1340 obtains a previous reconstructed image and sends it to the motion compensator 1341.

[0190] The motion compensator 1341 of the prediction decoder 1340 obtains a current prediction image by using a current motion vector and a previous reconstructed image.

[0191] The generator 1320 entropy encodes the motion vector feature data through cost calculation such as sum of transform differences (SATD) or rate distortion optimization (RDO) calculation based on the prediction result of the prediction decoder 1340 to generate a bitstream and transmit it to the image decoding device 1500 .

[0192] The motion vector encoder 1315 and the motion vector decoder 1342 may be implemented as a neural network including one or more layers (eg, convolutional layers). In detail, the motion vector encoder 1315 may include Figure 4 , Figure 5 and Figure 8 A neural network structure in a neural network structure. The motion vector decoder 1342 may include Figure 6 , Figure 7 and Fig. 9 A neural network structure in a neural network structure.

[0193] The motion vector encoder 1315 and the motion vector decoder 1342 may be stored in a memory. The motion vector encoder 1315 and the motion vector decoder 1342 may be implemented as at least one dedicated processor for AI.

[0194] According to an embodiment of the present disclosure, the prediction encoder 1310 may further include a residual encoder. The prediction encoder 1310 may obtain a current prediction image from the prediction decoder 1340. The prediction encoder 1310 may obtain a current residual image corresponding to a difference between the current prediction image and the current image, and may obtain feature data of the current residual image by applying the current residual image to the residual encoder.

[0195] According to an embodiment of the present disclosure, the generator 1320 may generate a bitstream further including feature data of the current residual image obtained by the prediction encoder 1310 .

[0196] According to an embodiment of the present disclosure, the obtainer 1330 may also obtain feature data of the current residual image from the bitstream generated by the generator 1320 .

[0197] According to an embodiment of the present disclosure, the prediction decoder 1340 may further include a residual decoder. The prediction decoder 1340 may obtain a current residual image by applying feature data of the current residual image to the residual decoder. The prediction decoder 1340 may obtain a current reconstructed image by using the current prediction image and the current residual image.

[0198] The residual encoder and residual decoder may be implemented as a neural network including one or more layers (eg, convolutional layers).

[0199] The residual encoder and the residual decoder may be stored in a memory. The residual encoder and the residual decoder may be implemented as at least one dedicated processor for AI.

[0200] Fig.14 is a flowchart of an image decoding method according to an embodiment of the present disclosure.

[0201] In operation S1410, the image decoding device 1500 obtains motion vector feature data of a current image of multiple resolutions from a bitstream.

[0202] According to an embodiment of the present disclosure, the plurality of resolutions may include an original resolution of the current image and a resolution reduced from the original resolution of the current image.

[0203] According to an embodiment of the present disclosure, the current image may be feature data output by applying pixel data of the current image to a convolutional neural network, and the previous reconstructed image may be feature data output by applying pixel data of the previous reconstructed image to a convolutional neural network. In other words, the current image and the previous reconstructed image may be feature map domains.

[0204] According to an embodiment of the present disclosure, the current image may be pixel data of the current image, and the previous reconstructed image may be pixel data of the previous reconstructed image. In other words, the current image and the previous reconstructed image may be pixel domains.

[0205] In operation S1430, the image decoding device 1500 obtains a current motion vector by applying motion vector characteristic data to a motion vector decoder.

[0206] According to an embodiment of the present disclosure, a motion vector decoder may include a plurality of first decoding neural network layers, each of the first decoding neural network layers may include a first decoding layer and a second decoding layer operating in parallel, the first decoding layer may include a first upper convolution layer, the second decoding layer may include a second upper convolution layer and a first operation layer for multiplying an input by an amplification ratio corresponding to a reduction ratio. The first decoding layer and the second decoding layer may operate simultaneously to process separate data streams in parallel.

[0207] According to an embodiment of the present disclosure, a motion vector decoder may include a plurality of second decoding neural network layers, each of the second decoding neural network layers may include a third decoding layer and a fourth decoding layer operating in parallel, the third decoding layer may include a first convolution layer and a first upsampling layer, and the fourth decoding layer may include a second convolution layer and a first operation layer for multiplying an input by an amplification ratio corresponding to a reduction ratio. The third decoding layer and the fourth decoding layer may operate simultaneously to process separate data streams in parallel.

[0208] According to an embodiment of the present disclosure, the motion vector decoder may include a plurality of third decoding neural network layers, and each of the third decoding neural network layers may include a first inverse operation layer for dilated convolution, a second inverse operation layer for dilated convolution, and a third inverse operation layer for dilated convolution operating in parallel. The first inverse operation layer, the second inverse operation layer, and the third inverse operation layer may operate simultaneously to process separate data streams in parallel.

[0209] In operation S1450, the image decoding device 1500 obtains a current predicted image by using a current motion vector and a previous reconstructed image.

[0210] According to an embodiment of the present disclosure, the image decoding device 1500 can obtain feature data of a current residual image from a bit stream, obtain the current residual image by applying the feature data of the current residual image to a residual decoder, and obtain a current reconstructed image by using a current predicted image and the current residual image.

[0211] Fig.15 is a flowchart of an image decoding method according to an embodiment of the present disclosure.

[0212] refer to Fig.15 , the image decoding device 1500 according to an embodiment may include an obtainer 1510 and a prediction decoder 1520 .

[0213] The obtainer 1510 and the prediction decoder 1520 may be implemented as a processor. The obtainer 1510 and the prediction decoder 1520 may operate according to instructions stored in a memory.

[0214] Despite Fig.15 15 and 15, respectively, but the obtainer 1510 and the prediction decoder 1520 may be implemented by one processor. In this case, the obtainer 1510 and the prediction decoder 1520 may be implemented as a dedicated processor, or may be implemented by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). The dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processing unit for using an external memory.

[0215] The obtainer 1510 and the prediction decoder 1520 may be implemented as a plurality of processors. In this case, the obtainer 1510 and the prediction decoder 1520 may be implemented as a combination of dedicated processors, or may be implemented by a combination of software and a plurality of general-purpose processors (e.g., AP, CPU, or GPU). The processor may include a dedicated AI processor. As another example, the dedicated AI processor may be configured as a chip separate from the processor.

[0216] The obtainer 1510 receives a bitstream from an image encoding device.

[0217] The obtainer 1510 obtains motion vector feature data from a bitstream.

[0218] The prediction decoder 1520 includes a motion compensator 1522 and a motion vector decoder 1521. The prediction decoder 1520 obtains a current motion vector by applying motion vector feature data to the motion vector decoder 1521. The prediction decoder 1520 sends the current motion vector to the motion compensator 1522. The prediction decoder 1520 obtains a previous reconstructed image and sends it to the motion compensator 1522.

[0219] The motion compensator 1522 of the prediction decoder 1520 obtains a current prediction image by using a current motion vector and a previous reconstructed image.

[0220] The motion vector decoder 1521 may be implemented as a neural network including one or more layers (eg, convolutional layers). In detail, the motion vector decoder 1521 may include Figure 6 , Figure 7 and Fig. 9 A neural network structure in a neural network structure.

[0221] The motion vector decoder 1521 may be stored in a memory. The motion vector decoder 1521 may be implemented as at least one dedicated processor for AI.

[0222] According to an embodiment of the present disclosure, the obtainer 1510 may also obtain feature data of the current residual image from the bitstream.

[0223] According to an embodiment of the present disclosure, the prediction decoder 1520 may further include a residual decoder. The prediction decoder 1520 may obtain a current residual image by applying feature data of the current residual image to the residual decoder. The prediction decoder 1340 may obtain a current reconstructed image by using the current prediction image and the current residual image.

[0224] The residual decoder can be implemented as a neural network including one or more layers (e.g., convolutional layers).

[0225] The residual decoder may be stored in a memory. The residual decoder may be implemented as at least one dedicated processor for AI.

[0226] Fig.16 is a diagram for describing a method of training a motion vector encoder, a motion vector decoder, a residual encoder, and a residual decoder according to an embodiment of the present disclosure.

[0227] exist Fig.16 In the example, the current training image 1620, the previously reconstructed training image 1610, and the current predicted training image x' i and the current reconstructed training image 1630 correspond to the aforementioned current image, the aforementioned previously reconstructed image, the aforementioned current predicted image and the aforementioned current reconstructed image, respectively.

[0228] When training the motion vector encoder 110, the motion vector decoder 150, the residual encoder 130 and the residual decoder 170, the similarity between the current reconstructed training image 1630 and the current training image 1620 and the bit rate of the bitstream to be generated by encoding the current training image 1620 need to be considered.

[0229] To this end, according to an embodiment of the present disclosure, the motion vector encoder 110, the motion vector decoder 150, the residual encoder 130 and the residual decoder 170 can be trained based on first loss information 1640 and second loss information 1650 corresponding to the size of the bit stream and third loss information 1660 corresponding to the similarity between the current training image 1620 and the current reconstructed training image 1630.

[0230] refer to Fig.16 , the current training image 1620 and the previously reconstructed training image 1610 may be input to the motion vector encoder 110. The motion vector encoder 110 may output motion vector feature data w of the current image of multiple resolutions by processing the current training image 1620 and the previously reconstructed training image 1610. i .

[0231] Motion vector feature data w of the current image at multiple resolutions i may be input to the motion vector decoder 150, which may process the motion vector feature data w of the current image at multiple resolutions. i , to output the current motion vector g i .

[0232] This can be done by using the previously reconstructed training image 1610 and the current motion vector g i Motion compensation 190 is performed to generate the current predicted training image x' i .

[0233] Residual training image r i The current prediction training image x' can be used i and the current training image 1620 , and can be input to the residual encoder 130 .

[0234] The residual encoder 130 can process the residual training image r i To output the feature data v of the residual image i .

[0235] The feature data v of the residual image i can be input to the residual decoder 170. The residual decoder 170 can output the reconstructed residual training image r' i .

[0236] The current reconstructed training image 1630 can be obtained by combining the previously reconstructed training image 1610 and the current motion vector g with the motion compensation 190. i The generated current prediction image x' i And the reconstructed residual image data r' i Combine to obtain.

[0237] The first loss information 1640 may be obtained from the motion vector feature data w of the current image at multiple resolutions. i The entropy or motion vector feature data w of the current image with multiple resolutions i The corresponding bit rate of the bit stream is calculated.

[0238] The second loss information 1650 can be obtained from the feature data v of the residual image i The entropy or feature data v of the residual image i The corresponding bit rate of the bit stream is calculated.

[0239] Since the first loss information 1640 and the second loss information 1650 are related to the efficiency of encoding the current training image 1620, the first loss information 1640 and the second loss information 1650 may be referred to as compression loss information.

[0240] According to the embodiments of the present disclosure, although Fig.16 The first loss information 1640 and the second loss information 1650 related to the bit rate of the bitstream are derived, but one piece of loss information corresponding to the bit rate of one bitstream generated by encoding the current training image 1620 may be derived.

[0241] The third loss information 1660 may correspond to a difference between the current training image 1620 and the current reconstructed training image 1630. The difference between the current training image 1620 and the current reconstructed training image 1630 may include at least one of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, or a video multi-method assessment fusion (VMAF) value between the current training image 1620 and the current reconstructed training image 1630.

[0242] Since the third loss information 1660 is related to the quality of the current reconstructed training image 1630, the third loss information 1660 may be referred to as quality loss information.

[0243] The motion vector encoder 310 , the motion vector decoder 150 , the residual encoder 130 , and the residual decoder 170 may be trained to reduce or minimize final loss information derived from at least one of the first loss information 1640 , the second loss information 1650 , or the third loss information 1660 .

[0244] In detail, the motion vector encoder 110, the motion vector decoder 150, the residual encoder 130, and the residual decoder 170 may be trained to reduce or minimize final loss information while changing preset parameter values.

[0245] According to an embodiment of the present disclosure, the final loss information may be calculated according to Equation 1 below.

[0246] [Equation 1]

[0247] Final loss information = a*first loss information + b*second loss information + c*third loss information

[0248] In Equation 1, a, b, and c represent weights applied to the first loss information 1640, the second loss information 1650, and the third loss information 1660, respectively.

[0249] According to Equation 1, it is found that the motion vector encoder 110, the motion vector decoder 150, the residual encoder 130 and the residual decoder 170 can be trained so that the current reconstructed training image 1630 is as similar as possible to the current training image 1620, and the size of the bit stream corresponding to the data output from the motion vector encoder 110 and the residual encoder 130 is minimized.

[0250] According to an embodiment of the present disclosure, an AI-based image encoding method may include: obtaining motion vector feature data of a current image of multiple resolutions by applying the current image and a previously reconstructed image to a motion vector encoder; obtaining a current motion vector by applying the motion vector feature data to a motion vector decoder; obtaining a current predicted image by using the current motion vector and a previously reconstructed image; and generating a bitstream including the motion vector feature data. The multiple resolutions may include an original resolution of the current image and a resolution reduced from the original resolution of the current image.

[0251] According to an embodiment of the present disclosure, the AI-based image encoding method may further include: obtaining a current residual image corresponding to the difference between a current predicted image and a current image; and obtaining feature data of the current residual image by applying the current residual image to a residual encoder, wherein the bitstream may further include the feature data of the current residual image.

[0252] According to an embodiment of the present disclosure, a motion vector decoder may include multiple first decoding neural network layers, each of the multiple first decoding neural network layers may include a parallel first decoding layer and a second decoding layer, the first decoding layer may include a first upper convolutional layer, the second decoding layer may include a second upper convolutional layer and a first multiplication layer that multiplies the input by an amplification ratio corresponding to the reduction ratio, and a motion vector encoder may include multiple first encoding neural network layers, each of the multiple first encoding neural network layers may include a parallel first encoding layer and a second encoding layer, the first encoding layer may include a first convolutional layer, and the second encoding layer may include a pooling layer and a second convolutional layer.

[0253] According to an embodiment of the present disclosure, a motion vector decoder may include multiple second decoding neural network layers, each of the multiple second decoding neural network layers may include parallel third and fourth decoding layers, the third decoding layer may include a first convolutional layer and a first upsampling layer, the fourth decoding layer may include a second upsampling layer, a second convolutional layer, and a first multiplication layer that multiplies the input by an amplification ratio corresponding to the reduction ratio, the motion vector encoder may include multiple second encoding neural network layers, each of the multiple second encoding neural network layers may include parallel third and fourth encoding layers, the third encoding layer may include a third convolutional layer and a first pooling layer, and the fourth encoding layer may include a second pooling layer and a fourth convolutional layer.

[0254] According to an embodiment of the present disclosure, a motion vector decoder may include multiple third decoding neural network layers, each of which may include a parallel first inverse operation layer for dilated convolution, a second inverse operation layer for dilated convolution, and a third inverse operation layer for dilated convolution, and a motion vector decoder may include multiple third encoding neural network layers, and each of which may include a parallel first dilated convolution layer, a second dilated convolution layer, and a third dilated convolution layer.

[0255] According to an embodiment of the present disclosure, the current image may be feature data output by applying pixel data of the current image to a convolutional neural network, and the previously reconstructed image may be feature data output by applying pixel data of the previously reconstructed image to a convolutional neural network.

[0256] According to an embodiment of the present disclosure, the current image may be pixel data of the current image, and the previous reconstructed image may be pixel data of the previous reconstructed image.

[0257] The AI-based image encoding method according to an embodiment of the present disclosure obtains motion vector feature data of a current image of multiple resolutions by using a motion vector encoder having a neural network structure including a neural network layer for maintaining the resolution of an input image and a neural network layer for reducing the resolution of an input image, and obtains the current motion vector by applying the motion vector feature data to a motion vector decoder, so that by applying an extended effective receptive field, not only is it effective to estimate motion vectors of small sizes, but it is also effective to estimate motions of large sizes. Because the motion vector feature data of the current image of multiple resolutions obtained from the motion vector encoder is a feature map of a small size output from the final layer, and is also a feature map of all scales, the image encoding method is effective in encoding feature data of multiple scales in terms of data compression, and is suitable for video encoding and decoding.

[0258] According to an embodiment of the present disclosure, an AI-based image decoding method may include: obtaining motion vector feature data of a current image of multiple resolutions from a bitstream; obtaining a current motion vector by applying the motion vector feature data to a motion vector decoder; and obtaining a current predicted image by using the current motion vector and a previously reconstructed image. The multiple resolutions may include an original resolution of the current image and a resolution reduced from the original resolution of the current image.

[0259] According to an embodiment of the present disclosure, the AI-based image decoding method may also include: obtaining feature data of a current residual image from a bit stream; obtaining the current residual image by applying the feature data of the current residual image to a residual decoder; and obtaining a current reconstructed image by using a current predicted image and the current residual image.

[0260] According to an embodiment of the present disclosure, a motion vector decoder may include multiple first decoding neural network layers, each of the first decoding neural network layers may include a parallel first decoding layer and a second decoding layer, the first decoding layer may include a first upper convolution layer, the second decoding layer may include a second upper convolution layer and a first operation layer for multiplying the input by an amplification ratio corresponding to the reduction ratio.

[0261] According to an embodiment of the present disclosure, a motion vector decoder may include multiple second decoding neural network layers, each of the second decoding neural network layers may include a third decoding layer and a fourth decoding layer in parallel, the third decoding layer may include a first convolutional layer and a first upsampling layer, and the fourth decoding layer may include a second convolutional layer and a first operation layer for multiplying the input by an amplification ratio corresponding to the reduction ratio.

[0262] According to an embodiment of the present disclosure, the motion vector decoder may include multiple third decoding neural network layers, and each of the third decoding neural network layers may include a first inverse operation layer for dilated convolution, a second inverse operation layer for dilated convolution, and a third inverse operation layer for dilated convolution in parallel.

[0263] According to an embodiment of the present disclosure, the current image may be feature data output by applying pixel data of the current image to a convolutional neural network, and the previously reconstructed image may be feature data output by applying pixel data of the previously reconstructed image to a convolutional neural network.

[0264] According to an embodiment of the present disclosure, the current image may be pixel data of the current image, and the previous reconstructed image may be pixel data of the previous reconstructed image.

[0265] According to the AI-based image decoding method of an embodiment of the present disclosure, motion vector feature data of a current image of multiple resolutions is obtained from a bitstream, and the current motion vector is obtained by applying the motion vector feature data to a motion vector decoder, so that by applying an extended effective receptive field, not only the estimation of small-sized motion vectors is effective, but also the estimation of large-sized motion is effective. Because the motion vector feature data of the current image of multiple resolutions obtained from the bitstream is a small-sized feature map output from the final layer, and is also a feature map of all scales, the image decoding method is effective in encoding feature data of multiple scales in terms of data compression, and is suitable for video encoding and decoding.

[0266] The AI-based image encoding device may include: a prediction encoder 1310 configured to obtain motion vector feature data of a current image of multiple resolutions by applying the current image and a previously reconstructed image to a motion vector encoder; a prediction decoder 1340 configured to obtain a current motion vector by applying the motion vector feature data to a motion vector decoder, and obtain a current predicted image by using the current motion vector and a previously reconstructed image; and a generator 1320 configured to generate a bitstream including the motion vector feature data. The multiple resolutions may include an original resolution of the current image and a resolution reduced from the original resolution of the current image.

[0267] According to an embodiment of the present disclosure, the prediction decoder 1340 may obtain a current residual image corresponding to the difference between the current prediction image and the current image. The prediction decoder 1340 may obtain feature data of the current residual image by applying the current residual image to the residual encoder. The generator 1320 may generate a bitstream further including feature data of the current residual image.

[0268] According to an embodiment of the present disclosure, a motion vector decoder may include multiple first decoding neural network layers, each of the multiple first decoding neural network layers may include a parallel first decoding layer and a second decoding layer, the first decoding layer may include a first upper convolutional layer, the second decoding layer may include a second upper convolutional layer and a first multiplication layer that multiplies the input by an amplification ratio corresponding to the reduction ratio, and a motion vector encoder may include multiple first encoding neural network layers, each of the multiple first encoding neural network layers may include a parallel first encoding layer and a second encoding layer, the first encoding layer may include a first convolutional layer, and the second encoding layer may include a pooling layer and a second convolutional layer.

[0269] According to an embodiment of the present disclosure, a motion vector decoder may include multiple second decoding neural network layers, each of the multiple second decoding neural network layers may include parallel third and fourth decoding layers, the third decoding layer may include a first convolutional layer and a first upsampling layer, the fourth decoding layer may include a second upsampling layer, a second convolutional layer, and a first multiplication layer that multiplies the input by an amplification ratio corresponding to the reduction ratio, the motion vector encoder may include multiple second encoding neural network layers, each of the multiple second encoding neural network layers may include parallel third and fourth encoding layers, the third encoding layer may include a third convolutional layer and a first pooling layer, and the fourth encoding layer may include a second pooling layer and a fourth convolutional layer.

[0270] According to an embodiment of the present disclosure, a motion vector decoder may include multiple third decoding neural network layers, each of which may include a parallel first inverse operation layer for dilated convolution, a second inverse operation layer for dilated convolution, and a third inverse operation layer for dilated convolution, and a motion vector decoder may include multiple third encoding neural network layers, and each of which may include a parallel first dilated convolution layer, a second dilated convolution layer, and a third dilated convolution layer.

[0271] According to an embodiment of the present disclosure, the current image may be feature data output by applying pixel data of the current image to a convolutional neural network, and the previously reconstructed image may be feature data output by applying pixel data of the previously reconstructed image to a convolutional neural network.

[0272] According to an embodiment of the present disclosure, the current image may be pixel data of the current image, and the previous reconstructed image may be pixel data of the previous reconstructed image.

[0273] According to an embodiment of the present disclosure, an AI-based image encoding device obtains motion vector feature data of a current image of multiple resolutions by using a motion vector encoder having a neural network structure including a neural network layer for maintaining the resolution of an input image and a neural network layer for reducing the resolution of an input image, and obtains a current motion vector by applying the motion vector feature data to a motion vector decoder, so that by applying an extended effective receptive field, not only is it effective to estimate a motion vector of a small size, but it is also effective to estimate a motion of a large size. Because the motion vector feature data of the current image of multiple resolutions obtained from the motion vector encoder is a feature map of a small size output from the final layer, and is also a feature map of all scales, the image encoding device is effective in encoding feature data of multiple scales in terms of data compression, and is suitable for video encoding and decoding.

[0274] According to an embodiment of the present disclosure, an AI-based image decoding device may include: an obtainer 1510 configured to obtain motion vector feature data of a current image of multiple resolutions from a bitstream; and a prediction decoder 1520 configured to obtain a current motion vector by applying the motion vector feature data to a motion vector decoder, and obtain a current predicted image by using the current motion vector and a previously reconstructed image. The multiple resolutions may include an original resolution of the current image and a resolution reduced from the original resolution of the current image.

[0275] According to an embodiment of the present disclosure, the obtainer 1510 may obtain feature data of the current residual image from the bitstream. The prediction decoder 1520 may obtain the current residual image by applying the feature data of the current residual image to the residual decoder. The prediction decoder 1520 may obtain the current reconstructed image by using the current prediction image and the current residual image.

[0276] According to an embodiment of the present disclosure, a motion vector decoder may include multiple first decoding neural network layers, each of the first decoding neural network layers may include a parallel first decoding layer and a second decoding layer, the first decoding layer may include a first upper convolution layer, the second decoding layer may include a second upper convolution layer and a first operation layer for multiplying the input by an amplification ratio corresponding to the reduction ratio.

[0277] According to an embodiment of the present disclosure, a motion vector decoder may include multiple second decoding neural network layers, each of the second decoding neural network layers may include a third decoding layer and a fourth decoding layer in parallel, the third decoding layer may include a first convolutional layer and a first upsampling layer, and the fourth decoding layer may include a second convolutional layer and a first operation layer for multiplying the input by an amplification ratio corresponding to the reduction ratio.

[0278] According to an embodiment of the present disclosure, the motion vector decoder may include multiple third decoding neural network layers, and each of the third decoding neural network layers may include a first inverse operation layer for dilated convolution, a second inverse operation layer for dilated convolution, and a third inverse operation layer for dilated convolution in parallel.

[0279] According to an embodiment of the present disclosure, the current image may be feature data output by applying pixel data of the current image to a convolutional neural network, and the previously reconstructed image may be feature data output by applying pixel data of the previously reconstructed image to a convolutional neural network.

[0280] According to an embodiment of the present disclosure, the current image may be pixel data of the current image, and the previous reconstructed image may be pixel data of the previous reconstructed image.

[0281] According to an AI-based image decoding device of an embodiment of the present disclosure, motion vector feature data of a current image of multiple resolutions is obtained from a bitstream, and the current motion vector is obtained by applying the motion vector feature data to a motion vector decoder, so that by applying an extended effective receptive field, not only the estimation of small-sized motion vectors is effective, but also the estimation of large-sized motion is effective. Because the motion vector feature data of the current image of multiple resolutions obtained from the bitstream is a small-sized feature map output from the final layer, and is also a feature map of all scales, the image decoding device is effective in encoding feature data of multiple scales in terms of data compression, and is suitable for video encoding and decoding.

[0282] The machine-readable storage medium may be provided as a non-transitory storage medium. A "non-transitory storage medium" is a tangible device, which simply means that it does not contain signals (e.g., electromagnetic waves). The term does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored. For example, a non-transitory recording medium may include a buffer that temporarily stores data.

[0283] According to an embodiment of the present disclosure, the method according to various disclosed embodiments may be provided by being included in a computer program product. The computer program product as a commodity may be traded between a seller and a buyer. The computer program product is distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed (e.g., downloaded or uploaded) directly and online through an application store or between two user devices (e.g., a smartphone). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be at least temporarily stored in a device-readable storage medium (such as a memory of a manufacturer's server, a server of an application store, or a relay server), or may be temporarily generated.

Claims

1. An image coding method based on artificial intelligence (AI), the image coding method comprising: obtaining motion vector feature data of a current frame by inputting the current frame and the reference frame into a motion vector encoder and applying a plurality of resolution scales to the current frame and the reference frame via the motion vector encoder (S1210); Obtaining a current motion vector by inputting the motion vector feature data into a motion vector decoder (S1230); obtaining a current predicted image based on the current motion vector and the reference frame (S1250); and A bitstream is generated based on the motion vector feature data and the current prediction image (S1270).

2. The image encoding method according to claim 1, wherein: The reference frame is the previously reconstructed image.

3. The image encoding method according to claim 1 or 2, wherein the plurality of resolution scales include a first resolution scale for maintaining an original resolution of the current frame and a second resolution scale for reducing the original resolution of the current frame.

4. The image encoding method according to any one of claims 1 to 3, further comprising: Obtaining a current residual image corresponding to a difference between a current predicted image and a current frame; as well as The feature data of the current residual image is obtained by applying the current residual image to the residual encoder, The bit stream also includes feature data of the current residual image.

5. The image encoding method according to any one of claims 1 to 4, wherein The motion vector decoder comprises a plurality of first decoding neural network layers, Each of the plurality of first decoding neural network layers comprises a first decoding layer and a second decoding layer operating in parallel, The first decoding layer includes the first up-convolutional layer, The second decoding layer includes a second up-convolutional layer and a first multiplication layer, wherein the first multiplication layer multiplies the input by an up-scaling ratio corresponding to the down-scaling ratio, The motion vector encoder comprises a plurality of first encoding neural network layers, Each of the plurality of first encoding neural network layers includes a first encoding layer and a second encoding layer operating in parallel, The first encoding layer includes a first convolutional layer, and The second encoding layer includes a pooling layer and a second convolutional layer.

6. The image encoding method according to any one of claims 1 to 4, wherein The motion vector decoder comprises a plurality of second decoding neural network layers, Each of the plurality of second decoding neural network layers includes a third decoding layer and a fourth decoding layer operating in parallel, The third decoding layer includes the first convolutional layer and the first upsampling layer. The fourth decoding layer includes a second upsampling layer, a second convolutional layer, and a first multiplication layer, wherein the first multiplication layer multiplies the input by an upscaling ratio corresponding to the reduction ratio, The motion vector encoder comprises a plurality of second encoding neural network layers, Each of the plurality of second encoding neural network layers includes a third encoding layer and a fourth encoding layer operating in parallel, The third encoding layer includes a third convolutional layer and a first pooling layer, and The fourth encoding layer includes a second pooling layer and a fourth convolutional layer.

7. The image encoding method according to any one of claims 1 to 4, wherein The motion vector decoder comprises a plurality of third decoding neural network layers, Each of the plurality of third decoding neural network layers includes a first inverse operation layer for dilated convolution, a second inverse operation layer for dilated convolution, and a third inverse operation layer for dilated convolution operating in parallel, The motion vector decoder comprises a plurality of third coding neural network layers, and Each of the plurality of third encoding neural network layers includes a first dilated convolutional layer, a second dilated convolutional layer, and a third dilated convolutional layer operating in parallel.

8. The image encoding method according to any one of claims 1 to 7, wherein: The current frame includes feature data output by applying pixel data of the current frame to the convolutional neural network, and the reference frame includes feature data output by applying pixel data of the reference frame to the convolutional neural network.

9. The image encoding method according to any one of claims 1 to 7, wherein: The current frame includes pixel data of the current frame, and the reference frame includes pixel data of the reference frame.

10. An image decoding method based on artificial intelligence (AI), the image decoding method comprising: Obtain motion vector feature data of a current frame at multiple resolutions from a bitstream; obtaining a current motion vector by applying the motion vector feature data to a motion vector decoder; as well as The current predicted image is obtained by using the current motion vector and the reference frame. The multiple resolutions include an original resolution of the current frame and a resolution reduced from the original resolution of the current frame.

11. The image decoding method according to claim 10, further comprising: Obtain feature data of the current residual image from the bit stream; Obtaining a current residual image by applying feature data of the current residual image to a residual decoder; A current reconstructed image is obtained by using the current prediction image and the current residual image.

12. The image decoding method according to claim 10 or 11, wherein The motion vector decoder comprises a plurality of first decoding neural network layers, Each of the plurality of first decoding neural network layers comprises a first decoding layer and a second decoding layer operating in parallel, The first decoding layer includes a first up-convolutional layer, and The second decoding layer includes a second up-convolution layer and a first operation layer, wherein the first operation layer multiplies the input by a magnification ratio corresponding to the reduction ratio.

13. The image decoding method according to claim 10 or 11, wherein The motion vector decoder comprises a plurality of second decoding neural network layers, Each of the plurality of second decoding neural network layers includes a third decoding layer and a fourth decoding layer operating in parallel, The third decoding layer includes the first convolutional layer and the first upsampling layer, and The fourth decoding layer includes a second upsampling layer, a second convolutional layer, and a first operation layer, wherein the first operation layer multiplies the input by a magnification ratio corresponding to the reduction ratio.

14. The image decoding method according to claim 10 or 11, wherein The motion vector decoder comprises a plurality of third decoding neural network layers, and Each of the plurality of third decoding neural network layers includes a first inverse operation layer for dilated convolution, a second inverse operation layer for dilated convolution, and a third inverse operation layer for dilated convolution, which operate in parallel.

15. An image encoding device 1300 based on artificial intelligence (AI), the image encoding device comprising: A prediction encoder 1310 configured to obtain motion vector feature data of a current frame by inputting the current frame and the reference frame into a motion vector encoder and applying a plurality of resolution scales to the current frame and the reference frame via the motion vector encoder; The prediction decoder 1340 is configured to obtain a current motion vector by inputting the motion vector feature data into a motion vector decoder; Obtaining a current predicted image based on the current motion vector and the reference frame; as well as The generator 1320 is configured to generate a bitstream based on the motion vector feature data and the current prediction image.

Images