Method and apparatus for encoding image by using filtered optical flow, and method and apparatus for decoding image

Abstract

Disclosed is an image decoding method according to an embodiment comprising the steps of: acquiring feature data and filtering information of a preliminary optical flow from a bitstream, the filtering information including at least one of type information of a filter and parameter information of the filter; acquiring the preliminary optical flow by applying the feature data of the preliminary optical flow to a neural network-based first decoder; generating an optical flow by applying the preliminary optical flow to the filter corresponding to the filtering information; and generating the current reconstructed image by using previous data and the optical flow.

Description

Image encoding method and device using filtered optical flow, and image decoding method and device

[0001] The present disclosure relates to image encoding and decoding. More specifically, the present disclosure relates to a technique for encoding and decoding images using artificial intelligence (AI), such as a neural network.

[0002] In codecs such as H.264 AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding), an image can be divided into blocks, and each block can be predicted and decoded through inter prediction or intra prediction.

[0003] Intra prediction is a method of compressing images by removing spatial redundancy within the image, and inter prediction is a method of compressing images by removing temporal redundancy between images.

[0004] A representative example of inter prediction is motion estimation coding. Motion estimation coding uses a reference image to predict blocks in the current image. Using a predetermined evaluation function, the reference block most similar to the current block can be searched for within a predetermined range within the reference image. The current block is predicted based on the reference block, and the predicted block generated as a result of the prediction is subtracted from the current block to generate and encode a residual block.

[0005] Recently, technologies for encoding / decoding images using AI (Artificial Intelligence) have been proposed, and a method for effectively encoding / decoding images using AI, for example, neural networks, is required.

[0006] A method for decoding an image according to one embodiment may include a step of obtaining feature data and filtering information of a preliminary optical flow from a bitstream.

[0007] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0008] A method for decoding an image according to one embodiment may include a step of obtaining a preliminary optical flow by applying feature data of the preliminary optical flow to a neural network-based first decoder.

[0009] A method of decoding an image according to one embodiment may include a step of generating an optical flow by applying a preliminary optical flow to a filter corresponding to filtering information.

[0010] A method for decoding an image according to one embodiment may include a step of generating a current restored image using previous data and optical flow.

[0011] A method for encoding an image according to one embodiment may include a step of obtaining feature data of a preliminary optical flow by applying a current image and previous data to a neural network-based first encoder.

[0012] A video encoding method according to one embodiment may include a step of obtaining a preliminary optical flow by applying feature data of the preliminary optical flow to a neural network-based first decoder.

[0013] A video encoding method according to one embodiment may include a step of selecting a filter used for filtering preliminary optical flow among a plurality of filters.

[0014] A method of encoding an image according to one embodiment may include a step of generating an optical flow by applying a preliminary optical flow to a selected filter.

[0015] A method of encoding an image according to one embodiment may include a step of encoding a current image using optical flow and previous data.

[0016] A method of encoding an image according to one embodiment may include generating a bitstream including feature data of a preliminary optical flow and filtering information for a filter.

[0017] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0018] A computer-readable recording medium according to one embodiment can record a bitstream.

[0019] In one embodiment, the bitstream may include feature data and filtering information of the preliminary optical flow.

[0020] In one embodiment, feature data of the preliminary optical flow can be obtained by applying the current image and previous data to a neural network-based first encoder.

[0021] In one embodiment, filtering information can be obtained by applying feature data of the preliminary optical flow to a neural network-based first decoder to obtain the preliminary optical flow, and selecting a filter used for filtering the preliminary optical flow from among a plurality of filters.

[0022] In one embodiment, an optical flow is generated by applying a preliminary optical flow to a selected filter, and the current image can be encoded using the optical flow and previous data.

[0023] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0024] An image decoding device according to one embodiment may include an acquisition unit that acquires feature data and filtering information of a preliminary optical flow from a bitstream.

[0025] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0026] An image decoding device according to one embodiment may include a predictive decoding unit that obtains a preliminary optical flow by applying feature data of a preliminary optical flow to a neural network-based first decoder, generates an optical flow by applying the preliminary optical flow to a filter corresponding to filtering information, and generates a current restored image by using previous data and the optical flow.

[0027] An image encoding device according to one embodiment may include a prediction encoding unit that applies a current image and previous data to a neural network-based first encoder to obtain feature data of a preliminary optical flow, applies the feature data of the preliminary optical flow to a neural network-based first decoder to obtain a preliminary optical flow, selects a filter used for filtering the preliminary optical flow among a plurality of filters, applies the preliminary optical flow to the selected filter to generate an optical flow, and encodes a current image using the optical flow and previous data.

[0028] An image encoding device according to one embodiment may include a generation unit that generates a bitstream including feature data of a preliminary optical flow and filtering information for a filter.

[0029] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0030] FIG. 1 is a diagram illustrating an encoding and decoding process of an optical flow based on AI according to one embodiment.

[0031] FIG. 2 is a diagram illustrating an image encoding and decoding process based on AI according to one embodiment.

[0032] FIG. 3 is a diagram illustrating an image encoding and decoding process based on AI according to one embodiment.

[0033] FIG. 4 is a diagram illustrating a configuration of an image decoding device according to one embodiment.

[0034] FIG. 5 is a diagram illustrating the configuration of an acquisition unit according to one embodiment.

[0035] Figure 6 is a table for explaining filtering information according to one embodiment.

[0036] Figure 7 is a table for explaining filtering information according to one embodiment.

[0037] FIG. 8 is a diagram illustrating a neural network that can be used in a neural network filter according to one embodiment.

[0038] FIG. 9 is a diagram illustrating syntax for filtering information according to one embodiment.

[0039] Figure 10 is a flowchart of an image decoding method according to one embodiment.

[0040] FIG. 11 is a diagram illustrating a configuration of an image encoding device according to one embodiment.

[0041] Fig. 12 is a diagram illustrating the configuration of a generation unit according to one embodiment.

[0042] FIG. 13 is a diagram for explaining a method for a prediction encoding unit to select a filter according to one embodiment.

[0043] FIG. 14 is a diagram for explaining a method for a prediction encoding unit to select a filter according to one embodiment.

[0044] Figure 15 is a flowchart of an image encoding method according to one embodiment.

[0045] FIG. 16 is a diagram illustrating a method for training a neural network that can be used in a neural network filter according to one embodiment.

[0046] FIG. 17 is a diagram for explaining a training method of a first encoder, a first decoder, a second encoder, and a second decoder according to one embodiment.

[0047] FIG. 18 is a diagram for explaining a training method of a first encoder, a first decoder, a second encoder, and a second decoder according to one embodiment.

[0048] A method for decoding an image according to one embodiment may include a step of obtaining feature data and filtering information of a preliminary optical flow from a bitstream.

[0049] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0050] A method for decoding an image according to one embodiment may include a step of obtaining a preliminary optical flow by applying feature data of the preliminary optical flow to a neural network-based first decoder.

[0051] A method of decoding an image according to one embodiment may include a step of generating an optical flow by applying a preliminary optical flow to a filter corresponding to filtering information.

[0052] A method for decoding an image according to one embodiment may include a step of generating a current restored image using previous data and optical flow.

[0053] The present disclosure may be subject to various modifications and may have various embodiments. Therefore, embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the embodiments of the present disclosure, and the present disclosure may include all modifications, equivalents, and alternatives that fall within the spirit and technical scope of the various embodiments.

[0054] When describing embodiments, detailed descriptions of related known technologies may be omitted if they are deemed to unnecessarily obscure the gist of the present disclosure. Furthermore, numbers (e.g., "first," "second," etc.) used in the description of embodiments may correspond to identification symbols used to distinguish one component from another.

[0055] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0056] In the present disclosure, when a component is referred to as being “connected” or “connected” to another component, the component may be directly connected or connected to the other component, but unless there is a specific description to the contrary, the component may also be connected or connected via another component in between.

[0057] In the present disclosure, components expressed as "units", "modules", etc. may be two or more components combined into a single component, or a single component may be divided into two or more more detailed components. In addition, each component described below may additionally perform some or all of the functions performed by other components in addition to its own main function, and some of the main functions performed by each component may be exclusively performed by other components.

[0058] In the present disclosure, 'image' may refer to a picture, a still image, a frame, a moving image composed of a plurality of consecutive still images, or a video.

[0059] In the present disclosure, the "current image" may be an image that is currently being encoded and decoded. In one embodiment, the current image may be a block segmented from the image that is currently being encoded and decoded. For example, the current image may be a slice, tile, maximum coding unit, encoding unit, prediction unit, or transform unit segmented from the image.

[0060] In the present disclosure, a 'previous image' may be an image whose encoding order and decoding order are earlier than that of the current image. In one embodiment, the previous image may be a slice, tile, maximum encoding unit, encoding unit, prediction unit, or transform unit segmented from an image.

[0061] In this disclosure, "neural network" may be a representative example of an artificial neural network model that mimics brain neurons, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

[0062] In this disclosure, "neural network parameters" may refer to values ​​used in the computational process of each layer of the neural network. For example, neural network parameters may be used when applying input values ​​to a given computational formula. Neural network parameters are values ​​established as a result of training and may be updated using separate training data as needed.

[0063] In this disclosure, "feature data" may refer to data obtained by a neural network-based encoder processing input data. The feature data may be one-dimensional or two-dimensional data containing multiple samples. The feature data may also be referred to as a latent representation. The feature data may represent latent features in data output by a decoder, as described below.

[0064] In this disclosure, "sample" may refer to data assigned to a sampling location of image or feature data, and thus to be processed. For example, a pixel within a frame in a spatial domain may correspond to a sample. A unit containing multiple samples may be defined as a block.

[0065] FIG. 1 is a diagram illustrating an encoding and decoding process of an optical flow based on AI according to one embodiment.

[0066] Inter prediction may be a process of encoding and decoding a current image (10) by utilizing temporal redundancy between the current image (10) and a previous restored image. The previous restored image may be an image obtained through decoding of the previous image.

[0067] The positional differences (or motion vectors) between blocks or samples in the current image (10) and reference blocks or reference samples in the previously reconstructed image can be used for inter prediction for the current image (10). These positional differences can be referred to as optical flow. Optical flow can also be defined as a set of motion vectors corresponding to samples or blocks in the image.

[0068] Optical flow can indicate how the positions of samples in the previous restored image have changed within the current image (10), or where samples that are identical / similar to the samples in the current image (10) are located within the previous restored image.

[0069] For example, if a sample identical to or most similar to a sample located at (1, 1) in the current image (10) is located at (2, 1) in the previous restored image, the optical flow or motion vector for the sample located at (1, 1) can be derived as (1(=2-1), 0(=1-1)).

[0070] In the process of encoding and decoding optical flow using AI according to one embodiment, a first encoder (12) and a first decoder (14) may be used to obtain optical flow for a current image (10).

[0071] The first encoder (12) and the first decoder (14) can be implemented as neural networks. The first encoder (12) and the first decoder (14) can be understood as neural networks for extracting optical flow.

[0072] In one embodiment, the first encoder (12) may be referred to as an optical flow encoder, and the first decoder (14) may be referred to as an optical flow decoder. In one embodiment, the first encoder (12) may be referred to as a motion vector encoder, and the first decoder (14) may be referred to as a motion vector decoder.

[0073] Referring to FIG. 1, previous data (20) and a current image (10) can be input to a first encoder (12). The first encoder (12) can process the current image (10) and previous data (20) according to parameters set as a result of training to output feature data (w) of preliminary optical flow.

[0074] In the present disclosure, the previous data (20) may include at least one of a previous restored image, feature data of the previous restored image, a previous predicted image used to generate the previous restored image, feature data of the previous predicted image, a previous preliminary optical flow used to generate the previous restored image, feature data of the previous preliminary optical flow, a previous residual image used to generate the previous restored image, or feature data of the previous residual image.

[0075] In one embodiment, the previous data (20) may mean the previous restored image itself or data acquired during the process of generating the previous restored image.

[0076] In one embodiment, the feature data of the previous restored image, which is an example of the previous data (20), may be feature data output from a predetermined layer (the first layer or the middle layer) other than the last layer among the layers constituting the neural network (e.g., the second decoder (24) illustrated in FIG. 3) that outputs the previous restored image.

[0077] In one embodiment, a previous predicted image, which is an example of previous data (20), can be used to generate a previous restored image, and can be obtained by warping an image restored before the previous restored image. The feature data of the previous predicted image can be obtained by processing the previous predicted image with a neural network. In one embodiment, the feature data of the previous predicted image can also be obtained by warping the feature data of an image restored before the previous restored image.

[0078] The feature data (w) of the preliminary optical flow can be input to the first decoder (14). The first decoder (14) can process the input feature data (w) according to the parameters set as a result of training and output the preliminary optical flow (g).

[0079] In one embodiment, certain processing may be performed on the preliminary optical flow (g) for encoding and decoding of the current image (10).

[0080] The reason for performing some processing on the preliminary optical flow (g) is that there is a possibility that the preliminary optical flow (g) may not be accurate.

[0081] As described below with reference to FIGS. 17 and 18, since the first encoder (12) and the first decoder (14) are trained in a direction in which the bitrate of the bitstream is reduced, a preliminary optical flow (g) of relatively low quality may be output. In addition, since there is a limit to the number of layers that can be included in the first encoder (12) and the first decoder (14) due to the load and delay of the calculation, it may be difficult to generate a preliminary optical flow (g) of high quality. Due to the low quality of the preliminary optical flow (g), the size of the residual data may increase, and thus the bitrate of the bitstream may increase.

[0082] In one embodiment, the bitrate of the bitstream can be reduced by performing post-processing on the preliminary optical flow (g).

[0083] As illustrated in Fig. 1, a filter (30) may be used for post-processing of the preliminary optical flow (g). The filter (30) may be a filter selected from among a plurality of filters according to a predetermined criterion. A method for selecting a filter will be described later.

[0084] An optical flow (h) may be generated as the preliminary optical flow (g) is filtered by the filter (30). In one embodiment, as the preliminary optical flow (g) is applied to the filter (30), at least some of the sample values ​​of the preliminary optical flow (g) are changed, and an optical flow (h) including the changed sample values ​​may be obtained. In one embodiment, the optical flow (h) may also be referred to as a filtered optical flow.

[0085] FIG. 1 illustrates that previous data (20) and a current image (10) are applied to a first encoder (12), and feature data (w) of a preliminary optical flow is obtained from the first encoder (12). There may be various methods for obtaining the feature data (w) of the preliminary optical flow.

[0086] For example, a preliminary optical flow may be extracted from previous data (20) and a current image (10) according to a known optical flow extraction method, and the extracted preliminary optical flow may be applied to the first encoder (12). Then, feature data (w) of the preliminary optical flow output from the first encoder (12) may be input to the first decoder (14).

[0087] Additionally, for example, multiple neural networks connected in series or in parallel may be used to obtain feature data (w) of preliminary optical flow, and multiple neural networks connected in series or in parallel may be used to obtain preliminary optical flow (g) from feature data (w) of preliminary optical flow. When multiple neural networks are connected in series, it may mean that the output of one neural network is input to another neural network. When multiple neural networks are connected in parallel, it may mean that one neural network and another neural network separately process input data, and the output data of one neural network and another neural network are combined with each other.

[0088] In one embodiment, when the encoding and decoding process of the optical flow illustrated in FIG. 1 is implemented by an encoding device and a decoding device, the encoding device can obtain feature data (w) of the preliminary optical flow using the current image (10) and previous data (20). Then, the encoding device can generate a bitstream including feature data (w) of the preliminary optical flow and information about a filter (30) applied to the preliminary optical flow (g) (hereinafter, filtering information) and transmit the bitstream to the decoding device.

[0089] A decoding device can obtain feature data (w) and filtering information of a preliminary optical flow from a bitstream. Then, the decoding device can process the feature data (w) of the preliminary optical flow with a first decoder (14) to obtain a preliminary optical flow (g), and filter the preliminary optical flow (g) according to the filtering information to obtain an optical flow (h).

[0090] When the optical flow (h) for the current image (10) is obtained, the current image (10) can be encoded and decoded based on the optical flow (h).

[0091] In one embodiment, a current image (10) can be encoded and decoded based on optical flow (h) and previous data (20).

[0092] In one embodiment, motion compensation may be applied to previous data (20) based on optical flow (h) for encoding and decoding the current image (10). Motion compensation may generate a predicted image similar to the current image (10).

[0093] In one embodiment, the predicted image may be determined as the current reconstructed image.

[0094] In one embodiment, a residual image corresponding to the difference between the current image (10) and the predicted image is acquired, and the residual image and the predicted image are combined to generate the current reconstructed image. Data for the residual image can be transmitted from the encoder to the decoder via a bitstream.

[0095] Hereinafter, with reference to FIGS. 2 and 3, the encoding and decoding process of the current image (10) will be described.

[0096] FIG. 2 is a diagram illustrating an image encoding and decoding process based on AI according to one embodiment.

[0097] In the encoding and decoding process of the current image (10) using AI according to one embodiment, a second encoder (22) and a second decoder (24) may be used.

[0098] The second encoder (22) and the second decoder (24) can be implemented as a neural network. The second encoder (22) and the second decoder (24) can be understood as a neural network for encoding and decoding a residual image (60) corresponding to the difference between the current image (10) and the predicted image (50).

[0099] In one embodiment, the second encoder (22) may be referred to as a residual encoder, and the second decoder (24) may be referred to as a residual decoder.

[0100] In one embodiment, the second encoder (22) and the second decoder (24) are used to encode and decode pixel values ​​of the current image (10), so the second encoder (22) may be referred to as a pixel encoder, and the second decoder (24) may be referred to as a pixel decoder.

[0101] Referring to FIG. 2, previous data (20) is warped through warping (40) based on optical flow (h), and a predicted image (50) can be obtained as a result of warping (40). Warping (40) may be a type of geometric deformation that moves the positions of samples within an image.

[0102] In the embodiment illustrated in FIG. 2, a previously restored image can be used as previous data (20) to generate a predicted image (50). Accordingly, a predicted image (50) can be obtained through warping (40) on the previously restored image.

[0103] A predicted image (50) similar to the current image (10) can be obtained by applying warping (40) to the previous data (20) according to the optical flow (h) indicating the relative positional relationship between samples in the previous data (20) and samples in the current image (10).

[0104] For example, if the sample located at (1, 1) in the previous data (20) is most similar to the sample located at (2, 1) in the current image (10), the location of the sample located at (1, 1) in the previous data (20) can be changed to (2, 1) through warping (40) based on optical flow (h).

[0105] In one embodiment, warping (40) may be implemented based on a neural network, in which case, previous data (20) and optical flow (h) may be input into a neural network for warping (40), and a predicted image (50) may be output from the neural network.

[0106] Since the predicted image (50) generated from the previous data (20) is not the current image (10) itself, a residual image (60) corresponding to the difference between the predicted image (50) and the current image (10) can be obtained.

[0107] For example, a residual image (60) can be obtained by subtracting sample values ​​in a predicted image (50) from sample values ​​in a current image (10).

[0108] The residual image (60) can be input to the second encoder (22). The second encoder (22) can process the residual image (60) according to parameters set as a result of training and output feature data (v) of the residual image (60).

[0109] The feature data (v) of the residual image (60) can be input to the second decoder (24). The second decoder (24) can process the input feature data (v) according to parameters set as a result of training and output a restored residual image (70).

[0110] A current restored image (80) can be obtained by combining the predicted image (50) and the restored residual image (70). For example, a current restored image (80) can be obtained by adding sample values ​​in the predicted image (50) and sample values ​​in the restored residual image (70).

[0111] When the encoding and decoding process of the image illustrated in FIG. 2 is implemented by an encoding device and a decoding device, the encoding device can obtain feature data (v) of the residual image (60) using the second encoder (22). Then, the encoding device can generate a bitstream including the feature data (v) of the residual image (60) and transmit it to the decoding device.

[0112] The decoding device can obtain feature data (v) of a residual image (60) from a bitstream. Then, the decoding device can process the feature data (v) of the residual image (60) with a second decoder (24) to obtain a restored residual image (70), and can obtain a current restored image (80) by combining the predicted image (50) generated based on the optical flow (h) from the previous data (20) and the restored residual image (70).

[0113] FIG. 3 is a diagram illustrating an image encoding and decoding process based on AI according to one embodiment.

[0114] Referring to FIG. 3, the current image (10), previous data (20), and optical flow (h) can be input to the second encoder (22). The second encoder (22) can process the input data according to parameters set as a result of training and output feature data (v) of the residual image.

[0115] In the embodiment illustrated in FIG. 2, a residual image (60) corresponding to the difference between the predicted image (50) generated through warping (40) and the current image (10) is input to the second encoder (22), whereas in the embodiment illustrated in FIG. 3, it can be understood that the second encoder (22) performs the warping process for the previous data (20) and the process of obtaining the residual image together.

[0116] In FIG. 3, since the residual image (60) is not input to the second encoder (22), in one embodiment, it may be referenced that feature data of samples of the current image (10) are output from the second encoder (22).

[0117] In one embodiment, a predicted image generated from previous data (20), for example, a previous restored image, based on optical flow (h), and a current image (10) may be input to a second encoder (22).

[0118] In one embodiment, feature data of a predicted image generated from feature data of a previous restored image based on optical flow (h) and a current image (10) may be input to a second encoder (22).

[0119] In one embodiment, a predicted image generated from a previous restored image based on optical flow (h), feature data of a predicted image generated from feature data of a previous restored image based on optical flow (h), and a current image (10) may be input to a second encoder (22).

[0120] Previous data (20), optical flow (h), and feature data (v) of the residual image can be input to the second decoder (24). The second decoder (24) can process the input data according to parameters set as a result of training and output the current restored image (80).

[0121] In the embodiment illustrated in FIG. 2, the predicted image (50) generated through warping (40) and the residual image (70) output from the second decoder (24) are combined to obtain the current restored image (80), whereas in the embodiment illustrated in FIG. 3, it can be understood that the second decoder (24) performs the warping process for the previous data (20) and the process of combining the residual image and the predicted image together.

[0122] In one embodiment, previous data (20), for example, a predicted image (50) generated from a previous restored image based on optical flow (h), and feature data (v) of a residual image may be input to a second decoder (24).

[0123] In one embodiment, feature data of a predicted image generated from feature data of a previously restored image based on optical flow (h) and feature data (v) of a residual image may be input to a second decoder (24).

[0124] In one embodiment, a predicted image generated from a previous restored image based on optical flow (h), feature data of a predicted image generated from feature data of a previous restored image based on optical flow (h), and feature data (v) of a residual image may be input to the second decoder (24).

[0125] When the encoding and decoding process of the image illustrated in FIG. 3 is implemented by an encoding device and a decoding device, the encoding device can obtain feature data (v) of the residual image using the second encoder (22). Then, the encoding device can generate a bitstream including the feature data (v) of the residual image and transmit it to the decoding device.

[0126] The decoding device can obtain feature data (v) of a residual image from a bitstream. Then, the decoding device can obtain a current restored image (80) by processing the feature data (v) of the residual image, previous data (20), and optical flow (h) with a second decoder (24).

[0127] The encoding and decoding process of the optical flow (h) is described with reference to FIG. 1, and the encoding and decoding process of the current image (10) is described separately with reference to FIG. 2 and FIG. 3. The encoding and decoding process of the optical flow (h) can be understood as a part of the encoding and decoding process of the current image (10).

[0128] FIG. 4 is a diagram illustrating the configuration of an image decoding device (400) according to one embodiment.

[0129] Referring to FIG. 4, the image decoding device (400) may include an acquisition unit (410) and a prediction decoding unit (430).

[0130] The acquisition unit (410) and the prediction decoding unit (430) may be implemented with at least one processor. The acquisition unit (410) and the prediction decoding unit (430) may operate according to at least one instruction stored in a memory.

[0131] Although FIG. 4 illustrates the acquisition unit (410) and the prediction decoding unit (430) separately, the acquisition unit (410) and the prediction decoding unit (430) may be implemented through a single processor. For example, the acquisition unit (410) and the prediction decoding unit (430) may be implemented through a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software.

[0132] The acquisition unit (410) and the prediction decoding unit (430) may be configured with multiple processors. For example, the acquisition unit (410) and the prediction decoding unit (430) may be implemented with a combination of dedicated processors, or may be implemented with a combination of multiple general-purpose processors, such as an AP, CPU, or GPU, and software.

[0133] In one embodiment, the image decoding device (400) may utilize a first decoder (14) and a second decoder (24) to generate a current restored image. The first decoder (14) and the second decoder (24) may be stored in memory. In one embodiment, the first decoder (14) and the second decoder (24) may be implemented as an AI processor.

[0134] The acquisition unit (410) can acquire a bitstream including the encoding result for the current image.

[0135] The acquisition unit (410) can receive a bitstream from an image encoding device via a network. In one embodiment, the acquisition unit (410) can also acquire a bitstream from a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, and a magneto-optical medium such as a floptical disk.

[0136] The acquisition unit (410) can parse the bitstream to obtain data corresponding to the encoding result for the current image.

[0137] In one embodiment, data corresponding to the encoding result for the current image may include at least one of feature data of the preliminary optical flow, feature data of the residual image, or filtering information.

[0138] In one embodiment, the acquisition unit (410) may acquire a first bitstream corresponding to feature data and filtering information of a preliminary optical flow and a second bitstream corresponding to feature data of a residual image, and may parse the first bitstream and the second bitstream, respectively, to acquire feature data of the preliminary optical flow, feature data of the residual image, and filtering information.

[0139] Data corresponding to the encoding result for the current image is transmitted to the prediction decoding unit (430), and the prediction decoding unit (430) can use the acquired data to generate a current restored image corresponding to the current image.

[0140] In one embodiment, the currently restored image can be transmitted to a display device for playback.

[0141] FIG. 5 is a diagram illustrating the configuration of an acquisition unit (410) according to one embodiment.

[0142] Referring to FIG. 5, the acquisition unit (410) may include an entropy decoding unit (510) and an inverse quantization unit (530).

[0143] The entropy decoding unit (510) can entropy code the bitstream to obtain feature data of the preliminary optical flow, feature data of the residual image, and filtering information.

[0144] In one embodiment, at least one of the feature data of the preliminary optical flow, the feature data of the residual image, or the filtering information may be quantized. In this case, the dequantization unit (530) may dequantize the quantized data to obtain dequantized data. The dequantized data may be transmitted to the prediction decoding unit (430).

[0145] In one embodiment, when the feature data of the preliminary optical flow and the feature data of the residual image are quantized, the inverse quantization unit (530) can inverse quantize the quantized feature data of the preliminary optical flow and the quantized feature data of the residual image to obtain inverse quantized feature data of the preliminary optical flow and inverse quantized feature data of the residual image.

[0146] In one embodiment, the acquisition unit (410) may further include an inverse transform unit. The inverse transform unit may inversely transform the inverse quantized data output from the inverse quantization unit (530) from the frequency domain to the spatial domain.

[0147] When the image encoding device (1100) described below converts data corresponding to the encoding result for the current image from the spatial domain to the frequency domain, the inverse transform unit can inversely transform the inverse quantized data output from the inverse quantization unit (530) from the frequency domain to the spatial domain.

[0148] In one embodiment, the acquisition unit (410) may not include the inverse quantization unit (530). That is, data corresponding to the encoding result for the current image can be acquired through processing by the entropy decoding unit (510).

[0149] Referring again to FIG. 4, the prediction decoding unit (430) can generate a current restored image using data transmitted from the acquisition unit (410).

[0150] In one embodiment, the predictive decoding unit (430) can obtain the preliminary optical flow by applying the feature data of the preliminary optical flow to the first decoder (14). Then, the predictive decoding unit (430) can obtain the optical flow by filtering the preliminary optical flow based on a filter indicated by the filtering information.

[0151] In one embodiment, the predictive decoding unit (430) can generate a current restored image based on the optical flow and previous data.

[0152] For example, the prediction decoding unit (430) can perform motion compensation on previous data based on optical flow to generate a current restored image.

[0153] Additionally, as an example, the prediction decoding unit (430) can generate a predicted image by warping previous data based on optical flow. The prediction decoding unit (430) can obtain a restored residual image by applying feature data of the residual image to the second decoder (24). The prediction decoding unit (430) can generate a current restored image by combining the predicted image and the restored residual image.

[0154] Additionally, as an example, the prediction decoding unit (430) can obtain a current restored image by applying the optical flow, previous data, and feature data of the residual image to the second decoder (24).

[0155] Additionally, as an example, the prediction decoding unit (430) can obtain a current restored image by applying a predicted image generated from previous data based on optical flow and feature data of a residual image to the second decoder (24).

[0156] Hereinafter, the preliminary optical flow filtering process performed by the prediction decoding unit (430) will be described with reference to FIGS. 6 and 7.

[0157] Figure 6 is a table for explaining filtering information according to one embodiment.

[0158] In one embodiment, the predictive decoding unit (430) can filter the preliminary optical flow using a filter indicated by the filtering information.

[0159] Filtering information is information for specifying a filter used for filtering preliminary optical flow, and may include, for example, at least one of filter type information, filter parameter information, or weight information.

[0160] In one embodiment, the filter type information may indicate the type of filter used to filter the preliminary optical flow. For example, the filter type information may indicate at least one of a Gaussian filter, a median filter, a bilateral filter, or a neural network filter.

[0161] If the filter type information indicates a Gaussian filter, the prediction decoding unit (430) can filter the preliminary optical flow using the Gaussian filter.

[0162] The filter type information illustrated in FIG. 6 is merely an example, and various filters may be used for filtering preliminary optical flow. For example, various types of filters capable of filtering sample values ​​within two-dimensional data may be used by the prediction decoding unit (430).

[0163] In one embodiment, the filter type information may indicate multiple filters. For example, the filter type information may indicate a Gaussian filter and a neural network filter.

[0164] In one embodiment, if it is agreed between the image decoding device (400) and the image encoding device what type of filter to use, the filtering information may not include information on the type of filter.

[0165] In one embodiment, the parameter information of the filter may indicate the values ​​of the filter parameters. Since the filtering effect of the same Gaussian filter may vary depending on the values ​​of the parameters, the image encoding device may signal parameter information indicating the values ​​of the parameters set in the Gaussian filter to the image decoding device (400).

[0166] The parameters of the filter may vary depending on the type of the filter. For example, as illustrated in FIG. 6, the parameters of the Gaussian filter may include the size of the filter kernel, the standard deviation (σx) in the x-axis direction (or horizontal direction), and the standard deviation (σy) in the y-axis direction (or vertical direction). In addition, for example, the parameters of the median filter may include the size of the filter kernel, and the parameters of the Laplacian filter may include the size and standard deviation of the filter kernel. In addition, for example, the parameters of the neural network filter may indicate at least one neural network among a plurality of neural networks that can be used as the neural network filter.

[0167] In one embodiment, a parameter of a filter may be referenced as a setting item of the filter, and a value of the parameter may be referenced as a setting value.

[0168] In one embodiment, when the type information of the filter indicates multiple filters, parameter information of each of the multiple filters may be included in the filtering information.

[0169] In one embodiment, the parameter information of the filter may include an index or flag indicating any one of a plurality of candidate values ​​that may be set for a specific parameter. For example, the parameter information of a Gaussian filter may include an index indicating any one of {3, 5, 7} that may be used as a kernel size, an index indicating any one of {0, 1 / 2, 1, 2, 4, 8} that may be used as an x-axis standard deviation, and an index indicating any one of {0, 1 / 2, 1, 2, 4, 8} that may be used as a y-axis standard deviation.

[0170] In one embodiment, the parameter information of a filter may include the value of the parameter itself.

[0171] In one embodiment, if it is agreed in advance between the image decoding device (400) and the image encoding device which value to use for a specific parameter among the parameters of the filter, information indicating the value of the parameter agreed upon may not be included in the filtering information. For example, if the image decoding device (400) and the image encoding device agree to set the size of the Gaussian filter to 3, the image encoding device may transmit the value of the x-axis direction standard deviation and the value of the y-axis direction standard deviation as parameter information of the Gaussian filter to the image decoding device (400), but may not transmit the value of the size of the Gaussian filter to the image decoding device (400).

[0172] The prediction decoding unit (430) can filter the preliminary optical flow with a filter corresponding to the filtering information. For example, if the type information of the filtering information indicates a Gaussian filter and the parameter information indicates that the size is 3, the x-axis direction standard deviation is 1, and the y-axis direction standard deviation is 2, the prediction decoding unit (430) can filter the preliminary optical flow with a filter kernel having a size of 3 and sample values ​​corresponding to the x-axis direction standard deviation of 1 and the y-axis direction standard deviation of 2.

[0173] In one embodiment, the predictive decoding unit (430) may determine the values ​​of the filter parameters based on the filtering strength of the filter. For example, if the filtering strength represents 3, the predictive decoding unit (430) may determine the values ​​of the parameters corresponding to the filtering strength of 3. In other words, the values ​​of the parameters corresponding to the filtering strength of the filter may be determined in advance.

[0174] In one embodiment, if the type information of the filter indicates a Gaussian filter, a median filter, or a Laplacian filter, the prediction decoding unit (430) can obtain an optical flow through a convolution operation using a filter kernel corresponding to the Gaussian filter, the median filter, or the Laplacian filter and the preliminary optical flow.

[0175] In one embodiment, if the type information of the filter indicates a neural network filter, the prediction decoding unit (430) can obtain an optical flow by applying a preliminary optical flow to the neural network indicated by the parameter information.

[0176] In one embodiment, the weight information included in the filtering information may indicate a weight used in a weighted sum among a plurality of candidate weights.

[0177] In one embodiment, the predictive decoding unit (430) can obtain a filtered preliminary optical flow by filtering the preliminary optical flow using a filter specified by type information and parameter information. The predictive decoding unit (430) can obtain an optical flow by weighting the filtered preliminary optical flow and the preliminary optical flow according to a weight indicated by the weight information.

[0178] For example, optical flow can be generated according to the following mathematical expression 1.

[0179] [Mathematical Formula 1]

[0180] P3 = w1*P1+w2*P2

[0181] In Equation 1, P1 may correspond to the preliminary optical flow, P2 may correspond to the filtered preliminary optical flow, and P3 may correspond to the optical flow. In addition, w1 and w2 may be weights applied to P1 and P2, respectively, and may be identified from the weight information.

[0182] In one embodiment, w2 is a weight indicated by the weight information, and w1 may be a-w2. a is a predetermined integer, for example, 1. In one example, if the weight information indicates 1 / 8, w2 may be 1 / 8, and w1 may be 7 / 8.

[0183] In one embodiment, w1 is the weight represented by the weight, and w2 may be a-w1.

[0184] In one embodiment, the predictive decoding unit (430) can obtain a first filtered preliminary optical flow by filtering the preliminary optical flow with a first filter indicated by the type information, and can obtain a second filtered preliminary optical flow by filtering the preliminary optical flow with a second filter indicated by the type information. The predictive decoding unit (430) can obtain an optical flow by weighting the first filtered preliminary optical flow and the second filtered preliminary optical flow according to a weight indicated by the weight information.

[0185] For example, optical flow can be generated according to the following mathematical expression (2).

[0186] [Equation 2]

[0187] P3 = w1*P2_a+w2*P2_b

[0188] In Equation 2, P2_a may correspond to the first filtered preliminary optical flow, P2_b may correspond to the second filtered preliminary optical flow, and P3 may correspond to the optical flow. In addition, w1 and w2 may be weights applied to P2_a and P2_b, respectively, and may be identified from the weight information.

[0189] In one embodiment, w2 is a weight indicated by the weight information, and w1 may be a-w2. a is a predetermined integer, for example, 1. In one example, if the weight information indicates 1 / 8, w2 may be 1 / 8, and w1 may be 7 / 8. In one embodiment, w1 is a weight indicated by the weight, and w2 may be a-w1.

[0190] In one embodiment, if the type information of the filter indicates n filters (where n is an integer greater than or equal to 1), the weight information may indicate n-1 candidates among a plurality of candidate weights (e.g., 0, 1 / 8, 1 / 4, 1 / 2, and 1 as illustrated in FIG. 6). For example, if the type information of the filter indicates three filters, the weight information may indicate two candidate weights among the plurality of candidate weights. A first weight indicated by the weight information may be applied to a first filtered preliminary optical flow generated based on a first filter among the three filters indicated by the type information, and a second weight indicated by the weight information may be applied to a second filtered preliminary optical flow generated based on a second filter among the three filters indicated by the type information. And, for the third filtered preliminary optical flow generated based on the third filter indicated by the type information, a value obtained by subtracting the sum of the first weight and the second weight from a predetermined value (e.g., 1) may be applied.

[0191] In one embodiment, the parameter information may indicate the strength of the filter, and the prediction decoding unit (430) may determine the parameter values ​​of the filter by considering the strength of the filter. This will be described with reference to FIG. 7.

[0192] Figure 7 is a table for explaining filtering information according to one embodiment.

[0193] In one embodiment, the parameter information may indicate the strength of the filter indicated by the type information. The strength of the filter may be divided into multiple values. In one example, a higher strength value may indicate a stronger filter.

[0194] Parameter values ​​corresponding to the strength of the filter may be agreed upon between the image decoding device (400) and the image encoding device (1100). Accordingly, the prediction decoding unit (430) may determine the parameter values ​​of the filter by considering the strength of the filter indicated by the parameter information.

[0195] In one example, referring to FIG. 7, if the type information indicates a Gaussian filter and the parameter information indicates a strength of 3, the prediction decoding unit (430) can determine the size of the Gaussian filter to be 7, the x-axis direction standard deviation to be 8, and the y-axis direction standard deviation to be 8.

[0196] Also, in one example, if the type information represents a Gaussian filter and the parameter information represents a strength of 1, the prediction decoding unit (430) can determine the size of the Gaussian filter to be 3, the x-axis direction standard deviation to be 1 / 2, and the y-axis direction standard deviation to be 1 / 2.

[0197] In one embodiment, the parameter information may include an index or flag indicating the strength of the filter. Accordingly, the prediction decoding unit (430) may determine the values ​​of the filter parameters through the index or flag corresponding to the parameter information.

[0198] As described above, when the type information of the filter indicates a neural network filter, the parameter information may indicate a neural network that can be used as a neural network filter among a plurality of neural networks.

[0199] The prediction decoding unit (430) can store in advance multiple neural networks that can be used as neural network filters, and can apply a preliminary optical flow to a neural network indicated by parameter information among the multiple neural networks.

[0200] FIG. 8 is a diagram illustrating a neural network that can be used in a neural network filter according to one embodiment.

[0201] The neural network (800) illustrated in FIG. 8 may be any one of the first neural network, the second neural network, and the third neural network illustrated in FIGS. 6 and 7.

[0202] As illustrated in FIG. 8, the preliminary optical flow (805) can be input to the first convolution layer (810).

[0203] The 3X3X4 shown in the first convolution layer (810) exemplifies convolution processing for one preliminary optical flow (805) using four filter kernels of size 3x3. As a result of the convolution processing, four feature maps can be generated by the four filter kernels.

[0204] The feature maps generated by the first convolution layer (810) may represent unique characteristics of the preliminary optical flow (805). For example, each feature map may represent vertical characteristics, horizontal characteristics, or edge characteristics of the preliminary optical flow (805).

[0205] The feature maps generated by the first convolution layer (810) can be input to the first activation layer (820).

[0206] The first activation layer (820) can provide non-linear characteristics to each feature map. The first activation layer (820) can include, but is not limited to, a sigmoid function, a Tanh function, a ReLU (Rectified Linear Unit) function, etc.

[0207] Imparting nonlinear characteristics in the first activation layer (820) may mean changing some sample values ​​of the feature maps and outputting them. In this case, the change may be performed by applying nonlinear characteristics.

[0208] The first activation layer (820) can determine whether to pass the sample values ​​of the feature map to the second convolution layer (830). For example, some sample values ​​of the feature map may be activated by the first activation layer (820) and passed to the second convolution layer (830), and some sample values ​​may be deactivated by the first activation layer (820) and not passed to the second convolution layer (830). The unique characteristics of the preliminary optical flow (805) represented by the feature maps can be emphasized by the first activation layer (820).

[0209] The feature maps output from the first activation layer (820) can be input to the second convolution layer (830). 3X3X4 displayed in the second convolution layer (830) exemplifies convolution processing on the input feature maps using four filter kernels of size 3x3. The output of the second convolution layer (830) can be input to the second activation layer (840). The second activation layer (840) can impart nonlinear characteristics to the input feature maps.

[0210] The feature maps output from the second activation layer (840) can be input to the third convolution layer (850). 3X3X1 displayed in the third convolution layer (850) exemplifies convolution processing to create one output data (855) using one filter kernel of size 3x3.

[0211] In one embodiment, the output data (855) may be optical flow, or may be filtered preliminary optical flow used in the weighted sum.

[0212] Although FIG. 8 illustrates that the neural network (800) includes three convolutional layers (810, 830, 850) and two activation layers (820, 840), this is only an example. In one embodiment, the number of convolutional layers and activation layers included in the neural network (800) may vary.

[0213] In one embodiment, the neural network (800) may be implemented via a recurrent neural network (RNN).

[0214] In one embodiment, the image decoding device (400) and the image encoding device (1100) described below may include at least one ALU (Arithmetic logic unit) for the aforementioned convolution operation and activation layer operation.

[0215] The ALU may be implemented as a processor. For a convolution operation, the ALU may include a multiplier that performs a multiplication operation between sample values ​​of the input data and sample values ​​of the filter kernel, and an adder that adds the results of the multiplication.

[0216] For the operation of the activation layer, the ALU may include a multiplier that multiplies the input sample value by a weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and a comparator that compares the multiplication result with a predetermined value to determine whether to pass the input sample value to the next layer.

[0217] In one embodiment, multiple different neural networks (e.g., the first neural network, the second neural network, and the third neural network illustrated in FIG. 6) may be used as neural network filters, and filtering results based on the multiple different neural networks may differ from each other. To derive different filtering results, each of the multiple neural networks may have different parameters or different internal structures as they are trained with different training data.

[0218] For example, if the internal structures of the first neural network and the second neural network are the same, the filtering results based on the first neural network and the filtering results based on the second neural network may be different by differentiating the training data used for training the first neural network and the training data used for training the second neural network. Here, the fact that the internal structures of the first neural network and the second neural network are the same may mean, for example, that the number of layers included in the first neural network and the second neural network are the same, and that the sizes and number of filter kernels used in the layers are the same.

[0219] Additionally, for example, if the internal structures of the first neural network and the second neural network are different (for example, if the first neural network has three convolutional layers and the second neural network has five convolutional layers), the filtering results based on the first neural network and the filtering results based on the second neural network may be different from each other.

[0220] In one embodiment, multiple neural networks that can be used as neural network filters may have different filtering strengths. For example, the first neural network may have the strongest filtering strength, while the third neural network may have the weakest filtering strength. The filtering strengths of the multiple neural networks may be determined based on the internal structure of each neural network and the type of training data. A neural network training method is described below with reference to FIG. 16.

[0221] FIG. 9 is a diagram illustrating the syntax of filtering information according to one embodiment.

[0222] In one embodiment, filtering information can be obtained from the bitstream at the picture sequence level (e.g., sequence parameter set (SPS)), picture level (e.g., picture parameter set (PPS)), or block level (e.g., slice data).

[0223] For example, if filtering information is obtained in units of picture sequences, the same filtering information can be applied to the picture sequence containing the current image.

[0224] Additionally, as an example, when filtering information is acquired on a picture-by-picture basis, filtering information for the current image and filtering information for other images can be acquired independently of each other.

[0225] Additionally, as an example, if the current image corresponds to a current block segmented from the image, filtering information can be obtained on a block-by-block basis, and in this case, filtering information for the current block and filtering information for other blocks can be obtained independently of each other.

[0226] In one embodiment, the type information, parameter information, and weight information included in the filtering information may be obtained in different units.

[0227] For example, filter type information may be obtained in picture sequence units (e.g., SPS), and parameter information and / or weight information may be obtained in picture units (e.g., PPS) or block units (e.g., slice data).

[0228] Additionally, as an example, filter type information may be obtained in picture units (e.g., PPS), and parameter information and / or weight information may be obtained in block units (e.g., slice data).

[0229] Additionally, as an example, filter type information can be obtained in picture sequence units (e.g., SPS), parameter information can be obtained in picture units (e.g., PPS), and weight information can be obtained in block units (e.g., slice data).

[0230] In one embodiment, information indicating whether filtering of the preliminary optical flow is required may be included in the bitstream. If the information indicates that filtering is required, the prediction decoding unit (430) may obtain filtering information and filter the preliminary optical flow based on the obtained filtering information. If the information indicating whether filtering of the preliminary optical flow is required indicates that filtering is not required, the prediction decoding unit (430) may determine the preliminary optical flow as an optical flow without obtaining the filtering information.

[0231] In one embodiment, information indicating whether filtering of preliminary optical flow is required can be obtained from the bitstream at the picture sequence unit (e.g., SPS), picture unit (e.g., PPS), or block unit (e.g., slice data).

[0232] In one example, information indicating whether filtering of preliminary optical flow is required is obtained at the picture sequence unit (e.g., SPS), and filtering information can be obtained at the picture unit (e.g., PPS) or block unit (e.g., slice data).

[0233] Additionally, in one example, information indicating whether filtering of preliminary optical flow is required may be obtained at the picture level (e.g., PPS), and filtering information may be obtained at the block level (e.g., slice data).

[0234] Referring to FIG. 9, in S910, a type_idx indicating a type of filter can be obtained from a bitstream. Depending on the value indicated by type_idx, the type of filter to be applied to a preliminary optical flow among multiple different types of filters can be determined.

[0235] In one embodiment, type_idx may be obtained from the bitstream when it is determined that filtering of the preliminary optical flow is required.

[0236] In S920, if type_idx is 0, for example, if type_idx indicates a Gaussian filter, in S930, size_idx indicating a size, st_horizontal_idx indicating an x-axis direction standard deviation, and st_vertical_idx indicating a y-axis direction standard deviation can be obtained from the bitstream. The prediction decoding unit (430) can determine parameter values ​​of the Gaussian filter from size_idx, st_horizontal_idx, and st_vertical_idx, and filter the preliminary optical flow with the Gaussian filter having the determined parameter values.

[0237] In S940, if type_idx is 1, for example, if type_idx indicates a median filter, size_idx indicating a size can be obtained from the bitstream in S950. The prediction decoding unit (430) can determine a parameter value of the median filter from size_idx and filter the preliminary optical flow with the median filter having the determined parameter value.

[0238] In S960, if type_idx is 2, for example, if type_idx indicates a Laplacian filter, size_idx indicating a size and st_idx indicating a standard deviation can be obtained from the bitstream in S970. The prediction decoding unit (430) can determine parameter values ​​of the Laplacian filter from size_idx and st_idx, and filter the preliminary optical flow with the Laplacian filter having the determined parameter values.

[0239] In S980, if type_idx is 3, for example, if type_idx indicates a neural network filter, NN_idx indicating a neural network used as a neural network filter in S990 can be obtained from the bitstream. The prediction decoding unit (430) can apply a preliminary optical flow to the neural network indicated by NN_idx.

[0240] Although not shown in FIG. 9, when type_idx is 0, 1, 2, or 3, weighting information for the weighted sum of the preliminary optical flow and the filtered preliminary optical flow can be further obtained from the bitstream.

[0241] Figure 10 is a flowchart of an image decoding method according to one embodiment.

[0242] In step S1010, the image decoding device (400) can obtain feature data and filtering information of the preliminary optical flow from the bitstream.

[0243] In one embodiment, the image decoding device (400) may obtain information from the bitstream indicating whether filtering is required for the preliminary optical flow, and if the obtained information indicates that filtering is required, may obtain filtering information from the bitstream.

[0244] In one embodiment, the image decoding device (400) can further obtain feature data of the residual image from the bitstream.

[0245] In one embodiment, the image decoding device (400) can perform entropy decoding, inverse quantization, and / or inverse transformation on the bitstream to obtain feature data of the preliminary optical flow, feature data of the residual image, and filtering information.

[0246] In step S1020, the image decoding device (400) can obtain the preliminary optical flow by applying the feature data of the preliminary optical flow to the first decoder (14).

[0247] The first decoder (14) can process feature data of the preliminary optical flow according to parameters set through training and output the preliminary optical flow.

[0248] In step S1030, the image decoding device (400) can generate an optical flow by applying a preliminary optical flow to a filter corresponding to the filtering information.

[0249] The filtering process of the preliminary optical flow using a filter corresponding to the filtering information has been described above, so a detailed description is omitted.

[0250] At step S1040, the image decoding device (400) can generate a current restored image using previous data and optical flow. The current restored image can be transmitted to a display for output.

[0251] In one embodiment, the image decoding device (400) can obtain a current restored image by performing motion compensation on previous data based on optical flow.

[0252] In one embodiment, the image decoding device (400) can generate a predicted image by warping previous data based on optical flow. The image decoding device (400) can obtain a reconstructed residual image by applying feature data of the residual image to the second decoder (24). The image decoding device (400) can generate a current reconstructed image by combining the predicted image and the reconstructed residual image.

[0253] In one embodiment, the predicted image may be determined as the current reconstructed image.

[0254] In one embodiment, the image decoding device (400) can obtain a current restored image by applying the optical flow, previous data, and feature data of the residual image to the second decoder (24).

[0255] In one embodiment, the image decoding device (400) can obtain a current restored image by applying a predicted image generated from previous data based on optical flow and feature data of a residual image to a second decoder (24).

[0256] FIG. 11 is a diagram illustrating the configuration of an image encoding device (1100) according to one embodiment.

[0257] Referring to FIG. 11, the image encoding device (1100) may include a prediction encoding unit (1110) and a generation unit (1130).

[0258] The prediction encoding unit (1110) and the generation unit (1130) may be implemented as a processor. The prediction encoding unit (1110) and the generation unit (1130) may operate according to instructions stored in a memory (not shown).

[0259] Although FIG. 11 illustrates the prediction encoding unit (1110) and the generation unit (1130) separately, the prediction encoding unit (1110) and the generation unit (1130) may be implemented through a single processor. In one embodiment, the prediction encoding unit (1110) and the generation unit (1130) may be implemented through a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software.

[0260] The prediction encoding unit (1110) and the generation unit (1130) may be configured with multiple processors. In one embodiment, the prediction encoding unit (1110) and the generation unit (1130) may be implemented with a combination of dedicated processors, or may be implemented with a combination of multiple general-purpose processors, such as an AP, CPU, or GPU, and software.

[0261] In one embodiment, the video encoding device (1100) may utilize a first encoder (12), a first decoder (14), a second encoder (22), and a second decoder (24) to encode a current video.

[0262] The first encoder (12), the first decoder (14), the second encoder (22), and the second decoder (24) may be stored in memory. In one embodiment, the first encoder (12), the first decoder (14), the second encoder (22), and the second decoder (24) may be implemented as an AI processor.

[0263] The prediction encoding unit (1110) can encode the current image and generate data corresponding to the encoding result.

[0264] In one embodiment, as an encoding result for a current image, at least one of feature data of a preliminary optical flow, feature data of a residual image, or filtering information can be obtained.

[0265] In one embodiment, the predictive encoding unit (1110) can obtain feature data of the preliminary optical flow by applying previous data and the current image to the first encoder (12).

[0266] The prediction encoding unit (1110) can obtain a preliminary optical flow for the current image by applying feature data of the preliminary optical flow to the first decoder (14).

[0267] In one embodiment, the predictive encoding unit (1110) may apply quantization and inverse quantization to the feature data of the preliminary optical flow obtained from the first encoder (12), and may apply the inverse quantized feature data of the preliminary optical flow to the first decoder (14). The reason for applying quantization and inverse quantization to the feature data of the preliminary optical flow is to obtain a preliminary optical flow that is identical to the preliminary optical flow obtained by the image decoding device (400).

[0268] In one embodiment, the predictive encoding unit (1110) may select a filter among a plurality of filters to be used for filtering the preliminary optical flow. In one embodiment, the plurality of filters may be a Gaussian filter, a median filter, a Laplacian filter, or a neural network filter, but the type of filter is not limited thereto.

[0269] Multiple filters can be distinguished from each other based on their type and parameters. For example, the first filter may be a Gaussian filter and the second filter may be a median filter. Furthermore, for example, if the first and second filters are Gaussian filters, the parameter values ​​of the first filter and the second filter may be different.

[0270] The prediction encoding unit (1110) can select a filter to be used for filtering preliminary optical flow from among a plurality of filters that can be specified by type and / or parameters.

[0271] In one embodiment, the predictive encoder (1110) may select two or more filters for filtering the preliminary optical flow.

[0272] In one embodiment, filtering information for a filter selected for filtering the preliminary optical flow may be transmitted to the generation unit (1130).

[0273] In one embodiment, the predictive encoder (1110) can obtain an optical flow for a current block by filtering a preliminary optical flow with a filter selected from among a plurality of filters.

[0274] In one embodiment, the predictive encoding unit (1110) may obtain a filtered preliminary optical flow by filtering the preliminary optical flow using a filter. The predictive encoding unit (1110) may obtain an optical flow by weighting the filtered preliminary optical flow and the preliminary optical flow according to weights. In one embodiment, the weight used in the weighted sum may be selected from a plurality of candidate weights.

[0275] In one embodiment, the predictive encoding unit (1110) can obtain a first filtered preliminary optical flow by filtering the preliminary optical flow with a first filter selected from among a plurality of filters, and can obtain a second filtered preliminary optical flow by filtering the preliminary optical flow with a second filter selected from among a plurality of filters. The predictive encoding unit (1110) can obtain an optical flow by weighting the first filtered preliminary optical flow and the second filtered preliminary optical flow according to weights.

[0276] In one embodiment, the prediction encoding unit (1110) can obtain feature data of a residual image using optical flow and previous data.

[0277] In one embodiment, the predictive encoding unit (1110) can generate a predicted image from previous data based on optical flow and obtain a residual image corresponding to the difference between the predicted image and the current image. The predictive encoding unit (1110) can apply the residual image to the second encoder (22) to obtain feature data of the residual image.

[0278] In one embodiment, the predictive encoding unit (1110) can obtain feature data of a residual image by applying the current image, previous data, and optical flow to the second encoder (22).

[0279] In one embodiment, the prediction encoding unit (1110) can obtain feature data of a residual image by applying a predicted image generated from previous data based on optical flow and a current image to the second encoder (22).

[0280] In one embodiment, when feature data of a residual image is acquired, the predictive encoding unit (1110) can generate a current restored image using the feature data of the residual image, optical flow, previous data, and the second decoder (24). Since the method for generating the current restored image has been described with respect to the image decoding device (400), a detailed description thereof will be omitted. The current restored image can be used for encoding the next image.

[0281] When the generation unit (1130) obtains data corresponding to the encoding result of the current image, it can generate a bitstream including the data.

[0282] In one embodiment, the generation unit (1130) can generate a first bitstream corresponding to feature data and filtering information of the preliminary optical flow, and a second bitstream corresponding to feature data of the residual image.

[0283] The bitstream may be transmitted to the image decoding device (400) via a network. In one embodiment, the bitstream may be recorded on a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, etc.

[0284] Fig. 12 is a diagram illustrating the configuration of a generation unit (1130) according to one embodiment.

[0285] Referring to FIG. 12, the generation unit (1130) may include a quantization unit (1210) and an entropy encoding unit (1230).

[0286] The quantization unit (1210) can quantize data corresponding to the encoding result for the current image.

[0287] In one example, the quantization unit (1210) may quantize feature data of the preliminary optical flow and feature data of the residual image. In one example, quantization may be applied to the feature data of the preliminary optical flow and the feature data of the residual image, but quantization may not be applied to the filtering information. In this case, the quantized feature data of the preliminary optical flow, the quantized feature data of the residual image, and the filtering information may be transmitted to the entropy encoding unit (1230).

[0288] The entropy encoding unit (1230) can generate a bitstream by entropy encoding data transmitted from the quantization unit (1210).

[0289] In one embodiment, the generation unit (1130) may further include a transformation unit. The transformation unit may transform data corresponding to the encoding result for the current image from the spatial domain to the frequency domain and provide the data to the quantization unit (1210).

[0290] In one embodiment, the generation unit (1130) may not include a quantization unit (1210). That is, a bitstream including data corresponding to the encoding result for the current image can be obtained through processing by the entropy encoding unit (1230).

[0291] Hereinafter, a method for selecting a filter to be used for filtering preliminary optical flow among a plurality of filters (hereinafter, a plurality of candidate filters) will be described with reference to FIGS. 13 and 14.

[0292] FIG. 13 is a diagram for explaining a method for selecting a filter by a prediction encoding unit (1110) according to one embodiment.

[0293] The prediction encoding unit (1110) can select one or more filters among a plurality of candidate filters to be used for filtering preliminary optical flows.

[0294] In one embodiment, the predictive encoder (1110) can select a weight to be used in the weighted sum among a plurality of candidate weights (e.g., 0, 1 / 8, 1 / 4, 1 / 2, and 1). As described above, the weight can be used in the weighted sum between the filtered preliminary optical flow and the preliminary optical flow, or in the weighted sum of two or more filtered preliminary optical flows. If the weighted sum is not used to obtain the optical flow, the predictive encoder (1110) can omit the process of selecting the weight.

[0295] Multiple candidate filters can be distinguished by type and parameter values. The multiple candidate filters can differ from each other in at least one of type and parameter values.

[0296] For example, the plurality of candidate filters may be a Gaussian filter, a median filter, a Laplacian filter, or a neural network filter as illustrated in FIG. 6. For example, one candidate filter may be a Gaussian filter, having a size of 3, an x-direction standard deviation of 0, and a y-direction standard deviation of 0. Another candidate filter may be a Gaussian filter, having a size of 3, an x-direction standard deviation of 1 / 2, and a y-direction standard deviation of 0. Yet another candidate filter may be a median filter, having a size of 3.

[0297] Additionally, for example, the plurality of candidate filters may correspond to a Gaussian filter, a median filter, a Laplacian filter, or a neural network filter as illustrated in FIG. 6, and the strengths of the plurality of candidate filters may be 3, 2, or 1.

[0298] The prediction encoding unit (1110) can select the most suitable filter and weight for filtering the preliminary optical flow by applying each of the candidate filters and each of the candidate weights to the preliminary optical flow.

[0299] Referring to FIG. 13, the prediction encoding unit (1110) can generate a prediction image from the current image using each of a plurality of candidate filters and each of the candidate weights. In addition, the prediction encoding unit (1110) can select a filter and a weight for filtering the preliminary optical flow from among the plurality of candidate filters and candidate weights based on the comparison result between the current image and the prediction image.

[0300] In one embodiment, the comparison result between the current image and the predicted image may include at least one of an L1-norm value, an L2-norm value, an SSIM value, a PSNR-HVS value, an MS-SSIM value, a VIF value, a VMAF value, an MSE value, an RMSE value, or a SAD value between the current image and the predicted image.

[0301] A process for producing a comparison result between a current image and a predicted image using one candidate filter and one candidate weight is described.

[0302] The predictive encoding unit (1110) can obtain a preliminary optical flow from the current image and previous data. A first encoder (12) and a first decoder (14) can be used to obtain the preliminary optical flow.

[0303] The predictive encoding unit (1110) can filter the preliminary optical flow with a candidate filter to generate a filtered preliminary optical flow. The predictive encoding unit (1110) can obtain an optical flow by weighting and adding the preliminary optical flow and the filtered preliminary optical flow according to candidate weights.

[0304] The predictive encoding unit (1110) can generate a predicted image using optical flow and previous data. Warping may be applied to the previous data to generate the predicted image. Once the predicted image is generated, the predictive encoding unit (1110) can calculate a comparison result between the current image and the predicted image, and use the calculated comparison result to select a filter and weights for filtering the preliminary optical flow.

[0305] In one embodiment, the prediction encoding unit (1110) can select a filter and weights used to generate a predicted image most similar to the current image among a plurality of candidate filters and a plurality of candidate weights.

[0306] The process of selecting filters and weights can be expressed in the pseudo code in Table 1 below.

[0307]

[0308] In Table 1, the distance may correspond to the comparison result between the current image and the predicted image. Referring to Table 1, the initial Best distance may be set to infinity. Then, for each filter type in the type list, for each size in the size list, for each parameter in the parameter list, and for each weight in the weight list, the comparison result (distance) between the predicted image and the current image may be calculated. If the distance is smaller than the Best distance, the distance is determined as the new Best distance, and the type, size, parameter, and weight corresponding to the Best distance may be determined as the Best Type, Best Size, Best Parameter, and Best Weight, respectively. The best type, best size, best parameter, and best weight determined most recently may be used for filtering the preliminary optical flow.

[0309] FIG. 14 is a diagram for explaining a method for selecting a filter by a prediction encoding unit (1110) according to one embodiment.

[0310] In one embodiment, the prediction encoding unit (1110) can select a filter and weights used to generate a current restored image that are most similar to the current image among a plurality of candidate filters and a plurality of candidate weights.

[0311] In one embodiment, the comparison result between the current image and the current reconstructed image may include at least one of an L1-norm value, an L2-norm value, an SSIM value, a PSNR-HVS value, an MS-SSIM value, a VIF value, a VMAF value, an MSE value, an RMSE value, or a SAD value between the current image and the current reconstructed image.

[0312] In one embodiment, the predictive encoder (1110) may further consider the bitrate of the bitstream when selecting filters and weights.

[0313] Referring to FIG. 14, the prediction encoding unit (1110) can encode the current image based on each of a plurality of candidate filters and each of a plurality of candidate weights, and transmit data generated as a result of the encoding to the generation unit (1130).

[0314] In one embodiment, the prediction encoding unit (1110) can transmit feature data of the preliminary optical flow, feature data of the residual image, and filtering information indicating candidate filters and candidate weights to the generation unit (1130) as an encoding result for the current image.

[0315] The generation unit (1130) can generate a bitstream based on the transmitted data.

[0316] In one embodiment, the predictive encoding unit (1110) may select a filter and weights by considering the feature data of the preliminary optical flow, the feature data of the residual image, and the bit rate of the bitstream corresponding to the filtering information.

[0317] In one embodiment, the bitrate of the bitstream corresponding to the feature data of the preliminary optical flow, the feature data of the residual image, and the filtering information may be the bitrate of the result of entropy encoding the feature data of the preliminary optical flow, the feature data of the residual image, and the filtering information.

[0318] In one example, the prediction encoding unit (1110) can select a filter and weights that result in the smallest rate among a plurality of candidate filters and a plurality of candidate weights.

[0319] In one embodiment, the predictive encoding unit (1110) can select a filter and a weight to be used for filtering the preliminary optical flow among a plurality of candidate filters and a plurality of candidate weights based on a comparison result between the current image and the current restored image and a bitrate of the bitstream.

[0320] For example, the prediction encoding unit (1110) can calculate a loss value by weighting and adding a value corresponding to the comparison result between the current image and the current restored image and a value corresponding to the bit rate of the bitstream, and select a filter and weight that cause the smallest loss value.

[0321] Describes the process of calculating a loss value using one candidate filter and candidate weights.

[0322] The predictive encoding unit (1110) can obtain a preliminary optical flow from the current image and previous data. A first encoder (12) and a first decoder (14) can be used to obtain the preliminary optical flow.

[0323] The predictive encoding unit (1110) can filter the preliminary optical flow with one candidate filter to generate a filtered preliminary optical flow. The predictive encoding unit (1110) can obtain an optical flow by weighting and adding the preliminary optical flow and the filtered preliminary optical flow according to candidate weights.

[0324] The predictive encoding unit (1110) can generate a current restored image using optical flow and previous data. A second encoder (22) and a second decoder (24) can be used to generate the current restored image.

[0325] Once the current restored image is generated, a comparison result between the current image and the current restored image can be calculated. In addition, the bit rate of the bit stream can be calculated using the feature data of the preliminary optical flow output from the first encoder (12), the feature data of the residual image output from the second encoder (22), and filtering information indicating one candidate filter and one candidate weight. The predictive encoding unit (1110) can calculate a loss value corresponding to one candidate filter and one candidate weight from the comparison result between the current image and the current restored image and the bit rate of the bit stream.

[0326] Figure 15 is a flowchart of an image encoding method according to one embodiment.

[0327] At step S1510, the image encoding device (1100) can obtain feature data of a preliminary optical flow by applying the current image and previous data to the first encoder (12).

[0328] At step S1520, the image encoding device (1100) can obtain the preliminary optical flow by applying the feature data of the preliminary optical flow to the first decoder (14).

[0329] At step S1530, the image encoding device (1100) can select a filter among a plurality of filters to be used for filtering the preliminary optical flow.

[0330] The method of selecting a filter is described with reference to FIGS. 13 and 14, so a detailed description is omitted.

[0331] At step S1540, the image encoding device (1100) can generate an optical flow by applying a preliminary optical flow to the selected filter.

[0332] In one embodiment, the video encoding device (1100) can obtain a filtered preliminary optical flow by applying a preliminary optical flow to a filter, and obtain an optical flow by weighting the filtered preliminary optical flow and the preliminary optical flow.

[0333] In one embodiment, the video encoding device (1100) can obtain a first filtered preliminary optical flow by applying the preliminary optical flow to a first filter, and can obtain a second filtered preliminary optical flow by applying the preliminary optical flow to a second filter. The video encoding device (1100) can obtain an optical flow by weighting the first filtered preliminary optical flow and the second filtered preliminary optical flow.

[0334] At step S1550, the image encoding device (1100) can encode the current image using optical flow and previous data.

[0335] In one embodiment, feature data of a residual image can be obtained as an encoding result for the current image.

[0336] In one embodiment, the image encoding device (1100) can generate a predicted image from previous data based on optical flow and obtain a residual image corresponding to the difference between the predicted image and the current image. The image encoding device (1100) can obtain feature data of the residual image by applying the residual image to the second encoder (22).

[0337] In one embodiment, the image encoding device (1100) can obtain feature data of a residual image by applying the current image, previous data, and optical flow to the second encoder (22).

[0338] In one embodiment, the image encoding device (1100) can obtain feature data of a residual image by applying a predicted image generated from previous data based on optical flow and a current image to a second encoder (22).

[0339] At step S1560, the image encoding device (1100) can generate a bitstream including feature data and filtering information of the preliminary optical flow.

[0340] In one embodiment, the filtering information may include at least one of filter type information, parameter information, or weight information.

[0341] In one embodiment, the bitstream may further include feature data of the residual image.

[0342] As described above, when a neural network filter is used to filter the preliminary optical flow, the image encoding device (1100) and the image decoding device (400) can apply the preliminary optical flow to a neural network selected from among a plurality of neural networks.

[0343] Each of the multiple neural networks can process preliminary optical flow according to parameters set through training. A method for training a neural network used as a neural network filter is described with reference to FIG. 16.

[0344] FIG. 16 is a diagram illustrating a method for training a neural network that can be used in a neural network filter according to one embodiment.

[0345] Referring to FIG. 16, a neural network (1600) can input a preliminary optical flow for training, process the preliminary optical flow for training according to preset parameters, and output the optical flow for training.

[0346] The comparison result between the ground truth optical flow and the training optical flow can be used as loss information for training the neural network (1600). In one embodiment, the comparison result between the ground truth optical flow and the training optical flow can include at least one of an L1-norm value, an L2-norm value, an SSIM value, a PSNR-HVS value, an MS-SSIM value, a VIF value, a VMAF value, an MSE value, an RMSE value, or a SAD value.

[0347] The neural network (1600) can update parameters to reduce or minimize loss information, and process the next input training preliminary optical flow using the updated parameters. Once the optimization of the parameters of the neural network (1600) is completed, the neural network (1600) can be stored in the image encoding device (1100) and the image decoding device (400).

[0348] In one embodiment, multiple neural networks that can be used as neural network filters may have different internal structures, and thus different filtering strengths. For example, the filtering strength of a neural network having n convolutional layers (where n is an integer greater than or equal to 2) may be stronger than the filtering strength of a neural network having m convolutional layers (where m is an integer less than n). Since each of multiple neural networks having different internal structures is trained based on a preliminary optical flow for training, multiple neural networks having different filtering strengths can be used as neural network filters.

[0349] In one embodiment, multiple neural networks used in the neural network filter may be trained based on different types of preliminary training optical flows. Accordingly, multiple neural networks with different filtering strengths may be obtained.

[0350] For example, the filtering strength of a neural network trained on high-quality training reserve optical flow may be relatively weak, while the filtering strength of a neural network trained on low-quality training reserve optical flow may be relatively strong.

[0351] A method for training a neural network (1600) using different types of training pre-optical flows is described.

[0352] In one embodiment, a first correct optical flow can be obtained from a previous reconstructed image (or previous data) and a current image reconstructed based on a first quantization parameter. The first correct optical flow can be extracted according to a predetermined optical flow extraction method (e.g., Flownet, OpenCV optical flow estimation, etc.). The previous data and the current image are sequentially applied to a first encoder (12) and a first decoder (14), and a first preliminary optical flow for training can be output from the first decoder (14). The first preliminary optical flow for training can be processed by a neural network (1600), and the neural network (1600) can be trained based on a comparison result between the first training optical flow output from the neural network (1600) and the first correct optical flow.

[0353] Next, a second correct optical flow can be obtained from the previous restored image (or previous data) and the current image restored based on the second quantization parameter. The previous data and the current image are sequentially applied to the first encoder (12) and the first decoder (14), and a second preliminary optical flow for training can be output from the first decoder (14). The second preliminary optical flow for training can be processed by the neural network (1600), and the neural network (1600) can be trained based on the comparison result between the second training optical flow output from the neural network (1600) and the second correct optical flow.

[0354] In one embodiment, if the first quantization parameter is smaller than the second quantization parameter, the quality of the first preliminary optical flow for training may be better than the quality of the second preliminary optical flow for training. This is because the quality of the previous restored image restored according to the high quantization parameter is relatively poor, and thus the quality of the preliminary optical flow for training obtained based on the previous restored image with poor quality may also be poor. In other words, the strength of the neural network (1600) trained based on the first preliminary optical flow for training with good quality may be weaker than the strength of the neural network trained based on the second preliminary optical flow for training with poor quality.

[0355] In one embodiment, multiple neural networks with different filtering strengths can be obtained by varying the quantization parameters used to obtain the previous restored image (or previous data).

[0356] In one embodiment, the preliminary training optical flow may be obtained by modifying sample values ​​of the correct optical flow. The quality of the preliminary training optical flow may be determined based on the degree of modification of the sample values ​​of the correct optical flow.

[0357] In one embodiment, training of a plurality of neural networks that can be used in a neural network filter can be performed by an image encoding device (1100), and the plurality of trained neural networks can be transmitted to an image decoding device (400).

[0358] In one embodiment, multiple neural networks can be trained by a separate training device (e.g., a server), and the multiple neural networks that have completed training can be transmitted to an image encoding device (1100) and an image decoding device (400).

[0359] Hereinafter, a training method of the first encoder (12), the first decoder (14), the second encoder (22), and the second decoder (24) will be described with reference to FIGS. 17 and 18.

[0360] FIG. 17 is a diagram for explaining a training method of a first encoder (12), a first decoder (14), a second encoder (22), and a second decoder (24) according to one embodiment.

[0361] In FIG. 17, the current image for training (1710), previous data for training (1720), and current restored image for training (1780) may correspond to the current image, previous data, and current restored image described above, respectively.

[0362] In training the first encoder (12), the first decoder (14), the second encoder (22), and the second decoder (24), it may be considered how similar the current reconstructed image for training (1780) is to the current image for training (1710), and how large the bitrate of the bitstream generated through encoding the current image for training (1710) is. To this end, in one embodiment, the first encoder (12), the first decoder (14), the second encoder (22), and the second decoder (24) may be trained based on first loss information (1792) and second loss information (1794) corresponding to how large the size of the bitstream is, and third loss information (1796) corresponding to the similarity between the current image for training (1710) and the current reconstructed image for training (1780).

[0363] Referring to FIG. 17, the current training image (1710) and previous training data (1720) can be input to the first encoder (12). The first encoder (12) can process the current training image (1710) and previous training data (1720) to output feature data (w) of the preliminary optical flow.

[0364] The feature data (w) of the preliminary optical flow is input to the first decoder (14), and the first decoder (14) can process the feature data (w) of the preliminary optical flow and output the preliminary optical flow (g).

[0365] A training prediction image (1750) can be generated by warping (40) the training previous data (1720) according to the preliminary optical flow (g), and a training residual image (1760) corresponding to the difference between the training prediction image (1750) and the training current image (1710) can be obtained.

[0366] The training residual image (1760) is input to the second encoder (22), and the second encoder (22) can process the training residual image (1760) to output feature data (v) of the residual image.

[0367] The second decoder (24) processes the feature data (v) of the residual image to restore the training residual image (1760), and the restored training residual image (1770) and the training prediction image (1750) are combined to obtain the training current restored image (1780).

[0368] For training the first encoder (12), the first decoder (14), the second encoder (22), and the second decoder (24), at least one of the first loss information (1792), the second loss information (1794), or the third loss information (1796) can be obtained.

[0369] The first loss information (1792) may correspond to the entropy of the feature data (w) of the preliminary optical flow or the bitrate of the bitstream corresponding to the feature data (w) of the preliminary optical flow. In addition, the second loss information (1794) may correspond to the entropy of the feature data (v) of the residual image or the bitrate of the bitstream corresponding to the feature data (v) of the residual image.

[0370] In one embodiment, instead of the first loss information (1792) and the second loss information (1794), one loss information may be derived from the bitrate of the bitstream corresponding to the feature data (w) of the preliminary optical flow and the feature data (v) of the residual image.

[0371] The first loss information (1792) and the second loss information (1794) are related to the encoding efficiency for the current training image (1710), and thus can be referred to as compression loss information.

[0372] The third loss information (1796) may correspond to a difference between the current training image (1710) and the current training reconstructed image (1780). The difference between the current training image (1710) and the current training reconstructed image (1780) may include at least one of an L1-norm value, an L2-norm value, an SSIM (Structural Similarity) value, a PSNR-HVS (Peak Signal-To-Noise Ratio-Human Vision System) value, an MS-SSIM (Multiscale SSIM) value, a VIF (Variance Inflation Factor) value, or a VMAF (Video Multimethod Assessment Fusion) value between the current training image (1710) and the current training reconstructed image (1780).

[0373] The third loss information (1796) is related to the quality of the current restored image (1780) for training, and therefore may also be referred to as quality loss information.

[0374] The first encoder (12), the first decoder (14), the second encoder (22) and the second decoder (24) can be trained such that the final loss information derived from at least one of the first loss information (1792), the second loss information (1794) or the third loss information (1796) is reduced or minimized.

[0375] In one embodiment, the first encoder (12), the first decoder (14), the second encoder (22) and the second decoder (24) can operate to reduce or minimize the final loss information while changing the values ​​of preset parameters.

[0376] In one embodiment, the final loss information can be calculated according to the following mathematical expression 3.

[0377] [Equation 3]

[0378] Final loss information = a*first loss information+b*second loss information+c*third loss information

[0379] In mathematical expression 3, a, b, and c may be weights applied to the first loss information (1792), the second loss information (1794), and the third loss information (1796), respectively.

[0380] According to mathematical expression 3, it can be seen that the first encoder (12), the first decoder (14), the second encoder (22) and the second decoder (24) are trained in a direction in which the current reconstructed image for training (1780) becomes as similar as possible to the current image for training (1710) and the size of the bitstream corresponding to the data output from the first encoder (12) and the second encoder (22) is minimized.

[0381] FIG. 18 is a diagram for explaining a training method of a first encoder (12), a first decoder (14), a second encoder (22), and a second decoder (24) according to one embodiment.

[0382] Referring to FIG. 18, the current training image (1710) and previous training data (1720) can be input to the first encoder (12). The first encoder (12) can process the current training image (1710) and previous training data (1720) to output feature data (w) of the preliminary optical flow.

[0383] The feature data (w) of the preliminary optical flow is input to the first decoder (14), and the first decoder (14) can process the feature data (w) of the preliminary optical flow and output the preliminary optical flow (g).

[0384] The preliminary optical flow (g), the current training image (1710), and the previous training data (1720) are input to the second encoder (22), and the second encoder (22) can process the preliminary optical flow (g), the current training image (1710), and the previous training data (1720) to output feature data (v) of the residual image. In one embodiment, the training predicted image and the current training image (1710) generated from the preliminary optical flow (g) and the previous training data (1720) may also be input to the second encoder (22).

[0385] The second decoder (24) can process the feature data (v) of the residual image, the preliminary optical flow (g), and the previous data for training (1720) to generate a current reconstructed image (1780) for training. In one embodiment, the training predicted image generated from the preliminary optical flow (g) and the previous data for training (1720) and the feature data (v) of the residual image may be input to the second decoder (24).

[0386] For training of the first encoder (12), the first decoder (14), the second encoder (22) and the second decoder (24), at least one of the first loss information (1892), the second loss information (1894) or the third loss information (1896) can be obtained.

[0387] The first loss information (1892) may correspond to the entropy of the feature data (w) of the preliminary optical flow or the bitrate of the bitstream corresponding to the feature data (w) of the preliminary optical flow. In addition, the second loss information (1894) may correspond to the entropy of the feature data (v) of the residual image or the bitrate of the bitstream corresponding to the feature data (v) of the residual image.

[0388] In one embodiment, instead of the first loss information (1892) and the second loss information (1894), one loss information may be derived from the bitrate of the bitstream corresponding to the feature data (w) of the preliminary optical flow and the feature data (v) of the residual image.

[0389] The third loss information (1896) may correspond to a difference between the current training image (1710) and the current training reconstructed image (1780). The difference between the current training image (1710) and the current training reconstructed image (1780) may include at least one of an L1-norm value, an L2-norm value, an SSIM (Structural Similarity) value, a PSNR-HVS (Peak Signal-To-Noise Ratio-Human Vision System) value, an MS-SSIM (Multiscale SSIM) value, a VIF (Variance Inflation Factor) value, or a VMAF (Video Multimethod Assessment Fusion) value between the current training image (1710) and the current training reconstructed image (1780).

[0390] The first encoder (12), the first decoder (14), the second encoder (22) and the second decoder (24) can be trained such that the final loss information derived from at least one of the first loss information (1892), the second loss information (1894) or the third loss information (1896) is reduced or minimized.

[0391] In one embodiment, the first encoder (12), the first decoder (14), the second encoder (22) and the second decoder (24) can operate to reduce or minimize the final loss information while changing the values ​​of preset parameters.

[0392] In one embodiment, the final loss information can be calculated according to the mathematical expression 3 described above.

[0393] The training process of the first encoder (12), the first decoder (14), the second encoder (22), and the second decoder (24) described with reference to FIGS. 17 and 18 can be performed by the video encoding device (1100). Then, the first encoder (12), the first decoder (14), the second encoder (22), and the second decoder (24) for which training has been completed can be transmitted to the video decoding device (400).

[0394] In one embodiment, the training process of the first encoder (12), the first decoder (14), the second encoder (22), and the second decoder (24) may be performed by a separate training device (e.g., a server). Then, the first encoder (12), the first decoder (14), the second encoder (22), and the second decoder (24), for which training has been completed, may be transmitted to the image decoding device (400) and the image encoding device (1100).

[0395] One embodiment aims to reduce the bitrate of a bitstream generated as a result of encoding an image.

[0396] One embodiment aims to improve the quality of a restored image generated by decoding a bitstream.

[0397] One embodiment aims to provide an AI-based end-to-end encoding / decoding system.

[0398] The technical problems to be achieved through the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure pertains from the description below.

[0399] A method for decoding an image according to one embodiment may include a step of obtaining feature data and filtering information of a preliminary optical flow from a bitstream.

[0400] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0401] A method for decoding an image according to one embodiment may include a step of obtaining a preliminary optical flow by applying feature data of the preliminary optical flow to a neural network-based first decoder (14).

[0402] A method of decoding an image according to one embodiment may include a step of generating an optical flow by applying a preliminary optical flow to a filter corresponding to filtering information.

[0403] A method for decoding an image according to one embodiment may include a step of generating a current restored image using previous data and optical flow.

[0404] According to an image decoding method according to one embodiment, as the quality of optical flow improves, the quality of the currently restored image can also improve.

[0405] In one embodiment, the image decoding method further includes a step of obtaining feature data of a residual image from a bitstream, and the step of generating a current restored image may include a step of obtaining a current restored image by applying an optical flow, feature data of the residual image, and previous data to a neural network-based second decoder (24).

[0406] In one embodiment, the current restored image can be restored more accurately based on a neural network.

[0407] In one embodiment, the image decoding method further includes a step of obtaining feature data of a residual image from a bitstream, and the step of generating a current restored image may include a step of obtaining a residual image by applying the feature data of the residual image to a neural network-based second decoder (24); a step of generating a predicted image from previous data based on optical flow; and a step of generating a current restored image by combining the predicted image and the residual image.

[0408] In one embodiment, the current restored image can be restored more accurately based on a neural network.

[0409] In one embodiment, the filter information further includes weight information, and the step of generating the optical flow may include the step of applying the preliminary optical flow to a filter corresponding to the filtering information to generate a filtered preliminary optical flow; and the step of combining the preliminary optical flow and the filtered preliminary optical flow according to the weight information to generate the optical flow.

[0410] In one embodiment, more accurate optical flow can be generated through weighting information.

[0411] In one embodiment, the filter information further includes weight information, and the step of generating the optical flow may include: generating a first filtered preliminary optical flow by applying the preliminary optical flow to a first filter indicated by the type information; generating a second filtered preliminary optical flow by applying the preliminary optical flow to a second filter indicated by the type information; and generating an optical flow by combining the first filtered preliminary optical flow and the second filtered preliminary optical flow according to the weight information.

[0412] In one embodiment, more accurate optical flow can be generated through weighting information.

[0413] In one embodiment, the type information of the filter may indicate at least one of a Gaussian filter, a median filter, a bilateral filter, or a neural network filter.

[0414] In one embodiment, a filter suitable for filtering preliminary optical flow is selected among several filters, thereby generating a more accurate optical flow.

[0415] In one embodiment, when the type information of the filter indicates a neural network filter, the parameter information indicates one of a plurality of neural networks of different types, and the step of generating the optical flow may include a step of obtaining the optical flow by applying a preliminary optical flow to the neural network indicated by the parameter information.

[0416] In one embodiment, a more accurate optical flow can be generated by filtering the preliminary optical flow based on a neural network.

[0417] A method for encoding an image according to one embodiment may include a step of obtaining feature data of a preliminary optical flow by applying a current image and previous data to a neural network-based first encoder (12).

[0418] A video encoding method according to one embodiment may include a step of obtaining a preliminary optical flow by applying feature data of the preliminary optical flow to a neural network-based first decoder (14).

[0419] A video encoding method according to one embodiment may include a step of selecting a filter used for filtering preliminary optical flow among a plurality of filters.

[0420] A method of encoding an image according to one embodiment may include a step of generating an optical flow by applying a preliminary optical flow to a selected filter.

[0421] A method of encoding an image according to one embodiment may include a step of encoding a current image using optical flow and previous data.

[0422] A method of encoding an image according to one embodiment may include generating a bitstream including feature data of a preliminary optical flow and filtering information for a filter.

[0423] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0424] According to an image encoding method according to one embodiment, the bitrate of a bitstream can be reduced as the quality of an optical flow is improved.

[0425] In one embodiment, the step of encoding the current image includes the step of applying the current image, previous data, and optical flow to a neural network-based second encoder (22) to obtain feature data of a residual image, wherein the feature data of the residual image can be included in the bitstream.

[0426] In one embodiment, the current image can be encoded more effectively based on a neural network.

[0427] In one embodiment, the step of encoding the current image includes the step of generating a predicted image from previous data based on optical flow; and the step of applying a residual image corresponding to the difference between the predicted image and the current image to a neural network-based second encoder (22) to obtain feature data of the residual image, wherein the feature data of the residual image can be included in the bitstream.

[0428] In one embodiment, the current image can be encoded more effectively based on a neural network.

[0429] In one embodiment, the step of selecting a filter may include: generating a plurality of optical flows by applying a preliminary optical flow to a plurality of filters; and selecting at least one filter from the plurality of filters by considering a difference between each of the plurality of predicted images generated based on each of the plurality of optical flows and the current image.

[0430] In one embodiment, a filter suitable for filtering preliminary optical flow is selected among several filters, thereby generating a more accurate optical flow.

[0431] In one embodiment, the step of selecting a filter may include the step of selecting at least one filter from among the plurality of filters by taking into account at least one of a difference between each of the plurality of current restored images generated corresponding to each of the plurality of filters and the current image, or a comparison result of bitrates of bitstreams generated corresponding to each of the plurality of filters.

[0432] In one embodiment, a filter suitable for filtering preliminary optical flow is selected among several filters, thereby generating a more accurate optical flow.

[0433] In one embodiment, the plurality of filters include neural network filters that utilize different types of neural networks, wherein the neural networks utilized in the neural network filters process the preliminary training optical flow to output the training optical flow, and the neural networks can be trained based on the results of a comparison between the training optical flow and the ground truth optical flow.

[0434] In one embodiment, a neural network can be trained to generate accurate optical flow.

[0435] A computer-readable recording medium according to one embodiment can record a bitstream.

[0436] In one embodiment, the bitstream may include feature data and filtering information of the preliminary optical flow.

[0437] In one embodiment, feature data of the preliminary optical flow can be obtained by applying the current image and previous data to a neural network-based first encoder (12).

[0438] In one embodiment, filtering information can be obtained by applying feature data of the preliminary optical flow to a neural network-based first decoder (14) to obtain a preliminary optical flow, and selecting a filter used for filtering the preliminary optical flow from among a plurality of filters.

[0439] In one embodiment, an optical flow is generated by applying a preliminary optical flow to a selected filter, and the current image can be encoded using the optical flow and previous data.

[0440] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0441] An image decoding device according to one embodiment may include an acquisition unit (410) that acquires feature data and filtering information of a preliminary optical flow from a bitstream.

[0442] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0443] An image decoding device according to one embodiment may include a prediction decoding unit (430) that obtains a preliminary optical flow by applying feature data of a preliminary optical flow to a neural network-based first decoder (14), generates an optical flow by applying the preliminary optical flow to a filter corresponding to filtering information, and generates a current restored image by using previous data and the optical flow.

[0444] According to an image decoding device according to one embodiment, as the quality of the optical flow improves, the quality of the currently restored image can also improve.

[0445] An image encoding device according to one embodiment may include a prediction encoding unit (1110) that obtains feature data of a preliminary optical flow by applying a current image and previous data to a neural network-based first encoder (12), obtains a preliminary optical flow by applying the feature data of the preliminary optical flow to a neural network-based first decoder (14), selects a filter used for filtering the preliminary optical flow among a plurality of filters, applies the preliminary optical flow to the selected filter to generate an optical flow, and encodes a current image by using the optical flow and previous data.

[0446] An image encoding device according to one embodiment may include a generation unit (1130) that generates a bitstream including feature data of a preliminary optical flow and filtering information for a filter.

[0447] In one embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0448] According to an embodiment of the video encoding device, the bitrate of a bitstream can be reduced as the quality of an optical flow is improved.

[0449] One embodiment can reduce the bitrate of a bitstream generated as a result of encoding an image.

[0450] One embodiment can improve the quality of a restored image generated by decoding a bitstream.

[0451] One embodiment may provide an AI-based end-to-end encoding / decoding system.

[0452] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0453] Meanwhile, the embodiments of the present disclosure described above can be written as a program that can be executed on a computer, and the written program can be stored in a storage medium that can be read by a device.

[0454] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0455] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0456] Above, the technical idea of ​​the present disclosure has been described in detail with reference to preferred embodiments, but the technical idea of ​​the present disclosure is not limited to the above embodiments, and various modifications and changes are possible by a person having ordinary knowledge in the art within the scope of the technical idea of ​​the present disclosure.

Claims

1. A step of obtaining feature data and filtering information of a preliminary optical flow from a bitstream, wherein the filtering information includes at least one of type information of a filter and parameter information of a filter. A step of obtaining a preliminary optical flow by applying the feature data of the preliminary optical flow to a first decoder (14) based on a neural network; A step of generating an optical flow by applying the preliminary optical flow to a filter corresponding to the filtering information; and An image decoding method, comprising the step of generating a current restored image using previous data and the optical flow.

2. In paragraph 1, The above video decryption method is, Further comprising a step of obtaining feature data of a residual image from the above bitstream, The steps for generating the above current restored image are: An image decoding method comprising a step of applying the optical flow, feature data of the residual image, and the previous data to a neural network-based second decoder (24) to obtain the current restored image.

3. In any one of paragraphs 1 and 2, The above video decryption method is, Further comprising a step of obtaining feature data of a residual image from the above bitstream, The steps for generating the above current restored image are: A step of obtaining a residual image by applying feature data of the residual image to a neural network-based second decoder (24); A step of generating a predicted image from the previous data based on the optical flow; and An image decoding method, comprising the step of combining the predicted image and the residual image to generate the current restored image.

4. In any one of the clauses 1 to 3, The above filter information further includes weight information, The steps of generating the above optical flow are: A step of generating a filtered preliminary optical flow by applying the preliminary optical flow to a filter corresponding to the filtering information; and An image decoding method, comprising the step of generating the optical flow by combining the preliminary optical flow and the filtered preliminary optical flow according to the weight information.

5. In any one of paragraphs 1 to 4, The above filter information further includes weight information, The steps of generating the above optical flow are: A step of generating a first filtered preliminary optical flow by applying the preliminary optical flow to the first filter indicated by the above type information; A step of generating a second filtered preliminary optical flow by applying the preliminary optical flow to the second filter indicated by the above type information; and An image decoding method, comprising the step of generating the optical flow by combining the first filtered preliminary optical flow and the second filtered preliminary optical flow according to the weight information.

6. In any one of paragraphs 1 to 5, Information on the type of the above filter is: An image decoding method, wherein the image decoding method comprises at least one of a Gaussian filter, a median filter, a bilateral filter, or a neural network filter.

7. In any one of paragraphs 1 to 6, If the type information of the above filter indicates a neural network filter, the parameter information indicates one of multiple neural networks of different types, The steps of generating the above optical flow are: An image decoding method, comprising a step of obtaining the optical flow by applying the preliminary optical flow to a neural network indicated by the parameter information.

8. A step of obtaining feature data of preliminary optical flow by applying the current image and previous data to a first encoder (12) based on a neural network; A step of obtaining a preliminary optical flow by applying the feature data of the preliminary optical flow to a first decoder (14) based on a neural network; A step of selecting a filter among a plurality of filters to be used for filtering the preliminary optical flow; A step of generating an optical flow by applying the preliminary optical flow to the selected filter; A step of encoding the current image using the optical flow and the previous data; and A step of generating a bitstream including feature data of the preliminary optical flow and filtering information for the filter, A video encoding method, wherein the filtering information includes at least one of type information of the filter and parameter information of the filter.

9. In paragraph 8, The step of encoding the current image above is: A step of obtaining feature data of a residual image by applying the current image, the previous data, and the optical flow to a neural network-based second encoder (22), A method for encoding an image, wherein feature data of the residual image is included in the bitstream.

10. In any one of paragraphs 8 to 9, The step of encoding the current image above is: A step of generating a predicted image from the previous data based on the optical flow; and It includes a step of obtaining feature data of the residual image by applying the residual image corresponding to the difference between the predicted image and the current image to a neural network-based second encoder (22), A method for encoding an image, wherein feature data of the residual image is included in the bitstream.

11. In any one of clauses 8 to 10, The steps for selecting the above filter are: A step of generating a plurality of optical flows by applying the preliminary optical flows to the plurality of filters; and An image encoding method, comprising a step of selecting at least one filter among the plurality of filters by considering a difference between each of the plurality of predicted images generated based on each of the plurality of optical flows and the current image.

12. In any one of the clauses 8 to 11, The steps for selecting the above filter are: A method for encoding an image, comprising the step of selecting at least one filter from among the plurality of filters by considering at least one of a difference between each of the plurality of current restored images generated corresponding to each of the plurality of filters and the current image, or a comparison result of bitrates of bitstreams generated corresponding to each of the plurality of filters.

13. In any one of paragraphs 8 to 12, The above multiple filters include neural network filters using different types of neural networks, The neural networks used in the above neural network filter process the preliminary optical flow for training and output the optical flow for training. A method for image encoding, wherein the above neural networks are trained based on the comparison results between the training optical flow and the ground truth optical flow.

14. In a computer-readable recording medium that records a bitstream, The above bitstream includes feature data and filtering information of the preliminary optical flow, The feature data of the above preliminary optical flow is obtained by applying the current image and previous data to the first encoder (12) based on a neural network, The above filtering information is obtained by applying the feature data of the preliminary optical flow to the first decoder (14) based on a neural network to obtain the preliminary optical flow, and selecting a filter used for filtering the preliminary optical flow among a plurality of filters. The optical flow is generated by applying the preliminary optical flow to the selected filter, and the current image is encoded using the optical flow and the previous data. A recording medium, wherein the filtering information includes at least one of type information of the filter and parameter information of the filter.

15. An acquisition unit (410) for acquiring feature data and filtering information of a preliminary optical flow from a bitstream; the filtering information includes at least one of type information of the filter and parameter information of the filter, An image decoding device including a prediction decoding unit (430) that obtains a preliminary optical flow by applying feature data of the preliminary optical flow to a first decoder (14) based on a neural network, generates an optical flow by applying the preliminary optical flow to a filter corresponding to the filtering information, and generates a current restored image by using previous data and the optical flow.

Images