Method and apparatus for encoding image by using filted optical flow, and method and apparatus for decoding image

Abstract

Disclosed is an image decoding method according to an embodiment, comprising the steps of: acquiring feature data and filtering information of a preliminary optical stream from a bit stream, the filtering information including at least one of type information of a filter and parameter information of the filter; acquiring the preliminary optical flow by applying feature data of the preliminary optical flow to a first decoder based on a neural network; generating an optical flow by applying the preliminary optical flow to the filter corresponding to the filtering information; a current reconstructed image is generated by using previous data and the optical flow.

Description

Technical Field

[0001] This disclosure relates to the encoding and decoding of images. Specifically, this disclosure relates to techniques for encoding and decoding images using artificial intelligence (AI) (e.g., neural networks). Background Technology

[0002] In codecs such as H.264 Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC), images can be divided into blocks, and each block can be predictively encoded and predictedally decoded via inter-frame prediction or intra-frame prediction.

[0003] Intra-frame prediction is a method of compressing images by removing spatial redundancy within the image, while inter-frame prediction is a method of compressing images by removing temporal redundancy between images.

[0004] A typical example of inter-frame prediction is motion estimation coding. Motion estimation coding uses a reference image to predict blocks in the current image. A reference block most similar to the current block is searched within a predetermined range in the reference image using a predefined evaluation function. The current block is predicted based on the reference block, and the predicted block is subtracted from the current block to generate and encode residual blocks.

[0005] Recently, techniques for encoding / decoding images using artificial intelligence (AI) have been proposed, and methods for effectively encoding / decoding images using AI (e.g., neural networks) are needed. Summary of the Invention

[0006] Solution to the problem The image decoding method according to the embodiment may include: obtaining feature data and filtering information of preliminary optical flow from a bitstream.

[0007] In an embodiment, the filtering information may include at least one of: filter type information or filter parameter information.

[0008] The image decoding method according to an embodiment may include obtaining the preliminary optical flow by applying the feature data of the preliminary optical flow to a first decoder based on a neural network.

[0009] The image decoding method according to an embodiment may include generating an optical flow by applying the preliminary optical flow to a filter corresponding to the filtering information.

[0010] The image decoding method according to the embodiment may include generating a current reconstructed image by using previous data and the optical flow.

[0011] The image encoding method according to the embodiment may include obtaining preliminary optical flow feature data by applying the current image and previous data to a first encoder based on a neural network.

[0012] The image encoding method according to the embodiment may include obtaining the preliminary optical flow by applying the feature data of the preliminary optical flow to a first decoder based on a neural network.

[0013] The image encoding method according to the embodiment may include: selecting a filter from a plurality of filters for filtering the preliminary optical flow.

[0014] The image encoding method according to an embodiment may include generating an optical flow by applying the preliminary optical flow to a selected filter.

[0015] The image encoding method according to the embodiment may include encoding the current image by using the optical flow and the previous data.

[0016] The image encoding method according to the embodiment may include: generating a bitstream that includes feature data of the preliminary optical flow and filtering information for the filter.

[0017] In an embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0018] The computer-readable recording medium according to the embodiments can record bit streams.

[0019] In an embodiment, the bitstream may include characteristic data and filtering information of the preliminary optical flow.

[0020] In one embodiment, the feature data of the initial optical flow can be obtained by applying the current image and previous data to a first encoder based on a neural network.

[0021] In an embodiment, the initial optical flow can be obtained by applying the feature data of the initial optical flow to a first decoder based on a neural network, and filtering information can be obtained by selecting a filter from a plurality of filters to filter the initial optical flow.

[0022] In an embodiment, when the initial optical flow is applied to the selected filter, an optical flow can be generated, and the current image can be encoded using the optical flow and the previous data.

[0023] In an embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0024] The image decoding apparatus according to an embodiment may include an acquisition unit, wherein the acquisition unit acquires feature data and filtering information of a preliminary optical flow from a bitstream.

[0025] In an embodiment, the filtering information may include at least one of filter type information or filter parameter information.

[0026] The image decoding apparatus according to an embodiment may include a predictive decoding unit, wherein the predictive decoding unit obtains the preliminary optical flow by applying feature data of the preliminary optical flow to a first decoder based on a neural network, generates an optical flow by applying the preliminary optical flow to a filter corresponding to the filtering information, and generates a current reconstructed image by using previous data and the optical flow.

[0027] The image encoding apparatus according to an embodiment may include a predictive encoding unit, wherein the predictive encoding unit obtains feature data of a preliminary optical flow by applying a current image and previous data to a first encoder based on a neural network, obtains the preliminary optical flow by applying the feature data of the preliminary optical flow to a first decoder based on a neural network, selects a filter from a plurality of filters for filtering the preliminary optical flow, generates an optical flow by applying the preliminary optical flow to the selected filter, and encodes the current image using the optical flow and the previous data.

[0028] The image encoding apparatus according to an embodiment may include a generation unit, wherein the generation unit generates a bitstream including feature data of the preliminary optical flow and filtering information for the filter.

[0029] In an embodiment, the filtering information may include at least one of filter type information or filter parameter information. Attached Figure Description

[0030] Figure 1 This is a diagram illustrating AI-based optical flow encoding and decoding processing according to an embodiment.

[0031] Figure 2 This is a diagram illustrating AI-based image encoding and decoding processing according to an embodiment.

[0032] Figure 3 This is a diagram illustrating AI-based image encoding and decoding processing according to an embodiment.

[0033] Figure 4 This is a diagram illustrating the configuration of an image decoding device according to an embodiment.

[0034] Figure 5 This is a diagram illustrating the configuration of the obtaining unit according to an embodiment.

[0035] Figure 6 A table for interpreting filter information is shown according to an embodiment.

[0036] Figure 7 A table for interpreting filter information is shown according to an embodiment.

[0037] Figure 8 This is a diagram illustrating a neural network used for a neural network filter according to an embodiment.

[0038] Figure 9 This is a diagram illustrating the semantics of filtering information according to an embodiment.

[0039] Figure 10 This is a flowchart of an image decoding method according to an embodiment.

[0040] Figure 11 This is a diagram illustrating the configuration of an image encoding device according to an embodiment.

[0041] Figure 12 This is a diagram illustrating the configuration of the generation unit according to an embodiment.

[0042] Figure 13 This is a diagram illustrating the method for selecting a filter by a predictive coding unit according to an embodiment.

[0043] Figure 14 This is a diagram illustrating the method for selecting a filter by a predictive coding unit according to an embodiment.

[0044] Figure 15 This is a flowchart of an image encoding method according to an embodiment.

[0045] Figure 16 This is a diagram illustrating a method for training a neural network that can be used as a neural network filter according to an embodiment.

[0046] Figure 17 This is a diagram illustrating a method for training a first encoder, a first decoder, a second encoder, and a second decoder according to an embodiment.

[0047] Figure 18 This is a diagram illustrating a method for training a first encoder, a first decoder, a second encoder, and a second decoder according to an embodiment. Detailed Implementation

[0048] Optimal mode of carrying out the present invention The image decoding method according to the embodiment may include: obtaining feature data and filtering information of preliminary optical flow from a bitstream.

[0049] In this embodiment, the filtering information may include at least one of the filter type information or the filter parameter information.

[0050] The image decoding method according to the embodiment may include obtaining the initial optical flow by applying feature data of the initial optical flow to a first decoder based on a neural network.

[0051] The image decoding method according to the embodiment may include generating an optical flow by applying a preliminary optical flow to a filter corresponding to filtering information.

[0052] The image decoding method according to the embodiment may include generating a current reconstructed image by using previous data and optical flow.

[0053] Methods for use in this invention Because this disclosure allows for various modifications and numerous embodiments, embodiments will be shown in the accompanying drawings and described in detail in the written description. However, this is not intended to limit this disclosure to a particular mode of practice, and it should be understood that all modifications, equivalents, and alternatives that do not depart from the spirit and scope of this disclosure are included herein.

[0054] In the description of the embodiments, certain detailed explanations of the related technologies are omitted when it is believed that such detailed explanations may unnecessarily obscure the essence of this disclosure. Furthermore, numbers used in the description of the embodiments (e.g., first, second, etc.) may correspond to identification signals to distinguish one component from another.

[0055] In this disclosure, expressions such as "at least one of a, b or c" may mean "a", "b", "c", "a and b", "a and c", "b and c", "all of a, b and c" or variations thereof.

[0056] In this disclosure, unless otherwise expressly stated, when a component is “connected” or “connected to” another component, the component is in contact with or connected to that other component directly or through the other component.

[0057] In this disclosure, components described as “units” or “modules” can be implemented such that two or more components are combined into one component, or a component is subdivided into two or more components. Furthermore, each component described below, in addition to its primary function, may perform some or all of the functions of other components, and some of the primary functions of each component may be performed specifically by other components.

[0058] In this disclosure, "image" can refer to a picture, still image, frame, moving image or video composed of multiple consecutive still images.

[0059] In this disclosure, the "current image" can be an image that is currently the target of encoding and decoding. In embodiments, the current image can be a block partitioned from the image that is currently the target of encoding and decoding. For example, the current image may correspond to a slice, tile, maximum coding unit, coding unit, prediction unit, or transform unit partitioned from the image.

[0060] In this disclosure, a "previous image" can be an image that has an earlier encoding order and an earlier decoding order than the current image. In embodiments, a previous image may correspond to a stripe, parallel block, maximum coding unit, coding unit, prediction unit, or transform unit divided from an image.

[0061] In this disclosure, "neural network" can be a representative example of an artificial neural network model that mimics brain neurons, and neural networks are not limited to artificial neural network models based on specific algorithms. Neural networks can be referred to as deep neural networks.

[0062] In this disclosure, "parameters of a neural network" can be values ​​used in the computation process of each layer that constitutes the neural network. For example, the parameters of the neural network can be used when input values ​​are applied to a predetermined calculation formula. The parameters of the neural network can be updated as needed using individual training data, values ​​set as a result of training.

[0063] In this disclosure, "feature data" can refer to data obtained when a neural network-based encoder processes input data. Feature data can be one-dimensional or two-dimensional data including various samples. Feature data can be referred to as latent representation. Feature data can represent latent features in the data output by the decoder described below.

[0064] In this disclosure, a "sample point" can refer to a sampling location assigned to image or feature data and to be processed. For example, pixels in a frame in the spatial domain can correspond to sample points. A unit comprising multiple sample points can be defined as a block.

[0065] Figure 1 This is a diagram illustrating AI-based optical flow encoding and decoding processing according to an embodiment.

[0066] Inter-frame prediction can be a process of encoding and decoding the current image 10 using temporal redundancy between the current image 10 and the previously reconstructed image. The previously reconstructed image can be an image obtained by decoding a previous image.

[0067] The positional difference (or motion vector) between a block or sample in the current image 10 and a reference block or sample in a previously reconstructed image can be used for inter-frame prediction of the current image 10. This positional difference can be referred to as optical flow. Optical flow can be defined as the set of motion vectors corresponding to a sample or block in an image.

[0068] Optical flow can indicate how the position of a sample point in a previously reconstructed image changes in the current image 10, or the position of a sample point that is the same as or similar to a sample point in the current image 10 in a previously reconstructed image.

[0069] For example, when the same or most similar sample point located at (1,1) in the current image 10 is located at (2,1) in the previously reconstructed image, the optical flow or motion vector of the sample point located at (1,1) can be derived as (1 (=2-1), 0 (=1-1)).

[0070] In the process of encoding and decoding optical flow using AI according to the embodiment, the optical flow of the current image 10 can be obtained using the first encoder 12 and the first decoder 14.

[0071] The first encoder 12 and the first decoder 14 can be implemented as neural networks. The first encoder 12 and the first decoder 14 can be understood as neural networks used to extract optical flow.

[0072] In one embodiment, the first encoder 12 may be referred to as an optical flow encoder, and the first decoder 14 may be referred to as an optical flow decoder. In another embodiment, the first encoder 12 may be referred to as a motion vector encoder, and the first decoder 14 may be referred to as a motion vector decoder.

[0073] Reference Figure 1 The previous data 20 and the current image 10 can be input into the first encoder 12. The first encoder 12 can process the current image 10 and the previous data 20 according to the parameters set as a result of training to output preliminary optical flow feature data w.

[0074] In this disclosure, the prior data 20 may include at least one of the following: a prior reconstructed image, feature data of the prior reconstructed image, a prior predicted image for generating the prior reconstructed image, feature data of the prior predicted image, a prior preliminary optical flow for generating the prior reconstructed image, feature data of the prior preliminary optical flow, a prior residual image for generating the prior reconstructed image, or feature data of the prior residual image.

[0075] In an embodiment, the previous data 20 may refer to the previously reconstructed image itself or data obtained in the process of generating the previously reconstructed image.

[0076] In an embodiment, the feature data of the previously reconstructed image, as an example of the previous data 20, may be feature data output from a predetermined layer (a first layer or an intermediate layer), wherein the predetermined layer is not the neural network that constitutes the previously reconstructed image (e.g., Figure 3 The final layer in the second decoder 24 shown.

[0077] In an embodiment, a previously predicted image, as an example of previous data 20, can be used to generate a previously reconstructed image, and can be obtained by warping an image reconstructed earlier than the previously reconstructed image. Feature data of the previously predicted image can be obtained by processing the previously predicted image through a neural network. In an embodiment, feature data of the previously predicted image can be obtained by warping feature data of an image reconstructed earlier than the previously reconstructed image.

[0078] The feature data w of the preliminary optical flow can be input into the first decoder 14. The first decoder 14 can output the preliminary optical flow g by processing the input feature data w according to the parameters set as a training result.

[0079] In an embodiment, specific processing may be performed on the initial optical flow g to encode and decode the current image 10.

[0080] A certain process is performed on the initial optical flow g because there is a possibility that the initial optical flow g is inaccurate.

[0081] See below for reference. Figure 17 and Figure 18 When the first encoder 12 and the first decoder 14 are trained in the direction of decreasing bit rate of the bitstream, a relatively low-quality initial optical flow g may be output. Furthermore, due to computational load and latency, there is a limitation on the number of layers that can be included in the first encoder 12 and the first decoder 14, making it difficult to generate a high-quality initial optical flow g. Because of the low-quality initial optical flow g, the size of the residual data may increase, and therefore the bit rate of the bitstream may increase.

[0082] In this embodiment, the bit rate of the bit stream can be reduced by performing post-processing on the initial optical flow g.

[0083] like Figure 1 As shown, filter 30 can be used for post-processing of the initial optical flow g. Filter 30 can be selected from multiple filters according to predetermined criteria. The filter selection method is described below.

[0084] When the initial optical flow g is filtered by filter 30, an optical flow h can be generated. In an embodiment, when the initial optical flow g is applied to filter 30, at least some of the sample values ​​of the initial optical flow g are changed, and an optical flow h including the changed sample values ​​can be obtained. In an embodiment, the optical flow h may be referred to as the filtered optical flow.

[0085] although Figure 1 The diagram shows that previous data 20 and current image 10 are applied to the first encoder 12 and feature data w of the initial optical flow is obtained from the first encoder 12, but various methods may exist for obtaining feature data w of the initial optical flow.

[0086] For example, preliminary optical flow can be extracted from previous data 20 and current image 10 using a known optical flow extraction method, and the extracted preliminary optical flow can be applied to a first encoder 12. Feature data w of the preliminary optical flow output from the first encoder 12 can be input to a first decoder 14.

[0087] Furthermore, for example, multiple neural networks connected in series or parallel can be used to obtain the feature data w of the initial optical flow, and multiple neural networks connected in series or parallel can be used to obtain the initial optical flow g from the feature data w of the initial optical flow. Multiple neural networks connected in series can refer to the case where the output of any one neural network is input to another neural network. Multiple neural networks connected in parallel can refer to the case where any one neural network and another neural network process the input data independently, and the output data of one neural network is combined with the output data of another neural network.

[0088] In an embodiment, when Figure 1 When the optical flow encoding and decoding processes shown are implemented by an encoding device and a decoding device, the encoding device can obtain the feature data w of the initial optical flow using the current image 10 and the previous data 20. The encoding device can generate a bit stream including the feature data w of the initial optical flow and information about the filter 30 applied to the initial optical flow g (hereinafter referred to as filter information), and send the generated bit stream to the decoding device.

[0089] The decoding device can obtain the feature data w and filtering information of the initial optical flow from the bit stream. The decoding device can obtain the initial optical flow g by processing the feature data w of the initial optical flow using the first decoder 14, and obtain the optical flow h by filtering the initial optical flow g according to the filtering information.

[0090] When the optical flow h for the current image 10 is obtained, the current image 10 can be encoded and decoded based on the optical flow h.

[0091] In an embodiment, the current image 10 can be encoded and decoded based on optical flow h and previous data 20.

[0092] In this embodiment, in order to encode and decode the current image 10, motion compensation can be applied to the previous data 20 based on optical flow h. A predicted image similar to the current image 10 can be generated through motion compensation.

[0093] In this embodiment, the predicted image may be determined as the current reconstructed image.

[0094] In this embodiment, a residual image corresponding to the difference between the current image 10 and the predicted image can be obtained, and when the residual image and the predicted image are combined with each other, the current reconstructed image can be generated. The data of the residual image can be sent from the encoder to the decoder via a bitstream.

[0095] The following reference Figure 2 and Figure 3 This describes the encoding and decoding process for the current image 10.

[0096] Figure 2 This is a diagram illustrating AI-based image encoding and decoding processing according to an embodiment.

[0097] In the process of encoding and decoding the current image 10 using AI, according to an embodiment, a second encoder 22 and a second decoder 24 may be used.

[0098] The second encoder 22 and the second decoder 24 can be implemented by a neural network. The second encoder 22 and the second decoder 24 can be understood as a neural network for encoding and decoding the residual image 60 corresponding to the difference between the current image 10 and the predicted image 50.

[0099] In an embodiment, the second encoder 22 may be referred to as a residual encoder, and the second decoder 24 may be referred to as a residual decoder.

[0100] In this embodiment, since the second encoder 22 and the second decoder 24 are used to encode and decode the pixel values ​​of the current image 10, the second encoder 22 may be referred to as a pixel encoder and the second decoder 24 may be referred to as a pixel decoder.

[0101] Reference Figure 2 The previous data 20 can be distorted by warp 40 based on optical flow h, and the predicted image 50 can be obtained as a result of warp 40. Warp 40 can be a type of geometric transformation that moves the position of sample points within the image.

[0102] exist Figure 2 In the illustrated embodiment, to generate the predicted image 50, the previously reconstructed image can be used as the previous data 20. Therefore, the predicted image 50 can be obtained by warping 40 onto the previously reconstructed image.

[0103] A predicted image 50 similar to the current image 10 can be obtained by applying warping 40 to the previous data 20 based on optical flow h, which represents the relative positional relationship between samples in the previous data 20 and samples in the current image 10.

[0104] For example, when a sample located at (1,1) in the previous data 20 is most similar to a sample located at (2,1) in the current image 10, the position of the sample located at (1,1) in the previous data 20 can be changed to (2,1) based on optical flow h by twist 40.

[0105] In one embodiment, the distortion 40 can be implemented based on a neural network, in which case the previous data 20 and optical flow h can be input into the neural network for the distortion 40, and the predicted image 50 can be output from the neural network.

[0106] Since the predicted image 50 generated from the previous data 20 is not the current image 10 itself, a residual image 60 corresponding to the difference between the predicted image 50 and the current image 10 can be obtained.

[0107] For example, the residual image 60 can be obtained by subtracting the sample values ​​in the predicted image 50 from the sample values ​​in the current image 10.

[0108] The residual image 60 can be input into the second encoder 22. The second encoder 22 can output the feature data v of the residual image 60 by processing the residual image 60 according to the parameters set as training results.

[0109] The feature data v of the residual image 60 can be input to the second decoder 24. The second decoder 24 can output the reconstructed residual image 70 by processing the input feature data v according to the parameters set as training results.

[0110] When the predicted image 50 and the reconstructed residual image 70 are combined with each other, the current reconstructed image 80 can be obtained. For example, the current reconstructed image 80 can be obtained by adding the sample values ​​in the predicted image 50 to the sample values ​​in the reconstructed residual image 70.

[0111] when Figure 2 When the image encoding and decoding processes shown are implemented by an encoding device and a decoding device, the encoding device can obtain the feature data v of the residual image 60 by using the second encoder 22. The encoding device can generate a bitstream including the feature data v of the residual image 60 and send the generated bitstream to the decoding device.

[0112] The decoding device can obtain feature data v of the residual image 60 from the bitstream. The decoding device can obtain a reconstructed residual image 70 by processing the feature data v of the residual image 60 using a second decoder 24, and obtain a current reconstructed image 80 by combining a predicted image 50 generated from the previous data 20 based on optical flow h with the reconstructed residual image 70.

[0113] Figure 3 This is a diagram illustrating AI-based image encoding and decoding processing according to an embodiment.

[0114] Reference Figure 3 The current image 10, previous data 20, and optical flow h can be input into the second encoder 22. The second encoder 22 can output feature data v of the residual image by processing the input data according to parameters set as a training result.

[0115] exist Figure 2 In the illustrated embodiment, the residual image 60, corresponding to the difference between the predicted image 50 generated by warping 40 and the current image 10, is input to the second encoder 22, while... Figure 3 In the illustrated embodiment, the second encoder 22 can be understood to perform the distortion processing of the previous data 20 and the processing to obtain the residual image together.

[0116] exist Figure 3 Since the residual image 60 is not input to the second encoder 22, in this embodiment, it can be referred to as feature data of the sample points of the current image 10 that can be output from the second encoder 22.

[0117] In one embodiment, a predicted image generated from previous data 20 (e.g., a previously reconstructed image) based on optical flow h and the current image 10 can be input to a second encoder 22.

[0118] In one embodiment, feature data of the predicted image generated from feature data of a previously reconstructed image based on optical flow h, along with the current image 10, can be input to the second encoder 22.

[0119] In an embodiment, the predicted image generated from the previously reconstructed image based on optical flow h, the feature data of the predicted image generated from the feature data of the previously reconstructed image based on optical flow h, and the current image 10 can be input to the second encoder 22.

[0120] The previous data 20, optical flow h, and feature data v of the residual image can be input into the second decoder 24. The second decoder 24 can output the current reconstructed image 80 by processing the input data according to the parameters set as a result of training.

[0121] exist Figure 2 In the illustrated embodiment, the current reconstructed image 80 is obtained by combining the predicted image 50 generated by warping 40 with the residual image 70 output from the second decoder 24, while Figure 3 In the illustrated embodiment, the second decoder 24 can be understood to perform distortion processing on the previous data 20 and combined processing of the residual image and the predicted image together.

[0122] In one embodiment, a predicted image 50 generated from previous data 20 (e.g., a previously reconstructed image) based on optical flow h and feature data v of the residual image can be input to a second decoder 24.

[0123] In one embodiment, feature data of the predicted image generated from feature data of the previously reconstructed image based on optical flow h, and feature data v of the residual image, can be input into the second decoder 24.

[0124] In an embodiment, the predicted image generated from the previously reconstructed image based on optical flow h, the feature data of the predicted image generated from the feature data of the previously reconstructed image based on optical flow h, and the feature data v of the residual image can be input to the second decoder 24.

[0125] when Figure 3 When the image encoding and decoding processes shown are implemented by an encoding device and a decoding device, the encoding device can obtain the feature data v of the residual image by using a second encoder 22. The encoding device can generate a bitstream including the feature data v of the residual image and send the generated bitstream to the decoding device.

[0126] The decoding device can obtain the feature data v of the residual image from the bitstream. The decoding device can obtain the current reconstructed image 80 by processing the feature data v of the residual image, the previous data 20, and the optical flow h using the second decoder 24.

[0127] Although reference Figure 1 The process of encoding and decoding optical flow h is described, and references are made. Figure 2 and Figure 3 The encoding and decoding of the current image 10 is described separately, but the encoding and decoding of the optical flow h can be understood as part of the encoding and decoding of the current image 10.

[0128] Figure 4 This is a diagram illustrating the configuration of an image decoding device 400 according to an embodiment.

[0129] Reference Figure 4 The image decoding device 400 may include an acquisition unit 410 and a prediction decoding unit 430.

[0130] The acquisition unit 410 and the prediction / decoding unit 430 may be implemented by at least one processor. The acquisition unit 410 and the prediction / decoding unit 430 may operate according to at least one instruction stored in memory.

[0131] although Figure 4 The acquisition unit 410 and the prediction decoding unit 430 are shown separately, but they can be implemented by a single processor. For example, the acquisition unit 410 and the prediction decoding unit 430 can be implemented by a dedicated processor, or by a combination of a general-purpose processor (such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU)) and software.

[0132] The acquisition unit 410 and the prediction decoding unit 430 may be configured with multiple processors. For example, the acquisition unit 410 and the prediction decoding unit 430 may be implemented by a combination of dedicated processors, or by a combination of multiple general-purpose processors (such as AP, CPU or GPU) and software.

[0133] In one embodiment, the image decoding device 400 may use a first decoder 14 and a second decoder 24 to generate the current reconstructed image. The first decoder 14 and the second decoder 24 may be stored in a memory. In another embodiment, the first decoder 14 and the second decoder 24 may be implemented by an AI processor.

[0134] The obtaining unit 410 can obtain a bitstream including the encoding result for the current image.

[0135] The obtaining unit 410 can receive a bitstream from an image encoding device via a network. In an embodiment, the obtaining unit 410 can obtain the bitstream from a data storage medium, wherein the data storage medium includes magnetic media (such as hard disks, floppy disks, and magnetic tapes), optical recording media (such as CD-ROMs and DVDs), and magneto-optical media (such as floppy disks).

[0136] The obtaining unit 410 can obtain data corresponding to the encoding result for the current image by parsing the bit stream.

[0137] In an embodiment, the data corresponding to the encoding result for the current image may include at least one of the following: feature data of the preliminary optical flow, feature data of the residual image, or filtering information.

[0138] In an embodiment, the obtaining unit 410 can obtain the feature data of the preliminary optical flow, the feature data of the residual image, and the filtering information by the following operations: obtaining a first bit stream corresponding to the feature data of the preliminary optical flow and the filtering information, and a second bit stream corresponding to the feature data of the residual image, and parsing each bit stream in the first bit stream and the second bit stream.

[0139] Data corresponding to the encoding result for the current image can be sent to the prediction decoding unit 430, and the prediction decoding unit 430 can use the obtained data to generate a current reconstructed image corresponding to the current image.

[0140] In one embodiment, the currently reconstructed image may be sent to a display device for reproduction.

[0141] Figure 5 This is a diagram illustrating the configuration of the acquisition unit 410 according to an embodiment.

[0142] Reference Figure 5The obtaining unit 410 may include an entropy decoding unit 510 and an inverse quantization unit 530.

[0143] The entropy decoding unit 510 can obtain the feature data of the preliminary optical flow, the feature data of the residual image, and the filtering information by entropy encoding the bit stream.

[0144] In this embodiment, at least one of the feature data of the preliminary optical flow, the feature data of the residual image, or the filtering information can be quantized. In this case, the dequantization unit 530 can obtain dequantized data by dequantizing the quantized data. The dequantized data can be sent to the prediction decoding unit 430.

[0145] In the embodiment, when the feature data of the initial optical flow and the feature data of the residual image are quantized, the dequantization unit 530 can obtain the dequantized feature data of the initial optical flow and the dequantized feature data of the residual image by dequantizing the quantized feature data of the initial optical flow and the quantized feature data of the residual image.

[0146] In an embodiment, the obtaining unit 410 may further include an inverse transform unit. The inverse transform unit can perform an inverse transform from the frequency domain to the spatial domain on the inverse quantization data output from the inverse quantization unit 530.

[0147] When the image encoding device 1100, which will be described below, transforms the data corresponding to the encoding result for the current image from the spatial domain to the frequency domain, the inverse transform unit can inverse transform the inverse quantized data output from the inverse quantization unit 530 from the frequency domain to the spatial domain.

[0148] In this embodiment, the obtaining unit 410 may not include the inverse quantization unit 530. In other words, data corresponding to the encoding result for the current image can be obtained through the processing of the entropy decoding unit 510.

[0149] Return to reference Figure 4 The prediction decoding unit 430 can generate the current reconstructed image by using the data received from the acquisition unit 410.

[0150] In one embodiment, the prediction decoding unit 430 can obtain the initial optical flow by applying feature data of the initial optical flow to the first decoder 14. The prediction decoding unit 430 can also obtain the optical flow by filtering the initial optical flow based on a filter indicated by filtering information.

[0151] In one embodiment, the prediction decoding unit 430 may generate the current reconstructed image based on optical flow and previous data.

[0152] For example, the prediction decoding unit 430 can generate the current reconstructed image by performing motion compensation on previous data based on optical flow.

[0153] Furthermore, for example, the prediction decoding unit 430 can generate a predicted image by warping previous data based on optical flow. The prediction decoding unit 430 can obtain a reconstructed residual image by applying feature data from the residual image to the second decoder 24. The prediction decoding unit 430 can generate a current reconstructed image by combining the predicted image with the reconstructed residual image.

[0154] Furthermore, for example, the prediction decoding unit 430 can obtain the current reconstructed image by applying feature data of optical flow, previous data, and residual image to the second decoder 24.

[0155] Furthermore, for example, the prediction decoding unit 430 can obtain the current reconstructed image by applying feature data of the prediction image and residual image generated from previous data based on optical flow to the second decoder 24.

[0156] The following reference Figure 6 and Figure 7 The description describes the filtering process of the initial optical flow performed by the prediction decoding unit 430.

[0157] Figure 6 A table for interpreting filter information is shown according to an embodiment.

[0158] In an embodiment, the prediction decoding unit 430 can filter the initial optical flow by using a filter indicated by filtering information.

[0159] The filtering information specifies the filter used to filter the initial optical flow and may include at least one of, for example, filter type information, filter parameter information, or weight information.

[0160] In an embodiment, the filter type information may indicate the type of filter used to filter the initial optical flow. For example, the filter type information may indicate at least one of a Gaussian filter, a median filter, a bilateral filter, or a neural network filter.

[0161] When the filter type information indicates a Gaussian filter, the prediction decoding unit 430 can filter the initial optical flow by using a Gaussian filter.

[0162] Figure 6 The filter type information shown is merely an example, and various filters can be used to filter the initial optical flow. For example, the prediction decoding unit 430 can use various types of filters capable of filtering sample values ​​in two-dimensional data.

[0163] In this embodiment, the filter type information may indicate multiple filters. For example, the filter type information may indicate a Gaussian filter and a neural network filter.

[0164] In an embodiment, when there is an agreement between the image decoding device 400 and the image encoding device regarding which type of filter will be used, the filtering information may not include filter type information.

[0165] In this embodiment, the filter parameter information may indicate the values ​​of the filter parameters. Although the Gaussian filter is the same, the same Gaussian filter may have different filtering effects depending on the values ​​of the parameters. However, the image encoding device may send parameter information indicating the values ​​of the parameters to be set in the Gaussian filter to the image decoding device 400 via a signal.

[0166] The parameters of a filter can vary depending on the type of filter. For example, such as Figure 6 As shown, the parameters of a Gaussian filter may include the size of the filter kernel, the standard deviation σx in the x-axis direction (or horizontal direction), and the standard deviation σy in the y-axis direction (or vertical direction). Furthermore, for example, the parameters of a median filter may include the size of the filter kernel, and the parameters of a Laplace filter may include the size and standard deviation of the filter kernel. Additionally, for example, the parameters of a neural network filter may refer to at least one of a plurality of neural networks that will be used as a neural network filter.

[0167] In an embodiment, the parameters of the filter may be referred to as filter settings, and the values ​​of the parameters may be referred to as setting values.

[0168] In an embodiment, when the filter type information indicates multiple filters, the parameter information of each of the multiple filters can be included in the filtering information.

[0169] In an embodiment, the filter's parameter information may include an index or flag indicating any of a plurality of candidate values ​​that can be set for a particular parameter. For example, the parameter information for a Gaussian filter may include an index indicating any of {3,5,7} that will be used as the kernel size, an index indicating any of {0,1 / 2,1,2,4,8} that will be used as the standard deviation in the x-axis direction, and an index indicating any of {0,1 / 2,1,2,4,8} that will be used as the standard deviation in the y-axis direction.

[0170] In this embodiment, the filter's parameter information may include the values ​​of the parameters themselves.

[0171] In an embodiment, when the image decoding device 400 and the image encoding device pre-agree on the values ​​to be used for specific parameters in the filter parameters, the information indicating the agreed parameter values ​​may not be included in the filter information. For example, when the image decoding device 400 and the image encoding device agree to set the size of the Gaussian filter to 3, the image encoding device may send the values ​​of the standard deviation in the x-axis direction and the standard deviation in the y-axis direction as the parameter information of the Gaussian filter to the image decoding device 400, instead of sending the size value of the Gaussian filter to the image decoding device 400.

[0172] The predictive decoding unit 430 can filter the preliminary optical flow using a filter corresponding to the filtering information. For example, when the type information of the filtering information indicates a Gaussian filter, and the parameter information indicates a size of 3, a standard deviation of 1 in the x-axis direction, and a standard deviation of 2 in the y-axis direction, the predictive decoding unit 430 can use a filter kernel of size 3 and sample values ​​corresponding to a standard deviation of 1 in the x-axis direction and a standard deviation of 2 in the y-axis direction to filter the preliminary optical flow.

[0173] In this embodiment, the prediction decoding unit 430 can determine the values ​​of the filter parameters based on the filter strength. For example, when the filter strength is indicated as 3, the prediction decoding unit 430 can determine the values ​​of the parameters corresponding to the filter strength of 3. In other words, the values ​​of the parameters corresponding to the filter strength can be predetermined.

[0174] In an embodiment, when the filter type information indicates a Gaussian filter, a median filter, or a Laplace filter, the prediction decoding unit 430 can obtain the optical flow by using a convolution operation of the filter kernel corresponding to the Gaussian filter, median filter, or Laplace filter and the initial optical flow.

[0175] In an embodiment, when the filter type information indicates a neural network filter, the prediction decoding unit 430 can obtain optical flow by applying a preliminary optical flow to the neural network indicated by the parameter information.

[0176] In an embodiment, the weight information included in the filtering information may indicate the weights used for weighted summation among a plurality of candidate weights.

[0177] In this implementation, the prediction decoding unit 430 can obtain a filtered preliminary optical flow by filtering the preliminary optical flow using a filter specified by type information and parameter information. The prediction decoding unit 430 can also obtain the optical flow by calculating a weighted sum of the filtered preliminary optical flow and the preliminary optical flow according to weights indicated by weight information.

[0178] For example, optical flow can be generated according to Equation 1 below.

[0179] [Equation 1] P3 = w1 × P1 + w2 × P2 In Equation 1, P1 corresponds to the initial optical flow, P2 corresponds to the filtered initial optical flow, and P3 corresponds to the optical flow. Furthermore, w1 and w2 are weights applied to P1 and P2 respectively, and can be identified from the weight information.

[0180] In this embodiment, w2 can be a weight indicated by the weight information, and w1 can be a-w2. The predetermined integer a can be, for example, 1. For instance, when the weight information indicates 1 / 8, w2 can be 1 / 8, and w1 can be 7 / 8.

[0181] In an embodiment, w1 can be a weight indicated by weight information, and w2 can be a-w1.

[0182] In an embodiment, the prediction decoding unit 430 can obtain a first filtered preliminary optical flow by filtering the preliminary optical flow using a first filter indicated by type information, and obtain a second filtered preliminary optical flow by filtering the preliminary optical flow using a second filter indicated by type information. The prediction decoding unit 430 can obtain the optical flow by calculating a weighted sum of the first filtered preliminary optical flow and the second filtered preliminary optical flow according to weights indicated by weight information.

[0183] For example, optical flow can be generated according to Equation 2 below.

[0184] [Equation 2] P3 = w1 × P2_a + w2 × P2_b In Equation 2, P2_a corresponds to the initial optical flow of the first filter, P2_b corresponds to the initial optical flow of the second filter, and P3 corresponds to the optical flow. Furthermore, w1 and w2 are weights applied to P2_a and P2_b respectively, and can be identified from the weight information.

[0185] In this embodiment, w2 can be a weight indicated by the weight information, and w1 can be a-w2. The predetermined integer a can be, for example, 1. For instance, when the weight information indicates 1 / 8, w2 can be 1 / 8, and w1 can be 7 / 8. In this embodiment, w1 can be a weight indicated by the weight information, and w2 can be a-w1.

[0186] In an embodiment, when the filter type information indicates n filters (n is an integer of 1 or greater), the weight information can indicate n-1 candidates from a plurality of candidate weights (e.g., Figure 6(As shown in 0, 1 / 8, 1 / 4, 1 / 2, and 1). For example, when the filter type information indicates three filters, the weight information can indicate two candidate weights from a plurality of candidate weights. A first weight indicated by the weight information can be applied to a first preliminary filtered optical flow generated based on a first filter among the three filters indicated by the type information, and a second weight indicated by the weight information can be applied to a second preliminary filtered optical flow generated based on a second filter among the three filters indicated by the type information. A value obtained by subtracting the sum of the first and second weights from a predetermined value (e.g., 1) can be applied to a third preliminary filtered optical flow generated based on a third filter indicated by the type information.

[0187] In this embodiment, the parameter information may indicate the strength of the filter, and the prediction decoding unit 430 may determine the parameter values ​​of the filter based on the filter strength. (See also...) Figure 7 This needs to be explained.

[0188] Figure 7 A table for interpreting filter information is shown according to an embodiment.

[0189] In an embodiment, parameter information may indicate the strength of a filter as indicated by type information. Filter strength can be categorized by multiple values, and for example, a larger strength value may indicate a stronger filter.

[0190] The image decoding device 400 and the image encoding device 1100 may agree on parameter values ​​corresponding to the strength of the filter. Therefore, the prediction decoding unit 430 can determine the parameter values ​​of the filter based on the filter strength indicated by the parameter information.

[0191] For example, refer to Figure 7 When the type information indicates a Gaussian filter and the parameter information indicates an intensity of 3, the prediction decoding unit 430 can determine the size of the Gaussian filter to be 7, the standard deviation in the x-axis direction to be 8, and the standard deviation in the y-axis direction to be 8.

[0192] Furthermore, for example, when the type information indicates a Gaussian filter and the parameter information indicates an intensity of 1, the prediction decoding unit 430 can determine the size of the Gaussian filter to be 3, the standard deviation in the x-axis direction to be 1 / 2, and the standard deviation in the y-axis direction to be 1 / 2.

[0193] In this embodiment, the parameter information may include an index or flag indicating the strength of the filter. Therefore, the prediction decoding unit 430 can determine the value of the filter parameters by using the index or flag corresponding to the parameter information.

[0194] As mentioned above, when the filter type information indicates a neural network filter, the parameter information can indicate which neural network among multiple neural networks will be used as a neural network filter.

[0195] The prediction decoding unit 430 can pre-store multiple neural networks that will be used as neural network filters, and apply preliminary optical flow to the neural networks indicated by parameter information in the multiple neural networks.

[0196] Figure 8 This is a diagram illustrating a neural network used for a neural network filter according to an embodiment.

[0197] Figure 8 The neural network 800 shown can be Figure 6 and Figure 7 Any one of the first, second, and third neural networks shown.

[0198] like Figure 8 As shown, the initial optical flow 805 can be input into the first convolutional layer 810.

[0199] The 3×3×4 marked on the first convolutional layer 810 refers to an example of convolving a preliminary optical flow 805 using four filter kernels, each of which is 3×3 in size. As a result of the convolution process, four feature maps can be generated from the four filter kernels.

[0200] The feature maps generated by the first convolutional layer 810 can represent the unique characteristics of the initial optical flow 805. For example, each feature map can represent the vertical, horizontal, or edge characteristics of the initial optical flow 805.

[0201] The feature map generated by the first convolutional layer 810 can be input into the first activation layer 820.

[0202] The first activation layer 820 may provide nonlinear properties to each feature map. The first activation layer 820 may include a sigmoid function, a hyperbolic tangent function, or a rectified linear unit (ReLU) function, but this disclosure is not limited thereto.

[0203] The first activation layer 820 provides nonlinear characteristics that can be used to modify and output sample values ​​of the feature map. In this state, changes can be performed by applying nonlinear characteristics.

[0204] The first activation layer 820 can determine whether to pass sample values ​​of the feature map to the second convolutional layer 830. For example, some sample values ​​of the feature map can be activated by the first activation layer 820 and passed to the second convolutional layer 830, while some sample values ​​can be deactivated by the first activation layer 820 and not passed to the second convolutional layer 830. The unique characteristics of the preliminary optical flow 805 represented by the feature map can be emphasized by the first activation layer 820.

[0205] The feature map output from the first activation layer 820 can be input into the second convolutional layer 830. The 3×3×4 marked on the second convolutional layer 830 refers to an example of convolution processing the input feature map using four filter kernels (each filter kernel is 3×3 in size). The output of the second convolutional layer 830 can be input into the second activation layer 840. The second activation layer 840 can provide non-linear characteristics to the input feature map.

[0206] The feature map output from the second activation layer 840 can be fed into the third convolutional layer 850. The 3×3×1 marked on the third convolutional layer 850 refers to an example of convolutional processing used to generate an output data 855 using a filter kernel of size 3×3.

[0207] In an embodiment, the output data 855 may be the optical flow used for weighted summation or the filtered preliminary optical flow.

[0208] although Figure 8 The diagram shows a neural network 800 comprising three convolutional layers 810, 830, and 850, and two activation layers 820 and 840, but this is merely an example. In embodiments, the number of convolutional and activation layers included in the neural network 800 can be varied.

[0209] In this embodiment, the neural network 800 can be implemented using a recurrent neural network (RNN).

[0210] In the embodiments, the image decoding device 400 and image encoding device 1100 described below may include at least one arithmetic logic unit (ALU) for the above-described convolution operations and activation layer operations.

[0211] The ALU can be implemented by a processor. For convolution operations, the ALU may include multipliers and adders, wherein the multipliers perform the product operation between the sample values ​​of the input data and the sample values ​​of the filter kernel, and the adders sum the results of the multiplication.

[0212] For operations on activation layers, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by a weight for a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a predetermined value to determine whether to pass the input sample value to the next layer.

[0213] In this embodiment, multiple different neural networks can be used as neural network filters (e.g., Figure 6 The first, second, and third neural networks shown are illustrated, and the filtering results based on multiple different neural networks can differ from each other. To produce different filtering results, the multiple neural networks can each have different parameters trained using different training data, or they can have different internal structures.

[0214] For example, when the first neural network and the second neural network have the same internal structure, the filtering results based on the first neural network and the second neural network can be different from each other by distinguishing the training data used to train the first neural network and the second neural network. The first neural network and the second neural network having the same internal structure can mean, for example, that the first neural network and the second neural network include the same number of layers, and that the size and number of filter kernels used in the layers are the same.

[0215] Furthermore, for example, when the internal structures of the first neural network and the second neural network are different from each other (e.g., when the first neural network has three convolutional layers and the second neural network has five convolutional layers), the filtering results based on the first neural network and the filtering results based on the second neural network may be different from each other.

[0216] In this embodiment, the multiple neural networks used as neural network filters may have different filtering strengths than each other. For example, the first neural network may have the strongest filtering strength, and the third neural network may have the weakest filtering strength. The filtering strength of the multiple neural networks can be determined based on the internal structure of each neural network and the type of training data. See below for further details. Figure 16 Describe neural network training methods.

[0217] Figure 9 This is a diagram illustrating the semantics of filtering information according to an embodiment.

[0218] In embodiments, filtering information can be obtained from the bitstream in units of frame sequences (e.g., sequence parameter sets (SPS)), frames (e.g., frame parameter sets (PPS)) or blocks (e.g., strip data).

[0219] For example, when obtaining filtering information on a per-frame basis, the same filtering information can be applied to a per-frame sequence that includes the current image.

[0220] Furthermore, for example, when obtaining filtering information on a per-image basis, filtering information for the current image and filtering information for other images can be obtained independently.

[0221] Furthermore, for example, when the current image corresponds to the current block divided from the image, filtering information can be obtained on a block-by-block basis, and in this case, filtering information for the current block and filtering information for other blocks can be obtained independently of each other.

[0222] In the embodiments, type information, parameter information, and weight information included in the filtering information can be obtained according to different units.

[0223] For example, while filter type information can be obtained in units of frame sequences (e.g., SPS), parameter information and / or weight information can be obtained in units of frames (e.g., PPS) or blocks (e.g., strip data).

[0224] Furthermore, for example, while filter type information can be obtained in units of frames (e.g., PPS), parameter information and / or weight information can be obtained in units of blocks (e.g., strip data).

[0225] Furthermore, for example, filter type information can be obtained in units of frame sequences (e.g., SPS), parameter information can be obtained in units of frames (e.g., PPS), and weight information can be obtained in units of blocks (e.g., strip data).

[0226] In this embodiment, information indicating whether filtering of the initial optical flow is required may be included in the bitstream. When the corresponding information indicates that filtering is required, the prediction decoding unit 430 can obtain the filtering information and filter the initial optical flow based on the obtained filtering information. When the information indicating whether filtering of the initial optical flow is required indicates that filtering is not required, the prediction decoding unit 430 can determine the initial optical flow as an optical flow without obtaining the filtering information.

[0227] In an embodiment, information indicating whether the initial optical flow needs to be filtered can be obtained from the bitstream in units of frame sequences (e.g., SPS), frames (e.g., PPS), or blocks (e.g., strip data).

[0228] For example, while information indicating whether filtering of the initial optical flow is required can be obtained in units of frame sequences (e.g., SPS), filtering information can be obtained in units of frames (e.g., PPS) or blocks (e.g., strip data).

[0229] Furthermore, for example, while information indicating whether filtering of the initial optical flow is necessary can be obtained in units of frames (e.g., PPS), filtering information can be obtained in units of blocks (e.g., strip data).

[0230] Reference Figure 9 In S910, the type_idx indicating the type of filter can be obtained from the bitstream. Based on the value indicated by type_idx, the type of filter to be applied to the initial optical flow can be determined from among several different types of filters.

[0231] In an embodiment, when it is determined that filtering of the initial optical flow is required, type_idx can be obtained from the bit stream.

[0232] In S920, when type_idx is 0 (e.g., type_idx indicates a Gaussian filter), in S930, size_idx indicating the size, st_horizontal_idx indicating the standard deviation in the x-axis direction, and st_vertical_idx indicating the standard deviation in the y-axis direction can be obtained from the bitstream. The prediction decoding unit 430 can determine the parameter values ​​of the Gaussian filter based on size_idx, st_horizontal_idx, and st_vertical_idx, and filter the preliminary optical flow using the Gaussian filter with the determined parameter values.

[0233] In S940, when type_idx is 1 (e.g., type_idx indicates a median filter), in S950, size_idx, indicating the size, can be obtained from the bitstream. The prediction decoding unit 430 can determine the parameter values ​​of the median filter based on size_idx and filter the initial optical flow using the median filter with the determined parameter values.

[0234] In S960, when type_idx is 2 (e.g., type_idx indicates a Laplace filter), in S970, size_idx indicating the size and st_idx indicating the standard deviation can be obtained from the bitstream. The prediction decoding unit 430 can determine the parameter values ​​of the Laplace filter based on size_idx and st_idx, and filter the initial optical flow using the Laplace filter with the determined parameter values.

[0235] In S980, when type_idx is 3 (e.g., type_idx indicates a neural network filter), in S990, NN_idx, indicating the neural network used as a neural network filter, can be obtained from the bitstream. The prediction decoding unit 430 can then apply a preliminary optical flow to the neural network indicated by NN_idx.

[0236] Despite Figure 9 The information is not shown in the figure, but when type_idx is 0, 1, 2 or 3, the weight information for the weighted sum of the preliminary optical flow and the filtered preliminary optical flow can also be obtained from the bit stream.

[0237] Figure 10 This is a flowchart of an image decoding method according to an embodiment.

[0238] In S1010, the image decoding device 400 can obtain preliminary optical flow feature data and filtering information from the bitstream.

[0239] In an embodiment, the image decoding device 400 can obtain information from the bitstream indicating whether the initial optical stream needs to be filtered, and when the obtained information indicates that filtering is required, it obtains filtering information from the bitstream.

[0240] In an embodiment, the image decoding device 400 can also obtain feature data of the residual image from the bitstream.

[0241] In this embodiment, the image decoding device 400 can obtain feature data of the preliminary optical flow, feature data of the residual image, and filtering information by performing entropy decoding, dequantization, and / or inverse transformation on the bitstream.

[0242] In S1020, the image decoding device 400 can obtain the initial optical flow by applying the feature data of the initial optical flow to the first decoder 14.

[0243] The first decoder 14 can output the initial optical flow by processing the feature data of the initial optical flow according to the parameters set through training.

[0244] In S1030, the image decoding device 400 can generate an optical flow by applying a preliminary optical flow to a filter corresponding to the filtering information.

[0245] Since the above describes the process of filtering the initial optical flow using a filter corresponding to the filtering information, its detailed description is omitted.

[0246] In S1040, the image decoding device 400 can generate a currently reconstructed image using previous data and optical flow. The currently reconstructed image can be sent to a display for output.

[0247] In one embodiment, the image decoding device 400 can obtain the current reconstructed image by performing motion compensation on previous data based on optical flow.

[0248] In one embodiment, the image decoding device 400 can generate a predicted image by warping previous data based on optical flow. The image decoding device 400 can obtain a reconstructed residual image by applying feature data from the residual image to a second decoder 24. The image decoding device 400 can generate a current reconstructed image by combining the predicted image with the reconstructed residual image.

[0249] In this embodiment, the predicted image may be determined as the current reconstructed image.

[0250] In one embodiment, the image decoding device 400 can obtain the current reconstructed image by applying optical flow, previous data, and feature data of the residual image to the second decoder 24.

[0251] In one embodiment, the image decoding device 400 can obtain the current reconstructed image by applying feature data of the predicted image and residual image generated from previous data based on optical flow to the second decoder 24.

[0252] Figure 11This is a diagram illustrating the configuration of an image encoding device 1100 according to an embodiment.

[0253] Reference Figure 11 The image encoding device 1100 may include a prediction encoding unit 1110 and a generation unit 1130.

[0254] The predictive coding unit 1110 and the generation unit 1130 can be implemented by a processor. The predictive coding unit 1110 and the generation unit 1130 can operate according to instructions stored in a memory (not shown).

[0255] although Figure 11 Predictive coding unit 1110 and generation unit 1130 are shown separately, but they can be implemented by a single processor. In embodiments, predictive coding unit 1110 and generation unit 1130 can be implemented by a dedicated processor, or by a combination of a general-purpose processor such as an AP, CPU, or GPU and software.

[0256] The predictive coding unit 1110 and the generation unit 1130 can be configured by multiple processors. In embodiments, the predictive coding unit 1110 and the generation unit 1130 can be implemented by a combination of dedicated processors, or by a combination of general-purpose processors such as AP, CPU or GPU and software.

[0257] In an embodiment, the image encoding device 1100 may use a first encoder 12, a first decoder 14, a second encoder 22, and a second decoder 24 to encode the current image.

[0258] The first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 can be stored in a memory. In an embodiment, the first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 can be implemented by an AI processor.

[0259] The predictive coding unit 1110 can encode the current image to generate data corresponding to the coding result.

[0260] In an embodiment, at least one of the feature data of the preliminary optical flow, the feature data of the residual image, or the filtering information can be obtained as the encoding result of the current image.

[0261] In an embodiment, the predictive coding unit 1110 can obtain preliminary optical flow feature data by applying previous data and the current image to the first encoder 12.

[0262] The predictive coding unit 1110 can obtain the preliminary optical flow of the current image by applying the feature data of the preliminary optical flow to the first decoder 14.

[0263] In an embodiment, the predictive coding unit 1110 may apply quantization and dequantization to the feature data of the preliminary optical flow obtained from the first encoder 12, and apply the dequantized feature data of the preliminary optical flow to the first decoder 14. The reason for applying quantization and dequantization to the feature data of the preliminary optical flow is to obtain the same preliminary optical flow as that obtained by the image decoding device 400.

[0264] In an embodiment, the predictive coding unit 1110 may select a filter from a plurality of filters for filtering the initial optical flow. In an embodiment, the plurality of filters may include Gaussian filters, median filters, Laplace filters, or neural network filters, but the type of filter is not limited thereto.

[0265] Multiple filters can be distinguished from each other based on their type and parameters. For example, the first filter can be a Gaussian filter, and the second filter can be a median filter. Furthermore, for example, when the first and second filters are Gaussian filters, the parameter values ​​of the first and second filters can be different from each other.

[0266] The predictive coding unit 1110 can select from a plurality of filters specified by type and / or parameters to filter the initial optical flow.

[0267] In an embodiment, the prediction coding unit 1110 may select two or more filters for filtering the initial optical flow.

[0268] In one embodiment, filtering information for a filter selected for filtering the initial optical flow can be sent to the generation unit 1130.

[0269] In an embodiment, the prediction coding unit 1110 can obtain the optical flow of the current block by filtering the preliminary optical flow using a filter selected from a plurality of filters.

[0270] In an embodiment, the predictive coding unit 1110 can obtain a filtered preliminary optical flow by filtering the preliminary optical flow using a filter. The predictive coding unit 1110 can obtain the optical flow by calculating a weighted sum of the filtered preliminary optical flow and the preliminary optical flow based on weights. In an embodiment, weights for the weighted sum can be selected from a plurality of candidate weights.

[0271] In an embodiment, the predictive coding unit 1110 can obtain a first filtered preliminary optical flow by filtering the preliminary optical flow using a first filter selected from a plurality of filters, and obtain a second filtered preliminary optical flow by filtering the preliminary optical flow using a second filter selected from a plurality of filters. The predictive coding unit 1110 can obtain the optical flow by calculating a weighted sum of the first filtered preliminary optical flow and the second filtered preliminary optical flow according to weights.

[0272] In an embodiment, the prediction coding unit 1110 can obtain feature data of the residual image by using optical flow and previous data.

[0273] In one embodiment, the predictive coding unit 1110 can generate a predicted image from previous data based on optical flow and obtain a residual image corresponding to the difference between the predicted image and the current image. The predictive coding unit 1110 can obtain feature data of the residual image by applying the residual image to the second encoder 22.

[0274] In an embodiment, the prediction coding unit 1110 can obtain feature data of the residual image by applying the current image, previous data, and optical flow to the second encoder 22.

[0275] In an embodiment, the prediction coding unit 1110 can obtain a predicted image generated from previous data based on optical flow, and obtain feature data of the residual image by applying the current image to the second encoder 22.

[0276] In this embodiment, when the feature data of the residual image is obtained, the predictive coding unit 1110 can generate the current reconstructed image using the feature data of the residual image, optical flow, previous data, and the second decoder 24. Since a method for generating the current reconstructed image has been described with respect to the image decoding device 400, its detailed description is omitted. The current reconstructed image can be used to encode the next image.

[0277] When data corresponding to the encoding result of the current image is obtained, the generation unit 1130 can generate a bitstream including the data.

[0278] In an embodiment, the generation unit 1130 can generate a first bitstream corresponding to the feature data and filtering information of the preliminary optical flow and a second bitstream corresponding to the feature data of the residual image.

[0279] The bitstream can be sent to the image decoding device 400 via a network. In embodiments, the bitstream can be recorded on a data storage medium including magnetic media (such as hard disks, floppy disks, and magnetic tapes), optical recording media (such as CD-ROMs and DVDs), or magneto-optical media (such as floppy disks).

[0280] Figure 12This is a diagram illustrating the configuration of the generation unit 1130 according to an embodiment.

[0281] Reference Figure 12 The generation unit 1130 may include a quantization unit 1210 and an entropy encoding unit 1230.

[0282] The quantization unit 1210 can quantize the data corresponding to the encoding result for the current image.

[0283] For example, the quantization unit 1210 can quantize the feature data of the preliminary optical flow and the feature data of the residual image. For example, quantization can be applied to the feature data of the preliminary optical flow and the feature data of the residual image, but quantization can be omitted from the filtering information. In this case, the quantized feature data of the preliminary optical flow, the quantized feature data of the residual image, and the filtering information can be sent to the entropy coding unit 1230.

[0284] Entropy coding unit 1230 can generate a bit stream by entropy coding the data sent from quantization unit 1210.

[0285] In an embodiment, the generation unit 1130 may further include a transformation unit. The transformation unit can transform data corresponding to the encoding result for the current image from the spatial domain to the frequency domain, thereby providing the data to the quantization unit 1210.

[0286] In this embodiment, the generation unit 1130 may not include the quantization unit 1210. In other words, through the processing of the entropy coding unit 1230, a bitstream comprising data corresponding to the encoding result for the current image can be obtained.

[0287] In the following description, refer to Figure 13 and Figure 14 This describes a method for selecting a filter from a plurality of filters (hereinafter referred to as a plurality of candidate filters) for filtering an initial optical flow.

[0288] Figure 13 This is a diagram illustrating the method of selecting a filter by the predictive coding unit 1110 according to an embodiment.

[0289] The predictive coding unit 1110 can select one or more filters from a plurality of candidate filters to filter the initial optical flow.

[0290] In an embodiment, the predictive coding unit 1110 may select weights from a plurality of candidate weights (e.g., 0, 1 / 8, 1 / 4, 1 / 2, and 1) to be used for weighted summation. As described above, the weights may be used for weighted summation between filtered preliminary optical flows, or for weighted summation between two or more filtered preliminary optical flows. When the weighted summation is not used to obtain the optical flow, the predictive coding unit 1110 may omit the weight selection process.

[0291] Multiple candidate filters can be classified by type and parameter values. Multiple candidate filters can differ from each other in at least one aspect, namely type and parameter values.

[0292] For example, multiple candidate filters can be Figure 6 The examples shown are Gaussian filters, median filters, Laplace filters, or neural network filters. For instance, any candidate filter could be a Gaussian filter with a size of 3, a standard deviation of 0 in both the x and y directions. Another candidate filter could be a Gaussian filter with a size of 3, a standard deviation of 1 / 2 in the x-direction, and a standard deviation of 0 in the y-direction. Yet another candidate filter could be a median filter with a size of 3.

[0293] Furthermore, for example, multiple candidate filters can be combined with... Figure 6 The Gaussian filter, median filter, Laplace filter, or neural network filter shown are corresponding to the candidate filters, and the strengths of the multiple candidate filters can be 3, 2, or 1.

[0294] The predictive coding unit 1110 can apply each of the candidate filters and each of the candidate weights to the initial optical flow in order to select the most suitable filter and weights for filtering the initial optical flow.

[0295] Reference Figure 13 The prediction coding unit 1110 can generate a predicted image from the current image by using each candidate filter and each candidate weight in the candidate filters. The prediction coding unit 1110 can select filters and weights for filtering the initial optical flow from multiple candidate filters and multiple candidate weights based on the comparison results between the current image and the predicted image.

[0296] In an embodiment, the comparison result between the current image and the predicted image may include at least one of the following: L1 norm value, L2 norm value, SSIM value, PSNR-HVS value, MS-SSIM value, VIF value, VMAF value, MSE value, RMSE value, or SAD value.

[0297] This describes the process of calculating the comparison result between the current image and the predicted image using any candidate filter and any candidate weight.

[0298] The predictive coding unit 1110 can obtain preliminary optical flow from the current image and previous data. The first encoder 12 and the first decoder 14 can be used to obtain the preliminary optical flow.

[0299] The predictive coding unit 1110 can generate a filtered preliminary optical flow by filtering the preliminary optical flow using candidate filters. The predictive coding unit 1110 can obtain the optical flow by calculating a weighted sum of the preliminary optical flow and the filtered preliminary optical flow based on the candidate weights.

[0300] The prediction coding unit 1110 can generate a prediction image using optical flow and previous data. Distortion can be applied to the previous data to obtain the prediction image. When the prediction image is generated, the prediction coding unit 1110 can calculate a comparison result between the current image and the prediction image, and select filters and weights for filtering the initial optical flow by using the calculated comparison result.

[0301] In an embodiment, the prediction coding unit 1110 may select from a plurality of candidate filters and a plurality of candidate weights a filter and weights for generating a prediction image that is most similar to the current image.

[0302] The filter and weight selection process can be represented by the pseudocode in Table 1.

[0303] [Table 1]

[0304] In Table 1, the distance corresponds to the comparison result between the current image and the predicted image. Referring to Table 1, the first best distance can be set to infinity. The comparison result (distance) between the predicted image and the current image can be calculated for each filter type in the Type List, each size in the Size List, each parameter in the Parameter List, and each weight in the Weight List. When the distance is less than the best distance, the distance can be determined as the new best distance, and the type, size, parameter, and weight corresponding to the best distance can be determined as the best type, best size, best parameter, and best weight, respectively. The finally determined best type, best size, best parameter, and best weight can be used to filter the initial optical flow.

[0305] Figure 14 This is a diagram illustrating a method for selecting a filter by a predictive coding unit 1110 according to an embodiment.

[0306] In an embodiment, the prediction coding unit 1110 may select from a plurality of candidate filters and a plurality of candidate weights a filter and weights for generating a current reconstructed image that is most similar to the current image.

[0307] In an embodiment, the comparison result between the current image and the current reconstructed image may include at least one of the following: L1 norm value, L2 norm value, SSIM value, PSNR-HVS value, MS-SSIM value, VIF value, VMAF value, MSE value, RMSE value, or SAD value.

[0308] In an embodiment, the predictive coding unit 1110 may also consider the bit rate of the bit stream when selecting filters and weights.

[0309] Reference Figure 14 The prediction coding unit 1110 can encode the current image based on each candidate filter in the candidate filters and each candidate weight in the candidate weights, and send the data generated as the result of the encoding to the generation unit 1130.

[0310] In an embodiment, the prediction coding unit 1110 can send the feature data of the preliminary optical flow, the feature data of the residual image, and the filtering information indicating the candidate filter and candidate weight as the coding result of the current image to the generation unit 1130.

[0311] The generation unit 1130 can generate a bit stream based on the transmitted data.

[0312] In an embodiment, the predictive coding unit 1110 may select filters and weights based on the bit rate of the bit stream corresponding to the feature data of the preliminary optical flow, the feature data of the residual image, and the filtering information.

[0313] In an embodiment, the bit rate of the bit stream corresponding to the feature data of the preliminary optical flow, the feature data of the residual image, and the filtering information can be the bit rate of the result of entropy coding of the feature data of the preliminary optical flow, the feature data of the residual image, and the filtering information.

[0314] For example, the predictive coding unit 1110 can select the filter and weights that result in the minimum ratio from a plurality of candidate filters and a plurality of candidate weights.

[0315] In an embodiment, the predictive coding unit 1110 may select filters and weights from a plurality of candidate filters and a plurality of candidate weights for filtering the preliminary optical flow based on the comparison result between the current image and the current reconstructed image and the bit rate of the bit stream.

[0316] For example, the predictive coding unit 1110 can calculate the loss value by calculating the weighted sum of the value corresponding to the comparison result between the current image and the current reconstructed image and the value corresponding to the bit rate of the bit stream, and select the filter and weight that result in the minimum loss value.

[0317] The description describes the process of calculating the loss value using any candidate filter and any candidate weight.

[0318] The predictive coding unit 1110 can obtain preliminary optical flow from the current image and previous data. The first encoder 12 and the first decoder 14 can be used to obtain the preliminary optical flow.

[0319] The predictive coding unit 1110 can generate a filtered preliminary optical flow by filtering the preliminary optical flow using any candidate filter. The predictive coding unit 1110 can obtain the optical flow by calculating a weighted sum of the preliminary optical flow and the filtered preliminary optical flow based on the candidate weights.

[0320] Predictive coding unit 1110 can generate the current reconstructed image using optical flow and previous data. Second encoder 22 and second decoder 24 can be used to generate the current reconstructed image.

[0321] When generating the current reconstructed image, a comparison between the current image and the current reconstructed image can be calculated. Furthermore, the bit rate of the bitstream can be calculated using feature data of the preliminary optical flow output from the first encoder 12, feature data of the residual image output from the second encoder 22, and filtering information indicating any candidate filter and any candidate weight. The predictive coding unit 1110 can calculate the loss value corresponding to any candidate filter and any candidate weight based on the comparison result between the current image and the current reconstructed image and the bit rate of the bitstream.

[0322] Figure 15 This is a flowchart of an image encoding method according to an embodiment.

[0323] In operation S1510, the image encoding device 1100 can obtain preliminary optical flow feature data by applying the current image and previous data to the first encoder 12.

[0324] In operation S1520, the image encoding device 1100 can obtain the initial optical flow by applying the feature data of the initial optical flow to the first decoder 14.

[0325] During operation S1530, the image encoding device 1100 can select from a plurality of filters to filter the initial optical flow.

[0326] As already referred to above Figure 13 and Figure 14 The filter selection method is described, therefore its detailed description is omitted.

[0327] In operation S1540, the image encoding device 1100 can generate an optical flow by applying a preliminary optical flow to a selected filter.

[0328] In an embodiment, the image encoding device 1100 can obtain a filtered preliminary optical flow by applying a preliminary optical flow to a filter, and obtain an optical flow by calculating a weighted sum of the filtered preliminary optical flow and the preliminary optical flow.

[0329] In an embodiment, the image encoding device 1100 can obtain a first filtered preliminary optical flow by applying a preliminary optical flow to a first filter, and a second filtered preliminary optical flow by applying the preliminary optical flow to a second filter. The image encoding device 1100 can obtain the optical flow by calculating a weighted sum of the first filtered preliminary optical flow and the second filtered preliminary optical flow.

[0330] In operation S1550, the image encoding device 1100 can encode the current image using optical flow and previous data.

[0331] In this embodiment, the feature data of the residual image can be obtained as the encoding result of the current image.

[0332] In one embodiment, the image encoding device 1100 can generate a predicted image from previous data based on optical flow and obtain a residual image corresponding to the difference between the predicted image and the current image. The image encoding device 1100 can obtain feature data of the residual image by applying the residual image to the second encoder 22.

[0333] In an embodiment, the image encoding device 1100 can obtain feature data of the residual image by applying the current image, previous data, and optical flow to the second encoder 22.

[0334] In an embodiment, the image encoding device 1100 can obtain feature data of the residual image by applying a predicted image generated from previous data based on optical flow and the current image to the second encoder 22.

[0335] During operation S1560, the image encoding device 1100 can generate a bitstream that includes feature data of the initial optical flow and filtering information.

[0336] In an embodiment, the filtering information may include at least one of filter type information, parameter information, or weight information.

[0337] In this embodiment, the bitstream may also include feature data of the residual image.

[0338] As described above, when the neural network filter is used to filter the initial optical flow, the image encoding device 1100 and the image decoding device 400 can apply the initial optical flow to a neural network selected from multiple neural networks.

[0339] Although each neural network can process the initial optical flow based on the parameters set through training, reference Figure 16A method for training a neural network to be used as a filter for the neural network is described.

[0340] Figure 16 This is a diagram illustrating a method for training a neural network that can be used as a neural network filter according to an embodiment.

[0341] Reference Figure 16 The neural network 1600 can receive an input of preliminary optical flow for training and output an optical flow for training by processing the preliminary optical flow for training according to preset parameters.

[0342] The comparison between the ground truth optical flow and the optical flow used for training can be used as loss information to train the neural network 1600. In an embodiment, the comparison between the ground truth optical flow and the optical flow used for training may include L1 norm value, L2 norm value, SSIM value, PSNR-HVS value, MS-SSIM value, VIF value, VMAF value, MSE value, RMSE value, or SAD value.

[0343] The neural network 1600 can update its parameters to reduce or minimize the loss information, and process the initial optical flow input for training using the updated parameters. When the parameters of the neural network 1600 are optimized, the corresponding neural network 1600 can be stored in the image encoding device 1100 and the image decoding device 400.

[0344] In embodiments, multiple neural networks used as neural network filters may have different internal structures than each other, and therefore their filtering strengths may differ. For example, a neural network with n convolutional layers (n is an integer of 2 or greater) may have a stronger filtering strength than a neural network with m convolutional layers (m is an integer less than n). When each of multiple neural networks with different internal structures is trained based on an initial optical flow used for training, multiple neural networks with different filtering strengths can be used as neural network filters.

[0345] In this embodiment, multiple neural networks used in the neural network filter can be trained based on different types of initial optical flow used for training. Therefore, multiple neural networks with different filtering intensities can be obtained.

[0346] For example, the filtering strength of a neural network trained on high-quality initial optical flow for training can be relatively low, while the filtering strength of a neural network trained on low-quality initial optical flow for training can be relatively strong.

[0347] This describes a method for training a neural network 1600 by using different types of initial optical flow for training.

[0348] In an embodiment, a first reference true optical flow can be obtained from a previously reconstructed image (or previous data) reconstructed based on a first quantization parameter and a current image. The first reference true optical flow can be extracted according to a predetermined optical flow extraction method (e.g., Flownet, OpenCV optical flow estimation, etc.). The previous data and the current image can be sequentially applied to a first encoder 12 and a first decoder 14, and a first preliminary optical flow for training can be output from the first decoder 14. The first preliminary optical flow for training can be processed by a neural network 1600, and the neural network 1600 can be trained based on a comparison between the first optical flow for training output from the neural network 1600 and the first reference true optical flow.

[0349] Next, a second reference true optical flow can be obtained from a previously reconstructed image (or previous data) reconstructed based on the second quantization parameters and the current image. The previous data and the current image can be sequentially applied to the first encoder 12 and the first decoder 14, and a second preliminary optical flow for training can be output from the first decoder 14. The second preliminary optical flow for training can be processed by the neural network 1600, and the neural network 1600 can be trained based on the comparison between the second optical flow for training output from the neural network 1600 and the second reference true optical flow.

[0350] In an embodiment, when the magnitude of the first quantization parameter is smaller than the magnitude of the second quantization parameter, the quality of the first preliminary optical flow used for training may be higher than the quality of the second preliminary optical flow used for training. This is because the quality of the previously reconstructed image rebuilt based on the high quantization parameter is relatively low, and therefore the quality of the preliminary optical flow used for training obtained based on the low-quality previously reconstructed image may also be low. In other words, the strength of the neural network 1600 trained based on the high-quality first preliminary optical flow used for training may be lower than the strength of the neural network trained based on the low-quality second preliminary optical flow used for training.

[0351] In one embodiment, multiple neural networks with different filtering intensities can be obtained by changing the quantization parameters used to obtain the previously reconstructed image (or previous data).

[0352] In this embodiment, the initial optical flow for training can be obtained by changing the sample values ​​of the reference real optical flow. The quality of the initial optical flow for training can be determined based on the degree of change in the sample values ​​of the reference real optical flow.

[0353] In one embodiment, the image encoding device 1100 may perform training on a plurality of neural networks to be used as neural network filters, and the trained plurality of neural networks may be sent to the image decoding device 400.

[0354] In an embodiment, multiple neural networks can be trained by a separate training device (e.g., a server), and the trained neural networks can be sent to the image encoding device 1100 and the image decoding device 400.

[0355] In the following description, refer to Figure 17 and Figure 18 Describe the method for training the first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24.

[0356] Figure 17 This is a diagram illustrating a method for training a first encoder 12, a first decoder 14, a second encoder 22, and a second decoder 24 according to an embodiment.

[0357] exist Figure 17 In this context, the current image 1710 used for training, the previous data 1720 used for training, and the current reconstructed image 1780 used for training can correspond to the aforementioned current image, previous data, and current reconstructed image, respectively.

[0358] When training the first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24, considerations can be given to how similar the current reconstructed image 1780 used for training is to the current image 1710 used for training, and the bit rate of the bitstream generated by encoding the current image 1710 used for training. Therefore, in this embodiment, the first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 can be trained based on first loss information 1792 and second loss information 1794 corresponding to the size of the bitstream, and third loss information 1796 corresponding to the similarity between the current image 1710 used for training and the current reconstructed image 1780 used for training.

[0359] Reference Figure 17 The current image 1710 and the previous data 1720 used for training can be input into the first encoder 12. The first encoder 12 can output preliminary optical flow feature data w by processing the current image 1710 and the previous data 1720 used for training.

[0360] The feature data w of the preliminary optical flow can be input into the first decoder 14, and the first decoder 14 can output the preliminary optical flow g by processing the feature data w of the preliminary optical flow.

[0361] When the previous data 1720 used for training is warped 40 according to the initial optical flow g, a prediction image 1750 for training can be generated, and a residual image 1760 for training corresponding to the difference between the prediction image 1750 used for training and the current image 1710 used for training can be obtained.

[0362] The residual image 1760 used for training can be input into the second encoder 22, and the second encoder 22 can output feature data v of the residual image by processing the residual image 1760 used for training.

[0363] The second decoder 24 can reconstruct the residual image 1760 for training by processing the feature data v of the residual image, and when the reconstructed residual image 1770 for training and the predicted image 1750 for training are combined with each other, the current reconstructed image 1780 for training can be obtained.

[0364] In order to train the first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24, at least one of the first loss information 1792, the second loss information 1794, or the third loss information 1796 can be obtained.

[0365] The first loss information 1792 may correspond to the entropy of the feature data w of the initial optical flow or the bit rate of the bit stream corresponding to the feature data w of the initial optical flow. Furthermore, the second loss information 1794 may correspond to the entropy of the feature data v of the residual image or the bit rate of the bit stream corresponding to the feature data v of the residual image.

[0366] In an embodiment, a loss information can be calculated from the bit rate of the bit stream corresponding to the feature data w of the preliminary optical flow and the feature data v of the residual image, instead of the first loss information 1792 and the second loss information 1794.

[0367] Since the first loss information 1792 and the second loss information 1794 are related to the coding efficiency of the current image 1710 used for training, compression can also be referred to as loss information.

[0368] The third loss information 1796 may correspond to the difference between the current image 1710 used for training and the current reconstructed image 1780 used for training. The difference between the current image 1710 used for training and the current reconstructed image 1780 used for training may include at least one of the following: L1 norm value, L2 norm value, structural similarity (SSIM) value, peak signal-to-noise ratio-human visual system (PSNR-HVS) value, multi-scale SSIM (MS-SSIM) value, variance inflation factor (VIF) value, or video multi-method evaluation fusion (VMAF) value.

[0369] The third loss information 1796 related to the quality of the current reconstructed image 1780 used for training can be referred to as the quality loss information.

[0370] The first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 can be trained such that the final loss information derived from at least one of the first loss information 1792, the second loss information 1794, or the third loss information 1796 is reduced or minimized.

[0371] In an embodiment, the first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 are operable to reduce or minimize the final loss of information by changing the values ​​of preset parameters.

[0372] In this embodiment, the final loss information can be calculated according to Equation 3 below.

[0373] [Equation 3] Final loss information = a × first loss information + b × second loss information + c × third loss information In Equation 3, a, b, and c can be the weights applied to each of the first loss information 1792, the second loss information 1794, and the third loss information 1796, respectively.

[0374] According to Equation 3, it can be seen that the first encoder 12, the first decoder 14, the second encoder 22 and the second decoder 24 are trained in the direction in which the current reconstructed image 1780 used for training becomes most similar to the current image 1710 used for training, and the size of the bitstream corresponding to the data output from the first encoder 12 and the second encoder 22 is minimized.

[0375] Figure 18 This is a diagram illustrating a method for training a first encoder 12, a first decoder 14, a second encoder 22, and a second decoder 24 according to an embodiment.

[0376] Reference Figure 18 The current image 1710 and the previous data 1720 used for training can be input into the first encoder 12. The first encoder 12 can output preliminary optical flow feature data w by processing the current image 1710 and the previous data 1720 used for training.

[0377] The feature data w of the preliminary optical flow can be input into the first decoder 14, and the first decoder 14 can output the preliminary optical flow g by processing the feature data w of the preliminary optical flow.

[0378] The initial optical flow g, the current image 1710 for training, and the previous data 1720 for training can be input to the second encoder 22, and the second encoder 22 can output feature data v of the residual image by processing the initial optical flow g, the current image 1710 for training, and the previous data 1720 for training. In an embodiment, a predicted image for training generated from the initial optical flow g and the previous data 1720 for training, as well as the current image 1710 for training, can be input to the second encoder 22.

[0379] The second decoder 24 can generate a current reconstructed image 1780 for training by processing feature data v of the residual image, preliminary optical flow g, and previous data 1720 for training. In an embodiment, the predicted image for training generated from the preliminary optical flow g and the previous data 1720 for training, as well as the feature data v of the residual image, can be input to the second decoder 24.

[0380] In order to train the first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24, at least one of the first loss information 1892, the second loss information 1894, or the third loss information 1896 can be obtained.

[0381] The first loss information 1892 may correspond to the entropy of the feature data w of the initial optical flow or the bit rate of the bit stream corresponding to the feature data w of the initial optical flow. Furthermore, the second loss information 1894 may correspond to the entropy of the feature data v of the residual image or the bit rate of the bit stream corresponding to the feature data v of the residual image.

[0382] In an embodiment, a loss information can be calculated from the bit rate of the bit stream corresponding to the feature data w of the initial optical flow and the feature data v of the residual image, instead of the first loss information 1892 and the second loss information 1894.

[0383] The third loss information 1896 may correspond to the difference between the current image 1710 used for training and the current reconstructed image 1780 used for training. The difference between the current image 1710 used for training and the current reconstructed image 1780 used for training may include at least one of the following: L1 norm value, L2 norm value, structural similarity (SSIM) value, peak signal-to-noise ratio-human visual system (PSNR-HVS) value, multi-scale SSIM (MS-SSIM) value, variance inflation factor (VIF) value, or video multi-method evaluation fusion (VMAF) value.

[0384] The first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 can be trained such that the final loss information derived from at least one of the first loss information 1892, the second loss information 1894, or the third loss information 1896 is reduced or minimized.

[0385] In an embodiment, the first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 are operable to reduce or minimize the final loss of information while changing the values ​​of preset parameters.

[0386] In this embodiment, the final loss information can be calculated according to Equation 3 above.

[0387] The reference can be performed by the image encoding device 1100 Figure 17 and Figure 18 The process of training the first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 is described. The trained first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 can be sent to the image decoding device 400.

[0388] In an embodiment, the training of the first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 can be performed by a separate training device (e.g., a server). The trained first encoder 12, the first decoder 14, the second encoder 22, and the second decoder 24 can be sent to the image decoding device 400 and the image encoding device 1100.

[0389] The embodiments involve reducing the bit rate of the bitstream generated by image encoding.

[0390] The embodiments are designed to improve the quality of reconstructed images generated by decoding bitstreams.

[0391] The embodiments are intended to provide an AI-based end-to-end encoding / decoding system.

[0392] The technical problems to be solved by this disclosure are not limited to those described above, and other technical problems not explicitly described will be clearly understood by those skilled in the art from the following description.

[0393] The image decoding method according to the embodiments may include obtaining feature data and filtering information of preliminary optical flow from a bitstream.

[0394] In this embodiment, the filtering information may include at least one of the filter type information or the filter parameter information.

[0395] The image decoding method according to the embodiment may include obtaining the initial optical flow by applying feature data of the initial optical flow to a first decoder 14 based on a neural network.

[0396] The image decoding method according to the embodiments may include generating an optical flow by applying a preliminary optical flow to a filter corresponding to filtering information.

[0397] The image decoding method according to the embodiment may include generating the current reconstructed image by using previous data and optical flow.

[0398] According to the image decoding method of the embodiment, as the quality of the optical flow is improved, the quality of the currently reconstructed image can also be improved.

[0399] In an embodiment, the image decoding method may further include obtaining feature data of the residual image from the bitstream, and the step of generating the current reconstructed image may include obtaining the current reconstructed image by applying the optical flow, the feature data of the residual image, and the previous data to a second neural network-based decoder 24.

[0400] According to an embodiment, the currently reconstructed image can be reconstructed more accurately based on a neural network.

[0401] In an embodiment, the image decoding method may further include obtaining feature data of the residual image from the bitstream, and the step of generating the current reconstructed image may include: obtaining the residual image by applying the feature data of the residual image to a second decoder 24 based on a neural network; generating a predicted image from previous data based on optical flow; and generating the current reconstructed image by combining the predicted image with the residual image.

[0402] According to an embodiment, the currently reconstructed image can be reconstructed more accurately based on a neural network.

[0403] In an embodiment, the filtering information may further include weighting information, and the step of generating optical flow may include: generating a filtered preliminary optical flow by applying a preliminary optical flow to a filter corresponding to the filtering information; and generating an optical flow by combining the preliminary optical flow with the filtered preliminary optical flow according to the weighting information.

[0404] According to the embodiments, optical flow can be generated more accurately based on weight information.

[0405] In an embodiment, the filtering information may further include weighting information, and the step of generating optical flow may include: generating a first filtered preliminary optical flow by applying a preliminary optical flow to a first filter indicated by type information; generating a second filtered preliminary optical flow by applying the preliminary optical flow to a second filter indicated by type information; and generating an optical flow by combining the first filtered preliminary optical flow and the second filtered preliminary optical flow according to weighting information.

[0406] According to the embodiments, optical flow can be generated more accurately based on weight information.

[0407] In an embodiment, the filter type information may indicate at least one of a Gaussian filter, a median filter, a bilateral filter, or a neural network filter.

[0408] According to an embodiment, when a filter suitable for filtering the initial optical flow is selected from a plurality of filters, a more accurate optical flow can be generated.

[0409] In an embodiment, when the filter type information indicates a neural network filter, the parameter information indicates any one of a plurality of neural networks of different types, and the step of generating optical flow may include obtaining optical flow by applying a preliminary optical flow to the neural network indicated by the parameter information.

[0410] According to an embodiment, when the initial optical flow is filtered based on a neural network, a more accurate optical flow can be generated.

[0411] According to an embodiment, the image encoding method may include obtaining preliminary optical flow feature data by applying the current image and previous data to a neural network-based first encoder 12.

[0412] According to an embodiment, the image encoding method may include obtaining an initial optical flow by applying feature data of the initial optical flow to a first decoder 14 based on a neural network.

[0413] According to an embodiment, the image encoding method may include selecting a filter from a plurality of filters for filtering the initial optical flow.

[0414] According to an embodiment, the image encoding method may include generating an optical flow by applying a preliminary optical flow to a selected filter.

[0415] According to an embodiment, the image encoding method may include encoding the current image using optical flow and previous data.

[0416] According to an embodiment, the image encoding method may include generating a bitstream that includes feature data of preliminary optical flow and filtering information for a filter.

[0417] In this embodiment, the filtering information may include at least one of the filter type information or the filter parameter information.

[0418] According to the image encoding method of the embodiment, as the quality of the optical flow is improved, the bit rate of the bit stream can be reduced.

[0419] In an embodiment, the step of encoding the current image may include obtaining feature data of the residual image by applying the current image, previous data, and optical flow to a second encoder 22 based on a neural network, and the feature data of the residual image may be included in the bitstream.

[0420] According to an embodiment, the current image can be encoded more effectively based on a neural network.

[0421] In an embodiment, the step of encoding the current image may include: generating a predicted image from previous data based on optical flow; and obtaining feature data of the residual image by applying a residual image corresponding to the difference between the predicted image and the current image to a second encoder 22 based on a neural network, wherein the feature data of the residual image may be included in the bitstream.

[0422] According to an embodiment, the current image can be encoded more effectively based on a neural network.

[0423] In an embodiment, the step of selecting a filter may include: generating a plurality of optical flows by applying an initial optical flow to a plurality of filters; and selecting at least one filter from the plurality of filters based on the difference between the current image and each of a plurality of predicted images generated based on each of the plurality of optical flows.

[0424] According to an embodiment, when a filter suitable for filtering the initial optical flow is selected from a plurality of filters, a more accurate optical flow can be generated.

[0425] In an embodiment, the step of selecting a filter may include selecting at least one filter from a plurality of filters based on at least one of the following: the difference between the current image and each of a plurality of current reconstructed images generated in response to each of the plurality of filters, or the bit rate of the bit streams generated in response to each of the plurality of filters.

[0426] According to an embodiment, when a filter suitable for filtering the initial optical flow is selected from a plurality of filters, a more accurate optical flow can be generated.

[0427] In an embodiment, when multiple filters include neural network filters using different types of neural networks, the neural network used in the neural network filter can output an optical flow for training by processing an initial optical flow for training, and can train the neural network based on a comparison between the optical flow for training and a reference real optical flow.

[0428] According to an embodiment, a neural network can be trained to generate accurate optical flow.

[0429] The computer-readable recording medium according to the embodiments can record bit streams.

[0430] In this embodiment, the bitstream may include characteristic data of the initial optical flow and filtering information.

[0431] In one embodiment, preliminary optical flow feature data can be obtained by applying the current image and previous data to a neural network-based first encoder 12.

[0432] In an embodiment, a preliminary optical flow can be obtained by applying feature data of the preliminary optical flow to a first decoder 14 based on a neural network, and filtering information can be obtained by selecting a filter from a plurality of filters to filter the preliminary optical flow.

[0433] In one embodiment, when an initial optical flow is applied to a selected filter, an optical flow can be generated, and the current image can be encoded using the optical flow and previous data.

[0434] In this embodiment, the filtering information may include at least one of the filter type information or the filter parameter information.

[0435] The image decoding apparatus according to the embodiment may include an acquisition unit 410 for obtaining feature data and filtering information of preliminary optical flow from a bitstream.

[0436] In this embodiment, the filtering information may include at least one of the filter type information or the filter parameter information.

[0437] The image decoding device according to the embodiment may include a predictive decoding unit 430, wherein the predictive decoding unit 430 obtains an initial optical flow by applying feature data of the initial optical flow to a first decoder 14 based on a neural network, generates an optical flow by applying the initial optical flow to a filter corresponding to filtering information, and generates a current reconstructed image by using previous data and the optical flow.

[0438] According to the image decoding device of the embodiment, as the quality of the optical flow is improved, the quality of the currently reconstructed image can also be improved.

[0439] The image encoding apparatus according to an embodiment may include a predictive encoding unit 1110, wherein the predictive encoding unit 1110 obtains feature data of a preliminary optical flow by applying the current image and previous data to a first encoder 12 based on a neural network, obtains a preliminary optical flow by applying the feature data of the preliminary optical flow to a first decoder 14 based on a neural network, selects a filter from a plurality of filters for filtering the preliminary optical flow, generates an optical flow by applying the preliminary optical flow to the selected filter, and encodes the current image using the optical flow and previous data.

[0440] The image encoding apparatus according to an embodiment may include a generation unit 1130, wherein the generation unit 1130 generates a bitstream including feature data of preliminary optical flow and filtering information for a filter.

[0441] In this embodiment, the filtering information may include at least one of the filter type information or the filter parameter information.

[0442] According to the image encoding apparatus of the embodiment, as the quality of the optical flow improves, the bit rate of the bit stream can be reduced.

[0443] In one embodiment, the bit rate of the bitstream generated as a result of encoding the image can be reduced.

[0444] In this embodiment, the quality of the reconstructed image generated by decoding the bitstream can be improved.

[0445] In one embodiment, an AI-based end-to-end encoding / decoding system may be provided.

[0446] The effects of this disclosure are not limited to those described above, and those skilled in the art to which this disclosure pertains will clearly understand from the following description various other effects not described in this disclosure.

[0447] Embodiments of this disclosure can be written as programs that will be executed on a computer, and the written programs can be stored in a machine-readable recording medium.

[0448] Machine-readable storage media may be provided in the form of non-transitory storage media. Here, "non-transitory" simply means that the storage medium does not contain signals (e.g., electromagnetic waves) and is tangible, but does not distinguish between data that is semi-permanently or temporarily stored in the storage medium. For example, a non-transitory storage medium may include a buffer for temporarily storing data.

[0449] According to embodiments, the methods disclosed in various embodiments of this disclosure can be provided by including the methods disclosed in various embodiments of this disclosure in a computer program product. The computer program product, as a commodity, can be traded between a seller and a buyer. The computer program product can be distributed (e.g., downloaded or uploaded) in the form of a device-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)), distributed through an app store, or distributed directly online between two user devices (e.g., smartphones). For online distribution, at least a portion of the computer program product (e.g., a downloadable application) may be at least temporarily stored or generated on a device-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a relay server).

[0450] As described above, although the present disclosure has been specifically shown and described with reference to preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims.

Claims

1. An image decoding method, comprising: Preliminary optical flow characteristic data and filtering information are obtained from the bit stream, wherein the filtering information includes at least one of filter type information and filter parameter information; The preliminary optical flow is obtained by applying the feature data of the preliminary optical flow to a first decoder (14) based on a neural network; An optical flow is generated by applying the initial optical flow to a filter corresponding to the filtering information; and The current reconstructed image is generated using previous data and the optical flow.

2. The image decoding method according to claim 1, wherein, The image decoding method further includes: The feature data of the residual image are obtained from the bitstream, and The steps for generating the current reconstructed image include: The current reconstructed image is obtained by applying the optical flow, the feature data of the residual image, and the previous data to a second decoder (24) based on a neural network.

3. The image decoding method according to any one of claims 1 and 2, further comprising: Feature data of the residual image are obtained from the bitstream. The steps for generating the current reconstructed image include: The residual image is obtained by applying the feature data of the residual image to a second decoder (24) based on a neural network; A predicted image is generated from the previous data based on the optical flow; and The current reconstructed image is generated by combining the predicted image with the residual image.

4. The image decoding method according to any one of claims 1 to 3, wherein, The filtering information also includes weight information, and The steps for generating the optical flow include: A filtered preliminary optical flow is generated by applying the preliminary optical flow to a filter corresponding to the filtering information; and The optical flow is generated by combining the preliminary optical flow with the filtered preliminary optical flow according to the weight information.

5. The image decoding method according to any one of claims 1 to 4, wherein, The filtering information also includes weight information, and The steps for generating the optical flow include: A first filtered initial optical flow is generated by applying the initial optical flow to a first filter indicated by the type information; A second filtered initial optical flow is generated by applying the initial optical flow to a second filter indicated by the type information; and The optical flow is generated by combining the first filtered preliminary optical flow and the second filtered preliminary optical flow according to the weight information.

6. The image decoding method according to any one of claims 1 to 5, wherein, The type information of the filter indicates at least one of a Gaussian filter, a median filter, a bilateral filter, or a neural network filter.

7. The image decoding method according to any one of claims 1 to 6, wherein, When the filter type information indicates a neural network filter, the parameter information indicates any one of multiple neural networks of different types, and The steps for generating the optical flow include: The optical flow is obtained by applying the preliminary optical flow to the neural network indicated by the parameter information.

8. An image encoding method, comprising: Preliminary optical flow feature data is obtained by applying the current image and previous data to a neural network-based first encoder (12); The preliminary optical flow is obtained by applying the feature data of the preliminary optical flow to a first decoder (14) based on a neural network; Select a filter from a plurality of filters to filter the initial optical flow; An optical flow is generated by applying the initial optical flow to the selected filter; The current image is encoded using the optical flow and the previous data; and Generate a bitstream including the feature data of the initial optical flow and filtering information for the filter. The filtering information includes at least one of the filter type information or the filter parameter information.

9. The image encoding method according to claim 8, wherein, The steps for encoding the current image include: Feature data of the residual image is obtained by applying the current image, the previous data, and the optical flow to a second encoder (22) based on a neural network. The feature data of the residual image is included in the bitstream.

10. The image encoding method according to any one of claims 8 and 9, wherein, The steps for encoding the current image include: A predicted image is generated from the previous data based on the optical flow; and Feature data of the residual image is obtained by applying the residual image corresponding to the difference between the predicted image and the current image to a second neural network-based encoder (22). The feature data of the residual image is included in the bitstream.

11. The image encoding method according to any one of claims 8 to 10, wherein, The steps for selecting the filter include: Multiple optical flows are generated by applying the initial optical flow to the plurality of filters; and At least one filter is selected from the plurality of filters based on the difference between the current image and each of the plurality of predicted images generated based on each of the plurality of optical flows.

12. The image encoding method according to any one of claims 8 to 11, wherein, The steps for selecting the filter include: At least one filter is selected from the plurality of filters based on at least one of the following: the difference between the current image and each of the plurality of current reconstructed images generated in response to each of the plurality of filters, or the bit rate of the bit stream generated in response to each of the plurality of filters.

13. The image encoding method according to any one of claims 8 to 12, wherein, When the plurality of filters includes neural network filters that use different types of neural networks... The neural network used in the neural network filter outputs a training optical flow by processing the initial optical flow used for training, and The neural network is trained based on the comparison between the optical flow used for training and the reference real optical flow.

14. A computer-readable recording medium having a bit stream recorded thereon, wherein, The bitstream includes preliminary optical flow feature data and filtering information. The initial optical flow feature data is obtained by applying the current image and previous data to a first encoder (12) based on a neural network. The filtering information is obtained by applying the feature data of the preliminary optical flow to a first decoder (14) based on a neural network to obtain the preliminary optical flow, and by selecting a filter from a plurality of filters for filtering the preliminary optical flow. An optical flow is generated by applying the initial optical flow to the selected filter, and The current image is encoded using the optical flow and the previous data, and The filtering information includes at least one of the filter type information or the filter parameter information.

15. An image decoding device, comprising: The obtaining unit (410) is configured to obtain preliminary optical flow feature data and filtering information from the bitstream, wherein the filtering information includes at least one of filter type information or filter parameter information; and The prediction decoding unit (430) obtains the preliminary optical flow by applying the feature data of the preliminary optical flow to a first decoder (14) based on a neural network, generates an optical flow by applying the preliminary optical flow to a filter corresponding to the filtering information, and generates a current reconstructed image by using previous data and the optical flow.

Images