Device and method for performing artificial intelligence encoding and artificial intelligence decoding on images

Abstract

Methods and devices related to artificial intelligence (AI) downscaling and upscaling and techniques related to reducing artifact problems are provided. Some embodiments include: downscaling an original image through a deep neural network (DNN); and generating artifact information from the original image based on frequency transform coefficients, wherein the artifact information represents an area in a first image that includes an artifact in the first image. Post-processing can be performed based on the artifact information to change pixels in the first image to reduce the impact of the artifact.

Description

Technical Field

[0001] The present disclosure relates to image processing. More specifically, the present disclosure relates to a method and apparatus for encoding and decoding images based on artificial intelligence (AI). Background Art

[0002] After an image is encoded by a codec based on a preset data compression standard (eg, the Moving Picture Experts Group (MPEG) standard), the image is stored in a recording medium or transmitted through a communication channel in the form of a bit stream.

[0003] With the development and supply of hardware capable of reproducing and storing high-resolution / high-definition images, the demand for codecs capable of efficiently encoding and decoding high-resolution / high-definition images is increasing. Summary of the invention

[0004] Solution to the problem

[0005] A method and apparatus for artificial intelligence (AI) encoding and AI decoding of an image according to an embodiment of the present disclosure aims to encode and decode an image based on AI so as to achieve a low bit rate.

[0006] In addition, the method and apparatus for artificial intelligence (AI) encoding and AI decoding of an image according to an embodiment of the present disclosure aims to enhance the quality of an image by removing artifacts in the image.

[0007] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure.

[0008] The present invention discloses an artificial intelligence (AI) encoding device, comprising: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to perform the following operations: using a deep neural network (DNN) for downscaling to obtain an original image, wherein the original image includes multiple blocks; estimating an artifact map of the first image based on multiple frequency transform coefficients of the original image, wherein the presence of artifacts is indicated by a first pixel value in the artifact map and the absence of artifacts is indicated by a second pixel value in the artifact map; generating artifact information representing a first area including artifacts in the first image based on the multiple frequency transform coefficients; post-processing the first image based on the artifact map to obtain a post-processed first image, and encoding the post-processed first image to obtain image data.

[0009] In some embodiments of the AI ​​encoding device, the artifact map has a preset size.

[0010] In some embodiments of the AI ​​encoding device, the processor is also configured to execute the one or more instructions to perform the following operations: perform a frequency transform on a first block of the multiple blocks to generate a second multiple frequency transform coefficients corresponding to the first block, wherein the multiple frequency transform coefficients include a second multiple frequency transform coefficients, determine whether a criterion is met based on the distribution of the second multiple frequency transform coefficients, and generate the artifact map when the criterion is met, wherein the first pixel value in the artifact map represents a degree value of each frequency transform coefficient in the second multiple frequency transform coefficients, and each degree value in the multiple degree values ​​corresponds to a frequency transform coefficient in the multiple frequency transform coefficients.

[0011] In some embodiments of the AI ​​encoding device, when the criterion is not met, the artifact map includes a plurality of degree values, and each degree value of the plurality of degree values ​​is equal to the second pixel value.

[0012] In some embodiments of the AI ​​encoding device, the processor is further configured to execute the one or more instructions to perform the following operations: perform morphological processing, smoothing processing, or seamless processing on the first image based on the artifact map.

[0013] In some embodiments of the AI ​​encoding device, the original image is a most recent image among multiple original images in a video sequence, and the processor is further configured to execute the one or more instructions to perform the following operations: post-processing the first image based on comparing the artifact map with at least one second artifact map of a previous original image among the multiple original images in the video sequence.

[0014] In some embodiments of the AI ​​encoding device, the processor is also configured to execute the one or more instructions to post-process the first image by: determining whether K or more previous artifact maps among N previous artifact maps are inconsistent with the artifact map at the first position index of the second multiple frequency transform coefficients, and when K or more previous artifact maps are inconsistent with the artifact map at the first position index, changing the first pixel value of the first block, wherein when the first pixel value of the artifact map at the first position index is different from the first pixel value of each previous artifact map among K or more previous artifact maps at the first position index, the first previous artifact map is inconsistent with the artifact map.

[0015] In some embodiments of the AI ​​encoding device, the processor is also configured to execute the one or more instructions to perform the following operations: perform a frequency transform on a first block of the multiple blocks to generate a second multiple frequency transform coefficients corresponding to the first block, wherein the multiple frequency transform coefficients include a second multiple frequency transform coefficients, calculate the sum of absolute differences (SAD) between a first frequency transform coefficient at a second position index of the second multiple frequency transform coefficients and a second frequency transform coefficient at a second position index of a third multiple frequency transform coefficients associated with a second block of at least one previous original image, wherein the second block is co-located with the first block, determine whether the calculated SAD is within a preset range, and when the calculated SAD is not within the preset range, post-process the first image by changing a first pixel value of the first block.

[0016] In some embodiments of the AI ​​encoding device, the processor is also configured to execute the one or more instructions to perform the following operations: post-process the first image based on the artifact map and based on at least one of the motion vector of the original image or the optical flow of the original image.

[0017] In some embodiments of the AI ​​encoding device, the processor is further configured to execute the one or more instructions to perform the following operations: post-process the first image by applying a filter to the first region based on the artifact information.

[0018] In some embodiments of the AI ​​encoding device, the processor is also configured to execute the one or more instructions to perform the following operations: based on the artifact information, apply a filter to a second area of ​​the original image corresponding to the first area in the first image to produce a reduced area including a second pixel value; and change the third pixel value in the first area of ​​the first image including the artifact to the second pixel value of the reduced area.

[0019] In some embodiments of the AI ​​encoding device, the filter includes at least one of a Gaussian filter or a low-pass filter, and the processing is also configured to execute the one or more instructions to perform the following operations: perform a frequency transform on a first block of the multiple blocks to generate a second multiple frequency transform coefficients corresponding to the first block, wherein the multiple frequency transform coefficients include a second multiple frequency transform coefficients, and determine the cutoff frequency of the low-pass filter based on the maximum value of the transform coefficients in the second multiple frequency transform coefficients located in a high frequency region.

[0020] In some embodiments of the AI ​​encoding device, the filter includes at least one of a bicubic filter, a bilinear filter, or a Lanczos filter.

[0021] In some embodiments of the AI ​​encoding device, the criterion is met when one or more of the first condition, the second condition, the third condition, or the fourth condition is met, the first block includes a DC frequency transform coefficient, a low-frequency region, and a high-frequency region, the first condition is met when a first sum of first energy values ​​of a second plurality of frequency transform coefficients on the high-frequency region exceeds a second sum of second energy values ​​of a second plurality of frequency transform coefficients on the low-frequency region, wherein a scaling of the second sum is a first predetermined scaling, the second condition is met when a third sum of energy values ​​of a first maximum energy value at a third position index of a second frequency transform coefficient included in the second plurality of frequency transform coefficients on the high-frequency region and a frequency transform coefficient located at a fourth position index around the third position index exceeds a fourth sum of energy values ​​of a fifth position index of a third frequency transform coefficient included in the second plurality of frequency transform coefficients on the low-frequency region and a frequency transform coefficient located around the fifth position index, wherein a scaling of the fourth sum is a second predetermined scaling, the third condition is met when the third sum of the high-frequency region exceeds a scaled version of the first sum, wherein a scaling of the first sum is a third predetermined scaling, or the fourth condition is met when the number of times the frequency transform coefficient has energy exceeding a preset value is less than a threshold number.

[0022] The present invention also provides an artificial intelligence (AI) decoding method, comprising: obtaining image data generated as a result of encoding a first image; obtaining AI data including artifact information derived from an original image, wherein the artifact information represents an area including artifacts in the first image; performing decoding of the image data to generate a second image corresponding to the first image; using a deep neural network (DNN) for upscaling to perform AI upscaling on the second image to obtain a third image; and outputting the artifact information and the third image to an image quality engine; and post-processing the third image by the image quality engine by changing pixel values ​​of the third image based on the artifact information, wherein the artifact information is determined at the encoder based on frequency transform coefficients of the original image.

[0023] The present invention also provides an artificial intelligence (AI) encoding method, including: using a deep neural network (DNN) for downscaling to downscale an original image to obtain a first image, wherein the original image includes multiple blocks; estimating an artifact map of the first image based on multiple frequency transform coefficients of the original image, wherein the presence of artifacts is indicated by a first pixel value in the artifact map and the absence of artifacts is indicated by a second pixel value in the artifact map; generating artifact information representing a first area including artifacts in the first image based on the multiple frequency transform coefficients; post-processing the first image based on the artifact map to obtain a post-processed first image, and encoding the post-processed first image to obtain image data.

[0024] The present invention also provides an artificial intelligence (AI) decoding device, comprising: a memory storing one or more instructions; a processor configured to execute the one or more instructions stored in the memory to perform the following operations: obtain image data generated as a result of encoding of a first image and AI data including artifact information derived from an original image, wherein the artifact information represents an area in the first image including artifacts; perform decoding of the image data to generate a second image corresponding to the first image; perform AI upscaling on the second image using a deep neural network (DNN) for upscaling to obtain a third image; output the artifact information and the third image to an image quality engine, and the image quality engine post-processes the third image by changing the pixel values ​​of the third image based on the artifact information, wherein the artifact information is determined at the encoder based on the frequency transform coefficients of the original image.

[0025] In some embodiments of the AI ​​encoding device, when the fourth position index is accommodated within the square area of ​​the first block, the fourth position index is around the third position index, and the square area has a predetermined width.

[0026] According to an embodiment of the present disclosure, an artificial intelligence (AI) encoding device includes: a memory storing one or more instructions; a processor configured to execute one or more instructions stored in the memory to perform the following operations: obtaining a first image reduced by AI from an original image through a deep neural network (DNN) for reduction; generating artifact information representing an area including artifacts in the first image based on values ​​of frequency transform coefficients of the original image; performing post-processing for changing pixel values ​​of pixels in the first image based on the artifact information, and generating image data corresponding to a result of a first encoding of the post-processed first image and AI data including the artifact information.

[0027] The artifact information may include an artifact map of a preset size.

[0028] The processor may also be configured to execute the one or more instructions to perform the following operations: perform a frequency transform on each block of the original image to generate a frequency transform coefficient for each block, determine whether a preset criterion is met based on the distribution of the frequency transform coefficients for each block of the original image, and generate an artifact map including preset pixel values ​​for each block based on the determination result.

[0029] Pixels of a block satisfying a preset criterion among the pixels included in the artifact map may have a first pixel value, and pixels of a block not satisfying the preset criterion may have a second pixel value.

[0030] The processor may be further configured to execute the one or more instructions to perform the following operations: perform morphological processing, smoothing processing, or seamless processing on the artifact map.

[0031] The processor may be further configured to execute the one or more instructions to perform the following operations: compare the artifact map of the original image with at least one artifact map of a previous image, and perform post-processing of changing pixel values ​​of pixels in the first image based on a result of the comparison.

[0032] The processor can also be configured to execute the one or more instructions to perform the following operations: determine whether the number of blocks of first pixel values ​​in N second blocks in an artifact map of N (N is an integer) previous images (e.g., frames) is greater than or equal to K (K is an integer), wherein the N second blocks correspond to a first block of first pixel values ​​in the artifact map of the original image, and perform post-processing of changing the pixel values ​​of the pixels in the first block based on the result of the determination.

[0033] The processor can also be configured to execute the one or more instructions to perform the following operations: calculate the sum of absolute differences (SAD) between transform coefficients of a first block of an original image and transform coefficients of a second block of at least one previous original image, wherein the second block is co-located with the first block, determine whether the calculated SAD is within a preset range, and perform post-processing of changing pixel values ​​of pixels in the first image based on a result of the determination, wherein the first block is determined based on the artifact information.

[0034] The processor may be further configured to execute the one or more instructions to perform the following operations: performing post-processing of changing pixel values ​​of pixels in the first image based on at least one of a motion vector of the original image or an optical flow of the original image.

[0035] The post-processing may include applying a preset filter to a region including the artifact in the first image based on the artifact information.

[0036] The post-processing may include: applying a filter to an area of ​​the original image corresponding to an area including artifacts in the first image based on the artifact information to produce a reduced area; and changing the pixel values ​​of pixels in the area including artifacts in the first image to the pixel values ​​of pixels in the reduced area.

[0037] The filter may include at least one of a Gaussian filter or a low-pass filter, and determine the cutoff frequency of the low-pass filter based on the maximum value of the transformation coefficients located in the high-frequency area among the frequency transformation coefficients of the area of ​​the original image corresponding to the area including the artifact in the first image.

[0038] The filter may include at least one of a bicubic filter, a bilinear filter, or a Lanczos filter.

[0039] The preset criterion may be based on at least one of the following comparison results: a first comparison result obtained by comparing the sum of the values ​​of the transform coefficients of the low-frequency region of each block of the original image with the sum of the values ​​of the transform coefficients of the high-frequency region of the original image, a second comparison result obtained by comparing a first sum of the values ​​of the transform coefficients of the high-frequency region of each block and a first transform coefficient located around the maximum value with a sum of the values ​​of the transform coefficients of the low-frequency region and a second transform coefficient located around the maximum value, a third comparison result obtained by comparing the sum of the transform coefficients of the high-frequency region with the first sum of the high-frequency region, or a fourth comparison result obtained by comparing a preset value with the number of transform coefficients having a value greater than the preset value.

[0040] According to an embodiment of the present disclosure, an artificial intelligence (AI) decoding method includes: obtaining image data generated as a result of a first encoding of a first image and AI data including artifact information indicating an area including an artifact; performing a first decoding on the image data to generate a second image corresponding to the first image; obtaining a third image enlarged by AI from the second image through a deep neural network (DNN) for enlargement; and outputting the artifact information and the third image to an image quality engine, wherein the artifact information is determined based on the value of a frequency transform coefficient related to the original image, and the image quality engine performs post-processing of changing the pixel values ​​of pixels in the third image based on the artifact information.

[0041] According to an embodiment of the present disclosure, an artificial intelligence (AI) encoding method includes: obtaining a first image that is AI-reduced from an original image through a deep neural network (DNN) for reduction; generating artifact information representing an area including artifacts in the first image based on values ​​of frequency transform coefficients related to the original image; and performing post-processing of changing pixel values ​​of pixels in the first image based on the artifact information; and generating image data corresponding to a result of a first encoding of the post-processed first image and AI data including the artifact information.

[0042] According to an embodiment of the present disclosure, an artificial intelligence (AI) decoding device includes: a memory storing one or more instructions; a processor configured to execute the one or more instructions stored in the memory to perform the following operations: obtain image data generated as a result of a first encoding of a first image and artifact information representing an area including an artifact; perform a first decoding on the image data to generate a second image corresponding to the first image; obtain a third image enlarged by AI from the second image through a deep neural network (DNN) for enlargement; and output the artifact information and the third image to an image quality engine, wherein the artifact information is determined based on the value of a frequency transform coefficient related to the original image, and the image quality engine performs post-processing of changing the pixel values ​​of pixels in the third image based on the artifact information. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0044] A brief description of each figure is provided to provide a more complete understanding of the drawings described in this specification;

[0045] Figure 1 is a diagram for describing an artificial intelligence (AI) encoding process and an AI decoding process according to an embodiment;

[0046] Figure 2 is a block diagram of a configuration of an AI decoding device according to an embodiment;

[0047] Figure 3 is a diagram illustrating a second deep neural network (DNN) for performing AI upscaling on a second image;

[0048] Figure 4 is a diagram for describing a convolution operation performed by a convolutional layer;

[0049] Figure 5 is a table showing a mapping relationship between several pieces of image-related information and several pieces of DNN setting information;

[0050] Figure 6 is a diagram showing a second image including a plurality of frames;

[0051] Figure 7 is a block diagram of a configuration of an AI encoding device according to an embodiment;

[0052] Figure 8 is a diagram illustrating a first DNN for performing AI downscaling on an original image;

[0053] Fig. 9 is a diagram for describing a method of training a first DNN and a second DNN;

[0054] Fig.10 is a diagram for describing a training process of a training device on a first DNN and a second DNN;

[0055] Fig.11 is a diagram of an apparatus for performing AI downscaling on an original image and an apparatus for performing AI upscaling on a second image;

[0056] Fig.12 is a block diagram showing a configuration of an AI encoding device according to another embodiment of the present disclosure;

[0057] Fig.13 is a diagram for describing a method performed by an AI encoding device for determining an artifact map based on values ​​of frequency transform coefficients of an original image and post-processing a first image based on the artifact map;

[0058] Fig.14 is a diagram for describing a method of determining an artifact area based on distribution of transform coefficients in a block included in an original image;

[0059] Fig.15a is an example diagram showing spatial aliasing artifacts occurring in a first image reduced from an original image;

[0060] Fig.15b is an example diagram showing spatiotemporal aliasing artifacts occurring in a first image reduced from an original image;

[0061] Fig.15c is a diagram for describing a spatiotemporal aliasing artifact recognized by human eyes among spatiotemporal aliasing artifacts appearing in a first image reduced from an original image;

[0062] Fig.15d is a diagram for describing spatial aliasing artifacts recognized by human eyes among spatial aliasing artifacts appearing in a first image reduced from an original image;

[0063] Fig.16 An example of an original image and a first image partitioned in units of blocks is shown.

[0064] Fig.17a and Fig.17b is a diagram for illustrating an artifact map before and after outlier removal;

[0065] Fig.18a and Fig.18b is a diagram for describing a process of removing an outlier by an AI encoding device according to an embodiment of the present disclosure;

[0066] Fig.19a and Fig.19bis a diagram for describing a process of removing an outlier by an AI encoding device according to another embodiment of the present disclosure;

[0067] Figures 20a to 20f is a diagram for describing a method of correcting an artifact map by performing a smoothing process and a seamless process on a post-processing application area of ​​the artifact map according to an embodiment of the present disclosure;

[0068] Fig.21 is a diagram for describing a method of correcting an artifact area by comparing a block in an artifact map of a current frame with a co-located block in an artifact map of an immediately previous frame;

[0069] Fig.22a is a flowchart for describing an AI encoding method according to another embodiment of the present disclosure;

[0070] Figure 22b is a diagram for describing an AI encoding method according to another embodiment of the present disclosure;

[0071] Fig.23 is a block diagram showing a configuration of an AI decoding device according to another embodiment of the present disclosure;

[0072] Fig.24a is a diagram for illustrating an image quality engine;

[0073] Figure 24b is a diagram for describing a method performed by an AI decoding device for adjusting the strength of a low-pass filter based on artifact information and performing filtering;

[0074] Fig.25 is a block diagram showing a configuration of an AI decoding device according to another embodiment of the present disclosure; and

[0075] Fig.26 is a flowchart for describing an AI decoding method according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0076] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0077] Since the present disclosure allows various changes and many examples, specific embodiments will be described in detail in the drawings and in the written description. However, this is not intended to limit the present disclosure to a specific practice, and it should be understood that all changes, equivalents and substitutions that do not depart from the spirit and technical scope of the present disclosure are included in the present disclosure.

[0078] In the description of the embodiments, when it is considered that a specific detailed explanation of the related art may unnecessarily obscure the essence of the present disclosure, the detailed explanation of the related art is omitted. In addition, the numbers (e.g., first, second, etc.) used in the description of the specification are merely identification codes for distinguishing one element from another element.

[0079] Furthermore, in the present specification, it should be understood that when elements are “connected” or “coupled” to each other, the elements may be directly connected or coupled to each other or may be connected or coupled to each other through an intermediate element therebetween, unless otherwise specified.

[0080] In this specification, with respect to an element represented as a "unit" or a "module", two or more elements may be combined into one element, or one element may be divided into two or more elements according to subdivided functions. In addition, each element described below may additionally perform some or all of the functions performed by another element in addition to its own main function, and some of the main functions of each element may be completely performed by another component.

[0081] Furthermore, in the present specification, an “image” or a “picture” may refer to a still image, a moving image including a plurality of consecutive still images (or frames), or a video.

[0082] Furthermore, in this specification, a deep neural network (DNN) is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model that uses a specific algorithm.

[0083] In addition, in this specification, "parameter" is a value used in the operation process of each layer forming a neural network, and may include, for example, a weight used when an input value is applied to a specific operation expression. Here, the parameter may be expressed in a matrix form. The parameter is a value set as a training result, and may be updated by separate training data when necessary.

[0084] In addition, in this specification, a “first DNN” indicates a DNN used to perform artificial intelligence (AI) downscaling on an image, and a “second DNN” indicates a DNN used to perform artificial intelligence (AI) upscaling on an image.

[0085] In addition, in the present specification, "DNN setting information" includes information related to elements constituting the DNN. "DNN setting information" includes the above-mentioned parameters as information related to elements constituting the DNN. The first DNN or the second DNN can be set by using the DNN setting information.

[0086] In addition, in this specification, the "original image" means an image that is a target of AI encoding, and the "first image" means an image obtained as a result of performing AI reduction on the original image during the AI ​​encoding process. In addition, the "second image" means an image obtained via the first decoding during the AI ​​decoding process, and the "third image" means an image obtained by performing AI enlargement on the second image during the AI ​​decoding process.

[0087] In addition, in this specification, "AI reduction" means a process of reducing the image resolution based on AI, and "first encoding" means an encoding process according to an image compression method based on frequency transformation. In addition, "first decoding" means a decoding process according to an image reconstruction method based on frequency transformation, and "AI enlargement" means a process of increasing the image resolution based on AI.

[0088] Figure 1 is a diagram for describing an AI encoding process and an AI decoding process according to an embodiment.

[0089] As described above, when the resolution of an image significantly increases, the throughput of information for encoding and decoding the image increases, and thus, a method for improving encoding and decoding efficiency of an image is required.

[0090] like Figure 1 As shown, according to an embodiment of the present disclosure, a first image 115 is obtained by performing AI reduction 110 on an original image 105 having a high resolution. Then, a first encoding 120 and a first decoding 130 are performed on the first image 115 having a relatively low resolution, so that the bit rate can be greatly reduced compared to when the first encoding and the first decoding are performed on the original image 105.

[0091] Specifically, in Figure 1 According to an embodiment, during the AI ​​encoding process, a first image 115 is obtained by performing AI reduction 110 on an original image 105, and a first encoding 120 is performed on the first image 115. During the AI ​​decoding process, AI encoded data including AI data and image data obtained as a result of the AI ​​encoding is received, a second image 135 is obtained via the first decoding 130, and a third image 145 is obtained by performing AI enlargement 140 on the second image 135.

[0092] Referring to the AI ​​encoding process in detail, when the original image 105 is received, AI reduction 110 is performed on the original image 105 to obtain a first image 115 of a specific resolution or a specific quality. Here, the AI ​​reduction 110 is performed based on AI, and the AI ​​for the AI ​​reduction 110 needs to be jointly trained with the AI ​​for the AI ​​enlargement 140 of the second image 135. This is because when the AI ​​for the AI ​​reduction 110 and the AI ​​for the AI ​​enlargement 140 are trained separately, the difference between the original image 105 as the AI ​​encoding object and the third image 145 reconstructed by AI decoding increases.

[0093] In an embodiment of the present disclosure, the AI ​​data may be used to maintain this joint relationship during the AI ​​encoding process and the AI ​​decoding process. Therefore, the AI ​​data obtained by the AI ​​encoding process may include information indicating the magnification target, and during the AI ​​decoding process, the AI ​​magnification 140 is performed on the second image 135 according to the magnification target verified based on the AI ​​data.

[0094] The AI ​​for AI reduction 110 and the AI ​​for AI upscaling 140 may be implemented as DNNs. Fig. 9 As described, since the first DNN and the second DNN are jointly trained by sharing loss information under a specific target, the AI ​​encoding device may provide the target information used during the joint training of the first DNN and the second DNN to the AI ​​decoding device, and the AI ​​decoding device may perform AI upscaling 140 on the second image 135 based on the provided target information to achieve the target resolution.

[0095] about Figure 1 The first encoding 120 and the first decoding 130 can reduce the amount of information of the first image 115 obtained by performing AI reduction 110 on the original image 105 through the first encoding 120. The first encoding 120 may include a process of generating prediction data by predicting the first image 115, a process of generating residual data corresponding to the difference between the first image 115 and the prediction data, a process of transforming the residual data of the spatial domain component into a frequency domain component, a process of quantizing the residual data transformed into the frequency domain component, and a process of entropy encoding the quantized residual data. Such a first encoding 120 can be performed via one of the image compression methods using frequency transformation (such as MPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, High Efficiency Video Coding (HEVC), VC-1, VP8, VP9, ​​and Alliance for Open Media 1 (AV1)).

[0096] The second image 135 corresponding to the first image 115 may be reconstructed by performing the first decoding 130 on the image data. The first decoding 130 may include a process of generating quantized residual data by entropy decoding the image data, a process of inversely quantizing the quantized residual data, a process of transforming the residual data of the frequency domain component into the spatial domain component, a process of generating prediction data, and a process of reconstructing the second image 135 by using the prediction data and the residual data. Such a first decoding 130 may be performed via an image reconstruction method corresponding to one of the image compression methods using frequency transformation (such as MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1), wherein the image compression method is the image compression method used in the first encoding 120.

[0097] The AI ​​encoded data obtained through the AI ​​encoding process may include image data obtained as a result of performing the first encoding 120 on the first image 115, and AI data related to the AI ​​reduction 110 of the original image 105. The image data may be used during the first decoding 130, and the AI ​​data may be used during the AI ​​enlargement 140.

[0098] The image data may be transmitted in the form of a bitstream. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data as a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding 120 performed on the first image 115. For example, the image data may include prediction mode information, motion information, and information related to quantization parameters used during the first encoding 120. The image data may be generated according to the rules (e.g., according to the syntax) of the image compression method used during the first encoding 120 in MPEG-2, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1.

[0099] The AI ​​data is used in the AI ​​upscaling 140 based on the second DNN. As described above, because the first DNN and the second DNN are jointly trained, the AI ​​data includes information that enables the AI ​​upscaling 140 to be accurately performed on the second image 135 by the second DNN. During the AI ​​decoding process, the AI ​​upscaling 140 can be performed on the second image 135 based on the AI ​​data to have a target resolution and / or quality.

[0100] The AI ​​data may be transmitted together with the image data in the form of a bit stream. Alternatively, according to an embodiment, the AI ​​data may be transmitted separately from the image data in the form of a frame or a packet. The AI ​​data and the image data obtained as a result of the AI ​​encoding may be transmitted through the same network or through different networks.

[0101] Figure 2 is a block diagram of a configuration of the AI ​​decoding apparatus 200 according to an embodiment.

[0102] Reference Figure 2 , the AI ​​decoding device 200 according to the embodiment may include a receiver 210 and an AI decoder 230. The receiver 210 may include a communicator 212, a parser 214, and an outputter 216. The AI ​​decoder 230 may include a first decoder 232 and an AI amplifier 234.

[0103] The receiver 210 receives and parses AI-encoded data obtained as a result of AI encoding, and outputs image data and AI data to the AI ​​decoder 230, respectively.

[0104] Specifically, the communicator 212 receives AI-encoded data obtained as a result of AI encoding through a network. The AI-encoded data obtained as a result of performing AI encoding includes image data and AI data. The image data and AI data may be received through the same type of network or different types of networks.

[0105] The parser 214 receives the AI ​​coded data received by the communicator 212, and parses the AI ​​coded data to distinguish the image data from the AI ​​data. For example, the parser 214 can distinguish the image data from the AI ​​data by reading the header of the data obtained from the communicator 212. According to an embodiment, the parser 214 sends the image data and the AI ​​data to the outputter 216 via the header of the data received by the communicator 212, and the outputter 216 sends the distinguished image data and the AI ​​data to the first decoder 232 and the AI ​​amplifier 234, respectively. At this time, it can be verified that the image data included in the AI ​​coded data is image data generated by a specific codec (e.g., MPEG-2, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9 or AV1). In this case, the corresponding information can be sent to the first decoder 232 by the outputter 216, so that the image data is processed by the verified codec.

[0106] According to an embodiment, the AI ​​encoded data parsed by the parser 214 can be obtained from a data storage medium, which includes a magnetic medium (such as a hard disk, floppy disk or tape), an optical recording medium (such as a CD-ROM or DVD) or a magneto-optical medium (such as a magneto-optical floppy disk).

[0107] The first decoder 232 reconstructs a second image 135 corresponding to the first image 115 based on the image data. The second image 135 obtained by the first decoder 232 is provided to the AI ​​amplifier 234. According to an embodiment, first decoding related information (such as prediction mode information, motion information, quantization parameter information, etc.) included in the image data may be further provided to the AI ​​amplifier 234.

[0108] Upon receiving the AI ​​data, the AI ​​amplifier 234 performs AI up-amplification on the second image 135 based on the AI ​​data. According to an embodiment, the AI ​​up-amplification may be performed by further using first decoding related information (such as prediction mode information, quantization parameter information, etc.) included in the image data.

[0109] The receiver 210 and the AI ​​decoder 230 according to the embodiment are described as separate devices, but can be implemented by one processor. In this case, the receiver 210 and the AI ​​decoder 230 can be implemented by a dedicated processor or by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). The dedicated processor can be implemented by including a memory for implementing the embodiments of the present disclosure or by including a storage processor for using an external memory.

[0110] In addition, the receiver 210 and the AI ​​decoder 230 may be configured by a plurality of processors. In this case, the receiver 210 and the AI ​​decoder 230 may be implemented by a combination of dedicated processors or by a combination of software and a general-purpose processor such as an AP, a CPU, or a GPU. Similarly, the AI ​​amplifier 234 and the first decoder 232 may be implemented by different processors.

[0111] The AI ​​data provided to the AI ​​amplifier 234 includes information that enables the second image 135 to be processed via AI enlargement. Here, the enlargement target should correspond to the reduction target of the first DNN. Therefore, the AI ​​data includes information for verifying the reduction target of the first DNN.

[0112] Examples of the information included in the AI ​​data include difference information between the resolution of the original image 105 and the resolution of the first image 115 and information related to the first image 115 .

[0113] The difference information may be represented as information about the degree of resolution conversion of the first image 115 compared to the original image 105 (e.g., resolution conversion rate information). In addition, since the resolution of the first image 115 is verified by the resolution of the reconstructed second image 135, and the degree of resolution conversion is verified accordingly, the difference information may be represented only as the resolution information of the original image 105. Here, the resolution information may be represented as a vertical / horizontal size or as a ratio (16:9, 4:3, etc.) and the size of one axis. In addition, when there is pre-set resolution information, the resolution information may be represented in the form of an index or a flag.

[0114] The information related to the first image 115 may include information about at least one of a bit rate of image data obtained as a result of performing the first encoding on the first image 115 or a codec type used during the first encoding of the first image 115 .

[0115] The AI ​​amplifier 234 may determine an enlargement target of the second image 135 based on at least one of the difference information included in the AI ​​data or the information related to the first image 115. The enlargement target may indicate, for example, to what extent the resolution of the second image 135 will be enlarged. When the enlargement target is determined, the AI ​​amplifier 234 performs AI enlargement on the second image 135 through the second DNN to obtain a third image 145 corresponding to the enlargement target.

[0116] Before describing the method of performing AI up-scaling on the second image 135 according to the up-scaling target performed by the AI ​​amplifier 234, reference will be made to Figure 3 and Figure 4 Describe the AI ​​amplification process through the second DNN.

[0117] Figure 3 is a diagram showing a second DNN 300 for performing AI upscaling on a second image 135, and Figure 4 Is used to describe Figure 3 Schematic diagram of the convolution operation in the first convolutional layer 310 of FIG.

[0118] like Figure 3 As shown, the second image 135 is input to the first convolutional layer 310. Figure 3 The 3×3×4 indicated in the illustrated first convolution layer 310 indicates that a convolution process is performed on one input image by using four filter kernels having a size of 3×3. As a result of the convolution process, four feature maps are generated by the four filter kernels. Each feature map indicates an inherent characteristic of the second image 135. For example, each feature map may represent a vertical direction characteristic, a horizontal direction characteristic, an edge characteristic, etc. of the second image 135.

[0119] Will refer to Figure 4The convolution operation in the first convolutional layer 310 is described in detail.

[0120] One feature map 450 may be generated by multiplication and addition between the parameters of the filter kernel 430 of size 3×3 used in the first convolution layer 310 and the corresponding pixel values ​​in the second image 135. Since four filter kernels are used in the first convolution layer 310, four feature maps may be generated by convolution operations using the four filter kernels.

[0121] exist Figure 4 I1 to I49 indicated in the second image 135 in FIG. 4 indicate pixels in the second image 135, and F1 to F9 indicated in the filter kernel 430 indicate parameters of the filter kernel 430. In addition, M1 to M9 indicated in the feature map 450 indicate samples of the feature map 450.

[0122] exist Figure 4 , the second image 135 includes 49 pixels, but the number of pixels is only an example, and when the second image 135 has a resolution of 4K, the second image 135 may include, for example, 3840×2160 pixels.

[0123] During the convolution operation process, the pixel values ​​of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination (e.g., addition) of the multiplied result values ​​can be designated as the value of M1 of the feature map 450. When the stride of the convolution operation is 2, the pixel values ​​of I3, I4, I5, I10, I11, I12, I17, I18, and I19 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination of the multiplied result values ​​can be designated as the value of M2 of the feature map 450.

[0124] When the filter kernel 430 moves to the last pixel of the second image 135 along the stride, a convolution operation is performed between the pixel values ​​in the second image 135 and the parameters of the filter kernel 430, and thus a feature map 450 having a specific size may be generated.

[0125] According to the present disclosure, the values ​​of the parameters of the second DNN, for example, the values ​​of the parameters of the filter kernel used in the convolution layer of the second DNN (for example, F1 to F9 of the filter kernel 430), can be optimized by joint training of the first DNN and the second DNN. As described above, the AI ​​amplifier 234 can determine the enlargement target corresponding to the reduction target of the first DNN based on the AI ​​data, and determine the parameters corresponding to the determined enlargement target as the parameters of the filter kernel used in the convolution layer of the second DNN.

[0126] The convolutional layers included in the first DNN and the second DNN can be constructed according to the reference Figure 4 The convolution operation described above is used to perform the processing, but referring to Figure 4 The convolution operation process described is only an example and is not limited thereto.

[0127] Return to reference Figure 3 , the feature map output from the first convolution layer 310 may be input to the first activation layer 320 .

[0128] The first activation layer 320 may assign nonlinear features to each feature map. The first activation layer 320 may include a sigmoid function, a Tanh function, a rectified linear unit (ReLU) function, etc., but is not limited thereto.

[0129] The first activation layer 320 to which the nonlinear feature is assigned indicates that at least one sample value of the feature map, which is the output of the first convolution layer 310, is changed. Here, the change is performed by applying the nonlinear feature.

[0130] The first activation layer 320 determines whether to send the sample values ​​of the feature map output from the first convolution layer 310 to the second convolution layer 330. For example, some sample values ​​of the feature map are activated by the first activation layer 320 and sent to the second convolution layer 330, and some sample values ​​are deactivated by the first activation layer 320 and are not sent to the second convolution layer 330. The inherent characteristics of the second image 135 represented by the feature map are highlighted by the first activation layer 320.

[0131] The feature map 325 output from the first activation layer 320 is input to the second convolutional layer 330 . Figure 3 One of the feature maps 325 shown in FIG. 3 is a feature map in the first activation layer 320 for processing reference Figure 4 The result of the feature map 450 is described.

[0132] 3×3×4 indicated in the second convolution layer 330 indicates that a convolution process is performed on the feature map 325 by using four filters having a size of 3×3. The output of the second convolution layer 330 is input to the second activation layer 340. The second activation layer 340 may assign nonlinear features to input data.

[0133] The feature map 345 output from the second activation layer 340 is input to the third convolutional layer 350 . Figure 3 3×3×1 indicated in the third convolution layer 350 shown in indicates that a convolution process is performed by using a filter kernel having a size of 3×3 to generate an output image. The third convolution layer 350 is a layer for outputting a final image, and generates an output by using a filter kernel. According to an embodiment of the present disclosure, the third convolution layer 350 may output the third image 145 as a result of the convolution operation.

[0134] As will be described later, there may be a plurality of pieces of DNN setting information indicating the number of filter kernels of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, parameters of the filter kernels of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, and the like, and the plurality of pieces of DNN setting information should be associated with the plurality of pieces of DNN setting information of the first DNN. The association between the plurality of pieces of DNN setting information of the second DNN and the plurality of DNN setting information of the first DNN may be achieved via joint training of the first DNN and the second DNN.

[0135] exist Figure 3 , the second DNN 300 includes three convolutional layers (a first convolutional layer 310, a second convolutional layer 330, and a third convolutional layer 350) and two activation layers (a first activation layer 320 and a second activation layer 340), but this is only an example, and the number of convolutional layers and activation layers may vary according to the embodiment. In addition, according to the embodiment, the second DNN 300 may be implemented as a recurrent neural network (RNN). In this case, the convolutional neural network (CNN) structure of the second DNN 300 according to the embodiment of the present disclosure is changed to an RNN structure.

[0136] According to an embodiment, the AI ​​amplifier 234 may include at least one arithmetic logic unit (ALU) for the convolution operation described above and the operation of the activation layer. The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier and an adder, wherein the multiplier performs multiplication between the sample value of the second image 135 or the feature map output from the previous layer and the sample value of the filter kernel, and the adder adds the result value of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in the predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0137] Hereinafter, a method of performing AI up-scaling on the second image 135 according to the up-scaling target, performed by the AI ​​amplifier 234, will be described.

[0138] According to an embodiment, the AI ​​amplifier 234 may store a plurality of pieces of DNN setting information that may be set in the second DNN.

[0139] Here, the DNN setting information may include information about at least one of the number of convolutional layers included in the second DNN, the number of filter cores of each convolutional layer, or the parameters of each filter core. Multiple pieces of DNN setting information may correspond to various amplification targets, respectively, and the second DNN may operate based on the DNN setting information corresponding to the specific amplification target. Based on the DNN setting information, the second DNN may have a different structure. For example, the second DNN may include three convolutional layers based on any one piece of DNN setting information, and may include four convolutional layers based on another piece of DNN setting information.

[0140] According to an embodiment, the DNN setting information may include only the parameters of the filter kernel used in the second DNN. In this case, the structure of the second DNN does not change, but only the parameters of the internal filter kernel may be changed based on the DNN setting information.

[0141] The AI ​​amplifier 234 may obtain DNN setting information from among the plurality of DNN setting information for performing AI amplification on the second image 135. Each of the plurality of DNN setting information used at this time is information for obtaining the third image 145 of a predetermined resolution and / or a predetermined quality, and is jointly trained with the first DNN.

[0142] For example, one piece of DNN setting information among the multiple DNN setting information may include information for obtaining a third image 145 having a resolution twice as high as the resolution of the second image 135 (e.g., a third image 145 of 4K (4096×2160) which is twice as high as the 2K (2048×1080) of the second image 135), and another piece of DNN setting information may include information for obtaining a third image 145 having a resolution four times as high as the resolution of the second image 135 (e.g., a third image 145 of 8K (8192×4320) which is four times as high as the 2K (2048×1080) of the second image 135).

[0143] Each of the multiple DNN setting information is Figure 6 The AI ​​encoding device 600 of the embodiment obtains the DNN setting information of the first DNN jointly, and the AI ​​amplifier 234 obtains one of the multiple DNN setting information according to the magnification rate corresponding to the reduction rate in the DNN setting information of the first DNN. In this regard, the AI ​​amplifier 234 can verify the information of the first DNN. In order for the AI ​​amplifier 234 to verify the information of the first DNN, the AI ​​decoding device 200 according to the embodiment receives AI data including the information of the first DNN from the AI ​​encoding device 600.

[0144] In other words, the AI ​​amplifier 234 can verify the information targeted by the DNN setting information of the first DNN used to obtain the first image 115 by using the information received from the AI ​​encoding device 600, and obtain the DNN setting information of the second DNN jointly trained with the DNN setting information of the first DNN.

[0145] When DNN setting information for performing AI up-scaling on the second image 135 is obtained from a plurality of pieces of DNN setting information, input data may be processed based on a second DNN operated according to the obtained DNN setting information.

[0146] For example, when any DNN setting information is obtained, including Figure 3 The number of filter kernels included in each of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300 and the parameters of the filter kernels are set to the values ​​included in the obtained DNN setting information.

[0147] Specifically, in Figure 4 The parameters of the 3×3 filter kernel used in any convolutional layer of the second DNN are set to {1, 1, 1, 1, 1, 1, 1, 1}, and when the DNN setting information is changed later, these parameters are replaced by the parameters {2, 2, 2, 2, 2, 2, 2, 2} included in the changed DNN setting information.

[0148] The AI ​​amplifier 234 may obtain DNN setting information for AI amplification from a plurality of pieces of DNN setting information based on information included in the AI ​​data, and the AI ​​data for obtaining the DNN setting information will now be described.

[0149] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification from a plurality of pieces of DNN setting information based on the difference information included in the AI ​​data. For example, when it is verified based on the difference information that the resolution of the original image 105 (e.g., 4K (4096×2160)) is twice the resolution of the first image 115 (e.g., 2K (2048×1080)), the AI ​​amplifier 234 may obtain DNN setting information for increasing the resolution of the second image 135 by two times.

[0150] According to another embodiment, the AI ​​amplifier 234 may obtain DNN setting information for performing AI amplification on the second image 135 from a plurality of pieces of DNN setting information based on information related to the first image 115 included in the AI ​​data. The AI ​​amplifier 234 may predetermine a mapping relationship between image-related information and DNN setting information, and obtain DNN setting information mapped to information related to the first image 115.

[0151] Figure 5 is a table showing mapping relationships between several pieces of image-related information and several pieces of DNN setting information.

[0152] By following Figure 5 In the embodiment of the present disclosure, it will be determined that the AI ​​encoding and AI decoding processing according to the embodiment of the present disclosure will not only consider the change in resolution. Figure 5 As shown, the DNN setting information may be selected by considering resolution such as standard definition (SD), high definition (HD), or full HD, bit rate such as 10Mbps, 15Mbps, or 20Mbps, and codec information such as AV1, H.264, or HEVC, individually or collectively. With such consideration of resolution, bit rate, and codec information, training considering each element may be performed jointly with encoding and decoding processing during AI training processing (see Fig. 9 ).

[0153] Therefore, when training Figure 5 When multiple DNN setting information is provided based on image-related information including codec type, resolution of the image, etc. as shown in , DNN setting information for AI upscaling the second image 135 can be obtained based on information related to the first image 115 received during AI decoding processing.

[0154] In other words, AI amplifier 234 can be matched Figure 5 The image related information on the left side of the table and the DNN setting information on the right side of the table are used according to the image related information.

[0155] like Figure 5 As shown in , when it is verified from information related to the first image 115 that the resolution of the first image 115 is SD, the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 10Mbps, and the first encoding is performed on the first image 115 via the AV1 codec, the AI ​​amplifier 234 can use A DNN setting information among multiple DNN setting information.

[0156] In addition, when it is verified from information related to the first image 115 that the resolution of the first image 115 is HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the H.264 codec, the AI ​​amplifier 234 can use B DNN setting information among multiple DNN setting information.

[0157] In addition, when it is verified from the information related to the first image 115 that the resolution of the first image 115 is FUll HD, the bit rate of the image data obtained as a result of performing the first encoding is 20Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 can use the C DNN setting information among the multiple DNN setting information, and when it is verified that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 can use the DDNN setting information among the multiple DNN setting information. Whether it is C DNN setting information or D DNN setting information is selected based on whether the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 20Mbps or 15Mbps. Different bit rates of image data obtained when performing the first encoding on the first image 115 of the same resolution via the same codec indicate different qualities of the reconstructed image. Therefore, the first DNN and the second DNN can be jointly trained based on a specific image quality, and therefore, the AI ​​amplifier 234 can obtain the DNN setting information according to the bit rate of the image data indicating the quality of the second image 135.

[0158] According to another embodiment, the AI ​​amplifier 234 may consider both the information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and the information related to the first image 115 included in the AI ​​data, and obtain DNN setting information for performing AI amplification on the second image 135 from a plurality of DNN setting information. For example, the AI ​​amplifier 234 may receive the quantization parameter information used during the first encoding process of the first image 115 from the first decoder 232, verify the bit rate of the image data obtained from the AI ​​data as the encoding result of the first image 115, and obtain DNN setting information corresponding to the quantization parameter information and the bit rate. Even when the bit rate is the same, the quality of the reconstructed image may vary depending on the complexity of the image. The bit rate is a value representing the entire first image 115 on which the first encoding is performed, and the quality of each frame may vary even within the first image 115. Therefore, when the prediction mode information, motion information, and / or quantization parameter obtainable for each frame from the first decoder 232 are considered together, DNN setting information more suitable for the second image 135 can be obtained compared to when only the AI ​​data is used.

[0159] In addition, according to an embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information. The identifier of the DNN setting information is information for distinguishing a pair of DNN setting information jointly trained between the first DNN and the second DNN, so that AI amplification is performed on the second image 135 to an amplification target corresponding to the reduction target of the first DNN. After obtaining the identifier of the DNN setting information included in the AI ​​data, the AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information corresponding to the identifier of the DNN setting information. For example, an identifier indicating each of a plurality of DNN setting information that can be set in the first DNN and an identifier indicating each of a plurality of DNN setting information that can be set in the second DNN may be pre-specified. In this case, the same identifier may be specified for a pair of DNN setting information that can be set in each of the first DNN and the second DNN. The AI ​​data may include an identifier of the DNN setting information set in the first DNN for AI reduction of the original image 105. The AI ​​amplifier 234 receiving the AI ​​data may perform AI amplification on the second image 135 by using the DNN setting information indicated by the identifier included in the AI ​​data among the plurality of DNN setting information.

[0160] In addition, according to an embodiment, the AI ​​data may include DNN setting information. The AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information after obtaining the DNN setting information included in the AI ​​data.

[0161] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter cores of each convolutional layer, the parameters of each filter core, etc.) are stored in the form of a lookup table, the AI ​​amplifier 234 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on the information included in the AI ​​data, and perform AI amplification on the second image 135 by using the obtained DNN setting information.

[0162] According to an embodiment, when the structure of the DNN corresponding to the amplification target is determined, the AI ​​amplifier 234 may obtain DNN setting information corresponding to the determined structure of the DNN, for example, parameters of a filter kernel.

[0163] The AI ​​amplifier 234 obtains DNN setting information of the second DNN through AI data including information related to the first DNN, and performs AI amplification on the second image 135 through the second DNN set based on the obtained DNN setting information, and in this case, memory usage and throughput can be reduced compared to directly analyzing features of the second image 135 for amplification.

[0164] According to an embodiment, when the second image 135 includes a plurality of frames, the AI ​​amplifier 234 may independently obtain DNN setting information for a specific number of frames, or may obtain common DNN setting information for all frames.

[0165] Figure 6 is a diagram illustrating a second image 135 including a plurality of frames.

[0166] like Figure 6 As shown in , the second image 135 may include frames t0 to tn.

[0167] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information of the second DNN through the AI ​​data, and perform AI amplification on frames t0 to tn based on the obtained DNN setting information. In other words, frames t0 to tn may be processed via AI amplification based on the common DNN setting information.

[0168] According to another embodiment, the AI ​​amplifier 234 may perform AI amplification on some of the frames t0 to tn (e.g., frames t0 to ta) by using the "A" DNN setting information obtained from the AI ​​data, and perform AI amplification on the frames ta+1 to tb by using the "B" DNN setting information obtained from the AI ​​data. In addition, the AI ​​amplifier 234 may perform AI amplification on the frames tb+1 to tn by using the "C" DNN setting information obtained from the AI ​​data. In other words, the AI ​​amplifier 234 may independently obtain DNN setting information for each group including a specific number of frames among a plurality of frames, and perform AI amplification on the frames included in each group by using the independently obtained DNN setting information.

[0169] According to another embodiment, the AI ​​amplifier 234 may independently obtain DNN setting information for each frame constituting the second image 135. In other words, when the second image 135 includes three frames, the AI ​​amplifier 234 may perform AI amplification on the first frame by using the DNN setting information obtained in relation to the first frame, perform AI amplification on the second frame by using the DNN setting information obtained in relation to the second frame, and perform AI amplification on the third frame by using the DNN setting information obtained in relation to the third frame. According to a method of obtaining DNN setting information based on information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and information related to the first image 115 included in the above-mentioned AI data, DNN setting information may be independently obtained for each frame included in the second image 135. This is because mode information, quantization parameter information, etc. may be independently determined for each frame included in the second image 135.

[0170] According to another embodiment, the AI ​​data may include information about which frame the DNN setting information obtained based on the AI ​​data is valid for. For example, when the AI ​​data includes information indicating that the DNN setting information is valid until frame ta, the AI ​​amplifier 234 performs AI amplification on frames t0 to ta by using the DNN setting information obtained based on the AI ​​data. In addition, when another piece of AI data includes information indicating that the DNN setting information is valid until frame tn, the AI ​​amplifier 234 performs AI amplification on frames ta+1 to tn by using the DNN setting information obtained based on the other piece of AI data.

[0171] In the following, reference will be made to Figure 7 An AI encoding apparatus 600 for performing AI encoding on an original image 105 is described.

[0172] Figure 7 is a block diagram of a configuration of an AI encoding apparatus 600 according to an embodiment.

[0173] Reference Figure 7 , the AI ​​encoding device 600 may include an AI encoder 610 and a transmitter 630. The AI ​​encoder 610 may include an AI reducer 612 and a first encoder 614. The transmitter 630 may include a data processor 632 and a communicator 634.

[0174] exist Figure 7 In the embodiment, the AI ​​encoder 610 and the transmitter 630 are described as separate devices, but the AI ​​encoder 610 and the transmitter 630 may be implemented by one processor. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a graphics processor GPU). The dedicated processor may be implemented by including a memory for implementing an embodiment of the present disclosure or by including a storage processor for using an external memory.

[0175] In addition, the AI ​​encoder 610 and the transmitter 630 may be configured by a plurality of processors. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a combination of dedicated processors or by a combination of software and a plurality of general-purpose processors such as an AP, a CPU, or a GPU. The AI ​​reducer 612 and the first encoder 614 may be implemented by different processors.

[0176] The AI ​​encoder 610 performs AI reduction on the original image 105 and first encoding on the first image 115, and transmits the AI ​​data and the image data to the transmitter 630. The transmitter 630 transmits the AI ​​data and the image data to the AI ​​decoding apparatus 200.

[0177] The image data includes data obtained as a result of performing the first encoding on the first image 115. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data that is a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding process of the first image 115. For example, the image data may include prediction mode information, motion information, quantization parameter information, etc., used to perform the first encoding on the first image 115.

[0178] The AI ​​data includes information that enables AI up-scaling to be performed on the second image 135 to an up-scaling target corresponding to the down-scaling target of the first DNN. According to an embodiment, the AI ​​data may include difference information between the original image 105 and the first image 115. In addition, the AI ​​data may include information related to the first image 115. The information related to the first image 115 may include information about at least one of a resolution of the first image 115, a bit rate of image data obtained as a result of performing the first encoding on the first image 115, or a codec type used during the first encoding of the first image 115.

[0179] According to an embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information so that the AI ​​zoom-in is performed on the second image 135 to a zoom-in target corresponding to a zoom-out target of the first DNN.

[0180] In addition, according to an embodiment, the AI ​​data may include DNN setting information that can be set in the second DNN.

[0181] The AI ​​downsizer 612 may obtain the first image 115 obtained by performing AI downsizing on the original image 105 through the first DNN. The AI ​​downsizer 612 may determine a downsizing target of the original image 105 based on a predetermined criterion.

[0182] In order to obtain the first image 115 matching the reduction target, the AI ​​reducer 612 may store a plurality of DNN setting information that can be set in the first DNN. The AI ​​reducer 612 obtains DNN setting information corresponding to the reduction target from the plurality of DNN setting information, and performs AI reduction on the original image 105 by using the first DNN set according to the obtained DNN setting information.

[0183] Each of the plurality of pieces of DNN setting information may be trained to obtain a first image 115 of a predetermined resolution and / or a predetermined quality. For example, any one of the plurality of pieces of DNN setting information may include information for obtaining a first image 115 of half the resolution of the original image 105 (e.g., a first image 115 of 2k (2048×1080) which is half the resolution of the original image 105 of 4k (4096×2160)), and another piece of DNN setting information may include information for obtaining a first image 115 of a quarter the resolution of the original image 105 (e.g., a first image 115 of 2k (2048×1080) which is a quarter the resolution of the original image 105 of 8k (8192x4320)).

[0184] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter cores of each convolutional layer, the parameters of each filter core, etc.) are stored in the form of a lookup table, the AI ​​downsizer 612 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on the reduction target, and perform AI downsizing on the original image 105 by using the obtained DNN setting information.

[0185] According to an embodiment, the AI ​​reducer 612 may determine the structure of the DNN corresponding to the reduction target, and obtain DNN setting information corresponding to the determined structure of the DNN, for example, obtain parameters of a filter kernel.

[0186] As the first DNN and the second DNN are jointly trained, a plurality of pieces of DNN setting information for performing AI reduction on the original image 105 may have an optimized value. Here, each piece of DNN setting information includes at least one of the number of convolutional layers included in the first DNN, the number of filter cores of each convolutional layer, and a parameter of each filter core.

[0187] The AI ​​downsizer 612 may set the first DNN using the DNN setting information obtained to perform AI downsizing on the original image 105, so as to obtain the first image 115 of a specific resolution and / or a specific quality through the first DNN. When the DNN setting information for performing AI downsizing on the original image 105 is obtained from a plurality of pieces of DNN setting information, each layer in the first DNN may process input data based on information included in the DNN setting information.

[0188] Hereinafter, a method of determining a downscaling target performed by the AI ​​downscaling unit 612 will be described. The downscaling target may indicate, for example, how much to reduce the resolution from the original image 105 to obtain the first image 115.

[0189] According to an embodiment, the AI ​​downsizer 612 can determine the downsizing target based on at least one of the compression ratio (e.g., the resolution difference between the original image 105 and the first image 115, the target bit rate, etc.), the compression quality (e.g., the type of bit rate), the compression history information, or the type of the original image 105.

[0190] For example, the AI ​​downsizer 612 may determine a downsizing target based on a compression ratio, compression quality, etc. that is preset or input from a user.

[0191] As another example, the AI ​​downsizer 612 may determine the downsizing target by using the compression history information stored in the AI ​​encoding device 600. For example, based on the compression history information available to the AI ​​encoding device 600, the encoding quality, compression ratio, etc. preferred by the user may be determined, and the downsizing target may be determined based on the encoding quality determined based on the compression history information. For example, the resolution, quality, etc. of the first image 115 may be determined based on the most commonly used encoding quality based on the compression history information.

[0192] As another example, the AI ​​downsizer 612 may determine the downsizing target based on encoding qualities used more frequently than a certain threshold according to compression history information (eg, an average quality of encoding qualities used more frequently than a certain threshold).

[0193] As another example, the AI ​​downsizer 612 may determine the downsizing target based on the resolution, type (eg, file format), etc. of the original image 105 .

[0194] According to an embodiment, when the original image 105 includes a plurality of frames, the AI ​​downsizer 612 may independently determine downsizing targets for a specific number of frames, or may determine downsizing targets for all frames.

[0195] According to an embodiment, the AI ​​downsizer 612 may divide the frames included in the original image 105 into a certain number of groups and independently determine a downsizing target for each group. The same or different downsizing targets may be determined for each group. According to each group, the number of frames included in the group may be the same or different.

[0196] According to another embodiment, the AI ​​downsizer 612 may independently determine a downscaling target for each frame included in the original image 105. The same or different downscaling targets may be determined for each frame.

[0197] Hereinafter, an example of the structure of the first DNN 700 on which AI reduction is based will be described.

[0198] Figure 8 is a diagram illustrating a first DNN 700 for performing AI downscaling on an original image 105.

[0199] like Figure 8 As shown, the original image 105 is input to the first convolution layer 710. The first convolution layer 710 performs a convolution process on the original image 105 by using 32 filter kernels of size 5×5. The 32 feature maps generated as a result of the convolution process are input to the first activation layer 720. The first activation layer 720 can assign nonlinear features to the 32 feature maps.

[0200] The first activation layer 720 determines whether to send sample values ​​of the feature map output from the first convolution layer 710 to the second convolution layer 730. For example, some sample values ​​of the feature map are activated by the first activation layer 720 and sent to the second convolution layer 730, and some sample values ​​are deactivated by the first activation layer 720 and are not sent to the second convolution layer 730. Information represented by the feature map output from the first convolution layer 710 is highlighted by the first activation layer 720.

[0201] The output 725 of the first activation layer 720 is input to the second convolution layer 730. The second convolution layer 730 performs a convolution process on the input data by using 32 filter kernels of a size of 5×5. The 32 feature maps output as a result of the convolution process are input to the second activation layer 740, and the second activation layer 740 can assign nonlinear features to the 32 feature maps.

[0202] The output 745 of the second activation layer 740 is input to the third convolution layer 750. The third convolution layer 750 performs a convolution process on the input data by using one filter kernel of size 5×5. As a result of the convolution process, an image may be output from the third convolution layer 750. The third convolution layer 750 generates an output by using the one filter kernel as a layer for outputting a final image. According to an embodiment of the present disclosure, the third convolution layer 750 may output the first image 115 as a result of the convolution operation.

[0203] There may be a plurality of pieces of DNN setting information indicating the number of filter cores of the first convolutional layer 710, the second convolutional layer 730, and the third convolutional layer 750 of the first DNN 700, the parameters of each filter core of the first convolutional layer 710, the second convolutional layer 730, and the third convolutional layer 750 of the first DNN 700, and the plurality of pieces of DNN setting information may be associated with the plurality of pieces of DNN setting information of the second DNN. The association between the plurality of pieces of DNN setting information of the first DNN and the plurality of pieces of DNN setting information of the second DNN may be achieved via joint training of the first DNN and the second DNN.

[0204] exist Figure 8, the first DNN 700 includes three convolutional layers (a first convolutional layer 710, a second convolutional layer 730, and a third convolutional layer 750) and two activation layers (a first activation layer 720 and a second activation layer 740), but this is only an example, and the number of convolutional layers and activation layers may vary according to the embodiment. In addition, according to the embodiment, the first DNN 700 may be implemented as an RNN. In this case, the CNN structure of the first DNN 700 according to the embodiment of the present disclosure is changed to an RNN structure.

[0205] According to an embodiment, the AI ​​reducer 612 may include at least one ALU for the convolution operation and the operation of the activation layer described above. The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier and an adder, the multiplier performs multiplication between the sample value of the original image 105 or the feature map output from the previous layer and the sample value of the filter kernel, and the adder adds the result value of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in the predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0206] Return to reference Figure 7 After receiving the first image 115 from the AI ​​downsizer 612, the first encoder 614 may reduce the amount of information of the first image 115 by performing first encoding on the first image 115. As a result of the first encoding performed by the first encoder 614, image data corresponding to the first image 115 may be obtained.

[0207] The data processor 632 processes at least one of the AI ​​data or the image data to be sent in a specific form. For example, when the AI ​​data and the image data are to be sent in the form of a bit stream, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and send the image data and the AI ​​data in the form of one bit stream through the communicator 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and send each of the bit stream corresponding to the AI ​​data and the bit stream corresponding to the image data through the communicator 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a frame or a packet, and send the image data in the form of a bit stream and send the AI ​​data in the form of a frame or a packet through the communicator 634.

[0208] The communicator 634 transmits the AI-encoded data obtained as a result of performing the AI ​​encoding through the network. The AI-encoded data obtained as a result of performing the AI ​​encoding includes image data and AI data. The image data and AI data may be transmitted through the same type of network or different types of networks.

[0209] According to an embodiment, the AI ​​encoded data obtained as a result of processing by the data processor 632 can be stored in a data storage medium, which includes a magnetic medium such as a hard disk, a floppy disk or a tape, an optical recording medium such as a CD-ROM or a DVD, or a magneto-optical medium such as a magneto-optical floppy disk.

[0210] In the following, reference will be made to Fig. 9 A method of jointly training the first DNN 700 and the second DNN 300 is described.

[0211] Fig. 9 is a diagram for describing a method of training a first DNN 700 and a second DNN 300.

[0212] In an embodiment, the original image 105 on which AI encoding is performed by the AI ​​encoding process is reconstructed into the third image 145 via the AI ​​decoding process, and in order to maintain the similarity between the original image 105 and the third image 145 obtained as a result of AI decoding, the association between the AI ​​encoding process and the AI ​​decoding process is required. In other words, the information lost in the AI ​​encoding process needs to be reconstructed during the AI ​​decoding process, and in this regard, the first DNN 700 and the second DNN 300 need to be jointly trained.

[0213] For accurate AI decoding, ultimately, it is necessary to reduce Fig. 9 The quality loss information 830 corresponding to the comparison result of the third training image 804 and the original training image 801 is shown. Therefore, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0214] First, we will describe Fig. 9 The training process shown in .

[0215] exist Fig. 9 , the original training image 801 is an image on which AI reduction is to be performed, and the first training image 802 is an image obtained by performing AI reduction on the original training image 801. In addition, the third training image 804 is an image obtained by performing AI enlargement on the first training image 802.

[0216] The original training image 801 includes a still image or a moving image including a plurality of frames. According to an embodiment, the original training image 801 may include a luminance image extracted from a still image or a moving image including a plurality of frames. In addition, according to an embodiment, the original training image 801 may include a patch image extracted from a still image or a moving image including a plurality of frames. When the original training image 801 includes a plurality of frames, the first training image 802, the second training image, and the third training image 804 also each include a plurality of frames. When the plurality of frames of the original training image 801 are sequentially input to the first DNN 700, the plurality of frames of the first training image 802, the second training image, and the third training image 804 may be sequentially obtained by the first DNN 700 and the second DNN 300.

[0217] For joint training of the first DNN 700 and the second DNN 300, an original training image 801 is input to the first DNN 700. The original training image 801 input to the first DNN 700 is output as a first training image 802 via AI downscaling, and the first training image 802 is input to the second DNN 300. As a result of performing AI upscaling on the first training image 802, a third training image 804 is output.

[0218] Reference Fig. 9 , the first training image 802 is input to the second DNN 300, and according to an embodiment, the second training image obtained when the first encoding and the first decoding are performed on the first training image 802 may be input to the second DNN 300. In order to input the second training image to the second DNN 300, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 codecs may be used. Specifically, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 codecs may be used to perform the first encoding on the first training image 802, and perform the first decoding on the image data corresponding to the first training image 802.

[0219] Reference Fig. 9 , separate from the first training image 802 output by the first DNN 700, a reduced training image 803 obtained by performing conventional reduction on the original training image 801 is obtained. Here, the conventional reduction may include at least one of bilinear scaling, bicubic scaling, lanczos scaling, or step scaling.

[0220] In order to prevent the structural features of the first image 115 from greatly deviating from the structural features of the original image 105 , a reduced training image 803 is obtained to preserve the structural features of the original training image 801 .

[0221] Before performing training, the first DNN 700 and the second DNN 300 may be set to predetermined DNN setting information. When performing training, structure loss information 810, complexity loss information 820, and quality loss information 830 may be determined.

[0222] The structural loss information 810 may be determined based on a result of comparing the reduced training image 803 and the first training image 802. For example, the structural loss information 810 may correspond to a difference between structural information of the reduced training image 803 and structural information of the first training image 802. The structural information may include various features that may be extracted from an image, such as brightness, contrast, histogram, etc. of the image. The structural loss information 810 indicates how much structural information of the original training image 801 is retained in the first training image 802. When the structural loss information 810 is small, the structural information of the first training image 802 is similar to the structural information of the original training image 801.

[0223] The complexity loss information 820 may be determined based on the spatial complexity of the first training image 802. For example, the total variance value of the first training image 802 may be used as the spatial complexity. The complexity loss information 820 is related to the bit rate of the image data obtained by performing the first encoding on the first training image 802. It is defined that when the complexity loss information 820 is small, the bit rate of the image data is low.

[0224] The quality loss information 830 may be determined based on a result of comparing the original training image 801 and the third training image 804. The quality loss information 830 may include at least one of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, or a video multi-method assessment fusion (VMAF) value about the difference between the original training image 801 and the third training image 804. The quality loss information 830 indicates how similar the third training image 804 is to the original training image 801. When the quality loss information 830 is small, the third training image 804 is more similar to the original training image 801.

[0225] Reference Fig. 9 , the first DNN 700 is trained using the structure loss information 810, the complexity loss information 820, and the quality loss information 830, and the second DNN 300 is trained using the quality loss information 830. In other words, both the first DNN 700 and the second DNN 300 are trained using the quality loss information 830.

[0226] The first DNN 700 may update parameters so that the final loss information determined based on the structure loss information 810, the complexity loss information 820, and the quality loss information 830 is reduced or minimized. In addition, the second DNN 300 may update parameters so that the quality loss information 830 is reduced or minimized.

[0227] Final loss information for training the first DNN 700 and the second DNN 300 may be determined as Equation 1 below.

[0228] [Equation 1]

[0229] LossDS = a×structural loss information + b×complexity loss information + c×quality loss information

[0230] LossUS=d×quality loss information

[0231] In Equation 1, LossDS indicates final loss information to be reduced or minimized for training the first DNN 700, and LossUS indicates final loss information to be reduced or minimized for training the second DNN 300. In addition, a, b, c, and d may be predetermined specific weights.

[0232] In other words, the first DNN 700 updates parameters in a direction in which the LossDS in Equation 1 is reduced, and the second DNN 300 updates parameters in a direction in which the LossUS in Equation 1 is reduced. When the parameters of the first DNN 700 are updated according to the LossDS derived during training, the first training image 802 obtained based on the updated parameters becomes different from the previous first training image 802 obtained based on the non-updated parameters, and thus, the third training image 804 also becomes different from the previous third training image 804. When the third training image 804 becomes different from the previous third training image 804, the quality loss information 830 is also re-determined, and the second DNN 300 updates the parameters accordingly. When the quality loss information 830 is re-determined, the LossDS is also re-determined, and the first DNN 700 updates the parameters according to the re-determined LossDS. In other words, the update of the parameters of the first DNN 700 leads to the update of the parameters of the second DNN 300, and the update of the parameters of the second DNN 300 leads to the update of the parameters of the first DNN 700. In other words, because the first DNN 700 and the second DNN 300 are jointly trained by sharing the quality loss information 830, the parameters of the first DNN 700 and the parameters of the second DNN 300 can be jointly optimized.

[0233] Referring to Equation 1, it is verified that LossUS is determined according to the quality loss information 830 , but this is only an example, and LossUS may also be determined based on at least one of the structure loss information 810 and the complexity loss information 820 and the quality loss information 830 .

[0234] In the above, it has been described that the AI ​​amplifier 234 of the AI ​​decoding device 200 and the AI ​​reducer 612 of the AI ​​encoding device 600 store multiple DNN setting information, and now a method of training each of the multiple DNN setting information stored in the AI ​​amplifier 234 and the AI ​​reducer 612 will be described.

[0235] As described in reference equation 1, the first DNN 700 updates parameters by considering the similarity between the structural information of the first training image 802 and the structural information of the original training image 801 (structural loss information 810), the bit rate of the image data obtained as a result of performing the first encoding on the first training image 802 (complexity loss information 820), and the difference between the third training image 804 and the original training image 801 (quality loss information 830).

[0236] Specifically, the parameters of the first DNN 700 can be updated so that when the first encoding is performed on the first training image 802, the first training image 802 having structural information similar to the original training image 801 is obtained, and image data with a small bit rate is obtained, and at the same time, the second DNN 300 that performs AI amplification on the first training image 802 obtains a third training image 804 similar to the original training image 801.

[0237] The direction of optimizing the parameters of the first DNN 700 may be changed by adjusting the weights a, b, and c in Equation 1. For example, when the weight b is determined to be large, the parameters of the first DNN 700 may be updated by prioritizing a low bit rate over the high quality of the third training image 804. In addition, when the weight c is determined to be large, the parameters of the first DNN 700 may be updated by prioritizing the high quality of the third training image 804 over a low bit rate or maintaining the structural information of the original training image 801.

[0238] In addition, the direction of optimizing the parameters of the first DNN 700 may vary according to the type of codec used to perform the first encoding on the first training image 802. This is because the second training image to be input to the second DNN 300 may vary according to the type of codec.

[0239] In other words, the parameters of the first DNN 700 and the parameters of the second DNN 300 may be jointly updated based on the weights a, b, and c and the type of codec used to perform the first encoding on the first training image 802. Therefore, when the first DNN 700 and the second DNN 300 are trained after the weights a, b, and c are each determined to be a specific value and the type of codec is determined to be a specific type, the parameters of the first DNN 700 and the parameters of the second DNN 300 that are associated with and optimized to each other may be determined.

[0240] In addition, when the first DNN 700 and the second DNN 300 are trained after changing the weights a, b, and c and the type of codec, the parameters of the first DNN 700 and the parameters of the second DNN 300 that are associated and optimized with each other may be determined. In other words, when the first DNN 700 and the second DNN 300 are trained while changing the values ​​of the weights a, b, and c and the type of codec, a plurality of pieces of DNN setting information that are jointly trained with each other may be determined in the first DNN 700 and the second DNN 300.

[0241] As mentioned above Figure 5 As described, a plurality of pieces of DNN setting information of the first DNN 700 and the second DNN 300 may be mapped to information related to the first image. In order to set such a mapping relationship, the first training image 802 output from the first DNN 700 via a specific codec may be first encoded according to a specific bit rate, and the second training image obtained by performing a first decoding on the bit stream obtained as a result of performing the first encoding may be input to the second DNN 300. In other words, by training the first DNN 700 and the second DNN 300 after setting the environment to perform the first encoding on the first training image 802 of a specific resolution via a specific codec according to a specific bit rate, a pair of DNN setting information mapped to the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained as a result of performing the first encoding on the first training image 802 may be determined. By differently changing the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained according to the first encoding of the first training image 802, the mapping relationship between multiple DNN setting information of the first DNN 700 and the second DNN 300 and the multiple information related to the first image can be determined.

[0242] Fig.10 is a diagram for describing the training process of the training device 1000 on the first DNN 700 and the second DNN.

[0243] Reference Fig. 9 The described training of the first DNN 700 and the second DNN 300 may be performed by the training device 1000. The training device 1000 includes the first DNN 700 and the second DNN 300. The training device 1000 may be, for example, the AI ​​encoding device 600 or a separate server. The DNN setting information of the second DNN 300 obtained as a result of the training is stored in the AI ​​decoding device 200.

[0244] Reference Fig.10 , in operations S840 and S845, the training device 1000 initially sets the DNN setting information of the first DNN 700 and the second DNN 300. Therefore, the first DNN 700 and the second DNN 300 may operate according to the predetermined DNN setting information. The DNN setting information may include information about at least one of the number of convolutional layers included in the first DNN 700 and the second DNN 300, the number of filter kernels of each convolutional layer, the size of the filter kernel of each convolutional layer, or the parameters of each filter kernel.

[0245] In operation S850, the training apparatus 1000 inputs the original training image 801 into the first DNN 700. The original training image 801 may include a still image or at least one frame included in a moving image.

[0246] In operation S855, the first DNN 700 processes the original training image 801 according to the initially set DNN setting information, and outputs the first training image 802 obtained by performing AI reduction on the original training image 801. Fig.10 , the first training image 802 output from the first DNN 700 is directly input to the second DNN 300, but the first training image 802 output from the first DNN 700 may be input to the second DNN 300 by the training device 1000. In addition, the training device 1000 may perform first encoding and first decoding on the first training image 802 via a specific codec, and then input the second training image to the second DNN 300.

[0247] In operation S860, the second DNN 300 processes the first training image 802 or the second training image according to initially set DNN setting information, and outputs a third training image 804 obtained by performing AI up-scaling on the first training image 802 or the second training image.

[0248] In operation S865 , the training apparatus 1000 calculates complexity loss information 820 based on the first training image 802 .

[0249] In operation S870 , the training apparatus 1000 calculates structure loss information 810 by comparing the reduced training image 803 and the first training image 802 .

[0250] In operation S875 , the training apparatus 1000 calculates quality loss information 830 by comparing the original training image 801 and the third training image 804 .

[0251] In operation S880, the initially set DNN setting information is updated via a back propagation process based on the final loss information. The training apparatus 1000 may calculate the final loss information for training the first DNN 700 based on the complexity loss information 820, the structure loss information 810, and the quality loss information 830.

[0252] In operation S885, the second DNN 300 updates the initially set DNN setting information through a back propagation process based on the quality loss information 830 or the final loss information. The training apparatus 1000 may calculate the final loss information for training the second DNN 300 based on the quality loss information 830.

[0253] Then, the training device 1000, the first DNN 700, and the second DNN 300 may repeat operations S850 to S885 until the final loss information is minimized to update the DNN setting information. At this time, during each repetition, the first DNN 700 and the second DNN 300 operate according to the DNN setting information updated in the previous operation.

[0254] Table 1 below shows the effects when AI encoding and AI decoding are performed on the original image 105 according to an embodiment of the present disclosure and when encoding and decoding are performed on the original image 105 via HEVC.

[0255] [Table 1]

[0256]

[0257] As shown in Table 1, according to an embodiment of the present disclosure, in addition to the subjective image quality when AI encoding and AI decoding are performed on content including 300 frames of 8K resolution being higher than the subjective image quality when encoding and decoding are performed via HEVC, the bit rate is reduced by at least 50%.

[0258] Fig.11 1 is a diagram of an apparatus 20 for performing AI downscaling on an original image 105 and an apparatus 40 for performing AI upscaling on a second image 135 .

[0259] Device 20 receives original image 105 and provides image data 25 and AI data 30 to device 40 by using AI downsizer 1124 and transform-based encoder 1126. According to an embodiment, image data 25 corresponds to Figure 1 The AI ​​data 30 corresponds to the image data Figure 1 In addition, according to an embodiment, the transform-based encoder 1126 corresponds to Figure 7 The first encoder 614 and the AI ​​reducer 1124 correspond to Figure 7 AI reducer 612.

[0260] The device 40 receives the AI ​​data 30 and the image data 25, and obtains the third image 145 by using the transform-based decoder 1146 and the AI ​​amplifier 1144. According to an embodiment, the transform-based decoder 1146 corresponds to Figure 2 The first decoder 232, and the AI ​​amplifier 1144 corresponds to Figure 2 AI amplifier 234.

[0261] According to an embodiment, the device 20 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the device 20 executes the reference according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, referring to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0262] According to an embodiment, the device 40 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the device 40 executes the reference according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, referring to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0263] exist Fig.11 , the configuration controller 1122 receives at least one input value 10. According to an embodiment, the at least one input value 10 may include at least one of a target resolution difference for the AI ​​downsizer 1124 and the AI ​​upsizer 1144, a bit rate of the image data 25, a bit rate type of the image data 25 (e.g., a variable bit rate type, a constant bit rate type, or an average bit rate type), or a codec type of the transform-based encoder 1126. The at least one input value 10 may include a value pre-stored in the device 20 or a value input from a user.

[0264] The configuration controller 1122 controls the operations of the AI ​​downsizer 1124 and the transform-based encoder 1126 based on the received input value 10. According to an embodiment, the configuration controller 1122 obtains DNN setting information for the AI ​​downsizer 1124 according to the received input value 10, and sets the AI ​​downsizer 1124 using the obtained DNN setting information. According to an embodiment, the configuration controller 1122 may send the received input value 10 to the AI ​​downsizer 1124, and the AI ​​downsizer 1124 may obtain DNN setting information for performing AI downsizing on the original image 105 based on the received input value 10. According to an embodiment, the configuration controller 1122 may provide additional information to the AI ​​downsizer 1124 together with the input value 10, for example, color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI downsizing is applied and tone mapping information of a high dynamic range (HDR), and the AI ​​downsizer 1124 may obtain the DNN setting information in consideration of the input value 10 and the additional information. According to an embodiment, the configuration controller 1122 sends at least a portion of the received input value 10 to the transform-based encoder 1126, and the transform-based encoder 1126 performs a first encoding on the first image 115 by a bit rate of a specific value, a specific type of bit rate, and a specific codec.

[0265] The AI ​​reducer 1124 receives the original image 105 and performs a reference Figure 1 , Figure 7 , Figure 8 , Fig. 9 or Fig.10 At least one of the operations described in is performed to obtain a first image 115.

[0266] According to an embodiment, the AI ​​data 30 is provided to the device 40. The AI ​​data 30 may include at least one of resolution difference information between the original image 105 and the first image 115 or information related to the first image 115. The resolution difference information may be determined based on a target resolution difference in the input value 10, and the information related to the first image 115 may be determined based on at least one of a target bit rate, a bit rate type, or a codec type. According to an embodiment, the AI ​​data 30 may include parameters used during AI upscaling. The AI ​​data 30 may be provided to the device 40 from the AI ​​downsizer 1124.

[0267] When the transform-based encoder 1126 processes the first image 115, image data 25 is obtained, and the image data 25 is transmitted to the device 40. The transform-based encoder 1126 may process the first image 115 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or VA1. The configuration controller 1142 controls the operation of the AI ​​amplifier 1144 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 obtains DNN setting information for the AI ​​amplifier 1144 according to the received AI data 30, and sets the AI ​​amplifier 1144 with the obtained DNN setting information. According to an embodiment, the configuration controller 1142 may transmit the received AI data 30 to the AI ​​amplifier 1144, and the AI ​​amplifier 1144 may obtain DNN setting information for performing AI amplification on the second image 135 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 may provide additional information to the AI ​​amplifier 1144 together with the AI ​​data 30, for example, color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI amplification is applied and tone mapping information of HDR, and the AI ​​amplifier 1144 may obtain DNN setting information in consideration of the AI ​​data 30 and the additional information. According to an embodiment, the AI ​​amplifier 1144 may receive the AI ​​data 30 from the configuration controller 1142, receive at least one of the prediction mode information, motion information, or quantization parameter information from the transform-based decoder 1146, and obtain the DNN setting information based on at least one of the prediction mode information, motion information, and quantization parameter information and the AI ​​data 30.

[0268] The transform-based decoder 1146 may process the image data 25 to reconstruct the second image 135. The transform-based decoder 1146 may process the image data 25 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1.

[0269] The AI ​​amplifier 1144 may obtain the third image 145 by performing AI amplification on the second image 135 provided from the transform-based decoder 1146 based on the set DNN setting information.

[0270] The AI ​​reducer 1124 may include a first DNN, and the AI ​​amplifier 1144 may include a second DNN, and according to an embodiment, according to the reference Fig. 9 and Fig.10 The described training method trains the first DNN and the DNN setting information of the second DNN.

[0271] Fig.12 1 is a block diagram showing a configuration of an AI encoding device 1200 according to another embodiment of the present disclosure. Fig.12 According to another embodiment of the present disclosure, the AI ​​encoding device 1200 may include an AI encoder 1210 and a transmitter 1230. The AI ​​encoder 1210 may include an AI downsizer 1212, an artifact remover 1216, and a first encoder 1214.

[0272] As described above, the AI ​​downscaling unit 1212 may obtain the first image 115 AI-downscaled from the original image 105 by the first DNN. The first image 115 may have a resolution lower than that of the original image 105. The AI ​​downscaling performed by the AI ​​downscaling unit 1212 has been described above, and thus, a detailed description thereof will be omitted.

[0273] Fig.13 is a diagram for describing a method performed by the AI ​​encoding apparatus 1200 for determining an artifact map based on values ​​of frequency transform coefficients of an original image and post-processing the first image 115 based on the artifact map.

[0274] In operation 1310, the AI ​​encoding device 1200 may perform frequency transformation on each block of the original image 105 to generate a frequency transformation image 1315, and in operation 1320, determine an artifact region of each block based on the distribution of frequency transformation coefficients of the frequency transformation image 1315. The AI ​​encoding device 1200 may generate an artifact map 1325 based on the artifact region. At this time, abnormal values ​​included in the artifact map 1325 may be removed. Figures 17a to 19b A method of removing outliers included in the artifact map 1325 is described.

[0275] In operation 1310, the frequency transform may be one of fast Fourier transform (FFT), discrete cosine transform (DCT), and wavelet transform, but is not limited thereto.

[0276] In operation 1330, the AI ​​encoding apparatus 1200 may store the artifact map 1325 in a buffer. In operation 1340, the AI ​​encoding apparatus 1200 may compare the artifact map 1325 of the current frame with the artifact map 1335 of the previous frame to obtain an artifact map for post-processing 1345. However, the AI ​​encoding apparatus 1200 may determine the artifact map 1325 of the current frame as the artifact map for post-processing without comparing the artifact map 1325 with the artifact map 1335.

[0277] Here, the artifact map may be a map that represents an artifact area including aliasing artifacts in units of blocks. Aliasing artifacts are noises generated when the sampling frequency of an image is insufficiently greater than twice the maximum frequency of a signal in the image or when adjacent signal spectra overlap each other due to improper filtering of the image.

[0278] When the AI ​​downscaling unit 1212 performs AI downscaling on the original image 105 by using the first DNN, the sampling frequency may be reduced compared to the sampling frequency of the original image 105 , and when the sampling frequency becomes insufficiently greater than the maximum frequency of the signal in the original image 105 , aliasing artifacts may be generated.

[0279] In operation 1350, the AI ​​encoding apparatus 1200 may perform post-processing on the first image 115 based on the artifact map 1345 to generate a post-processed first image 1355. Here, the post-processing is a process of changing pixel values ​​in an image, and may be a process using various filters.

[0280] For example, the AI ​​encoding apparatus 1200 may filter an area of ​​the first image 115 corresponding to the artifact area. At this time, one of a low-pass filter or a Gaussian filter may be used. The post-processed first image 1355 may be input to the first encoder 1214.

[0281] Alternatively, the AI ​​encoding device 1200 may generate a reduced region from which high-frequency components are removed by performing low-pass filtering and using a typical filter including a reduction function (such as a bicubic filter and a bilinear filter), or by using a typical filter including a low-pass filter function and a reduction function (such as a Lanczos filter) to filter an area of ​​the original image 105 corresponding to the artifact area of ​​the artifact map 1345. The reduced region may be reduced by the same degree of scaling as the first image 115 reduced by the first DNN AI.

[0282] The AI ​​encoding apparatus 1200 may replace the pixel values ​​of the first image 115 corresponding to the artifact area with the pixel values ​​of the reduced area, but is not limited thereto.

[0283] However, the AI ​​encoding device 1200 may generate a reduced image by filtering the area of ​​the original image 105 corresponding to the artifact area of ​​the artifact map 1345 using a typical low-pass filter and reducing the filtered area using a typical reduction method. The AI ​​encoding device 1200 may replace the pixel values ​​of the first image 115 corresponding to the artifact area with the pixel values ​​of the reduced area.

[0284] The AI ​​encoding device 1200 may perform the first encoding by using the post-processed first image that is post-processed with respect to the first image 115 to reduce aliasing artifacts that may be recognized by the human eye, thereby improving the quality of the encoded image.

[0285] Refer again Fig.12, the AI ​​encoding device 1200 according to another embodiment of the present disclosure may include a central processor (not shown) for controlling the AI ​​encoder 1210 and the transmitter 1230. Alternatively, the AI ​​encoder 1210 and the transmitter 1230 may be operated by their own processors (not shown), and the processors (not shown) may operate in an organized manner with each other in order to operate the AI ​​encoding device 1200. Alternatively, the AI ​​encoder 1210 and the transmitter 1230 may be controlled by the control of an external processor (not shown).

[0286] The AI ​​encoding apparatus 1200 may include one or more data storage devices (not shown) for storing input / output data of the downsizer 1212, the artifact remover 1216, the first encoder 1214, and the transmitter 1230. The AI ​​encoding apparatus 1200 may include a memory controller (not shown) for controlling data output to or input from the data storage device (not shown).

[0287] The AI ​​encoding device 1200 can perform image encoding including prediction by interacting with an internal video encoding processor or an external video encoding processor to encode an image. The internal video encoding processor of the AI ​​encoding device 1200 according to an embodiment of the present disclosure may be set as a separate processor, or the central processing unit or the graphics processing unit may include an image encoding processing module for implementing basic image encoding operations.

[0288] Fig.12 The operations of the AI ​​scaler 1212, the first encoder 1214, and the transmitter 1230 of the AI ​​encoding device 1200 shown in FIG. 1 may be similar to those described above with reference to FIG. Figure 7 The operations of the AI ​​sealer 612, first encoder 614, and transmitter 630 are described to be the same.

[0289] The first image 115 may be provided to an artifact remover 1216. The artifact remover 1216 may detect artifacts in the first image 115 and remove the detected artifacts.

[0290] The artifact remover 1216 may perform post-processing for removing artifacts in the first image 115. The first image 115 reduced by the first DNN AI may include artifacts (e.g., aliasing artifacts). That is, because the AI ​​reduction reduces the resolution of the original image 105, the first image 115 may include shapes that do not exist in the original image 105 due to the change in resolution.

[0291] When encoding and decoding are performed based on the first image 115 including aliasing artifacts that do not exist in the original image 105, the image quality may be reduced. Therefore, the artifact remover 1216 may perform post-processing for removing or reducing artifacts of the first image 115. More specifically, the artifact remover 1216 may perform post-processing for removing or reducing aliasing artifacts that can be detected by the human eye from the first image 115, thereby minimizing the aliasing artifacts that can be detected by the human eye. Therefore, the quality of the encoded image can be improved.

[0292] An image in which an aliasing artifact detectable by human eyes is generated may be an image in which a pattern appears continuously, and when an aliasing artifact is generated in such an image, an image with a pattern similar to a moiré pattern may be generated.

[0293] That is, when aliasing artifacts are generated in continuous patterns having a small size, adjacent patterns overlap so that sample values ​​corresponding to a specific area of ​​an image cannot be smoothly arranged, resulting in a distorted pattern direction or an irregular pattern.

[0294] Hereinafter, various examples will be described in which sample values ​​cannot be smoothly arranged due to aliasing artifacts generated in an image, thereby causing a distorted pattern direction or an irregular pattern.

[0295] Fig.15a is an example diagram illustrating spatial aliasing artifacts occurring in a first image reduced from an original image.

[0296] For example, Fig.15a As shown, unlike the smoothly arranged pattern of another area 1505 of the original image 105, the area 1500 of the first image 115 output from the first DNN may include an aliasing artifact area 1510 having a characteristic that the pattern is distorted in different directions. That is, the high-frequency components included in the original image 105 may be distorted during reduction and represented as different frequency components. For this reason, spatial aliasing artifacts may be generated. The artifact remover 1216 may determine the area where the spatial aliasing artifact is generated based on the value of the frequency transform coefficient of the original image 105, and perform post-processing for removing or reducing the spatial aliasing artifact from the area in the first image 115, thereby removing or reducing the spatial aliasing artifact.

[0297] Fig.15b is an example diagram showing spatiotemporal aliasing artifacts occurring in a first image reduced from an original image.

[0298] like Fig.15bAs shown, unlike another region 1525 of the original image 105, some patterns cannot be arranged smoothly and the directions are distorted due to distortion, so even in a region where there is little motion so that distortion occurs, the continuous image has a pixel value that is temporally different from the pixel value of the previous image. Therefore, an aliasing artifact region 1530 may exist in region 1520 of the first image 115 output from the first DNN.

[0299] In this case, the aliasing artifacts included in the aliasing artifact area 1530 may be spatiotemporal aliasing artifacts. Fig.15c Briefly describe the causes of spatiotemporal aliasing artifacts.

[0300] Fig.15c is a diagram for describing a spatiotemporal aliasing artifact recognized by human eyes among spatiotemporal aliasing artifacts appearing in a first image reduced from an original image.

[0301] Reference Fig.15c , when the original image 105 is reduced, the resolution of the original image 105 may be reduced, and the original image 105 may have spatially different pixel values ​​(generating spatial aliasing artifacts). Because the same value exists continuously in the spatial region 1535 of the temporally continuous original image 105, no flicker occurs in the original image 105. However, in the spatial region 1540 of the temporally continuous first image 115, a spatial aliasing artifact may be generated due to the reduction to change the pixel value so that different values ​​are temporally continuous. As a result, the non-flickering area may flicker, so that the human eye can recognize such flicker.

[0302] The spatiotemporal aliasing artifact may be an aliasing artifact generated when a small amount of motion is generated in the temporally continuous original images 105. Therefore, by detecting an area in the original image 105 that is approximately still for a long time without any large motion, the spatiotemporal aliasing artifact may be removed or reduced by performing post-processing for removing or reducing the spatial aliasing artifact from an area of ​​the first image 115 corresponding to the detected area of ​​the original image 105.

[0303] Fig.15d is a diagram for describing spatial aliasing artifacts recognized by human eyes among spatial aliasing artifacts appearing in a first image reduced from an original image.

[0304] Reference Fig.15d, the first image 115 output from the first DNN may include a spatial aliasing artifact region in which the direction of the pattern included in the original image 105 is distorted into a different pattern direction or an irregular pattern appears, and the spatial aliasing artifact can be easily detected by the human eye. When the first image 115 includes an artifact that can be easily detected by the human eye, when the original image 105 is transformed into the frequency domain, the high-frequency transform coefficient components in the frequency domain block 1545 are concentrated at the transform coefficient components 1550 of several high-frequency components, and are not dispersed in the high-frequency region. In addition to the DC component, there may be few frequency transform coefficient components in the frequency domain block 1545. Therefore, the region including the aliasing artifact that can be easily recognized by the human eye can be determined based on the distribution of the frequency transform coefficients. Generally, the frequency transform coefficients of the frequency domain block 1545 can be represented algebraically using a variable index (such as Xvr). The virtual variable "v" can set an index in the horizontal direction of the frequency, and the virtual variable "r" can set an index in the vertical direction of the frequency. In general, a pair (v, r) may be referred to herein as a location, a place, an index pair, or a location index. For example, as a non-limiting example, in Fig.15d In FIG. 1 , the frequency transform coefficients in the frame labeled 1550 may be located at positions (v=0, r=30), (v=0, r=31), and (v=0, r=32) of the N×N transform block size of 64×64. Fig.15d In the non-limiting example of item 1550, the corresponding frequency transform coefficients are X(0,30), X(0,31), and X(0,32). The energy of each coefficient can be written as the square of the magnitude (such as |X(0,30)| 2 、|X(0,31)| 2 , and |X(0,32)| 2 ). That is, by determining whether most of the energy is concentrated at several frequency components in the high-frequency region, the region including the artifact can be determined.

[0305] According to an embodiment of the present disclosure, the artifact remover 1216 may generate artifact information representing an area including artifacts in the first image 115. The artifact information may represent the location of the area including artifacts in the first image 115. The artifact information may include a two-dimensional (2D) artifact map having a horizontal size and a vertical size. The resolution of the artifact map may be the same as the resolution of the first image 115. Alternatively, the resolution of the artifact map may be the same as the resolution of the original image 105. The artifact information may be one-dimensional (1D) data representing the location of the area including artifacts in the first image 115.

[0306] The artifact remover 1216 may generate artifact information representing a region including an artifact in the first image 115 based on the value of the frequency transform coefficient of the original image 105 .

[0307] Pixels in the artifact map may have the first pixel value or the second pixel value. Pixels or blocks in the first image 115 corresponding to the pixels or blocks having the first pixel value may be determined to include artifacts.

[0308] In addition, the pixel or block corresponding to the pixel or block having the second pixel value in the first image 115 can be determined as not including artifacts. For example, the first pixel value can be an integer value of 1 or greater, and the second pixel value can be 0. Therefore, in some embodiments, the first pixel value is an indicator. The value 0 indicates that there are no artifacts. A value greater than 0 indicates an artifact. The magnitude of the first pixel value greater than 0 indicates the degree of the artifact. However, the first pixel value and the second pixel value are not limited to the above values ​​and can be set to various other specific values.

[0309] More specifically, depending on the degree of aliasing artifacts, the first pixel value or the second pixel value may have different values. When the degree of aliasing artifacts is 0 to 100, the first pixel value may be an integer value between 1 and 100, and the second pixel value may be 0. The degree of aliasing artifacts may be determined based on the MHN / HF ratio or the position of the maximum frequency.

[0310] According to an embodiment of the present disclosure, the artifact remover 1216 may perform frequency transform on each block of the original image 105 to generate a frequency transform coefficient of each block to generate an artifact map. The artifact remover 1216 may determine whether a preset standard is met based on the distribution of the frequency transform coefficient of each block of the original image 105, and generate an artifact map with a preset value for each block according to the result of the determination.

[0311] Here, the preset criterion may be based on a first comparison result obtained by comparing the sum of the transform coefficients of the low frequency region of each block of the original image 105 with the sum of the transform coefficients of the high frequency region of the original image 105. In addition, the preset criterion may be based on a second comparison result obtained by comparing the maximum value of the transform coefficients of the high frequency region of each block and the first sum of the values ​​of the transform coefficients located around the first transform coefficient having the maximum value with the second sum of the values ​​of the transform coefficients of the low frequency region and the second sum of the values ​​of the transform coefficients located around the second transform coefficient having the maximum value. In addition, the preset criterion may be based on a third comparison result obtained by comparing the sum of the transform coefficients of the high frequency region with the first sum of the first transform coefficients in the high frequency region. In addition, the preset criterion may be based on a fourth comparison result obtained by comparing a preset value with the number of transform coefficients having a value greater than the preset value. The preset criterion may be based on at least one of the first comparison result, the second comparison result, the third comparison result, or the fourth comparison result.

[0312] For example, the artifact remover 1216 can generate an artifact map with a preset value for each block based on the first condition described below. Typically, the conditions described below refer to the sum of energy over a domain. The domain is indicated in the formula with a symbolic label (such as HF, LF, MHN, or LHN) and can correspond to a series of indexes (also referred to as position indexes). The independent variable of the sum can be a signed frequency transform coefficient, an absolute value frequency transform coefficient, or energy (such as the magnitude square of the frequency transform coefficient). In some cases, the independent variable is not energy, but simply a count of events. These cases are represented in the formula with the symbol "number".

[0313] [First condition]

[0314] If ((∑HF>α∑LF) and (∑HF>thd1) and (∑LF>thd2))

[0315] In some embodiments, the first condition uses a value of 0 for thd1 and thd2, so that the first condition is based only on the high frequency region (e.g., Fig.14 The sum of the HF 1410 excluding "DC" and excluding the low frequency region LF 1405 and the low frequency region (eg, Fig.14 In some embodiments, the independent variable of the sum refers to the energy of the frequency transform coefficient (such as |X(v,r)|2).

[0316] Fig.14 is a diagram for describing a method of determining an artifact area based on distribution of transform coefficients in a block included in an original image.

[0317] Reference Fig.14 , HF represents the high frequency region 1410, LF represents the low frequency region 1405, thd1 and thd2 represent specific thresholds, and α represents a preset coefficient. For example, thd1 may be 64, thd2 may be 1024, and α may be 1.2, but are not limited thereto. The high frequency region 1410 may represent a region in which at least one of the horizontal frequency or the vertical frequency in the block is greater than or equal to a preset value, and the low frequency region 1405 may represent a region in which the horizontal frequency and the vertical frequency in the block are less than or equal to a preset value, but are not limited thereto. The low frequency region 1405 may not include a coefficient of a DC frequency.

[0318] When the first condition is satisfied, the artifact remover 1216 may determine an artifact map for each block having a first pixel value indicating the presence of an artifact.

[0319] For example, the artifact remover 1216 may generate an artifact map having a preset value for each block based on a second condition described below.

[0320] [Second condition]

[0321] If ((∑MHN>β∑MLN) and (∑MHN>thd3))

[0322] Here, MHN 1415 represents a region including a frequency transform coefficient 1420 having a maximum value in the high-frequency region 1410 and a transform coefficient located around the frequency transform coefficient 1420, and MLN 1430 represents a region including a frequency transform coefficient 1435 having a maximum value in the low-frequency region 1405 and a transform coefficient located around the frequency transform coefficient 1435. The expression "located around" indicates the identification of a cluster. There are many methods known in the field of signal processing to identify clusters. For example, a cluster can be identified as being within a preset distance of a maximum value (amount or energy maximum). The preset distance can be an integer L. For example, if the maximum value |X| occurs at r=R and v=V, the frequency transform coefficients around the maximum value are located within a square defined by (v=V+ / -L, r=R+ / -L).

[0323] For example, in Fig.14 The maximum value in the LF region occurs at v = 2, r = 4 (see Fig.14 1435 in the example, the coefficient at position index (2,4). Note that the DC coefficient is at v=0, r=0, and the zero-based counting occurs from top to bottom for r and from left to right for v. For the example of L=2, a square of size 4×4 is defined around item 1435 (see Fig.14 The frequency transform coefficients around the maximum value are not limited to positions within a square. The frequency transform coefficients around the maximum value may be located within a rectangle or a circle or various geometric shapes, etc.

[0324] In addition, thd3 represents a specific threshold value, and β represents a preset coefficient. For example, thd3 may be 1024 and β may be 1.2, but is not limited thereto. In some embodiments, thd3 is zero, and the second condition depends only on satisfying the comparison between MHN and MLN.

[0325] When the second condition is satisfied, the artifact remover 1216 may determine an artifact map for each block having a first pixel value indicating the presence of an artifact.

[0326] For example, the artifact remover 1216 may generate an artifact map having a preset value for each block based on a third condition described below.

[0327] [Third Condition]

[0328] If (∑MHN>γ∑HF)

[0329] Here, HF represents the high frequency region 1410, and MHN 1415 represents a region including the frequency transform coefficient 1420 having the maximum value in the high frequency region 1410 and transform coefficients located around the frequency transform coefficient 1420. γ may be a preset coefficient less than 1. For example, γ may be 0.6, but is not limited thereto.

[0330] When the third condition is satisfied, the artifact remover 1216 may determine an artifact map for each block having a first pixel value indicating the presence of an artifact.

[0331] For example, the artifact remover 1216 may generate an artifact map having a preset value for each block based on a fourth condition described below.

[0332] [Fourth condition]

[0333] If ((number∑(HF(i)>thd4)+number∑(LF(i)>thd4))<thd5)

[0334] Here, number represents the number of frequency components that meet a specific condition, HF(i) represents the i-th frequency conversion coefficient in the high frequency region 1410, and LF(i) represents the i-th frequency conversion coefficient in the low frequency region 1405. thd4 and thd5 represent specific thresholds. For example, thd4 may be 10, and thd5 may be a value greater than or equal to 48, but are not limited thereto.

[0335] In some embodiments, the fourth condition is satisfied only based on the number of those frequency transform coefficients in the high frequency region 1410 exceeding thd4; that is, in some embodiments, the fourth condition does not require the number of frequency transform coefficients in the low frequency region to be greater than thd4.

[0336] In the above first condition, second condition, third condition and fourth condition, the sum may be an energy sum.

[0337] When the fourth condition is satisfied, the artifact remover 1216 may determine an artifact map for each block having a first pixel value indicating the presence of an artifact.

[0338] When at least one of the first to fourth conditions is met, the artifact remover 1216 may determine that each block has an artifact map of a first pixel value indicating the presence of an artifact. For example, when all of the first to fourth conditions are met, the artifact remover 1216 may determine that each block has an artifact map of a first pixel value indicating the presence of an artifact.

[0339] Fig.16 1 is a diagram for illustrating the original image 105 partitioned in units of blocks 106 and the first image 115 partitioned in units of blocks 116 .

[0340] Reference Fig.16 , the original image 105 and the first image 115 may be partitioned into the same number of blocks 106 and blocks 116. The blocks 106 and blocks 116 may have a preset size. However, because the resolution of the original image 105 is greater than the resolution of the first image 115, the size of the block 106 partitioning the original image 105 may be greater than the size of the block 116 partitioning the first image 115.

[0341] According to an embodiment of the present disclosure, the original image 105 may be partitioned in units of blocks 106 having the same size or in units of blocks 106 having different sizes. According to an embodiment of the present disclosure, the first image 115 may be partitioned in units of blocks 116 having the same size or in units of blocks 116 having different sizes.

[0342] A frequency transform may be performed on each block 106 of the original image 105 , and an artifact map may be generated based on the frequency transformed blocks 106 .

[0343] In the following, reference will be made to Figures 17a to 19b A process for removing outliers of an artifact map performed by the AI ​​encoding device 1200 is described.

[0344] Fig.17a and Fig.17b is a diagram for illustrating an artifact map before and after outlier removal.

[0345] Reference Fig.17a and Fig.17b The artifact remover 1216 may perform processing (such as morphological processing) on ​​the artifact map 1700 generated based on the distribution of the frequency transform coefficients of each block of the original image 105 to generate an artifact map 1710 from which abnormal values ​​have been removed. Figures 18a to 19b This process is described in detail.

[0346] A region of interest (ROI) may be set, and in a remaining region except the ROI, the first pixel value may be changed to a second pixel value. The ROI may be a region having a preset size ranging from the center of the artifact map.

[0347] Fig.18a and Fig.18b is a diagram for describing a process of removing an abnormal value by the AI ​​encoding apparatus 1200 according to an embodiment of the present disclosure.

[0348] An artifact map 1800 generated based on the distribution of frequency transform coefficients of each block of the original image 105 is shown. Fig.18a In , a1, b1, etc. are indexes used to distinguish blocks from each other.

[0349] Reference Fig.18a , block a1, block b1, etc. may have a first pixel value, and block c1, block d1, etc. may have a second pixel value.

[0350] The block having the first pixel value may represent an area where an artifact exists in the first image 115 , and the block having the second pixel value may represent an area where an artifact does not exist in the first image 115 .

[0351] According to an embodiment of the present disclosure, the artifact remover 1216 may perform morphological processing on the artifact map 1800 generated based on the distribution of the frequency transform coefficients of each block of the original image 105 .

[0352] Morphological processing may be a process of changing the pixel value of a hole in an image based on the pixel values ​​of adjacent pixels or changing the pixel value of an island in an image based on the pixel values ​​of adjacent pixels. Fig.18a , block i5 has a pixel value different from the pixel values ​​of the adjacent blocks h4, i4, j4, h5, j5, h6, i6, and j6, and therefore, block i5 corresponds to a hole. In addition, block f7 has a pixel value different from the pixel values ​​of the adjacent blocks e6, f6, g6, e7, g7, e8, f8, and g8, and therefore, block f7 corresponds to an island.

[0353] Fig.18b An artifact image 1850 is shown after morphological processing has been performed. Fig.18b As shown, the pixel value of block i5 may be changed from the second pixel value to the first pixel value, and the pixel value of block f7 may be changed from the first pixel value to the second pixel value.

[0354] The morphological processing may include an erosion processing and a dilation processing. The morphological processing may include an opening processing that continuously performs the erosion processing and the dilation processing. By performing the opening processing, noise may be removed by the erosion processing, and a region whose size has been reduced may be restored by the dilation processing.

[0355] However, as mentioned above Fig.18a and Fig.18b The operation of removing outliers may be accompanied by complex calculations (such as morphological processing). Fig.19a and Fig.19b A process of removing abnormal values ​​through simple calculation by the AI ​​encoding device 1200 is described.

[0356] Fig.19a and Fig.19b is a diagram for describing a process of removing an outlier by an AI encoding device according to an embodiment of the present disclosure.

[0357] Reference Fig.19aAccording to an embodiment of the present disclosure, the artifact remover 1216 may perform a process for removing abnormal values ​​on an artifact map generated based on the distribution of frequency transform coefficients of each block of the original image 105 .

[0358] When the current block 1910 included in the artifact map has a first pixel value and the number of blocks having the first pixel value among the blocks 1920 included in the preset area 1900 ranging from the current block 1910 is greater than a specific threshold value (e.g., 5), the artifact remover 1216 may determine not to change the first pixel value of the current block 1910. Fig.19b According to an embodiment of the present disclosure, when a current block 1940 included in an artifact map has a first pixel value and the number of blocks having the first pixel value among blocks 1950 included in a preset area 1930 ranging from the current block 1940 is less than or equal to a specific threshold value (e.g., 5), the artifact remover 1216 may change the first pixel value of the current block 1940 to a second pixel value.

[0359] According to an embodiment of the present disclosure, when the first image 115 is configured with a plurality of frames, the artifact remover 1216 may generate an artifact map corresponding to each frame. According to an embodiment of the present disclosure, the artifact remover 1216 may generate an artifact map corresponding to some of the plurality of frames of the first image 115 to prevent the bit rate from increasing. For example, the artifact remover 1216 may generate an artifact map corresponding to a frame and / or an I frame in which a scene change occurs among the plurality of frames.

[0360] According to an embodiment of the present disclosure, the artifact remover 1216 may compare an artifact map of the original image 105 with at least one artifact map of a previous image, and perform post-processing of changing pixel values ​​of pixels in the first image 115 based on a determined result.

[0361] Furthermore, a process of changing pixel values ​​based on a smoothing process and / or a seamless process may be performed on the artifact map. The smoothing process may be based on various filtering methods for smoothing, and includes morphological processing.

[0362] Figures 20a to 20f is a diagram for describing a method of correcting an artifact map by performing a smoothing process and a seamless process on a post-processing application area of ​​the artifact map according to an embodiment of the present disclosure.

[0363] Reference Figures 20a to 20f , the AI ​​encoding device 1200 may perform smoothing and / or seamless processing on the artifact region 2010 included in the artifact map 2000 to generate an artifact map 2020 having an artifact region 2030. When a basic unit of a block map is a block (e.g., N×N, where N is an integer), smoothing and seamless processing may be performed in units of pixels smaller than a block or in units of blocks smaller than N×N.

[0364] The smoothing process is a process for smoothing the boundary of a region. For example, by performing the smoothing process on the region 2040, the region 2050 can be generated. The smoothing process may include a morphological process (eg, an opening process).

[0365] The seamless process is a process for blurring the boundary of a region. For example, the seamless process may include a Poisson blending process. For example, by performing the seamless process on the region 2040, the region 2060 may be generated.

[0366] For example, by performing both seamless processing and smoothing processing on the region 2040, a region with a blurred and smooth boundary (such as the region 2070) can be generated.

[0367] By performing a process of changing pixel values ​​on the artifact map based on a smoothing process and / or a seamless process and performing post-processing on a region of the first image 115 corresponding to the artifact map, a more natural image with a smooth boundary may be generated.

[0368] Fig.21 is a diagram for describing a method of correcting an artifact area by comparing a block in an artifact map of a current frame with a co-located block in an artifact map of an immediately previous frame.

[0369] Reference Fig.21 , the artifact remover 1216 may determine whether the number of blocks having the first pixel value among the N second blocks in the artifact map 2100 of N (N is an integer) previous images (e.g., frames) is greater than or equal to K (K is an integer), wherein the N second blocks correspond to the first block having the first pixel value in the artifact map of the original image, and perform post-processing of changing the pixel values ​​of the pixels in the first block based on the result of the determination. At this time, the second block of the previous image corresponding to the first block may be a block co-located with the first block.

[0370] In addition, the artifact remover 1216 may perform post-processing of changing the pixel value of a pixel in the first image based on at least one of the motion vector or the optical flow of the original image. For example, when the artifact remover 1216 determines whether the number of blocks having the first pixel value among the N second blocks in the artifact map of N (N is an integer) previous images (e.g., frames) corresponding to the first block having the first pixel value in the artifact map 2100 of the original image is greater than or equal to K (K is an integer), the artifact remover 1216 may determine the second block of the previous image corresponding to the first block based on at least one of the motion vector or the optical flow of the original image. In this case, when the motion amplitude based on the amplitude of the motion vector and the optical flow is greater than a preset amplitude, when determining whether the number of blocks is greater than or equal to K, the second block in the artifact map of the corresponding previous image may not be counted.

[0371] The artifact remover 1216 may calculate the sum of absolute differences (SAD) between the transform coefficients of a first block of the original image and the transform coefficients of a second block of at least one previous original image, the second block being co-located with the first block. In some embodiments, the SAD is based on the sum of at least one of the absolute values ​​of the difference in the real part or the absolute values ​​of the difference in the imaginary part of the frequency transform coefficients X specified over some domain (range of images and indices). The artifact remover 1216 may determine whether the calculated SAD is within a preset range. That is, the artifact remover 1216 may determine whether the calculated SAD is within a preset range according to Equation 2 below.

[0372] [Equation 2]

[0373] Thd1 <SAD<Thd2

[0374] Here, Thd1 and Thd2 may have thresholds of preset values.

[0375] The artifact remover 1216 may perform post-processing of changing the pixel value of the pixel in the first image based on the determined result. That is, when the artifact remover 1216 determines that the calculated SAD is within a preset range, the artifact remover 1216 may determine post-processing of changing the pixel value of the pixel in the first image. The first block may be a block determined based on the artifact information, and may be a block having a first pixel value in the artifact map.

[0376] The artifact remover 1216 can be based on Fig.17a and Fig.17b as well as Fig.18a and Fig.18b Artifact maps 1700, 1710, 1800, and 1850 are shown to perform post-processing that changes the pixel values ​​of pixels in the first image.

[0377] According to an embodiment of the present disclosure, the artifact remover 1216 may apply a preset filter to an area including artifacts in the first image 115 based on the artifact map to perform post-processing of changing pixel values ​​of pixels in the first image 115. The preset filter may include at least one of a Gaussian filter or a low-pass filter.

[0378] For example, the artifact remover 1216 may apply a Gaussian filter to an area including artifacts in the first image 115 in the spatial domain based on the artifact map. The artifact remover 1216 may apply a low-pass filter to an area including artifacts in the first image 115 in the frequency domain based on the artifact map. At this time, the cutoff frequency of the low-pass filter may be determined based on the maximum value of the transform coefficient located in the high-frequency region among the frequency transform coefficients of the region in the original image, wherein the region in the original image corresponds to the region including artifacts in the first image 115. For example, the cutoff frequency of the low-pass filter may be determined as 1 / 2 of the frequency of the transform coefficient having the maximum value in the high-frequency region. Optionally, the cutoff frequency of the low-pass filter may be determined by the following equation 3.

[0379] [Equation 3]

[0380] Cutoff-freq=(pos_max_val_high-pos_dc) / 2

[0381] Here, pos_max_val_high represents the frequency of the transform coefficient having the maximum value in the high frequency region, and pos_dc represents the frequency of the DC coefficient in the low frequency region. The cutoff frequency may be determined for each of the horizontal direction and the horizontal direction.

[0382] According to an embodiment of the present disclosure, the artifact remover 1216 may filter the region in the original image corresponding to the region including the artifact in the first image 115 based on the artifact map by performing low-pass filtering and filtering using a typical filter including a reduction function (such as a bicubic filter and a bilinear filter), or by performing filtering using a typical filter including both a low-pass filtering function and a reduction function (such as a Lanczos filter), thereby generating a reduced region from which high-frequency components have been removed. The reduced region may be reduced by the same scaling degree as the first image 115 reduced by the first DNN AI.

[0383] The pixel values ​​of the pixels in the region including the artifacts in the first image 115 may be changed to the pixel values ​​of the pixels in the reduced region. The artifact remover 1216 may expand the corresponding region in the original image 105, apply the filter to the expanded region to generate a reduced region, reduce the reduced region to correspond to the region including the artifacts in the first image 115, and change the pixel values ​​of the pixels included in the region including the artifacts in the first image 115 to the pixel values ​​of the pixels in the reduced region. In addition, according to an embodiment of the present disclosure, the artifact remover 1216 may use an artifact detection network to generate an artifact map. The artifact detection network may be a DNN and include at least one convolution layer. The artifact detection network may be trained based on an input image and a ground truth (GT) image with confirmed artifacts. The artifact remover 1216 may input the first image 115 into the artifact detection network and obtain an artifact map output from the artifact detection network.

[0384] The first encoder 1214 may encode the first image 115 post-processed by the artifact remover 1216. As described above, the encoding may include a process of predicting the post-processed first image to generate prediction data, a process of generating residual data corresponding to a difference between the post-processed first image and the prediction data, a process of transforming the residual data as a spatial domain component into a frequency domain component, a process of quantizing the residual data transformed into the frequency domain component, and a process of entropy encoding the quantized residual data.

[0385] The data processor 1232 may process at least one of the AI ​​data or the image data so that at least one of the AI ​​data or the image data is sent in a preset form. For example, when the AI ​​data and the image data need to be sent in the form of a bit stream, the data processor 1232 may process the AI ​​data to be represented in the form of a bit stream, and send the AI ​​data and the image data in the form of a bit stream through the communicator 1234. According to another embodiment of the present disclosure, the data processor 1232 may process the AI ​​data to be represented in the form of a bit stream, and send a bit stream corresponding to the AI ​​data and a bit stream corresponding to the image data through the communicator 1234. According to another example, the data processor 1232 may process the AI ​​data to be represented in the form of a frame or a packet, and send the image data in the form of a bit stream and send the AI ​​data in the form of a frame or a packet through the communicator 1234.

[0386] The communicator 1234 may transmit the AI-encoded data generated as a result of the AI-encoding through the network. The AI-encoded data generated as a result of the AI-encoding may include image data and AI data.

[0387] Image data and AI data may be transmitted through the same type of network or different types of networks.

[0388] The image data may include data generated as a result of the first encoding of the post-processed first image. The image data may include data generated based on pixel values ​​in the post-processed first image, for example, residual data that is a difference between the post-processed first image and the prediction data. In addition, the image data may include information used in the process of performing the first encoding on the post-processed first image. For example, the image data may include information about a mode for performing the first encoding on the post-processed first image, and information related to a quantization parameter for performing the first encoding on the post-processed first image.

[0389] The AI ​​data may include information for enabling the AI ​​decoding device to AI-enlarge the second image 135 as the enlarged target corresponding to the reduced target of the first DNN. According to an example, the AI ​​data may include information about the difference between the original image 105 and the first image 115 (or the post-processed first image). In addition, the AI ​​data may include information related to the first image 115 (or the post-processed first image). The information related to the first image 115 (or the post-processed first image) may include information about at least one of the resolution of the first image 115 (or the post-processed first image), the bit rate of the image data generated as a result of the first encoding of the post-processed first image, or the codec type used to perform the first encoding on the post-processed first image. In addition, the AI ​​data may include artifact information generated by the artifact remover 1216. For example, the AI ​​data may include data of an artifact map.

[0390] According to an embodiment of the present disclosure, the artifact map may also be firstly encoded by the first encoder 1214 and then transmitted as image data.

[0391] Fig.22a is a flowchart for describing an AI encoding method according to another embodiment of the present disclosure.

[0392] In operation S2210, the AI ​​encoding apparatus 1200 may obtain a first image 115 AI-reduced from an original image 105 by using a first DNN.

[0393] In operation S2220, the AI ​​encoding device 1200 may generate artifact information representing an area including an artifact in the first image 115. The artifact may include at least one of a spatial aliasing artifact or a spatiotemporal aliasing artifact. The artifact information may include an artifact map having a preset size. For example, when the resolution of the original image 105 is 3840×2160, an artifact map of a size of 240×135 may be generated. In this case, the basic unit of the block map may correspond to a block of the original image 105 of a size of 16×16. When the resolution of the reduced image is 1920×1080, the basic unit of the block map may correspond to a block of the reduced image of a size of 8×8, but is not limited thereto. However, the basic unit of the block map may correspond to a block of the original image 105 of a size of M×M (M is an integer). When the basic unit of the block map corresponds to a pixel of the original image 105, the block map may have a size of 3840×2160. The artifact may represent at least one of a spatial aliasing artifact or a spatiotemporal aliasing artifact.

[0394] The AI ​​encoding apparatus 1200 may generate an artifact map based on a value of a frequency transform coefficient of the original image 105. The AI ​​encoding apparatus 1200 may generate an artifact map based on a distribution of frequency transform coefficients of each block of the original image 105. According to another example, the AI ​​encoding apparatus 1200 may generate an artifact map by using an artifact detection network.

[0395] In operation S2230, the AI ​​encoding device 1200 may perform post-processing of changing pixel values ​​of pixels in the first image 115 based on the artifact information. For example, the AI ​​encoding device 1200 may apply a preset filter to a region including artifacts in the first image 115 based on the artifact information, thereby changing the pixel values ​​of the pixels in the first image 115. In this case, the preset filter may include at least one of a Gaussian filter or a low-pass filter.

[0396] Alternatively, the AI ​​encoding device 1200 may apply a filter to an area of ​​the original image 105 corresponding to the area including the artifact in the first image 115 based on the artifact information to generate a reduced area, and change the pixel values ​​of the pixels in the area including the artifact in the first image 115 to the pixel values ​​of the pixels in the reduced area. In this case, the filter may be a typical filter. For example, the filter may include a bicubic filter, a bilinear filter, and a Lanczos filter.

[0397] In operation S2240, the AI ​​encoding device 1200 may perform first encoding on the post-processed first image. The AI ​​encoding device 1200 may transmit AI data and image data generated as a result of encoding the post-processed first image to the AI ​​decoding device. The AI ​​data may include artifact information.

[0398] Figure 22b is a diagram for describing an AI encoding method according to another embodiment of the present disclosure. In operation S2250, the AI ​​encoding apparatus 1200 may obtain a first image 115 AI-reduced from an original image 105 by using a first DNN.

[0399] In operation S2260, the AI ​​encoding device 1200 may perform frequency transformation on the blocks of the original image 105, and generate first artifact information representing an area including artifacts in the first image 115 based on the value of the transformation coefficient of each block generated by the frequency transformation. The first artifact information may include an artifact map. As described above, abnormal values ​​of the artifact map may be removed.

[0400] In operation S2270, the AI ​​encoding apparatus 1200 may generate second artifact information about the current image based on at least one of artifact information of at least one previous image or first artifact information of the current image. The artifact information of the previous image may include an artifact map, and outliers of the artifact map may be removed.

[0401] In operation S2280, the AI ​​encoding apparatus 1200 may perform post-processing of changing pixel values ​​of pixels in the first image 115 based on at least one of the first artifact information or the second artifact information. When the AI ​​encoding apparatus 1200 performs post-processing of changing pixel values ​​of pixels in the first image 115 based on the first artifact information, operation S2270 of generating the second artifact information may be omitted.

[0402] Fig.23 23 is a block diagram illustrating a configuration of an AI decoding device 2300 according to another embodiment of the present disclosure.

[0403] Reference Fig.23 , the AI ​​decoding device 2300 may include a receiver 2310 and an AI decoder 2330. The receiver 2310 may include a communicator 2312, a parser 2314, and an outputter 2316, and the AI ​​decoder 2330 may include a first decoder 2332 and an AI amplifier 2334. The communicator 2312 may receive AI encoded data including image data and AI data through a network. The image data may include information generated as a result of the first encoding of the post-processed first image, and the AI ​​data may include artifact information.

[0404] The parser 2314 may separate the AI ​​encoded data received through the communicator 2312 into image data and AI data, and transmit the image data to the first decoder 2332 and the AI ​​data to the AI ​​amplifier 2334 through the outputter 2316 .

[0405] As above Figure 2The operations of the communicator 212, the parser 214 and the outputter 216 of the AI ​​decoding device 200 are similar to Fig.23 The operations of the communicator 2312, the parser 2314, and the outputter 2316 of the AI ​​decoding device 2300 are the same, and therefore, a detailed description thereof will be omitted.

[0406] The AI ​​decoding device 2300 according to an embodiment of the present disclosure may include a receiver 2310 and an AI decoder 2330. The AI ​​decoding device 2300 may include a central processor (not shown) for controlling the receiver 2310 and the AI ​​decoder 2330. Alternatively, the receiver 2310 and the AI ​​decoder 2330 may be operated by their own processors (not shown), and the processors (not shown) may be operated in an organized manner with each other in order to operate the AI ​​decoding device 2300. Alternatively, the receiver 2310 and the AI ​​decoder 2330 may be controlled by the control of an external processor (not shown) of the AI ​​decoding device 2300.

[0407] The AI ​​decoding device 2300 may include one or more data storage devices (not shown) for storing input / output data of the communicator 2312, the parser 2314, the outputter 2316, the first decoder 2332, and the AI ​​amplifier 2334. The AI ​​decoding device 2300 may include a memory controller (not shown) for controlling data input / output of the data storage device (not shown).

[0408] The AI ​​decoding device 2300 can perform image decoding including prediction by interacting with an internal video decoding processor or an external video decoding processor to decode the image. The internal video decoding processor of the AI ​​decoding device 2300 according to an embodiment of the present disclosure may be set as a separate processor, or the central processing unit or the graphics processing unit may include an image decoding processing module for implementing basic image decoding operations.

[0409] The first decoder 2332 may perform a first decoding on the image data to generate a second image 135 corresponding to the post-processed first image. According to an embodiment of the present disclosure, when the artifact information is first encoded and then transmitted to the communicator 2312, the first decoder 2332 may perform a first decoding on the image data corresponding to the artifact information to generate the artifact information.

[0410] The AI ​​amplifier 2334 may receive the second image 135 from the first decoder 2332 and amplify the second image 135 to a third image 145 based on the second DNN. Figure 2 The second DNN-based AI amplification has been described in detail, so its description will be omitted.

[0411] The AI ​​amplifier 2334 may output the third image 145 and the artifact information. According to an embodiment of the present disclosure, the AI ​​amplifier 2334 may amplify the artifact map before outputting the artifact map. As described above, since the artifact map may have the same resolution as the first image 115, the AI ​​amplifier 2334 may also amplify the artifact map so that the resolution of the artifact map is the same as the resolution of the third image 145. At this time, the AI ​​amplifier 2334 may perform conventional amplification on the artifact map.

[0412] According to an embodiment of the present disclosure, the AI ​​amplifier 2334 may output artifact information and the third image 145 to an image quality engine. The image quality engine may perform image quality enhancement processing on the image before a display device (such as a TV) displays the image. The image quality engine may include a plurality of filters for processing the image. For example, the image quality engine may apply at least one of image signal analysis, noise removal, detail enhancement, or amplification to the image to enhance image quality.

[0413] According to an embodiment of the present disclosure, when the AI ​​decoding device 2300 is included in a display device, the AI ​​amplifier 2334 may output the artifact information and the third image 145 to a storage device (e.g., a main storage device or an auxiliary storage device) in the display device to store the artifact information and the third image 145 in the storage device so that the image quality engine performs image quality enhancement of the third image 145. The image quality engine may load the third image 145 and the artifact information from the storage device and perform image quality enhancement on the third image 145. Alternatively, the AI ​​amplifier 2334 may directly send the artifact information and the third image 145 to the image quality engine in the display device so that the image quality engine performs image quality enhancement of the third image 145.

[0414] According to an embodiment of the present disclosure, the AI ​​decoding device 2300 may output or transmit the third image 145 and artifact information to a display device through a wired and / or wireless network.

[0415] The image quality engine may perform post-processing for changing pixel values ​​of pixels in the third image 145 based on the artifact information.

[0416] Fig.24a is a diagram illustrating the image quality engine 2400 .

[0417] The image quality engine 2400 may include a plurality of filters, and the artifact map may be input to the filter that post-processes the third image 145 based on the artifact map, that is, Fig.24a The anti-aliasing filter may include a low-pass filter or a Gaussian filter. The low-pass filter may be a filter in the space domain or a filter in the frequency domain formed in the shape of a mask having a preset size.

[0418] According to an embodiment of the present disclosure, the image quality engine 2400 may change the weight of the low-pass filter so that the aliasing artifacts present in the third image 145 can be more effectively removed. For example, the image quality engine 2400 may change the preset weight of the low-pass filter. For example, the image quality engine 2400 may increase or decrease the preset weight of the low-pass filter.

[0419] According to an embodiment of the present disclosure, the image quality engine 2400 may apply a low-pass filter to a region including artifacts in the third image 145 based on the artifact map.

[0420] More specifically, the image quality engine 2400 may apply a low pass filter to an area of ​​the third image 145 corresponding to an area having the first pixel value in the artifact map.

[0421] For example, the AI ​​decoding device 2300 may receive information about the cutoff frequency sent from the AI ​​encoding device 1200, and the image quality engine 2400 may apply a low-pass filter in the frequency domain to a region in the third image 145 based on the information about the cutoff frequency output from the AI ​​decoding device 2300.

[0422] According to another example, the image quality engine 2400 may apply a low-pass filter formed in the shape of a mask having a preset size to an area including artifacts in the third image 145 based on the artifact map. The strength of the low-pass filter applied to the third image 145 may be determined according to a pixel unit of the third image 145 or a value of the artifact map.

[0423] Will refer to Figure 24b An example of adjusting the strength of a low-pass filter formed in the shape of a mask having a preset size according to a value of an artifact map is described.

[0424] The image quality engine 2400 may determine the strength range of the low-pass filter according to a preset standard. For example, the image quality engine 2400 may determine the strength of the filter based on at least one of the type, resolution, size, and image quality of the original image 105, the type of subject included in the original image 105, or artifact information. The third image 145 that has passed through the plurality of filters of the image quality engine 2400 may be displayed through a display device.

[0425] As described above, when it is determined that the second image 135 is not enlarged, the second image 135 that has passed through a plurality of filters of the image quality engine 2400 may be displayed through the display device.

[0426] Figure 24b is a diagram for describing a method for adjusting the strength of a low-pass filter based on artifact information and performing filtering, performed by the AI ​​decoding device 2300 .

[0427]

[0428] Reference Figure 24b , the low-pass filter may be a filter formed in the shape of a 3×3 mask 2450. The AI ​​decoding device 2300 may adjust the strength of the low-pass filter by adjusting the value of the coefficient in the 3×3 mask 2450. For example, the AI ​​decoding device 2300 may change the coefficient located at the center of the 3×3 mask 2450 from 2 to 5 among the coefficients in the 3×3 mask 2450 based on the artifact map. In this case, the blurriness of the image may be weakened by filtering. Optionally, the AI ​​decoding device 2300 may change the remaining coefficients of the coefficients in the 3×3 mask 2450 other than the coefficient located at the center from 1 to 2 based on the artifact map.

[0429] At this time, the ratio and value of the coefficients may be determined based on various methods.

[0430] For example, when the coefficient is based on a Gaussian filter, the coefficient may be determined based on Equation 4 below.

[0431] [Equation 4]

[0432] The AI ​​decoding apparatus 2300 may determine the values ​​of each coefficient by adjusting σ based on the artifact map.

[0433] In addition, the AI ​​decoding device 2300 may determine the strength of the low-pass filter based on the first pixel value of the artifact map. For example, when the first pixel value is less than a preset value, the AI ​​decoding device 2300 may determine the strength of the low-pass filter as a first strength, and when the first pixel value is greater than or equal to the preset value, the AI ​​decoding device 2300 may determine the strength of the low-pass filter as a second strength.

[0434] Fig.25 25 is a block diagram showing a configuration of an AI decoding device 2500 according to another embodiment of the present disclosure.

[0435] Reference Fig.25 , the AI ​​decoding device 2500 may include a receiver 2510 and an AI decoder 2530. The receiver 2510 may include a communicator 2512, a parser 2514, and an outputter 2516, and the AI ​​decoder 2530 may include a first decoder 2532, an AI amplifier 2534, and an artifact remover 2536.

[0436] The communicator 2512 may receive AI encoding data including image data and AI data through a network. The image data may include information generated as a result of first encoding of the post-processed first image, and the AI ​​data may include artifact information.

[0437] The parser 2514 may separate the AI ​​encoded data received through the communicator 2512 into image data and AI data, and transmit the image data to the first decoder 2532 through the outputter 2516 and send the AI ​​data to the AI ​​amplifier 2534 .

[0438] Refer to above Figure 2 The operations of the communicator 212, the parser 214, and the outputter 216 of the AI ​​decoding device 200 described above are similar to those of Fig.25 The operations of the communicator 2512, the parser 2514, and the outputter 2516 of the AI ​​decoding device 2500 are the same, and therefore, a detailed description thereof will be omitted.

[0439] The AI ​​decoder 2532 may perform a first decoding on the image data to generate a second image 135 corresponding to the post-processed first image. According to an embodiment of the present disclosure, when the artifact information is first encoded and then transmitted to the communicator 2512, the first decoder 2532 may perform a first decoding on the image data corresponding to the artifact information to generate the artifact information. In this case, the artifact information may be provided from the first decoder 2532 to the artifact remover 2536.

[0440] The artifact remover 2536 may receive the artifact information included in the AI ​​data from the outputter 2516. Alternatively, the artifact remover 2536 may receive the artifact information restored as a result of the first decoding of the image data from the first decoder 2532. Alternatively, the artifact remover 2536 may receive the artifact information included in the AI ​​data from the AI ​​amplifier 2534.

[0441] The artifact remover 2536 may perform post-processing on the region including the artifact in the second image 135 .

[0442] The artifact remover 2536 may receive the second image generated by the first decoder 2532 and perform post-processing of changing pixel values ​​of pixels of the second image 135 based on the artifact information. More specifically, the artifact remover 2536 may apply a low-pass filter to an area in the second image 135 including artifacts.

[0443] The post-processed second image output from the artifact remover 2536 may be sent to the AI ​​amplifier 2534, and the AI ​​amplifier 2534 may output the third image 145 amplified by the second DNN AI. The third image 145 may be post-processed as needed and then displayed by the display device.

[0444] In addition, according to an embodiment of the present disclosure, the artifact remover 2536 may perform post-processing on the third image 145 instead of the second image 135. In this case, the AI ​​amplifier 2534 may provide the third image 145 AI-enlarged from the second image 135 generated by the first decoder 2532 to the artifact remover 2536. For post-processing of the third image 145, the AI ​​amplifier 2534 may enlarge an artifact map having a low resolution to generate an artifact map having a high resolution, and provide the artifact map having a high resolution to the artifact remover 2536.

[0445] The artifact remover 2536 may apply a low pass filter to the area including the artifacts in the third image 145 .

[0446] The post-processed third image may be output from the artifact remover 2536 for display.

[0447] Fig.26 is a flowchart for describing an AI decoding method according to another embodiment of the present disclosure.

[0448] In operation S2610, the AI ​​decoding apparatus 2500 may obtain image data generated as a result of first encoding of the post-processed first image and AI data related to AI reduction.

[0449] The image data may be received in the form of a bitstream. The image data may include data generated based on pixel values ​​in the post-processed first image, for example, residual data that is a difference between the post-processed first image and the prediction data. In addition, the image data may include information used in the first encoding process of the post-processed first image. For example, the image data may include information about a mode for performing the first encoding on the post-processed first image, and information related to a quantization parameter for performing the first encoding on the post-processed first image.

[0450] The AI ​​data may include information for enabling the second DNN to AI-enlarge the second image 135 as the enlargement target corresponding to the reduction target of the first DNN. In addition, the AI ​​data may include artifact information (eg, an artifact map) indicating an area including artifacts in the first image 115.

[0451] In operation S2620, the AI ​​decoding apparatus 2500 may obtain the second image 135 based on the image data. More specifically, the AI ​​decoding apparatus 2500 may generate the second image 135 corresponding to the post-processed first image by performing first decoding on the image data based on an image restoration method using frequency transformation.

[0452] In operation S2630, the AI ​​decoding apparatus 2500 may obtain the third image 145 AI-enlarged from the second image 135 by using the second DNN.

[0453] In operation S2640, the AI ​​decoding apparatus 2500 may output the artifact information and the third image 145 to the image quality engine. The AI ​​decoding apparatus 2500 may request a change in the weight of a filter that processes the third image 145 using the artifact information among a plurality of filters of the image quality engine.

[0454] The third image 145 on which the image quality enhancement is performed by the image quality engine may be displayed by the display device.

[0455] In addition, the above reference Fig. 9 The process of training the first DNN 700 and the second DNN 300 is described. According to an embodiment of the present disclosure, the first training image may be replaced with a first training image after processing based on artifact information. Fig. 9 The first training image 802 can be used to train the first DNN 700 and the second DNN 300. In addition, the second training image generated by performing the first encoding and the first decoding on the first training image 802 can be used as the input of the second DNN 300. In addition, the third training image 804 can be replaced with a third training image post-processed based on the artifact information, so that the first DNN 700 and the second DNN 300 can be trained.

[0456] Furthermore, the above-described embodiments of the present disclosure may be written as computer-executable programs or instructions that may be stored in a medium.

[0457] The medium can store computer executable programs or instructions continuously, or temporarily store computer executable programs or instructions for execution or downloading. In addition, the medium can be any of various recording media or storage media combined with single or multiple pieces of hardware, and the medium is not limited to the medium directly connected to the computer system, but can be published on the network. Examples of media include magnetic media (such as hard disks, floppy disks and tapes) configured to store program instructions, optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as optical floppy disks), and ROM, RAM and flash memory. Other examples of media include recording media and storage media managed by application stores that publish applications or by websites, servers, etc. that provide or publish various other types of software.

[0458] In addition, the model related to the above-mentioned DNN can be implemented via a software module. When the DNN model is implemented via a software module (e.g., a program module including instructions), the DNN model can be stored in a computer-readable recording medium.

[0459] In addition, the DNN model can be integrated in the form of a hardware chip to become part of the above-mentioned AI decoding device 200 or AI encoding device 600. For example, the DNN model can be manufactured in the form of a dedicated hardware chip for AI, or can be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU).

[0460] In addition, the DNN model can be provided in the form of downloadable software. The computer program product may include a product in the form of a software program electronically released by a manufacturer or an electronic market (e.g., a downloadable application). For electronic release, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of the manufacturer or electronic market, or a storage medium of a forwarding server.

[0461] The method and apparatus for AI encoding and AI decoding according to the embodiment may process an image at a low bit rate through AI-based image encoding and decoding.

[0462] In addition, the method and apparatus for AI encoding and AI decoding according to an embodiment of the present disclosure can remove artifacts in an image to enhance the quality of the image.

[0463] It should be noted that the effects of the method and device for AI encoding and decoding according to the embodiments of the present disclosure are not limited to those described above, and other effects of the present disclosure will be apparent to those skilled in the art based on the following description.

[0464] Although one or more embodiments of the present disclosure have been described with reference to the drawings, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope defined by the following claims.

Claims

1. A server for providing an image by using artificial intelligence (AI), wherein: The server comprises: The one or more processors, when executing one or more instructions stored in the server, are configured to perform the following operations: Based on the narrowed target, obtaining NN setting information from a plurality of pieces of neural network NN setting information pre-stored in the server, Performing AI downscaling on the original image using a neural network NN for downscaling to obtain a first image, wherein the NN for downscaling is set by the obtained NN setting information, and the original image includes a plurality of blocks, Based on a plurality of frequency transform coefficients of the original image, artifact information of an artifact map including a first image is obtained, wherein the artifact information indicates a first region including an artifact in the first image, changing a pixel value of a pixel in the first image based on the artifact information, encoding the first image whose pixel values ​​are changed to obtain image data, and The image data and AI data related to AI reduction are transmitted to an electronic device, wherein the AI ​​data includes the artifact information and includes an identifier of the obtained NN setting information.

2. The server according to claim 1, wherein: The artifact map has a preset size.

3. The server according to claim 1, in, indicating the presence of an artifact in the artifact map by a first pixel value, and indicating the absence of an artifact in the artifact map by a second pixel value, wherein the one or more processors are further configured to execute the one or more instructions to perform the following operations: performing a frequency transform on a first block of the plurality of blocks to obtain a second plurality of frequency transform coefficients corresponding to the first block, wherein the plurality of frequency transform coefficients include the second plurality of frequency transform coefficients, determining whether a criterion is satisfied based on the distribution of the second plurality of frequency transform coefficients, and When the criterion is met, the artifact map is obtained, wherein the first pixel value in the artifact map represents the degree value of the artifact.

4. The server according to claim 3, wherein: When the criterion is not met, the artifact map includes a value of the extent of the artifact as the second pixel value.

5. The server according to claim 3, wherein: The original image is a current image among multiple original images in a video sequence, and the one or more processors are also configured to execute the one or more instructions to perform the following operations: changing the pixel values ​​of pixels in the first image based on comparing the artifact map with at least one second artifact map of a previous original image among the multiple original images in the video sequence.

6. The server according to claim 5, wherein: The one or more processors are further configured to execute the one or more instructions to change a pixel value of a pixel in the first image by: determining whether equal to or greater than K previous artifact maps among the N previous artifact maps are inconsistent with the artifact map at the first position index of the second plurality of frequency transform coefficients, and When the K or more previous artifact images are inconsistent with the artifact image at the first position index, the first pixel value of the first block is changed, wherein when the first pixel value of the artifact image at the first position index is different from the first pixel value of each previous artifact image in the K or more previous artifact images at the first position index, the K or more previous artifact images are inconsistent with the artifact image.

7. The server according to claim 3, wherein: The criterion is satisfied when one or more of the first condition, the second condition, the third condition or the fourth condition is satisfied, The first block includes the DC frequency conversion coefficient, the low frequency area and the high frequency area. The first condition is satisfied when a first sum of first energy values ​​of the second plurality of frequency transform coefficients over the high frequency region exceeds a scaled version of a second sum of second energy values ​​of the second plurality of frequency transform coefficients over the low frequency region, wherein the scaling of the second sum is a first predetermined scaling, The second condition is satisfied when a third sum of the first maximum energy value at a third position index of a second frequency transform coefficient including a second plurality of frequency transform coefficients on the high frequency region and the energy values ​​of the frequency transform coefficients at a fourth position index located around the third position index exceeds a scaled version of a fourth sum of the second maximum energy value at a fifth position index of a third frequency transform coefficient including a second plurality of frequency transform coefficients on the low frequency region and the energy values ​​of the frequency transform coefficients located around the fifth position index, wherein the scaling of the fourth sum is a second predetermined scaling, A third condition is satisfied when the third sum of the high frequency regions exceeds a scaled version of the first sum, wherein the scaling of the first sum is a third predetermined scaling, or The fourth condition is satisfied when the number of times that the second plurality of frequency transform coefficients on the high frequency region and the low frequency region have energies exceeding a preset value is less than a threshold number.

8. The server according to claim 1, wherein: The one or more processors are further configured to execute the one or more instructions to perform the following operations: perform morphological processing, smoothing processing, or seamless processing on the first image based on the artifact map.

9. The server according to claim 1, wherein: The one or more processors are further configured to execute the one or more instructions to: performing a frequency transform on a first block of the plurality of blocks to obtain a second plurality of frequency transform coefficients corresponding to the first block, wherein the plurality of frequency transform coefficients comprises a second plurality of frequency transform coefficients, calculating a sum of absolute differences (SAD) between a first frequency transform coefficient at a second position index of a second plurality of frequency transform coefficients and a second frequency transform coefficient at a second position index of a third plurality of frequency transform coefficients associated with a second block of at least one previous original image, wherein the second block is co-located with the first block, Determine if the calculated SAD is within a preset range, and When the calculated SAD is within a preset range, the first pixel value of the first block is changed.

10. The server according to claim 1, wherein: The one or more processors are further configured to execute the one or more instructions to change pixel values ​​of pixels in the first image based on the artifact map and based on at least one of a motion vector of the original image or an optical flow of the original image.

11. The server according to claim 1, wherein: The one or more processors are further configured to execute the one or more instructions to change pixel values ​​of pixels in the first image by applying a filter to the first region based on the artifact information.

12. The server according to claim 11, wherein: The filter comprises at least one of a Gaussian filter or a low pass filter, and the one or more processors are further configured to execute the one or more instructions to perform the following operations: performing a frequency transform on a first block of the plurality of blocks to obtain a second plurality of frequency transform coefficients corresponding to the first block, wherein the plurality of frequency transform coefficients include the second plurality of frequency transform coefficients, and The cutoff frequency of the low pass filter is determined based on a maximum value of transform coefficients located in a high frequency region among the second plurality of frequency transform coefficients.

13. The server according to claim 1, wherein: The one or more processors are further configured to execute the one or more instructions to: applying a filter to a second region of the original image corresponding to the first region in the first image based on the artifact information to obtain a reduced region including a second pixel value; as well as A third pixel value in a first region including an artifact in the first image is changed to a second pixel value of the reduced region.

14. The server according to claim 13, wherein: The filter includes at least one of a bicubic filter, a bilinear filter, or a Lanczos filter.

15. A method for providing an image by a server configured to use artificial intelligence (AI), wherein: The method comprises: Based on the narrowed target, obtain NN setting information from a plurality of pieces of neural network NN setting information pre-stored in a server; Performing AI downscaling on an original image using a neural network NN for downscaling to obtain a first image, wherein the NN for downscaling is set by the obtained NN setting information, and the original image includes a plurality of blocks; Obtaining artifact information including an artifact map of a first image based on a plurality of frequency transform coefficients of the original image, wherein the artifact information indicates a first region including an artifact in the first image; changing a pixel value of a pixel in the first image based on the artifact information; encoding the first image whose pixel values ​​are changed to obtain image data; and The image data and AI data related to AI reduction are transmitted to an electronic device, wherein the AI ​​data includes the artifact information and includes an identifier of the obtained NN setting information.

Images