Coding method and device thereof, and decoding method and device thereof

Abstract

Provided is a computer-readable recording medium storing a video file containing AI encoded data, wherein the AI ​​encoded data includes: image data including encoding information of a low-resolution image generated by AI downscaling of a high-resolution image; and AI data for AI upscaling of the low-resolution image reconstructed based on the image data, wherein the AI ​​data includes: AI target data indicating whether AI upscaling is applied to one or more frames, and when AI upscaling is applied to the one or more frames, AI supplementary data regarding upscaling DNN information for AI upscaling of the one or more frames in multiple preset default DNN configuration information.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing. More specifically, the present disclosure relates to a method and device for encoding and decoding images based on a deep neural network. Background Art

[0002] The image is stored in a recording medium or transmitted via a communication channel in the form of a bit stream after being encoded via a codec complying with a specific data compression standard such as the Moving Picture Experts Group (MPEG) standard.

[0003] With the development and supply of hardware capable of reproducing and storing high-resolution and high-quality images, the demand for codecs capable of efficiently encoding and decoding high-resolution and high-quality images has increased. Summary of the invention

[0004] Technical issues

[0005] According to an aspect of the present invention, a method and apparatus for encoding and decoding an image are provided, wherein the method and apparatus encode and decode the image at a low bit rate using a deep neural network (DNN).

[0006] Furthermore, a computer-readable recording medium recording a program for executing the image encoding method and the image decoding method according to the embodiment of the present disclosure in a computer is disclosed.

[0007] Solution to the problem

[0008] In the present disclosure, a computer-readable recording medium storing a video file including artificial intelligence (AI) encoded data, wherein the AI ​​encoded data includes: image data including encoding information of a low-resolution image generated by AI downscaling of a high-resolution image; and AI data related to AI upscaling of the low-resolution image reconstructed based on the image data, wherein the AI ​​data includes: AI target data indicating whether AI upscaling is to be applied to at least one frame; and when AI upscaling is applied to the at least one frame, AI supplementary data regarding upscaling DNN information for AI upscaling of the at least one frame among multiple preset default deep neural network (DNN) configuration information.

[0009] According to another embodiment of the present disclosure, a video decoding method using artificial intelligence (AI) magnification includes: receiving a video file including AI encoded data, wherein the AI ​​encoded data includes image data and AI-enlarged AI data about the image data; obtaining the AI ​​data of the AI ​​encoded data from a metadata box of the video file, and obtaining the image data of the AI ​​encoded data from a media data box of the video file; reconstructing a low-resolution image of a current frame by decoding the image data; obtaining magnified deep neural network (DNN) information of the current frame from the AI ​​data; and generating a high-resolution image corresponding to the low-resolution image by performing AI magnification on the low-resolution image according to the magnified DNN information of the current frame.

[0010] According to another embodiment of the present disclosure, a video encoding method using artificial intelligence (AI) reduction includes: determining reduced deep neural network (DNN) information for AI-reducing a high-resolution image of a current frame to a low-resolution image; generating a low-resolution image of the current frame by AI-reducing the high-resolution image of the current frame according to the reduced DNN information; generating AI data about enlarged DNN information corresponding to the reduced DNN information, wherein the enlarged DNN information is used to AI-enlarge the low-resolution image of the current frame; obtaining image data by encoding the low-resolution image of the current frame; generating AI-encoded data including the image data and the AI ​​data; and outputting a video file including a media data box and a metadata box, wherein the image data of the AI-encoded data is embedded in the media data box, and the AI ​​data of the AI-encoded data is embedded in the metadata box.

[0011] According to another embodiment of the present disclosure, a video decoding device for performing a video decoding method using artificial intelligence (AI) amplification is provided. The video decoding device includes: a communicator configured to receive a video file including AI encoded data, wherein the AI ​​encoded data includes image data and AI amplified AI data about the image data; a parser configured to obtain the AI ​​data of the AI ​​encoded data from a metadata box of the video file, and obtain the image data of the AI ​​encoded data from a media data box of the video file; a first decoder configured to reconstruct a low-resolution image of a current frame by decoding the image data; and an AI amplifier configured to obtain amplified deep neural network (DNN) information of the current frame from the AI ​​data, and generate a high-resolution image corresponding to the low-resolution image by AI amplifying the low-resolution image according to the amplified DNN information of the current frame.

[0012] According to another embodiment of the present disclosure, a video encoding device for performing a video encoding method using artificial intelligence (AI) reduction is provided. The video encoding device includes: an AI reducer configured to determine reduced deep neural network (DNN) information for AI-scaling a high-resolution image of a current frame into a low-resolution image, generating a low-resolution image of the current frame by AI-scaling the high-resolution image of the current frame, and generating AI data for AI-enlarging the low-resolution image of the current frame; a first encoder configured to determine image data by encoding the low-resolution image of the current frame; a data processor configured to determine AI-encoded data including the image data and the AI ​​data; and a communicator configured to output a video file including a media data frame and a metadata frame, wherein the image data of the AI-encoded data is embedded in the media data frame, and the AI ​​data of the AI-encoded data is embedded in the metadata frame.

[0013] The technical problems of the present disclosure are not limited to the above-mentioned technical features, and other unstated technical problems can be inferred from the following embodiments.

[0014] The beneficial effects of disclosure

[0015] According to an artificial intelligence (AI) encoding method and apparatus and an AI decoding method and apparatus according to an embodiment, an image may be processed at a low bit rate via AI-based image encoding and decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] A brief description of each figure is provided to provide a more complete understanding of the figures described in this specification.

[0017] Figure 1 is a diagram for describing an artificial intelligence (AI) encoding process and an AI decoding process according to an embodiment.

[0018] Figure 2 is a block diagram of a configuration of an AI decoding device according to an embodiment.

[0019] Figure 3 is a diagram illustrating a second deep neural network (DNN) for performing AI upscaling on a second image.

[0020] Figure 4 is a diagram for describing the convolution operation performed by a convolutional layer.

[0021] Figure 5 is a table showing the mapping relationship between several pieces of image-related information and several pieces of DNN setting information.

[0022] Figure 6 is a diagram illustrating a second image including a plurality of frames.

[0023] Figure 7 is a block diagram of a configuration of an AI encoding device according to an embodiment.

[0024] Figure 8 is a diagram illustrating a first DNN for performing AI downscaling on an original image.

[0025] Fig. 9 is a diagram for describing a method of training a first DNN and a second DNN.

[0026] Fig.10 2 is a diagram for describing a training process of a training device on a first DNN and a second DNN.

[0027] Fig.11 is a diagram of an apparatus for performing AI downscaling on an original image and an apparatus for performing AI upscaling on a second image.

[0028] Fig.12 It is a diagram for describing the correspondence relationship between image data and AI data and the structures of image data and AI data.

[0029] Fig.13a is a diagram for describing a data flow in an AI decoding device when AI data is embedded in image data.

[0030] Fig.13b is a diagram for describing a data flow in an AI decoding device when AI data and image data are separated and included in two files.

[0031] Fig.14 An example of AI encoded data when AI data and image data are separated in a single file is shown.

[0032] Fig.15a An example of AI encoded data is shown when AI data is embedded in image data in a single file.

[0033] Fig.15b An example of AI encoded data is shown when AI data is embedded in image data in a single file.

[0034] Fig.15c An example of AI encoded data is shown when part of the AI ​​data is embedded in the image data and the remaining AI data is separated from the image data in a single file.

[0035] Fig.16 When AI data and image data are Fig.14 An example of AI encoded data divided by video segment units when separated as in .

[0036] Fig.17An example of AI data and image data being separated and sent in two files is shown.

[0037] Fig.18a Shown to be applicable Figures 14 to 17 An example of the data structure of the video AI data described in .

[0038] Fig.18b Shown for Fig.16 Video clip AI data described in or Fig.14 , Figures 15a to 15c and Fig.17 An example of the data structure of frame group AI data.

[0039] Fig.19 Show implementation Fig.18a The syntax table of the data structure.

[0040] Fig. 20 Shown for Figures 14 to 17 An example of the data structure of frame group AI data or frame AI data described in .

[0041] Fig.21 Show implementation Fig. 20 The syntax table of the data structure.

[0042] Fig. 22 is a flowchart of an image decoding method according to an AI decoder according to the present disclosure.

[0043] Fig.23 is a flowchart of an image encoding method according to an AI encoder according to the present disclosure.

[0044] Fig.24 is a block diagram showing a configuration of an image decoding device according to the present disclosure.

[0045] Fig.25 is a block diagram showing the configuration of an image encoding device according to the present disclosure.

[0046] Best Mode

[0047] In the present disclosure, a computer-readable recording medium storing a video file including artificial intelligence (AI) encoded data, wherein the AI ​​encoded data includes: image data including encoding information of a low-resolution image generated by AI downscaling of a high-resolution image; and AI data related to AI upscaling of the low-resolution image reconstructed based on the image data, wherein the AI ​​data includes: AI target data indicating whether AI upscaling is to be applied to at least one frame; and when AI upscaling is applied to the at least one frame, AI supplementary data regarding upscaling DNN information for AI upscaling of the at least one frame among multiple preset default deep neural network (DNN) configuration information. DETAILED DESCRIPTION

[0048] Since the present disclosure allows various changes and many examples, specific embodiments will be described in detail in the drawings and in the written description. However, this is not intended to limit the present disclosure to a specific mode of practice, and it will be understood that all changes, equivalents and substitutions that do not depart from the spirit and technical scope of the present disclosure are included in the present disclosure.

[0049] In the description of the embodiments, when it is considered that a specific detailed explanation of the related art may unnecessarily obscure the essence of the present disclosure, the specific detailed explanation of the related art is omitted. In addition, the numbers (e.g., first, second, etc.) used in the description of the specification are only identifier codes for distinguishing one element from another element.

[0050] Furthermore, in the present specification, it will be understood that when elements are “connected” or “coupled” to each other, the elements may be directly connected or coupled to each other, but may alternatively be connected or coupled to each other through intermediate elements between the elements unless otherwise specified.

[0051] In this specification, for an element represented as a "unit" or a "module", two or more elements may be combined into one element, or one element may be divided into two or more elements according to subdivided functions. In addition, each element described below may perform some or all of the functions performed by another element in addition to its main function, and some of the main functions of each element may be completely performed by another component.

[0052] Furthermore, in the present specification, an “image” or a “picture” may refer to a still image, a moving image including a plurality of consecutive still images (or frames), or a video.

[0053] Furthermore, in this specification, a deep neural network (DNN) is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model that uses a specific algorithm.

[0054] In addition, in this specification, "parameter" is a value used in the operation process of each layer forming a neural network, and may include, for example, a weight used when applying an input value to a specific operation expression. Here, the parameter may be expressed in a matrix form. The parameter is a value set as a result of training, and may be updated by separate training data when necessary.

[0055] Furthermore, in this specification, a “first DNN” indicates a DNN for performing artificial intelligence (AI) downscaling on an image, and a “second DNN” indicates a DNN for performing AI upscaling on an image.

[0056] In addition, in the present specification, "DNN setting information" includes information related to elements constituting the DNN. "DNN setting information" includes the parameters described above as information related to elements constituting the DNN. The first DNN or the second DNN can be set by using the DNN setting information.

[0057] In addition, in this specification, the "original image" means an image that is a target of AI encoding, and the "first image" means an image obtained as a result of performing AI reduction on the original image during the AI ​​encoding process. In addition, the "second image" means an image obtained via the first decoding during the AI ​​decoding process, and the "third image" means an image obtained by performing AI enlargement on the second image during the AI ​​decoding process.

[0058] In addition, in this specification, "AI reduction" means a process of reducing the resolution of an image based on AI, and "first encoding" means an encoding process according to an image compression method based on frequency transformation. In addition, "first decoding" means a decoding process according to an image reconstruction method based on frequency transformation, and "AI enlargement" means a process of increasing the resolution of an image based on AI.

[0059] Figure 1 is a diagram for describing an AI encoding process and an AI decoding process according to an embodiment.

[0060] As described above, when the resolution of an image significantly increases, the throughput of information for encoding and decoding the image increases, and thus, a method for improving encoding and decoding efficiency of an image is required.

[0061] like Figure 1 As shown in , according to an embodiment of the present disclosure, a first image 115 is obtained by performing AI reduction 110 on an original image 105 having a high resolution. Then, a first encoding 120 and a first decoding 130 are performed on the first image 115 having a relatively low resolution, so that the bit rate can be greatly reduced compared to when the first encoding and the first decoding are performed on the original image 105.

[0062] Specifically, in Figure 1 According to an embodiment, during the AI ​​encoding process, a first image 115 is obtained by performing AI reduction 110 on an original image 105 and a first encoding 120 is performed on the first image 115. During the AI ​​decoding process, AI encoded data including AI data and image data obtained as a result of the AI ​​encoding is received, a second image 135 is obtained via the first decoding 130, and a third image 145 is obtained by performing AI enlargement 140 on the second image 135.

[0063] Referring to the AI ​​encoding process in detail, when the original image 105 is received, AI reduction 110 is performed on the original image 105 to obtain a first image 115 of a specific resolution or a specific quality. Here, the AI ​​reduction 110 is performed based on AI, and the AI ​​for the AI ​​reduction 110 needs to be jointly trained with the AI ​​for the AI ​​enlargement 140 of the second image 135. This is because when the AI ​​for the AI ​​reduction 110 and the AI ​​for the AI ​​enlargement 140 are trained separately, the difference between the original image 105 as the object of AI encoding and the third image 145 reconstructed by AI decoding increases.

[0064] In an embodiment of the present disclosure, AI data may be used to maintain such a joint relationship during the AI ​​encoding process and the AI ​​decoding process. Therefore, the AI ​​data obtained by the AI ​​encoding process may include information indicating a magnification target, and during the AI ​​decoding process, AI magnification 140 is performed on the second image 135 according to the magnification target verified based on the AI ​​data.

[0065] The AI ​​for AI reduction 110 and the AI ​​for AI upscaling 140 may be implemented as DNNs. Fig. 9 As described, since the first DNN and the second DNN are jointly trained by sharing loss information under a specific target, the AI ​​encoding device may provide the target information used during the joint training of the first DNN and the second DNN to the AI ​​decoding device, and the AI ​​decoding device may perform AI upscaling 140 on the second image 135 to the target resolution based on the provided target information.

[0066] about Figure 1 The first encoding 120 and the first decoding 130 can reduce the amount of information of the first image 115 obtained by performing AI reduction 110 on the original image 105 through the first encoding 120. The first encoding 120 may include a process of generating prediction data by predicting the first image 115, a process of generating residual data corresponding to the difference between the first image 115 and the prediction data, a process of transforming the residual data of the spatial domain component into a frequency domain component, a process of quantizing the residual data transformed into the frequency domain component, and a process of entropy encoding the quantized residual data. Such a first encoding 120 can be performed via one of an image compression method using frequency transformation, such as MPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, High Efficiency Video Coding (HEVC), VC-1, VP8, VP9, ​​and AOMedia Video1 (AV1).

[0067] The second image 135 corresponding to the first image 115 may be reconstructed by performing the first decoding 130 on the image data. The first decoding 130 may include a process of generating quantized residual data by entropy decoding the image data, a process of inversely quantizing the quantized residual data, a process of transforming the residual data of the frequency domain component into the spatial domain component, a process of generating prediction data, and a process of reconstructing the second image 135 by using the prediction data and the residual data. Such first decoding 130 may be performed via an image reconstruction method corresponding to one of the image compression methods using frequency transformation used in the first encoding 120, such as MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1.

[0068] The AI ​​encoded data obtained through the AI ​​encoding process may include image data obtained as a result of performing the first encoding 120 on the first image 115 and AI data related to the AI ​​reduction 110 of the original image 105. The image data may be used during the first decoding 130, and the AI ​​data may be used during the AI ​​enlargement 140.

[0069] The image data may be transmitted in the form of a bitstream. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data that is a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding 120 performed on the first image 115. For example, the image data may include prediction mode information, motion information, and information related to quantization parameters used during the first encoding 120. The image data may be generated according to the rules (for example, according to the syntax) of the image compression method used during the first encoding 120 in MPEG-2, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1.

[0070] The AI ​​data is used in the AI ​​upscaling 140 based on the second DNN. As described above, because the first DNN and the second DNN are jointly trained, the AI ​​data includes information that enables the AI ​​upscaling 140 to be accurately performed on the second image 135 by the second DNN. During the AI ​​decoding process, the AI ​​upscaling 140 may be performed on the second image 135 based on the AI ​​data to have a target resolution and / or quality.

[0071] The AI ​​data may be transmitted together with the image data in the form of a bitstream. Alternatively, according to an embodiment, the AI ​​data may be transmitted separately from the image data in the form of a frame or a packet. The AI ​​data and the image data obtained as a result of the AI ​​encoding may be transmitted over the same network or over different networks.

[0072] Figure 2is a block diagram of a configuration of the AI ​​decoding apparatus 100 according to an embodiment.

[0073] Reference Figure 2 , the AI ​​decoding device 200 according to the embodiment may include a receiver 210 and an AI decoder 230. The receiver 210 may include a communicator 212, a parser 214, and an outputter 216. The AI ​​decoder 230 may include a first decoder 232 and an AI amplifier 234.

[0074] The receiver 210 receives and parses AI-encoded data obtained as a result of AI encoding, and outputs the image data and the AI ​​data to the AI ​​decoder 230 so as to be distinguishable.

[0075] Specifically, the communicator 212 receives AI-encoded data obtained as a result of AI encoding through a network. The AI-encoded data obtained as a result of performing AI encoding includes image data and AI data. The image data and AI data may be received through the same type of network or different types of networks.

[0076] The parser 214 receives the AI ​​coded data received through the communicator 212, and parses the AI ​​coded data to distinguish the image data from the AI ​​data. For example, the parser 214 can distinguish the image data from the AI ​​data by reading the header of the data obtained from the communicator 212. According to an embodiment, the parser 214 sends the image data and the AI ​​data to the outputter 216 via the header of the data received through the communicator 212, and the outputter 216 sends the distinguished image data and AI data to the first decoder 232 and the AI ​​amplifier 234, respectively. At this time, it can be verified that the image data included in the AI ​​coded data is image data generated via a specific codec (e.g., MPEG-2, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9 or AV1). In this case, the corresponding information can be sent to the first decoder 232 by the outputter 216, so that the image data is processed by the verified codec.

[0077] According to an embodiment, the AI ​​encoded data parsed by the parser 214 may be obtained from a data storage medium, wherein the data storage medium includes a magnetic medium such as a hard disk, a floppy disk or a tape, an optical recording medium such as a CD-ROM or a DVD, or a magneto-optical medium such as a floppy disk.

[0078] The first decoder 232 reconstructs a second image 135 corresponding to the first image 115 based on the image data. The second image 135 obtained by the first decoder 232 is provided to the AI ​​amplifier 234. According to an embodiment, first decoding related information (such as prediction mode information, motion information, quantization parameter information, etc.) included in the image data may also be provided to the AI ​​amplifier 234.

[0079] Upon receiving the AI ​​data, the AI ​​amplifier 234 performs AI up-amplification on the second image 135 based on the AI ​​data. According to an embodiment, the AI ​​up-amplification may also be performed by using first decoding related information (such as prediction mode information, quantization parameter information, etc.) included in the image data.

[0080] The receiver 210 and the AI ​​decoder 230 according to the embodiment are described as separate devices, but can be implemented by one processor. In this case, the receiver 210 and the AI ​​decoder 230 can be implemented by a dedicated processor or by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). The dedicated processor can be implemented by including a memory for implementing the embodiments of the present disclosure or by including a memory processor for using an external memory.

[0081] In addition, the receiver 210 and the AI ​​decoder 230 may be configured by a plurality of processors. In this case, the receiver 210 and the AI ​​decoder 230 may be implemented by a combination of dedicated processors or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a GPU). Similarly, the AI ​​amplifier 234 and the first decoder 232 may be implemented by different processors.

[0082] The AI ​​data provided to the AI ​​amplifier 234 includes information that enables the second image 135 to be processed via AI enlargement. Here, the enlargement target should correspond to the reduction of the first DNN. Therefore, the AI ​​data includes information for verifying the reduction target of the first DNN.

[0083] Examples of the information included in the AI ​​data include difference information between the resolution of the original image 105 and the resolution of the first image 115 and information related to the first image 115 .

[0084] The difference information may be represented as information about the degree of resolution conversion of the first image 115 compared to the original image 105 (e.g., resolution conversion rate information). In addition, since the resolution of the first image 115 is verified by the resolution of the reconstructed second image 135 and thus the degree of resolution conversion is verified, the difference information may be represented as only the resolution information of the original image 105. Here, the resolution information may be represented as a vertical size / horizontal size, or a ratio (16:9, 4:3, etc.) and a size of one axis. In addition, when there is pre-set resolution information, the resolution information may be represented in the form of an index or a flag.

[0085] The information related to the first image 115 may include information about at least one of a bit rate of image data obtained as a result of performing the first encoding on the first image 115 or a codec type used during the first encoding of the first image 115 .

[0086] The AI ​​amplifier 234 may determine an enlargement target of the second image 135 based on at least one of the difference information included in the AI ​​data or the information related to the first image 115. The enlargement target may indicate, for example, to what extent the resolution is to be enlarged for the second image 135. When the enlargement target is determined, the AI ​​amplifier 234 performs AI enlargement on the second image 135 through the second DNN to obtain a third image 145 corresponding to the enlargement target.

[0087] Before describing the method of performing AI up-scaling on the second image 135 according to the up-scaling target performed by the AI ​​amplifier 234, reference will be made to Figure 3 and Figure 4 Describe the AI ​​amplification process through the second DNN.

[0088] Figure 3 is a diagram illustrating a second DNN 300 for performing AI upscaling on a second image 135, and Figure 4 Is used to describe Figure 3 Schematic diagram of the convolution operation in the first convolutional layer 310 of FIG.

[0089] like Figure 3 As shown in , the second image 135 is input to the first convolutional layer 310. Figure 3 3×3×4 indicated in the first convolution layer 310 shown in indicates that a convolution process is performed on one input image by using four filter kernels of size 3×3. Four feature maps are generated by the four filter kernels as a result of the convolution process. Each feature map indicates an inherent characteristic of the second image 135. For example, each feature map may represent a vertical direction characteristic, a horizontal direction characteristic, an edge characteristic, etc. of the second image 135.

[0090] Will refer to Figure 4 The convolution operation in the first convolutional layer 310 is described in detail.

[0091] One feature map 450 may be generated by multiplication and addition between parameters of a filter kernel 430 having a size of 3×3 used in the first convolution layer 310 and corresponding pixel values ​​in the second image 135. Since four filter kernels are used in the first convolution layer 310, four feature maps may be generated through convolution operations using the four filter kernels.

[0092] Figure 4I1 to I49 indicated in the second image 135 in FIG. 4 indicate pixels in the second image 135, and F1 to F9 indicated in the filter kernel 430 indicate parameters of the filter kernel 430. In addition, M1 to M9 indicated in the feature map 450 indicate samples of the feature map 450.

[0093] exist Figure 4 , the second image 135 includes 49 pixels, but the number of pixels is only an example, and when the second image 135 has a resolution of 4K, the second image 135 may include, for example, 3840×2160 pixels.

[0094] During the convolution operation process, the pixel values ​​of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination (e.g., addition) of the multiplication result values ​​may be assigned as the value of M1 of the feature map 450. When the stride of the convolution operation is 2, the pixel values ​​of I3, I4, I5, I10, I11, I12, I17, I18, and I19 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination of the multiplication result values ​​may be assigned as the value of M2 of the feature map 450.

[0095] When the filter kernel 430 moves to the last pixel of the second image 135 along the stride, a convolution operation is performed between the pixel values ​​in the second image 135 and the parameters of the filter kernel 430, and thus a feature map 450 having a specific size may be generated.

[0096] According to the present disclosure, the values ​​of the parameters of the second DNN (e.g., the values ​​of the parameters of the filter kernel used in the convolution layer of the second DNN (e.g., the values ​​of F1 to F9 of the filter kernel 430)) can be optimized by joint training of the first DNN and the second DNN. As described above, the AI ​​amplifier 234 can determine the enlargement target corresponding to the reduction target of the first DNN based on the AI ​​data, and determine the parameters corresponding to the determined enlargement target as the parameters of the filter kernel used in the convolution layer of the second DNN.

[0097] The convolutional layers included in the first DNN and the second DNN can be based on the reference Figure 4 The convolution operation described above is used to perform the processing, but referring to Figure 4 The described convolution operation processing is only an example and is not limited thereto.

[0098] Return to reference Figure 3 , the feature map output from the first convolution layer 310 may be input to the first activation layer 320 .

[0099] The first activation layer 320 may impart nonlinear features to each feature map. The first activation layer 320 may include a sigmoid function, a Tanh function, a rectified linear unit (ReLU) function, etc., but is not limited thereto.

[0100] The first activation layer 320 imparting the nonlinear feature indicates that at least one sample value of the feature map as the output of the first convolution layer 310 is changed. Here, the change is performed by applying the nonlinear feature.

[0101] The first activation layer 320 determines whether to send the sample values ​​of the feature map output from the first convolution layer 310 to the second convolution layer 330. For example, some sample values ​​of the feature map are activated by the first activation layer 320 and sent to the second convolution layer 330, and some sample values ​​are deactivated by the first activation layer 320 and are not sent to the second convolution layer 330. The inherent characteristics of the second image 135 represented by the feature map are emphasized by the first activation layer 320.

[0102] The feature map 325 output from the first activation layer 320 is input to the second convolutional layer 330 . Figure 3 One of the feature maps 325 shown in FIG. 3 is a feature map of the reference in the first activation layer 320. Figure 4 The result of processing the feature map 450 is described.

[0103] 3×3×4 indicated in the second convolution layer 330 indicates that a convolution process is performed on the feature map 325 by using four filters of size 3×3. The output of the second convolution layer 330 is input to the second activation layer 340. The second activation layer 340 can impart nonlinear features to input data.

[0104] The feature map 345 output from the second activation layer 340 is input to the third convolutional layer 350 . Figure 3 3×3×1 indicated in the third convolution layer 350 shown in indicates that a convolution process is performed by using one filter kernel of size 3×3 to generate one output image. The third convolution layer 350 is a layer for outputting a final image, and generates one output by using one filter kernel. According to an embodiment of the present disclosure, the third convolution layer 350 may output the third image 145 as a result of the convolution operation.

[0105] As will be described later, there may be a plurality of pieces of DNN setting information indicating the number of filter kernels of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, parameters of the filter kernels of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, and the like, and the plurality of pieces of DNN setting information should be associated with the plurality of pieces of DNN setting information of the first DNN. The association between the plurality of pieces of DNN setting information of the second DNN and the plurality of DNN setting information of the first DNN may be achieved via joint training of the first DNN and the second DNN.

[0106] exist Figure 3 , the second DNN 300 includes three convolutional layers (a first convolutional layer 310, a second convolutional layer 330, and a third convolutional layer 350) and two activation layers (a first activation layer 320 and a second activation layer 340), but this is only an example, and the number of convolutional layers and activation layers may vary according to the embodiment. In addition, according to the embodiment, the second DNN 300 may be implemented as a recurrent neural network (RNN). In this case, the convolutional neural network (CNN) structure of the second DNN 300 according to the embodiment of the present disclosure is changed to an RNN structure.

[0107] According to an embodiment, the AI ​​amplifier 234 may include at least one arithmetic logic unit (ALU) for the above-mentioned convolution operation and the operation of the activation layer. The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier and an adder, wherein the multiplier performs multiplication between the sample value of the second image 135 or the feature map output from the previous layer and the sample value of the filter kernel, and the adder adds the result value of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0108] Hereinafter, a method of performing AI up-scaling on the second image 135 according to the up-scaling target, performed by the AI ​​amplifier 234, will be described.

[0109] According to an embodiment, the AI ​​amplifier 234 may store a plurality of pieces of DNN setting information that may be set in the second DNN.

[0110] Here, the DNN setting information may include information about at least one of the number of convolutional layers included in the second DNN, the number of filter kernels for each convolutional layer, or the parameters of each filter kernel. Multiple pieces of DNN setting information may correspond to various amplification targets, respectively, and the second DNN may operate based on the DNN setting information corresponding to a specific amplification target. The second DNN may have different structures based on the DNN setting information. For example, the second DNN may include three convolutional layers based on any one piece of DNN setting information, and may include four convolutional layers based on another piece of DNN setting information.

[0111] According to an embodiment, the DNN setting information may include only the parameters of the filter kernel used in the second DNN. In this case, the structure of the second DNN does not change, but only the parameters of the internal filter kernel may be changed based on the DNN setting information.

[0112] The AI ​​amplifier 234 may obtain DNN setting information from among the plurality of DNN setting information for performing AI amplification on the second image 135. Each of the plurality of DNN setting information used at this time is information for obtaining the third image 145 of a predetermined resolution and / or a predetermined quality, and is jointly trained with the first DNN.

[0113] For example, one piece of DNN setting information among the multiple DNN setting information may include information for obtaining a third image 145 having a resolution that is twice as high as the resolution of the second image 135 (for example, a third image 145 of 4K (4096×2160) that is twice as high as the 2K (2048×1080) of the second image 135), and another piece of DNN setting information may include information for obtaining a third image 145 having a resolution that is four times as high as the resolution of the second image 135 (for example, a third image 145 of 8K (8192×4320) that is four times as high as the 2K (2048×1080) of the second image 135).

[0114] Each of the plurality of pieces of DNN setting information is Figure 6 The AI ​​encoding device 600 of the embodiment of the present invention is jointly obtained with the DNN setting information of the first DNN, and the AI ​​amplifier 234 obtains one piece of the DNN setting information according to the enlargement ratio corresponding to the reduction ratio of the DNN setting information of the first DNN. In this regard, the AI ​​amplifier 234 can verify the information of the first DNN. In order for the AI ​​amplifier 234 to verify the information of the first DNN, the AI ​​decoding device 200 according to the embodiment receives AI data including the information of the first DNN from the AI ​​encoding device 600.

[0115] In other words, the AI ​​amplifier 234 can verify the information targeted by the DNN setting information of the first DNN used to obtain the first image 115 by using the information received from the AI ​​encoding device 600, and obtain the DNN setting information of the second DNN jointly trained with the DNN setting information of the first DNN.

[0116] When DNN setting information for performing AI up-scaling on the second image 135 is obtained from a plurality of pieces of DNN setting information, input data may be processed based on the second DNN operating according to the obtained DNN setting information.

[0117] For example, when any DNN setting information is obtained, Figure 3 The number of filter kernels included in each of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300 and the parameters of the filter kernels are set to the values ​​included in the obtained DNN setting information.

[0118] Specifically, in Figure 4 The parameters of the 3×3 filter kernel used in any convolutional layer of the second DNN are set to {1,1,1,1,1,1,1,1,1}, and when the DNN setting information is subsequently changed, the parameters are replaced by {2,2,2,2,2,2,2,2,2} as parameters included in the changed DNN setting information.

[0119] The AI ​​amplifier 234 may obtain DNN setting information for AI amplification from a plurality of pieces of DNN setting information based on information included in the AI ​​data, and the AI ​​data for obtaining the DNN setting information will now be described.

[0120] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification from a plurality of pieces of DNN setting information based on the difference information included in the AI ​​data. For example, when it is verified based on the difference information that the resolution of the original image 105 (e.g., 4K (4096×2160)) is twice as high as the resolution of the first image 115 (e.g., 2K (2048×1080)), the AI ​​amplifier 234 may obtain DNN setting information for increasing the resolution of the second image 135 by two times.

[0121] According to another embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification of the second image 135 from a plurality of pieces of DNN setting information based on information related to the first image 115 included in the AI ​​data. The AI ​​amplifier 234 may predetermine a mapping relationship between image-related information and DNN setting information, and obtain DNN setting information mapped to information related to the first image 115.

[0122] Figure 5 is a table showing the mapping relationship between several pieces of image-related information and several pieces of DNN setting information.

[0123] By following Figure 5 In the embodiment of the present disclosure, it will be determined that the AI ​​encoding and AI decoding processing according to the embodiment of the present disclosure does not only consider the change of resolution. Figure 5 As shown in , the DNN setting information can be selected by considering resolution (such as standard definition (SD), high definition (HD), or full HD), bit rate (such as 10Mbps, 15Mbps, or 20Mbps), and codec information (such as AV1, H.264, or HEVC) alone or together. Due to such consideration of resolution, bit rate, and codec information, it is considered that the training of each element should be performed jointly with the encoding and decoding processes during the AI ​​training process (see Fig. 9 ).

[0124] Therefore, when training is based on including Figure 5 When multiple DNN setting information is provided by the image-related information such as the codec type, image resolution, etc. shown in , the DNN setting information for AI upscaling the second image 135 can be obtained based on the information related to the first image 115 received during the AI ​​decoding process.

[0125] In other words, AI amplifier 234 can Figure 5 The image related information on the left side of the table and the DNN setting information on the right side of the table are matched to use the DNN setting information according to the image related information.

[0126] like Figure 5 As shown in , when it is verified from information related to the first image 115 that the resolution of the first image 115 is SD, the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 10Mbps, and the first encoding is performed on the first image 115 via the AV1 codec, the AI ​​amplifier 234 may use the ADNN setting information among multiple DNN setting information.

[0127] In addition, when it is verified from information related to the first image 115 that the resolution of the first image 115 is HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the H.264 codec, the AI ​​amplifier 234 can use B DNN setting information among multiple DNN setting information.

[0128] In addition, when it is verified from the information related to the first image 115 that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 20Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 may use the C DNN setting information among the plurality of DNN setting information, and when it is verified that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 may use the D DNN setting information among the plurality of DNN setting information. One of the C DNN setting information and the DDNN setting information is selected based on whether the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 20Mbps or 15Mbps. Different bit rates of image data obtained when performing the first encoding on the first image 115 of the same resolution via the same codec indicate different qualities of the reconstructed image. Therefore, the first DNN and the second DNN may be jointly trained based on a specific image quality, and therefore, the AI ​​amplifier 234 may obtain the DNN setting information according to the bit rate of the image data indicating the quality of the second image 135.

[0129] According to another embodiment, the AI ​​amplifier 234 may obtain DNN setting information for performing AI amplification on the second image 135 from a plurality of pieces of DNN setting information, taking into account both information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and information related to the first image 115 included in the AI ​​data. For example, the AI ​​amplifier 234 may receive quantization parameter information used during the first encoding process of the first image 115 from the first decoder 232, verify the bit rate of image data obtained as a result of encoding the first image 115 from the AI ​​data, and obtain DNN setting information corresponding to the quantization parameter information and the bit rate. Even when the bit rate is the same, the quality of the reconstructed image may vary according to the complexity of the image. The bit rate is a value representing the entire first image 115 on which the first encoding is performed, and the quality of each frame may vary even within the first image 115. Therefore, when the prediction mode information, motion information, and / or quantization parameter obtainable for each frame from the first decoder 232 are considered together, DNN setting information more suitable for the second image 135 may be obtained compared to when only the AI ​​data is used.

[0130] In addition, according to an embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information. The identifier of the DNN setting information is information for distinguishing a pair of DNN setting information jointly trained between the first DNN and the second DNN, so that AI amplification is performed on the second image 135 to an amplification target corresponding to the reduction target of the first DNN. After obtaining the identifier of the DNN setting information included in the AI ​​data, the AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information corresponding to the identifier of the DNN setting information. For example, an identifier indicating each of the multiple DNN setting information that can be set in the first DNN and an identifier indicating each of the multiple DNN setting information that can be set in the second DNN may be pre-specified. In this case, the same identifier may be specified for a pair of DNN setting information that can be set in each of the first DNN and the second DNN. The AI ​​data may include an identifier of the DNN setting information set in the first DNN for AI reduction of the original image 105. The AI ​​amplifier 234 receiving the AI ​​data may perform AI up-amplification on the second image 135 by using DNN setting information indicated by an identifier included in the AI ​​data among the plurality of pieces of DNN setting information.

[0131] In addition, according to an embodiment, the AI ​​data may include DNN setting information. The AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information included in the AI ​​data after obtaining the DNN setting information.

[0132] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter cores for each convolutional layer, parameters of each filter core, etc.) are stored in the form of a lookup table, the AI ​​amplifier 234 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on the information included in the AI ​​data, and perform AI amplification on the second image 135 by using the obtained DNN setting information.

[0133] According to an embodiment, when the structure of the DNN corresponding to the amplification target is determined, the AI ​​amplifier 234 may obtain DNN setting information corresponding to the determined structure of the DNN, for example, parameters of a filter kernel.

[0134] The AI ​​amplifier 234 obtains DNN setting information of the second DNN through AI data including information related to the first DNN, and performs AI amplification on the second image 135 through the second DNN set based on the obtained DNN setting information, and in this case, memory usage and throughput can be reduced compared to directly analyzing features of the second image 135 for amplification.

[0135] According to an embodiment, when the second image 135 includes a plurality of frames, the AI ​​amplifier 234 may independently obtain DNN setting information for a specific number of frames, or may obtain common DNN setting information for all frames.

[0136] Figure 6 is a diagram illustrating a second image 135 including a plurality of frames.

[0137] like Figure 6 As shown in , the second image 135 may include frames t0 to tn.

[0138] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information of the second DNN through the AI ​​data, and perform AI amplification on frames t0 to tn based on the obtained DNN setting information. In other words, frames t0 to tn may be processed via AI amplification based on the common DNN setting information.

[0139] According to another embodiment, the AI ​​amplifier 234 may perform AI amplification on some frames (e.g., frame t0 to frame ta) among frame t0 to frame tn by using the "A" DNN setting information obtained from the AI ​​data, and perform AI amplification on frame ta+1 to frame tb by using the "B" DNN setting information obtained from the AI ​​data. In addition, the AI ​​amplifier 234 may perform AI amplification on frame tb+1 to frame tn by using the "C" DNN setting information obtained from the AI ​​data. In other words, the AI ​​amplifier 234 may independently obtain DNN setting information for each group including a specific number of frames among a plurality of frames, and perform AI amplification on the frames included in each group by using the independently obtained DNN setting information.

[0140] According to another embodiment, the AI ​​amplifier 234 may independently obtain DNN setting information for each frame forming the second image 135. In other words, when the second image 135 includes three frames, the AI ​​amplifier 234 may perform AI amplification on the first frame by using the DNN setting information obtained about the first frame, perform AI amplification on the second frame by using the DNN setting information obtained about the second frame, and perform AI amplification on the third frame by using the DNN setting information obtained about the third frame. According to the method of obtaining DNN setting information based on the information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and the information related to the first image 115 included in the AI ​​data as described above, DNN setting information may be independently obtained for each frame included in the second image 135. This is because the mode information, quantization parameter information, etc. may be independently determined for each frame included in the second image 135.

[0141] According to another embodiment, the AI ​​data may include information about which frame the DNN setting information obtained based on the AI ​​data is valid for. For example, when the AI ​​data includes information indicating that the DNN setting information is valid until frame ta, the AI ​​amplifier 234 performs AI amplification on frame t0 to frame ta by using the DNN setting information obtained based on the AI ​​data. In addition, when another piece of AI data includes information indicating that the DNN setting information is valid until frame tn, the AI ​​amplifier 234 performs AI amplification on frame ta+1 to frame tn by using the DNN setting information obtained based on the other piece of AI data.

[0142] In the following, reference will be made to Figure 7 An AI encoding apparatus 600 for performing AI encoding on an original image 105 is described.

[0143] Figure 7 is a block diagram of a configuration of an AI encoding apparatus 600 according to an embodiment.

[0144] Reference Figure 7 , the AI ​​encoding device 600 may include an AI encoder 610 and a transmitter 630. The AI ​​encoder 610 may include an AI reducer 66 and a first encoder 614. The transmitter 630 may include a data processor 632 and a communicator 634.

[0145] exist Figure 7 , the AI ​​encoder 610 and the transmitter 630 are shown as independent devices, but the AI ​​encoder 610 and the transmitter 630 may be implemented by one processor. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a graphics processing unit GPU). The dedicated processor may be implemented by including a memory for implementing an embodiment of the present disclosure or by including a memory processor for using an external memory.

[0146] In addition, the AI ​​encoder 610 and the transmitter 630 may be composed of multiple processors. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a combination of dedicated processors or by a combination of software and multiple general-purpose processors (such as AP, CPU or GPU). The AI ​​reducer 612 and the first encoder 614 may be implemented by different processors.

[0147] The AI ​​encoder 610 performs AI reduction on the original image 105 and first encoding on the first image 115, and transmits the AI ​​data and the image data to the transmitter 630. The transmitter 630 transmits the AI ​​data and the image data to the AI ​​decoding apparatus 200.

[0148] The image data includes data obtained as a result of performing the first encoding on the first image 115. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data that is a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding process of the first image 115. For example, the image data may include prediction mode information, motion information, quantization parameter information, etc. used to perform the first encoding on the first image 115.

[0149] The AI ​​data includes information that enables AI up-scaling to be performed on the second image 135 to an up-scaling target corresponding to the down-scaling target of the first DNN. According to an embodiment, the AI ​​data may include difference information between the original image 105 and the first image 115. In addition, the AI ​​data may include information related to the first image 115. The information related to the first image 115 may include information about at least one of a resolution of the first image 115, a bit rate of image data obtained as a result of performing the first encoding on the first image 115, and a codec type used during the first encoding of the first image 115.

[0150] According to an embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information such that AI enlargement is performed on the second image 135 to an enlarged target corresponding to a reduced target of the first DNN.

[0151] In addition, according to an embodiment, the AI ​​data may include DNN setting information that can be set in the second DNN.

[0152] The AI ​​downsizer 612 may obtain the first image 115 obtained by performing AI downsizing on the original image 105 through the first DNN. The AI ​​downsizer 612 may determine a downsizing target of the original image 105 based on a predetermined criterion.

[0153] In order to obtain the first image 115 matching the reduction target, the AI ​​reducer 612 may store a plurality of DNN setting information that may be set in the first DNN. The AI ​​reducer 612 obtains DNN setting information corresponding to the reduction target from the plurality of DNN setting information, and performs AI reduction on the original image 105 through the first DNN set in the obtained DNN setting information.

[0154] Each of the plurality of pieces of DNN setting information may be trained to obtain a first image 115 of a predetermined resolution and / or a predetermined quality. For example, any one of the plurality of pieces of DNN setting information may include information for obtaining a first image 115 having a resolution half that of the original image 105 (e.g., a first image 115 of 2K (2048×1080) which is half that of 4K (4096×2160) of the original image 105), and another piece of DNN setting information may include information for obtaining a first image 115 having a resolution one-quarter that of the original image 105 (e.g., a first image 115 of 2K (2048×1080) which is one-quarter that of 8K (8192×4320) of the original image 105).

[0155] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter kernels for each convolutional layer, parameters of each filter kernel, etc.) are stored in the form of a lookup table, the AI ​​downsizer 612 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on the reduction target, and perform AI downsizing on the original image 105 by using the obtained DNN setting information.

[0156] According to an embodiment, the AI ​​reducer 612 may determine the structure of the DNN corresponding to the reduction target, and obtain DNN setting information corresponding to the determined structure of the DNN, for example, obtain parameters of a filter kernel.

[0157] As the first DNN and the second DNN are jointly trained, a plurality of pieces of DNN setting information for performing AI reduction on the original image 105 may have an optimized value. Here, each piece of DNN setting information includes at least one of the number of convolutional layers included in the first DNN, the number of filter kernels for each convolutional layer, or a parameter of each filter kernel.

[0158] The AI ​​downsizer 612 may set the first DNN using the DNN setting information obtained for performing AI downsizing on the original image 105 to obtain the first image 115 of a specific resolution and / or a specific quality through the first DNN. When the DNN setting information for performing AI downsizing on the original image 105 is obtained from a plurality of pieces of DNN setting information, each layer in the first DNN may process input data based on information included in the DNN setting information.

[0159] Hereinafter, a method of determining a downscaling target performed by the AI ​​downscaler 612 will be described. The downscaling target may indicate, for example, how much the resolution is reduced from the original image 105 to obtain the first image 115.

[0160] According to an embodiment, the AI ​​reducer 612 may determine the reduction target based on at least one of the compression ratio of the original image 105 (e.g., the resolution difference between the original image 105 and the first image 115, the target bit rate, etc.), the compression quality (e.g., the type of bit rate), the compression history information or the type.

[0161] For example, the AI ​​downsizer 612 may determine a downsizing target based on a compression ratio, compression quality, etc. that is preset or input from a user.

[0162] As another example, the AI ​​downsizer 612 may determine the downsizing target by using the compression history information stored in the AI ​​encoding device 600. For example, according to the compression history information that may be used by the AI ​​encoding device 600, the encoding quality, compression ratio, etc. preferred by the user may be determined, and the downsizing target may be determined according to the encoding quality determined based on the compression history information. For example, the resolution, quality, etc. of the first image 115 may be determined based on the most frequently used encoding quality according to the compression history information.

[0163] As another example, the AI ​​downsizer 612 may determine the downsizing target based on encoding qualities used more frequently than a specific threshold (eg, an average quality of encoding qualities used more frequently than a specific threshold) according to the compression history information.

[0164] As another example, the AI ​​downsizer 612 may determine the downscaling target based on the resolution, type (eg, file format), etc. of the original image 105 .

[0165] According to an embodiment, when the original image 105 includes a plurality of frames, the AI ​​downsizer 612 may independently determine downsizing targets for a specific number of frames, or may determine downsizing targets for all frames.

[0166] According to an embodiment, the AI ​​downsizer 612 may divide the frames included in the original image 105 into a specific number of groups and independently determine a downsizing target for each group. The same or different downsizing targets may be determined for each group. According to each group, the number of frames included in the group may be the same or different.

[0167] According to another embodiment, the AI ​​downsizer 612 may independently determine a downscaling target for each frame included in the original image 105. The same or different downscaling targets may be determined for each frame.

[0168] Hereinafter, an example of the structure of the first DNN 700 on which the AI ​​reduction is based will be described.

[0169] Figure 8 is a diagram illustrating a first DNN 700 for performing AI downscaling on an original image 105.

[0170] like Figure 8As shown in FIG. 1 , the original image 105 is input to the first convolution layer 710. The first convolution layer 710 performs a convolution process on the original image 105 by using 32 filter kernels of a size of 5×5. The 32 feature maps generated as a result of the convolution process are input to the first activation layer 720. The first activation layer 720 may impart nonlinear features to the 32 feature maps.

[0171] The first activation layer 720 determines whether to send sample values ​​of the feature map output from the first convolution layer 710 to the second convolution layer 730. For example, some sample values ​​of the feature map are activated by the first activation layer 720 and sent to the second convolution layer 730, and some sample values ​​are deactivated by the first activation layer 720 and are not sent to the second convolution layer 730. Information represented by the feature map output from the first convolution layer 710 is emphasized by the first activation layer 720.

[0172] The output 725 of the first activation layer 720 is input to the second convolution layer 730. The second convolution layer 730 performs a convolution process on the input data by using 32 filter kernels of a size of 5×5. The 32 feature maps output as a result of the convolution process are input to the second activation layer 740, and the second activation layer 740 can give nonlinear features to the 32 feature maps.

[0173] The output 745 of the second activation layer 740 is input to the third convolution layer 750. The third convolution layer 750 performs a convolution process on the input data by using a filter kernel of size 5×5. As a result of the convolution process, an image may be output from the third convolution layer 750. The third convolution layer 750 generates an output by using a filter kernel as a layer for outputting a final image. According to an embodiment of the present disclosure, the third convolution layer 750 may output the first image 115 as a result of the convolution operation.

[0174] There may be a plurality of pieces of DNN setting information indicating the number of filter cores of the first convolution layer 710, the second convolution layer 730, and the third convolution layer 750 of the first DNN 700, a parameter of each filter core of the first convolution layer 710, the second convolution layer 730, and the third convolution layer 750 of the first DNN 700, and the like, and the plurality of pieces of DNN setting information may be associated with the plurality of pieces of DNN setting information of the second DNN. The association between the plurality of pieces of DNN setting information of the first DNN and the plurality of pieces of DNN setting information of the second DNN may be achieved through joint training of the first DNN and the second DNN.

[0175] exist Figure 8, the first DNN 700 includes three convolutional layers (a first convolutional layer 710, a second convolutional layer 730, and a third convolutional layer 750) and two activation layers (a first activation layer 720 and a second activation layer 740), but this is only an example, and the number of convolutional layers and activation layers may vary according to the embodiment. In addition, according to the embodiment, the first DNN 700 may be implemented as an RNN. In this case, the CNN structure of the first DNN 700 according to the embodiment of the present disclosure is changed to an RNN structure.

[0176] According to an embodiment, the AI ​​reducer 612 may include at least one ALU for the above-mentioned convolution operation and the operation of the activation layer. The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier and an adder, wherein the multiplier performs multiplication between the sample value of the original image 105 or the feature map output from the previous layer and the sample value of the filter kernel, and the adder adds the result value of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0177] Return to reference Figure 7 Upon receiving the first image 115 from the AI ​​downsizer 612, the first encoder 614 may reduce the amount of information of the first image 115 by performing the first encoding on the first image 115. Image data corresponding to the first image 115 may be obtained as a result of the first encoding performed by the first encoder 614.

[0178] The data processor 632 processes at least one of the AI ​​data or the image data to be sent in a specific form. For example, when the AI ​​data and the image data are to be sent in the form of a bit stream, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and send the image data and the AI ​​data in the form of one bit stream through the communicator 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and send each of the bit streams corresponding to the AI ​​data and the bit stream corresponding to the image data through the communicator 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a frame or a packet, and send the image data in the form of a bit stream and send the AI ​​data in the form of a frame or a packet through the communicator 634.

[0179] The communicator 634 transmits the AI-encoded data obtained as a result of performing the AI ​​encoding through the network. The AI-encoded data obtained as a result of performing the AI ​​encoding includes image data and AI data. The image data and the AI ​​data may be transmitted through the same type of network or different types of networks.

[0180] According to an embodiment, the AI ​​encoded data obtained as a result of processing by the data processor 632 may be stored in a data storage medium, wherein the data storage medium includes a magnetic medium such as a hard disk, a floppy disk, or a tape, an optical recording medium such as a CD-ROM or a DVD, or a magneto-optical medium such as a floppy disk.

[0181] In the following, reference will be made to Fig. 9 A method of jointly training the first DNN 700 and the second DNN 300 is described.

[0182] Fig. 9 is a diagram for describing a method of training a first DNN 700 and a second DNN 300.

[0183] In an embodiment, the original image 105 that is AI-encoded by the AI ​​encoding process is reconstructed into the third image 145 via the AI ​​decoding process, and in order to maintain the similarity between the original image 105 and the third image 145 obtained as a result of AI decoding, the association between the AI ​​encoding process and the AI ​​decoding process is required. In other words, it is necessary to reconstruct the information lost in the AI ​​encoding process during the AI ​​decoding process, and in this regard, it is necessary to jointly train the first DNN 700 and the second DNN 300.

[0184] In order to perform accurate AI decoding, it is ultimately necessary to reduce Fig. 9 The quality loss information 830 corresponding to the result of comparing the third training image 804 and the original training image 801 shown in FIG. Therefore, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0185] First, we will describe Fig. 9 The training process shown in .

[0186] exist Fig. 9 , the original training image 801 is an image to be subjected to AI reduction, and the first training image 802 is an image obtained by performing AI reduction on the original training image 801. In addition, the third training image 804 is an image obtained by performing AI enlargement on the first training image 802.

[0187] The original training image 801 includes a still image or a moving image including a plurality of frames. According to an embodiment, the original training image 801 may include a luminance image extracted from a still image or a moving image including a plurality of frames. In addition, according to an embodiment, the original training image 801 may include a block image extracted from a still image or a moving image including a plurality of frames. When the original training image 801 includes a plurality of frames, the first training image 802, the second training image, and the third training image 804 also each include a plurality of frames. When the plurality of frames of the original training image 801 are sequentially input to the first DNN 700, the plurality of frames of the first training image 802, the second training image, and the third training image 804 may be sequentially obtained by the first DNN 700 and the second DNN 300.

[0188] For joint training of the first DNN 700 and the second DNN 300, an original training image 801 is input to the first DNN 700. The original training image 801 input to the first DNN 700 is output as a first training image 802 via AI downscaling, and the first training image 802 is input to the second DNN 300. A third training image 804 is output as a result of performing AI upscaling on the first training image 802.

[0189] Reference Fig. 9 , the first training image 802 is input to the second DNN 850, and according to an embodiment, the second training image obtained when the first encoding and the first decoding are performed on the first training image 802 may be input to the second DNN 300. In order to input the second training image to the second DNN 300, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 codecs may be used. Specifically, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 codecs may be used to perform the first encoding on the first training image 802 and perform the first decoding on the image data corresponding to the first training image 802.

[0190] Reference Fig. 9 , separately from the first training image 802 output by the first DNN 700, a reduced training image 803 obtained by performing conventional downscaling on the original training image 801 is obtained. Here, the conventional downscaling may include at least one of bilinear scaling, bicubic scaling, lanczos scaling, or step scaling.

[0191] In order to prevent the structural features of the first image 115 from greatly deviating from the structural features of the original image 105 , a reduced training image 803 is obtained to preserve the structural features of the original training image 801 .

[0192] Before performing training, the first DNN 700 and the second DNN 300 may be set to predetermined DNN setting information. When performing training, structure loss information 810, complexity loss information 820, and quality loss information 830 may be determined.

[0193] The structural loss information 810 may be determined based on a result of comparing the reduced training image 803 and the first training image 802. For example, the structural loss information 810 may correspond to a difference between the structural information of the reduced training image 803 and the structural information of the first training image 802. The structural information may include various features that may be extracted from an image, such as brightness, contrast, histogram, etc. of the image. The structural loss information 810 indicates how much structural information of the original training image 801 is maintained in the first training image 802. When the structural loss information 810 is small, the structural information of the first training image 802 is similar to the structural information of the original training image 801.

[0194] The complexity loss information 820 may be determined based on the spatial complexity of the first training image 802. For example, the total variance value of the first training image 802 may be used as the spatial complexity. The complexity loss information 820 is related to the bit rate of the image data obtained by performing the first encoding on the first training image 802. When the complexity loss information 820 is small, the bit rate of the image data is low.

[0195] The quality loss information 830 may be determined based on a result of comparing the original training image 801 and the third training image 804. The quality loss information 830 may include at least one of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, or a video multi-method assessment fusion (VMAF) value about the difference between the original training image 801 and the third training image 804. The quality loss information 830 indicates how similar the third training image 804 is to the original training image 801. When the quality loss information 830 is small, the third training image 804 is more similar to the original training image 801.

[0196] Reference Fig. 9 , the structure loss information 810, the complexity loss information 820, and the quality loss information 830 are used to train the first DNN 700, and the quality loss information 830 is used to train the second DNN 300. In other words, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0197] The first DNN 700 may update parameters so that the final loss information determined based on the loss information 810 to 830 is reduced or minimized. In addition, the second DNN 300 may update parameters so that the quality loss information 830 is reduced or minimized.

[0198] Final loss information for training the first DNN 700 and the second DNN 300 may be determined as in Equation 1 below.

[0199] [Equation 1]

[0200] LossDS = a×structural loss information + b×complexity loss information + c×quality loss information

[0201] LossUS=d×quality loss information

[0202] In Equation 1, LossDS indicates final loss information to be reduced or minimized to train the first DNN 700, and LossUS indicates final loss information to be reduced or minimized to train the second DNN 300. In addition, a, b, c, and d may be predetermined specific weights.

[0203] In other words, the first DNN 700 updates the parameters in the direction in which the LossDS of Equation 1 decreases, and the second DNN 300 updates the parameters in the direction in which the LossDS decreases. When the parameters of the first DNN 700 are updated according to the LossDS derived during training, the first training image 802 obtained based on the updated parameters becomes different from the previous first training image 802 obtained based on the non-updated parameters, and therefore, the third training image 804 also becomes different from the previous third training image 804. When the third training image 804 becomes different from the previous third training image 804, the quality loss information 830 is also re-determined, and the second DNN 300 updates the parameters accordingly. When the quality loss information 830 is re-determined, the LossDS is also re-determined, and the first DNN 700 updates the parameters according to the re-determined LossDS. In other words, the update of the parameters of the first DNN 700 leads to the update of the parameters of the second DNN 300, and the update of the parameters of the second DNN 300 leads to the update of the parameters of the first DNN 700. In other words, because the first DNN 700 and the second DNN 300 are jointly trained by sharing the quality loss information 830, the parameters of the first DNN 700 and the parameters of the second DNN 300 can be jointly optimized.

[0204] Referring to Equation 1, it is verified that LossUS is determined according to the quality loss information 830 , but this is only an example, and LossUS may be determined based on at least one of the structure loss information 810 and the complexity loss information 820 , as well as the quality loss information 830 .

[0205] In the above, it has been described that the AI ​​amplifier 234 of the AI ​​decoding device 200 and the AI ​​reducer 612 of the AI ​​encoding device 600 store multiple DNN setting information, and now a method of training each of the multiple DNN setting information stored in the AI ​​amplifier 234 and the AI ​​reducer 612 will be described.

[0206] As described with reference to Equation 1, the first DNN 700 updates parameters by considering the similarity between the structural information of the first training image 802 and the structural information of the original training image 801 (structural loss information 810), the bit rate of the image data obtained as a result of performing the first encoding on the first training image 802 (complexity loss information 820), and the difference between the third training image 804 and the original training image 801 (quality loss information 830).

[0207] Specifically, the parameters of the first DNN 700 can be updated to obtain a first training image 802 having similar structural information to the original training image 801 and to obtain image data with a small bit rate when the first encoding is performed on the first training image 802, and at the same time, the second DNN 300 that performs AI amplification on the first training image 802 obtains a third training image 804 similar to the original training image 801.

[0208] The direction in which the parameters of the first DNN 700 are optimized may be changed by adjusting the weights a, b, and c of Equation 1. For example, when the weight b is determined to be high, the parameters of the first DNN 700 may be updated by prioritizing the low bit rate over the high quality of the third training image 804. In addition, when the weight c is determined to be high, the parameters of the first DNN 700 may be updated by prioritizing the high quality of the third training image 804 over the high bit rate or maintaining the structural information of the original training image 801.

[0209] In addition, the direction in which the parameters of the first DNN 700 are optimized may vary according to the type of codec used to perform the first encoding on the first training image 802. This is because the second training image to be input to the second DNN 300 may vary according to the type of codec.

[0210] In other words, the parameters of the first DNN 700 and the parameters of the second DNN 300 may be jointly updated based on the weights a, b, and c and the type of codec used to perform the first encoding on the first training image 802. Therefore, when the first DNN 700 and the second DNN 300 are trained after the weights a, b, and c are each determined to be a specific value and the type of codec is determined to be a specific type, the parameters of the first DNN 700 and the parameters of the second DNN 300 that are associated with and optimized to each other may be determined.

[0211] In addition, when the first DNN 700 and the second DNN 300 are trained after changing the weights a, b, and c and the type of codec, the parameters of the first DNN 700 and the parameters of the second DNN 300 that are associated and optimized with each other may be determined. In other words, when the first DNN 700 and the second DNN 300 are trained while changing the values ​​of the weights a, b, and c and the type of codec, a plurality of pieces of DNN setting information that are jointly trained with each other may be determined in the first DNN 700 and the second DNN 300.

[0212] As mentioned above Figure 5 As described, a plurality of pieces of DNN setting information of the first DNN 700 and the second DNN 300 may be mapped to information related to the first image. In order to set such a mapping relationship, the first training image 802 output from the first DNN 700 may be first encoded via a specific codec according to a specific bit rate, and the second training image obtained by performing a first decoding on a bit stream obtained as a result of performing the first encoding may be input to the second DNN 300. In other words, by training the first DNN 700 and the second DNN 300 after setting an environment such that the first encoding is performed on the first training image 802 of a specific resolution via a specific codec according to a specific bit rate, a pair of DNN setting information mapped to the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained as a result of performing the first encoding on the first training image 802 may be determined. By differently changing the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained according to the first encoding of the first training image 802, the mapping relationship between multiple DNN setting information of the first DNN 700 and the second DNN 300 and multiple information related to the first image can be determined.

[0213] Fig.10 is a diagram for describing the training process of the training device 1000 on the first DNN 700 and the second DNN.

[0214] Reference Fig. 9The training of the described first DNN 700 and the second DNN 300 may be performed by the training device 1000. The training device 1000 includes the first DNN 700 and the second DNN 300. The training device 1000 may be, for example, the AI ​​encoding device 600 or a separate server. The DNN setting information of the second DNN 300 obtained as a result of the training is stored in the AI ​​decoding device 200.

[0215] Reference Fig.10 In operation S840 and operation S845, the training device 1000 initially sets the DNN setting information of the first DNN 700 and the second DNN 300. Therefore, the first DNN 700 and the second DNN 300 may operate according to the predetermined DNN setting information. The DNN setting information may include information about at least one of the number of convolutional layers included in the first DNN 700 and the second DNN 300, the number of filter kernels for each convolutional layer, the size of the filter kernel for each convolutional layer, or the parameters of each filter kernel.

[0216] In operation S850, the training apparatus 1000 inputs the original training image 801 into the first DNN 700. The original training image 801 may include at least one frame included in a still image or a moving image.

[0217] In operation S855, the first DNN 700 processes the original training image 801 according to the initially set DNN setting information, and outputs the first training image 802 obtained by performing AI reduction on the original training image 801. Fig.10 , the first training image 802 output from the first DNN 700 is directly input to the second DNN 300, but the first training image 802 output from the first DNN 700 may be input to the second DNN 300 by the training device 1000. In addition, the training device 1000 may perform first encoding and first decoding on the first training image 802 via a specific codec, and then input the second training image to the second DNN 300.

[0218] In operation S860, the second DNN 300 processes the first training image 802 or the second training image according to initially set DNN setting information, and outputs a third training image 804 obtained by performing AI up-scaling on the first training image 802 or the second training image.

[0219] In operation S865 , the training apparatus 1000 calculates complexity loss information 820 based on the first training image 802 .

[0220] In operation S870 , the training apparatus 1000 calculates structure loss information 810 by comparing the reduced training image 803 and the first training image 802 .

[0221] In operation S875 , the training apparatus 1000 calculates quality loss information 830 by comparing the original training image 801 and the third training image 804 .

[0222] In operation S880 , the initially set DNN setting information is updated via a back propagation process based on the final loss information. The training apparatus 1000 may calculate the final loss information for training the first DNN 700 based on the complexity loss information 820 , the structure loss information 810 , and the quality loss information 830 .

[0223] In operation S885, the second DNN 300 updates the initially set DNN setting information through a back propagation process based on the quality loss information 830 or the final loss information. The training apparatus 1000 may calculate the final loss information for training the second DNN 300 based on the quality loss information 830.

[0224] Then, the training device 1000, the first DNN 700, and the second DNN 300 may repeat operations S850 to S885 until the final loss information is minimized to update the DNN setting information. At this time, during each repetition, the first DNN 700 and the second DNN 300 operate according to the DNN setting information updated in the previous operation.

[0225] Table 1 below shows the effects when AI encoding and AI decoding are performed on the original image 105 according to an embodiment of the present disclosure and when encoding and decoding are performed on the original image 105 via HEVC.

[0226] [Table 1]

[0227]

[0228] As shown in Table 1, although the subjective image quality when AI encoding and AI decoding are performed on content including 300 frames of 8K resolution according to an embodiment of the present disclosure is higher than the subjective image quality when encoding and decoding are performed via HEVC, the bit rate is reduced by at least 50%.

[0229] Fig.11 1 is a diagram of an apparatus 20 for performing AI downscaling on an original image 105 and an apparatus 40 for performing AI upscaling on a second image 135 .

[0230] Device 20 receives original image 105 and provides image data 25 and AI data 30 to device 40 by using AI downsizer 1124 and transform-based encoder 1126. According to an embodiment, image data 25 corresponds to Figure 1 The AI ​​data 30 corresponds to the image data Figure 1 In addition, according to an embodiment, the transform-based encoder 1126 corresponds to Figure 7 The first encoder 614 and the AI ​​reducer 1124 correspond to Figure 7 AI reducer 612.

[0231] The device 40 receives the AI ​​data 30 and the image data 25, and obtains the third image 145 by using the transform-based decoder 1146 and the AI ​​amplifier 1144. According to an embodiment, the transform-based decoder 1146 corresponds to Figure 2 The first decoder 232, and the AI ​​amplifier 1144 corresponds to Figure 2 AI amplifier 234.

[0232] According to an embodiment, the device 20 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the device 20 executes the reference to the computer program according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, reference will be made to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0233] According to an embodiment, the device 40 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the device 40 executes the reference to the computer program according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, reference will be made to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0234] exist Fig.11 , the configuration controller 1122 receives at least one input value 10. According to an embodiment, the at least one input value 10 may include at least one of a target resolution difference for the AI ​​downsizer 1124 and the AI ​​upsizer 1144, a bit rate of the image data 25, a bit rate type of the image data 25 (e.g., a variable bit rate type, a constant bit rate type, or an average bit rate type), or a codec type for the transform-based encoder 1126. The at least one input value 10 may include a value pre-stored in the device 20 or a value input from a user.

[0235] The configuration controller 1122 controls the operations of the AI ​​downsizer 1124 and the transform-based encoder 1126 based on the received input value 10. According to an embodiment, the configuration controller 1122 obtains DNN setting information for the AI ​​downsizer 1124 according to the received input value 10, and sets the AI ​​downsizer 1124 using the obtained DNN setting information. According to an embodiment, the configuration controller 1122 may send the received input value 10 to the AI ​​downsizer 1124, and the AI ​​downsizer 1124 may obtain DNN setting information for performing AI downsizing on the original image 105 based on the received input value 10. According to an embodiment, the configuration controller 1122 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI downsizing is applied and tone mapping information of a high dynamic range (HDR)) to the AI ​​downsizer 1124 together with the input value 10, and the AI ​​downsizer 1124 may obtain the DNN setting information in consideration of the input value 10 and the additional information. According to an embodiment, the configuration controller 1122 sends at least a portion of the received input value 10 to the transform-based encoder 1126, and the transform-based encoder 1126 performs a first encoding on the first image 115 by a bit rate of a specific value, a specific type of bit rate, and a specific codec.

[0236] The AI ​​reducer 1124 receives the original image 105 and performs a reference Figure 1 , Figure 7 , Figure 8 , Fig. 9 or Fig.10 At least one of the operations described in is performed to obtain a first image 115.

[0237] According to an embodiment, the AI ​​data 30 is provided to the device 40. The AI ​​data 30 may include at least one of resolution difference information between the original image 105 and the first image 115 or information related to the first image 115. The resolution difference information may be determined based on a target resolution difference of the input value 10, and the information related to the first image 115 may be determined based on at least one of a target bit rate, a bit rate type, or a codec type. According to an embodiment, the AI ​​data 30 may include parameters used during AI upscaling. The AI ​​data 30 may be provided to the device 40 from the AI ​​downsizer 1124.

[0238] Image data 25 is obtained as the original image 105 is processed by a transform-based encoder 1126 and sent to the device 40. The transform-based encoder 1126 may process the first image 115 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or VA1.

[0239] The configuration controller 1142 controls the operation of the AI ​​amplifier 1144 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 obtains DNN setting information for the AI ​​amplifier 1144 according to the received AI data 30, and sets the AI ​​amplifier 1144 using the obtained DNN setting information. According to an embodiment, the configuration controller 1142 may send the received AI data 30 to the AI ​​amplifier 1144, and the AI ​​amplifier 1144 may obtain DNN setting information for performing AI amplification on the second image 135 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI amplification is applied and tone mapping information of HDR) to the AI ​​amplifier 1144 together with the AI ​​data 30, and the AI ​​amplifier 1144 may obtain the DNN setting information in consideration of the AI ​​data 30 and the additional information. According to an embodiment, the AI ​​amplifier 1144 may receive AI data 30 from the configuration controller 1142, receive at least one of prediction mode information, motion information, or quantization parameter information from the transform-based decoder 1146, and obtain DNN setting information based on at least one of the prediction mode information, motion information, and quantization parameter information and the AI ​​data 30.

[0240] The transform-based decoder 1146 may process the image data 25 to reconstruct the second image 135. The transform-based decoder 1146 may process the image data 25 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1.

[0241] The AI ​​amplifier 1144 may obtain the third image 145 by performing AI amplification on the second image 135 provided from the transform-based decoder 1146 based on the set DNN setting information.

[0242] The AI ​​reducer 1124 may include a first DNN, and the AI ​​amplifier 1144 may include a second DNN, and according to an embodiment, according to the reference Fig. 9 and Fig.10 The training method described is to train DNN setting information for a first DNN and a second DNN.

[0243] exist Figures 12 to 25 In the present invention, the relationship between AI data and image data in AI encoded data and the data structure of AI data will be described. Hereinafter, the upscaling DNN is a DNN for upscaling a low-resolution image such as the second image 135 to a high-resolution image such as the third image 145, similar to Figure 3The second DNN 300 of the present invention. The enlarged DNN information indicates DNN configuration information specified according to the AI ​​data, and the enlarged DNN can be configured based on the enlarged DNN information. The low-resolution image indicates an image of low resolution, such as the first image 115 and the second image 135. The high-resolution image indicates an image of high resolution, such as the original image 105 and the third image 145.

[0244] exist Fig.12 , the correspondence between the image data 1200 and the AI ​​data 1240 is described together with the structures of the image data 1200 and the AI ​​data 1240.

[0245] exist Fig.12 , the video-frame group-frame layer structure of the image data 1200 is described. Fig.12 The video 1202 is a data unit including all consecutive frames of the image data 1200. Parameter information of the video parameter set may be applied to all frames included in the video 1202. The video parameter set is included in the video header 1204.

[0246] The video 1202 may include multiple frame groups. A frame group is a data unit of one or more continuous frames including parameter information of a shared frame group parameter set. A frame group may be a picture group (GOP) or a coded video sequence (CVS). A frame group parameter set may be included in a frame group header. For example, a frame group parameter set of a first frame group 1210 may be included in a first frame group header 1212. In addition, a frame group parameter set of a second frame group 1214 may be included in a second frame group header 1216.

[0247] A frame group is independently encoded without reference to another frame group. The frame group may include an instantaneous decoding refresh (IDR) frame or an intra-frame random access picture (IRAP) frame that is encoded without reference to another frame. In addition, the remaining frames of the frame group are encoded by referring to the IDR frame (or IRAP frame). Therefore, the first frame group 1210 can be encoded independently without reference to another frame group of the video 1202. In addition, the first frame 1220, which is the first frame to be encoded of the first frame group 1210, is an IDR frame (or IRAP frame). The remaining frames of the first frame group 1210 including the second frame 1230 are encoded by referring to the first frame 1220.

[0248] A frame indicates a still image included in a video. A frame header may include a frame parameter set, wherein the frame parameter set includes parameter information applied to the frame. For example, a first frame header 1222 of a first frame 1220 may include a frame parameter set applied to the first frame 1220. Similarly, a second frame header 1232 of a second frame 1230 may include a frame parameter set applied to the second frame 1230.

[0249] Will describe according to Fig.121202. The AI ​​data 1240 of the video-frame group-frame layer structure described in the present invention can be classified into video AI data 1242, frame group AI data 1250, and frame AI data 1260 according to the application scope. Video AI data 1242 represents AI data commonly applied to all frame groups included in the video 1202. In addition, frame group AI data 1250 represents AI data commonly applied to frames included in the current frame group. In addition, frame AI data 1260 represents AI data applied to the current frame.

[0250] The video AI data 1242 corresponds to the video header 1204. Therefore, the video AI data 1242 may be decoded in parallel with the video header 1204. Alternatively, the video AI data 1242 may be decoded immediately before the decoding of the video header 1204. Alternatively, the video AI data 1242 may be decoded immediately after the decoding of the video header 1204.

[0251] The frame group AI data 1250 corresponds to the frame group header. Fig.12 , the first frame group AI data 1252 corresponds to the first frame group header 1212. In addition, the second frame group AI data 1254 corresponds to the second frame group header 1216. The first frame group AI data 1252 and the second frame group AI data 1254 may be decoded in parallel with the first frame group header 1212 and the second frame group header 1216, respectively. Optionally, the first frame group AI data 1252 and the second frame group AI data 1254 may be decoded immediately before decoding of the first frame group header 1212 and the second frame group header 1216, respectively. Optionally, the first frame group AI data 1252 and the second frame group AI data 1254 may be decoded immediately after decoding of the first frame group header 1212 and the second frame group header 1216, respectively.

[0252] The frame AI data 1260 corresponds to the frame header. Fig.12 , the first frame AI data 1262 corresponds to the first frame header 1222. In addition, the second frame AI data 1264 corresponds to the second frame header 1232. The first frame AI data 1262 and the second frame AI data 1264 may be decoded in parallel with the first frame header 1222 and the second frame header 1232, respectively. Optionally, the first frame AI data 1262 and the second frame AI data 1264 may be decoded immediately before the decoding of the first frame header 1222 and the second frame header 1232, respectively. Optionally, the first frame AI data 1262 and the second frame AI data 1264 may be decoded immediately after the decoding of the first frame header 1222 and the second frame header 1232, respectively.

[0253] According to an embodiment of the present disclosure, Figure 7The data processor 632 may generate AI encoded data in a single file format including both the image data 1200 and the AI ​​data 1240. The communicator 634 transmits the AI ​​encoded data in a single file format to the Figure 2 Communicator 212. In this specification, a file means a group of data stored in a memory. In addition, a video file means a group of image data stored in a memory, and the image data can be implemented in the form of a bit stream.

[0254] According to an embodiment of the present disclosure, the AI ​​data 1240 may not be embedded in the image data 1200, and may be configured in a single file separately from the image data 1200. Therefore, although the AI ​​encoding data is configured in a single file, since the AI ​​data 1240 and the image data 1200 are separated, the AI ​​data 1240 and / or the image data 1200 may include information about synchronization of the AI ​​data 1240 and the image data 1200.

[0255] The communicator 212 may receive the AI ​​encoded data. In addition, the parser 214 may extract the AI ​​data and the image data from the AI ​​encoded data. In addition, the outputter 216 sends the image data to the first decoder 232, and sends the AI ​​data to the AI ​​amplifier 234. In addition, the first decoder 232 generates a low-resolution image by decoding the image data. In addition, the AI ​​amplifier 234 obtains the amplification DNN information suitable for amplification of the low-resolution image based on the AI ​​data, and the AI ​​amplifier 234 amplifies the low-resolution image by using the amplification DNN set according to the amplification DNN information.

[0256] According to an embodiment of the present disclosure, synchronization data for synchronization of AI data and image data may be included in the AI ​​encoded data. The synchronization data may be included in the AI ​​encoded data independently of the AI ​​data and the image data. Optionally, the synchronization data may be included in the AI ​​data or the image data. The parser 214 may synchronize the image data and the AI ​​data according to the synchronization data. Optionally, the AI ​​amplifier 234 may synchronize the image data and the AI ​​data according to the synchronization data. Amplification DNN information suitable for AI amplification of a low-resolution image may be selected according to the synchronization data.

[0257] According to an embodiment of the present disclosure, the AI ​​data 1240 may be embedded in the image data 1200. For example, the video AI data 1242 may be embedded in the video header 1204. Therefore, the video header 1204 may include the video AI data 1242 and the video parameter set. Therefore, the video AI data 1242 may be decoded together with the video parameter set. Alternatively, the video AI data 1242 may be embedded in a single file independently of the video header 1204 to be located before or after the video header 1204. Therefore, the video AI data 1242 may be decoded immediately before or after the decoding of the video header 1204.

[0258] Similarly, the first frame group header 1212 may be included in the first frame group AI data 1252 together with the frame group parameter set. Therefore, the first frame group AI data 1252 may be decoded together with the frame group parameter set. Alternatively, the first frame group AI data 1252 may be embedded in a single file independently of the first frame group header 1212 to be located before or after the first frame group header 1212. Therefore, the first frame group AI data 1252 may be decoded immediately before or after the decoding of the first frame group header 1212.

[0259] Similarly, the first frame header 1222 may include the first frame AI data 1262 and the frame parameter set. Therefore, the first frame AI data 1262 may be decoded together with the frame parameter set. Alternatively, the first frame AI data 1262 may be embedded in a single file independently of the first frame header 1222 to be located before or after the first frame header 1222. Therefore, the first frame AI data 1262 may be decoded immediately before or after the decoding of the first frame header 1222.

[0260] Will refer to Fig.13a The data flow in the AI ​​decoding device 200 when AI data is embedded in image data is described.

[0261] In the AI ​​coded data, when the AI ​​data is embedded in the image data, the AI ​​data cannot be independently decoded without decoding the image data. Therefore, when the first decoder 232 decodes the image data, the AI ​​data embedded in the image data is extracted from the image data. In addition, the AI ​​data extracted from the image data is sent from the first decoder 232 to the AI ​​amplifier 234.

[0262] According to an embodiment of the present disclosure, a portion of the AI ​​data may be embedded in the image data, and the rest of the AI ​​data may be included in the AI ​​encoding data independently of the image data. For example, video AI data may be included in the AI ​​encoding data independently of the image data, and frame group AI data and frame AI data may be embedded in the image data.

[0263] The first AI data existing independently of the image data may be separated from the image data by the parser 214. Also, the first AI data separated from the image data may be transmitted from the outputter 216 to the AI ​​amplifier 234. The first AI data may be video AI data and / or video segment AI data.

[0264] In addition, the second AI data embedded in the image data is extracted from the image data by the first decoder 232. The extracted second AI data is sent to the AI ​​amplifier 234 from the first decoder 232. In addition, the AI ​​amplifier 234 obtains the amplification DNN information required for AI amplification of the low-resolution image based on the first AI data and the second AI data. The second AI data may be frame group AI data and / or frame AI data.

[0265] According to an embodiment of the present disclosure, the data processor 632 may separately generate a file corresponding to the image data 1200 and a file corresponding to the AI ​​data 1240. Therefore, the communicator 634 transmits the AI ​​encoded data to the communicator 212 in two file formats. Therefore, the communicator 634 may transmit the file corresponding to the image data 1200 and the file corresponding to the AI ​​data 1240 via different communication channels. In addition, the communicator 634 may sequentially transmit the file corresponding to the image data 1200 and the file corresponding to the AI ​​data 1240 with a time difference.

[0266] According to the embodiments of the present disclosure, according to the reference Fig.12 The correspondence between the described image data 1200 and the AI ​​data 1240 may decode the file corresponding to the AI ​​data 1240 according to the decoding process of the file corresponding to the image data 1200. In addition, since the file corresponding to the image data 1200 and the file corresponding to the AI ​​data 1240 are separated, the file corresponding to the image data 1200 and the file corresponding to the AI ​​data 1240 may include information about synchronization of the two files.

[0267] Fig.13b is a diagram for describing a data flow in the AI ​​decoding device 200 when AI data and image data are separated and configured in two files.

[0268] The communicator 212 may receive a file including image data and a file including AI data separately, rather than a single file including AI encoded data. In addition, the communicator 212 may obtain synchronization data required for synchronization of the image data and the AI ​​data from the file including image data or the file including AI data. According to an embodiment of the present disclosure, the synchronization data may be sent from a separate file. Fig.13b In the embodiment, the synchronization data is represented as data independent of the AI ​​data, but according to an embodiment of the present disclosure, the AI ​​data or the image data may include the synchronization data.

[0269] According to an embodiment of the present disclosure, the parser 214 may synchronize the image data and the AI ​​data according to the synchronization data. In addition, the outputter 216 may transmit the synchronized image data to the first decoder 232 and transmit the synchronized AI data to the AI ​​amplifier 234.

[0270] According to an embodiment of the present disclosure, the outputter 216 may transmit the image data to the first decoder 232. In addition, the AI ​​data and the synchronization data may be transmitted to the AI ​​amplifier 234. The AI ​​amplifier 234 AI-amplifies the low-resolution image output from the first decoder 232 by using the amplification DNN information obtained according to the synchronization data and the AI ​​data.

[0271] Fig.14 14 shows an example of AI encoded data when AI data 1420 and image data 1431 are separated in a single file. Fig.14 In the embodiment, the AI ​​encoded data is included in a video file 1400 in a specific container format. The specific container format may be MP4, AVI, MKV or FLV.

[0272] The video file 1400 includes a metadata box 1410 and a media data box 1430. Information about media data included in the media data box 1430 is included in the metadata box 1410. For example, the metadata box 1410 may include information about the type of media data, the type of codec used to encode the media data, and the reproduction time of the media.

[0273] The metadata box 1410 may include synchronization data 1415 and AI data 1420. The synchronization data 1415 and the AI ​​data 1420 are stored in the metadata box 1410 after being encoded according to an encoding method provided by a specific container format.

[0274] The parser 214 may extract the synchronization data 1415 and the AI ​​data 1420 from the metadata box 1410. In addition, the parser 214 may extract the image data 1431 from the media data box 1430. The outputter 216 may transmit the image data 1431 to the first decoder 232 according to the synchronization data 1415 and transmit the AI ​​data 1420 to the AI ​​amplifier 234. Alternatively, the outputter 216 may transmit the synchronization data 1415 to the AI ​​amplifier 234, and the AI ​​amplifier 234 may synchronize the image data 1431 and the AI ​​data 1420 according to the synchronization data 1415.

[0275] The AI ​​data 1420 may include video AI data 1422, frame group AI data 1424, and frame AI data 1426. The video AI data 1422 may be set to correspond to the video header 1432, the frame group AI data 1424 may be set to correspond to the frame group header 1436, and the frame AI data 1426 may be set to correspond to the frame header 1440.

[0276] According to an embodiment of the present disclosure, when the same magnification DNN information is used for all frames of a video, the frame group AI data 1424 and the frame AI data 1426 may be omitted from the AI ​​data 1420. Alternatively, when the magnification DNN information is adaptively obtained for each frame group unit and the same magnification DNN information is used for all frames of the frame group, the frame AI data 1426 may be omitted from the AI ​​data 1420.

[0277] The synchronization data 1415 includes information about synchronization of the video AI data 1422, the frame group AI data 1424, and the frame AI data 1426, as well as the video header 1432, the frame group header 1436, and the frame header 1440. For example, the synchronization data 1415 may include reproduction order information or decoding order information of the image data 1431 of the media data frame 1430. Therefore, the AI ​​amplifier 234 may obtain the amplification DNN information required for AI amplification of the low-resolution image from the AI ​​data determined according to the reproduction order information or the decoding order information of the synchronization data 1415.

[0278] For example, in order to AI upscale the low-resolution image of frame 1438, the parser 214 or the AI ​​amplifier 234 may determine the frame group AI data 1424 corresponding to the frame group 1434 or the frame AI data 1426 corresponding to the frame 1438 based on the synchronization data 1415. In addition, considering the video AI data 1422 applied to all frames as well as the frame group AI data 1424 and the frame AI data 1426, the AI ​​amplifier 234 may obtain upscaled DNN information for AI upscale of the low-resolution image of frame 1438.

[0279] The AI ​​data 1420 of the metadata box 1410 may be decoded before the image data 1431 of the media data box 1430. Therefore, the AI ​​amplifier 234 may obtain the amplification DNN information before the decoding of the image data 1431 according to the AI ​​data 1420. The amplification DNN information may be applied to the entire video. Optionally, different multiple amplification DNN information may be obtained for each frame group for adaptive AI amplification of the frame group unit. Optionally, the amplification DNN information may be obtained differently in advance for each frame for adaptive AI amplification of the frame unit.

[0280] The AI ​​amplifier 234 may decode the AI ​​data 1420 of the metadata frame 1410 according to the decoding order of the image data 1431 of the media data frame 1430. The video AI data 1422 may be decoded immediately before or after the decoding of the video header 1432. Alternatively, the video AI data 1422 may be decoded in parallel with the video header 1432. In order to decode the video AI data 1422 according to the decoding order of the video header 1432, the synchronization data 1415 may be referred to.

[0281] Similarly, the AI ​​amplifier 234 may decode the frame group AI data 1424 immediately before or after the first decoder 232 decodes the frame group header 1436. Alternatively, the AI ​​amplifier 234 may decode the frame group AI data 1424 in parallel with the first decoder 232 decoding the frame group header 1436. In order to decode the frame group AI data 1424 according to the decoding order of the frame group header 1436, the synchronization data 1415 may be referred to.

[0282] In addition, the AI ​​amplifier 234 may decode the frame AI data 1426 immediately before or after the first decoder 232 decodes the frame header 1440. Alternatively, the AI ​​amplifier 234 may decode the frame AI data 1426 in parallel with the first decoder 232 decoding the frame header 1440. In order to decode the frame AI data 1426 according to the decoding order of the frame header 1440, the synchronization data 1415 may be referred to.

[0283] exist Fig.14 , for ease of description, a metadata box 1410 and a media data box 1430 are included in the video file 1400. However, the video file 1400 may include two or more metadata boxes and two or more media data boxes. Therefore, two or more image data segments in which the image data is divided in a specific time unit may be stored in two or more media data boxes. In addition, information about the image data segments stored in the two or more media data boxes may be included in the two or more metadata boxes. In addition, the two or more metadata boxes may include AI data.

[0284] Fig.15a An example of AI encoded data when AI data is embedded in image data 1505 is shown.

[0285] Similar to Fig.14The video file 1400 includes a metadata box 1502 and a media data box 1504. Since the AI ​​data is embedded in the image data 1505, the metadata box 1502 does not include the AI ​​data. Alternatively, the image data 1505 in which the AI ​​data is embedded is included in the media data box 1504. The AI ​​data may be encoded according to a video codec of the image data 1505. For example, the video codec of the image data 1505 may be H.264, HEVC, AVS2.0, and Xvid.

[0286] Therefore, the parser 214 does not extract the AI ​​data from the metadata box 1502. Instead, the first decoder 232 may extract the AI ​​data from the image data 1505 and send the extracted AI data to the AI ​​amplifier 234. In addition, the AI ​​amplifier 234 may AI-amplify the low-resolution image reconstructed by the first decoder 232 based on the amplification DNN information obtained via the AI ​​data.

[0287] and Fig.14 Unlike the video file 1400 of FIG. 1505 , the video file 1500 may not include synchronization data because the AI ​​data is embedded in the image data 1505 according to the decoding order of the image data 1505. Therefore, when the image data 1505 is decoded, the AI ​​data is sequentially decoded.

[0288] The video AI data 1508 may be immediately after the video header 1506 including parameter information of the video. Therefore, the video AI data 1508 may be decoded after decoding the video parameters included in the video header 1506. Fig.15a Differently, the video AI data 1508 may be immediately preceding the video header 1506 .

[0289] The frame group AI data 1514 may be immediately after the frame group header 1512 including the parameter information of the frame group 1510. Therefore, the frame group AI data 1514 may be decoded after decoding the frame group parameters included in the frame group header 1512. Fig.15a Differently, the frame group AI data 1514 may be immediately before the frame group header 1512. The decoding order of the pieces of frame group AI data and the frame group header of the remaining frame groups decoded after the frame group 1510 may be determined in the same manner as the decoding order of the frame group AI data 1514 and the frame group header 1512.

[0290] The frame AI data 1520 may be immediately after the frame header 1518 including the parameter information of the frame 1516. Therefore, the frame AI data 1520 may be decoded after decoding the frame group parameters included in the frame header 1518. Fig.15aDifferently, the frame AI data 1520 may be immediately before the frame header 1518. The decoding order of the pieces of frame AI data and the frame header of the remaining frames decoded after the frame 1516 may be determined in the same manner as the decoding order of the frame AI data 1520 and the frame header 1518.

[0291] Fig.15a Another example of AI encoding data when AI data is embedded in image data 1525 is shown.

[0292] Similar to Fig.15a 1500, the video file 1521 includes a metadata box 1522 and a media data box 1524. In addition, the metadata box 1522 does not include AI data, and instead, the image data 1525 embedded with AI data is included in the media data box 1524. However, unlike Fig.15a Unlike video file 1500 , a data header corresponding to the AI ​​data is embedded in video file 1521 .

[0293] The video AI data 1528 may be included in the video header 1526 including parameter information of the video. Therefore, the video AI data 1528 may be decoded together with the video parameters included in the video header 1526.

[0294] The frame group AI data 1534 may be included in the frame group header 1532, wherein the frame group header 1532 includes parameter information of the frame group 1530. Therefore, the frame group AI data 1534 may be decoded together with the frame group parameters included in the frame group header 1532. Frame group AI data of the remaining frame groups decoded after the frame group 1530 may also be included in the frame group header 1532.

[0295] The frame AI data 1540 may be included in the frame header 1538, wherein the frame header 1538 includes parameter information of the frame 1536. Therefore, the frame AI data 1540 may be decoded together with the frame parameters included in the frame header 1538. The frame AI data of the remaining frames decoded after the frame 1536 may also be included in the frame header 1538.

[0296] Fig.15c An example of AI encoded data is shown when part of the AI ​​data is embedded in the image data 1557 and the remaining AI data is separated from the image data 1557.

[0297] Similar to Fig.15a The video file 1500 includes a metadata box 1552 and a media data box 1556. The metadata box 1552 includes video AI data 1554 applied to all frames of the video. In addition, frame group AI data 1564 and frame AI data 1570 are included in the image data 1557 of the media data box 1556.

[0298] and Fig.15a Video AI data 1508 and Fig.15b Unlike the video AI data 1528, the video AI data 1554 included in the metadata box 1552 may be decoded before decoding the image data 1557. In addition, when the image data 1557 is decoded, the frame group AI data 1564 and the frame AI data 1570 may be sequentially decoded.

[0299] Therefore, the parser 214 may extract the video AI data 1554 from the metadata box 1552. In addition, the communicator 212 may send the video AI data 1554 to the AI ​​amplifier 234. In addition, the communicator 212 may send the image data 1557 to the first decoder 232. In addition, the first decoder 232 may decode the image data 1557 to reconstruct the low-resolution image, and may extract the frame group AI data 1564 and the frame AI data 1570. In addition, the first decoder 232 may send the frame group AI data 1564 and the frame AI data 1570 to the AI ​​amplifier 234. In addition, the AI ​​amplifier 234 may obtain the amplification DNN information for AI amplification of the low-resolution image based on the video AI data 1554, the frame group AI data 1564, and the frame AI data 1570.

[0300] The frame group AI data 1564 may be located immediately after the frame group header 1562 including parameter information of the frame group 1560. However, according to an embodiment of the present disclosure, the frame group AI data 1564 may be located immediately before the frame group header 1562. In addition, the frame group AI data 1564 may be included in the frame group header 1562.

[0301] The frame AI data 1570 may be located immediately after the frame header 1568 including the parameter information of the frame 1566. However, according to an embodiment of the present disclosure, the frame AI data 1570 may be located immediately before the frame header 1568. In addition, the frame AI data 1570 may be included in the frame header 1568.

[0302] exist Fig.15c , only the video AI data 1554 is included in the metadata box 1552, but the frame group AI data 1564 may be additionally included in the metadata box 1552. Optionally, a portion of the frame group AI data 1564 may be included in the metadata box 1552. Optionally, a portion of the frame AI data 1570 may be included in the metadata box 1552.

[0303] exist Fig.15a and Fig.15cIn the embodiment of the present invention, the frame group AI data 1514 and 1564 and the frame AI data 1520 and 1570 embedded in the media data boxes 1504 and 1556 may be embedded in the form of a supplemental enhancement information (SEI) message. The SEI message is a data unit including additional information about an image, wherein the additional information is not necessarily required when decoding the image. The SEI message may be transmitted in units of frame groups or in units of frames. In addition, similar to the reference Fig.13a The second AI data described above, the SEI message may be extracted by the first decoder 232 and sent to the AI ​​amplifier 234 .

[0304] Fig.16 It shows that when AI data and image data are Fig.14 An example of AI encoded data divided by video segment units when separated as in .

[0305] When the size of the metadata including AI data is large or the image data is streamed in real time through a communication channel, the AI ​​encoded data may be divided by video segment units and stored in the video file 1600 to reduce the overhead required for processing the metadata including AI data. A video segment is a portion of a video and includes frames at a specific time. For example, a video segment may include only one frame group or more frame groups. When a video segment includes one frame group, the video file 1600 may include video segments equal in number to the number of frame groups of the image data.

[0306] Each video segment may include a metadata box and a media data box. Metadata including AI data is divided and stored in multiple video segments. Therefore, the AI ​​data is divided and stored not only in the metadata box 1610 regarding the entire image data, but also in the metadata box regarding the video segment. Therefore, when a specific portion of the image data is AI-enlarged from a low-resolution image, when the AI ​​data is divided and stored in the metadata box regarding the video segment and the metadata box 1610, the AI ​​data for obtaining the enlarged DNN information suitable for the specific portion can be obtained more quickly than when all the AI ​​data is stored in the metadata box 1610 regarding the entire image data.

[0307] For example, when performing reproduction from the current video segment, for AI upscaling of a low-resolution image of the current video segment, only the video AI data 1612 of the metadata box 1610 and the AI ​​data of the segment metadata box 1630 of the current video segment data box 1620 may be referenced. In addition, the AI ​​data of the metadata boxes of other video segments are not referenced. Therefore, the overhead of decoding according to the AI ​​data is reduced.

[0308] In addition, when the image data is streamed in real time, the AI ​​data is not transmitted immediately during the initial reproduction, but may be divided and transmitted by video segment units. Therefore, since the AI ​​data is divided and transmitted sequentially, the overhead according to the decoding of the AI ​​data may be reduced. Therefore, the division and transmission by video segment units may be advantageous.

[0309] exist Fig.16 In FIG. 1 , a metadata box 1610 about the entire image data includes video AI data 1612. The video AI data 1612 is applied to all video segments included in the video. The metadata box 1610 may be decoded before the current video segment data box 1620.

[0310] The current video segment data box 1620 includes a segment metadata box 1630 and a segment media data box 1640. The segment metadata box 1630 may include synchronization data 1631 and AI data 1632. In addition, the segment media data box 1640 includes video segment data 1641.

[0311] The AI ​​data 1632 of the current video segment data frame 1620 may include video segment AI data 1634, frame group AI data 1636, and frame AI data 1638. The video segment AI data 1634 is applied to all frame groups included in the current video segment. In addition, the frame group AI data 1636 is applied to all frames included in the current frame group. The frame AI data 1638 is applied to the current frame.

[0312] According to an embodiment of the present disclosure, when the same magnification DNN information is used for all frames of a video segment, the frame group AI data 1636 and the frame AI data 1638 may be omitted from the AI ​​data 1632 of the current video segment data frame 1620. Alternatively, when the magnification DNN information is adaptively determined for each frame group unit and the same magnification DNN information is used for all frames of the frame group, the frame AI data 1638 may be omitted from the AI ​​data 1632 of the current video segment data frame 1620.

[0313] According to an embodiment of the present disclosure, when a video segment includes one frame group, the video segment AI data 1634 may be omitted from the AI ​​data 1632 of the current video segment data frame 1620. Here, the frame group AI data 1636 may perform a function as the video segment AI data 1634. Alternatively, when the video segment includes only one frame group and the same magnification DNN information is used for all frames of the frame group, the video segment AI data 1634 and the frame AI data 1638 may be omitted from the AI ​​data 1632 of the current video segment data frame 1620. Then, AI magnification may be applied to all frames of the video segment based on the magnification DNN information according to the frame group AI data 1636. The synchronization data 1631 includes information on synchronization of the video segment AI data 1634, the frame group AI data 1636, and the frame AI data 1638, as well as the video segment header 1642, the frame group header 1646, and the frame header 1648. The video segment header 1642 includes video segment parameters commonly applied to the frames included in the video segment. For example, the synchronization data 1631 may include reproduction order information or decoding order information of the video segment data 1641 of the segment media data box 1640. Therefore, the AI ​​upscaling 234 may obtain upscaling information required for AI upscaling of the low-resolution image from the AI ​​data determined according to the synchronization data 1631.

[0314] For example, in order to upscale the low-resolution image of frame 1649, the parser 214 or the AI ​​amplifier 234 may obtain the frame group AI data 1636 corresponding to the frame group 1644 and the frame AI data 1638 corresponding to the frame 1649 based on the synchronization data 1631. Then, considering the video segment AI data 1634 applied to all frames as well as the frame group AI data 1636 and the frame AI data 1638, the AI ​​amplifier 234 may obtain upscaled DNN information for AI upscale of the low-resolution image of frame 1649.

[0315] According to an embodiment of the present disclosure, the AI ​​data 1632 of the segment metadata box 1630 may be decoded before the video segment data 1641 of the segment media data box 1640. Therefore, the AI ​​amplifier 234 may obtain the amplification DNN information before decoding the video segment data 1641 according to the AI ​​data 1632. The obtained amplification DNN information may be applied to the entire video segment. Optionally, for adaptive AI amplification in frame group units, the amplification DNN information may be pre-obtained differently for each frame group. Optionally, for adaptive AI amplification in frame units, the amplification DNN information may be pre-obtained differently for each frame.

[0316] According to an embodiment of the present disclosure, the AI ​​amplifier 234 may decode the AI ​​data 1632 of the segment metadata box 1630 according to the decoding order of the video segment data 1641 of the segment media data box 1640. For example, the AI ​​amplifier 234 may decode the frame group AI data 1636 according to the decoding order of the frame group header 1646 by the first decoder 232. In addition, the AI ​​amplifier 234 may decode the frame AI data 1638 according to the decoding of the frame header 1648 by the first decoder 232. The decoding order of the video segment AI data 1634, the frame group AI data 1636, and the frame AI data 1638 may be synchronized with the decoding order of the video segment header 1642, the frame group header 1646, and the frame header 1648 by referring to the synchronization data 1631.

[0317] The remaining video segment data frames following the current video segment data frame 1620 may be sequentially decoded in the same method as the current video segment data frame 1620 .

[0318] Fig.17 An example of AI data 1740 and image data 1700 being sent in two separate files is shown.

[0319] When the communicator 212 does not receive the AI ​​data 1740, the low-resolution image obtained from the image data 1700 is not AI-enlarged. When the communicator 212 receives the AI ​​data 1740, the enlarged DNN information required for the AI ​​enlargement of the low-resolution image is obtained according to the AI ​​data 1740 sent to the AI ​​amplifier 234 through the outputter 216.

[0320] The image data 1700 may include a video header 1710, a frame group header 1722 of a frame group 1720, and a frame header 1732 of a frame 1730. In addition, the AI ​​data 1740 may include video AI data 1742, frame group AI data 1750, and frame AI data 1760. Since the image data 1700 and the AI ​​data 1740 are transmitted in separate files, the image data 1700 and / or the AI ​​data 1740 may include synchronization data required for synchronization of the image data 1700 and the AI ​​data 1740. The synchronization data may indicate a decoding order or a reproduction order of the image data 1700.

[0321] The parser 214 or the AI ​​amplifier 234 may set the reproduction order or decoding order of the video AI data 1742 , the frame group AI data 1750 , and the frame AI data 1760 according to the synchronization data to match the reproduction order or decoding order of the video header 1710 , the frame group header 1722 , and the frame header 1732 . Fig.17The dotted arrows between the video AI data 1742 and the video header 1710, the dotted arrows between the frame group AI data 1750 and the frame group header 1722, and the dotted arrows between the frame AI data 1760 and the frame header 1732 indicate synchronization between the AI ​​data and the data header.

[0322] The image data 1700 and the AI ​​data 1740 may include an identification number for matching the two data. For example, the AI ​​data 1740 may include an identification number of the image data 1700 to which the AI ​​data 1740 is applied. On the other hand, the image data 1700 may include an identification number of the AI ​​data 1740 to which the image data 1700 is applied. Alternatively, both the image data 1700 and the AI ​​data 1740 may include an identification number. Therefore, even when the image data 1700 and the AI ​​data 1740 are not transmitted at the same time, the image data 1700 and the AI ​​data 1740 may be matched with each other according to the identification number.

[0323] According to an embodiment of the present disclosure, when the enlarged DNN information is obtained in frame group units, the frame AI data 1760 may be omitted from the AI ​​data 1740. In addition, when the same enlarged DNN information is used for all frames of the video, the frame group AI data 1750 and the frame AI data 1760 may be omitted from the AI ​​data 1740.

[0324] Fig.18a Shown for Figures 14 to 17 An example of data structure 1800 of video AI data described in .

[0325] The data structure 1800 of the video AI data includes elements related to upscaling DNN information for AI upscaling, including ai_codec_info 1802, ai_codec_applied_channel_info 1804, target_bitrate_info 1806, res_info 1808, ai_codec_DNN_info 1814, and ai_codec_supplementary_info 1816. Fig.18a The arrangement order of elements shown in FIG. 1 is only an example and may be changed and implemented by those having ordinary skill in the art.

[0326] According to an embodiment of the present disclosure, ai_codec_info 1802 represents AI target data indicating whether AI upscaling will be applied to a low-resolution image. When ai_codec_info 1802 indicates that AI upscaling is applied to a low-resolution image reconstructed from image data, data structure 1800 includes AI supplementary data for obtaining upscaling DNN information for AI upscaling. On the other hand, when ai_codec_info 1802 indicates that AI upscaling is not applied to a low-resolution image reconstructed from image data, data structure 1800 may not include AI supplementary data regarding AI upscaling. The AI ​​supplementary data includes ai_codec_applied_channel_info 1804, target_bitrate_info 1806, res_info 1808, ai_codec_DNN_info 1814, and ai_codec_supplementary_info 1816.

[0327] ai_codec_applied_channel_info 1804 is channel information indicating the color channel to which AI amplification is applied. The image may be represented in RGB format, YUV format, or YCbCr format. When the low-resolution image reconstructed from the image data is in YCbCr format, the low-resolution image includes a low-resolution image of the Y channel for luminance, a low-resolution image of the Cb channel for chrominance, and a low-resolution image of the Cr channel. ai_codec_applied_channel_info 1804 may indicate the color channel to which AI amplification is applied among the three channels. In addition, the AI ​​amplifier 234 may perform AI amplification on the low-resolution image of the color channel indicated by ai_codec_applied_channel_info 1804. The AI ​​amplifier 234 may obtain different multiple pieces of amplification DNN information for each color channel.

[0328] According to an embodiment of the present disclosure, AI upscaling may be applied only to a low-resolution image of the Y channel with respect to brightness to reduce throughput according to AI upscaling. Because the human eye is more sensitive to image quality for brightness rather than chrominance, the difference in people's subjective perception of image quality may not be significant between when low-resolution images of all color channels are upscaled by AI and when low-resolution images of only the Y channel are upscaled by AI.

[0329] Therefore, ai_codec_applied_channel_info 1804 may indicate whether the low-resolution image of the Cb channel and the low-resolution image of the Cr channel are AI-enlarged. When ai_codec_applied_channel_info 1804 indicates that the low-resolution image of the Cb channel and the low-resolution image of the Cr channel are not AI-enlarged, the AI ​​amplifier 234 may apply AI enlargement only to the low-resolution image of the Y channel. On the other hand, when ai_codec_applied_channel_info 1804 indicates that the low-resolution image of the Cb channel and the low-resolution image of the Cr channel are AI-enlarged, the AI ​​amplifier 234 may apply AI enlargement to the low-resolution images of all channels.

[0330] The target_bitrate_info 1806 is information indicating a bit rate of image data obtained as a result of the first encoding performed by the first encoder 614. The AI ​​amplifier 234 may obtain amplification DNN information suitable for image quality of a low-resolution image according to the target_bitrate_info 1806.

[0331] res_info 1808 indicates a resolution image related to the resolution of a high-resolution image (such as the third image 145) obtained via AI upscaling. res_info 1808 may include pic_width_org_luma 1810 and pic_height_org_luma 1812. pic_width_org_luma 1810 and pic_height_org_luma 1812 are high-resolution image width information and high-resolution image height information indicating the width and height of the high-resolution image, respectively. The AI ​​amplifier 234 may determine an AI magnification ratio according to the resolution of the high-resolution image determined by pic_width_org_luma 1810 and pic_height_org_luma 1812 and the resolution of the low-resolution image reconstructed by the first decoder 232.

[0332] According to an embodiment of the present disclosure, res_info 1808 may include resolution ratio information indicating a resolution ratio of the low-resolution image and the high-resolution image, instead of pic_width_org_luma 1810 and pic_height_org_luma 1812. Therefore, the AI ​​amplifier 234 may determine the resolution of the high-resolution image according to the resolution ratio based on the resolution ratio information and the resolution of the low-resolution image reconstructed by the first decoder 232. The resolution ratio information may include vertical resolution ratio information and horizontal resolution ratio information.

[0333] The AI ​​amplifier 234 may obtain the amplified DNN information suitable for AI amplification of the low-resolution image according to the AI ​​amplification ratio determined based on res_info 1808. Alternatively, the AI ​​amplifier 234 may obtain the amplified DNN information suitable for AI amplification of the low-resolution image according to the resolution of the low-resolution image and the resolution of the high-resolution image determined based on res_info 1808.

[0334] ai_codec_DNN_info 1814 is information indicating pre-agreed upscaling DNN information for AI upscaling of low-resolution images. The AI ​​amplifier 234 may determine one of a plurality of pre-stored default DNN configuration information as upscaling DNN information according to ai_codec_applied_channel_info 1804, target_bitrate_info 1806, and res_info 1808. In addition, the AI ​​amplifier 234 may additionally consider other features of the high-resolution image (image type, maximum brightness, color gamut, etc.) to determine one of a plurality of pre-stored default DNN configuration information as upscaling DNN information.

[0335] When there are two or more pieces of default DNN configuration information matching a specific condition, ai_codec_DNN_info1814 may indicate one of the two or more pieces of default DNN configuration information. In addition, the AI ​​amplifier 234 may AI-amplify the low-resolution image by using the amplification DNN information indicated by ai_codec_DNN_info1814.

[0336] According to an embodiment of the present disclosure, when there are two or more pieces of default DNN configuration information matching a specific condition, ai_codec_DNN_info 1814 may indicate two or more pieces of enlarged DNN information applicable to the current video file among the two or more pieces of default DNN configuration information. In addition, one of the two or more pieces of enlarged DNN information indicated by ai_codec_DNN_info 1814 may be adaptively selected by frame group or frame unit.

[0337] ai_codec_supplementary_info 1816 indicates supplementary information about AI upscaling. ai_codec_supplementary_info 1816 may include information required to determine upscaling DNN information applied to a video. ai_codec_supplementary_info 1816 may include information about category, high dynamic range (HDR) maximum brightness, HDR color gamut, HDR perceptual quantizer (PQ), codec, and rate control type.

[0338] According to the embodiments of the present disclosure, the Fig.18a ai_codec_applied_channel_info 1804, target_bitrate_info 1806, res_info 1808, ai_codec_DNN_info 1814, and ai_codec_supplementary_info 1816. The video AI data having the data structure 1800 including the elements may be applied to all frame groups.

[0339] Fig.18b Shown to be applicable Fig.16 Video clip AI data described in or Fig.14 , Figures 15a to 15c and Fig.17 An example of a data structure 1820 of frame group AI data.

[0340] Data structure 1820 has Fig.18a The data structure 1800 is similar to the structure. Fig.18b ai_codec_info 1822, ai_codec_applied_channel_info 1826, target_bitrate_info 1828, res_info 1830, pic_width_org_luma 1832, pic_height_org_luma 1834, ai_codec_DNN_info 1836, and ai_codec_supplementary_info 1838 executable with Fig.18a ai_codec_info 1802, ai_codec_applied_channel_info 1804, target_bitrate_info 1806, res_info 1808, pic_width_org_luma 1810, pic_height_org_luma 1812, ai_codec_DNN_info 1814, and ai_codec_supplementary_info 1816 have the same functions.

[0341] However, the data structure 1820 may further include AI supplementary data related information (dependent_ai_condition_info 1824) indicating whether the AI ​​supplementary data is the same between consecutive previous data units and the current data unit. When ai_codec_info 1822 indicates that AI upscaling is applied to the low-resolution image, the data structure 1820 may include dependent_ai_condition_info 1824. When ai_codec_info 1822 indicates that AI upscaling is not applied to the low-resolution image, dependent_ai_condition_info 1824 may be omitted from the data structure 1820.

[0342] When dependent_ai_condition_info 1824 indicates that AI supplementary data is the same between consecutive previous data units and a current data unit, ai_codec_applied_channel_info 1826 , target_bitrate_info 1828 , res_info 1830 , pic_width_org_luma 1832 , pic_height_org_luma 1834 , ai_codec_DNN_info 1836 , and ai_codec_supplementary_info 1838 may be omitted from the data structure 1820 . In addition, channel information, target bit rate information, resolution information, DNN information, and supplementary information are determined from ai_codec_applied_channel_info 1826, target_bitrate_info 1828, res_info 1830, pic_width_org_luma 1832, pic_height_org_luma 1834, ai_codec_DNN_info 1836, and ai_codec_supplementary_info 1838 of a previous data unit. Therefore, when the same AI supplementary data is applied to a plurality of data units, the size of the AI ​​data may be reduced according to dependent_ai_condition_info 1824.

[0343] When dependent_ai_condition_info 1824 indicates that the AI ​​supplementary data is different between consecutive previous data units and the current data unit, the data structure 1820 includes ai_codec_applied_channel_info 1826, target_bitrate_info 1828, res_info 1830, pic_width_org_luma 1832, pic_height_org_luma 1834, ai_codec_DNN_info 1836, and ai_codec_supplementary_info 1838. In addition, the enlarged DNN information of the current data unit may be obtained independently regardless of the enlarged DNN information of the previous data unit.

[0344] According to an embodiment of the present disclosure, the supplementary data related information of the data unit sent initially may be omitted. Therefore, the AI ​​supplementary data related information is included in the video segment header from the video segment sent for the second time.

[0345] According to an embodiment of the present disclosure, AI supplementary data related information may only indicate the relevance of specific AI supplementary data. For example, AI supplementary data related information may indicate whether the current data unit inherits the resolution information of the previous data unit. The AI ​​data inherited according to the AI ​​supplementary data related information may include at least one of ai_codec_info 1822, ai_codec_applied_channel_info 1804, target_bitrate_info 1806, res_info 1808, ai_codec_DNN_info 1814, or ai_codec_supplementary_info 1816.

[0346] exist Fig.18b In the embodiment of the present invention, the data unit may be a video segment or a frame group. When the data unit is a video segment, dependent_ai_condition_info 1824 may indicate whether the AI ​​supplementary data is the same between the consecutive previous video segment and the current video segment. When the data unit is a frame group, dependent_ai_condition_info 1824 may indicate whether the AI ​​supplementary data is the same between the consecutive previous frame group and the current frame group.

[0347] Fig.19 Show implementation Fig.18a Syntax table 1900 of data structure 1800.

[0348] According to syntax table 1900 , syntax elements such as ai_codec_info, ai_codec_applied_channel_info, target_bitrate, pic_width_org_luma, pic_height_org_luma, ai_codec_DNN_info, and ai_codec_supplementary_info_flag are parsed.

[0349] ai_codec_info is the same as Fig.18a The AI ​​target data corresponding to the ai_codec_info 1802 of the ai_codec_info is obtained. When ai_codec_info indicates that AI upscaling is allowed (if (ai_codec_info)), the syntax elements corresponding to the AI ​​supplementary data are parsed. When ai_codec_info indicates that AI upscaling is not allowed, the syntax elements corresponding to the AI ​​supplementary data are not parsed.

[0350] The syntax elements corresponding to the AI ​​supplementary data include ai_codec_applied_channel_info, target_bitrate, pic_width_org_luma, pic_height_org_luma, ai_codec_DNN_info, and ai_codec_supplementary_info_flag.

[0351] ai_codec_applied_channel_info and Fig.18a ai_codec_applied_channel_info1804. target_bitrate is the same as Fig.18a The target bit rate information corresponding to target_bitrate_info 1806. pic_width_org_luma and pic_height_org_luma are respectively Fig.18a The pic_width_org_luma1810 and pic_height_org_luma1812 correspond to the high-resolution image width information and high-resolution image height information. ai_codec_DNN_info is related to Fig.18a ai_codec_DNN_info 1814 corresponds to the DNN information.

[0352] ai_codec_supplementary_info_flag is an indication Fig.18aThe ai_codec_supplementary_info 1816 of the syntax table 1900 is a supplementary information flag. When ai_codec_supplementary_info_flag indicates that supplementary information for AI up-scaling is not parsed, additional supplementary information is not obtained. However, when ai_codec_supplementary_info_flag indicates that supplementary information for AI up-scaling is parsed (if (ai_codec_supplementary_info_flag)), additional supplementary information is obtained.

[0353] The obtained additional supplementary information may include genre_info, hdr_max_luminance, hdr_color_gamut, hdr_pq_type, and rate_control_type. Genre_info indicates the kind of content of the image data, hdr_max_luminance indicates the HDR maximum brightness applied to the high-resolution image, hdr_color_gamut indicates the HDR color gamut applied to the high-resolution image, hdr_pq_type indicates HDR PQ information applied to the high-resolution image, and rate_control_type indicates the rate control type applied to the image data obtained as a result of the first encoding. According to an embodiment of the present disclosure, a specific syntax element may be parsed in the syntax element corresponding to the supplementary information.

[0354] Fig.19 The syntax table 1900 is only an example, and Fig.18a Some of the elements of the data structure 1800 may be included in the syntax table 1900. In addition, elements not included in the data structure 1800 may be included in the syntax table 1900.

[0355] According to an embodiment of the present disclosure, similar to Fig.18b Data structure 1820, Fig.19 The syntax table 1900 may include AI supplementary data related information such as dependent_ai_condition_info. Therefore, the syntax table 1900 to which the AI ​​supplementary data related information is added may be applied to a video segment or a frame group.

[0356] Fig. 20 Shown to be applicable Figures 14 to 17 An example of a data structure 2000 of (frame group AI data or) frame AI data described in FIG.

[0357] Elements for adaptively determining the amplification DNN information in frame units are included in the data structure 2000. The elements may include ai_codec_frame_info 2002, dependent_ai_condition_frame_info 2004, ai_codec_frame_DNN_info 2006, ai_codec_enhancement_flag 2008, and ai_codec_artifact_removal_flag 2014. Fig. 20 The arrangement order of elements shown in FIG. 1 is only an example and may be changed and implemented by those having ordinary skill in the art.

[0358] ai_codec_frame_info 2002 is frame AI target data indicating whether AI upscaling is allowed for the current frame. According to ai_codec_frame_info 2002, when AI upscaling is allowed for the current frame, the data structure 2000 includes frame AI supplementary data related to AI upscaling of the current frame. Otherwise, AI upscaling is not applied to the current frame, and the data structure 2000 does not include frame AI supplementary data. The frame AI supplementary data represents AI supplementary data applied to the frame.

[0359] According to an embodiment of the present disclosure, when AI amplification is allowed for the current frame according to ai_codec_frame_info 2002, the data structure 2000 may include dependent_ai_condition_frame_info 2004. Dependent_ai_condition_frame_info 2004 is frame AI supplementary data related information indicating whether the frame AI supplementary data is the same between the consecutive previous frames and the current frame. When dependent_ai_condition_frame_info 2004 indicates that the frame AI supplementary data is the same between the consecutive previous frames and the current frame, the data structure 2000 may not include additional frame AI supplementary data about the current frame, and the amplified DNN information of the current frame is determined to be the same as the amplified DNN information of the previous frame. When dependent_ai_condition_frame_info 2004 indicates that the frame AI supplementary data is not the same between the consecutive previous frames and the current frame, the data structure 2000 includes additional frame AI supplementary data about the current frame, and the amplified DNN information of the current frame is obtained independently of the amplified DNN information of the previous frame. The additional frame AI supplemental data may include ai_codec_frame_DNN_info 2006 , ai_codec_enhancement_flag 2008 , and ai_codec_artifact_removal_flag 2014 .

[0360] ai_codec_frame_DNN_info 2006 is frame DNN information indicating enlarged DNN information of the current frame among a plurality of pieces of enlarged DNN information about higher data units of the current frame. Fig.18a When ai_codec_DNN_info 1814 indicates two or more pieces of zoom-in DNN information about a video, ai_codec_frame_DNN_info 2006 may determine the zoom-in DNN information of the current frame from the two or more pieces of zoom-in DNN information. Fig.18b When ai_codec_DNN_info1836 indicates two or more pieces of zoom-in DNN information about a video segment or a frame group, ai_codec_frame_DNN_info2006 may determine the zoom-in DNN information of the current frame from the two or more pieces of zoom-in DNN information. Fig.18a ai_codec_DNN_info 1814 or Fig.18b When ai_codec_DNN_info 1836 indicates only one piece of amplified DNN information, ai_codec_frame_DNN_info 2006 may be omitted.

[0361] ai_codec_enhancement_flag 2008 is AI enhancement information indicating whether to activate the enhancement process of AI enlargement accuracy. When ai_codec_enhancement_flag 2008 indicates to activate the enhancement process of AI enlargement accuracy, some samples of the high-resolution image after AI enlargement are adjusted according to the encoding parameter information. When ai_codec_enhancement_flag 2008 indicates not to activate the enhancement process of AI enlargement accuracy, the enhancement process is omitted.

[0362] When encoding the original image 105 or the first image 115 at the encoding end, encoding parameters are generated. The encoding parameters may be generated based on prediction, transformation, and in-loop filtering of a data unit (maximum coding unit, coding unit, prediction unit, transformation unit, or pixel unit). The encoding parameters include motion vectors, predicted motion vectors, intra-frame modes, residual signal related information, and SAO parameters.

[0363] The coding parameter information is information required for performing enhancement processing according to coding parameters. The coding parameter information may include coding parameter type information indicating the type of coding parameter referenced by the enhancement processing and coding parameter mapping information indicating an application area of ​​the enhancement processing in the current frame.

[0364] When ai_codec_enhancement_flag 2008 indicates activation of the enhancement process of AI enlargement accuracy, the data structure 2000 may include encod_param_type 2010 as encoding parameter type information and encod_param_map 2012 as encoding parameter mapping information. For example, encod_param_type 2010 may indicate a motion vector. In addition, encod_param_map 2012 may indicate an application area of ​​the enhancement process according to the motion vector in the high-resolution image after AI enlargement. In addition, the pixels of the application area may be modified according to the motion vector.

[0365] According to an embodiment of the present disclosure, encod_param_type 2010 may indicate two or more encoding parameters. In addition, encod_param_map 2012 may indicate an application area of ​​an enhancement process with respect to each encoding parameter.

[0366] According to an embodiment of the present disclosure, encod_param_type 2010 and encod_param_map 2012 may be omitted from the data structure 2000. Therefore, one or more encoding parameters referred to in the enhancement process and an application area of ​​each encoding parameter may be determined in advance.

[0367] ai_codec_artifact_removal_flag 2014 is artifact removal information indicating whether artifact removal of the high-resolution image after AI enlargement is performed. For artifact removal of the high-resolution image, the pixels of the low-resolution image before AI enlargement may be corrected, or the pixels of the high-resolution image after AI enlargement according to the second DNN may be corrected according to the artifact removal information, thereby removing artifacts.

[0368] The artifact removal information may include artifact type information indicating the artifact type and artifact map information indicating the artifact area where the artifact is located. In addition, the artifact removal information may include artifact type quantity information indicating the number of artifact types of the image. Therefore, the data structure 2000 may include the number of multiple artifact type information and artifact map information equal to the number of artifact type quantity information.

[0369] Artifact types include contour artifacts, ringing artifacts, and aliasing artifacts. In addition, one or more artifact regions are determined for each artifact type. For example, one or more artifact regions may be determined for contour artifacts, and one or more artifact regions may be determined for ringing artifacts.

[0370] When ai_codec_artifact_removal_flag 2014 indicates that artifact removal of the AI-upscaled high-resolution image is performed, the data structure 2000 may include num_artifact_type 2016 as artifact type number information. When ai_codec_artifact_removal_flag 2014 indicates that artifact removal of the AI-upscaled high-resolution image is not performed, num_artifact_type 2016 is omitted from the data structure 2000.

[0371] The data structure 2000 may include artifact_type 2018 as much artifact type information as the number indicated by num_artifact_type 2016. In addition, the data structure 2000 may include artifact map information for each artifact_type 2018. The artifact map information of the data structure 2000 may include num_artifact_map 2020 indicating the number of artifact regions. In addition, the data structure 2000 may include map_x_pos 2022, map_y_pos 2024, map_width 2026, map_height 2028 indicating the position and size of each artifact region.

[0372] According to an embodiment of the present disclosure, the data structure 2000 may be omitted. Fig. 20 Some of the dependent_ai_condition_frame_info 2004, ai_codec_frame_DNN_info 2006, ai_codec_enhancement_flag 2008 and ai_codec_artifact_removal_flag 2014.

[0373] According to an embodiment of the present disclosure, Fig. 20 The data structure 2000 may be applied to a frame group instead of a frame. When the data structure 2000 is applied to a frame group, the same up-scaling DNN information, the same AI enhancement information, and the same artifact removal information may be applied to all frames included in the frame group. In addition, when the data structure 2000 is applied to a frame group, the data structure 2000 may include ai_codec_frame_group_info, dependent_ai_condition_frame_group_info, and ai_codec_frame_group_DNN_info applied to the frame group instead of ai_codec_frame_info 2002, dependent_ai_condition_frame_info 2004, and ai_codec_frame_DNN_info 2006. In addition, ai_codec_enhancement_flag 2008, encod_param_type 2010, encod_param_map 2012, ai_codec_artifact_removal_flag 2014, num_artifact_type 2016, artifact_type 2018, num_artifact_map 2020, map_x_pos 2022, map_y_pos 2024, map_width 2026, and map_height 2028 may be applied to all frames of a frame group.

[0374] According to an embodiment of the present disclosure, Fig.18a The data structure 1800 can be applied to video AI data. In addition, Fig. 20 The data structure 2000 may be applied to frame AI data. The video AI data according to the data structure 1800 includes AI data commonly applied to all frames, and the frame AI data according to the data structure 2000 includes AI data adaptively applied to frames.

[0375] According to an embodiment of the present disclosure, Fig.18aThe data structure 1800 can be applied to video AI data. In addition, Fig. 20 The data structure 2000 may be applied to the frame group AI data. The video AI data according to the data structure 1800 includes AI data commonly applied to all frames, and the frame group AI data according to the data structure 2000 includes AI data adaptively applied to the frame group. In addition, the same AI data is applied to all frames included in the frame group.

[0376] According to an embodiment of the present disclosure, Fig.18b The data structure 1820 can be applied to video segment AI data or frame group AI data. Fig. 20 The data structure 2000 may be applied to frame AI data. The video segment AI data or frame group AI data according to the data structure 1820 includes AI data commonly applied to all frames of the video segment or frame group, and the frame AI data according to the data structure 2000 includes AI data adaptively applied to the frame.

[0377] According to an embodiment of the present disclosure, Fig.18b The data structure 1820 can be applied to video segment AI data. In addition, Fig. 20 The data structure 2000 of the embodiment of the present invention may be applied to the frame group AI data. The video segment AI data according to the data structure 1820 includes AI data commonly applied to all frames of the video segment, and the frame group AI data according to the data structure 2000 includes AI data adaptively applied to the frame group. In addition, the same AI data is applied to all frames included in the frame group.

[0378] Fig.21 Show implementation Fig. 20 Syntax table 2100 of data structure 2000.

[0379] According to the syntax table 2100 , syntax elements such as ai_codec_frame_info, dependent_ai_condition_frame_info, ai_codec_frame_DNN_info, ai_codec_enhancement_flag, and ai_codec_artifact_removal_flag are parsed.

[0380] ai_codec_frame_info is the same as Fig. 20The frame AI target data corresponding to ai_codec_frame_info 2002 is obtained. When ai_codec_frame_info indicates that AI upscaling is allowed (if (ai_codec_frame_info)), the syntax elements corresponding to the frame AI supplementary data are parsed. When ai_codec_frame_info indicates that AI upscaling is not allowed, the syntax elements corresponding to the frame AI supplementary data are not parsed.

[0381] Syntax elements corresponding to the frame AI supplementary data may include dependent_ai_condition_frame_info, ai_codec_frame_DNN_info, ai_codec_enhancement_flag, and ai_codec_artifact_removal_flag. Fig.21 The dependent_ai_condition_frame_info, ai_codec_frame_DNN_info, ai_codec_enhancement_flag, and ai_codec_artifact_removal_flag correspond to Fig. 20 dependent_AI_condition_frame_info 2004, AI_codec_frame_DNN_info 2006, ai_codec_enhancement_flag 2008 and ai_codec_artifact_removal_flag 2014.

[0382] When ai_codec_frame_info indicates that AI upscaling is allowed, dependent_ai_condition_frame_info is obtained. When dependent_ai_condition_frame_info indicates that the frame AI supplementary data is the same between the consecutive previous frame and the current frame, the upscaling DNN information of the current frame is determined to be the same as the upscaling DNN information of the previous frame. When dependent_ai_condition_frame_info indicates that the frame AI supplementary data is not the same between the consecutive previous frame and the current frame, the upscaling DNN information, AI enhancement information, and artifact removal information of the current frame are determined according to ai_codec_frame_DNN_info, ai_codec_enhancement_flag, and ai_codec_artifact_removal_flag.

[0383] The ai_codec_frame_DNN_info indicates the magnified DNN information of the current frame among multiple magnified DNN information of higher data units of the current frame.

[0384] The ai_codec_enhancement_flag indicates whether to activate the enhancement process of AI magnification accuracy. When the ai_codec_enhancement_flag indicates to activate the enhancement process (if(ai_codec_enhancement_flag)), the encod_param_type indicating the type of encoding parameter and the encod_param_map indicating the encoding parameter area are obtained.

[0385] The ai_codec_artifact_removal_flag indicates whether the artifact removal of the high-resolution image magnified by AI is performed. When the ai_codec_artifact_removal_flag indicates that the artifact removal of the high-resolution image is performed (if(ai_codec_artifact_removal_flag)), the num_artifact_type indicating the number of artifact types is obtained. The artifact_type indicating the artifact types as many as the number indicated by num_artifact_type and the num_artifact_map indicating the number of artifact areas are obtained (for(i = 0; i < num_artifact_type; i++)). In addition, the map_x_pos, map_y_pos, map_width, and map_height indicating the positions and sizes of the artifact areas as many as the number of num_artifact_map are obtained (for(i = 0; i < num_artifact_map; i++)).

[0386] Fig.21 The syntax table 2100 is only an example, and Fig. 20 Some of the elements in the data structure 2000 of may be included in the syntax table 2100. In addition, the elements not included in the data structure 2000 may be included in the syntax table 2100.

[0387] Fig. 22 is a flowchart of an image decoding method according to an AI decoder according to an embodiment of the present disclosure.

[0388] In operation 2210, a video file including AI-encoded data is received, where the AI-encoded data includes image data and AI data regarding the AI magnification of the image data. The reception of the AI-encoded data in operation 2210 may be performed by the communicator 212.

[0389] In operation 2220, AI data of the AI-encoded data is obtained from the metadata box of the video file, and image data of the AI-encoded data is obtained from the media data box of the video file. When the image data and the AI ​​data are separated, the obtaining of the image data and the AI ​​data in operation 2220 may be performed by the parser 214. Alternatively, when the AI ​​data is included in the image data, the AI ​​data may be obtained from the image data by the first decoder 232.

[0390] According to an embodiment of the present disclosure, the AI ​​encoded data may include synchronization data about synchronization of the image data and the AI ​​data. The synchronization data may indicate the relationship between the image data and the AI ​​data according to the decoding order or reproduction order of the image data. The synchronization data may include information about synchronization of the video header with the video AI data, synchronization of the frame group header with the frame group AI data, and synchronization of the frame header with the frame AI data.

[0391] In operation 2230 , a low resolution image of the current frame is reconstructed by decoding the image data. The reconstruction of the low resolution image according to operation 2230 may be performed by the first decoder 232 .

[0392] In operation 2240 , the amplified DNN information of the current frame is obtained from the AI ​​data. The obtaining of the amplified DNN information of the current frame according to operation 2240 may be performed by the AI ​​amplifier 234 .

[0393] According to an embodiment of the present disclosure, AI data may include Fig.12 The AI ​​data may include video AI data, frame group AI data, and frame AI data of the layer structure of AI data. In addition, the AI ​​data may include Fig.16 The AI ​​data according to the layer structure may include up-scaled DNN information applied to the corresponding layer.

[0394] According to an embodiment of the present disclosure, the AI ​​data may include video AI data. When only the video AI data is included in the AI ​​data, the enlarged DNN information applied to all frames of the image data may be obtained from a plurality of pieces of default DNN configuration information according to the video AI data.

[0395] According to an embodiment of the present disclosure, the AI ​​data may include frame group AI data. When only the frame group AI data is included in the AI ​​data, the enlarged DNN information applied to all frames of the frame group may be adaptively obtained from a plurality of pieces of default DNN configuration information relative to the frame group according to the frame group AI data.

[0396] According to an embodiment of the present disclosure, the AI ​​data may include frame AI data. When only the frame AI data is included in the AI ​​data, the amplified DNN information applied to the frame may be adaptively obtained from a plurality of pieces of default DNN configuration information relative to the frame according to the frame AI data.

[0397] According to an embodiment of the present disclosure, the AI ​​data may include frame group AI data and video AI data. When only video AI data and frame group AI data are included in the AI ​​data, one or more pieces of enlarged DNN information may be obtained from a plurality of pieces of default DNN configuration information according to the video AI data. In addition, enlarged DNN information applied to a frame of a frame group may be selected from the one or more pieces of enlarged DNN information according to the frame group AI data.

[0398] According to an embodiment of the present disclosure, AI data may include frame AI data and video AI data. When only video AI data and frame AI data are included in the AI ​​data, one or more pieces of enlarged DNN information may be obtained from a plurality of pieces of default DNN configuration information according to the video AI data. In addition, enlarged DNN information applied to a frame may be selected from the one or more pieces of enlarged DNN information according to the frame AI data.

[0399] According to an embodiment of the present disclosure, the AI ​​data may include frame AI data and frame group AI data. When only the frame group AI data and the frame AI data are included in the AI ​​data, one or more pieces of enlarged DNN information may be obtained from a plurality of pieces of default DNN configuration information according to the frame group AI data. In addition, the enlarged DNN information applied to the frame may be selected from the one or more pieces of enlarged DNN information according to the frame AI data.

[0400] According to an embodiment of the present disclosure, the AI ​​data may include video segment AI data. When only the video segment AI data is included in the AI ​​data, the enlarged DNN information applied to all frames of the video segment may be obtained from a plurality of pieces of default DNN configuration information according to the video segment AI data.

[0401] According to an embodiment of the present disclosure, the AI ​​data may include video AI data and video segment AI data. When only the video AI data and the video segment AI data are included in the AI ​​data, one or more pieces of enlarged DNN information may be obtained from a plurality of pieces of default DNN configuration information according to the video AI data. In addition, the enlarged DNN information applied to all frames of the video segment may be selected from the one or more pieces of enlarged DNN information according to the video segment AI data.

[0402] According to an embodiment of the present disclosure, AI data may include frame group AI data or frame AI data and video segment AI data. When frame group AI data or frame AI data is included in AI data together with video segment AI data, one or more pieces of enlarged DNN information may be obtained from a plurality of pieces of default DNN configuration information according to the video segment AI data. In addition, enlarged DNN information applied to a frame group or a frame may be selected from the one or more pieces of enlarged DNN information according to the frame group AI data or frame AI data.

[0403] According to an embodiment of the present disclosure, AI data may include all of video AI data, video segment AI data, frame group AI data, and frame AI data. When AI data includes all of video AI data, video segment AI data, frame group AI data, and frame AI data, one or more pieces of enlarged DNN information applicable to the video may be limited from multiple pieces of default DNN configuration information according to the video AI data. In addition, one or more pieces of enlarged DNN information applicable to the video segment may be selected from the one or more pieces of enlarged DNN information applicable to the video according to the video segment AI data. In addition, one or more pieces of enlarged DNN information applicable to the frame group may be selected from the one or more pieces of enlarged DNN information applicable to the video segment according to the frame group AI data. In addition, the enlarged DNN information applied to the frame may be selected from the one or more pieces of enlarged DNN information applicable to the frame group according to the frame AI data. According to the above-mentioned embodiment of the present disclosure, some of the video AI data, video segment AI data, frame group AI data, and frame AI data may be removed from the above-mentioned hierarchical AI data structure.

[0404] According to an embodiment of the present disclosure, the AI ​​data may include AI target data indicating whether AI upscaling is to be applied and AI supplementary data regarding upscaling DNN information used in the AI ​​upscaling when the AI ​​upscaling is to be applied.

[0405] The video AI data may include video AI target data indicating whether AI upscaling is to be applied to the image data and video AI supplemental data regarding one or more upscaling DNN information applicable to AI upscaling of frames included in the image data. When the video AI target data indicates that AI upscaling is to be applied to the frames included in the image data, the one or more upscaling DNN information is obtained according to the video AI supplemental data.

[0406] The video segment AI data may include video segment AI target data indicating whether AI upscaling is to be applied to the video segment and video segment AI supplementary data regarding one or more upscaling DNN information applicable to AI upscaling of frames included in the video segment. When the video segment AI target data indicates that AI upscaling is to be applied to frames included in the video segment, the one or more upscaling DNN information of the video segment is obtained according to the video segment AI supplementary data.

[0407] The frame group AI data may include frame group AI target data indicating whether AI amplification is to be applied to the frame group and frame group AI supplementary data regarding one or more pieces of amplification DNN information applicable to AI amplification of frames included in the frame group. When the frame group AI target data indicates that AI amplification is applied to the frames included in the frame group, the one or more pieces of amplification DNN information of the frame group are obtained according to the frame group AI supplementary data.

[0408] The frame AI data may include frame AI target data indicating whether AI amplification is to be applied to the frame and frame AI supplementary data regarding amplification DNN information for AI amplification of the frame. When the frame AI target data indicates that AI amplification is applied to the current frame, the amplification DNN information of the current frame is obtained according to the frame AI supplementary data.

[0409] According to an embodiment of the present disclosure, the video segment AI data may include video segment AI supplementary data related information indicating whether the video segment AI supplementary data is the same between the consecutive previous video segments and the current video segment. When the video segment AI supplementary data related information indicates that the video segment AI supplementary data is the same between the consecutive previous video segments and the current video segment, the video segment AI supplementary data of the current video segment is determined to be the same as the video segment AI supplementary data of the previous video segment.

[0410] In addition, the frame group AI data may include frame group AI supplementary data related information indicating whether the frame group AI supplementary data is the same between the consecutive previous frame group and the current frame group. When the frame group AI supplementary data related information indicates that the frame group AI supplementary data is the same between the consecutive previous frame group and the current frame group, the frame group AI supplementary data of the current frame group is determined to be the same as the frame group AI supplementary data of the previous frame group.

[0411] In addition, the frame AI data may include frame AI supplementary data related information indicating whether the frame AI supplementary data is the same between the consecutive previous frames and the current frame. When the frame AI supplementary data related information indicates that the frame AI supplementary data is the same between the consecutive previous frames and the current frame, the frame AI supplementary data of the current frame may be determined to be the same as the frame AI supplementary data of the previous frame.

[0412] According to an embodiment of the present disclosure, video AI data applied to the entire image data, video segment AI data corresponding to the video segment, frame group AI data applied to the frame group, and frame AI data applied to the frame can be determined according to the synchronization data.

[0413] According to an embodiment of the present disclosure, the AI ​​data may include channel information indicating a color channel to which AI is applied for amplification. In addition, amplification DNN information may be obtained for the color channel indicated by the channel information.

[0414] According to an embodiment of the present disclosure, the AI ​​data may include at least one of target bit rate information indicating the bit rate of a low-resolution image according to the image data or resolution information related to the resolution of a high-resolution image enlarged by AI. In addition, two or more pieces of enlarged DNN information about a video, a video clip, or a frame group may be determined based on at least one of the target bit rate information or the resolution information. In addition, the enlarged DNN information of the current frame may be determined from the two or more pieces of enlarged DNN information based on the frame AI data.

[0415] In operation 2250 , a high-resolution image corresponding to the low-resolution image is generated by AI upscaling the low-resolution image according to the up-scaled DNN information of the current frame. The generation of the high-resolution image according to operation 2250 may be performed by the AI ​​amplifier 234 .

[0416] According to an embodiment of the present disclosure, the accuracy of the generated high-resolution image can be enhanced according to the AI ​​enlargement enhancement process based on the encoding parameters. Optionally, the artifacts of the generated high-resolution image can be removed according to the artifact removal process.

[0417] Reference Figures 12 to 21 The AI-amplified functions described can be applied to Fig. 22 Image decoding method.

[0418] Fig.23 is a flowchart of an image encoding method according to an AI encoder according to an embodiment of the present disclosure.

[0419] In operation 2310 , downscaling DNN information for downscaling a high-resolution image of a current frame into a low-resolution image is determined. The determination of the AI ​​data according to operation 2310 may be performed by the AI ​​downscaling unit 612 .

[0420] According to at least one of the target bit rate information or the resolution information, two or more reduced DNN information may be determined for a video, a video segment, or a frame group. In addition, the reduced DNN information of the current frame may be determined from the two or more reduced DNN information according to the frame AI data.

[0421] In operation 2320 , a low-resolution image of the current frame is generated by performing AI downscaling on the high-resolution image of the current frame using the downscaling DNN information. The generation of the low-resolution image according to operation 2320 may be performed by the AI ​​downscaling unit 612 .

[0422] In operation 2330, AI data for AI upscaling of the low-resolution image of the current frame is generated. The generation of the low-resolution image according to operation 2330 may be performed by the AI ​​downscaling unit 612. The AI ​​data for AI upscaling may be generated by referring to the downscaling DNN information generated in operation 2310.

[0423] According to an embodiment of the present disclosure, AI data may include Fig.12 The AI ​​data may include video AI data, frame group AI data, and frame AI data of the layer structure of AI data. In addition, the AI ​​data may include Fig.16 The AI ​​data according to the layer structure may include up-scaled DNN information applied to the corresponding layer.

[0424] According to an embodiment of the present disclosure, the AI ​​data may include video AI data. When only the video AI data is included in the AI ​​data, the video AI data may indicate the enlarged DNN information applied to all frames of the image data among the plurality of pieces of default DNN configuration information.

[0425] According to an embodiment of the present disclosure, the AI ​​data may include frame group AI data. When only the frame group AI data is included in the AI ​​data, the frame group AI data may indicate the enlarged DNN information applied to all frames of the frame group among the plurality of pieces of default DNN configuration information.

[0426] According to an embodiment of the present disclosure, the AI ​​data may include frame AI data. When only the frame AI data is included in the AI ​​data, the frame AI data may indicate the enlarged DNN information applied to the frame among the plurality of pieces of default DNN configuration information.

[0427] According to an embodiment of the present disclosure, the AI ​​data may include frame group AI data and video AI data. When only the video AI data and the frame group AI data are included in the AI ​​data, the video AI data may include one or more pieces of enlarged DNN information among a plurality of pieces of default DNN configuration information. In addition, the enlarged DNN information applied to the frame of the frame group in the one or more pieces of enlarged DNN information may be indicated according to the frame group AI data.

[0428] According to an embodiment of the present disclosure, AI data may include frame AI data and video AI data. When only video AI data and frame AI data are included in the AI ​​data, the video AI data may include one or more pieces of enlarged DNN information among multiple pieces of default DNN configuration information. In addition, the frame AI data may indicate the enlarged DNN information applied to the frame among the one or more pieces of enlarged DNN information.

[0429] According to an embodiment of the present disclosure, AI data may include frame AI data and frame group AI data. When only frame group AI data and frame AI data are included in the AI ​​data, the frame group AI data may indicate one or more pieces of enlarged DNN information among multiple pieces of default DNN configuration information. In addition, the frame AI data may indicate the enlarged DNN information applied to the frame among the one or more pieces of enlarged DNN information.

[0430] According to an embodiment of the present disclosure, the AI ​​data may include video segment AI data. When only the video segment AI data is included in the AI ​​data, the video segment AI data may indicate the enlarged DNN information applied to all frames of the video segment among the plurality of pieces of default DNN configuration information.

[0431] According to an embodiment of the present disclosure, the AI ​​data may include video AI data and video segment AI data. When only the video AI data and the video segment AI data are included in the AI ​​data, the video AI data may include one or more pieces of enlarged DNN information among a plurality of pieces of default DNN configuration information. In addition, the video segment AI data may indicate the enlarged DNN information applied to all frames of the video segment among the one or more pieces of enlarged DNN information.

[0432] According to an embodiment of the present disclosure, AI data may include frame group AI data or frame AI data and video segment AI data. When frame group AI data or frame AI data is included in AI data together with video segment AI data, video segment AI data may indicate one or more pieces of enlarged DNN information among multiple pieces of default DNN configuration information. In addition, frame group AI data or frame AI data may indicate enlarged DNN information applied to frame groups or frames among the one or more pieces of enlarged DNN information.

[0433] According to an embodiment of the present disclosure, AI data may include all of video AI data, video segment AI data, frame group AI data, and frame AI data. When AI data includes all of video AI data, video segment AI data, frame group AI data, and frame AI data, the video AI data may indicate one or more pieces of enlarged DNN information applicable to the video among multiple pieces of default DNN configuration information. In addition, the video segment AI data may indicate one or more pieces of enlarged DNN information applicable to the video segment among the one or more pieces of enlarged DNN information applicable to the video. In addition, the frame group AI data may indicate one or more pieces of enlarged DNN information applicable to the frame group among the one or more pieces of enlarged DNN information applicable to the video segment. In addition, the frame AI data may indicate the enlarged DNN information applied to the frame among the one or more pieces of enlarged DNN information applicable to the frame group. According to the above-mentioned embodiment of the present disclosure, one of the video AI data, video segment AI data, frame group AI data, and frame AI data may be removed from the above-mentioned hierarchical AI data structure.

[0434] According to an embodiment of the present disclosure, the AI ​​data may include AI target data indicating whether AI upscaling is to be applied. In addition, when AI upscaling is applied, the AI ​​data may include AI supplementary data regarding upscaling DNN information corresponding to downscaling DNN information for AI downscaling.

[0435] The video AI data may include video AI target data indicating whether AI upscaling will be applied to the image data and video AI supplemental data regarding one or more upscaling DNN information applicable to AI upscaling of the frames included in the image data. The video AI target data may be determined based on whether AI upscaling will be applied to the image data. In addition, the video AI supplemental data may be determined based on one or more upscaling DNN information of the frames included in the image data.

[0436] The video segment AI data may include video segment AI target data indicating whether AI upscaling is to be applied to the video segment and video segment AI supplementary data regarding one or more upscaling DNN information applicable to AI upscaling of frames included in the video segment. The video segment AI target data may be determined based on whether AI upscaling is to be applied to the video segment. The video segment AI supplementary data may be determined based on one or more upscaling DNN information of frames included in the video segment.

[0437] The frame group AI data may include frame group AI target data indicating whether AI amplification is to be applied to the frame group and frame group AI supplementary data regarding one or more pieces of amplification DNN information applicable to AI amplification of frames included in the frame group. The frame group AI target data may be determined based on whether AI amplification is to be applied to the frame group. In addition, the frame group AI supplementary data may be determined based on one or more pieces of amplification DNN information applicable to AI amplification of frames included in the frame group.

[0438] The frame AI data may include frame AI target data indicating whether AI upscaling is to be applied to the frame and frame AI supplemental data regarding upscaling DNN information for AI upscaling of the frame.

[0439] The frame AI target data may be determined based on whether AI upscaling is to be applied to the current frame. The frame AI supplemental data may be determined according to upscaling DNN information for AI upscaling of the current frame.

[0440] According to an embodiment of the present disclosure, the video segment AI data may include video segment AI supplementary data related information indicating whether the video segment AI supplementary data is the same between the consecutive previous video segments and the current video segment. The video segment AI supplementary data related information is determined based on whether the video segment AI supplementary data is the same between the consecutive previous video segments and the current video segment.

[0441] In addition, the frame group AI data may include frame group AI supplementary data related information indicating whether the frame group AI supplementary data is the same between the consecutive previous frame group and the current frame group. The frame group AI supplementary data related information is determined based on whether the frame group AI supplementary data is the same between the consecutive previous frame group and the current frame group.

[0442] In addition, the frame AI data may include frame AI supplementary data related information indicating whether the frame AI supplementary data is the same between the consecutive previous frames and the current frame. The frame AI supplementary data related information is determined based on whether the frame AI supplementary data is the same between the consecutive previous frames and the current frame.

[0443] According to an embodiment of the present disclosure, the AI ​​encoded data may include synchronization data about synchronization of image data and AI data. The synchronization data may include data about synchronization of image data and video AI data, synchronization of video segments and video segment AI data, synchronization of frame groups and frame group AI data, and synchronization of current frame and frame AI data.

[0444] According to an embodiment of the present disclosure, the AI ​​data may include channel information indicating the color channel of the AI ​​amplification applied to the current frame. In addition, the amplification DNN information may be determined for the color channel indicated by the channel information.

[0445] According to an embodiment of the present disclosure, the AI ​​data may include at least one of target bit rate information indicating a bit rate of a low-resolution image or resolution information indicating a resolution of a high-resolution image.

[0446] In operation 2340 , image data is obtained by encoding the low resolution image of the current frame. The obtaining of the image data according to operation 2340 may be performed by the first encoder 614 .

[0447] In operation 2350, AI encoding data including image data and AI data is generated. The generation of AI encoding data according to operation 2350 may be performed by the data processor 632. According to an embodiment of the present disclosure, the image data and the AI ​​data may not be included in a single file but may be configured as separate files.

[0448] In operation 2360, a video file including a media data box in which the image data of the generated AI coded data is embedded and a metadata box is output. The output of the AI ​​coded data according to operation 2360 may be performed by the communicator 634.

[0449] With reference Figures 12 to 21 The AI ​​zoom-in and AI zoom-out functions described can be applied to Fig.23 image coding method.

[0450] Fig.24 is a block diagram showing the configuration of an image decoding device according to an embodiment of the present disclosure.

[0451] Reference Fig.24, the image decoding device 2400 according to an embodiment of the present disclosure may include a communicator 2410, a processor 2420 and a memory 2430.

[0452] The communicator 2410 may receive AI encoded data. Alternatively, the communicator 2410 may receive AI data and image data from an external device (eg, a server) according to the control of the processor 2420.

[0453] The processor 2420 may generally control the image decoding apparatus 2400. The processor 2420 according to an embodiment of the present disclosure may execute one or more programs stored in the memory 2430. The processor 2420 may perform the functions of the first decoder 232 and the AI ​​amplifier 234. The processor 2420 may be configured as one or more general-purpose processors.

[0454] According to an embodiment of the present disclosure, the processor 2420 may include a graphics processor 2422 and an AI dedicated processor 2424. In addition, the processor 2420 may be implemented in the form of a system on chip (SoC) integrating at least one of the graphics processor 2422 or the AI ​​dedicated processor 2424. The processor 2420 controls the overall operation of the image decoding device 2400 and the signal flow between the internal components of the image decoding device 2400, and processes data.

[0455] The graphics processor 2422 is a processor specially designed for decoding and post-processing of images. Therefore, the graphics processor 2422 can process the image data received by the image decoding device 2400 and effectively perform the low-resolution image reconstruction function of the first decoder 232. The AI ​​dedicated processor 2424 is a processor specially designed for AI operations. Therefore, the AI ​​dedicated processor 2424 can effectively perform the low-resolution image AI amplification function of the AI ​​amplifier 234.

[0456] The image data and AI data input to the image decoding device 2400 through the communicator 2410 are processed by the processor 2420. When the operation of decoding the image data is performed by the graphics processor 2422, a low-resolution image may be generated. In addition, the operation of AI enlargement of the low-resolution image may be performed by the AI ​​dedicated processor 2424. Therefore, when the AI ​​dedicated processor 2424 performs AI enlargement on the low-resolution image, a high-resolution image may be generated.

[0457] According to an embodiment of the present disclosure, the processor 2420 is described as including a graphics processor 2422, but may include one or more graphics processors 2422. In addition, according to an embodiment of the present disclosure, the processor 2420 is described as including an AI dedicated processor 2424, but may include one or more AI dedicated processors 2424. In addition, the processor 2420 may include one or more general-purpose processors. Additional processing required for AI amplification may be performed according to the one or more general-purpose processors. The AI ​​dedicated processor 2424 may be implemented as a hardware field programmable gate array (FPGA).

[0458] The memory 2430 according to an embodiment of the present disclosure may include various types of data, programs or applications for driving and controlling the image decoding device 2400. The program stored in the memory 2430 may include one or more instructions. In addition, the program (one or more instructions) or application stored in the memory 2430 may be executed by the processor 2420. The memory 2430 may store data from the communicator 2410 and the processor 2420. In addition, the memory 2430 may send data required by the processor 2420 to the processor 2420.

[0459] The image decoding device 2400 according to an embodiment of the present disclosure may execute Figure 2 The functions of the AI ​​decoding device 200 described in Fig. 22 At least one of the operations of the image decoding method described in.

[0460] Fig.25 is a block diagram showing the configuration of an image encoding device according to an embodiment of the present disclosure.

[0461] Reference Fig.25 , the image encoding device 2500 according to an embodiment of the present disclosure may include a communicator 2510, a processor 2520 and a memory 2530.

[0462] The processor 2520 according to an embodiment of the present disclosure may generally control the image encoding apparatus 2500. The processor 2520 according to an embodiment of the present disclosure may execute one or more programs stored in the memory 2530. The processor 2520 may perform the functions of the AI ​​downsizer 612 and the first encoder 614. The processor 2520 may be configured as one or more general-purpose processors.

[0463] According to an embodiment of the present disclosure, the processor 2520 may include a graphics processor 2522 and an AI dedicated processor 2524. In addition, the processor 2520 may be implemented in the form of an SoC integrating at least one of the graphics processor 2522 or the AI ​​dedicated processor 2524. The processor 2520 controls the overall operation of the image encoding device 2500 and the signal flow between the internal components of the image encoding device 2500, and processes data.

[0464] The graphics processor 2522 is a processor specially designed for encoding and post-processing of images. Therefore, the graphics processor 2522 can effectively perform the low-resolution image encoding function of the first encoder 614. The AI ​​dedicated processor 2524 is a processor specially designed for AI operations. Therefore, the AI ​​dedicated processor 2524 can effectively perform the high-resolution image AI reduction function of the AI ​​reducer 612. The AI ​​dedicated processor 2524 can be implemented as a hardware FPGA.

[0465] The AI ​​reduction of the high-resolution image and the encoding of the low-resolution image are performed by the processor 2520. When the operation for AI reduction of the high-resolution image is performed by the AI ​​dedicated processor 2524, a low-resolution image may be generated, and AI data required for AI enlargement of the low-resolution image may be generated by the AI ​​dedicated processor 2524. When the operation for encoding the low-resolution image is performed by the graphic processor 2522, image data may be generated.

[0466] The communicator 2510 may generate a single file of AI-encoded data including AI data and image data according to the control of the processor 2520. In addition, the communicator 2510 may output the single file of AI-encoded data to the outside of the image encoding device 2500 according to the control of the processor 2520. Alternatively, the communicator 2510 may generate a file including AI data and a file including image data, respectively, according to the control of the processor 2520. In addition, the communicator 2510 may output each of the file including AI data and the file including image data to the outside of the image encoding device 2500 according to the control of the processor 2520.

[0467] According to an embodiment of the present disclosure, the processor 2520 is described as including one graphics processor 2522, but may include one or more graphics processors 2522. In addition, according to an embodiment of the present disclosure, the processor 2520 is described as including one AI dedicated processor 2524, but may include one or more AI dedicated processors 2524. In addition, the processor 2520 may include one or more general-purpose processors. Additional processing required for AI amplification may be performed according to the one or more general-purpose processors.

[0468] The memory 2530 according to an embodiment of the present disclosure may include various types of data, programs or applications for driving and controlling the image encoding device 2500. The program stored in the memory 2530 may include one or more instructions. In addition, the program (one or more instructions) or application stored in the memory 2530 may be executed by the processor 2520. The memory 2530 may store a high-resolution image such as the original image 105. The memory 2530 may store data from the communicator 2510 and the processor 2520. In addition, the memory 2530 may send data required by the processor 2520 to the processor 2520.

[0469] The image encoding device 2500 according to an embodiment of the present disclosure may execute Figure 7 The functions of the AI ​​encoding device 600 described in Fig.23 At least one of the operations of the image encoding method described in.

[0470] In addition, the above-described embodiments of the present disclosure may be written as computer-executable programs or instructions that may be stored in a medium.

[0471] The medium may store computer executable programs or instructions continuously, or temporarily store computer executable programs or instructions for execution or downloading. In addition, the medium may be any of a variety of recording media or storage media combined with a single or multiple pieces of hardware, and the medium is not limited to a medium directly connected to a computer system, but may be distributed over a network. Examples of media include magnetic media (such as hard disks, floppy disks, and magnetic tapes) configured to store program instructions, optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as floppy disks, as well as ROM, RAM, and flash memory). Other examples of media include recording media and storage media managed by application stores that distribute applications or by websites, servers, etc. that supply or distribute various other types of software.

[0472] In addition, the model related to the above-mentioned DNN can be implemented via a software module. When the DNN model is implemented via a software module (eg, a program module including instructions), the DNN model can be stored in a computer-readable recording medium.

[0473] In addition, the DNN model can be integrated in the form of a hardware chip to become part of the above-mentioned AI decoding device 200 or AI encoding device 600. For example, the DNN model can be manufactured in the form of a dedicated hardware chip for AI, or can be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU).

[0474] In addition, the DNN model can be provided in the form of downloadable software. The computer program product may include a product in the form of a software program that is electronically distributed through a manufacturer or an electronic market (e.g., a downloadable application). For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a server or relay server of the manufacturer or electronic market.

[0475] Although one or more embodiments of the present disclosure have been described with reference to the drawings, it will be understood by those skilled in the art that various changes in form and details may be made in the present disclosure without departing from the spirit and scope defined by the following claims.

Claims

1. A computer-readable recording medium storing a program for executing an image encoding method in a computer, in, A video file including artificial intelligence AI encoded data is generated by the image encoding method, The AI ​​coded data includes: Image data, including encoded information of a reduced image generated by performing AI reduction on an original image; and AI data relating to an AI enlargement of the reduced image reconstructed from the image data, The AI ​​data includes: AI target data indicating whether AI upscaling is to be applied to at least one frame of the reduced image; and when AI upscaling is applied to the at least one frame of the reduced image, AI supplementary data related to upscaling DNN configuration information for AI upscaling of the at least one frame of the reduced image among a plurality of preset default deep neural network DNN configuration information, The AI ​​supplementary data includes channel information indicating a color channel to which AI amplification is applied, target bit rate information indicating a bit rate of the image data, and at least one of resolution information related to a resolution of the enlarged image, wherein the amplification DNN configuration information for AI amplification of at least one frame of the reduced image is obtained based on the at least one information.

2. The computer-readable recording medium according to claim 1, wherein: The AI ​​target data includes at least one of the following data: video AI target data indicating whether AI upscaling is to be applied to frames included in the image data; Video clip AI target data indicating whether AI upscaling will be applied to frames included in the video clip; Frame group AI target data indicating whether AI upscaling is to be applied to frames included in the frame group; or Frame AI target data indicating whether AI upscaling will be applied to the current frame.

3. The computer-readable recording medium according to claim 1, wherein: The AI ​​supplementary data includes at least one of the following data: video AI supplemental data relating to at least one piece of upscaling DNN configuration information commonly applied to AI upscaling of frames included in the image data; video segment AI supplemental data relating to at least one piece of upscaling DNN configuration information commonly applied to AI upscaling of frames included in the video segment; frame group AI supplementary data relating to at least one piece of upscaling DNN configuration information commonly applied to AI upscaling of frames included in the frame group; or Frame AI supplemental data, related to the upscaling DNN configuration information for the AI ​​upscaling applied to the current frame.

4. The computer-readable recording medium according to claim 1, wherein: The video file includes: a media data frame, comprising the image data; and a metadata box including metadata about the image data, Wherein, the AI ​​data is included in the metadata box.

5. The computer-readable recording medium according to claim 4, wherein: The metadata box includes synchronization data regarding synchronization of the image data and the AI ​​data, and The AI ​​data is configured to be decoded according to a reproduction order or a decoding order of frames of the image data based on the synchronization data.

6. The computer-readable recording medium according to claim 1, wherein: The AI ​​data is embedded in the image data, AI-upscaled video AI data about the image data is located in the image data to be decoded together with video encoding parameters commonly applied to frames of the image data, AI data on the AI ​​upscaled video segment of the current video segment including the current frame are located in the image data to be decoded together with video segment encoding parameters commonly applied to the frames of the current video segment, AI upscaled frame group AI data about a current frame group including the current frame is located in the image data to be decoded together with frame group parameters commonly applied to frames of the current frame group, and The AI-upscaled frame AI data about the current frame is located in the image data to be decoded together with the encoding information of the current frame.

7. The computer-readable recording medium according to claim 6, wherein: The image data includes: A video header, including video encoding parameters of the image data; A video segment header, including video segment encoding parameters of the current video segment; A frame group header, including frame group coding parameters of the current frame group; and Frame header, including the encoding information of the current frame, The video AI data is included in the video header or immediately before or after the video header, The video segment AI data is included in the video segment header or immediately before or after the video segment header, The frame group AI data is included in the frame group header or immediately before or after the frame group header, and The frame AI data is included in the frame header or immediately before or after the frame header.

8. The computer-readable recording medium according to claim 1, wherein: The video file includes: a metadata box including metadata about the image data; and a plurality of video segment data frames, in which the AI ​​encoded data is divided according to specific time, Wherein, each video segment data frame in the plurality of video segment data frames comprises: A segment media data frame, including video segment data; and a segment metadata box comprising metadata about the video segment data, and The AI ​​data includes AI-enlarged video AI data applied to all frames of the image data and AI-enlarged video segment AI data applied to all frames of the video segment, The video AI data is included in the metadata box, and the video segment AI data is included in the segment metadata box.

9. The computer-readable recording medium according to claim 3, wherein: The AI ​​data includes at least one of the following data: Information related to the video segment AI supplementary data, indicating whether the video segment AI supplementary data is the same between the current video segment and the consecutive previous video segment, Frame Group AI Supplemental Data related information, indicating whether the Frame Group AI Supplemental Data is the same between the current frame group and the consecutive previous frame group, or Frame AI supplementary data related information, indicating whether the frame AI supplementary data is the same between the current frame and the consecutive previous frames, When the video segment AI supplementary data related information indicates that the video segment AI supplementary data is the same between the current video segment and the consecutive previous video segment, omitting the video segment AI supplementary data about the current video segment from the AI ​​data, when the frame group AI supplementary data related information indicates that the frame group AI supplementary data is the same between the current frame group and the consecutive previous frame group, omitting the frame group AI supplementary data regarding the current frame group from the AI ​​data, and When the frame AI supplementary data related information indicates that the frame AI supplementary data is the same between the current frame and the consecutive previous frame, the frame AI supplementary data regarding the current frame is omitted from the AI ​​data.

10. A video decoding method using artificial intelligence AI amplification, the video decoding method comprising: receiving a video file including AI-encoded data, wherein the AI-encoded data includes image data and AI-enlarged AI data regarding the image data; Obtain the AI ​​data of the AI ​​coded data from a metadata box of the video file, and obtain the image data of the AI ​​coded data from a media data box of the video file; Reconstructing a reduced image of the current frame by decoding the image data; Obtaining the enlarged deep neural network DNN configuration information of the current frame from the AI ​​data; and Generate an enlarged image corresponding to the reduced image by performing AI enlargement on the reduced image according to the enlarged DNN configuration information of the current frame, The AI ​​data includes: AI target data indicating whether AI upscaling is to be applied to at least one frame of the downscaled image, and when AI upscaling is applied to the at least one frame of the reduced image, AI supplementary data related to upscaling DNN configuration information for AI upscaling of the at least one frame of the reduced image among a plurality of preset default DNN configuration information, The AI ​​supplementary data includes channel information indicating a color channel to which AI amplification is applied, target bit rate information indicating a bit rate of the image data, and at least one of resolution information related to a resolution of the enlarged image, wherein the amplification DNN configuration information for AI amplification of at least one frame of the reduced image is obtained based on the at least one information.

11. The video decoding method according to claim 10, wherein: The AI ​​data includes at least one of the following AI data: AI-enlarged frame AI data about the current frame, AI-enlarged frame group AI data about the frame group including the current frame, AI-enlarged video segment AI data about the video segment including the current frame, or AI-enlarged video AI data about the video including the current frame.

12. The video decoding method according to claim 10, The step of obtaining the amplified DNN configuration information includes: determining whether AI upscaling is to be applied to the at least one frame based on the AI ​​target data; and When it is determined that AI amplification is applied to the at least one frame, at least one piece of amplification DNN configuration information applied to the at least one frame is determined according to the AI ​​supplementary data.

13. The video decoding method according to claim 10, wherein: The step of obtaining the amplified DNN configuration information includes: obtaining two or more amplified DNN configuration information of the at least one frame from multiple default DNN configuration information according to at least one of the target bit rate information or the resolution information.

14. The video decoding method according to claim 10, wherein: The metadata box of the video file includes: synchronization data regarding synchronization of said image data and said AI data, and The steps of obtaining the AI ​​data include: According to the encoding order or the reproduction order of the image data and the AI ​​data indicated by the synchronization data, the AI ​​data corresponding to the current frame is obtained.

15. A video encoding method using artificial intelligence AI for downscaling, the video encoding method comprising: Determine the downscaling deep neural network (DNN) configuration information for downscaling the original image AI of the current frame into a downscaled image; Generate the reduced image of the current frame by performing AI reduction on the original image of the current frame according to the reduced DNN configuration information; Generate AI data about the enlarged DNN configuration information corresponding to the reduced DNN configuration information, wherein the enlarged DNN configuration information is used to perform AI enlargement on the reduced image of the current frame; Obtaining image data by encoding the reduced image of the current frame; generating AI coded data including the image data and the AI ​​data; and outputting a video file including a media data box and a metadata box, wherein the image data of the AI ​​coded data is embedded in the media data box, and the AI ​​data of the AI ​​coded data is embedded in the metadata box, The AI ​​data includes: AI target data indicating whether AI upscaling is to be applied to at least one frame of the downscaled image, and when AI upscaling is applied to the at least one frame of the reduced image, AI supplementary data related to upscaling DNN configuration information for AI upscaling of the at least one frame of the reduced image among a plurality of preset default DNN configuration information, The AI ​​supplementary data includes channel information indicating a color channel to which AI amplification is applied, target bit rate information indicating a bit rate of the image data, and at least one of resolution information related to a resolution of the enlarged image, wherein the amplification DNN configuration information for AI amplification of at least one frame of the reduced image is obtained based on the at least one information.

16. The video encoding method according to claim 15, wherein: The AI ​​data corresponds to AI-enlarged frame AI data about the current frame, AI-enlarged frame group AI data about the frame group including the current frame, AI-enlarged video segment AI data about the video segment including the current frame, or AI-enlarged video AI data about the video including the current frame.

17. The video encoding method according to claim 15, wherein: The steps of generating the AI ​​data include: determining the AI ​​target data based on whether AI upscaling is to be applied to the at least one frame; and The AI ​​supplementary data is determined according to the up-scaling DNN configuration information of the at least one frame to which AI up-scaling is applied.

18. The video encoding method according to claim 15, wherein: The step of generating the AI ​​data includes: determining two or more pieces of enlarged DNN configuration information of the at least one frame from a plurality of pieces of default DNN configuration information based on at least one of the target bit rate information or the resolution information.

19. The video encoding method according to claim 15, wherein: The metadata box of the video file includes synchronization data regarding synchronization of the AI ​​data and the image data, and The step of generating the AI ​​data includes determining the synchronization data according to an encoding order or a reproduction order of the image data and the AI ​​data.

20. A video decoding device for performing a video decoding method using artificial intelligence (AI) amplification, the video decoding device comprising: a communicator configured to receive a video file including AI-encoded data, wherein the AI-encoded data includes image data and AI-amplified AI data regarding the image data; a parser configured to obtain the AI ​​data of the AI ​​coded data from a metadata box of the video file, and to obtain the image data of the AI ​​coded data from a media data box of the video file; A first decoder configured to reconstruct a reduced image of a current frame by decoding the image data; and The AI ​​amplifier is configured to obtain the magnification deep neural network DNN configuration information of the current frame from the AI ​​data, and generate an enlarged image corresponding to the reduced image by performing AI magnification on the reduced image according to the magnification DNN configuration information of the current frame, The AI ​​data includes: AI target data indicating whether AI upscaling is to be applied to at least one frame of the downscaled image, and when AI upscaling is applied to the at least one frame of the reduced image, AI supplementary data related to upscaling DNN configuration information for AI upscaling of the at least one frame of the reduced image among a plurality of preset default DNN configuration information, The AI ​​supplementary data includes channel information indicating a color channel to which AI amplification is applied, target bit rate information indicating a bit rate of the image data, and at least one of resolution information related to a resolution of the enlarged image, wherein the amplification DNN configuration information for AI amplification of at least one frame of the reduced image is obtained based on the at least one information.

21. A video encoding device for performing a video encoding method for downscaling using artificial intelligence (AI), the video encoding device comprising: an AI downscaling device configured to determine downscaling deep neural network (DNN) configuration information for AI downscaling an original image of a current frame into a downscaled image, generate the downscaled image of the current frame by performing AI downscaling on the original image of the current frame, and generate AI data for AI upscaling the downscaled image of the current frame; a first encoder configured to determine image data by encoding the reduced image of the current frame; a data processor configured to determine AI coded data including the image data and the AI ​​data; and a communicator configured to output a video file including a media data box and a metadata box, wherein the image data of the AI ​​encoded data is embedded in the media data box, and the AI ​​data of the AI ​​encoded data is embedded in the metadata box, The AI ​​data includes: AI target data indicating whether AI upscaling is to be applied to at least one frame of the downscaled image, and when AI upscaling is applied to the at least one frame of the reduced image, AI supplementary data related to upscaling DNN configuration information for AI upscaling of the at least one frame of the reduced image among a plurality of preset default DNN configuration information, The AI ​​supplementary data includes channel information indicating a color channel to which AI amplification is applied, target bit rate information indicating a bit rate of the image data, and at least one of resolution information related to a resolution of the enlarged image, wherein the amplification DNN configuration information for AI amplification of at least one frame of the reduced image is obtained based on the at least one information.

Images