AI encoding device and its operation method and AI decoding device and its operation method

Abstract

An artificial intelligence (AI) encoding device includes: at least one processor configured to: determine a downscaling target based on a target resolution of a first image; obtain a first image by performing AI downscaling on an original image using an AI downscaling neural network corresponding to the downscaling target; generate image data by encoding the first image; select AI upscaling neural network setting identification information based on the target resolution of the first image, characteristic information of the original image, and target detail intensity; generate AI data including the target resolution of the first image, bit depth information of the first image, AI upscaling neural network setting identification information, and encoding control information; and generate AI encoded data including image data and AI data; and a communication interface configured to send the AI ​​encoded data to an AI decoding device, wherein the AI ​​data includes information about the AI ​​upscaling neural network corresponding to the AI ​​downscaling neural network.

Description

Technical Field

[0001] This disclosure relates to an artificial intelligence (AI) encoding device, an operating method of the AI ​​encoding device, an AI decoding device, and an operating method of the AI ​​decoding device. More specifically, this disclosure relates to an AI encoding device capable of performing AI downscaling and encoding on an original image and sending the results of AI downscaling and encoding to an AI decoding device, an operating method of the AI ​​encoding device, an AI decoding device capable of decoding and performing AI upscaling on an image received from the AI ​​encoding device, and an operating method of the AI ​​decoding device. Background Technology

[0002] Images can be encoded using a codec that follows a specified data compression standard (e.g., the Moving Picture Experts Group (MPEG) compression standard), and then stored on a recording medium or transmitted as a bitstream using a communication channel.

[0003] With the development and availability of hardware for reproducing and storing high-resolution / high-quality images, the demand for codecs capable of efficiently encoding and decoding high-resolution / high-quality images is increasing. Summary of the Invention

[0004] Technical issues

[0005] An AI encoding device is provided that can perform AI downscaling on images and send the AI ​​data required for AI upscaling and the encoded image data to an AI decoding device, as well as a method for operating the AI ​​encoding device.

[0006] In addition, an AI decoding device is provided that can receive image data and AI data from an AI encoding device, decode the image data, and perform AI upscaling on the decoded image using an upscaling neural network model corresponding to the AI ​​data, as well as a method for operating the AI ​​decoding device.

[0007] Technical solutions to the problem

[0008] According to one aspect of this disclosure, an artificial intelligence (AI) encoding apparatus includes: at least one processor configured to: determine a downscaling target based on a target resolution of a first image; obtain a first image by performing AI downscaling on an original image using an AI downscaling neural network corresponding to the downscaling target; generate image data by encoding the first image; select AI upscaling neural network setting identification information based on the target resolution of the first image, characteristic information of the original image, and target detail intensity; generate AI data including the target resolution of the first image, bit depth information of the first image, AI upscaling neural network setting identification information, and encoding control information; and generate AI encoded data including image data and AI data; and a communication interface configured to send the AI ​​encoded data to an AI decoding apparatus, wherein the AI ​​data includes information about the AI ​​upscaling neural network corresponding to the AI ​​downscaling neural network.

[0009] The at least one processor can also be configured to determine the quality information of the original image by using a quality measurement network, and to determine the target detail intensity based on the quality information of the original image.

[0010] The quality information indicates at least one of high quality, low quality, or noise quality, and the quality measurement network can be configured to extract features representing at least one of detail, noise, or contrast of the original image, and determine the quality information based on the extracted features.

[0011] The AI ​​data may also include decoding control information, which includes the update method of the AI ​​upscaling neural network and the AI ​​upscaling ratio.

[0012] The encoding control information may include at least one of quantization level or picture type, wherein the quantization level may be determined to a preset first value, and the picture type may be determined to be an intra-frame (I) picture, a predictive (P) picture, or a bidirectional (B) picture.

[0013] The AI-encoded data may also include a metadata box and a media data box. The AI ​​data can be included in the metadata box, and the image data can be included in the media data box.

[0014] This AI data can be included in the Supplemental Enhancement Information (SEI) region, which is an additional information region of the image data.

[0015] According to one aspect of this disclosure, an AI decoding apparatus includes: a communication interface configured to receive AI encoded data generated by AI downscaling and encoding an original image; and at least one processor configured to: divide the AI ​​encoded data into image data and AI data; decode the image data to obtain a second image corresponding to a first image obtained by downscaling the original image; extract the resolution of the first image, bit depth information of the first image, AI upscaling neural network setting identification information, and first encoding control information from the AI ​​data; obtain second encoding control information for encoding the first image; update the first encoding control information based on the second encoding control information; select an AI upscaling neural network based on the resolution of the original image, the bit depth information of the original image, the AI ​​upscaling neural network setting identification information, and the second encoding control information; and perform AI upscaling on the second image using the selected AI upscaling neural network.

[0016] The first encoding control information may include an initial value for a preset quantization level, and the second encoding control information may include a quantization level value applied to encode the first image.

[0017] The at least one processor may also be configured to: select neural network setting information from multiple neural network setting information corresponding to AI upscaling neural network setting identification information based on the resolution of the first image, the bit depth information of the first image, and the quantization level value included in the second encoding control information, and set the parameters of the AI ​​upscaling neural network using the selected neural network setting information.

[0018] The at least one processor can also be configured to determine the update information and upsampling rate of the AI ​​upscaling neural network from the AI ​​data, and to perform AI upscaling on the second image based on the selected AI upscaling neural network, the update information, and the upsampling rate.

[0019] According to one aspect of this disclosure, an operation method of an AI encoding device includes: determining a downscaling target based on the target resolution of a first image; obtaining a first image by performing AI downscaling on the original image using an AI downscaling neural network corresponding to the downscaling target; generating image data by encoding the first image; selecting AI upscaling neural network setting identification information based on the target resolution of the first image, characteristic information of the original image, and target detail intensity; generating AI data including the target resolution of the first image, bit depth information of the first image, AI upscaling neural network setting identification information, and encoding control information; generating AI encoded data including image data and AI data; and sending the AI ​​encoded data to an AI decoding device, wherein the AI ​​data represents information about the AI ​​upscaling neural network corresponding to the AI ​​downscaling neural network.

[0020] The method may also include using a quality measurement network to determine the quality information of the original image; and determining the target detail intensity based on the quality information of the original image.

[0021] The quality information can indicate at least one of high quality, low quality, or noise quality, and the quality measurement network can be configured to extract features representing at least one of detail, noise, or contrast of the original image, and determine the quality information based on the extracted features.

[0022] The AI ​​data may also include decoding control information, which includes the update method of the AI ​​upscaling neural network and the AI ​​upscaling ratio.

[0023] The encoding control information may include at least one of quantization level or picture type, wherein the quantization level may be determined to a preset first value, and the picture type may be determined to be an intra-frame (I) picture, a predictive (P) picture, or a bidirectional (B) picture.

[0024] According to one aspect of this disclosure, an operation method of an AI decoding apparatus includes: receiving AI encoded data generated by AI downscaling and encoding an original image; dividing the AI ​​encoded data into image data and AI data; obtaining a second image corresponding to a first image obtained by downscaling the original image by decoding the image data; extracting the resolution of the first image, bit depth information of the first image, AI upscaling neural network setting identification information, and first encoding control information from the AI ​​data; obtaining second encoding control information for encoding the first image; updating the first encoding control information based on the second encoding control information; selecting an AI upscaling neural network based on the resolution of the original image, the bit depth information of the original image, the AI ​​upscaling neural network setting identification information, and the second encoding control information; and performing AI upscaling on the second image by using the selected AI upscaling neural network.

[0025] The first encoding control information may include an initial value for a preset quantization level, and the second encoding control information may include a quantization level value applied to encode the first image.

[0026] Determining the AI ​​upscaling neural network may include selecting neural network setting information from multiple neural network setting information corresponding to AI upscaling neural network setting identification information based on the resolution of the first image, the bit depth information of the first image, and the quantization level value included in the second encoding control information; and setting the parameters of the AI ​​upscaling neural network using the selected neural network setting information.

[0027] The method may also include determining the update information and upsampling rate of the AI ​​upscaling neural network from the AI ​​data, and the AI ​​upscaling of the second image may include AI upscaling of the second image based on the selected AI upscaling neural network, update information and upsampling rate. Attached Figure Description

[0028] Figure 1 This is a diagram illustrating the artificial intelligence (AI) encoding and decoding processes according to an embodiment.

[0029] Figure 2 This is a diagram illustrating a method for training a first deep neural network (DNN) and a second DNN according to an embodiment.

[0030] Figure 3 This is a diagram illustrating the process of training a first DNN and a second DNN according to an embodiment.

[0031] Figure 4 This is a block diagram of the structure of an AI encoding device according to an embodiment.

[0032] Figure 5 and Figure 6 This is a diagram illustrating an AI data setting controller and an AI data generator according to an embodiment.

[0033] Figure 7 This is a block diagram of the structure of an AI decoding device according to an embodiment.

[0034] Figure 8 This is a diagram illustrating an AI data restorer and a model determiner according to an embodiment.

[0035] Figure 9 An example is shown where the model determiner determines the setup information for a second DNN to be used in AI upscaling.

[0036] Figure 10 It is a syntax table for AI data according to the embodiment.

[0037] Figure 11 It is a syntax table for AI data according to another embodiment.

[0038] Figure 12 An example of an AI decoding apparatus according to an embodiment is shown, comprising a decoding device and an AI upscaling device that are separate from each other.

[0039] Figure 13 The header and content structures of a Vendor Specific Infoframe (VSIF) packet according to an embodiment are shown.

[0040] Figure 14The header and content structures of a VSIF packet according to another embodiment are shown.

[0041] Figure 15 This is a flowchart of the operation method of the AI ​​encoding device according to an embodiment.

[0042] Figure 16 This is a flowchart of the operation method of the AI ​​decoding device according to an embodiment. Detailed Implementation

[0043] Throughout the disclosure, the phrase "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0044] Because this disclosure allows for various changes and numerous examples, specific embodiments will be shown in the accompanying drawings and described in detail in the written description. However, this is not intended to limit this disclosure to a particular mode of practice, and it will be understood that all changes, equivalents, and substitutions without departing from the spirit and technical scope of this disclosure are included herein.

[0045] In the description of the embodiments, certain detailed explanations of the related technologies are omitted where it is believed that such detailed explanations might unnecessarily obscure the essence of this disclosure. Furthermore, the ordinal numbers (e.g., first, second, etc.) used in the description are merely identifier codes used to distinguish one element from another.

[0046] Furthermore, it should be understood in this specification that when elements are “connected” or “coupled” to each other, the elements may be directly connected or coupled to each other, but may also be connected or coupled to each other through intermediate elements, unless otherwise described.

[0047] In this specification, embodiments may be described and illustrated based on blocks that perform one or more of the functions described herein. These blocks, which may be referred to herein as “units” or “modules” or given names such as drivers, controllers, devices, etc., may be physically implemented using analog or digital circuitry (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, etc.) and may be driven by firmware and software. For example, the circuitry may be implemented in one or more semiconductor chips or on a substrate support such as a printed circuit board. The circuitry included in a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware performing some functions of the block and a processor performing other functions of the block. Each block of an embodiment may be physically divided into two or more discrete blocks that interact with each other. Similarly, the blocks of an embodiment may be physically combined into more complex blocks.

[0048] In this specification, regarding elements referred to as "units" or "modules," depending on their subdivided functions, two or more elements may be combined into one element, or one element may be divided into two or more elements. Furthermore, each element described below, in addition to performing its own primary function, may perform some or all of the functions performed by other elements, and some of the primary functions of each element may be entirely performed by other components.

[0049] In addition, in this specification, "image" or "picture" can mean a still image, a moving image including multiple consecutive still images (or frames), or a video.

[0050] Furthermore, in this specification, deep neural networks (DNNs) are representative examples of artificial neural network models that simulate brain nerves or brain-like nerves, and are not limited to artificial neural network models that use specific algorithms.

[0051] Furthermore, in this specification, "parameters" are values ​​used in the operations that form each layer of a neural network; for example, they may include weights used when input values ​​are applied to a specific operational expression. Here, parameters can be represented in matrix form. Parameters are values ​​set as training results and can be updated as needed using individual training data.

[0052] Furthermore, in this specification, "first DNN" refers to a DNN used for artificial intelligence (AI) downscaling of images, while "second DNN" refers to a DNN used for artificial intelligence upscaling of images.

[0053] Furthermore, in this specification, the "DNN setup information" includes the aforementioned parameters as information related to the elements constituting the DNN. The first DNN or the second DNN can be configured using the DNN setup information.

[0054] Furthermore, in this specification, "original image" refers to the image that is the object of AI encoding, while "first image" refers to the result of performing AI downscaling on the original image during the AI ​​encoding process. "Second image" refers to the image obtained through decoding during the AI ​​decoding process, while "third image" refers to the image obtained by performing AI upscaling on the second image during the AI ​​decoding process.

[0055] Furthermore, in this specification, "AI downscaling" refers to the process of reducing the resolution of an image based on AI, while "encoding" refers to the encoding process according to an image compression method based on frequency transform. Additionally, "first decoding" refers to the decoding process according to an image reconstruction method based on frequency transform, and "AI upscaling" refers to the process of increasing the resolution of an image based on AI.

[0056] Figure 1 This is a diagram illustrating the AI ​​encoding and AI decoding processes according to embodiments of the present disclosure.

[0057] As mentioned above, when the resolution of an image increases significantly, the throughput of information used to encode / decode the image increases. Therefore, a method is needed to improve the efficiency of image encoding and decoding.

[0058] like Figure 1 As shown, according to an embodiment of this disclosure, a first image 115 is obtained by performing AI downscaling 110 on an original image 105 with high resolution. Then, encoding 120 and decoding 130 are performed on the first image 115 with relatively low resolution, so the bit rate can be greatly reduced compared to performing encoding 120 and decoding 130 on the original image 105.

[0059] For example, refer to Figure 1 According to embodiments of this disclosure, during the AI ​​encoding process, a first image 115 is obtained by performing AI downscaling 110 on the original image 105, and encoding 120 is performed on the first image 115. During the AI ​​decoding process, AI-encoded data, including AI data and image data, obtained as a result of AI encoding, is received, a second image 135 is obtained by decoding 130, and a third image 145 is obtained by performing AI upscaling 140 on the second image 135.

[0060] Referring to the example AI encoding process, when the original image 105 is received, AI downscaling 110 is performed on the original image 105 to obtain a first image 115 with a specific resolution and / or a specific quality. AI downscaling 110 is performed based on AI, and for the second image 135, the AI ​​used for AI downscaling 110 can be jointly trained with the AI ​​used for AI upscaling 140. This is because when the AI ​​used for AI downscaling 110 and the AI ​​used for AI upscaling 140 are trained separately, the difference between the original image 105, the object of AI encoding, and the third image 145 reconstructed through AI decoding increases.

[0061] According to embodiments of this disclosure, AI data can be used to maintain this joint relationship during the AI ​​encoding and AI decoding processes. Therefore, the AI ​​data obtained through the AI ​​encoding process can include information indicating the upscaling target, and during the AI ​​decoding process, AI upscaling 140 can be performed on the second image 135 based on the upscaling target verified by the AI ​​data.

[0062] The AI ​​used for AI downscaling 110 and the AI ​​used for AI upscaling 140 can be implemented as DNNs. For ease of explanation, the DNN used for AI downscaling 110 will be referred to as the first DNN, and the DNN used for AI upscaling 140 will be referred to as the second DNN. (See later...) Figure 2 and Figure 3 As described, because the first DNN and the second DNN are jointly trained by sharing loss information under a specific target, the AI ​​encoding device can provide the target information used during the joint training of the first DNN and the second DNN to the AI ​​decoding device, and the AI ​​decoding device can perform AI upscaling 140 on the second image 135 based on the received target information to achieve target quality and / or resolution.

[0063] about Figure 1 The encoding 120 and decoding 130 can reduce the amount of information in the first image 115 obtained by performing AI downscaling 110 on the original image 105 through encoding 120. Encoding 120 can include a process of generating prediction data by predicting the first image 115, a process of generating residual data corresponding to the difference between the first image 115 and the prediction data, a process of transforming the residual data of the spatial domain components into frequency domain components, a process of quantizing the residual data transformed into frequency domain components, and a process of entropy coding on the quantized residual data. This encoding 120 can be performed using one of the image compression methods that use frequency transformation (such as MPEG-2, H.264 Advanced Video Codec (AVC), MPEG-4, High Efficiency Video Codec (HEVC), VC-1, VP8, VP9, ​​and AOMedia Video 1 (AV1)).

[0064] A second image 135 corresponding to the first image 115 can be reconstructed by performing decoding 130 on the image data. Decoding 130 may include a process of generating quantized residual data by entropy decoding of the image data, a process of inverse quantization of the quantized residual data, a process of transforming the residual data of the frequency domain components into spatial domain components, a process of generating prediction data, and a process of reconstructing the second image 135 using the prediction data and the residual data. This decoding 130 can be performed using an image reconstruction method corresponding to one of the image compression methods that use frequency transformation (such as MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 used in encoding 120).

[0065] The AI-encoded data obtained through the AI ​​encoding process may include image data obtained as a result of encoding 120 on the first image 115, and AI data related to AI upscaling 140 on the second image 135. The image data can be used during decoding 130, and the AI ​​data can be used during AI upscaling 140.

[0066] Image data can be transmitted in the form of a bitstream. The image data may include data obtained based on pixel values ​​in the first image 115, such as residual data as the difference between the first image 115 and its predicted data. Furthermore, the image data includes multiple pieces of information used during the encoding 120 of the first image 115. For example, the image data may include prediction mode information, motion information, and quantization parameter-related information used during encoding 120. The image data can be generated according to the rules (e.g., according to the syntax) of the image compression method used during encoding 120 in image compression methods using frequency transformation (such as MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1).

[0067] AI data is used for AI upscaling 140 based on the second DNN. As described above, because the first DNN and the second DNN are jointly trained, the AI ​​data includes information that enables the second DNN to accurately perform AI upscaling 140 on the second image 135. During the AI ​​decoding process, AI upscaling 140 can be performed on the second image 135 based on the AI ​​data to achieve a target resolution and / or quality.

[0068] AI data can be transmitted along with image data in bitstream form. According to embodiments, AI data can be transmitted separately from image data in frame or packet form. Alternatively, according to embodiments, AI data can be included in the image data and transmitted accordingly. Image data and AI data can be transmitted over the same network or different networks.

[0069] Now refer to Figure 2 Describe a method for jointly training a first DNN and a second DNN.

[0070] Figure 2 This is a diagram illustrating a method for training a first DNN 200 and a second DNN 300 according to embodiments of the present disclosure.

[0071] According to an embodiment, an AI decoding process is used to reconstruct a third image 145 from an original image 105 that has undergone AI encoding. Furthermore, to maintain the similarity between the third image 145 obtained as a result of AI decoding and the original image 105, connectivity between the AI ​​encoding and AI decoding processes is desired. In other words, information lost during AI encoding can be reconstructed during AI decoding, and for this purpose, a first DNN 200 (which may be a DNN for AI downscaling) and a second DNN 300 (which may be a DNN for AI upscaling) can be jointly trained.

[0072] For accurate AI decoding, ultimately, compared to... Figure 2 The quality loss information 230 corresponding to the results of the third training image 204 and the original training image 201 can be reduced. Therefore, the quality loss information 230 is used to train both the first DNN 200 and the second DNN 300.

[0073] First, the description Figure 2 The training process is shown.

[0074] exist Figure 2 In this diagram, the original training image 201 is the image to be downscaled using AI, while the first training image 202 is the image obtained by downscaling the original training image 201 using AI. Furthermore, the third training image 204 is the image obtained by upscaling the first training image 202 using AI.

[0075] The original training image 201 includes a still image or a moving image comprising multiple frames. According to an embodiment, the original training image 201 may include a brightness image extracted from the still image or the moving image comprising multiple frames. According to an embodiment, the original training image 201 may include a patch image extracted from the still image or the moving image comprising multiple frames. When the original training image 201 comprises multiple frames, the first training image 202 and the third training image 204 also each comprise multiple frames. When the multiple frames of the original training image 201 are sequentially input into the first DNN 200, the multiple frames of the first training image 202 and the third training image 204 can be obtained sequentially through the first DNN 200 and the second DNN 300.

[0076] For the joint training of the first DNN 200 and the second DNN 300, the original training image 201 is input into the first DNN 200. Using AI downscaling, the original training image 201 input into the first DNN 200 is output as the first training image 202, and the first training image 202 is input into the second DNN 300. The third training image 204 is output as the result of performing AI upscaling on the first training image 202.

[0077] refer to Figure 2 The first training image 202 is input to the second DNN 300. According to an embodiment, a second training image obtained by encoding and decoding the first training image 202 can be input to the second DNN 300. To input the second training image to the second DNN 300, any one of the following codecs can be used: MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1. For example, any one of these codecs can be used to encode the first training image 202 and decode the image data corresponding to the first training image 202.

[0078] refer to Figure 2 Unlike the first training image 202 output by the first DNN 200, a scaled-down training image 203 is obtained by performing conventional downscaling on the original training image 201. Here, conventional downscaling may include at least one of bilinear scaling, bicubic scaling, lanczos scaling, or stairstep scaling.

[0079] In order to prevent the structural features of the first image 115 from deviating significantly from the structural features of the original image 105, a scaled-down training image 203 is obtained to maintain the structural features of the original training image 201.

[0080] Before training, the first DNN 200 and the second DNN 300 can be configured with predetermined DNN settings. During training, structural loss information 210, complexity loss information 220, and quality loss information 230 can be determined.

[0081] Structural loss information 210 can be determined based on the result of comparing the scaled-down training image 203 and the first training image 202. For example, structural loss information 210 can correspond to the difference between the structural information of the scaled-down training image 203 and the structural information of the first training image 202. Structural information can include various features that can be extracted from the image, such as the image's brightness, contrast, histogram, etc. Structural loss information 210 indicates how much of the original training image 201's structural information is maintained in the first training image 202. When structural loss information 210 is small, the structural information of the first training image 202 is similar to the structural information of the original training image 201.

[0082] The complexity loss information 220 can be determined based on the spatial complexity of the first training image 202. For example, the total variance of the first training image 202 can be used as the spatial complexity. The complexity loss information 220 is related to the bit rate of the image data obtained by performing a first encoding on the first training image 202. When the complexity loss information 220 is small, the bit rate of the image data is low.

[0083] The quality loss information 230 can be determined based on the result of comparing the original training image 201 with the third training image 204. The quality loss information 230 may include at least one of the following: L1 norm value, L2 norm value, structural similarity (SSIM) value, peak signal-to-noise ratio-human vision system (PSNR-HVS) value, multiscale SSIM (MS-SSIM) value, variance inflation factor (VIF) value, or video multimethod assessment fusion (VMAF) value regarding the difference between the original training image 201 and the third training image 204. The quality loss information 230 indicates the degree of similarity between the third training image 204 and the original training image 201. When the quality loss information 230 is low, the third training image 204 is more similar to the original training image 201.

[0084] refer to Figure 2 Structural loss information 210, complexity loss information 220, and quality loss information 230 are used to train the first DNN 200, and quality loss information 230 is used to train the second DNN 300. In other words, quality loss information 230 is used to train the first DNN 200 and the second DNN 300.

[0085] The first DNN 200 can update its parameters to reduce or minimize the final loss information determined based on structural loss information 210, complexity loss information 220, and quality loss information 230. Furthermore, the second DNN 300 can update its parameters to reduce or minimize the quality loss information 230.

[0086] The final loss information used to train the first DNN 200 and the second DNN 300 can be determined by the following Equation 1.

[0087] [Equation 1]

[0088] LossDS=a*STRUCTURAL LOSS INFORMATION+b*COMPLEXITY LOSS INFORMATION+c*QUALITY LOSS INFORMATION

[0089] LossUS=d*QUALITY LOSS INFORMATION

[0090] In Equation 1, LossDS indicates that it will be reduced or minimized to train the final loss information of the first DNN 200, and LossUS indicates that it will be reduced or minimized to train the final loss information of the second DNN 300. Furthermore, a, b, c, and d can correspond to predetermined weights.

[0091] In other words, the first DNN 200 updates its parameters in the direction that reduces LossDS according to Equation 1, and the second DNN 300 updates its parameters in the direction that reduces LossUS. When the parameters of the first DNN 200 are updated based on the LossDS derived during training, the first training image 202 obtained based on the updated parameters becomes different from the previous first training image 202 obtained during previous training, and therefore, the third training image 204 also becomes different from the previous third training image 204 obtained during previous training. When the third training image 204 becomes different from the previous third training image 204, the quality loss information 230 is also redefined, and the second DNN 300 updates its parameters accordingly. When the quality loss information 230 is redefined, the LossDS is also redefined, so the first DNN 200 updates its parameters based on the newly defined LossDS. In other words, updating the parameters of the first DNN 200 leads to updating the parameters of the second DNN 300, and updating the parameters of the second DNN 300 leads to updating the parameters of the first DNN 200. In other words, because the first DNN 200 and the second DNN 300 are jointly trained by sharing the quality loss information 230, the parameters of the first DNN 200 and the second DNN 300 can be jointly optimized.

[0092] Referring to Equation 1, it is verified that LossUS is determined based on quality loss information 230, but this is only an example. LossUS can be determined based on at least one of structural loss information 210 or complexity loss information 220, as well as quality loss information 230.

[0093] The AI ​​encoding and decoding apparatus according to embodiments of the present disclosure can store multiple DNN setup information. A method for training multiple DNN setup information stored in the AI ​​encoding and decoding apparatus will be described.

[0094] As described with reference to Equation 1, the first DNN 200 updates its parameters based on the similarity between the structural information of the first training image 202 and the structural information of the original training image 201 (e.g., represented as structural loss information 210), the bit rate of the image data obtained as a result of performing the first encoding on the first training image 202 (e.g., represented as complexity loss information 220), and the difference between the third training image 204 and the original training image 201 (e.g., represented as quality loss information 220).

[0095] For example, the parameters of the first DNN 200 can be updated to obtain a first training image 202 with structural information similar to the original training image 201, and image data with a small bit rate can be obtained when encoding the first training image 202, and the second DNN 300 performing AI upscaling on the first training image 202 can also obtain a third training image 204 similar to the original training image 201.

[0096] The direction in which the parameters of the first DNN 200 are optimized varies by adjusting the weights a, b, and c in Equation 1. For example, when weight b is determined to be high, the parameters of the first DNN 200 can be updated by prioritizing the low bit rate over the high quality of the third training image 204. Furthermore, when weight c is determined to be high, the parameters of the first DNN 200 can be updated by prioritizing the high quality of the third training image 204 over the high bit rate or by maintaining the structural information of the original training image 201.

[0097] Furthermore, the direction in which the parameters of the first DNN 200 are optimized can vary depending on the type of codec used to encode the first training image 202. This is because the second training image to be input into the second DNN 300 can vary depending on the type of codec.

[0098] In other words, the parameters of the first DNN 200 and the second DNN 300 can be jointly updated based on the weights a, b, and c, and the type of the codec used to encode the first training image 202. Therefore, when the first DNN 200 and the second DNN 300 are trained after the weights a, b, and c are each determined to a specific value and the type of the codec is determined to a specific type, the parameters of the first DNN 200 and the second DNN 300, which are connected and optimized to each other, can be determined.

[0099] Furthermore, when training the first DNN 200 and the second DNN 300 after changing the weights a, b, and c, as well as the type of codec, the parameters of the first DNN 200 and the parameters of the second DNN 300, which are connected and optimized together, can be determined. In other words, when training the first DNN 200 and the second DNN 300 while changing the values ​​of the weights a, b, and c, as well as the values ​​of each type of codec, the settings information of multiple DNNs jointly trained together can be determined in the first DNN 200 and the second DNN 300.

[0100] According to embodiments of the present disclosure, multiple DNN setup information of the first DNN 200 and the second DNN 300 can be mapped to multiple pieces of information related to the first image 115. To establish this mapping, a first training image 202 output from the first DNN 200 can be encoded using a specific codec based on a specific bit rate, and a second training image obtained by decoding the bitstream obtained as a result of the encoding can be input to the second DNN 300. In other words, the DNN setup information pairs mapped to the resolution of the first training image 202, the type of codec used to encode the first training image 202, and the bit rate of the bitstream obtained as a result of encoding the first training image 202 can be determined by training the first DNN 200 and the second DNN 300 after setting up the environment, such that the first training image 202 with a specific resolution is encoded using a specific codec based on a specific bit rate. By varying the resolution of the first training image 202, the type of codec used to encode the first training image 202, and the bit rate of the bitstream obtained from the encoding of the first training image 202, the mapping relationship between multiple DNN setup information of the first DNN 200 and the second DNN 300 and multiple pieces of information related to the first image 115 can be determined.

[0101] Figure 3 This is a diagram illustrating the process of training a first DNN 200 and a second DNN 300 according to embodiments of the present disclosure.

[0102] The above references Figure 2 The training of the first DNN 200 and the second DNN 300 described herein can be performed by a training device 301. The training device 301 includes the first DNN 200 and the second DNN 300. The training device 301 can be, for example, an AI encoding device or a separate server. Multiple DNN configuration details of the second DNN 300 obtained as a training result are stored in an AI decoding device.

[0103] refer to Figure 3 In operations S310 and S320, the training device 301 initially sets the DNN configuration information of the first DNN 200 and the second DNN 300, respectively. Therefore, the first DNN 200 and the second DNN 300 can operate according to the predetermined DNN configuration information. The DNN configuration information may include information about at least one of the following: the number of convolutional layers included in the first DNN 200 and the second DNN 300; the number of filter kernels in each convolutional layer; the size of each filter kernel; or the parameters of each filter kernel.

[0104] In operation S330, the training device 301 inputs the original training image 201 into the first DNN 200. The original training image 201 may include at least one frame included in a still image or a moving image.

[0105] In operation S340, the first DNN 200 processes the original training image 201 according to the initially set DNN settings information and outputs the first training image 202 obtained by AI downscaling the original training image 201. Figure 3 In this process, the first training image 202 output from the first DNN 200 is directly input into the second DNN 300. However, the first training image 202 output from the first DNN 200 can also be input into the second DNN 300 via the training device 301. Furthermore, the training device 301 can use a specific codec to encode and decode the first training image 202, and then input the second training image into the second DNN 300.

[0106] In operation S350, the second DNN 300 processes the first training image 202 or the second training image according to the initially set DNN setting information, and outputs the third training image 204 obtained by AI upscaling of the first training image 202 or the second training image.

[0107] In operation S360, the training device 301 calculates complexity loss information 220 based on the first training image 202.

[0108] In operation S370, the training device 301 calculates structural loss information 210 by comparing the scaled-down training image 203 with the first training image 202.

[0109] In operation S380, the training device 301 calculates quality loss information 230 by comparing the original training image 201 with the third training image 204.

[0110] In operation S390, the first DNN 200 updates the initially set DNN settings information based on the final loss information using a backpropagation process. The training device 301 can calculate the final loss information used to train the first DNN 200 based on complexity loss information 220, structural loss information 210, and quality loss information 230.

[0111] In operation S395, the second DNN 300 updates the initially set DNN settings information based on the quality loss information 230 or the final loss information using a backpropagation process. The training device 301 can calculate the final loss information used to train the second DNN 300 based on the quality loss information 230.

[0112] Then, the training device 301, the first DNN 200, and the second DNN 300 repeat operations S330 to S395 until multiple final loss information is minimized to update the DNN setting information. During each repetition, the first DNN 200 and the second DNN 300 operate according to the DNN setting information updated in the previous operation.

[0113] Table 1 below shows the effects of performing AI encoding and AI decoding on the original image 105 according to embodiments of the present disclosure, as well as when performing encoding and decoding on the original image 105 using HEVC.

[0114] [Table 1]

[0115]

[0116] As shown in Table 1, even if the subjective image quality when performing AI encoding and AI decoding on content comprising 300 8K resolution frames according to the embodiments of this disclosure is higher than the subjective image quality when performing encoding and decoding using HEVC, the bit rate will decrease by 50% or more.

[0117] Figure 4 This is a block diagram illustrating the structure of an AI encoding apparatus 400 according to an embodiment of the present disclosure.

[0118] refer to Figure 4 The AI ​​encoding device 400 may include an AI encoder 401 and a transmitter 460. The AI ​​encoder 401 may include an AI downscaling unit 410, an AI data setting controller 430, an AI data generator 440, an encoder 420, and a stream generator 450.

[0119] exist Figure 4 In this design, the AI ​​encoder 401 and transmitter 460 are shown as separate devices. However, the AI ​​encoder 401 and transmitter 460 can be implemented using a single processor. In this case, the AI ​​encoder 401 and transmitter 460 can be implemented using a dedicated processor or a combination of software and a general-purpose processor (such as an application processor (AP), central processing unit (CPU), or graphics processing unit (GPU)). The dedicated processor may include memory for implementing the embodiments of this disclosure, or it may include a memory processor for using external memory.

[0120] Furthermore, the AI ​​encoder 401 and transmitter 460 can be implemented using multiple processors. In this case, the AI ​​encoder 401 and transmitter 460 can be implemented using a combination of dedicated processors or a combination of software and multiple general-purpose processors (such as AP, CPU, or GPU). According to an embodiment, the encoder 420 can be configured by a first processor, the AI ​​data setting controller 430, the AI ​​data generator 440, and the stream generator 450 can be implemented using a second processor different from the first processor, and the transmitter 460 can be implemented using a third processor different from the first and second processors.

[0121] AI encoder 401 can downscale the original image 105 and encode the first image 115. AI encoder 401 can obtain one or more input pieces of information and generate AI data based on the obtained input pieces of information. AI encoder 401 sends AI encoded data, including the image data and AI data obtained as a result of encoding, to transmitter 460. Transmitter 460 sends the AI ​​encoded data to AI decoding device.

[0122] AI downscaling unit 410 can obtain a first image 115 obtained by performing AI downscaling on the original image 105 through a first DNN. AI downscaling unit 410 can perform AI downscaling on the original image 105 by using target resolution information of the first image 115 corresponding to the downscaling target provided by AI data setting controller 430.

[0123] The AI ​​data setting controller 430 can determine the downscaling target of the original image 105 based on predetermined criteria. The downscaling target indication is used to indicate how much the resolution of the original image 105 should be reduced to obtain information for the first image 115. For example, the AI ​​data setting controller 430 can determine the downscaling ratio based on the difference between the resolution of the original image 105 and the target resolution information of the first image 115, and can send the determined downscaling ratio to the AI ​​downscaler 410. The AI ​​data setting controller 430 sets the downscaling target resolution of the first image 115 without setting the resolution information of the original image 105, and uses the set downscaling target resolution to determine the setup information of the second DNN to be used in the AI ​​upscaling 140.

[0124] To obtain a first image 115 that matches the downscaling target, the AI ​​downscaler 410 can store multiple DNN setting information that can be set in the first DNN. The AI ​​downscaler 410 can obtain DNN setting information corresponding to the downscaling ratio from the multiple DNN setting information, and can set the obtained DNN setting information to the first DNN.

[0125] Each of multiple DNN setup messages can be trained to obtain a first image 115 with predetermined resolutions and / or predetermined qualities. For example, any one of the multiple DNN setup messages may include multiple messages for obtaining a first image 115 with a resolution half that of the original image 105 (e.g., the first image 115 has a 2K resolution (e.g., 1920×1080), which is half that of the original image 105's 4K resolution (e.g., 3840×2160)). Another DNN setup message may include multiple messages for obtaining a first image 115 with a resolution one-quarter that of the original image 105 (e.g., the first image 115 has a 1K resolution (e.g., 960×540), which is one-quarter that of the original image 105's 4K resolution (e.g., 3840×2160)).

[0126] According to an embodiment, when multiple pieces of information included in the DNN setup information (which may be, for example, the number of convolutional layers, the number of filter kernels in each convolutional layer, the parameters of each filter kernel, etc.) are stored in the form of a lookup table, the AI ​​downscaling unit 410 can obtain the DNN setup information by combining some values ​​selected from the lookup table based on the downscaling target, and set the obtained DNN setup information in the first DNN.

[0127] According to an embodiment, the AI ​​downscaling unit 410 can determine the structure of the DNN corresponding to the downscaling target, and obtain DNN setting information corresponding to the determined DNN structure, for example, by obtaining the parameters of the filter kernel.

[0128] Because the first DNN and the second DNN are trained jointly, the multiple DNN settings used to perform AI downscaling on the original image 105 can have optimized values. Here, each DNN setting can include at least one of the following: the number of convolutional layers included in the first DNN, the number of filter kernels in each convolutional layer, or the parameters of each filter kernel.

[0129] AI downscaling unit 410 can set up a first DNN using DNN setup information determined for AI downscaling of the original image 105, thereby obtaining a first image 115 with a predetermined resolution and / or predetermined quality through the first DNN. When DNN setup information for performing AI downscaling on the original image 105 is obtained from multiple DNN setup information, each layer in the first DNN can process the input data based on the multiple pieces of information included in the DNN setup information.

[0130] AI downscaling unit 410 can send the first image 115 obtained by performing AI downscaling on the original image 105 through the first DNN to encoder 420.

[0131] Upon receiving the first image 115 from the AI ​​downscaling unit 410, the encoder 420 can reduce the information content of the first image 115 by encoding it according to a frequency-transform-based image compression method. Image data is obtained as a result of encoding using a predetermined codec (e.g., MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1). Image data is generated according to the rules (e.g., syntax) of the predetermined codec. The image data may include data obtained based on pixel values ​​in the first image 115, such as residual data as the difference between the first image 115 and its predicted data. Furthermore, the image data includes multiple pieces of information used during the encoding process of the first image 115. For example, the image data may include scene type information, prediction mode information, motion information, and quantization parameter-related information (e.g., quantization level) used to encode the first image 115.

[0132] The image data obtained as a result of encoding by encoder 420 is provided to stream generator 450.

[0133] The AI ​​data setting controller 430 can receive one or more input pieces of information. For example, reference Figure 5 The AI ​​data setting controller 430 can obtain the target resolution of the first image 115, the bit depth of the first image 115, the characteristic information of the original image (e.g., the input image), the target detail intensity, the encoding control information, the decoding control information, etc.

[0134] The AI ​​data setting controller 430 may also obtain at least one of the following as input information: the target bit rate of the image data, the bit rate type of the image data (e.g., variable bit rate type, constant bit rate type, or average bit rate type), the color format (e.g., the luminance component, chrominance component, red component, green component, or blue component to which AI downscaling has been applied), the type of codec used to encode the first image 115, or compression history information.

[0135] One or more input messages may include information pre-stored in the AI ​​encoding device 400 or information received from the user.

[0136] The AI ​​data setting controller 430 can control the operation of the AI ​​downscaling unit 410 based on input information. According to an embodiment, the AI ​​data setting controller 430 can determine a downscaling target based on the input information and can provide the determined downscaling target to the AI ​​downscaling unit 410.

[0137] According to an embodiment, the AI ​​data setting controller 430 can send at least a portion of the input information to the encoder 420, so that the encoder 420 can encode the first image 115 using a specific bit rate, a specific type of bit rate, and a specific codec.

[0138] According to an embodiment, the AI ​​data setting controller 430 can determine the downscaling target based on at least one of the compression ratio (such as the downscaling ratio, for example, the resolution difference between the original image 105 and the first image 115 or the target bit rate), compression quality (e.g., bit rate type), compression history information, or the type of the original image 105.

[0139] For example, the AI ​​data setting controller 430 can determine the downscaling target based on preset compression ratios, compression quality, etc., or received from the user.

[0140] As another example, the AI ​​data setting controller 430 can determine the downscaling target by using compression history information stored in the AI ​​encoding device 400. For example, based on the compression history information available to the AI ​​encoding device 400, the user's preferred encoding quality, compression ratio, etc., can be determined, and the downscaling target can be determined based on the encoding quality determined based on the compression history information. For example, the resolution, quality, etc. of the first image 115 can be determined based on the compression history information according to the most frequently used encoding quality.

[0141] As another example, the AI ​​data setting controller 430 can determine the downscaling target based on compression history information, based on the coding quality used more frequently than a predetermined threshold (e.g., the average quality of coding quality used more frequently than a predetermined threshold).

[0142] As another example, the AI ​​data setting controller 430 can determine the downscaling target based on the resolution, type (e.g., file format) of the original image 105.

[0143] The AI ​​data setting controller 430 can determine the image quality of the original image. For example, referencing... Figure 6The AI ​​data setting controller 430 may include an image quality measurement network 610, which outputs image quality information 620 when an original image is input to it. The image quality information 620 may be determined as at least one of high quality, low quality, or noisy quality. The image quality measurement network 610 may extract features representing at least one of detail, noise, or contrast of the input original image, and may determine the image quality information 620 based on the extracted features. For example, the image quality may be determined as high quality when the detail level is equal to or greater than a first value and the noise is less than a second value. The image quality may be determined as low quality when the detail level is less than the first value and the noise is less than the second value. The image quality may be determined as noisy quality when the detail level is less than the first value and the noise is equal to or greater than the second value.

[0144] The AI ​​data setting controller 430 can determine the target detail intensity 630 based on the determined image quality information 620. For example, when the image quality is low, the AI ​​data setting controller 430 can determine that the target detail intensity is strong. However, the embodiments are not limited to this.

[0145] Return to reference Figure 5 The AI ​​data generator 440 can generate AI data based on input information obtained by the AI ​​data setting controller 430. AI data according to embodiments of this disclosure may include at least one of the following: target resolution of a first image, bit depth of the first image, second DNN setting identification information, quantization level initialization value, quantization parameter initialization value, image type initialization value, second DNN update information, or upscaling ratio.

[0146] According to the embodiment, when an operation method is predefined between the AI ​​encoding device 400 and the AI ​​decoding device 700, the AI ​​data may not include image type information, second DNN update information, and upscaling ratio.

[0147] For example, refer to Figure 5The AI ​​data generator 440 can generate second DNN setting identification information based on the target resolution, target detail intensity, and characteristic information of the original image of the first image. The AI ​​data generator 440 can also generate quantization level initialization values ​​and image type initialization values ​​based on encoding control information. The AI ​​data generator 440 can determine second DNN update information and upscaling ratio based on decoding control information. However, the embodiments are not limited to this, and the AI ​​data generator 440 can include various information that enables the AI ​​upscaler of the AI ​​decoding device 700 to upscale the second image 135AI to an upscaling target corresponding to the downscaling target.

[0148] AI data generator 440 can send the generated AI data to stream generator 450.

[0149] The stream generator 450 generates AI-encoded data that includes image data received from the encoder 420 and AI data received from the AI ​​data generator 440.

[0150] According to an embodiment, the stream generator 450 can generate AI-encoded data that includes image data and AI data that are separate from each other. In this case, the AI-encoded data can be a container format such as MP4, AVI, MKV, or FLV, and the AI-encoded data can consist of a metadata box and a media data box.

[0151] For example, according to an embodiment, the media data box may include image data generated according to the syntax of a predetermined image compression method, while the metadata box may include information about the image data and AI data included in the media data box. The AI ​​data may be encoded according to an encoding method provided in a predetermined container format and may be stored in the metadata box.

[0152] According to an embodiment, the stream generator 450 can generate AI-encoded data such that the AI ​​data is included in the image data. When the AI ​​data is included in the image data, the AI ​​data can be included in supplemental enhancement information (SEI), which is an additional information region of the image data.

[0153] The AI-encoded data generated by the stream generator 450 is sent to the transmitter 460. The transmitter 460 can then send the AI-encoded data, obtained as a result of AI encoding, over the network.

[0154] According to an embodiment, AI encoded data can be stored in a data storage medium, which includes a magnetic medium (e.g., a hard disk, floppy disk, or magnetic tape), an optical recording medium (e.g., a CD-ROM or DVD), or a magneto-optical medium (e.g., a magnetic disk).

[0155] Figure 7 This is a block diagram illustrating the structure of an AI decoding apparatus 700 according to an embodiment of the present disclosure.

[0156] refer to Figure 7 The AI ​​decoding device 700 may include a receiver 710 and an AI decoder 702. The AI ​​decoder 702 may include a parser 720, a decoder 750, an AI data restorer 730, a model determiner 740, and an AI upscaler 760.

[0157] exist Figure 7 In this embodiment, receiver 710 and AI decoder 702 are shown as separate devices, but receiver 710 and AI decoder 702 can be implemented using a single processor. In this case, receiver 710 and AI decoder 702 can be implemented using a dedicated processor or a combination of software and a general-purpose processor (such as an application processor (AP), central processing unit (CPU), or graphics processing unit (GPU)). The dedicated processor may include memory for implementing embodiments of this disclosure, or may include a memory processor for using external memory.

[0158] The receiver 710 and AI decoder 702 can be implemented using multiple processors. In this case, the receiver 710 and AI decoder 702 can be implemented using a combination of dedicated processors or a combination of software and multiple general-purpose processors (such as an AP, CPU, or GPU). According to an embodiment, the receiver 710 can be implemented using a first processor, the decoder 750 can be implemented using a second processor different from the first processor, and the parser 720, AI data restorer 730, model determiner 740, and AI upscaler 760 can be implemented using a third processor different from the first and second processors.

[0159] Receiver 710 receives AI-encoded data obtained as a result of AI encoding. For example, AI-encoded data can be a video file with a file format such as mp4 or mov.

[0160] Receiver 710 can receive AI-encoded data transmitted over a network. Receiver 710 can output the AI-encoded data to AI decoder 702.

[0161] According to an embodiment, AI-encoded data can be obtained from a data storage medium, including a magnetic medium (e.g., a hard disk, floppy disk, or magnetic tape), an optical recording medium (e.g., a CD-ROM or DVD), or a magneto-optical medium (e.g., a magnetic disk).

[0162] The parser 720 parses the AI-encoded data to send the image data obtained by encoding the first image 115 to the decoder 750 and the AI ​​data to the AI ​​data restorer 730.

[0163] According to an embodiment, parser 720 can parse AI-encoded data that includes separate image data and AI data. Parser 720 can distinguish between AI data and image data by reading the header within the AI-encoded data.

[0164] According to another embodiment, parser 720 can parse AI encoded data to obtain image data, extract AI data from the image data, and then send the AI ​​data to AI data restorer 730 and the remaining image data to decoder 750. In other words, AI data can be included in the image data. For example, AI data can be included in Supplemental Enhancement Information (SEI), which is an additional information region of the bitstream corresponding to the image data.

[0165] According to another embodiment, the parser 720 can divide the bitstream corresponding to the image data into a bitstream to be processed by the decoder 750 and a bitstream corresponding to the AI ​​data, and can output the two divided bitstreams to the decoder 750 and the AI ​​data restorer 730 respectively.

[0166] The parser 720 can verify that the image data included in the AI ​​encoded data is image data obtained using a predetermined codec (e.g., MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1). In this case, the parser 720 can send the corresponding information to the decoder 750, thereby processing the image data using the verified codec.

[0167] The decoder 750 reconstructs a second image 135 corresponding to the first image 115 based on the image data received from the parser 720. The second image 135 obtained by the decoder 750 is provided to the AI ​​upscaler 760.

[0168] According to an embodiment, the parser 720 can provide decoding-related information, such as prediction pattern information, motion information, and quantization parameter information, to the AI ​​data restorer 730 or the model determiner 740. The decoding-related information can be used to obtain second DNN setup information.

[0169] The AI ​​data restorer 730 can extract information from the received AI data that causes the second image 135 to undergo AI upscaling. The upscaling target of the second image 135 can correspond to the downscaling target of the first DNN 200. Therefore, the AI ​​data can include information for verifying the downscaling target of the first DNN 200.

[0170] refer to Figure 8 The AI ​​data recovery unit 730 may include an AI data parser 810 and an encoding control information updater 820.

[0171] AI data parser 810 can parse the received AI data to obtain first encoding control information, first image resolution, first image bit depth, second DNN setting identification information, and decoding control information from the AI ​​data.

[0172] According to the embodiment, when an operation method is predefined between the AI ​​encoding device 400 and the AI ​​decoding device 700, the AI ​​data may not include the image type information and decoding control information in the first encoding control information, such as the second DNN update information and upscaling ratio.

[0173] The first encoding control information obtained from AI data may include frame type information and quantization level information, and the frame type information and quantization level information obtained from AI data may be arbitrarily set initial values.

[0174] The encoding control information updater 820 can obtain second encoding control information. This second encoding control information is obtained from the header information of the AI ​​encoded data, such as a bitstream, and therefore can be the actual encoding control information applied when encoding the first image. The encoding control information updater 820 can update the first encoding control information with the second encoding control information by comparing the first encoding control information with the second encoding control information.

[0175] The model determiner 740 can determine the settings information of the second DNN to be used in AI upscaling based on at least one of the updated encoding control information, the resolution information of the first image, the bit depth information of the first image, the second DNN setting identifier information, or the decoding control information. The model determiner 740 can determine the upscaling target of the second image 135 using the upscaling ratio of the decoding control information. The upscaling target can indicate, for example, to what extent the resolution of the second image 135 will be upscaled. When the AI ​​data does not include an upscaling ratio, the model determiner 740 can determine the upscaling target of the second image 135 based on a predetermined output resolution.

[0176] According to an embodiment, the pattern determiner 740 may store multiple DNN setting information that can be set in the second DNN. Here, the DNN setting information may include information about at least one of the following: the number of convolutional layers included in the second DNN, the number of filter kernels in each convolutional layer, or the parameters of each filter kernel.

[0177] Multiple DNN configurations can correspond to various upscaling objectives, and the second DNN can operate based on the DNN configuration corresponding to a specific upscaling objective. The second DNN can have different structures based on the DNN configuration. For example, the second DNN can include three convolutional layers based on any one DNN configuration, or it can include four convolutional layers based on another DNN configuration.

[0178] According to an embodiment, the DNN configuration information may include only the parameters of the filter kernels used in the second DNN. In this case, not only the structure of the second DNN remains unchanged, but only the parameters of the internal filter kernels can be changed based on the DNN configuration information.

[0179] The model determiner 740 can obtain DNN setup information for performing AI upscaling on the second image 135 from multiple DNN setup information. Each of the multiple DNN setup information used at this time is information for obtaining a third image 145 with a predetermined output resolution and / or predetermined quality, and is jointly trained with the first DNN.

[0180] For example, one of the multiple DNN setting information may include multiple pieces of information for obtaining a third image 145 with a resolution twice that of the second image 135 (e.g., the third image 145 has a 4K resolution (e.g., 3840×2160), which is twice the 2K resolution (e.g., 1920×1080) of the second image 135), and another DNN setting information may include multiple pieces of information for obtaining a third image 145 with a resolution four times that of the second image 135 (e.g., the third image 145 has a 4K resolution (e.g., 3840×2160), which is four times the 1K resolution (e.g., 960×540) of the second image 135).

[0181] Each of the multiple DNN setting information is obtained in conjunction with the DNN setting information of the first DNN of the AI ​​encoding device 400, and the model determiner 740 determines one DNN setting information from the multiple DNN setting information according to the magnification ratio corresponding to the scaling ratio of the DNN setting information of the first DNN.

[0182] The model determiner 740 can verify the target information used to obtain the DNN setup information of the first DNN for obtaining the first image 115, and can obtain the DNN setup information of the second DNN jointly trained with the DNN setup information of the first DNN by using multiple pieces of information received from the AI ​​encoding device 400.

[0183] Figure 9An example is shown where the model determiner 740 determines the setup information for a second DNN to be used in AI upscaling.

[0184] refer to Figure 9 The model determiner 740 includes multiple DNN setup pieces. Each of the multiple DNN setup pieces is determined through joint training with the DNN setup pieces of the first DNN. The multiple DNN setup pieces can be divided into multiple groups, and a second DNN setup identifier (e.g., DNN model setup ID) can be assigned to each group. For example, the multiple groups may include a first group 911 where the second DNN setup identifier is 0 and a second group 912 where the second DNN setup identifier is 1.

[0185] The model determiner 740 can select one group from multiple groups based on the second DNN setting identifier information included in the AI ​​data, and can determine one DNN setting information from multiple DNN setting information included in the selected group based on resolution information, bit depth and quantization level.

[0186] For example, when the second DNN setting identifier information is 0, the model determiner 740 can select the first group 911, and when the resolution is 1920×1080, the bit depth is 8, and the quantization level is L3, the model determiner 740 can select the third setting information 930 from the multiple DNN setting information included in the first group 911.

[0187] The model determiner 740 can send the selected DNN setting information to the AI ​​upscaler 760 and can process the input data based on a second DNN that operates according to the DNN setting information.

[0188] Once the upscaling target is determined, the AI ​​upscaler 760 performs AI upscaling on the second image 135 through the second DNN to obtain a third image 145 corresponding to the upscaling target.

[0189] When any DNN setting information is obtained, the AI ​​upscaler 760 can set the number of filter kernels in each layer of the second DNN and the parameters of each filter kernel to the values ​​included in the obtained DNN setting information, and can perform AI upscaling on the second image by using the set parameters.

[0190] Figure 10 This is the syntax table 1010 for AI data according to embodiments of this disclosure.

[0191] refer to Figure 10 AI data syntax table 1010 is an example of an SEI syntax table for AI data transmission in a bitstream encoded according to a codec conforming to MPEG standards, etc.

[0192] AI data syntax table 1010 includes elements related to second DNN information used for AI upscaling of a second image recovered from image data.

[0193] according to Figure 10 In the AI ​​data syntax table 1010, `input_picture_width_in_luma` and `input_picture_height_in_luma` indicate the width and height information of the downscaled input image (e.g., the first image), respectively. For example, `input_picture_width_in_luma` and `input_picture_height_in_luma` are information related to the resolution of the downscaled input image. `input_bit_depth` indicates the bit depth of the downscaled input image, and `input_bit_depth` can be 8 or 10. `dnn_model_update_type` is information related to the second DNN update. `quantization_level` indicates the quantization level of the input video stream, and the initial value of the quantization level can be, but is not limited to, 12. `picture_type` is a value related to the encoding control method and indicates one of I, P, and B. `up_scaling_factor` is information indicating the upscaling factor. `dnn_model_set_index` indicates the mutually agreed-upon second DNN setting identifier information used in AI upscaling.

[0194] The model determiner 740 can determine one of the pre-stored settings for AI upscaling from multiple pre-stored settings for AI based on the elements included in the AI ​​data syntax table, as the settings for the second DNN 300.

[0195] Figure 11 It is syntax table 1110 for AI data according to another embodiment of this disclosure.

[0196] AI data syntax table 1110 includes elements related to the second DNN information used for AI upscaling of the second image recovered from the image data.

[0197] The above has been referenced Figure 10 The parameters input_picture_width_in_luma, input_picture_height_in_luma, input_bit_depth, quantization_level, and dnn_model_set_index are described, so their descriptions will be omitted.

[0198] Figure 11AI data syntax table 1110 may not include Figure 10 The AI ​​data syntax table 1010 includes dnn_model_update_type, picture_type, and up_scaling_factor.

[0199] The model determiner 740 can determine one of the pre-stored settings for AI upscaling from multiple pre-stored settings for AI based on the elements included in the AI ​​data syntax table, as the second DNN.

[0200] Figure 12 An example of an AI decoding apparatus according to an embodiment of the present disclosure is shown, comprising a decoding device 1200 and an AI upscaling device 1300 that are separate from each other.

[0201] The decoding device 1200 and the AI ​​upscaling device 1300 can be connected to each other via an HDMI cable. When the HDMI transmitter 1250 of the decoding device 1200 and the HDMI receiver 1310 of the AI ​​upscaling device 1300 are connected to each other via the HDMI cable, the four channels providing the TMDS data channel and the TMDS clock channel can be paired. The TMDS data channel includes three data transmission channels and can be used to transmit video data, audio data, and additional data. At this time, the packet structure is used to transmit audio data and additional data through the TMDS data channel.

[0202] The HDMI transmitter 1250 of the decoding device 1200 and the HDMI receiver 1310 of the AI ​​upscaling device 1300 can provide a Display Data Channel (DDC). DDC is a protocol standard defined by the Video Electronics Standards Association (VESA) for the transmission of digital information between a monitor and a computer adapter. DDC is used for the exchange of structural and status information between a source device (e.g., a decoding device) and a destination device (e.g., an AI upscaling device).

[0203] refer to Figure 12 The decoding device 1200 may include a receiver 1210, a parser 1220, a decoder 1230, a vendor-specific information frame (VSIF) structurator 1240, and an HDMI transmitter 1250.

[0204] Figure 12 The receiver 1210, parser 1220 and decoder 1230 respectively correspond to Figure 7 The receiver 710, parser 720 and decoder 750 are included, so a detailed description of them will be omitted.

[0205] The VSIF structurator 1240 can structure the AI ​​data transmitted by the parser 1220 into VSIF packets.

[0206] The VSIF structuring unit 1240 can generate VSIF packets corresponding to each of multiple frames. For example, when AI data is received once for multiple frames, the VSIF structuring unit 1240 can generate VSIF packets corresponding to each of the multiple frames using the received AI data once. For example, VSIF packets corresponding to multiple frames respectively can be generated based on the same AI data.

[0207] On the other hand, when multiple AI data are received for multiple frames, the VSIF structurator 1240 can generate new VSIF packets by using the newly received AI data.

[0208] Now refer to Figure 13 and Figure 14 A detailed description of the VSIF packets generated by the VSIF structurator 1240.

[0209] Figure 13 The header and content structures of a VSIF packet according to an embodiment of this disclosure are shown.

[0210] refer to Figure 13 A VSIF packet includes a VSIF packet header 1410 and a VSIF packet content 1420. The VSIF packet header 1410 can include 3 bytes, where the first byte HB0 is a value indicating the type, with the VSIF type value represented as 0x01; the second byte HB1 indicates version information, with the VSIF version represented as 0x01; and the lower five bits 0 to 4 of the third byte HB2 indicate the length of the VSIF packet content.

[0211] The VSIF structurator 1240 can structure AI data into VSIF packets. For example, the VSIF structurator 1240 can generate VSIF packets, such that the AI ​​data is included in the VSIF packet content 1420.

[0212] The VSIF structurer 1240 can determine the packet bytes used to describe the AI ​​data based on the amount of AI data. The VSIF structurer 1240 can generate VSIF packet content, making the above reference... Figure 10 The AI ​​data described is described in bytes 5 (PB4) through byte 18 (PB17) of the VSIF packet content.

[0213] For example, the VSIF structurator 1240 can define input_picture_width_in_luma using PB05 and PB06, and can define input_picture_height_in_luma using PB07 and PB08. input_picture_width_in_luma and input_picture_height_in_luma indicate the resolution of the downscaled first image.

[0214] The VSIF structuring class 1240 can define input_bit_depth using PB09. input_bit_depth indicates the bit depth of the first image.

[0215] The VSIF structuring unit 1240 can define the quantization level using PB10. The quantization level indicates the quantization level of the input video stream.

[0216] The VSIF structurator 1240 can define dnn_model_set_index using PB11 and PB12. dnn_model_set_index indicates mutually agreed-upon second DNN setup identifier information used in AI upscaling.

[0217] However, the embodiments disclosed herein are not limited thereto. The VSIF structurator 1240 can use various methods to structure AI data into VSIF packet form.

[0218] Figure 14 The header and content structures of a VSIF packet according to another embodiment of this disclosure are shown.

[0219] Figure 14 The VSIF packet header 1415 can be configured to be similar to... Figure 13 The VSIF packet header 1410 is the same.

[0220] The VSIF structurer 1240 can generate VSIF packet content, making the above reference... Figure 11 The AI ​​data described is described in bytes 5 (PB4) through byte 18 (PB17) of the VSIF packet content. Figure 14 AI data can also include Figure 11 The AI ​​data includes dnn_model_update_type, picture_type, and up_scaling_factor.

[0221] Therefore, the VSIF structurator 1240 can define input_picture_width_in_luma using PB05 and PB06, and can define input_picture_height_in_luma using PB07 and PB08. input_picture_width_in_luma and input_picture_height_in_luma indicate the resolution of the downscaled first image.

[0222] The VSIF structurator 1240 can define the picture_type using bits 1 and 2 of PB13, and the picture_type is a value associated with the encoding control method and represents I, P, or B.

[0223] The VSIF structurator 1240 can define dnn_model_update_type using bits 3 and 4 of PB13, and dnn_model_update_type indicates information related to the update of the AI ​​upscaling neural network model.

[0224] The VSIF structuring unit 1240 can define input_bit_depth using bits 5 through 8 of PB13. input_bit_depth indicates the bit depth of the first image.

[0225] The VSIF structuring unit 1240 can define the quantization level using PB14. The quantization level indicates the quantization level of the input video stream.

[0226] The VSIF structuring module 1240 can define up_scaling_factor using PB15. up_scaling_factor indicates the upscaling factor.

[0227] The VSIF structurator 1240 can define dnn_model_set_index using PB16 and PB17. dnn_model_set_index indicates mutually agreed-upon second DNN setup information used in AI upscaling.

[0228] However, the embodiments disclosed herein are not limited thereto. The VSIF structurator 1240 can use various methods to structure AI data into VSIF packet form.

[0229] Figure 13 and Figure 14 The structure of the VSIF packet is merely an example, and therefore the embodiments disclosed herein are not limited thereto. Define as needed. Figure 13 and 14 The position or size of fields in the AI ​​data included in the VSIF packet can be changed, and additional data can be further included in the VSIF packet.

[0230] Refer again Figure 12 The VSIF structuring unit 1240 can send the generated VSIF packets to the HDMI transmitter 1250, which can then send the VSIF packets to the AI ​​upscaling device 1300 via the TMDS channel.

[0231] The HDMI transmitter 1250 can send the second image received from the decoder 1230 to the AI ​​upscaling device 1300 via the TMDS channel.

[0232] The AI ​​upscaling device 1300 may include an HDMI receiver 1310, an AI data restorer 1320, a model determiner 1330, and an AI upscaler 1340.

[0233] The AI ​​upscaling device 1300 can store extended display identification data (EDID) information. EDID information is a data structure that includes various information about the AI ​​upscaling device and can be transmitted to the decoding device 1200 using the DDC channel.

[0234] The EDID information may include information about the AI ​​upscaling capability of the AI ​​upscaling device 1300. For example, the EDID information may include information about whether the AI ​​upscaling device 1300 is capable of performing AI upscaling.

[0235] The HDMI receiver 1310 of the AI ​​upscaling device 1300 can receive AI data and a second image structured in VSIF packet form via the TMDS channel.

[0236] The HDMI receiver 1310 of the AI ​​upscaling device 1300 can search for VSIF packets by checking the header information of the HDMI packets, and then determine whether the VSIF packets contain AI data.

[0237] For example, the HDMI receiver 1310 can determine whether a received HDMI packet is a VSIF packet by checking whether the first byte HB0, which indicates the packet type in the header information of the received HDMI packet, is 0x01. When it is determined that the HDMI packet is a VSIF packet, the HDMI receiver 1310 can also determine whether AI data is included in the VSIF packet content.

[0238] For example, the HDMI receiver 1310 can obtain AI data by using the bit values ​​included in the VSIF packet content. For example, the AI ​​data restorer 1320 can obtain the resolution information of the first image by using PB05 to PB08 of the VSIF packet content.

[0239] The AI ​​data restorer 1320 can obtain the bit depth of the first image by using bits 4 to 7 of PB09 or PB13, and can obtain quantization level information by using PB14 or PB10. The AI ​​data restorer 1320 can also obtain second DNN setting identification information by using PB16 and PB17 or by using PB11 and PB12.

[0240] In an embodiment, the AI ​​data restorer 1320 can obtain image type information by using bits 0 and 1 of PB13, and can obtain second DNN update information by using bits 2 and 3 of PB13.

[0241] AI data restorer 1320 can provide AI data obtained from VSIF packet content to model determiner 1330. Model determiner 1330 can determine one of the pre-stored settings for AI upscaling as the second DNN based on the obtained AI data.

[0242] The HDMI receiver 1310 can provide the received second image to the AI ​​upscaler 1340. The AI ​​upscaler 1340 can set the parameters of the second DNN using the determined DNN setting information, and can perform AI upscaling on the second image by using the set parameters.

[0243] exist Figures 12 to 14 In this embodiment, the decoding device 1200 and the AI ​​upscaling device 1300 are connected to each other using an HDMI cable. However, embodiments of this disclosure are not limited thereto; the decoding device and the AI ​​upscaling device according to embodiments of this disclosure are connected to each other using a DisplayPort (DP) cable. When the decoding device and the AI ​​upscaling device are connected to each other using a DP cable, the decoding device can send the second image and AI data to the AI ​​upscaling device via DP in a manner similar to HDMI.

[0244] The decoding device 1200 can send the second image and AI data to the AI ​​upscaling device 1300 via input / output interfaces other than HDMI and DP.

[0245] The decoding device 1200 can send the second image and AI data to the AI ​​upscaling device 1300 via different interfaces. For example, the decoding device 1200 can send the second image to the AI ​​upscaling device 1300 via an HDMI cable and send the AI ​​data to the AI ​​upscaling device 1300 via a DP cable. As another example, the decoding device 1200 can send the second image to the AI ​​upscaling device 1300 via a DP cable and send the AI ​​data to the AI ​​upscaling device 1300 via an HDMI cable.

[0246] Figure 15 This is a flowchart of an operation method of an AI encoding apparatus according to an embodiment of the present disclosure.

[0247] refer to Figure 15 In operation S1510, the AI ​​encoding device 400 can determine the downscaling target. For example, the AI ​​encoding device 400 can determine the downscaling ratio based on the target resolution of the first image.

[0248] The AI ​​encoding device 400 can obtain DNN setting information corresponding to the downscaling ratio, and can set the parameters of the first DNN by using the obtained DNN setting information. In operation S1520, the AI ​​encoding device 400 can obtain the first image by performing AI downscaling on the original image using the parameter-set first DNN.

[0249] In operation S1530, the AI ​​encoding device 400 can generate image data by encoding the first image.

[0250] The AI ​​encoding device 400 can obtain one or more input information for setting AI data. For example, the AI ​​encoding device 400 can obtain the target resolution of a first image, the bit depth of the first image, characteristic information of the original image (e.g., the input image), target detail intensity, encoding control information, decoding control information, etc. The one or more input information may include information pre-stored in the AI ​​encoding device 400 or information received from the user. The AI ​​encoding device 400 can determine the quality of the original image by using an image quality measurement network, and can determine the target detail intensity based on the quality information of the original image.

[0251] In operation S1540, the AI ​​data generator 440 can generate second DNN setting identification information based on the target resolution, target detail intensity and characteristic information of the first image and the original image.

[0252] In operation S1550, the AI ​​encoding device 400 can generate AI data based on the obtained input information. The AI ​​data according to embodiments of this disclosure may include at least one of the following: resolution of a first image, bit depth of the first image, second DNN setting identification information, quantization level initialization value, quantization parameter initialization value, image type initialization value, second DNN update information, or upscaling ratio.

[0253] The AI ​​encoding device 400 can generate AI encoded data including image data and AI data in operation S1560, and can send the AI ​​encoded data to the AI ​​decoding device 700 in operation S1570.

[0254] Figure 16 This is a flowchart of an operation method of an AI decoding apparatus according to an embodiment of the present disclosure.

[0255] refer to Figure 16 In operation S1610, the AI ​​decoding device 700 can receive AI encoded data. The received AI encoded data may include image data and AI data.

[0256] In operation S1620, the AI ​​decoding device 700 can divide the AI ​​encoded data into image data and AI data. For example, the AI ​​decoding device 700 can distinguish between AI data and image data by reading the header within the AI ​​encoded data. In an embodiment, the AI ​​decoding device 700 can extract the AI ​​data included in the SEI, which is an additional information region of the bitstream corresponding to the image data.

[0257] In operation S1630, the AI ​​decoding device 700 can obtain a second image by decoding the image data.

[0258] In operation S1640, the AI ​​decoding device 700 can extract multiple AI upscaling related information from the AI ​​data that enables the second image to be AI upscaled. For example, the AI ​​decoding device 700 can extract the resolution of the first image, the bit depth of the first image, the second DNN setting identification information, and the first encoding control information. The first encoding control information may include image type information and quantization level information, and the image type information and quantization level information included in the first encoding control information can be arbitrarily set initial values.

[0259] In operation S1650, the AI ​​decoding device 700 can obtain second encoding control information as actual encoding control information applied when encoding the first image, and can update the first encoding control information with the second encoding control information by comparing the first encoding control information with the second encoding control information.

[0260] In operation S1660, the AI ​​decoding device 700 can determine the setting information of the second DNN to be used in AI upscaling based on at least one of the updated encoding control information, the resolution information of the downscaled first image, the bit depth information of the first image, the second DNN setting identifier information, or the decoding control information. For example, the AI ​​decoding device 700 can obtain DNN setting information corresponding to the upscaling ratio and can set the parameters of the second DNN by using the obtained DNN setting information.

[0261] In operation S1670, the AI ​​decoding device 700 can perform AI upscaling on the second image by using the set parameters.

[0262] The operation methods of the AI ​​encoding apparatus and the AI ​​decoding apparatus according to the embodiments can be implemented as program commands executable by various computer devices and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., individually or in combination. The program commands to be recorded on the computer-readable recording medium may be specifically designed and configured for the embodiments or may be well-known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, or magnetic tapes; optical media such as CD-ROMs or DVDs; magneto-optical media such as optical-magnetic floppy disks; and hardware devices specifically configured to store and execute program commands, such as ROMs, RAMs, or flash memory. Examples of program commands are high-level language code executable by a computer using an interpreter, etc., and machine language code generated by a compiler.

[0263] Furthermore, the computer program product may provide methods for operating the AI ​​encoding device and the AI ​​decoding device according to the disclosed embodiments. The computer program product can be traded as a commodity between a seller and a buyer.

[0264] Computer program products may include software programs and computer-readable storage media on which the software programs are stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by a manufacturer of an electronic device or an electronic marketplace (e.g., the Google Play Store, the App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or may be temporarily created. In this case, the storage medium may be a server of the manufacturer, a server of the electronic marketplace, or a relay server used for temporary storage of the software program.

[0265] In a system including server and client devices, the computer program product may include the storage medium of the server or the storage medium of the client device. In an embodiment, if a third device (e.g., a smartphone) is present that communicates with the server or client device, the computer program product may include the storage medium of the third device. Alternatively, the computer program product may include the S / W program itself, transmitted from the server to the client device or the third device, or from the third device to the client device.

[0266] In this scenario, one of the server, client device, and third device may execute a computer program product to perform the method according to embodiments of this disclosure. Alternatively, at least two of the server, client device, and third device may execute a computer program product to distribute and perform the method according to the disclosed embodiments.

[0267] For example, a server (e.g., a cloud server or an artificial intelligence server) can execute a computer program product stored on the server to control client devices communicating with the server to perform methods according to the disclosed embodiments.

[0268] An AI encoding apparatus according to an embodiment of the present disclosure sends AI data, including information related to an AI upscaling neural network and AI downscaling and encoded image data, to an AI decoding apparatus according to an embodiment of the present disclosure. The AI ​​decoding apparatus sets the parameters of the AI ​​upscaling neural network based on the received AI data, thereby improving the quality of the recovered image.

[0269] Although one or more embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made to one or more embodiments of the present disclosure without departing from the spirit and scope defined by the appended claims.

Claims

1. An artificial intelligence (AI) coding device, comprising: At least one processor is configured as follows: Based on the target resolution of the first image, a downscaling target is determined, wherein the downscaling target indicates how much the resolution is reduced from the original image to obtain information from the first image. The first image is obtained by using an AI downscaling neural network corresponding to the downscaling target to perform AI downscaling on the original image. Image data is generated by encoding the first image. Based on the target resolution of the first image, the characteristic information of the original image, and the target detail intensity, an AI upscaling neural network is selected to set the labeling information. Generate AI data including the target resolution of the first image, the bit depth information of the first image, AI upscaling neural network setting identification information and encoding control information, and Generate AI-encoded data that includes image data and AI data; and The communication interface is configured to send AI-encoded data to the AI ​​decoding device. The AI ​​data includes information about AI upscaling neural networks corresponding to AI downscaling neural networks.

2. The AI ​​encoding device according to claim 1, wherein, The at least one processor is further configured to: By using a quality measurement network, the quality information of the original image is determined, and Determine the target detail intensity based on the quality information of the original image.

3. The AI ​​encoding device according to claim 2, wherein, The quality information indicates at least one of high quality, low quality, or noise quality, and The quality measurement network is configured as follows: Extract features representing at least one of the details, noise, or contrast of the original image, and Quality information is determined based on the extracted features.

4. The AI ​​encoding device according to claim 1, wherein, The AI ​​data also includes decoding control information, which includes the update method of the AI ​​upscaling neural network and the AI ​​upscaling ratio.

5. The AI ​​encoding device according to claim 1, wherein, The encoding control information includes at least one of quantization level or picture type. Wherein, the quantization level is determined to be a preset first value, and The image type is defined as intra-frame (I) image, predictive (P) image, or bidirectional (B) image.

6. The AI ​​encoding device according to claim 1, wherein, The AI-encoded data also includes metadata boxes and media data boxes. The AI ​​data is included in the metadata box, and The image data is included in the media data box.

7. The AI ​​encoding device according to claim 1, wherein, The AI ​​data is included in the Supplemental Enhancement Information (SEI) within the additional information region of the image data.

8. An artificial intelligence (AI) decoding device, comprising: The communication interface is configured to receive AI-encoded data generated by AI downscaling and encoding of the original image; and At least one processor is configured as follows: The AI-encoded data is divided into image data and AI data. By decoding the image data, a second image corresponding to the first image obtained by downscaling the original image is obtained. Extract the resolution of the first image, the bit depth information of the first image, the AI ​​upscaling neural network setting identification information, and the first encoding control information from the AI ​​data. Obtain second encoding control information for encoding the first image. Update the first coding control information based on the second coding control information. Based on the resolution of the original image, the bit depth information of the original image, the AI ​​upscaling neural network setting identification information, and the second encoding control information, the AI ​​upscaling neural network is selected, and Using the selected AI upscaling neural network, the second image is AI upscaled to obtain a third image corresponding to the upscaling target, which indicates the extent to which the resolution of the second image will be upscaled.

9. The AI ​​decoding device according to claim 8, wherein, The first encoding control information includes the initial value of the preset quantization level, and The second encoding control information includes quantization level values ​​applied to the encoding of the first image.

10. The AI ​​decoding device according to claim 9, wherein, The at least one processor is further configured to: Based on the resolution of the first image, the bit depth information of the first image, and the quantization level value included in the second encoding control information, neural network setting information is selected from multiple neural network setting information corresponding to the AI ​​upscaling neural network setting identifier information. Based on the selected neural network settings, configure the parameters of the AI ​​upscaling neural network.

11. The AI ​​decoding device according to claim 8, wherein, The at least one processor is further configured to: Determine the update information and upsampling rate of the AI ​​upscaling neural network from AI data, and The second image is upscaled using AI based on the selected AI upscaling neural network, update information, and upsampling rate.

12. A method of operating an artificial intelligence (AI) coding device, the method comprising: Based on the target resolution of the first image, a downscaling target is determined, wherein the downscaling target indicator is used to indicate how much the resolution is reduced from the original image to obtain information from the first image; The first image is obtained by using an AI downscaling neural network corresponding to the downscaling target to perform AI downscaling on the original image; Image data is generated by encoding the first image; Based on the target resolution of the first image, the characteristic information of the original image, and the target detail intensity, an AI upscaling neural network is selected to set the label information; Generate AI data including the target resolution of the first image, the bit depth information of the first image, the AI ​​upscaling neural network setting identification information, and the encoding control information; Generate AI-encoded data that includes image data and AI data; and Send AI-encoded data to the AI ​​decoding device. The AI ​​data refers to information about the AI ​​upscaling neural network corresponding to the AI ​​downscaling neural network.

13. The operating method according to claim 12, further comprising: A quality measurement network is used to determine the quality information of the original image; as well as Determine the target detail intensity based on the quality information of the original image.

14. The operating method according to claim 13, wherein, The quality information indicates at least one of high quality, low quality, or noise quality, and The quality measurement network is configured to extract features representing at least one of detail, noise, or contrast of the original image, and to determine quality information based on the extracted features.

15. The operating method according to claim 12, wherein, The AI ​​data also includes decoding control information, which includes the update method of the AI ​​upscaling neural network and the AI ​​upscaling ratio.

Images