AI encoding device and operation method thereof, and AI decoding device and operation method thereof

Abstract

An artificial intelligence (AI) decoding device is provided, the AI ​​decoding device comprising: a memory storing one or more instructions; and a processor configured to execute the one or more instructions to perform the following operations: when an image is input to a second DNN comprising multiple layers, a first layer comprising a first filter kernel and a second filter kernel among the multiple layers is used to obtain a first result value based on an operation between the image and the first filter kernel and a second result value is obtained based on an operation between the second image and the second filter kernel, normalization is performed by transforming the first result value into a first value using a first scaling factor, and normalization is performed by transforming the second result value into a second value using a second scaling factor, and the first value and the second value are transformed into integer values ​​included in a preset range.

Description

Technical Field

[0001] The present disclosure relates to an AI encoding device including a deep neural network (DNN) for performing artificial intelligence (AI) downscaling on an image and an operating method of the AI ​​encoding device, and to an AI decoding device including a DNN for performing AI upscaling on an image and an operating method of the AI ​​decoding device, and more specifically, to an AI encoding device and an operating method of the AI ​​encoding device for reducing the amount of memory and the amount of calculation required to perform convolution operations in multiple convolution layers included in the DNN, and to an AI decoding device and an operating method of the AI ​​decoding device for reducing the amount of memory and the amount of calculation required to perform convolution operations in multiple convolution layers included in the DNN. Background Art

[0002] The image data is encoded by a codec conforming to a specified data compression standard (eg, the Moving Picture Experts Group (MPEG) compression standard), and then stored in a recording medium or transmitted in the form of a bit stream via a communication channel.

[0003] With the development and supply of hardware for reproducing and storing high-resolution / high-quality images, the demand for codecs capable of efficiently encoding and decoding high-resolution / high-quality images is increasing.

[0004] A convolutional neural network (CNN) may include one or more convolutional layers, in each of which a convolution operation may be performed on input data and weights included in a kernel.

[0005] CNN requires a lot of memory to store the intermediate result values ​​output between convolutional layers. In CNN, low-precision transformations can be performed to reduce the amount of memory required. When low-precision transformations are performed, transformation errors may occur, so a method is needed to minimize the transformation errors. Summary of the invention

[0006] Technical Solution

[0007] Provided are an AI decoding device and an operating method of the AI ​​decoding device capable of minimizing a transform error when an intermediate result value generated in a second DNN included in the AI ​​decoding device is represented with low precision.

[0008] Best Mode

[0009] Provided are an AI decoding device and an operating method of the AI ​​decoding device capable of minimizing a transform error when an intermediate result value generated in a second DNN included in the AI ​​decoding device is represented with low precision.

[0010] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure.

[0011] According to an embodiment of the present disclosure, an AI decoding device includes: a memory storing one or more instructions; and a processor configured to execute the one or more stored instructions to perform the following operations: obtain image data corresponding to a first image obtained by reducing an original image using a first deep neural network (DNN); obtain a second image as a reconstructed image of the first image based on the image data; input the second image to a second DNN including multiple layers, wherein the multiple layers include a first layer and a second layer, and the first layer includes a first filter kernel and a second filter kernel; obtain a first result value based on an operation between the second image and the first filter kernel, and obtain a second result value based on an operation between the second image and the second filter kernel; perform normalization by transforming the first result value into a first value using a first scaling factor, and perform normalization by transforming the second result value into a second value using a second scaling factor; transform the first value and the second value into integer values ​​included in a preset range; input the integer value to a second layer, obtain a third result value based on the integer value in the second layer, and obtain a third image by enlarging the second image based on the third result value, wherein the first scaling factor and the second scaling factor are set in association with the scaling factor of the first DNN.

[0012] The second DNN may be a network trained in combination with the first DNN based on an image obtained during training of the first DNN, and the first scaling factor and the second scaling factor may be set via training of the second DNN and the first DNN.

[0013] The first scaling factor and the second scaling factor may be trained such that a normal distribution of the first values ​​becomes equal to a normal distribution of the second values.

[0014] The processor may be further configured to clip the first value and the second value to the preset range and convert the clipped first value and the second value into the integer value.

[0015] The processor may be further configured to transform the clipped first value and the second value into the integer value by rounding the clipped first value and the second value.

[0016] The processor may be further configured to store the integer value in the memory.

[0017] A size of a memory storing the integer value may be determined based on the preset range.

[0018] The processor may also be configured to generate a first feature map by performing a convolution operation between the second image and the first filter kernel, and obtain a first result value by applying the first activation function to the first feature map. The processor may also be configured to generate a second feature map by performing a convolution operation between the second image and the second filter kernel, and obtain a second result value by applying the second activation function to the second feature map.

[0019] The first activation function and the second activation function can be expressed as: Wherein, f(x) represents the first activation function or the second activation function, a and b represent slopes with different values, and the value of b is determined through training of the second DNN.

[0020] The value of b in the first activation function may be different from the value of b in the second activation function.

[0021] According to an embodiment of the present disclosure, an AI encoding device includes: a memory storing one or more instructions; and a processor configured to execute the one or more stored instructions to perform the following operations: input an original image to a first deep neural network (DNN) including multiple layers, wherein the multiple layers include a first layer and a second layer, and the first layer includes a first filter kernel and a second filter kernel; obtain a first result value based on an operation between the original image and the first filter kernel, and obtain a second result value based on an operation between the original image and the second filter kernel; perform normalization by transforming the first result value into a first value using a first scaling factor, and perform normalization by transforming the second result value into a second value using a second scaling factor; transform the first value and the second value into integer values ​​included in a preset range; input the integer value to a second layer, obtain a third result value based on the integer value in the second layer, obtain a first image by reducing the original image based on the third result value, and encode the first image, wherein the first scaling factor and the second scaling factor are set in association with the scaling factor of a second DNN configured to magnify the first image.

[0022] The first DNN may be a network trained in conjunction with the second DNN based on loss information obtained during training of the second DNN. The first scaling factor and the second scaling factor may be set via training of the second DNN and the first DNN.

[0023] The first DNN may be a network trained based on first loss information generated by performing AI downscaling during training of the first DNN and second loss information generated by performing AI upscaling during training of the second DNN.

[0024] The first scaling factor and the second scaling factor may be trained such that a normal distribution of the first values ​​becomes equal to a normal distribution of the second values.

[0025] The processor may be further configured to clip the first value and the second value to the preset range and convert the clipped first value and the second value into the integer value.

[0026] The processor may be further configured to transform the clipped first value and the second value into the integer value by rounding the clipped first value and the second value.

[0027] The processor may be further configured to store the integer value in the memory.

[0028] A size of a memory storing the integer value may be determined based on the preset range.

[0029] According to an embodiment of the present disclosure, an operation method of an AI decoding device may include: obtaining image data corresponding to a first image obtained by reducing an original image using a first deep neural network (DNN); obtaining a second image as a reconstructed image of the first image based on the image data; inputting the second image to a second DNN including multiple layers, wherein the multiple layers include a first layer and a second layer, and the first layer includes a first filter kernel and a second filter kernel; obtaining a first result value based on an operation between the second image and the first filter kernel, and obtaining a second result value based on an operation between the second image and the second filter kernel; performing normalization by transforming the first result value into a first value by using a first scaling factor, and performing normalization by transforming the second result value into a second value by using a second scaling factor; transforming the first value and the second value into integer values ​​included in a preset range; inputting the integer value to a second layer, and obtaining a third result value based on the integer value in the second layer; and obtaining a third image by enlarging the second image based on the third result value, wherein the first scaling factor and the second scaling factor are set in association with the scaling factor of the first DNN.

[0030] According to an embodiment of the present disclosure, an operating method of an AI encoding device may include: inputting an original image into a first deep neural network (DNN) including multiple layers, wherein the multiple layers include a first layer and a second layer, and the first layer includes a first filter kernel and a second filter kernel; obtaining a first result value based on an operation between the original image and the first filter kernel, and obtaining a second result value based on an operation between the original image and the second filter kernel; performing normalization by transforming the first result value into a first value using a first scaling factor, and performing normalization by transforming the second result value into a second value using a second scaling factor; transforming the first value and the second value into integer values ​​included in a preset range; inputting the integer value into a second layer, and obtaining a third result value based on the integer value in the second layer; obtaining a first image by reducing the original image based on the third result value; and encoding the first image, wherein the first scaling factor and the second scaling factor are set in association with a scaling factor of a second DNN configured to magnify the first image.

[0031] According to an embodiment of the present disclosure, an artificial intelligence (AI) decoding device may include: a memory storing one or more instructions; and a processor configured to execute the one or more stored instructions to perform the following operations: receiving a reduced image reduced by a first deep neural network (DNN), inputting the reduced image to a first layer of a second deep neural network, obtaining a first value and a second value from the first layer of the second DNN, wherein the first value and the second value respectively represent a first feature map and a second feature map of the reduced image; performing a transformation operation on the first value and the second value by normalizing the first value and the second value based on a first scaling factor and a second scaling factor, respectively, and transforming the normalized first value and the normalized second value into integer values ​​within a preset range, inputting the integer values ​​to a second layer of the second DNN, and obtaining an enlarged image corresponding to the reduced image from the second layer of the second DNN, wherein the first scaling factor and the second scaling factor are set in association with the scaling factor of the first DNN.

[0032] According to an embodiment of the present disclosure, an artificial intelligence (AI) encoding device may include: a memory storing one or more instructions; and a processor configured to execute the one or more stored instructions to perform the following operations: input an original image to a first layer of a first deep neural network (DNN), obtain a first value and a second value from the first layer of the first DNN, wherein the first value and the second value respectively represent a first feature map and a second feature map of the original image; perform a transformation operation on the first value and the second value by normalizing the first value and the second value based on a first scaling factor and a second scaling factor, respectively, and transforming the normalized first value and the normalized second value into integer values ​​within a preset range, input the integer values ​​into a second layer of the first DNN to obtain a reduced image, and send the reduced image to a second DNN configured to enlarge the reduced image, wherein the first scaling factor and the second scaling factor are set in association with the scaling factor of the second DNN.

[0033] Beneficial Effects

[0034] According to an embodiment of the present disclosure, the AI ​​encoding device can reduce the amount of necessary memory by representing intermediate result values ​​generated during image reduction by using the first DNN with low precision, and can improve the performance of the first DNN by minimizing the transformation error during the transformation of the intermediate result values ​​into low-precision values.

[0035] The AI ​​encoding device according to an embodiment of the present disclosure may also improve the performance of the first DNN by applying a trainable activation function to the first DNN.

[0036] According to an embodiment of the present disclosure, the AI ​​decoding device can reduce the amount of necessary memory by representing intermediate result values ​​generated during upscaling of an image by using a second DNN with low precision, and can improve the performance of the second DNN by minimizing transformation errors during transformation of the intermediate result values ​​into low-precision values.

[0037] In a system in which an AI encoding device includes a server that provides AI-reduced images and an AI decoding device includes a terminal that receives AI-reduced images, the amount of memory required to store intermediate result values ​​generated during AI upscaling of images received from the terminal can be reduced, thereby improving the efficiency of the system.

[0038] The AI ​​decoding device according to an embodiment of the present disclosure may improve the performance of the second DNN by applying a trainable activation function to the second DNN. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0040] Figure 1 is a diagram for describing an artificial intelligence (AI) encoding process and an AI decoding process according to an embodiment;

[0041] Figure 2 is a block diagram of a configuration of an AI decoding device according to an embodiment;

[0042] Figure 3 is a diagram illustrating a second deep neural network (DNN) for performing AI upscaling on a second image;

[0043] Figure 4 is a diagram for describing a convolution operation performed by a convolutional layer;

[0044] Figure 5 is a table showing a mapping relationship between several pieces of image-related information and several pieces of DNN setting information;

[0045] Figure 6 is a diagram showing a second image including a plurality of frames;

[0046] Figure 7 is a block diagram of a configuration of an AI encoding device according to an embodiment;

[0047] Figure 8 is a diagram illustrating a first DNN for performing AI downscaling on an original image;

[0048] Fig. 9 is a diagram for describing a method of training a first DNN and a second DNN.

[0049] Fig.10 2 is a diagram for describing a training process of a first DNN and a second DNN by a training device.

[0050] Fig.11 is a diagram of an apparatus for performing AI downscaling on an original image and an apparatus for performing AI upscaling on a second image.

[0051] Fig.12 is a block diagram showing the structure of a DNN according to an embodiment of the present disclosure;

[0052] Fig.13 Is used to describe Fig.12 A diagram of a data processing method in the first layer;

[0053] Fig.14 is a block diagram of a structure of a low-precision transform unit according to an embodiment of the present disclosure;

[0054] Fig.15 is a diagram for describing a data processing method performed by a low-precision transform unit according to an embodiment of the present disclosure;

[0055] Fig.16 is a diagram for describing a method of determining a scaling factor according to an embodiment of the present disclosure;

[0056] Fig.17 An activation function according to an embodiment of the present disclosure is shown;

[0057] Fig.18 is a flowchart of an operating method of an AI encoding device according to an embodiment of the present disclosure;

[0058] Fig.19 is a flowchart of an operating method of an AI decoding device according to an embodiment of the present disclosure;

[0059] Fig. 20 is a block diagram of a structure of an AI encoding device according to an embodiment of the present disclosure; and

[0060] Fig.21 is a block diagram of a structure of an AI decoding device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0061] Embodiments of the present disclosure provide an apparatus and method for performing artificial intelligence (AI) encoding and AI decoding on an image, wherein the image is encoded and decoded based on AI to achieve a low bit rate and minimize low-precision transformation errors.

[0062] Since the present disclosure allows various changes and many examples, the embodiments will be described in detail in the drawings and in the written description. However, this is not intended to limit the present disclosure to the mode of practice, and it will be understood that all changes, equivalents and substitutions that do not depart from the spirit and technical scope of the present disclosure are included in the present disclosure.

[0063] In the description of the embodiments, when it is considered that the detailed explanation of the related art may unnecessarily obscure the essence of the present disclosure, the detailed explanation of the related art is omitted. In addition, the numbers (e.g., first, second, etc.) used in the description of the specification are identifier codes for distinguishing one element from another element.

[0064] Furthermore, in this specification, it will be understood that when elements are “connected” or “coupled” to each other, the elements may be directly connected or coupled to each other, but may alternatively be connected or coupled to each other with intervening elements between the elements, unless otherwise specified.

[0065] In this specification, for an element represented as a "unit" or a "module", two or more elements may be combined into one element, or one element may be divided into two or more elements according to subdivided functions. In addition, each element described below may perform some or all of the functions performed by another element in addition to its main function, and some of the main functions of each element may be completely performed by another component.

[0066] Furthermore, in the present specification, an “image” or a “picture” may refer to a still image, a moving image including a plurality of consecutive still images (or frames), or a video.

[0067] Furthermore, in this specification, a deep neural network (DNN) is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model using an algorithm.

[0068] In addition, in this specification, a "parameter" is a value used in the operation process of each layer forming a neural network, and for example, a "parameter" may include a weight used when applying an input value to an operation expression. Here, the parameter can be expressed in a matrix form. The parameter is a value set as a result of training and can be updated by separate training data.

[0069] Furthermore, in this specification, a “first DNN” indicates a DNN for performing artificial intelligence (AI) downscaling on an image, and a “second DNN” indicates a DNN for performing AI upscaling on an image.

[0070] In addition, in this specification, "DNN setting information" includes information related to elements constituting the DNN. "DNN setting information" includes the above-mentioned parameters as information related to elements constituting the DNN. The first DNN or the second DNN can be set by using the DNN setting information.

[0071] In addition, in this specification, the "original image" means an image that is a target of AI encoding, and the "first image" means an image obtained as a result of performing AI reduction on the original image during the AI ​​encoding process. In addition, the "second image" means an image obtained via the first decoding during the AI ​​decoding process, and the "third image" means an image obtained by performing AI enlargement on the second image during the AI ​​decoding process.

[0072] In addition, in this specification, "AI reduction" means a process of reducing the resolution of an image based on AI, and "first encoding" means an encoding process according to an image compression method based on frequency transformation. In addition, "first decoding" means a decoding process according to an image reconstruction method based on frequency transformation, and "AI enlargement" means a process of increasing the resolution of an image based on AI.

[0073] Expressions such as "at least one of..." when following a list of elements modify the entire list of elements rather than the individual elements of the list. For example, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0074] Figure 1 is a diagram for describing an AI encoding process and an AI decoding process according to an embodiment.

[0075] As described above, when the resolution of an image is significantly increased, the throughput of information for encoding and decoding the image increases, and thus, a method for improving the encoding and decoding efficiency of the image is required.

[0076] like Figure 1 As shown in , according to an embodiment of the present disclosure, a first image 115 is obtained by performing AI reduction 110 on an original image 105 having a high resolution. Then, a first encoding 120 and a first decoding 130 are performed on the first image 115 having a relatively low resolution, so that the bit rate can be greatly reduced compared to when the first encoding 120 and the first decoding 130 are performed on the original image 105.

[0077] exist Figure 1 In the embodiment, a first image 115 is obtained by performing AI reduction 110 on an original image 105, and a first encoding 120 is performed on the first image 115 during an AI encoding process. During an AI decoding process, AI encoded data including AI data and image data obtained as a result of the AI ​​encoding is received, a second image 135 is obtained via a first decoding 130, and a third image 145 is obtained by performing AI enlargement 140 on the second image 135.

[0078] Referring to the AI ​​encoding process in detail, when the original image 105 is received, AI reduction 110 is performed on the original image 105 to obtain a first image 115 of a specific resolution or a specific quality. Here, the AI ​​reduction 110 is performed based on AI, and the AI ​​for the AI ​​reduction 110 is jointly trained with the AI ​​for the AI ​​enlargement 140 of the second image 135. This is because the AI ​​reduction 110 and the AI ​​enlargement 140 have two competing goals of reducing and enlarging the image, and therefore, when the AI ​​for the AI ​​reduction 110 and the AI ​​for the AI ​​enlargement 140 are trained separately, the difference between the original image 105 as the object of AI encoding and the third image 145 reconstructed by AI decoding increases.

[0079] In an embodiment of the present disclosure, AI data may be used to maintain such a joint relationship during the AI ​​encoding process and the AI ​​decoding process. Therefore, the AI ​​data obtained by the AI ​​encoding process may include information indicating a magnification target, and during the AI ​​decoding process, AI magnification 140 is performed on the second image 135 according to the magnification target verified based on the AI ​​data.

[0080] The AI ​​for AI reduction 110 and the AI ​​for AI upscaling 140 may be implemented as DNNs. Fig. 9 As described, since the first DNN and the second DNN are jointly trained by sharing loss information under a target, the AI ​​encoding device may provide the target information used during the joint training of the first DNN and the second DNN to the AI ​​decoding device, and the AI ​​decoding device may perform AI upscaling 140 on the second image 135 based on the provided target information to achieve the target resolution.

[0081] about Figure 1 The first encoding 120 and the first decoding 130 may reduce the amount of information of the first image 115 obtained by performing AI reduction 110 on the original image 105 through the first encoding 120. The first encoding 120 may include a process of generating prediction data by predicting the first image 115, a process of generating residual data corresponding to the difference between the first image 115 and the prediction data, a process of transforming the residual data of the spatial domain component into a frequency domain component, a process of quantizing the residual data transformed into the frequency domain component, and a process of entropy encoding the quantized residual data. Such a first encoding 120 may be performed via one of an image compression method using frequency transformation, such as MPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, High Efficiency Video Coding (HEVC), VC-1, VP8, VP9, ​​and AOMedia Video1 (AV1).

[0082] The second image 135 corresponding to the first image 115 may be reconstructed by performing the first decoding 130 on the image data. The first decoding 130 may include a process of generating quantized residual data by entropy decoding the image data, a process of inversely quantizing the quantized residual data, a process of transforming the residual data of the frequency domain component into the spatial domain component, a process of generating prediction data, and a process of reconstructing the second image 135 by using the prediction data and the residual data. Such first decoding 130 may be performed via an image reconstruction method corresponding to one of the image compression methods using frequency transformation used in the first encoding 120, such as MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1.

[0083] The AI ​​encoded data obtained through the AI ​​encoding process may include image data obtained as a result of performing the first encoding 120 on the first image 115 and AI data related to the AI ​​reduction 110 of the original image 105. The image data may be used during the first decoding 130, and the AI ​​data may be used during the AI ​​enlargement 140.

[0084] The image data may be transmitted in the form of a bitstream. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data that is a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding 120 performed on the first image 115. For example, the image data may include prediction mode information, motion information, and information related to a quantization parameter used during the first encoding 120. The image data may be generated according to the rules (for example, according to syntax) of the image compression method used during the first encoding 120 in MPEG-2, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1.

[0085] The AI ​​data is used based on the second DNN in the AI ​​upscaling 140. As described above, because the first DNN and the second DNN are jointly trained, the AI ​​data includes information that enables the AI ​​upscaling 140 to be accurately performed on the second image 135 by the second DNN. During the AI ​​decoding process, the AI ​​upscaling 140 may be performed on the second image 135 based on the AI ​​data to have a target resolution and / or quality.

[0086] The AI ​​data may be transmitted together with the image data in the form of a bitstream. Alternatively, according to an embodiment, the AI ​​data may be transmitted separately from the image data in the form of a frame or packet. The AI ​​data and the image data obtained as a result of the AI ​​encoding may be transmitted through the same network or through different networks.

[0087] Figure 2 is a block diagram of a configuration of the AI ​​decoding apparatus 200 according to an embodiment.

[0088] Reference Figure 2 , the AI ​​decoding device 200 according to the embodiment may include a receiver 210 and an AI decoder 230. The receiver 210 may include a communication interface 212, a parser 214, and an output interface 216. The AI ​​decoder 230 may include a first decoder 232 and an AI amplifier 234.

[0089] The receiver 210 receives and parses AI-encoded data obtained as a result of AI encoding, and outputs the image data and the AI ​​data distinguishably to the AI ​​decoder 230 .

[0090] The communication interface 212 receives AI-encoded data obtained as a result of AI encoding through a network. The AI-encoded data obtained as a result of performing AI encoding includes image data and AI data. The image data and AI data may be received through the same type of network or different types of networks.

[0091] The parser 214 receives the AI ​​coded data received through the communication interface 212, and parses the AI ​​coded data to distinguish the image data from the AI ​​data. For example, the parser 214 can distinguish the image data from the AI ​​data by reading the header of the data obtained from the communication interface 212. According to an embodiment, the parser 214 sends the image data and the AI ​​data to the output interface 216 via the header of the data received through the communication interface 212, and the output interface 216 sends the distinguished image data and AI data to the first decoder 232 and the AI ​​amplifier 234, respectively. At this time, it can be verified that the image data included in the AI ​​coded data is image data generated by a codec (e.g., MPEG-2, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9 or AV1). In this case, the corresponding information can be sent to the first decoder 232 through the output interface 216, so that the image data is processed by the verified codec.

[0092] According to an embodiment, the AI ​​encoded data parsed by the parser 214 may be obtained from a data storage medium, wherein the data storage medium includes a magnetic medium such as a hard disk, a floppy disk or a tape, an optical recording medium such as a CD-ROM or a DVD, or a magneto-optical medium such as a floppy disk.

[0093] The first decoder 232 reconstructs a second image 135 corresponding to the first image 115 based on the image data. The second image 135 obtained by the first decoder 232 is provided to the AI ​​amplifier 234. According to an embodiment, first decoding related information (such as prediction mode information, motion information, quantization parameter information, etc.) included in the image data may also be provided to the AI ​​amplifier 234.

[0094] Upon receiving the AI ​​data, the AI ​​amplifier 234 performs AI up-amplification on the second image 135 based on the AI ​​data. According to an embodiment, the AI ​​up-amplification may also be performed by using first decoding related information (such as prediction mode information, quantization parameter information, etc.) included in the image data.

[0095] The receiver 210 and the AI ​​decoder 230 according to the embodiment are described as separate devices, but can be implemented by one processor. In this case, the receiver 210 and the AI ​​decoder 230 can be implemented by a dedicated processor or by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). The dedicated processor can be implemented by including a memory for implementing the embodiments of the present disclosure or by including a memory processor for using an external memory.

[0096] In addition, the receiver 210 and the AI ​​decoder 230 may be configured by a plurality of processors. In this case, the receiver 210 and the AI ​​decoder 230 may be implemented by a combination of dedicated processors or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a GPU). Similarly, the AI ​​amplifier 234 and the first decoder 232 may be implemented by different processors.

[0097] The AI ​​data provided to the AI ​​amplifier 234 includes information that enables the second image 135 to be processed via AI enlargement. Here, the enlargement target corresponds to the reduction of the first DNN. Therefore, the AI ​​data includes information for verifying the reduction target of the first DNN.

[0098] Examples of the information included in the AI ​​data include difference information between the resolution of the original image 105 and the resolution of the first image 115 , and information related to the first image 115 .

[0099] The difference information may be represented as information about the degree of resolution conversion of the first image 115 compared to the original image 105 (e.g., resolution conversion rate information). In addition, since the resolution of the first image 115 is verified by the resolution of the reconstructed second image 135 and the degree of resolution conversion is verified accordingly, the difference information may be represented only as the resolution information of the original image 105. Here, the resolution information may be represented as a vertical size / horizontal size, or a ratio (16:9, 4:3, etc.) and a size of one axis. In addition, when there is pre-set resolution information, the resolution information may be represented in the form of an index or a flag.

[0100] The information related to the first image 115 may include information on any one or any combination of a bit rate of image data obtained as a result of performing the first encoding on the first image 115 or a codec type used during the first encoding of the first image 115 .

[0101] The AI ​​amplifier 234 may determine an enlargement target of the second image 135 based on any one or both of the difference information included in the AI ​​data and the information related to the first image 115. The enlargement target may indicate, for example, to what extent the resolution is to be enlarged for the second image 135. When the enlargement target is determined, the AI ​​amplifier 234 performs AI enlargement on the second image 135 through the second DNN to obtain a third image 145 corresponding to the enlargement target.

[0102] Before describing the method of performing AI up-scaling on the second image 135 according to the up-scaling target performed by the AI ​​amplifier 234, reference will be made to Figure 3 and Figure 4 Describe the AI ​​amplification process through the second DNN.

[0103] Figure 3 is a diagram illustrating a second DNN 300 for performing AI upscaling on a second image 135, and Figure 4 Is used to describe Figure 3 Schematic diagram of the convolution operation in the first convolutional layer 310 of FIG.

[0104] like Figure 3 As shown in , the second image 135 is input to the first convolutional layer 310. Figure 3 3×3×4 indicated in the first convolution layer 310 shown in indicates that a convolution process is performed on one input image by using four filter kernels of size 3×3. Four feature maps are generated by the four filter kernels as a result of the convolution process. Each feature map indicates an inherent characteristic of the second image 135. For example, each feature map may represent a vertical direction characteristic, a horizontal direction characteristic, an edge characteristic, etc. of the second image 135.

[0105] Will refer to Figure 4 The convolution operation in the first convolutional layer 310 is described in detail.

[0106] One feature map 450 may be generated by multiplying and adding the parameters of the filter kernel 430 of size 3×3 used in the first convolution layer 310 and the corresponding pixel values ​​in the second image 135. Four filter kernels are used in the first convolution layer 310, and four feature maps may be generated by convolution operations using the four filter kernels.

[0107] Figure 4 I1 to I49 indicated in the second image 135 in FIG. 4 indicate pixels in the second image 135, and F1 to F9 indicated in the filter kernel 430 indicate parameters of the filter kernel 430. In addition, M1 to M9 indicated in the feature map 450 indicate samples of the feature map 450.

[0108] exist Figure 4, the second image 135 includes 49 pixels, but the number of pixels is only an example, and when the second image 135 has a resolution of 4K, the second image 135 may include, for example, 3840×2160 pixels.

[0109] During the convolution operation process, the pixel values ​​of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination (e.g., addition) of the multiplication result values ​​may be assigned as the value of M1 of the feature map 450. When the stride of the convolution operation is 2, the pixel values ​​of I3, I4, I5, I10, I11, I12, I17, I18, and I19 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination of the multiplication result values ​​may be assigned as the value of M2 of the feature map 450.

[0110] When the filter kernel 430 moves to the last pixel of the second image 135 along the stride, a convolution operation is performed between the pixel values ​​in the second image 135 and the parameters of the filter kernel 430, and thus a feature map 450 having a specific size may be generated.

[0111] According to the present disclosure, the values ​​of the parameters of the second DNN, for example, the values ​​of the parameters of the filter kernel used in the convolution layer of the second DNN (for example, F1 to F9 of the filter kernel 430), can be optimized by joint training of the first DNN and the second DNN. As described above, the AI ​​amplifier 234 can determine the enlargement target corresponding to the reduction target of the first DNN based on the AI ​​data, and determine the parameters corresponding to the determined enlargement target as the parameters of the filter kernel used in the convolution layer of the second DNN.

[0112] The convolutional layers included in the first DNN and the second DNN can be based on the reference Figure 4 The convolution operation described above is used to perform the processing, but referring to Figure 4 The described convolution operation processing is only an example and is not limited thereto.

[0113] Return to reference Figure 3 , the feature map output from the first convolution layer 310 may be input to the first activation layer 320 .

[0114] The first activation layer 320 may assign a nonlinear feature to each feature map. The first activation layer 320 may include a sigmoid function, a Tanh function, a rectified linear unit (ReLU) function, etc., but is not limited thereto.

[0115] The first activation layer 320 to which the nonlinear feature is assigned indicates that at least one sample value of the feature map, which is the output of the first convolution layer 310, is changed. Here, the change is performed by applying the nonlinear feature.

[0116] The first activation layer 320 determines whether to send the sample values ​​of the feature map output from the first convolution layer 310 to the second convolution layer 330. For example, some sample values ​​of the feature map are activated by the first activation layer 320 and sent to the second convolution layer 330, and some sample values ​​are deactivated by the first activation layer 320 and are not sent to the second convolution layer 330. The inherent characteristics of the second image 135 represented by the feature map are emphasized by the first activation layer 320.

[0117] The feature map 325 output from the first activation layer 320 is input to the second convolutional layer 330 . Figure 3 One of the feature maps 325 shown in FIG. 3 is a feature map of the reference in the first activation layer 320. Figure 4 The result of processing the feature map 450 is described.

[0118] 3×3×4 indicated in the second convolution layer 330 indicates that a convolution process is performed on the feature map 325 by using four filters of size 3×3. The output of the second convolution layer 330 is input to the second activation layer 340. The second activation layer 340 may assign nonlinear features to input data.

[0119] The feature map 345 output from the second activation layer 340 is input to the third convolutional layer 350 . Figure 3 3×3×1 indicated in the third convolution layer 350 shown in indicates that a convolution process is performed by using one filter kernel of size 3×3 to generate one output image. The third convolution layer 350 is a layer for outputting a final image, and generates one output by using one filter kernel. According to an embodiment of the present disclosure, the third convolution layer 350 may output the third image 145 as a result of the convolution operation.

[0120] As will be described later, there may be a plurality of pieces of DNN setting information indicating the number of filter kernels of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, parameters of the filter kernels of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, and the plurality of pieces of DNN setting information may be associated with the plurality of pieces of DNN setting information of the first DNN. The association between the plurality of pieces of DNN setting information of the second DNN and the plurality of pieces of DNN setting information of the first DNN may be achieved via joint training of the first DNN and the second DNN.

[0121] exist Figure 3, the second DNN 300 includes three convolutional layers (a first convolutional layer 310, a second convolutional layer 330, and a third convolutional layer 350) and two activation layers (a first activation layer 320 and a second activation layer 340), but this is only an example, and the number of convolutional layers and activation layers may vary according to the embodiment. In addition, according to the embodiment, the second DNN 300 may be implemented as a recurrent neural network (RNN). In this case, the convolutional neural network (CNN) structure of the second DNN 300 according to the embodiment of the present disclosure is changed to an RNN structure.

[0122] According to an embodiment, the AI ​​amplifier 234 may include at least one arithmetic logic unit (ALU) for the above-mentioned convolution operation and the operation of the activation layer. The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier and an adder, wherein the multiplier performs multiplication between the sample value of the second image 135 or the feature map output from the previous layer and the sample value of the filter kernel, and the adder adds the multiplied result values. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0123] Hereinafter, a method of performing AI up-scaling on the second image 135 according to the up-scaling target, performed by the AI ​​amplifier 234, will be described.

[0124] According to an embodiment, the AI ​​amplifier 234 may store a plurality of pieces of DNN setting information that may be set in the second DNN.

[0125] Here, the DNN setting information may include information about any one or any combination of the number of convolutional layers included in the second DNN, the number of filter cores of each convolutional layer, and the parameters of each filter core. Multiple DNN setting information may correspond to various amplification targets, respectively, and the second DNN may perform operations based on the DNN setting information corresponding to the amplification target. The second DNN may have different structures based on the DNN setting information. For example, the second DNN may include three convolutional layers based on any one DNN setting information, and may include four convolutional layers based on another DNN setting information.

[0126] According to an embodiment, the DNN setting information may include only the parameters of the filter kernel used in the second DNN. In this case, the structure of the second DNN does not change, but only the parameters of the internal filter kernel may be changed based on the DNN setting information.

[0127] The AI ​​amplifier 234 may obtain DNN setting information for performing AI amplification on the second image 135 from among the plurality of DNN setting information. Each of the plurality of DNN setting information used at this time is information for obtaining the third image 145 of a predetermined resolution and / or a predetermined quality, and is jointly trained with the first DNN.

[0128] For example, one piece of DNN setting information among the multiple DNN setting information may include information for obtaining a third image 145 having a resolution twice as high as the resolution of the second image 135 (for example, a third image 145 of 4K (4096×2160) which is twice as high as 2K (2048×1080) of the second image 135), and another piece of DNN setting information may include information for obtaining a third image 145 having a resolution four times as high as the resolution of the second image 135 (for example, a third image 145 of 8K (8192×4320) which is four times as high as 2K (2048×1080) of the second image 135).

[0129] and Figure 7 The AI ​​encoding device 600 of the embodiment obtains each of the multiple DNN setting information by combining the DNN setting information of the first DNN of the AI ​​encoding device 600, and the AI ​​amplifier 234 obtains one of the multiple DNN setting information according to the enlargement ratio corresponding to the reduction ratio of the DNN setting information of the first DNN. In this regard, the AI ​​amplifier 234 can verify the information of the first DNN. In order for the AI ​​amplifier 234 to verify the information of the first DNN, the AI ​​decoding device 200 according to the embodiment receives AI data including the information of the first DNN from the AI ​​encoding device 600.

[0130] In other words, the AI ​​amplifier 234 may verify information targeted by the DNN setting information of the first DNN used to obtain the first image 115 by using information received from the AI ​​encoding device 600, and obtain the DNN setting information of the second DNN jointly trained with the DNN setting information of the first DNN.

[0131] When DNN setting information for performing AI up-scaling on the second image 135 is obtained from a plurality of pieces of DNN setting information, input data may be processed based on the second DNN operated according to the obtained DNN setting information.

[0132] For example, when any DNN setting information is obtained, Figure 3 The number of filter kernels included in each of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300 and the parameters of the filter kernels are set to the values ​​included in the obtained DNN setting information.

[0133] Figure 4 The parameters of the 3×3 filter kernel used in any convolutional layer of the second DNN are set to {1,1,1,1,1,1,1,1}, and when the DNN setting information is subsequently changed, the parameters are replaced by {2,2,2,2,2,2,2,2,2} as parameters included in the changed DNN setting information.

[0134] The AI ​​amplifier 234 may obtain DNN setting information for AI amplification from a plurality of pieces of DNN setting information based on information included in the AI ​​data, and the AI ​​data for obtaining the DNN setting information will now be described.

[0135] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification from a plurality of pieces of DNN setting information based on difference information included in the AI ​​data. For example, when it is verified based on the difference information that the resolution of the original image 105 (e.g., 4K (4096×2160)) is twice as high as the resolution of the first image 115 (e.g., 2K (2048×1080)), the AI ​​amplifier 234 may obtain DNN setting information for increasing the resolution of the second image 135 by two times.

[0136] According to another embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification of the second image 135 from a plurality of pieces of DNN setting information based on information related to the first image 115 included in the AI ​​data. The AI ​​amplifier 234 may predetermine a mapping relationship between image-related information and DNN setting information, and obtain DNN setting information mapped to information related to the first image 115.

[0137] Figure 5 is a table showing the mapping relationship between several pieces of image-related information and several pieces of DNN setting information.

[0138] By following Figure 5 In the embodiment of the present disclosure, it will be determined that the AI ​​encoding and AI decoding processing according to the embodiment of the present disclosure not only considers the change in resolution. Figure 5 As shown in , the DNN setting information may be selected by considering resolutions such as standard definition (SD), high definition (HD), or full HD, bit rates such as 10 Mbps, 15 Mbps, or 20 Mbps, and codec information such as AV1, H.264, or HEVC, either individually or collectively. With such considerations of resolution, bit rate, and codec information, it is considered that the training of each element may be performed jointly with the encoding and decoding processes during the AI ​​training process (see Fig. 9 ).

[0139] Therefore, when training Figure 5When multiple DNN setting information is provided based on image-related information including codec type, image resolution, etc. as shown in , DNN setting information for AI upscaling of the second image 135 can be obtained based on information related to the first image 115 received during AI decoding processing.

[0140] In other words, AI amplifier 234 can Figure 5 The image-related information on the left side of the table and the DNN setting information on the right side of the table are matched to use the DNN setting information based on the image-related information.

[0141] like Figure 5 As shown in , when it is verified from information related to the first image 115 that the resolution of the first image 115 is SD, the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 10Mbps, and the first encoding is performed on the first image 115 via the AV1 codec, the AI ​​amplifier 234 can use A DNN setting information among multiple DNN setting information.

[0142] In addition, when it is verified from information related to the first image 115 that the resolution of the first image 115 is HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the H.264 codec, the AI ​​amplifier 234 can use B DNN setting information among multiple DNN setting information.

[0143] In addition, when it is verified from the information related to the first image 115 that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 20Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 may use the C DNN setting information among the plurality of DNN setting information, and when it is verified that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 may use the D DNN setting information among the plurality of DNN setting information. One of the C DNN setting information and the D DNN setting information is selected based on whether the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 20Mbps or 15Mbps. Different bit rates of image data obtained when performing the first encoding on the first image 115 of the same resolution via the same codec indicate different qualities of the reconstructed image. Therefore, the first DNN and the second DNN may be jointly trained based on the image quality, and therefore, the AI ​​amplifier 234 may obtain the DNN setting information according to the bit rate of the image data indicating the quality of the second image 135.

[0144] According to another embodiment, the AI ​​amplifier 234 may obtain DNN setting information for performing AI amplification on the second image 135 from a plurality of pieces of DNN setting information, taking into account both information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and information related to the first image 115 included in the AI ​​data. For example, the AI ​​amplifier 234 may receive quantization parameter information used during the first encoding process of the first image 115 from the first decoder 232, verify the bit rate of image data obtained from the AI ​​data as the encoding result of the first image 115, and obtain DNN setting information corresponding to the quantization parameter information and the bit rate. Even when the bit rate is the same, the quality of the reconstructed image may vary according to the complexity of the image. The bit rate is a value representing the entire first image 115 on which the first encoding is performed, and the quality of each frame may vary even within the first image 115. Therefore, when the prediction mode information, motion information, and / or quantization parameter that can be obtained from the first decoder 232 for each frame are considered together, DNN setting information more suitable for the second image 135 may be obtained compared to when only the AI ​​data is used.

[0145] In addition, according to an embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information. The identifier of the DNN setting information is information for distinguishing a pair of DNN setting information jointly trained between the first DNN and the second DNN, so that AI amplification is performed on the second image 135 to an amplification target corresponding to the reduction target of the first DNN. After obtaining the identifier of the DNN setting information included in the AI ​​data, the AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information corresponding to the identifier of the DNN setting information. For example, an identifier indicating each of the multiple DNN setting information that can be set in the first DNN and an identifier indicating each of the multiple DNN setting information that can be set in the second DNN may be pre-specified. In this case, the same identifier may be specified for a pair of DNN setting information that can be set in each of the first DNN and the second DNN. The AI ​​data may include an identifier of the DNN setting information set in the first DNN for AI reduction of the original image 105. The AI ​​amplifier 234 receiving the AI ​​data may perform AI up-amplification on the second image 135 by using DNN setting information indicated by an identifier included in the AI ​​data among the plurality of pieces of DNN setting information.

[0146] In addition, according to an embodiment, the AI ​​data may include DNN setting information. The AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information included in the AI ​​data after obtaining the DNN setting information.

[0147] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter cores of each convolutional layer, the parameters of each filter core, etc.) are stored in the form of a lookup table, the AI ​​amplifier 234 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on the information included in the AI ​​data, and perform AI amplification on the second image 135 by using the obtained DNN setting information.

[0148] According to an embodiment, when the structure of the DNN corresponding to the amplification target is determined, the AI ​​amplifier 234 may obtain DNN setting information corresponding to the determined structure of the DNN, for example, parameters of a filter kernel.

[0149] The AI ​​amplifier 234 obtains DNN setting information of the second DNN through AI data including information related to the first DNN, and performs AI amplification on the second image 135 through the second DNN set based on the obtained DNN setting information, and in this case, memory usage and throughput can be reduced compared to directly analyzing features of the second image 135 for amplification.

[0150] According to an embodiment, when the second image 135 includes a plurality of frames, the AI ​​amplifier 234 may independently obtain DNN setting information for a specific number of frames, or may obtain common DNN setting information for all frames.

[0151] Figure 6 is a diagram illustrating a second image 135 including a plurality of frames.

[0152] like Figure 6 As shown in , the second image 135 may include frames t0 to tn.

[0153] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information of the second DNN through the AI ​​data, and perform AI amplification on frames t0 to tn based on the obtained DNN setting information. In other words, frames t0 to tn may be processed via AI amplification based on the common DNN setting information.

[0154] According to another embodiment, the AI ​​amplifier 234 may perform AI amplification on some frames (e.g., frame t0 to frame ta) among frame t0 to frame tn by using the "A" DNN setting information obtained from the AI ​​data, and perform AI amplification on frame ta+1 to frame tb by using the "B" DNN setting information obtained from the AI ​​data. In addition, the AI ​​amplifier 234 may perform AI amplification on frame tb+1 to frame tn by using the "C" DNN setting information obtained from the AI ​​data. In other words, the AI ​​amplifier 234 may independently obtain DNN setting information for each group including several frames among a plurality of frames, and perform AI amplification on the frames included in each group by using the independently obtained DNN setting information.

[0155] According to another embodiment, the AI ​​amplifier 234 may independently obtain DNN setting information for each frame forming the second image 135. In other words, when the second image 135 includes three frames, the AI ​​amplifier 234 may perform AI amplification on the first frame by using the DNN setting information obtained in relation to the first frame, perform AI amplification on the second frame by using the DNN setting information obtained in relation to the second frame, and perform AI amplification on the third frame by using the DNN setting information obtained in relation to the third frame. According to the method of obtaining DNN setting information based on the information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and the information related to the first image 115 included in the AI ​​data as described above, DNN setting information may be independently obtained for each frame included in the second image 135. This is because the mode information, quantization parameter information, etc. may be independently determined for each frame included in the second image 135.

[0156] According to another embodiment, the AI ​​data may include information about which frame the DNN setting information obtained based on the AI ​​data is valid for. For example, when the AI ​​data includes information indicating that the DNN setting information is valid until frame ta, the AI ​​amplifier 234 performs AI amplification on frame t0 to frame ta by using the DNN setting information obtained based on the AI ​​data. In addition, when another piece of AI data includes information indicating that the DNN setting information is valid until frame tn, the AI ​​amplifier 234 performs AI amplification on frame ta+1 to frame tn by using the DNN setting information obtained based on the other piece of AI data.

[0157] In the following, reference will be made to Figure 7 An AI encoding apparatus 600 for performing AI encoding on an original image 105 is described.

[0158] Figure 7 is a block diagram of a configuration of an AI encoding apparatus 600 according to an embodiment.

[0159] Reference Figure 7, the AI ​​encoding device 600 may include an AI encoder 610 and a transmitter 630. The AI ​​encoder 610 may include an AI reducer 612 and a first encoder 614. The transmitter 630 may include a data processor 632 and a communication interface 634.

[0160] exist Figure 7 , the AI ​​encoder 610 and the transmitter 630 are shown as independent devices, but the AI ​​encoder 610 and the transmitter 630 may be implemented by one processor. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a graphics processing unit GPU). The dedicated processor may be implemented by including a memory for implementing an embodiment of the present disclosure or by including a memory processor for using an external memory.

[0161] In addition, the AI ​​encoder 610 and the transmitter 630 may be configured by a plurality of processors. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a combination of dedicated processors or by a combination of software and a plurality of general-purpose processors (such as an AP, a CPU, or a GPU). The AI ​​reducer 612 and the first encoder 614 may be implemented by different processors.

[0162] The AI ​​encoder 610 performs AI reduction on the original image 105 and first encoding on the first image 115, and transmits the AI ​​data and the image data to the transmitter 630. The transmitter 630 transmits the AI ​​data and the image data to the AI ​​decoding apparatus 200.

[0163] The image data includes data obtained as a result of performing the first encoding on the first image 115. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data that is a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding process of the first image 115. For example, the image data may include prediction mode information, motion information, quantization parameter information, etc., used to perform the first encoding on the first image 115.

[0164] The AI ​​data includes information that enables AI up-scaling to be performed on the second image 135 to an up-scaling target corresponding to the down-scaling target of the first DNN. According to an embodiment, the AI ​​data may include difference information between the original image 105 and the first image 115. In addition, the AI ​​data may include information related to the first image 115. The information related to the first image 115 may include information about any one or any combination of a resolution of the first image 115, a bit rate of image data obtained as a result of performing the first encoding on the first image 115, or a codec type used during the first encoding of the first image 115.

[0165] According to an embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information such that the AI ​​up-scaling is performed on the second image 135 to a zoom-in target corresponding to a zoom-out target of the first DNN.

[0166] In addition, according to an embodiment, the AI ​​data may include DNN setting information that can be set in the second DNN.

[0167] The AI ​​downsizer 612 may obtain the first image 115 obtained by performing AI downsizing on the original image 105 through the first DNN. The AI ​​downsizer 1212 may determine a downsizing target of the original image 105 based on a predetermined criterion.

[0168] In order to obtain the first image 115 matching the reduction target, the AI ​​reducer 612 may store a plurality of DNN setting information that may be set in the first DNN. The AI ​​reducer 1212 obtains DNN setting information corresponding to the reduction target from the plurality of DNN setting information, and performs AI reduction on the original image 105 by using the first DNN set according to the obtained DNN setting information.

[0169] Each of the plurality of pieces of DNN setting information may be trained to obtain a first image 115 of a predetermined resolution and / or a predetermined quality. For example, any one of the plurality of pieces of DNN setting information may include information for obtaining a first image 115 having a resolution half that of the original image 105 (e.g., a first image 115 of 2K (2048×1080) which is half that of the 4K (4096×2160) of the original image 105), and another piece of DNN setting information may include information for obtaining a first image 115 having a resolution one-quarter that of the original image 105 (e.g., a first image 115 of 2K (2048×1080) which is one-quarter that of the 8K (8192×4320) of the original image 105).

[0170] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter cores of each convolutional layer, the parameters of each filter core, etc.) are stored in the form of a lookup table, the AI ​​downsizer 612 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on the reduction target, and perform AI downsizing on the original image 105 by using the obtained DNN setting information.

[0171] According to an embodiment, the AI ​​reducer 612 may determine the structure of the DNN corresponding to the reduction target, and obtain DNN setting information corresponding to the determined structure of the DNN, for example, obtain parameters of a filter kernel.

[0172] Since the first DNN and the second DNN are jointly trained, the plurality of pieces of DNN setting information for performing AI reduction on the original image 105 may have an optimal value. Here, each piece of DNN setting information includes any one or any combination of the number of convolutional layers included in the first DNN, the number of filter cores of each convolutional layer, and the parameters of each filter core.

[0173] The AI ​​downsizer 612 may set the first DNN using the DNN setting information obtained for performing AI downsizing on the original image 105 to obtain the first image 115 of a specific resolution and / or a specific quality through the first DNN. When the DNN setting information for performing AI downsizing on the original image 105 is obtained from a plurality of pieces of DNN setting information, each layer in the first DNN may process input data based on information included in the DNN setting information.

[0174] Hereinafter, a method of determining a downscaling target performed by the AI ​​downscaling unit 612 will be described. The downscaling target may indicate, for example, how much to reduce the resolution from the original image 105 to obtain the first image 115.

[0175] According to an embodiment, the AI ​​reducer 612 may determine the reduction target based on any one or any combination of the compression ratio of the original image 105 (e.g., the resolution difference between the original image 105 and the first image 115, the target bit rate, etc.), the compression quality (e.g., the type of bit rate), the compression history information and the type.

[0176] For example, the AI ​​downsizer 612 may determine a downsizing target based on a compression ratio, compression quality, etc. that is preset or input from a user.

[0177] As another example, the AI ​​downsizer 612 may determine the downsizing target by using the compression history information stored in the AI ​​encoding device 600. For example, according to the compression history information that can be used by the AI ​​encoding device 600, the encoding quality, compression ratio, etc. preferred by the user may be determined, and the downsizing target may be determined according to the encoding quality determined based on the compression history information. For example, the resolution, quality, etc. of the first image 115 may be determined based on the encoding quality most frequently used according to the compression history information.

[0178] As another example, the AI ​​downsizer 612 may determine the downsizing target based on encoding qualities used more frequently than a threshold value according to compression history information (eg, an average quality of encoding qualities used more frequently than the threshold value).

[0179] As another example, the AI ​​downsizer 612 may determine the downscaling target based on the resolution, type (eg, file format), etc. of the original image 105 .

[0180] According to an embodiment, when the original image 105 includes a plurality of frames, the AI ​​downsizer 612 may independently determine downsizing targets for a specific number of frames, or may determine downsizing targets for all frames.

[0181] According to an embodiment, the AI ​​downsizer 612 may divide the frames included in the original image 105 into a certain number of groups and independently determine a downsizing target for each group. The same or different downsizing targets may be determined for each group. According to each group, the number of frames included in the group may be the same or different.

[0182] According to another embodiment, the AI ​​downsizer 612 may independently determine a downscaling target for each frame included in the original image 105. The same or different downscaling targets may be determined for each frame.

[0183] Hereinafter, an example of the structure of the first DNN 700 on which the AI ​​reduction is based will be described.

[0184] Figure 8 is a diagram illustrating a first DNN 700 for performing AI downscaling on an original image 105.

[0185] like Figure 8 As shown in FIG. 1 , the original image 105 is input to the first convolution layer 710. The first convolution layer 710 performs a convolution process on the original image 105 by using 32 filter kernels of a size of 5×5. The 32 feature maps generated as a result of the convolution process are input to the first activation layer 720. The first activation layer 720 may assign nonlinear features to the 32 feature maps.

[0186] The first activation layer 720 determines whether to transmit sample values ​​of the feature map output from the first convolution layer 710 to the second convolution layer 730. For example, some sample values ​​of the feature map are activated by the first activation layer 720 and transmitted to the second convolution layer 730, and some sample values ​​are deactivated by the first activation layer 720 and are not transmitted to the second convolution layer 730. Information represented by the feature map output from the first convolution layer 710 is emphasized by the first activation layer 720.

[0187] The output 725 of the first activation layer 720 is input to the second convolution layer 730. The second convolution layer 730 performs a convolution process on the input data by using 32 filter kernels of a size of 5×5. The 32 feature maps output as a result of the convolution process are input to the second activation layer 740, and the second activation layer 740 may assign nonlinear features to the 32 feature maps.

[0188] The output 745 of the second activation layer 740 is input to the third convolution layer 750. The third convolution layer 750 performs a convolution process on the input data by using one filter kernel of size 5×5. As a result of the convolution process, an image may be output from the third convolution layer 750. The third convolution layer 750 generates an output by using the one filter kernel as a layer for outputting a final image. According to an embodiment of the present disclosure, the third convolution layer 750 may output the first image 115 as a result of the convolution operation.

[0189] There may be a plurality of pieces of DNN setting information indicating the number of filter cores of the first convolution layer 710, the second convolution layer 730, and the third convolution layer 750 of the first DNN 700, a parameter of each filter core of the first convolution layer 710, the second convolution layer 730, and the third convolution layer 750 of the first DNN 700, and the like, and the plurality of pieces of DNN setting information may be associated with the plurality of pieces of DNN setting information of the second DNN. The association between the plurality of pieces of DNN setting information of the first DNN and the plurality of pieces of DNN setting information of the second DNN may be achieved through joint training of the first DNN and the second DNN.

[0190] exist Figure 8 , the first DNN 700 includes three convolutional layers (a first convolutional layer 710, a second convolutional layer 730, and a third convolutional layer 750) and two activation layers (a first activation layer 720 and a second activation layer 740), but this is only an example, and the number of convolutional layers and activation layers may vary according to the embodiment. In addition, according to the embodiment, the first DNN 700 may be implemented as an RNN. In this case, the CNN structure of the first DNN 700 according to the embodiment of the present disclosure is changed to an RNN structure.

[0191] According to an embodiment, the AI ​​reducer 712 may include at least one ALU for the above-mentioned convolution operation and the operation of the activation layer. The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier and an adder, wherein the multiplier performs multiplication between the sample value of the original image 105 or the feature map output from the previous layer and the sample value of the filter kernel, and the adder adds the multiplied result values. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0192] Return to reference Figure 7 Upon receiving the first image 115 from the AI ​​downsizer 612, the first encoder 614 may reduce the amount of information of the first image 115 by performing the first encoding on the first image 115. Image data corresponding to the first image 115 may be obtained as a result of the first encoding performed by the first encoder 614.

[0193] The data processor 632 processes one or both of the AI ​​data and the image data to be sent in a specific form. For example, when the AI ​​data and the image data are to be sent in the form of a bit stream, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and send the image data and the AI ​​data in the form of one bit stream through the communication interface 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and send each of the bit streams corresponding to the AI ​​data and the bit stream corresponding to the image data through the communication interface 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a frame or a packet, and send the image data in the form of a bit stream and send the AI ​​data in the form of a frame or a packet through the communication interface 634.

[0194] The communication interface 634 transmits the AI-encoded data obtained as a result of performing the AI ​​encoding through the network. The AI-encoded data obtained as a result of performing the AI ​​encoding includes image data and AI data. The image data and the AI ​​data may be transmitted through the same type of network or different types of networks.

[0195] According to an embodiment, the AI ​​encoded data obtained as a result of processing by the data processor 632 may be stored in a data storage medium, wherein the data storage medium includes a magnetic medium such as a hard disk, a floppy disk, or a tape, an optical recording medium such as a CD-ROM or a DVD, or a magneto-optical medium such as a floppy disk.

[0196] In the following, reference will be made to Fig. 9A method of jointly training the first DNN 700 and the second DNN 300 is described.

[0197] Fig. 9 is a diagram for describing a method of training a first DNN 700 and a second DNN 300.

[0198] In an embodiment, the original image 105 on which AI encoding is performed is reconstructed into the third image 145 via the AI ​​decoding process, and in order to maintain the similarity between the original image 105 and the third image 145 obtained as a result of AI decoding, an association is established between the AI ​​encoding process and the AI ​​decoding process. In other words, information lost in the AI ​​encoding process is reconstructed during the AI ​​decoding process, and in this regard, the first DNN 700 and the second DNN 300 are jointly trained.

[0199] In order to perform accurate AI decoding, the Fig. 9 The quality loss information 830 corresponding to the result of comparing the third training image 804 and the original training image 801 shown in FIG. Therefore, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0200] First, we will describe Fig. 9 The training process shown in .

[0201] exist Fig. 9 , the original training image 801 is an image to be subjected to AI reduction, and the first training image 802 is an image obtained by performing AI reduction on the original training image 801. In addition, the third training image 804 is an image obtained by performing AI enlargement on the first training image 802.

[0202] The original training image 801 includes a still image or a moving image including a plurality of frames. According to an embodiment, the original training image 801 may include a luminance image extracted from a still image or a moving image including a plurality of frames. In addition, according to an embodiment, the original training image 801 may include a block image extracted from a still image or a moving image including a plurality of frames. When the original training image 801 includes a plurality of frames, the first training image 802, the second training image, and the third training image 804 also each include a plurality of frames. When the plurality of frames of the original training image 801 are sequentially input to the first DNN 700, the plurality of frames of the first training image 802, the second training image, and the third training image 804 may be sequentially obtained by the first DNN 700 and the second DNN 300.

[0203] For joint training of the first DNN 700 and the second DNN 300, an original training image 801 is input to the first DNN 700. The original training image 801 input to the first DNN 700 is output as a first training image 802 via AI downscaling, and the first training image 1702 is input to the second DNN 300. A third training image 804 is output as a result of performing AI upscaling on the first training image 802.

[0204] Reference Fig. 9 , the first training image 802 is input to the second DNN 300, and according to an embodiment, the second training image obtained when the first encoding and the first decoding are performed on the first training image 802 may be input to the second DNN 300. In order to input the second training image to the second DNN 300, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 codecs may be used. Any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 codecs may be used to perform the first encoding on the first training image 802 and perform the first decoding on the image data corresponding to the first training image 802.

[0205] Reference Fig. 9 , separately from the first training image 802 output by the first DNN 700, a reduced training image 803 obtained by performing conventional downscaling on the original training image 801 is obtained. Here, conventional downscaling may include any one or any combination of bilinear scaling, bicubic scaling, lanczos scaling, or step scaling.

[0206] In order to prevent the structural features of the first image 115 from greatly deviating from the structural features of the original image 105 , a reduced training image 803 is obtained to preserve the structural features of the original training image 801 .

[0207] Before performing training, the first DNN 700 and the second DNN 300 may be set to predetermined DNN setting information. When performing training, structure loss information 810, complexity loss information 820, and quality loss information 830 may be determined.

[0208] The structural loss information 810 may be determined based on a result of comparing the reduced training image 803 and the first training image 802. For example, the structural loss information 810 may correspond to a difference between structural information of the reduced training image 803 and structural information of the first training image 802. The structural information may include various features that may be extracted from an image, such as brightness, contrast, a histogram, etc. of the image. The structural loss information 810 indicates how much structural information of the original training image 801 is retained in the first training image 802. When the structural loss information 810 is small, the structural information of the first training image 802 is similar to the structural information of the original training image 801.

[0209] The complexity loss information 820 may be determined based on the spatial complexity of the first training image 802. For example, the total variance value of the first training image 802 may be used as the spatial complexity. The complexity loss information 820 is related to the bit rate of the image data obtained by performing the first encoding on the first training image 802. It is defined that when the complexity loss information 820 is small, the bit rate of the image data is low.

[0210] The quality loss information 830 may be determined based on a result of comparing the original training image 801 and the third training image 804. The quality loss information 830 may include any one or any combination of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, and a video multi-method assessment fusion (VMAF) value about the difference between the original training image 801 and the third training image 804. The quality loss information 830 indicates how similar the third training image 804 is to the original training image 801. When the quality loss information 830 is small, the third training image 804 is more similar to the original training image 801.

[0211] Reference Fig. 9 , the structure loss information 810, the complexity loss information 820, and the quality loss information 830 are used to train the first DNN 700, and the quality loss information 830 is used to train the second DNN 3000. In other words, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0212] The first DNN 700 may update parameters so that the final loss information determined based on the loss information 810 to 830 is reduced or minimized. In addition, the second DNN 300 may update parameters so that the quality loss information 830 is reduced or minimized.

[0213] Final loss information for training the first DNN 700 and the second DNN 300 may be determined as Equation 1 below.

[0214] [Equation 1]

[0215] LossDS = a×structural loss information + b×complexity loss information + c×quality loss information

[0216] LossUS=d×quality loss information

[0217] In Equation 1, LossDS indicates final loss information to be reduced or minimized to train the first DNN 700, and LossUS indicates final loss information to be reduced or minimized to train the second DNN 300. In addition, a, b, c, and d may be predetermined weights.

[0218] In other words, the first DNN 700 updates the parameters in the direction in which the LossDS of Equation 1 decreases, and the second DNN 300 updates the parameters in the direction in which the LossDS decreases. When the parameters of the first DNN 700 are updated according to the LossDS derived during training, the first training image 802 obtained based on the updated parameters becomes different from the previous first training image 802 obtained based on the non-updated parameters, and therefore, the third training image 804 also becomes different from the previous third training image 804. When the third training image 804 becomes different from the previous third training image 804, the quality loss information 830 is also re-determined, and the second DNN 300 updates the parameters accordingly. When the quality loss information 830 is re-determined, the LossDS is also re-determined, and the first DNN 700 updates the parameters according to the re-determined LossDS. In other words, the update of the parameters of the first DNN 700 leads to the update of the parameters of the second DNN 300, and the update of the parameters of the second DNN 300 leads to the update of the parameters of the first DNN 700. In other words, because the first DNN 700 and the second DNN 300 are jointly trained by sharing the quality loss information 830, the parameters of the first DNN 700 and the parameters of the second DNN 300 can be jointly optimized.

[0219] Referring to Equation 1, it is verified that LossUS is determined according to the quality loss information 830 , but this is only an example, and LossUS may be determined based on one or both of the structure loss information 810 and the complexity loss information 820 , as well as the quality loss information 800 .

[0220] In the above, it has been described that the AI ​​amplifier 234 of the AI ​​decoding device 200 and the AI ​​reducer 612 of the AI ​​encoding device 600 store multiple DNN setting information, and now a method of training each of the multiple DNN setting information stored in the AI ​​amplifier 234 and the AI ​​reducer 612 will be described.

[0221] As described with reference to Equation 1, the first DNN 700 updates parameters by considering the similarity between the structural information of the first training image 802 and the structural information of the original training image 801 (structural loss information 810), the bit rate of the image data obtained as a result of performing the first encoding on the first training image 802 (complexity loss information 820), and the difference between the third training image 804 and the original training image 801 (quality loss information 830).

[0222] The parameters of the first DNN 700 can be updated so that: a first training image 802 having similar structural information to the original training image 801 is obtained and image data having a small bit rate is obtained when the first encoding is performed on the first training image 802, and at the same time, the second DNN 300 that performs AI amplification on the first training image 802 obtains a third training image 804 similar to the original training image 801.

[0223] The direction in which the parameters of the first DNN 700 are optimized may be changed by adjusting the weights a, b, and c of Equation 1. For example, when the weight b is determined to be high, the parameters of the first DNN 700 may be updated by prioritizing a low bit rate over the high quality of the third training image 804. In addition, when the weight c is determined to be high, the parameters of the first DNN 700 may be updated by prioritizing the high quality of the third training image 804 over a high bit rate or maintaining the structural information of the original training image 801.

[0224] In addition, the direction in which the parameters of the first DNN 700 are optimized may vary according to the type of codec used to perform the first encoding on the first training image 802. This is because the second training image to be input to the second DNN 300 may vary according to the type of codec.

[0225] In other words, the parameters of the first DNN 700 and the parameters of the second DNN 300 may be jointly updated based on the weights a, b, and c and the type of codec used to perform the first encoding on the first training image 802. Therefore, when the first DNN 700 and the second DNN 300 are trained after the weights a, b, and c are each determined to be a specific value and the type of codec is determined to be a specific type, the parameters of the first DNN 700 and the parameters of the second DNN 300 that are associated with and optimized to each other may be determined.

[0226] In addition, when the first DNN 700 and the second DNN 300 are trained after changing the weights a, b, and c and the type of codec, the parameters of the first DNN 700 and the parameters of the second DNN 300 that are associated and optimized with each other may be determined. In other words, when the first DNN 700 and the second DNN 300 are trained while changing the values ​​of the weights a, b, and c and the type of codec, a plurality of pieces of DNN setting information that are jointly trained with each other may be determined in the first DNN 700 and the second DNN 300.

[0227] As mentioned above Figure 5 As described, a plurality of pieces of DNN setting information of the first DNN 700 and the second DNN 300 may be mapped to information related to the first image. In order to set such a mapping relationship, the first training image 802 output from the first DNN 700 may be first encoded via a specific codec according to a specific bit rate, and the second training image obtained by performing a first decoding on a bit stream obtained as a result of performing the first encoding may be input to the second DNN 300. In other words, by training the first DNN 700 and the second DNN 300 after setting an environment such that the first encoding is performed on the first training image 802 of a specific resolution via a specific codec according to a specific bit rate, a pair of DNN setting information mapped to the following items may be determined: the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained as a result of performing the first encoding on the first training image 802. By differently changing the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained according to the first encoding of the first training image 802, the mapping relationship between multiple DNN setting information of the first DNN 700 and the second DNN 300 and multiple information related to the first image can be determined.

[0228] Fig.10 is a diagram for describing a training process of the first DNN 700 and the second DNN by the training device 1000.

[0229] Reference Fig. 9 The training of the described first DNN 700 and the second DNN 300 may be performed by the training device 1000. The training device 1000 includes the first DNN 700 and the second DNN 300. The training device 1000 may be, for example, the AI ​​encoding device 600 or a separate server. The DNN setting information of the second DNN 300 obtained as a result of the training is stored in the AI ​​decoding device 200.

[0230] Reference Fig.10In operation S840 and operation S845, the training device 1000 initially sets the DNN setting information of the first DNN 700 and the second DNN 300. Therefore, the first DNN 700 and the second DNN 300 may operate according to the predetermined DNN setting information. The DNN setting information may include information about any one or any combination of the number of convolutional layers included in the first DNN 700 and the second DNN 300, the number of filter kernels of each convolutional layer, the size of the filter kernel of each convolutional layer, and the parameters of each filter kernel.

[0231] In operation S850, the training apparatus 1000 inputs the original training image 801 into the first DNN 700. The original training image 801 may include at least one frame included in a still image or a moving image.

[0232] In operation S855, the first DNN 700 processes the original training image 801 according to the initially set DNN setting information, and outputs the first training image 802 obtained by performing AI reduction on the original training image 801. Fig.10 , the first training image 802 output from the first DNN 700 is directly input to the second DNN 300, but the first training image 802 output from the first DNN 700 may be input to the second DNN 300 by the training device 1000. In addition, the training device 1000 may perform first encoding and first decoding on the first training image 802 via a specific codec, and then input the second training image to the second DNN 300.

[0233] In operation S860, the second DNN 300 processes the first training image 802 or the second training image according to initially set DNN setting information, and outputs a third training image 804 obtained by performing AI up-scaling on the first training image 802 or the second training image.

[0234] In operation S865 , the training apparatus 1000 calculates complexity loss information 820 based on the first training image 802 .

[0235] In operation S870 , the training apparatus 1000 calculates structure loss information 810 by comparing the reduced training image 803 and the first training image 802 .

[0236] In operation S875 , the training apparatus 1000 calculates quality loss information 830 by comparing the original training image 801 and the third training image 804 .

[0237] In operation S880 , the initially set DNN setting information is updated via a back propagation process based on the final loss information. The training apparatus 1000 may calculate the final loss information for training the first DNN 700 based on the complexity loss information 820 , the structure loss information 810 , and the quality loss information 830 .

[0238] In operation S885, the second DNN 300 updates the initially set DNN setting information through a back propagation process based on the quality loss information 830 or the final loss information. The training apparatus 1000 may calculate the final loss information for training the second DNN 300 based on the quality loss information 830.

[0239] Then, the training device 1000, the first DNN 700, and the second DNN 300 may repeat operations S850 to S885 until the final loss information is minimized to update the DNN setting information. At this time, during each repetition, the first DNN 700 and the second DNN 300 operate according to the DNN setting information updated in the previous operation.

[0240] Table 1 below shows the effects when AI encoding and AI decoding are performed on the original image 105 according to an embodiment of the present disclosure and when encoding and decoding are performed on the original image 105 via HEVC.

[0241]

Table 1

[0242]

[0243] As shown in Table 1, although the subjective image quality when AI encoding and AI decoding are performed on content including 300 frames of 8K resolution according to an embodiment of the present disclosure is higher than the subjective image quality when encoding and decoding are performed via HEVC, the bit rate is reduced by at least 50%.

[0244] Fig.11 is a diagram of a first device 20 for performing AI downscaling on an original image 105 and a second device 40 for performing AI upscaling on a second image 135 .

[0245] The first device 20 receives the original image 105 and provides the image data 25 and the AI ​​data 30 to the second device 40 by using the AI ​​downsizer 1124 and the transform-based encoder 1126. According to an embodiment, the image data 25 corresponds to Figure 1 The AI ​​data 30 corresponds to the image data Figure 1 In addition, according to an embodiment, the transform-based encoder 1126 corresponds to Figure 7 The first encoder 614 and the AI ​​reducer 1124 correspond to Figure 7 AI reducer 612.

[0246] The second device 40 receives the AI ​​data 30 and the image data 25, and obtains a third image 145 by using a transform-based decoder 1146 and an AI amplifier 1144. According to an embodiment, the transform-based decoder 1146 corresponds to Figure 2 The first decoder 232, and the AI ​​amplifier 1144 corresponds to Figure 2 AI amplifier 234.

[0247] According to an embodiment, the first device 20 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the first device 20 executes the reference to the computer program according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, reference will be made to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0248] According to an embodiment, the second device 40 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the second device 40 executes the reference to the computer program according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, reference will be made to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0249] exist Fig.11 , the configuration controller 1122 of the first device 20 receives at least one input value 10. According to an embodiment, the at least one input value 10 may include any one or any combination of a target resolution difference for the AI ​​reducer 1124 and the AI ​​amplifier 1144, a bit rate of the image data 25 for the transform-based encoder 1126, a bit rate type (e.g., a variable bit rate type, a constant bit rate type, or an average bit rate type) and a codec type of the image data 25. The at least one input value 10 may include a value pre-stored in the first device 20 or a value input from a user.

[0250] The configuration controller 1122 controls the operations of the AI ​​downsizer 1124 and the transform-based encoder 1126 based on the received input value 10. According to an embodiment, the configuration controller 1122 obtains DNN setting information for the AI ​​downsizer 1124 according to the received input value 10, and sets the AI ​​downsizer 1124 using the obtained DNN setting information. According to an embodiment, the configuration controller 1122 may send the received input value 10 to the AI ​​downsizer 1124, and the AI ​​downsizer 1124 may obtain DNN setting information for performing AI downsizing on the original image 105 based on the received input value 10. According to an embodiment, the configuration controller 1122 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI downsizing is applied and tone mapping information of a high dynamic range (HDR)) to the AI ​​downsizer 1124 together with the input value 10, and the AI ​​downsizer 1124 may obtain the DNN setting information in consideration of the input value 10 and the additional information. According to an embodiment, the configuration controller 1122 sends at least a portion of the received input value 10 to the transform-based encoder 1126, and the transform-based encoder 1126 performs a first encoding on the first image 115 via a bit rate of a specific value, a specific type of bit rate, and a specific codec.

[0251] The AI ​​reducer 1124 receives the original image 105 and performs a reference Figure 1 , Figure 7 , Figure 8 , Fig. 9 and Fig.10 The operations described in any one or any combination thereof are performed to obtain the first image 115.

[0252] According to an embodiment, the AI ​​data 30 is provided to the second device 40. The AI ​​data 30 may include one or both of the resolution difference information between the original image 105 and the first image 115 and the information related to the first image 115. The resolution difference information may be determined based on the target resolution difference of the input value 10, and the information related to the first image 115 may be determined based on any one or any combination of the target bit rate, the bit rate type, and the codec type. According to an embodiment, the AI ​​data 30 may include parameters used during AI upscaling. The AI ​​data 30 may be provided to the second device 40 from the AI ​​downsizer 1124.

[0253] The image data 25 is obtained when the original image 105 is processed by the transform-based encoder 1126 and is sent to the second device 40. The transform-based encoder 1126 may process the first image 115 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or VA1.

[0254] The configuration controller 1142 of the second device 40 controls the operation of the AI ​​amplifier 1144 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 obtains DNN setting information for the AI ​​amplifier 1144 according to the received AI data 30, and sets the AI ​​amplifier 1144 using the obtained DNN setting information. According to an embodiment, the configuration controller 1142 may send the received AI data 30 to the AI ​​amplifier 1144, and the AI ​​amplifier 1144 may obtain DNN setting information for performing AI amplification on the second image 135 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI amplification is applied and tone mapping information of HDR) to the AI ​​amplifier 1144 together with the AI ​​data 30, and the AI ​​amplifier 1144 may obtain the DNN setting information in consideration of the AI ​​data 30 and the additional information. According to an embodiment, the AI ​​amplifier 1144 may receive AI data 30 from the configuration controller 1142, receive any one or any combination of prediction mode information, motion information, and quantization parameter information from the transform-based decoder 1146, and obtain DNN setting information based on the AI ​​data 30 and any one or any combination of prediction mode information, motion information, and quantization parameter information.

[0255] The transform-based decoder 1146 may process the image data 25 to reconstruct the second image 135. The transform-based decoder 1146 may process the image data 25 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1.

[0256] The AI ​​amplifier 1144 may obtain the third image 145 by performing AI amplification on the second image 135 provided from the transform-based decoder 1146 based on the set DNN setting information.

[0257] The AI ​​reducer 1124 may include a first DNN, and the AI ​​amplifier 1144 may include a second DNN, and according to an embodiment, according to the reference Fig. 9 and Fig.10 The described training method trains the first DNN and the DNN setting information of the second DNN.

[0258] Fig.12 is a block diagram showing the structure of DNN 900 according to an embodiment of the present disclosure.

[0259] Reference Fig.12 , DNN 900 may include multiple layers and may also include low-precision transformation units between the multiple layers.

[0260] The plurality of layers are now described as including a first layer 910 and a second layer 920, and the second layer 920 is now described as a layer adjacent to the first layer 910. The DNN 900 may include a low-precision transform unit 950 between the first layer 910 and the second layer 920.

[0261] Fig.12 Only two layers are shown, but the present disclosure is not limited thereto. DNN 900 may include more than two layers. For example, DNN 900 may further include a third layer adjacent to the second layer 920, and may further include a low-precision transform unit between the second layer 920 and the third layer.

[0262] The first layer 910 and the second layer 920 may include a first convolution layer 911 and a second convolution layer 921, respectively, and include a first activation layer 912 and a second activation layer 922, respectively. The first convolution layer 911 and the second convolution layer 921 may perform a convolution operation between input data (image) and a filter kernel to generate a feature map, and the first activation layer 912 and the second activation layer 922 may apply an activation function to the generated feature map.

[0263] Fig.12 The DNN 900 may be an example of the first DNN 700 included in the AI ​​encoding device 600 or the second DNN 300 included in the AI ​​decoding device 200. For example, the first convolution layer (Conv1) 911 may correspond to Figure 3 The first convolutional layer 310 and Figure 8 The first convolutional layer 710 of . The second convolutional layer (Conv2) 921 may correspond to Figure 3 The second convolutional layer 330 and Figure 8 The first activation layer 912 may correspond to the second convolution layer 730 of Figure 3 The first activation layer 320 and Figure 8 The second activation layer 922 may correspond to the first activation layer 720. Figure 3 The second activation layer 340 and Figure 8 The second activation layer 740 is formed by a plurality of layers of a plurality of active layers.

[0264] Now refer to Fig.13 A data processing method performed by DNN 900 according to an embodiment of the present disclosure is described.

[0265] The first convolution layer 911 and the first activation layer 912 included in the first layer 910 will be focused on and described.

[0266] A case where the DNN 900 according to an embodiment of the present disclosure is the first DNN 700 included in the AI ​​encoding device 600 will now be described.

[0267] Fig.13Is used to describe Fig.12 9 is a diagram of a data processing method in the first layer 910.

[0268] Reference Fig.13 According to an embodiment of the present disclosure, the first convolution layer 911 may include a plurality of filter kernels, that is, the first filter kernel K 1.1 , the second filter kernel K 1.2 , ... and the Nth filter kernel K 1.N The first convolution layer 911 according to an embodiment of the present disclosure may generate a feature map corresponding to each of the plurality of filter kernels by performing a convolution operation between data input to the first convolution layer 911 and each of the plurality of filter kernels.

[0269] For example, when the original image 105 is input, the first convolution layer 911 may perform the first filter kernel K among the plurality of filter kernels. 1.1 The convolution operation between the original image 105 and the first feature map F is generated. 1.1 When the original image 105 is input, the first convolution layer 911 can also perform the second filter kernel K in the plurality of filter kernels. 1.2 The convolution operation between the original image 105 and the original image 105 is used to generate the second feature map F 1.2 .

[0270] The first activation layer 912 may apply an activation function to a plurality of feature maps (i.e., the first feature map F 1.1 , the second feature map F 1.2 , ... and the Nth feature map F 1.N The activation function is used to change the value of the feature map extracted by performing the convolution operation into a nonlinear value such as "having the feature or not having the feature".

[0271] The first activation layer 912 may apply different activation functions A1, A2, ..., and AN to the plurality of feature maps F output by the first convolutional layer 911, respectively. 1.1 、F 1.2 , … and F 1.N This will be referred to below. Fig.15 Describe in more detail.

[0272] The first activation layer 912 can be implemented by applying an activation function to the first feature map F 1.1 To obtain the first result value 1010. The first activation layer 912 can be applied to the second feature map F by applying the activation function 1.2 To obtain the second result value 1020.

[0273] The result values ​​(e.g., the first result value 1010 and the second result value 1020) obtained in the first layer 910 are stored in the memory, and the result values ​​stored in the memory are input to a layer (e.g., the second layer 920) adjacent to the first layer 910. The result values ​​obtained in the first layer 910 may be represented as real numbers (values ​​with high precision), and in order to reduce the amount of memory storing the result values, the low-precision transform unit 950 may transform the result values ​​obtained in the first layer 910 into integer values ​​(values ​​with low precision) having a preset range. The integer values ​​have a transform error.

[0274] Fig.13 The first graph 1035 shows the distribution of the first result value, and the second graph 1045 shows the distribution of the second result value. Comparing the first graph 1035 with the second graph 1045, the first filter kernel K 1.1 The corresponding first result value 1010 and the second filter kernel K 1.2 The corresponding second result value 1020 may have different distributions. When the first result value 1010 and the second result value 1020 having different distributions are transformed into integer values ​​by performing clipping in the same range, a transformation error increases.

[0275] Therefore, in order to reduce the transformation error, normalization may be performed, wherein different scaling factors are applied to the first result value 1010 and the second result value 1020 having different distributions, so that the first result value 1010 and the second result value 1020 to which different scaling factors have been applied have the same or similar distributions, respectively.

[0276] Fig.14 is a block diagram showing a structure of a low-precision transform unit 950 according to an embodiment of the present disclosure, and Fig.15 2 is a diagram for describing a data processing method in the low-precision transform unit 950 according to an embodiment of the present disclosure.

[0277] Reference Fig.14 According to an embodiment of the present disclosure, the low-precision transform unit 950 may include a normalization unit 1110 , a cropping unit 1120 , and a rounding unit 1130 .

[0278] The normalization unit 1110 may normalize the result value obtained by the first layer 910. For example, referring to Fig.15 , the normalization unit 1110 may perform normalization by applying a first scaling factor (Scale factor 1) to the first result value 1035, and may perform normalization by applying a second scaling factor (Scale factor 2) to the second result value 1045, so that the first result value 1035 and the second result value 1045 have the same or similar distribution.

[0279] For example, the normalization unit 1110 may be configured to obtain the first filter kernel F 1.1 The first result value 1035 is multiplied by the first scaling factor to transform the first result value 1035 into a first value 1210, and the first result value 1035 can be converted into a first value 1210 by combining the first result value 1035 with the second filter kernel F 1.2 The corresponding second result value 1045 is multiplied by the second scaling factor to transform the second result value 1045 into a second value 1220 .

[0280] The first scaling factor and the second scaling factor according to an embodiment of the present disclosure are values ​​determined through the training of the first DNN according to an embodiment of the present disclosure, and when the first DNN and the second DNN according to the embodiment of the present disclosure are jointly trained, the first scaling factor and the second scaling factor of the first DNN are determined as values ​​associated with the first scaling factor and the second scaling factor of the second DNN. For example, during the training of the first DNN and the second DNN, when the first parameter of the filter kernel included in the first DNN is updated, the result value obtained in each of the multiple layers included in the first DNN is changed, and when the result value obtained in each of the multiple layers is changed, the scaling factor of the first DNN is updated so that the value to which the result value has been changed has the same or similar distribution. For example, the first scaling factor and the second scaling factor of the first DNN may be updated so that the first value and the second value obtained through the transformation by applying the first scaling factor and the second scaling factor have the same normal distribution. Optionally, the first scaling factor and the second scaling factor of the first DNN may be updated based on the statistical distribution of the result value obtained in each of the multiple layers. This will be referred to later. Fig.16 Describe in more detail.

[0281] When the first parameter and scaling factor of the first DNN are updated, the first training image output by the first DNN is changed. When the first training image input to the second DNN is changed, the third training image output by the second DNN is also changed. When the third training image is changed, the quality loss information for training the second DNN is re-determined.

[0282] Therefore, the second parameter of the filter kernel included in the second DNN is updated in the direction that minimizes the newly determined quality loss information. When the second parameter of the second DNN is updated, the result value obtained in each of the plurality of layers included in the second DNN is changed, and when the result value obtained in each of the plurality of layers is changed, the scaling factor of the second DNN applied so that the value to which the result value has been changed has a similar distribution is also updated.

[0283] When the quality loss information is re-determined, the final loss information for training the first DNN is also re-determined, and the first parameter and scaling factor of the first DNN are updated in a direction that minimizes the newly determined final loss information for training the first DNN. In this way, the first parameter and scaling factor of the first DNN and the second parameter and scaling factor of the second DNN are jointly updated, and thus the scaling factor of the first DNN that has completed training and the scaling factor of the second DNN that has completed training have associated values.

[0284] Therefore, the DNN setting information of the first DNN determined after the training is completed (e.g., the first parameters and the number of filter cores included in the first DNN), the scaling factor of the first DNN, the DNN setting information of the second DNN (e.g., the second parameters and the number of filter cores included in the second DNN), and the scaling factor of the second DNN are associated with each other. Therefore, when the DNN setting information of the first DNN is determined from the multiple pieces of DNN setting information of the first DNN, the scaling factor of the first DNN corresponding to the determined DNN setting information of the first DNN can also be determined, and the DNN setting information and the scaling factor of the second DNN corresponding to the determined DNN setting information of the first DNN can be determined.

[0285] Return to reference Fig.14 and Fig.15 , the clipping unit 1120 according to an embodiment of the present disclosure may clip the first value 1210 and the second value 1220 to a preset range. For example, the preset range may be expressed as [minimum value, maximum value]. The range to which the first value 1210 and the second value 1220 are clipped may be determined based on the number of bits used to represent the output value or based on an integer range. For example, when the output value is represented by 8 bits, 8 bits may represent 256 integer values, and therefore, the clipping range may be determined to be [-127, 128]. However, this is an example, and when the output value is expected to be represented by 8 bits, the clipping range may be determined to be various ranges including 256 integer values.

[0286] The cropping unit 1120 may transform a value smaller than a minimum value among the first value and the second value into the minimum value, and may transform a value greater than a maximum value among the first value and the second value into the maximum value.

[0287] For example, Fig.15 As shown, when the minimum value is -127 and the maximum value is 128, the clipping unit 1120 may transform values ​​less than -127 into -127, and values ​​greater than 128 into 128. Therefore, the clipped value may be represented as a real number equal to or greater than -127 and less than or equal to 128.

[0288] According to an embodiment of the present disclosure, the rounding unit 1130 may transform the clipped (real number) value into an integer value within a preset range. For example, the rounding unit 1130 may transform the clipped (real number) value into an integer value by rounding the clipped (real number) value. Alternatively, the rounding unit 1130 may transform the clipped value into an integer value by rounding the clipped value up or down. However, the present disclosure is not limited thereto.

[0289] The integer value transformed within a preset range may be input to the second layer 920 adjacent to the first layer 910. In the second layer 920, as in the first layer 910, a convolution operation may be performed, and thus a feature map may be generated in the second convolution layer 921, and an activation function may be applied to the feature map generated in the second convolution layer 921 in the second activation layer 922.

[0290] Despite Figures 13 to 15 The DNN 900 according to the embodiment of the present disclosure is the first DNN, but referring to Figures 13 to 15 The described data processing method performed by the DNN 900 may be similarly applied to the second DNN included in the AI ​​decoding device 200 .

[0291] Fig.16 is a diagram for describing a method of determining a scaling factor according to an embodiment of the present disclosure.

[0292] Reference Fig.16 , the DNN 900 according to an embodiment of the present disclosure may be an example of the first DNN 700 or the second DNN 300.

[0293] The DNN 900 may be trained based on a plurality of training images (training data). When the DNN 900 is being trained based on the plurality of training images, a scaling factor to be included in the DNN 900 may be determined. For example, when the first training image 1311 of the plurality of training images (training data 1310) is input to the DNN 900, each of the plurality of layers included in the DNN 900 may obtain a plurality of pieces of training result data respectively corresponding to the plurality of filter kernels. For example, in the first layer 910, the first training image 1311 may be trained by performing a first filter kernel K 1.1 A first feature map is generated by performing a convolution operation between them, and the first training result data 1321 can be obtained by applying the activation function A1 to the first feature map.

[0294] In the first layer 910, the first training image 1311 and the second filter kernel K 1.2 A second feature map is generated by performing a convolution operation between them, and the second training result data 1331 can be obtained by applying the activation function A2 to the second feature map.

[0295] When the DNN 900 is trained based on a plurality of training images and the input training image is changed, the first filter kernel K of the first layer 910 1.1 The corresponding first training result data 1321 is changed and combined with the second filter kernel K of the first layer 910. 1.2 The corresponding second training result data 1331 is changed. Therefore, in the first layer 910, a plurality of pieces of first training result data 1320 and a plurality of pieces of second training result data 1330 corresponding to a plurality of training images may be obtained.

[0296] The deviation of the value included in the first training result data 1321 with respect to the first training image 1311 may be determined as the deviation from the first filter kernel K 1.1 The initial value of the corresponding first deviation, and the deviation of the value included in the second training result data 1331 with respect to the first training image 1311 can be determined as the deviation of the second filter kernel K 1.2 The initial value of the corresponding second deviation.

[0297] When the first training result data 1321 and the second training result data 1331 change as the input training image changes, the dispersion of the values ​​included in the changed first training result data can be obtained, and the obtained dispersion can be used to update the first dispersion corresponding to the first filter kernel. In addition, the dispersion of the values ​​included in the changed second training result data can be obtained, and the obtained dispersion can be used to update the second dispersion corresponding to the second filter kernel.

[0298] The scaling factor can be calculated using the parameter σ, the parameter A, and the parameter B. The dispersion σ can be calculated using 1 2 and deviation σ 2 2 The square value of parameter σ is obtained by summing the values ​​of σ and σ. For example, the parameter σ is calculated based on the following formula:

[0299] [Equation 2]

[0300]

[0301] In Equation 2, σ 1 2 indicates the first filter kernel K updated during the training of the DNN 900 by using the previous training image 1.1 The corresponding first deviation, and σ 2 2 Indicates the first filter kernel K obtained in the first layer 910 after the current training image is input to the DNN 900 1.1 The dispersion of the values ​​included in the corresponding first training result data 1321. The coefficient 0.99 or the coefficient 0.01 is an example, and various other constant values ​​may be used.

[0302] Similarly, the second filter kernel K of the first layer 910 may be updated 1.2 The corresponding dispersion may be updated, and the dispersion corresponding to each of the plurality of filter cores included in each layer except the first layer 910 among the plurality of layers.

[0303] As in Equation 3 below, a scaling factor corresponding to each of a plurality of filter kernels included in the DNN 900 may be determined based on a deviation value corresponding to each of the plurality of filter kernels.

[0304] [Equation 3]

[0305]

[0306] In Equation 3, parameter A is a value determined based on a preset range. For example, when the preset range is equal to or greater than -127 and less than or equal to 128, parameter A may be determined to be 128. Assuming that the value included in the training result data corresponding to each of the plurality of filter kernels follows a Gaussian distribution, when the probability that the value included in the training result data is included in the preset range is determined, parameter B may be determined according to the determined probability. For example, when the probability that the value included in the training result data is included in the preset range is determined to be 99.7%, parameter B may be determined to be 3. When the probability that the value included in the training result data is included in the preset range is determined to be 95.4%, parameter B may be determined to be 2. The value of parameter B may increase as the probability that the value included in the training result data is within the preset range increases. However, the present disclosure is not limited thereto.

[0307] Fig.17 An activation function according to an embodiment of the present disclosure is shown.

[0308] Reference Fig.17 , the activation function 1410 according to an embodiment of the present disclosure can be expressed as Equation 4.

[0309] [Equation 4]

[0310]

[0311] In Equation 4, x indicates a feature map generated as a result of the convolution operation. According to an embodiment of the present disclosure, the slope b in the interval in which x is a positive number may be determined by training the first DNN 700 and the second DNN 300, and the slope b may have different values ​​for different layers and different filter cores included in the first DNN 700 and the second DNN 300. For example, the slope b of the first activation function applied to the first feature map generated by the first filter core of the first layer and the slope b of the second activation function applied to the second feature map generated by the second filter core of the first layer may have different values.

[0312] The activation function 1420 according to another embodiment of the present disclosure may be expressed as Equation 5.

[0313] [Equation 5]

[0314]

[0315] In Equation 5, according to an embodiment of the present disclosure, the slope α in the interval where x is a negative number may be determined via training of the first DNN 700 and the second DNN 300, or may have a fixed value. According to an embodiment of the present disclosure, the slope b in the interval where x is a positive number may be determined via training the first DNN 700 and the second DNN 300. Therefore, the slope b may have different values ​​for different layers and different filter cores included in the first DNN 700 and the second DNN 300. For example, the slope b of the first activation function applied to the first feature map generated by the first filter core of the first layer and the slope b of the second activation function applied to the second feature map generated by the second filter core of the first layer may have different values.

[0316] The slope a and the slope b may have different values ​​(non-linear), and the slope a may be less than 1. However, the present disclosure is not limited thereto.

[0317] Fig.18 is a flowchart of an operating method of an AI encoding device according to an embodiment of the present disclosure.

[0318] Reference Fig.18 In operation S1510, when an original image is input to a first DNN 700 including a plurality of layers, the AI ​​encoding device 600 according to an embodiment of the present disclosure may obtain a first result value based on a result of an operation between the original image and a first filter kernel from a first layer among the plurality of layers, and may obtain a second result value based on a result of an operation between the original image and a second filter kernel.

[0319] The first DNN 700 according to an embodiment of the present disclosure is a network that performs AI reduction on an image, and thus may be a network that corresponds to the second DNN 300 that performs AI enlargement on an image and is trained in conjunction with the second DNN 300. The first DNN 700 may include a plurality of layers. For example, the first DNN 700 may include a first layer and a second layer. According to an embodiment of the present disclosure, each of the plurality of layers may include a convolution layer and an activation layer. A convolution layer may include a plurality of filter cores. For example, a first convolution layer included in a first layer may include a first filter core and a second filter core.

[0320] When an original image is input to the first DNN 700, in the first convolution layer of the first layer, a first feature map may be generated via a convolution operation between the original image and a first filter kernel, and a second feature map may be generated via a convolution operation between the original image and a second filter kernel.

[0321] The first feature map and the second feature map generated in the first convolution layer of the first layer may be input to the first activation layer of the first layer. In the first activation layer of the first layer, the activation function may be applied to the first feature map and thus the first result value may be output, and the activation function may be applied to the second feature map and thus the second result value may be output. Therefore, the AI ​​encoding device 600 may obtain the first result value corresponding to the first filter kernel from the first layer, and may obtain the second result value corresponding to the second filter kernel from the first layer.

[0322] In operation S1520, the AI ​​encoding device 600 may normalize the first result value and the second result value. For example, the AI ​​encoding device 600 may perform normalization by applying a first scaling factor to the first result value, and may perform normalization by applying a second scaling factor to the second result value. For example, the AI ​​encoding device 600 may transform the first result value into a first value by multiplying the first result value by the first scaling factor, and may transform the second result value into a second value by multiplying the second result value by the second scaling factor.

[0323] The first scaling factor and the second scaling factor according to an embodiment of the present disclosure are values ​​determined by training the first DNN, and when the first DNN and the second DNN according to an embodiment of the present disclosure are trained in conjunction with each other, the first scaling factor and the second scaling factor of the first DNN are determined as values ​​associated with the scaling factor of the second DNN. For example, during the training of the first DNN and the second DNN, when the first parameter of the filter kernel included in the first DNN is updated, the result value obtained in each of the multiple layers included in the first DNN is changed, and when the result value obtained in each of the multiple layers is changed, the scaling factor of the first DNN is updated so that the value to which the result value has been changed has the same or similar distribution. For example, the first scaling factor and the second scaling factor of the first DNN may be updated so that the first value and the second value obtained by the transformation by applying the first scaling factor and the second scaling factor have the same normal distribution. Optionally, the first scaling factor and the second scaling factor of the first DNN may be updated based on the statistical distribution of the result value obtained in each of the multiple layers.

[0324] When the first parameter and scaling factor of the first DNN are updated, the first training image output by the first DNN is changed. When the first training image input to the second DNN is changed, the third training image output by the second DNN is also changed. When the third training image is changed, the quality loss information for training the second DNN is re-determined.

[0325] Therefore, the second parameter of the filter kernel included in the second DNN is updated in the direction that minimizes the newly determined quality loss information. When the second parameter of the second DNN is updated, the result value obtained in each of the plurality of layers included in the second DNN is changed, and when the result value obtained in each of the plurality of layers is changed, the scaling factor of the second DNN applied so that the value to which the result value has been changed has a similar distribution is changed.

[0326] When the quality loss information is re-determined, the final loss information for training the first DNN is also re-determined, and the first parameter and scaling factor of the first DNN are updated in a direction that minimizes the newly determined final loss information for training the first DNN. Therefore, the first parameter and scaling factor of the first DNN and the second parameter and scaling factor of the second DNN are updated in relation to each other, and thus the scaling factor of the first DNN that has completed training and the scaling factor of the second DNN that has completed training have associated values.

[0327] Therefore, the DNN setting information of the first DNN determined after the training is completed (e.g., the first parameters and the number of filter cores included in the first DNN), the scaling factor of the first DNN, the DNN setting information of the second DNN (e.g., the second parameters and the number of filter cores included in the second DNN), and the scaling factor of the second DNN are associated with each other. Therefore, when the DNN setting information of the first DNN is determined from the multiple pieces of DNN setting information of the first DNN, the scaling factor of the first DNN corresponding to the determined DNN setting information of the first DNN can also be determined, and the DNN setting information and the scaling factor of the second DNN corresponding to the determined DNN setting information of the first DNN can be determined.

[0328] In operation S1530, the AI ​​encoding apparatus 600 may transform the normalized first value and the normalized second value into integer values ​​included in a preset range.

[0329] For example, the AI ​​encoding device 600 may clip the first value and the second value to a preset range. The range to which the first value and the second value are clipped may be determined based on the number of bits used to represent the output value or based on an integer range. When the clipping range is [minimum value, maximum value], the AI ​​encoding device 600 may transform a value of the first value and the second value that is less than the minimum value into the minimum value, and may transform a value of the first value and the second value that is greater than the maximum value into the maximum value.

[0330] The AI ​​encoding device 600 may transform the cropped value into an integer value by rounding the cropped value. Alternatively, the AI ​​encoding device 600 may transform the cropped value into an integer value by rounding up or rounding down the cropped value. The integer value may be input to a second layer adjacent to the first layer.

[0331] In operation S1540, when an integer value is input to the second layer in the same manner as in the first layer, in the second convolution layer of the second layer, a plurality of feature maps may be generated via a convolution operation between the input integer value and a plurality of filter cores, and in the second activation layer of the second layer, a third result value may be obtained by applying an activation function to each of the plurality of feature maps. The AI ​​encoding device 600 may perform a low-precision transformation by performing normalization and clipping on the third result value to transform the third result value into an integer value within a preset range, and may input the integer value to the next layer.

[0332] In this way, the following process may be repeatedly performed: the original image is subjected to convolution operation and activation function operation using the filter kernel in each of the multiple layers when passing through the multiple layers included in the first DNN 700, thereby outputting a result value, and performing a low-precision transformation on the result value output by each layer, and inputting the result of the low-precision transformation to the next layer. Therefore, in operation S1550, the AI ​​encoding device 600 may obtain the first image 115 by performing AI reduction on the original image using the first DNN 700. At this time, the integer value obtained by performing a low-precision transformation on the result value output by each layer of the first DNN may be stored in a memory before being input to the next layer, thereby reducing the amount of memory required for storing intermediate result values ​​generated when the first image (the image obtained by performing AI reduction on the original image) is obtained by using the first DNN 700.

[0333] In operation S1560, the AI ​​encoding apparatus 600 may encode the first image.

[0334] Fig.19 is a flowchart of an operating method of an AI decoding device according to an embodiment of the present disclosure.

[0335] Reference Fig.19 , in operation S1610, the AI ​​decoding apparatus 200 may obtain image data corresponding to a first image.

[0336] The first image may be an image obtained by performing AI downscaling on an original image using the first DNN 700 , and the image data may include image data generated by performing AI encoding on the first image.

[0337] In operation S1620, the AI ​​decoding apparatus 200 may restore a second image corresponding to the first image based on the image data.

[0338] In operation S1630, when the second image is input to the second DNN 300 including multiple layers, the AI ​​decoding device 200 can obtain a first result value from a first layer among the multiple layers based on the result of the operation between the second image and the first filter kernel and can obtain a second result value based on the result of the operation between the second image and the second filter kernel.

[0339] The second DNN 300 according to an embodiment of the present disclosure is a network that performs AI amplification on an image, and thus may be a network that corresponds to the first DNN 700 that performs AI amplification on an image and is trained in conjunction with the first DNN 700. The second DNN 300 may include a plurality of layers. For example, the second DNN 300 may include a first layer and a second layer. According to an embodiment of the present disclosure, each of the plurality of layers may include a convolutional layer and an activation layer. A convolutional layer may include a plurality of filter cores. For example, a first convolutional layer included in a first layer may include a first filter core and a second filter core.

[0340] When the second image is input to the second DNN 300, in the first convolution layer of the first layer, a first feature map may be generated via a convolution operation between the second image and the first filter kernel, and a second feature map may be generated via a convolution operation between the second image and the second filter kernel.

[0341] The first feature map and the second feature map generated in the first convolution layer of the first layer may be input to the first activation layer of the first layer. In the first activation layer of the first layer, the activation function may be applied to the first feature map and thus the first result value may be output, and the activation function may be applied to the second feature map and thus the second result value may be output. Therefore, the AI ​​decoding device 200 may obtain the first result value corresponding to the first filter kernel from the first layer, and may obtain the second result value corresponding to the second filter kernel from the first layer.

[0342] In operation S1640, the AI ​​decoding device 200 may normalize the first result value and the second result value. For example, the AI ​​decoding device 200 may perform normalization by applying a first scaling factor to the first result value, and may perform normalization by applying a second scaling factor to the second result value. For example, the AI ​​decoding device 200 may transform the first result value into a first value by multiplying the first result value by the first scaling factor, and may transform the second result value into a second value by multiplying the second result value by the second scaling factor.

[0343] The first scaling factor and the second scaling factor according to an embodiment of the present disclosure are values ​​determined by training the first DNN, and when the first DNN and the second DNN according to an embodiment of the present disclosure are trained in conjunction with each other, the first scaling factor and the second scaling factor of the first DNN are determined as values ​​associated with the scaling factor of the second DNN. For example, during the training of the first DNN and the second DNN, when the first parameter of the filter kernel included in the first DNN is updated, the result value obtained in each of the multiple layers included in the first DNN is changed, and when the result value obtained in each of the multiple layers is changed, the scaling factor of the first DNN is updated so that the value to which the result value has been changed has the same or similar distribution. For example, the first scaling factor and the second scaling factor of the first DNN may be updated so that the first value and the second value obtained by the transformation by applying the first scaling factor and the second scaling factor have the same normal distribution. Optionally, the first scaling factor and the second scaling factor of the first DNN may be updated based on the statistical distribution of the result value obtained in each of the multiple layers.

[0344] When the first parameter and scaling factor of the first DNN are updated, the first training image output by the first DNN is changed. When the first training image input to the second DNN is changed, the third training image output by the second DNN is also changed. When the third training image is changed, the quality loss information for training the second DNN is re-determined.

[0345] Therefore, the second parameter of the filter kernel included in the second DNN is updated in the direction that minimizes the newly determined quality loss information. When the second parameter of the second DNN is updated, the result value obtained in each of the multiple layers included in the second DNN is changed, and when the result value obtained in each of the multiple layers is changed, the scaling factor of the second DNN applied so that the changed result value has the same or similar distribution is updated.

[0346] When the quality loss information is re-determined, the final loss information for training the first DNN is also re-determined, and the first parameter and scaling factor of the first DNN are updated in a direction that minimizes the newly determined final loss information for training the first DNN. Therefore, the first parameter and scaling factor of the first DNN and the second parameter and scaling factor of the second DNN are updated in relation to each other, and thus the scaling factor of the first DNN that has completed training and the scaling factor of the second DNN that has completed training have associated values.

[0347] Therefore, the DNN setting information of the first DNN determined after the training is completed (e.g., the first parameters and the number of filter cores included in the first DNN), the scaling factor of the first DNN, the DNN setting information of the second DNN (e.g., the second parameters and the number of filter cores included in the second DNN), and the scaling factor of the second DNN are associated with each other. Therefore, when the DNN setting information of the first DNN is determined from the multiple pieces of DNN setting information of the first DNN, the scaling factor of the first DNN corresponding to the determined DNN setting information of the first DNN can also be determined, and the DNN setting information and the scaling factor of the second DNN corresponding to the determined DNN setting information of the first DNN can be determined.

[0348] In operation S1650, the AI ​​decoding apparatus 200 may transform the normalized first value and the normalized second value into integer values ​​included in a preset range.

[0349] For example, the AI ​​decoding device 200 may clip the first value and the second value to a preset range. The range to which the first value and the second value are clipped may be determined based on the number of bits used to represent the output value or based on an integer range. When the clipping range is [minimum value, maximum value], the AI ​​decoding device 200 may transform a value of the first value and the second value that is less than the minimum value into the minimum value, and may transform a value of the first value and the second value that is greater than the maximum value into the maximum value.

[0350] The AI ​​decoding device 200 may transform the cropped value into an integer value by rounding the cropped value. Alternatively, the AI ​​decoding device 200 may transform the cropped value into an integer value by rounding up or rounding down the cropped value. The integer value may be input to a second layer adjacent to the first layer.

[0351] In operation S1660, when an integer value is input to the second layer in the same manner as in the first layer, in the second convolution layer of the second layer, a plurality of feature maps may be generated via a convolution operation between the input integer value and a plurality of filter cores, and in the second activation layer of the second layer, a third result value may be obtained by applying an activation function to each of the plurality of feature maps. The AI ​​decoding device 200 may perform a low-precision transformation by normalizing and clipping the third result value to transform the third result value into an integer value within a preset range, and may input the integer value to the next layer.

[0352] In this way, the following process may be repeatedly performed: the second image performs a convolution operation and an activation function operation using a filter kernel in each of the plurality of layers when passing through the plurality of layers included in the second DNN 300, thereby outputting a result value, and performing a low-precision transformation on the result value output by each layer, and inputting the result of the low-precision transformation to the next layer. Therefore, in operation S1670, the AI ​​decoding device 200 may obtain a third image by performing AI upscaling on the second image using the second DNN 300.

[0353] At this time, the integer values ​​obtained by low-precision transforming the result values ​​output by each layer of the second DNN can be stored in the memory before being input to the next layer, and thus the amount of memory required to store intermediate result values ​​generated when obtaining the third image (the image obtained by AI upscaling the second image) by using the second DNN 300 can be reduced.

[0354] Fig. 20 17 is a block diagram of a structure of an AI encoding device 1700 according to an embodiment of the present disclosure.

[0355] Fig. 20 The AI ​​encoding device 1700 may be Figure 7 An embodiment of the AI ​​encoding device 600. Fig. 20 , the AI ​​encoding device 1700 may include a processor 1720 and a memory 1730.

[0356] The processor 1720 according to an embodiment of the present disclosure may completely control the AI ​​encoding apparatus 1700. The processor 1720 according to an embodiment of the present disclosure may execute one or more programs stored in the memory 1730.

[0357] The memory 1730 according to an embodiment of the present disclosure may store various data, programs, or applications for driving and controlling the AI ​​encoding device 1700. The program stored in the memory 1730 may include at least one instruction. The program (one or more instructions) or application stored in the memory 1730 may be executed by the processor 1720.

[0358] The processor 1720 according to an embodiment of the present disclosure may execute Figure 7 shown in and referenced above Figure 7 The operations of the AI ​​encoding unit 610 and the sending unit 630 described in Figures 12 to 15 shown in and referenced above Figures 12 to 15 The operations of the first convolution layer 911 and the second convolution layer 921, the first activation layer 912 and the second activation layer 922, and the low-precision transform unit 950 described above, and Fig.18 At least one of the operations of the AI ​​encoding device 600 described.

[0359] When an original image is input to a first DNN including multiple layers, the processor 1720 according to an embodiment of the present disclosure may obtain a first result value based on the result of the operation between the original image and the first filter kernel from the first layer of the multiple layers, and may obtain a second result value based on the result of the operation between the original image and the second filter kernel. For example, the processor 1720 may generate a first feature map via a convolution operation between the original image and the first filter kernel, and may obtain the first result value by applying an activation function to the first feature map in a first activation layer of the first layer. The processor 1720 may also generate a second feature map via a convolution operation between the original image and the second filter kernel, and may also obtain a second result value by applying an activation function to the second feature map in a second activation layer of the first layer. As described above with reference to Fig.17 As described above, the activation function applied to the first feature map (first activation function) and the activation function applied to the second feature map (second activation function) may be activation functions in the form of Equation 4 or 5, and the slopes a or b of the first activation function and the second activation function may be different from each other.

[0360] The processor 1720 may perform normalization by applying a first scaling factor to the first result value, and may perform normalization by applying a second scaling factor to the second result value. The processor 1720 may clip the normalized value to a preset range and convert the clipped value to an integer value. The processor 1720 may convert the clipped value to an integer value by rounding, rounding up, or rounding down the clipped value, but the present disclosure is not limited thereto.

[0361] The processor 1720 may input the integer value to a second layer adjacent to the first layer.

[0362] In this way, the following process may be repeatedly performed: the original image performs convolution operation and activation function operation using the filter kernel in each of the multiple layers when passing through the multiple layers included in the first DNN 700, performs low-precision transformation on the result value output by each layer, and inputs the result of the low-precision transformation to the next layer. Therefore, the processor 1720 can obtain the first image 115 by performing AI reduction on the original image using the first DNN 700.

[0363] Any one or any combination of the convolutional layers, activation layers, or low-precision transform units included in the first DNN 700 may be manufactured in the form of a hardware chip and may be installed on the AI ​​encoding device 1700. For example, any one or any combination of the convolutional layers, activation layers, or low-precision transform units included in the first DNN 700 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) or a processor dedicated to graphics (e.g., a graphics processing unit (GPU)), and may be installed on any of the above-mentioned various image processing devices.

[0364] A software module may be used to implement any one or any combination of the convolutional layers, activation layers, and low-precision transform units included in the first DNN 700. When a software module (or a program module including instructions) is used to implement any one or any combination of the convolutional layers, activation layers, or low-precision transform units included in the first DNN 700, the software module may be stored in a non-transitory computer-readable medium. In this case, at least one software module may be provided by an operating system (OS) or by a specific application. Optionally, some of the at least one software module may be provided by the OS, and other software modules may be provided by a specific application.

[0365] Fig.21 18 is a block diagram of a structure of an AI decoding device 1800 according to an embodiment of the present disclosure.

[0366] Fig.21 The AI ​​decoding device 1800 may be Figure 2 An embodiment of the AI ​​decoding device 200. Fig.21 , the AI ​​decoding device 1800 may include a processor 1820 and a memory 1830.

[0367] The processor 1820 according to an embodiment of the present disclosure may completely control the AI ​​decoding device 1800. The processor 1820 according to an embodiment of the present disclosure may execute one or more programs stored in the memory 1830.

[0368] The memory 1830 according to an embodiment of the present disclosure may store various data, programs, or applications for driving and controlling the AI ​​decoding device 1800. The program stored in the memory 1830 may include at least one instruction. The program (one or more instructions) or application stored in the memory 1830 may be executed by the processor 1820.

[0369] The processor 1820 according to an embodiment of the present disclosure may execute Figure 2 shown in and referenced above Figure 2The operations of the receiving unit 210 and the AI ​​decoding unit 230 described in Figures 12 to 15 shown in and referenced above Figures 12 to 15 The operations of the first convolution layer 911 and the second convolution layer 921, the first activation layer 912 and the second activation layer 922, and the low-precision transform unit 950 described above, and Fig.21 At least one of the operations of the AI ​​decoding device 200 described.

[0370] When the second image is input to the second DNN including multiple layers, the processor 1820 according to an embodiment of the present disclosure may obtain a first result value based on the result of the operation between the second image and the first filter kernel from the first layer of the multiple layers, and may obtain a second result value based on the result of the operation between the second image and the second filter kernel. For example, the processor 1820 may generate a first feature map via a convolution operation between the second image and the first filter kernel, and may obtain the first result value by applying an activation function to the first feature map in a first activation layer of the first layer. The processor 1820 may also generate a second feature map via a convolution operation between the second image and the second filter kernel, and may also obtain a second result value by applying an activation function to the second feature map in a second activation layer of the first layer. As described above with reference to Fig.17 As described above, the activation function applied to the first feature map (first activation function) and the activation function applied to the second feature map (second activation function) may be activation functions in the form of Equation 4 or 5, and the slopes a or b of the first activation function and the second activation function may be different from each other.

[0371] The processor 1820 may perform normalization by applying a first scaling factor to the first result value, and may perform normalization by applying a second scaling factor to the second result value. The processor 1820 may clip the normalized value to a preset range and convert the clipped value to an integer value. The processor 1820 may convert the clipped value to an integer value by rounding, rounding up, or rounding down the clipped value, but the present disclosure is not limited thereto.

[0372] The processor 1820 may input the integer value to a second layer adjacent to the first layer.

[0373] In this way, the following process may be repeatedly performed: the second image performs convolution operation and activation function operation using the filter kernel in each of the multiple layers when passing through the multiple layers included in the second DNN 300, performs low-precision transformation on the result value output by each layer, and inputs the result of the low-precision transformation to the next layer. Therefore, the processor 1820 can obtain a third image by performing AI amplification on the second image using the second DNN 300.

[0374] Any one or any combination of the convolutional layer, activation layer, or low-precision transform unit included in the second DNN 300 may be manufactured in the form of a hardware chip and may be installed on the AI ​​decoding device 1800. For example, at least one of the convolutional layer, activation layer, or low-precision transform unit included in the second DNN 300 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) or a processor dedicated to graphics (e.g., a graphics processing unit (GPU)), and may be installed on any of the above-mentioned various image processing devices.

[0375] A software module may be used to implement any one or any combination of the convolutional layers, activation layers, and low-precision transform units included in the second DNN 300. When a software module (or a program module including instructions) is used to implement any one or any combination of the convolutional layers, activation layers, and low-precision transform units included in the second DNN 300, the software module may be stored in a non-transitory computer-readable medium. In this case, at least one software module may be provided by the OS or by a specific application. Alternatively, some of the at least one software module may be provided by the OS, and other software modules may be provided by a specific application.

[0376] Fig. 20 and Fig.21 The block diagrams of the AI ​​encoding device 1700 and the AI ​​decoding device 1800 shown are only exemplary embodiments of the present disclosure. In actual implementation, the AI ​​encoding device 1700 and the AI ​​decoding device 1800 may be combined or omitted according to the specifications of the AI ​​encoding device 1700 and the AI ​​decoding device 1800. Fig. 20 and Fig.21 components shown, or can be Fig. 20 and Fig.21 In other words, two or more components may be combined into a single component, or a single component may be divided into two or more components. The functions performed in each block are merely examples for explaining the embodiments of the present disclosure, and the detailed operations or devices of each block do not limit the scope of the embodiments of the present disclosure.

[0377] According to an embodiment of the present disclosure, the AI ​​encoding device can reduce the amount of necessary memory by representing intermediate result values ​​generated during image reduction by using the first DNN with low precision, and can improve the performance of the first DNN by minimizing the transformation error during the transformation of the intermediate result values ​​into low precision values.

[0378] The AI ​​encoding device according to an embodiment of the present disclosure may also improve the performance of the first DNN by applying a trainable activation function to the first DNN.

[0379] According to an embodiment of the present disclosure, the AI ​​decoding device can reduce the amount of necessary memory by representing intermediate result values ​​generated during image enlargement by using the second DNN with low precision, and can improve the performance of the second DNN by minimizing the transformation error during the transformation of the intermediate result values ​​into low-precision values.

[0380] In a system in which an AI encoding device includes a server that provides AI-reduced images and an AI decoding device includes a terminal that receives AI-reduced images, the amount of memory required to store intermediate result values ​​generated during AI upscaling of images received from the terminal can be reduced, thereby improving the efficiency of the system.

[0381] The AI ​​decoding device according to an embodiment of the present disclosure may improve the performance of the second DNN by applying a trainable activation function to the second DNN.

[0382] Furthermore, the embodiments of the present disclosure described above may be written as computer-executable programs or instructions that may be stored in a medium.

[0383] The medium can store computer executable programs or instructions continuously, or temporarily store computer executable programs or instructions for execution or downloading. In addition, the medium can be any of a variety of recording media or storage media combined with single or multiple pieces of hardware, and the medium is not limited to the medium directly connected to the computer system, but can be distributed on the network. Examples of media include magnetic media (such as hard disks, floppy disks, and tapes) configured to store program instructions, optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as floppy disks, and ROM, RAM, and flash memory). Other examples of media include recording media and storage media managed by application stores that distribute applications or by websites, servers, etc. that supply or distribute various other types of software.

[0384] The model related to the above-mentioned DNN can be implemented via a software module. When the DNN model is implemented via a software module (eg, a program module including instructions), the DNN model can be stored in a computer-readable recording medium.

[0385] In addition, the DNN model can be integrated in the form of a hardware chip to become part of the above-mentioned AI decoding device 200 or AI encoding device 600. For example, the DNN model can be manufactured in the form of a dedicated hardware chip for AI, or can be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU).

[0386] In addition, the DNN model can be provided in the form of downloadable software. The computer program product may include a product in the form of a software program that is electronically distributed through a manufacturer or an electronic market (e.g., a downloadable application). For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a server or relay server of the manufacturer or electronic market.

[0387] Although one or more embodiments of the present disclosure have been described with reference to the drawings, it will be understood by those skilled in the art that various changes in form and details may be made in the present disclosure without departing from the spirit and scope defined by the following claims.

Claims

1. An artificial intelligence (AI) decoding device, comprising: a memory storing one or more instructions; as well as A processor configured to execute the one or more stored instructions to perform the following operations: obtaining AI data related to AI downscaling of the original image to a first image, wherein the AI ​​data includes an index indicating deep neural network (DNN) setting information for AI upscaling, obtaining image data corresponding to the encoding result of the first image, Obtaining a second image by decoding the obtained image data, selecting DNN setting information from a plurality of DNN setting information based on the index, wherein the DNN setting information is used to perform AI upscaling on the obtained second image, inputting the second image to an upscaling DNN including a plurality of layers, wherein the plurality of layers include a first layer and a second layer, the first layer includes a first filter kernel and a second filter kernel, and the upscaling DNN is set using the selected DNN setting information, A first result value is obtained based on an operation between the second image and the first filter kernel, and a second result value is obtained based on an operation between the second image and the second filter kernel, performing normalization by transforming the first result value into a first value using a first scaling factor, and performing normalization by transforming the second result value into a second value using a second scaling factor, converting the first value and the second value into integer values ​​included in a preset range, inputting the integer value to the second layer, obtaining a third result value based on the integer value in the second layer, and obtaining a third image by upscaling the second image based on the third result value, The first scaling factor and the second scaling factor are set in association with a scaling factor of the reduced DNN.

2. The AI ​​decoding device according to claim 1, wherein: The enlarged DNN is a network trained based on images obtained during the training of the reduced DNN in combination with the reduced DNN, and The first scaling factor and the second scaling factor are set via training of the up-scaling DNN and the down-scaling DNN.

3. The AI ​​decoding device according to claim 1, wherein: The first scaling factor and the second scaling factor are trained such that the normal distribution of the first values ​​becomes equal to the normal distribution of the second values.

4. The AI ​​decoding device according to claim 1, wherein: The processor is further configured to execute the one or more instructions to perform the following operations: clip the first value and the second value to the preset range, and convert the clipped first value and the second value into the integer value.

5. The AI ​​decoding device according to claim 4, wherein: The processor is further configured to execute the one or more instructions to convert the clipped first value and the second value to the integer value by rounding the clipped first value and the second value.

6. The AI ​​decoding device according to claim 1, wherein: The processor is further configured to execute the one or more instructions to store the integer value in the memory.

7. The AI ​​decoding device according to claim 6, wherein: The size of the memory storing the integer value is determined based on the preset range.

8. The AI ​​decoding device according to claim 1, wherein: The processor is further configured to execute the one or more instructions to perform the following operations: generating a first feature map by performing a convolution operation between the second image and the first filter kernel, and obtaining a first result value by applying a first activation function to the first feature map, and A second feature map is generated by performing a convolution operation between the second image and a second filter kernel, and a second result value is obtained by applying a second activation function to the second feature map.

9. The AI ​​decoding device according to claim 8, wherein: The first activation function and the second activation function are expressed as: Wherein, f(x) represents the first activation function or the second activation function, a and b represent slopes with different values, and The value of b is determined through training of the enlarged DNN.

10. The AI ​​decoding device according to claim 9, wherein: The value of b in the first activation function is different from the value of b in the second activation function.

11. An artificial intelligence (AI) coding device, comprising: a memory storing one or more instructions; as well as A processor configured to execute the one or more stored instructions to perform the following operations: Select DNN setting information for AI reduction from multiple deep neural network DNN setting information, inputting the original image to a reduced DNN including a plurality of layers, wherein the plurality of layers include a first layer and a second layer, the first layer includes a first filter kernel and a second filter kernel, and the reduced DNN is set using the selected DNN setting information, A first result value is obtained based on an operation between the original image and a first filter kernel, and a second result value is obtained based on an operation between the original image and a second filter kernel, performing normalization by transforming the first result value into a first value using a first scaling factor, and performing normalization by transforming the second result value into a second value using a second scaling factor, converting the first value and the second value into integer values ​​included in a preset range, inputting the integer value into the second layer, obtaining a third result value based on the integer value in the second layer, and obtaining the first image by reducing the original image based on the third result value, encoding the first image to obtain image data, and providing the image data and AI data related to the AI ​​downscaling to an AI decoding device, wherein the AI ​​data includes an index indicating DNN setting information for AI upscaling, The first scaling factor and the second scaling factor are set in association with a scaling factor of an upscaling DNN configured to upscale the first image.

12. The AI ​​encoding device according to claim 11, wherein: The reduced DNN is a network trained based on the loss information obtained during the training of the enlarged DNN in combination with the enlarged DNN, and The first scaling factor and the second scaling factor are set via training of the up-scaling DNN and the down-scaling DNN.

13. The AI ​​encoding device according to claim 12, wherein: The reduced DNN is a network trained based on first loss information generated by performing AI reduction during training of the reduced DNN and second loss information generated by performing AI upscaling during training of the upscaling DNN.

14. An operating method of an artificial intelligence (AI) decoding device, the operating method comprising: Obtaining AI data related to AI downscaling of an original image to a first image, wherein the AI ​​data includes an index indicating deep neural network (DNN) setting information for AI upscaling; Obtaining image data corresponding to the encoding result of the first image; Obtaining a second image by decoding the obtained image data; Selecting DNN setting information from a plurality of DNN setting information based on the index, wherein the DNN setting information is used to perform AI upscaling on the obtained second image; inputting the second image to an upscaling DNN including a plurality of layers, wherein the plurality of layers include a first layer and a second layer, the first layer includes a first filter kernel and a second filter kernel, and the upscaling DNN is set using the selected DNN setting information; Obtaining a first result value based on an operation between the second image and the first filter kernel, and obtaining a second result value based on an operation between the second image and the second filter kernel; performing normalization by transforming the first result value into a first value using a first scaling factor, and performing normalization by transforming the second result value into a second value using a second scaling factor; transforming the first value and the second value into integer values ​​included in a preset range; inputting the integer value into the second layer, and obtaining a third result value based on the integer value in the second layer; obtaining a third image by upscaling the second image based on the third result value, The first scaling factor and the second scaling factor are set in association with a scaling factor of the reduced DNN.

15. An operating method of an artificial intelligence (AI) encoding device, the operating method comprising: Selecting DNN setting information for AI reduction from a plurality of deep neural network DNN setting information; Inputting the original image to a reduced DNN including a plurality of layers, wherein the plurality of layers include a first layer and a second layer, the first layer includes a first filter kernel and a second filter kernel, and the reduced DNN is set using the selected DNN setting information; Obtaining a first result value based on an operation between the original image and a first filter kernel, and obtaining a second result value based on an operation between the original image and a second filter kernel; performing normalization by transforming the first result value into a first value using a first scaling factor, and performing normalization by transforming the second result value into a second value using a second scaling factor; transforming the first value and the second value into integer values ​​included in a preset range; inputting the integer value into the second layer, and obtaining a third result value based on the integer value in the second layer; Obtaining a first image by reducing the original image based on the third result value; encoding the first image to obtain image data, and providing the image data and AI data related to the AI ​​downscaling to an AI decoding device, wherein the AI ​​data includes an index indicating DNN setting information for AI upscaling, The first scaling factor and the second scaling factor are set in association with a scaling factor of an upscaling DNN configured to upscale the first image.

Images