Device and method for performing artificial intelligence AI encoding and AI decoding on images

Abstract

An artificial intelligence (AI) decoding method includes: obtaining image data generated by performing first encoding on a first image and AI data related to AI downscaling of at least one original image, wherein the at least one original image is related to the first image; obtaining a second image corresponding to the first image by performing first decoding on the image data; obtaining deep neural network (DNN) setting information for performing AI upscaling of the second image based on the AI ​​data; and generating a third image by performing AI upscaling on the second image via an upscaling DNN operated according to the obtained DNN setting information. The DNN setting information is DNN information updated via joint training of an upscaling DNN and a downscaling DNN for AI downscaling to perform AI upscaling of at least one second image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and more particularly, to a method and apparatus for encoding and decoding an image based on artificial intelligence (AI). Background Art

[0002] The image is encoded using a codec conforming to a predefined data compression standard such as the Moving Picture Experts Group (MPEG) standard, and the encoded image is then stored in a recording medium or transmitted in the form of a bit stream through a communication channel.

[0003] With the development and spread of hardware capable of reproducing and storing high-resolution / high-definition images, there is an increasing demand for codecs capable of efficiently encoding and decoding high-resolution / high-definition images. Summary of the invention

[0004] Technical Solution

[0005] According to an embodiment of the present disclosure, a method and apparatus for performing artificial intelligence (AI) encoding and AI decoding on an image are provided, whereby a low bit rate can be achieved by encoding and decoding an image based on AI.

[0006] In addition, according to an embodiment of the present disclosure, a method and apparatus for performing AI encoding and AI decoding on an image are provided, whereby image quality can be improved by performing upscaling after periodically or when necessary updating upscaling DNN setting information optimized for the original image.

[0007] In addition, according to an embodiment of the present disclosure, a method and apparatus for performing AI encoding and AI decoding on an image are provided, whereby the amount of information to be encoded / decoded can be effectively reduced by effectively signaling DNN setting information periodically or when necessary for updating the DNN setting information optimized for the original image of the amplified DNN.

[0008] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure.

[0009] According to an embodiment of the present disclosure, an AI decoding device includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to perform the following operations: obtain image data generated by performing first encoding on a first image and AI data related to AI reduction of at least one original image, wherein the at least one original image is related to the first image; obtain a second image corresponding to the first image by performing first decoding on the image data; obtain DNN setting information for performing AI upscaling of the second image based on the AI ​​data; and generate a third image by performing AI upscaling on the second image via an enlarged DNN operated according to the obtained DNN setting information, wherein the DNN setting information is updated via joint training of an enlarged DNN and a reduced DNN for AI downscaling of the at least one original image to perform AI upscaling of at least one second image corresponding to the at least one original image, and the joint training is performed using at least one original image.

[0010] The obtained DNN setting information may include enlarging the weights and biases of the filter kernels in at least one convolutional layer of the DNN.

[0011] The processor may also be configured to generate a first training image via a reduction DNN by using the at least one original image, generate a second training image via an enlargement DNN by using the first training image, and update the enlargement DNN and the reduction DNN based on first loss information and third loss information, and second loss information, wherein the first loss information and the third loss information correspond to a result of comparing the second training image with an original image of at least one original image that has not undergone AI reduction, and the second loss information is generated based on the first training image.

[0012] The first loss information may be generated based on a result of comparing the quality parameter of the second training image with the quality parameter of the at least one original image.

[0013] The third loss information may be generated based on a result of comparing the feature-related parameters of the second training image with the feature-related parameters of the at least one original image.

[0014] The second loss information may be related to the spatial complexity of the first training image.

[0015] The processor may also be configured to generate a first training image via a reduction DNN by using the at least one original image, perform a first encoding on the first training image, generate a second training image via an enlargement DNN by using the first training image that has undergone the first encoding, and update the enlargement DNN based on the first loss information and the third loss information, wherein the first loss information and the third loss information correspond to a result of comparing the second training image with an original image of the at least one original image that has not undergone AI reduction.

[0016] The updated DNN setting information of the amplified DNN may include weight residual information / bias residual information, wherein the weight residual information / bias residual information indicates the difference between the weights / biases of all or some filter kernels in all or some convolutional layers in the amplified DNN before updating the weights / biases and the weights / biases of all or some filter kernels in the amplified DNN after updating the weights / biases.

[0017] The updated DNN setting information of the amplified DNN may include information about weight residuals / bias residuals obtained by performing frequency transformation, and the information about weight residuals / bias residuals indicates the difference between the weights / biases of all or some filter kernels in all or some convolutional layers in the amplified DNN before updating the weights / biases and the weights / biases of all or some filter kernels in the amplified DNN after updating the weights / biases.

[0018] The weight residual information / bias residual information may be information encoded using one of differential pulse code modulation (DPCM), run length coding (RLC), and Huffman coding techniques.

[0019] The weight residual information / bias residual information may be information about weight residual / bias residual generated through model compression.

[0020] Model compression may include at least one of pruning or quantization.

[0021] The updated DNN setting information of the amplified DNN may be information updated for performing AI amplification, wherein the information for performing AI amplification is obtained by entropy encoding the weights / biases of all or some filter kernels in all or some convolutional layers in the amplified DNN after updating the weights / biases based on context model information related to the weights / biases of all or some filter kernels in all or some convolutional layers in the amplified DNN before updating the weights / biases.

[0022] The DNN setting information may include flag information indicating whether AI up-scaling is performed by using a filter kernel of a convolutional layer in a predetermined DNN, or by using a filter kernel of a convolutional layer in a DNN updated for performing AI up-scaling of the at least one second image corresponding to the at least one original image via joint training of an up-scaling DNN and a down-scaling DNN for AI down-scaling of the at least one original image, the joint training being performed using the at least one original image.

[0023] According to another embodiment of the present disclosure, an AI encoding device includes: a memory storing one or more instructions; and a processor configured to execute one or more instructions stored in the memory to perform the following operations: obtain a first image by performing AI downscaling on at least one original image via a downscaling DNN; generate AI encoded data including the image data and AI data, wherein the AI ​​data includes information related to the AI ​​downscaling, wherein the AI ​​data includes DNN setting information of an upscaling DNN for performing AI upscaling on a second image, and the second image is generated by performing a first decoding on the image data, and wherein the DNN setting information is updated via joint training of an upscaling DNN and a downscaling DNN for AI downscaling of the at least one original image to perform AI upscaling of at least one second image corresponding to the at least one original image, and the joint training is performed using the at least one original image.

[0024] According to another embodiment of the present disclosure, an AI decoding method includes: obtaining image data generated by performing first encoding on a first image and AI data related to AI reduction of at least one original image, wherein the at least one original image is related to the first image; obtaining a second image corresponding to the first image by performing first decoding on the image data; obtaining DNN setting information for performing AI upscaling of the second image based on the AI ​​data; and generating a third image by performing AI upscaling on the second image via an enlarged DNN operated according to the obtained DNN setting information, wherein the DNN setting information is updated via joint training of an enlarged DNN and a reduced DNN for AI downscaling of at least one original image to perform AI upscaling of at least one second image corresponding to the at least one original image, and the joint training is performed using at least one original image.

[0025] According to another embodiment of the present disclosure, an AI encoding method includes: obtaining a first image by performing AI downscaling on at least one original image via a downscaling DNN; generating image data by performing first encoding on the first image; and generating AI encoded data including the image data and AI data, the AI ​​data including information related to the AI ​​downscaling, wherein the AI ​​data includes DNN setting information of an upscaling DNN for performing AI upscaling on a second image, and the second image is generated by performing first decoding on the image data, and wherein the DNN setting information is updated via joint training of an upscaling DNN and a downscaling DNN for AI downscaling of the at least one original image to perform AI upscaling of at least one second image corresponding to the at least one original image, and the joint training is performed using the at least one original image.

[0026] According to another embodiment of the present disclosure, a computer-readable recording medium has recorded thereon a program for executing the above-mentioned AI decoding method and AI encoding method. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0028] Figure 1 is a diagram for describing an artificial intelligence (AI) encoding process and an AI decoding process according to an embodiment;

[0029] Figure 2 is a block diagram of an AI decoding device according to an embodiment;

[0030] Figure 3 is a diagram illustrating a deep neural network (DNN) for performing AI upscaling on an image;

[0031] Figure 4 is a diagram for describing a convolution operation performed by a convolutional layer;

[0032] Figure 5 is a table showing a mapping relationship between image related information and DNN setting information;

[0033] Figure 6 is a diagram showing an image including a plurality of frames;

[0034] Figure 7 is a block diagram of an AI encoding device according to an embodiment;

[0035] Figure 8 is a diagram illustrating a DNN for performing AI downscaling on an original image;

[0036] Fig. 9 is a diagram for describing a method of training a first DNN and a second DNN;

[0037] Fig.10 is a diagram for describing a training process of a training device on a first DNN and a second DNN;

[0038] Fig.11 is a diagram of an apparatus for performing AI downscaling on an original image and an apparatus for performing AI upscaling on a second image;

[0039] Fig.12 is a block diagram of a configuration of an AI encoding device according to an embodiment of the present disclosure;

[0040] Fig.13 is a block diagram of a configuration of an AI decoding device according to an embodiment of the present disclosure;

[0041] Fig.14a is a flowchart of an AI encoding method according to an embodiment of the present disclosure;

[0042] Fig.14b is a flowchart of an AI encoding method via DNN joint training based on original images according to an embodiment of the present disclosure;

[0043] Fig.14c is a flowchart of an AI encoding method through separate training of a DNN based on an original image according to an embodiment of the present disclosure;

[0044] Fig.15 is a flowchart of an AI decoding method according to an embodiment of the present disclosure;

[0045] Fig.16a is a diagram for describing a process of determining a plurality of optimal setting information of a reduced DNN and an enlarged DNN through joint training of the reduced DNN and the enlarged DNN by using an original image as a training image, performed by an AI encoding device as a first stage of joint training according to an embodiment of the present disclosure;

[0046] Fig.16b is used to describe the selection according to the second stage of joint training performed by the AI ​​encoding device according to an embodiment of the present disclosure Fig.16a A diagram of a process of determining setting information of a reduced DNN, determining optimal setting information of an enlarged DNN via separate training of the enlarged DNN, and generating AI encoded data including the optimal setting information of the enlarged DNN;

[0047] Fig.16cis a diagram of a process of performing AI upscaling on a second image via an upscaling DNN based on setting information of the upscaling DNN included in AI encoded data, performed by an AI decoding device according to an embodiment of the present disclosure;

[0048] Fig.17a is a diagram for describing a process of selecting setting information of a reduced DNN, determining optimal DNN setting information of an enlarged DNN through separate training of an enlarged DNN, and generating AI encoded data including the optimal DNN setting information of the enlarged DNN, performed by an AI encoding device according to an embodiment of the present disclosure;

[0049] Fig.17b is a diagram for describing a process of performing AI upscaling on a second image via an upscaling DNN based on DNN setting information of an upscaling DNN included in AI encoded data, performed by an AI decoding device according to an embodiment of the present disclosure;

[0050] Fig.18 is a flowchart of a process of upscaling a second image by updating DNN setting information of an upscaling DNN predetermined based on a flag obtained from AI encoded data, or upscaling the second image by updating DNN setting information of the upscaling DNN optimized for an original image, performed by an AI decoding device according to an embodiment of the present disclosure;

[0051] Fig.19a An example of default weights and biases, weights and biases optimized for original images in an amplified DNN, and weight differences and bias differences in an amplified DNN according to an embodiment of the present disclosure is shown;

[0052] Fig.19b An example of weights and biases of an enlarged DNN optimized for an original image and weights and biases in the enlarged DNN determined via quantization and pruning according to an embodiment of the present disclosure is shown;

[0053] Fig.20a is a diagram for describing a method of encoding weights optimized for an original image in an up-scaled DNN according to an embodiment of the present disclosure;

[0054] Fig.20b is a diagram for describing a method of encoding weights optimized for an original image in an up-scaled DNN according to an embodiment of the present disclosure;

[0055] Fig.21a is a diagram for describing a process of entropy encoding weight information optimized for an original image of an up-scaled DNN based on a context model, performed by an AI encoding device according to an embodiment of the present disclosure; and

[0056] Figure 21b2 is a diagram for describing a process of entropy decoding weight information of an up-scaled DNN optimized for an original image based on a context model, performed by an AI decoding device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] Since the present disclosure allows various changes and many examples, specific embodiments will be shown in the drawings and described in detail in the written description. However, this is not intended to limit the present disclosure to a specific mode of practice, and it should be understood that all changes, equivalents and substitutes that do not depart from the spirit and technical scope of the present disclosure are included in the present disclosure.

[0058] In the description of the embodiments, when some detailed explanations of the related art may unnecessarily obscure the essence of the present disclosure, these explanations are omitted. In addition, the numbers (eg, first, second, etc.) used in the description of the specification are merely identification marks for distinguishing one element from another.

[0059] Throughout the disclosure, the expression "at least one of A, B, or C" includes only A, only B, only C, both A and B, both A and C, both B and C, all of A, B, and C, or variations thereof.

[0060] Furthermore, in the present disclosure, it should be understood that when elements are “connected” or “coupled” to each other, the elements may be directly connected or coupled to each other, but may optionally be connected or coupled to each other through intervening elements therebetween, unless otherwise stated.

[0061] In the present invention, regarding an element represented as a "unit" or a "module", two or more elements may be combined into one element, or one element may be divided into two or more elements according to subdivided functions. In addition, each element described below may additionally perform some or all of the functions performed by another element in addition to its own main function, and some of the main functions of each element may be completely performed by another component.

[0062] Furthermore, in the present disclosure, an “image” or a “picture” may refer to a still image, a moving image including a plurality of consecutive still images (or frames), or a video.

[0063] Furthermore, in the present disclosure, a deep neural network (DNN) is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model that uses a specific algorithm.

[0064] In addition, in the present disclosure, a "parameter" is a value used in the operation process of each layer forming a neural network, and may include, for example, a weight used when an input value is applied to a specific operation expression. Here, the parameter may be expressed in a matrix form. The parameter is a set of numerical values ​​as a result of training or iteration of the neural network, and may be updated by separate training data when necessary.

[0065] Furthermore, in the present disclosure, a “first DNN” indicates a DNN used for artificial intelligence (AI) downscaling of an image, and a “second DNN” indicates a DNN used for AI upscaling of an image.

[0066] In addition, in the present disclosure, "DNN setting information" includes information related to elements constituting the DNN. "DNN setting information" includes the above-mentioned parameters as information related to elements constituting the DNN. The first DNN or the second DNN can be set by using the DNN setting information.

[0067] In addition, in the present disclosure, the "original image" refers to an image that is an object of AI encoding, and the "first image" refers to an image obtained as a result of performing AI reduction on the original image during the AI ​​encoding process. In addition, the "second image" refers to an image obtained via the first decoding during the AI ​​decoding process, and the "third image" refers to an image obtained by AI-enlarging the second image during the AI ​​decoding process.

[0068] In addition, in the present disclosure, "AI reduction" (or reduction) means a process of reducing the resolution of an image based on AI, and "first encoding" means an encoding process according to an image compression method based on frequency transformation. In addition, "first decoding" means a decoding process according to an image reconstruction method based on frequency transformation, and "AI enlargement" (or enlargement) means a process of increasing the resolution of an image based on AI.

[0069] Figure 1 is a diagram for describing an AI encoding process and an AI decoding process according to an embodiment.

[0070] When the resolution of an image is significantly increased, the throughput of information used to encode and decode the image increases. Therefore, a method for improving the efficiency of encoding and decoding an image is needed.

[0071] like Figure 1 As shown, according to an embodiment of the present disclosure, a first image 115 is obtained by performing AI reduction 110 on an original image 105 having a high resolution. Then, a first encoding 120 and a first decoding 130 are performed on the first image 115 having a relatively low resolution, so that the bit rate can be significantly reduced compared to simply performing the first encoding and the first decoding on the high-resolution original image 105.

[0072] Specifically, in Figure 1 According to an embodiment, a first image 115 is obtained by performing AI reduction 110 on an original image 105, and a first encoding 120 is performed on the first image 115 by an encoding device, a transmission source, etc. during an AI encoding process. During an AI decoding process, AI encoded data including AI data and image data obtained as a result of the AI ​​encoding is received, a second image 135 is obtained via a first decoding 130, and a third image 145 is obtained by performing AI enlargement 140 on the second image 135 by a decoding device, a receiving device, etc.

[0073] Referring to the AI ​​encoding process in detail, when the original image 105 is received, AI reduction 110 is performed on the original image 105 to obtain a first image 115 of a specific resolution or a specific quality. Here, the AI ​​reduction 110 is performed based on AI, and the AI ​​model for performing the AI ​​reduction 110 needs to be jointly trained with the AI ​​model for performing the AI ​​enlargement 140 of the second image 135. This is because, when the AI ​​model for AI reduction 110 and the AI ​​model for AI enlargement 140 are trained separately, there may be distortion caused by the difference between the original image 105 subjected to AI encoding and the third image 145 reconstructed by AI decoding.

[0074] In an embodiment of the present disclosure, the AI ​​data may be a signaling for maintaining such a joint relationship during the AI ​​encoding process and the AI ​​decoding process. Therefore, the AI ​​data obtained by the AI ​​encoding process may include information indicating an enlargement target, and during the AI ​​decoding process, the AI ​​enlargement 140 is performed on the second image 135 according to the enlargement target verified based on the AI ​​data.

[0075] The AI ​​model for AI reduction 110 and the AI ​​model for AI enlargement 140 may be implemented as a DNN. Fig. 9 As described, since the first DNN and the second DNN are jointly trained by sharing loss information under a specific target, the AI ​​encoding device may provide the target information used during the joint training of the first DNN and the second DNN to the AI ​​decoding device, and the AI ​​decoding device may perform AI upscaling 140 on the second image 135 based on the provided target information to achieve the target resolution.

[0076] about Figure 1The first encoding 120 and the first decoding 130 may reduce the amount of information of the first image 115 obtained by performing AI reduction 110 on the original image 105 through the first encoding 120. The first encoding 120 may include: a process of generating prediction data by predicting the first image 115, a process of generating residual data corresponding to the difference between the first image 115 and the prediction data, a process of transforming the residual data of the spatial domain component into a frequency domain component, a process of quantizing the residual data transformed into the frequency domain component, and a process of entropy encoding the quantized residual data. Such a first encoding 120 may be performed via one of an image compression method using frequency transformation, such as MPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, H.265 / High Efficiency Video Coding (HEVC), VC-1, VP8, VP9, ​​and AOMediaVideo1 (AV1).

[0077] The second image 135 corresponding to the first image 115 may be reconstructed by performing the first decoding 130 on the image data. The first decoding 130 may include a process of generating quantized residual data by entropy decoding the image data, a process of inversely quantizing the quantized residual data, a process of transforming the residual data of the frequency domain component into the spatial domain component, a process of generating prediction data, and a process of reconstructing the second image 135 by using the prediction data and the residual data. Such first decoding 130 may be performed via a corresponding image reconstruction method corresponding to one of the image compression methods using frequency transformation used in the first encoding 120, such as MPEG-2, H.264AVC, MPEG-4, H.265 / HEVC, VC-1, VP8, VP9, ​​and AV1.

[0078] The AI ​​encoded data obtained through the AI ​​encoding process may include image data obtained as a result of performing the first encoding 120 on the first image 115, and AI data related to the AI ​​reduction 110 of the original image 105. The image data may be used during the first decoding 130, and the AI ​​data may be used during the AI ​​enlargement 140.

[0079] The image data may be transmitted in the form of a bitstream. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data as a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding 120 performed on the first image 115. For example, the image data may include prediction mode information, motion information, and information related to quantization parameters used during the first encoding 120. The image data may be generated according to the rules (e.g., according to the syntax) of the image compression method used during the first encoding 120 in MPEG-2, H.264AVC, MPEG-4, H.265 / HEVC, VC-1, VP8, VP9, ​​and AV1.

[0080] The AI ​​data is used in the AI ​​upscaling 140 based on the second DNN. As described above, because the first DNN and the second DNN are jointly trained, the AI ​​data includes information that enables the AI ​​upscaling 140 to be consistently performed on the second image 135 by the second DNN. During the AI ​​decoding process, the AI ​​upscaling 140 may be performed on the second image 135 based on the AI ​​data to have a target resolution and / or quality.

[0081] The AI ​​data may be transmitted together with the image data in the form of a bit stream. Alternatively, according to an embodiment, the AI ​​data may be transmitted separately from the image data in the form of a frame or a packet. The AI ​​data and the image data obtained as a result of the AI ​​encoding may be transmitted via the same network or via different networks.

[0082] Figure 2 is a block diagram of an AI decoding device according to an embodiment.

[0083] Reference Figure 2 , the AI ​​decoding device 200 according to the embodiment may include a receiver 210 and an AI decoder 230. The receiver 210 may include a communicator 212, a parser 214, and an outputter 216. The AI ​​decoder 230 may include a first decoder 232 and an AI amplifier 234.

[0084] The receiver 210 receives and parses AI-encoded data obtained as a result of AI encoding, and outputs image data and AI data to the AI ​​decoder 230 .

[0085] Specifically, the communicator 212 receives AI-encoded data obtained as a result of AI encoding through a wireless network. The AI-encoded data obtained as a result of performing AI encoding includes image data and AI data. The image data and AI data may be received through the same type of network or different types of networks.

[0086] The parser 214 receives the AI ​​coded data received through the communicator 212, and parses the AI ​​coded data to separate the image data from the AI ​​data. For example, the parser 214 can distinguish the image data from the AI ​​data by reading the header of the data obtained from the communicator 212, wherein the header describes the payload including the image data and the AI ​​data or the configuration of the AI ​​data. According to an embodiment, the parser 214 sends the image data and the AI ​​data to the outputter 216 via the header of the data received through the communicator 212, and the outputter 216 sends the image data and the AI ​​data to the first decoder 232 and the AI ​​amplifier 234, respectively. At this time, the image data included in the AI ​​coded data can be verified as image data generated via a specific codec (e.g., MPEG-2, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1). In this case, the corresponding information can be sent to the first decoder 232 through the outputter 216 so that the image data is processed via the appropriate codec.

[0087] According to an embodiment, the AI ​​encoded data parsed by the parser 214 may be obtained from a data storage medium including a magnetic medium (such as a hard disk, a floppy disk, or a magnetic tape), an optical recording medium (such as a CD-ROM or a DVD), or a magneto-optical medium (such as a floppy disk).

[0088] The first decoder 232 reconstructs a second image 135 corresponding to the first image 115 based on the image data. The second image 135 obtained by the first decoder 232 is provided to the AI ​​amplifier 234. According to an embodiment, first decoding related information (such as prediction mode information, motion information, quantization parameter information, etc.) included in the image data may be further provided to the AI ​​amplifier 234.

[0089] Upon receiving the AI ​​data, the AI ​​amplifier 234 performs AI up-amplification on the second image 135 based on the AI ​​data. According to an embodiment, the AI ​​up-amplification may be performed by further using first decoding related information (such as prediction mode information, quantization parameter information, etc.) included in the image data.

[0090] The receiver 210 and the AI ​​decoder 230 according to the embodiment are described as separate components or devices, but may be jointly implemented by one processor for controlling the joint functions of the receiver 210 and the decoder 230. In this case, the receiver 210 and the AI ​​decoder 230 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processor (GPU). The dedicated processor may be implemented by including a memory for implementing the embodiments of the present disclosure or by including a memory processor for using an external memory. The communicator 212 may include a communication interface such as a wireless communication interface including a radio and / or an antenna or a wired communication interface such as HDMI, Ethernet, etc. to receive AI encoded data through a network or from a data storage medium, wherein the data storage medium includes a magnetic medium such as a hard disk, a floppy disk, or a tape, an optical recording medium such as a CD-ROM or a DVD, or a magneto-optical medium such as a floppy disk. Alternatively, the communicator 212 may include a part or component of a dedicated processor, or a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processor (GPU) to receive AI encoded data.

[0091] In addition, the receiver 210 and the AI ​​decoder 230 may be configured by multiple processors. In this case, the receiver 210 and the AI ​​decoder 230 may be implemented by a combination of dedicated processors or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a GPU). Similarly, the AI ​​amplifier 234 and the first decoder 232 may be implemented by different processors.

[0092] The AI ​​data provided to the AI ​​amplifier 234 includes information that enables the second image 135 to be processed via AI magnification. Here, the magnification target or ratio should correspond to the reduction target or ratio of the first DNN. Therefore, the AI ​​data includes information for verifying the reduction target of the first DNN.

[0093] Examples of the information included in the AI ​​data include difference information between the resolution of the original image 105 and the resolution of the first image 115 and information related to the first image 115 .

[0094] The difference information may be represented as information about the degree of resolution conversion of the first image 115 compared to the original image 105 (e.g., resolution conversion rate information). In addition, since the resolution of the first image 115 is verified by the resolution of the reconstructed second image 135, and the degree of resolution conversion is verified accordingly, the difference information may be represented as only the resolution information of the original image 105. Here, the resolution information may be represented as a vertical / horizontal size or as a ratio (16:9, 4:3, etc.) and a size of one axis. In addition, when there is preset resolution information, the resolution information may be represented in the form of an index or a flag by one or more bits transmitted by signaling.

[0095] The information related to the first image 115 may include information about at least one of a bit rate of image data obtained as a result of performing the first encoding on the first image 115 or a codec type used during the first encoding of the first image 115 .

[0096] The AI ​​amplifier 234 may determine an enlargement target of the second image 135 based on at least one of the difference information included in the AI ​​data or the information related to the first image 115. The enlargement target may indicate, for example, to what extent the resolution is to be enlarged for the second image 135. When the enlargement target is determined, the AI ​​amplifier 234 performs AI enlargement on the second image 135 through the second DNN to obtain a third image 145 corresponding to the enlargement target.

[0097] Before describing the method of performing AI up-scaling on the second image 135 according to the up-scaling target performed by the AI ​​amplifier 234, reference will be made to Figure 3 and Figure 4 Describe the AI ​​amplification process through the second DNN.

[0098] Figure 3 is a diagram illustrating a DNN 300 for performing AI upscaling on a second image 135, and Figure 4 Is used to describe Figure 3 Schematic diagram of the convolution operation in the first convolutional layer 310 of FIG.

[0099] like Figure 3 As shown, the second image 135 is input to the first convolutional layer 310. Figure 3 The mark 3×3×4 indicated in the illustrated first convolution layer 310 indicates that a convolution process is performed on one input image by using four filter kernels having a size of 3×3. As a result of the convolution process, four feature maps are generated by the four filter kernels. Each feature map indicates an inherent characteristic of the second image 135. For example, each feature map may represent a vertical direction characteristic, a horizontal direction characteristic, an edge characteristic, etc. of the second image 135.

[0100] Will refer to Figure 4The convolution operation in the first convolution layer 310 is described in detail.

[0101] One feature map 450 may be generated by multiplication and addition between parameters of a filter kernel 430 having a size of 3×3 used in the first convolution layer 310 and corresponding pixel values ​​in the second image 135. Since four filter kernels are used in the first convolution layer 310, four feature maps may be generated by convolution operations using the four filter kernels.

[0102] exist Figure 4 In the second image 135 in FIG. 4 , I1 to I49 indicate pixels in the second image 135, and F1 to F9 indicated in the filter kernel 430 indicate parameters of the filter kernel 430. In addition, M1 to M9 indicated in the feature map 450 indicate samples of the feature map 450.

[0103] exist Figure 4 , the second image 135 includes 49 pixels, but the number of pixels is only a simplified example, and when the second image 135 has a resolution of 4K, the second image 135 may include, for example, 3840×2160 pixels.

[0104] During the convolution operation process, the pixel values ​​of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination (e.g., addition) of the multiplication result values ​​may be assigned as the value of M1 of the feature map 450. When the stride of the convolution operation is 2, the pixel values ​​of I3, I4, I5, I10, I11, I12, I17, I18, and I19 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination of the multiplication result values ​​may be assigned as the value of M2 of the feature map 450.

[0105] When the filter kernel 430 advances to the last pixel of the second image 135 according to the stride, a convolution operation is performed between the pixel value in the second image 135 and the parameter of the filter kernel 430, and thus a feature map 450 having a specific size may be generated.

[0106] According to the present disclosure, the values ​​of the parameters of the second DNN, for example, the values ​​of the parameters of the filter kernel used in the convolution layer of the second DNN (for example, F1 to F9 of the filter kernel 430) can be optimized by joint training of the first DNN and the second DNN. As described above, the AI ​​amplifier 234 can determine the enlargement target corresponding to the reduction target of the first DNN based on the AI ​​data, and determine the parameters corresponding to the determined enlargement target as the parameters of the filter kernel used in the convolution layer of the second DNN.

[0107] The convolutional layers included in the first DNN and the second DNN can be based on the reference Figure 4 The convolution operation described above is used to perform the processing, but referring to Figure 4 The described convolution operation processing is only an example, and the convolution operation processing is not limited thereto.

[0108] Return to reference Figure 3 , the feature map output from the first convolution layer 310 may be input to the first activation layer 320 .

[0109] The first activation layer 320 may assign a nonlinear feature to each feature map. The first activation layer 320 may include a sigmoid function, a Tanh function, a rectified linear unit (ReLU) function, etc., but the first activation layer 320 is not limited thereto.

[0110] The first activation layer 320 assigns a nonlinear feature indicating that at least one sample value of the feature map as the output of the first convolutional layer 310 is changed. Here, the change is performed by applying a nonlinear feature.

[0111] The first activation layer 320 determines whether to send sample values ​​of the feature map output from the first convolution layer 310 to the second convolution layer 330. For example, some sample values ​​of the feature map are activated by the first activation layer 320 and sent to the second convolution layer 330, and some sample values ​​are deactivated by the first activation layer 320 and are not sent to the second convolution layer 330. The inherent characteristics of the second image 135 represented by the feature map are emphasized by the first activation layer 320.

[0112] The feature map 325 output from the first activation layer 320 is input to the second convolutional layer 330 . Figure 3 One of the feature maps 325 shown in FIG. 325 is a feature map 325 in which a reference is processed in the first activation layer 320. Figure 4 The result of the feature map 450 is described.

[0113] The notation 3×3×4 indicated in the second convolution layer 330 indicates that a convolution process is performed on the feature map 325 by using four filters having a size of 3×3. The output of the second convolution layer 330 is input to the second activation layer 340. The second activation layer 340 may assign nonlinear features to input data.

[0114] The feature map 345 output from the second activation layer 340 is input to the third convolutional layer 350. Figure 3 The mark 3×3×1 indicated in the third convolution layer 350 shown indicates that one output image is generated by performing a convolution process using one filter kernel of size 3×3. The third convolution layer 350 is a layer for outputting a final image, and generates one output by using one filter kernel. According to an embodiment of the present disclosure, the third convolution layer 350 may output the third image 145 as a result of the convolution operation.

[0115] As will be described later, there may be a plurality of pieces of DNN setting information indicating the number of filter kernels of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, parameters of the filter kernels of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, and the like, and the plurality of pieces of DNN setting information should be associated with the plurality of pieces of DNN setting information of the first DNN. The association between the plurality of pieces of DNN setting information of the second DNN and the plurality of DNN setting information of the first DNN may be achieved via joint training of the first DNN and the second DNN.

[0116] exist Figure 3 , the second DNN 300 includes three convolutional layers (a first convolutional layer 310, a second convolutional layer 330, and a third convolutional layer 350) and two activation layers (a first activation layer 320 and a second activation layer 340), but this is only an example, and the configuration and number of the convolutional layers and the activation layers may vary according to the embodiment. In addition, according to the embodiment, the second DNN 300 may be implemented as a recurrent neural network (RNN). In this case, the convolutional neural network (CNN) structure of the second DNN 300 according to the embodiment of the present disclosure is changed to an RNN structure.

[0117] According to an embodiment, the AI ​​amplifier 234 may include at least one arithmetic logic unit (ALU) for convolution operations and operations of the above-mentioned activation layers. The ALU may be implemented as a processor. For convolution operations, the ALU may include a multiplier that performs multiplication between the sample values ​​of the second image 135 or the feature map output from the previous layer and the sample values ​​of the filter kernel, and an adder that adds the result values ​​of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0118] Hereinafter, a method of performing AI up-scaling on the second image 135 according to the up-scaling target, performed by the AI ​​amplifier 234, will be described.

[0119] According to an embodiment, the AI ​​amplifier 234 may store a plurality of pieces of DNN setting information configurable in the second DNN.

[0120] Here, the DNN setting information may include information about at least one of the number of convolutional layers included in the second DNN, the number of filter kernels of each convolutional layer, or the parameters of each filter kernel. A plurality of DNN setting information may correspond to various magnification targets, respectively, and the second DNN may operate based on the DNN setting information corresponding to a certain magnification target. Based on the DNN setting information, the second DNN may have a different structure. For example, the second DNN may include three convolutional layers based on any one DNN setting information, and may include four convolutional layers based on another DNN setting information.

[0121] According to an embodiment, the DNN setting information may include only the parameters of the filter kernel used in the second DNN. In this case, the structure of the second DNN does not change, but only the parameters of the internal filter kernel may be changed based on the DNN setting information.

[0122] The AI ​​amplifier 234 may obtain DNN setting information from among the plurality of DNN setting information for performing AI amplification on the second image 135. Each of the plurality of DNN setting information may be information for obtaining the third image 145 of a predetermined resolution and / or a predetermined quality, and is jointly trained with the first DNN.

[0123] For example, one piece of DNN setting information among the multiple DNN setting information may include information for obtaining a third image 145 having a resolution twice that of the second image 135 (e.g., a third image 145 having a 4K (4096×2160) resolution that is twice the 2K (2048×1080) resolution of the second image 135), and another piece of DNN setting information may include information for obtaining a third image 145 having a resolution four times higher than the resolution of the second image 135 (e.g., a third image 145 having an 8K (8192×4320) resolution that is four times higher than the 2K (2048×1080) resolution of the second image 135).

[0124] Each of the multiple DNN setting information is Figure 6 The AI ​​encoding device 600 of the embodiment obtains the DNN setting information of the first DNN together, and the AI ​​amplifier 234 obtains one of the multiple DNN setting information according to the magnification rate corresponding to the reduction rate of the DNN setting information of the first DNN. In this regard, the AI ​​amplifier 234 can verify the information of the first DNN. In order for the AI ​​amplifier 234 to verify the information of the first DNN, the AI ​​decoding device 200 according to the embodiment receives AI data including the information of the first DNN from the AI ​​encoding device 600.

[0125] In other words, the AI ​​amplifier 234 can verify the information targeted by the DNN setting information of the first DNN used to obtain the first image 115 by using the information received from the AI ​​encoding device 600, and obtain the DNN setting information of the second DNN jointly trained with the DNN setting information of the first DNN.

[0126] When DNN setting information for performing AI up-scaling on the second image 135 is obtained from the plurality of pieces of DNN setting information, input data may be processed based on a second DNN operated according to the obtained DNN setting information.

[0127] For example, when obtaining any DNN setting information, including Figure 3 The number of filter kernels in each of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300 and the parameters of the filter kernels are set to the values ​​included in the obtained DNN setting information.

[0128] Specifically, in Figure 4 The parameters of the 3×3 filter kernel used in any convolutional layer of the second DNN are set to {1, 1, 1, 1, 1, 1, 1}, and when the DNN setting information is changed later, the parameters are replaced by {2, 2, 2, 2, 2, 2, 2, 2} as parameters included in the changed DNN setting information.

[0129] The AI ​​amplifier 234 may obtain DNN setting information for AI amplification from the plurality of pieces of DNN setting information based on information included in the AI ​​data, and the AI ​​data for obtaining the DNN setting information will now be described.

[0130] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification from the plurality of pieces of DNN setting information based on difference information included in the AI ​​data. For example, when the resolution of the original image 105 (e.g., 4K (4096×2160)) is twice the resolution of the first image 115 (e.g., 2K (2048×1080)), the AI ​​amplifier 234 may obtain DNN setting information for increasing the resolution of the second image 135 by two times based on the difference information.

[0131] According to another embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification of the second image 135 from the plurality of pieces of DNN setting information based on the information related to the first image 115 included in the AI ​​data. The AI ​​amplifier 234 may predetermine a mapping relationship between the image-related information and the DNN setting information, and obtain the DNN setting information mapped to the information related to the first image 115.

[0132] Figure 5 is a table showing the mapping relationship between image related information and DNN setting information.

[0133] according to Figure 5 , the AI ​​encoding and AI decoding processing according to the embodiments of the present disclosure may not only consider the change in resolution. Figure 5 As shown, the DNN setting information may be selected by considering resolution (such as standard definition (SD), high definition (HD), or full HD), bit rate (such as 10Mbps, 15Mbps, or 20Mbps), and codec information (such as AV1, H.264, or HEVC) individually or collectively. For such consideration of resolution, bit rate, and codec information, it is considered that the training of each element should be performed jointly with the encoding and decoding processes during the AI ​​training process (see Fig. 9 ).

[0134] Therefore, when training Figure 5 As shown, when multiple DNN setting information is provided based on image-related information including codec type, image resolution, etc., DNN setting information for AI upscaling the second image 135 can be obtained based on information related to the first image 115 received during AI decoding processing.

[0135] In other words, AI amplifier 234 is able to Figure 5 The image related information on the left side of the table is associated with the DNN setting information on the right side of the table to use the DNN setting information according to the image related information.

[0136] like Figure 5 As shown, when it is verified from information related to the first image 115 that the resolution of the first image 115 is SD, the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 10Mbps, and the first encoding is performed on the first image 115 via the AV1 codec, the AI ​​amplifier 234 can use the ADNN setting information among the multiple DNN setting information.

[0137] In addition, when it is verified from information related to the first image 115 that the resolution of the first image 115 is HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the H.264 codec, the AI ​​amplifier 234 can use B DNN setting information among the multiple DNN setting information.

[0138] In addition, when it is verified from the information related to the first image 115 that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 20Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 may use the C DNN setting information among the multiple DNN setting information. Finally, when it is verified that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 may use the DDNN setting information among the multiple DNN setting information. One of the C DNN setting information and the D DNN setting information may be selected based on whether the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 20Mbps or 15Mbps. Different bit rates of image data obtained when the first encoding is performed on the first image 115 of the same resolution via the same codec indicates different qualities of the reconstructed image. Therefore, the first DNN and the second DNN may be jointly trained based on a specific image quality, and thus, the AI ​​amplifier 234 may obtain DNN setting information according to a bit rate of image data indicating the quality of the second image 135 .

[0139] According to another embodiment, the AI ​​amplifier 234 may obtain DNN setting information for performing AI amplification on the second image 135 from the plurality of pieces of DNN setting information, taking into account both information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and information related to the first image 115 included in the AI ​​data. For example, the AI ​​amplifier 234 may receive quantization parameter information used during the first encoding process of the first image 115 from the first decoder 232, verify the bit rate of image data obtained from the AI ​​data as an encoding result of the first image 115, and obtain DNN setting information corresponding to the quantization parameter information and the bit rate. Even when the bit rate is the same, the quality of the reconstructed image may vary depending on the complexity of the image. The bit rate is a value representing the entire first image 115 on which the first encoding is performed, and the quality of each frame may vary even within the first image 115. Therefore, when the prediction mode information, motion information, and / or quantization parameter obtainable for each frame from the first decoder 232 are considered together, DNN setting information more suitable for the second image 135 may be obtained compared to when only the AI ​​data is used.

[0140] In addition, according to an embodiment, the AI ​​data may include an identifier of the mutually agreed DNN setting information. The identifier of the DNN setting information is information for distinguishing a pair of DNN setting information obtained by jointly training between the first DNN and the second DNN so that AI amplification is performed on the second image 135 to an amplification target corresponding to the reduction target of the first DNN. After obtaining the identifier of the DNN setting information included in the AI ​​data, the AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information corresponding to the identifier of the DNN setting information. For example, an identifier indicating each of a plurality of DNN setting information configurable or selectable in the first DNN and an identifier indicating each of a plurality of DNN setting information configurable or selectable in the second DNN may be pre-specified. In this case, the same identifier may be specified for a pair of DNN setting information configurable or selectable in each of the first DNN and the second DNN. The AI ​​data may include an identifier of the DNN setting information set in the first DNN for AI reduction of the original image 105. The AI ​​amplifier 234 receiving the AI ​​data may perform AI up-amplification on the second image 135 by using DNN setting information indicated by an identifier included in the AI ​​data among a plurality of DNN setting information.

[0141] In addition, according to an embodiment, the AI ​​data may include DNN setting information. The AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information after obtaining the DNN setting information included in the AI ​​data.

[0142] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter cores of each convolutional layer, the parameters of each filter core, etc.) are stored in the form of a lookup table, the AI ​​amplifier 234 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on information included in the AI ​​data, and perform AI amplification on the second image 135 by using the obtained DNN setting information.

[0143] According to an embodiment, when determining the structure of the DNN corresponding to the amplification target, the AI ​​amplifier 234 may obtain DNN setting information corresponding to the structure of the DNN, for example, parameters of a filter kernel.

[0144] The AI ​​amplifier 234 obtains DNN setting information of the second DNN through the AI ​​data including information related to the first DNN, and performs AI amplification on the second image 135 through the second DNN set based on the obtained DNN setting information. Therefore, compared with directly analyzing the features of the second image 135 for amplification, memory usage and required throughput can be reduced.

[0145] According to an embodiment, when the second image 135 includes a plurality of frames, the AI ​​amplifier 234 may independently obtain DNN setting information of a specific number of frames, or may obtain common DNN setting information of all frames.

[0146] Figure 6 is a diagram showing an image including a plurality of frames.

[0147] like Figure 6 As shown in , the second image 135 may include frames t0 to tn.

[0148] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information of the second DNN through the AI ​​data, and perform AI amplification on frames t0 to tn based on the DNN setting information. In other words, frames t0 to tn may be processed via AI amplification based on the common DNN setting information.

[0149] According to another embodiment, the AI ​​amplifier 234 may perform AI amplification on some of the frames t0 to tn (e.g., frames t0 to ta) by using the "A" DNN setting information obtained from the AI ​​data, and perform AI amplification on the frames ta+1 to tb by using the "B" DNN setting information obtained from the AI ​​data. In addition, the AI ​​amplifier 234 may perform AI amplification on the frames Tb+1 to Tn by using the "C" DNN setting information obtained from the AI ​​data. In other words, the AI ​​amplifier 234 may independently obtain DNN setting information for each frame group including a specific number of frames among a plurality of frames, and perform AI amplification on the frames included in each group by using the independently obtained DNN setting information.

[0150] According to another embodiment, the AI ​​amplifier 234 may independently obtain DNN setting information for each frame forming the second image 135. In other words, when the second image 135 includes three frames, the AI ​​amplifier 234 may perform AI amplification on the first frame by using the DNN setting information obtained about the first frame, perform AI amplification on the second frame by using the DNN setting information obtained about the second frame, and perform AI amplification on the third frame by using the DNN setting information obtained about the third frame. According to a method of obtaining DNN setting information based on information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and information related to the first image 115 included in the above-mentioned AI data, DNN setting information may be independently obtained for each frame included in the second image 135. This is because mode information, quantization parameter information, etc. may be independently determined for each frame included in the second image 135.

[0151] According to another embodiment, the AI ​​data may include information about which frame the DNN setting information obtained based on the AI ​​data is applicable to. For example, when the AI ​​data includes information indicating that the DNN setting information is valid until frame ta, the AI ​​amplifier 234 performs AI amplification on frames t0 to ta by using the DNN setting information obtained based on the AI ​​data. In addition, when another piece of AI data includes information indicating that the DNN setting information is valid until frame tn, the AI ​​amplifier 234 performs AI amplification on frames ta+1 to tn by using the DNN setting information obtained based on the other piece of AI data.

[0152] In the following, reference will be made to Figure 7 An AI encoding apparatus 600 for performing AI encoding on an original image 105 is described.

[0153] Figure 7 is a block diagram of an AI encoding device according to an embodiment.

[0154] Reference Figure 7 , the AI ​​encoding device 600 may include an AI encoder 610 and a transmitter 630. The AI ​​encoder 610 may include an AI reducer 612 and a first encoder 614. The transmitter 630 may include a data processor 632 and a communicator 634.

[0155] exist Figure 7 , the AI ​​encoder 610 and the transmitter 630 are shown as separate components or devices, but the AI ​​encoder 610 and the transmitter 630 may be jointly implemented by one processor for controlling the joint functions of the AI ​​encoder 610 and the transmitter 630. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a graphics processor GPU). The dedicated processor may be implemented by including a memory for implementing the embodiments of the present disclosure or by including a memory processor for using an external memory.

[0156] In addition, the AI ​​encoder 610 and the transmitter 630 may be configured by multiple processors. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a combination of dedicated processors or by a combination of software and multiple general-purpose processors (such as AP, CPU or GPU). The AI ​​reducer 612 and the first encoder 614 may be implemented by different processors.

[0157] The AI ​​encoder 610 performs AI reduction on the original image 105 and first encoding on the first image 115, and sends the AI ​​data and the image data to the transmitter 630. The transmitter 630 sends the AI ​​data and the image data to the AI ​​decoding device 200. The transmitter 630 may include a communication interface, such as a wireless communication interface including a radio device and / or an antenna or a wired communication interface such as HDMI, Ethernet, etc. Optionally, the transmitter 630 may include a portion or component of a dedicated processor, or a combination of a general-purpose processor (e.g., an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU)) through software.

[0158] The image data includes data obtained as a result of performing the first encoding on the first image 115. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data that is a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding process of the first image 115. For example, the image data may include prediction mode information, motion information, quantization parameter information used to perform the first encoding on the first image 115, and the like.

[0159] The AI ​​data includes information that enables AI up-scaling to be performed on the second image 135 to an up-scaling target corresponding to the down-scaling target of the first DNN. According to an embodiment, the AI ​​data may include difference information between the original image 105 and the first image 115. In addition, the AI ​​data may include information related to the first image 115. The information related to the first image 115 may include information about at least one of a resolution of the first image 115, a bit rate of image data obtained as a result of performing the first encoding on the first image 115, or a codec type used during the first encoding of the first image 115.

[0160] According to an embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information so that the AI ​​up-scaling is performed on the second image 135 to an up-scaling target corresponding to the down-scaling target of the first DNN.

[0161] Furthermore, according to an embodiment, the AI ​​data may include DNN setting information configurable in the second DNN.

[0162] The AI ​​downsizer 612 may obtain the first image 115 obtained by performing AI downsizing on the original image 105 through the first DNN. The AI ​​downsizer 612 may determine a downsizing target of the original image 105 based on a predetermined criterion.

[0163] In order to obtain the first image 115 matching the reduction target, the AI ​​reducer 612 may store a plurality of DNN setting information that may be set in the first DNN. The AI ​​reducer 612 obtains DNN setting information corresponding to the reduction target from the plurality of DNN setting information, and performs AI reduction on the original image 105 through the first DNN set according to the DNN setting information.

[0164] Each of the plurality of pieces of DNN setting information may be trained to obtain a first image 115 of a predetermined resolution and / or a predetermined quality. For example, any one of the plurality of pieces of DNN setting information may include information for obtaining a first image 115 having a resolution half that of the original image 105 (e.g., a first image 115 having a resolution of 2K (2048×1080) half that of the original image 105 having a resolution of 4K (4096×2160)), and another piece of DNN setting information may include information for obtaining a first image 115 having a resolution one-quarter that of the original image 105 (e.g., a first image 115 having a resolution of 2K (2048×1080) one-quarter that of the original image 105 having a resolution of 8K (8192×4320)).

[0165] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter cores of each convolutional layer, the parameters of each filter core, etc.) are stored in the form of a lookup table, the AI ​​downsizer 612 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on the reduction target, and perform AI downsizing on the original image 105 by using the DNN setting information.

[0166] According to an embodiment, the AI ​​downsizer 612 may determine the structure of a DNN corresponding to a downsizing target, and obtain DNN setting information, such as parameters of a filter kernel, corresponding to the structure of the DNN.

[0167] When the first DNN and the second DNN are jointly trained, a plurality of pieces of DNN setting information for performing AI reduction on the original image 105 may have an optimized value. Here, each piece of DNN setting information includes at least one of the number of convolutional layers included in the first DNN, the number of filter kernels of each convolutional layer, or a parameter of each filter kernel.

[0168] The AI ​​downsizer 612 may set the first DNN using the obtained DNN setting information for performing AI downsizing on the original image 105 to obtain the first image 115 of a specific resolution and / or a specific quality through the first DNN. When the DNN setting information for performing AI downsizing on the original image 105 is obtained from the plurality of pieces of DNN setting information, each layer in the first DNN may process input data based on information included in the DNN setting information.

[0169] Hereinafter, a method of determining a downscaling target performed by the AI ​​downscaling unit 612 will be described. The downscaling target may indicate, for example, how much resolution is reduced from the original image 105 to obtain the first image 115.

[0170] According to an embodiment, the AI ​​downsizer 612 may determine the downsizing target based on at least one of a compression ratio (e.g., a resolution difference between the original image 105 and the first image 115 , a target bit rate, etc.), a compression quality (e.g., a type of bit rate), compression history information, or a type of the original image 105 .

[0171] For example, the AI ​​downsizer 612 may determine a downsizing target based on a compression ratio, compression quality, etc. that is preset or input from a user.

[0172] As another example, the AI ​​downsizer 612 may determine the downsizing target by using the compression history information stored in the AI ​​encoding device 600. For example, based on the compression history information available to the AI ​​encoding device 600, the encoding quality, compression ratio, etc. preferred by the user may be determined, and the downsizing target may be determined based on the encoding quality determined based on the compression history information. For example, the resolution, quality, etc. of the first image 115 may be determined based on the most frequently used encoding quality according to the compression history information.

[0173] As another example, the AI ​​downsizer 612 may determine the downsizing target based on encoding qualities used more frequently than a specific threshold (eg, an average quality of encoding qualities used more frequently than a specific threshold) according to the compression history information.

[0174] As another example, the AI ​​downsizer 612 may determine the downscaling target based on the resolution, type (eg, file format), etc. of the original image 105 .

[0175] According to an embodiment, when the original image 105 includes a plurality of frames, the AI ​​downsizer 612 may independently determine downsizing targets for a specific number of frames, or may determine downsizing targets for all frames.

[0176] According to an embodiment, the AI ​​downsizer 612 may divide the frames included in the original image 105 into a certain number of groups or subsets, and independently determine the downsizing target for each group. The same or different downsizing targets may be determined for each group. According to each group, the number of frames included in the group may be the same or different.

[0177] According to another embodiment, the AI ​​downsizer 612 may independently determine a downscaling target for each frame included in the original image 105. The same or different downscaling targets may be determined for each frame.

[0178] Hereinafter, an example of the structure of the first DNN 700 on which the AI ​​reduction is based will be described.

[0179] Figure 8 is a diagram illustrating a DNN for performing AI downscaling on an original image.

[0180] like Figure 8 As shown, the original image 105 is input to the first convolution layer 710 of the first DNN 700. The first convolution layer 710 performs a convolution process on the original image 105 by using 32 filter kernels of size 5×5. Therefore, 32 feature maps generated as a result of the convolution process are input to the first activation layer 720. The first activation layer 720 may assign nonlinear features to the 32 feature maps.

[0181] The first activation layer 720 determines whether to send a sample value of a feature map output from the first convolution layer 710 to the second convolution layer 730. For example, some sample values ​​of the feature map are activated by the first activation layer 720 and sent to the second convolution layer 730, and some sample values ​​are deactivated by the first activation layer 720 and are not sent to the second convolution layer 730. Information represented by the feature map output from the first convolution layer 710 is emphasized by the first activation layer 720.

[0182] The output 725 of the first activation layer 720 is input to the second convolution layer 730. The second convolution layer 730 performs a convolution process on the input data by using 32 filter kernels of a size of 5×5. Therefore, 32 feature maps output as a result of the convolution process are input to the second activation layer 740, and the second activation layer 740 may assign nonlinear features to the 32 feature maps.

[0183] The output 745 of the second activation layer 740 is input to the third convolution layer 750. The third convolution layer 750 performs a convolution process on the input data by using one filter kernel of size 5×5. As a result of the convolution process, an image may be output from the third convolution layer 750. The third convolution layer 750 generates an output by using the one filter kernel as a layer for outputting a final image. According to an embodiment of the present disclosure, the third convolution layer 750 may output the first image 115 as a result of the convolution operation.

[0184] There may be a plurality of pieces of DNN setting information indicating the number of filter cores of the first convolution layer 710, the second convolution layer 730, and the third convolution layer 750 of the first DNN 700, a parameter of each filter core of the first convolution layer 710, the second convolution layer 730, and the third convolution layer 750 of the first DNN 700, and the plurality of pieces of DNN setting information may be associated with the plurality of pieces of DNN setting information of the second DNN. The association between the plurality of pieces of DNN setting information of the first DNN and the plurality of pieces of DNN setting information of the second DNN may be achieved through joint training of the first DNN and the second DNN.

[0185] exist Figure 8 , the first DNN 700 includes three convolutional layers (a first convolutional layer 710, a second convolutional layer 730, and a third convolutional layer 750) and two activation layers (a first activation layer 720 and a second activation layer 740), but this is only an example configuration, and the number and configuration of the convolutional layers and the activation layers may vary according to the embodiment. In addition, according to the embodiment, the first DNN 700 may be implemented as an RNN. In this case, the CNN structure of the first DNN 700 according to the embodiment of the present disclosure is changed to an RNN structure.

[0186] According to an embodiment, the AI ​​reducer 612 may include at least one ALU for convolution operations and operations of the above-mentioned activation layers. The ALU may be implemented as a processor. For convolution operations, the ALU may include a multiplier and an adder, the multiplier performs multiplication between the sample value of the original image 105 or the feature map output from the previous layer and the sample value of the filter kernel, and the adder adds the result values ​​of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, the multiplier multiplies the input sample value by the weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0187] Return to reference Figure 7, after receiving the first image 115 from the AI ​​downsizer 612, the first encoder 614 may reduce the amount of information or the amount and size of data of the first image 115 by performing first encoding on the first image 115. As a result of the first encoding performed by the first encoder 614, image data corresponding to the first image 115 may be obtained.

[0188] The data processor 632 processes at least one of the AI ​​data or the image data to be transmitted in some form. For example, when the AI ​​data and the image data are to be transmitted in the form of a bit stream, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and transmit the image data and the AI ​​data in the form of one bit stream through the communicator 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and transmit each of the bit stream corresponding to the AI ​​data and the bit stream corresponding to the image data through the communicator 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a frame or a packet, and transmit the image data in the form of a bit stream and transmit the AI ​​data in the form of a frame or a packet through the communicator 634.

[0189] The communicator 634 transmits the AI-encoded data obtained as a result of performing the AI ​​encoding through the network. The AI-encoded data obtained as a result of performing the AI ​​encoding includes image data and AI data. The image data and the AI ​​data may be transmitted through the same type of network or different types of networks.

[0190] According to an embodiment, the AI ​​encoded data obtained as a result of processing by the data processor 632 may be stored in a data storage medium including a magnetic medium (such as a hard disk, a floppy disk, or a magnetic tape), an optical recording medium (such as a CD-ROM or a DVD), or a magneto-optical medium (such as a floppy disk).

[0191] In the following, reference will be made to Fig. 9 A method of jointly training the first DNN 700 and the second DNN 300 is described.

[0192] Fig. 9 is a diagram for describing a method of training a first DNN and a second DNN.

[0193] In an embodiment, the original image 105 on which AI encoding is performed by the AI ​​encoding process is reconstructed into the third image 145 via the AI ​​decoding process, and in order to maintain consistency between the original image 105 and the third image 145 obtained as a result of AI decoding, coordination between the AI ​​encoding process and the AI ​​decoding process is required. In other words, information lost in the AI ​​encoding process needs to be reconstructed during the AI ​​decoding process, and in this regard, the first DNN 700 and the second DNN 300 need to be jointly trained to similarly consider information loss.

[0194] For accurate AI decoding, ultimately, the Fig. 9 The third training image 804 shown is compared with the original training image 801, and the corresponding quality loss information 830 is obtained. Therefore, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0195] First, we will describe Fig. 9 The training process shown.

[0196] exist Fig. 9 , the original training image 801 is an image on which AI reduction is to be performed, and the first training image 802 is an image obtained by performing AI reduction on the original training image 801. In addition, the third training image 804 is an image obtained by performing AI enlargement on the first training image 802.

[0197] The original training image 801 includes a still image or a moving image including a plurality of frames. According to an embodiment, the original training image 801 may include a luminance image extracted from a still image or a moving image including a plurality of frames. In addition, according to an embodiment, the original training image 801 may include a patch image extracted from a still image or a moving image including a plurality of frames. When the original training image 801 includes a plurality of frames, the first training image 802, the second training image, and the third training image 804 also each include a plurality of frames. When the plurality of frames of the original training image 801 are sequentially input to the first DNN 700, the plurality of frames of the first training image 802, the second training image, and the third training image 804 may be sequentially obtained by the first DNN 700 and the second DNN 300.

[0198] For joint training of the first DNN 700 and the second DNN 300, an original training image 801 is input to the first DNN 700. The original training image 801 input to the first DNN 700 is output as a first training image 802 via AI downscaling, and the first training image 802 is input to the second DNN 300. As a result of performing AI upscaling on the first training image 802, a third training image 804 is output.

[0199] Reference Fig. 9 , the first training image 802 is input to the second DNN 300, and according to an embodiment, the second training image obtained when the first encoding and the first decoding are performed on the first training image 802 may be input to the second DNN 300. In order to input the second training image to the second DNN 300, any one of the codecs of MPEG-2, H.264, MPEG-4, H.265 / HEVC, VC-1, VP8, VP9, ​​and AV1 may be used. Specifically, any one of the codecs of MPEG-2, H.264, MPEG-4, H.265 / HEVC, VC-1, VP8, VP9, ​​and AV1 may be used to perform the first encoding on the first training image 802, and to perform the first decoding on the image data corresponding to the first training image 802.

[0200] Reference Fig. 9 , separately from outputting the first training image 802 through the first DNN 700, a simplified training image 803 obtained by performing conventional downscaling on the original training image 801 is obtained. Here, conventional downscaling may include at least one of bilinear scaling, bicubic scaling, Lanczos scaling, or step scaling.

[0201] In order to prevent the structural features of the first image 115 from greatly deviating from the structural features of the original image 105 , a simplified training image 803 is obtained to retain the structural features of the original training image 801 .

[0202] Before performing training, the first DNN 700 and the second DNN 300 may be configured according to predetermined DNN setting information. When performing training, structure loss information 810, complexity loss information 820, and quality loss information 830 may be determined.

[0203] The structural loss information 810 may be determined based on the result of comparing the simplified training image 803 and the first training image 802. For example, the structural loss information 810 may correspond to the difference between the structural information of the simplified training image 803 and the structural information of the first training image 802. The structural information may include various features that can be extracted from the image, such as brightness, contrast, histogram, etc. of the image. The structural loss information 810 indicates how much structural information of the original training image 801 is maintained in the first training image 802. When the structural loss information 810 is small, the structural information of the first training image 802 is similar to the structural information of the original training image 801.

[0204] The complexity loss information 820 may be determined based on the spatial complexity of the first training image 802. For example, the total variance value of the first training image 802 may be used as the spatial complexity. The complexity loss information 820 is related to the bit rate of the image data obtained by performing the first encoding on the first training image 802. It is defined that when the complexity loss information 820 is small, the bit rate of the image data is low.

[0205] The quality loss information 830 may be determined based on a result of comparing the original training image 801 and the third training image 804. The quality loss information 830 may include at least one of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, or a video multi-method assessment fusion (VMAF) value about the difference between the original training image 801 and the third training image 804. The quality loss information 830 indicates how similar the third training image 804 is to the original training image 801. When the quality loss information 830 is small, the third training image 804 is more similar to the original training image 801.

[0206] Reference Fig. 9 , the structure loss information 810, the complexity loss information 820, and the quality loss information 830 are used to train the first DNN 700, and the quality loss information 830 is used to train the second DNN 300. In other words, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0207] The first DNN 700 may update parameters so that the final loss information determined based on the structure loss information 810, the complexity loss information 820, and the quality loss information 830 is reduced or minimized. In addition, the second DNN 300 may update parameters so that the quality loss information 830 is reduced or minimized.

[0208] Final loss information for training the first DNN 700 and the second DNN 300 may be determined as Equation 1 below.

[0209] [Equation 1]

[0210] LossDS = a × structure loss information + b × complexity loss information + c × quality loss information

[0211] LossUS=d×quality loss information

[0212] In Equation 1, LossDS indicates final loss information to be reduced or minimized to train the first DNN 700, and LossUS indicates final loss information to be reduced or minimized to train the second DNN 300. In addition, a, b, c, and d may be predetermined certain weights.

[0213] In other words, the first DNN 700 updates parameters so that the LossDS of Equation 1 decreases, and the second DNN 300 updates parameters so that the LossUS decreases. When the parameters of the first DNN 700 are updated according to the LossDS derived during training, the first training image 802 obtained based on the updated parameters becomes different from the previous first training image 802 obtained based on the parameters that have not been updated, and therefore, the third training image 804 also becomes different from the previous third training image 804. When the third training image 804 becomes different from the previous third training image 804, the quality loss information 830 is also newly determined, and the second DNN 300 updates the parameters accordingly. When the quality loss information 830 is newly determined, the LossDS is also newly determined, and the first DNN 700 updates the parameters according to the newly determined LossDS. In other words, the update of the parameters of the first DNN 700 leads to the update of the parameters of the second DNN 300, and the update of the parameters of the second DNN 300 leads to the update of the parameters of the first DNN 700. Therefore, because the first DNN 700 and the second DNN 300 are jointly trained by sharing the quality loss information 830, the parameters of the first DNN 700 and the parameters of the second DNN 300 can be jointly optimized.

[0214] Referring to Equation 1, it is verified that LossUS is determined according to the quality loss information 830 , but this is only an example, and the loss may be determined based on at least one of the structure loss information 810 and the complexity loss information 820 , as well as the quality loss information 830 .

[0215] In the above, it has been described that the AI ​​amplifier 234 of the AI ​​decoding device 200 and the AI ​​reducer 612 of the AI ​​encoding device 600 store multiple DNN setting information, and now a method of training each of the multiple DNN setting information stored in the AI ​​amplifier 234 and the AI ​​reducer 612 will be described.

[0216] As described with reference to Equation 1, the first DNN 700 updates parameters by considering the similarity between the structural information of the first training image 802 and the structural information of the original training image 801 (structural loss information 810), the bit rate of the image data obtained as a result of performing the first encoding on the first training image 802 (complexity loss information 820), and the difference between the third training image 804 and the original training image 801 (quality loss information 830).

[0217] Specifically, the parameters of the first DNN 700 can be updated so that a first training image 802 having structural information similar to that of the original training image 801 is obtained, and image data having a small bit rate is obtained when the first encoding is performed on the first training image 802, and at the same time, the second DNN 300 that performs AI amplification on the first training image 802 obtains a third training image 804 similar to the original training image 801.

[0218] The direction of optimizing the value of the parameter of the first DNN 700 may be changed by adjusting the weights a, b, and c of Equation 1. For example, when the weight b is determined to be high, the value of the parameter of the first DNN 700 may be updated by giving priority to the low bit rate of the third training image 804 over the high quality. In addition, when the weight c is determined to be high, the value of the parameter of the first DNN 700 may be updated by giving priority to the high quality of the third training image 804 over the high bit rate or maintaining the structural information of the original training image 801.

[0219] In addition, the direction of optimizing the values ​​of the parameters of the first DNN 700 may vary according to the type of codec used to perform the first encoding on the first training image 802. This is because the second training image to be input to the second DNN 300 may vary according to the type of codec.

[0220] In other words, the values ​​of the parameters of the first DNN 700 and the values ​​of the parameters of the second DNN 300 may be jointly updated based on the weights a, b, and c and the type of codec used to perform the first encoding on the first training image 802. Therefore, when the first DNN 700 and the second DNN 300 are trained after the weights a, b, and c are each determined to be a specific value and the type of codec is determined to be a specific type, the values ​​of the parameters of the first DNN 700 and the corresponding values ​​of the parameters of the second DNN 300 may be jointly optimized.

[0221] In addition, when the first DNN 700 and the second DNN 300 are trained after changing the weights a, b, and c and the type of codec, the values ​​of the parameters of the first DNN 700 and the values ​​of the parameters of the second DNN 300 that are jointly optimized may be determined. In other words, when the first DNN 700 and the second DNN 300 are trained while changing the values ​​of the weights a, b, and c and the type of codec, a plurality of pieces of DNN setting information that are jointly trained with each other may be determined in the first DNN 700 and the second DNN 300.

[0222] As mentioned above Figure 5As described above, a plurality of pieces of DNN setting information of the first DNN 700 and the second DNN 300 may be mapped to information related to the first image. In order to set such a mapping relationship, the first training image 802 output from the first DNN 700 via a specific codec may be first encoded according to a specific bit rate, and the second training image obtained by performing a first decoding on a bit stream obtained as a result of performing the first encoding may be input to the second DNN 300. In other words, by training the first DNN 700 and the second DNN 300 after setting an environment such that the first encoding is performed on the first training image 802 of a specific resolution via a specific codec according to a specific bit rate, a pair of DNN setting information mapped to the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained as a result of performing the first encoding on the first training image 802 may be determined. By differently changing the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained according to the first encoding of the first training image 802, the mapping relationship between multiple DNN setting information of the first DNN 700 and the second DNN 300 and multiple information related to the first image can be determined.

[0223] Fig.10 2 is a diagram for describing a training process of a training device on a first DNN and a second DNN.

[0224] Reference Fig. 9 The training of the first DNN 700 and the second DNN 300 described above may be performed by the training device 1000. The training device 1000 may include the first DNN 700 and the second DNN 300. The training device 1000 may be, for example, the AI ​​encoding device 600 or a separate server. The DNN setting information of the second DNN 300 obtained as a result of the training may be stored in the AI ​​decoding device 200.

[0225] Reference Fig.10 In operation S840 and operation S845, the training device 1000 initially sets the DNN setting information of the first DNN 700 and the second DNN 300. Therefore, the first DNN 700 and the second DNN 300 may operate according to the initialized DNN setting information. The DNN setting information may include information about at least one of the number of convolutional layers included in the first DNN 700 and the second DNN 300, the number of filter kernels of each convolutional layer, the size of the filter kernel of each convolutional layer, or the parameters of each filter kernel.

[0226] In operation S850, the training apparatus 1000 provides the original training image 801 as an input to the first DNN 700. The original training image 801 may include a still image or at least one frame included in a moving image.

[0227] In operation S855, the first DNN 700 processes the original training image 801 according to the initialized DNN setting information, and outputs a first training image 802 obtained by performing AI reduction on the original training image 801. Fig.10 , the first training image 802 output from the first DNN 700 is directly input to the second DNN 300, but the first training image 802 output from the first DNN 700 may be input to the second DNN 300 via the training device 1000. In addition, the training device 1000 may perform first encoding and first decoding on the first training image 802 via a specific codec, and provide the second training image to the second DNN 300 as an input.

[0228] In operation S860, the second DNN 300 processes the first training image 802 or the second training image according to the initialized DNN setting information, and outputs a third training image 804 obtained by performing AI up-scaling on the first training image 802 or the second training image.

[0229] In operation S865 , the training apparatus 1000 calculates complexity loss information 820 based on the first training image 802 .

[0230] In operation S870 , the training apparatus 1000 calculates structure loss information 810 by comparing the simplified training image 803 and the first training image 802 .

[0231] In operation S875 , the training apparatus 1000 calculates quality loss information 830 by comparing the original training image 801 and the third training image 804 .

[0232] In operation S880, the initial DNN setting information is updated via a feedback propagation process based on the final loss information. The training apparatus 1000 may calculate the final loss information for training the first DNN 700 based on the complexity loss information 820, the structure loss information 810, and the quality loss information 830.

[0233] In operation S885, the second DNN 300 updates the DNN setting information via a feedback propagation process based on the quality loss information 830 or the final loss information. The training apparatus 1000 may calculate the final loss information for training the second DNN 300 based on the quality loss information 830.

[0234] Then, the training device 1000, the first DNN 700, and the second DNN 300 may repeat operations S850 to S885 until the final loss information is minimized to update the DNN setting information. At this time, during each repetition, the first DNN 700 and the second DNN 300 operate according to the DNN setting information updated in the previous operation.

[0235] Table 1 below shows the effects when AI encoding and AI decoding are performed on the original image 105 according to an embodiment of the present disclosure and when encoding and decoding are performed on the original image 105 via HEVC.

[0236] Table 1

[0237]

[0238] As shown in Table 1, although the subjective image quality when AI encoding and AI decoding are performed on content with 8K resolution including 300 frames is higher than when encoding and decoding are performed via HEVC, the bit rate is still reduced by at least 50%.

[0239] Fig.11 is a diagram of an apparatus for performing AI downscaling on an original image and an apparatus for performing AI upscaling on a second image.

[0240] Device 20 receives the original image 105 and provides image data 25 and AI data 30 to device 40 by using AI downsizer 1124 and transform-based encoder 1126. According to an embodiment, image data 25 and AI data 30 are Figure 1 The image data corresponds to the AI ​​data 30. Figure 1 In addition, according to an embodiment, the transform-based encoder 1126 and Figure 7 The first encoder 614 corresponds to the AI ​​reducer 1124 and Figure 7 The AI ​​reducer 612 responds accordingly.

[0241] The device 40 receives the AI ​​data 30 and the image data 25, and obtains the third image 145 by using the transform-based decoder 1146 and the AI ​​amplifier 1144. According to an embodiment, the transform-based decoder 1146 and the AI ​​amplifier 1144 are Figure 2 The first decoder 232 corresponds to the AI ​​amplifier 1144 and Figure 2 The AI ​​amplifier 234 responds accordingly.

[0242] According to an embodiment, the device 20 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the device 20 executes the reference to the computer program according to the execution of the computer program by the CPU. Fig.11According to the embodiment, reference will be made to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0243] According to an embodiment, the device 40 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the device 40 executes the reference to the computer program according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, reference will be made to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0244] exist Fig.11 , the configuration controller 1122 receives at least one input value 10. According to an embodiment, the at least one input value 10 may include at least one of a target resolution difference of the AI ​​downsizer 1124 and the AI ​​amplifier 1144, a bit rate of the image data 25, a bit rate type of the image data 25 (e.g., a variable bit rate type, a constant bit rate type, or an average bit rate type), or a codec type of the transform-based encoder 1126. The at least one input value 10 may include a value pre-stored in the device 20 or a value input from a user.

[0245] The configuration controller 1122 controls the operations of the AI ​​downsizer 1124 and the transform-based encoder 1126 based on the received input value I0. According to an embodiment, the configuration controller 1122 obtains DNN setting information for the AI ​​downsizer 1124 according to the received input value 10, and configures the AI ​​downsizer 1124 with the DNN setting information. According to an embodiment, the configuration controller 1122 may send the input value 10 to the AI ​​downsizer 1124, and the AI ​​downsizer 1124 may obtain DNN setting information for performing AI downsizing on the original image 105 based on the input value 10. According to an embodiment, the configuration controller 1122 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI downsizing is applied and tone mapping information of a high dynamic range (HDR)) together with the input value 10 to the AI ​​downsizer 1124, and the AI ​​downsizer 1124 may obtain the DNN setting information in consideration of the input value 10 and the additional information. According to an embodiment, the configuration controller 1122 sends at least a portion of the received input value 10 to the transform-based encoder 1126, and the transform-based encoder 1126 performs a first encoding on the first image 115 via a bit rate of a specific value, a specific type of bit rate, and a specific codec.

[0246] The AI ​​reducer 1124 receives the original image 105 and performs a reference Figure 1 , Figure 7 , Figure 8 , Fig. 9or Fig.10 At least one of the operations described in is performed to obtain a first image 115.

[0247] According to an embodiment, the AI ​​data 30 is provided to the device 40. The AI ​​data 30 may include at least one of resolution difference information between the original image 105 and the first image 115 or information related to the first image 115. The resolution difference information may be determined based on a target resolution difference in the input value 10, and the information related to the first image 115 may be determined based on at least one of a target bit rate, a bit rate type, or a codec type. According to an embodiment, the AI ​​data 30 may include parameters used during AI upscaling. The AI ​​data 30 may be provided to the device 40 from the AI ​​downsizer 1124.

[0248] Image data 25 is obtained when the original image 105 is processed by a transform-based encoder 1126 and sent to the device 40. The transform-based encoder 1126 may process the first image 115 according to MPEG-2, H.264 AVC, MPEG-4, H.265 / HEVC, VC-1, VP8, VP9, ​​or VA1.

[0249] The configuration controller 1142 controls the operation of the AI ​​amplifier 1144 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 obtains DNN setting information for the AI ​​amplifier 1144 according to the received AI data 30, and configures the AI ​​amplifier 1144 according to the DNN setting information. According to an embodiment, the configuration controller 1142 may send the received AI data 30 to the AI ​​amplifier 1144, and the AI ​​amplifier 1144 may obtain DNN setting information for performing AI amplification on the second image 135 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI amplification is applied and tone mapping information of HDR) to the AI ​​amplifier 1144 together with the AI ​​data 30, and the AI ​​amplifier 1144 may obtain the DNN setting information in consideration of the AI ​​data 30 and the additional information. According to an embodiment, the AI ​​amplifier 1144 may receive AI data 30 from the configuration controller 1142, receive at least one of prediction mode information, motion information, or quantization parameter information from the transform-based decoder 1146, and obtain DNN setting information based on the AI ​​data 30 and at least one of the prediction mode information, motion information, and quantization parameter information.

[0250] The transform-based decoder 1146 may process the image data 25 to reconstruct the second image 135. The transform-based decoder 1146 may process the image data 25 according to MPEG-2, H.264 AVC, MPEG-4, H.265 / HEVC, VC-1, VP8, VP9, ​​or AV1.

[0251] The AI ​​amplifier 1144 may obtain the third image 145 by performing AI up-amplification on the second image 135 provided from the transform-based decoder 1146 based on the set DNN setting information.

[0252] The AI ​​reducer 1124 may include a first DNN, and the AI ​​amplifier 1144 may include a second DNN, and according to an embodiment, according to the reference Fig. 9 and Fig.10 The described training method trains the first DNN and the DNN setting information of the second DNN.

[0253] Referenced above Figures 1 to 11 The AI ​​encoding technology and AI decoding technology according to the embodiments of the present disclosure are described. Figure 12 to Figure 2 1 describes in detail an AI encoding technique and an AI decoding technique for performing upscaling by updating upscaling DNN setting information optimized for an original image according to another embodiment of the present disclosure. The DNN setting information may include at least one of the number of convolutional layers, the number of filter kernels in each convolutional layer, the size of each filter kernel, or information about a parameter of each filter kernel.

[0254] Hereinafter, the "parameters" included in the "DNN setting information" are values ​​used in mathematical operations of each layer constituting the neural network, and may include, for example, weights and biases used when applying an input value to a predefined operation formula. The weight may be a value for performing multiplication with the input value, and the bias may be a value for performing addition with a value obtained as a result of performing multiplication between the input value and the weight. In addition, the parameter may be represented in a matrix form. For example, the weight may be a parameter of a 3×3 filter kernel used in a convolution layer for AI magnification or AI reduction, and the bias may be a parameter in a 1×1 matrix form added before applying an activation function to a result obtained after performing a number of convolution operations equal to the number of input channels (depth) in the convolution layer for AI magnification or AI reduction and adding together the result values ​​of each convolution operation. The weights and biases according to an embodiment of the present disclosure will be described in more detail below with reference to Equation 9. The parameter is a set of numbers as a training result, and may be updated by separate training data or training data consisting of original images.

[0255] Hereinafter, "model compression" refers to a compression technique for reducing the amount of data while maintaining the highest possible accuracy by reducing the number and size of parameters in the artificial neural network model to reduce the complexity of the artificial neural network model. Examples of "model compression" include pruning and quantization, but model compression is not limited thereto.

[0256] Fig.12 is a block diagram of a configuration of an AI encoding device according to an embodiment of the present disclosure. Fig.12 , the AI ​​encoding device 1200 includes an AI encoder 1210 and a transmitter 1230. The AI ​​encoder 1210 includes a DNN setting information updater 1212, an AI reducer 1216, and a first encoder 1214.

[0257] As described above, the AI ​​downsizer 1216 downsamples the original image (e.g., Figure 1 105) performs AI downscaling to obtain a first image (e.g., Figure 1 115). The first image is an image with a resolution lower than that of the original image. Since the AI ​​reduction performed by the AI ​​reducer 1216 has been described above, its redundant description is omitted below. The AI ​​encoding device 1200 may include a central processor for controlling the AI ​​encoder 1210 and the transmitter 1230. Optionally, the AI ​​encoder 1210 and the transmitter 1230 are operated by respective processors, and when the processors work closely together, the AI ​​encoding device 1200 may be operated as a whole. Optionally, the AI ​​encoder 1210 and the transmitter 1230 may be controlled by an external processor.

[0258] The AI ​​encoding device 1200 may further include one or more data memories or storages for storing data input to or output from the DNN setting information updater 1212, the AI ​​reducer 1216, the first encoder 1214, and the transmitter 1230. The AI ​​encoding device 1200 may further include a memory controller that controls input / output of data stored in the data memories.

[0259] In order to encode an image, the AI ​​encoding device 1200 can perform image encoding operations including prediction by interacting with a built-in or external video encoding processor. According to an embodiment of the present disclosure, the built-in video encoding processor of the AI ​​encoding device 1200 can implement basic image encoding operations together with a CPU or GPU including an image encoding processing module and a separate processor.

[0260] The DNN setting information updater 1212 may update DNN setting information of a DNN for performing AI upscaling corresponding to a DNN for performing AI downscaling. The DNN setting information updater 1212 may update setting information for performing AI upscaling of at least one second image corresponding to at least one original image by using at least one original image as a training image, through joint training of an upscaling DNN and a downscaling DNN for AI downscaling of at least one original image, or separate training of an upscaling DNN.

[0261] The DNN setting information updater 1212 may not select the DNN setting information of the reduced DNN for AI reduction of the original image, but determine the DNN setting information of the reduced DNN and the DNN setting information of the enlarged DNN through joint training of the enlarged DNN and the reduced DNN using the original image as a training image.

[0262] In this case, the number of convolutional layers, the number of filter kernels in each convolutional layer, and the size of each filter kernel included in the DNN setting information may be determined. Then, joint training may be performed based on the number of convolutional layers, the number of filter kernels in each convolutional layer, and the size of each filter kernel. For example, the number of convolutional layers, the number of filter kernels in each convolutional layer, and the size of each filter kernel may be determined based on the structure of the most complex enlarged DNN included in the default enlarged DNN setting information. In other words, the number of convolutional layers may be determined to be less than or equal to the number of convolutional layers in the most complex enlarged DNN included in the default enlarged DNN setting information, and the number of filter kernels in each convolutional layer may be determined to be less than or equal to the number of filter kernels in each most complex convolutional layer included in the default enlarged DNN setting information. The size of each filter kernel may be determined to be less than or equal to the size of each filter kernel in the most complex enlarged DNN included in the default enlarged DNN setting information.

[0263] In addition, in this case, the initial parameters included in the DNN setting information of the reduced / enlarged DNN required for joint training may be randomly initialized parameters. For example, the initial parameters of the reduced / enlarged DNN may be values ​​randomly sampled based on a specific probability distribution. For example, the weights between the initial parameters of the reduced / enlarged DNN may be values ​​randomly sampled based on a Gaussian probability distribution. In addition, for example, the bias between the initial parameters of the reduced / enlarged DNN may be initialized to 0.

[0264] Optionally, for example, the initial parameters of the downscaling / upscaling DNN required for the joint training may be one of a plurality of predetermined downscaling / upscaling DNN parameters. Optionally, the initial parameters of the downscaling / upscaling DNN required for the joint training may be default downscaling / upscaling DNN parameters. Optionally, the initial parameters of the downscaling / upscaling DNN required for the joint training may be parameters of the downscaling / upscaling DNN used in an immediately preceding group of pictures (GOP) unit, an immediately preceding intra random access point (IRAP) period unit, an immediately preceding sequence unit, an immediately preceding unit of a preset number of frames, and the like.

[0265] The image input to the upscaling DNN during joint training may be a first image generated by downscaling the original image through the downscaling DNN, but the image is not limited thereto. The input image may be a second image generated by performing first encoding and first decoding on the first image obtained after downscaling (e.g., Figure 1 135).

[0266] Thereafter, the DNN setting information updater 1212 may select the determined parameters of the reduced DNN and determine the parameters of the enlarged DNN through separate training of the enlarged DNN using at least one original image as a training image. In this case, the initial parameters of the enlarged DNN available before the separate training may be the parameters of the enlarged DNN determined during the joint training, but the initial parameters are not limited thereto. The weights in the enlarged DNN may be values ​​randomly sampled based on a specific probability distribution, and the bias in the enlarged DNN may be determined to be 0.

[0267] The image input to the enlarged DNN during the separate training for separate training of the enlarged DNN may be a second image generated by performing first encoding and first decoding on a first image, wherein the first image is generated by reducing the original image via the reduction DNN.

[0268] In this case, the parameters of the upscaled DNN generated during or after training may be ultimately generated via model compression.

[0269] Model compression as used herein refers to a compression technique for reducing the amount of data while maintaining the highest possible accuracy by reducing the number and size of parameters in the artificial neural network model to reduce the complexity of the artificial neural network model. Information loss may occur during model compression. Representative examples of model compression include quantization and pruning, but model compression is not limited to this. Quantization refers to the process of dividing continuous changes into a finite number of levels of discontinuous changes and assigning a unique value to each level. A typical example of quantization is the process of converting non-integer values ​​into integer values ​​via rounding operations such as rounding, rounding down, and rounding up. Specifically, quantization may include transforming the data type of the parameter information of the amplified DNN, and, for example, may include transforming real data types (such as float32 and double64) into integer data types (such as int16 and int8).

[0270] Pruning refers to a process of changing a value with a small amount of data to zero. A typical example of pruning includes an operation of setting a DNN parameter value to 0 (the DNN parameter value is less than a specific value close to 0 and included in the parameter information of the enlarged DNN), but pruning is not limited thereto.

[0271] Alternatively, instead of performing joint training of the reduced DNN and the enlarged DNN, the DNN setting information updater 1212 may select parameters of the reduced DNN, and determine parameters of the enlarged DNN for enlarging at least one original image through separate training using at least one original image as a training image. For example, the initial parameters of the reduced DNN required for separate training may be one of a plurality of predetermined reduced DNN parameters or a default reduced DNN parameter. In addition, as initial parameters of the enlarged DNN available before separate training, the weights in the enlarged DNN may be values ​​randomly sampled based on a specific probability distribution, and the bias in the enlarged DNN may be determined to be 0.

[0272] However, the embodiments of the present disclosure are not limited thereto, and the initial parameters of the reduction / enlargement DNN required for separate training may be parameters of the reduction / enlargement DNN used in an immediately preceding group of pictures (GOP) unit, an immediately preceding instantaneous decoding refresh (IDR) picture period (IRAP) period unit, an immediately preceding sequence unit, an immediately preceding unit of a preset number of frames, and the like.

[0273] In this case, the image input to the enlargement DNN for separate training of the enlargement DNN may be a second image generated by performing first encoding and first decoding on a first image generated by reducing the original image through the reduction DNN.

[0274] The DNN setting information updater 1212 may generate a first training image via a reduced DNN by using at least one original image as a training image, generate a third training image via an enlarged DNN by using the first training image, and update the enlarged DNN and the reduced DNN based on first loss information and third loss information and second loss information generated based on the first training image, the first loss information and the third loss information each corresponding to a result of comparing the third training image with the original image that has not undergone reduction. Here, the first loss information may be information based on a result of comparing a quality parameter of the third training image with a quality parameter of the original image. In other words, when the first loss information has a smaller value, this may indicate that the quality of the original image is more similar to the quality of the third training image. The first loss information may be information used for training using dedicated structural information of the original image. The first loss information may be related to Fig. 9 The quality loss information 830 corresponds.

[0275] In this case, the comparison result of the quality parameter may be a quality comparison parameter about the difference between the third training image and the original image, such as an L1 norm value, an L2 norm value, an SSIM value, a PSNR-HVS value, an MS-SSIM value, a VIF value, or a VMAF value, but the quality comparison parameter is not limited thereto. The comparison result of the quality parameter may be determined based on a combination of quality comparison parameters. For example, the first loss information may be determined based on the following equation 2. In equation 2, i, j, and k each represent a weight for adjusting the relative importance of each quality comparison parameter.

[0276] [Equation 2]

[0277] First loss information = i×L1 norm value + j×SSIM value, i×L2 norm value + j×VIF value, or i×L1 norm value + j×L2 norm value + k×SSIM value…

[0278] The third loss information may be information based on a result of comparing the feature-related parameters of the third training image with the feature-related parameters of the original image. In other words, when the third loss information has a smaller value, this may indicate that the features of the original image are more similar to the features of the third training image. The third loss information may be information for training using dedicated detail information (i.e., feature-related information) of the original image.

[0279] For example, the result of comparing the feature-related parameters may be an L1 norm value or an L2 norm value of the feature of each image, but the result of comparing the feature-related parameters is not limited thereto. Edge information determined based on a Sobel edge detector or a Canny edge detector may be a feature-related parameter.

[0280] The second loss information may be information related to the spatial complexity of the first training image. For example, the second loss information may represent the total variance of the first training image. The smaller the second loss, the smaller the amount of image data transmitted through the bitstream. The second loss information may be related to Fig. 9 The complexity loss information 820 corresponds.

[0281] Therefore, the first loss information and the third loss information are information related to distortion, and the second loss information is information related to rate, and when training is performed by considering all of the first loss information, the second loss information, and the third loss information, the DNN setting information updater 1212 can determine the optimal up-scaling DNN parameters based on the rate-distortion (RD) cost.

[0282] The DNN setting information updater 1212 may determine the loss value LOSS based on the following equation 3. DS , and based on the loss value LOSS DS To update the parameters of the enlarged DNN and the reduced DNN, the loss value LOSS DS Reduce.

[0283] [Equation 3]

[0284] LOSS DS =a×first loss information+b×second loss information+c×third loss information

[0285] In addition, the DNN setting information updater 1212 may determine the loss value Loss based on the following Equation 4 by additionally considering the fourth loss information: DS , and based on the loss value Loss DS Update the parameters of the enlarged DNN and the reduced DNN so that the loss value Loss DS In this case, the fourth loss information can be Fig. 9 The structural loss information 810 corresponds.

[0286] [Equation 4]

[0287] Loss DS =a×first loss information+b×second loss information+c×third loss information+d×fourth loss information

[0288] The DNN setting information updater 1212 may select parameters of a downscaling DNN, perform first encoding on a first training image obtained by downscaling at least one original image via the downscaling DNN, generate a third training image by upscaling a second training image obtained by performing first decoding on the first training image that has undergone the first encoding via the upscaling DNN, and update the upscaling DNN based on first loss information and third loss information, wherein the first loss information and the third loss information each correspond to a result of comparing the third training image with the original image that has not undergone AI downscaling. The first loss information and the third loss information may be the first loss information and the third loss information described above.

[0289] For example, the DNN setting information updater 1212 may determine the loss value Loss based on the following equation 5: US , and based on the loss value Loss US To update the parameters of the amplified DNN, so that the loss value Loss US Reduce.

[0290] [Equation 5]

[0291] Loss US = d×first loss information+e×third loss information

[0292] The DNN setting information updater 1212 may generate information about parameters of the enlarged DNN obtained through joint training of the reduced DNN and the enlarged DNN or separate training of the enlarged DNN.

[0293] In addition, the DNN setting information updater 1212 may generate information about at least one of the number of convolutional layers, the number of filter kernels in each convolutional layer, or the size of each filter kernel.

[0294] The DNN setting information updater 1212 may encode the DNN setting information of the enlarged DNN. For example, the DNN setting information updater 1212 may encode at least one of the number of convolutional layers, the number of filter kernels in each convolutional layer, or the size of each filter kernel. In addition, the DNN setting information updater 1212 may encode information about the parameters of the enlarged DNN. For example, the DNN setting information updater 1212 may encode the weight residual or bias residual of the filter kernel in the enlarged DNN. In this case, the weight residual or bias residual may be the difference between the weight or bias encoded in the immediately preceding time period unit and the weight or bias determined for the current time period unit.

[0295] In this case, encoding may be performed using differential pulse code modulation (DPCM), run length coding (RLC), and Huffman coding, but encoding is not limited thereto. Encoding may be performed using various other entropy coding techniques.

[0296] The embodiments of the present disclosure are not limited thereto, and the DNN setting information updater 1212 may perform entropy coding on the DNN setting information of the enlarged DNN based on the context model. For example, the DNN setting information updater 1212 may generate a context model based on the DNN setting information of the enlarged DNN for the previous time period unit, update the DNN setting information of the enlarged DNN for the current time period unit, and entropy code the updated DNN setting information of the enlarged DNN by using the context model. In this case, the context model refers to a model representing the probability of occurrence of a symbol estimated based on various context information of the surrounding environment.

[0297] Whenever the weight or bias of the filter kernel in the enlarged DNN of the current time period unit is encoded, the context model may be updated, and information about the next weight or bias of the filter kernel in the enlarged DNN of the current time period unit may be entropy encoded based on the updated context model. The context model may be determined for the entire enlarged DNN, but the determination is not limited thereto, and the context model may be determined for each convolution layer, for at least one filter kernel corresponding to one output channel in the convolution layer, and for each filter kernel in the convolution layer. Examples of context-based entropy coding may include range coding, arithmetic coding, and Huffman coding, but entropy coding is not limited thereto.

[0298] The DNN setting information updater 1212 may update the weights or biases of the filter kernels in all layers of the enlarged DNN, and encode information about the updated weights or biases of the filter kernels in all layers, but the update is not limited to this. The DNN setting information updater 1212 may update the weights or biases in some layers, and encode information about the updated weights or biases in the corresponding layers. Optionally, the DNN setting information updater 1212 may update the weights or biases of some filter kernels in the layer, and encode information about the updated weights or biases of some filter kernels. For example, the DNN setting information updater 1212 may update the weights or biases in the last layer of the enlarged DNN, and encode information about the updated weights or biases in the last layer. The DNN setting information updater 1212 may encode the updated DNN setting information including weight residual information or bias residual information of the enlarged DNN. The weight residual information or the bias residual information indicates the difference between the weights or biases of all or some filter kernels in all or some convolutional layers in the DNN before updating the weights or biases and the weights or biases of all or some filter kernels in all or some convolutional layers in the DNN after updating the weights or biases.

[0299] The DNN setting information updater 1212 may perform frequency transformation on the weight residuals and encode updated DNN setting information of the up-scaled DNN, wherein the updated DNN setting information includes information about the weight residuals that have undergone the frequency transformation.

[0300] The DNN setting information updater 1212 may perform entropy encoding on the weight or bias updated for use in the current period unit based on the context model, but entropy encoding is not limited thereto. The DNN setting information updater 1212 may entropy encode the difference between the weight or bias updated for use in the current period unit and the weight or bias used in the immediately preceding period unit based on the context model. In this case, the context model may be based on information about the difference between the weight or bias determined to be used in the amplified DNN in at least one of the immediately preceding period unit or the period unit before the immediately preceding period unit.

[0301] The DNN setting information updater 1212 may perform entropy encoding on weight information or bias information in conjunction with a model compression technique (eg, pruning or quantization).

[0302] The DNN setting information updater 1212 may periodically update the DNN setting information of the amplified DNN. For example, the DNN setting information updater 1212 may update the DNN setting information for each GOP, each IRAP period (e.g., an instantaneous decoding refresh (IDR) picture period), each sequence, or each unit of a preset number of frames.

[0303] For example, the AI ​​encoding device 1200 may perform scene change detection while encoding an image in each IDR picture period, and when a scene change is detected in a frame, the frame is determined to be an IDR picture and subsequent images are encoded in each IDR picture period. In this case, the DNN setting information updater 1212 may update the DNN setting information for each IDR picture period in which a scene change is detected.

[0304] The DNN setting information updater 1212 may update the DNN setting information when necessary. For example, the DNN setting information updater 1212 may determine whether to update the DNN setting information for each preset period, and update the DNN setting information when the DNN setting information updater 1212 determines to update the DNN setting information. The DNN setting information updater 1212 may determine the DNN setting information of the enlarged DNN based on at least some of the at least one original image, and determine whether to update the DNN setting information by comparing the results of performing AI reduction and AI enlargement based on the determined DNN setting information with the results of performing AI reduction and AI enlargement based on the DNN setting information that has not been updated based on the RD cost.

[0305] In this case, the DNN setting information updater 1212 may encode flag information about whether to update the DNN setting information. When the DNN setting information updater 1212 determines to update the DNN setting information, the DNN setting information updater 1212 may encode the flag information as a first value. On the other hand, when the DNN setting information updater 1212 determines not to update the DNN setting information, the DNN setting information updater 1212 may encode the flag information as a second value.

[0306] When the DNN setting information updater 1212 determines not to update the DNN setting information, the flag information may not be encoded. The first value or the second value may be 0 or 1, but the value is not limited thereto.

[0307] The DNN setting information updater 1212 may determine whether to enlarge the second image by using predetermined DNN setting information of the AI-enlarged DNN or by using DNN setting information updated using the original image as the training image of the AI-enlarged DNN. In this document, the terms "AI-enlarged DNN" and "enlarged DNN" or "AI-reduced DNN" and "reduced DNN" may be used interchangeably. The DNN setting information updater 1212 may determine the DNN setting information based on at least some of the at least one original image, and determine whether to enlarge the second image based on the predetermined DNN setting information of the AI-enlarged DNN or the DNN setting information updated using the original image as the training image of the AI-enlarged DNN by comparing the results of performing AI reduction and AI enlargement based on the determined DNN setting information with the results of performing AI reduction and AI enlargement based on the predetermined DNN setting information based on the RD cost.

[0308] The DNN setting information updater 1212 may encode the flag information to indicate whether the second image is to be enlarged based on the predetermined DNN setting information of the AI-enlarged DNN or the DNN setting information updated using the original image as the training image based on the AI-enlarged DNN. When the flag information has a first value, the flag information may indicate that the second image is to be enlarged by using the predetermined setting information of the AI-enlarged DNN, and when the flag information has a second value, the flag information may indicate that the second image is to be enlarged by using the DNN setting information updated using the original image as the training image of the AI-enlarged DNN. When the second image is to be enlarged by using the predetermined setting information of the AI-enlarged DNN, the DNN setting information updater 1212 may not encode the flag information.

[0309] The DNN setting information updater 1212 may update the DNN setting information of the AI ​​amplification DNN based on at least one of a resolution of the original image or the first image or a bit rate of the image data.

[0310] For example, the DNN setting information updater 1212 may change the DNN structure by changing at least one of the number of convolutional layers, the number of filter kernels in each convolutional layer, or the size of each filter kernel based on at least one of the resolution of the original image or the first image or the bit rate of the image data. The AI ​​encoding device 1200 may update the DNN setting parameters by performing training of the enlarged DNN based on the changed DNN structure. For example, the DNN structure may be changed to have 3 convolutional layers, 3×3 filter kernels, and 4 channels, and the DNN setting information including the setting parameters of the AI ​​enlarged DNN may be updated based on the configured DNN structure. The DNN structure may also be changed to include 4 convolutional layers, 3×3 filter kernels, and 8 channels, and the setting parameters of the AI ​​enlarged DNN may be updated based on the changed DNN structure. In addition, the DNN structure may be configured to include 5 convolutional layers, 5×5 filter kernels, and 8 channels, and the setting parameters of the AI ​​enlarged DNN may be updated based on the changed DNN structure. When joint training of the upscaling DNN and the downscaling DNN is performed by considering the same bit rate of the image data and the same resolution of the original image and the first image, the image data to be encoded may change accordingly as the DNN structure changes. However, in the case of training the upscaling DNN separately, the image data may remain intact, wherein during the training of the upscaling DNN separately, the first image that has undergone downscaling is selected even when the DNN structure changes. In addition, in the case of training the upscaling DNN separately, when the DNN setting information is updated, the third image ( Figure 1 The quality of the third image is after performing AI upscaling by the AI ​​decoding device ( Fig.13 The image was obtained at 1300.

[0311] In addition, the DNN setting information updater 1212 may generate multiple pieces of DNN setting information for multiple DNN structures by changing the DNN structure in various ways, and may encode the multiple pieces of DNN setting information for the multiple DNN structures. In this case, the multiple DNN structures may be DNN structures corresponding to the same bit rate of the image data and the same resolution of the original image and the first image. However, the embodiments of the present disclosure are not limited thereto, and the multiple DNN structures may be DNN structures corresponding to various bit rates of the image data and various resolutions of the original image and the first image.

[0312] Although referenced Fig.12The DNN setting information updater 1212 is described as encoding the DNN setting information of the enlarged DNN, but the embodiments of the present disclosure are not limited thereto, and a person of ordinary skill in the art will understand that the DNN setting information updater 1212 may send the DNN setting information of the enlarged DNN to the data processor 1232, and the data processor 1232 may perform the above-mentioned encoding operation of the DNN setting information updater 1212.

[0313] The first encoder 1214 encodes the first image obtained by the AI ​​downsizer 1216 that performs downscaling.

[0314] As described above, encoding may include a process of generating prediction data by predicting a first image, a process of generating residual data corresponding to the difference between the first image and the prediction data, a process of transforming the residual data in the spatial domain into frequency domain components, a process of quantizing the residual data that has undergone the transformation into frequency domain components, and a process of entropy encoding the quantized residual data.

[0315] The data processor 1232 processes at least one of the AI ​​data or the image data to be sent in a predefined format. For example, when the AI ​​data and the image data need to be sent in the form of a bit stream, the data processor 1232 may process the AI ​​data so that the AI ​​data is represented in the form of a bit stream, and send the AI ​​data and the image data in the form of one bit stream via the communicator 1234. As another example, the data processor 1232 may process the AI ​​data so that the AI ​​data is represented in the form of a bit stream, and send the bit stream corresponding to the AI ​​data and the bit stream corresponding to the image data via the communicator 1234, respectively. As another example, the data processor 1232 may process the AI ​​data so that the AI ​​data is represented as a frame or a packet, and send the image data in the form of a bit stream and the AI ​​data in the form of a frame or a packet via the communicator 1234.

[0316] The communicator 1234 transmits the AI-encoded data generated as a result of the AI-encoding via the network. The AI-encoded data generated as a result of the AI-encoding includes image data and AI data.

[0317] Image data and AI data can be sent via homogeneous or heterogeneous networks.

[0318] The image data includes data generated as a result of a first encoding of a first image. The image data may include data generated based on pixel values ​​in the first image, such as residual data that is a difference between the first image and prediction data. In addition, the image data includes information used during the first encoding of the first image. For example, the image data may include mode information for performing the first encoding on the first image and information related to a quantization parameter.

[0319] The AI ​​data includes information that enables the AI ​​decoding device 1300 to perform AI upscaling on the second image according to an upscaling target corresponding to the downscaling target of the first DNN (downscaling DNN). For example, the AI ​​data may include difference information between the original image and the first image. In addition, the AI ​​data may include information related to the first image. The information related to the first image may include information about at least one of the resolution of the first image, the bit rate of the image data generated as a result of the first encoding of the first image, or the type of codec used in the first encoding of the first image.

[0320] Fig.13 is a block diagram of a configuration of an AI decoding device according to an embodiment of the present disclosure. Fig.13 , the AI ​​decoding device 1300 includes a receiver 1310 and an AI decoder 1330. The AI ​​decoder 1330 includes a first decoder 1332, an AI amplifier 1334, and a DNN setting information updater 1336.

[0321] The AI ​​decoding device 1300 may include a central processor that controls the receiver 1310 and the AI ​​decoder 1330. Optionally, the receiver 1310 and the AI ​​decoder 1330 are operated by respective processors, and when the processors work closely together, the AI ​​decoding device 1300 may be fully operated. Optionally, the receiver 1310 and the AI ​​decoder 1330 may also be controlled by an external processor.

[0322] The AI ​​decoding device 1300 may further include one or more data memories for storing data input to or output from the DNN setting information updater 1336, the AI ​​amplifier 1334, the first decoder 1332, and the receiver 1310. The AI ​​decoding device 1300 may further include a memory controller that controls input / output of data stored in the data memories.

[0323] In order to decode the image, the AI ​​decoding device 1300 can perform image decoding operations including prediction by interacting with a built-in or external video decoding processor. According to an embodiment of the present disclosure, the built-in video decoding processor of the AI ​​decoding device 1300 can implement basic image decoding operations together with a CPU or GPU including an image decoding processing module and a separate processor.

[0324] The communicator 1312 receives AI encoding data including image data and AI data via a network. The image data includes information generated as a result of first encoding of a first image, and the AI ​​data includes DNN setting information for enlarging the DNN.

[0325] The parser 1314 divides the AI ​​encoded data received via the communicator 1312 into image data and AI data, and transmits the image data to the first decoder 1332 and the AI ​​data to the AI ​​amplifier 1334 via the outputter 1316 .

[0326] because Fig.13 The operations of the communicator 1312, the parser 1314, and the outputter 1316 of the AI ​​decoding device 1300 are respectively similar to those of the reference Figure 2 The operations of the communicator 212 , the parser 214 , and the outputter 216 of the described AI decoding device 200 are corresponding, so a redundant description thereof is omitted.

[0327] The AI ​​amplifier 1334 obtains a third image (eg, Figure 1 145). The third image is an image having a higher resolution than that of the second image. Since the AI ​​amplification performed by the AI ​​amplifier 1334 has been described above, its redundant description is omitted.

[0328] The DNN setting information updater 1336 may be provided separately from the AI ​​amplifier 1334 , but the DNN setting information updater 1336 is not limited thereto and the DNN setting information updater 1336 may be included in the AI ​​amplifier 1334 .

[0329] The DNN setting information updater 1336 may update the DNN setting information of the AI ​​up-scaled DNN corresponding to the AI ​​down-scaled DNN based on the DNN setting information.

[0330] The DNN setting information updater 1336 may obtain AI data related to AI reduction of at least one original image to the first image. The DNN setting information updater 1336 may obtain DNN setting information for performing AI upscaling based on the AI ​​data. The DNN setting information updater 1336 may update the DNN setting information based on the DNN setting information obtained according to the AI ​​data.

[0331] In this case, the DNN setting information may be DNN information that is updated by using at least one original image as a training image, through joint training of an upscaling DNN and a downscaling DNN for AI downscaling of at least one original image, or separate training of the upscaling DNN for performing AI upscaling of at least one second image corresponding to the at least one original image.

[0332] Referenced above Fig.12It is described that the AI ​​encoding device 1200 updates DNN setting information for performing AI upscaling of at least one second image corresponding to at least one original image through joint training of an upscaling DNN and a downscaling DNN for AI downscaling of at least one original image, or separate training of the upscaling DNN, by using at least one original image as a training image. Because the AI ​​decoding device 1300 uses the parameters of the AI ​​upscaling DNN updated using the original image as the training image, rather than the parameters of the AI ​​upscaling DNN previously determined by using a separately provided image as the training image, the AI ​​decoding device 1300 can optimally perform AI upscaling on the second image.

[0333] The DNN setting information may include at least one of the number of convolutional layers, the number of filter kernels in each convolutional layer, or information about the size of each filter kernel.

[0334] In addition, the DNN setting information may include a weight of at least one filter kernel in at least one convolutional layer constituting the enlarged DNN. The embodiments of the present disclosure are not limited thereto, and the DNN setting information may include a bias of at least one output channel.

[0335] The DNN setting information updater 1336 may obtain a first flag based on the AI ​​data, and determine whether to enlarge the second image by using the predetermined DNN setting information of the AI-enlarged DNN or by using the DNN setting information updated using the original image as the training image based on the AI-enlarged DNN based on the first flag. The first flag may be a flag indicating that the second image will be enlarged by using the predetermined DNN setting information of the AI-enlarged DNN or by using the DNN setting information updated using the original image as the training image based on the AI-enlarged DNN. When the first flag has a first value, the first flag may indicate that the second image will be enlarged by using the predetermined DNN setting information of the AI-enlarged DNN or by using the DNN setting information updated using the original image as the training image based on the AI-enlarged DNN. Fig. 9 The trained AI amplification DNN uses predetermined DNN setting information to amplify the second image, wherein the predetermined DNN setting information is stored in the AI ​​decoding device 1300.

[0336] In this case, the DNN setting information updater 1336 may determine the DNN setting information of the AI ​​amplified DNN from a plurality of predetermined DNN setting information of a plurality of AI amplified DNNs, and determine the DNN setting information of the AI ​​amplified DNN as the DNN setting information for performing AI amplification on the second image.

[0337] When the first flag has the second value, the first flag may indicate that the second image is to be upscaled based on DNN setting information updated using the original image as a training image of the AI ​​upscaled DNN.

[0338] In this configuration, the DNN setting information updater 1336 may determine the DNN setting information updated using the original image as the training image of the AI ​​up-scaling DNN as the DNN setting information for performing AI up-scaling on the second image.

[0339] The first value or the second value may be 0 or 1, but the value is not limited thereto.

[0340] In this configuration, DNN setting information updated using the original image as a training image for AI enlarged DNN may be generated via model compression. The AI ​​encoding device 1200 may encode the DNN setting information of the enlarged DNN obtained by joint training of the reduced DNN and the enlarged DNN or separate training of the enlarged DNN using the original image as the training image, and the DNN setting information updater 1336 may obtain information about the encoded DNN setting information.

[0341] For example, the DNN setting information updater 1336 may obtain information about the structure of the enlarged DNN (e.g., the number of convolutional layers, the number of filter kernels in each convolutional layer, and information about the size of each filter kernel). The DNN setting information updater 1336 may obtain information about the number of convolutional layers in the enlarged DNN, the number of filter kernels in each convolutional layer, and the size of each filter kernel, and update the number of convolutional layers in the enlarged DNN, the number of filter kernels in each convolutional layer, and the size of each filter kernel based on the obtained information.

[0342] The DNN setting information updater 1336 may obtain multiple pieces of DNN setting information for multiple enlarged DNN structures, determine a DNN structure from the multiple enlarged DNN structures based on at least one of the bit rate of the image data or the resolution of the original image or the first image, and update the number of convolutional layers in the enlarged DNN, the number of filter cores in each convolutional layer, and the size of each filter core based on the DNN setting information corresponding to the determined DNN structure.

[0343] Optionally, the DNN setting information updater 1336 may obtain multiple pieces of DNN setting information for multiple DNN structures, determine a DNN structure from the multiple DNN structures based on at least one available DNN structure, and update the number of convolutional layers in the amplified DNN, the number of filter cores in each convolutional layer, and the size of each filter core based on the DNN setting information corresponding to the determined DNN structure. In this case, at least one available DNN structure may be determined based on the capabilities of the processor implementing the AI ​​amplifier 1334 in the AI ​​decoding device 1300. For example, the DNN setting information updater 1336 may obtain information about weight residuals or bias residuals in the amplified DNN. In this case, the weight residual or bias residual may represent the difference between the weight or bias used to perform amplification via the amplified DNN in the immediately preceding time period unit and the weight or bias used to perform amplification via the amplified DNN in the current time period unit. In this case, the encoded information about the weight residual or bias residual in the enlarged DNN is information encoded using various entropy coding techniques such as differential pulse code modulation (DPCM), run length coding (RLC), and Huffman coding. The DNN setting information updater 1336 can obtain the information about the weight residual or bias residual in the enlarged DNN by performing an inverse encoding operation.

[0344] The DNN setting information updater 1336 may update the weight or bias for performing amplification via the amplified DNN in the current time period unit by adding the weight residual or bias residual in the amplified DNN to the weight or bias for performing amplification via the amplified DNN in the immediately preceding time period unit. The embodiments of the present disclosure are not limited thereto, and the DNN setting information updater 1336 may perform entropy decoding on the encoded DNN setting information based on the context model. For example, the DNN setting information updater 1336 may generate a context model based on the DNN setting information of the amplified DNN for the previous time period unit, and perform entropy decoding on the encoded DNN setting information of the amplified DNN by using the generated context model.

[0345] Whenever the weight or bias of the filter kernel in the enlarged DNN for the current time period unit is decoded, the context model may be updated, and information about the next weight or bias of the filter kernel in the enlarged DNN for the current time period unit may be entropy decoded based on the updated context model. Examples of context-based entropy decoding may include range decoding, arithmetic decoding, and Huffman decoding, but entropy decoding is not limited thereto. The context model may be determined for the entire enlarged DNN, but the context model is not limited thereto, and the context model may be determined for each convolution layer, for at least one filter kernel corresponding to one output channel in the convolution layer, and for each filter kernel in the convolution layer.

[0346] The DNN setting information updater 1336 may decode the weight information or bias information of all layers for the enlarged DNN, and update the weights or biases in all layers based on the decoded weight information or bias information, but the DNN setting information updater 1336 is not limited thereto. The DNN setting information updater 1336 may decode the weight information or bias information of some layers, and update the weights or biases in the corresponding layers. Alternatively, the DNN setting information updater 1336 may decode the weight information or bias information of some filter cores in the layer, and update the weights or biases in the corresponding filter cores in the layer. For example, the DNN setting information updater 1336 may decode the weight information / bias information in the last layer of the enlarged DNN, and update the weights or biases in the last layer.

[0347] The DNN setting information updater 1336 may decode the weight residual information or the bias residual information, and then update the weights or biases of all or some filter kernels in all or some convolutional layers based on the decoded weight residual information or the bias residual information, wherein the weight residual information or the bias residual information indicates the difference between the weights or biases of all or some filter kernels in all or some convolutional layers in the enlarged DNN before the weights or biases are updated and the weights or biases of all or some filter kernels in all or some convolutional layers in the enlarged DNN after the weights or biases are updated. The DNN setting information updater 1336 may decode the weight residual information and obtain the weight residual by performing an inverse frequency transform on the decoded weight residual information.

[0348] Although the DNN setting information updater 1336 may perform entropy decoding on information about weights or biases in the enlarged DNN for performing AI enlargement on the second image based on the context model, the DNN setting information updater 1336 is not limited thereto. The DNN setting information updater 1336 may entropy decode the difference between the weights or biases updated for the current time period unit and the weights or biases used for enlargement in the immediately preceding time period unit based on the context model. In this case, the context model may be based on information about the difference between the weights or biases determined for enlarging at least one of the immediately preceding image or the image before the immediately preceding image. The encoded DNN setting information may be generated by entropy encoding the weight information or the bias information in combination with a model compression technique (e.g., pruning or quantization).

[0349] The DNN setting information updater 1336 may periodically update the DNN setting information of the enlarged DNN. For example, the DNN setting information updater 1336 may decode the encoded DNN setting information for each GOP, each IRAP period (e.g., each IDR picture period), each sequence, or each unit of a preset number of frames, and update the DNN setting information of the enlarged DNN based on the decoded DNN setting information.

[0350] The DNN setting information updater 1336 may update the DNN setting information when necessary. For example, the DNN setting information updater 1336 may decode the first flag information encoded for each preset period, determine whether to update the DNN setting information of the amplified information based on the decoded first flag information, and update the DNN setting information when the DNN setting information updater 1336 determines to update the DNN setting information. In this case, the encoded first flag information may be flag information about whether to update the DNN setting information of the amplified DNN.

[0351] When the first flag information has the first value, the DNN setting information updater 1336 may determine not to update the DNN setting information of the enlarged DNN.

[0352] When the first flag information has the second value, the DNN setting information updater 1336 may determine to update the DNN setting information of the enlarged DNN.

[0353] The first value or the second value may be 0 or 1, but the value is not limited thereto.

[0354] The DNN setting information updater 1336 may parse the first flag information for each preset period, and when the first flag information is not parsed for each preset period, determine not to update the DNN setting information of the amplified DNN.

[0355] The DNN setting information updater 1336 may decode the encoded second flag information, and determine whether to enlarge the second image by using the predetermined DNN setting information of the AI-enlarged DNN or by using the DNN setting information updated using the original image as the training image of the AI-enlarged DNN based on the decoded second flag information. In this case, the encoded second flag information may be flag information indicating whether to enlarge the second image based on the predetermined DNN setting information of the AI-enlarged DNN or based on the DNN setting information updated using the original image as the training image of the AI-enlarged DNN.

[0356] When the second flag information has the first value, the DNN setting information updater 1336 may determine that the second image is to be enlarged by using predetermined DNN setting information for enlarging the DNN.

[0357] When the second flag information has the second value, the DNN setting information updater 1336 may determine to upscale the second image by using the DNN setting information updated using the original image as a training image using the upscaled DNN.

[0358] The DNN setting information updater 1336 may parse the second flag information for each preset period, and when the second flag information is not parsed for each preset period, determine that the second image is to be enlarged by using predetermined DNN setting information of the enlargement DNN.

[0359] Although it has been described above that the DNN setting information updater 1336 decodes the encoded DNN setting information of the amplified DNN, the embodiments of the present disclosure are not limited to this, and a person of ordinary skill in the art will understand that the parser 1314 may perform the above-mentioned decoding operation of the DNN setting information updater 1336 and send the decoded DNN setting information to the DNN setting information updater 1336 via the outputter 1316.

[0360] The AI ​​amplifier 1334 may generate a third image by performing upscaling on the second image via an upscaling DNN operating according to the DNN setting information. The DNN setting information may be received from the DNN setting information updater 1336.

[0361] Fig.14a is a flowchart of an AI encoding method according to an embodiment of the present disclosure.

[0362] The AI ​​encoding apparatus 1200 generates upscaling DNN setting information for performing AI upscaling corresponding to AI downscaling, the upscaling DNN setting information being DNN information updated through joint or separate training using a DNN of an original image (operation S1405). Fig.16a , Fig.16b and Fig.17a A process of generating DNN setting information updated via DNN joint or individual training using an original image, performed by the AI ​​encoding device 1200, is described in detail.

[0363] The AI ​​encoding apparatus 1200 may obtain image data generated by performing first encoding on a first image, wherein the first image is obtained by performing AI reduction on an original image (operation S1410).

[0364] The AI ​​encoding apparatus 1200 transmits the image data and the AI ​​data including the up-scaling DNN setting information (operation S1415 ).

[0365] Fig.14b is a flowchart of an AI encoding method via DNN joint training based on original images according to an embodiment of the present disclosure.

[0366] The AI ​​encoding apparatus 1200 may obtain a first training image by performing AI downscaling on the original image through the downscaling DNN, and obtain a third training image by performing AI upscaling on the first training image through the upscaling DNN (operation S1420).

[0367] The AI ​​encoding apparatus 1200 may update DNN information through DNN joint training based on the original image, the first training image, and the third training image (operation S1425).

[0368] The AI ​​encoding device 1200 can obtain a first image by performing AI reduction on the original image through the reduction DNN based on the updated DNN information, and obtain a third training image by performing enlargement on the second training image through the enlargement DNN, wherein the second training image is generated by performing first encoding and first decoding on the first training image (operation S1430).

[0369] The AI ​​encoding apparatus 1200 may generate upscaling DNN setting information for performing AI upscaling corresponding to AI downscaling by updating DNN information through training of the upscaling DNN based on the original image and the third training image (operation S1435 ).

[0370] The AI ​​encoding apparatus 1200 may obtain image data generated by performing first encoding on a first image, wherein the first image is generated by performing reduction on an original image (operation S1440). In this case, AI reduction may be performed based on DNN information set through DNN joint training using the original image.

[0371] The AI ​​encoding apparatus 1200 may generate AI encoding data including the image data and the AI ​​data including the enlarged DNN setting information (operation S1445 ).

[0372] The following will refer to Fig.16a and Fig.16b The process of generating DNN setting information updated via DNN joint training using an original image, performed by the AI ​​encoding device 1200, is described in detail.

[0373] Fig.14c is a flowchart of an AI encoding method through separate training of a DNN based on an original image according to an embodiment of the present disclosure.

[0374] The AI ​​encoding apparatus 1200 may obtain a first training image by performing AI downscaling on an original image through a downscaling DNN, and obtain a third training image by performing AI upscaling on a second image generated by performing first encoding and first decoding on the first training image through an upscaling DNN (operation 1450).

[0375] The AI ​​encoding apparatus 1200 may update DNN information through separate training of the up-scaled DNN based on the original image and the third training image, and generate up-scaled DNN setting information for performing AI up-scale corresponding to AI down-scale (operation S1455).

[0376] The AI ​​encoding apparatus 1200 may obtain image data generated by performing first encoding on a first image, wherein the first image is generated by performing reduction on an original image (operation S1460).

[0377] The AI ​​encoding apparatus 1200 may generate AI encoding data including the image data and the AI ​​data including the up-scaling DNN setting information (operation S1465 ).

[0378] The following will refer to Fig.17a A process of generating DNN setting information updated via DNN separate training using an original image, performed by the AI ​​encoding apparatus 1200, is described in detail.

[0379] Fig.15 is a flowchart of an AI decoding method according to an embodiment of the present disclosure.

[0380] The AI ​​decoding apparatus 1300 obtains image data and AI data (operation S1510 ).

[0381] The AI ​​decoding apparatus 1300 obtains a second image based on the image data (operation S1520). The AI ​​decoding apparatus 1300 may obtain the second image by performing the first decoding on the image data.

[0382] The AI ​​decoding apparatus 1300 obtains upscaling DNN setting information for performing AI upscaling, which is DNN information updated through joint or separate training using a DNN of an original image, from the AI ​​data (operation S1530 ).

[0383] The following will refer to Fig.16c and Fig.17b A process of obtaining upscaling DNN setting information for performing AI upscaling from AI data, which is DNN information updated via joint or separate training of a DNN using an original image, performed by the AI ​​decoding apparatus 1300 is described in detail.

[0384] The AI ​​decoding apparatus 1300 obtains a third image by performing up-scaling on the second image based on the up-scaling DNN setting information (operation S1540 ).

[0385] Fig.16aThis is a diagram for describing a process of determining multiple optimal DNN setting information of a reduced DNN and an enlarged DNN through joint training of the reduced DNN and the enlarged DNN using an original image as a training image, performed by the AI ​​encoding device 1200 as a first stage of joint training according to another embodiment of the present disclosure.

[0386] Reference Fig.16a , the AI ​​encoding device 1200 may generate a first training image 1602 as a result of performing AI reduction based on the first DNN 700 by using the original image 1601 as a training image. In this case, the original image 1601 may be a frame in a current GOP unit, a current IRAP time period unit, a current sequence unit, or a current unit of a predetermined number of frames to be encoded as one or more frames, but the original image 1601 is not limited thereto. In addition, the initial DNN setting information of the first DNN 700 and the initial DNN setting information of the second DNN 300 may be used to reduce and enlarge the image in the immediately preceding time period unit, respectively. Optionally, the initial DNN setting information of the first DNN 700 or the second DNN 300 may be one of a plurality of predetermined DNN setting information or DNN setting information set as default.

[0387] However, the embodiments of the present disclosure are not limited thereto, and the initial weights in the first DNN 700 and the second DNN 300 may be values ​​obtained using random sampling following a specific probability distribution. The initial biases in the first DNN 700 and the second DNN 300 may be determined to be zero.

[0388] The AI ​​encoding apparatus 1200 may generate the second loss information 1620 based on the complexity of the first training image 1602. In this case, the second loss information 1620 may be the same as that described above with reference to Fig.12 The second loss information described corresponds.

[0389] The AI ​​encoding apparatus 1200 may generate a third training image 1604 as a result of performing AI up-scaling based on the second DNN 300 by using the first training image 1602 .

[0390] The AI ​​encoding apparatus 1200 may generate first loss information 1610 and third loss information 1630, each of which corresponds to a result of comparing information related to the original image 1601 and information related to the third training image 1604. In this case, the first loss information 1610 and the third loss information 1630 may be respectively the same as those described above with reference to Fig.12 The described first loss information and third loss information correspond.

[0391] The AI ​​encoding apparatus 1200 may determine a loss value Loss based on the first loss information 1610, the second loss information 1620, and the third loss information 1630 according to the following Equation 6: DS .

[0392] [Equation 6]

[0393] Loss DS = a×first loss information+b×second loss information+c×third loss information

[0394] The AI ​​encoding device 1200 may be based on the loss value Loss DS A plurality of pieces of DNN setting information of the first DNN 700 and the second DNN 300 are updated.

[0395] The AI ​​encoding device 1200 may perform joint training of the first DNN 700 and the second DNN 300 by iteratively repeating the above process, and DS The joint training of the first DNN 700 and the second DNN 300 is used to determine the optimal DNN setting information.

[0396] Fig.16b is used to describe the AI ​​encoding device 1200 selecting the second stage of joint training according to an embodiment of the present disclosure according to Fig.16a The illustrated process is a diagram of a process of determining DNN setting information of a reduced DNN, determining optimal DNN setting information of an enlarged DNN through separate training of the enlarged DNN, and transmitting the optimal DNN setting information of the enlarged DNN through a bitstream.

[0397] Reference Fig.16b , the AI ​​encoding device 1200 selects the above reference Fig.16a The optimal DNN setting information determined by the joint training of the first DNN 700 and the second DNN 300. In this case, the initial DNN setting information of the second DNN 300 may be determined based on the optimal DNN setting information of the second DNN 300 determined by the joint training of the first DNN 700 and the second DNN 300, but the initial DNN setting information is not limited thereto. The initial weight in the second DNN 300 may be a value randomly sampled based on a specific probability distribution, and the initial bias in the second DNN 300 may be 0.

[0398] The AI ​​encoding apparatus 1200 may generate a first training image 1602 as a result of performing AI reduction based on the first DNN 700 by using the original image 1601 as a training image. The AI ​​encoding apparatus 1200 may perform first encoding 1603 and first decoding 1605 on the first training image 1602. The AI ​​encoding apparatus 1200 may generate a third training image 1604 by performing AI up-scaling on the image generated via the first decoding 1605 based on the second DNN 300. The AI ​​encoding apparatus 1200 may generate first loss information 1610 and third loss information 1630, each corresponding to a result of comparing image information related to the original image 1601 and image information related to the third training image 1604.

[0399] The AI ​​encoding apparatus 1200 may determine a loss value Loss based on the first loss information 1610 and the third loss information 1630 according to the following Equation 7: US .

[0400] [Equation 7]

[0401] Loss US = d×first loss information+e×third loss information

[0402] The AI ​​encoding device 1200 may be based on the loss value Loss US Update the DNN setting information of the second DNN 300. In this case, the DNN setting information of the second DNN 300 may be updated via model compression.

[0403] The AI ​​encoding device 1200 may perform separate training of the second DNN 300 by repeating the above-mentioned process, and US The optimal DNN setting information for the second DNN 300 is determined through separate training.

[0404] The AI ​​encoding device 1200 may encode the determined optimal DNN setting information of the second DNN 300, and the transmitter 1230 of the AI ​​encoding device 1200 may generate AI encoded data including the encoded DNN setting information and the image data generated via the first encoding 1603. In this case, the DNN setting information may include weight residual information or bias residual information of the second DNN 300. In addition, the AI ​​encoded data may be represented in the form of a bitstream.

[0405] Fig.16c is a diagram for describing a process in which the AI ​​decoding apparatus 1300 performs AI up-scaling on a second image via an up-scaling DNN based on DNN setting information of the up-scaling DNN included in AI encoded data according to an embodiment of the present disclosure.

[0406] As mentioned above Fig.16b As described above, the AI ​​encoding device 1200 can generate AI encoding data.

[0407] Reference Fig.16c , the receiver 1310 of the AI ​​decoding device 1300 receives the AI ​​encoded data.

[0408] As described above, the AI ​​encoded data may be represented as a bitstream. The receiver 1310 of the AI ​​decoding device 1300 obtains image data and DNN setting information from the AI ​​encoded data. The DNN setting information updater 1336 of the AI ​​decoding device 1300 may configure the second DNN 300 based on the DNN setting information. The AI ​​decoding device 1300 may obtain the image data from the AI ​​encoded data, and generate the second image 1670 by performing the first decoding 1660 on the image data.

[0409] The AI ​​decoding device 1300 may generate a third image 1680 by enlarging the second image 1670 via the second DNN 300 based on the DNN setting information. In this case, since the DNN setting information is as described above with reference to Fig.16a and 16b The DNN setting information of the second DNN 300 optimized using the original image as the training image, so the quality of the third image 1680 can be improved compared with the quality of the third image generated by enlarging the second image based on the DNN setting information predetermined by the second DNN based on the training image separately provided by the AI ​​encoding device 1200 and the AI ​​decoding device 1300.

[0410] Fig.17a It is a diagram for describing the process of an AI encoding device selecting DNN setting information of a reduced DNN, determining optimal DNN setting information of an enlarged DNN through separate training of an enlarged DNN, and sending the optimal DNN setting information of the enlarged DNN in a bitstream according to an embodiment of the present disclosure.

[0411] Reference Fig.17a , the AI ​​encoding device 1200 selects the weight of the first DNN 700. In this case, the selected DNN setting information of the first DNN 700 may be setting information for reducing the original image in the immediately preceding time period unit. However, the embodiments of the present disclosure are not limited thereto, and the DNN setting information of the first DNN 700 may be one of a plurality of predetermined DNN setting information or DNN setting information set as default.

[0412] In addition, the initial DNN setting information of the second DNN 300 may be DNN setting information for enlarging the original image in the immediately preceding unit. However, the embodiments of the present disclosure are not limited thereto, and the initial weight in the second DNN 300 may be a value randomly sampled based on a specific probability distribution, and the initial bias in the second DNN 300 may be 0.

[0413] The AI ​​encoding apparatus 1200 may generate a first training image 1702 as a result of performing AI reduction based on the first DNN 700 by using the original image 1701 as a training image. The AI ​​encoding apparatus 1200 may perform first encoding 1703 and first decoding 1705 on the first training image 1702. The AI ​​encoding apparatus 1200 may generate a third training image 1704 by performing AI up-scaling on the image generated via the first decoding 1705 based on the second DNN 300. The AI ​​encoding apparatus 1200 may generate first loss information 1710 and third loss information 1730, each corresponding to a result of comparing image information related to the original image 1701 and image information related to the third training image 1704.

[0414] The AI ​​encoding device 1200 may determine a loss value Loss based on the first loss information 1710 and the third loss information 1730 according to the following Equation 8: US .

[0415] [Equation 8]

[0416] Loss US = d×first loss information+e×third loss information

[0417] The AI ​​encoding device 1200 may be based on the loss value Loss US Update the DNN setting information of the second DNN 300. In this case, the DNN setting information of the second DNN 300 may be updated via model compression.

[0418] The AI ​​encoding device 1200 may perform separate training of the second DNN 300 by repeating the above-mentioned process, and US The optimal DNN setting information for the second DNN 300 is determined through separate training.

[0419] The AI ​​encoding device 1200 may encode the determined optimal DNN setting information of the second DNN 300, and the transmitter 1230 of the AI ​​encoding device 1200 may generate AI encoded data including the encoded DNN setting information and the image data generated via the first encoding 1703. In this case, the DNN setting information may include weight residual information or bias residual information of the second DNN 300.

[0420] Fig.17b is a diagram for describing a process of performing AI upscaling on a second image via an upscaling DNN based on DNN setting information of an upscaling DNN included in AI encoded data, performed by the AI ​​decoding apparatus 1300 according to an embodiment of the present disclosure.

[0421] As mentioned above Fig.17a As described above, the AI ​​encoding device 1200 can generate AI encoding data.

[0422] Reference Fig.17b , the receiver 1310 of the AI ​​decoding device 1300 receives the AI ​​encoded data. As described above, the AI ​​encoded data may be represented as a bitstream. The receiver 1310 of the AI ​​decoding device 1300 obtains the image data and the DNN setting information from the AI ​​encoded data. The DNN setting information updater 1336 of the AI ​​decoding device 1300 may determine the second DNN 300 based on the DNN setting information.

[0423] The AI ​​decoding device 1300 may generate a second image 1770 by performing the first decoding 1760 on the image data. The AI ​​decoding device 1300 may generate a third image 1780 by enlarging the second image 1770 via the second DNN 300 based on the DNN setting information. In this case, since the DNN setting information is as described above with reference to Fig.17a The DNN setting information of the second DNN 300 is optimized using the original image as a training image, so the quality of the third image 1780 can be improved compared to the quality of the third image generated by enlarging the second image based on the DNN setting information predetermined by the second DNN based on a separately provided training image.

[0424] Fig.18 It is a flowchart of a process of enlarging a second image by updating DNN setting information of an enlarged DNN predetermined based on a flag obtained from AI encoded data, or by updating DNN setting information of an enlarged DNN optimized for an original image, performed by the AI ​​decoding device 1300 according to an embodiment of the present disclosure.

[0425] Reference Fig.18, the AI ​​decoding apparatus 1300 may determine, based on the value of the first flag, whether the first flag indicates whether to use predetermined DNN setting information of the amplified DNN (operation S1805).

[0426] When the first flag indicates that the predetermined DNN setting information of the enlarged DNN is to be used, the AI ​​decoding apparatus 1300 may enlarge the image by using the predetermined DNN setting information of the enlarged DNN (operation S1810). In this case, the predetermined DNN setting information of the enlarged DNN may be stored in the AI ​​decoding apparatus 1300.

[0427] The AI ​​decoding apparatus 1300 may periodically obtain the second flag, and determine whether the second flag indicates whether to use DNN setting information of the enlarged DNN other than the previous DNN setting information among a plurality of predetermined DNN setting information of the enlarged DNN based on the value of the second flag (operation S1815).

[0428] When the second flag indicates that DNN setting information of an enlarged DNN other than the previous DNN setting information among multiple predetermined DNN setting information of the enlarged DNN will be used, the AI ​​decoding device 1300 may change the DNN setting information used for the enlarged DNN to DNN setting information of the enlarged DNN other than the DNN setting information used in the previous period among the multiple predetermined DNN setting information (operation S1820).

[0429] When the second flag indicates that DNN setting information of the enlarged DNN other than the previous DNN setting information among multiple predetermined DNN setting information of the enlarged DNN will not be used, the AI ​​decoding device 1300 may determine that the DNN setting information used in the previous period is maintained as the DNN setting information to be used for the enlarged DNN (operation S1825).

[0430] The AI ​​decoding device 1300 may enlarge the image by using predetermined DNN setting information of the enlarged DNN (operation S1830). In this case, the predetermined DNN setting information of the enlarged DNN may be the DNN setting information changed in operation S1820 or the DNN setting information maintained in operation S1825.

[0431] When the first flag indicates that the predetermined DNN setting information of the enlarged DNN will not be used, the AI ​​decoding device 1300 may enlarge the image by using the DNN setting information of the enlarged DNN optimized for the original image (operation S1835). Figures 16a to 17b A description of DNN setting information optimized for original images regarding the up-scaling DNN is provided, and therefore, a redundant description is not repeated below.

[0432] The AI ​​decoding apparatus 1300 may periodically obtain the third flag, and determine whether the third flag indicates whether to update the DNN setting information with the DNN setting information optimized for the current period of the amplified DNN based on the value of the third flag (operation S1840).

[0433] When the third flag indicates that the DNN setting information is to be updated, the AI ​​decoding apparatus 1300 may change the DNN setting information to be used for the up-scaled DNN to the DNN setting information of the up-scaled DNN optimized for the current period of the original image (operation S1845).

[0434] When the third flag indicates that the DNN setting information will not be updated, the AI ​​decoding device 1300 may determine to maintain the DNN setting information of the enlarged DNN used in the previous period (operation S1850). The AI ​​decoding device 1300 may enlarge the image by using the DNN setting information optimized for the original image of the enlarged DNN (operation S1855). In this case, the DNN setting information optimized for the original image may be the DNN setting information changed in S1845 or the DNN setting information maintained in S1850.

[0435] Fig.19a Examples of default weights and biases, weights and biases optimized for original images in an enlarged DNN, and weight differences and bias differences in an enlarged DNN according to an embodiment of the present disclosure are shown.

[0436] Reference Fig.19a , the AI ​​encoding device 1200 may determine a default weight matrix 1910 of a size of 3×3 and a default bias matrix 1920 of a size of 1×1 for a filter kernel in a convolution layer of the enlarged DNN. In this case, the default weight matrix 1910 and the default bias matrix 1920 may be weights and biases pre-determined by using a separate training image in the AI ​​encoding device 1200, but the weights and biases are not limited thereto. The weights and biases may be weights and biases determined in an immediately preceding period unit by using an original image as a training image.

[0437] According to Equation 9 below, the input matrix can be used , default weight matrix (e.g., 1910) and the default bias matrix (e.g., 1920) to generate the output matrix In Equation 9, activation() and convolution() represent the activation function and convolution operation function respectively. L , and Represent the convolutional layer and the layer L The input channels (depth) and layersL output channels, and t Indicates time.

[0438] [Equation 9]

[0439]

[0440] Reference Fig.19a , the AI ​​encoding device 1200 may determine an optimal weight matrix 1930 having a size of 3×3 and an optimal bias matrix 1940 having a size of 1×1 by using the current original image as a training image. The values ​​of the elements in the optimal weight matrix 1930 and the optimal bias matrix 1940 may be different from the values ​​of the elements of the default weight matrix 1910 and the default bias matrix 1920, respectively.

[0441] The AI ​​encoding device 1200 may entropy encode the optimal weight matrix 1930 and the optimal bias matrix 1940 based on the context model for transmission.

[0442] The AI ​​encoding device 1200 may determine a weight difference matrix 1950 having a size of 3×3 and a bias difference matrix 1960 having a size of 1×1 based on the default weight matrix 1910 , the default bias matrix 1920 , the optimal weight matrix 1930 , and the optimal bias matrix 1940 .

[0443] The value of an element in each weight difference matrix 1950 may be the difference between the value of an element in a corresponding one of the default weight matrices 1910 and the value of an element in the corresponding optimal weight matrix 1930, and the value of an element in each bias difference matrix 1960 may be the difference between the value of an element in a corresponding one of the default bias matrices 1920 and the value of an element in the corresponding optimal bias matrix 1940.

[0444] The AI ​​encoding apparatus 1200 may entropy encode the weight difference matrix 1950 and the bias difference matrix 1960 for transmission.

[0445] Fig.19b An example of weights and biases in an upscaled DNN optimized for an original image and weights and biases in an upscaled DNN determined via quantization and pruning according to an embodiment of the present disclosure is shown.

[0446] Reference Fig.19b , the AI ​​encoding device 1200 can determine an optimal weight matrix 1970 of a size of 3×3 and an optimal bias matrix 1975 of a size of 1×1 by using the current original image as a training image.

[0447] The AI ​​encoding apparatus 1200 may determine a weight matrix 1980 having a size of 3×3 by performing pruning on the optimal weight matrix 1970. In other words, referring to Fig.19b , the AI ​​encoding device 1200 may determine the weight matrix 1980 by changing all values ​​of elements whose absolute values ​​are less than 0.02 in the optimal weight matrix 1970 to zero. However, the absolute value for changing the value of the element to zero is not limited to 0.02, and a person of ordinary skill in the art will understand that the value of the element less than a value close to 0 may be changed to 0.

[0448] The AI ​​encoding apparatus 1200 may determine a weight matrix 1990 having a size of 3×3 and a bias matrix 1995 having a size of 1×1 by performing quantization on the weight matrix 1980 and the optimal bias matrix 1975, respectively. Fig.19b , the AI ​​encoding device 1200 may determine the weight matrix 1990 by performing a multiplication operation of the values ​​of the elements in the weight matrix 1980 by 128 and a rounding operation on the element product, and determine the bias matrix 1995 by performing a rounding operation on the values ​​of the elements in the optimal bias matrix 1975. However, the embodiments of the present disclosure are not limited thereto, and it will be understood by those of ordinary skill in the art that the quantization of various embodiments of the present disclosure may be performed on the weight / bias matrix.

[0449] Fig.20a is a diagram for describing a method of encoding weights optimized for an original image in an up-scaled DNN according to an embodiment of the present disclosure.

[0450] Reference Fig.20a , the AI ​​encoding device 1200 may determine the weight difference matrix 2010 of the 4×4 filter kernel. In this case, the weight difference values ​​in the weight difference matrix 2010 are weight differences optimized for the original image in the magnified DNN, as described above with reference to Fig.19a As described above, a detailed description thereof will be omitted below.

[0451] The AI ​​encoding apparatus 1200 may determine a weight matrix 2015 by performing pruning on a weight difference matrix 2010 of a 4×4 filter kernel.

[0452] The AI ​​encoding device 1200 may determine the weight matrix 220 by performing quantization on the weight matrix 215. Fig.19b Descriptions on pruning and quantization are provided and thus will not be repeated.

[0453] Then, the AI ​​encoding apparatus 1200 may generate AI data 2040 by performing RLC 2030 on the weights in the weight matrix 2020. In this case, the AI ​​data 2040 may be represented in the form of a bitstream.

[0454] Fig.20b is a diagram for describing a method of encoding weights optimized for an original image in an up-scaled DNN according to another embodiment of the present disclosure.

[0455] Reference Fig.20b , the AI ​​encoding device 1200 may determine the weight difference matrix 2055 of the 4×4 filter kernel. In this case, the weight difference values ​​in the weight difference matrix 2055 are weight differences optimized for the original image in the magnified DNN, as described above with reference to Fig.19a As described above, and its redundant description is omitted below.

[0456] The AI ​​encoding apparatus 1200 may determine a weight matrix 2060 by performing a transform (eg, a discrete cosine transform) on the weight difference matrix 2055 of the 4×4 filter kernel.

[0457] The AI ​​encoding apparatus 1200 may determine the weight matrix 2065 by performing pruning on the weight matrix 2060 .

[0458] The AI ​​encoding apparatus 1200 may determine the weight matrix 2070 by performing quantization on the weight matrix 2065 .

[0459] The AI ​​encoding apparatus 1200 may generate AI data 2090 by performing RLC 2080 on the weights of the weight matrix 2070 .

[0460] The AI ​​decoding device 1300 may perform run length decoding on the AI ​​data 2090 to generate a two-dimensional (2D) weight matrix, and restore the weight difference matrix through inverse quantization and inverse transformation. In this case, the restored weight difference matrix may be different from the weight difference matrix 2055. The mismatch is caused by data loss due to pruning and quantization.

[0461] according to Fig.20b The AI ​​encoding device 1200 may transform the weight difference matrix 2055 so that the data values ​​may be concentrated at low frequencies and the data values ​​at high frequencies may be closer to zero. Fig.20a Compared with the described method, unnecessary information may be discarded more effectively during subsequent pruning and quantization. The AI ​​decoding apparatus 1300 may perform an inverse transform corresponding to the transform performed by the AI ​​encoding apparatus 1200 .

[0462] In this case, the AI ​​decoding apparatus 1300 may generate a 2D weight matrix by performing run-length decoding on the AI ​​data 2090, and restore a weight difference matrix by performing inverse quantization and inverse transformation on the 2D weight matrix.

[0463] Fig.21ais a diagram for describing a process of entropy encoding weight information optimized for an original image of an up-scaled DNN based on a context model, performed by the AI ​​encoding device 1200 according to an embodiment of the present disclosure, and Figure 21b is a diagram for describing a process of entropy decoding weight information of an enlarged DNN optimized for an original image based on a context model, performed by the AI ​​decoding device 1300 according to an embodiment of the present disclosure.

[0464] Reference Fig.21a In the AI ​​encoding device 1200, the context model generator 2110 may be based on the weight w t-1 (which is the weight of the image at the previous time t-1) to generate the context model M t-1 , and the entropy encoder 2120 can be based on the context model M t-1 For weight w t (which is the weight of the image at the current time t) is entropy encoded to generate weight information. In this case, entropy encoding may be performed in combination with model compression techniques such as pruning and quantization. In this case, the image at the previous time t-1 may represent a frame immediately before the current frame at time t, but the time is not limited thereto, and the time may represent a time period unit immediately before the current time period unit including the frame at time t. In this case, the time period unit may be one of the following: a preset number of frames, a frame in a GOP, a frame in a sequence, and a frame in an IRAP period, but is not limited thereto.

[0465] Reference Figure 21b In the AI ​​decoding device 1300, the context model generator 2130 may generate a context model based on the weight w in the same manner as in the AI ​​encoding device 1200. t-1 (Weight w t-1 is the restored weight of the image at the previous time t-1) to generate the context model M t-1 , and the entropy decoder 2140 can be decoded by t-1 The weight information received from the AI ​​encoding device 1200 is entropy decoded to obtain the weight w t , where the weight w t is the weight of the image at the current time t. In this case, when entropy coding is performed in combination with a model compression technique (such as pruning and quantization), the AI ​​decoding device 1300 may perform an operation corresponding to the model compression technique in combination with the entropy decoding operation. For example, when entropy coding is performed in combination with a quantization operation, the AI ​​decoding device 1300 may perform an inverse quantization operation in combination with the entropy decoding operation.

[0466] In addition, the image data and DNN setting information generated by the AI ​​encoding device 1200 and corresponding to at least one quality (for example, a quality based on at least one of resolution or bit rate) are provided to the AI ​​decoding device 1300 through a streaming system. The streaming system is a system including at least one server (for example, a service server or a content server) and a terminal, and refers to a system in which the server provides image-related data corresponding to a request according to a request from the terminal via a network. In this case, the generated image data and DNN setting information may be stored in a content server separate from the service server, and the terminal may receive location information of image data of at least one quality from the service server according to the request, and then receive the image data and DNN setting information from the content server corresponding to the location information of the image data of at least one quality. In this case, the content server is a server for storing image data and DNN setting information, and may be the AI ​​encoding device 1200, but the content server is not limited thereto. The content server may be separated from the AI ​​encoding device 1200, and in this case, the content server may receive the image data and DNN setting information from the AI ​​encoding device 1200 for storage.

[0467] Meanwhile, the above-described embodiments of the present disclosure may be written as computer-executable programs or instructions that may be stored in a medium.

[0468] The medium may store computer executable programs or instructions permanently, or store computer executable programs or instructions for execution or download. In addition, the medium may be any of a variety of recording media or storage media in which single or multiple pieces of hardware are combined, and the medium is not limited to media directly connected to the computer system, but may be distributed over a network. Examples of media include magnetic media (such as hard disks, floppy disks, and tapes) configured to store program instructions, optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as floppy disks), and ROM, RAM, and flash memory. Other examples of media include recording media and storage media managed by application stores that publish applications or by websites, servers, etc. that provide or publish various other types of software.

[0469] Meanwhile, the model related to the above-mentioned DNN may be implemented via software. When the DNN model is implemented via software (eg, a program module including instructions), the DNN model may be stored in a computer-readable recording medium.

[0470] In addition, the DNN model can be integrated in the form of a hardware chip to become part of the above-mentioned AI decoding device 200 or AI encoding device 600. For example, the DNN model can be manufactured in the form of a dedicated hardware chip for AI, or can be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU).

[0471] In addition, the DNN model can be provided in the form of downloadable software. The computer program product may include a product in the form of a software program electronically distributed by a manufacturer or an electronic market (e.g., a downloadable application). For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of the manufacturer or the electronic market, or a storage medium of a relay server.

[0472] According to an embodiment of the present disclosure, a method and apparatus for performing AI encoding and AI decoding of an image can process an image at a low bit rate via AI-based image encoding and decoding.

[0473] In addition, according to an embodiment of the present disclosure, a method and apparatus for performing AI encoding and AI decoding of an image can improve image quality by performing upscaling after updating upscaling DNN setting information optimized for an original image periodically or when necessary.

[0474] In addition, according to an embodiment of the present disclosure, a method and apparatus for performing AI encoding and AI decoding of an image can effectively reduce the amount of information to be encoded and decoded by effectively signaling DNN setting information periodically or when necessary to update the DNN setting information optimized for the original image of the amplified DNN.

[0475] However, a person of ordinary skill in the art will understand that the effects achievable by the method and apparatus for performing AI encoding and AI decoding of an image according to an embodiment of the present disclosure are not limited to those described above, and other effects of the present disclosure not described herein will be more clearly understood from the following description.

[0476] Although one or more embodiments of the present disclosure have been described with reference to the drawings, people skilled in the art will understand that various changes in form and details may be made therein without departing from the spirit and scope defined by the following claims.

Claims

1. An artificial intelligence (AI) decoding device, comprising: a memory storing one or more instructions; as well as A processor configured to execute the one or more instructions stored in the memory to perform the following operations: obtaining image data generated by first encoding a first image; Obtaining AI data related to AI downscaling from the original image to the first image by downscaling the deep neural network (DNN); Obtaining a second image corresponding to the first image by performing a first decoding on the image data; updating predetermined DNN setting information for AI amplification based on the AI ​​data; as well as generating a third image by performing AI upscaling on the second image via an upscaling DNN operating according to the updated DNN setting information, The predetermined DNN setting information is obtained through joint training of an enlarged DNN and a reduced DNN, and Wherein, the updated DNN setting information is obtained through separate training of an amplified DNN operated according to the predetermined DNN setting information.

2. The AI ​​decoding device according to claim 1, wherein: The AI ​​data includes weights and biases of filter kernels used to update the predetermined DNN setting information.

3. The AI ​​decoding device according to claim 1, wherein: The predetermined DNN setting information is obtained by the following operations: generating a first training image by using the original image through a downscaling DNN, generating a second training image by upscaling the DNN using the first training image, and updating the enlarged DNN and the reduced DNN based on the first loss information, the second loss information, and the third loss information, The first loss information and the third loss information correspond to the result of comparing the second training image with the original image that has not undergone AI reduction, and the second loss information is generated based on the first training image.

4. The AI ​​decoding device according to claim 3, wherein: The first loss information is generated based on a result of comparing a quality parameter of the second training image with a quality parameter of the original image.

5. The AI ​​decoding device according to claim 3, wherein: The third loss information is generated based on a result of comparing the feature-related parameters of the second training image with the feature-related parameters of the original image.

6. The AI ​​decoding device according to claim 3, wherein: The second loss information is related to the spatial complexity of the first training image.

7. The AI ​​decoding device according to claim 1, wherein: The updated DNN setting information is obtained by the following operations: generating a first training image by using the original image through a downscaling DNN, performing a first encoding on a first training image, generating a second training image by upscaling the DNN using the first training image that has undergone the first encoding, and Based on the first loss information and the third loss information, the DNN is updated and amplified. The first loss information and the third loss information correspond to the result of comparing the second training image with the original image that has not undergone AI reduction.

8. The AI ​​decoding device according to claim 1, wherein: The AI ​​data includes weight residual information and bias residual information, wherein the weight residual information and the bias residual information indicate a difference between the weights and biases of the predetermined DNN setting information and the weights and biases of the updated DNN setting information.

9. The AI ​​decoding device according to claim 8, wherein: The weight residual information and the bias residual information are information encoded using one of differential pulse code modulation DPCM, run length coding RLC, and Huffman coding schemes.

10. The AI ​​decoding device according to claim 8, wherein: The weight residual information and the bias residual information are information about the weight residual and the bias residual generated through model compression.

11. The AI ​​decoding device according to claim 8, wherein: The weight residual information and the bias residual information are obtained by performing frequency transformation on the weight residual and the bias residual.

12. The AI ​​decoding device according to claim 1, wherein: The AI ​​data includes information obtained by entropy encoding weights and biases of the updated DNN setting information based on context model information about the weights and biases of the predetermined DNN setting information.

13. The AI ​​decoding device according to claim 1, wherein: The AI ​​data includes flag information indicating whether to perform AI upscaling by using the predetermined DNN setting information or to perform AI upscaling by using the updated DNN setting information.

14. A device for performing artificial intelligence (AI) encoding of an image, the device comprising: a memory storing one or more instructions; as well as A processor configured to execute the one or more instructions stored in the memory to perform the following operations: Obtaining a first image by performing AI downscaling on the original image via a downscaling deep neural network (DNN); generating image data by performing first encoding on a first image; and generating AI encoded data including the image data and AI data, wherein the AI ​​data includes information related to AI reduction, wherein the AI ​​data includes information for updating predetermined DNN setting information stored in the AI ​​decoding device, The predetermined DNN setting information is obtained through joint training of an enlarged DNN and a reduced DNN, and Wherein, the updated DNN setting information is obtained through separate training of an amplified DNN operated according to the predetermined DNN setting information.

15. A method for performing artificial intelligence (AI) decoding on an image, the method comprising: obtaining image data generated by first encoding a first image; Obtaining AI data related to AI downscaling from the original image to the first image by downscaling the deep neural network (DNN); Obtaining a second image corresponding to the first image by performing a first decoding on the image data; updating predetermined DNN setting information for AI amplification based on the AI ​​data; as well as generating a third image by performing AI upscaling on the second image via an upscaling DNN operating according to the updated DNN setting information, The predetermined DNN setting information is obtained through joint training of an enlarged DNN and a reduced DNN, and Wherein, the updated DNN setting information is obtained through separate training of an amplified DNN operated according to the predetermined DNN setting information.

Images