Image encoding method and device, and image decoding method and device

Abstract

A method and apparatus for image encoding and image decoding are provided. The image decoding method includes: obtaining DNN update permission information indicating whether to update one or more deep neural network (DNN) setting information; obtaining DNN update information necessary to determine the one or more DNN setting information to be updated based on the DNN update permission information indicating to update the one or more DNN setting information; determining the one or more updated DNN setting information according to the DNN update information; and obtaining a third image by performing artificial intelligence (AI) upscaling on a second image according to the one or more updated DNN setting information.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and more particularly, to an apparatus and method for encoding and decoding an image by using a deep neural network (DNN). Background Art

[0002] The image is encoded using a codec conforming to a predefined data compression standard such as the Moving Picture Experts Group (MPEG) standard, etc., and then stored in a recording medium or transmitted in the form of a bit stream through a communication channel.

[0003] With the development and spread of hardware capable of reproducing and storing high-resolution / high-definition images, there is an increasing demand for codecs capable of efficiently encoding and decoding high-resolution / high-definition images. Summary of the invention

[0004] Technical Solution

[0005] Provided are methods and apparatus for encoding and decoding an image at a low bit rate by using a deep neural network (DNN).

[0006] Furthermore, a computer-readable recording medium is provided in which a program for executing the image encoding method and the image decoding method according to the embodiment of the present disclosure on a computer is recorded.

[0007] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure.

[0008] According to an embodiment, an image decoding method is provided, including: obtaining DNN update permission information indicating whether to update one or more deep neural network (DNN) setting information; obtaining DNN update information necessary to determine one or more updated DNN setting information based on the DNN update permission information indicating to update the one or more DNN setting information; determining the one or more updated DNN setting information based on the DNN update information; and obtaining a third image by performing artificial intelligence (AI) upscaling on a second image based on the one or more updated DNN setting information.

[0009] The image decoding method further includes: obtaining the third image by performing AI upscaling on the second image according to one or more pieces of default DNN setting information based on an indication by the DNN update permission information that the one or more pieces of DNN setting information are not updated.

[0010] The DNN update information includes DNN quantity information indicating the number of the one or more updated DNN setting information, and wherein the step of determining the one or more updated DNN setting information includes: determining whether the number of the one or more updated DNN setting information matches the number of the one or more updated DNN setting information indicated by the DNN quantity information.

[0011] The DNN update information also includes at least one of the following information: layer number information indicating the number of layers included in the DNN, input channel information indicating the number of input channels in each of the layers, output channel information indicating the number of output channels in each layer, filter kernel size information indicating the size of the filter kernel in each layer, weight information indicating the weight of the filter kernel, and bias information indicating the bias used to adjust the output of the filter kernel.

[0012] The step of obtaining the DNN update information includes: obtaining default DNN reuse permission information indicating whether the structure of the default DNN is reused for the one or more updated DNN setting information; and obtaining reference DNN index information based on the default DNN reuse permission information indicating that the structure of the default DNN is reused for the one or more updated DNN setting information, wherein the reference DNN index information indicates the index of the default DNN whose information is reused, and wherein the step of determining the one or more updated DNN setting information includes: determining the one or more updated DNN setting information based on the structure of the default DNN indicated by the reference DNN index information and the DNN update information based on the default DNN reuse permission information indicating that the structure of the default DNN is reused for the one or more updated DNN setting information.

[0013] The step of determining the one or more updated DNN setting information also includes: determining the number of layers, the number of input channels in each of the layers, the number of output channels in each layer, and the size of the filter kernel of each layer based on the number of default layers in the default DNN indicated by the reference DNN index information, the number of default input channels in each of the default layers, the number of output channels in each default layer, and the size of the filter kernel of each default layer, and determining the weights and biases of the filter kernels in the one or more updated DNN setting information based on the DNN update information.

[0014] The DNN update information includes at least one of weight information indicating a weight of a filter kernel or bias information indicating a bias for adjusting an output of the filter kernel, the weight information indicating a difference between weights in the default DNN and the updated DNN, respectively, and the bias information indicating a difference between biases in the default DNN and the updated DNN, respectively.

[0015] The steps of obtaining the DNN update permission information, obtaining the DNN update information, and determining the one or more updated DNN setting information are performed for a frame group, wherein the frame group is an encoded video sequence including multiple frames, and wherein the step of obtaining the third image includes: for a frame in the frame group, determining updated DNN setting information for the frame based on the one or more updated DNN setting information; and obtaining the third image in the frame by performing AI amplification on the second image in the frame according to the updated DNN setting information for the frame.

[0016] The step of obtaining the third image includes: for a frame, obtaining DNN index information indicating updated DNN setting information for the frame based on the one or more updated DNN setting information; determining the updated DNN setting information for the frame based on the one or more updated DNN setting information according to the DNN index information; obtaining secondary DNN update permission information, wherein the secondary DNN update permission information indicates whether a secondary update of the updated DNN setting information for the frame will be performed; obtaining secondary DNN update information based on the secondary DNN update permission information indicating that a secondary update of the updated DNN setting information for the frame will be performed; and obtaining the third image by performing AI amplification on the second image using the secondary updated DNN setting information determined according to the secondary DNN update information.

[0017] According to an embodiment, an image decoding method is provided, comprising: obtaining deep neural network (DNN) index information for a frame in a plurality of frames, wherein the DNN index information indicates one or more default DNN setting information among a plurality of default DNN setting information for enlarging a second image; obtaining DNN update permission information indicating whether to update the default DNN setting information indicated by the DNN index information; obtaining DNN update information necessary to determine one or more updated DNN setting information based on the DNN update permission information indicating updating the default DNN setting information; determining the one or more updated DNN setting information based on the DNN update information; and obtaining a third image in the frame by performing artificial intelligence (AI) enlargement on the second image in the frame based on the one or more updated DNN setting information.

[0018] According to an embodiment, an image decoding device is provided, comprising: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to perform the following operations: obtaining DNN update permission information indicating whether to update one or more deep neural network (DNN) setting information; obtaining DNN update information necessary to determine one or more updated DNN setting information based on the DNN update permission information indicating to update the one or more DNN setting information; determining the one or more updated DNN setting information based on the DNN update information; and obtaining a third image by performing artificial intelligence (AI) upscaling on a second image based on the one or more updated DNN setting information.

[0019] According to an embodiment, there is provided a non-transitory computer-readable recording medium on which a program for executing the image decoding method according to an embodiment of the present disclosure is recorded.

[0020] According to an embodiment, a non-transitory computer-readable recording medium storing artificial intelligence (AI) data is provided, wherein the AI ​​data includes DNN update permission information indicating whether to update one or more deep neural network (DNN) setting information, and indicating to update the one or more DNN setting information based on the DNN update permission information, and the AI ​​data also includes: DNN update information required to determine one or more updated DNN setting information, and DNN index information indicating whether to enlarge a second image based on the one or more updated DNN setting information.

[0021] The DNN update information includes DNN quantity information indicating the number of the one or more updated DNN setting information, and wherein the DNN index information indicates whether the number of the one or more updated DNN setting information matches the number of the one or more updated DNN setting information indicated by the DNN quantity information.

[0022] The DNN update information includes at least one of the following information: layer number information indicating the number of layers included in the DNN, input channel information indicating the number of input channels in each of the layers, output channel information indicating the number of output channels in each layer, filter kernel size information indicating the size of the filter kernel in each layer, weight information indicating the weight of the filter kernel, and bias information indicating the bias used to adjust the output of the filter kernel.

[0023] The DNN update information also includes: default DNN reuse permission information indicating whether the structure of the default DNN is reused for the one or more updated DNN setting information; and when the default DNN reuse permission information indicates that the structure of the default DNN is reused for the one or more updated DNN setting information, the DNN update information also includes reference DNN index information, wherein the reference DNN index information indicates the index of the default DNN whose information is reused, and the DNN update information also includes information related to the update of the DNN setting information in which the structure of the default DNN indicated by the reference DNN index information is not reused.

[0024] The DNN setting information in which the structure of the default DNN indicated by the reference DNN index information is reused includes: the number of layers in the default DNN, the number of input channels in each of the layers, the number of output channels in each layer, and the size of the filter kernel of each layer, wherein the DNN setting information in which the structure of the default DNN indicated by the reference DNN index information is not reused includes the weights and biases of the filter kernel.

[0025] The DNN update information includes at least one of weight information indicating the weight of the filter kernel and bias information indicating a bias for adjusting the output of the filter kernel, wherein the weight information indicates a difference between a weight in a default DNN and a weight in an updated DNN, and wherein the bias information indicates a difference between a bias in the default DNN and a bias in the updated DNN.

[0026] The DNN update permission information and the DNN update information are set for a frame group including a plurality of frames, and wherein the DNN index information is set for each of the plurality of frames in the frame group.

[0027] The DNN index information is set for each of a plurality of frames in a frame group, and wherein the AI ​​data further includes: secondary DNN update permission information indicating whether a secondary update of the updated DNN setting information determined according to the DNN index information will be performed, and when the secondary DNN update permission information indicates that a secondary update of the updated DNN setting information determined according to the DNN index information will be performed, determining the secondary DNN update information required for the secondary updated DNN setting information for enlarging the second image. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0029] Figure 1 is a diagram for describing an artificial intelligence (AI) encoding process and an AI decoding process according to an embodiment;

[0030] Figure 2 is a block diagram of an AI decoding device according to an embodiment;

[0031] Figure 3 is a diagram illustrating a second deep neural network (DNN) for performing AI upscaling on a second image according to an embodiment;

[0032] Figure 4 is a diagram for describing a convolution operation through a convolution layer according to an embodiment;

[0033] Figure 5 is a table showing a mapping relationship between several pieces of image-related information and several pieces of DNN setting information according to an embodiment;

[0034] Figure 6 is a diagram illustrating a second image including a plurality of frames according to an embodiment;

[0035] Figure 7 is a block diagram of an AI encoding device according to an embodiment;

[0036] Figure 8 is a diagram illustrating a first DNN for performing AI downscaling on an original image according to an embodiment;

[0037] Fig. 9 is a diagram for describing a method of training a first DNN and a second DNN according to an embodiment;

[0038] Fig.10 is a sequence flow chart for describing a training process of a first DNN and a second DNN by a training device according to an embodiment;

[0039] Fig.11 is a diagram of an apparatus for performing AI downscaling on an original image and an apparatus for performing AI upscaling on a second image according to an embodiment;

[0040] Fig.12 shows the structure of a convolutional layer according to an embodiment;

[0041] Fig.13 According to the embodiment Fig.12 The weights and biases of the first filter kernel and the second filter kernel in the convolutional layer of ;

[0042] Fig.14 shows the structures of image data and AI data and the corresponding relationship between the image data and AI data according to an embodiment;

[0043] Fig.15 shows AI encoded data in a case where AI data and image data are separated from each other in a single file according to an embodiment;

[0044] Fig.16 shows AI encoded data in a case where AI data according to an embodiment is embedded in image data in a single file;

[0045] Fig.17 According to an embodiment, Figure 2 A block diagram of an AI decoding device having more detailed functions than an AI amplifier of an AI decoding device;

[0046] Fig.18 A DNN setting information determination method for updating DNN setting information based on each video according to an embodiment is shown;

[0047] Fig.19 A DNN setting information determination method for updating one or more pieces of DNN setting information based on each video according to an embodiment is shown;

[0048] Fig. 20 A DNN setting information determination method for updating one or more pieces of DNN setting information on a per-video and per-frame-group basis according to an embodiment is shown;

[0049] Fig.21 A DNN setting information determination method for updating one or more pieces of DNN setting information on a per-video and per-frame-group basis according to an embodiment is shown;

[0050] Fig. 22 A DNN setting information determination method for updating DNN setting information based on each frame according to an embodiment is shown;

[0051] Fig.23 shows a data structure including AI target data and AI auxiliary data and a data structure including AI data related to updating of DNN setting information according to an embodiment;

[0052] Fig.24 shows a data structure including AI data related to updating of DNN setting information according to an embodiment;

[0053] Fig.25 shows a data structure including AI data related to updating of DNN setting information according to an embodiment;

[0054] Fig.26 The implementation according to the embodiment is shown Fig.23 an AI codec high-level syntax including a data structure of AI data related to the update of DNN setting information;

[0055] Fig. 27 shows a data structure including AI data related to updating of DNN setting information for a frame or a frame group according to an embodiment;

[0056] Fig.28 The implementation according to the embodiment is shown Fig. 27 an AI codec high-level syntax including a data structure of AI data related to the update of DNN setting information;

[0057] Fig.29 is a flowchart of an image decoding method performed by an AI decoding device according to an embodiment; and

[0058] Fig.30 is a block diagram of an image decoding apparatus according to an embodiment. DETAILED DESCRIPTION

[0059] Since the present disclosure allows various changes and many examples, specific embodiments will be described in detail in the drawings and in the written description. However, this is not intended to limit the present disclosure to a specific mode of practice, and it will be understood that all changes, equivalents and substitutions that do not depart from the spirit and technical scope of the present disclosure are included in the present disclosure.

[0060] In the description of the embodiments, when a specific detailed explanation of the related art may unnecessarily obscure the essence of the present disclosure, such explanation is omitted. In addition, the numbers (e.g., first, second, etc.) used in the description are only identification marks used to distinguish one element from another element.

[0061] Throughout this disclosure, the expression "at least one of a, b, or c" includes only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0062] Furthermore, in the present disclosure, it will be understood that when elements are “connected” or “coupled” to each other, the elements may be directly connected or coupled to each other, but may alternatively be connected or coupled to each other through intermediate elements between the elements unless otherwise specified.

[0063] In the present disclosure, for an element represented as a "unit" or "module", two or more elements may be combined into one element, or one element may be divided into two or more elements according to subdivided functions. In addition, each element described below may also perform some or all of the functions performed by another element in addition to its main function, and some of the main functions of each element may be completely performed by another component.

[0064] Furthermore, in the present disclosure, an “image” or a “picture” may mean a still image, a moving image including a plurality of consecutive still images (or frames), or a video.

[0065] Furthermore, in the present disclosure, a deep neural network (DNN) is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model that uses a specific algorithm.

[0066] In addition, in the present disclosure, "parameter" is a value used in the operation process of each layer forming a neural network, and for example, may include a weight used when applying an input value to a specific operation expression. Here, the parameter can be expressed in a matrix form. The parameter is a value set as a result of training, and can be updated by separate training data when necessary.

[0067] Furthermore, in the present disclosure, a “first DNN” indicates a DNN for performing artificial intelligence (AI) downscaling on an image, and a “second DNN” indicates a DNN for performing AI upscaling on an image.

[0068] In addition, in the present disclosure, "DNN setting information" includes information related to elements constituting the DNN. "DNN setting information" includes the parameters described above as information related to elements constituting the DNN. The first DNN or the second DNN can be set by using the DNN setting information.

[0069] In addition, in the present disclosure, the "original image" refers to an image that is an object of AI encoding, and the "first image" refers to an image obtained as a result of performing AI reduction on the original image during the AI ​​encoding process. In addition, the "second image" refers to an image obtained via the first decoding during the AI ​​decoding process, and the "third image" refers to an image obtained by performing AI enlargement on the second image during the AI ​​decoding process.

[0070] In addition, in the present disclosure, "AI reduction" means a process of reducing the resolution of an image based on AI, and "first encoding" means an encoding process according to an image compression method based on frequency transformation. In addition, "first decoding" means a decoding process according to an image reconstruction method based on frequency transformation, and "AI enlargement" means a process of increasing the resolution of an image based on AI.

[0071] Figure 1 is a diagram for describing an AI encoding process and an AI decoding process according to an embodiment.

[0072] As described above, when the resolution of an image significantly increases, the throughput of information for encoding and decoding the image increases, and thus, a method for improving encoding and decoding efficiency of an image is required.

[0073] As in Figure 1 As shown in FIG, a first image 115 is obtained by performing AI reduction 110 on an original image 105 having a high resolution. Then, a first encoding 120 and a first decoding 130 are performed on the first image 115 having a relatively low resolution, so that the bit rate can be significantly reduced compared to performing the first encoding and the first decoding only on the original image 105 having a high resolution.

[0074] Specifically, in Figure 1 , during the AI ​​encoding process, a first image 115 is obtained by performing AI reduction 110 on an original image 105, and a first encoding 120 is performed on the first image 115 by an encoding device or the like. During the AI ​​decoding process, AI encoded data including AI data and image data obtained as a result of the AI ​​encoding is received by a decoding device or the like, a second image 135 is obtained via the first decoding 130, and a third image 145 is obtained by performing AI enlargement 140 on the second image 135.

[0075] Referring to the AI ​​encoding process in detail, when the original image 105 is received, AI reduction 110 is performed on the original image 105 to obtain a first image 115 of a specific resolution or a specific quality. Here, the AI ​​reduction 110 is performed based on an AI model, and the AI ​​model for performing the AI ​​reduction 110 needs to be jointly trained with the AI ​​model for performing the AI ​​enlargement 140 of the second image 135. This is because when the AI ​​model for the AI ​​reduction 110 and the AI ​​model for the AI ​​enlargement 140 are trained separately, the difference between the original image 105 as the object of AI encoding and the third image 145 reconstructed by AI decoding increases.

[0076] In an embodiment of the present disclosure, AI data may be used to maintain such a joint relationship during the AI ​​encoding process and the AI ​​decoding process. Therefore, the AI ​​data obtained by the AI ​​encoding process may include information indicating a magnification target, and during the AI ​​decoding process, AI magnification 140 is performed on the second image 135 according to the magnification target included in the AI ​​data.

[0077] The AI ​​model for AI reduction 110 and the AI ​​model for AI enlargement 140 may be implemented as DNNs. Fig. 9 As described, since the first DNN and the second DNN are jointly trained by sharing loss information under a specific target, the AI ​​encoding device may provide target information used during the joint training of the first DNN and the second DNN to the AI ​​decoding device, and the AI ​​decoding device may perform AI upscaling 140 on the second image 135 to a target resolution based on the provided target information.

[0078] about Figure 1The first encoding 120 and the first decoding 130 may reduce the amount of information of the first image 115 obtained by performing AI reduction 110 on the original image 105 through the first encoding 120. The first encoding 120 may include a process of generating prediction data by predicting the first image 115, a process of generating residual data corresponding to the difference between the first image 115 and the prediction data, a process of transforming the residual data of the spatial domain component into a frequency domain component, a process of quantizing the residual data transformed into the frequency domain component, and a process of entropy encoding the quantized residual data. Such a first encoding 120 may be performed via one of an image compression method using frequency transformation, such as MPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, High Efficiency Video Coding (HEVC), VC-1, VP8, VP9, ​​and AOMedia Video1 (AV1).

[0079] The second image 135 corresponding to the first image 115 may be reconstructed by performing the first decoding 130 on the image data. The first decoding 130 may include a process of generating quantized residual data by entropy decoding the image data, a process of inversely quantizing the quantized residual data, a process of transforming the residual data of the frequency domain component into the spatial domain component, a process of generating prediction data, and a process of reconstructing the second image 135 by using the prediction data and the residual data. Such first decoding 130 may be performed via an image reconstruction method corresponding to one of the image compression methods using frequency transformation used in the first encoding 120, such as MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1.

[0080] The AI ​​encoded data obtained through the AI ​​encoding process may include image data obtained as a result of performing the first encoding 120 on the first image 115 and AI data related to the AI ​​reduction 110 of the original image 105. The image data may be used during the first decoding 130, and the AI ​​data may be used during the AI ​​enlargement 140.

[0081] The image data may be transmitted in the form of a bitstream. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data that is a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding 120 performed on the first image 115. For example, the image data may include prediction mode information, motion information, and information related to quantization parameters used during the first encoding 120. The image data may be generated according to the rules (for example, according to the syntax) of the image compression method used during the first encoding 120 in MPEG-2, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1.

[0082] The AI ​​data is used in the AI ​​upscaling 140 based on the second DNN. As described above, because the first DNN and the second DNN are jointly trained, the AI ​​data includes information that enables the AI ​​upscaling 140 to be accurately performed on the second image 135 by the second DNN. During the AI ​​decoding process, the AI ​​upscaling 140 may be performed on the second image 135 based on the AI ​​data to have a target resolution and / or quality.

[0083] The AI ​​data may be transmitted together with the image data in the form of a bitstream. In addition, according to an embodiment, the AI ​​data may be transmitted separately from the image data in the form of a frame or a packet. The AI ​​data and the image data obtained as a result of the AI ​​encoding may be transmitted through the same network or through different networks.

[0084] Figure 2 is a block diagram of an AI decoding apparatus 200 according to an embodiment.

[0085] Reference Figure 2 , the AI ​​decoding device 200 according to the embodiment may include a receiver 210 and an AI decoder 230. The receiver 210 may include a communicator 212, a parser 214, and an outputter 216. The AI ​​decoder 230 may include a first decoder 232 and an AI amplifier 234.

[0086] The receiver 210 receives and parses AI encoded data obtained as a result of AI encoding, and outputs image data and AI data to the AI ​​decoder 230 .

[0087] Specifically, the communicator 212 receives AI-encoded data obtained as a result of AI encoding through a network. The AI-encoded data obtained as a result of performing AI encoding includes image data and AI data. The image data and AI data may be received through the same type of network or different types of networks.

[0088] The parser 214 receives the AI ​​coded data received through the communicator 212, and parses the AI ​​coded data to separate the image data and the AI ​​data. For example, the parser 214 can distinguish the image data and the AI ​​data by reading the header of the data obtained from the communicator 212. According to an embodiment, the parser 214 sends the image data and the AI ​​data to the outputter 216 via the header of the data received through the communicator 212, and the outputter 216 sends the image data and the AI ​​data to the first decoder 232 and the AI ​​amplifier 234, respectively. At this time, the image data included in the AI ​​coded data can be verified as image data generated by a specific codec (e.g., MPEG-2, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9 or AV1). In this case, the corresponding information can be sent to the first decoder 232 through the outputter 216, so that the image data is processed by the verified codec.

[0089] According to an embodiment, the AI ​​encoded data parsed by the parser 214 can be obtained from a data storage medium, wherein the data storage medium includes a magnetic medium (such as a hard disk, a floppy disk, or a tape), an optical recording medium (such as a CD-ROM or a DVD), or a magneto-optical medium (such as a floppy disk).

[0090] The first decoder 232 reconstructs a second image 135 corresponding to the first image 115 based on the image data. The second image 135 obtained by the first decoder 232 is provided to the AI ​​amplifier 234. According to an embodiment, first decoding related information (such as prediction mode information, motion information, quantization parameter information, etc.) included in the image data may also be provided to the AI ​​amplifier 234.

[0091] Upon receiving the AI ​​data, the AI ​​amplifier 234 performs AI up-amplification on the decoded second image 135 based on the AI ​​data. According to an embodiment, the AI ​​up-amplification may be performed by further using first decoding related information (such as prediction mode information, quantization parameter information, etc.) included in the image data.

[0092] The receiver 210 and the AI ​​decoder 230 according to the embodiment are described as separate devices, but can be implemented by one processor. In this case, the receiver 210 and the AI ​​decoder 230 can be implemented by a dedicated processor or by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). The dedicated processor can be implemented by including a memory for implementing an embodiment of the present disclosure or by including a memory processor for using an external memory.

[0093] In addition, the receiver 210 and the AI ​​decoder 230 may be configured by a plurality of processors. In this case, the receiver 210 and the AI ​​decoder 230 may be implemented by a combination of dedicated processors or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a GPU). Similarly, the AI ​​amplifier 234 and the first decoder 232 may be implemented by different processors.

[0094] The AI ​​data provided to the AI ​​amplifier 234 includes information that enables the second image 135 to be processed via AI enlargement. Here, the enlargement target should correspond to the reduction target of the first DNN. Therefore, the AI ​​data includes information for verifying the reduction target of the first DNN.

[0095] Examples of the information included in the AI ​​data include difference information between the resolution of the original image 105 and the resolution of the first image 115 and information related to the first image 115 .

[0096] The difference information may include information about the degree of resolution conversion of the first image 115 compared to the original image 105 (e.g., resolution conversion rate information). In addition, since the resolution of the first image 115 is verified by the resolution of the reconstructed second image 135 and thus the degree of resolution conversion is verified, the difference information may be represented only as the resolution information of the original image 105. Here, the resolution information may include a vertical screen size / horizontal size, or a ratio (16:9, 4:3, etc.) and a size of one axis. In addition, when there is pre-set resolution information, the resolution information may be represented in the form of an index or a flag.

[0097] The information related to the first image 115 may include information about at least one of a bit rate of image data obtained as a result of performing the first encoding on the first image 115 or a codec type used during the first encoding of the first image 115 .

[0098] The AI ​​amplifier 234 may determine an enlargement target of the second image 135 based on at least one of the difference information included in the AI ​​data or the information related to the first image 115. The enlargement target may indicate, for example, to what extent the resolution is to be enlarged for the second image 135. When the enlargement target is determined, the AI ​​amplifier 234 performs AI enlargement on the second image 135 through the second DNN to obtain a third image 145 corresponding to the enlargement target.

[0099] Before describing a method in which the AI ​​amplifier 234 performs AI up-scaling on the second image 135 according to the up-scaling target, reference will be made to Figure 3 and Figure 4 Describe the AI ​​amplification process through the second DNN.

[0100] Figure 3is a diagram illustrating a second DNN 300 for performing AI upscaling on a second image 135 according to an embodiment, and Figure 4 It is used to describe the Figure 3 Schematic diagram of the convolution operation in the first convolutional layer 310 of FIG.

[0101] As in Figure 3 As shown in , the second image 135 is input to the first convolution layer 310. Here, the annotation 3×3×4 in the first convolution layer 310 indicates that a convolution process is performed on one input image by using four filter kernels of size 3×3. Four feature maps are generated by the four filter kernels as a result of the convolution process. Each feature map indicates an inherent characteristic of the second image 135. For example, each feature map may represent a vertical direction characteristic, a horizontal direction characteristic, an edge characteristic, etc. of the second image 135.

[0102] Will refer to Figure 4 The convolution operation in the first convolutional layer 310 is described in detail.

[0103] One feature map 450 may be generated by multiplication and addition between parameters of a filter kernel 430 having a size of 3×3 used in the first convolution layer 310 and corresponding pixel values ​​in the second image 135. Since four filter kernels are used in the first convolution layer 310, four feature maps may be generated through a convolution operation using the four filter kernels.

[0104] Reference Figure 4 , I1 to I49 indicate pixels in the second image 135, and F1 to F9 in the filter kernel 430 indicate parameters of the filter kernel 430. In addition, M1 to M9 in the feature map 450 indicate samples of the feature map 450.

[0105] exist Figure 4 , the second image 135 includes 49 pixels, but the number of pixels is only an example, and when the second image 135 has a resolution of 4K, the second image 135 may include, for example, 3840×2160 pixels.

[0106] During the convolution operation process, the pixel values ​​of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination (e.g., addition) of the multiplication result values ​​may be assigned as the value of M1 of the feature map 450. When the step size of the convolution operation is 2, the pixel values ​​of I3, I4, I5, I10, I11, I12, I17, I18, and I19 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination of the multiplication result values ​​may be assigned as the value of M2 of the feature map 450. Similarly, M3 to M9 of the feature map 450 may be obtained by performing the above-mentioned operation process.

[0107] While the filter kernel 430 moves to the last pixel of the second image 135 along the step size, a convolution operation is performed between the pixel values ​​in the second image 135 and the parameters of the filter kernel 430, and thus a feature map 450 having a specific size may be generated.

[0108] According to the present disclosure, the values ​​of the parameters of the second DNN (e.g., the values ​​of the parameters of the filter kernel used in the convolution layer of the second DNN (e.g., the values ​​of F1 to F9 of the filter kernel 430)) can be optimized by joint training of the first DNN and the second DNN. As described above, the AI ​​amplifier 234 can determine the enlargement target corresponding to the reduction target of the first DNN based on the AI ​​data, and determine the parameters corresponding to the determined enlargement target as the parameters of the filter kernel used in the convolution layer of the second DNN.

[0109] The convolutional layers included in the first DNN and the second DNN can be based on the reference Figure 4 The convolution operation described above is used to perform the processing, but referring to Figure 4 The described convolution operation process is only an example and is not limited thereto.

[0110] Return to reference Figure 3 , the feature map output from the first convolution layer 310 may be input to the first activation layer 320 .

[0111] The first activation layer 320 may impart nonlinear features to each feature map. The first activation layer 320 may include a sigmoid function, a hyperbolic tangent (Tanh) function, a rectified linear unit (ReLU) function, etc., but is not limited thereto.

[0112] The first activation layer 320 imparting the nonlinear feature indicates changing at least one sample value of the feature map which is the output of the first convolution layer 310. Here, the change is performed by applying the nonlinear feature.

[0113] The first activation layer 320 determines whether to send the sample values ​​of the feature map output from the first convolution layer 310 to the second convolution layer 330. For example, some sample values ​​of the feature map are activated by the first activation layer 320 and sent to the second convolution layer 330, and some sample values ​​are deactivated by the first activation layer 320 and are not sent to the second convolution layer 330. The inherent characteristics of the second image 135 represented by the feature map are emphasized by the first activation layer 320.

[0114] The feature map 325 output from the first activation layer 320 is input to the second convolutional layer 330 . Figure 3 One of the feature maps 325 shown in FIG. 3 is a feature map of the reference in the first activation layer 320. Figure 4 The result of processing the feature map 450 is described.

[0115] The notation 3×3×4 in the second convolution layer 330 indicates that a convolution process is performed on the feature map 325 by using four filters of size 3×3. The output of the second convolution layer 330 is input to the second activation layer 340. The second activation layer 340 can impart nonlinear features to the input data.

[0116] The feature map 345 output from the second activation layer 340 is input to the third convolutional layer 350 . Figure 3 The annotation 3×3×1 in the third convolution layer 350 shown in FIG. 1 indicates that a convolution process is performed by using a filter kernel of size 3×3 to generate an output image. The third convolution layer 350 is a layer for outputting a final image and generates an output by using a filter kernel. According to an embodiment of the present disclosure, the third convolution layer 350 may output the third image 145 as a result of the convolution operation.

[0117] As will be described later, there may be a plurality of pieces of DNN setting information indicating the number of filter kernels of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, parameters of the filter kernels of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, and the like, and the plurality of pieces of DNN setting information should be associated with the plurality of pieces of DNN setting information of the first DNN. The association between the plurality of pieces of DNN setting information of the second DNN and the plurality of DNN setting information of the first DNN may be achieved via joint training of the first DNN and the second DNN.

[0118] exist Figure 3, the second DNN 300 includes three convolutional layers (a first convolutional layer 310, a second convolutional layer 330, and a third convolutional layer 350) and two activation layers (a first activation layer 320 and a second activation layer 340), but this is only an example, and the number of convolutional layers and the number of activation layers may vary according to the embodiment. In addition, according to the embodiment, the second DNN 300 may be implemented as a recurrent neural network (RNN). In this case, the convolutional neural network (CNN) structure of the second DNN 300 according to the embodiment of the present disclosure is changed to an RNN structure.

[0119] According to an embodiment, the AI ​​amplifier 234 may include at least one arithmetic logic unit (ALU) for the above-mentioned convolution operation and the operation of the activation layer. The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier and an adder, wherein the multiplier performs multiplication between the sample value of the second image 135 or the feature map output from the previous layer and the sample value of the filter kernel, and the adder adds the result value of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0120] Hereinafter, a method of performing AI up-scaling on the second image 135 according to the up-scaling target, performed by the AI ​​amplifier 234, will be described.

[0121] According to an embodiment, the AI ​​amplifier 234 may store a plurality of pieces of DNN setting information that may be configured in the second DNN.

[0122] Here, the DNN setting information may include information about the number of convolutional layers included in the second DNN, the number of filter kernels for each convolutional layer, or at least one of one or more parameters of each filter kernel. Multiple pieces of DNN setting information may correspond to various amplification targets, respectively, and the second DNN may operate based on the DNN setting information corresponding to the specific amplification target. The second DNN may have different structures based on the DNN setting information. For example, the second DNN may include three convolutional layers based on any one piece of DNN setting information, and may include four convolutional layers based on another piece of DNN setting information.

[0123] According to an embodiment, the DNN setting information may include only the parameters of the filter kernel used in the second DNN. In this case, the structure of the second DNN does not change, but only the parameters of the internal filter kernel may be changed based on the DNN setting information.

[0124] The AI ​​amplifier 234 may obtain DNN setting information for performing AI amplification on the second image 135 from among the plurality of DNN setting information. Each of the plurality of DNN setting information may be information for obtaining the third image 145 of a predetermined resolution and / or a predetermined quality, and is jointly trained with the first DNN.

[0125] For example, one piece of the multiple DNN setting information may include information for obtaining a third image 145 having a resolution that is twice the resolution of the second image 135 (for example, the third image 145 may have a 4K (4096×2160) resolution that is twice the 2K (2048×1080) resolution of the second image 135), and another piece of the DNN setting information may include information for obtaining a third image 145 having a resolution that is four times higher than the resolution of the second image 135 (for example, a third image 145 having an 8K (8192×4320) resolution that is four times higher than the 2K (2048×1080) resolution of the second image 135).

[0126] Each of the plurality of pieces of DNN setting information is Figure 7 The DNN setting information of the first DNN of the AI ​​encoding device 600 is jointly obtained, and the AI ​​amplifier 234 obtains one of the multiple DNN setting information according to the enlargement ratio corresponding to the reduction ratio of the DNN setting information of the first DNN. In this regard, the AI ​​amplifier 234 can verify the information of the first DNN. In order for the AI ​​amplifier 234 to verify the information of the first DNN, the AI ​​decoding device 200 according to the embodiment receives AI data including the information of the first DNN from the AI ​​encoding device 600.

[0127] In other words, the AI ​​amplifier 234 can verify the information targeted by the DNN setting information of the first DNN used to obtain the first image 115 by using the information received from the AI ​​encoding device 600, and obtain the DNN setting information of the second DNN jointly trained with the DNN setting information of the first DNN.

[0128] When DNN setting information for performing AI up-scaling on the second image 135 is obtained from a plurality of pieces of DNN setting information, input data may be processed based on the second DNN operating according to the obtained DNN setting information.

[0129] For example, when any DNN setting information is obtained, Figure 3 The number of filter kernels included in each of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300 and the parameters of the filter kernels are set to the values ​​included in the obtained DNN setting information.

[0130] Specifically, in Figure 3 The parameters of the 3×3 filter kernel used in any convolutional layer of the second DNN are set to {1,1,1,1,1,1,1,1,1}, and when the DNN setting information is subsequently changed, the parameters are replaced by {2,2,2,2,2,2,2,2,2} as parameters included in the changed DNN setting information.

[0131] The AI ​​amplifier 234 may obtain DNN setting information for AI amplification from the plurality of pieces of DNN setting information based on information included in the AI ​​data, and the AI ​​data for obtaining the DNN setting information will now be described.

[0132] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification from the plurality of pieces of DNN setting information based on the difference information included in the AI ​​data. For example, when it is verified based on the difference information that the resolution of the original image 105 (e.g., 4K (4096×2160) resolution) is twice as high as the resolution of the first image 115 (e.g., 2K (2048×1080) resolution), the AI ​​amplifier 234 may obtain DNN setting information for increasing the resolution of the second image 135 by two times.

[0133] According to another embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification of the second image 135 from a plurality of pieces of DNN setting information based on information related to the first image 115 included in the AI ​​data. The AI ​​amplifier 234 may predetermine a mapping relationship between image-related information and DNN setting information, and obtain DNN setting information mapped to information related to the first image 115.

[0134] Figure 5 is a table showing a mapping relationship between image related information and DNN setting information according to an embodiment.

[0135] according to Figure 5 , AI encoding and AI decoding processing not only considers the change in resolution, but also considers other factors used to perform AI encoding and decoding processing. Figure 5 As shown in , the DNN setting information can be selected by considering resolution (such as standard definition (SD), high definition (HD), or full HD), bit rate (such as 10Mbps, 15Mbps, or 20Mbps), and codec information (such as AV1, H.264, or HEVC) alone or together. Considering such resolution, bit stream, and codec information, it is considered that the training of each element should be performed jointly with the encoding and decoding process during the AI ​​training process (see Fig. 9 ).

[0136] Therefore, when based on including Figure 5 When multiple DNN setting information is provided for the image-related information such as the codec type, image resolution, etc. shown in , the DNN setting information for AI upscaling the second image 135 can be obtained based on the information related to the first image 115 received during the AI ​​decoding process.

[0137] In other words, AI amplifier 234 can Figure 5 The image-related information on the left side of the table is matched with the DNN setting information on the right side of the table to use the DNN setting information according to the image-related information.

[0138] As in Figure 5 As shown in , when it is verified from information related to the first image 115 that the resolution of the first image 115 is SD, the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 10Mbps, and the first encoding is performed on the first image 115 via the AV1 codec, the AI ​​amplifier 234 can use the A DNN setting information among the multiple DNN setting information.

[0139] In addition, when it is verified from information related to the first image 115 that the resolution of the first image 115 is HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the H.264 codec, the AI ​​amplifier 234 can use B DNN setting information among the multiple DNN setting information.

[0140] In addition, when it is verified from the information related to the first image 115 that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 20Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 may use the C DNN setting information among the plurality of DNN setting information. When it is verified that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 may use the D DNN setting information among the plurality of DNN setting information. One of the C DNN setting information and the D DNN setting information is selected based on whether the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 20Mbps or 15Mbps. Different bit rates of image data obtained when performing the first encoding on the first image 115 of the same resolution via the same codec indicate different qualities of the reconstructed image. Therefore, the first DNN and the second DNN may be jointly trained based on a specific image quality, and therefore, the AI ​​amplifier 234 may obtain the DNN setting information according to the bit rate of the image data indicating the quality of the second image 135.

[0141] According to another embodiment, the AI ​​amplifier 234 may obtain DNN setting information for performing AI amplification on the second image 135 from a plurality of pieces of DNN setting information in consideration of both information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and information related to the first image 115 included in the AI ​​data. For example, the AI ​​amplifier 234 may receive quantization parameter information used during the first encoding process of the first image 115 from the first decoder 232, verify the bit rate of image data obtained as a result of encoding the first image 115 from the AI ​​data, and obtain DNN setting information corresponding to the quantization parameter information and the bit rate. Even when the bit rate is the same, the quality of the reconstructed image may vary according to the complexity of the image. The bit rate is a value representing the entire first image 115 on which the first encoding is performed, and the quality of each frame may vary even within the first image 115. Therefore, when considering the prediction mode information, motion information, and / or quantization parameter obtainable for each frame from the first decoder 232, DNN setting information more suitable for the second image 135 may be obtained compared to when only the AI ​​data is used.

[0142] In addition, according to an embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information. The identifier of the DNN setting information is information for distinguishing a pair of DNN setting information jointly trained between the first DNN and the second DNN, so that AI amplification is performed on the second image 135 based on the enlargement target corresponding to the reduction target of the first DNN. The AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information corresponding to the identifier of the DNN setting information after obtaining the identifier of the DNN setting information included in the AI ​​data. For example, an identifier indicating each of the multiple DNN setting information that can be configured or set in the first DNN and an identifier indicating each of the multiple DNN setting information that can be configured or set in the second DNN may be pre-specified. In this case, the same identifier may be specified for a pair of DNN setting information that can be configured or set in each of the first DNN and the second DNN. The AI ​​data may include an identifier of the DNN setting information set in the first DNN for AI reduction of the original image 105. The AI ​​amplifier 234 receiving the AI ​​data may perform AI up-amplification on the second image 135 by using DNN setting information indicated by an identifier included in the AI ​​data among the plurality of DNN setting information.

[0143] In addition, according to an embodiment, the AI ​​data may include DNN setting information. The AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information after obtaining the DNN setting information included in the AI ​​data.

[0144] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter kernels for each convolutional layer, one or more parameters of each filter kernel, etc.) are stored in the form of a lookup table, the AI ​​amplifier 234 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on the information included in the AI ​​data, and perform AI amplification on the second image 135 by using the obtained DNN setting information.

[0145] According to an embodiment, when the structure of the DNN corresponding to the amplification target is determined, the AI ​​amplifier 234 may obtain DNN setting information corresponding to the determined structure of the DNN, for example, parameters of a filter kernel.

[0146] The AI ​​amplifier 234 obtains DNN setting information of the second DNN through the AI ​​data including information related to the first DNN, and performs AI amplification on the second image 135 through the second DNN set based on the obtained DNN setting information. As such, memory usage and throughput can be reduced compared to when features of the second image 135 are directly analyzed for amplification.

[0147] According to an embodiment, when the second image 135 includes a plurality of frames, the AI ​​amplifier 234 may independently obtain DNN setting information for a specific number of frames, or may obtain common DNN setting information for all frames.

[0148] Figure 6 is a diagram illustrating a second image 135 including a plurality of frames.

[0149] As in Figure 6 As shown in , the second image 135 may include frames t0 to tn.

[0150] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information of the second DNN through the AI ​​data, and perform AI amplification on frames t0 to tn based on the obtained DNN setting information. In other words, frames t0 to tn may be processed via AI amplification based on the common DNN setting information.

[0151] According to another embodiment, the AI ​​amplifier 234 may perform AI amplification on some frames (e.g., frame t0 to frame ta) among frame t0 to frame tn by using the "A" DNN setting information obtained from the AI ​​data, and perform AI amplification on frame ta+1 to frame tb by using the "B" DNN setting information obtained from the AI ​​data. In addition, the AI ​​amplifier 234 may perform AI amplification on frame tb+1 to frame tn by using the "C" DNN setting information obtained from the AI ​​data. In other words, the AI ​​amplifier 234 may independently obtain DNN setting information for each group including a specific number of frames among a plurality of frames, and perform AI amplification on the frames included in each group by using the independently obtained DNN setting information.

[0152] According to another embodiment, the AI ​​amplifier 234 may independently obtain DNN setting information for each frame forming the second image 135. In other words, when the second image 135 includes three frames, the AI ​​amplifier 234 may perform AI amplification on the first frame by using the DNN setting information obtained about the first frame, perform AI amplification on the second frame by using the DNN setting information obtained about the second frame, and perform AI amplification on the third frame by using the DNN setting information obtained about the third frame. Based on the information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and the information related to the first image 115 included in the above-mentioned AI data, the DNN setting information may be independently obtained for each frame included in the second image 135. This is because the mode information, the quantization parameter information, etc. may be independently determined for each frame included in the second image 135.

[0153] According to another embodiment, the AI ​​data may include information about which frame the DNN setting information is applicable, wherein the DNN setting information is obtained based on the AI ​​data. For example, when the AI ​​data includes information indicating that the DNN setting information is applicable until frame ta, the AI ​​amplifier 234 performs AI amplification on frame t0 to frame ta by using the DNN setting information obtained based on the AI ​​data. In addition, when another piece of AI data includes information indicating that the DNN setting information is applicable until frame tn, the AI ​​amplifier 234 performs AI amplification on frame ta+1 to frame tn by using the DNN setting information obtained based on the other piece of AI data.

[0154] In the following, reference will be made to Figure 7 An AI encoding apparatus 600 for performing AI encoding on an original image 105 is described.

[0155] Figure 7 is a block diagram of an AI encoding apparatus 600 according to an embodiment.

[0156] Reference Figure 7 , the AI ​​encoding device 600 may include an AI encoder 610 and a transmitter 630. The AI ​​encoder 610 may include an AI reducer 66 and a first encoder 614. The transmitter 630 may include a data processor 632 and a communicator 634.

[0157] exist Figure 7 In the embodiment of the present invention, the AI ​​encoder 610 and the transmitter 630 are shown as independent devices, but the AI ​​encoder 610 and the transmitter 630 may be implemented by one processor. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a graphics processor GPU). The dedicated processor may be implemented by including a memory for implementing an embodiment of the present disclosure or by including a memory processor for using an external memory.

[0158] In addition, the AI ​​encoder 610 and the transmitter 630 may be composed of multiple processors. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a combination of dedicated processors or by a combination of software and multiple general-purpose processors (such as AP, CPU or GPU). The AI ​​reducer 612 and the first encoder 614 may be implemented by different processors.

[0159] The AI ​​encoder 610 performs AI reduction on the original image 105 and performs first encoding on the first image 115. The AI ​​encoder 610 performs AI reduction on the original image 105 and performs first encoding on the first image 115, and transmits the AI ​​data and the image data to the transmitter 630. The transmitter 630 transmits the AI ​​data and the image data to the AI ​​decoding device 200. The transmitter 630 may include a communication interface, such as a wireless communication interface including a radio and / or an antenna, or a wired communication interface (such as HDMI, Ethernet, etc.).

[0160] The image data includes data obtained as a result of performing the first encoding on the first image 115. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data that is a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding process of the first image 115. For example, the image data may include prediction mode information, motion information, quantization parameter information, etc. used to perform the first encoding on the first image 115.

[0161] The AI ​​data includes information that enables AI up-scaling to be performed on the second image 135 to an up-scaling target corresponding to the down-scaling target of the first DNN. According to an embodiment, the AI ​​data may include difference information between the original image 105 and the first image 115. In addition, the AI ​​data may include information related to the first image 115. The information related to the first image 115 may include information about at least one of a resolution of the first image 115, a bit rate of image data obtained as a result of performing the first encoding on the first image 115, and a codec type used during the first encoding of the first image 115.

[0162] According to an embodiment, the AI ​​data may include an identifier of mutually agreed or unified DNN setting information so that the AI ​​up-scaling is performed on the second image 135 to an up-scaling target corresponding to the down-scaling target of the first DNN.

[0163] Furthermore, according to an embodiment, the AI ​​data may include DNN setting information configurable in the second DNN.

[0164] The AI ​​downsizer 612 may obtain the first image 115 obtained by performing AI downsizing on the original image 105 through the first DNN. The AI ​​downsizer 612 may determine a downsizing target of the original image 105 based on a predetermined criterion.

[0165] In order to obtain the first image 115 based on the reduction target, the AI ​​reducer 612 may store a plurality of DNN setting information configurable in the first DNN. The AI ​​reducer 612 obtains DNN setting information corresponding to the reduction target from the plurality of DNN setting information, and performs AI reduction on the original image 105 through the first DNN set in the obtained DNN setting information.

[0166] Each of the plurality of pieces of DNN setting information may be trained to obtain a first image 115 having a predetermined resolution and / or a predetermined quality. For example, any one of the plurality of pieces of DNN setting information may include information for obtaining a first image 115 having a resolution half that of the original image 105 (e.g., the first image 115 may be reduced to a 2K (2048×1080) resolution that is half that of the 4K (4096×2160) resolution of the original image 105), and another piece of DNN setting information may include information for obtaining a first image 115 having a resolution one-fourth that of the original image 105 (e.g., the first image 115 may be reduced to a 2K (2048×1080) resolution that is one-fourth that of the 8K (8192×4320) resolution of the original image 105).

[0167] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter kernels for each convolutional layer, one or more parameters of each filter kernel, etc.) are stored in the form of a lookup table, the AI ​​downsizer 612 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on the reduction target, and perform AI downsizing on the original image 105 by using the obtained DNN setting information.

[0168] According to an embodiment, the AI ​​downsizer 612 may determine the structure of the DNN corresponding to the downsizing target, and obtain DNN setting information corresponding to the determined structure of the DNN, for example, obtain parameters of a filter kernel.

[0169] As the first DNN and the second DNN are jointly trained, a plurality of pieces of DNN setting information for performing AI reduction on the original image 105 may have an optimized value. Here, each piece of DNN setting information includes at least one of the number of convolutional layers included in the first DNN, the number of filter kernels for each convolutional layer, or one or more parameters of each filter kernel.

[0170] The AI ​​downsizer 612 may set the first DNN using the DNN setting information for performing AI downsizing on the original image 105 to obtain the first image 115 of a specific resolution and / or a specific quality through the first DNN. When the DNN setting information for performing AI downsizing on the original image 105 is obtained from a plurality of pieces of DNN setting information, each layer in the first DNN may process input data based on information included in the DNN setting information.

[0171] Hereinafter, a method of determining a downscaling target performed by the AI ​​downscaler 612 will be described. The downscaling target may indicate, for example, how much the resolution is reduced from the original image 105 to obtain the first image 115.

[0172] According to an embodiment, the AI ​​reducer 612 may determine the reduction target based on at least one of the compression ratio of the original image 105 (e.g., the resolution difference between the original image 105 and the first image 115, the target bit rate, etc.), the compression quality (e.g., the type of bit rate), the compression history information or the type.

[0173] For example, the AI ​​downsizer 612 may determine a downsizing target based on a compression ratio, compression quality, etc. that is preset or input from a user.

[0174] As another example, the AI ​​downsizer 612 may determine the downsizing target by using the compression history information stored in the AI ​​encoding device 600. For example, based on the compression history information of the AI ​​encoding device 600, the encoding quality, compression ratio, etc. preferred by the user may be determined, and the downsizing target may be determined based on the encoding quality determined based on the compression history information. For example, the resolution, quality, etc. of the first image 115 may be determined based on the most frequently used encoding quality according to the compression history information.

[0175] As another example, the AI ​​downsizer 612 may determine the downsizing target based on encoding qualities used more frequently than a specific threshold (eg, an average quality of encoding qualities used more frequently than a specific threshold) according to the compression history information.

[0176] As another example, the AI ​​downsizer 612 may determine the downscaling target based on the resolution, type (eg, file format), etc. of the original image 105 .

[0177] According to an embodiment, when the original image 105 includes a plurality of frames, the AI ​​downsizer 612 may independently determine downsizing targets for a specific number of frames among the plurality of frames, or may determine downsizing targets for all of the plurality of frames.

[0178] According to an embodiment, the AI ​​downsizer 612 may divide the frames included in the original image 105 into a specific number of groups and independently determine a downsizing target for each group. The same or different downsizing targets may be determined for each group. According to each group, the number of frames included in the group may be the same or different.

[0179] According to another embodiment, the AI ​​downsizer 612 may independently determine a downscaling target for each frame included in the original image 105. The same or different downscaling targets may be determined for each frame.

[0180] Hereinafter, an example of the structure of the first DNN 700 on which AI reduction is based will be described.

[0181] Figure 8 is a diagram illustrating a first DNN 700 for performing AI downscaling on an original image 105 according to an embodiment.

[0182] As in Figure 8 As shown in FIG. 1 , the original image 105 is input to the first convolution layer 710. The first convolution layer 710 performs a convolution process on the original image 105 by using 32 filter kernels of a size of 5×5. Here, 32 feature maps generated as a result of the convolution process are input to the first activation layer 720. The first activation layer 720 may impart nonlinear features to the 32 feature maps.

[0183] The first activation layer 720 determines whether to send sample values ​​of the feature map output from the first convolution layer 710 to the second convolution layer 730. For example, some sample values ​​of the feature map are activated by the first activation layer 720 and sent to the second convolution layer 730, and some sample values ​​are deactivated by the first activation layer 720 and are not sent to the second convolution layer 730. Information represented by the feature map output from the first convolution layer 710 is emphasized by the first activation layer 720.

[0184] The output 725 of the first activation layer 720 is input to the second convolution layer 730. The second convolution layer 730 performs a convolution process on the input data by using 32 filter kernels of a size of 5×5. Here, 32 feature maps output as a result of the convolution process are input to the second activation layer 740, and the second activation layer 740 can give nonlinear features to the 32 feature maps.

[0185] The output 745 of the second activation layer 740 is input to the third convolution layer 750. The third convolution layer 750 performs a convolution process on the input data by using a filter kernel of size 5×5. As a result of the convolution process, an image may be output from the third convolution layer 750. The third convolution layer 750 generates an output by using the one filter kernel as a layer for outputting a final image. According to an embodiment of the present disclosure, the third convolution layer 750 may output the first image 115 as a result of the convolution operation. Although embodiments for the convolution layer and the activation layer have been described above, the convolution layer and the activation layer are not limited to the above-mentioned specific embodiments and may be variously configured.

[0186] There may be a plurality of pieces of DNN setting information indicating the number of filter kernels of the first convolution layer 710, the second convolution layer 730, and the third convolution layer 750 of the first DNN 700, one or more parameters of each filter kernel of the first convolution layer 710, the second convolution layer 730, and the third convolution layer 750 of the first DNN 700, and the plurality of pieces of DNN setting information may be associated with the plurality of pieces of DNN setting information of the second DNN. The association between the plurality of pieces of DNN setting information of the first DNN and the plurality of pieces of DNN setting information of the second DNN may be achieved via joint training of the first DNN and the second DNN.

[0187] exist Figure 8 , the first DNN 700 includes three convolutional layers (respectively, a first convolutional layer 710, a second convolutional layer 730, and a third convolutional layer 750) and two activation layers (respectively, a first activation layer 720 and a second activation layer 740), but this is only an example, and the number of convolutional layers and activation layers may vary according to various embodiments. In addition, according to an embodiment, the first DNN 700 may be implemented as an RNN. In this case, the CNN structure of the first DNN 700 may be changed to an RNN structure.

[0188] According to an embodiment, the AI ​​reducer 612 may include at least one ALU for the above-mentioned convolution operation and the operation of the activation layer. The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier and an adder, wherein the multiplier performs multiplication between the sample value of the original image 105 or the feature map output from the previous layer and the sample value of the filter kernel, and the adder adds the result value of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0189] Return to reference Figure 7Upon receiving the first image 115 from the AI ​​downsizer 612, the first encoder 614 may reduce the amount of information of the first image 115 by performing first encoding on the first image 115. Image data corresponding to the first image 115 may be obtained as a result of the first encoding performed by the first encoder 614.

[0190] The data processor 632 processes at least one of the AI ​​data or the image data to be sent in a specific form. For example, when the AI ​​data and the image data are to be sent in the form of a bit stream, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and send the image data and the AI ​​data in the form of one bit stream through the communicator 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and send each of the bit streams corresponding to the AI ​​data and the bit streams corresponding to the image data through the communicator 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a frame or a packet, and send the image data in the form of a bit stream and send the AI ​​data in the form of a frame or a packet through the communicator 634.

[0191] The communicator 634 transmits the AI-encoded data obtained as a result of performing the AI ​​encoding through the network. The AI-encoded data obtained as a result of performing the AI ​​encoding includes image data and AI data. The image data and the AI ​​data may be transmitted through the same type of network or different types of networks.

[0192] According to an embodiment, the AI ​​encoded data obtained as a result of processing by the data processor 632 may be stored in a data storage medium, wherein the data storage medium includes a magnetic medium (such as a hard disk, a floppy disk, or a tape), an optical recording medium (such as a CD-ROM or a DVD), or a magneto-optical medium (such as a floppy disk).

[0193] In the following, reference will be made to Fig. 9 A method of jointly training the first DNN 700 and the second DNN 300 is described.

[0194] Fig. 9 is a diagram for describing a method of training a first DNN 700 and a second DNN 300.

[0195] In an embodiment, the original image 105 that is AI-encoded by the AI ​​encoding process is reconstructed into the third image 145 via the AI ​​decoding process, and in order to maintain the similarity or consistency between the original image 105 and the third image 145 obtained as a result of AI decoding, communication or cooperation between the AI ​​encoding process and the AI ​​decoding process is required. In other words, it is necessary to reconstruct the information lost in the AI ​​encoding process during the AI ​​decoding process, and in this regard, it is necessary to jointly train the first DNN 700 and the second DNN 300 to compensate for the information loss.

[0196] In order to perform accurate AI decoding, it is necessary to reduce Fig. 9 The difference between the third training image 804 and the original training image 801 shown in FIG. 8 corresponds to quality loss information 830. Therefore, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0197] First, we will describe Fig. 9 The training process shown in .

[0198] exist Fig. 9 , the original training image 801 is an image to be subjected to AI reduction, and the first training image 802 is an image obtained by performing AI reduction on the original training image 801. In addition, the third training image 804 is an image obtained by performing AI enlargement on the first training image 802.

[0199] The original training image 801 may be a still image or a moving image including a plurality of frames. According to an embodiment, the original training image 801 may include a luminance image extracted from a still image or a moving image including a plurality of frames. In addition, according to an embodiment, the original training image 801 may include a block image extracted from a still image or a moving image including a plurality of frames. When the original training image 801 includes a plurality of frames, the first training image 802, the second training image, and the third training image 804 may each include a plurality of frames. When the plurality of frames of the original training image 801 are sequentially input to the first DNN 700, the plurality of frames of the first training image 802, the plurality of frames of the second training image, and the plurality of frames of the third training image 804 may be sequentially obtained by the first DNN 700 and the second DNN 300.

[0200] For joint training of the first DNN 700 and the second DNN 300, an original training image 801 is input to the first DNN 700. The original training image 801 input to the first DNN 700 is output as a first training image 802 via AI downscaling, and the first training image 802 is input to the second DNN 300. A third training image 804 is output as a result of performing AI upscaling on the first training image 802.

[0201] Reference Fig. 9 , the first training image 802 is input to the second DNN 850, and according to an embodiment, the second training image obtained when the first encoding and the first decoding are performed on the first training image 802 may be input to the second DNN 300. In order to input the second training image to the second DNN 300, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 codecs may be used. Specifically, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 codecs may be used to perform the first encoding on the first training image 802 and perform the first decoding on the image data corresponding to the first training image 802.

[0202] Reference Fig. 9 , separately from the first training image 802 output by the first DNN 700, a reduced training image 803 obtained by performing conventional downscaling on the original training image 801 is obtained. Here, the conventional downscaling may include at least one of bilinear scaling, bicubic scaling, lanczos scaling, or step scaling.

[0203] In order to prevent the structural features of the first image 115 from greatly deviating from the structural features of the original image 105 , a reduced training image 803 is obtained to preserve the structural features of the original training image 801 .

[0204] Before performing training, the first DNN 700 and the second DNN 300 may be configured according to predetermined DNN setting information. When performing training, structure loss information 810, complexity loss information 820, and quality loss information 830 may be determined.

[0205] The structural loss information 810 may be determined based on a result of comparing the reduced training image 803 and the first training image 802. For example, the structural loss information 810 may correspond to a difference between structural information of the reduced training image 803 and structural information of the first training image 802. The structural information may include various features that may be extracted from an image, such as brightness, contrast, a histogram, etc. of the image. The structural loss information 810 indicates how much structural information of the original training image 801 is maintained in the first training image 802. When the structural loss information 810 is small, the structural information of the first training image 802 is similar to the structural information of the original training image 801.

[0206] The complexity loss information 820 may be determined based on the spatial complexity of the first training image 802. For example, the total variance value of the first training image 802 may be used as the spatial complexity. The complexity loss information 820 is related to the bit rate of the image data obtained by performing the first encoding on the first training image 802. When the complexity loss information 820 is small, the bit rate of the image data is low.

[0207] The quality loss information 830 may be determined based on a result of comparing the original training image 801 with the third training image 804. The quality loss information 830 may include at least one of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, or a video multi-method assessment fusion (VMAF) value about the difference between the original training image 801 and the third training image 804. The quality loss information 830 indicates how similar the third training image 804 is to the original training image 801. When the quality loss information 830 is small, the third training image 804 is more similar to the original training image 801.

[0208] Reference Fig. 9 , the structure loss information 810, the complexity loss information 820, and the quality loss information 830 are used to train the first DNN 700, and the quality loss information 830 is used to train the second DNN 300. In other words, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0209] The first DNN 700 may update parameters so that the final loss information determined based on the structure loss information 810, the complexity loss information 820, and the quality loss information 830 is reduced or minimized. In addition, the second DNN 300 may update parameters so that the quality loss information 830 is reduced or minimized.

[0210] The final loss information for training the first DNN 700 and the second DNN 300 may be determined according to Equation 1 below.

[0211] [Equation 1]

[0212] LossDS = a×structural loss information + b×complexity loss information + c×quality loss information

[0213] LossUS=d×quality loss information

[0214] In Equation 1, LossDS indicates final loss information to be reduced or minimized to train the first DNN 700, and LossUS indicates final loss information to be reduced or minimized to train the second DNN 300. In addition, coefficients a, b, c, and d may be predetermined weights.

[0215] In other words, the first DNN 700 updates the parameters so that the LossDS of Equation 1 decreases, and the second DNN 300 updates the parameters so that the LossUS decreases. When the parameters of the first DNN 700 are updated according to the LossDS derived during training, the first training image 802 obtained based on the updated parameters becomes different from the previous first training image 802 obtained based on the parameters that are not updated, and therefore, the third training image 804 also becomes different from the previous third training image 804. When the third training image 804 becomes different from the previous third training image 804, the quality loss information 830 is also re-determined, and the second DNN 300 updates the parameters accordingly. When the quality loss information 830 is re-determined, the LossDS is also re-determined, and the first DNN 700 updates the parameters according to the re-determined LossDS. In other words, the update of the parameters of the first DNN 700 leads to the update of the parameters of the second DNN 300, and the update of the parameters of the second DNN 300 leads to the update of the parameters of the first DNN 700. Therefore, because the first DNN 700 and the second DNN 300 are jointly trained by sharing the quality loss information 830, the parameters of the first DNN 700 and the parameters of the second DNN 300 can be jointly optimized.

[0216] Referring to Equation 1, it may be assumed that LossUS is determined according to the quality loss information 830 , but this is only an example, and LossUS may be determined based on at least one of the structure loss information 810 and the complexity loss information 820 , as well as the quality loss information 830 .

[0217] In the above, it has been described that the AI ​​amplifier 234 of the AI ​​decoding device 200 and the AI ​​reducer 612 of the AI ​​encoding device 600 store multiple DNN setting information, and now a method of training each of the multiple DNN setting information stored in the AI ​​amplifier 234 and the AI ​​reducer 612 will be described.

[0218] As described with reference to Equation 1, the first DNN 700 updates parameters by taking into account the similarity between structural information of the first training image 802 and structural information of the original training image 801 (structural loss information 810), the bit rate of image data obtained as a result of performing the first encoding on the first training image 802 (complexity loss information 820), and the difference between the third training image 804 and the original training image 801 (quality loss information 830).

[0219] Specifically, the parameters of the first DNN 700 can be updated to obtain a first training image 802 having similar structural information to the original training image 801 and to obtain image data having a small bit rate when the first encoding is performed on the first training image 802, and at this time, the second DNN 300 that performs AI amplification on the first training image 802 obtains a third training image 804 similar to the original training image 801.

[0220] The parameters of the first DNN 700 may be optimized by adjusting the weights a, b, and c of Equation 1. For example, when the weight b is set to a high value, the parameters of the first DNN 700 may be updated to prioritize the low bit rate over the high quality of the third training image 804. In addition, when the weight c is set to a high value, the parameters of the first DNN 700 may be updated to prioritize the high quality of the third training image 804 over the high bit rate or to maintain the structural information of the original training image 801.

[0221] In addition, the parameters of the first DNN 700 may be optimized according to the type of codec used to perform the first encoding on the first training image 802. This is because the second training image to be input to the second DNN 300 may vary according to the type of codec.

[0222] In other words, the parameters of the first DNN 700 and the parameters of the second DNN 300 may be jointly updated based on the weights a, b, and c and the type of codec used to perform the first encoding on the first training image 802. Therefore, when the first DNN 700 and the second DNN 300 are trained after the weights a, b, and c are each determined to be a specific value and the type of codec is determined to be a specific type, the parameters of the first DNN 700 and the parameters of the second DNN 300 may be jointly optimized.

[0223] In addition, when the first DNN 700 and the second DNN 300 are trained after changing the weights a, b, and c and the type of codec, the parameters of the first DNN 700 and the parameters of the second DNN 300 may be jointly optimized. In other words, when the first DNN 700 and the second DNN 300 are trained while changing the values ​​of the weights a, b, and c and the type of codec, a plurality of pieces of DNN setting information that are jointly trained with each other may be determined in the first DNN 700 and the second DNN 300.

[0224] As mentioned above Figure 5 As described, a plurality of pieces of DNN setting information of the first DNN 700 and the second DNN 300 may be mapped to information related to the first image. In order to set such a mapping relationship, the first training image 802 output from the first DNN 700 may be first encoded via a specific codec according to a specific bit rate, and the second training image obtained by performing a first decoding on a bit stream obtained as a result of performing the first encoding may be input to the second DNN 300. In other words, by training the first DNN 700 and the second DNN 300 after setting an environment such that the first encoding is performed on the first training image 802 of a specific resolution via a specific codec according to a specific bit rate, a pair of DNN setting information mapped to the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained as a result of performing the first encoding on the first training image 802 may be determined. By differently changing the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained according to the first encoding of the first training image 802, the mapping relationship between the multiple DNN setting information of the first DNN 700 and the second DNN 300 and the multiple information related to the first image can be determined.

[0225] Fig.10 is a diagram for describing the training process of the training device 1000 on the first DNN 700 and the second DNN.

[0226] Reference Fig. 9 The training of the described first DNN 700 and the second DNN 300 may be performed by the training device 1000. The training device 1000 includes the first DNN 700 and the second DNN 300. The training device 1000 may be, for example, the AI ​​encoding device 600 or a separate server. The DNN setting information of the second DNN 300 obtained as a result of the training is stored in the AI ​​decoding device 200.

[0227] Reference Fig.10In operation S840 and operation S845, the training device 1000 initially sets the DNN setting information of the first DNN 700 and the second DNN 300. Therefore, the first DNN 700 and the second DNN 300 may operate according to the initial DNN setting information. The DNN setting information may include information about at least one of the number of convolutional layers included in the first DNN 700 and the second DNN 300, the number of filter kernels for each convolutional layer, the size of the filter kernel for each convolutional layer, or one or more parameters of each filter kernel.

[0228] In operation S850, the training apparatus 1000 inputs the original training image 801 into the first DNN 700. The original training image 801 may include at least one frame included in a still image or a moving image.

[0229] In operation S855, the first DNN 700 processes the original training image 801 according to the initial DNN setting information and outputs a first training image 802 obtained by performing AI reduction on the original training image 801. Fig.10 In the embodiment, the first training image 802 output from the first DNN 700 is directly input to the second DNN 300, or the first training image 802 output from the first DNN 700 may be input to the second DNN 300 by the training device 1000. In addition, the training device 1000 may perform first encoding and first decoding on the first training image 802 via a specific codec, and then provide the second training image to the second DNN 300 as an input.

[0230] In operation S860, the second DNN 300 processes the first training image 802 or the second training image according to the initial DNN setting information, and outputs a third training image 804 obtained by performing AI up-scaling on the first training image 802 or the second training image.

[0231] In operation S865 , the training apparatus 1000 calculates complexity loss information 820 based on the first training image 802 .

[0232] In operation S870 , the training apparatus 1000 calculates structure loss information 810 by comparing the reduced training image 803 with the first training image 802 .

[0233] In operation S875 , the training apparatus 1000 calculates quality loss information 830 by comparing the original training image 801 with the third training image 804 .

[0234] In operation S880 , the initially set DNN setting information is updated via a back propagation process based on the final loss information. The training apparatus 1000 may calculate the final loss information for training the first DNN 700 based on the complexity loss information 820 , the structure loss information 810 , and the quality loss information 830 .

[0235] In operation S885, the second DNN 300 updates the DNN setting information through a back propagation process based on the quality loss information 830 or the final loss information. The training apparatus 1000 may calculate the final loss information for training the second DNN 300 based on the quality loss information 830.

[0236] Then, the training device 1000, the first DNN 700, and the second DNN 300 may repeat operations S850 to S885 until the final loss information is minimized, and the DNN setting information is generated. During each repetition, the first DNN 700 and the second DNN 300 operate according to the DNN setting information updated in the previous operation.

[0237] Table 1 below shows effects when the original image 105 is AI-encoded and AI-decoded according to an embodiment of the present disclosure and when the original image 105 is encoded and decoded via HEVC.

[0238]

Table 1

[0239]

[0240] As shown in Table 1, although the subjective image quality when AI encoding and AI decoding are performed on content including 300 frames of 8K resolution is higher than when encoding and decoding are performed via HEVC, the bit rate is reduced by at least 50%.

[0241] Fig.11 1 is a diagram of an apparatus 20 for performing AI downscaling on an original image 105 and an apparatus 40 for performing AI upscaling on a second image 135 .

[0242] Device 20 receives original image 105 and provides image data 25 and AI data 30 to device 40 by using AI downsizer 1124 and transform-based encoder 1126. According to an embodiment, image data 25 corresponds to Figure 1 The AI ​​data 30 corresponds to the image data Figure 1 In addition, according to an embodiment, the transform-based encoder 1126 corresponds to Figure 7 The first encoder 614, and the AI ​​reducer 1124 corresponds to Figure 7 AI reducer 612.

[0243] The device 40 receives the AI ​​data 30 and the image data 25, and obtains the third image 145 by using the transform-based decoder 1146 and the AI ​​amplifier 1144. According to an embodiment, the transform-based decoder 1146 corresponds to Figure 2 The first decoder 232 and the AI ​​amplifier 1144 correspond to Figure 2 AI amplifier 234.

[0244] According to an embodiment, the device 20 includes a CPU, a memory, and a computer program including instructions. The computer program may be stored in the memory. According to an embodiment, the device 20 executes the reference to the computer program according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, reference will be made to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0245] According to an embodiment, the device 40 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the device 40 executes the reference to the computer program according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, reference will be made to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0246] exist Fig.11 , the configuration controller 1122 receives at least one input value 10. According to an embodiment, the at least one input value 10 may include at least one of a target resolution for each of the AI ​​downsizer 1124 and the AI ​​upsizer 1144, a bit rate of the image data 25, a bit rate type of the image data 25 (e.g., a variable bit rate type, a constant bit rate type, or an average bit rate type), or a codec type for the transform-based encoder 1126. The at least one input value 10 may include a value pre-stored in the device 20 or a value input from a user.

[0247] The configuration controller 1122 controls the operations of the AI ​​downsizer 1124 and the transform-based encoder 1126 based on the received input value 10. According to an embodiment, the configuration controller 1122 obtains DNN setting information for the AI ​​downsizer 1124 according to the received input value 10, and configures the AI ​​downsizer 1124 using the obtained DNN setting information. According to an embodiment, the configuration controller 1122 may send the received input value 10 to the AI ​​downsizer 1124, and the AI ​​downsizer 1124 may obtain DNN setting information for performing AI downsizing on the original image 105 based on the received input value 10. According to an embodiment, the configuration controller 1122 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI downsizing can be applied and tone mapping information of a high dynamic range (HDR)) to the AI ​​downsizer 1124 together with the input value 10, and the AI ​​downsizer 1124 may obtain the DNN setting information in consideration of the input value 10 and the additional information. According to an embodiment, the configuration controller 1122 sends at least a portion of the received input value 10 to the transform-based encoder 1126, and the transform-based encoder 1126 performs a first encoding on the first image 115 by a bit rate of a specific value, a specific type of bit rate, and a specific codec.

[0248] The AI ​​reducer 1124 receives the original image 105 and performs a reference Figure 1 , Figure 7 , Figure 8 , Fig. 9 and Fig.10 The operations are described to obtain the first image 115 .

[0249] According to an embodiment, the AI ​​data 30 is provided to the device 40. The AI ​​data 30 may include at least one of resolution difference information between the original image 105 and the first image 115 or information related to the first image 115. The resolution difference information may be determined based on a target resolution difference of the input value 10. The information related to the first image 115 may be determined based on at least one of a target bit rate, a bit rate type, or a codec type. According to an embodiment, the AI ​​data 30 may include parameters used during AI upscaling. The AI ​​data 30 may be provided to the device 40 from the AI ​​downsizer 1124.

[0250] Image data 25 is obtained as the original image 105 is processed by a transform-based encoder 1126 and sent to the device 40. The transform-based encoder 1126 may process the first image 115 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or VA1.

[0251] The configuration controller 1142 controls the operation of the AI ​​amplifier 1144 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 obtains DNN setting information for the AI ​​amplifier 1144 according to the received AI data 30, and configures the AI ​​amplifier 1144 according to the obtained DNN setting information. According to an embodiment, the configuration controller 1142 may send the received AI data 30 to the AI ​​amplifier 1144, and the AI ​​amplifier 1144 may obtain DNN setting information for performing AI amplification on the second image 135 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI amplification is applied and tone mapping information of HDR) to the AI ​​amplifier 1144 together with the AI ​​data 30, and the AI ​​amplifier 1144 may obtain the DNN setting information based on the AI ​​data 30 and the additional information. According to an embodiment, the AI ​​amplifier 1144 may receive AI data 30 from the configuration controller 1142, receive at least one of prediction mode information, motion information, or quantization parameter information from the transform-based decoder 1146, and obtain DNN setting information based on at least one of the prediction mode information, motion information, and quantization parameter information and the AI ​​data 30.

[0252] The transform-based decoder 1146 may process the image data 25 to reconstruct the second image 135. The transform-based decoder 1146 may process the image data 25 according to MPEG-2, H.264 AVC, MPEG-4, H.265 / HEVC, VC-1, VP8, VP9, ​​or AV1.

[0253] The AI ​​amplifier 1144 may obtain the third image 145 by performing AI amplification on the second image 135 provided from the transform-based decoder 1146 based on the set DNN setting information.

[0254] The AI ​​reducer 1124 may include a first DNN, and the AI ​​amplifier 1144 may include a second DNN, and according to an embodiment, according to the reference Fig. 9 and Fig.10 The training method described is to train DNN setting information for a first DNN and a second DNN.

[0255] Fig.12 shows a convolutional layer 1200 according to an embodiment of the present disclosure, and Fig.13 The convolutional layer 1300 according to an embodiment of the present disclosure is shown. Now referring to Fig.12 and Fig.13 The structure of the convolution layer 1200 and DNN setting information about parameters according to an embodiment of the present disclosure are described. Fig.12 and Fig.13 The structure of the convolutional layer 1200 is described to determine Figure 3 The convolutional layers 310, 330, 350 and 360 of the second DNN 300 are Figure 8 Convolutional layer 710, convolutional layer 730, and convolutional layer 750 of the first DNN 700.

[0256] Fig.12 The structure of the convolution layer 1200 according to an embodiment of the present disclosure is shown.

[0257] The convolution layer 1200 is composed of one or more filter kernels. Each filter kernel includes parameters for generating elements for output channels from elements for corresponding input channels. The parameters of the filter kernel include weights and biases.

[0258] The weights of the filter kernel are the parameters required to compute the weighted sum of the elements with respect to the input channels. For example, Figure 4 F1 to F9 in the filter kernel 430 of the second image 135 are weights of the filter kernel 430. I1, I2, I3, I8, I9, I10, I15, I16, and I17 in the second image 135 are elements for the input channel. M1 in the feature map 450 for the output channel may be determined as a weighted sum of I1, I2, I3, I8, I9, I10, I15, I16, and I17 in the second image 135 based on the weights F1 to F9 of the filter kernel 430.

[0259] The bias of the filter kernel is a parameter for adjusting the elements output to the output channel. Each element for the output channel can be determined by adding the bias to the weighted sum of the elements for the input channel determined according to the weight of the filter kernel. For example, M1 in the feature map 450 as an element for the output channel can be determined by adding the bias to the weighted sum of I1, I2, I3, I8, I9, I10, I15, I16, and I17 in the second image 135.

[0260] as Figure 3 The second convolution layer 330 of the convolution layer 1200 can output the values ​​obtained by filtering the elements received from four input channels (i.e., the first input channel 1202, the second input channel 1204, the third input channel 1206, and the fourth input channel 1208) using four filter kernels (i.e., the first filter kernel 1210, the second filter kernel 1220, the third filter kernel 1230, and the fourth filter kernel 1240) to four output channels (i.e., the first output channel 1260, the second output channel 1262, the third output channel 1264, and the fourth output channel 1266), respectively.

[0261] The first filter kernel 1210 may include a three-dimensional (3D) weight matrix and a first bias 1250. For example, the size of the 3D weight matrix of the first filter kernel 1210 may be represented as 3×3×4. The notation 3×3×4 represents the row, column, and depth of the filter kernel 1210, respectively. According to an embodiment of the present disclosure, the sizes of the 3D weight matrices of the first filter kernel 1210, the second filter kernel 1220, the third filter kernel 1230, and the fourth filter kernel 1240 may be determined differently. For ease of description, Fig.12 The 3D weight matrix of the first filter kernel 1210 is shown to include four two-dimensional (2D) weight matrices of size 3×3. Similarly, Fig.12 It is also shown that the 3D weight matrix of each of the second filter kernel 1220, the third filter kernel 1230 and the fourth filter kernel 1240 includes four 2D weight matrices of size 3×3.

[0262] The 2D weight matrix 1212 of the first depth of the first filter kernel 1210 is used to filter the elements for the first input channel 1202. In addition, the 2D weight matrix 1214 of the second depth, the 2D weight matrix 1216 of the third depth, and the 2D weight matrix 1218 of the fourth depth of the first filter kernel 1210 are used to filter the elements for the second input channel 1204, the third input channel 1206, and the fourth input channel 1208, respectively.

[0263] Similarly, the 2D weight matrix 1222, 2D weight matrix 1224, 2D weight matrix 1226, and 2D weight matrix 1228 of the depth of the second filter kernel 1220 are respectively used to filter the elements for the first input channel 1202, the second input channel 1204, the third input channel 1206, and the fourth input channel 1208. In addition, the 2D weight matrix 1232, 2D weight matrix 1234, 2D weight matrix 1236, and 2D weight matrix 1238 of the depth of the third filter kernel 1230 are respectively used to filter the elements for the first input channel 1202, the second input channel 1204, the third input channel 1206, and the fourth input channel 1208. In addition, the depth 2D weight matrix 1242, 2D weight matrix 1244, 2D weight matrix 1246 and 2D weight matrix 1248 of the fourth filter kernel 1240 are used to filter the elements of the first input channel 1202, the second input channel 1204, the third input channel 1206 and the fourth input channel 1208, respectively.

[0264] The first filter kernel 1210 includes a first bias 1250. Similarly, the second filter kernel 1220, the third filter kernel 1230, and the fourth filter kernel 1240 include a second bias 1252, a third bias 1254, and a fourth bias 1256, respectively. The first bias 1250, the second bias 1252, the third bias 1254, and the fourth bias 1256 are used to adjust the elements for the first output channel 1260, the second output channel 1262, the third output channel 1264, and the fourth output channel 1266.

[0265] A first weighted sum for the elements of the first input channel 1202 is determined according to the weights in the 2D weight matrix 1212 of the first depth of the first filter kernel 1210. A second weighted sum for the elements of the second input channel 1204 is determined according to the weights in the 2D weight matrix 1214 of the second depth of the first filter kernel 1210. In addition, a third weighted sum for the elements of the third input channel 1206 is determined according to the weights in the 2D weight matrix 1216 of the third depth of the first filter kernel 1210. A fourth weighted sum for the elements of the first input channel 1202 is determined according to the weights in the 2D weight matrix 1218 of the fourth depth of the first filter kernel 1210. In addition, an element for the first output channel 1260 may be determined by adding the first bias 1250 to the sum of the first weighted sum to the fourth weighted sum.

[0266] Similar to the above method of determining the elements for the first output channel 1260, the elements for the second output channel 1262, the third output channel 1264, and the fourth output channel 1266 may be determined. In detail, the elements for the second output channel 1262 may be determined by filtering the elements for the first input channel 1202, the second input channel 1204, the third input channel 1206, and the fourth input channel 1208 according to the four 2D weight matrices 1222, 1224, 1226, and 1228 of the second filter kernel 1220 and the second bias 1252. Similarly, the elements for the third output channel 1264 may be determined by filtering the elements for the first input channel 1202, the second input channel 1204, the third input channel 1206, and the fourth input channel 1208 according to the four 2D weight matrices 1232, 1234, 1236, and 1238 of the third filter kernel 1230 and the third bias 1254. In addition, elements for the fourth output channel 1266 can be determined by filtering the elements for the first input channel 1202, the second input channel 1204, the third input channel 1206 and the fourth input channel 1208 according to the four 2D weight matrices 1242, 1244, 1246 and 1248 of the fourth filter kernel 1240 and the fourth bias 1256.

[0267] Although referenced Fig.12 The convolution layer 1200 is described as having four input channels (ie, the first input channel 1202, the second input channel 1204, the third input channel 1206, and the fourth input channel 1208), but the number of input channels may be configured differently according to embodiments. Fig.12 The convolution layer 1200 is described as having four output channels (i.e., a first output channel 1260, a second output channel 1262, a third output channel 1264, and a fourth output channel 1266), but according to an embodiment of the present disclosure, the number of output channels may be configured differently. The number of filter kernels included in the convolution layer 1200 is determined according to the number of output channels. In addition, although for convenience, Fig.12 It is shown that the first filter kernel 1210, the second filter kernel 1220, the third filter kernel 1230 and the fourth filter kernel 1240 all have a size of 3×3 (3 rows and 3 columns), but according to an embodiment of the present disclosure, the number of rows and columns in each of the first filter kernel 1210, the second filter kernel 1220, the third filter kernel 1230 and the fourth filter kernel 1240 can be set or arranged differently.

[0268] Fig.13 The embodiment according to the present disclosure is shown Fig.12 The weights and biases of the first filter kernel 1210 and the second filter kernel 1220 in the convolution layer 1200 are as follows. Figure 4 In addition to the description of the filter kernel 430, reference is now also made to Fig.13 Describes the functionality of a filter kernel. Fig.12 The reference numerals 12XX and Fig.13 The figures correspond to the reference numerals 13XX. Fig.13 Part of the reference numerals in the figure may not be Fig.12 Any reference numerals in the accompanying drawings correspond to those in the accompanying drawings.

[0269] For the convenience of description, Fig.13 Only weights and biases are shown for the first filter kernel 1210 and the second filter kernel 1220. However, similar to the first filter kernel 1210 and the second filter kernel 1220, the third filter kernel 1230 and the fourth filter kernel 1240 may each include weights and biases.

[0270] Reference Fig.13 , the size of the 2D data array for each of the first input channel 1202, the second input channel 1204, the third input channel 1206, and the fourth input channel 1208 is 3×3. In addition, the size of the 2D data array for each of the first output channel 1260 and the second output channel 1262 is 3×3. Although for convenience, Fig.13The size of the 2D data array of each input channel or each output channel is shown to be 3×3, but according to an embodiment of the present disclosure, the size of the 2D data array of each of the first input channel 1202, the second input channel 1204, the third input channel 1206 and the fourth input channel 1208 and the size of the 2D data array of each of the first output channel 1260 and the second output channel 1262 can be configured differently.

[0271] As in Fig.13 As shown in , padding (such as zero padding or mirror padding) may be performed on the top, bottom, right, and left sides of the 2D data array for each of the first input channel 1202, the second input channel 1204, the third input channel 1206, and the fourth input channel 1208. Zero padding refers to a process of adding an element with a value of 0 to the 2D data array. Mirror padding is a process of adding an element with the same value as an element located in the boundary of the 2D data array to the 2D data array. Fig.13 , zero padding may be performed on the 2D data array for each of the first input channel 1202 , the second input channel 1204 , the third input channel 1206 , and the fourth input channel 1208 to generate a 2D data array having a size of 5×5.

[0272] The upper left element 1380 in the 2D data array for the first output channel 1360 is determined by filtering the 3×3 2D data array 1370, 1372, 1374, and 1376 at the upper left corner of the 5×5 zero-padded 2D data arrays for the first input channel 1302, the second input channel 1304, the third input channel 1306, and the fourth input channel 1308, respectively. Fig.12 The position of each element in the 2D data array 1370, 2D data array 1372, 2D data array 1374, and 2D data array 1376 corresponding to the element corresponds to the position of the 2D data array 1370, 2D data array 1372, 2D data array 1374, and 2D data array 1376. Fig.12 The positions of 2D data array 1370, 2D data array 1372, 2D data array 1374 and 2D data array 1376 corresponding to the upper left element 1380 are determined based on the position of the upper left element 1380 in the 2D data array (shown in FIG. 1 ).

[0273] The element [0,0,0; 0,0,1; 0,0,1] in the 2D data array 1370 is element-wise multiplied by the element [1,0,-1; 1,0,-1; 1,0,-1] of the 2D weight matrix 1312 of the first depth of the first filter kernel 1310. Then, -2, which is the sum of all element-wise products, is determined as a value obtained by filtering the 2D data array 1370. (0×1+0×0+0×(-1)+0×1+0×0+1×(-1)+0×1+0×0+1×(-1)=-2).

[0274] According to the above filtering method, the value obtained by filtering the 2D data array 1372 using the 2D weight matrix 1314 of the second depth of the first filter kernel 1310 is determined to be -2. Similarly, the value obtained by filtering the 2D data array 1374 according to the 2D weight matrix 1316 of the third depth of the first filter kernel 1310 is determined to be 0. Then, the value obtained by filtering the 2D data array 1376 according to the 2D weight matrix 1318 of the fourth depth of the first filter kernel 1310 is determined to be -1.

[0275] Finally, -4 ((-2)+(-2)+0+(-1)+1=-4), which is a result obtained by adding the first bias 1350 to the sum of all values ​​obtained by filtering the 2D data array 1370, the 2D data array 1372, the 2D data array 1374, and the 2D data array 1376, is determined as the value of the upper left element 1380 for the first output channel 1260. Then, the values ​​of the remaining elements in the 2D data array for the first output channel 1260 are determined in the same manner as the value of the upper left element 1380. In addition, the elements of the 2D data array for the second output channel 1262 are determined in the same manner as the elements of the 2D data array for the first output channel 1260.

[0276] In order to improve the performance of AI amplification, the above references can be optimized according to image characteristics, network status, content provider selection, etc. Fig.12 and Fig.13 The DNN of the convolution layer 1200 described above. Therefore, the number of convolution layers included in the DNN and the structure and parameters of each convolution layer in the DNN may be determined differently according to image characteristics, network status, selection of content providers, etc.

[0277] Reference Fig.12 and Fig.13 The structure and parameters of the convolutional layer 1200 described above can be applied to Figure 3 The first convolution layer 310, the second convolution layer 330 and the third convolution layer 350 in the second DNN 300 and Figure 8The first convolutional layer 710, the second convolutional layer 730 and the third convolutional layer 750 in the first DNN 700.

[0278] Now refer to Figures 14 to 16 The structure of AI encoded data including image data 1400 and AI data 1440 is described.

[0279] Fig.14 The structures of the image data 1400 and the AI ​​data 1440 and the corresponding relationship between the image data 1400 and the AI ​​data 1440 are shown.

[0280] Reference Fig.14 Describes the video-frame-group-frame hierarchy of image data 1400. Fig.14 The video 1402 is a data unit including all consecutive frames of the image data 1400. Parameter information of the video parameter set may be applied to all frames in the video 1402. The video parameter set is included in the video header 1404.

[0281] Video 1402 may include multiple frame groups (or coded video sequences). A frame group is a data unit composed of one or more continuous frames of parameter information of a shared frame group parameter set (or sequence parameter set). The frame group parameter set may be included in a frame group header. For example, the frame group parameter set of a first frame group 1410 may be included in a first frame group header 1412. In addition, the frame group parameter set of a second frame group 1414 may be included in a second frame group header 1416.

[0282] Each frame group can be independently encoded without reference to other frame groups. Each frame group includes an instant decoding refresh (IDR) frame or an intra-frame random access point (IRAP) frame encoded without reference to other frames. All remaining frames in the frame group are encoded with reference to the IDR frame (or IRAP frame). Therefore, the first frame group 1410 is independently encoded without reference to other frame groups in the video 1402. The first frame 1420 as the first coded frame in the first frame group 1410 is an IDR frame (or IRAP frame). All remaining frames in the first frame group 1410 including the second frame 1430 are encoded with reference to the first frame 1420.

[0283] Each frame represents a still image included in the video. The frame header of the frame may include a frame parameter set, wherein the frame parameter set includes parameter information applied to the frame. For example, the first frame header 1422 of the first frame 1420 may include a frame parameter set applied to the first frame 1420. Similarly, the second frame header 1432 of the second frame 1430 may include a frame parameter set applied to the second frame 1430.

[0284] Reference Fig.14A method of classifying AI data 1440 according to the above-mentioned video-frame group-frame hierarchical structure is described. AI data 1440 may be classified into video AI data 1442, frame group AI data 1450, and frame AI data 1460 according to the type of data unit to which AI data 1440 is applied. Video AI data 1442 refers to AI data applied to a video, and frame group AI data 1450 refers to AI data applied to a frame group. In addition, frame AI data 1460 refers to AI data applied to a frame.

[0285] The video AI data 1442 corresponds to the video header 1404. Therefore, the video AI data 1442 may be decoded in parallel with the video header 1404. In addition, the video AI data 1442 may be decoded before decoding the video header 1404. Alternatively, the video AI data 1442 may be decoded after decoding the video header 1404.

[0286] The frame group AI data 1450 corresponds to the frame group header. Fig.14 , the first frame group AI data 1452 corresponds to the first frame group header 1412. In addition, the second frame group AI data 1454 corresponds to the second frame group header 1416. The first frame group AI data 1452 and the second frame group AI data 1454 may be decoded in parallel with the first frame group header 1412 and the second frame group header 1416, respectively. Optionally, for example, the first frame group AI data 1452 and the second frame group AI data 1454 may be decoded immediately before the first frame group header 1412 and the second frame group header 1416 are decoded, respectively. Optionally, the first frame group AI data 1452 and the second frame group AI data 1454 may be decoded immediately after the first frame group header 1412 and the second frame group header 1416 are decoded, respectively.

[0287] The frame AI data 1460 corresponds to the frame header. Fig.14 , the first frame AI data 1462 corresponds to the first frame header 1422. In addition, the second frame AI data 1464 corresponds to the second frame header 1432. The first frame AI data 1462 and the second frame AI data 1464 may be decoded in parallel with the first frame header 1422 and the second frame header 1432, respectively. Optionally, for example, the first frame AI data 1462 and the second frame AI data 1464 may be decoded immediately before the first frame header 1422 and the second frame header 1432 are decoded, respectively. Optionally, the first frame AI data 1462 and the second frame AI data 1464 may be decoded immediately after the first frame header 1422 and the second frame header 1432 are decoded, respectively.

[0288] According to an embodiment of the present disclosure, Figure 7The data processor 632 may generate a file corresponding to the image data 1400 separately from a file corresponding to the AI ​​data 1440. Therefore, the communicator 634 transmits the AI ​​encoded data to the image data 1400 as two separate files. Figure 2 Communicator 212. Therefore, communicator 634 may send files corresponding to image data 1400 and files corresponding to AI data 1440 via different communication channels. Communicator 634 may also sequentially send files corresponding to image data 1400 and files corresponding to AI data 1440 at different times. In the present disclosure, a file refers to a collection of data stored in a memory. A video file is a collection of image data stored in a memory, and the image data may be implemented in the form of a bitstream.

[0289] In the above embodiments of the present disclosure, according to the reference Fig.14 The corresponding relationship between the described image data 1400 and the AI ​​data 1440 may be such that the file corresponding to the AI ​​data 1440 may be decoded according to the process of decoding the file corresponding to the image data 1400. In addition, since the file (or bitstream) corresponding to the image data 1400 and the file corresponding to the AI ​​data 1440 are separated, the file corresponding to the image data 1400 and the file corresponding to the AI ​​data 1440 may each include information about synchronization of the two files.

[0290] According to an embodiment of the present disclosure, Figure 7 The data processor 632 may generate the AI ​​coded data as a single file including both the image data 1400 and the AI ​​data 1440. The communicator 634 transmits the AI ​​coded data as a single file to the Figure 2 The communicator 212 in FIG.

[0291] According to an embodiment of the present disclosure, the AI ​​data 1440 may not be embedded in the image data 1400, but may be separated from the image data 1400. Therefore, since the AI ​​encoded data is configured as a single file, but the AI ​​data 1440 is separated from the image data 1400, the AI ​​data 1440 and / or the image data 1400 may contain information about synchronization of the AI ​​data 1440 and the image data 1400. Optionally, the second DNN 300 required for AI amplification may be determined based on the AI ​​data 1440, and then the image data 1400 may be decoded.

[0292] Optionally, AI data 1440 may also be embedded in image data 1400. For example, video AI data 1442 may be embedded in video header 1404. In this case, video header 1404 may include video AI data 1442 and a video parameter set. Optionally, video AI data 1442 may be embedded in a single file independently of video header 1404 to be placed before or after video header 1404. Therefore, decoding of video AI data 1442 may be performed simultaneously with decoding of video header 1404, decoding of video AI data 1442 may be performed immediately before decoding of video header 1404, or decoding of video AI data 1442 may be performed immediately after decoding of video header 1404.

[0293] Similarly, the first frame group header 1412 may include the first frame group AI data 1452 and the frame group parameter set. Alternatively, the first frame group AI data 1452 may be embedded in a single file independently of the first frame group header 1412 to be placed before or after the first frame group header 1412. Therefore, the decoding of the first frame group AI data 1452 may be performed simultaneously with the decoding of the first frame group header 1412, or the decoding of the first frame group AI data 1452 may be performed before or after the decoding of the first frame group header 1412.

[0294] Similarly, the first frame header 1422 may include the first frame AI data 1462 and the frame parameter set. Optionally, the first frame AI data 1462 may be embedded in a single file independently of the first frame header 1422 to be placed before or after the first frame header 1422. Therefore, the decoding of the first frame AI data 1462 may be performed simultaneously with the decoding of the first frame header 1422, the decoding of the first frame AI data 1462 may be performed before the decoding of the first frame header 1422, or the decoding of the first frame AI data 1462 may be performed after the decoding of the first frame header 1422.

[0295] Fig.15 FIG. 15 shows AI encoded data when the AI ​​data 1520 according to an embodiment of the present disclosure is separated from the image data in a single file. Fig.15 As shown in , the AI ​​encoded data 1500 is included in a video file 1500 in a predefined container format. The predefined container format may be MP4, Audio Video Interleave (AVI), Matroska (MKV), Flash Video (FLV), etc.

[0296] The video file 1500 includes a metadata box 1510 and a media data box 1530. The metadata box 1510 includes information about the media data included in the media data box 1530. For example, the metadata box 1510 may include pieces of information about the type of media data, the type of codec used to encode the media data, and the playback time of the media.

[0297] The metadata box 1510 may include AI data 1520. The AI ​​data 1520 is encoded according to an encoding method provided in a predefined container format, and then the encoded AI data 1520 is stored in the metadata box 1510. The AI ​​data 1520 stored in the metadata box 1510 may be configured independently of image data.

[0298] The AI ​​data 1520 includes video AI data 1522, frame group AI data 1524, and frame AI data 1526. The video AI data 1522, frame group AI data 1524, and frame AI data 1526 are respectively arranged to correspond to the video header 1532, the frame group header 1536, and the frame header 1540.

[0299] The AI ​​data 1520 in the metadata box 1510 may be decoded before the image data in the media data box 1530. Therefore, the second DNN 300 required for AI upscaling may be determined based on the AI ​​data 1520 before the image data is decoded. The second DNN 300 may be determined for the entire video (i.e., all frames in the plurality of frames in the video). Alternatively, for adaptive AI upscaling based on each frame group, the second DNN 300 may be determined differently for each frame group. Alternatively, for adaptive AI upscaling based on each frame, the second DNN 300 may be determined differently for each frame.

[0300] The AI ​​data 1520 in the metadata box 1510 may be decoded together with the image data in the media data box 1530. The decoding of the video AI data 1522 may be performed immediately before or after the decoding of the video header 1532. Optionally, the video AI data 1522 may be decoded in parallel with the video header 1532. Similarly, the decoding of the frame group AI data 1524 may be performed immediately before or after the decoding of the frame group header 1536. Optionally, the frame group AI data 1524 may be decoded in parallel with the frame group header 1536. In addition, the decoding of the frame AI data 1526 may be performed immediately before or after the decoding of the frame header 1540. Optionally, the frame AI data 1526 may be decoded in parallel with the frame header 1540.

[0301] Although for ease of description, Fig.15The video file 1500 is shown to include one metadata box 1510 and one media data box 1530, but the video file 1500 may include two or more metadata boxes 1510 and two or more media data boxes. In this case, two or more image data segments obtained by dividing the image data into units at preset time intervals may be stored in two or more media data boxes. In addition, a plurality of pieces of information about the two or more image data segments stored in the two or more media data boxes may be included in the two or more metadata boxes, respectively. In addition, the two or more metadata boxes may each include AI data.

[0302] Although referenced Fig.15 It is described that the AI ​​data 1520 is included in the metadata box 1510, but according to an embodiment of the present disclosure, the AI ​​data 1520 may be included in the media data box 1530. In this case, the AI ​​data 1520 may be configured independently of the image data within the media data box 1530.

[0303] Fig.16 AI encoding data in a case where AI data according to an embodiment of the present disclosure is embedded in image data in a single file is shown.

[0304] The video file 1600 includes a metadata box 1610 and a media data box 1620. Since the AI ​​data is embedded in the image data, the metadata box 1610 does not include the AI ​​data. Instead, the media data box 1620 includes the image data in which the AI ​​data is embedded. The AI ​​data may be encoded according to a video codec for the image data. Examples of the video codec for the image data may include H.264, HEVC, AVS2.0, XviD, etc.

[0305] Reference Fig.16 , the video AI data 1624 may be included in the video header 1622. Therefore, the video AI data 1624 may be decoded together with the video parameters included in the video header 1622. In addition, the frame group AI data 1634 may be included in the frame group header 1632. The frame group AI data 1634 may be decoded together with the frame group parameters included in the frame group header 1632. In addition, the frame AI data 1644 may be included in the frame header 1642. The frame AI data 1644 may be decoded together with the frame parameters included in the frame header 1642.

[0306] According to an embodiment of the present disclosure, Fig.16Differently, the video AI data 1624 may be located before or after the video header 1622. In this case, the video AI data 1624 may be decoded before or after the video parameters included in the video header 1622 are decoded. In addition, the frame group AI data 1634 may be located before or after the frame group header 1632. Therefore, the frame group AI data 1634 may be decoded before or after the frame group parameters included in the frame group header 1632 are decoded. In addition, the frame AI data 1644 may be located before or after the frame header 1642. Therefore, the frame AI data 1644 may be decoded before or after the frame parameters included in the frame header 1642 are decoded. According to an embodiment of the present disclosure, the video AI data 1624, the frame group AI data 1634, and the frame AI data 1644 are all in the form of supplemental enhancement information (SEI) messages, and are separated from the video header 1622, the frame group header 1632, and the frame header 1642, respectively.

[0307] Fig.15 The AI ​​data 1520 is shown to be included only in the metadata box 1510, while Fig.16 It is shown that the AI ​​data is embedded in the image data included in the media data box 1620. However, according to an embodiment of the present invention, a portion of the AI ​​data may be included in the metadata box 1610, while the rest of the AI ​​data may be included in the media data box 1620. For example, video AI data applied to the entire image data may be included in the metadata box 1610, while AI data applied to a frame group and a frame may be included in the media data box 1620. In addition, according to an embodiment of the present disclosure, a portion of the AI ​​data may not be included in the video file 1600 and may be transmitted separately.

[0308] Reference Figures 17 to 30 A method of adaptively determining DNN setting information of a second DNN 300 for performing AI upscaling for a video, a frame group, and / or a frame is described. In addition, a method of updating DNN setting information for a video, a frame group, and / or a frame to adaptively determine DNN setting information for a video, a frame group, and / or a frame is described. Therefore, the accuracy of AI upscaling can be improved by applying updated DNN setting information adaptively determined for a current video, a current frame group, and / or a current frame instead of default DNN setting information pre-set based on a training image.

[0309] Fig.17 is included with Figure 2 17 is a block diagram of an AI decoding device 1700 having an AI amplifier 1734 with detailed functions compared to the AI ​​amplifier 234 of the AI ​​decoding device 200 .

[0310] The communicator 1712, the parser 1714, and the outputter 1716 included in the receiver 1710 of the AI ​​decoding device 1700 may respectively perform the functions of the communicator 212, the parser 214, and the outputter 216 included in the receiver 210 of the AI ​​decoding device 200. In addition, the first decoder 1732 and the AI ​​amplifier 1734 included in the AI ​​decoder 1730 of the AI ​​decoding device 1700 may respectively perform the functions of the first decoder 232 and the AI ​​amplifier 234 included in the AI ​​decoder 230 of the AI ​​decoding device 200.

[0311] When Fig.14 When the image data and the AI ​​data are sent as separate files as shown in , the image data and the AI ​​data are received by the communicator 1712. Then, the image data and the AI ​​data are sent to the first decoder 1732 and the AI ​​amplifier 1734 respectively by the outputter 1716.

[0312] When Fig.15 When the image data and AI data shown in FIG. 1 are separated from each other in a single video file and transmitted, a video file including the image data and the AI ​​data is received by the communicator 1712. Then, the image data and the AI ​​data are extracted from the video file by the parser 1714, and the image data and the AI ​​data are transmitted to the first decoder 1732 and the AI ​​amplifier 1734, respectively, by the outputter 1716.

[0313] When Fig.16 When a video file including image data in which AI data is embedded is transmitted as shown in , the communicator 1712 receives the video file. Then, the image data in which AI data is embedded is transmitted to the first decoder 1732 by the outputter 1716. When the image data is decoded, the AI ​​data embedded in the image data is extracted by the first decoder 1732. Then, the second image 135 obtained by decoding the image data is transmitted to the AI ​​amplifier 1734.

[0314] The AI ​​amplifier 1734 included in the AI ​​decoder 1730 of the AI ​​decoding device 1700 may include a DNN controller 1736 and an image processor 1738. The DNN controller 1736 may determine video DNN setting information applied to a video, frame group DNN setting information applied to a frame group, and frame DNN setting information applied to a frame based on the AI ​​data.

[0315] The DNN controller 1736 may determine the video DNN setting information, the frame group DNN setting information, and the frame DNN setting information based on one or more preset default DNN setting information. In addition, the DNN controller 1736 may set the second DNN 300 by using the video DNN setting information, the frame group DNN setting information, and / or the frame DNN setting information. The one or more default DNN setting information may be pre-stored in the AI ​​amplifier 1734.

[0316] In addition, the DNN controller 1736 may use updated DNN setting information generated according to the DNN update information instead of the default DNN setting information. In addition, the DNN controller 1736 may determine video DNN setting information, frame group DNN setting information, and frame DNN setting information optimized according to image characteristics of the video, frame group, and frame based on the updated DNN setting information.

[0317] The DNN update information may include information for generating DNN setting information optimized for a video, a frame group, or a frame. In addition, the updated DNN setting information may include DNN setting information generated according to the DNN update information. In addition, the updated DNN is a DNN set according to the updated DNN setting information.

[0318] In addition, the image processor 1738 may perform AI upscaling on the second image 135 according to the second DNN 300 set by the DNN controller 1736 .

[0319] The AI ​​decoding device 1700 can process as shown in FIG. Figures 14 to 16 The AI ​​encoded data is sent in various forms as described.

[0320] In the following, reference will be made to Figures 18 to 22 Describes a method performed by the DNN controller 1736 to determine video DNN setting information, frame group DNN setting information and / or frame DNN setting information.

[0321] Fig.18 A DNN setting information determination method for updating DNN setting information based on each video according to an embodiment of the present disclosure is shown.

[0322] DNN controller 1736 ( Fig.17 ) determines whether the DNN setting information will be updated for the video 1830 (operation 1802). The DNN controller 1736 may determine whether the DNN setting information will be updated according to the video DNN update permission information. The video DNN update permission information indicates whether to update the DNN setting information for the video 1830.

[0323] When the DNN setting information is not updated for the video 1830, the DNN controller 1736 determines the video DNN setting information from one or more pieces of default DNN setting information (operation 1804). The DNN controller 1736 may determine the video DNN setting information from one or more pieces of default DNN setting information according to the video DNN index information. The video DNN index information indicates the DNN setting information for the video 1830 among the plurality of candidate DNN setting information.

[0324] When the DNN setting information is updated for the video 1830, the DNN controller 1736 determines the updated DNN setting information generated according to the video DNN update information as the video DNN setting information (operation 1806). The video DNN update information may include the number of layers, the number of input channels and the number of output channels, the size of the filter kernel, the weight and bias of the filter kernel, etc. related to the up-scaling DNN applied to the video 1830.

[0325] The DNN controller 1736 sets the video DNN setting information determined in operation 1804 or operation 1806 as frame group DNN setting information for the first frame group 1832 , the second frame group 1834 , the third frame group 1836 , and the like (operation 1808 ).

[0326] Fig.19 A DNN setting information determination method for updating one or more pieces of DNN setting information based on each video according to an embodiment of the present disclosure is illustrated.

[0327] DNN controller 1736 ( Fig.17 ) determines whether DNN setting information will be updated for video 1930 (operation 1902). DNN controller 1736 can determine whether DNN setting information will be updated according to video DNN update permission information.

[0328] When the DNN setting information is not updated for the video 1930, the DNN controller 1736 determines one or more video DNN setting information from one or more default DNN setting information (operation 1904). The DNN controller 1736 may determine one or more video DNN setting information from one or more default DNN setting information based on the video DNN index information.

[0329] When the DNN setting information is updated for the video 1930, the DNN controller 1736 determines one or more pieces of updated DNN setting information generated according to the video DNN update information as one or more pieces of video DNN setting information (operation 1906).

[0330] The DNN controller 1736 determines one piece of video DNN setting information among the one or more pieces of video DNN setting information determined in operation 1904 or operation 1906 as frame group DNN setting information for the current frame group (operation 1908).

[0331] The video 1930 is composed of a plurality of frame groups including a first frame group 1932, a second frame group 1934, a third frame group 1936, and the like. The current frame group is a frame group currently being decoded among the frame groups included in the video 1930. Operation 1908 is performed for each frame group in the frame group. In other words, the DNN controller 1736 may independently select a piece of video DNN setting information for each frame group from the one or more video DNN setting information. Therefore, different video DNN setting information may be selected for the first frame group 1932 and the second frame group 1934, respectively.

[0332] The DNN controller 1736 may select a piece of video DNN setting information from one or more pieces of video DNN setting information according to the frame group DNN index information. The frame group DNN index information indicates DNN setting information for a frame group from among a plurality of available candidate DNN setting information.

[0333] Fig. 20 A DNN setting information determination method for updating one or more pieces of DNN setting information based on each video and based on each frame group according to an embodiment of the present disclosure is illustrated.

[0334] DNN controller 1736 ( Fig.17 ) determines whether DNN setting information will be updated for video 2030 (operation 2002). DNN controller 1736 can determine whether DNN setting information will be updated according to video DNN update permission information.

[0335] When the DNN setting information is not updated for the video 2030, the DNN controller 1736 determines one or more video DNN setting information from one or more default DNN setting information (operation 2004). The DNN controller 1736 may determine one or more video DNN setting information from one or more default DNN setting information based on the video DNN index information.

[0336] When updating the DNN setting information for the video 2030, the DNN controller 1736 determines one or more pieces of updated DNN setting information generated according to the video DNN update information as one or more pieces of video DNN setting information (operation 2006).

[0337] Operations 2008 to 2012 are performed for each frame group. Video 2030 includes a plurality of frame groups including a first frame group 2032, a second frame group 2034, a third frame group 2036, etc. A current frame group is a frame group currently being decoded among the plurality of frame groups included in video 2030.

[0338] The DNN controller 1736 determines whether the DNN setting information will be updated for the current frame group (operation 2008). The DNN controller 1736 may determine whether the DNN setting information will be updated according to the frame group DNN update permission information. The frame group DNN update permission information indicates whether the DNN setting information will be updated for the current frame group.

[0339] When the DNN setting information is not updated for the current frame group, the DNN controller 1736 determines one piece of video DNN setting information among the one or more pieces of video DNN setting information determined in operation 2004 or operation 2006 as the frame group DNN setting information for the current frame group (operation 2010). The DNN controller 1736 may select one piece of video DNN setting information from the one or more pieces of video DNN setting information according to the frame group DNN index information.

[0340] When the DNN setting information is updated for the current frame group, the DNN controller 1736 determines the updated DNN setting information generated according to the frame group DNN update information as the frame group DNN setting information (operation 2012).

[0341] According to an embodiment of the present disclosure, the frame group DNN update information may include DNN setting information related to the structure of the second DNN 300 applied to the frame group, so that the zoom DNN can be set without referring to the video DNN setting information. The frame group DNN update information may also include DNN setting information about the parameters of the filter kernel in the second DNN 300.

[0342] According to another embodiment of the present disclosure, DNN setting information related to the structure of the second DNN 300 may be determined based on the video DNN setting information. In addition, the frame group DNN update information may include only DNN setting information about the parameters of the filter kernel to be set in the second DNN 300. Therefore, operation 2012 may include selecting one video DNN setting information from the one or more video DNN setting information. Then, the structure of the enlarged DNN may be determined based on the selected video DNN setting information. In addition, the weights and biases of the filter kernel in the enlarged DNN may be determined based on the frame group DNN update information.

[0343] Fig.21A DNN setting information determination method for updating one or more pieces of DNN setting information based on each video and each frame group according to an embodiment of the present disclosure is shown. Fig. 20 The DNN setting information determination method 2000 is different, according to Fig.21 The DNN setting information determining method 2100 can determine the DNN setting information for the frame group by referring to the DNN setting information for the previously decoded frame group.

[0344] according to Figures 18 to 20 In the DNN setting information determination method, for a frame group as a lower-level data unit of a video, video DNN setting information is referenced vertically. However, when multiple pieces of frame group DNN setting information for temporally continuous frame groups are identical or similar to each other, the size of AI data required to determine the DNN setting information can be reduced by horizontally referencing the DNN setting information between the frame groups.

[0345] The DNN controller 1736 determines whether the DNN setting information is to be updated for the video 2130 (operation 2102). The DNN controller 1736 may determine whether the DNN setting information is to be updated according to the video DNN update permission information.

[0346] When the DNN setting information is not updated for the video 2130, the DNN controller 1736 determines one or more video DNN setting information from one or more default DNN setting information (operation 2104). The DNN controller 1736 may determine one or more video DNN setting information from one or more default DNN setting information based on the video DNN index information.

[0347] When updating the DNN setting information for the video 2030, the DNN controller 1736 determines one or more updated DNN setting information generated according to the video DNN update information as one or more video DNN setting information (operation 2106).

[0348] The video 2130 includes a plurality of frame groups including a first frame group 2132, a second frame group 2134, a third frame group 2136, and the like. The frame group DNN setting information for the first frame group 2132 that is initially decoded may be determined by referring only to one or more pieces of video DNN setting information determined in operation 2104 or operation 2106. On the other hand, the frame group DNN setting information for frame groups decoded after the first frame group 2132 (such as the second frame group 2134 and the third frame group 2136) may be determined by referring to one or more pieces of video DNN setting information determined in operation 2104 or operation 2106 and the frame group DNN setting information for the previously decoded frame groups.

[0349] Operation 2108 , operation 2110 , and operation 2112 are performed for the first frame group 2132 .

[0350] The DNN controller 1736 determines whether the DNN setting information is to be updated for the first frame group 2132 (operation 2108). The DNN controller 1736 may determine whether the DNN setting information is to be updated according to the frame group DNN update permission information.

[0351] When the DNN setting information is not updated for the first frame group 213 as the current frame group, the DNN controller 1736 determines one piece of video DNN setting information among the one or more pieces of video DNN setting information determined in operation 2104 or 2106 as the frame group DNN setting information for the current frame group (operation 2110). The DNN controller 1736 may select one piece of video DNN setting information from among the one or more pieces of video DNN setting information according to the frame group DNN index information.

[0352] When the DNN setting information is updated for the current frame group, the DNN controller 1736 determines the updated DNN setting information set based on the frame group DNN update information as the frame group DNN setting information (operation 2112). According to an embodiment of the present disclosure, the DNN controller 1736 may determine the frame group DNN setting information without referring to the video DNN setting information. According to another embodiment, the DNN controller 1736 may determine the frame group DNN setting information related to the structure of the second DNN 300 based on the video DNN setting information. In addition, the DNN controller 1736 may determine the frame group DNN setting information about the parameters of the filter kernel in the second DNN 300 based on the frame group DNN update information.

[0353] Operation 2114 , operation 2116 , and operation 2118 are performed on each of the remaining frame groups except the first frame group 2132 .

[0354] The DNN controller 1736 determines whether the DNN setting information will be updated for the current frame group (operation 2114). Similar to operation 2108, the DNN controller 1736 may determine whether the DNN setting information will be updated according to the frame group DNN update permission information.

[0355] When the DNN setting information is not updated for the current frame group, the DNN controller 1736 determines one or more pieces of video DNN setting information determined in operation 2104 or operation 2106 and one piece of DNN setting information from one or more pieces of frame group DNN setting information for a previous frame group decoded before the current frame group as frame group DNN setting information for the current frame group (operation 2116).

[0356] When the DNN setting information is updated for the current frame group, the DNN controller 1736 determines the updated DNN setting information set based on the frame group DNN update information as the frame group DNN setting information for the current frame group (operation 2118). According to an embodiment of the present disclosure, the DNN controller 1736 may determine the frame group DNN setting information for the current frame group without referring to the video DNN setting information or the frame group DNN setting information for the previous frame group. According to another embodiment of the present disclosure, the DNN controller 1736 may determine the frame group DNN setting information related to the structure of the second DNN 300 based on the video DNN setting information or the frame group DNN setting information for the previous frame group. In addition, the DNN controller 1736 may determine the frame group DNN setting information about the parameters of the filter kernel to be set in the second DNN 300 based on the frame group DNN update information.

[0357] According to an embodiment of the present disclosure, before operation 2114, the DNN controller 1736 may obtain frame group DNN dependency information for each of the remaining frame groups except the first frame group 2132, wherein the frame group DNN dependency information indicates whether to refer to the frame group DNN setting information for the immediately preceding frame group. In addition, before operation 2114, the DNN controller 1736 may determine whether to determine the frame group DNN setting information for the current frame group by referring to the frame group DNN setting information for the immediately preceding frame group based on the frame group DNN dependency information. Because the same or similar frame group DNN setting information is likely to be used for consecutive frame groups, the size of the data required to determine the frame group DNN setting information can be reduced by using the frame group DNN dependency information.

[0358] Fig. 22 A DNN setting information determination method for updating DNN setting information based on each frame according to an embodiment of the present disclosure is shown. Fig. 22 The DNN setting information determination method shown can apply different DNN setting information to each frame.

[0359] DNN controller 1736 is based on Figures 18 to 21 The method for determining frame group DNN setting information shown in is used to determine the frame group DNN setting information for the current frame group 2210 (operation 2202).

[0360] The DNN controller 1736 determines whether to update the DNN setting information for the current frame (operation 2204). The current frame is a frame currently being decoded among the frames included in the current frame group 2210. The current frame group 2210 includes a plurality of frames including a first frame 2212, a second frame 2214, and a third frame 2216, etc. For example, because the frame DNN setting information is determined independently for each frame, different frame DNN setting information may be applied to the first frame 2212 and the second frame 2214, respectively.

[0361] When the DNN setting information is not updated for the current frame, the DNN controller 1736 determines the frame group DNN setting information as the frame DNN setting information for the current frame (operation 2206).

[0362] When the DNN setting information is updated for the current frame, the DNN controller 1736 determines the updated DNN setting information generated according to the frame DNN update information as the frame DNN setting information (operation 2208). According to an embodiment of the present disclosure, the DNN controller 1736 may determine the frame DNN setting information without referring to the frame group DNN setting information. According to another embodiment, the DNN controller 1736 may determine the frame DNN setting information about the structure of the second DNN 300 by referring to the frame group DNN setting information. In addition, the frame DNN setting information about the parameters of the filter kernel to be set in the second DNN 300 may be determined based on the frame DNN update information.

[0363] According to an embodiment of the present disclosure, before operation 2204, the DNN controller 1736 may obtain frame DNN dependency information for each of the remaining frames except the first frame 2212, wherein the frame DNN dependency information indicates whether the current frame will refer to the frame DNN setting information for the immediately preceding frame. In addition, before operation 2114, the DNN controller 1736 may determine whether to determine the frame DNN setting information for the current frame by referring to the frame DNN setting information for the immediately preceding frame based on the frame DNN dependency information. Because the same or similar frame DNN setting information is likely to be used for consecutive frames, the size of the data required to determine the frame DNN setting information can be reduced by using the frame DNN dependency information.

[0364] Figure 23 to Figure 26 The structure of DNN update information for determining at least one piece of DNN setting information for a video or a frame group according to an embodiment of the present disclosure is shown.

[0365] Figure 23 to Figure 25Data structures 2320, 2400, and 2500 according to embodiments of the present disclosure are shown respectively, wherein each data structure includes AI data related to the update of DNN setting information for a video or a frame group. The AI ​​data according to each of the data structures 2320, 2400, and 2500 may be included in a video parameter set or a frame group parameter set. Optionally, the AI ​​data according to each of the data structures 2320, 2400, and 2500 may not be included in the video parameter set or the frame group parameter set, but may be included in the video header or the frame group header as a separate item.

[0366] According to an embodiment of the present disclosure, AI data according to each of the data structure 2320, the data structure 2400, and the data structure 2500 may be stored in Fig.15 The AI ​​data may be sent separately from the image data in the metadata box 1510 of the video file 1500. Alternatively, the AI ​​data according to each of the data structures 2320, 2400, and 2500 may be sent separately from the image data. The AI ​​data may also be synchronized with the image data and applied to the video or frame group.

[0367] When AI data according to each of data structures 2320, 2400, and 2500 is applied to a video, AI data related to the update of DNN setting information may be applied to all frames of the video. When AI data according to each of data structures 2320, 2400, and 2500 is applied to a frame group, AI data related to the update of DNN setting information may be applied to all frames of the frame group.

[0368] According to Figure 23 to Figure 25 The data structures 2320, 2400 and 2500 are used to determine the Fig.18 Operation 1802 to operation 1806, Fig.19 Operation 1902 to operation 1906, Fig. 20 Operations 2002 to 2006 and Fig.21 The characteristics of one or more video DNN setting information optimized for each video in operations 2102 to 2106. Figure 23 to Figure 25 The data structures 2320, 2400 and 2500 are used to determine the Fig. 20 Operation 2008 to Operation 2012 and Fig.21 Characteristics of one or more frame group DNN setting information optimized for each frame group in operations 2108 to 2118.

[0369] Fig.23A data structure 2300 including AI target data and AI auxiliary data and a data structure 2320 including AI data related to the update of DNN setting information according to an embodiment of the present disclosure are shown. Fig.23 In the embodiment, updated DNN setting information is determined according to the DNN update information without referring to the default DNN setting information or the DNN setting information for a data unit at a higher level than the current data unit.

[0370] Fig.23 The data structure 2300 is a higher-level structure compared to the data structure 2320. The data structure 2300 includes ai_codec_info 2302, where ai_codec_info 2302 is AI target data indicating whether AI upscaling is applied to the current data unit. When ai_codec_info 2302 indicates that AI upscaling is applied to the current data unit, the data structure 2300 includes AI auxiliary data ai_codec_applied_channel_info 2304 indicating channel information, AI auxiliary data target_bitrate_info 2306 indicating target bit rate information, AI auxiliary data res_info 2308 indicating resolution information, AI auxiliary data ai_codec_DNN_info 2310 indicating DNN-related information, and AI auxiliary data ai_codec_supplementary_info 2312 indicating supplementary information.

[0371] Reference Fig.23 , ai_codec_DNN_info 2310 may include Fig.23 The DNN update information related to the update of the DNN setting information of the data structure 2320 may include ai_codec_DNN_update_flag 2322, ai_codec_num_DNN 2324, ai_codec_num_layers 2326, ai_codec_out_channel 2328, ai_codec_in_channel 2330, ai_codec_filter_size 2332, ai_codec_weights 2334, and ai_codec_bias 2336.

[0372] ai_codec_DNN_update_flag 2322 in the data structure 2320 is DNN update permission information. The DNN update permission information indicates whether to update one or more pieces of DNN setting information.

[0373] When ai_codec_DNN_update_flag 2322 indicates that one or more pieces of DNN setting information are not updated, ai_codec_num_DNN 2324 and some DNN update information below ai_codec_num_DNN 2324 are not included in the DNN update flag. Fig.23 In the data structure 2320 of . When the current data unit is a video, one or more video DNN setting information can be selected from multiple pre-stored default DNN setting information. When the current data unit is a frame group, the frame group DNN setting information for the current frame group can be selected from the one or more video DNN setting information. The DNN index information referenced when selecting the video DNN setting information or the frame group DNN setting information can be included in the data structure 2320.

[0374] When ai_codec_DNN_update_flag 2322 indicates updating of one or more pieces of DNN setting information, DNN update information is included in the data structure 2320. The DNN update information indicates information related to updating of the one or more pieces of DNN setting information used to set the second DNN 300.

[0375] For example, the DNN update information may include DNN quantity information about the amount of DNN setting information to be updated. The DNN update information may also include layer number information about the number of layers in the DNN. The DNN update information may also include input channel information indicating the number of input channels in the layer and output channel information indicating the number of output channels in the layer. In addition, the DNN update information may include filter kernel size information indicating the size of the filter kernel of the layer, and information related to weights and biases representing parameters of the filter kernel.

[0376] ai_codec_num_DNN 2324 indicates DNN number information. When ai_codec_DNN_update_flag 2322 indicates updating of the one or more pieces of DNN setting information, ai_codec_num_DNN 2324 may be included in the data structure 2320. The number of updated DNN setting information equal to the number indicated by ai_codec_num_DNN 2324 is defined for a video or frame group. For example, when ai_codec_num_DNN 2324 indicates four (4), four pieces of updated DNN setting information are defined.

[0377] The pieces of information about layers and filter kernels included in the updated DNN setting information are determined according to ai_codec_num_layers 2326, ai_codec_out_channel 2328, ai_codec_in_channel 2330, ai_codec_filter_size 2332, ai_codec_weights 2334, and ai_codec_bias 2336, which may be individually defined for each of the plurality of updated DNNs. Alternatively, the values ​​of ai_codec_num_layers 2326, ai_codec_out_channel 2328, ai_codec_in_channel 2330, and ai_codec_filter_size 2332 may be set to be commonly applied to the plurality of pieces of updated DNN setting information.

[0378] ai_codec_num_layers 2326 may be defined for the updated DNN setting information. ai_codec_num_layers 2326 is layer number information indicating the number of layers included in the updated DNN setting information. For example, when ai_codec_num_layers 2326 indicates three (3), the updated DNN is set to include three layers.

[0379] The number of the plurality of pieces of ai_codec_num_layers 2326 may be defined to be equal to the number of the plurality of updated DNNs, respectively, for the updated DNNs. When the number of the updated DNNs is four, four pieces of ai_codec_num_layers 2326 respectively associated with the four updated DNNs may be included in the data structure 2320.

[0380] In addition, only one ai_codec_num_layers 2326 may be defined for a plurality of updated DNNs. Therefore, the number of layers in each of the updated DNNs may be determined by the one ai_codec_num_layers 2326. Therefore, the number of layers in the updated DNNs may be determined to be equal to each other for all updated DNNs.

[0381] ai_codec_out_channel 2328 and ai_codec_in_channel 2330 may be defined for each layer in the updated DNN. ai_codec_out_channel 2328 indicates output channel information about the number of output channels in each layer. In addition, ai_codec_in_channel 2330 indicates input channel information about the number of input channels in each layer. For example, when ai_codec_out_channel 2328 indicates 3, the number of output channels in each layer is set to 3. When ai_codec_in_channel 2330 indicates 3, the number of input channels in each layer is set to 3.

[0382] When there are two or more layers in the DNN, ai_codec_in_channel 2330 of the subsequent layer may be determined according to ai_codec_out_channel 2328 of the previous layer. Therefore, ai_codec_in_channel 2330 may be omitted from the data structure 2320 for the remaining layers except the first layer of the DNN.

[0383] In addition, when the number of input channels in the DNN is fixed, the ai_codec_in_channel 2330 of the first layer may be omitted from the data structure 2320. For example, when the number of input channels in the DNN is fixed to 3, the ai_codec_in_channel 2330 of the first layer is omitted from the data structure 2320.

[0384] Similarly, when the number of output channels in the DNN is fixed, the ai_codec_out_channel 2328 of the last layer of the DNN may be omitted from the data structure 2320. For example, when the number of output channels in the DNN is fixed to 3, the ai_codec_out_channel 2328 of the last layer (i.e., the third layer) may be omitted from the data structure 2320.

[0385] ai_codec_out_channel 2328 and ai_codec_in_channel 2330 may be defined for each layer in each of the multiple updated DNNs. For example, when there are two updated DNNs (i.e., a first updated DNN and a second updated DNN), multiple ai_codec_out_channel 2328 and multiple ai_codec_in_channel 2330 for each layer of the first updated DNN and multiple ai_codec_out_channel 2328 and multiple ai_codec_in_channel 2330 for each layer of the second updated DNN may be included in the data structure 2320.

[0386] In addition, when each of two or more updated DNNs includes an equal number of layers, a plurality of ai_codec_out_channel 2328 and ai_codec_in_channel 2330 respectively defined for the layers in one updated DNN are applied to the corresponding layers in another updated DNN. Therefore, each of the two or more updated DNNs can be determined to have an equal number of layers, and the layers arranged at corresponding positions in the two or more updated DNNs are each determined to have an equal number of input channels and an equal number of output channels. For example, the first layers in the two or more updated DNNs can each be determined to have an equal number of input channels and an equal number of output channels. Similarly, the second layers in the two or more updated DNNs can each be determined to have an equal number of input channels and an equal number of output channels.

[0387] ai_codec_filter_size 2332 may be defined for each layer of the updated DNN. ai_codec_filter_size 2332 is filter kernel size information indicating the size of the filter kernel of the layer. Because the depth of the filter kernel is determined according to the number of input channels, ai_codec_filter_size 2332 represents the number of rows and columns of the filter kernel, and the depth of the filter kernel may be omitted in ai_codec_filter_size 2332. When the number of rows in the filter kernel is equal to the number of columns therein, ai_codec_filter_size 2332 may include only one value of the number of rows and columns of the filter kernel. For example, when ai_codec_filter_size 2332 indicates 3, the size of the filter kernel is determined to be 3×3. When the number of rows in the filter kernel is not necessarily equal to the number of columns therein, both the number of rows and the number of columns of the filter kernel may be used to indicate the size of the filter kernel. For example, when ai_codec_filter_size 2332 indicates 3 and 5, the size of the filter kernel is determined to be 3×5 (ie, 3 rows by 5 columns).

[0388] ai_codec_filter_size 2332 may be defined for each layer in each of the multiple updated DNNs. For example, when there are two updated DNNs (i.e., a first updated DNN and a second updated DNN), multiple ai_codec_filter_size 2332 for each layer of the first updated DNN and multiple ai_codec_filter_size 2332 for each layer of the second updated DNN may be included in the data structure 2320.

[0389] In addition, when each of two or more updated DNNs includes an equal number of layers, and the layers arranged at corresponding positions in the two or more updated DNNs have an equal number of input channels and an equal number of output channels, a plurality of ai_codec_filter_size 2332 respectively defined for the layers in one updated DNN may be applied to the corresponding layers in another updated DNN. Therefore, each of the two or more updated DNNs may be determined to have an equal number of layers, and the layers arranged at corresponding positions in the two or more updated DNNs may be determined to have an equal number of input channels, an equal number of output channels, and an equal filter kernel size. For example, the first layers in the two or more updated DNNs may be determined to have an equal filter kernel size. Similarly, the second layers in the two or more updated DNNs may be determined to have an equal filter kernel size.

[0390] ai_codec_weights 2334 is defined for the filter kernel of each layer in the updated DNN. ai_codec_weights 2334 is information indicating the weights included in the filter kernel. Each layer in the updated DNN includes a number of filter kernels equal to the number of output channels. Therefore, when a layer has two output channels, the layer includes two filter kernels.

[0391] The size of the filter kernel is determined by the number of rows, columns, and depth. Therefore, each filter kernel includes a number of parameters corresponding to (number of rows)×(number of columns)×(depth). Because the depth of the filter kernel is equal to the number of input channels, each filter kernel includes a matrix of weights corresponding to (number of rows)×(number of columns)×(number of input channels).

[0392] Therefore, the number of weights included in all filter kernels of a layer is equal to (number of rows)×(number of columns)×(number of input channels)×(number of output channels). Therefore, for a layer, a number of ai_codec_weights 2334 equal to (number of rows)×(number of columns)×(number of input channels)×(number of output channels) may be included in the data structure 2320. For example, Fig.13 As shown in , when the number of rows and the number of columns are both 3, and the number of input channels and the number of output channels are both 4, 144 (=3×3×4×4) pieces of ai_codec_weights 2334 may be included in the data structure 2320 .

[0393] The bit size of ai_codec_weights 2334 may be limited. For example, ai_codec_weights 2334 may be limited to indicate one of values ​​-1, 0, and 1. Therefore, ai_codec_weights 2334 may be represented by 1 to 2 bits. However, the range of ai_codec_weights 2334 may be determined in various ways and is not limited thereto.

[0394] ai_codec_bias 2336 is defined for the filter kernel of each layer in the updated DNN. ai_codec_bias 2336 is bias information indicating the bias of the filter kernel. The output value of the filter kernel is adjusted according to the bias corresponding to the filter kernel. For example, when the output value of the filter kernel is 3 and the bias is 1, the output value of the filter kernel is adjusted to 4 (=3+1).

[0395] The layer includes a number of filter kernels equal to the number of output channels. Therefore, a number of biases equal to the number of output channels in the layer can be determined. For example, when the number of output channels in the layer is 2, two ai_codec_bias 2336 for the layer are included in the data structure 2320.

[0396] exist Fig.23 The continuous dots shown in the figure represent the repetition of the same data structure. Therefore, the continuous dots appearing below the information about the second layer of the first DNN indicate that the data structures for the other layers of the first DNN are also the same as the data structures for the first and second layers of the first DNN. In addition, the continuous dots below the information about the second DNN (ai_codec_num_layers2326) indicate that the data structures for the other DNNs including the second DNN are the same as the data structures for the first DNN.

[0397] Fig.23 The data structure 2320 is only an example, and according to an embodiment of the present disclosure, the data structure 2320 may include Fig.23 DNN update information not shown in the data structure 2320 but necessary for configuring the updated DNN. For example, input channel size information about the size of the 2D data array for the input channel and output channel size information about the size of the 2D data array for the output channel may be included in the data structure 2320.

[0398] In addition, extension information about the extension of the 2D data array for the input channel may be included in the data structure 2320. Thus, as in Fig.13 As shown in, according to the extended information, zero value elements can be added to the top, bottom, left and right of the 2D data array for input channels. Alternatively, zero value elements can be added between elements in the 2D data array for input channels. Alternatively, interpolation elements can be added between elements in the 2D data array for input channels. Therefore, the size of the 2D data array for input channels can be increased according to the extended information.

[0399] Fig.24 2400 including AI data related to the update of DNN setting information according to an embodiment of the present disclosure. Fig.23 Data structure 2320, data structure 2400 and Fig.23 The sub-information of ai_codec_DNN_info 2310 is related.

[0400] exist Fig.24In the DNN update information, the structure of the updated DNN is determined with reference to the structure of the default DNN. In addition, the weights and biases of the filter kernels in the updated DNN are determined according to the DNN update information. In the present disclosure, the structure of the DNN represents multiple pieces of information about the properties of the layers, input channels, and output channels other than the weights and biases of the filter kernels, and the size of the filter kernels. For example, the number of layers, the number of input channels and output channels in each of the layers, and the size of the filter kernels of each layer are included in the DNN setting information related to the structure of the DNN.

[0401] exist Fig.24 ai_codec_DNN_update_flag 2402 and ai_codec_num_DNN 2404 shown in FIG. Fig.23 ai_codec_DNN_update_flag 2322 and ai_codec_num_DNN 2324 have the same function.

[0402] and Fig.23 The data structure 2320 is different, Fig.24 The data structure 2400 does not include ai_codec_num_layers 2326, ai_codec_out_channel 2328, ai_codec_in_channel 2330, and ai_codec_filter_size 2332. Alternatively, the structure of the updated DNN is determined with reference to the structure of the default DNN. Therefore, ai_codec_DNN_struct_reuse(ref)_idx 2406 indicating reference DNN index information may be included in the data structure 2400. The reference DNN index information indicates the default DNN setting information or video DNN setting information referenced when setting the updated DNN setting information. When the current data unit is a video, the reference DNN index information indicates one of the multiple default DNN setting information. When the current data unit is a frame group, the reference DNN index information may indicate one of the multiple video DNN setting information. According to an embodiment of the present disclosure, when the current data unit is a frame group, the reference DNN index information may indicate one DNN setting information among a plurality of pieces of video DNN setting information and a plurality of pieces of frame group DNN setting information of a previous frame group.

[0403] When the DNN setting information referenced when generating the updated DNN setting information is determined according to ai_codec_DNN_struct_reuse(ref)_idx 2406, the number of layers, the number of input channels and output channels in each layer, and the size of the filter kernel of each layer in the updated DNN setting information are determined according to the number of layers included in the referenced DNN setting information, the number of input channels and output channels in each layer, and the size of the filter kernel of each layer. In addition, the size of the 2D data array for input channels and the size of the 2D data array for output channels included in the updated DNN setting information can be determined according to the size of the 2D data array for input channels and the size of the 2D data array for output channels included in the referenced DNN setting information.

[0404] ai_codec_layer_update_flag 2408 may be included in the data structure 2400. ai_codec_layer_update_flag 2408 is a layer update flag indicating whether the filter kernel parameters in the layer of the updated DNN are the same as the filter kernel parameters in the layer of the default DNN, or a flag for indicating that it needs to be updated with new values. Because the number of layers in the referenced DNN setting information is equal to the number of layers in the updated DNN, multiple ai_codec_layer_update_flag 2408 equal to the number of layers in the updated DNN are included in the data structure 2400. When ai_codec_layer_update_flag 2408 indicates that the filter kernel parameters in the layer of the updated DNN need to be updated with new values, ai_codec_weights 2410 indicating the weights of the filter kernel in the updated DNN and ai_codec_bias 2412 indicating the bias are included in the data structure 2400. Therefore, the weights and biases of the filter kernels in the updated DNN may be different from the weights and biases of the filter kernels in the default DNN.

[0405] When ai_codec_layer_update_flag 2408 indicates that the filter kernel parameters in the layer of the updated DNN are the same as the filter kernel parameters in the layer of the default DNN, ai_codec_weights 2410 and / or ai_codec_bias 2412 may be omitted from the data structure 2400, and the filter kernel parameters in the layer of the updated DNN are determined to be the same as the filter kernel parameters in the layer of the default DNN.

[0406] According to an embodiment of the present disclosure, filter core identification information may be included in the data structure 2400, wherein the filter core identification information indicates whether the parameters of some filter cores in a layer of the updated DNN are the same as the parameters of the filter cores in the layer of the default DNN. When the filter core identification information indicates that the parameters of the filter cores in the updated DNN are the same as the parameters of the filter cores in the default DNN, ai_codec_weights 2410 and / or ai_codec_bias 2412 may be omitted from the data structure 2400. The filter core identification information may be defined for one or more filter cores of the layer, respectively. Therefore, when the layer includes three filter cores, three filter core identification information for the three filter cores may be included in the data structure 2400.

[0407] According to an embodiment of the present disclosure, when the updated DNN setting information refers to the default DNN setting information, ai_codec_weights 2410 may indicate the difference between the weight of the filter core in the default DNN and the weight of the filter core in the updated DNN, wherein the weight is arranged at the same position in the filter core. For example, ai_codec_weights 2410 for the first weight of the first filter core in the first layer of the updated DNN may indicate the difference between the first weight of the first filter core in the first layer of the updated DNN and the first weight of the first filter core in the first layer of the default DNN.

[0408] Similarly, when the updated DNN setting information refers to the default DNN setting information, ai_codec_bias 2412 may indicate the difference between the biases at the same positions in the default DNN and the updated DNN, respectively, for example, ai_codec_bias 2412 for the bias of the first filter kernel in the first layer of the updated DNN may indicate the difference between the bias of the first filter kernel in the first layer of the updated DNN and the bias of the first filter kernel in the first layer of the default DNN.

[0409] exist Fig.24 The consecutive dots shown in 2406 indicate that the same data structure is repeated. Therefore, the consecutive dots appearing below the information about the second layer of the first DNN indicate that the data structures for other layers of the first DNN are also the same as the data structures for the first and second layers of the first DNN. In addition, the consecutive dots below the information about the second DNN (ai_codec_DNN_struct_reuse_idx 2406) indicate that the data structures for other DNNs including the second DNN are the same as the data structures for the first DNN.

[0410] Fig.25FIG. 2500 shows a data structure including AI data related to the update of DNN setting information according to an embodiment of the present disclosure. Fig.23 Data structures 2320, data structures 2500 and Fig.23 The sub-information of ai_codec_DNN_info 2310 is related.

[0411] Fig.25 The data structure 2500 includes default DNN reuse permission information. Fig.23 The updated DNN determination method and Fig.24 The updated DNN setting information is determined according to the selected updated DNN determination method. Therefore, in Fig.25 In, as reference Fig.23 As described above, the updated DNN setting information may be determined based on the DNN update information without referring to the default DNN setting information. Fig.25 In, as reference Fig.24 As described above, the updated DNN setting information may be determined by referring to the structure of the default DNN.

[0412] Fig.25 ai_codec_DNN_update_flag 2502 and ai_codec_num_DNN 2504 respectively have the same Fig.23 ai_codec_DNN_update_flag 2322 and ai_codec_num_DNN 2324 have the same function.

[0413] and Fig.23 Data structures 2320 and Fig.24 The data structure of 2400 is different, Fig.25 The data structure 2500 includes ai_codec_DNN_struct_reuse_flag 2518 as default DNN reuse permission information. The default DNN reuse permission information indicates whether to determine the structure of the updated DNN with reference to the default DNN. The data structure 2500 includes ai_codec_DNN_struct_reuse_flag 2518 for each updated DNN setting information.

[0414] When ai_codec_DNN_struct_reuse_flag 2518 indicates that the updated DNN setting information refers to the structure of the default DNN (yes), as shown in Fig.24 As described above, the updated DNN setting information is determined by referring to the structure of the default DNN. Fig.24The characteristics of the described data structure 2400 can be applied to Fig.25 Data structure 2500.

[0415] Therefore, the default DNN setting information, the video DNN setting information, or the frame group DNN setting information for the previous frame group referred to by the updated DNN setting information may be determined according to ai_codec_DNN_struct_reuse_idx 2520. In addition, whether the layer of the updated DNN is the same as the layer of the default DNN may be determined according to ai_codec_layer_update_flag 2522. Then, the weight and bias of the filter kernel may be determined according to ai_codec_weights 2524 and ai_codec_bias 2526, respectively.

[0416] When ai_codec_DNN_struct_reuse_flag 2518 indicates that the updated DNN setting information does not refer to the structure of the default DNN (ie, ai_codec_DNN_struct_reuse_flag = No), as shown in FIG. Fig.23 As described above, the updated DNN setting information is determined without referring to the structure of the default DNN. Fig.23 The characteristics of the described data structure 2320 can be applied to Fig.25 Data structure 2500.

[0417] Therefore, ai_codec_num_layers 2506, ai_codec_out_channel 2508, ai_codec_in_channel 2510, and ai_codec_filter_size 2512 are included in the data structure 2500. Therefore, the number of layers in the updated DNN, the number of output channels and input channels in each of the layers, and the size of the filter kernel of each layer are determined according to ai_codec_num_layers 2506, ai_codec_out_channel 2508, ai_codec_in_channel 2510, and ai_codec_filter_size 2512. In addition, the weights and biases of the filter kernels are determined according to ai_codec_weights 2514 and ai_codec_bias 2516, respectively.

[0418] exist Fig.25The continuous dots shown in the figure represent the repetition of the same data structure. Therefore, the continuous dots appearing below the information about the second layer of the first DNN indicate that the data structures for other layers of the first DNN are also the same as the data structures for the first and second layers of the first DNN. In addition, the continuous dots below the information about the second DNN (ai_codec_DNN_struct_reuse_flag 2518) indicate that the data structures for other DNNs including the second DNN are the same as the data structure of the first DNN.

[0419] Fig.26 The AI ​​codec high-level syntax 2600 of the data structure including AI data related to the update of DNN setting information according to an embodiment of the present disclosure is shown. The AI ​​codec high-level syntax 2600 represents the high-level syntax of the data structure implementing the AI ​​data. The high-level syntax refers to the syntax including parameters that are generally applied when encoding a specific data unit. For example, the high-level syntax includes a video parameter set applied to a video, a frame group parameter set applied to a frame group, a frame parameter set applied to a frame, etc.

[0420] The syntax describes the data structure in the bit stream represented by consecutive bits. Therefore, the bit stream is interpreted according to the data structure described by the syntax. Fig.26 The AI ​​codec high-level syntax 2600 can be applied to a video or a group of frames.

[0421] According to an embodiment of the present disclosure, it can be implemented in the form of SEI message Fig.26 The AI ​​codec high-level syntax 2600. SEI message is a collection of supplementary information related to the video and is sent separately from the coded data of the image. Fig.26 The AI ​​codec high-level syntax 2600 of the SEI message.

[0422] exist Fig.26 The left side of the table describes the order of obtaining syntax elements related to the update of DNN setting information and the conditions for obtaining syntax elements. In addition, Fig.26 The right side of the table shows how to extract syntax elements related to the update of DNN setting information from the bitstream. Here, u(n) indicates that the corresponding syntax element has a size of n bits, where n is a positive integer. ue(v) indicates that the corresponding syntax element is encoded by an exponential Golomb coding scheme.

[0423] As in Fig.26As shown on the left side of the table, the AI ​​codec high-level syntax 2600 starts with ai_codec_info, where ai_codec_info is AI target information indicating whether AI upscaling is applied to the current data unit. ai_codec_info is a syntax element having a size of 1 bit, and when ai_codec_info indicates 0, the AI ​​codec high-level syntax 2600 ends.

[0424] When ai_codec_info indicates 1, the AI ​​codec high-level syntax 2600 includes ai_codec_DNN_update_flag as DNN update permission information. Fig.26 In the table, ai_codec_DNN_update_flag is a syntax element having a size of 1 bit. When ai_codec_DNN_update_flag indicates "0", the AI ​​codec high-level syntax 2600 includes ai_codec_DNN_index indicating one of the default DNN setting information and the video DNN setting information. In addition, the DNN setting information for the current data unit is determined according to ai_codec_DNN_index.

[0425] When ai_codec_DNN_update_flag indicates "1", the AI ​​codec high-level syntax 2600 includes ai_codec_num_DNN as DNN number information. ai_codec_num_DNN is a syntax element having a size of 8 bits. The number of multiple updated DNN setting information according to ai_codec_num_DNN may be in the range of 1 to 256. However, according to an embodiment of the present disclosure, the bit size of ai_codec_num_DNN may be changed differently. According to an embodiment of the present disclosure, when only one updated DNN setting information is required, ai_codec_num_DNN may be omitted from the AI ​​codec high-level syntax 2600.

[0426] The AI codec advanced syntax 2600 includes several ai_codec_network_struct_reuse_flag[i], where ai_codec_network_struct_reuse_flag[i] is DNN reuse permission information, and the number of ai_codec_network_struct_reuse_flag[i] is equal to the number of updated DNNs indicated by ai_codec_num_DNN (e.g., for(i = 0; i < ai_codec_num_DNN; i++)). For example, when ai_codec_num_DNN indicates 4, the AI codec advanced syntax 2600 may include ai_codec_network_struct_reuse_flag[0], ai_codec_network_struct_reuse_flag[1], ai_codec_network_struct_reuse_flag[2], and ai_codec_network_struct_reuse_flag[3]. ai_codec_network_struct_reuse_flag[i] is a syntax element with a size of 1 bit.

[0427] When ai_codec_network_struct_reuse_flag[i] indicates 1, the AI codec advanced syntax 2600 includes ai_codec_network_struct_reuse_idx[i] as reference DNN index information. ai_codec_network_struct_reuse_idx[i] is a syntax element with a size of 8 bits. The structure of the updated DNN is determined according to the default DNN setting information indicated by ai_codec_network_struct_reuse_idx[i]. For example, the number of layers (ai_codec_num_layers[i]) in the updated DNN, the number of input channels (ai_codec_in_channel[i][j]) in each layer, the number of output channels (ai_codec_out_channel[i][j]) in each layer, the size of the filter kernel (ai_codec_filter_width[i][j] and ai_codec_filter_height[i][j]) in each layer, etc. can be determined according to the default DNN setting information to be referenced. In addition, the size of the 2D data array for the input channels and the size of the 2D data array for the output channels in the updated DNN can be determined according to the structure of the default DNN to be referenced.

[0428] According to an embodiment of the present disclosure, different from Fig.26 , the AI codec high-level syntax 2600 may not include ai_codec_network_struct_reuse_idx[i]. The default DNN setting information to be referred to when setting the updated DNN setting information may be determined according to the identification number assigned to the updated DNN. For example, when the first updated DNN has the same identification number as the first default DNN, the first updated DNN setting information may be determined by referring to the first default DNN setting information. Similarly, when the second updated DNN has the same identification number as the second default DNN, the second updated DNN setting information may be determined by referring to the second default DNN setting information.

[0429] A number of ai_codec_bias[i][j][k] indicating the bias information of the current layer can be obtained, where the number of ai_codec_bias[i][j][k] is equal to the number of output channels indicated by ai_codec_out_channel[i][j] (for (k = 0; k < ai_codec_out_channel[i][j]; k++)). In Fig.26 the AI codec high-level syntax 2600, ai_codec_bias[i][j][k] is encoded by the exponential Golomb coding scheme.

[0430] A number of ai_codec_weights[i][j][k][1][m] indicating the weight information related to the parameters of the filter kernel in the current layer can be obtained, where the number of ai_codec_weights[i][j][k][1][m] is equal to the number corresponding to the product obtained by multiplying the number of output channels indicated by ai_codec_out_channel[i][j], the number of input channels indicated by ai_codec_in_channel[i][j], and the size of the filter kernel indicated by ai_codec_filter_width[i][j] * ai_codec_filter_height[i][j] (for (k = 0; k < ai_codec_out_channel[i][j]; k++) { for (l = 0; l < ai_codec_in_channel[i][j]; l++) { for (m = 0; m < ai_codec_filter_width[i][j] * ai_codec_filter_height[i][j]; m++)}}). In Fig.26In the AI codec advanced syntax 2600, ai_codec_weights[i][j][k][1][m] is encoded by the exponential Golomb coding scheme.

[0431] According to an embodiment of the present disclosure, ai_codec_weights[i][j][k][l][m] may indicate the difference between the weights in the default DNN and the weights in the updated DNN. In addition, ai_codec_bias[i][j][k] may indicate the difference between the biases in the default DNN and the biases in the updated DNN.

[0432] When ai_codec_network_struct_reuse_flag[i] indicates "0", the AI codec advanced syntax 2600 includes several ai_codec_num_layers[i] as the layer number information, where the number of ai_codec_num_layers[i] is equal to the number of updated DNNs according to ai_codec_num_DNN (for (i = 0; i < ai_codec_num_DNN; i++)). For example, when ai_codec_num_DNN indicates 4, the AI codec advanced syntax 2600 may include ai_codec_num_layers[0], ai_codec_num_layers[1], ai_codec_num_layers[2], and ai_codec_num_layers[3].

[0433] ai_codec_num_layers[i] is a syntax element with an 8-bit size. Therefore, the number of layers according to ai_codec_num_layers[i] can be determined within the range of 1 to 256 (i.e., 2 8 = 256). However, according to an embodiment of the present disclosure, the bit size of ai_codec_num_layers[i] may be changed differently.

[0434] Several ai_codec_layer_update_flag are included in the AI codec high-level syntax 2600, where the number of ai_codec_layer_update_flag is equal to the number of layers according to ai_codec_num_layers[i] (for (j = 0; j < ai_codec_num_layers[i]; j++)). The ai_codec_layer_update_flag is a syntax element with a size of 1 bit and may include layer update information indicating whether the current layer is updated. When the ai_codec_layer_update_flag indicates "0", the DNN setting information of the default layer stored in the AI amplifier 234 is applied to the current layer. When the ai_codec_layer_update_flag indicates "1", ai_codec_in_channel[i][j] (i.e., input channel information), ai_codec_out_channel[i][j] (i.e., output channel information), ai_codec_filter_width[i][j], and ai_codec_filter_height[i][j] (which are filter kernel size information) are included in the high-level syntax 2600. Although Fig.26 it is shown that the ai_codec_layer_update_flag is included in the AI codec high-level syntax 2600, according to an embodiment of the present disclosure, the ai_codec_layer_update_flag may not be included in the AI codec high-level syntax 2600. When the ai_codec_layer_update_flag is not included in the AI codec high-level syntax 2600, ai_codec_in_channel[i][j], ai_codec_out_channel[i][j], ai_codec_filter_width[i][j], and ai_codec_filter_height[i][j] may be configured to be included in the AI codec high-level syntax without decoding the layer update information.

[0435] ai_codec_in_channel[i][j], ai_codec_out_channel[i][j], ai_codec_filter_width[i][j], and ai_codec_filter_height[i][j] are syntax elements, each of which has an 8-bit size. Therefore, the values ​​indicated by ai_codec_in_channel[i][j], ai_codec_out_channel[i][j], ai_codec_filter_width[i][j], and ai_codec_filter_height[i][j], respectively, may be determined within a range of 1 to 256. However, according to an embodiment of the present disclosure, the bit sizes of ai_codec_in_channel[i][j], ai_codec_out_channel[i][j], ai_codec_filter_width[i][j], and ai_codec_filter_height[i][j] may be changed differently.

[0436] In addition, the AI ​​codec high-level syntax includes ai_codec_weights[i][j][k][1][m] and ai_codec_bias[i][j][k], where ai_codec_weights[i][j][k][1][m] indicates the weights of parameters included in the filter kernel whose values ​​are determined, and ai_codec_bias[i][j][k] indicates a bias used to adjust the output of the filter kernel.

[0437] Fig.26 The AI ​​codec high-level syntax 2600 may be divided into a first syntax 2610, a second syntax 2620, and a third syntax 2630. The first syntax 2610 includes ai_codec_DNN_update_flag, ai_codec_num_DNN, and ai_codec_DNN_struct_reuse_flag[i]. A method for determining updated DNN setting information is determined according to ai_codec_DNN_struct_reuse_flag[i]. When ai_codec_DNN_struct_reuse_flag[i] indicates "1", the updated DNN setting information is determined according to the syntax elements of the second syntax 2620. When ai_codec_DNN_struct_reuse_flag[i] indicates "0", the updated DNN setting information is determined according to the syntax elements of the third syntax 2630.

[0438] By setting ai_codec_DNN_struct_reuse_flag[i] of the first syntax 2610 to always indicate "1" and excluding the third syntax 2630, Fig.26 The AI ​​codec high-level syntax 2600 is implemented Fig.24 In addition, by setting ai_codec_DNN_struct_reuse_flag[i] of the first syntax 2610 to always indicate "0" and excluding the second syntax 2620, the Fig.26 The AI ​​codec high-level syntax 2600 is implemented Fig.23 Data structure 2320.

[0439] Fig.26 The AI ​​codec high-level syntax 2600 is only an example, and according to an embodiment of the present invention, the AI ​​codec high-level syntax 2600 may be excluded. Fig.26 Some of the grammatical elements of Fig.26 The syntax not shown in FIG. 2600 may be included in the AI ​​codec high-level syntax 2600. Fig.26 The AI ​​codec high-level syntax 2600 is implemented in different ways Figure 23 to Figure 25 The algorithm shown in .

[0440] Fig. 27 A data structure 2700 including AI data related to the update of DNN setting information for a frame group or a frame according to an embodiment of the present disclosure is shown. The AI ​​data according to the data structure 2700 may be included in a frame group parameter set or a frame parameter set. Optionally, the AI ​​data according to the data structure 2700 may not be included in the frame group parameter set or the frame parameter set, but may be separately included in the frame group header or the frame header. Optionally, the AI ​​data according to the data structure 2700 may be sent separately from the image data. The AI ​​data according to the data structure 2700 may also be synchronized with the image data and applied to the frame group or the frame.

[0441] Used in Fig. 20 Operation 2008 to Operation 2012 and Fig.21 Determining the characteristics of the frame group DNN setting information optimized for each frame group in operations 2108 to 2118 may be implemented in Fig. 27 The data structure 2700. In addition, for Fig. 22 The characteristics of the DNN setting information optimized for each frame determined in operations 2204 to 2208 may be applied to Fig. 27 Data structure 2700.

[0442] Reference Fig. 27, ai_codec_info 2702 is AI target information indicating whether AI upscaling is applied to the current data unit. When ai_codec_info 2702 indicates that AI upscaling is not applied to the current data unit, AI upscaling for the current data unit is not performed.

[0443] When ai_codec_info 2702 indicates that AI amplification is applied to the current data unit, the data structure 2700 includes dependent_ai_condition_info 2704 as AI auxiliary data dependency information, wherein the AI ​​auxiliary data dependency information indicates whether AI auxiliary data for the previous data unit is applied to the current data unit. When dependent_ai_condition_info 2704 indicates that AI auxiliary data for the previous data unit is applied to the current data unit, DNN setting information for the current data unit is not updated. When dependent_ai_condition_info 2704 indicates that AI auxiliary data of the immediately preceding data unit is not applied to the current data unit, ai_codec_DNN_index 2706 as DNN index information is included in the data structure 2700.

[0444] ai_codec_DNN_index 2706 indicates the DNN setting information applied to the current data unit among multiple candidate DNN setting information. For example, when the current data unit is a frame group, ai_codec_DNN_index 2706 may indicate the DNN setting information applied to the current frame group among multiple video DNN setting information. Optionally, when the current data unit is a frame group, ai_codec_DNN_index 2706 may indicate the DNN setting information applied to the current frame group among one or more video DNN setting information and one or more frame group DNN setting information for the previous frame group. Optionally, when the current data unit is a frame, ai_codec_DNN_index 2706 may indicate the DNN setting information applied to the current frame among one or more frame group DNN setting information and one or more frame DNN setting information of the previous frame. When there is one candidate DNN setting information, ai_codec_DNN_index 2706 may be omitted from the data structure 2700.

[0445] The data structure 2700 includes ai_codec_DNN_update_flag 2708 as DNN update permission information. Figure 22 to Figure 24 The DNN update permission information described in dictates whether to update multiple DNNs, but Fig. 27The DNN update permission information shown in 2706 indicates whether to update the DNN setting information of the higher-level DNN indicated by ai_codec_DNN_index 2706.

[0446] When ai_codec_DNN_update_flag 2708 indicates that the DNN setting information of the higher-level DNN is not updated, the DNN update information is not included in the data structure 2700. Therefore, the second image in the frame may be AI-enlarged according to the DNN setting information of the higher-level DNN indicated by ai_codec_DNN_index 2706. When ai_codec_DNN_update_flag 2708 indicates that the DNN setting information is updated, the second image in the frame may be AI-enlarged according to the updated DNN setting information determined from the DNN update information.

[0447] The structure of the updated DNN is determined with reference to the DNN setting information of the higher-level DNN. Therefore, the number of layers, the number of input channels, the number of output channels, the size of the filter kernel, etc. in the updated DNN are determined according to the DNN setting information of the higher-level DNN.

[0448] ai_codec_layer_update_flag 2710 may be included in the data structure 2700. ai_codec_layer_update_flag 2710 is a layer update flag indicating whether the layers of the updated DNN need to be updated with new values ​​without referring to the layers of the higher-level DNN. Several ai_codec_layer_update_flag 2710 are included in the data structure 2700, wherein the number of ai_codec_layer_update_flag 2710 is equal to the number of layers in the updated DNN.

[0449] When ai_codec_layer_update_flag 2710 indicates that the layers of the updated DNN directly refer to the layers of the higher-level DNN, the layers of the updated DNN are determined to be the same as the layers of the higher-level DNN. When ai_codec_layer_update_flag 2710 indicates that the layers of the updated DNN are updated with new values ​​without referring to the layers of the higher-level DNN, the parameters of the filter kernels in the layers of the updated DNN are determined differently from the parameters of the filter kernels in the layers of the higher-level DNN.

[0450] The weights and biases of the filter kernel in the updated DNN are determined according to ai_codec_weights 2712 as weight information and ai_codec_bias 2714 as bias information.

[0451] According to an embodiment of the present disclosure, filter core identification information may be included in the data structure 2700, wherein the filter core identification information indicates whether the parameters of the filter core in the updated DNN are the same as the parameters of the filter core in the higher-level DNN. When the filter core identification information indicates that the parameters of the filter core in the updated DNN are the same as the parameters of the filter core in the higher-level DNN, ai_codec_weights 2712 and ai_codec_bias 2714 may be omitted from the data structure 2700.

[0452] According to an embodiment of the present disclosure, ai_codec_weights 2712 may indicate the difference between the weights of the filter kernel in the higher-level DNN and the weights of the filter kernel in the updated DNN, wherein the weights may be arranged at the same position in each filter kernel. Similarly, ai_codec_bias 2714 may indicate the difference between the biases at the same position in the higher-level DNN and the updated DNN, respectively.

[0453] exist Fig. 27 The continuous dots shown in the figure indicate that the same data structure can be repeated. Therefore, the continuous dots below the information about the second layer indicate that the data structure of other layers can also be the same as the data structure of the first layer and the second layer.

[0454] Fig.28 The implementation of the embodiment according to the present disclosure is shown Fig. 27 The AI ​​codec high-level syntax 2800 includes a data structure 2700 of AI data related to the update of DNN setting information. Fig.28 The AI ​​codec high-level syntax 2800 can be applied to a frame group or a frame.

[0455] exist Fig.28 The left side of the table describes the order of obtaining syntax elements related to the update of DNN setting information and the conditions for obtaining syntax elements. In addition, Fig.28 The right side of the table shows how to extract syntax elements related to the update of DNN setting information from the bitstream.

[0456] like Fig.28As shown on the left side of the table, the AI ​​codec high-level syntax 2800 includes ai_codec_info, where ai_codec_info is AI target information indicating whether AI upscaling is applied to the current data unit. ai_codec_info is a syntax element having a size of 1 bit, and when ai_codec_info indicates '0', the AI ​​codec high-level syntax 2800 ends.

[0457] When ai_codec_info indicates "1", the AI ​​codec high-level syntax 2800 includes dependent_ai_condition_info as AI auxiliary data dependency information, wherein the AI ​​auxiliary data dependency information indicates whether AI auxiliary data for the previous data unit is applied to the current data unit. dependent_ai_condition_info is a syntax element having a size of 1 bit. When dependent_ai_condition_info indicates "1", the AI ​​codec high-level syntax 2800 ends.

[0458] When dependent_ai_condition_info indicates '0', the AI ​​codec high level syntax 2800 includes ai_codec_DNN_index as DNN index information. ai_codec_DNN_index indicates a syntax element encoded by the Exponential Golomb coding scheme.

[0459] The AI ​​codec high-level syntax 2800 includes ai_codec_DNN_update_flag as DNN update permission information. ai_codec_DNN_update_flag indicates a syntax element having a size of 1 bit. When ai_codec_DNN_update_flag indicates '0', the frame DNN setting information is determined with reference to the DNN setting information of a higher-level DNN indicated by ai_codec_DNN_index.

[0460] When ai_codec_DNN_update_flag indicates "1", ai_codec_bias[i][j][k] as bias information and ai_codec_weights[i][j][k][1][m] as weight information are included in the AI ​​codec high-level syntax 2800. According to an embodiment of the present disclosure, ai_codec_weights[i][j][k][l][m] may indicate the difference between the weight of the filter kernel in the higher-level DNN and the weight of the filter kernel in the updated DNN, the weights being arranged at the same position in the filter kernel. Similarly, ai_codec_bias[i][j][k] may indicate the difference between the biases at the same positions in the higher-level DNN and the updated DNN, respectively.

[0461] In addition, the AI ​​codec high-level syntax 2800 may include filter core identification information, wherein the filter core identification information indicates whether the parameters of the filter core in the updated DNN are the same as the parameters of the filter core in the higher-level DNN. When the filter core identification information indicates that the parameters of the filter core in the updated DNN are the same as the parameters of the filter core in the higher-level DNN, AI_codec_weights[i][j][k][1][m] and / or ai_codec_bias[i][j][k] may be omitted from the AI ​​codec high-level syntax 2800.

[0462] Fig.28 The AI ​​codec high-level syntax 2800 is only an example, and according to an embodiment of the present invention, the AI ​​codec high-level syntax 2800 may be excluded. Fig.28 Some of the grammatical elements of Fig.28 The syntax not shown in FIG. 2800 may be included in the AI ​​codec high-level syntax 2800. Fig.28 The AI ​​codec high-level syntax 2800 is implemented in different ways Fig. 27 The algorithm shown in .

[0463] Can be implemented in the form of SEI message Fig.28 AI codec high-level syntax 2800. Implementation for frame transmission Fig.28 The AI ​​codec high-level syntax 2800 of the SEI message.

[0464] According to an embodiment of the present disclosure, Fig.28ai_codec_info and dependent_ai_condition_info shown in FIG. 2800 may be included in syntax other than the AI ​​codec high-level syntax 2800. In this case, when both ai_codec_info and dependent_ai_condition_info included in other syntaxes indicate '0', the AI ​​codec high-level syntax 2800 may be set to be activated.

[0465] Fig.29 is a flowchart of an image decoding method performed by an AI decoding device according to an embodiment of the present disclosure.

[0466] DNN update permission information indicating whether to update one or more pieces of DNN setting information is obtained (operation 2910).

[0467] When the DNN update permission information indicates to update one or more pieces of DNN setting information, DNN update information for determining the one or more updated pieces of DNN setting information is obtained (operation 2920).

[0468] When the DNN update permission information indicates that one or more pieces of DNN setting information are not updated, the DNN update information is not obtained. Therefore, the third image can be generated by performing AI upscaling on the second image according to one or more pieces of default DNN setting information.

[0469] According to an embodiment of the present disclosure, the DNN update information may include DNN quantity information indicating the number of one or more updated DNN setting information. In addition, the DNN update information may include at least one of the following information: layer number information indicating the number of layers included in the DNN, input channel information indicating the number of input channels in the layer, output channel information indicating the number of output channels in the layer, filter kernel size information indicating the size of the filter kernel of the layer, weight information indicating the weight of the filter kernel, or bias information indicating the bias for adjusting the output of the filter kernel.

[0470] According to an embodiment of the present disclosure, the DNN update information may include default DNN reuse permission information, wherein the default DNN reuse permission information indicates whether information about the structure of the default DNN is reused for the updated DNN setting information. When the default DNN reuse permission information indicates that information about the structure of the default DNN will be reused for the updated DNN setting information, the DNN update information may include reference DNN index information indicating the index of the default DNN. In addition, the DNN update information may include at least one of weight information indicating the weight of the filter kernel or bias information indicating the bias for adjusting the output of the filter kernel. The weight information may include the difference between the weights in the default DNN and the updated DNN, respectively, and the bias information may indicate the difference between the biases in the default DNN and the updated DNN, respectively.

[0471] One or more updated DNN setting information is obtained according to the DNN update information (operation 2930).

[0472] According to an embodiment of the present disclosure, the amount of updated DNN setting information is determined based on DNN quantity information.

[0473] According to an embodiment of the present disclosure, when the default DNN reuse permission information indicates that information about the structure of the default DNN is reused for the updated DNN setting information, one or more updated DNN setting information may be determined based on the information about the structure of the default DNN indicated by the reference DNN index information and the DNN update information. For example, the number of layers in the updated DNN setting information, the number of input channels of each layer in the layers, the number of output channels of each layer, and the size of the filter kernel of each layer may be determined based on the number of layers in the default DNN indicated by the reference DNN index information, the number of input channels of each layer in the layers, the number of output channels of each layer, and the size of the filter kernel of each layer. In addition, the weights and biases of the filter kernels in the updated DNN setting information may be determined based on the DNN update information.

[0474] According to an embodiment of the present disclosure, operations 2910 to 2930 may be performed on a frame group including a plurality of frames. Alternatively, operations 2910 to 2930 may be performed on a video.

[0475] A third image is obtained by performing AI up-scaling on the second image according to the one or more pieces of updated DNN setting information (operation 2940).

[0476] According to an embodiment of the present disclosure, operation 2940 may be performed for a frame. Based on the DNN index information obtained for the frame, updated DNN setting information for the frame may be determined from the one or more updated DNN setting information. In addition, a third image in the frame may be obtained by performing AI amplification on the second image in the frame according to the updated DNN setting information.

[0477] According to an embodiment of the present disclosure, secondary DNN update permission information may be obtained, wherein the secondary DNN update permission information indicates whether a secondary update of updated DNN setting information determined for a frame will be performed. When the secondary DNN update permission information indicates a secondary update of updated DNN setting information, the secondary DNN update information may be obtained. In addition, a third image in the frame may be obtained by performing AI amplification on a second image in the frame according to the secondary updated DNN setting information determined from the secondary DNN update information.

[0478] According to an embodiment of the present disclosure, the determination of one or more pieces of updated DNN setting information according to operations 2910 to 2930 may be omitted. Instead, one or more pieces of default DNN setting information optimized for a video or a frame group may be obtained.

[0479] exist Fig.29 The image decoding method 2900 shown in FIG. 2 is only an example and is not identical to the reference Figures 14 to 28 The embodiments described in connection with updating the DNN setting information may be applied to Fig.29 Image decoding method 2900.

[0480] Fig.30 30 is a block diagram showing a configuration of an image decoding apparatus 3000 according to an embodiment of the present disclosure.

[0481] The image decoding apparatus 3000 may include a communicator 3010 , a processor 3020 , and a memory 3030 .

[0482] The communicator 3010 may receive AI data and image data from an external device (eg, a server) according to the control of the processor 3020 .

[0483] The processor 3020 may control all operations of the image decoding apparatus 3000. According to an embodiment of the present disclosure, the processor 3020 may execute one or more programs stored in the memory 3030. The processor 3020 may also execute the first decoder 1732 and the AI ​​amplifier 1734 ( Fig.17 The processor 3020 may include one or more processors.

[0484] According to an embodiment of the present disclosure, the processor 3020 may include a graphics processor 3022 and an AI dedicated processor 3024. In addition, the processor 3020 may be implemented in the form of a system on chip (SoC) including at least one of the graphics processor 3022 or the AI ​​dedicated processor 3024. The graphics processor 3022 controls all operations of the image decoding device 3000 and signal flows between internal components of the image decoding device 3000 and processes data.

[0485] The graphics processor 3022 is a processor with a design specifically for decoding and post-processing images. Therefore, the graphics processor 3022 can perform the calculations required to process the image data received through the communicator 3010 or retrieved from the memory of the image decoding device 3000. Therefore, the low-resolution image reconstruction function of the first decoder 1732 can be efficiently performed by the graphics processor 3022. The AI ​​dedicated processor 3024 is a processor with a design specifically for AI calculations. Therefore, the AI ​​dedicated processor 3024 can effectively perform the calculations required for the AI ​​amplifier 1734 to perform the AI ​​amplification function of the low-resolution image. Specifically, the AI ​​dedicated processor 3024 can perform the calculations required for the DNN controller 1736 to update the DNN and determine the DNN setting information. In addition, the AI ​​dedicated processor 3024 can perform the calculations required for the image processor 1738 to perform AI amplification on the low-resolution image.

[0486] The image data and AI data input to the image decoding device 3000 via the communicator 3010 are processed by the processor 3020. The graphics processor 3022 may decode the image data to generate a low-resolution image. The AI ​​dedicated processor 3024 may then perform AI upscaling on the low-resolution image generated by the graphics processor 3022. Thus, the AI ​​dedicated processor 3024 may generate a high-resolution image by performing AI upscaling on the low-resolution image. In addition, the upscaling DNN necessary for AI upscaling may be set by the AI ​​dedicated processor 3024.

[0487] Although the processor 3020 has been described as including one graphics processor 3022, according to an embodiment of the present disclosure, the processor 3020 may include two or more graphics processors 3022. In addition, although the processor 3020 has been described as including one AI dedicated processor 3024, according to an embodiment of the present disclosure, the processor 3020 may include two or more AI dedicated processors 3024. In addition, the processor 3020 may include one or more general-purpose processors. Additional processing required for AI amplification may be performed under the control of one or more general-purpose processors. The AI ​​dedicated processor 3024 may be implemented as a hardware field programmable gate array (FPGA).

[0488] According to an embodiment of the present disclosure, the memory 3030 may store various data, programs, or applications for driving and controlling the image decoding device 3000. The program stored in the memory 3030 may include one or more instructions. In addition, the program (one or more instructions) or application stored in the memory 3030 may be executed by the processor 3020. The memory 3030 may store data from the communicator 3010 and the processor 3020. In addition, the memory 3030 may send data requested by the processor 3020 to the processor 3020.

[0489] According to an embodiment of the present disclosure, the image decoding device 3000 may perform a reference Figure 2 The functions and references of the AI ​​decoding device 200 are described Fig.29 At least one of the operations of the described image decoding method.

[0490] Available through Fig.30 The image decoding device 3000 implements Fig.29 The image decoding method 2900. With reference to Figures 17 to 28 The technical features associated with updating the DNN setting information described can be applied by Fig.30 The image decoding device 3000 performs Fig.29 Image decoding method 2900.

[0491] According to various embodiments of the present disclosure, Fig.30 The image decoding device 3000 performs Fig.29 The image decoding method 2900 may be written as a program executable on a computer. Figure 23 to Figure 28 The described data structure 2320, data structure 2400, data structure 2500, AI codec high-level syntax 2600, data structure 2700, and AI codec high-level syntax 2800, therefore, DNN setting information for AI amplification can be determined.

[0492] In addition, the above-described embodiments of the present disclosure may be written as computer-executable programs or instructions that may be stored in a storage medium.

[0493] The storage medium can store computer executable programs or instructions continuously, or temporarily store computer executable programs or instructions for execution or download. In addition, the storage medium can be any of various recording media or storage media combined with a single piece of hardware or multiple pieces of hardware, and the storage medium is not limited to a medium directly connected to a computer system, but can be distributed on a network. Here, the storage medium may include one or more computer-readable storage media representing any type of physical memory that can store information or data that can be read by a processor. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals (i.e., non-temporary). Examples of storage media include magnetic media (such as hard disks, floppy disks, and tapes) configured to store program instructions, optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as optical floppy disks), and ROM, RAM, and flash memory. Other examples of media include recording media and storage media managed by application stores that distribute applications or by websites, servers, etc. that provide or distribute various other types of software.

[0494] In addition, the model related to the above-mentioned DNN can be implemented via a software module. When the DNN model is implemented via a software module (eg, a program module including instructions), the DNN model can be stored in a computer-readable recording medium.

[0495] In addition, the DNN model can be integrated in the form of a hardware chip to become part of the above-mentioned AI decoding device 200 or AI encoding device 600. For example, the DNN model can be manufactured in the form of a dedicated hardware chip for AI, or can be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU).

[0496] In addition, the DNN model can be provided in the form of downloadable software. The computer program product may include a product in the form of a software program distributed electronically by a manufacturer or an electronic market (e.g., a downloadable application). For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a server of a manufacturer or an electronic market, or a relay server.

[0497] According to an embodiment of the present disclosure, a method for decoding an image by using updated DNN setting information for image upscaling is provided. The quality of the upscaled image can be improved by using a DNN configured according to the DNN setting information optimized for the image.

[0498] While one or more embodiments of the present disclosure have been shown and described above, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope as defined by the following claims.

Claims

1. An image decoding method, comprising: obtaining, for a frame group including a plurality of frames, DNN update permission information indicating whether to update one or more pieces of default deep neural network (DNN) setting information; Based on the DNN update permission information indicating updating the one or more pieces of default DNN setting information, obtaining, for the frame group, DNN update information necessary to determine the one or more pieces of updated DNN setting information, wherein the DNN update information includes DNN quantity information indicating the quantity of the one or more pieces of updated DNN setting information for the frame group; Determining the one or more updated DNN setting information according to the DNN update information; For a frame in the frame group, obtaining DNN index information indicating updated DNN setting information for the frame among the one or more updated DNN setting information; Determining updated DNN setting information for the frame based on the DNN index information; and A third image is obtained by performing artificial intelligence (AI) upscaling on a second image corresponding to the frame according to the determined updated DNN setting information.

2. The image decoding method according to claim 1, in, The step of determining the one or more updated DNN setting information includes determining whether the number of the one or more updated DNN setting information matches the number indicated by the DNN number information.

3. The image decoding method according to claim 1, wherein: The DNN update information also includes at least one of the following information: layer number information indicating the number of layers included in the DNN, input channel information indicating the number of input channels in each layer, output channel information indicating the number of output channels in each layer, filter kernel size information indicating the size of the filter kernel in each layer, weight information indicating the weight of the filter kernel, and bias information indicating the bias used to adjust the output of the filter kernel.

4. The image decoding method according to claim 1, wherein: The step of obtaining the DNN update information includes: obtaining default DNN reuse permission information indicating whether a structure of a default DNN is reused for the one or more pieces of updated DNN setting information; and obtaining reference DNN index information based on the default DNN reuse permission information indicating that the structure of the default DNN is reused for the one or more updated DNN setting information, wherein the reference DNN index information indicates an index of the default DNN whose information is reused, and Among them, the step of determining the one or more updated DNN setting information includes: determining the one or more updated DNN setting information based on the structure of the default DNN indicated by the reference DNN index information and the DNN update information.

5. The image decoding method according to claim 4, wherein: The step of determining the one or more updated DNN setting information further includes: determining the number of layers, the number of input channels in each layer, the number of output channels in each layer, and the size of the filter kernel of each layer according to the number of default layers in the default DNN indicated by the reference DNN index information, the number of input channels in each default layer, the number of output channels in each default layer, and the size of the filter kernel of each default layer; and The weights and biases of the filter kernels in the one or more updated DNN setting information are determined based on the DNN update information.

6. The image decoding method according to claim 5, wherein: The DNN update information includes at least one of weight information indicating a weight of a filter kernel or bias information indicating a bias for adjusting an output of the filter kernel, The weight information indicates a difference between weights in the default DNN and the updated DNN, respectively, and The bias information indicates a difference between biases in the default DNN and the updated DNN, respectively.

7. The image decoding method according to claim 1, wherein: The frame group is an encoded video sequence including the plurality of frames.

8. The image decoding method according to claim 1, wherein: The step of obtaining the third image comprises: obtaining secondary DNN update permission information, wherein the secondary DNN update permission information indicates whether a secondary update of updated DNN setting information for the frame determined according to the DNN index information will be performed; obtaining secondary DNN update information based on the secondary DNN update permission information indicating that a secondary update of updated DNN setting information for the frame is to be performed; and The third image is obtained by performing AI upscaling on the second image using secondary updated DNN setting information determined according to the secondary DNN update information.

9. An image decoding device, comprising: a memory storing one or more instructions; as well as A processor configured to execute the one or more instructions stored in the memory to perform the following operations: obtaining, for a frame group including a plurality of frames, DNN update permission information indicating whether to update one or more pieces of default deep neural network (DNN) setting information; Based on the DNN update permission information indicating updating the one or more pieces of default DNN setting information, obtaining, for the frame group, DNN update information necessary to determine the one or more pieces of updated DNN setting information, wherein the DNN update information includes DNN quantity information indicating the quantity of the one or more pieces of updated DNN setting information for the frame group; Determining the one or more updated DNN setting information according to the DNN update information; For a frame in the frame group, obtaining DNN index information indicating updated DNN setting information for the frame among the one or more updated DNN setting information; Determining updated DNN setting information for the frame based on the DNN index information; and A third image is obtained by performing artificial intelligence (AI) upscaling on a second image corresponding to the frame according to the determined updated DNN setting information.

10. A method for storing artificial intelligence (AI) data, comprising: storing the AI ​​data on a non-transitory computer-readable recording medium, wherein the AI ​​data includes DNN update permission information indicating whether to update one or more pieces of default deep neural network (DNN) setting information for a frame group including a plurality of frames, Based on the DNN update permission information indicating updating the one or more pieces of default DNN setting information, the AI ​​data further includes: DNN update information necessary for determining one or more pieces of updated DNN setting information for the frame group, and DNN index information indicating updated DNN setting information for the frame among the one or more pieces of updated DNN setting information for the frame in the frame group, wherein the DNN update information includes DNN quantity information indicating the quantity of the one or more updated DNN setting information for the frame group, and The updated DNN setting information indicated by the DNN index information is used to perform artificial intelligence (AI) upscaling on the second image corresponding to the frame to obtain a third image.

11. The method according to claim 10, wherein: The DNN update information further includes: default DNN reuse permission information indicating whether a structure of a default DNN is reused for the one or more updated DNN setting information, and wherein, when the default DNN reuse permission information indicates that the structure of the default DNN is reused for the one or more updated DNN setting information, the DNN update information further includes reference DNN index information, wherein the reference DNN index information indicates an index of the default DNN whose information is reused, and The DNN update information also includes: information related to the update of default DNN setting information in which the structure of the default DNN indicated by the reference DNN index information is not reused.

12. The method according to claim 11, wherein: The information that the structure of the default DNN indicated by the reference DNN index information is reused includes: the number of layers, the number of input channels in each layer, the number of output channels in each layer, and the size of the filter kernel of each layer, Among them, the information that is not reused in the structure of the default DNN indicated by the reference DNN index information includes the weights and biases of the filter kernel.

13. The method according to claim 10, wherein: setting the DNN index information for each of the plurality of frames in the frame group, and The AI ​​data also includes: secondary DNN update permission information indicating whether a secondary update of updated DNN setting information determined according to the DNN index information is to be performed, and When the secondary DNN update permission information indicates that a secondary update of updated DNN setting information determined according to the DNN index information will be performed, secondary DNN update information necessary for the secondary updated DNN setting information for AI upscaling of the second image is determined.

Images