A method and an apparatus for performing artificial intelligence encoding and artificial intelligence decoding

Abstract

An AI encoding device is proposed comprising a memory and a processor, wherein the memory stores instructions executable by the processor, and the processor acquires an original image, previously encoded frame information, and network environment information; acquires deblocking filter setting information based on the original image, previously encoded frame information, and network environment information; applies deblocking filtering to the original image based on the deblocking filter setting information and acquires a first image AI downscaled through a DNN for downscaled; generates image data by first encoding the first image; and transmits AI data including the deblocking filter setting information and information related to the AI ​​downscaled, and the image data.

Description

Technology Field

[0001] The present disclosure relates to a method and apparatus for processing images, and more specifically, the present disclosure relates to a method and apparatus for performing artificial intelligence (AI) encoding and artificial intelligence decoding. Background Technology

[0002] The video is encoded by a codec that follows a specified data compression standard, such as the MPEG (Moving Picture Expert Group) standard, and then stored in the form of a bitstream on a recording medium or transmitted through a communication channel.

[0003] With the development and widespread adoption of hardware capable of playing and storing high-resolution / high-quality video, the need for codecs capable of effectively encoding and decoding high-resolution / high-quality video is increasing. The problem to be solved

[0004] The AI ​​encoding and AI decoding method and apparatus for an image according to one embodiment have the technical objective of predicting the occurrence of blocking artifacts in a variable and limited network environment in a real-time streaming service, additionally performing deblocking filtering, and encoding and decoding the image by downscaling and upscaling based on AI. means of solving the problem

[0005] To solve the above technical problem, the AI ​​decoding device proposed in the present disclosure includes: a memory that stores one or more instructions; and a processor that executes the one or more instructions stored in the memory. The processor obtains image data generated as a result of a first encoding of a first image, AI data related to AI downscaling from an original image to the first image, and deblocking filter setting information. It obtains a second image corresponding to the first image by first decoding the image data, obtains DNN setting information for AI upscaling of the second image among a plurality of DNN setting information based on the AI ​​data, and generates a restored image that is AI upscaled from the second image and has deblocking filtering applied based on the deblocking filter setting information through a Deep Neural Network (DNN) for upscaling that operates with the obtained DNN setting information.

[0006] To solve the above technical problem, the AI ​​decoding method proposed in the present disclosure may include: a step of obtaining image data generated as a result of a first encoding of a first image, AI data related to AI downscaling from an original image to the first image, and deblocking filter setting information; a step of obtaining a second image corresponding to the first image by first decoding the image data; a step of obtaining DNN setting information for AI upscaling of the second image among a plurality of DNN setting information based on the AI ​​data; and a step of generating a restored image that is AI upscaled from the second image and has deblocking filtering applied based on the deblocking filter setting information through a Deep Neural Network (DNN) for upscaling that operates with the obtained DNN setting information.

[0007] To solve the above technical problem, the AI ​​encoding device proposed in the present disclosure includes a memory for storing one or more instructions; and a processor for executing the one or more instructions stored in the memory. The processor acquires an original image, previously encoded frame information, and network environment information, acquires deblocking filter setting information based on the original image, the previously encoded frame information, and the network environment information, applies deblocking filtering to the original image based on the deblocking filter setting information, acquires a first image AI downscaled through a DNN for downscaled, generates image data by first encoding the first image, and can transmit AI data including the deblocking filter setting information and information related to the AI ​​downscaled, and the image data.

[0008] To solve the above technical problem, the AI ​​encoding method proposed in the present disclosure may include: a step of acquiring an original image, previously encoded frame information, and network environment information; a step of acquiring deblocking filter setting information based on the original image, the previously encoded frame information, and the network environment information; a step of applying deblocking filtering to the original image based on the deblocking filter setting information and acquiring a first image AI downscaled through a DNN for downscaled; a step of generating image data by first encoding the first image; and a step of transmitting the AI ​​data including the deblocking filter setting information and information related to the AI ​​downscaled, and the image data. Effects of the invention

[0009] During encoding, deblocking filter setting information is obtained based on the original image, previous frame information, and network environment information; deblocking filtering is applied to the original image based on the deblocking filter setting information; the deblocking filtered original image is AI downscaled and the downscaled image and deblocking filter setting information are transmitted; and during decoding, the transmitted downscaled image is AI upscaled to obtain an upscaled image; and deblocking filtering is applied to the upscaled image based on the transmitted deblocking filter setting information to obtain a restored image, thereby enabling high-resolution images to be effectively encoded and decoded based on AI even when network conditions change suddenly. Brief explanation of the drawing

[0010] A brief description of each drawing is provided to help to better understand the drawings cited in this specification. FIG. 1 is a diagram illustrating an AI encoding process and an AI decoding process according to one embodiment. FIG. 2 is a block diagram showing the configuration of an AI decoding device according to one embodiment. Figure 3 is an exemplary diagram showing a second DNN for AI upscaling of a second image. Figure 4 is a diagram illustrating a convolution operation by a convolution layer. Figure 5 is an exemplary diagram showing the mapping relationship between various image-related information and various DNN setting information. FIG. 6 is a drawing illustrating a second image composed of multiple frames. FIG. 7 is a block diagram showing the configuration of an AI encoding device according to one embodiment. Figure 8 is an exemplary diagram showing a first DNN for AI downscaling of an original image. FIG. 9 is a diagram showing the structure of AI encoding data according to one embodiment. FIG. 10 is a diagram showing the structure of AI encoding data according to another embodiment. Figure 11 is a diagram illustrating a method for training the first DNN and the second DNN. Figure 12 is a diagram illustrating the training process of the first DNN and the second DNN by a training device. FIG. 13 is a diagram illustrating an AI encoding process and an AI decoding process that add deblocking filtering according to one embodiment. FIG. 14 is a block diagram showing the configuration of an AI decoding device according to one embodiment. FIG. 15 is a block diagram showing the configuration of an AI encoding device according to one embodiment. FIG. 16 is a diagram illustrating an AI encoding process and an AI decoding process that adaptively add deblocking filtering according to another embodiment. FIG. 17 is a diagram illustrating an AI encoding process and an AI decoding process that adaptively add deblocking filtering according to another embodiment. FIG. 18 is a diagram illustrating an AI encoding process and an AI decoding process that adaptively add deblocking filtering according to another embodiment. FIG. 19 is an exemplary diagram showing a third DNN and a fourth DNN for an AI-based deblocking filter. FIG. 20 is a diagram illustrating a method for training the third DNN (2010) and the fourth DNN (2020). FIG. 21 is a diagram illustrating the training process of the third DNN (2010) and the fourth DNN (2020) by the training device (2100). FIG. 22 is a flowchart for explaining an AI encoding method according to one embodiment. FIG. 23 is a flowchart illustrating an AI decoding method according to one embodiment. Specific details for implementing the invention

[0011] The present disclosure is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the embodiments of the present disclosure, and it should be understood that the present disclosure includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the various embodiments.

[0012] In describing the embodiments, if it is determined that a detailed description of related prior art could unnecessarily obscure the gist of the present disclosure, such detailed description is omitted. Additionally, numbers used in the description of the specification (e.g., first, second, etc.) are merely identifiers to distinguish one component from another.

[0013] In addition, when a component is described in this specification as being "connected" or "connected" to another component, it should be understood that the component may be directly connected to or directly connected to the other component, but unless otherwise specifically stated, it may also be connected or connected through another component in between.

[0014] In addition, components expressed as '~part (unit)', 'module', etc. in this specification may consist of two or more components combined into a single component, or a single component may be divided into two or more components according to more detailed functions. Furthermore, each component described below may additionally perform some or all of the functions performed by other components in addition to the primary function it is responsible for, and it goes without saying that some of the primary functions performed by each component may be exclusively performed by other components.

[0015] Additionally, in this specification, 'image' or 'picture' may refer to a still image, a video composed of a plurality of consecutive still images (or frames), or a video.

[0016] Additionally, in this specification, 'DNN (deep neural network)' is a representative example of an artificial neural network model that mimics brain neurons and is not limited to an artificial neural network model using a specific algorithm.

[0017] Additionally, in this specification, 'parameter' refers to a value used in the computation process of each layer constituting a neural network, and may include, for example, weights used when applying input values ​​to a predetermined calculation formula. Furthermore, parameters may be expressed in the form of a matrix. Parameters are values ​​set as a result of training and may be updated through separate training data as needed.

[0018] In addition, in this specification, 'first DNN' refers to a DNN used for AI downscaling of an image, and 'second DNN' refers to a DNN used for AI upscaling of an image.

[0019] In addition, in this specification, 'DNN configuration information' is information related to elements constituting a DNN and includes the aforementioned parameters. A first DNN or a second DNN can be configured using the DNN configuration information.

[0020] Additionally, in this specification, 'original image' refers to an image that is the subject of AI encoding, and 'first image' refers to an image obtained as a result of AI downscaling of the original image during the AI ​​encoding process. Additionally, 'second image' refers to an image obtained through the first decoding during the AI ​​decoding process, and 'third image' refers to an image obtained by AI upscaling the second image during the AI ​​decoding process.

[0021] Additionally, in this specification, 'AI downscale' refers to a process of reducing the resolution of an image based on AI, and 'first encoding' refers to encoding processing by a frequency conversion-based image compression method. Additionally, 'first decoding' refers to decoding processing by a frequency conversion-based image restoration method, and 'AI upscale' refers to a process of increasing the resolution of an image based on AI.

[0022] FIG. 1 is a diagram illustrating an artificial intelligence (AI) encoding process and an AI decoding process according to one embodiment.

[0023] As mentioned above, as image resolution increases rapidly, the amount of information processing required for encoding and decoding increases; accordingly, measures are needed to improve the efficiency of image encoding and decoding.

[0024] As illustrated in FIG. 1, according to one embodiment of the present disclosure, a first image (115) is obtained by AI downscaling (110) an original image (105) with high resolution. Then, since a first encoding (120) and a first decoding (130) are performed on the first image (115) with relatively low resolution, the bitrate can be significantly reduced compared to the case where the first encoding (120) and the first decoding (130) are performed on the original image (105).

[0025] Referring to FIG. 1, the description is as follows: In one embodiment, during the AI ​​encoding process, an original image (105) is AI downscaled (110) to obtain a first image (115), and the first image (115) is first encoded (120). In the AI ​​decoding process, AI encoded data including AI data and image data obtained as a result of AI encoding is received, a second image (135) is obtained through first decoding (130), and a third image (145) is obtained by AI upscaled (140) the second image (135).

[0026] Looking more closely at the AI ​​encoding process, when an original video (105) is input, the original video (105) is AI downscaled (110) to obtain a first video (115) of a predetermined resolution and / or predetermined quality. At this time, the AI ​​downscaled (110) is performed based on AI, and the AI ​​for the AI ​​downscaled (110) must be jointly trained with the AI ​​for the AI ​​upscaled (140) of the second video (135). This is because if the AI ​​for the AI ​​downscaled (110) and the AI ​​for the AI ​​upscaled (140) are trained separately, the difference between the original video (105) that is the subject of AI encoding and the third video (145) restored through AI decoding becomes large.

[0027] In an embodiment of the present disclosure, AI data may be used to maintain this relationship during the AI ​​encoding process and the AI ​​decoding process. Accordingly, the AI ​​data obtained through the AI ​​encoding process must include information indicating an upscale target, and during the AI ​​decoding process, the second image (135) must be AI upscaled (140) according to the upscale target identified based on the AI ​​data.

[0028] AI for AI downscaling (110) and AI for AI upscaling (140) can be implemented as a deep neural network (DNN). As described below with reference to FIG. 11, the first DNN and the second DNN are trained in conjunction by sharing loss information under a predetermined target. Therefore, the AI ​​encoding device provides the target information used when the first DNN and the second DNN are trained in conjunction to the AI ​​decoder, and the AI ​​decoder can AI upscale (140) the second image (135) to the target quality and / or resolution based on the provided target information.

[0029] To explain in detail the first encoding (120) and the first decoding (130) illustrated in FIG. 1, the first image (115) that has been AI downscaled (110) from the original image (105) can have its amount of information reduced through the first encoding (120). The first encoding (120) may include a process of predicting the first image (115) to generate prediction data, a process of generating residual data corresponding to the difference between the first image (115) and the prediction data, a process of transforming the residual data, which is a spatial domain component, into a frequency domain component, a process of quantizing the residual data transformed into a frequency domain component, and a process of entropy encoding the quantized residual data. This first encoding process (120) can be implemented through one of the video compression methods using frequency conversion, such as MPEG-2, H.264 AVC (Advanced Video Coding), MPEG-4, HEVC (High Efficiency Video Coding), VC-1, VP8, VP9, ​​and AV1 (AOMedia Video 1).

[0030] A second image (135) corresponding to a first image (115) can be restored through a first decoding (130) of image data. The first decoding (130) may include a process of generating quantized residual data by entropy decoding image data, a process of inversely quantizing the quantized residual data, a process of converting residual data of frequency domain components into spatial domain components, a process of generating prediction data, and a process of restoring the second image (135) using the prediction data and residual data. Such a first decoding (130) process can be implemented through an image restoration method corresponding to one of the image compression methods using frequency conversion, such as MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1, used in the first encoding (120) process.

[0031] The AI ​​encoded data obtained through the AI ​​encoding process may include image data obtained as a result of the first encoding (120) of the first image (115) and AI data related to the AI ​​downscale (110) of the original image (105). The image data may be used in the first decoding (130) process, and the AI ​​data may be used in the AI ​​upscale (140) process.

[0032] The image data may be transmitted in the form of a bitstream. The image data may include data obtained based on pixel values ​​within the first image (115), for example, residual data which is the difference between the first image (115) and the prediction data of the first image (115). Additionally, the image data includes information used in the first encoding (120) process of the first image (115). For example, the image data may include prediction mode information, motion information, and quantization parameter-related information used in the first encoding (120) used to first encode (120) the first image (115). The image data may be generated according to the rules, for example, the syntax, of the image compression method used in the first encoding (120) process among image compression methods using frequency conversion, such as MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1.

[0033] AI data is used for AI upscaling (140) based on the second DNN. As described above, since the first DNN and the second DNN are trained in conjunction, the AI ​​data contains information that enables accurate AI upscaling (140) of the second image (135) through the second DNN. In the AI ​​decoding process, the second image (135) can be AI upscaled (140) to a target resolution and / or quality based on the AI ​​data.

[0034] AI data can be transmitted along with video data in the form of a bitstream. Depending on the implementation, AI data may also be transmitted separately from video data in the form of frames or packets.

[0035] Alternatively, depending on the implementation example, AI data may be included in and transmitted with video data.

[0036] Video data and AI data can be transmitted through the same network or different networks.

[0037] FIG. 2 is a block diagram showing the configuration of an AI decoding device (200) according to one embodiment.

[0038] Referring to FIG. 2, an AI decoding device (200) according to one embodiment includes a receiving unit (210) and an AI decoding unit (230). The AI ​​decoding unit (230) may include a parsing unit (232), a first decoding unit (234), an AI upscaling unit (236), and an AI setting unit (238).

[0039] In FIG. 2, the receiver (210) and the AI ​​decoder (230) are depicted as separate devices, but the receiver (210) and the AI ​​decoder (230) can be implemented through a single processor. In this case, the receiver (210) and the AI ​​decoder (230) may be implemented as a dedicated processor, or they may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU) and software. Additionally, in the case of a dedicated processor, it may include memory for implementing the embodiment of the present disclosure or a memory processing unit for using external memory.

[0040] The receiver (210) and the AI ​​decoding unit (230) may be composed of multiple processors. In this case, they may be implemented by a combination of dedicated processors, or by a combination of multiple general-purpose processors such as AP, CPU, and GPU, and S / W. In one embodiment, the receiver (210) is implemented as a first processor, the first decoding unit (234) is implemented as a second processor different from the first processor, and the parsing unit (232), the AI ​​upscaling unit (236), and the AI ​​setting unit (238) may be implemented as a third processor different from the first processor and the second processor.

[0041] The receiver (210) receives AI encoded data obtained as a result of AI encoding. For example, the AI ​​encoded data may be a video file having a file format such as mp4, mov, etc.

[0042] The receiver (210) can receive AI encoded data transmitted through the network. The receiver (210) outputs the AI ​​encoded data to the AI ​​decoder (230).

[0043] In one embodiment, the AI ​​encoded data may be obtained from a data storage medium including a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, etc.

[0044] The parsing unit (232) parses the AI ​​encoded data and transmits the image data generated as a first encoding result of the first image (115) to the first decoding unit (234), and transmits the AI ​​data to the AI ​​setting unit (238).

[0045] In one embodiment, the parsing unit (232) can parse video data and AI data that are separated and included within the AI ​​encoded data. The parsing unit (232) can distinguish between the AI ​​data and video data included within the AI ​​encoded data by reading the header within the AI ​​encoded data. In one example, the AI ​​data may be included in the VSIF (Vendor Specific InfoFrame) within the HDMI stream.

[0046] The structure of AI encoded data including AI data and video data separated from each other will be described later with reference to FIG. 9.

[0047] In another embodiment, the parsing unit (232) parses image data from the AI ​​encoded data, extracts AI data from the image data, transmits the AI ​​data to the AI ​​setting unit (238), and transmits the remaining image data to the first decoding unit (234). That is, the AI ​​data may be included in the image data; for example, the AI ​​data may be included in the SEI (Supplemental enhancement information), which is an additional information area of ​​the bitstream corresponding to the image data. The structure of the AI ​​encoded data including the image data containing the AI ​​data will be described later with reference to FIG. 10.

[0048] In another embodiment, the parsing unit (232) may divide a bitstream corresponding to image data into a bitstream to be processed by the first decoding unit (234) and a bitstream corresponding to AI data, and output each divided bitstream to the first decoding unit (234) and the AI ​​setting unit (238).

[0049] The parsing unit (232) may determine that the video data included in the AI ​​encoded data is video data obtained through a predetermined codec (e.g., MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1). In this case, the information may be transmitted to the first decoding unit (234) so ​​that the video data can be processed by the confirmed codec.

[0050] The first decoding unit (234) restores a second image (135) corresponding to the first image (115) based on image data received from the parsing unit (232). The second image (135) obtained by the first decoding unit (234) is provided to the AI ​​upscaling unit (236).

[0051] According to an embodiment, first decoding-related information, such as prediction mode information, motion information, and quantization parameter information, may be provided from the first decoding unit (234) to the AI ​​setting unit (238). The first decoding-related information may be used to obtain DNN setting information.

[0052] The AI ​​data provided to the AI ​​setting unit (238) includes information that enables AI upscaling of the second image (135). At this time, the upscaling target of the second image (135) must correspond to the downscaling target of the first DNN. Therefore, the AI ​​data must include information that can identify the downscaling target of the first DNN.

[0053] Specifically, examples of the information included in the AI ​​data include information on the difference between the resolution of the original image (105) and the resolution of the first image (115), and information related to the first image (115).

[0054] The difference information can be expressed as information regarding the degree of resolution conversion of the first image (115) relative to the original image (105) (e.g., resolution conversion rate information). Also, since the resolution of the first image (115) can be determined through the resolution of the restored second image (135) and the degree of resolution conversion can be verified through this, the difference information may be expressed only by the resolution information of the original image (105). Here, the resolution information may be expressed as a horizontal / vertical screen size, or as a ratio (16:9, 4:3, etc.) and a size of one axis. Additionally, if there is pre-set resolution information, it may be expressed in the form of an index or a flag.

[0055] In addition, information related to the first image (115) may include information regarding at least one of the resolution of the first image (115), the bit rate of the image data obtained as a result of the first encoding of the first image (115), and the type of codec used during the first encoding of the first image (115).

[0056] The AI ​​setting unit (238) can determine an upscale target for the second image (135) based on at least one of the difference information included in the AI ​​data and information related to the first image (115). The upscale target may indicate, for example, to what resolution the second image (135) should be upscaled. When the upscale target is determined, the AI ​​upscale unit (236) AI upscales the second image (135) through the second DNN to obtain a third image (145) corresponding to the upscale target.

[0057] Before explaining how the AI ​​setting unit (238) determines an upscale target based on AI data, the AI ​​upscale process through the second DNN is explained with reference to FIG. 3 and FIG. 4.

[0058] FIG. 3 is an exemplary diagram showing a second DNN (300) for AI upscaling of a second image (135), and FIG. 4 shows a convolution operation in the first convolution layer (310) shown in FIG. 3.

[0059] As illustrated in FIG. 3, the second image (135) is input into the first convolution layer (310). The 3x3x4 shown in FIG. 3 illustrates convolution processing on one input image using four filter kernels of size 3 x 3. As a result of the convolution processing, four feature maps are generated by the four filter kernels. Each feature map represents unique characteristics of the second image (135). For example, each feature map may represent vertical characteristics, horizontal characteristics, or edge characteristics of the second image (135).

[0060] Referring to FIG. 4, the convolution operation in the first convolution layer (310) will be described in detail.

[0061] A single feature map (450) can be generated through multiplication and addition operations between the parameters of a filter kernel (430) having a size of 3 x 3 used in the first convolution layer (310) and the corresponding pixel values ​​in the second image (135). Since four filter kernels are used in the first convolution layer (310), four feature maps can be generated through a convolution operation process using four filter kernels.

[0062] In FIG. 4, I1 to I49 displayed in the second image (135) represent pixels of the second image (135), and F1 to F9 displayed in the filter kernel (430) represent parameters of the filter kernel (430). Additionally, M1 to M9 displayed in the feature map (450) represent samples of the feature map (450).

[0063] FIG. 4 illustrates that the second image (135) contains 49 pixels, but this is merely an example, and if the second image (135) has a resolution of 4K, for example, it may contain 3840 x 2160 pixels.

[0064] In the convolution operation, each of the pixel values ​​of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the second image (135) and each of F1, F2, F3, F4, F5, F6, F7, F8, and F9 of the filter kernel (430) are multiplied, and the result of the multiplication operation combined (e.g., addition operation) can be assigned as the value of M1 of the feature map (450). If the stride of the convolution operation is 2, the multiplication operation is performed on each of the pixel values ​​of I3, I4, I5, I10, I11, I12, I17, I18, and I19 of the second image (135) and each of F1, F2, F3, F4, F5, F6, F7, F8, and F9 of the filter kernel (430), and the combined result of the multiplication operation can be assigned as the value of M2 of the feature map (450).

[0065] A feature map (450) having a predetermined size can be obtained by performing a convolution operation between the pixel values ​​in the second image (135) and the parameters of the filter kernel (430) while the filter kernel (430) moves according to the stride until it reaches the last pixel of the second image (135).

[0066] According to the present disclosure, through the combined training of the first DNN and the second DNN, the values ​​of the parameters of the second DNN, for example, the parameters of the filter kernel used in the convolution layers of the second DNN (e.g., F1, F2, F3, F4, F5, F6, F7, F8, and F9 of the filter kernel (430)), can be optimized. The AI ​​setting unit (238) can determine an upscale target corresponding to a downscale target of the first DNN based on AI data, and determine the parameters corresponding to the determined upscale target as the parameters of the filter kernel used in the convolution layers of the second DNN.

[0067] The convolution layers included in the first DNN and the second DNN can perform processing according to the convolution operation process described in relation to FIG. 4, but the convolution operation process described in FIG. 4 is merely one example and is not limited thereto.

[0068] Referring again to FIG. 3, the feature maps output from the first convolution layer (310) are input to the first activation layer (320).

[0069] The first activation layer (320) can impart non-linear characteristics to each feature map. The first activation layer (320) may include a sigmoid function, a Tanh function, a ReLU (Rectified Linear Unit) function, etc., but is not limited thereto.

[0070] Imposing non-linear characteristics in the first activation layer (320) means changing and outputting some sample values ​​of the feature map, which is the output of the first convolution layer (310). At this time, the change is performed by applying non-linear characteristics.

[0071] The first activation layer (320) determines whether to pass sample values ​​of feature maps output from the first convolution layer (310) to the second convolution layer (330). For example, some sample values ​​of the feature maps are activated by the first activation layer (320) and passed to the second convolution layer (330), while some sample values ​​are deactivated by the first activation layer (320) and not passed to the second convolution layer (330). The unique characteristics of the second image (135) represented by the feature maps are emphasized by the first activation layer (320).

[0072] The feature maps (325) output from the first activation layer (320) are input to the second convolution layer (330). One of the feature maps (325) illustrated in FIG. 3 is the result of processing the feature map (450) described in relation to FIG. 4 in the first activation layer (320).

[0073] The 3x3x4 shown in the second convolution layer (330) exemplifies convolution processing on input feature maps (325) using four filter kernels of size 3 x 3. The output of the second convolution layer (330) is input to the second activation layer (340). The second activation layer (340) can impart non-linear characteristics to the input data.

[0074] Feature maps (345) output from the second activation layer (340) are input to the third convolution layer (350). The 3x3x1 shown in the third convolution layer (350) illustrated in FIG. 3 exemplifies performing convolution processing to create one output image using one filter kernel of size 3 x 3. The third convolution layer (350) is a layer for outputting a final image and generates one output using one filter kernel. According to an example of the present disclosure, the third convolution layer (350) can output a third image (145) through convolution operations.

[0075] DNN configuration information indicating the number of filter kernels, parameters of filter kernels, etc. of the first convolution layer (310), the second convolution layer (330), and the third convolution layer (350) of the second DNN (300) may be multiple as described below, and the multiple DNN configuration information must be linked with the multiple DNN configuration information of the first DNN. The linkage between the multiple DNN configuration information of the second DNN and the multiple DNN configuration information of the first DNN can be implemented through linked learning of the first DNN and the second DNN.

[0076] FIG. 3 illustrates that the second DNN (300) includes three convolution layers (310, 330, 350) and two activation layers (320, 340), but this is merely an example, and depending on the embodiment, the number of convolution layers and activation layers may vary. Additionally, depending on the embodiment, the second DNN (300) may be implemented through a recurrent neural network (RNN). In this case, it means changing the CNN structure of the second DNN (300) according to the example of the present disclosure to an RNN structure.

[0077] In one embodiment, the AI ​​upscaler (236) may include at least one Arithmetic Logic Unit (ALU) for the aforementioned convolution operation and the operation of the activation layer. The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier that performs a multiplication operation between the sample values ​​of the second image (135) or the feature map output from the previous layer and the sample values ​​of the filter kernel, and an adder that adds the result values ​​of the multiplication. Additionally, for the operation of the activation layer, the ALU may include a multiplier that multiplies the input sample value by a weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and a comparator that compares the result of the multiplication with a predetermined value to determine whether to pass the input sample value to the next layer.

[0078] In the following, the AI ​​setting unit (238) determines an upscale target, and the AI ​​upscale unit (236) explains how to AI upscale the second image (135) according to the upscale target.

[0079] In one embodiment, the AI ​​setting unit (238) can store a plurality of DNN setting information that can be set in the second DNN.

[0080] Here, the DNN configuration information may include information on at least one of the number of convolution layers included in the second DNN, the number of filter kernels per convolution layer, and the parameters of each filter kernel. Multiple DNN configuration information may correspond to various upscale targets, and the second DNN may operate based on the DNN configuration information corresponding to a specific upscale target. Depending on the DNN configuration information, the second DNN may have different structures. For example, depending on one DNN configuration information, the second DNN may include three convolution layers, and depending on another DNN configuration information, the second DNN may include four convolution layers.

[0081] In one embodiment, the DNN configuration information may include only the parameters of the filter kernel used in the second DNN. In this case, the structure of the second DNN is not changed, but only the parameters of the internal filter kernel may change according to the DNN configuration information.

[0082] The AI ​​setting unit (238) can obtain DNN setting information for AI upscaling of the second image (135) among the plurality of DNN setting information. Each of the plurality of DNN setting information used here is information for obtaining a third image (145) of a predetermined resolution and / or predetermined quality, and is trained in conjunction with the first DNN.

[0083] For example, one of the multiple DNN setting information may include information for obtaining a third image (145) with a resolution twice as large as the resolution of the second image (135), for example, a third image (145) of 4K (4096*2160) which is twice as large as the second image (135) of 2K (2048*1080), and another DNN setting information may include information for obtaining a third image (145) with a resolution four times larger than the resolution of the second image (135), for example, a third image (145) of 8K (8192*4320) which is four times larger than the second image (135) of 2K (2048*1080).

[0084] Each of the multiple DNN setting information is created in conjunction with the DNN setting information of the first DNN of the AI ​​encoding device (700), and the AI ​​setting unit (238) obtains one of the multiple DNN setting information according to an enlargement ratio corresponding to the reduction ratio of the DNN setting information of the first DNN. To do this, the AI ​​setting unit (238) must verify the information of the first DNN. In order for the AI ​​setting unit (238) to verify the information of the first DNN, the AI ​​decoding device (200) according to one embodiment receives AI data including the information of the first DNN from the AI ​​encoding device (700).

[0085] In other words, the AI ​​setting unit (238) can use the information received from the AI ​​encoding device (700) to identify the information targeted by the DNN setting information of the first DNN used to acquire the first image (115), and acquire the DNN setting information of the second DNN trained in conjunction with it.

[0086] When DNN setting information for AI upscaling of the second image (135) is obtained among multiple DNN setting information, the DNN setting information is transmitted to the AI ​​upscaling unit (236), and input data can be processed based on the second DNN that operates according to the DNN setting information.

[0087] For example, when the AI ​​upscaler (236) obtains any one DNN configuration information, for each of the first convolution layer (310), the second convolution layer (330), and the third convolution layer (350) of the second DNN (300) shown in FIG. 3, the number of filter kernels included in each layer and the parameters of the filter kernels are set to the values ​​included in the obtained DNN configuration information.

[0088] Specifically, the AI ​​upscaler (236) can replace the parameters of the 3 x 3 filter kernel used in any one of the convolution layers of the second DNN shown in FIG. 4 with {1, 1, 1, 1, 1, 1, 1, 1}, and if there is a change in the DNN configuration information, the parameters of the filter kernel can be replaced with the parameters included in the changed DNN configuration information, which are {2, 2, 2, 2, 2, 2, 2, 2}.

[0089] The AI ​​setting unit (238) can obtain DNN setting information for upscaling the second image (135) among a plurality of DNN setting information based on information included in the AI ​​data, and the AI ​​data used to obtain the DNN setting information is described in detail.

[0090] In one embodiment, the AI ​​setting unit (238) can obtain DNN setting information for upscaling a second image (135) among a plurality of DNN setting information based on difference information included in the AI ​​data. For example, if it is confirmed based on the difference information that the resolution of the original image (105) (e.g., 4K (4096*2160)) is twice as large as the resolution of the first image (115) (e.g., 2K (2048*1080)), the AI ​​setting unit (238) can obtain DNN setting information that can increase the resolution of the second image (135) by two times.

[0091] In another embodiment, the AI ​​setting unit (238) may obtain DNN setting information for AI upscaling a second image (135) among a plurality of DNN setting information based on information related to a first image (115) included in the AI ​​data. The AI ​​setting unit (238) may predetermine a mapping relationship between the image-related information and the DNN setting information and obtain DNN setting information mapped to the information related to the first image (115).

[0092] Figure 5 is an exemplary diagram showing the mapping relationship between various image-related information and various DNN setting information.

[0093] Through the embodiment according to FIG. 5, it can be seen that the AI ​​encoding / AI decoding process of one embodiment of the present disclosure does not consider only changes in resolution. As shown in FIG. 5, the selection of DNN configuration information can be made by considering resolutions such as SD, HD, and Full HD, bitrates such as 10 Mbps, 15 Mbps, and 20 Mbps, and codec information such as AV1, H.264, and HEVC, either individually or all together. For such consideration, training that takes each of these elements into account must be performed in conjunction with the encoding and decoding processes during the AI ​​training process (see FIG. 11).

[0094] Accordingly, depending on the training content, as shown in FIG. 5, if multiple DNN setting information is provided based on image-related information including codec type, image resolution, etc., then DNN setting information for AI upscaling of the second image (135) can be obtained based on the first image (115) related information received during the AI ​​decoding process.

[0095] That is, the AI ​​setting unit (238) matches the image-related information shown on the left side of the table shown in FIG. 5 with the DNN setting information on the right side of the table, thereby enabling the use of DNN setting information according to the image-related information.

[0096] As illustrated in FIG. 5, if it is confirmed from the information related to the first image (115) that the resolution of the first image (115) is SD, the bit rate of the image data obtained as a result of the first encoding of the first image (115) is 10 Mbps, and the first image (115) is first encoded with an AV1 codec, the AI ​​setting unit (238) can obtain A DNN setting information among a plurality of DNN setting information.

[0097] Additionally, if it is confirmed from the information related to the first image (115) that the resolution of the first image (115) is HD, the bit rate of the image data obtained as a result of the first encoding is 15 Mbps, and the first image (115) was first encoded with an H.264 codec, the AI ​​setting unit (238) can obtain B DNN setting information among the plurality of DNN setting information.

[0098] Additionally, if it is confirmed from the information related to the first image (115) that the resolution of the first image (115) is Full HD, the bitrate of the image data obtained as a result of the first encoding of the first image (115) is 20 Mbps, and the first image (115) is first encoded with an HEVC codec, the AI ​​setting unit (238) obtains C DNN setting information among the plurality of DNN setting information, and if it is confirmed that the resolution of the first image (115) is Full HD, the bitrate of the image data obtained as a result of the first encoding of the first image (115) is 15 Mbps, and the first image (115) is first encoded with an HEVC codec, the AI ​​setting unit (238) can obtain D DNN setting information among the plurality of DNN setting information. Depending on whether the bit rate of the image data obtained as a result of the first encoding of the first image (115) is 20 Mbps or 15 Mbps, either C DNN setting information or D DNN setting information is selected. When the first image (115) of the same resolution is first encoded with the same codec, the fact that the bit rates of the image data are different means that the quality of the restored image is different. Accordingly, the first DNN and the second DNN can be trained in conjunction based on a predetermined quality, and accordingly, the AI ​​setting unit (238) can obtain DNN setting information according to the bit rate of the image data representing the quality of the second image (135).

[0099] In another embodiment, the AI ​​setting unit (238) may obtain DNN setting information for AI upscaling the second image (135) among a plurality of DNN setting information by considering all information provided from the first decoding unit (234) (prediction mode information, motion information, quantization parameter information, etc.) and information related to the first image (115) included in the AI ​​data. For example, the AI ​​setting unit (238) may receive quantization parameter information used in the first encoding process of the first image (115) from the first decoding unit (234), check the bitrate of the image data obtained as a result of encoding the first image (115) from the AI ​​data, and obtain DNN setting information corresponding to the quantization parameter and bitrate. Even with the same bit rate, there may be differences in the quality of the restored image depending on the complexity of the image, and the bit rate is a value that represents the entire first image (115) being encoded, so the quality of each frame within the first image (115) may differ. Therefore, by considering together the prediction mode information, motion information, and / or quantization parameters that can be obtained for each frame from the first decoding unit (234), it is possible to obtain DNN setting information that is more suitable for the second image (135) compared to using only AI data.

[0100] Additionally, according to an embodiment, the AI ​​data may include an identifier of mutually agreed-upon DNN setting information. The identifier of the DNN setting information is information for distinguishing pairs of linked DNN setting information between the first DNN and the second DNN so that the second image (135) can be AI upscaled as an upscale target corresponding to the downscale target of the first DNN. The AI ​​setting unit (238) obtains the identifier of the DNN setting information included in the AI ​​data, obtains DNN setting information corresponding to the identifier of the DNN setting information, and the AI ​​upscale unit (236) can AI upscale the second image (135) using the corresponding DNN setting information. For example, an identifier pointing to each of a plurality of DNN setting information that can be set in the first DNN and an identifier pointing to each of a plurality of DNN setting information that can be set in the second DNN may be pre-specified. In this case, the same identifier may be assigned to pairs of DNN setting information that can be set for each of the first DNN and the second DNN. The AI ​​data may include an identifier of DNN setting information set for the first DNN for AI downscaling of the original image (105). The AI ​​setting unit (238) that receives the AI ​​data obtains the DNN setting information pointed to by the identifier included in the AI ​​data among a plurality of DNN setting information, and the AI ​​upscaling unit (236) can AI upscale the second image (135) using the corresponding DNN setting information.

[0101] Additionally, depending on the embodiment, the AI ​​data may include DNN setting information. The AI ​​setting unit (238) obtains the DNN setting information included in the AI ​​data, and the AI ​​upscale unit (236) may use the DNN setting information to AI upscale the second image (135).

[0102] According to an implementation example, if the information constituting the DNN setting information (e.g., the number of convolution layers, the number of filter kernels per convolution layer, the parameters of each filter kernel, etc.) is stored in the form of a lookup table, the AI ​​setting unit (238) obtains the DNN setting information by combining a selected portion of the lookup table values ​​based on the information included in the AI ​​data, and the AI ​​upscale unit (236) may AI upscale the second image (135) using the DNN setting information.

[0103] According to an embodiment, when the structure of the DNN corresponding to the upscale target is determined, the AI ​​setting unit (238) may obtain DNN setting information corresponding to the determined DNN structure, such as parameters of a filter kernel.

[0104] As described above, the AI ​​setting unit (238) obtains DNN setting information of the second DNN through AI data containing information related to the first DNN, and the AI ​​upscaling unit (236) AI upscales the second image (135) through the second DNN set with the DNN setting information, which can reduce memory usage and computational load compared to upscaling by directly analyzing the features of the second image (135).

[0105] In one embodiment, when the second image (135) is composed of a plurality of frames, the AI ​​setting unit (238) may independently obtain DNN setting information for a predetermined number of frames, or obtain common DNN setting information for all frames.

[0106] FIG. 6 is a drawing illustrating a second image (135) composed of multiple frames.

[0107] As shown in FIG. 6, the second image (135) may consist of frames corresponding to t0 to tn.

[0108] In one example, the AI ​​setting unit (238) obtains DNN setting information of the second DNN through AI data, and the AI ​​upscaling unit (236) can AI upscale frames corresponding to t0 to tn based on the DNN setting information. That is, frames corresponding to t0 to tn can be AI upscaled based on common DNN setting information.

[0109] In another example, the AI ​​setting unit (238) may obtain 'A' DNN setting information from AI data for some of the frames corresponding to t0 to tn, for example, frames corresponding to t0 to ta, and obtain 'B' DNN setting information from AI data for frames corresponding to ta+1 to tb. Additionally, the AI ​​setting unit (238) may obtain 'C' DNN setting information from AI data for frames corresponding to tb+1 to tn. In other words, the AI ​​setting unit (238) independently obtains DNN setting information for each group containing a predetermined number of frames among a plurality of frames, and the AI ​​upscale unit (236) may AI upscale the frames included in each group using the independently obtained DNN setting information.

[0110] In another example, the AI ​​setting unit (238) may independently acquire DNN setting information for each frame constituting the second image (135). For example, if the second image (135) consists of three frames, the AI ​​setting unit (238) may acquire DNN setting information in relation to the first frame, acquire DNN setting information in relation to the second frame, and acquire DNN setting information in relation to the third frame. That is, DNN setting information may be independently acquired for each of the first frame, the second frame, and the third frame. Depending on the method in which DNN setting information is acquired based on the information provided from the aforementioned first decoding unit (234) (prediction mode information, motion information, quantization parameter information, etc.) and the information related to the first image (115) included in the AI ​​data, DNN setting information may be independently acquired for each frame constituting the second image (135). This is because the mode information, quantization parameter information, etc., can be independently determined for each frame constituting the second image (135).

[0111] In another example, the AI ​​data may include information indicating up to which frame the DNN setting information obtained based on the AI ​​data is valid. For example, if the AI ​​data contains information that the DNN setting information is valid up to frame ta, the AI ​​setting unit (238) obtains the DNN setting information based on the AI ​​data, and the AI ​​upscale unit (236) AI upscales frames t0 to ta using the corresponding DNN setting information. And, if other AI data contains information that the DNN setting information is valid up to frame tn, the AI ​​setting unit (238) obtains the DNN setting information based on the other AI data, and the AI ​​upscale unit (236) AI upscales frames ta+1 to tn using the obtained DNN setting information.

[0112] Hereinafter, with reference to FIG. 7, an AI encoding device (700) for AI encoding of an original image (105) will be described.

[0113] FIG. 7 is a block diagram showing the configuration of an AI encoding device (700) according to one embodiment.

[0114] Referring to FIG. 7, the AI ​​encoding device (700) may include an AI encoding unit (710) and a transmission unit (730). The AI ​​encoding unit (710) may include an AI downscale unit (712), a first encoding unit (714), a data processing unit (716), and an AI setting unit (718).

[0115] FIG. 7 illustrates the AI ​​encoding unit (710) and the transmission unit (730) as separate devices, but the AI ​​encoding unit (710) and the transmission unit (730) can be implemented through a single processor. In this case, it may be implemented with a dedicated processor, or it may be implemented through a combination of a general-purpose processor such as an AP, CPU, or GPU and S / W. In addition, in the case of a dedicated processor, it may include memory for implementing the embodiments of the present disclosure or a memory processing unit for using external memory.

[0116] Additionally, the AI ​​encoding unit (710) and the transmission unit (730) may be composed of multiple processors. In this case, they may be implemented by a combination of dedicated processors, or by a combination of multiple general-purpose processors such as APs, CPUs, and GPUs, and S / W.

[0117] In one embodiment, the first encoding unit (714) is configured as a first processor, the AI ​​downscaling unit (712), the data processing unit (716), and the AI ​​setting unit (718) are implemented as a second processor different from the first processor, and the transmission unit (730) may be implemented as a third processor different from the first processor and the second processor. The AI ​​encoding unit (710) performs AI downscaling of the original image (105) and first encoding of the first image (115), and transmits the AI ​​encoded data to the transmission unit (730). The transmission unit (730) transmits the AI ​​encoded data to the AI ​​decoding device (200).

[0118] The image data includes data obtained as a result of the first encoding of the first image (115). The image data may include data obtained based on pixel values ​​within the first image (115), for example, residual data which is the difference between the first image (115) and the prediction data of the first image (115). Additionally, the image data includes information used in the first encoding process of the first image (115). For example, the image data may include prediction mode information, motion information, and quantization parameter-related information used in the first encoding of the first image (115).

[0119] The AI ​​data includes information that enables the AI ​​upscaler (236) to AI upscale the second image (135) to an upscale target corresponding to the downscale target of the first DNN. In one example, the AI ​​data may include difference information between the original image (105) and the first image (115). In one example, the AI ​​data may include information related to the first image (115). The information related to the first image (115) may include information about at least one of the resolution of the first image (115), the bitrate of the image data obtained as a result of the first encoding of the first image (115), and the codec type used during the first encoding of the first image (115).

[0120] In one embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information so that the second image (135) can be AI upscaled to an upscale target corresponding to the downscale target of the first DNN.

[0121] In addition, in one embodiment, the AI ​​data may include DNN setting information that can be set in the second DNN.

[0122] The AI ​​downscaler (712) can obtain a first image (115) that has been AI downscaled from the original image (105) through the first DNN. The AI ​​downscaler (712) can AI downscale the original image (105) using DNN setting information provided by the AI ​​setting unit (718). The AI ​​setting unit (718) can determine a downscale target for the original image (105) based on predetermined criteria.

[0123] In order to acquire a first image (115) that corresponds to a downscale target, the AI ​​setting unit (718) can store multiple DNN setting information that can be set in the first DNN. The AI ​​setting unit (718) acquires DNN setting information corresponding to the downscale target among the multiple DNN setting information and provides the acquired DNN setting information to the AI ​​downscale unit (712).

[0124] Each of the above multiple DNN setting information may be trained to obtain a first image (115) of a predetermined resolution and / or predetermined quality. For example, one of the multiple DNN setting information may include information for obtaining a first image (115) with a resolution that is 1 / 2 times smaller than the resolution of the original image (105), for example, a first image (115) of 2K (2048*1080) that is 1 / 2 times smaller than the original image (105) of 4K (4096*2160), and another DNN setting information may include information for obtaining a first image (115) with a resolution that is 1 / 4 times smaller than the resolution of the original image (105), for example, a first image (115) of 2K (2048*1080) that is 1 / 4 times smaller than the original image (105) of 8K (8192*4320).

[0125] According to an implementation example, if the information constituting the DNN setting information (e.g., the number of convolution layers, the number of filter kernels per convolution layer, the parameters of each filter kernel, etc.) is stored in the form of a lookup table, the AI ​​setting unit (718) may provide the DNN setting information obtained by combining some of the lookup table values ​​according to the downscale target to the AI ​​downscale unit (712).

[0126] According to an embodiment, the AI ​​setting unit (718) may determine the structure of a DNN corresponding to a downscale target and obtain DNN setting information corresponding to the determined structure of the DNN, such as parameters of a filter kernel.

[0127] Multiple DNN configuration information for AI downscaling of the original image (105) can have optimized values ​​by training the first DNN and the second DNN together. Here, each DNN configuration information includes at least one of the number of convolution layers included in the first DNN, the number of filter kernels per convolution layer, and the parameters of each filter kernel.

[0128] The AI ​​downscaler (712) sets the first DNN with the DNN setting information determined for AI downscale of the original image (105), and can obtain the first image (115) of a predetermined resolution and / or predetermined quality through the first DNN. When the DNN setting information for AI downscale of the original image (105) is obtained among the plurality of DNN setting information, each layer within the first DNN can process the input data based on the information included in the DNN setting information.

[0129] Below, the method by which the AI ​​setting unit (718) determines a downscale target is described. The downscale target may indicate, for example, how much the resolution of the first image (115) should be reduced from the original image (105).

[0130] The AI ​​setting unit (718) acquires one or more input information. In one embodiment, the input information may include at least one of the target resolution of the first image (115), the target bitrate of the image data, the bitrate type of the image data (e.g., variable bitrate type, constant bitrate type or average bitrate type, etc.), the color format to which AI downscale is applied (luminance component, chrominance component, red component, green component or blue component, etc.), the codec type for the first encoding of the first image (115), compression history information, the resolution of the original image (105), and the type of the original image (105).

[0131] One or more input information may include information that is pre-stored in the AI ​​encoding device (700) or received from the user.

[0132] The AI ​​setting unit (718) controls the operation of the AI ​​downscale unit (712) based on input information. In one embodiment, the AI ​​setting unit (718) determines a downscale target according to the input information and can provide DNN setting information corresponding to the determined downscale target to the AI ​​downscale unit (712).

[0133] In one embodiment, the AI ​​setting unit (718) may transmit at least a portion of the input information to the first encoding unit (714) so ​​that the first encoding unit (714) first encodes the first image (115) with a bit rate of a specific value, a bit rate of a specific type, and a specific codec.

[0134] In one embodiment, the AI ​​setting unit (718) can determine a downscale target based on at least one of a compression rate (e.g., a difference in resolution between the original image (105) and the first image (115), a target bitrate), a compression quality (e.g., a bitrate type), compression history information, and a type of the original image (105).

[0135] In one example, the AI ​​setting unit (718) can determine a downscale target based on a compression rate or compression quality that is pre-set or input by the user.

[0136] As another example, the AI ​​setting unit (718) may determine a downscale target using compression history information stored in the AI ​​encoding device (700). For example, according to the compression history information available to the AI ​​encoding device (700), the encoding quality or compression ratio preferred by the user may be determined, and the downscale target may be determined according to the encoding quality, etc. determined based on the compression history information. For example, the resolution, image quality, etc. of the first video (115) may be determined according to the encoding quality that has been used most frequently according to the compression history information.

[0137] As another example, the AI ​​setting unit (718) may determine a downscale target based on encoding quality that has been used more than a predetermined threshold value according to compression history information (e.g., the average quality of encoding quality that has been used more than a predetermined threshold value).

[0138] As another example, the AI ​​setting unit (718) may determine a downscale target based on the resolution and type (e.g., file format) of the original image (105).

[0139] In one embodiment, when the original image (105) is composed of a plurality of frames, the AI ​​setting unit (718) may independently acquire DNN setting information for a predetermined number of frames and provide the independently acquired DNN setting information to the AI ​​downscale unit (712).

[0140] In one example, the AI ​​setting unit (718) may divide the frames constituting the original image (105) into a predetermined number of groups and independently obtain DNN setting information for each group. For each group, identical or different DNN setting information may be obtained. The number of frames included in the groups may be the same or different for each group.

[0141] In another example, the AI ​​setting unit (718) can independently determine DNN setting information for each frame constituting the original image (105). For each frame, identical or different DNN setting information can be obtained.

[0142] Below, an exemplary structure of the first DNN (800) that serves as the basis for AI downscaling is described.

[0143] FIG. 8 is an exemplary diagram showing a first DNN (800) for AI downscaling of an original image (105).

[0144] As illustrated in FIG. 8, the original image (105) is input to the first convolution layer (810). The first convolution layer (810) performs convolution processing on the original image (105) using 32 filter kernels of size 5 x 5. The 32 feature maps generated as a result of the convolution processing are input to the first activation layer (820). The first activation layer (820) can impart non-linear characteristics to the 32 feature maps.

[0145] The first activation layer (820) determines whether to transmit sample values ​​of feature maps output from the first convolution layer (810) to the second convolution layer (830). For example, some sample values ​​of the feature maps are activated by the first activation layer (820) and transmitted to the second convolution layer (830), while some sample values ​​are deactivated by the first activation layer (820) and not transmitted to the second convolution layer (830). The information represented by the feature maps output from the first convolution layer (810) is highlighted by the first activation layer (820).

[0146] The output (825) of the first activation layer (820) is input to the second convolution layer (830). The second convolution layer (830) performs convolution processing on the input data using 32 filter kernels of size 5 x 5. The 32 feature maps output as a result of the convolution processing are input to the second activation layer (840), and the second activation layer (840) can impart non-linear characteristics to the 32 feature maps.

[0147] The output (845) of the second activation layer (840) is input to the third convolution layer (850). The third convolution layer (850) performs convolution processing on the input data using one filter kernel of size 5 x 5. As a result of the convolution processing, one image can be output from the third convolution layer (850). The third convolution layer (850) obtains one output using one filter kernel as a layer for outputting the final image. According to an example of the present disclosure, the third convolution layer (850) can output the first image (115) through the result of the convolution operation.

[0148] There may be multiple DNN configuration information indicating the number of filter kernels, parameters of filter kernels, etc. of the first convolution layer (810), the second convolution layer (830), and the third convolution layer (850) of the first DNN (800), and the multiple DNN configuration information must be linked with the multiple DNN configuration information of the second DNN. The linkage between the multiple DNN configuration information of the first DNN and the multiple DNN configuration information of the second DNN can be implemented through linked learning of the first DNN and the second DNN.

[0149] FIG. 8 illustrates that the first DNN (800) includes three convolution layers (810, 830, 750) and two activation layers (820, 840), but this is merely an example, and depending on the embodiment, the number of convolution layers and activation layers may vary. Additionally, depending on the embodiment, the first DNN (800) may be implemented through a recurrent neural network (RNN). In this case, it means changing the CNN structure of the first DNN (800) according to the example of the present disclosure to an RNN structure.

[0150] In one embodiment, the AI ​​downscaler (712) may include at least one ALU for convolution operations and operations of an activation layer. The ALU may be implemented as a processor. For convolution operations, the ALU may include a multiplier that performs a multiplication operation between sample values ​​of a feature map output from the original image (105) or a previous layer and sample values ​​of a filter kernel, and an adder that adds the results of the multiplication. Additionally, for operations of an activation layer, the ALU may include a multiplier that multiplies an input sample value by a weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and a comparator that compares the result of the multiplication with a predetermined value to determine whether to pass the input sample value to the next layer.

[0151] Referring again to FIG. 7, the AI ​​setting unit (718) transmits AI data to the data processing unit (716). The AI ​​data includes information that enables the AI ​​upscale unit (236) to AI upscale the second image (135) to an upscale target corresponding to the downscale target of the first DNN. The first encoding unit (714), which receives the first image (115) from the AI ​​downscale unit (712), can reduce the amount of information contained in the first image (115) by first encoding the first image (115) according to a frequency conversion-based image compression method. As a result of the first encoding through a predetermined codec (e.g., MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1), image data is obtained. The image data is obtained according to the rules of the predetermined codec, that is, the syntax. For example, the image data may include residual data, which is the difference between the first image (115) and the prediction data of the first image (115), prediction mode information used to first encode the first image (115), motion information, and quantization parameter-related information used to first encode the first image (115). The image data obtained as a result of the first encoding of the first encoding unit (714) is provided to the data processing unit (716).

[0152] The data processing unit (716) generates AI encoded data including image data received from the first encoding unit (714) and AI data received from the AI ​​setting unit (718).

[0153] In one embodiment, the data processing unit (716) can generate AI encoded data that includes video data and AI data in separate states. For example, the AI ​​data may be included in a VSIF (Vendor Specific InfoFrame) within an HDMI stream.

[0154] In another embodiment, the data processing unit (716) may include AI data within the image data obtained as a result of the first encoding by the first encoding unit (714) and generate AI encoded data including the image data. For example, the data processing unit (716) may combine a bitstream corresponding to the image data and a bitstream corresponding to the AI ​​data to generate image data in the form of a single bitstream. To this end, the data processing unit (716) may represent the AI ​​data as bits having a value of 0 or 1, i.e., a bitstream. In one embodiment, the data processing unit (716) may include the bitstream corresponding to the AI ​​data in the SEI (Supplemental enhancement information), which is an additional information area of ​​the bitstream obtained as a result of the first encoding.

[0155] AI encoded data is transmitted to a transmission unit (730). The transmission unit (730) transmits the AI ​​encoded data obtained as an AI encoding result through a network. In one embodiment, the AI ​​encoded data may be stored in a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, and a magneto-optical medium such as a floptical disk.

[0156] FIG. 9 is a diagram showing the structure of AI encoded data (900) according to one embodiment.

[0157] As described above, AI data (912) and video data (932) may be included separately within the AI ​​encoded data (900). Here, the AI ​​encoded data (900) may be a container format such as MP4, AVI, MKV, FLV, etc. The AI ​​encoded data (900) may consist of a metadata box (910) and a media data box (930).

[0158] The metadata box (910) contains information regarding video data (932) included in the media data box (930). For example, the metadata box (910) may contain information regarding the type of the first video (115), the type of codec used to encode the first video (115), and the playback time of the first video (115). Additionally, the metadata box (910) may contain AI data (912). The AI ​​data (912) may be encoded according to an encoding method provided by a predetermined container format and stored in the metadata box (910).

[0159] The media data box (930) may include image data (932) generated according to the syntax of a predetermined image compression method.

[0160] FIG. 10 is a diagram showing the structure of AI encoded data (1000) according to another embodiment.

[0161] Referring to FIG. 10, AI data (1034) may be included in video data (1032). AI encoded data (1000) may include a metadata box (1010) and a media data box (1030), but when AI data (1034) is included in video data (1032), the AI ​​data (1034) may not be included in the metadata box (1010).

[0162] The media data box (1030) contains video data (1032) including AI data (1034). For example, the AI ​​data (1034) may be included in an additional information area of ​​the video data (1032).

[0163] Below, with reference to FIG. 11, a method for training the first DNN (800) and the second DNN (300) in combination will be described.

[0164] FIG. 11 is a diagram illustrating a method for training the first DNN (800) and the second DNN (300).

[0165] In one embodiment, the original video (105) that has been AI encoded through an AI encoding process is restored to a third video (145) through an AI decoding process. In order to maintain similarity between the third video (145) obtained as a result of AI decoding and the original video (105), a correlation between the AI ​​encoding process and the AI ​​decoding process is required. That is, information lost during the AI ​​encoding process must be recoverable during the AI ​​decoding process, and for this purpose, linked training of the first DNN (800) and the second DNN (300) is required.

[0166] For accurate AI decoding, it is ultimately necessary to reduce quality loss information (1130) corresponding to the comparison result between the third training image (1104) and the original training image (1101) shown in FIG. 11. Accordingly, quality loss information (1130) is used for training both the first DNN (800) and the second DNN (300).

[0167] First, the training process illustrated in Fig. 11 will be explained.

[0168] In FIG. 11, the original training image (1101) is the image to be AI downscaled, and the first training image (1102) is the image AI downscaled from the original training image (1101). Additionally, the third training image (1104) is the image AI upscaled from the first training image (1102).

[0169] The original training image (1101) includes a still image or a video composed of multiple frames. In one embodiment, the original training image (1101) may include a luminance image extracted from a still image or a video composed of multiple frames. In addition, in one embodiment, the original training image (1101) may include a patch image extracted from a still image or a video composed of multiple frames. When the original training image (1101) is composed of multiple frames, the first training image (1102), the second training image, and the third training image (1104) are also composed of multiple frames. When multiple frames of the original training image (1101) are sequentially input to the first DNN (800), multiple frames of the first training image (1102), the second training image, and the third training image (1104) can be sequentially acquired through the first DNN (800) and the second DNN (300).

[0170] For the combined training of the first DNN (800) and the second DNN (300), the original training image (1101) is input into the first DNN (800). The original training image (1101) input into the first DNN (800) is AI downscaled and output as the first training image (1102), and the first training image (1102) is input into the second DNN (300). The third training image (1104) is output as a result of AI upscaling the first training image (1102).

[0171] Referring to FIG. 11, a first training image (1102) is input to a second DNN (300). Depending on the embodiment, a second training image obtained through the first encoding and first decoding processes of the first training image (1102) may also be input to the second DNN (300). To input the second training image to the second DNN, any one of the codecs MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 may be used. Specifically, any one of the codecs MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 may be used for the first encoding of the first training image (1102) and the first decoding of the image data corresponding to the first training image (1102).

[0172] Referring to FIG. 11, a legacy downscaled reduced training image (1103) is obtained from the original training image (1101), separate from the output of the first training image (1102) through the first DNN (800). Here, the legacy downscale may include at least one of a bilinear scale, a bicubic scale, a lanczos scale, and a stair step scale.

[0173] In order to prevent the structural features of the first image (115) from deviating significantly from the structural features of the original image (105), a reduced training image (1103) that preserves the structural features of the original training image (1101) is obtained.

[0174] Before training proceeds, the first DNN (800) and the second DNN (300) can be set with predetermined DNN configuration information. As training proceeds, structural loss information (1110), complexity loss information (1120), and quality loss information (1130) can be determined.

[0175] Structural loss information (1110) can be determined based on the comparison result between the reduced training image (1103) and the first training image (1102). In one example, structural loss information (1110) may correspond to the difference between the structural information of the reduced training image (1103) and the structural information of the first training image (1102). Structural information may include various features extractable from the image, such as the image's brightness, contrast, and histogram. Structural loss information (1110) indicates the extent to which the structural information of the original training image (1101) is preserved in the first training image (1102). The smaller the structural loss information (1110), the more similar the structural information of the first training image (1102) is to the structural information of the original training image (1101).

[0176] Complexity loss information (1120) can be determined based on the spatial complexity of the first training image (1102). In one example, the total variance value of the first training image (1102) can be used as the spatial complexity. Complexity loss information (1120) is related to the bitrate of the image data obtained by first encoding the first training image (1102). It is defined that the smaller the complexity loss information (1120), the smaller the bitrate of the image data.

[0177] Quality loss information (1130) can be determined based on the comparison results between the original training video (1101) and the third training video (1104). Quality loss information (1130) may include at least one of an L1-norm value, an L2-norm value, a SSIM (Structural Similarity) value, a PSNR-HVS (Peak Signal-To-Noise Ratio-Human Vision System) value, an MS-SSIM (Multiscale SSIM) value, a VIF (Variance Inflation Factor) value, and a VMAF (Video Multimethod Assessment Fusion) value regarding the difference between the original training video (1101) and the third training video (1104). Quality loss information (1130) indicates the degree to which the third training video (1104) is similar to the original training video (1101). The smaller the quality loss information (1130), the more similar the third training image (1104) is to the original training image (1101).

[0178] Referring to FIG. 11, structural loss information (1110), complexity loss information (1120), and quality loss information (1130) are used for training the first DNN (800), and the quality loss information (1130) is used for training the second DNN (300). That is, the quality loss information (1130) is used for training both the first DNN (800) and the second DNN (300).

[0179] The first DNN (800) can update parameters so that the final loss information determined based on structural loss information (1110), complexity loss information (1120), and quality loss information (1130) is reduced or minimized. Additionally, the second DNN (300) can update parameters so that the quality loss information (1130) is reduced or minimized.

[0180] The final loss information for training the first DNN (800) and the second DNN (300) can be determined as shown in Equation 1 below.

[0181] [Mathematical Formula 1]

[0182]

[0183] In the above mathematical formula 1, LossDS represents the final loss information to be reduced or minimized for training the first DNN (800), and LossUS represents the final loss information to be reduced or minimized for training the second DNN (300). Additionally, a, b, c, and d may correspond to predetermined weights.

[0184] That is, the first DNN (800) updates the parameters in the direction in which the LossDS of Equation 1 decreases, and the second DNN (300) updates the parameters in the direction in which the LossUS decreases. When the parameters of the first DNN (800) are updated according to the LossDS derived during the training process, the first training image (1102) obtained based on the updated parameters becomes different from the first training image (1102) in the previous training process, and accordingly, the third training image (1104) also becomes different from the third training image (1104) in the previous training process. When the third training image (1104) becomes different from the third training image (1104) in the previous training process, the quality loss information (1130) is also newly determined, and accordingly, the second DNN (300) updates the parameters. When the quality loss information (1130) is newly determined, the LossDS is also newly determined, so the first DNN (800) updates the parameters according to the newly determined LossDS. That is, the parameter update of the first DNN (800) causes the parameter update of the second DNN (300), and the parameter update of the second DNN (300) causes the parameter update of the first DNN (800). In other words, since the first DNN (800) and the second DNN (300) are trained in conjunction through the sharing of the quality loss information (1130), the parameters of the first DNN (800) and the parameters of the second DNN (300) can be optimized with a correlation to each other.

[0185] Referring to Equation 1, it can be seen that LossUS is determined according to quality loss information (1130), but this is just one example, and LossUS may also be determined based on at least one of structural loss information (1110) and complexity loss information (1120) and quality loss information (1130).

[0186] Previously, it was explained that the AI ​​setting unit (238) of the AI ​​decoding device (200) and the AI ​​setting unit (718) of the AI ​​encoding device (700) store multiple DNN setting information. Now, a method for training each of the multiple DNN setting information stored in the AI ​​setting unit (238) and the AI ​​setting unit (718) will be described.

[0187] As explained in relation to Equation 1, in the case of the first DNN (800), parameters are updated by considering the degree of similarity between the structural information of the first training image (1102) and the structural information of the original training image (1101) (structural loss information (1110)), the bitrate of the image data obtained from the first encoding result of the first training image (1102) (complexity loss information (1120)), and the difference between the third training image (1104) and the original training image (1101) (quality loss information (1130)).

[0188] To explain in detail, the parameters of the first DNN (800) can be updated so that a first training image (1102) can be obtained which is similar to the structural information of the original training image (1101) and has a small bit rate of image data obtained when the first encoding is performed, and at the same time, a second DNN (300) that AI upscales the first training image (1102) can obtain a third training image (1104) similar to the original training image (1101).

[0189] By adjusting the weights of a, b, and c in Equation 1, the direction in which the parameters of the first DNN (800) are optimized becomes different. For example, if the weight of b is set high, the parameters of the first DNN (800) may be updated with greater importance placed on lowering the bitrate than on the quality of the third training video (1104). Also, if the weight of c is set high, the parameters of the first DNN (800) may be updated with greater importance placed on increasing the quality of the third training video (1104) than on increasing the bitrate or maintaining the structural information of the original training video (1101).

[0190] In addition, the direction in which the parameters of the first DNN (800) are optimized may differ depending on the type of codec used to first encode the first training image (1102). This is because, depending on the type of codec, the second training image to be input to the second DNN (300) may differ.

[0191] That is, the parameters of the first DNN (800) and the parameters of the second DNN (300) can be updated in conjunction based on weights a, b, and c, and the type of codec for the first encoding of the first training image (1102). Therefore, when weights a, b, and c are each determined to a predetermined value and the type of codec is determined to a predetermined type, and the first DNN (800) and the second DNN (300) are trained, the parameters of the first DNN (800) and the parameters of the second DNN (300) that are optimized in conjunction with each other can be determined.

[0192] Then, after changing the weights a, b, c, and the type of codec, the first DNN (800) and the second DNN (300) are trained, and the parameters of the first DNN (800) and the parameters of the second DNN (300) that are optimized in conjunction with each other can be determined. In other words, by training the first DNN (800) and the second DNN (300) while changing the values ​​of each of the weights a, b, c, and the type of codec, multiple DNN configuration information that is trained in conjunction with each other can be determined in the first DNN (800) and the second DNN (300).

[0193] As previously explained in relation to FIG. 5, multiple DNN configuration information of the first DNN (800) and the second DNN (300) may be mapped to first image-related information. To establish this mapping relationship, a first training image (1102) output from the first DNN (800) may be first encoded with a specific codec according to a specific bit rate, and a second training image obtained by first decoding the bitstream obtained from the first encoding result may be input to the second DNN (300). That is, by setting up an environment so that a first training image (1102) of a specific resolution is first encoded by a specific codec at a specific bit rate, and then training the first DNN (800) and the second DNN (300), a pair of DNN setting information mapped to the resolution of the first training image (1102), the type of codec used for the first encoding of the first training image (1102), and the bit rate of the bitstream obtained as a result of the first encoding of the first training image (1102) can be determined. By varying the resolution of the first training image (1102), the type of codec used for the first encoding of the first training image (1102), and the bit rate of the bitstream obtained according to the first encoding of the first training image (1102), a mapping relationship between the multiple DNN setting information of the first DNN (800) and the second DNN (300) and the first image-related information can be determined.

[0194] FIG. 12 is a diagram illustrating the training process of the first DNN (800) and the second DNN (300) by the training device (1200).

[0195] Training of the first DNN (800) and the second DNN (300) described in relation to FIG. 11 can be performed by a training device (1200). The training device (1200) includes the first DNN (800) and the second DNN (300). The training device (1200) may be, for example, an AI encoding device (700) or a separate server. The DNN configuration information of the second DNN (300) obtained as a result of training is stored in an AI decoding device (200).

[0196] Referring to FIG. 12, the training device (1200) initializes the DNN configuration information of the first DNN (800) and the second DNN (300) (S1240, S1245). By doing so, the first DNN (800) and the second DNN (300) can operate according to the predetermined DNN configuration information. The DNN configuration information may include information on at least one of the number of convolution layers included in the first DNN (800) and the second DNN (300), the number of filter kernels per convolution layer, the size of the filter kernels per convolution layer, and the parameters of each filter kernel.

[0197] The training device (1200) inputs the original training image (1101) into the first DNN (800) (S1250). The original training image (1101) may include at least one frame constituting a still image or a video.

[0198] The first DNN (800) processes the original training image (1101) according to the initial DNN setting information and outputs the first training image (1102) that has been AI downscaled from the original training image (1101) (S1255). FIG. 12 is illustrated as the first training image (1102) output from the first DNN (800) being directly input to the second DNN (300), but the first training image (1102) output from the first DNN (800) may be input to the second DNN (300) by the training device (1200). Additionally, the training device (1200) may first encode and first decode the first training image (1102) using a predetermined codec, and then input the second training image to the second DNN (300).

[0199] The second DNN (300) processes the first training image (1102) or the second training image according to the initial DNN setting information and outputs the third training image (1104) that has been AI upscaled from the first training image (1102) or the second training image (S1260).

[0200] The training device (1200) produces complexity loss information (1120) based on the first training image (1102) (S1265).

[0201] The training device (1200) compares the reduced training image (1103) and the first training image (1102) to produce structural loss information (1110) (S1270).

[0202] The training device (1200) compares the original training image (1101) and the third training image (1104) to produce quality loss information (1130) (S1275).

[0203] The first DNN (800) updates the initial DNN setting information through a back propagation process based on the final loss information (S1280). The training device (1200) can calculate the final loss information for training the first DNN (800) based on the complexity loss information (1120), the structural loss information (1110), and the quality loss information (1130).

[0204] The second DNN (300) updates the initial DNN setting information through a backtranslation process based on quality loss information or final loss information (S1285). The training device (1200) can calculate final loss information for training the second DNN (300) based on quality loss information (1130).

[0205] Subsequently, the training device (1200), the first DNN (800), and the second DNN (300) update the DNN setting information by repeating the process S1250 to S1285 until the final loss information is minimized. At this time, during each iteration, the first DNN (800) and the second DNN operate according to the DNN setting information updated in the previous process.

[0206] Table 1 below shows the effects of AI encoding and AI decoding of the original image (105) according to one embodiment of the present disclosure, and of HEVC encoding and decoding of the original image (105).

[0207] [Table 1]

[0208]

[0209] As can be seen from Table 1, according to one embodiment of the present disclosure, when content consisting of 300 frames of 8K resolution is AI encoded and AI decoded, the subjective image quality is higher than when encoded and decoded with HEVC, yet the bitrate is reduced by more than 50%.

[0210] In the following, a method for transmitting video by adding a deblocking filter to resolve the blocking phenomenon that occurs when the bandwidth is reduced due to changes in the environment of the streaming channel when streaming video in real time using an AI upscale model and an AI downscale model determined according to the resolution and bitrate during video transmission according to the AI-based upscale and downscale described in FIGS. 1 to 12 is described.

[0211] FIG. 13 is a diagram illustrating an AI encoding process and an AI decoding process that add deblocking filtering according to one embodiment.

[0212] As described above in FIG. 1, when the streaming channel environment is normal, the original video (105) is AI downscaled (110) based on an AI downscale model and an AI upscale model suitable for the streaming channel environment to obtain a first video (115), the first video (115) is first encoded (120) and transmitted, AI encoded data including AI data and video data obtained as a result of AI encoding is received, a second video (135) is obtained through a first decoding (130), and the second video (135) is AI upscaled (140) to obtain a third video (145).

[0213] On the other hand, when changes occur in the streaming channel environment—for example, when a large number of users suddenly flock to the channel or the terminal's communication environment deteriorates outdoors—bandwidth decreases. Consequently, quantization parameters are increased to lower the bitrate, which may result in blocking depending on the video resolution and screen characteristics. When blocking, or blocking artifacts, occur, viewers perceive the video quality as poor. To prevent such blocking, video quality can be improved by adding a deblocking filter and performing AI downscaling and AI upscaling.

[0214] Referring to FIG. 13, in the deblocking prediction (1370), information on the network environment of the current streaming channel, the original video (1305), and information on the previously encoded frame are obtained to predict whether a blocking phenomenon will occur in the input original video, and filter setting information, such as a region map for the area where the block occurs, is transmitted when AI encoding and AI decoding are performed. In the AI ​​encoding process, based on filter setting information for such a deblocking filter, a deblocking filter (1310) is applied to the original image (1305) to obtain a deblocking filtered original image (1315), the deblocking filtered image (1315) is AI downscaled (1320) to obtain a first image (1325), and the first image (1325) is first encoded (1330) and transmitted. In the AI ​​decoding process, AI encoded data including AI data and image data obtained as a result of AI encoding is received, a second image (1345) is obtained through first decoding (1340), the second image (1345) is AI upscaled (1350) to obtain an AI upscaled image (1355), and a deblocking filter (1360) according to the filter setting information transmitted in the deblocking prediction (1370) is applied to the AI ​​upscaled The restored image (1365) is obtained by applying it to the image (1355).

[0215] Network environment information is information indicating changes in the network environment transmitted according to real-time protocols (e.g., Real-time Transport Protocol (RTP), Real-time Transport Control Protocol (RTCP)) used in network communication, such as communication between mobile terminals or communication between a server and a client.

[0216] Blocking may occur if bandwidth decreases due to changes in the network environment while streaming is being performed at a predetermined resolution and bitrate. For example, in cases where the resolution is 4K and the bitrate is 20Mbps, 4K and the bitrate is 15Mbps, FHD and the bitrate is 10Mbps, FHD and the bitrate is 7.5Mbps, or HD and the bitrate is 5Mbps, an appropriate AI downscale and AI upscale model is selected for each case. During the AI ​​encoding and AI decoding processes, if there is a change in the network environment (such as a sudden increase in users or poor communication conditions outdoors), deblocking filtering is performed through a deblocking prediction process based on changes in the network environment, characteristics of the original video, and whether blocking artifacts from previous frames are detected. This allows for the prediction and recognition of blocking artifacts in variable and limited network bandwidth, thereby removing the blocking artifacts and improving the video quality. Generally, in video encoding and decoding, there is a deblocking filter that removes blocking artifacts in block units using an in-loop filter. However, since it is an in-loop filter, applying strong deblocking filtering can damage the actual data. Therefore, deblocking filtering can be performed through an additional deblocking filter to improve the image quality.

[0217] Deblocking prediction (1370) obtains whether there is a region where blocking may occur from the original image (1305), obtains whether a blocking phenomenon of the previous frame of the current frame is detected from previous frame information, obtains information on changes in bandwidth, bit rate, etc. from network environment information, and transmits filter setting information including a blocking region map for the region where blocking may occur when AI decoding and AI encoding are performed. Based on the transmitted filter setting information, deblocking filtering is additionally performed when each of the AI ​​decoding and AI encoding processes is performed.

[0218] The deblocking filter (1310) in the AI ​​encoding process acts as a pre-filter and redistributes the image quality between complex areas and non-complex areas when the total complexity is more complex than the allocated bits. For example, if there is a simple area such as a stationary tree in the video and a complex and fast-moving area such as a water jet coming out of a fountain, the complex area is simply blurred to reduce complexity and reduce the bit rate, and the bit rate corresponding to the reduced complexity is distributed to the simple area during encoding. Since blocking is a phenomenon that occurs when image quality distribution is not done properly, the probability of blocking can be reduced by redistributing the image quality. Complex and fast-moving areas are blurred so that blocking artifacts are less noticeable, and the bit rate reduced through blurring of complex and fast-moving areas is allocated to the simple area, thereby preventing the occurrence of blocking in the simple area with a higher bit rate than before.

[0219] The deblocking filter (1360) in the AI ​​decoding process is a post-filter that applies deblocking filtering according to the blocking area map included in the filter setting information, thereby producing a sharpen effect that makes the boundary surface of the blurred part clearly visible, and restores it by removing blocking artifacts.

[0220] In the deblocking prediction (1370), filter setting information including a blocking area map for the probability of occurrence of blocking artifacts in the current frame and the probability of occurrence by area can be generated. In the original image, blocks are relatively more visible in areas with fast and complex movement and flat areas. Based on this, a blocking area map indicating the probability of occurrence of blocking artifacts can be generated.

[0221] In addition, a score map can be generated according to various criteria, and a blocking area map can be generated based on the weighted average.

[0222] For example, an A score map can be generated by calculating complexity by block or pixel from the original image to find areas of high complexity and assigning high scores to those areas of high complexity, and the complexity can be calculated based on the total variance value, etc. A B score map can be generated by identifying motion from the N-1th frame and optical flow, etc., and assigning high scores to areas with many changes in motion.

[0223] In addition, if there are many blocking artifacts in the previous image, there may be many blocking artifacts in the current image as well. Accordingly, the C score map can be generated by giving high scores to areas with many blocking artifacts in the previous area, and the D score map can be generated by giving high scores to areas with large differences in pixel boundary values ​​at block boundaries.

[0224] Finally, blocking artifacts may occur during encoding depending on the complexity of the current frame, the target bitrate, and the quantization parameter (QP). For example, the E score map can be generated based on a DQP (delta quantization parameter) map representing the difference between the average QP of the pixels in a block of the current frame and the QP of each pixel. The F score map is generated based on the complexity of encoding, i.e., the coding score, and the coding score is determined as the sum of the intra-coding score and the inter-coding score. The intra-coding score is determined based on the difference between the original image and the intra-coded image and the prediction bits used to code the image, using the current image as input. The inter-coding score can be determined based on the difference between the original image and the inter-coded image and the prediction bits used to code the image, using the current image and at least one previous image as input. The G score map can be generated based on the value obtained by dividing the coding score of the F score map by the target bitrate (channel information, bandwidth). Accordingly, the G-score map can be generated based on rate control, in which the encoder estimates the bitrate of the encoded bitstream based on the bandwidth available on the network. Even if the coding score is large, if the target bitrate is relatively large, the score in the G-score map may not be large.

[0225] As described above, the weighted average of score maps A through G can be the blocking area map. However, score maps A through G are examples and are not limited thereto.

[0226] In the deblocking prediction process, deblocking filtering can be performed according to filter setting information including such blocking area maps.

[0227] According to one embodiment, the deblocking filter may be a legacy deblocking filter.

[0228] According to another embodiment, the deblocking filter may be a trained DNN. Accordingly, the filter setting information transmitted for the deblocking filter in the deblocking prediction may include a blocking area map and DNN setting information. In FIGS. 20 and 21, a method and training process for training a third DNN for deblocking filtering for a preprocessing filter performed before AI downscaling and a fourth DNN for deblocking filtering for a postprocessing filter performed after AI upscaling are described below when the deblocking filter is a DNN.

[0229] Accordingly, even if changes occur in the communication environment, the quality of the video can be adjusted while maintaining the AI ​​downscale model and AI scale model by performing deblocking filtering based on the network environment, original video, and previous frame information without changing the AI ​​downscale model and AI scale model that were initially selected.

[0230] According to one embodiment, information regarding a DQP map or rate control based on the original image (1305), previous frame information, and network environment information obtained from the deblocking prediction (1370) may be additionally transmitted to the first encoding (1330) and used to perform the first encoding.

[0231] FIG. 14 is a block diagram showing the configuration of an AI decoding device (1400) according to one embodiment.

[0232] Referring to FIG. 14, an AI decoding device (1400) according to one embodiment includes a receiving unit (1410), an AI decoding unit (1420), and a deblocking prediction unit (1430). The AI ​​decoding unit (230) may include a parsing unit (1421), a first decoding unit (1422), an AI upscaling unit (1423), a deblocking filter unit (1424), and an AI setting unit (1425).

[0233] In FIG. 14, the receiver (1410), the AI ​​decoder (1420), and the deblocking prediction unit (1430) are depicted as separate devices, but the receiver (1410), the AI ​​decoder (1420), and the deblocking prediction unit (1430) may be implemented through a single processor. In this case, the receiver (1410), the AI ​​decoder (1420), and the deblocking prediction unit (1430) may be implemented as a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU) and software. Additionally, in the case of a dedicated processor, it may include memory for implementing the embodiments of the present disclosure or a memory processing unit for using external memory.

[0234] The receiver (1410), the AI ​​decoding unit (1420), and the deblocking prediction unit (1430) may be composed of multiple processors. In this case, they may be implemented by a combination of dedicated processors, or by a combination of multiple general-purpose processors such as AP, CPU, and GPU, and S / W. In one embodiment, the receiver (1410) is implemented with a first processor, the first decoding unit (1422) is implemented with a second processor different from the first processor, the parsing unit (1421), the AI ​​upscaling unit (1423), the deblocking filter unit (1424), and the AI ​​setting unit (1425) are implemented with a third processor different from the first processor and the second processor, and the deblocking prediction unit (1430) may be implemented with a third processor different from the first processor, the second processor, and the third processor.

[0235] The receiver (1410) receives AI encoded data obtained as a result of AI encoding and filter setting information transmitted from the deblocking prediction unit of the AI ​​encoding device. For example, the AI ​​encoded data may be a video file having a file format such as mp4, mov, etc.

[0236] The receiver (1410) can receive AI encoded data and filter setting information transmitted through the network. The receiver (1410) outputs the AI ​​encoded data and filter setting information to the AI ​​decoder (1420).

[0237] In another embodiment, the receiver (1410) can receive AI encoded data including filter setting information.

[0238] In one embodiment, the AI ​​encoded data may be obtained from a data storage medium including a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, etc.

[0239] The parsing unit (1421) parses AI encoded data and transmits the image data generated as a first encoding result of the first image (1325) to the first decoding unit (1422), transmits the AI ​​data to the AI ​​setting unit (1425), and transmits the filter setting information to the deblocking prediction unit (1430).

[0240] In one embodiment, the deblocking prediction unit (1430) transmits filter setting information to the deblocking filter unit (1424). The filter setting information may include a blocking area map indicating the area where block artifacts occur.

[0241] In another embodiment, the filter setting information may include a blocking area map indicating the area where block artifacts occur and DNN setting information of the DNN for deblocking filtering.

[0242] In one embodiment, the parsing unit (1421) can parse video data and AI data that are separated and included within the AI ​​encoded data. The parsing unit (1421) can distinguish between the AI ​​data and video data included within the AI ​​encoded data by reading the header within the AI ​​encoded data. In one example, the AI ​​data may be included in the VSIF (Vendor Specific InfoFrame) within the HDMI stream. The structure of the AI ​​encoded data containing the separated AI data and video data is omitted as described above with reference to FIG. 9.

[0243] In another embodiment, the parsing unit (1421) parses image data from AI encoded data, extracts AI data from the image data, transmits the AI ​​data to the AI ​​setting unit (1425), and transmits the remaining image data to the first decoding unit (1422). That is, the AI ​​data may be included in the image data; for example, the AI ​​data may be included in the SEI (Supplemental enhancement information), which is an additional information area of ​​the bitstream corresponding to the image data. The structure of the AI ​​encoded data containing the image data containing the AI ​​data is omitted as described above with reference to FIG. 10.

[0244] In another embodiment, the parsing unit (1421) parses image data, AI data, and filter setting information that are separated and included within the AI ​​encoded data, outputs the image data generated as a first encoding result of the first image (1325) to the first decoding unit (1422), outputs the AI ​​data to the AI ​​setting unit (1425), and outputs the filter setting information to the deblocking prediction unit (1430).

[0245] In another embodiment, the parsing unit (1421) may divide a bitstream corresponding to image data into a bitstream to be processed by the first decoding unit (1422) and a bitstream corresponding to AI data, and output each divided bitstream to the first decoding unit (1422) and the AI ​​setting unit (1425).

[0246] The parsing unit (1421) may determine that the video data included in the AI ​​encoded data is video data obtained through a predetermined codec (e.g., MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1). In this case, the information may be transmitted to the first decoding unit (1422) so that the video data can be processed by the confirmed codec.

[0247] The first decoding unit (1422) restores a second image (1345) corresponding to the first image (1325) based on image data received from the parsing unit (1421). The second image (1345) obtained by the first decoding unit (1422) is provided to the AI ​​upscaling unit (1423).

[0248] According to an embodiment, first decoding-related information, such as prediction mode information, motion information, and quantization parameter information, may be provided from the first decoding unit (1422) to the AI ​​setting unit (1425). The first decoding-related information may be used to obtain DNN setting information.

[0249] The AI ​​data provided to the AI ​​setting unit (1425) includes information that enables AI upscaling of the second image (1345). At this time, the upscaling target of the second image (1345) must correspond to the downscaling target of the first DNN. Therefore, the AI ​​data must include information that can identify the downscaling target of the first DNN.

[0250] Specific examples of the information included in the AI ​​data include information on the difference between the resolution of the original image (1305) and the resolution of the first image (1325), and information related to the first image (132).

[0251] The difference information can be expressed as information regarding the degree of resolution conversion of the first image (1325) relative to the original image (1305) (e.g., resolution conversion rate information). Also, since the resolution of the first image (1325) can be determined through the resolution of the restored second image (1345) and the degree of resolution conversion can be verified through this, the difference information may be expressed only by the resolution information of the original image (1305). Here, the resolution information may be expressed as a horizontal / vertical screen size, or as an aspect ratio (16:9, 4:3, etc.) and a size of one axis. Additionally, if there is pre-set resolution information, it may be expressed in the form of an index or a flag.

[0252] Information related to the first image (1325) may include information on at least one of the resolution of the first image (1325), the bitrate of the image data obtained as a result of the first encoding of the first image (1325), and the type of codec used during the first encoding of the first image (1325).

[0253] The AI ​​setting unit (1425) can determine an upscale target for the second image (1345) based on at least one of the difference information included in the AI ​​data and information related to the first image (1325). The upscale target may indicate, for example, to what resolution the second image (1345) should be upscaled.

[0254] When the AI ​​upscale unit (1423) determines the upscale target, it AI upscales the second image (1345) through the second DNN to obtain an upscaled image (1355) corresponding to the upscale target.

[0255] The deblocking filter unit (1424) obtains a third image (1465) by applying deblocking filtering to the upscaled second image (1355) based on filter setting information obtained from the deblocking prediction unit (1430).

[0256] The AI ​​upscaling process through the second DNN is omitted as described in Figures 3 and 4.

[0257] In the following, the AI ​​setting unit (1425) determines an upscale target based on AI data, and the AI ​​upscale unit (1423) explains how to AI upscale a second image (1345) according to the upscale target.

[0258] In one embodiment, the AI ​​setting unit (1425) can store a plurality of DNN setting information that can be set in the second DNN.

[0259] Here, the DNN configuration information may include information on at least one of the number of convolution layers included in the second DNN, the number of filter kernels per convolution layer, and the parameters of each filter kernel.

[0260] Multiple DNN configuration information can correspond to various upscale targets, and a second DNN can operate based on DNN configuration information corresponding to a specific upscale target. Depending on the DNN configuration information, the second DNN may have different structures. For example, depending on one DNN configuration information, the second DNN may include three convolution layers, and depending on another DNN configuration information, the second DNN may include four convolution layers.

[0261] In one embodiment, the DNN configuration information may include only the parameters of the filter kernel used in the second DNN. In this case, the structure of the second DNN is not changed, but only the parameters of the internal filter kernel may change according to the DNN configuration information.

[0262] The AI ​​setting unit (1425) can obtain DNN setting information for AI upscaling of the second image (1345) among a plurality of DNN setting information. Each of the plurality of DNN setting information used here is information for obtaining an AI upscaled second image (1355) with a predetermined resolution and / or predetermined quality, and is trained in conjunction with the first DNN.

[0263] For example, one of the multiple DNN setting information may include information for obtaining a third image (1365) with a resolution twice as large as the resolution of the second image (1345), for example, a third image (1365) of 4K (4096*2160) which is twice as large as the second image (1345) of 2K (2048*1080), and another DNN setting information may include information for obtaining a third image (1365) with a resolution four times larger than the resolution of the second image (1345), for example, a third image (1365) of 8K (8192*4320) which is four times larger than the second image (1345) of 2K (2048*1080).

[0264] Each of the multiple DNN setting information is created in conjunction with the DNN setting information of the first DNN of the AI ​​encoding device (1500), and the AI ​​setting unit (1425) obtains one of the multiple DNN setting information according to an enlargement ratio corresponding to the reduction ratio of the DNN setting information of the first DNN. To do this, the AI ​​setting unit (1425) must verify the information of the first DNN. In order for the AI ​​setting unit (1425) to verify the information of the first DNN, the AI ​​decoding device (1400) according to one embodiment receives AI data including the information of the first DNN from the AI ​​encoding device (1500).

[0265] In other words, the AI ​​setting unit (1425) can use the information received from the AI ​​encoding device (1500) to identify the information targeted by the DNN setting information of the first DNN used to acquire the first image (1325), and acquire the DNN setting information of the second DNN trained in conjunction with it.

[0266] When DNN setting information for AI upscaling of the second image (1345) is obtained among multiple DNN setting information, the DNN setting information is transmitted to the AI ​​upscaling unit (1423), and input data can be processed based on the second DNN operating according to the DNN setting information.

[0267] For example, when the AI ​​upscaler (1423) obtains any one DNN configuration information, for each of the first convolution layer (310), the second convolution layer (330), and the third convolution layer (350) of the second DNN (300) shown in FIG. 3, the number of filter kernels included in each layer and the parameters of the filter kernels are set to the values ​​included in the obtained DNN configuration information.

[0268] Specifically, the AI ​​upscaler (1423) can replace the parameters of the 3 x 3 filter kernel used in any one of the convolution layers of the second DNN shown in FIG. 4 with {1, 1, 1, 1, 1, 1, 1, 1}, and if there is a change in the DNN configuration information, the parameters of the filter kernel can be replaced with the parameters included in the changed DNN configuration information, which are {2, 2, 2, 2, 2, 2, 2, 2}.

[0269] The AI ​​setting unit (1425) can obtain DNN setting information for upscaling the second image (1345) among a plurality of DNN setting information based on information included in the AI ​​data, and the AI ​​data used to obtain the DNN setting information is described in detail.

[0270] In one embodiment, the AI ​​setting unit (1425) can obtain DNN setting information for upscaling a second image (1345) among a plurality of DNN setting information based on difference information included in the AI ​​data. For example, if it is confirmed based on the difference information that the resolution of the original image (1305) (e.g., 4K (4096*2160)) is twice as large as the resolution of the first image (1325) (e.g., 2K (2048*1080)), the AI ​​setting unit (238) can obtain DNN setting information that can increase the resolution of the second image (1345) by two times.

[0271] In another embodiment, the AI ​​setting unit (1425) may obtain DNN setting information for AI upscaling a second image (1345) among a plurality of DNN setting information based on information related to a first image (1325) included in the AI ​​data. The AI ​​setting unit (1425) may predetermine a mapping relationship between the image-related information and the DNN setting information and obtain DNN setting information mapped to the information related to the first image (1325).

[0272] FIG. 15 is a block diagram showing the configuration of an AI encoding device (1500) according to one embodiment.

[0273] Referring to FIG. 15, the AI ​​encoding device (1500) may include an AI encoding unit (1510), a transmission unit (1520), and a deblocking prediction unit (1530). The AI ​​encoding unit (1510) may include a deblocking filter unit (1511), an AI downscale unit (1512), a first encoding unit (1513), a data processing unit (1514), and an AI setting unit (1515).

[0274] FIG. 15 illustrates the AI ​​encoding unit (1510), transmission unit (1520), and deblocking prediction unit (1530) as separate devices, but the AI ​​encoding unit (1510), transmission unit (1520), and deblocking prediction unit (1530) can be implemented through a single processor. In this case, it may be implemented with a dedicated processor, or it may be implemented through a combination of a general-purpose processor such as an AP, CPU, or GPU and S / W. In addition, in the case of a dedicated processor, it may include memory for implementing the embodiments of the present disclosure or a memory processing unit for using external memory.

[0275] Additionally, the AI ​​encoding unit (1510), transmission unit (1520), and deblocking prediction unit (1530) may be composed of multiple processors. In this case, they may be implemented by a combination of dedicated processors, or by a combination of multiple general-purpose processors such as AP, CPU, and GPU, and S / W. In one embodiment, the first encoding unit (1513) is composed of a first processor, the deblocking filter unit (1511), the AI ​​downscale unit (1512), the data processing unit (1514), and the AI ​​setting unit (1515) are implemented by a second processor different from the first processor, the transmission unit (1520) is implemented by a third processor different from the first processor and the second processor, and the deblocking prediction unit (1530) may be implemented by a fourth processor different from the first processor, the second processor, and the third processor.

[0276] The deblocking prediction unit (1530) acquires the original image (1305), previous frame information, and network environment information, and transmits filter setting information to the deblocking filter unit (1511) based on the original image (1305), previous frame information, and network environment information.

[0277] The deblocking prediction unit (1530) obtains previous frame information from the first encoding unit (1513) and obtains network environment information from the AI ​​decoder (1400) according to a real-time protocol.

[0278] In one embodiment, the deblocking prediction unit (1530) transmits filter setting information to the deblocking filter unit (1511). The filter setting information may include a blocking area map indicating the area where block artifacts occur.

[0279] In another embodiment, the filter setting information may include a blocking area map indicating the area where block artifacts occur and DNN setting information of the DNN for deblocking filtering.

[0280] The deblocking filter unit (1511) applies deblocking filtering to the original image (1305) based on filter setting information obtained from the deblocking prediction unit (1530) to obtain the deblocking filtered original image (1315).

[0281] The AI ​​encoding unit (1513) performs AI downscaling of the deblocking filtered original image (1315) and first encoding of the first image (1325), and transmits the AI ​​encoded data to the transmission unit (1520). The transmission unit (1520) transmits the AI ​​encoded data and filter setting information to the AI ​​decoding device (1400).

[0282] In another embodiment, the transmission unit (1520) transmits AI encoded data including filter setting information, AI data, and image data to the AI ​​decoder (1400).

[0283] The image data includes data obtained as a result of the first encoding of the first image (1325). The image data may include data obtained based on pixel values ​​within the first image (1325), for example, residual data which is the difference between the first image (1325) and the prediction data of the first image (1325). Additionally, the image data includes information used in the first encoding process of the first image (1325). For example, the image data may include prediction mode information, motion information, and quantization parameter-related information used in the first encoding of the first image (1325).

[0284] The AI ​​data includes information that enables the AI ​​upscaler (1513) to AI upscale the second image (1345) to an upscale target corresponding to the downscale target of the first DNN.

[0285] In one example, the AI ​​data may include difference information between the original image (1305) and the first image (1325).

[0286] In one example, the AI ​​data may include information related to the first image (1325). The information related to the first image (1325) may include information about at least one of the resolution of the first image (1325), the bitrate of the image data obtained as a result of the first encoding of the first image (1325), and the type of codec used during the first encoding of the first image (1325).

[0287] In one embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information so that the second image (1345) can be AI upscaled to an upscale target corresponding to the downscale target of the first DNN.

[0288] In addition, in one embodiment, the AI ​​data may include DNN setting information that can be set in the second DNN.

[0289] The AI ​​downscaler (1512) can obtain an AI downscaled image (1325) through a first DNN from a deblocking filtered original image (1315). The AI ​​downscaler (1512) can AI downscale the filtered original image (1315) using DNN setting information provided by the AI ​​setting unit (1515).

[0290] The AI ​​setting unit (1515) can determine the downscale target of the original image (105) based on predetermined criteria.

[0291] In order to acquire a first image (1325) corresponding to a downscale target, the AI ​​setting unit (1515) can store multiple DNN setting information that can be set in the first DNN. The AI ​​setting unit (1515) acquires DNN setting information corresponding to the downscale target among the multiple DNN setting information and provides the acquired DNN setting information to the AI ​​downscale unit (1512).

[0292] Each of the above multiple DNN setting information may be trained to obtain a first image (1325) of a predetermined resolution and / or predetermined quality. For example, one of the multiple DNN setting information may include information for obtaining a first image (1325) with a resolution that is 1 / 2 times smaller than the resolution of the original image (1305), for example, a first image (1325) of 2K (2048*1080) that is 1 / 2 times smaller than the original image (1305) of 4K (4096*2160), and another DNN setting information may include information for obtaining a first image (1325) with a resolution that is 1 / 4 times smaller than the resolution of the original image (1305), for example, a first image (1325) of 2K (2048*1080) that is 1 / 4 times smaller than the original image (1305) of 8K (8192*4320).

[0293] According to an implementation example, if the information constituting the DNN setting information (e.g., the number of convolution layers, the number of filter kernels per convolution layer, the parameters of each filter kernel, etc.) is stored in the form of a lookup table, the AI ​​setting unit (1515) may obtain the DNN setting information by combining a selected portion of the lookup table values ​​according to the downscale target, and provide the obtained DNN setting information to the AI ​​downscale unit (1512).

[0294] According to an embodiment, the AI ​​setting unit (1515) may determine the structure of a DNN corresponding to a downscale target and obtain DNN setting information corresponding to the determined structure of the DNN, such as parameters of a filter kernel.

[0295] Multiple DNN configuration information for AI downscaling of a filtered original image (1315) can have optimized values ​​by training the first DNN and the second DNN in conjunction. Here, each DNN configuration information includes at least one of the number of convolution layers included in the first DNN, the number of filter kernels per convolution layer, and the parameters of each filter kernel.

[0296] The AI ​​downscaler (1512) sets the first DNN with DNN setting information determined for AI downscale of the filtered original image (1315), and can obtain the first image (1325) of a predetermined resolution and / or predetermined quality through the first DNN. When DNN setting information for AI downscale of the original image (1305) is obtained among the plurality of DNN setting information, each layer within the first DNN can process the input data based on the information included in the DNN setting information.

[0297] The AI ​​setting unit (1515) describes how to determine a downscale target. The downscale target may indicate, for example, how much resolution a first image (1325) with reduced resolution should be obtained from a deblocking filtered original image (1315).

[0298] The AI ​​setting unit (1515) acquires one or more input information. In one embodiment, the input information may include at least one of the target resolution of the first image (1325), the target bitrate of the image data, the bitrate type of the image data (e.g., variable bitrate type, constant bitrate type or average bitrate type, etc.), the color format to which AI downscale is applied (luminance component, chrominance component, red component, green component or blue component, etc.), the codec type for the first encoding of the first image (1325), compression history information, the resolution of the original image (1305), and the type of the original image (1305).

[0299] One or more input information may include information that is pre-stored in the AI ​​encoding device (1500) or received from the user.

[0300] The AI ​​setting unit (1515) controls the operation of the AI ​​downscale unit (1512) based on input information. In one embodiment, the AI ​​setting unit (1515) determines a downscale target according to the input information and can provide DNN setting information corresponding to the determined downscale target to the AI ​​downscale unit (1512).

[0301] In one embodiment, the AI ​​setting unit (1515) may transmit at least a portion of the input information to the first encoding unit (1513) so that the first encoding unit (1513) first encodes the first image (1325) with a bit rate of a specific value, a bit rate of a specific type, and a specific codec.

[0302] In another embodiment, the deblocking prediction unit (1530) may additionally acquire information regarding the DQP map or rate control based on the original image (1305), previous frame information, and network environment information, and transmit the information regarding the DQP map or rate control to the first encoding unit (1513). Additionally, the first encoding unit (1513) may use the information regarding the DQP map or rate control to first encode the first image (1325).

[0303] In one embodiment, the AI ​​setting unit (1515) can determine a downscale target based on at least one of a compression rate (e.g., a difference in resolution between the original image (1305) and the first image (1325), a target bitrate), a compression quality (e.g., a bitrate type), compression history information, and a type of the original image (1305).

[0304] In one example, the AI ​​setting unit (1515) can determine a downscale target based on a compression rate or compression quality that is pre-set or input by the user.

[0305] As another example, the AI ​​setting unit (1515) may determine a downscale target using compression history information stored in the AI ​​encoding device (1500). For example, according to the compression history information available to the AI ​​encoding device (1500), the encoding quality or compression ratio preferred by the user may be determined, and the downscale target may be determined according to the encoding quality, etc. determined based on the compression history information. For example, the resolution, image quality, etc. of the first video (115) may be determined according to the encoding quality that has been used most frequently according to the compression history information.

[0306] As another example, the AI ​​setting unit (1515) may determine a downscale target based on encoding quality that has been used more than a predetermined threshold value according to compression history information (e.g., the average quality of encoding quality that has been used more than a predetermined threshold value).

[0307] As another example, the AI ​​setting unit (1515) may determine a downscale target based on the resolution and type (e.g., file format) of the original image (1305).

[0308] In one embodiment, when the original image (1305) is composed of a plurality of frames, the AI ​​setting unit (1515) may independently acquire DNN setting information for a predetermined number of frames and provide the independently acquired DNN setting information to the AI ​​downscale unit (1512).

[0309] In one example, the AI ​​setting unit (1515) divides the frames constituting the original image (1305) into a predetermined number of groups and can independently obtain DNN setting information for each group. For each group, identical or different DNN setting information may be obtained. The number of frames included in the groups may be the same or different for each group.

[0310] In another example, the AI ​​setting unit (1515) can independently determine DNN setting information for each frame constituting the original image (1305). For each frame, identical or different DNN setting information can be obtained.

[0311] Hereinafter, in FIGS. 16 to 18, embodiments in which the position of the deblocking filter differs from FIG. 13 are described.

[0312] FIG. 16 is a diagram illustrating an AI encoding process and an AI decoding process that adaptively add deblocking filtering according to another embodiment.

[0313] FIG. 16 shows an embodiment in which the position of the deblocking filter in the AI ​​encoding process is different compared to the AI ​​encoding process and AI decoding process that adaptively add the deblocking filtering of FIG. 13.

[0314] Referring to FIG. 16, in the deblocking prediction process (1670), information on the network environment of the current streaming channel, the original video (1605), and information on a previously encoded frame are obtained to predict whether a blocking phenomenon will occur in the input original video, and filter setting information, such as a blocking area map for the area where blocks occur, is transmitted when AI encoding and AI decoding are performed. In the AI ​​encoding process, the original image (1605) is AI downscaled (1610) to obtain a first image (1615), and based on the filter setting information obtained in the deblocking prediction process (1670), a deblocking filter (1620) is applied to the first image (1615) to obtain a deblocking filtered first image (1625), and the deblocking filtered first image (1625) is first encoded (1630) and transmitted. In the AI ​​decoding process, AI encoded data including AI data and image data obtained as a result of AI encoding is received, a second image (1645) is obtained through the first decoding (1640), the second image (1645) is AI upscaled (1650) to obtain an AI upscaled image (1355), and deblocking according to the filter setting information transmitted in the deblocking prediction (1370). A third image (1665) is obtained by applying a filter (1660).

[0315] FIG. 17 is a diagram illustrating an AI encoding process and an AI decoding process that adaptively add deblocking filtering according to another embodiment.

[0316] FIG. 17 shows an embodiment in which the positions of the deblocking filter in the AI ​​encoding process and the deblocking filter in the AI ​​decoding process are both different compared to the AI ​​encoding process and the AI ​​decoding process that adaptively add the deblocking filtering of FIG. 13.

[0317] Referring to FIG. 17, in the deblocking prediction process (1770), information on the network environment of the current streaming channel, the original video (1705), and information on the previously encoded frame are obtained to predict whether a blocking phenomenon will occur in the input original video, and filter setting information, such as a blocking area map for the area where blocks occur, is transmitted when AI encoding and AI decoding are performed. In the AI ​​encoding process, the original image (1705) is AI downscaled (1710) to obtain a first image (1715), and based on the filter setting information obtained in the deblocking prediction process (1770), a deblocking filter (1720) is applied to the first image (1715) to obtain a deblocking filtered first image (1725), and the deblocking filtered first image (1725) is first encoded (1730) and transmitted. In the AI ​​decoding process, AI encoded data including AI data and image data obtained as a result of AI encoding is received, a second image (1745) is obtained through the first decoding (1740), and a deblocking filter (1750) according to the filter setting information transmitted in the deblocking prediction (1370) is applied to the second image (1745) to obtain a deblocking filtered second image (1755). The deblocking filtered second image (1755) is AI upscaled (1760) to obtain the third image (1765).

[0318] FIG. 18 is a diagram illustrating an AI encoding process and an AI decoding process that adaptively add deblocking filtering according to another embodiment.

[0319] FIG. 18 shows an embodiment in which the position of the deblocking filter in the AI ​​decoding process is different compared to the AI ​​encoding process and AI decoding process that adaptively add the deblocking filtering of FIG. 13.

[0320] Referring to FIG. 18, in the deblocking prediction process (1870), information on the network environment of the current streaming channel, the original video (1805), and information on a previously encoded frame are obtained to predict whether a blocking phenomenon will occur in the input original video, and filter setting information, such as a blocking area map for the area where blocks occur, is transmitted when AI encoding and AI decoding are performed. In the AI ​​encoding process, based on the filter setting information obtained in the deblocking prediction process (1870), a deblocking filter (1810) is applied to the original image (1805) to obtain a first image (1825) by AI downscaling (1820) the deblocking filtered original image (1815), and the first image (1825) is encoded in the first encoding process (1830) and transmitted. In the AI ​​decoding process, AI encoded data including AI data and image data obtained as a result of AI encoding is received, a second image (1845) is obtained through the first decoding process (1840), a deblocking filter (1850) according to the filter setting information transmitted in the deblocking prediction process (1870) is applied to the second image (1845) to obtain a deblocking filtered second image (1855), and the deblocking filtered second image (1855) is AI The third image (1865) is obtained by upscaling (1860).

[0321] Below, a DNN for deblocking filtering is described in the case where the deblocking filtering of FIG. 13 is an AI deblocking filter trained with a DNN rather than a legacy deblocking filter.

[0322] FIG. 19 is an exemplary diagram showing a third DNN and a fourth DNN for an AI-based deblocking filter.

[0323] As illustrated in FIG. 19, the original image (1305) or the upscaled second image (1355) is input to the first convolution layer (1910). The first convolution layer (1910) performs convolution processing on the original image (1305) or the upscaled second image (1355) using four filter kernels of size 3 x 3. As a result of the convolution processing, four feature maps are generated. Each feature map represents the unique characteristics of the original image (1305) or the upscaled second image (1355). For example, each feature map may represent the vertical characteristics, horizontal characteristics, edge characteristics, or the possibility of blocking artifacts occurring of the original image (1305) or the upscaled second image (1355). The generated four feature maps are input to the first activation layer (1920).

[0324] The first activation layer (1920) can impart non-linear characteristics to the four feature maps.

[0325] The first activation layer (1920) determines whether to pass sample values ​​of feature maps output from the first convolution layer (1910) to the second convolution layer (1930). For example, some sample values ​​of the feature maps are activated by the first activation layer (1920) and passed to the second convolution layer (1930), while some sample values ​​are deactivated by the first activation layer (1920) and not passed to the second convolution layer (1930). The information represented by the feature maps output from the first convolution layer (1910) is highlighted by the first activation layer (1920).

[0326] The output (1925) of the first activation layer (1920) is input to the second convolution layer (1930). The second convolution layer (1930) performs convolution processing on the input data using four filter kernels of size 3 x 3. The four feature maps output as a result of the convolution processing are input to the second activation layer (1940), and the second activation layer (1940) can impart non-linear characteristics to the four feature maps.

[0327] The output (1945) of the second activation layer (1940) is input to the third convolution layer (1950). The third convolution layer (1950) performs convolution processing on the input data using one filter kernel of size 3 x 3. As a result of the convolution processing, one image may be output from the third convolution layer (1950). The third convolution layer (1950) obtains one output using one filter kernel as a layer for outputting the final image. According to an example of the present disclosure, the third convolution layer (1950) may output a deblocking filtered original image (1315) or a third image (1365) through the result of the convolution operation.

[0328] There may be multiple DNN configuration information indicating the number of filter kernels, parameters of the filter kernels, etc. of the first convolution layer (1910), the second convolution layer (1930), and the third convolution layer (1950) of the third DNN or the fourth DNN.

[0329] Multiple DNN configuration information of the third DNN and the fourth DNN must be linked with each other. Linkage between the multiple DNN configuration information of the third DNN and the multiple DNN configuration information of the fourth DNN can be implemented through linked learning of the third DNN and the fourth DNN.

[0330] FIG. 19 illustrates that the third or fourth DNN includes three convolution layers (1910, 1930, 1950) and two activation layers (1920, 1940), but this is merely an example, and depending on the embodiment, the number of convolution layers and activation layers may vary. Additionally, depending on the embodiment, the third or fourth DNN may be implemented through a recurrent neural network (RNN). In this case, it means changing the CNN structure of the third or fourth DNN according to the example of the present disclosure to an RNN structure.

[0331] In one embodiment, the deblocking filter unit (1424) or the deblocking filter unit (1511) may include at least one ALU for convolution operations and operations of the activation layer. The ALU may be implemented as a processor. For convolution operations, the ALU may include a multiplier that performs a multiplication operation between sample values ​​of the original image (1305), the upscaled second image (1355), or the feature map output from the previous layer and sample values ​​of the filter kernel, and an adder that adds the results of the multiplication. Additionally, for operations of the activation layer, the ALU may include a multiplier that multiplies the input sample value by a weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and a comparator that compares the result of the multiplication with a predetermined value to determine whether to pass the input sample value to the next layer.

[0332] Hereinafter, with reference to FIG. 20, a method for training the third DNN (2010) and the fourth DNN (2020) in combination will be described. The first DNN (800) and the second DNN (300) used for training the third DNN (2010) and the fourth DNN (2020) in combination are DNNs trained in the manner described in FIG. 11 and FIG. 12.

[0333] FIG. 20 is a diagram illustrating a method for training the third DNN (2010) and the fourth DNN (2020).

[0334] In one embodiment, when there is a change in the streaming environment (when there is a large number of users or a change in the communication environment outdoors), the original video (1305), previous frame information, and network environment information are obtained through a deblocking prediction process, and based on the original video (1305), previous frame information, and network environment information, deblocking filter setting information is transmitted for AI encoding and AI decoding. Through the AI ​​encoding process, the original video is deblocked filtered based on the deblocking filter setting information transmitted through the deblocking prediction process, and the first video (1325) that has been AI downscaled is first encoded and transmitted, the second video (1345) that has been first decoded is AI upscaled, and is restored to a third video (1365) that has been deblocked filtered based on the filter setting information transmitted through the deblocking prediction process.

[0335] In the AI ​​encoding and AI decoding processes described in FIGS. 1 to 12, when a streaming environment is maintained, the DNN setting information of the first DNN and the second DNN is selected to be suitable for the resolution and bitrate according to the initial network environment, so that video is transmitted without additional deblocking filters; however, when a problem occurs in the streaming environment, video can be transmitted while maintaining video quality through additional deblocking filters without changing the DNN setting information of the first DNN and the second DNN.

[0336] Accordingly, the third DNN and the fourth DNN for deblocking filtering are AI downscaled and AI upscaled with the DNN setting information of the first DNN and the second DNN, respectively, trained to correspond to a specific resolution and a specific bit rate, and are trained under conditions where the bandwidth or bit rate of the network channel is partially changed assuming a change in the network environment.

[0337] Accordingly, in order to maintain similarity between the original image (1305) and the third image (1365) obtained from AI encoding and AI decoding with added deblocking filtering, a correlation between the AI ​​encoding process and the AI ​​decoding process is required. That is, information lost during the AI ​​encoding process must be recoverable during the AI ​​decoding process, and for this purpose, the training of the third DNN (2010) and the fourth DNN (2020) in conjunction is required.

[0338] For accurate deblocking filtering and AI decoding, it is ultimately necessary to reduce quality loss information (2050) corresponding to the comparison result between the third training image (2004) and the original training image (2001) shown in FIG. 20. Thus, quality loss information (2050) is used for training both the third DNN (2010) and the fourth DNN (2020).

[0339] First, the training process illustrated in Fig. 20 will be explained.

[0340] In FIG. 20, the original training image (2001) is an image subject to deblocking filtering and AI downscaling, and the first training image (2002) is an image deblocking filtered and AI downscaled from the original training image (1101). Additionally, the third training image (2004) is an image AI upscaled and deblocking filtered from the first training image (2002).

[0341] The original training image (2001) includes a still image or a video composed of multiple frames. In one embodiment, the original training image (2001) may include a luminance image extracted from a still image or a video composed of multiple frames. Additionally, in one embodiment, the original training image (2001) may include a patch image extracted from a still image or a video composed of multiple frames. If the original training image (2001) consists of multiple frames, the first training image (2002), the second training image, and the third training image (2004) are also composed of multiple frames. When multiple frames of the original training video (2001) are sequentially input into the third DNN (2010) and the trained first DNN (800), multiple frames of the first training video (2002), the second training video, and the third training video (2004) can be sequentially acquired through the third DNN (2010), the trained first DNN (800), the trained second DNN (300), and the fourth DNN (2020).

[0342] For the combined training of the third DNN (2010) and the fourth DNN (2020), the original training image (2001) is input into the third DNN (2010) and the trained first DNN (800). The original training image (2001) input into the third DNN (2010) and the trained first DNN (800) is deblocked filtered and AI downscaled to be output as the first training image (2002), and the first training image (2002) is input into the trained second DNN (300) and the fourth DNN (2020). As a result of AI upscaling and deblocking filtering for the first training image (2002), the third training image (2004) is output.

[0343] Referring to FIG. 20, a first training image (2002) is input to a trained second DNN (300). Depending on the embodiment, a second training image obtained through a first encoding and first decoding process of the first training image (2002) may be input to the trained second DNN (300). Any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 may be used to input the second training image to the second DNN. Specifically, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 may be used for the first encoding of the first training image (2002) and the first decoding of the image data corresponding to the first training image (2002).

[0344] Referring to FIG. 20, a legacy downscaled reduced training image (2003) is obtained from the original training image (2001), separate from the output of the first training image (2002) through the third DNN (2010) and the trained first DNN (800). Here, the legacy downscale may include at least one of a bilinear scale, a bicubic scale, a lanczos scale, and a stair step scale.

[0345] In order to prevent the structural features of the first image (1325) from deviating significantly from the structural features of the original image (1305), a reduced training image (2003) that preserves the structural features of the original training image (2001) is obtained.

[0346] Before training proceeds, the 3rd DNN (2010) and the 4th DNN (2020) can be set with predetermined DNN configuration information. As training proceeds, structural loss information (2030), complexity loss information (2040), and quality loss information (2050) can be determined.

[0347] Structural loss information (2030) can be determined based on the comparison result between the reduced training image (2003) and the first training image (1102). In one example, structural loss information (2030) may correspond to the difference between the structural information of the reduced training image (2003) and the structural information of the first training image (2002). Structural information may include various features extractable from the image, such as luminance, contrast, and histogram. Structural loss information (2030) indicates the extent to which the structural information of the original training image (2001) is preserved in the first training image (2002). The smaller the structural loss information (2030), the more similar the structural information of the first training image (2002) is to the structural information of the original training image (2001).

[0348] Complexity loss information (2040) can be determined based on the spatial complexity of the first training image (2002). In one example, the total variance value of the first training image (2002) can be used as the spatial complexity. Complexity loss information (2040) is related to the bitrate of the image data obtained by first encoding the first training image (2002). It is defined that the bitrate of the image data is smaller the complexity loss information (2040).

[0349] Quality loss information (2050) can be determined based on the comparison results between the original training image (2001) and the third training image (2004). Quality loss information (2050) may include at least one of an L1-norm value, an L2-norm value, a SSIM (Structural Similarity) value, a PSNR-HVS (Peak Signal-To-Noise Ratio-Human Vision System) value, an MS-SSIM (Multiscale SSIM) value, a VIF (Variance Inflation Factor) value, and a VMAF (Video Multimethod Assessment Fusion) value regarding the difference between the original training image (2001) and the third training image (2004). Quality loss information (2050) indicates the degree to which the third training image (2004) is similar to the original training image (2001). The smaller the quality loss information (2050), the more similar the third training image (2004) is to the original training image (2001).

[0350] Referring to FIG. 20, structural loss information (2030), complexity loss information (2040), and quality loss information (2050) are used for training the third DNN (2010), and quality loss information (2050) is used for training the fourth DNN (2020). That is, quality loss information (2050) is used for training both the third DNN (2010) and the fourth DNN (2020).

[0351] The third DNN (2010) can update parameters so that the final loss information determined based on structural loss information (2030), complexity loss information (2040), and quality loss information (2050) is reduced or minimized. Additionally, the fourth DNN (2020) can update parameters so that the quality loss information (2050) is reduced or minimized.

[0352] The final loss information for training the 3rd DNN (2010) and 4th DNN (2020) can be determined as shown in Equation 2 below.

[0353] [Mathematical Formula 2]

[0354] LossDDS = e * Structural Loss Information + f * Complexity Loss Information + g * Quality Loss Information

[0355] LossDUS = h * Quality Loss Information

[0356] In the above mathematical formula 2, LossDDS represents the final loss information to be reduced or minimized for training the third DNN (2010), and LossDUS represents the final loss information to be reduced or minimized for training the fourth DNN (2020). Additionally, e, f, g, and h may correspond to predetermined weights.

[0357] That is, the third DNN (2010) updates the parameters in the direction in which the LossDDS of Equation 2 decreases, and the fourth DNN (2020) updates the parameters in the direction in which the LossDUS decreases. When the parameters of the third DNN (2010) are updated according to the LossDDS derived during the training process, the first training image (2002) obtained based on the updated parameters becomes different from the first training image (2002) in the previous training process, and accordingly, the third training image (2004) also becomes different from the third training image (2004) in the previous training process. When the third training image (2004) becomes different from the third training image (2004) in the previous training process, the quality loss information (2050) is also newly determined, and accordingly, the fourth DNN (2020) updates the parameters. When the quality loss information (2050) is newly determined, the LossDDS is also newly determined, so the third DNN (2010) updates its parameters according to the newly determined LossDDS. That is, the parameter update of the third DNN (2010) causes the parameter update of the fourth DNN (2020), and the parameter update of the fourth DNN (2020) causes the parameter update of the third DNN (2010). In other words, since the third DNN (2010) and the fourth DNN (2020) are trained in conjunction through the sharing of the quality loss information (2050), the parameters of the third DNN (2010) and the parameters of the fourth DNN (2020) can be optimized with a correlation to each other.

[0358] Referring to Equation 2, it can be seen that LossDUS is determined based on quality loss information (2050), but this is just one example, and LossDUS may also be determined based on at least one of structural loss information (2030) and complexity loss information (2040) and quality loss information (2050).

[0359] Previously, it was explained that the deblocking prediction unit (1430) of the AI ​​decoding device (1400) and the deblocking prediction unit (1530) of the AI ​​encoding device (1500) store filter setting information, and the filter setting information may also include multiple DNN setting information. A method for training each of the multiple DNN setting information stored in the deblocking prediction unit (1430) and the deblocking prediction unit (1530) will be described.

[0360] As explained in relation to Equation 2, in the case of the third DNN (2010), parameters are updated by considering the degree of similarity between the structural information of the first training image (2002) and the structural information of the original training image (2001) (structural loss information (2030)), the bitrate of the image data obtained from the first encoding result of the first training image (2002) (complexity loss information (2040)), and the difference between the third training image (2004) and the original training image (2001) (quality loss information (2050)).

[0361] To explain in detail, the parameters of the third DNN (2010) can be updated so that a first training image (2002) can be obtained which is similar to the structural information of the original training image (2001) and has a small bit rate of image data obtained when the first encoding is performed, and at the same time, the fourth DNN (2020) that AI upscales the first training image (2002) can obtain a third training image (2004) similar to the original training image (2001).

[0362] As the weights of e, f, and g in Equation 2 are adjusted, the direction in which the parameters of the third DNN (2010) are optimized becomes different. For example, if the weight of f is set high, the parameters of the third DNN (2010) may be updated with greater importance placed on lowering the bitrate than on the quality of the third training video (2004). Also, if the weight of g is set high, the parameters of the third DNN (2010) may be updated with greater importance placed on increasing the quality of the third training video (2004) than on increasing the bitrate or maintaining the structural information of the original training video (2001).

[0363] In addition, the direction in which the parameters of the third DNN (2010) are optimized may differ depending on the type of codec used to first encode the first training image (2002). This is because, depending on the type of codec, the second training image to be input to the fourth DNN (2020) may differ.

[0364] That is, the parameters of the third DNN (2010) and the parameters of the fourth DNN (2020) can be updated in conjunction based on the weights e, f, g, and the type of codec for the first encoding of the first training image (2002). Therefore, when the weights e, f, and g are each determined to a predetermined value and the type of codec is determined to a predetermined type, and then the third DNN (2010) and the fourth DNN (2020) are trained, the parameters of the third DNN (2010) and the parameters of the fourth DNN (2020) that are optimized in conjunction with each other can be determined.

[0365] Then, after changing the weights e, f, g, and the type of codec, the third DNN (2010) and the fourth DNN (2020) are trained, and the parameters of the third DNN (2010) and the parameters of the fourth DNN (2020) that are optimized in conjunction with each other can be determined. In other words, by training the third DNN (2010) and the fourth DNN (2020) while changing the values ​​of the weights e, f, g, and the type of codec respectively, multiple DNN configuration information that is trained in conjunction with each other can be determined in the third DNN (2010) and the fourth DNN (2020).

[0366] FIG. 21 is a diagram illustrating the training process of the third DNN (2010) and the fourth DNN (2020) by the training device (2100).

[0367] Training of the third DNN (2010) and the fourth DNN (2020) described in relation to FIG. 20 can be performed by a training device (2100). The training device (2100) includes the third DNN (2010), the fourth DNN (2020), the trained first DNN (800), and the trained second DNN (300). The training device (2100) may be, for example, an AI encoding device (1500) or a separate server. The DNN configuration information of the fourth DNN (2020) obtained as a result of training is stored in an AI decoding device (1400).

[0368] Referring to FIG. 21, the training device (2100) initializes the DNN configuration information of the third DNN (2010) and the fourth DNN (2020) (S2110, S2115). By doing so, the third DNN (2010) and the fourth DNN (2020) can operate according to the predetermined DNN configuration information. The DNN configuration information may include information on at least one of the number of convolution layers included in the third DNN (2010) and the fourth DNN (2020), the number of filter kernels per convolution layer, the size of the filter kernels per convolution layer, and the parameters of each filter kernel.

[0369] The training device (2100) inputs the original training image (2001) into the third DNN (2010) (S2120). The original training image (1101) may include at least one frame constituting a still image or a video. A filtering result is output from the third DNN (2010) and input into the trained first DNN (S2125).

[0370] The third DNN (2010) processes the original training image (2001) according to the initial DNN setting information and outputs the first training image (2002) that has been AI downscaled through the first DNN (800) that has been deblocked filtered from the original training image (2001) (S2130). FIG. 21 is illustrated as the first training image (2002) output from the third DNN (2010) and the trained first DNN (800) being directly input into the trained second DNN (300), but the first training image (2002) output from the third DNN (2010) and the trained first DNN (800) may be input into the trained second DNN (300) by the training device (2100). Additionally, the training device (2100) can first encode and first decode the first training image (2002) using a predetermined codec, and then input the second training image into the trained second DNN (300).

[0371] The trained second DNN (300) AI upscales the first training image (2002) or the second training image (S2135) and outputs a third training image (2004) that has been deblocked filtered from the first training image (2002) or the second training image (S2140).

[0372] The training device (2100) produces complexity loss information (2040) based on the first training image (2002) (S2145).

[0373] The training device (2100) compares the reduced training image (2003) and the first training image (2002) to produce structural loss information (2030) (S2150).

[0374] The training device (2100) compares the original training image (2001) and the third training image (2004) to produce quality loss information (2050) (S1255).

[0375] The third DNN (2010) updates the initial DNN setting information through a back propagation process based on the final loss information (S2160). The training device (2100) can calculate the final loss information for training the third DNN (2010) based on the complexity loss information (2040), the structural loss information (2030), and the quality loss information (2050).

[0376] The 4th DNN (2020) updates the initial DNN setting information through a backtranslation process based on quality loss information or final loss information (S2165). The training device (2100) can calculate final loss information for training the 4th DNN (2020) based on quality loss information (2050).

[0377] Subsequently, the training device (2100), the third DNN (2010), and the fourth DNN (2020) update the DNN configuration information by repeating the process S2110 to S2165 until the final loss information is minimized. At this time, during each iteration, the third DNN (2010) and the fourth DNN (2020) operate according to the DNN configuration information updated in the previous process.

[0378] FIG. 22 is a flowchart for explaining an AI encoding method according to one embodiment.

[0379] In step S2210, the AI ​​encoding device (1500) obtains the original image, previously encoded frame information, and network environment information.

[0380] According to one embodiment, previous frame information may indicate whether block artifacts of the encoded previous frame have been detected.

[0381] According to one embodiment, network environment information may indicate a change in at least one of the bit rate and bandwidth of a network channel.

[0382] In step S2220, the AI ​​encoding device (1500) obtains deblocking filter setting information based on the original image, the previously encoded frame information, and the transmission environment network information.

[0383] According to one embodiment, if network environment information indicates that there is no change in the network environment, the deblocking filter setting information is not obtained, and deblocking filtering may not be performed. Specifically, in normal situations where multiple people are connected simultaneously or there is no change in the network channel outdoors, blocking phenomena that degrade image quality do not occur, so there is no need to additionally perform deblocking filtering.

[0384] According to one embodiment, the deblocking filter setting information may include a blocking area map indicating an area where block artifacts occur. Specifically, as described above in FIG. 13, the blocking area map may be determined through the weighted average of A score maps to G score maps.

[0385] In step S2230, the AI ​​encoding device (1500) applies deblocking filtering to the original image based on the deblocking filter setting information and obtains a first AI downscaled image through a DNN for downscaling.

[0386] According to one embodiment, deblocking filtering in AI encoding may be a filtering that redistributes the allocation of bits based on deblocking filter setting information. Specifically, the deblocking filter can reduce the amount of bits by blurring complex and fast-moving areas of the image, and redistribute the allocation of bits by distributing the reduced amount of bits to simple areas so that the blocking phenomenon in simple areas is less noticeable.

[0387] In step S2240, the AI ​​encoding device (1500) first encodes the first image to generate image data.

[0388] In step S2250, the AI ​​encoding device (1500) transmits the deblocking filter setting information, AI data including information related to the AI ​​downscale, and the image data.

[0389] FIG. 23 is a flowchart illustrating an AI decoding method according to one embodiment.

[0390] In step S2310, the AI ​​decoder (1400) obtains image data generated as a first encoding result of the first image, AI data related to AI downscaling from the original image to the first image, and deblocking filter setting information.

[0391] According to one embodiment, if deblocking filter setting information is not obtained, deblocking filtering may not be performed. Specifically, since the fact that deblocking filter setting information is not obtained implies a normal situation where a deblocking filter should be applied—that is, when multiple people connect simultaneously or there is a change in the network channel outdoors—there is no need to perform additional deblocking filtering.

[0392] According to one embodiment, the deblocking filter setting information may include a blocking area map indicating an area where block artifacts occur. Specifically, as described above in FIG. 13, the blocking area map may be determined through the weighted average of A score maps to G score maps.

[0393] In step S2320, the AI ​​decoding device (1400) first decodes the image data to obtain a second image corresponding to the first image.

[0394] In step S2330, the AI ​​decoding device (1400) obtains DNN setting information for AI upscaling of the second image among a plurality of DNN setting information based on the AI ​​data.

[0395] In step S2340, the AI ​​decoding device (1400) generates a restored image that is AI upscaled from the second image and has deblocking filtering applied based on the one deblocking filter through an upscale DNN (Deep Neural Network) operating with the acquired DNN setting information.

[0396] According to one embodiment, deblocking filtering in AI decoding may be a filtering that blurs the block boundaries of an image and removes blocking artifacts based on deblocking filter setting information. Specifically, the deblocking filter can perform deblocking filtering according to each area by blurring the boundaries between blocks of an image and removing blocking artifacts.

[0397] Meanwhile, the embodiments of the present disclosure described above can be written as a program or instruction that can be executed on a computer, and the written program or instruction can be stored on a medium.

[0398] A medium may be used to continuously store programs or instructions executable by a computer, or to temporarily store them for execution or download. Additionally, a medium may be a variety of recording or storage means in the form of a single or multiple pieces of hardware combined, and is not limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Furthermore, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0399] Meanwhile, the model related to the aforementioned DNN can be implemented as a software module. When implemented as a software module (e.g., a program module containing instructions), the DNN model can be stored on a computer-readable recording medium.

[0400] Additionally, the DNN model may be integrated in the form of a hardware chip and become part of the aforementioned AI decoding device (1400) or AI encoding device (1500). For example, the DNN model may be manufactured in the form of a dedicated hardware chip for artificial intelligence, or it may be manufactured as part of an existing general-purpose processor (e.g., CPU or application processor) or a graphics-dedicated processor (e.g., GPU).

[0401] Furthermore, the DNN model may be provided in the form of downloadable software. The computer program product may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically through a manufacturer or an electronic market. For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer or electronic market, or a storage medium of a relay server.

[0402] Although the technical concept of the present disclosure has been described in detail with reference to preferred embodiments, the technical concept of the present disclosure is not limited to the above embodiments, and various modifications and changes can be made by those skilled in the art within the scope of the technical concept of the present disclosure.

Claims

Claim 1 An AI encoding device comprising: a memory; and a processor, wherein the memory stores instructions executable by the processor, and the processor acquires an original image, previously encoded frame information, and network environment information; acquires deblocking filter setting information based on the original image, the previously encoded frame information, and the network environment information; applies deblocking filtering to the original image based on the deblocking filter setting information and acquires a first image AI downscaled through a DNN for downscaled, generates image data by first encoding the first image, and transmits AI data including the deblocking filter setting information and information related to the AI ​​downscaled, and the image data. Claim 2 An AI encoding device according to claim 1, wherein the deblocking filter setting information includes a blocking area map indicating an area where block artifacts occur. Claim 3 In paragraph 2, the AI ​​encoding device wherein the blocking region map is determined based on at least one of the complexity and motion change of the original image. Claim 4 In claim 1, the AI ​​encoding device wherein the deblocking filtering is a filtering that redistributes the allocation of bits based on the deblocking filter setting information. Claim 5 In claim 1, the DNN setting information of the downscale DNN is obtained through the combined training of the downscale DNN and the upscale DNN for AI upscale of the first image, an AI encoding device. Claim 6 An AI encoding device according to claim 1, wherein if the network environment information indicates that there is no change in the network environment, the deblocking filter setting information is not obtained and the deblocking filtering is not performed. Claim 7 In claim 1, the AI ​​encoding device wherein the network environment information represents a change in at least one of the bitrate and bandwidth of the network channel. Claim 8 In claim 1, the AI ​​encoding device, wherein the previously encoded frame information indicates whether block artifacts of the previously encoded frame have been detected. Claim 9 An AI encoding device according to claim 1, which further acquires information regarding a quantization parameter difference map and rate control based on the original image, the previously encoded frame information, and the network environment information, and performs the first encoding based on the information regarding the quantization parameter difference map and rate control. Claim 10 An AI decoding device comprising: a memory; and a processor, wherein the memory stores instructions executable by the processor, and the processor acquires image data generated as a first encoding result of a first image, AI data related to AI downscaling from an original image to the first image, and deblocking filter setting information, and first decodes the image data to acquire a second image corresponding to the first image, acquires DNN setting information for AI upscaling of the second image among a plurality of DNN setting information based on the AI ​​data, and generates a restored image that is AI upscaled from the second image and has deblocking filtering applied based on the deblocking filter setting information through a Deep Neural Network (DNN) for upscaling operating with the acquired DNN setting information. Claim 11 In claim 10, the AI ​​decoding device, wherein the deblocking filter setting information includes a blocking area map indicating an area where block artifacts occur. Claim 12 In claim 11, the AI ​​decoder, wherein the blocking region map is determined based on at least one of the complexity and motion change of the original image. Claim 13 In claim 10, the AI ​​decoding device wherein the deblocking filtering is a filtering that blurs the block boundaries of an image and removes blocking artifacts based on the deblocking filter setting information. Claim 14 In claim 10, the above plurality of DNN setting information is an AI decoding device obtained through the combined training of the upscale DNN and the downscale DNN used for AI downscale of the original image. Claim 15 An AI decoding device according to claim 10, wherein if the above deblocking filter setting information is not obtained, the above deblocking filtering is not performed. Claim 16 A method for AI encoding using an AI encoding device, comprising: a step of acquiring an original image, previously encoded frame information, and network environment information; a step of acquiring deblocking filter setting information based on the original image, the previously encoded frame information, and the network environment information; a step of applying deblocking filtering to the original image based on the deblocking filter setting information and acquiring a first image AI downscaled through a DNN for downscaled; a step of generating image data by first encoding the first image; and a step of transmitting AI data including the deblocking filter setting information and information related to the AI ​​downscaled, and the image data. Claim 17 A method for AI decoding using an AI decoding device, comprising: a step of obtaining image data generated as a result of first encoding of a first image, AI data related to AI downscaling from an original image to the first image, and deblocking filter setting information; a step of obtaining a second image corresponding to the first image by first decoding the image data; a step of obtaining DNN setting information for AI upscaling of the second image among a plurality of DNN setting information based on the AI ​​data; and a step of generating a restored image that is AI upscaled from the second image and has deblocking filtering applied based on the deblocking filter setting information through a Deep Neural Network (DNN) for upscaling operating with the obtained DNN setting information.

Images