Method and apparatus for streaming data

Abstract

A terminal corresponding to the device is provided. The terminal receives information of a plurality of image data of different qualities; based on the information, requests a server to send image data among the plurality of image data to the terminal, receives image data and artificial intelligence (AI) data corresponding to the request, determines whether to perform AI upscaling on the received image data based on the AI ​​data, and based on the result of the determination, performs AI upscaling on the received image data through an upscaling deep neural network (DNN) jointly trained with a downscaling DNN of the server, confirms the state of a network between the terminal and the server, and requests image data based on the state of the network.

Description

Technical Field

[0001] The present disclosure relates to data streaming technology. More specifically, the present disclosure relates to a method and apparatus for adaptively streaming image data encoded by artificial intelligence (AI) using a deep neural network (DNN). Background Art

[0002] Schemes for transmitting image data through a network include a downloading scheme and a streaming scheme. The streaming scheme refers to a scheme in which a server transmits image data in real time and a terminal reproduces the received image data in real time.

[0003] Unlike a downloading scheme in which the reproduction of image data starts after the image data is completely transceived (i.e., completely sent and received), according to a streaming scheme, image data is transceived and reproduced in real time via a logical channel established between a server and a terminal, so that the quality of service (QoS) of the image data reproduction can be maintained while reflecting changes in the streaming environment.

[0004] Artificial intelligence (AI) systems are computer systems used to achieve human-level intelligence. Unlike general rule-based intelligent systems, AI systems learn and make decisions autonomously, thereby improving their capabilities. The more an AI system is used, the more the recognition rate of the AI ​​system increases, and the more accurately the AI ​​system understands user preferences. In this way, general rule-based intelligent systems can be replaced by deep learning-based AI systems. Summary of the invention

[0005] Technical issues

[0006] As interest in AI systems increases, many attempts are being actively made to apply AI systems to various technical fields. For example, research is being conducted to merge AI systems with technical fields including image processing, data processing, and the like.

[0007] Technical Solution

[0008] A terminal corresponding to the device is provided. The terminal receives information of a plurality of image data of different qualities; based on the information, requests a server to send image data among the plurality of image data to the terminal, receives image data and artificial intelligence (AI) data corresponding to the request, determines whether to perform AI amplification on the received image data based on the AI ​​data, and based on the result of the determination, performs AI amplification on the received image data through an amplification deep neural network (DNN) jointly trained with a reduction DNN of the server, confirms the state of a network between the terminal and the server, and requests image data based on the state of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0010] A brief description of each figure is provided to provide a more complete understanding of the drawings referenced in this specification.

[0011] Figure 1 is a diagram for describing an artificial intelligence (AI) encoding process and an AI decoding process according to an embodiment;

[0012] Figure 2 is a block diagram of a configuration of an AI decoding device according to an embodiment;

[0013] Figure 3 is a diagram illustrating a second deep neural network (DNN) for performing AI upscaling on a second image;

[0014] Figure 4 is a diagram for describing a convolution operation performed by a convolutional layer;

[0015] Figure 5 is a table showing a mapping relationship between a plurality of pieces of image-related information and a plurality of pieces of DNN setting information;

[0016] Figure 6 is a diagram showing a second image including a plurality of frames;

[0017] Figure 7 is a block diagram of a configuration of an AI encoding device according to an embodiment;

[0018] Figure 8 is a diagram illustrating a first DNN for performing AI downscaling on an original image;

[0019] Fig. 9 is a diagram for describing a method of training a first DNN and a second DNN;

[0020] Fig.10 is a diagram for describing a training process of a first DNN and a second DNN by a training device;

[0021] Fig.11 is a diagram of an apparatus for performing AI downscaling on an original image and an apparatus for performing AI upscaling on a second image;

[0022] Fig.12 is a diagram for describing the concept of a streaming system according to an embodiment of the present disclosure;

[0023] Fig.13A is a flowchart for describing a method of streaming data performed by a server according to an embodiment of the present disclosure;

[0024] Fig. 13B is a flowchart for describing a method for streaming data performed by a terminal according to an embodiment of the present disclosure;

[0025] Fig.14A is a flowchart for describing a method of streaming data performed by a server according to an embodiment of the present disclosure;

[0026] Fig. 14B is a flowchart for describing a method for streaming data performed by a terminal according to an embodiment of the present disclosure;

[0027] Fig.15A is a flowchart for describing a method of streaming data performed by a server according to an embodiment of the present disclosure;

[0028] Fig. 15B is a flowchart for describing a method for streaming data performed by a terminal according to an embodiment of the present disclosure;

[0029] Fig.16 is a diagram for describing a method of performing streaming between a server and a first terminal according to whether the first terminal supports AI amplification according to an embodiment of the present disclosure;

[0030] Fig.17 is a diagram for describing a method of performing streaming between a server and a first terminal according to whether the first terminal supports AI amplification according to an embodiment of the present disclosure;

[0031] Fig.18 is a diagram for describing a method of transmitting image data according to a capability stream of a terminal, which is performed by a server according to an embodiment of the present disclosure;

[0032] Fig.19 is a diagram for describing a method of streaming image data according to a state of a network and a capability of a terminal, which is performed by a server according to an embodiment of the present disclosure;

[0033] Fig. 20 is a diagram for describing a method of streaming image data corresponding to a state of a network based on additional information and capabilities, performed by a terminal according to an embodiment of the present disclosure;

[0034] Fig.21 is a diagram for describing additional information provided for streaming according to an embodiment of the present disclosure;

[0035] Fig. 22 is a diagram for describing a detailed configuration of additional information according to an embodiment of the present disclosure;

[0036] Fig.23 is a diagram for describing a detailed configuration of additional information according to an embodiment of the present disclosure;

[0037] Fig.24 is a diagram for describing a detailed configuration of additional information according to an embodiment of the present disclosure;

[0038] Fig.25 is a diagram for describing AI data and image data streamed from a server to a terminal according to an embodiment of the present disclosure;

[0039] Fig.26 is a diagram for describing a streaming system according to an embodiment of the present disclosure;

[0040] Fig. 27 is a block diagram showing a configuration of a server according to an embodiment of the present disclosure; and

[0041] Fig.28 is a block diagram illustrating a configuration of a terminal according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Best Practices

[0043] Provided are a method and apparatus for streaming data encoded by artificial intelligence (AI) using a deep neural network (DNN).

[0044] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure.

[0045] According to an embodiment of the present disclosure, a method for streaming data is provided, including: receiving information of a plurality of image data of different qualities; based on the information, requesting a server to send image data of the plurality of image data of different qualities to a terminal; receiving image data and artificial intelligence (AI) data corresponding to the request; determining whether to perform AI upscaling on the received image data based on the AI ​​data; based on a result of the determination whether to perform AI upscaling, performing AI upscaling on the received image data by an upscaling deep neural network (DNN) jointly trained with a downscaling DNN of the server; confirming a state of a network between the terminal and the server; and requesting the server to send image data of different qualities of the plurality of image data of different qualities based on the information of the plurality of image data of different qualities and the state of the network.

[0046] The method may further include: determining whether AI downscaling has been performed on the received image data by the downscaling DNN of the server based on the AI ​​data, and wherein the step of determining whether to perform AI upscaling on the received image data includes: determining to perform AI upscaling on the received image data when it is confirmed that AI downscaling has been performed on the received image data.

[0047] The information of the multiple image data includes quality information and AI scaling conversion information of the multiple image data, and wherein the step of requesting the server to send image data of different qualities includes requesting image data corresponding to the state of the network based on one or both of the quality information and the AI ​​scaling conversion information.

[0048] The method may further include determining image data corresponding to the state of the network based on capability information including information indicating whether the terminal supports AI zoom-in and information on an AI zoom-in level that the terminal can support.

[0049] The server may be a content provider server, wherein the method further comprises requesting information of the plurality of image data from a service server, and wherein the step of receiving information of the plurality of image data comprises receiving information of the plurality of image data and an identifier of the content provider server from the service server.

[0050] According to an embodiment of the present disclosure, a method for streaming data is provided, comprising: receiving a request for image data from a terminal among a plurality of image data of different qualities from a server; in response to the request, sending to the terminal artificial intelligence (AI) data and image data that has been AI-encoded by a reduced DNN of the server jointly trained with an enlarged DNN of the terminal; and receiving, from the terminal, a request for image data of different qualities among the plurality of image data of different qualities according to a state of a network between the terminal and the server.

[0051] The AI ​​data may include information about the downscaled DNN that has been applied to the AI-encoded image data.

[0052] The step of receiving a request for image data of different qualities may include receiving a request for image data corresponding to a state of a network and determined based on one or both of the AI ​​scaling conversion information and the quality information of each of the plurality of image data of different qualities.

[0053] The image data of different qualities may include image data corresponding to a state of the network and determined from among the plurality of image data of different qualities based on capability information including information indicating whether the terminal supports AI upscaling and information about an AI upscaling level supported by the terminal.

[0054] The method may further include providing an identifier of the server to the terminal.

[0055] According to an embodiment of the present disclosure, a terminal for receiving streaming data includes: a memory storing one or more instructions; and at least one processor configured to execute one or more instructions to: receive information of multiple image data of different qualities; based on the information, request a server to send image data of the multiple image data of different qualities to the terminal; receive the requested image data and artificial intelligence (AI) data corresponding to the requested image data; based on the AI ​​data, determine whether to perform AI upscaling on the received image data; based on the result of the determination whether to perform AI upscaling, perform AI upscaling on the received image data by an upscaling deep neural network (DNN) of the terminal jointly trained with a downscaling DNN of the server; confirm the state of a network between the terminal and the server; and based on the information of the multiple image data of different qualities and the state of the network, request the server to send image data of different qualities of the multiple image data of different qualities.

[0056] The at least one processor may also be configured to determine, based on the AI ​​data, whether AI downscaling has been performed on the received image data by the downscaling DNN of the server, and when it is confirmed that AI downscaling has been performed on the received image data, determine to perform AI upscaling on the received image data.

[0057] The information of the multiple image data includes quality information and AI scaling conversion information of the multiple image data, and wherein the at least one processor is also configured to execute the one or more instructions to request image data corresponding to the status of the network based on one or both of the quality information and the AI ​​scaling conversion information.

[0058] The at least one processor may be further configured to determine image data corresponding to a state of the network based on capability information including information indicating whether the terminal supports AI zoom-in and information about an AI zoom-in level supported by the terminal.

[0059] The server may be a content provider server, and wherein the at least one processor is further configured to execute one or more instructions to: request a service server to provide information of the plurality of image data to the terminal, and receive an identifier of the content provider server and information of the plurality of image data from the service server.

[0060] According to an embodiment of the present disclosure, a server for streaming data is provided, comprising: a memory storing one or more instructions; and at least one processor configured to execute the one or more instructions to: receive a request for image data from a terminal for a plurality of image data of different qualities of a server, in response to the request, send to the terminal artificial intelligence (AI) data and image data that has been AI-encoded by a reduced deep neural network (DNN) jointly trained with an enlarged DNN of the terminal, and receive from the terminal a request for image data of different qualities from the plurality of image data of different qualities according to a state of a network between the terminal and the server.

[0061] The AI ​​data may include information about the downscaled DNN that has been applied to the AI-encoded image data.

[0062] The at least one processor may also be configured to execute the one or more instructions to: receive a request for a second version of the image content, wherein the second version of the image content corresponds to the state of the network and is determined based on one or both of AI scaling conversion information and quality information of each version of the image content in multiple different versions of the image content.

[0063] The image data of different qualities may include image data corresponding to the state of the network and determined based on capability information including information indicating whether the terminal supports AI upscaling and information about an AI upscaling level supported by the terminal.

[0064] The at least one processor may also be configured to execute one or more instructions to provide an identifier of the server to the terminal.

[0065] According to an embodiment of the present disclosure, there is provided a non-transitory computer-readable recording medium having recorded thereon a program for executing a method of streaming data.

[0066] Invention Mode

[0067] Since the present disclosure allows various changes and many examples, specific embodiments will be shown in the drawings and described in detail in the written description. However, this is not intended to limit the present disclosure to a specific practice mode, and it will be understood that all changes, equivalents and substitutes that do not depart from the spirit and technical scope of the present disclosure are included in the present disclosure.

[0068] In the description of the embodiments, when it is considered that the detailed explanation of the related art may unnecessarily obscure the essence of the present disclosure, the detailed explanation of the related art is omitted. In addition, the ordinal numbers (e.g., first, second, etc.) used in the description of the specification are identifier codes used to distinguish one element from another element.

[0069] Furthermore, in the present specification, it will be understood that when elements are “connected” or “coupled” to each other, the elements may be directly connected or coupled to each other, but may instead be connected or coupled to each other through an intervening element between them, unless otherwise specified.

[0070] In this specification, with respect to an element represented as a "unit" or a "module", two or more elements may be combined into one element, or one element may be divided into two or more elements according to subdivided functions. In addition, each element described below may perform some or all of the functions performed by another element in addition to performing its own main function, and some of the main functions of each element may be completely performed by another component.

[0071] Furthermore, in this specification, an “image” or a “screen” may refer to a still image, a moving image including a plurality of consecutive still images (or frames), or a video.

[0072] Furthermore, in this specification, a deep neural network (DNN) is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model using an algorithm.

[0073] In addition, in this specification, "parameter" is a value used in the operation process of each layer forming a neural network, and for example, may include a weight used when an input value is applied to a specific operation expression. Here, the parameter can be expressed in a matrix form. The parameter is a value set as a training result and can be updated by separate training data.

[0074] In addition, in this specification, a “first DNN” indicates a DNN used to perform artificial intelligence (AI) downscaling on an image, and a “second DNN” indicates a DNN used to perform artificial intelligence (AI) upscaling on an image.

[0075] In addition, in the present specification, "DNN setting information" includes information related to elements constituting the DNN. "DNN setting information" includes the above-mentioned parameters as information related to elements constituting the DNN. The first DNN or the second DNN can be set by using the DNN setting information.

[0076] In addition, in this specification, the "original image" means an image that is an object of AI encoding, and the "first image" means an image obtained as a result of performing AI reduction on the original image during the AI ​​encoding process. In addition, the "second image" means an image obtained via the first decoding during the AI ​​decoding process, and the "third image" means an image obtained by AI-enlarging the second image during the AI ​​decoding process.

[0077] In addition, in this specification, 'AI reduction' means a process of reducing the resolution of an image based on AI, 'first encoding' means an encoding process according to an image compression method based on frequency transformation. In addition, 'first decoding' means a decoding process according to an image reconstruction method based on frequency transformation, and 'AI enlargement' means a process of increasing the resolution of an image based on AI.

[0078] Expressions such as “at least one of…” when following a list of elements modify the entire list of elements and do not modify the individual elements in the list. For example, the expression “at least one of a, b, or c” means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0079] Figure 1 is a diagram for describing an AI encoding process and an AI decoding process according to an embodiment.

[0080] As described above, when the resolution of an image significantly increases, the throughput of information for encoding and decoding the image increases, and thus, a method for improving the efficiency of encoding and decoding of the image is required.

[0081] like Figure 1 As shown, according to an embodiment of the present disclosure, a first image 115 is obtained by performing AI reduction 110 on an original image 105 having a high resolution. Then, a first encoding 120 and a first decoding 130 are performed on the first image 115 having a relatively low resolution, so that the bit rate can be greatly reduced compared to when the first encoding 120 and the first decoding 130 are performed on the original image 105.

[0082] exist Figure 1 According to an embodiment, a first image 115 is obtained by performing AI reduction 110 on an original image 105, and a first encoding 120 is performed on the first image 115 during an AI encoding process. During an AI decoding process, AI encoded data including AI data and image data obtained as a result of the AI ​​encoding is received, a second image 135 is obtained through a first decoding 130, and a third image 145 is obtained by performing AI enlargement 140 on the second image 135.

[0083] Referring to the AI ​​encoding process in detail, when the original image 105 is received, AI reduction 110 is performed on the original image 105 to obtain a first image 115 of a specific resolution or a specific quality. Here, the AI ​​reduction 110 is performed based on AI, and the AI ​​for the AI ​​reduction 110 is jointly trained with the AI ​​for the AI ​​enlargement 140 of the second image 135. This is because the AI ​​reduction 110 and the AI ​​enlargement 120 have two competing goals of reducing and enlarging the image, and therefore, when the AI ​​for the AI ​​reduction 110 and the AI ​​for the AI ​​enlargement 140 are trained separately, the difference between the original image 105 as the object of AI encoding and the third image 145 reconstructed by AI decoding increases.

[0084] In an embodiment of the present disclosure, the AI ​​data may be used to maintain this joint relationship during the AI ​​encoding process and the AI ​​decoding process. Therefore, the AI ​​data obtained through the AI ​​encoding process may include information indicating an enlargement target, and during the AI ​​decoding process, the AI ​​enlargement 140 is performed on the second image 135 according to the enlargement target verified based on the AI ​​data.

[0085] The AI ​​for AI reduction 110 and the AI ​​for AI upscaling 140 may be implemented as DNNs. Fig. 9 As described, since the first DNN and the second DNN are jointly trained by sharing loss information under a target, the AI ​​encoding device may provide the target information used during the joint training of the first DNN and the second DNN to the AI ​​decoding device, and the AI ​​decoding device may perform AI upscaling 140 on the second image 135 to the target resolution based on the provided target information.

[0086] about Figure 1 The first encoding 120 and the first decoding 130 can reduce the amount of information of the first image 115 obtained by performing AI reduction 110 on the original image 105 through the first encoding 120. The first encoding 120 may include a process of generating prediction data by predicting the first image 115, a process of generating residual data corresponding to the difference between the first image 115 and the prediction data, a process of transforming the residual data of the spatial domain component into a frequency domain component, a process of quantizing the residual data transformed into the frequency domain component, and a process of entropy encoding the quantized residual data. Such a first encoding 120 can be performed by one of the image compression methods using frequency transformation (such as MPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, High Efficiency Video Coding (HEVC), VC-1, VP8, VP9, ​​and Open Media Alliance Video 1 (AV1)).

[0087] The second image 135 corresponding to the first image 115 may be reconstructed by performing the first decoding 130 on the image data. The first decoding 130 may include a process of generating quantized residual data by entropy decoding the image data, a process of inversely quantizing the quantized residual data, a process of transforming the residual data of the frequency domain component into the spatial domain component, a process of generating prediction data, and a process of reconstructing the second image 135 by using the prediction data and the residual data. Such first decoding 130 may be performed by an image reconstruction method corresponding to one of the image compression methods using frequency transformation used in the first encoding 120, such as MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1.

[0088] The AI ​​encoded data obtained through the AI ​​encoding process may include image data obtained as a result of performing the first encoding 120 on the first image 115, and AI data related to the AI ​​reduction 110 of the original image 105. The image data may be used during the first decoding 130, and the AI ​​data may be used during the AI ​​enlargement 140.

[0089] The image data may be transmitted in the form of a bitstream. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data that is a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding 120 performed on the first image 115. For example, the image data may include prediction mode information, motion information, and information related to a quantization parameter used during the first encoding 120. The image data may be generated according to a rule (e.g., according to a syntax) of an image compression method in MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 used during the first encoding 120.

[0090] The AI ​​data is used in the AI ​​upscaling 140 based on the second DNN. As described above, because the first DNN and the second DNN are jointly trained, the AI ​​data includes information that enables the AI ​​upscaling 140 to be accurately performed on the second image 135 by the second DNN. During the AI ​​decoding process, the AI ​​upscaling 140 can be performed on the second image 135 based on the AI ​​data to have a target resolution and / or quality.

[0091] The AI ​​data may be transmitted together with the image data in the form of a bit stream. Alternatively, according to an embodiment, the AI ​​data may be transmitted separately from the image data in the form of a frame or a packet. The AI ​​data and the image data obtained as a result of the AI ​​encoding may be transmitted over the same network or over different networks.

[0092] Figure 2 is a block diagram of a configuration of the AI ​​decoding apparatus 200 according to an embodiment.

[0093] Reference Figure 2 , the AI ​​decoding device 200 according to the embodiment may include a receiver 210 and an AI decoder 230. The receiver 210 may include a communication interface 212, a parser 214, and an output interface 216. The AI ​​decoder 230 may include a first decoder 232 and an AI amplifier 234.

[0094] The receiver 210 receives and parses AI-encoded data obtained as a result of AI encoding, and outputs the image data and the AI ​​data distinguishably to the AI ​​decoder 230 .

[0095] The communication interface 212 receives AI-encoded data obtained as a result of AI encoding through a network. The AI-encoded data obtained as a result of performing AI encoding includes image data and AI data. The image data and AI data may be received through the same type of network or different types of networks.

[0096] The parser 214 receives the AI ​​coded data received through the communication interface 212, and parses the AI ​​coded data to distinguish between the image data and the AI ​​data. For example, the parser 214 can distinguish between the image data and the AI ​​data by reading the header of the data obtained from the communication interface 212. According to an embodiment, the parser 214 sends the image data and the AI ​​data to the output interface 216 distinguishably by means of the header of the data received through the communication interface 212, and the output interface 216 sends the distinguished image data and the AI ​​data to the first decoder 232 and the AI ​​amplifier 234, respectively. At this time, it can be verified that the image data included in the AI ​​coded data is image data generated by a codec (e.g., MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1). In this case, the corresponding information can be sent to the first decoder 232 through the output interface 216 so that the image data is processed by the verified codec.

[0097] According to an embodiment, the AI ​​encoded data parsed by the parser 214 may be obtained from a data storage medium including a magnetic medium such as a hard disk, a floppy disk or a tape, an optical recording medium such as a CD-ROM or a DVD, or a magneto-optical medium such as a magneto-optical floppy disk.

[0098] The first decoder 232 reconstructs a second image 135 corresponding to the first image 115 based on the image data. The second image 135 obtained by the first decoder 232 is provided to the AI ​​amplifier 234. According to an embodiment, the first decoding related information (such as prediction mode information, motion information, quantization parameter information, etc. included in the image data) may be further provided to the AI ​​amplifier 234.

[0099] Upon receiving the AI ​​data, the AI ​​amplifier 234 performs AI up-amplification on the second image 135 based on the AI ​​data. According to an embodiment, the AI ​​up-amplification may be performed by further using first decoding related information such as prediction mode information, quantization parameter information, etc. included in the image data.

[0100] The receiver 210 and the AI ​​decoder 230 according to the embodiment are described as separate devices, but can be implemented by one processor. In this case, the receiver 210 and the AI ​​decoder 230 can be implemented by a dedicated processor or by a combination of software and a general-purpose processor (such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU)). The dedicated processor can be implemented by including a memory for implementing the embodiments of the present disclosure or by including a memory processor for using an external memory.

[0101] In addition, the receiver 210 and the AI ​​decoder 230 may be configured by a plurality of processors. In this case, the receiver 210 and the AI ​​decoder 230 may be implemented by a combination of dedicated processors or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a GPU). Similarly, the AI ​​amplifier 234 and the first decoder 232 may be implemented by different processors.

[0102] The AI ​​data provided to the AI ​​amplifier 234 includes information that enables the second image 135 to be processed by AI enlargement. Here, the enlargement target should correspond to the reduction target of the first DNN. Therefore, the AI ​​data includes information for verifying the reduction target of the first DNN.

[0103] Examples of the information included in the AI ​​data include difference information between the resolution of the original image 105 and the resolution of the first image 115 and information related to the first image 115 .

[0104] The difference information may be expressed as information about the degree of resolution conversion of the first image 115 compared to the original image 105 (e.g., resolution conversion rate information). In addition, since the resolution of the first image 115 is verified by the resolution of the reconstructed second image 135, and the degree of resolution conversion is verified accordingly, the difference information may be expressed as only the resolution information of the original image 105. Here, the resolution information may be expressed as a vertical / horizontal size or as a ratio (16:9, 4:3, etc.) and a size of one axis. In addition, when there is pre-set resolution information, the resolution information may be expressed in the form of an index or a flag.

[0105] The information related to the first image 115 may include information about one or both of a bit rate of image data obtained as a result of performing the first encoding on the first image 115 and a codec type used during the first encoding of the first image 115 .

[0106] The AI ​​amplifier 234 may determine an enlargement target of the second image 135 based on one or both of the difference information included in the AI ​​data and the information related to the first image 115. The enlargement target may indicate, for example, to what extent the resolution is to be enlarged for the second image 135. When the enlargement target is determined, the AI ​​amplifier 234 performs AI enlargement on the second image 135 through the second DNN to obtain a third image 145 corresponding to the enlargement target.

[0107] Before describing the method of performing AI up-scaling on the second image 135 according to the up-scaling target performed by the AI ​​amplifier 234, reference will be made to Figure 3 and Figure 4 Describe the AI ​​amplification process through the second DNN.

[0108] Figure 3 is a diagram illustrating a second DNN 300 for performing AI upscaling on a second image 135, and Figure 4 Is used to describe Figure 3 310 of the first convolutional layer 310.

[0109] like Figure 3 As shown, the second image 135 is input to the first convolutional layer 310. Figure 3 The 3×3×4 indicated in the illustrated first convolution layer 310 indicates that a convolution process is performed on one input image by using four filter kernels having a size of 3×3. Four feature maps are generated as a result of the convolution process by the four filter kernels. Each feature map indicates an inherent feature of the second image 135. For example, each feature map may represent a vertical direction feature, a horizontal direction feature, an edge feature, etc. of the second image 135.

[0110] Will refer to Figure 4The convolution operation in the first convolution layer 310 is described in detail.

[0111] One feature map 450 may be generated by multiplication and addition between parameters of a filter kernel 430 having a size of 3×3 used in the first convolution layer 310 and corresponding pixel values ​​in the second image 135. Four filter kernels are used in the first convolution layer 310, and four feature maps may be generated by convolution operations using the four filter kernels.

[0112] exist Figure 4 I1 to I49 indicated in the second image 135 in FIG. 4 indicate pixels in the second image 135, and F1 to F9 indicated in the filter kernel 430 indicate parameters of the filter kernel 430. In addition, M1 to M9 indicated in the feature map 450 indicate samples of the feature map 450.

[0113] exist Figure 4 , the second image 135 includes 49 pixels, but the number of pixels is only an example, and when the second image 135 has a resolution of 4K, the second image 135 may include, for example, 3840×2160 pixels.

[0114] During the convolution operation process, the pixel values ​​of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination (e.g., addition) of the resultant values ​​of the multiplication may be assigned as the value of M1 of the feature map 450. When the step size of the convolution operation is 2, the pixel values ​​of I3, I4, I5, I10, I11, I12, I17, I18, and I19 of the second image 135 are multiplied by F1 to F9 of the filter kernel 430, respectively, and the value of the combination of the resultant values ​​of the multiplication may be assigned as the value of M2 of the feature map 450.

[0115] While the filter kernel 430 moves toward the last pixel of the second image 135 along the step size, a convolution operation is performed between the pixel values ​​in the second image 135 and the parameters of the filter kernel 430, and thus a feature map 450 having a specific size may be generated.

[0116] According to the present disclosure, the values ​​of the parameters of the second DNN, for example, the values ​​of the parameters of the filter kernel used in the convolution layer of the second DNN (for example, F1 to F9 of the filter kernel 430), can be optimized by joint training of the first DNN and the second DNN. As described above, the AI ​​amplifier 234 can determine the enlargement target corresponding to the reduction target of the first DNN based on the AI ​​data, and determine the parameters corresponding to the determined enlargement target as the parameters of the filter kernel used in the convolution layer of the second DNN.

[0117] The convolutional layers included in the first DNN and the second DNN can be constructed according to the reference Figure 4 The convolution operation described in this section is used to perform the processing, but refer to Figure 4 The convolution operation process described is only an example and is not limited to this.

[0118] Return to reference Figure 3 , the feature map output from the first convolution layer 310 may be input to the first activation layer 320 .

[0119] The first activation layer 320 may assign nonlinear features to each feature map. The first activation layer 320 may include a sigmoid function, a Tanh function, a rectified linear unit (ReLU) function, etc., but is not limited thereto.

[0120] The first activation layer 320 to which the nonlinear feature is assigned indicates that at least one sample value of the feature map, which is the output of the first convolutional layer 310, is changed. Here, the change is performed by applying the nonlinear feature.

[0121] The first activation layer 320 determines whether to send sample values ​​of the feature map output from the first convolution layer 310 to the second convolution layer 330. For example, some of the sample values ​​of the feature map are activated by the first activation layer 320 and sent to the second convolution layer 330, and some of the sample values ​​are deactivated by the first activation layer 320 and are not sent to the second convolution layer 330. Intrinsic features of the second image 135 represented by the feature map are highlighted by the first activation layer 320.

[0122] The feature map 325 output from the first activation layer 320 is input to the second convolutional layer 330 . Figure 3 One of the characteristic graphs 325 shown in FIG. 3 is a graph in the first activation layer 320 referring to FIG. Figure 4 The result of processing feature map 450 is described.

[0123] 3×3×4 indicated in the second convolution layer 330 indicates that a convolution process is performed on the feature map 325 by using four filter kernels having a size of 3×3. The output of the second convolution layer 330 is input to the second activation layer 340. The second activation layer 340 may assign nonlinear features to input data.

[0124] The feature map 345 output from the second activation layer 340 is input to the third convolutional layer 350 . Figure 3 3×3×1 indicated in the third convolution layer 350 shown in indicates that a convolution process is performed by using one filter kernel having a size of 3×3 to generate one output image. The third convolution layer 350 is a layer for outputting a final image, and generates one output by using one filter kernel. According to an embodiment of the present disclosure, the third convolution layer 350 may output the third image 145 as a result of the convolution operation.

[0125] As will be described later, there may be a plurality of pieces of DNN setting information indicating the number of filter kernels of the first convolutional layer 310, the second convolutional layer 330, and the third convolutional layer 350 of the second DNN 300, parameters of the filter kernels of the first convolutional layer 310, the second convolutional layer 330, and the third convolutional layer 350 of the second DNN 300, and the plurality of pieces of DNN setting information may be linked to the plurality of pieces of DNN setting information of the first DNN. The link between the plurality of pieces of DNN setting information of the second DNN and the plurality of DNN setting information of the first DNN may be achieved via joint training of the first DNN and the second DNN.

[0126] exist Figure 3 , the second DNN 300 includes three convolutional layers (a first convolutional layer 310, a second convolutional layer 330, and a third convolutional layer 350) and two activation layers (a first activation layer 320 and a second activation layer 340), but this is only an example, and the number of convolutional layers and activation layers may vary according to the embodiment. In addition, according to the embodiment, the second DNN 300 may be implemented as a recurrent neural network (RNN). In this case, the convolutional neural network (CNN) structure of the second DNN 300 according to the embodiment of the present disclosure is changed to an RNN structure.

[0127] According to an embodiment, the AI ​​amplifier 234 may include at least one arithmetic logic unit (ALU) for the convolution operation and the operation of the activation layer described above. The ALU may be implemented as a processor. For the convolution operation, the ALU may include a multiplier and an adder, wherein the multiplier performs multiplication between the sample value of the second image 135 or feature map output from the previous layer and the sample value of the filter kernel, and the adder adds the result value of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0128] Hereinafter, a method of performing AI up-scaling on the second image 135 according to the up-scaling target, performed by the AI ​​amplifier 234, will be described.

[0129] According to an embodiment, the AI ​​amplifier 234 may store a plurality of pieces of DNN setting information that may be set in the second DNN.

[0130] Here, the DNN setting information may include information about any one or any combination of the number of convolutional layers included in the second DNN, the number of filter kernels of each convolutional layer, and the parameters of each filter kernel. Multiple DNN setting information may correspond to various amplification targets, respectively, and the second DNN may operate based on the DNN setting information corresponding to the amplification target. The second DNN may have different structures based on the DNN setting information. For example, the second DNN may include three convolutional layers based on any one DNN setting information, and may include four convolutional layers based on another DNN setting information.

[0131] According to an embodiment, the DNN setting information may include only the parameters of the filter kernel used in the second DNN. In this case, the structure of the second DNN does not change, but only the parameters of the internal filter kernel may be changed based on the DNN setting information.

[0132] The AI ​​amplifier 234 may obtain DNN setting information from among the plurality of DNN setting information for performing AI amplification on the second image 135. Each of the plurality of DNN setting information used at this time is information for obtaining the third image 145 of a predetermined resolution and / or a predetermined quality, and is jointly trained with the first DNN.

[0133] For example, one piece of DNN setting information among the multiple DNN setting information may include information for obtaining a third image 145 having a resolution that is twice the resolution of the second image 135 (for example, 4K (4096×2160) of the third image 145 is twice the 2K (2048×1080) of the second image 135), and another piece of DNN setting information may include information for obtaining a third image 145 having a resolution that is four times the resolution of the second image 135 (for example, 8K (8192×4320) of the third image 145 is four times the 2K (2048×1080) of the second image 135).

[0134] Each of the multiple DNN setting information is Figure 6 The AI ​​encoding device 600 of the embodiment obtains the DNN setting information of the first DNN together, and the AI ​​amplifier 234 obtains one DNN setting information of the multiple DNN setting information according to the enlargement ratio corresponding to the reduction ratio of the DNN setting information of the first DNN. In this regard, the AI ​​amplifier 234 can verify the information of the first DNN. In order for the AI ​​amplifier 234 to verify the information of the first DNN, the AI ​​decoding device 200 according to the embodiment receives AI data including the information of the first DNN from the AI ​​encoding device 600.

[0135] In other words, the AI ​​amplifier 234 may verify information targeted by the DNN setting information of the first DNN used to obtain the first image 115 by using information received from the AI ​​encoding device 600, and obtain DNN setting information of the second DNN jointly trained with the DNN setting information of the first DNN.

[0136] When DNN setting information for performing AI up-scaling on the second image 135 is obtained from a plurality of pieces of DNN setting information, input data may be processed based on a second DNN operated according to the obtained DNN setting information.

[0137] For example, when obtaining any DNN setting information, including Figure 3 The number of filter kernels in each of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300 and the parameters of the filter kernels are set to the values ​​included in the obtained DNN setting information.

[0138] exist Figure 4 The parameters of the 3×3 filter kernel used in any one convolutional layer of the second DNN are set to {1,1,1,1,1,1,1,1,1} and when the DNN setting information is changed later, the parameters are replaced by {2,2,2,2,2,2,2,2,2} as parameters included in the changed DNN setting information.

[0139] The AI ​​amplifier 234 may obtain DNN setting information for AI amplification from a plurality of pieces of DNN setting information based on information included in the AI ​​data, and the AI ​​data for obtaining the DNN setting information will now be described.

[0140] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification from a plurality of pieces of DNN setting information based on the difference information included in the AI ​​data. For example, when it is verified based on the difference information that the resolution of the original image 105 (e.g., 4K (4096×2160)) is twice the resolution of the first image 115 (e.g., 2K (2048×1080)), the AI ​​amplifier 234 may obtain DNN setting information for doubling the resolution of the second image 135.

[0141] According to an embodiment, the AI ​​amplifier 234 may obtain DNN setting information for AI amplification of the second image 135 from a plurality of pieces of DNN setting information based on information related to the first image 115 included in the AI ​​data. The AI ​​amplifier 234 may predetermine a mapping relationship between image-related information and DNN setting information, and obtain DNN setting information mapped to information related to the first image 115.

[0142] Figure 5 is a table showing a mapping relationship between multiple pieces of image-related information and multiple pieces of DNN setting information.

[0143] By following Figure 5 In the embodiment of the present disclosure, it will be determined that the AI ​​encoding and AI decoding processes according to the embodiment of the present disclosure do not only consider the change in resolution. Figure 5 As shown, the DNN setting information can be selected by considering resolution such as standard definition (SD), high definition (HD), or full HD, bit rate such as 10Mbps, 15Mbps, or 20Mbps, and codec information such as AV1, H.264, or HEVC, individually or collectively. For such consideration of resolution, bit rate, and codec information, the training of each element can be performed jointly with the encoding and decoding process during the AI ​​training process (see Fig. 9 ).

[0144] Therefore, when a plurality of pieces of DNN setting information are provided based on image-related information including a codec type, a resolution of an image, etc., such as Figure 5 As shown, according to the training, DNN setting information for AI upscaling the second image 135 can be obtained based on the information related to the first image 115 received during the AI ​​decoding process.

[0145] In other words, AI amplifier 234 is able to Figure 5 The image related information on the left side of the table and the DNN setting information on the right side of the table are matched to use the DNN setting information according to the image related information.

[0146] like Figure 5 As shown, when it is verified from information related to the first image 115 that the resolution of the first image 115 is SD, the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 10Mbps, and the first encoding is performed on the first image 115 through the AV1 codec, the AI ​​amplifier 234 can use the ADNN setting information among multiple DNN setting information.

[0147] In addition, when it is verified from information related to the first image 115 that the resolution of the first image 115 is HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the H.264 codec, the AI ​​amplifier 234 can use B DNN setting information among multiple DNN setting information.

[0148] In addition, when it is verified from the information related to the first image 115 that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 20Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 can use the C DNN setting information among the multiple DNN setting information, and when it is verified that the resolution of the first image 115 is full HD, the bit rate of the image data obtained as a result of performing the first encoding is 15Mbps, and the first encoding is performed via the HEVC codec, the AI ​​amplifier 234 can use the D DNN setting information among the multiple DNN setting information. One of the C DNN setting information and the D DNN setting information is selected based on whether the bit rate of the image data obtained as a result of performing the first encoding on the first image 115 is 20Mbps or 15Mbps. Different bit rates of image data obtained when performing the first encoding on the first image 115 of the same resolution via the same codec indicate reconstructed images of different qualities. Therefore, the first DNN and the second DNN can be jointly trained based on the image quality, and therefore, the AI ​​amplifier 234 can obtain the DNN setting information according to the bit rate of the image data indicating the quality of the second image 135.

[0149] According to an embodiment, the AI ​​amplifier 234 may consider information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and information related to the first image 115 included in the AI ​​data, and obtain DNN setting information for performing AI amplification on the second image 135 from a plurality of DNN setting information. For example, the AI ​​amplifier 234 may receive quantization parameter information used during the first encoding process of the first image 115 from the first decoder 232, verify the bit rate of image data obtained as a result of encoding the first image 115 from the AI ​​data, and obtain DNN setting information corresponding to the quantization parameter information and the bit rate. Even when the bit rate is the same, the quality of the reconstructed image may vary depending on the complexity of the image. The bit rate is a value representing the entire first image 115 on which the first encoding is performed, and even within the first image 115, the quality of each frame may vary. Therefore, when the prediction mode information, motion information, and / or quantization parameter obtainable for each frame from the first decoder 232 are considered together, DNN setting information more suitable for the second image 135 may be obtained compared to when only the AI ​​data is used.

[0150] In addition, according to an embodiment, the AI ​​data may include an identifier of the mutually agreed DNN setting information. The identifier of the DNN setting information is information for distinguishing a pair of DNN setting information jointly trained between the first DNN and the second DNN, so that AI enlargement is performed on the second image 135 to an enlargement target corresponding to the reduction target of the first DNN. After obtaining the identifier of the DNN setting information included in the AI ​​data, the AI ​​amplifier 234 may perform AI enlargement on the second image 135 by using the DNN setting information corresponding to the identifier of the DNN setting information. For example, an identifier indicating each of the multiple DNN setting information that can be set in the first DNN and an identifier indicating each of the multiple DNN setting information that can be set in the second DNN may be pre-specified. In this case, the same identifier may be specified for a pair of DNN setting information that can be set in each of the first DNN and the second DNN. The AI ​​data may include an identifier of the DNN setting information set in the first DNN for AI reduction of the original image 105. The AI ​​amplifier 234 receiving the AI ​​data may perform AI enlargement on the second image 135 by using the DNN setting information indicated by the identifier included in the AI ​​data among the multiple DNN setting information.

[0151] In addition, according to an embodiment, the AI ​​data may include DNN setting information. The AI ​​amplifier 234 may perform AI amplification on the second image 135 by using the DNN setting information after obtaining the DNN setting information included in the AI ​​data.

[0152] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter kernels of each convolutional layer, the parameters of each filter kernel, etc.) are stored in the form of a lookup table, the AI ​​amplifier 234 can obtain the DNN setting information by combining some values ​​selected from the values ​​in the lookup table based on the information included in the AI ​​data, and perform AI amplification on the second image 135 by using the obtained DNN setting information.

[0153] According to an embodiment, when the structure of the DNN corresponding to the amplification target is determined, the AI ​​amplifier 234 may obtain DNN setting information corresponding to the determined structure of the DNN, for example, parameters of a filter kernel.

[0154] The AI ​​amplifier 234 obtains DNN setting information of the second DNN through AI data including information related to the first DNN, and performs AI amplification on the second image 135 through the second DNN set based on the obtained DNN setting information, and in this case, memory usage and throughput can be reduced compared to directly analyzing features of the second image 135 for amplification.

[0155] According to an embodiment, when the second image 135 includes a plurality of frames, the AI ​​amplifier 234 may independently obtain DNN setting information for a specific number of frames, or may obtain common DNN setting information for all frames.

[0156] Figure 6 is a diagram illustrating a second image 135 including a plurality of frames.

[0157] like Figure 6 As shown in , the second image 135 may include frames t0 to tn.

[0158] According to an embodiment, the AI ​​amplifier 234 may obtain the DNN setting information of the second DNN through the AI ​​data, and perform AI amplification on the frames t0 to tn based on the obtained DNN setting information. In other words, the frames t0 to tn may be processed through AI amplification based on the common DNN setting information.

[0159] According to an embodiment, the AI ​​amplifier 234 may perform AI amplification on some of the frames t0 to tn (e.g., frames t0 to ta) by using the "A" DNN setting information obtained from the AI ​​data, and perform AI amplification on frames ta+1 to tb by using the "B" DNN setting information obtained from the AI ​​data. In addition, the AI ​​amplifier 234 may perform AI amplification on frames tb+1 to tn by using the "C" DNN setting information obtained from the AI ​​data. In other words, the AI ​​amplifier 234 may independently obtain DNN setting information for each group including some of the plurality of frames, and perform AI amplification on the frames included in each group by using the independently obtained DNN setting information.

[0160] According to an embodiment, the AI ​​amplifier 234 may independently obtain DNN setting information for each frame forming the second image 135. In other words, when the second image 135 includes three frames, the AI ​​amplifier 234 may perform AI amplification on the first frame by using the DNN setting information obtained about the first frame, perform AI amplification on the second frame by using the DNN setting information obtained about the second frame, and perform AI amplification on the third frame by using the DNN setting information obtained about the third frame. According to a method of obtaining DNN setting information based on information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 232 and information related to the first image 115 included in the above-mentioned AI data, DNN setting information may be independently obtained for each frame included in the second image 135. This is because mode information, quantization parameter information, etc. may be independently determined for each frame included in the second image 135.

[0161] According to an embodiment, the AI ​​data may include information about which frame the DNN setting information obtained based on the AI ​​data is valid for. For example, when the AI ​​data includes information indicating that the DNN setting information is valid until frame ta, the AI ​​amplifier 234 performs AI amplification on frame t0 to frame ta by using the DNN setting information obtained based on the AI ​​data. In addition, when another piece of AI data includes information indicating that the DNN setting information is valid until frame tn, the AI ​​amplifier 234 performs AI amplification on frame ta+1 to frame tn by using the DNN setting information obtained based on the other piece of AI data.

[0162] In the following, reference will be made to Figure 7 An AI encoding apparatus 600 for performing AI encoding on an original image 105 is described.

[0163] Figure 7 is a block diagram of a configuration of an AI encoding apparatus 600 according to an embodiment.

[0164] Reference Figure 7 , the AI ​​encoding device 600 may include an AI encoder 610 and a transmitter 630. The AI ​​encoder 610 may include an AI reducer 612 and a first encoder 614. The transmitter 630 may include a data processor 632 and a communication interface 634.

[0165] exist Figure 7 , the AI ​​encoder 610 and the transmitter 630 are shown as separate devices, but the AI ​​encoder 610 and the transmitter 630 may be implemented by one processor. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor (such as an AP, a CPU, or a graphics processing unit GPU). The dedicated processor may be implemented by including a memory for implementing an embodiment of the present disclosure or by including a memory processor for using an external memory.

[0166] In addition, the AI ​​encoder 610 and the transmitter 630 may be configured by a plurality of processors. In this case, the AI ​​encoder 610 and the transmitter 630 may be implemented by a combination of dedicated processors or by a combination of software and a plurality of general-purpose processors (such as an AP, a CPU, or a GPU). The AI ​​reducer 612 and the first encoder 614 may be implemented by different processors.

[0167] The AI ​​encoder 610 performs AI reduction on the original image 105 and first encoding on the first image 115, and transmits the AI ​​data and the image data to the transmitter 630. The transmitter 630 transmits the AI ​​data and the image data to the AI ​​decoding apparatus 200.

[0168] The image data includes data obtained as a result of performing the first encoding on the first image 115. The image data may include data obtained based on pixel values ​​in the first image 115, for example, residual data that is a difference between the first image 115 and prediction data of the first image 115. In addition, the image data includes information used during the first encoding process of the first image 115. For example, the image data may include prediction mode information, motion information, quantization parameter information, etc., used to perform the first encoding on the first image 115.

[0169] The AI ​​data includes information that enables AI up-scaling to be performed on the second image 135 to an up-scaling target corresponding to the down-scaling target of the first DNN. According to an embodiment, the AI ​​data may include difference information between the original image 105 and the first image 115. In addition, the AI ​​data may include information related to the first image 115. The information related to the first image 115 may include information about any one or any combination of a resolution of the first image 115, a bit rate of image data obtained as a result of performing the first encoding on the first image 115, and a codec type used during the first encoding of the first image 115.

[0170] According to an embodiment, the AI ​​data may include an identifier of mutually agreed DNN setting information so that the AI ​​up-scaling is performed on the second image 135 to a scale-up target corresponding to a scale-down target of the first DNN.

[0171] In addition, according to an embodiment, the AI ​​data may include DNN setting information that can be set in the second DNN.

[0172] The AI ​​downsizer 612 may obtain the first image 115 obtained by performing AI downsizing on the original image 105 via the first DNN. The AI ​​downsizer 612 may determine a downsizing target of the original image 105 based on a predetermined criterion.

[0173] In order to obtain the first image 115 matching the reduction target, the AI ​​reducer 612 may store a plurality of pieces of DNN setting information that can be set in the first DNN. The AI ​​reducer 612 obtains DNN setting information corresponding to the reduction target from the plurality of pieces of DNN setting information, and performs AI reduction on the original image 105 through the first DNN set by the obtained DNN setting information.

[0174] Each of the plurality of pieces of DNN setting information may be trained to obtain a first image 115 of a predetermined resolution and / or a predetermined quality. For example, any one of the plurality of pieces of DNN setting information may include information for obtaining a first image 115 having a resolution half that of the original image 105 (e.g., 2k (2048×1080) of the first image 115 is half that of 4k (4096×2160) of the original image 105), and another piece of DNN setting information may include information for obtaining a first image 115 having a resolution one-quarter that of the original image 105 (e.g., 2k (2048×1080) of the first image 115 is one-quarter that of 8k (8192×4320) of the original image 105).

[0175] According to an embodiment, when multiple pieces of information constituting DNN setting information (for example, the number of convolutional layers, the number of filter kernels of each convolutional layer, the parameters of each filter kernel, etc.) are stored in the form of a lookup table, the AI ​​downsizer 612 can obtain the DNN setting information by selecting some values ​​from the values ​​in the lookup table based on the reduction target combination, and perform AI downsizing on the original image 105 by using the obtained DNN setting information.

[0176] According to an embodiment, the AI ​​downsizer 612 may determine the structure of the DNN corresponding to the downsizing target, and obtain DNN setting information corresponding to the determined structure of the DNN, for example, obtain parameters of a filter kernel.

[0177] When the first DNN and the second DNN are jointly trained, a plurality of pieces of DNN setting information for performing AI reduction on the original image 105 may have an optimized value. Here, each piece of DNN setting information includes any one or any combination of the number of convolutional layers included in the first DNN, the number of filter kernels of each convolutional layer, and the parameters of each filter kernel.

[0178] The AI ​​downsizer 612 may set the first DNN using the DNN setting information obtained to perform AI downsizing on the original image 105, so as to obtain the first image 115 of a specific resolution and / or a specific quality through the first DNN. When the DNN setting information for performing AI downsizing on the original image 105 is obtained from a plurality of pieces of DNN setting information, each layer in the first DNN may process input data based on information included in the DNN setting information.

[0179] Hereinafter, a method of determining a downscaling target performed by the AI ​​downscaler 612 will be described. The downscaling target may indicate, for example, how much the resolution is reduced from the original image 105 to obtain the first image 115.

[0180] According to an embodiment, the AI ​​downsizer 612 can determine the downsizing target based on any one or any combination of the compression ratio (e.g., the resolution difference between the original image 105 and the first image 115, the target bit rate, etc.), the compression quality (e.g., the type of bit rate), the compression history information, and the type of the original image 105.

[0181] For example, the AI ​​downsizer 612 may determine a downsizing target based on a compression ratio, compression quality, etc. that is preset or input from a user.

[0182] As another example, the AI ​​downsizer 612 may determine the downsizing target by using the compression history information stored in the AI ​​encoding device 600. For example, based on the compression history information available by the AI ​​encoding device 600, the encoding quality, compression ratio, etc. preferred by the user may be determined, and the downsizing target may be determined based on the encoding quality determined based on the compression history information. For example, the resolution, quality, etc. of the first image 115 may be determined based on the most frequently used encoding quality based on the compression history information.

[0183] As another example, the AI ​​downsizer 612 may determine the downsizing target based on encoding qualities that have been used more frequently than a certain threshold (eg, an average quality of encoding qualities that have been used more frequently than the threshold) according to the compression history information.

[0184] As another example, the AI ​​downsizer 612 may determine the downsizing target based on the resolution, type (eg, file format), etc. of the original image 105 .

[0185] According to an embodiment, when the original image 105 includes a plurality of frames, the AI ​​downsizer 612 may independently determine downsizing targets for a specific number of frames, or may determine downsizing targets for all frames.

[0186] According to an embodiment, the AI ​​downsizer 612 may divide the frames included in the original image 105 into a certain number of groups and independently determine a downsizing target for each group. The same or different downsizing targets may be determined for each group. The number of frames included in the group may be the same or different according to each group.

[0187] According to an embodiment, the AI ​​downsizer 612 may independently determine a downscaling target for each frame included in the original image 105. The same or different downscaling targets may be determined for each frame.

[0188] Hereinafter, an example of the structure of the first DNN 700 on which the AI ​​reduction is based will be described.

[0189] Figure 8 is a diagram illustrating a first DNN 700 for performing AI downscaling on an original image 105.

[0190] like Figure 8 As shown, the original image 105 is input to the first convolution layer 710. The first convolution layer 710 performs a convolution process on the original image 105 by using 32 filter kernels having a size of 5×5. The 32 feature maps generated as a result of the convolution process are input to the first activation layer 720. The first activation layer 720 can assign nonlinear features to the 32 feature maps.

[0191] The first activation layer 720 determines whether to transmit sample values ​​of the feature map output from the first convolution layer 710 to the second convolution layer 730. For example, some of the sample values ​​of the feature map are activated by the first activation layer 720 and transmitted to the second convolution layer 730, and some of the sample values ​​are deactivated by the first activation layer 720 and are not transmitted to the second convolution layer 730. Information represented by the feature map output from the first convolution layer 710 is highlighted by the first activation layer 720.

[0192] The output 725 of the first activation layer 720 is input to the second convolution layer 730. The second convolution layer 730 performs a convolution process on the input data by using 32 filter kernels having a size of 5×5. The 32 feature maps output as a result of the convolution process are input to the second activation layer 740, and the second activation layer 740 can assign nonlinear features to the 32 feature maps.

[0193] The output 745 of the second activation layer 740 is input to the third convolution layer 750. The third convolution layer 750 performs a convolution process on the input data by using a filter kernel having a size of 5×5. As a result of the convolution process, an image may be output from the third convolution layer 750. The third convolution layer 750 generates an output by using a filter kernel as a layer for outputting a final image. According to an embodiment of the present disclosure, the third convolution layer 750 may output the first image 115 as a result of the convolution operation.

[0194] There may be a plurality of pieces of DNN setting information indicating the number of filter kernels of the first convolutional layer 710, the second convolutional layer 730, and the third convolutional layer 750 of the first DNN 700, the parameters of each filter kernel of the first convolutional layer 710, the second convolutional layer 730, and the third convolutional layer 750 of the first DNN 700, and the plurality of pieces of DNN setting information may be linked to the plurality of pieces of DNN setting information of the second DNN. The link between the plurality of pieces of DNN setting information of the first DNN and the plurality of DNN setting information of the second DNN may be achieved via joint training of the first DNN and the second DNN.

[0195] exist Figure 8, the first DNN 700 includes three convolutional layers (a first convolutional layer 710, a second convolutional layer 730, and a third convolutional layer 750) and two activation layers (a first activation layer 720 and a second activation layer 740), but this is only an example, and the number of convolutional layers and activation layers may vary according to the embodiment. In addition, according to the embodiment, the first DNN 700 may be implemented as an RNN. In this case, the CNN structure of the first DNN 700 according to the embodiment of the present disclosure is changed to an RNN structure.

[0196] According to an embodiment, the AI ​​reducer 612 may include at least one ALU for convolution operations and operations of the above-mentioned activation layers. The ALU may be implemented as a processor. For convolution operations, the ALU may include a multiplier and an adder, wherein the multiplier performs multiplication between the sample values ​​of the original image 105 or the feature map output from the previous layer and the sample values ​​of the filter kernel, and the adder adds the result values ​​of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by the weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and the comparator compares the multiplication result with a specific value to determine whether to send the input sample value to the next layer.

[0197] Return to reference Figure 7 , after receiving the first image 115 from the AI ​​downsizer 612, the first encoder 614 may reduce the amount of information of the first image 115 by performing first encoding on the first image 115. The first encoder 614 may obtain image data corresponding to the first image 115 as a result of performing the first encoding.

[0198] The data processor 632 processes one or both of the AI ​​data or the image data to be sent in a specific form. For example, when the AI ​​data and the image data are to be sent in the form of a bit stream, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and send the image data and the AI ​​data in the form of one bit stream through the communication interface 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a bit stream, and send each of the bit streams corresponding to the AI ​​data and the bit stream corresponding to the image data through the communication interface 634. As another example, the data processor 632 may process the AI ​​data to be represented in the form of a frame or a packet, and send the image data in the form of a bit stream and send the AI ​​data in the form of a frame or a packet through the communication interface 634.

[0199] The communication interface 634 transmits the AI-encoded data obtained as a result of performing the AI ​​encoding through the network. The AI-encoded data obtained as a result of performing the AI ​​encoding includes image data and AI data. The image data and AI data may be transmitted through the same type of network or different types of networks.

[0200] According to an embodiment, the AI ​​encoded data obtained as a result of processing by the data processor 632 can be stored in a data storage medium including a magnetic medium such as a hard disk, floppy disk or tape, an optical recording medium such as a CD-ROM or DVD, or a magneto-optical medium such as a floppy disk.

[0201] In the following, reference will be made to Fig. 9 A method of jointly training the first DNN 700 and the second DNN 300 is described.

[0202] Fig. 9 is a diagram for describing a method of training a first DNN 700 and a second DNN 300.

[0203] In an embodiment, the original image 105 for which AI encoding is performed is reconstructed into the third image 145 through the AI ​​decoding process, and in order to maintain the similarity between the original image 105 and the third image 145 obtained as a result of AI decoding, a connection is established between the AI ​​encoding process and the AI ​​decoding process. In other words, the information lost in the AI ​​encoding process is reconstructed during the AI ​​decoding process, and in this regard, the first DNN 700 and the second DNN 300 are jointly trained.

[0204] For accurate AI decoding, ultimately, the reduction Fig. 9 The quality loss information 830 corresponding to the result of comparing the third training image 804 with the original training image 801 is shown. Therefore, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0205] First, we will describe Fig. 9 The training process is shown.

[0206] exist Fig. 9 , the original training image 801 is an image to be subjected to AI reduction, and the first training image 802 is an image obtained by performing AI reduction on the original training image 801. In addition, the third training image 804 is an image obtained by performing AI enlargement on the first training image 802.

[0207] The original training image 801 includes a still image or a moving image including a plurality of frames. According to an embodiment, the original training image 801 may include a luminance image extracted from a still image or a moving image including a plurality of frames. In addition, according to an embodiment, the original training image 801 may include a block image extracted from a still image or a moving image including a plurality of frames. When the original training image 801 includes a plurality of frames, the first training image 802, the second training image, and the third training image 804 also each include a plurality of frames. When the plurality of frames of the original training image 801 are sequentially input to the first DNN 700, the plurality of frames of the first training image 802, the second training image, and the third training image 804 may be sequentially obtained by the first DNN 700 and the second DNN 300.

[0208] For joint training of the first DNN 700 and the second DNN 300, an original training image 801 is input to the first DNN 700. The original training image 801 input to the first DNN 700 is output as a first training image 802 through AI downscaling, and the first training image 802 is input to the second DNN 300. A third training image 804 is output as a result of performing AI upscaling on the first training image 802.

[0209] Reference Fig. 9 , the first training image 802 is input to the second DNN 300, and according to an embodiment, the second training image obtained by performing the first encoding and the first decoding on the first training image 802 may be input to the second DNN 300. In order to input the second training image to the second DNN 300, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 codecs may be used. Any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, ​​and AV1 codecs may be used to perform the first encoding on the first training image 802, and perform the first decoding on the image data corresponding to the first training image 802.

[0210] Reference Fig. 9 , separate from the first training image 802 output by the first DNN 700, a reduced training image 803 obtained by performing conventional downscaling on the original training image 801 is obtained. Here, conventional downscaling may include any one or any combination of bilinear scaling, bicubic scaling, Lanzos scaling, or step scaling.

[0211] In order to prevent the structural features of the first image 115 from greatly deviating from the structural features of the original image 105 , a reduced training image 803 is obtained to preserve the structural features of the original training image 801 .

[0212] Before performing training, the first DNN 700 and the second DNN 300 may be set to predetermined DNN setting information. When performing training, structure loss information 810, complexity loss information 820, and quality loss information 830 may be determined.

[0213] The structural loss information 810 may be determined based on a result of comparing the reduced training image 803 and the first training image 802. For example, the structural loss information 810 may correspond to a difference between the structural information of the reduced training image 803 and the structural information of the first training image 802. The structural information may include various features that may be extracted from an image, such as brightness, contrast, a histogram, etc. of the image. The structural loss information 810 indicates how much structural information of the original training image 801 is maintained in the first training image 802. When the structural loss information 810 is small, the structural information of the first training image 802 is similar to the structural information of the original training image 801.

[0214] The complexity loss information 820 may be determined based on the spatial complexity of the first training image 802. For example, the total variance value of the first training image 802 may be used as the spatial complexity. The complexity loss information 820 is related to the bit rate of the image data obtained by performing the first encoding on the first training image 802. When the complexity loss information 820 is small, the bit rate of the image data is low.

[0215] The quality loss information 830 may be determined based on a result of comparing the original training image 801 and the third training image 804. The quality loss information 830 may include any one or any combination of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, and a video multi-method assessment fusion (VMAF) value about the difference between the original training image 801 and the third training image 804. The quality loss information 830 indicates how similar the third training image 804 is to the original training image 801. When the quality loss information 830 is small, the third training image 804 is more similar to the original training image 801.

[0216] Reference Fig. 9 , the structure loss information 810, the complexity loss information 820, and the quality loss information 830 are used to train the first DNN 700, and the quality loss information 830 is used to train the second DNN 300. In other words, the quality loss information 830 is used to train both the first DNN 700 and the second DNN 300.

[0217] The first DNN 700 may update parameters so that the final loss information determined based on the structure loss information 810, the complexity loss information 820, and the quality loss information 830 is reduced or minimized. In addition, the second DNN 300 may update parameters so that the quality loss information 830 is reduced or minimized.

[0218] Final loss information for training the first DNN 700 and the second DNN 300 may be determined as Equation 1 below.

[0219] [Equation 1]

[0220] LossDS = a*structural loss information + b*complexity loss information + c*quality loss information

[0221] LossUS=d*quality loss information

[0222] In Equation 1, LossDS indicates final loss information to be reduced or minimized to train the first DNN 700, and LossUS indicates final loss information to be reduced or minimized to train the second DNN 300. In addition, a, b, c, and d may be predetermined weights.

[0223] In other words, the first DNN 700 updates parameters in a direction in which the LossDS of Equation 1 is reduced, and the second DNN 300 updates parameters in a direction in which the LossUS of Equation 1 is reduced. When the parameters of the first DNN 700 are updated according to the LossDS derived during training, the first training image 802 obtained based on the updated parameters becomes different from the previous first training image 802 obtained based on the non-updated parameters, and therefore, the third training image 804 also becomes different from the previous third training image 804. When the third training image 804 becomes different from the previous third training image 804, the quality loss information 830 is also newly determined, and the second DNN 300 updates the parameters accordingly. When the quality loss information 830 is newly determined, the LossDS is also newly determined, and the first DNN 700 updates the parameters according to the newly determined LossDS. In other words, the update of the parameters of the first DNN 700 leads to the update of the parameters of the second DNN 300, and the update of the parameters of the second DNN 300 leads to the update of the parameters of the first DNN 700. In other words, because the first DNN 700 and the second DNN 300 are jointly trained by sharing the quality loss information 830, the parameters of the first DNN 700 and the parameters of the second DNN 300 can be jointly optimized.

[0224] Referring to Equation 1, it is verified that LossUS is determined according to the quality loss information 830 , but this is only an example, and LossUS may be determined based on any one or any combination of the structure loss information 810 and the complexity loss information 820 , and the quality loss information 830 .

[0225] In the above, it has been described that the AI ​​amplifier 234 of the AI ​​decoding device 200 and the AI ​​reducer 612 of the AI ​​encoding device 600 store multiple DNN setting information, and now a method of training each of the multiple DNN setting information stored in the AI ​​amplifier 234 and the AI ​​reducer 612 will be described.

[0226] As described with reference to Equation 1, the first DNN 700 updates parameters based on the similarity between the structural information of the first training image 802 and the structural information of the original training image 801 (structural loss information 810), the bit rate of the image data obtained as a result of performing the first encoding on the first training image 802 (complexity loss information 820), and the difference between the third training image 804 and the original training image 801 (quality loss information 830).

[0227] The parameters of the first DNN 700 can be updated so that a first training image 802 having structural information similar to that of the original training image 801 is obtained, and image data having a small bit rate is obtained when the first encoding is performed on the first training image 802, and at the same time, the second DNN 300 that performs AI amplification on the first training image 802 obtains a third training image 804 similar to the original training image 801.

[0228] The direction along which the parameters of the first DNN 700 are optimized may be changed by adjusting the weights a, b, and c of Equation 1. For example, when the weight b is determined to be high, the parameters of the first DNN 700 may be updated by prioritizing the low bit rate over the high quality of the third training image 804. In addition, when the weight c is determined to be high, the parameters of the first DNN 700 may be updated by prioritizing the high quality of the third training image 804 over the high bit rate or maintaining the structural information of the original training image 801.

[0229] In addition, the direction along which the parameters of the first DNN 700 are optimized may vary depending on the type of codec used to perform the first encoding on the first training image 802. This is because the second training image to be input to the second DNN 300 may vary depending on the type of codec.

[0230] In other words, the parameters of the first DNN 700 and the parameters of the second DNN 300 may be jointly updated based on the weights a, b, and c and the type of codec used to perform the first encoding on the first training image 802. Therefore, when the first DNN 700 and the second DNN 300 are trained after the weights a, b, and c are determined to be specific values, respectively, and the type of codec is determined to be a specific type, the parameters of the first DNN 700 and the parameters of the second DNN 300 that are linked and optimized to each other may be determined.

[0231] In addition, when the first DNN 700 and the second DNN 300 are trained after changing the weights a, b, and c and the type of codec, the parameters of the first DNN 700 and the parameters of the second DNN 300 that are linked and optimized with each other can be determined. In other words, when the first DNN 700 and the second DNN 300 are trained while changing the values ​​of the weights a, b, and c and the type of codec, a plurality of pieces of DNN setting information that are jointly trained with each other can be determined in the first DNN 700 and the second DNN 300.

[0232] As mentioned above Figure 5 As described, a plurality of pieces of DNN setting information of the first DNN 700 and the second DNN 300 may be mapped to information related to the first image. In order to set such a mapping relationship, the first training image 802 output from the first DNN 700 may be first encoded via a specific codec according to a specific bit rate, and the second training image obtained by performing a first decoding on the bit stream obtained as a result of performing the first encoding may be input to the second DNN 300. In other words, by training the first DNN 700 and the second DNN 300 after setting an environment so that the first training image 802 of a specific resolution is first encoded via a specific codec according to a specific bit rate, a pair of DNN setting information mapped to the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained as a result of performing the first encoding on the first training image 802 may be determined. By differently changing the resolution of the first training image 802, the type of codec used to perform the first encoding on the first training image 802, and the bit rate of the bit stream obtained according to the first encoding of the first training image 802, the mapping relationship between multiple DNN setting information of the first DNN 700 and the second DNN 300 and multiple information related to the first image can be determined.

[0233] Fig.10 is a diagram for describing a training process of a first DNN 700 and a second DNN 300 by a training device 1000.

[0234] Reference Fig. 9 The training of the described first DNN 700 and the second DNN 300 can be performed by the training device 1000. The training device 1000 includes the first DNN 700 and the second DNN 300. The training device 1000 may be, for example, the AI ​​encoding device 600 or a separate server. The DNN setting information of the second DNN 300 obtained as a result of the training is stored in the AI ​​decoding device 200.

[0235] Reference Fig.10 In operations S840 and S845, the training device 1000 initially sets the DNN setting information of the first DNN 700 and the second DNN 300. Therefore, the first DNN 700 and the second DNN 300 may operate according to the predetermined DNN setting information. The DNN setting information may include information about any one or any combination of the number of convolutional layers included in the first DNN 700 and the second DNN 300, the number of filter kernels of each convolutional layer, the size of the filter kernel of each convolutional layer, or the parameters of each filter kernel.

[0236] In operation S850, the training apparatus 1000 inputs the original training image 801 into the first DNN 700. The original training image 801 may include a still image or at least one frame included in a moving image.

[0237] In operation S855, the first DNN 700 processes the original training image 801 according to the initially set DNN setting information, and outputs the first training image 802 obtained by performing AI reduction on the original training image 801. Fig.10 , the first training image 802 output from the first DNN 700 is directly input to the second DNN 300, but the first training image 802 output from the first DNN 700 may be input to the second DNN 300 through the training device 1000. In addition, the training device 1000 may perform first encoding and first decoding on the first training image 802 via a specific codec, and then input the second training image to the second DNN 300.

[0238] In operation S860, the second DNN 300 processes the first training image 802 or the second training image according to initially set DNN setting information, and outputs a third training image 804 obtained by performing AI up-scaling on the first training image 802 or the second training image.

[0239] In operation S865 , the training apparatus 1000 calculates complexity loss information 820 based on the first training image 802 .

[0240] In operation S870 , the training apparatus 1000 calculates structure loss information 810 by comparing the reduced training image 803 and the first training image 802 .

[0241] In operation S875 , the training apparatus 1000 calculates quality loss information 830 by comparing the original training image 801 and the third training image 804 .

[0242] In operation S880, the initially set DNN setting information is updated via a back propagation process based on the final loss information. The training apparatus 1000 may calculate the final loss information for training the first DNN 700 based on the complexity loss information 820, the structure loss information 810, and the quality loss information 830.

[0243] In operation S885, the second DNN 300 updates the initially set DNN setting information via a back propagation process based on the quality loss information 830 or the final loss information. The training apparatus 1000 may calculate the final loss information for training the second DNN 300 based on the quality loss information 830.

[0244] Then, the training device 1000, the first DNN 700, and the second DNN 300 may repeat operations S850 to S885 until the final loss information is minimized to update the DNN setting information. At this time, during each repetition, the first DNN 700 and the second DNN 300 operate according to the DNN setting information updated in the previous operation.

[0245] Table 1 below shows the effects when AI encoding and AI decoding are performed on the original image 105 according to an embodiment of the present disclosure and when encoding and decoding are performed on the original image 105 via HEVC.

[0246] Table 1

[0247]

[0248] As shown in Table 1, although the subjective image quality when AI encoding and AI decoding are performed on content including 300 frames of 8K resolution according to an embodiment of the present disclosure is higher than the subjective image quality when encoding and decoding are performed through HEVC, the bit rate is reduced by at least 50%.

[0249] Fig.11 is a diagram of a first device 20 for performing AI downscaling on an original image 105 and a second device 40 for performing AI upscaling on a second image 135 .

[0250] The first device 20 receives the original image 105 and provides the image data 25 and the AI ​​data 30 to the second device 40 by using the AI ​​downsizer 1124 and the transform-based encoder 1126. According to an embodiment, the image data 25 and the AI ​​data 30 are Figure 1 The image data corresponds to the AI ​​data 30. Figure 1 In addition, according to an embodiment, the transform-based encoder 1126 and Figure 7 The first encoder 614 corresponds to the AI ​​reducer 1124 and Figure 7 The AI ​​reducer 612 responds accordingly.

[0251] The second device 40 receives the AI ​​data 30 and the image data 25, and obtains a third image 145 by using a transform-based decoder 1146 and an AI amplifier 1144. According to an embodiment, the transform-based decoder 1146 and the AI ​​amplifier 1144 are Figure 2 The first decoder 232 corresponds to the AI ​​amplifier 1144 and Figure 2 The AI ​​amplifier 234 responds accordingly.

[0252] According to an embodiment, the first device 20 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the first device 20 executes the reference to the computer program according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, reference will be made to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0253] According to an embodiment, the second device 40 includes a CPU, a memory, and a computer program including instructions. The computer program is stored in the memory. According to an embodiment, the second device 40 executes the reference to the computer program according to the execution of the computer program by the CPU. Fig.11 According to the embodiment, reference will be made to Fig.11 The functions described are performed by dedicated hardware chips and / or CPUs.

[0254] exist Fig.11 , the configuration controller 1122 of the first device 20 receives at least one input value 10. According to an embodiment, the at least one input value 10 may include any one or any combination of a target resolution difference for the AI ​​downsizer 1124 and the AI ​​amplifier 1144, a bit rate of the image data 25, a bit rate type of the image data 25 (e.g., a variable bit rate type, a constant bit rate type, or an average bit rate type), and a codec type for the transform-based encoder 1126. The at least one input value 10 may include a value pre-stored in the first device 20 or a value input from a user.

[0255] The configuration controller 1122 controls the operations of the AI ​​downsizer 1124 and the transform-based encoder 1126 based on the received input value 10. According to an embodiment, the configuration controller 1122 obtains DNN setting information for the AI ​​downsizer 1124 according to the received input value 10, and sets the AI ​​downsizer 1124 using the obtained DNN setting information. According to an embodiment, the configuration controller 1122 may send the received input value 10 to the AI ​​downsizer 1124, and the AI ​​downsizer 1124 may obtain DNN setting information for performing AI downsizing on the original image 105 based on the received input value 10. According to an embodiment, the configuration controller 1122 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI downsizing is applied and tone mapping information of a high dynamic range (HDR)) together with the input value 10 to the AI ​​downsizer 1124, and the AI ​​downsizer 1124 may obtain the DNN setting information in consideration of the input value 10 and the additional information. According to an embodiment, the configuration controller 1122 sends at least a portion of the received input value 10 to the transform-based encoder 1126, and the transform-based encoder 1126 performs a first encoding on the first image 115 by a bit rate of a specific value, a specific type of bit rate, and a specific codec.

[0256] The AI ​​reducer 1124 receives the original image 105 and performs a reference Figure 1 , Figure 7 , Figure 8 , Fig. 9 and Fig.10 Any one or any combination of the operations described in is performed to obtain the first image 115 .

[0257] According to an embodiment, the AI ​​data 30 is provided to the second device 40. The AI ​​data 30 may include one or both of the resolution difference information between the original image 105 and the first image 115 and the information related to the first image 115. The resolution difference information may be determined based on the target resolution difference of the input value 10, and the information related to the first image 115 may be determined based on at least one of the target bit rate, the bit rate type, or the codec type. According to an embodiment, the AI ​​data 30 may include parameters used during AI upscaling. The AI ​​data 30 may be provided to the second device 40 from the AI ​​downsizer 1124.

[0258] The image data 25 is obtained when the original image 105 is processed by the transform-based encoder 1126 and is sent to the second device 40. The transform-based encoder 1126 may process the first image 115 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or VA1.

[0259] The configuration controller 1142 of the second device 40 controls the operation of the AI ​​amplifier 1144 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 obtains DNN setting information for the AI ​​amplifier 1144 according to the received AI data 30, and sets the AI ​​amplifier 1144 with the obtained DNN setting information. According to an embodiment, the configuration controller 1142 may send the received AI data 30 to the AI ​​amplifier 1144, and the AI ​​amplifier 1144 may obtain DNN setting information for performing AI amplification on the second image 135 based on the AI ​​data 30. According to an embodiment, the configuration controller 1142 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component) information to which AI amplification is applied and tone mapping information of HDR) to the AI ​​amplifier 1144 together with the AI ​​data 30, and the AI ​​amplifier 1144 may obtain the DNN setting information in consideration of the AI ​​data 30 and the additional information.

[0260] According to an embodiment, the AI ​​amplifier 1144 may receive AI data 30 from the configuration controller 1142, receive at least one of prediction mode information, motion information, or quantization parameter information from the transform-based decoder 1146, and obtain DNN setting information based on any one or any combination of the prediction mode information, motion information, and quantization parameter information and the AI ​​data 30.

[0261] The transform-based decoder 1146 may process the image data 25 to reconstruct the second image 135. The transform-based decoder 1146 may process the image data 25 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, ​​or AV1.

[0262] The AI ​​amplifier 1144 may obtain the third image 145 by performing AI up-amplification on the second image 135 provided from the transform-based decoder 1146 based on the set DNN setting information.

[0263] The AI ​​reducer 1124 may include a first DNN, and the AI ​​amplifier 1144 may include a second DNN, and according to an embodiment, according to the reference Fig. 9 and Fig.10 A training method is described to train DNN setting information for a first DNN and a second DNN.

[0264] Fig.12 12 is a diagram for describing the concept of a streaming system 1200 according to an embodiment of the present disclosure.

[0265] Reference Fig.12, the streaming system 1200 may include a server 1210 and a terminal 1220. However, this is an example, and the elements of the streaming system 1200 are not limited to the server 1210 and the terminal 1220.

[0266] The server 1210 may stream image data to the terminal 1220. In the present disclosure, streaming refers to an operation of transmitting and receiving image data between the server 1210 and the terminal 1220 so that the terminal 1220 can reproduce the image data in real time. In addition, the server 1210 may stream various types of data including audio data and text data as well as image data to the terminal 1220, but in the present disclosure, a method of streaming image data according to an embodiment of the present disclosure will be described.

[0267] For streaming, the server 1210 and the terminal 1220 may be connected via a network 1230. The server 1210 may stream image data to the terminal 1220 via the network 1230.

[0268] For example, when the terminal 1220 requests predefined image data from the server 1210 in multiple image data, the predefined image data may be sent to the terminal 1220. Multiple image data may also be referred to as multiple versions of image content (e.g., movies, television content, videos, etc.) or multiple different quality versions of image content. The predefined image data may be image data corresponding to the user's settings. However, the present disclosure is not limited to this example, and therefore, in another example of the present disclosure, the predefined image data may be image data with a quality set as default when streaming between the server 1210 and the terminal 1220 starts. While the server 1210 streams the image data, the state of the network 1230 may be variable. Information about the state of the network 1230 may be determined based on the amount of communication in the transmission and reception paths between the server 1210 and the terminal 1220, and this may be described as a congestion level. However, this is an example, and the state of the network 1230 is not described as being based only on the communication occurring in the transmission and reception paths.

[0269] In order to adaptively perform streaming based on the variable state of the network, the server 1210 may adjust one or both of the bit rate and resolution of the image data to be transmitted from the server 1210 to the terminal 1220. The server 1210 may store multiple pieces of image data 10 (e.g., a high-definition (HD)-level Iron Man movie, a standard-definition (SD)-level Iron Man movie, a 15-Mbps Iron Man movie, a 10-Mbps Iron Man movie, etc.) for the same image content (e.g., an Iron Man movie), and the multiple pieces of image data 10 are obtained from the same image content by adjusting one or both of the bit rate and the resolution. However, this is an example, and the parameters for adjusting the quality of the image data may also include a sampling frequency, a frame rate, a window size (e.g., 1920×1080 (1080p) HD, 1280×720 (720p) HD, etc.), a video codec (e.g., H.264, H.265, advanced video coding, etc.), a pixel aspect ratio, an audio codec (e.g., advanced audio coding), etc.

[0270] The server 1210 according to an embodiment of the present disclosure may store multiple items of image data 10 having different qualities, and the multiple items of image data 10 may include one or both of the AI-encoded image data 12 or the non-AI-encoded image data 14. The AI-encoded image data 12 is generated through the above-mentioned AI encoding process, and the AI ​​encoding process includes a process of performing AI reduction on the original image through the first DNN 700. In this regard, the first DNN 700 is jointly trained with the second DNN 300 of the terminal 1220, and when the terminal 1220 receives the AI-encoded image data, the terminal 1220 may perform AI upscaling on the image data through the second DNN 300. In addition, the AI-encoded image data 12 may be stored together with the AI ​​data related to the AI ​​downscaling, and the AI ​​data may be used by the terminal 1220 during the AI ​​upscaling process.

[0271] The server 1210 may provide the terminal 1220 with additional information of the multiple items of image data 10 to allow the terminal 1220 to request image data corresponding to the state of the network 1230 among the multiple items of image data 10. The additional information may include quality information about each item of image data among the multiple items of image data 10 and AI scale conversion information.

[0272] The quality of each of the multiple image data 10 may be determined according to the resolution and the bit rate, and the quality information may include the values ​​of the resolution and the bit rate of each of the multiple image data 10. However, this is an example, and the quality may be determined according to the sampling frequency, the frame rate, the window size, the video codec, the pixel aspect ratio, the audio codec, etc. The AI ​​scaling conversion information may include information indicating whether the image data is AI-encoded image data, the value of the AI ​​scaling conversion level, etc. In this regard, the AI ​​scaling conversion level is an index indicating the difference between the AI-reduced image data and the original image data, and may be defined for resolution, bit rate, etc. For example, when the resolutions of 8K, 4K, full HD (FHD), and HD are supported in the streaming system 1200, the difference between two adjacent resolutions may be defined as a level interval. In this case, it may be described that the level difference between 8K and 4K corresponds to one level interval, and the level difference between 8K and FHD corresponds to two level intervals. For example, the resolutions of 8K, 4K, full HD (FHD), and HD may be set as the first resolution level, the second resolution level, the third resolution level, and the fourth resolution level, respectively, and the level interval between the resolutions of 8K, 4K, full HD (FHD), and HD (also referred to as "resolution level interval") may be determined as the difference between the resolution levels set for the resolutions of 8K, 4K, full HD (FHD), and HD. In addition, when the bit rates of 40 Mbps, 30 Mbps, 20 Mbps, and 10 Mbps are supported in the streaming system 1200, the difference between two adjacent bit rates may be defined as a level interval. For example, the bit rates of 40 Mbps, 30 Mbps, 20 Mbps, and 10 Mbps may be set as the first bit rate level, the second bit rate level, the third bit rate level, and the fourth bit rate level, respectively, and the level interval between the bit rates of 40 Mbps, 30 Mbps, 20 Mbps, and 10 Mbps (also referred to as "bit rate level interval") may be determined as the difference between the bit rate levels set for the bit rates of 40 Mbps, 30 Mbps, 20 Mbps, and 10 Mbps. According to an embodiment of the present disclosure, the level and the level interval may be defined based on a combination of resolution and bit rate. That is, the difference between 8K & 40Mbps and 8K & 30Mbps may be defined as one level interval, and the difference between 8K & 30Mbps and 4K & 20Mbps may be defined as two level intervals. However, this is an example, and the AI ​​scaling conversion level may be determined based on other factors other than resolution and bit rate.

[0273] The terminal 1220 according to an embodiment of the present disclosure may check the resolution and bit rate of the multiple image data 10 included in the additional information, and may request the server 1210 to send the image data that has been AI-encoded at a specific resolution or bit rate. In addition, the terminal 1220 may determine the resolution and bit rate of the image data to be requested from the server 1210 according to the state of the network 1230. For example, while the terminal 1220 receives the AI-encoded image data of FHD and 5Mbps from the server 1210, when it is confirmed that the congestion level of the network is improved due to the reduction in the bit error rate (BER) of the received image data, the terminal 1220 may request the server 1210 to increase or decrease the level of the resolution level, the bit rate level, or the combination of the resolution level and the bit rate level.

[0274] However, this is an example, and the terminal 1220 may request to transmit image data of a specific resolution or a specific bit rate, and the server 1210 may determine whether to transmit AI-encoded image data or non-AI-encoded image data. Figures 18 to 20 An embodiment of the present disclosure in which the terminal 1220 requests image data from the server 1210 based on the additional information is further described.

[0275] Fig.13A is a flowchart for describing a method of streaming data performed by a server according to an embodiment of the present disclosure.

[0276] In operation S1310, the server 1210 may transmit additional information of multiple items of image data of different qualities to the terminal 1220.

[0277] In response to a request from the terminal 1220, the server 1210 may send additional information of the plurality of image data to the terminal 1220. The additional information may include quality information and AI scaling conversion information about each of the plurality of image data. The quality information may include resolution and bit rate values ​​of the plurality of image data, respectively, and the AI ​​scaling conversion information may include information indicating whether the image data is AI-encoded image data, a value of an AI scaling conversion level, etc. However, this is an example, and reference will be made below. Figure 21 to Figure 24 The pieces of information included in the additional information are further described.

[0278] The additional information may be a media presentation description (MPD) according to the Moving Picture Experts Group (MPEG)-Dynamic Adaptive Streaming over Hypertext Transfer Protocol (HTTP) (DASH) standard. However, this is an example, and the additional information may be provided as a manifest file of a different type stored in an Extensible Markup Language (XML) format.

[0279] In operation S1320, the server 1210 may receive a request for image data having a quality corresponding to a state of a network between the terminal and the server 1210 from the terminal 1220 based on the additional information.

[0280] The server 1210 may receive a request message requesting image data of a specific quality among the multiple items of image data from the terminal 1220 , and the request message may include information for specifying one of the multiple items of image data.

[0281] According to an embodiment of the present disclosure, the request message may include quality information about the image data requested by the terminal 1220. For example, the request message may include information about one or both of a bit rate or a resolution.

[0282] According to an embodiment of the present disclosure, the request message may include quality information and information indicating whether AI reduction has been applied. For example, when AI-encoded image data is requested, the quality information indicates the quality of the AI-encoded image data. That is, when AI-encoded image data is requested and its quality information indicates FHD and 5Mbps, the server 1210 can determine that the terminal requests AI-encoded image data with a resolution of FHD and a bit rate of 5Mbps, which is obtained as a result of performing AI reduction. According to an embodiment of the present disclosure, when non-AI-encoded image data is requested, the quality information indicates the quality of the original image data. That is, when non-AI-encoded image data is requested and its quality information indicates FHD and 5Mbps, the server 1210 can determine that the terminal 1220 requests original image data with a resolution of FHD and a bit rate of 5Mbps.

[0283] The request message according to an embodiment of the present disclosure may include capability information of the terminal 1220 and quality information about the image data requested by the terminal 1220. For example, when the quality information indicates FHD and 20Mbps and the terminal 1220 supports AI upscaling, the server 1210 may determine that the terminal 1220 requests AI-encoded image data having a resolution of FHD and a bit rate of 20Mbps, which is obtained as a result of performing AI downscaling. According to an embodiment of the present disclosure, when the quality information indicates FHD and 20Mbps and the terminal 1220 does not support AI upscaling, the server 1210 may determine that the terminal 1220 requests original image data having a resolution of FHD and a bit rate of 20Mbps.

[0284] In operation S1330 , in response to the request, the server 1210 may transmit, to the terminal 1220 , the AI ​​data and the image data that has been AI-encoded by the first DNN jointly trained with the second DNN of the terminal 1220 .

[0285] When the terminal 1220 requests AI-encoded image data, the server 1210 may transmit the AI-encoded image data to the terminal 1220 together with AI data including information that can be used to perform AI upscaling on the AI-encoded image data. For example, the AI ​​data may include information about any one or any combination of information indicating whether AI downscaling has been applied, an AI scaling conversion level, and DNN configuration information used in AI upscaling. However, this is an example, and the AI ​​data may include other information that can be used in performing AI upscaling.

[0286] The server 1210 may transmit the AI-encoded image data to the terminal 1220 in units of segments. The segments may be generated by dividing the AI-encoded image data based on a preset time unit. However, this is an example, and the transmission unit of the AI-encoded image data transmitted from the server 1210 is not limited to the unit of the segment.

[0287] In operation S1340, when the state of the network between the terminal 1220 and the server 1210 changes, the server 1210 may receive a request for image data of different quality corresponding to the changed state of the network from the terminal 1220.

[0288] The terminal 1220 may periodically determine the state of the network. For example, the terminal 1220 may periodically measure the timestamp and BER of the received image data, and thus may determine the state of the network. In addition, when the state of the network changes, the terminal 1220 may change the quality of the image data to be requested from the server 1210. For example, in the case where the terminal 1220 requests FHD and 5Mbps image data from the server 1210 at a first time point when the network is congested, if the congestion is reduced and the condition of the network is improved at a second time point, the terminal 1220 may request 4K and 10Mbps image data from the server 1210 at a second time point after the first time point. When any one or any combination of delay (or waiting time), bit error rate, packet loss, and timeout (e.g., lost connection) is observed, the terminal 1220 or the server 1210 may determine that network congestion occurs. For example, when the time taken for the destination to receive a packet sent by the sender (i.e., delay or waiting time) is longer than a threshold delay, the terminal 1220 or the server 1210 may determine that network congestion occurs. In another example, when the terminal 1220 experiences buffering longer than a threshold buffering time while reproducing a video transmitted from the server 1210 , the terminal 1220 or the server 1210 may determine that network congestion occurs.

[0289] Information included in the request message to be transmitted from the terminal 1220 to the server 1210 in order to request image data of different quality corresponding to the changed state of the network may correspond to the description provided with reference to S1320.

[0290] Fig. 13B is a flowchart for describing a method of streaming data performed by the terminal 1220 according to an embodiment of the present disclosure.

[0291] In operation S1315, the terminal 1220 may request image data of a quality corresponding to the state of the network from among the multiple items of image data from the server 1210 based on the additional information. The additional information may include quality information and AI scaling conversion information about each of the multiple items of image data stored in the server 1210. The quality information may include resolutions and bit rates of the multiple items of image data, respectively, and the AI ​​scaling conversion information may include information indicating whether the image data is AI-encoded image data and a value of a level at which AI downscaling has been performed.

[0292] The terminal 1220 may determine the quality of the image data corresponding to the state of the network of the terminal 1220 based on the state of the network. The state of the network may be determined based on the timestamp and BER of the image data received by the terminal 1220.

[0293] A timestamp refers to information indicating the elapsed time from a reference time to the reception time of the image data. For example, when the average value of the timestamps in the first time period is 3ms and the average value of the timestamps in the second time period is 5ms, the terminal 1220 can determine that the state of the network in the second time period is congested compared to the first time period. In addition, BER refers to the ratio of the number of error bits to the total number of transmission bits. For example, when the BER is lower than a preset reference, the terminal 1220 can determine that the state of the network is not congested. As another example, when the value of the BER in the first time period is 0.005 and the value of the BER in the second time period is 0.01, the terminal 1220 can determine that the state of the network in the second time period is congested compared to the first time period. However, this is an example, and the state of the network can be determined based on another information.

[0294] For example, when the terminal 1220 estimates that the state of the network between the server 1210 and the terminal 1220 is congested, the terminal 1220 may select image data of 20 Mbps as a relatively low bit rate from 50 Mbps, 40 Mbps, 30 Mbps, and 20 Mbps as respective bit rates of the plurality of image data. However, this is an example, and the method of determining the quality of image data corresponding to the state of the network by the terminal 1220 is not limited to this example.

[0295] The terminal 1220 may send a request message to the server 1210 requesting the image data of the determined quality. According to an embodiment of the present disclosure, the request message may include quality information about the image data requested by the terminal 1220. For example, the request message may include information about at least one of the bit rate or the resolution. The request message according to an embodiment of the present disclosure may include quality information and information indicating whether AI reduction has been applied. The request message according to an embodiment of the present disclosure may include capability information of the terminal 1220 and quality information about the image data requested by the terminal 1220. The capability information may include information about whether the terminal 1220 supports AI amplification.

[0296] In operation S1325, when the terminal 1220 receives the image data and the AI ​​data corresponding to the request, the terminal 1220 may determine whether to perform AI up-scaling on the received image data based on the AI ​​data.

[0297] According to an embodiment of the present disclosure, the terminal 1220 may determine whether the received image data has been AI-encoded by a first DNN jointly trained with a second DNN based on the AI ​​data. The AI ​​data may include information about at least one of the following: information indicating whether AI reduction has been applied, an AI scaling conversion level, or DNN configuration information used in AI amplification. The DNN configuration information may be provided as an indicator indicating the number of convolutional layers, the number of filter kernels for each convolutional layer, the parameters of each filter kernel, and the like. However, this is an example, and the DNN configuration information may be provided as a lookup table, and as another example, the second DNN may be provided as the DNN configuration information. However, this is an example, and the AI ​​data may include other information required for the terminal 1220 to perform AI amplification.

[0298] According to an embodiment of the present disclosure, when the terminal 1220 includes information specifying AI-encoded image data in the request message for image data in the above operation S1315, the terminal 1220 may determine that AI reduction has been applied to the image data received in response to the request message.

[0299] In operation S1335, the terminal 1220 may perform AI upscaling on the received image data through the second DNN jointly trained with the first DNN based on the result of determining whether to perform AI upscaling.

[0300] When the terminal 1220 determines that the received image data is image data that has been AI-encoded by the first DNN jointly trained with the second DNN, the terminal may perform AI upscaling on the received image data by the second DNN.

[0301] According to an embodiment of the present disclosure, the terminal 1220 may determine the DNN configuration information of the second DNN based on at least one of the resolution or the bit rate of the AI-encoded image data. For example, when the resolution and the bit rate of the AI-encoded image data are 4K and 10Mbps, the terminal 1220 may select DNN configuration information optimized for resolution and bit rate from multiple pieces of DNN configuration information. In this regard, multiple pieces of DNN configuration information optimized for resolution and bit rate, respectively, may be pre-trained in the terminal 1220, and their information may be included in the AI ​​data, as will be described in the following embodiments. According to an embodiment of the present disclosure, the terminal 1220 may obtain DNN configuration information optimized for performing AI amplification on AI-encoded image data based on the DNN configuration information included in the AI ​​data.

[0302] The terminal 1220 may perform AI upscaling on the AI-encoded image data based on the selected DNN configuration information through a second DNN jointly trained with the first DNN.

[0303] In operation S1345, when the state of the network changes, the terminal 1220 may request image data of different quality corresponding to the changed state of the network from the server 1210 based on the additional information.

[0304] For example, although the terminal 1220 requests FHD and 5Mbps AI-encoded image data in the above operation S1315, when the interval of the timestamps of the image data received thereafter becomes shorter or the BER decreases, the terminal 1220 can determine that the congestion level of the network is alleviated and improved, and therefore can request 4K and 10Mbps AI-encoded image data from the server 1210.

[0305] As another example, although the terminal 1220 requests 4K and 10Mbps AI-encoded image data in the above operation S1315, when the interval of the timestamps of the image data received thereafter becomes longer or the BER increases, the terminal 1220 can determine that the congestion level of the network has deteriorated, and therefore can request FHD and 5Mbps AI-encoded image data from the server 1210.

[0306] As another example, although the terminal 1220 requests FHD and 5Mbps AI-encoded image data in the above operation S1315, when the interval of the timestamps of the image data received thereafter becomes longer or the BER increases, the terminal 1220 may determine that the congestion level of the network has deteriorated, and thus may request HD and 1Mbps image data from the server 1210. That is, when the resolution and the bit rate are less than a predetermined reference, the terminal 1220 may consider the level of the image data to be reconstructed by AI upscaling, and thus may request the server 1210 for image data on which AI downscaling has not been performed. However, this is an example, and the method of changing the quality of image data based on the change in the state of the network, performed by the terminal 1220, is not limited to the above example.

[0307] Information included in the request message transmitted from the terminal 1220 to the server 1210 in order to request image data of different qualities corresponding to the state of the network may correspond to the above description provided with reference to operation S1315.

[0308] Fig.14A is a flowchart for describing a method of streaming data performed by a server according to an embodiment of the present disclosure.

[0309] In operation S1410, the server 1210 may receive, from the terminal 1220, a request for one item of image data among a plurality of image data of different qualities determined based on additional information of the plurality of image data.

[0310] The additional information may be provided to the terminal 1220 from a service server independently separated from the server 1210. However, this is an example, and the additional information may be provided from the server 1210 to the terminal 1220.

[0311] According to an embodiment of the present disclosure, the server 1210 may receive a request for image data whose quality is set by the user based on the additional information from the terminal 1220. For example, when the user of the terminal 1220 selects the quality of FHD and 5Mbps, the server 1210 may receive a request for image data of FHD and 5Mbps from the terminal 1220. However, this is an example, and according to an embodiment of the present disclosure, the server 1210 may receive a request for image data of a quality set as a default from the terminal 1220. For example, in the case where streaming of image data is started between the terminal 1220 and the server 1210, when the state of the network between the terminal 1220 and the server 1210 is not confirmed, the server 1210 may receive a request for image data of the lowest quality among a plurality of qualities from the terminal 1220. According to an embodiment of the present disclosure, the server 1210 may receive a request for image data of a specific quality (e.g., HD and 4Mbps) set as a default from the terminal 1220. According to an embodiment of the present disclosure, in addition to the quality, the server 1210 may also receive a request for image data specifying information on whether AI encoding has been performed from the terminal 1220.

[0312] In operation S1420 , in response to the request, the server 1210 may transmit, to the terminal 1220 , the AI ​​data and the image data that has been AI-encoded by the DNN for downscaling jointly trained with the DNN for upscaling of the terminal 1220 .

[0313] When the terminal 1220 requests AI-encoded image data, the server 1210 may send the AI-encoded image data to the terminal 1220 together with AI data including information that may be required to enlarge the AI-encoded image data. For example, the AI ​​data may include information about at least one of: information indicating whether AI reduction has been applied, an AI scaling conversion level, or DNN configuration information used in AI enlargement. However, this is an example, and the AI ​​data may include other information that may be required for the terminal 1220 to perform AI enlargement.

[0314] The server 1210 may transmit the AI-encoded image data to the terminal 1220 in units of segments. The segments may be generated by dividing the AI-encoded image data based on a preset time unit. However, this is an example, and the transmission unit of the AI-encoded image data transmitted from the server 1210 is not limited to the unit of the segment.

[0315] In operation S1430, according to the state of the network between the terminal 1220 and the server 1210, the server 1210 may receive a request for image data of different qualities among the multiple items of image data from the terminal 1220 based on the additional information.

[0316] According to an embodiment of the present disclosure, the request may include quality information about the image data requested by the terminal 1220. For example, the request may include information about at least one of a bit rate or a resolution. As another example, the request may include quality information and information indicating whether AI reduction has been applied. As another example, the request message may include capability information of the terminal and quality information about the image data requested by the terminal 1220.

[0317] Fig. 14B is a flowchart for describing a method of streaming data performed by the terminal 1220 according to an embodiment of the present disclosure.

[0318] In operation S1405, the terminal 1220 may request specific image data. The terminal 1220 according to an embodiment of the present disclosure may request image data of a specific quality. For example, the terminal 1220 may request image data of FHD and 5Mbps. However, this is an example, and according to an embodiment of the present disclosure, the terminal 1220 may request image data from the server 1210 without specifying its quality.

[0319] According to an embodiment of the present disclosure, the terminal 1220 may request image data from the server 1210 by specifying information on whether AI encoding has been performed in addition to quality.

[0320] In operation S1415, the terminal 1220 may receive image data corresponding to the request. The terminal 1220 according to an embodiment of the present disclosure may receive additional information as well as image data corresponding to the request. The additional information may include quality information about multiple image data that may be provided from the server 1210 to the terminal 1220, AI scaling conversion information, and the like. However, this is an example, and the additional information may include multiple other pieces of information for identifying the multiple image data, respectively.

[0321] Furthermore, the additional information may include respective uniform resource locators (URLs) for receiving a plurality of pieces of image data identifiable based on quality information, AI scaling conversion information, and the like.

[0322] According to an embodiment of the present disclosure, the additional information to be received by the terminal 1220 may be determined according to the capability information of the terminal 1220. For example, when the terminal 1220 is a device supporting AI decoding, the server 1210 may send the additional information including the AI ​​scaling conversion information to the terminal 1220, and when the terminal 1220 is a device not supporting AI decoding, the server 1210 may send the additional information not including the AI ​​scaling conversion information to the terminal 1220. However, this is an example, and in the case where the terminal 1220 is a device not supporting AI decoding, even when the terminal 1220 receives the additional information including the AI ​​scaling conversion information, the terminal 1220 may not interpret but may ignore the additional information. In this embodiment of the present disclosure, the capability information of the terminal 1220 may be provided to the server 1210 in advance, or may be included in the request for specific image data.

[0323] The above-described embodiments of the present disclosure are examples, and thus the additional information may be provided from the server 1210 to the terminal 1220 after receiving the image data during a predetermined period, or may be provided from the server 1210 to the terminal 1220 before receiving the image data.

[0324] In operation S1425, the terminal 1220 may determine whether the state of the network is changed.

[0325] The state of the network may be determined based on the timestamp and BER of the image data received by the terminal 1220. For example, as a result of the determination based on the timestamp, when the terminal 1220 determines that the time of receiving the image data from the server 1210 is delayed, the terminal 1220 may determine that the state of the network is congested. As another example, when the BER is less than a predetermined reference, the terminal 1220 may determine that the state of the network is not congested. However, this is an example, and the state of the network may be determined based on other information.

[0326] According to an embodiment of the present disclosure, when at least one of the timestamp or BER of the image data changes, the terminal 1220 may determine that the state of the network has changed, and according to an embodiment of the present disclosure, when at least one of the timestamp or BER exceeds a preset range, or when a difference of at least a predetermined value from a previous measured value occurs or the difference is maintained for a certain period of time, the terminal 1220 may determine that the state of the network has changed. For example, when the previous BER is 0.001, and the BER measured thereafter is within a range between 0.0095 and 0.005, the terminal 1220 determines that the state of the network is maintained, but when the BER exceeds the corresponding range, the terminal 1220 may determine that the state of the network has changed. However, this is an example, and reference to making a determination as to whether the state of the network has changed is not limited to the above example.

[0327] As a result of the determination, when the terminal 1220 determines that the state of the network has not changed, the terminal 1220 may receive image data corresponding to the quality requested in operation S1405 or corresponding to whether AI downscaling has been performed.

[0328] In operation S1435, the terminal 1220 may change the image data to be requested based on the additional information. In the above operation S1425, when the terminal 1220 determines that the state of the network has changed, the terminal 1220 may determine the requestable image data (or image data quality setting) based on the additional information and the capabilities of the terminal 1220.

[0329] In order to determine the required image data (or image data quality setting), the terminal 1220 may determine whether the terminal 1220 is capable of supporting AI decoding. According to an embodiment of the present disclosure, the terminal 1220 may determine whether the terminal 1220 is capable of performing AI amplification. According to an embodiment of the present disclosure, the terminal 1220 may determine whether the AI-encoded image data of the corresponding quality can be the type of DNN configuration information that can be AI-enlarged according to the type of DNN configuration information based on the quality corresponding to the state of the changed network and the DNN configuration information jointly trained with the first DNN of the server 1210 in the second DNN of the terminal 1220. For example, when the quality corresponding to the state of the changed network is FHD and 5Mbps, the terminal 1220 may determine whether the second DNN has been jointly trained with the first DNN of the server 1210 to AI amplify the AI-encoded image data of FHD and 5Mbps.

[0330] According to an embodiment of the present disclosure, in the case of image data of the same quality, the terminal 1220 may determine whether AI reduction has been performed on the image data to be requested, the AI ​​reduction level, the type of DNN configuration information used in AI reduction, etc., based on the hardware device specification of the terminal 1220, the type of codec, etc. Information on whether AI reduction has been performed on each image data, the AI ​​reduction level of each image data, the type of DNN configuration information used in AI reduction, etc. may be included in the additional information and provided to the terminal 1220, and will be referred to below. Figure 20 to Figure 23 A description thereof is further provided.

[0331] Fig.15A is a flowchart for describing a method of streaming data performed by a server according to an embodiment of the present disclosure.

[0332] In operation S1510, the server 1210 may transmit additional information of multiple items of image data of different qualities to the terminal 1220.

[0333] The server 1210 may transmit additional information of the multiple items of image data in response to a request from the terminal 1220. However, this is an example, and the server 1210 may transmit the additional information to the terminal 1220 when a communication session for streaming image data is established between the server 1210 and the terminal 1220.

[0334] In this embodiment of the present disclosure, the additional information may correspond to the reference Fig.13A Additional information for the description.

[0335] In operation S1520, the server 1210 may receive a request for image data among the multiple items of image data from the terminal 1220 based on the additional information.

[0336] According to an embodiment of the present disclosure, the server 1210 may receive a request for image data whose quality is set by the user from the terminal 1220. For example, when the user of the terminal 1220 selects the quality of FHD and 5Mbps, the server 1210 may receive a request for image data of FHD and 5Mbps from the terminal 1220. However, this is an example, and according to an embodiment of the present disclosure, the server 1210 may receive a request for image data of a quality set as a default from the terminal 1220. For example, in the case where streaming of image data is started between the terminal 1220 and the server 1210, when the state of the network between the terminal 1220 and the server is not confirmed, the server 1210 may receive a request for image data of the lowest quality among a plurality of qualities from the terminal 1220. According to an embodiment of the present disclosure, the server 1210 may receive a request for image data of a specific quality (e.g., HD and 4Mbps) set as a default from the terminal 1220. According to an embodiment of the present disclosure, in addition to the quality, the server 1210 may also receive a request for image data specifying information on whether AI encoding has been performed from the terminal.

[0337] In operation S1530 , in response to the request, the server 1210 may transmit, to the terminal 1220 , the AI ​​data and the image data that has been AI-encoded by the DNN for downscaling jointly trained with the DNN for upscaling of the terminal 1220 .

[0338] When the terminal 1220 requests AI-encoded image data, the server 1210 may transmit the AI-encoded image data to the terminal 1220 together with AI data including information required to enlarge the AI-encoded image data. For example, the AI ​​data may include information about at least one of: information indicating whether AI reduction has been applied, an AI scaling conversion level, or DNN configuration information used in AI enlargement. However, this is an example, and the AI ​​data may include other information required for the terminal 1220 to perform AI enlargement.

[0339] The server 1210 may transmit the AI-encoded image data to the terminal 1220 in units of segments. The segments may be generated by segmenting the AI-encoded image data based on a preset time unit. However, this is merely an example, and the transmission unit of the AI-encoded image data transmitted from the server 1210 is not limited to the unit of segments.

[0340] In operation S1540, according to the state of the network between the terminal 1220 and the server 1210, the server 1210 may receive a request for image data of different qualities among the plurality of pieces of image data from the terminal 1220 based on the additional information.

[0341] According to an embodiment of the present disclosure, the request may include quality information about the image data requested by the terminal 1220. For example, the request may include information about at least one of a bit rate or a resolution. As another example, the request may include quality information and information indicating whether AI reduction has been applied. As another example, the request message may include capability information of the terminal 1220 and quality information about the image data requested by the terminal 1220.

[0342] Fig. 15B is a flowchart for describing a method of streaming data performed by the terminal 1220 according to an embodiment of the present disclosure.

[0343] In operation S1515, the terminal 1220 may receive additional information of the multiple image data of different qualities from the server 1210. According to an embodiment of the present disclosure, the terminal 1220 may request the additional information of the multiple image data from the server 1210. However, this is an example, and when a communication session for streaming image data is established between the terminal 1220 and the server 1210, the server 1210 may transmit the additional information to the terminal 1220.

[0344] In this embodiment of the present disclosure, the additional information may be associated with the reference Fig. 13B Additional information described accordingly.

[0345] In operation S1525, the terminal 1220 may request the server 1210 for image data among the multiple image data based on the additional information. According to an embodiment of the present disclosure, the terminal 1220 may request the server 1210 for image data whose quality is set by the user. For example, when the user of the terminal 1220 selects the quality of FHD and 5Mbps, the terminal 1220 may request the server 1210 for image data of FHD and 5Mbps from the terminal 1220. However, this is an example, and according to an embodiment of the present disclosure, the terminal 1220 may request the server 1210 for image data of a quality set as a default. For example, in the case where streaming of image data is started between the terminal 1220 and the server 1210, when the state of the network between the terminal 1220 and the server 1210 is not confirmed, the terminal 1220 may request the server 1210 for image data of the lowest quality among the multiple qualities. According to an embodiment of the present disclosure, the terminal 1220 may request the server 1210 for image data of a specific quality (e.g., HD and 4Mbps) set as a default.

[0346] According to an embodiment of the present disclosure, the terminal 1220 may also request image data from the server 1210 by specifying information on whether AI encoding has been performed in addition to quality.

[0347] In operation S1535, when the terminal 1220 receives the image data and the AI ​​data corresponding to the request, the terminal 1220 may determine whether to perform AI up-scaling on the received image data based on the AI ​​data.

[0348] According to an embodiment of the present disclosure, the terminal 1220 may determine whether the received image data has been AI-encoded by a first DNN jointly trained with a second DNN based on the AI ​​data. The AI ​​data may include information about at least one of the following: information indicating whether AI reduction has been applied, an AI scaling conversion level, or DNN configuration information used in AI amplification.

[0349] According to an embodiment of the present disclosure, when the terminal 1220 includes information specifying AI-encoded image data in the request message for image data in operation S1525, the terminal 1220 may determine that AI reduction has been applied to the image data received in response to the request message.

[0350] In operation S1545 , the terminal 1220 may perform AI upscaling on the received image data through the DNN for upscaling jointly trained with the DNN for downscaling of the server 1210 , based on the result of the determination as to whether to perform AI upscaling.

[0351] When the terminal 1220 determines that the received image data is image data that has been AI-encoded by the first DNN jointly trained with the second DNN, the terminal 1220 may perform AI upscaling on the received image data by the second DNN. In this embodiment of the present disclosure, the method of performing AI upscaling on the received image data by the second DNN performed by the terminal 1220 may be the same as that described above with reference to Fig. 13B The operation S1335 is described accordingly.

[0352] In operation S1555, the terminal 1220 may confirm the state of the network between the terminal 1220 and the server 1210. The state of the network may be determined based on the timestamp and BER of the image data received by the terminal 1220. For example, as a result of the determination based on the timestamp, when the terminal 1220 determines that the time of receiving the image data from the server 1210 is delayed, the terminal 1220 may determine that the state of the network is congested. As another example, when the BER is less than a predetermined reference, the terminal 1220 may determine that the state of the network is not congested. However, this is an example, and the state of the network may be determined based on other information.

[0353] In operation S1565, according to the state of the network, the terminal 1220 may request image data of different qualities from among the multiple items of image data to the server 1210 based on the additional information.

[0354] For example, although the terminal 1220 requests FHD and 5Mbps AI-encoded image data in the above operation S1525, when the interval of the timestamps of the image data received thereafter becomes shorter or the BER decreases, the terminal 1220 can determine that the congestion level of the network has improved, and therefore can request 4K and 10Mbps bit rate AI-encoded image data from the server 1210.

[0355] As another example, although the terminal 1220 requests 4K and 10Mbps AI-encoded image data in the above operation S1525, when the interval of the timestamps of the image data received thereafter becomes longer or the BER increases, the terminal 1220 can determine that the congestion level of the network has deteriorated, and therefore can request FHD and 5Mbps AI-encoded image data from the server 1210.

[0356] As another example, although the terminal 1220 requests FHD and 5Mbps AI-encoded image data in the above operation S1515, when the interval of the timestamps of the image data received thereafter becomes longer or the BER increases, the terminal 1220 may determine that the congestion level of the network has deteriorated, and thus may request HD and 1Mbps image data from the server 1210. That is, when the resolution and the bit rate are less than a predetermined reference, the terminal 1220 may consider the level of the image data to be reconstructed by AI upscaling, and thus may request the server 1210 for image data on which AI downscaling has not been performed. However, this is an example, and the method of changing the quality of image data based on the change in the state of the network, performed by the terminal 1220, is not limited to the above example.

[0357] Information included in the request message transmitted from the terminal 1220 to the server 1210 in order to request image data of different qualities corresponding to the state of the network may correspond to the above description provided with reference to operation S1515.

[0358] Fig.16 16 is a diagram for describing a method of performing streaming between a server 1610 and a first terminal 1620 according to whether the first terminal 1620 supports AI up-scaling according to an embodiment of the present disclosure.

[0359] exist Fig.16 In the embodiment of the present invention, it is assumed that the first terminal 1620 corresponds to a terminal capable of supporting AI amplification through a second DNN jointly trained with the first DNN of the server 1610, and the second terminal 1630 corresponds to a terminal that does not support AI amplification.

[0360] The server 1610 stores multiple pieces of image data of different qualities for adaptive streaming, and transmits the image data in response to a request from a terminal (e.g., the first terminal 1620). The terminal (e.g., the first terminal 1620) may obtain additional information of the multiple pieces of image data from the server 1610, and may request one piece of image data from the multiple pieces of image data from the server 1610 based on the additional information. Figure 20 to Figure 23 The pieces of information included in the additional information are further described.

[0361] For example, the server 1610 may perform AI downscaling on the original image data 1640 of 8K and 60Mbps through the 1a-1 DNN 1612 to obtain AI downscaled image data 1642 of 4K and 20Mbps. The server 1610 may transmit the AI-encoded image data 1642 of 4K and 20Mbps and the AI ​​data related to the AI ​​downscaling to 4K and 20Mbps in response to the request from the first terminal 1620. The first terminal 1620 may perform AI upscaling on the received image data through the 2a-1 DNN 1622 jointly trained with the 1a-1 DNN 1612, and thus may obtain AI upscaled image data 1652. In this regard, the first terminal 1620 may perform the above-mentioned AI upscaling by obtaining at least one of information indicating whether AI downscaling has been applied, an AI scaling conversion level, or DNN configuration information used in AI upscaling based on the information included in the received AI data.

[0362] As another example, the server 1610 may obtain AI-reduced FHD and 7Mbps image data 1644 by performing AI reduction on the original image data 1640 of 8K and 60Mbps via the 1a-2 DNN 1614. The server 1610 may transmit the AI-encoded image data 1644 of FHD and 7Mbps together with the AI ​​data related to the AI ​​reduction to FHD and 7Mbps in response to the request from the first terminal 1620. For example, while the first terminal 1620 requests and receives the AI-encoded image data 1642 of 4K and 20Mbps as in the above example, when the first terminal 1620 determines that the congestion level of the network has deteriorated, the first terminal 1620 may request the AI-encoded image data 1644 of FHD and 7Mbps. The first terminal 1620 may perform AI upscaling on the image data received in response to the request through the 2a-2 DNN 1624 jointly trained with the 1a-2DNN 1614, and thus may obtain the AI ​​upscaling image data 1654. As in the above example, the first terminal 1620 may use the information included in the AI ​​data in AI amplification.

[0363] As another example, the server 1610 may obtain the HD and 4Mbps image data 1646 that is AI-reduced by performing AI downscaling on the original image data 1640 of 8K and 60Mbps via the 1a-3 DNN 1616. The server 1610 may transmit the HD and 4Mbps AI-encoded image data 1646 together with the AI ​​data related to the AI ​​downscaling to the HD and 4Mbps in response to a request from the first terminal 1620. The first terminal 1620 may perform AI upscaling on the received image data through the 2a-3DNN 1626 jointly trained with the 1a-3 DNN 1616, and thus may obtain the AI ​​upscaling image data 1656. As in the above example, the first terminal 1620 may use the information included in the AI ​​data in the AI ​​upscaling.

[0364] In addition, the first terminal 1620 may additionally perform conventional upscaling on the AI-enlarged image data. For example, due to the state of the network, the first terminal 1620 may receive the AI-encoded image data 1646 of HD and 4Mbps obtained by applying AI downscaling via the 1a-3 DNN 1616. The first terminal 1620 may obtain the AI-enlarged image data 1656 of FHD and 7Mbps by performing AI upscaling via the 2a-3 DNN 1626 jointly trained with the 1a-3 DNN 1616. The first terminal 1620 may perform conventional upscaling on the AI-enlarged image data 1656 of FHD and 7Mbps, and thus may obtain the image data of 4K and 20Mbps.

[0365] In the above example, it is described that the first terminal 1620 receives the image data encoded by AI through the first DNN, but the first terminal 1620 may receive image data (eg, image data 1650) that is not AI-reduced.

[0366] The second terminal 1630 is a terminal that does not support upscaling by a second DNN jointly trained with the first DNN. The second terminal 1630 may not determine whether the image data 1660, 1662, 1664, or 1666 received from the server 1610 is image data that has been AI-reduced by the first DNN, and may process the received image data 1660, 1662, 1664, or 1666. In the case of image data that has been downscaled by the first DNN, the image data may have less quality loss than image data that has been subjected to a general downscaling technique, and therefore, even when the second terminal 1630 does not support upscaling by the second DNN, high-quality image data compared to image data based on related techniques may be provided to the second terminal 1630.

[0367] Fig.1717 is a diagram for describing a method of performing streaming between a server 1710 and a first terminal 1720 according to whether the first terminal 1720 supports AI up-scaling according to an embodiment of the present disclosure.

[0368] exist Fig.17 In the embodiment of the present invention, it is assumed that the first terminal 1720 corresponds to a terminal capable of supporting AI amplification through a second DNN jointly trained with the first DNN of the server 1710, and the second terminal 1730 corresponds to a terminal that does not support AI amplification.

[0369] The server 1710 stores multiple pieces of image data of different qualities for adaptive streaming, and transmits the image data in response to a request from a terminal (e.g., the first terminal 1720). The terminal (e.g., the first terminal 1720) may obtain additional information of the multiple pieces of image data from the server 1710, and may request one piece of image data from the multiple pieces of image data from the server 1710 based on the additional information. Figure 21 to Figure 24 The pieces of information included in the additional information are further described.

[0370] For example, the server 1710 may perform AI downscaling on the original image data 1740 of 8K and 60Mbps through the 1b-1 DNN 1712 to obtain AI downscaled image data 1742 of 4K and 20Mbps. The server 1710 may transmit the AI-encoded image data 1742 of 4K and 20Mbps together with the AI ​​data related to the AI ​​downscaling to 4K and 20Mbps in response to a request from the first terminal 1720. The first terminal 1720 may perform AI upscaling on the received image data through the 2b-1 DNN 1722 jointly trained with the 1b-1 DNN 1712, and thus may obtain AI upscaled image data 1752. In this regard, the first terminal 1720 may perform the above-mentioned AI upscaling by obtaining at least one of information indicating whether AI downscaling has been applied, an AI scaling conversion level, or DNN configuration information used in AI upscaling based on information included in the received AI data.

[0371] As another example, the server 1710 may perform AI downscaling on the AI ​​downscaled image data 1742 of 4K through the 1b-2 DNN 1714, and thus may obtain the AI ​​downscaled image data 1744 of FHD and 7Mbps. The server 1710 may transmit the AI ​​downscaled image data 1744 of FHD and 7Mbps together with the AI ​​data related to the AI ​​downscaling to FHD and 7Mbps in response to the request from the first terminal 1720. For example, while the first terminal 1720 requests and receives the AI-encoded image data 1742 of 4K and 20Mbps as in the above example, when the first terminal 1720 determines that the congestion level of the network has deteriorated, the first terminal 1720 may request the AI-encoded image data 1744 of FHD and 7Mbps. The first terminal 1720 may perform AI upscaling on the image data received in response to the request through the 2b-2 DNN 1724 jointly trained with the 1b-2 DNN 1714, and thus may obtain the AI ​​upscaled image data 1754. As in the above example, the first terminal 1720 may use information included in the AI ​​data in AI amplification.

[0372] As another example, the server 1710 may perform AI downscaling on the AI-encoded image data 1744 of FHD and 7Mbps through the 2b-3 DNN 1716, and thus may obtain the AI ​​downscaling image data 1746 of HD and 4Mbps. The server 1710 may transmit the AI ​​downscaling image data 1746 of HD and 4Mbps together with the AI ​​data related to the AI ​​downscaling to HD and 4Mbps in response to a request from the first terminal 1720. The first terminal 1720 may perform AI upscaling on the received image data through the 2b-3 DNN 1726 jointly trained with the 1b-3 DNN 1716, and thus may obtain the AI ​​upscaling image data 1756. As in the above example, the first terminal 1720 may use the information included in the AI ​​data in the AI ​​upscaling.

[0373] exist Fig.17 , the server 1710 is shown as obtaining the AI ​​downscaled image data 1744 of FHD and 7Mbps through two separate downscaling processes using the 1b-1 DNN 1712 and the 1b-2 DNN 1714, but the embodiment is not limited thereto, and the server 1710 may directly downscale the original image data 1740 of 8K and 60Mbps to the image data 1744 of FHD using a single DNN jointly trained with the corresponding DNN of the first terminal 1720. In addition, the server 1710 may directly downscale the original image data 1740 of 8K and 60Mbps to the image data 1746 of HD using a single DNN jointly trained with the corresponding DNN of the first terminal 1720.

[0374] In the above example, it is described that the first terminal 1720 receives the image data encoded by AI through the first DNN, but the first terminal 1720 may receive image data (eg, image data 1750) that is not AI-reduced.

[0375] Fig.17 The structure of the DNN shown is an example, and at least one of 1b-1 DNN 1712, 1b-2 DNN 1714, or 1b-3 DNN 1716 may be replaced with a conventional scaler. In addition, in association therewith, at least one of 2b-1 DNN 1722, 2b-2 DNN 1724, or 2b-3 DNN 1726 may be replaced with a conventional scaler.

[0376] The second terminal 1730 is a terminal that does not support upscaling by the second DNN jointly trained with the first DNN. The second terminal 1730 may not determine whether the image data 1760, 1762, 1764, or 1766 received from the server 1710 is image data that has been AI-reduced by the first DNN, and may process the received image data 1760, 1762, 1764, or 1766. In the case of image data that has been downscaled by the first DNN, the image data may have less quality loss than image data that has been subjected to a general downscaling technique, and therefore, even when the second terminal 1730 does not support upscaling by the second DNN, high-quality image data compared to image data based on related techniques may be provided to the second terminal 1730.

[0377] Fig.16 and Fig.17 The image data shown are examples, and the multiple items of image data stored at different qualities are not limited to AI-encoded 4K image data, AI-encoded FHD image data, AI-encoded HD image data, and the like. For example, the server 1710 may perform AI reduction on 8K original image data, and thus may store multiple items of AI-encoded image data including 5K (5120×2880) image data, 3K (2560×1440) image data, 540p (960×540) image data, 360p (640×360) image data, and the like. In addition, the server 1710 may store multiple items of image data at different bit rates for image data of a specific resolution. For example, AI-encoded 4K image data may be stored as 4K and 20Mbps image data, 4K and 15Mbps image data, and the like. To this end, the server 1710 may be configured in various ways. Fig.16 and Fig.17That is, a first DNN structure for converting 8K image data into AI-encoded 3K image data, a first DNN structure for converting 3K image data into AI-encoded 540p image data, a first DNN structure for converting 8K and 60Mbps image data into 4K and 15Mbps AI-encoded image data, and the like can be used.

[0378] Fig.18 12 is a diagram for describing a method of streaming image data according to a capability of a terminal 1220, performed by a server 1210, according to an embodiment of the present disclosure.

[0379] In operation S1810, the terminal 1220 may transmit information about the capability to the server 1210. According to an embodiment of the present disclosure, the information about the capability may include at least one of information indicating whether the terminal 1220 can change the quality of the image data requested to adapt to the state of the network, information indicating whether the terminal 1220 can support AI upscaling by the second DNN, or information about an AI upscaling level supportable by the terminal 1220. However, this is an example, and the information about the capability may include information about a codec supported by the terminal 1220.

[0380] In operation S1820, the terminal 1220 may request additional information from the server 1210. For adaptive streaming between the server 1210 and the terminal 1220, it is necessary to check the quality of each of the multiple image data that can be provided from the server 1210 and whether AI encoding has been performed thereon. Therefore, the terminal 1220 may request additional information from the server 1210. Figure 21 to Figure 24 The additional information is further described. Therefore, the terminal 1220 may request the server 1210 for additional information of multiple items of image data.

[0381] In operation S1830, the server 1210 may send the additional information to the terminal 1220. When the server 1210 receives the request from the terminal 1220, the server 1210 may determine the additional information corresponding to the request. For example, the server 1210 may determine the additional information from a plurality of pieces of additional information stored in the server 1210 and requested by the terminal 1220 based on an identifier of the additional information included in the request from the terminal 1220. The additional information may be directly generated by the server 1210, but according to an embodiment of the present disclosure, the additional information may be received from a different server.

[0382] In operation S1840, the terminal 1220 may request the server 1210 for image data of a quality corresponding to the state of the network based on the additional information.

[0383] In this embodiment of the present disclosure, it is assumed that the request message sent from the terminal 1220 to the server 1210 to request image data includes only quality information corresponding to the state of the network. For example, the terminal 1220 may send a request message including information indicating that the resolution of the image data corresponding to the state of the network corresponds to FHD and the bit rate of the image data corresponds to 5Mbps to the server 1210.

[0384] In operation S1850 , the server 1210 may determine image data corresponding to the request based on the capability of the terminal 1220 .

[0385] When requesting image data of a specific quality, server 1210 may determine whether to send image data on which downscaling has been performed. As a result of the determination by server 1210 based on the capability of terminal 1220, when terminal 1220 can support AI upscaling by a second DNN jointly trained with a first DNN, server 1210 may send AI-encoded image data.

[0386] In addition, the server 1210 may determine which image data to send, wherein the image data is AI-reduced to a specific level, based on the AI ​​upscaling level supportable by the terminal 1220. For example, in the case where the terminal 1220 requests FHD and 5Mbps AI-encoded image data, the server 1210 may determine whether to send FHD and 5Mbps image data obtained by performing AI downscaling on 8K and 30Mbps image data via 1a DNN, or FHD and 5Mbps image data obtained by performing AI downscaling on 4K and 10Mbps image data via 1b DNN. The 1a DNN and the 1b DNN may have different DNN configuration information.

[0387] Table 2 below includes values ​​of resolution and bit rate that can be provided from the server 1210 to the terminal 1220. When describing the scaling conversion level with reference to Table 2, the difference between 4K & 20Mbps and 4K & 10Mbps, and the difference between 4K & 10Mbps and FHD & 5Mbps can be defined as one level, respectively. However, this is an example, and the resolution and bit rate supported by the streaming system according to the present disclosure are not limited to the values ​​in Table 2.

[0388] Table 2

[0389]

[0390] In operation S1860, the server 1210 may transmit the AI ​​data and the image data to the terminal 1220 based on the determination. The AI ​​data may include information required for the terminal 1220 to perform AI upscaling on the AI-encoded image data, and may be used together with the reference image data. Figure 5 Provide description accordingly.

[0391] In operation S1870, the terminal 1220 may perform AI upscaling on the received image data through the second DNN jointly trained with the first DNN.

[0392] The terminal 1220 may determine whether to apply AI amplification to the received image data through a second DNN jointly trained with the first DNN based on the AI ​​data. When the AI ​​data includes information indicating that the received image data is AI-encoded image data, the terminal 1220 may perform AI amplification on the received image data through the second DNN. In addition, the AI ​​data may include information about at least one of the AI ​​scaling conversion levels or the DNN configuration information used in the AI ​​amplification. For example, the AI ​​data may include information indicating whether the AI-encoded image data of FHD and 5Mbps is generated by AI reduction of the image data of 8K and 30Mbps via the 1a DNN or by AI reduction of the image data of 4K and 10Mbps via the 1b DNN.

[0393] Fig.19 12 is a diagram for describing a method of streaming image data according to a state of a network and a capability of a terminal 1220, which is performed by a server 1210 according to an embodiment of the present disclosure.

[0394] In operation S1910, the terminal 1220 may transmit information about the capability to the server 1210. According to an embodiment of the present disclosure, the information about the capability may include at least one of information indicating whether the terminal 1220 can change the quality of the image data requested to adapt to the state of the network, information indicating whether the terminal 1220 can support AI upscaling by the second DNN, or information about an AI upscaling level supportable by the terminal 1220. However, this is an example, and the information about the capability may include information about a codec supported by the terminal 1220.

[0395] In operation S1920, the terminal 1220 may request the server 1210 for additional information. The terminal 1220 may request the server 1210 for additional information of the image data. Operation S1920 may be similar to the above-mentioned Fig.18 The operation S1820 is described accordingly.

[0396] In operation S1930, the server 1210 may transmit the additional information to the terminal 1220. The server 1210 may transmit the additional information in response to a request from the terminal 1220. Operation S1930 may be similar to the above-mentioned Fig.18 The operation S1830 is described accordingly.

[0397] In operation S1940, the terminal 1220 may request the server 1210 for image data of a specific quality based on the additional information. For example, the terminal 1220 may request the server 1210 for image data of 8K and 30Mbps according to the user's selection. In addition, the terminal 1220 may include information about the state of the network in the request for the image data of a specific quality, and may send the request. For example, the terminal 1220 may include information about the BER, timestamp, etc. of the previously received image data in the request message, and may send the request message to the server 1210.

[0398] In operation S1950 , the server 1210 may determine image data corresponding to the request based on the state of the network and the capability of the terminal 1220 .

[0399] The server 1210 may determine the quality of the image data corresponding to the state of the network based on the information about the state of the network included in the request received from the terminal 1220. For example, the server 1210 may determine the quality of the image data as 4K and 20 Mbps corresponding to the state of the network.

[0400] As described above in operation S1940, when it is determined that the terminal 1220 requests 8K and 30Mbps image data but the request from the terminal 1220 does not correspond to the state of the network, the server 1210 may determine whether the terminal 1220 supports AI upscaling by the second DNN jointly trained with the first DNN based on the information about the capability of the terminal 1220. As a result of the determination of the server 1210, when the terminal 1220 supports AI upscaling, the server 1210 may determine to transmit 4K and 20Mbps image data generated by performing AI downscaling on the 8K and 30Mbps image data via the first DNN to the terminal 1220.

[0401] In operation S1960, the server 1210 may transmit the AI ​​data and the image data to the terminal 1220 based on the determination. The AI ​​data may include information required for the terminal 1220 to perform AI upscaling on the AI-encoded image data, and may be used together with the reference image data. Fig.12 Provide description accordingly.

[0402] In operation S1970, the terminal 1220 may perform AI upscaling on the received image data through the second DNN jointly trained with the first DNN.

[0403] The terminal 1220 may determine whether to apply AI amplification to the received image data through a second DNN jointly trained with the first DNN based on the AI ​​data. When the AI ​​data includes information indicating that the received image data is AI-encoded image data, the terminal 1220 may perform AI amplification on the received image data through the second DNN. In addition, the AI ​​data may include information about at least one of the AI ​​scaling conversion levels or the DNN configuration information used in the AI ​​amplification. For example, the AI ​​data may include information indicating that AI-encoded image data of 4K and 20Mbps is generated by AI encoding image data of 8K and 30Mbps. As another example, the AI ​​data may include an AI scaling conversion level, and the terminal 1220 may determine the DNN configuration information based on the resolution and bit rate.

[0404] Fig. 20 is a diagram for describing a method of streaming image data corresponding to a state of a network based on additional information and capability, performed by the terminal 1220 according to an embodiment of the present disclosure.

[0405] In operation S2010, the terminal 1220 may transmit information about the capability to the server 1210. According to an embodiment of the present disclosure, the information about the capability may include at least one of information indicating whether the terminal 1220 is capable of changing the quality of the image data requested to adapt to the state of the network, information indicating whether the terminal 1220 is capable of supporting AI upscaling by the second DNN, or information about an AI upscaling level supportable by the terminal 1220. However, this is an example, and the information about the capability may include information about a codec supported by the terminal 1220.

[0406] In operation S2020, the terminal 1220 may request the server 1210 for additional information. The terminal 1220 may request the server 1210 for additional information of the image data. Operation S2020 may be similar to the above-mentioned Fig.18 The description of S1820 corresponds.

[0407] In operation S2030, the server 1210 may transmit the additional information to the terminal 1220. The server 1210 may transmit the additional information to the terminal 1220 in response to a request from the terminal 1220. Operation S2030 may be similar to the above-mentioned operation. Fig.18 The description of S1830 corresponds to that of S1830.

[0408] In operation S2040, the terminal 1220 may confirm image data corresponding to the state of the network and from among the plurality of image data based on the additional information and the capability of the terminal 1220.

[0409] For example, the terminal 1220 may check, based on the additional information, the quality of each of the plurality of image data that can be provided by the server 1210 and whether AI encoding has been performed, and various types of DNN configuration information that can be used to perform AI upscaling on image data of a specific quality. Figure 21 to Figure 24 Describes the additional information in detail.

[0410] The terminal 1220 may determine image data corresponding to the state of the network and from among the plurality of image data checked based on the additional information based on the capability of the terminal 1220. For example, when the quality of the image data corresponding to the state of the network is FHD and 5Mbps, the terminal 1220 may determine one of AI-encoded image data of FHD and 5Mbps generated by performing AI downscaling on image data of 8K and 30Mbps via 1a DNN and AI-encoded image data of FHD and 5Mbps generated by performing AI downscaling on image data of 4K and 10Mbps via 1b DNN based on the capability of the terminal 1220, wherein the image data of 8K and 30Mbps and the image data of 4K and 10Mbps are provided from the server 1210. When the terminal 1220 supports AI upscaling through a 2b DNN jointly trained with the 1b DNN, the terminal 1220 may determine AI-encoded image data of FHD and 5Mbps generated by performing AI downscaling on image data of 4K and 10Mbps via 1b DNN from among the plurality of image data.

[0411] In operation S2050, the terminal 1220 may request the determined image data from the server 1210. The request message sent from the terminal 1220 to the server 1210 to request the determined image data may include an identifier of the determined image data. For example, the request message may include an identifier of the AI-encoded image data of FHD and 5Mbps generated by performing AI downscaling on the image data of 4K and 10Mbps via the 1b DNN.

[0412] The terminal 1220 may request the determined image data from the server 1210 in units of segments. The segments may be generated by segmenting the image data based on time units. When the terminal 1220 requests the determined image data in units of segments, the request message may include not only information about the quality of the determined image data, whether AI encoding has been performed, etc., but also an identifier of the segment. The identifier of the segment may include a segment number, an offset, etc., but this is an example, and therefore the identifier of the segment is not limited to the above example. The segment number represents each of the numbers assigned to the multiple segments included in the image data. In addition, the offset refers to the difference between the preset reference time and the start time of the segment. Here, the preset reference time may be the start time of the first segment or the initial segment among the multiple segments included in the image data.

[0413] In operation S2060, the server 1210 may send AI data and image data corresponding to the request. The AI ​​data may include information required for the terminal 1220 to perform AI upscaling on the AI-encoded image data, and may be used together with the reference Figure 5 Provide description accordingly.

[0414] In operation S2070, the terminal 1220 may perform AI upscaling on the received image data through the second DNN jointly trained with the first DNN.

[0415] The terminal 1220 may perform AI upscaling on the received image data based on at least one of the information about the AI ​​scaling conversion level or the DNN configuration information used in the AI ​​upscaling included in the AI ​​data. The method in which the terminal 1220 performs AI upscaling on the received image data may be the same as that described in reference to Figure 2 Provide description accordingly.

[0416] Fig.21 is a diagram for describing additional information provided for streaming according to an embodiment of the present disclosure.

[0417] Reference Fig.21 , the additional information may hierarchically include an image data set element 2110, an image data element 2120, and a segment element 2130. Each of the above elements 2110, 2120, and 2130 may include a plurality of pieces of information indicating properties of an image data set, image data, and a segment. An image data set may be a group of multiple interchangeable image data. For example, an image data set may be a group of multiple image data generated by encoding a first period of content at different qualities, and may correspond to an adaptation set in MPEG-DASH. A segment may be a portion generated by segmenting image data based on time.

[0418] The image dataset element 2110 may include information about a type 2112 and an identification (ID) 2114 of each of the image datasets. In this regard, the type 2112 may indicate the type of content included in the image dataset, and examples of the type may include image, audio, text, etc. The ID 2114 may include identifiers for identifying image datasets, respectively.

[0419] The image data element 2120 may include an ID 2122 of the image data, quality 2124, AI scaling conversion information 2126, etc. The ID 2122 may include an identifier for identifying the image data, and the quality 2124 may include various attributes including a bit rate, a resolution, etc. The AI ​​scaling conversion information 2126 may also include information about a codec suitable for a second DNN used in the AI ​​upscaling conversion, information about AI upscaling levels available in respective conversions of a plurality of image data, information about a parameter set of the second DNN corresponding to a parameter set used in the first DNN, etc.

[0420] The segment element 2130 may include information about an ID 2132, an offset 2134, and the like of the segment. The ID 2132 may include an identifier for identifying the segment, and the offset 2134 may include information about the position of the segment on the timeline. In the case where the quality of the image data to be received must be changed due to a change in the state of the network, the offset 2134 may be used to synchronize the image data of the previous quality with the image data of the quality to be newly received. For example, when multiple pieces of image data are composed of segments with time offsets of 2ms, 4ms, 6ms, and 8ms, respectively, the terminal may receive segments up to 4ms of image data of 4K and 10Mbps, and thereafter, when the terminal requests image data of FHD and 5Mbps due to a change in the state of the network, the terminal may process the 6ms segment to be reproduced according to synchronization with the image data of 4K and 10Mbps.

[0421] Fig.21 The structure of the additional information shown is an example, and the additional information for adaptive streaming is not limited thereto. As another example, the AI ​​scaling conversion information 2126 may be included in the segment element 2130. As another example, the additional information may additionally include parameter update information, wherein the terminal is able to jointly update the parameters of the first DNN of the server with the parameters of the second DNN through the parameter update information. However, this is an example, and the information for updating the parameters of the second DNN may be provided to the terminal separately from the additional information.

[0422] Fig. 22 is a diagram for describing a detailed configuration of additional information according to an embodiment of the present disclosure.

[0423] Reference Fig. 22 , a "mediadataset" attribute may be defined in the additional information according to an embodiment of the present disclosure. The "mediadataset" attribute indicates an attribute of a media data set consisting of multiple media data of different qualities, and may include an "id" element indicating an identifier of the media data set, a "type" element indicating the type of content, and the like. This embodiment of the present disclosure corresponds to a case where the media data is image data, and the "type" element may be set to video. In the following, a description will be provided assuming that the media data is image data.

[0424] The "mediadataset" attribute defines the attributes of data for each item of multiple image data of different qualities, and may include an "id" element indicating an identifier of the media data, a "resolution" element indicating a resolution, a "bitrate" element indicating a bit rate, an "AIupscale" element indicating whether AI upscaling needs to be applied, etc. The terminal may check whether the image data has been AI downscaled by the first DNN jointly trained with the second DNN of the terminal based on the "AIupscale" element of each image data. The "AIupscale" element may be included in the above-mentioned AI scaling conversion information.

[0425] The terminal can check the attributes of each image data included in the additional information, and can request specific image data based on the attributes. For example, the terminal can request 4K and 10Mbps AI-encoded image data from the server among multiple image data. In this case, the terminal can send a request message including information about id=2 to the server.

[0426] When the terminal receives image data corresponding to the request, the terminal can perform AI amplification on the received image data through the second DNN jointly trained with the first DNN of the server. In this regard, the terminal can obtain DNN configuration information optimized for the second DNN based on information about the resolution, bit rate, etc. of the image data to perform AI amplification on the image data. The DNN configuration information may include information about filter kernels (e.g., the number of convolutional layers, the number of filter kernels according to each convolutional layer, the parameters of each filter kernel, etc.). For example, the terminal may include information that has been jointly trained with the first DNN of the server to indicate that amplification by the second DNN must be performed on 4K and 10Mbps image data by using A DNN configuration information.

[0427] According to an embodiment of the present disclosure, DNN configuration information optimized for the second DNN to perform AI upscaling may vary not only according to the resolution and bit rate of the image data, but also according to the type of content consisting of multiple image data. For example, the terminal may include information that has been jointly trained with the first DNN of the server to indicate that for the sports genre, upscaling by the second DNN must be performed on the AI-encoded image data of 4K and 10Mbps by using the DNN configuration information corresponding to the sports genre and 4K and 10Mbps, and for the drama genre, upscaling by the second DNN must be performed on the AI-encoded image data of 4K and 10Mbps by using the DNN configuration information corresponding to the drama genre and 4K and 10Mbps.

[0428] Fig.23 is a diagram for describing a detailed configuration of additional information according to an embodiment of the present disclosure.

[0429] Reference Fig.23 , the "mediadataset" attribute and the "mediadata" attribute may be defined in the additional information according to the embodiment of the present disclosure. In this embodiment of the present disclosure, no information is provided in accordance with the above reference. Fig. 22 The embodiments of the present disclosure are described above, and the "AI upscale level" element different therefrom will now be described in detail.

[0430] The "AIupscalelevel" element included in the "mediadata" attribute indicates the difference between the AI-encoded image data and the original image data.

[0431] As mentioned above Figure 5 As described, since the AI ​​encoding process according to an embodiment of the present invention is performed based on both resolution and bit rate, information on the difference between AI-encoded image data and original image data can be provided. Fig.23 In the embodiment of the present disclosure, the 8K and 40Mbps image data corresponds to the original image data. In the present embodiment of the present disclosure, the value of the "AIupscalelevel" element may be determined based on the difference between the bit rate and resolution of the 8K and 40Mbps image data (id=n) and the bit rate and resolution of the AI-encoded image data.

[0432] For example, AI-encoded image data with an id of 1 may be generated by performing AI encoding on original image data based on a bit rate of 30 Mbps, and AI-encoded image data with an id of 2 may be generated by performing AI encoding on original image data based on a resolution of 4K and a bit rate of 20 Mbps. The value of the "AIupscalelevel" element according to an embodiment of the present disclosure is an example of a difference in resolution and bit rate between original image data and AI-encoded image data, and is therefore not limited to this example.

[0433] The terminal may select image data on which the terminal may perform AI upscaling from AI-encoded image data of secondary 4K and 20Mbps and AI-encoded image data of tertiary 4K and 10Mbps based on the capabilities of the terminal, and may request the selected image data.

[0434] exist Fig.23 In the embodiment, the image data with id n+1 and the image data with id n+2 correspond to image data obtained by performing reduction using a conventional reducer.

[0435] Fig.24 is a diagram for describing a detailed configuration of additional information according to an embodiment of the present disclosure.

[0436] Reference Fig.24 , the "mediadataset" attribute and the "mediadata" attribute may be defined in the additional information according to the embodiment of the present disclosure. In this embodiment of the present disclosure, no information similar to that in the above reference is provided. Fig. 22 The embodiments of the present disclosure are described above, and the "AI upscale parameter set" element different therefrom will now be described in detail.

[0437] The "AIupscaleparameterset" element included in the "mediadata" attribute may provide information about a plurality of various DNN configuration information that may be used to perform AI upscaling on AI-reduced image data. For example, even for multiple AI-encoded image data of the same quality at 4K and 20Mbps, the number of DNN convolutional layers used for AI upscaling the AI-encoded image data and the size and number of filter kernels may vary, and thus various DNN configuration information may exist.

[0438] When multiple items of AI-encoded image data of the same quality have different DNN configuration information, the terminal can determine one of the multiple items of AI-encoded image data of the same quality based on the capabilities of the terminal. For example, when the AI ​​amplifier of the terminal includes B AI upscale parameter set, the terminal can select AI-encoded image data that can be reconstructed in the corresponding DNN from multiple items of AI-encoded image data of the same quality. As another example, when the AI ​​amplifier of the terminal includes A=AI upscale parameter set with a complex configuration compared to the AI ​​scaler of the above example, the terminal can reconstruct all multiple items of AI-encoded image data of the same quality. In this case, the terminal can select AI-encoded image data that uses relatively less network resources in streaming or can be reconstructed by the terminal into higher quality from multiple items of AI-encoded image data of the same quality according to the configuration.

[0439] The terminal may obtain at least one DNN configuration information from multiple pieces of DNN configuration information based on the hardware specification of the terminal or the codec. For example, the terminal may obtain DNN configuration information corresponding to the terminal and from multiple pieces of DNN configuration information applicable to AI amplification of AI-encoded image data for the same 4K and 20Mbps. Therefore, the terminal may request from the server image data that has been AI-encoded based on the DNN configuration information corresponding to 4K and 20Mbps.

[0440] Fig.25 25 is a diagram for describing AI data 2510 and image data 2520 streamed from a server to a terminal according to an embodiment of the present disclosure.

[0441] Reference Fig.25 In response to a request from the terminal, the server may send AI data 2510 and image data 2520 corresponding to the request to the terminal.

[0442] The AI ​​data 2510 may include information indicating whether AI downscaling has been performed on the image data 2520. In addition, when the image data 2520 has been AI-encoded, the AI ​​data 2510 may include information about at least one of an AI scaling conversion level or DNN configuration information used in AI upscaling. According to an embodiment of the present disclosure, the AI ​​data 2510 may include an AI scaling conversion level, and the DNN configuration information used for AI upscaling may be determined by the terminal based on the resolution and the bit rate.

[0443] The AI ​​data 2510 may correspond to an initial segment of MPEG-DASH, and the terminal may determine whether the image data 2520 has been AI upscaled by a second DNN jointly trained with a first DNN of the server.

[0444] The AI ​​data 2510 may also include other information required for the terminal to decode the image data 2520 in addition to the above information. For example, the AI ​​data 2510 may include information about the type, ID, offset, etc. of the codec.

[0445] The image data 2520 may be composed of a plurality of segments 2522 to 2524. The plurality of segments 2522 to 2524 may be generated by segmenting the image data 2520 based on time. In response to a request from a terminal, the server may transmit the image data 2520 to the terminal in units of segments. Therefore, when the state of the network between the server and the terminal changes, the quality of the image data requested by the terminal to the server may be effectively changed.

[0446] However, the configuration of the AI ​​data 2510 and the image data 2520 is an example, and the configuration of the AI ​​data and the image data for streaming according to an embodiment of the present disclosure is not limited thereto.

[0447] Fig.26 26 is a diagram for describing a streaming system 2600 according to an embodiment of the present disclosure.

[0448] Reference Fig.26 , according to the embodiment of the present disclosure, the streaming system 2600 may include a service server 2610, a plurality of content servers 2622 and 2624 (also referred to as a first content server 2622 and an Nth content server 2624), and a terminal 2630. However, this is an example, and the streaming system 2600 may also include additional elements. For example, the streaming system 2600 may include a service server 2610. The service server 2610 may be provided in a plurality of quantities. In addition, the present embodiment of the present disclosure will now be described with reference to one terminal 2630, but the service server 2610 and the plurality of content servers 2622 and 2624 may stream image data to a plurality of terminals.

[0449] The service server 2610 may provide the terminal 2630 with additional information of the multiple image data to allow the terminal 2630 to request image data corresponding to the state of the network and from the multiple image data. The additional information may include the respective quality of the multiple image data, whether AI encoding has been performed, various DNN configuration information that can be used to perform AI amplification on image data of a specific quality, etc. In addition, the additional information may include location information on the network where the multiple image data are stored. For example, the additional information may include a uniform resource identifier (URI) of the first content server 2622 or the Nth content server 2624. In addition, as described above with reference to Fig.21 As described above, when the plurality of image data are divided into segments, the additional information may include a URI of each segment.

[0450] The terminal 2630 may request image data corresponding to the state of the network and from multiple image data based on the additional information. For example, according to the result of the determination based on the state of the network, when the terminal 2630 determines to request FHD and 5Mbps image data generated by performing AI reduction on 8K and 30Mbps image data in multiple image data via 1a DNN, the terminal 2630 may obtain URI information of the determined image data based on the additional information. In this embodiment of the present disclosure, the FHD and 5Mbps image data generated by performing AI reduction on 8K and 30Mbps image data via 1aDNN may be stored in the first content server 2622. The terminal 2630 may request the determined image data from the first content server 2622 based on the URI information. When receiving the request, the first content server 2622 may send the AI ​​data and image data corresponding to the request to the terminal 2630. However, this is an example, and the terminal 2630 may request the determined image data from the first content server 2622 in units of segments.

[0451] The plurality of content servers 2622 and 2624 may provide information about content to the service server 2610. The content server (e.g., content server 2622) may provide information about content stored in the content server (e.g., content server 2622) or information about newly added content to the service server 2610. The service server 2610 may generate or update additional information based on the plurality of pieces of information about content provided from the plurality of content servers 2622 and 2624.

[0452] Reference above Fig.26 The streaming system 2600 described is an example, and the system for performing streaming according to the present disclosure is not limited thereto. For example, at least one cache server for transmitting image data and AI data may be further provided between the terminal 2630 and the plurality of content servers 2622 and 2624. As another example, the streaming system 2600 may further include a source server for providing content to each of the plurality of content servers 2622 and 2624.

[0453] Fig. 27 is a block diagram showing a configuration of a server 2700 according to an embodiment of the present disclosure.

[0454] Reference Fig. 27 , the server 2700 according to an embodiment of the present disclosure may include a communication interface 2710, a processor 2720, and a memory 2730. However, this is an example, and the server 2700 may further include other elements. For example, the server 2700 may include a plurality of processors.

[0455] The communication interface 2710 according to an embodiment of the present disclosure may provide an interface for communicating with another device (eg, a terminal). The communication interface 2710 may receive a request for additional information or image data from the terminal. In addition, the communication interface 2710 may send additional information or media to the terminal.

[0456] The processor 2720 according to an embodiment of the present disclosure may generally control the server 2700 to execute one or more programs stored in the memory 2730 to perform the same operations as described above with reference to FIG. Figures 1 to 11 The operations related to the image encoding device described above, and Figures 12 to 26 Describes server-related operations.

[0457] The memory 2730 according to an embodiment of the present disclosure may store various data, programs, or applications for driving and controlling the server 2700. Each of the one or more programs stored in the memory 2730 may include one or more instructions. Each program (one or more instructions) or each application stored in the memory 2730 may be executed by the processor 2720.

[0458] Fig.28 is a block diagram showing a configuration of a terminal 2800 according to an embodiment of the present disclosure.

[0459] Reference Fig.28 According to an embodiment of the present disclosure, the terminal 2800 may include a communication interface 2810, a processor 2820, and a memory 2830. However, this is an example, and the terminal 2800 may also include other elements. For example, the terminal 2800 may include a plurality of processors including a central processing unit (CPU), a graphics processing unit (GPU), a neutral processing unit (NPU), etc.

[0460] The communication interface 2810 according to an embodiment of the present disclosure may provide an interface for communicating with another device (e.g., a server). The communication interface 2810 may send a request for additional information or image data to the server. In addition, the communication interface 2810 may receive additional information or media from the server, and may output the received additional information or media to the processor 2820.

[0461] The processor 2820 according to an embodiment of the present disclosure may generally control the terminal 2800 to execute one or more programs stored in the memory 2830 to perform the above-mentioned Figures 1 to 11 The operations related to the image decoding device described above, and Figures 12 to 26 Describes terminal-related operations.

[0462] For example, the processor 2820 may send a request for additional information of multiple pieces of image data of different qualities to the server via the communication interface 2810. In response to the request, the processor 2820 may obtain the additional information from the server via the communication interface 2810.

[0463] The processor 2820 may send a request for predefined image data among the plurality of image data to the server via the communication interface 2810 based on the additional information. When the processor 2820 obtains the image data and the AI ​​data corresponding to the request, the processor 2820 may determine whether to perform AI upscaling on the received image data based on the AI ​​data. Based on the result of determining whether to perform AI upscaling, the processor 2820 may perform AI upscaling on the received image data through a DNN for upscaling jointly trained with a DNN for downscaling of the server.

[0464] The processor 2820 may confirm the state of the network based on the BER or timestamp of the image data received from the server via the communication interface 2810. Based on the confirmed state of the network, the processor 2820 may send a request for image data of different qualities among the multiple image data to the server via the communication interface 2810 based on the additional information. The processor 2820 may obtain the image data and AI data corresponding to the request.

[0465] When the terminal 2800 includes a plurality of processors, each of the processors may perform at least some of the operations of the processor 2820. For example, the CPU may confirm the state of the network and may request image data corresponding thereto. The NPU may perform AI upscaling on the AI-encoded image data, and the GPU may perform processing other than the processing performed by the NPU, including processing in reference to Figure 2 In the AI ​​decoding process described above, the GPU may support the NPU to perform processing in order to accelerate the processing performed by the NPU. However, this is an example, and the operations to be performed by the processor are not limited to the above examples.

[0466] The memory 2830 according to an embodiment of the present disclosure may store various data, programs or applications for driving and controlling the terminal 2800. Each of the one or more programs stored in the memory 2830 may include one or more instructions. Each program (one or more instructions) or each application stored in the memory 2830 may be executed by the processor 2820.

[0467] According to the actual specifications of the device, the elements in the block diagram can be combined, elements can be added thereto, or at least one element can be omitted. That is, when necessary, at least two elements can be combined into one element, or one element can be divided into two elements. In addition, the functions performed by each element are used to describe the embodiments of the present disclosure, and the detailed operations or devices do not limit the scope of the present disclosure.

[0468] The aforementioned embodiments of the present disclosure may be written as a computer-executable program that may be stored in a medium.

[0469] The medium can store the computer executable program continuously, or can temporarily store the computer executable program for execution or downloading. In addition, the medium can be any of various recording media or storage media combined with single or multiple pieces of hardware, and the medium is not limited to the medium directly connected to the computer system, but can be distributed through a network. Examples of media include magnetic media (such as hard disks, floppy disks and tapes) configured to store program instructions, optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as optical floppy disks), and ROM, RAM and flash memory. Other examples of media include recording media and storage media, servers, etc. managed by application stores or websites that distribute applications to provide or distribute various other types of software.

[0470] The model related to the above-mentioned DNN can be implemented as a software module. When the DNN model is implemented as a software module (e.g., a program module including instructions), the DNN model can be stored in a computer-readable recording medium.

[0471] In addition, the DNN model can be a part of at least one of the above-mentioned image decoding device, image encoding device, server or terminal by being integrated into a hardware chip. For example, the DNN model can be manufactured as a dedicated hardware chip for AI, or can be manufactured as a part of an existing general-purpose processor (e.g., CPU or AP) or a graphics-specific processor (e.g., GPU).

[0472] In addition, the DNN model can be provided as downloadable software. The computer program product can include a product (e.g., a downloadable application) that is a software program that is electronically distributed through a manufacturer or an electronic market. For electronic distribution, at least a portion of the software program can be stored in a storage medium or can be temporarily generated. In this case, the storage medium can be a server of the manufacturer or electronic market, or a storage medium of a forwarding server.

[0473] The method and apparatus for streaming data according to an embodiment of the present disclosure may transceive image data AI-encoded by using a DNN based on the state of a network, and thus may always maintain QoS of reproduction of image data under a variable state of the network.

[0474] Effects that can be achieved by the method and apparatus for streaming data according to the embodiments of the present disclosure are not limited to the above-mentioned features, and other unstated effects will be clearly understood by those of ordinary skill in the art in view of the following description.

[0475] Although one or more embodiments of the present disclosure have been described with reference to the drawings, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope defined by the following claims.

Claims

1. A method for streaming data, the method comprising: include: receiving information of a plurality of image data of different qualities; Based on the information, requesting the server to send image data from among the plurality of image data of different qualities to the terminal; receiving image data and artificial intelligence (AI) data corresponding to the request, wherein the AI ​​data is related to AI downscaling of an original image corresponding to the image data into a first image by a downscaling deep neural network (DNN) of the server, the first image being obtained based on the downscaling DNN set with first DNN setting information selected from a plurality of DNN setting information for AI downscaling, and the AI ​​data including an identifier indicating second DNN setting information for AI upscaling; determining whether to perform AI upscaling on the received image data based on the AI ​​data; Based on the result of the determining whether to perform AI upscaling, selecting second DNN setting information from a plurality of DNN setting information for AI upscaling based on the AI ​​data, and performing AI upscaling on the received image data through an upscaling DNN jointly trained with the downscaling DNN and set with the second DNN setting information; confirming a status of a network between the terminal and the server; and requesting the server to send image data of different qualities among the plurality of image data of different qualities according to information of the plurality of image data of different qualities and the state of the network, Among them, the multiple DNN setting information for AI reduction and the multiple DNN setting information for AI enlargement are obtained by joint training of the reduced DNN and the enlarged DNN.

2. The method according to claim 1, further comprising: include: determining, based on the AI ​​data, whether AI downscaling has been performed on the received image data by the downscaling DNN of the server, and The step of determining whether to perform AI upscaling on the received image data includes: when it is confirmed that AI downscaling has been performed on the received image data, determining to perform AI upscaling on the received image data.

3. The method according to claim 1, in, The information of the plurality of image data includes quality information and AI scaling conversion information of the plurality of image data, and The step of requesting the server to send image data of different qualities includes: requesting image data corresponding to the state of the network based on one or both of the quality information and the AI ​​scaling conversion information.

4. The method according to claim 1, further comprising: include: Image data corresponding to the state of the network is determined based on capability information including information indicating whether the terminal supports AI up-scaling and information on an AI up-scaling level that the terminal can support.

5. The method according to claim 1, in, The server is a content provider server, The method further comprises requesting information of the plurality of image data from a service server, and The step of receiving the information of the plurality of image data includes: receiving the information of the plurality of image data and an identifier of the content provider server from the service server.

6. A method for streaming data, the method comprising: include: receiving, from a terminal, a request for image data from a server among a plurality of image data of different qualities; In response to the request, sending artificial intelligence AI data and image data that has been AI-encoded by a reduction DNN of the server jointly trained with an enlarged DNN of the terminal to the terminal, wherein the AI ​​data is related to AI reduction of an original image corresponding to the image data to a first image by the reduction DNN and is used to select second DNN setting information from a plurality of DNN setting information for AI enlargement, the second DNN setting information is used to set an enlarged DNN, the enlarged DNN set with the second DNN setting information is used to AI enlarge the AI-encoded image data, the first image is obtained based on the reduction DNN set with the first DNN setting information selected from a plurality of DNN setting information for AI reduction, and the AI ​​data includes an identifier indicating the second DNN setting information for AI enlargement; and receiving, from the terminal, a request for image data of different qualities among the plurality of image data of different qualities according to a state of a network between the terminal and the server, Among them, the multiple DNN setting information for AI reduction and the multiple DNN setting information for AI enlargement are obtained by joint training of the reduced DNN and the enlarged DNN.

7. A terminal for streaming data, the terminal include: a memory storing one or more instructions; as well as at least one processor configured to execute the one or more instructions to: receiving information of a plurality of image data of different qualities; Based on the information, requesting the server to send image data from among the plurality of image data of different qualities to the terminal; Receiving requested image data and artificial intelligence (AI) data corresponding to the requested image data, wherein the AI ​​data is related to AI-scaling an original image corresponding to the image data into a first image by downscaling a deep neural network (DNN) and includes an identifier indicating second DNN setting information for AI upscaling, the second DNN setting information being used to set an upscaling DNN, the upscaling DNN set with the second DNN setting information being used to AI upscaling the AI-encoded image data, the first image being obtained based on the downscaling DNN set with the first DNN setting information selected from a plurality of DNN setting information for AI downscaling; determining whether to perform AI upscaling on the received image data based on the AI ​​data; Based on the result of determining whether to perform AI upscaling, selecting second DNN setting information from a plurality of DNN setting information for AI upscaling based on the AI ​​data, and performing AI upscaling on the received image data through the upscaling DNN of the terminal that is jointly trained with the downscaling DNN and set with the second DNN setting information; confirming a status of a network between the terminal and the server; and requesting the server to send image data of different qualities among the plurality of image data of different qualities according to information of the plurality of image data of different qualities and the state of the network, Among them, the multiple DNN setting information for AI reduction and the multiple DNN setting information for AI enlargement are obtained by joint training of the reduced DNN and the enlarged DNN.

8. The terminal according to claim 7, in, The at least one processor is further configured to execute the one or more instructions to: determining, based on the AI ​​data, whether AI downscaling has been performed on the received image data by the downscaling DNN of the server, and When it is confirmed that AI reduction has been performed on the received image data, it is determined to perform AI enlargement on the received image data.

9. The terminal according to claim 7, in, The information of the plurality of image data includes quality information and AI scaling conversion information of the plurality of image data, and The at least one processor is further configured to execute the one or more instructions to request image data corresponding to the status of the network based on one or both of the quality information and the AI ​​scaling conversion information.

10. The terminal according to claim 7, in, The at least one processor is further configured to execute the one or more instructions to: Image data corresponding to the state of the network is determined based on capability information including information indicating whether the terminal supports AI up-scaling and information on an AI up-scaling level supported by the terminal.

11. The terminal according to claim 10, in, The server is a content provider server, and The at least one processor is further configured to execute the one or more instructions to request a service server to provide the terminal with information of the plurality of image data, and receive an identifier of the content provider server and the information of the plurality of image data from the service server.

12. A server for streaming data, the server include: a memory storing one or more instructions; as well as at least one processor configured to execute the one or more instructions to: receiving a request for image data from a plurality of image data of different qualities from a server from a terminal, In response to the request, sending artificial intelligence AI data and image data that has been AI-encoded by a reduction DNN jointly trained with an enlarged deep neural network DNN of the terminal to the terminal, wherein the AI ​​data is related to AI reduction of an original image corresponding to the image data to a first image by the reduction DNN and includes an identifier indicating second DNN setting information for AI enlargement, the second DNN setting information is used to set the enlarged DNN, the enlarged DNN set with the second DNN setting information is used to AI enlarge the AI-encoded image data, and the first image is obtained based on the reduction DNN set with the first DNN setting information selected from a plurality of DNN setting information for AI reduction, and receiving, from the terminal, a request for image data of different qualities among the plurality of image data of different qualities according to a state of a network between the terminal and the server, Among them, the multiple DNN setting information for AI reduction and the multiple DNN setting information for AI enlargement are obtained by joint training of the reduced DNN and the enlarged DNN.

13. The server according to claim 12, in, The AI ​​data includes information about the downsized DNN that has been applied to the AI-encoded image data.

14. The server according to claim 12, in, The at least one processor is also configured to execute the one or more instructions to receive a request for a second version of the image content, wherein the second version of the image content corresponds to the state of the network and is determined based on one or both of AI scaling conversion information and quality information of each version of the image content in multiple different versions of the image content.

15. The server according to claim 12, in, The different quality image data includes image data corresponding to a state of the network and determined based on capability information including information indicating whether the terminal supports AI upscaling and information on an AI upscaling level supported by the terminal.

Images