Electronic device, system and control method thereof
By using a trained AI model to downscale and encode images, and combining image quality and network status information, image parameters are dynamically adjusted, solving the problems of image quality and latency in real-time live streaming and improving the user experience.
Patent Information
- Application Number
- CN202010273164.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-11-29
- Filing Date
- 2020-04-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-04-09
AI Technical Summary
In real-time live streaming applications, existing technologies struggle to effectively optimize image quality and reduce latency in different network environments, leading to a perceived decrease in quality for users.
The image is downscaled and encoded using a trained first artificial intelligence (AI) model, and image quality information is obtained by combining it with a trained second AI model. The downscaling ratio and bit rate of the image are dynamically adjusted through multiple quality identifiers and network state information to optimize image quality and the encoding process.
It optimizes image quality and reduces latency under different network environments, improves the user's perceived quality, and effectively utilizes network bandwidth.
Smart Images

Figure CN112887739B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an electronic device and a control method thereof, and, for example, to an electronic device, a system and a control method thereof that perform image processing using an artificial intelligence (AI) model. Background Art
[0002] With the development of electronic technology, various types of electronic devices have been developed and released. In recent years, display devices used in various places such as homes, offices, public places, etc. have been increasingly developed.
[0003] With the cord-cutting trend of reducing traditional pay broadcasting (such as terrestrial / cable / IPTV / satellite broadcasting), the increasing proliferation of over-the-top (OTT) services through public Internet networks (such as Netflix / Amazon Prime), and the proliferation of single media broadcasting platforms (such as YouTube / Twitter / Facebook, etc.), consumer demand for real-time live streaming applications is increasing.
[0004] OTT service providers provide live streaming services for real-time important content such as live broadcasts or sports events, and in the future, are preparing for expansion of services that require interaction with viewers such as live betting. Single-person media broadcasting platforms are providing a service that enables interaction with viewers through a chat window and are evolving into platforms that enable multi-party video calls / video conferences.
[0005] In this real-time live streaming application, very important factors that affect the viewer's perceived quality are the quality of the image and the delay. For example, since services such as supply chain (live event> capture> encoding> first mile delivery> CDN propagation> last mile delivery> user device) are provided using a public Internet network, streaming should be performed with the optimal image quality in a given network environment, and since the service emphasizes real-time performance, the delay felt by consumers should be minimized or reduced. For example, services such as sporting events should minimize or reduce spoiler effects by making the delay less than or equal to one second, and in interactive services such as video calls, the service must be provided with a delay of at least less than or equal to 200 milliseconds so that consumers do not feel the delay.
[0006] Due to the proliferation of services via the Internet and the Internet of Things (IoT) / all-IP trend, communication modules (such as Wi-Fi capable of Internet Protocol (IP) communication) are provided not only for mobile devices (such as mobile phones or tablets) but also for laptops, smart TVs, etc. In homes, application scenarios for direct streaming between devices are being developed.
[0007] Screen mirroring is a representative example of an inter-device direct streaming application scenario, where content that has been reproduced on a relatively small screen (such as a mobile phone and a tablet) is sent to a large screen (such as a TV for viewing). In the TV industry, bulky components (such as power components, cable connectors, etc.) are divided into an external box to maintain a thin display form factor. Here, a wireless communication module can be used to replace the complex cable connection between the external box and the display with direct data streaming.
[0008] When comparing the data path in direct streaming applications between devices and the supply chain of real-time live streaming applications over the Internet, the technical similarity of the end systems is very high only in different network environments, and the viewer's requirements for perceived quality are relatively high. For example, the image quality of wireless TV scenes should be close to visually lossless, and the delay should be within 10 milliseconds to prevent or reduce the user's perceived quality degradation. Summary of the invention
[0009] The present disclosure provides an electronic device, a system and a control method thereof that are capable of predicting the quality of an image being streamed in real time and processing the image at an optimal and / or improved resolution and bit rate.
[0010] Embodiments of the present disclosure can solve the above-mentioned problems, and can provide an electronic device, comprising: a processor, configured to downscale an image and encode the downscaled image using a trained first artificial intelligence (AI) model, and the processor is configured to control the downscaling of the image based on quality information of the image obtained using a trained second AI model, and the second AI model can be trained using feature information of the image obtained from the first AI model.
[0011] According to an example embodiment, an electronic device includes: a processor configured to downscale an image using a trained first artificial intelligence (AI) model and encode the downscaled image. Here, the processor may control downscaling the image based on quality information of the image obtained using a trained second AI model, and the second AI model may be trained using feature information of the image obtained from the first AI model.
[0012] According to example embodiments, the processor may control encoding of the downscaled image based on quality information of the image obtained using the second AI model.
[0013] According to an example embodiment, the quality information of the image includes a plurality of quality identifiers corresponding to different combinations of resolution information and bit rate information related to the image. The processor may determine a downscaling ratio of the image and a bit rate of the downscaled image based on the resolution information and the bit rate information corresponding to any one of the plurality of quality identifiers, and control downscaling the image based on the determined downscaling ratio, and control encoding the downscaled image based on the determined bit rate.
[0014] According to an example embodiment, the electronic device may include: a communication interface including a communication circuit. The processor may obtain state information of a network to which the communication interface is connected, and control downscaling of the image based on the state information of the network and quality information of the image.
[0015] According to an example embodiment, the quality information of the image includes a plurality of quality identifiers corresponding to different combinations of information about a plurality of resolutions and information about a plurality of bit rates associated with the image. The processor may determine at least one quality identifier of the plurality of quality identifiers based on state information of the network, determine one quality identifier of the at least one quality identifier based on a target quality identifier, and control downscaling of the image based on resolution information corresponding to the determined one quality identifier, and control encoding of the downscaled image based on information about the bit rate corresponding to the determined one quality identifier.
[0016] According to example embodiments, the second AI model includes at least one neural network layer, and parameters of the at least one neural network layer may be trained together with parameters of at least some layers included in the first AI model.
[0017] According to an example embodiment, the processor may control downscaling of the current frame based on quality information obtained from the current frame of the image, or control downscaling of the current frame based on quality information predicted from at least one previous frame of the image.
[0018] According to an example embodiment, the second AI model includes at least one recurrent neural network (RNN) layer trained to predict quality information of a current frame based on feature information corresponding to at least one previous frame.
[0019] According to an example embodiment, the quality information of an image includes multiple quality identifiers corresponding to different combinations of information about multiple resolutions and information about multiple bit rates related to the image, and the second AI model can be trained based on supervised training using the difference between the multiple quality identifiers corresponding to different combinations of information about multiple resolutions and information about multiple bit rates related to the training image and the multiple quality identifiers output by inputting the training image into the second AI model as a loss function.
[0020] According to an example embodiment, the processor may determine a downscaling ratio of an image and a bit rate of the downscaled image based on quality information of the image, downscale the image based on the determined downscaling ratio, encode the downscaled image based on the determined bit rate, and generate a data packet based on the encoded image and send the generated data packet, and the resolution information and the bit rate information are included in a header of the data packet.
[0021] According to an example embodiment, a method for controlling an electronic device, the method comprising: downscaling an image using a trained first artificial intelligence (AI) model; and encoding the downscaled image, wherein the step of downscaling the image comprises controlling the downscaling of the image based on quality information of the image obtained using a trained second AI model, and wherein the second AI model is trained using feature information of the image obtained from the first AI model.
[0022] According to an example embodiment, encoding the down-scaled image includes controlling encoding the down-scaled image based on quality information of the image obtained using the second AI model.
[0023] According to an example embodiment, the quality information of the image includes a plurality of quality identifiers corresponding to different combinations of resolution information and bit rate information associated with the image, the step of downscaling the image includes determining a downscaling ratio of the image based on any one of the plurality of quality identifiers and controlling the downscaling of the image based on the determined downscaling ratio, and the step of encoding the downscaled image includes determining a bit rate of the downscaled image based on any one of the quality identifiers and controlling the encoding of the downscaled image based on the determined bit rate.
[0024] According to an example embodiment, the method further comprises: obtaining status information of a network, wherein the step of downscaling the image comprises controlling the downscaling of the image based on the status information of the network and quality information of the image, and wherein the step of encoding the downscaled image comprises controlling the encoding of the downscaled image based on the status information of the network and quality information of the image.
[0025] According to an example embodiment, the quality information of the image includes a plurality of quality identifiers corresponding to different combinations of information about a plurality of resolutions and information about a plurality of bit rates associated with the image, and the method further includes: determining at least one quality identifier of the plurality of quality identifiers based on state information of a network, and determining any one quality identifier of the at least one quality identifier based on a target quality identifier, wherein the step of downscaling the image includes controlling the downscaling of the image based on the resolution information corresponding to the determined any one quality identifier, and wherein the step of encoding the downscaled image includes controlling the encoding of the downscaled image based on the information about the bit rate corresponding to the determined any one quality identifier.
[0026] According to an example embodiment, the second AI model includes at least one neural network layer, and wherein parameters of the at least one neural network layer are trained together with parameters of at least some layers included in the first AI model.
[0027] According to an example embodiment, downscaling the image comprises controlling downscaling the current frame based on quality information obtained from the current frame of the image; or controlling downscaling the current frame based on quality information predicted from at least one previous frame of the image.
[0028] According to an example embodiment, a system including a transmitting device and a receiving device includes: a transmitting device including a circuit configured to downscale an image using a trained first artificial intelligence (AI) model and encode the downscaled image; and a receiving device including a circuit configured to decode an image received from the transmitting device and obtain an output image by upscaling the decoded image, wherein the transmitting device is configured to control downscaling of the image based on quality information of the image obtained using a trained second AI model, and the second AI model is configured to be trained using feature information of the image obtained from the first AI model.
[0029] According to an example embodiment, the transmitting device is configured to control encoding of the downscaled image based on quality information of the image obtained using the second AI model.
[0030] According to an example embodiment, the received image includes at least one of bit rate information or resolution information, and the receiving device is configured to: decode the encoded image based on the bit rate information included in the received image, and up-scale the decoded image based on the resolution information included in the received image. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The above and other aspects and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:
[0032] Figure 1A is a diagram illustrating an example image processing method based on an artificial intelligence neural network according to an embodiment;
[0033] Figure 1B is a diagram illustrating an example image processing method based on an artificial intelligence neural network according to an embodiment;
[0034] Figure 1C is a diagram illustrating an example image processing method based on an artificial intelligence neural network according to an embodiment;
[0035] Figure 2A is a block diagram showing an example configuration of an electronic device according to an embodiment;
[0036] Figure 2B is a block diagram showing an example configuration of an electronic device according to an embodiment;
[0037] Figure 3 is a block diagram illustrating an example operation of a processor according to an embodiment;
[0038] Figure 4 is a diagram illustrating an example relationship between a first artificial intelligence model and a second artificial intelligence model according to an embodiment;
[0039] Figure 5 is a diagram illustrating an example learning method of a second artificial intelligence model according to an embodiment;
[0040] Figure 6 is a diagram illustrating an example learning method of a second artificial intelligence model according to an embodiment;
[0041] Figure 7 is a diagram illustrating an example learning method of a second artificial intelligence model according to an embodiment;
[0042] Fig. 8A is a diagram illustrating an example of using quality information of a current frame or a previous frame according to various embodiments;
[0043] Figure 8B is a diagram illustrating an example of using quality information of a current frame or a previous frame according to various embodiments;
[0044] Fig. 9 is a block diagram showing an example configuration of an electronic device according to an embodiment;
[0045] Fig. 10A is a diagram illustrating an example upscaling method using an artificial intelligence model according to an embodiment;
[0046] Fig. 10Bis a diagram illustrating an example upscaling method using an artificial intelligence model according to an embodiment;
[0047] Fig. 10C is a diagram illustrating an example upscaling method using an artificial intelligence model according to an embodiment;
[0048] Fig.11A is a diagram illustrating an example electronic device and an example receiving device according to an embodiment;
[0049] Fig. 11B is a block diagram illustrating an example electronic device and an example receiving device according to an embodiment;
[0050] Fig.12 is a diagram illustrating an example operation of a transmission and reception system according to an embodiment;
[0051] Fig.13 is a diagram illustrating example quality information according to an embodiment;
[0052] Fig.14 is a flow chart illustrating an example method for controlling an electronic device according to an embodiment;
[0053] Fig.15 is a diagram illustrating an example AI encoding process and an example AI decoding process according to an embodiment;
[0054] Fig.16 is a block diagram showing an example configuration of an example AI decoding device according to an embodiment;
[0055] Fig.17 is a diagram illustrating an example second deep neural network (DNN) for AI upscaling according to an embodiment;
[0056] Fig.18 is a diagram illustrating an example convolution operation through a convolutional layer according to an embodiment;
[0057] Fig.19 is a diagram showing an example mapping relationship between various image-related information and DNN configuration information according to an embodiment;
[0058] Fig. 20 is a diagram illustrating an example second image including a plurality of frames according to an embodiment;
[0059] Fig.21 is a block diagram showing an example configuration of an AI encoding device according to an embodiment;
[0060] Fig. 22 is a diagram illustrating a first DNN for AI downscaling an image according to an embodiment;
[0061] Fig.23is a diagram illustrating an example method for training a first DNN and a second DNN according to an embodiment;
[0062] Fig.24 is a signal flow diagram illustrating an example training process of a first DNN and a second DNN by a training apparatus according to an embodiment; and
[0063] Fig.25 is a block diagram illustrating an example device for AI downscaling and an example device for AI upscaling according to an embodiment.
[0064] The same reference numbers may be used throughout the drawings to refer to the same (or similar) elements. DETAILED DESCRIPTION
[0065] The present disclosure may include various modifications and embodiments, and specific embodiments of the present disclosure have been shown by way of example in the accompanying drawings and are described in more detail herein. However, it should be understood that the present disclosure is not intended to be limited to the embodiments of the present disclosure, but is intended to cover all modifications, equivalents and alternatives that fall within the spirit and scope of the various embodiments and the present disclosure.
[0066] In the following description, when it is determined that the subject matter of the present disclosure may make the main points of the present disclosure obscure, detailed description of the prior art may be omitted.In addition, the numbers (e.g., first, second, etc.) used in the description herein are merely identifiers for distinguishing one component from another.
[0067] Furthermore, when an element is referred to herein as being “connected to” or “connected to” another element, it should be understood that the element may be directly connected to or connected to another element, but unless otherwise specified, the element may be connected or connected via another element therebetween.
[0068] In this disclosure, “ Elements represented by "part (unit)", "module" and the like may be divided into two or more components, or one component may be divided into two or more components for each more specific function. In addition, each of the components described below may perform some or all of the functions that other components are responsible for in addition to the main functions that these components are responsible for, and it is understood that some of the functions that each of the components is responsible for may be performed by other components.
[0069] In the present disclosure, an “image” or a “picture” may mean a still image, a moving picture including a plurality of consecutive still images (or frames), or a video.
[0070] In addition, the term "deep neural network (DNN)" is a non-limiting illustrative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model that uses a specific algorithm.
[0071] In the present disclosure, "parameters" may refer to, for example, values used in the operation process of each layer forming a neural network, and may include, for example, weight values used when applying an input value to a predetermined equation. In addition, the parameters may be expressed in a matrix form. The parameters may be set as a result of training, and may be updated via separate training data as needed.
[0072] In the present disclosure, a “first deep neural network (DNN)” may refer to, for example, a DNN for performing AI downscaling on an image, and a “second DNN” may refer to, for example, a DNN for performing AI upscaling on an image.
[0073] In the present disclosure, "DNN configuration information" may refer to, for example, information related to components forming a DNN, and may include the above-mentioned parameters. Using the DNN configuration information, the first DNN or the second DNN may be set.
[0074] In the present disclosure, "image" may refer to, for example, an AI-encoded image, and "first image" may refer to, for example, an image acquired as a result of AI downscaling in an AI encoding process. In addition, "second image" may refer to, for example, an image acquired by a first decoding process during an AI decoding process, and "third image" may refer to, for example, an image acquired by AI upscaling the second image in an AI decoding process.
[0075] In the present disclosure, "AI downscaling" may refer to, for example, a process of reducing the resolution of an image based on, for example, AI, and "first encoding" may refer to, for example, an encoding process by an image encoding method based on frequency conversion. "First decoding" may refer to, for example, a decoding process by an image reconstruction method based on frequency conversion, and "AI upscaling" may refer to, for example, a process of increasing the resolution of an image based on, for example, AI.
[0076] Figure 1A , Figure 1B and Figure 1C is a diagram illustrating an example image processing method based on an artificial intelligence (AI) neural network according to an embodiment.
[0077] In order to stream high-definition / high-resolution images (such as 4K and 8K) over a network, image encoding techniques and upscaling / downscaling techniques that can reduce network request bandwidth are important. For image encoding techniques, standard codecs (such as H.264 / 265AVC, VP8 / 9, and AV1 algorithms) are widely used, and for OTT service providers, for example, for H.265, 4K can be compressed to about 15Mbps for service. In order to provide services to each user according to different network environments, the technology used to compress with various combinations of image resolution and bit rate is upscaling / downscaling technology. For example, when an 8K image is to be sent at a level of about 15Mbps, the transmitting end 10 can perform downscaling of the image to 4K, and the image can be sent to the codec encoder, and the receiving end 20 can perform upscaling to restore the 4K output of the codec encoder to an 8K image. Although simple interpolation (such as bilinear or bicubic) is used in upscaling / downscaling, recent studies have shown that upscaling / downscaling can further improve the perceived quality of consumers by using, for example, a convolutional neural network (CNN). For example, the method is easily compatible with any compression codec and thus can be easily scaled by being applied to the currently widely used H.265 / VP9 standard codec.
[0078] Generally, compression efficiency varies depending on the image resolution and image type. For example, if two images with different characteristics are compressed at the same resolution and bit rate, then Figure 1B As shown in , the consumer's perceived quality degradation is more likely to be more sensitive to the case of a relatively complex lower image. On the other hand, in the example of a relatively less complex upper image, the compression efficiency is good, so sending an image with a lower bit rate can reduce unnecessary wasted network bandwidth.
[0079] like Figure 1C As shown in the above graph of , since increasing the bit rate helps improve the user's perceived quality as the resolution increases, the resolution is also considered when optimizing the bit rate according to the type of image. Figure 1C As shown in the following graph, content providers find the optimum points for various resolution and bit rate combinations to encode in order to provide service with the best quality for each user's given network environment.
[0080] In the example of a video on demand (VoD) service, the images are pre-encoded and stored in the server used to provide the service, and the image quality can be optimized offline by investing sufficient time and computing power, but in real-time live streaming applications, this approach may not be used due to latency and computing resource limitations.
[0081] In the following, various embodiments for providing content with optimized and / or improved quality in real-time streaming of content will be described in more detail.
[0082] Figure 2A and Figure 2B is a block diagram showing an example configuration of an electronic device according to an embodiment.
[0083] Reference Figure 2A , the electronic device 100 includes a processor (eg, including a processing circuit) 110, and refers to Figure 2B , the electronic device 100 ′ includes a processor (eg, including a processing circuit) 110 and a memory 120 .
[0084] The information about the AI model according to the embodiment may be stored in the internal memory of the processor 110 or in the external memory (e.g., the memory 120), and thus, Figure 2A and Figure 2B In the following, reference will be made to Figure 2B Embodiments are described.
[0085] The electronic device 100 may be implemented as various types of devices that can provide content, such as a server (for example, but not limited to a content providing server, a personal computer (PC), etc.). The electronic device 100 may be the system itself that establishes a cloud computer environment, but is not limited thereto. The electronic device 100 may be implemented as various types of image processing devices that process images and transmit the images to external electronic devices, such as, for example, but not limited to, a set-top box, a digital versatile disk (DVD) player, a game console (for example, an Xbox), and a computer system. TM , PlayStation TM ), virtual reality (VR) devices, tablet computers, mobile phones, e-books, desktop computers, laptop computers, netbook computers, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, home appliances (such as refrigerators), cameras, camcorders, electronic frames, wearable devices (e.g., head-mounted devices (HMDs), smart watches, electronic clothing, electronic bracelets, electronic necklaces, glasses, etc.), near-eye displays (NEDs), etc.
[0086] According to an example, the electronic device 100 may convert an image into images with various resolutions and various compressed images, and transmit the image. The image processed by the electronic device 100 may be, for example, an image prepared by a content provider, but is not limited thereto, and may be an image additionally processed on the image. Hereinafter, for ease of description, it is assumed that the image is processed by the electronic device 100.
[0087] For example, the electronic device 100 may convert an image (or an input image) into at least one image, not only a standard definition (SD) of 720×480, a high definition (HD) of 1280×720, a full high definition (FHD) of 1920×1080, a quad high definition (QHD) of 2560×1440, a 4K ultra high definition (UHD) of 3840×2160, an 8K UHD of 7680×4320, but also an image of a higher resolution (e.g., 16K, 32K), and transmit the image. The electronic device 100 may compress the image in a compressed form (such as, for example, Moving Picture Experts Group (MPEG) (e.g., MP2, MP4, MP7, etc.), Joint Photographic Experts Group (JPEG), Advanced Video Coding (AVC), H.264, H.265, High Efficiency Video Codec (HEVC), VC-1, VP8, VP9, AOMedian Image 1 (AV1), etc., but the present disclosure is not limited thereto).
[0088] The memory 120 may store data required by various embodiments of the present disclosure. Depending on the purpose of data use, the memory 120 may be implemented as a memory embedded in the electronic device 100, or may be implemented as a detachable memory in the electronic device 100. For example, data for driving the electronic device 100 may be stored in a memory embedded in the electronic device 100, and data for additional functions of the electronic device 100 may be stored in a memory detachable from the electronic device 100. The memory embedded in the electronic device 100 can be a volatile memory (such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM)) or a non-volatile memory (for example, a one-time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (for example, a NAND flash memory or a NOR flash memory), a hard disk drive or a solid-state drive (SSD)), etc.), and in the case where the memory is detachably installed to the electronic device 100, the memory can be implemented as a memory card (for example, a compact flash (CF), a secure digital (SD), a micro secure digital (micro-SD), a mini secure digital (mini-SD), an extreme digital (xD), a multimedia card (MMC), etc.), an external memory connectable to a USB port (for example, a USB memory), etc., but the present disclosure is not limited to this.
[0089] According to an embodiment, the memory 120 may store a computer program including at least one instruction or a plurality of instructions for controlling the electronic device 100 .
[0090] According to another example, the memory 120 may store information about an AI model including multiple layers. Storing information about the AI model may refer to, for example, storing various information related to the operation of the AI model, for example, information about multiple layers included in the AI model, information about parameters used in each of the multiple layers (e.g., filter coefficients, biases, etc.). For example, the memory 120 may store information about a first AI model trained to downscale an image according to an example embodiment. The memory 120 may store information about a second AI model trained to obtain quality information of an image according to an example embodiment. However, it should be understood that the information about the AI model may be stored in the internal memory of the processor 110 according to the implementation of the processor 110 described below. For example, if the processor 110 is implemented with dedicated hardware, the information about the AI model may be stored in the internal memory of the processor 110.
[0091] According to another embodiment, the memory 120 may store images received from an external device, an external storage medium (eg, a universal serial bus (USB)), an external server (eg, a web hard). Here, the image may be a digital motion image, but is not limited thereto.
[0092] According to another example, the memory 120 may store, for example but not limited to, information required for downscaling, information required for compression, various information required for performing image quality processing such as at least one of noise reduction, detail enhancement, tone mapping, contrast enhancement, color enhancement, frame rate conversion, etc. The memory 120 may store a final output image generated by image processing.
[0093] According to an example embodiment, the memory 120 may be implemented as a single memory storing data generated in various operations according to the present disclosure. However, according to another embodiment, the memory 120 may be implemented to include a plurality of memories, each storing different types of data, or storing data generated in different steps, respectively.
[0094] It has been described in the examples that various data are stored in the external memory 120 of the processor 110 , but according to at least one example, at least one of the above data may be stored in the internal memory of the processor 110 .
[0095] The processor 110 may include various processing circuits and may be electrically connected to the memory 120 to control the overall operation of the electronic device 100. The processor 110 may be configured with one or more processors. For example, the processor 110 may perform the operation of the electronic device 100 according to various embodiments of the present disclosure by executing at least one instruction stored in the memory 120.
[0096] The processor 110 according to the embodiment may be implemented with, for example, but not limited to: a digital signal processor (DSP) for image processing of digital image signals, a microprocessor, a graphics processor (GPU), an AI processor, a neural processor (NPU), a time controller (TCON), etc., but not limited thereto. The processor 110 may include, for example, but not limited to, a central processing unit (CPU), a microcontroller unit (MCU), a microprocessor (MPU), a controller, an application processor (AP), a communication processor (CP), an advanced reduced instruction set computing (RISC) machine (ARM) processor, a dedicated processor, or one or more, or may be defined as a corresponding term. The processor 110 may be implemented in a system on chip (SoC) type or a large-scale integration (LSI) type, an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA) type with a processing algorithm embedded therein.
[0097] In addition, the processor 110 for executing the AI model according to the example embodiment may be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), a dedicated processor, etc.), a pure graphics processor (such as a graphics processing unit (GPU), a visual processing unit (VPU)), a pure AI processor (such as a neural network processor (NPU)), etc.), but is not limited thereto. The processor 110 may perform control to process input data according to a predefined operation rule or AI model stored in the memory 120. If the processor 110 is a pure AI processor, it may be designed with a hardware structure dedicated to the processing of a specific AI model. For example, hardware dedicated to the processing of a specific AI model may be designed into a hardware chip (such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.).
[0098] The processor 110 processes the input data to obtain output data. The input data may include, for example, but not limited to, at least one of text, an image, a user voice, etc. For example, the input data may be input via a user input (such as a communicator (e.g., including a communication circuit) capable of communicating with an external device, a keyboard or touchpad, a camera, a microphone, etc.). The output data may take various forms depending on the type of AI model. For example, the output data may be an image with improved resolution, object-related information included in the image, text corresponding to the voice, etc.
[0099] According to an example, the processor 110 may perform image processing on the image to obtain an output image. The image may include a still image, a plurality of continuous still images (or frames). The image may be an original image, but is not limited thereto. The image processing may be digital image processing including at least one of image enhancement, image reconstruction, image transformation, image analysis, image understanding, or image compression, but is not limited thereto.
[0100] According to an example embodiment, the processor 110 may perform image processing on an image (e.g., a raw image) using an AI model. For example, the processor 110 may load and use AI model related information stored in the memory 120 (e.g., an external memory such as a dynamic random access memory (DRAM)) to use the AI model.
[0101] According to an example embodiment, the processor 110 may downscale the image and encode (or compress) the downscaled image. In addition, the processor 110 may control at least one of downscaling the image or encoding the downscaled image based on quality information of the image. Downscaling the image may be performed using a first AI model. The quality information of the image may be obtained by a second AI model. In this case, the second AI model may be learned based on feature information of the image obtained from the first AI model.
[0102] Figure 3 is a diagram illustrating an example operation of the processor 110 according to an embodiment.
[0103] According to an example embodiment, the processor 110 may include a downscaler (e.g., including a processing circuit and / or an executable program element) 111, an encoder (e.g., including a processing circuit and / or an executable program element) 112, and a quality estimator (e.g., including a processing circuit and / or an executable program element) 113. For ease of description, the downscaler 111, the encoder 112, and the quality estimator 113 distinguish functions of the processor 110, and the downscaler 111, the encoder 112, and the quality estimator 113 may be implemented in the processor 110 by at least one software, at least one hardware, or a combination thereof.
[0104] According to an example embodiment, the downscaler 111 may include various processing circuits and / or executable program elements, and downscales the image using the first AI model. The output of the first AI model may be a downscaled image. In addition, the first AI model may be jointly trained with an AI model for upscaling the image, wherein the AI model for upscaling the image is included in a receiving device ( Figure 1A and Fig. 20 )middle.
[0105] The trained AI model may refer to a basic AI model (e.g., an AI model including any random parameters) that is trained using a plurality of training data, for example, through a learning algorithm, thereby generating a predetermined action rule or AI model that is set to perform an expected attribute (or purpose). Learning may be implemented through a separate server and / or system, but is not limited thereto, and may be implemented in an electronic device. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.
[0106] The first AI model can be implemented as, for example but not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, etc., but not limited to these.
[0107] Regarding downscaling, for example, at least one interpolation may be used, such as, for example, but not limited to, bilinear interpolation, nearest neighbor interpolation, bicubic interpolation, deconvolution interpolation, sub-pixel convolution interpolation, polyphase interpolation, trilinear interpolation, linear interpolation, etc.
[0108] According to an embodiment, the encoder 112 may include various processing circuits and / or executable program elements, and encode (or compress) the downscaled image through the downscaler 111. For example, the processor 110 may encode the downscaled image using one of various compression methods (such as, for example, but not limited to, Moving Picture Experts Group (MPEG) (e.g., MP2, MP4, MP7, etc.), Joint Photographic Experts Group (JPEG), Advanced Video Coding (AVC), H.264, H.265, High Efficiency Video Codec (HEVC), VC-1, VP8, VP9, AOMedian Image 1 (AV1), etc.).
[0109] The quality estimator 113 may include various processing circuits and / or executable program elements, and estimate the quality of the image and control at least one of the downscaler 111 or the encoder 112 based on the estimated quality. For example, the quality estimator 113 may obtain quality information of the image, and control at least one of the downscaling process of the downscaler 111 or the encoding process of the encoder 112 based on the obtained quality information. The quality information may include a quality identifier corresponding to at least one of the resolution information or the bit rate (or compression strength) information. However, the quality information may be implemented together with other types of information that may control at least one of the downscaler 111 or the encoder 112, but is not limited thereto. For example, the quality information may include a quality identifier corresponding to at least one of the downscaling information (e.g., downscaling ratio) or the bit rate (or compression strength) information.
[0110] When resolution information and bit rate information corresponding to quality information optimal for an image are obtained in the image, the quality estimator 113 may provide the obtained resolution information to the downscaler 111, and provide the obtained bit rate information to the encoder 112. That is, the processor 110 may control the optimal resolution and bit rate of the image in real time based on the image quality.
[0111] According to an embodiment, the quality estimator 113 may obtain the quality information of the image using a second AI model trained to obtain the quality information of the image, and may control at least one of downscaling the image or encoding the downscaled image based on the obtained quality information. The second AI model may be trained to obtain the quality information of the image using the feature information of the image obtained from the first AI model. As an example, the second AI model may estimate and output an image quality identifier in the form of a vector for a predefined resolution and bit rate combination. The vector form may refer to various quality identifiers obtained, for example, for a predefined resolution and bit rate combination. However, in another example, the second AI model may estimate and output an image quality identifier in the form of a vector for a predefined downscaling ratio and bit rate combination.
[0112] Figure 4 is a diagram illustrating an example relationship between a first artificial intelligence model and a second artificial intelligence model according to an embodiment.
[0113] like Figure 4 As shown in , the first AI model 410 may include multiple neural network layers, and each of the multiple neural network layers may include multiple parameters. The first AI model 410 may perform neural network operations through operations between the operation results of the previous layer and the multiple parameters.
[0114] According to an example, after applying the convolution filter to an arbitrary layer, operation data may be output through an activation function (e.g., a rectified linear unit (ReLU) operation). In this case, the operation data output from the arbitrary layer is multi-channel data, for example, 64 feature map (or activation map) data may be output and provided to the next layer. According to an example, the feature map data may be stored in a memory (an internal buffer or an external memory) and then provided to the next layer. However, in Figure 4 The configuration is not shown in . The parameter may refer to, for example, the weight (or coefficient) of the filter. In this case, the first AI model 410 may perform operations using various types of activation functions (such as, for example, but not limited to, identity function, log sigmoid function, hyperbolic tangent (tanh) function, ReLU function, leaky ReLU function, etc.).
[0115] like Figure 4As shown in , the second AI model 420 may include multiple neural network layers, and each of the multiple neural network layers may include multiple parameters. At least one layer 421 included in the second AI model 420 may be trained in conjunction with at least one layer 411 included in the first AI model 410.
[0116] As an example, the second AI model 420 may be implemented to share at least one layer 411 with the first AI model 420. As another example, the second AI model 420 may be implemented to include only one layer 421 using feature information output from the first AI model 410. However, for ease of description, it is assumed that the second AI model 420 is implemented to share at least one layer 411 with the first AI model 420.
[0117] For example, the second AI model 420 may include at least one layer 421 for receiving feature map data output from a specific layer 412 of the plurality of layers included in the first AI model 410. Therefore, the second AI model 420 may be implemented in a form including at least some of the layers 411 included in the first AI model 410 and at least one layer 421 added. The specific layer 412 may be a layer belonging to the original resolution domain. For example, since the resolution information of the quality information output from the second AI model is fed back to the first AI model 410 to control the downscaling of the first AI model 410, the first AI model 410 may perform downscaling through an interpolation operation (e.g., bicubic), wherein the feature map data output from the layer of the domain may be provided to the second AI model 420. Therefore, the feature map data before performing downscaling should be provided to the second AI model 420.
[0118] According to an example embodiment, the second AI model 420 may be implemented to fix the parameters of at least one layer 411 shared with the first AI model 410, and train the parameters of the remaining layers 421 not shared with the first AI model 410. For example, transfer learning may be used to train the second AI model 420. Transfer learning may refer to a method of, for example, reusing and learning a model that has been trained for a specific application in another application. For example, assuming that there is a Figure 5 The trained model for classifying animals in images is shown in the upper part of . Figure 5When there is a model with more types of animals as shown in the lower part of , the existing trained model can be reused without retraining from the beginning, and only the classifier part can be additionally trained to produce a new model. The transition learning method has the advantage of quickly learning data with a small amount of data, and is known to be excellent in performance. When a trained model already exists and you are trying to solve a problem similar to that model, the overlearning method is an effective method. In this way, the second AI model 420 can be implemented as using at least some layers 411 of the first AI model 410 trained for downscaling, and only the remaining additional layers 421 are further trained.
[0119] However, as described above, even if the second AI model 420 is implemented to include only one layer 421 for receiving feature map data output from at least one layer included in the first AI model 410, the parameters of at least one layer 421 included in the second AI model 420 may be trained together with the parameters of at least some layers 411 included in the first AI model 410.
[0120] By enabling the second AI model 420 to share feature information of an image obtained from the first AI model 410 , the second AI model 420 may be efficiently trained and the H / W implementation complexity may be minimized and / or reduced.
[0121] According to an example embodiment, the second AI model 420 may include a convolutional neural network (CNN) for extracting feature information of an input image and a fully connected network (FCN) for converting a two-dimensional image into one-dimensional quality information. In some cases, the second AI model 420 may also include a recurrent neural network (RNN) layer for predicting the temporal changes of each frame of the image. In this way, the second AI model 420 may be implemented to include various types of AI models suitable for predicting image quality.
[0122] According to an embodiment, the quality information of the image output from the second AI model 420 may include a plurality of quality identifiers corresponding to different combinations of a plurality of bit rate information and a plurality of resolution information associated with the image. The combination of the plurality of resolution information and the plurality of bit rate information may be predefined for learning the second AI model 420.
[0123] According to an embodiment, the quality information of the image may include a quality index corresponding to the resolution information and the bit rate information. For example, the resolution information may include, for example, but not limited to, standard definition (SD), high definition (HD), full high definition (FHD), quadruple high definition (QHD), 4K ultra high definition (4KUHD), 8K ultra high definition (8K UHD) or others, but not limited thereto. In addition, the bit rate information may be in the form of bit rate information of the image (e.g., bit rate_0, bit rate_1, bit rate_2, ..., bit rate_N). In addition, the quality identifier may be in the form of VQ metric_0, VQ metric_1, VQ metric_2, ..., VQ metric_N. The VQ metric may be implemented as various quality metrics that can measure image quality, such as, for example, but not limited to, peak signal-to-noise ratio (PSNR), image multivariate method assessment fusion (VMAF), structural similarity (SSIM), natural image quality evaluator (NIQE), MSAD, mean square error (MSE), etc.
[0124] According to an example, the quality information of the image output from the second AI model 420 may be in the form of a pair of data types including resolution, bit rate, and quality identifier. For example, the second AI model 420 may be trained to output a quality identifier for a predefined resolution and bit rate combination.
[0125] For example, the quality information of the image output from the second AI model 420 may include multiple quality identifiers corresponding to different resolution information and different bit rate information, such as, for example, but not limited to (FHD, bitrate_0, VQ metric_0), (FHD, bitrate_1, VQ metric_1), (FHD, bitrate_2, VQ metric_2), (FHD, bitrate_3, VQ metric_3), (2K, bitrate_4, VQ metric_4), (2K, bitrate_5, VQ metric_5), (2K, bitrate_6, VQ metric_6), (2K, bitrate_7, VQ metric_7), (4K, bitrate_8, VQ metric_8), (4K, bitrate_9, VQ metric_9) and (4K bitrate_10, VQ metric_10), (4K, bitrate_11, VQ metric_11). 1 to N are just numerical values for identifying the bit rate and VQ metric, and the bit rate and VQ metric can be represented by various values.
[0126] According to an embodiment, the second AI model 420 may be trained based on the difference between a plurality of first sample quality identifiers corresponding to different combinations of a plurality of bit rate information and a plurality of resolution information associated with a sample image and a plurality of second quality identifiers output by inputting the sample image into the second AI model 420. The first sample quality identifier may be an actual quality identifier of a reconstructed image reconstructed by the receiving device 200 from an image processed by the electronic device 100.
[0127] According to an embodiment, Figure 6 As shown in , the second AI model 420 can be trained by supervised learning using a loss function by the difference between the quality information output from the second AI model 420 and the quality information of the actual image. The quality information of the actual image can be obtained by downscaling and encoding the original training image in the electronic device 100 according to the example, sending the image to the receiving device 200 through the network, and then decoding and upscaling the image received from the receiving device 200 to compare the reconstructed image with the original training image. Figure 6 As shown in , the second AI model 420 may include a convolutional neural network (CNN) for extracting feature information of the input image and a fully connected network (FCN) for converting the two-dimensional image into one-dimensional quality information. In some cases, the second AI model 420 may also include a recurrent neural network (RNN) layer for estimating the temporal changes of each frame of the image.
[0128] For example, when the actual image quality identifier is implemented as a PSNR, the PSNR indicating the actual image quality identifier may be calculated based on Equations 1 and 2 as shown below.
[0129] [Equation 1]
[0130]
[0131] [Equation 2]
[0132]
[0133] Here, C(i,j) is an image, and R(i,j) is a reconstructed image.
[0134] return Figure 2A and Figure 2B , the processor 110 may determine a downscaling ratio of the image and a bit rate of the downscaled image based on resolution information and bit rate information corresponding to any one of a plurality of quality identifiers output from the second AI model 420.
[0135] According to an example, the processor 110 may determine the downscaling ratio of the image and the bit rate of the downscaled image based on the best quality identifier among the multiple quality identifiers, for example, the resolution information and the bit rate information corresponding to the first quality identifier. For example, the processor 110 may control the downscaling operation of the first AI model 410 based on the resolution information corresponding to the first quality identifier, and may control the compression strength of the downscaled image output from the first AI model 410 based on the bit rate information corresponding to the first quality identifier. However, the highest quality identifier is an example, and the processor 110 may utilize an average quality identifier among the multiple quality identifiers, or may use a quality identifier corresponding to a predefined threshold quality identifier (for example, a quality identifier corresponding to a quality that is 10% less than the highest quality).
[0136] According to another embodiment, the processor 110 may obtain network status information, and control at least one of downscaling the image or encoding the downscaled image based on the network status information and the quality information of the image. The network status information may include, for example, but not limited to, at least one of the available bandwidth of the network, the data packet loss rate, the round trip time of the data packet, the delay metric gradient of the data packet, the received signal strength identifier (RSSI) information, the communication channel information, the link speed information, the channel interference information, the retry rate information, etc. The network status information may be various network-related information affected by the congestion, distance and transmission speed of the network. The round trip time (or round trip delay) of the data packet may refer to, for example, the time taken for the data packet to go back and forth from the network to the receiving end. The delay gradient of the data packet may be the difference between the sending time interval of sending the data packet at the transmitting end and the receiving time interval of receiving the data packet at the receiving end. The communication channel information may include information about at least one of the channel and the frequency. The link speed information may include information about the rate of the data packet sent from the transmitting end to the receiving end. The channel interference information may be obtained based on the interference factor of each channel measured using various channel RF parameters (such as channel activity time, channel busy time and channel transmission time). The retry rate information may indicate a percentage of transmission data packets (TX data packets) that are retried in seconds.
[0137] According to an example, the content may be received from a content receiving device (or content output device) (eg, Figure 1A and Fig. 20 ) receives network status information. In this example, the content receiving device (or content output device) (e.g., Figure 1A and Fig. 20) can monitor the network status to collect network status information, and if there is a predetermined period of time or a request from the electronic devices 100 and 100', the content receiving device can send the collected network status information to the electronic devices 100 and 100'. For example, if the network is implemented using an Internet-based transmission control protocol (TCP) / Internet protocol (IP) communication protocol, TCP / IP can be implemented as a link layer, an Internet layer, a transport layer, and an application layer. In this example, the network status information can be collected by monitoring the network status in real time in the application layer, but is not limited to this. However, in this case, the electronic devices 100, 100' can collect the network status information by directly monitoring the network status.
[0138] According to an example, the processor 110 may control at least one of downscaling an image or encoding a downscaled image based on the state information of the network and the quality information of the image. For example, the processor 110 may determine at least one quality identifier of a plurality of quality identifiers based on the state information of the network, and may determine a quality identifier of the at least one quality identifier based on a target quality identifier. In this case, the processor 110 may determine a quality identifier based on a bit rate currently available under the network state. When there are multiple bit rates that can be transmitted under the current network state (i.e., multiple quality identifiers corresponding to the corresponding bit rates), the processor 110 may select a quality identifier of the lowest bit rate. Therefore, the network bandwidth may be reduced as much as possible.
[0139] The processor 110 may determine a downscaling ratio of the image and a bit rate of the downscaled image based on resolution information and bit rate information corresponding to the determined image quality identifier.
[0140] For example, if it is assumed that the quality identifier vector value of the image is called image quality metric (VQM), the network state vector value is called network state (NS), and the target VQM value (maximum quality standard (e.g., PSNR 45dB or higher)) is set, the optimal resolution and bit rate can be determined, for example, by the following method: i) estimating a set of VQMs (VQM') that can be transmitted under the current network state among the multiple VQMs output from the second AI model, ii) obtaining a set of VQM's that meet the target VQM value, and iii) if there is no set that meets the target VQM value (e.g., the network condition is poor), reducing the target VQM by 10% and repeating steps i) and ii). If there is a set that meets the target VQM value, selecting the bit rate and resolution combination with the lowest bit rate.
[0141] According to another example, the quality information of the image output from the second AI model 420 may additionally include paired data in the form of resolution, bit rate, and quality identifier, and probability information of the data.
[0142] For example, the quality information output from the second AI model 420 may include a plurality of quality identifiers corresponding to different bit rate information and different resolution information, such as, for example, but not limited to (FHD, bit rate_0, VQ metric_0, probability_0), (FHD, bit rate_1, VQ identifier_1, probability_1), (FHD, bit rate_2, VQ identifier_2, probability_2), (FHD, bit rate_3, VQ identifier_3, probability_3), (2K, bit rate_4, VQ identifier_4, probability_4), (2K, bit rate_5, VQ metric_5, probability_5), (2K, bit rate_6, VQ metric_6, probability_6), (2K, bit rate_7, VQ metric_7, probability_07), (4K, bit rate_8, VQ metric_8, probability_8), (4K, bit rate_9, VQ metric_9, probability_9), (4K, bit rate_10, VQ metric_10, probability_10), and (4K, bit rate_11, VQ metric_11, probability_11) and probability values corresponding to each quality identifier. Here, 1 to N are numbers for identifying bit rates and VQ metrics, and the bit rates, VQ metrics, and probabilities can be represented by various numbers.
[0143] The probability value may refer to, for example, the probability of providing a minimum bit rate that provides a threshold range of quality. Figure 7 , if the quality of the threshold range required by the electronic device 100 is set to range 710, the specific bit rate 720 may be the minimum bit rate that satisfies the quality of the resolution threshold range. However, for ease of description, it is assumed that the resolution is 1. In this way, the probability value may indicate the probability that a specific data pair corresponds to the minimum bit rate. If the electronic device 100 provides similar quality and critical range quality even when compressed to bit rate A and bit rate B (A<B), it is advantageous to compress to a small bit rate A rather than a large bit rate B in terms of compression efficiency. Therefore, the probability value may be large.
[0144] In this case, the second AI model may be trained to not only additionally output paired data in a format including image quality information (such as resolution, bit rate, and quality identifier) but also additionally output probability information of the paired data.
[0145] According to the above embodiments, in a given network environment, the network bandwidth can be effectively used and the user's perceived quality can be maximized.
[0146] return Figure 2A and Figure 2BAccording to an embodiment, the processor 110 may control at least one of downscaling or encoding the current frame based on quality information predicted (or estimated) from the current frame of the image. According to another embodiment, the processor 110 may control at least one of downscaling or encoding the current frame based on quality information predicted from at least one previous frame of the image.
[0147] Fig. 8A and Figure 8B is a diagram illustrating an example of using quality information of a current frame or a previous frame according to various embodiments.
[0148] Fig. 8A is a diagram of an example of controlling, by the processor 110 , at least one of down-scaling or encoding a current frame based on quality information corresponding to the current frame according to an embodiment.
[0149] Reference Fig. 8A , when the first image frame (current image frame) is input to the first AI model 810, the processor 110 may input the image feature information (e.g., feature map data corresponding to the first image frame (current image frame)) output from a specific layer included in the first AI model 810 to the second AI model 820. The specific layer may be a layer belonging to the original resolution domain, for example, a layer corresponding to the resolution before the interpolation process is performed. The processor 110 may store the feature information (e.g., feature map data) of the first image frame (current image frame) output from the specific layer in the frame buffer 80, and down-scale the feature map data stored in the frame buffer 80 based on the quality information output from the second AI model 820 to obtain a down-scaled frame. Therefore, the down-scaled frame may be one frame delayed (e.g., approximately 16.7ms based on 60FPS) and input to the image encoder. However, the prediction accuracy may be improved and the complexity may be reduced.
[0150] Figure 8B 1 is a diagram showing an example in which the processor 110 controls at least one of downscaling or encoding of a current frame based on quality information corresponding to a previous frame of an image according to another embodiment. For example, the processor 110 may predict quality information of the Nth frame based on the N1th, N-2th, ... Nkth frames, and control at least one of downscaling or encoding of the Nth frame based on the quality information.
[0151] Reference Figure 8B, when the second image frame (previous image frame) is input to the first AI model 810, the processor 110 may input the image feature information (e.g., feature map data corresponding to the second image frame (previous image frame)) output from the specific layer included in the first AI model 810 to the second AI model 820. The specific layer may be a layer belonging to the resolution domain of the image before substantial downscaling (e.g., bicubic processing) is performed. The processor 110 may store the feature information (i.e., feature map data) of the second image frame (previous image frame) output from the specific layer in a frame buffer 81 (e.g., first-in, first-out (FIFO)), and obtain quality information based on the feature map data stored in the frame buffer 81. When the third image frame (current frame) is input to the first AI model 810, the processor 110 may perform downscaling (e.g., bicubic) on the third image frame (current frame) based on the quality information predicted based on at least one second image frame (previous image frame). Therefore, the down-scaled current frame may be input to the image encoder without delay.
[0152] For this reason, Figure 8B As shown in , the second AI model 830 may include at least one recurrent neural network (RNN) layer, wherein the recurrent neural network layer is trained to predict the quality information of the current frame based on feature information corresponding to at least one previous frame. Figure 8B In the embodiment, since the RNN is a network suitable for analyzing data input sequentially using time differences and making future predictions based on the analysis, the quality information of the current image frame should be predicted based on the quality information of the previous image frame.
[0153] exist Fig. 8A and Figure 8B In the last layer of the second AI model 820, the fully connected network (FCN) is divided into three FCNs to divide and represent different resolutions (e.g., Figure 6 4K, 2K, FHD), but the FCN can be implemented as one FCN.
[0154] In the above embodiments, it has been explained that the bicubic processing in the first AI model 810 is performed at the back end of the last neural network layer, but the bicubic processing may be performed at the back end of the intermediate neural network layer. In this case, the feature map data output from the neural network layer before the bicubic processing may be provided to the second AI models 820, 830.
[0155] In the above embodiments, the interpolation processing (e.g., bicubic) and neural network layers included in the first AI model 810, the second AI model 820, and 830 may be implemented as software, hardware, or a combination thereof (e.g., processing circuits and / or executable program elements).
[0156] In the above method, the processor 110 may generate a data packet (or frame) based on the encoded image and send the generated data packet. According to one example, the processor 110 may send a data packet (or frame) including the encoded image to the receiving device 20 ( Figure 1A ). In this example, the resolution information and the bit rate information may be included in the header area of the data packet, and the encoded image may be included in the payload area of the data packet. According to an embodiment, the resolution and the bit rate may be changed in at least one frame unit, and the resolution and the bit rate corresponding to at least one frame unit may be included in the header of the data packet.
[0157] Fig. 9 is a block diagram showing an example configuration of an electronic device according to an embodiment.
[0158] Fig. 9 The electronic device shown in FIG. 1 may be implemented as a FIG. 2A to FIG. 2B The electronic devices 100 and 100 ′ shown in FIG. 1 are devices that receive content and output content, and may be referred to as, for example, a receiving device distinguished from the electronic devices 100 and 100 ′.
[0159] Reference Fig. 9 , the receiving device 200 includes a memory 210 , a communication interface (eg, including a communication circuit) 220 , and a processor (eg, including a processing circuit) 230 .
[0160] The receiving device 200 can be implemented as, for example, a television (TV) or a set-top box, but is not limited thereto, and can be implemented as various types of devices with a display function (such as, for example, but not limited to, a smart phone, a tablet personal computer (PC), a mobile phone, an e-book, a desktop computer, a laptop computer, a netbook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a Moving Picture Experts Group Stage 1 or Stage 2 (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) player, a mobile medical device, a camera, a camcorder, an electronic photo frame, a wearable device (for example, a head-mounted device (HMD), a smart watch, an electronic clothing, an electronic bracelet, an electronic necklace, etc.), a near-eye display (NED), a large format display (LFD), a digital signage, a digital information display (DID) / image wall, a projector display, etc.), but is not limited thereto, and can be a device that processes data using an AI model.
[0161] According to an example, images of various resolutions and various compressed images may be received. For example, the receiving device 200 may receive images of at least one resolution of SD, HD, FHD, 4K UHD, 8K UHD or more. The electronic device 100 may receive images in a compressed form such as MPEG, JPEG, AVC, H.264, H.265, HEVC, VC-1, VP8, VP9, AV1 (AOMedian Image 1), etc. The received image may be an image encoded from the electronic devices 100 and 100'.
[0162] The memory 210 can store data required by various embodiments. The implementation format of the memory 210 is similar to Figure 2B The format of memory 120 is similar and will not be described further.
[0163] According to an example, the memory 210 may store at least one instruction or a computer program including instructions for controlling the receiving apparatus 200 .
[0164] According to another example, the memory 210 may store information about an AI model including a plurality of layers. Storing information about the AI model may refer to, for example, storing various information related to the operation of the AI model, such as information about a plurality of layers included in the AI model, information about parameters used in each of the plurality of layers (e.g., filter coefficients, biases, etc.), etc. For example, the memory 210 may store information about an AI model trained to upscale an image according to an example embodiment. The information about the AI model may be stored in the internal memory of the processor 230 according to the implementation of the processor 230 described below. For example, if the processor 230 is implemented as dedicated hardware, the information about the AI model may be stored in the internal memory of the processor 230.
[0165] According to another example, the memory 210 may store images received from an external device, an external storage medium (eg, USB), an external server (eg, a network disk). The images may be digital motion images, but are not limited thereto.
[0166] According to another example, the memory 210 may store information required for upscaling, algorithms, image quality parameters, etc., information required for compression, various information required for performing image quality processing such as noise reduction, detail enhancement, tone mapping, contrast enhancement, color enhancement, or frame rate conversion. The memory 210 may store a final output image generated by image processing.
[0167] In the above embodiments, although it has been described that various data are stored in the external memory 210 of the processor 230, at least a portion of the aforementioned data may be stored in the internal memory of the processor 230 according to at least one example of the receiving device 200 or the processor 230.
[0168] The communication interface 220 may include various processing circuits and receive various types of content. For example, the communication interface 220 may receive an image signal by streaming or downloading from an external device (e.g., a source device), an external storage medium (e.g., a universal serial bus (USB) device), an external server (e.g., a network server, etc.) via a communication method (such as Wi-Fi (wireless LAN network) based on an access point (AP), Bluetooth, Zigbee, wired / wireless local area network (LAN), wide area network (WAN), Ethernet, IEEE 1394, high-definition multimedia interface (HDMI), universal serial bus (USB), mobile high-definition link (MHL), advanced encryption standard (AES) / European Broadcasting Union (EBU), optical fiber, coaxial, etc.). According to an embodiment, the communication interface 220 may receive a real-time image from the electronic device 100, 100' through an Internet communication network.
[0169] The processor 230 may include various processing circuits and be electrically connected to the memory 210, and may control the overall operation of the receiving device 200. The processor 230 may include one or more processors. The processor 230 may perform the operation of the receiving device 200 according to various embodiments by executing at least one instruction stored in the memory 210. The implementation format of the processor 230 is similar to Figure 2A and Figure 2B The format of processor 110 is the same or similar and will not be described further.
[0170] The processor 230 may obtain output data by processing input data. The input data may include at least one of text, image, or user voice. For example, the input data may be input through a communicator that can communicate with an external device, a user input device (such as a keyboard or touchpad, a camera, a microphone, etc.). Depending on the type of AI model, the output data may have various formats. For example, the output data may be an image with improved resolution, information related to an object included in the image, text corresponding to the voice, etc.
[0171] According to an embodiment, the processor 230 may perform image processing on the input image to obtain an output image. The image processing may be digital image processing including at least one of image enhancement, image restoration, image transformation, image analysis, image understanding, or image compression. In one example, when the input image is a compressed image, the processor 230 may decode the compressed image and perform image processing.
[0172] According to an embodiment, when an input image (eg, an image encoded after being down-scaled by the electronic device 100, 100') is received, the processor 230 may obtain an output image by decoding (or releasing compression) the received image and up-scaling the decoded image.
[0173] According to an example, the processor 230 may decode the received image based on the compression method of the received image. The processor 230 may upscale the decoded image using the AI model.
[0174] According to an embodiment, since the image resolution and the image bit rate may be changed in at least one frame unit, the processor 230 needs to know the resolution and bit rate of the transmitted image. For example, when the processor 230 receives an input image of a data packet type, the processor 230 may unpack the data packet to obtain the resolution information and bit rate information included in the header area, and obtain the encoded image included in the payload area. The resolution information and bit rate information included in the header area may be the resolution information associated with downscaling performed in the electronic devices 100 and 100' and the bit rate information associated with image compression.
[0175] The processor 230 may perform decoding on the encoded image based on the obtained bit rate information, and perform up-scaling on the decoded image based on the obtained resolution information.
[0176] Fig. 10A , Fig. 10B and Fig. 10C is a diagram illustrating in more detail an example scaling method using an artificial intelligence model according to an embodiment.
[0177] The AI model 1010 for upscaling according to an embodiment may be implemented as a learning network model for super-resolution processing. Super-resolution may refer to a process of converting a low-resolution image into a high-resolution image through a series of media processing, for example.
[0178] According to the example, Fig. 10AAs shown in , the processor 230 may upscale the decoded image 11 using an AI model 1010 including a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of parameters (or a plurality of weight values), and a neural network operation may be performed by an operation between the result of a previous layer and a plurality of parameters. The parameters included in the plurality of neural network layers may be optimized by the learning results of the AI model. For example, the parameters may be updated so that the loss value or cost value obtained by the AI model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN) (for example, but not limited to a convolutional neural network (CNN), a recurrent neural network (RNN), a general adversarial network (GAN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, etc.), but is not limited to the above examples.
[0179] According to another example, Fig. 10B As shown in , the processor 230 may perform interpolation processing 1020 on the decoded image 11, and input the interpolation image 12 to the AI model 1010 to obtain a residual image 13. For example, the AI model 1010 may be implemented as a residual neural network. Each of the multiple layers of the residual neural network may generate a residual image for the interpolated image using a filter including different parameters. However, the AI model 1010 does not have to generate only a residual image, and may process the input image in various ways according to the implementation example of the AI model 1010, and output the processed image. In this case, the processor 230 may combine the interpolation processed image 12 with the residual image 13 to obtain an output image 14, for example, a high-resolution image. The interpolation process may include, for example, scaling an image with a low resolution to a high resolution, and at least one interpolation technique (for example, but not limited to bilinear interpolation, nearest neighbor interpolation, bicubic interpolation, deconvolution interpolation, sub-pixel convolution interpolation, polyphase interpolation, trilinear interpolation, linear interpolation, etc.) may be used. The residual image may refer to an image including only residual information. The residual information may include but is not limited to at least one of edge direction, edge strength, noise information or texture information as information according to the difference between the input image and the reference image. According to another example, the residual information may include at least one of grayscale information, brightness information or gamma information.
[0180] According to another example, Fig. 10CAs shown in , the processor 230 may input the decoded image 11 to the AI model 1010 to obtain a residual image 13', and interpolate the residual image 13' 1030 to obtain an interpolated residual image 13". The processor 230 may also perform an interpolation process 1020 on the decoded image 11 to obtain an interpolated image 12. The processor 230 may then combine the interpolated image 12 with the interpolated residual image 13" to obtain an output image 15, for example, a high-resolution image. According to Fig. 10C The embodiment shown, with Fig. 10B Unlike the illustrated embodiment, the decoded image 11 may be input into the AI model 1010 to obtain a residual image 13 ′.
[0181] However, according to another embodiment, an AI model other than the AI model 1010 may also be included. In this example, the AI model 1010 and another AI model may be operated sequentially or in parallel. As an example, the processor 230 may input the decoded image 11 to the AI model 1010, input the output of the AI model 1010 to another AI model, and then obtain an output image based on the image output from the other AI model. As another example, the processor 230 may input the decoded image 11 to each of the AI model 1010 and another AI model, and obtain an output image based on a plurality of images output in parallel from the AI model 1010 and another AI model. For example, the AI model 1010 may be a model for generating a first residual image, and another AI model may include a model for generating a second residual image. The AI model 1010 may refer to, for example, a model for upscaling the resolution, and another AI model may be a model for one of the above-mentioned various image processing (e.g., noise reduction) image processing. The AI model 1010 may be a model for object area processing, and another AI model may be a model for background area processing.
[0182] Fig.11A and Fig. 11B is a block diagram illustrating an example of an electronic device and a receiving device according to an embodiment.
[0183] Fig.11A An example of an electronic device according to an embodiment is shown. Fig.11A The electronic device 100 includes a processor (eg, including a processing circuit) 110, a memory 120, and a communication interface (eg, including a communication circuit) 130. The processor 110 and the memory 120 are connected to Figure 2A and 2B The configuration is repeated and will not be described further.
[0184] The communication interface 130 may include various communication circuits and communicate with external devices. For example, the communication interface 130 may receive an image signal by streaming or downloading from an external device (e.g., a source device), an external storage medium (e.g., a universal serial bus (USB) device), an external server (e.g., a network server, etc.) via a communication method (such as, for example, but not limited to, Wi-Fi (wireless LAN network) based on an access point (AP), Bluetooth, Zigbee, wired / wireless local area network (LAN), wide area network (WAN), Ethernet, IEEE 1394, High-Definition Multimedia Interface (HDMI), Universal Serial Bus (USB), Mobile High-Definition Link (MHL), Advanced Encryption Standard (AES) / European Broadcasting Union (EBU), optical, coaxial, etc.). According to an embodiment, the communication interface 130 may transmit the image signal to the external device through an Internet communication network. Fig. 9 The receiving device 200 shown in transmits a real-time image.
[0185] Fig. 11B An example of a receiving device according to an embodiment is shown. Fig. 11B The receiving device 200' includes a memory 210, a communication interface (eg, including a communication circuit) 220, a processor (eg, including a processing circuit) 230, a display 240, an output device (eg, including an output circuit) 250, and a user interface 260. The memory 210, the communication interface 220, and the processor 230 are connected to Fig. 9 The configuration is repeated and therefore will not be described further.
[0186] The display 240 may be implemented as a display including a self-luminous element or a display including a non-self-luminous element and a backlight. For example, the display 240 may be implemented as various types of displays, such as, for example, but not limited to, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a light emitting diode (LED), a micro LED, a mini LED, a plasma display panel (PDP), a quantum dot (QD) display, a quantum dot light emitting diode (QLED), etc. In the display 240, a backlight unit, a driving circuit, etc. that may be implemented as an a-si TFT, a low temperature polysilicon (LTPS) TFT, an organic TFT (OTFT), etc. may also be included. The display 140 may be implemented as a touch screen coupled to a touch sensor, a flexible display, a bendable display, a three-dimensional (3D) display, a display physically connecting multiple display modules, etc. The processor 230 may control the display 240 to output an output image obtained according to various embodiments. Here, the output image may be a high-resolution image of 4K, 8K or higher.
[0187] The outputter 250 may include various output circuits and output a sound signal. For example, the outputter 250 may convert a digital sound signal processed by the processor 230 into an analog sound signal, amplify and output the analog sound signal. For example, the outputter 250 may include various output circuits, such as, for example, but not limited to, at least one speaker unit capable of outputting at least one channel, a D / A converter, an audio amplifier, etc. According to an example, the outputter 250 may be implemented to output various multi-channel sound signals. The processor 230 may control the outputter 250 to process the input sound signal according to the enhanced processing of the input image. For example, the processor 230 may convert an input two-channel sound signal into a virtual multi-channel (e.g., 5.1 channel) sound signal, identify the position of the receiving device 200' to process the signal into a stereo sound signal optimized for space, or provide an optimized sound signal according to the type of the input image (e.g., content type).
[0188] The user interface 260 may include various user interface circuits, and may be implemented as devices such as, for example, but not limited to, buttons, a touch pad, a mouse and keyboard, a touch screen, a remote control transceiver capable of performing the above-mentioned display function and operation input function, etc. The remote control transceiver may receive a remote control signal from an external remote controller via at least one communication method such as infrared communication, Bluetooth communication, or Wi-Fi communication, or transmit a remote control signal.
[0189] According to an embodiment, the receiving device 200' may further include a tuner and a demodulator. The tuner (not shown) may receive a radio frequency (RF) broadcast signal by tuning a user-selected channel or all pre-stored channels in the RF broadcast signal received via an antenna. The demodulator (not shown) may receive and demodulate a digital intermediate frequency (IF) signal and an IF (DIF) signal converted by the tuner, and perform channel decoding, etc. An input image received via a tuner according to an exemplary embodiment may be processed via a demodulator (not shown) and then provided to the processor 230 for image processing according to an exemplary embodiment.
[0190] Fig.12 is a diagram illustrating an example operation of an example transmission and reception system according to an embodiment.
[0191] exist Fig.12 , the upper left side of the dotted line may represent the operation of the transmitting device (e.g., electronic devices 100, 100', 100"), and the right side may represent the operation of the receiving device (e.g., receiving devices 200 and 200'). For ease of description, it is assumed that the operation on the left is performed by the electronic device 100 and the operation on the right is performed by the receiving device 200.
[0192] The electronic device 100 may include a down-scaler (e.g., including a processing circuit and / or an executable program element) 1210, a video encoder (e.g., including a processing circuit and / or an executable program element) 1220, a transmitter 1230, a video quality estimator (e.g., including a processing circuit and / or an executable program element) 1240, and a video quality controller (e.g., including a processing circuit and / or an executable program element) 1250, and the receiving device 200 may include a receiver 1260, a video decoder (e.g., including a processing circuit and / or an executable program element) 1270, and an up-scaler (e.g., including a processing circuit and / or an executable program element) 1280. Each structure 1210 to 1280 may be implemented with IP for various processing. IP may refer to, for example, a reusable functional block, and may be implemented as at least one hardware or software functional block or a combination of at least one software and at least one hardware. Each component 1210 - 1250 in the electronic device 100 may be implemented within at least one processing chip, and each component 1260 - 1280 in the receiving device 200 may also be implemented within at least one processing chip.
[0193] The downscaler 1210 may include various processing circuits and / or executable program elements, and downscale the image frame using the first AI model 410, 810. The image (e.g., video) encoder 1220 may include various processing circuits and / or executable program elements, and encode the downscaled image frame. The transmitter 1230 may include various transmission circuits, and transmit the encoded image frame to the receiving device 200.
[0194] In this example, the image quality estimator 1240 may include various processing circuits and / or executable program elements, and estimates the quality of the image frame based on feature information of the image frame obtained in the downscaling process of the downscaler 1210. The feature information of the image frame may be, for example, feature map information output from a specific layer included in the first AI model 410 or 810.
[0195] For example, the image quality estimator 1240 may be implemented using the second AI model 420, 820. The second AI model 420, 820 may include at least some of the layers included in the first AI model 410, 810, at least some of the layers 1241, and an additional layer 1242 for converting feature information of the image frame into quality information. In some cases, such as Figure 8B As shown in , an RNN layer 1243 for predicting the temporal change of the image may also be included. The quality information of the current image frame (or previous image frame) output from the second AI model 420, 820 may include multiple quality identifiers (VQ_metrics) corresponding to multiple resolution and bit rate sets. For example, Fig.13As shown in , the quality information may be quality information corresponding to each frame index, for example, resolution and bit rate.
[0196] The image (e.g., video) quality controller 1250 may include various processing circuits and / or executable program elements, and selects a quality identifier in consideration of a target quality index of the electronic device 100, a network status, etc. from among a plurality of quality identifiers output from the second AI model 420, 820, and provides resolution information and bit rate information corresponding to the selected quality identifier to the down-scaler 1210 and the image encoder 1220, respectively. The provided resolution information and bit rate information may be obtained based on the current image frame, or the provided resolution information and bit rate information may be information predicted based on a previous image frame.
[0197] The downscaler 1210 and the image encoder 1220 may perform downscaling and encoding based on the received resolution information and bit rate information, respectively.
[0198] The transmitter 1230 may include various circuits, and packetizes the encoded image output from the image encoder 1220 and transmits the image to the receiving apparatus 200 .
[0199] When a data packet is received from the electronic device 100, the receiver 1260 included in the receiving device 200 may unpack the received data packet into a compressed image format. The image (e.g., video) decoder 1270 may include various processing circuits and / or executable program elements, and decode (or decompress) the image data output from the receiver 1260. The upper scaler 1280 may include various processing circuits and / or executable program elements, and scale the decoded image data up to the output resolution to obtain an output image. The obtained output image may be displayed by a display. The header of the received data packet may include resolution information associated with the down scaling of the down scaler 1210 and bit rate information associated with the encoding of the image encoder 1220. The image decoder 1270 may perform decoding based on the bit rate information included in the header, and the upper scaler 1280 may perform up scaling based on the resolution information included in the header.
[0200] exist Fig.12 In the illustrated embodiment, the difference between the quality information output from the second AI models 420 and 820 and the quality information of the actual image can be used as a loss function to train the learning of the second AI models 420 and 820 using supervised learning. According to an example, the quality information of the actual image can be obtained by comparing the original training image with the image reconstructed via the downscaler 1210, the image encoder 1220, the transmitter 1230, the receiver 1260, the image decoder 1270, and the upscaler 1280.
[0201] Fig.14 is a flow chart illustrating an example method for controlling an electronic device according to an embodiment.
[0202] according to Fig.14 In the control method of the electronic device, in operation S1410, the electronic devices 100, 100' and 100" downscale the image using the trained first AI model. In operation S1420, the electronic devices 100, 100' and 100" encode the downscaled image.
[0203] In operation S1430, downscaling of the image using the first AI model may be controlled based on quality information of the image obtained using the trained second AI model. The second AI model may be trained using feature information of the image obtained from the first AI model.
[0204] In operation S1420, the down-scaled image may be controlled to be encoded based on quality information of the image obtained using the second AI model.
[0205] The quality information of the image may include a plurality of quality identifiers corresponding to different combinations of resolution information and bit rate information associated with the image. In operation S1410, a downscaling ratio of the image may be determined based on a quality identifier of any one of the plurality of quality identifiers, and the image may be downscaled based on the determined downscaling ratio control. In operation S1420, a bit rate of the downscaled image may be determined based on a quality identifier of any one of the plurality of quality identifiers, and the downscaled image may be encoded based on the determined bit rate control.
[0206] In addition, the method of controlling the electronic device 100, 100', 100" may further include obtaining status information of the network. In this case, in operation S1410, the image may be down-scaled based on the status information of the network and the quality information of the image. In operation S1420, the down-scaled image may be encoded based on the status information of the network and the quality information of the image.
[0207] The method of controlling the electronic device 100, 100', 100" may further include: determining at least one quality identifier among the plurality of quality identifiers based on state information of the network; and determining a quality identifier among the at least one quality identifier based on a target quality identifier. In operation S1410, down-scaling of the image may be controlled based on resolution information corresponding to the determined quality identifier. In addition, in operation S1420, encoding of the down-scaled image may be controlled based on bit rate information corresponding to the determined quality identifier.
[0208] The second AI model may include, for example, at least one neural network layer, and parameters of the at least one neural network layer may be trained in association with parameters of at least some layers included in the first AI model.
[0209] In operation S1410, down-scaling of a current frame may be controlled based on quality information obtained from a current frame of an image, and down-scaling of the current frame may be controlled based on quality information predicted from at least one previous frame of the image.
[0210] The second AI model may include, for example, at least one RNN layer, wherein the at least one RNN layer is trained to predict quality information of a current frame based on feature information corresponding to at least one previous frame.
[0211] The second AI model may be trained using supervised learning using the difference between a plurality of quality identifiers output by inputting a plurality of actual quality identifiers corresponding to different combinations of a plurality of resolution information and a plurality of bit rate information related to the training image into the second AI model as a loss function. Figure 6 The actual quality identifier has been described in , and the detailed description will not be repeated.
[0212] The method of controlling the electronic devices 100, 100', and 100" may further include generating a data packet based on the encoded image and transmitting the generated data packet. Resolution information and bit rate information of the image may be included in a header area of the data packet.
[0213] According to the various embodiments described above, the optimal resolution and bit rate can be controlled based on the real-time predicted image quality and network status. Therefore, the network bandwidth can be effectively utilized, and the perceived quality of consumers in a given network environment can be maximized. In particular, by optimally controlling the resolution and bit rate of images in real-time live streaming with delay constraints, the network bandwidth can be effectively used.
[0214] By sharing feature information of images obtained during the downscaling process to predict image quality, AI models can be efficiently trained and H / W implementation complexity can be minimized and / or reduced.
[0215] An AI encoding process and an AI decoding process according to an embodiment will be described with reference to the drawings.
[0216] Fig.15 is a diagram illustrating an example AI encoding process and an example AI decoding process according to an embodiment.
[0217] As described above, when the resolution of an image increases, the amount of information processing for encoding / decoding increases, and a method for enhancing encoding efficiency and decoding efficiency of an image is required.
[0218] like Fig.15As shown, according to an embodiment, a first image 1515 can be obtained by performing AI downscaling 1510 on an image 1505 having a high resolution. By performing the first encoding 1520 and the first decoding 1530 on the first image 1515 having a relatively small resolution, the bit rate to be processed can be significantly reduced compared to performing the first encoding 1520 and the first decoding 1530 on the image 1505.
[0219] Reference Fig.15 In an embodiment, in the AI encoding process, a first image 1515 is obtained by performing AI downscaling 1510 on an image 1505, and a first encoding 1520 is performed on the first image 1515. In the AI decoding process, AI data obtained as a result of the AI encoding and AI encoded data including image data are received, a second image 1535 is obtained by performing the first decoding 1530, and a third image 1545 is obtained by performing AI upscaling 1540 on the second image 1535.
[0220] Referring to the AI encoding process in more detail, when the image 1505 is received, the image 1505 is AI down-scaled 1510 to obtain a first image 1515 or a predetermined image of a predetermined resolution. The AI down-scaling 1510 is performed based on AI, wherein the AI for the AI down-scaling 1510 should be trained together with the AI for the AI up-scaling 15410 of the second image 1535, because if the AI for the AI down-scaling 1510 and the AI for the AI up-scaling 1540 are trained separately, the difference between the image 1505 as the target of AI encoding and the third image 1545 reconstructed by AI decoding increases.
[0221] In an embodiment, in order to maintain a common relationship in the AI encoding process and the AI decoding process, AI data may be used. Therefore, the AI data obtained by the AI encoding process needs to include information indicating an upscaling target, and in the AI decoding process, the second image 1535 needs to be AI upscaled 1540 according to the upscaling target identified based on the AI data.
[0222] The AI for AI downscaling 1510 and the AI for AI upscaling 1540 may be implemented as, for example but not limited to, DNNs. Fig.23 As described, since the first DNN and the second DNN are jointly trained by sharing loss information under a predetermined target, the AI encoding device provides the target information used when jointly training the first DNN and the second DNN to the AI decoding device, and the AI decoding device can perform AI upscaling 1540 on the second image 1535 using the target resolution based on the received target information.
[0223] When the first encoding 1520 and the first decoding 1530 are described in more detail (eg, Figure 1A ), the first image 1515 scaled down from the image 1505AI may have a reduced amount of information via the first encoding 1520. The first encoding 1520 may include: generating prediction data by predicting the first image 1515, generating residual data corresponding to the difference between the first image 1515 and the prediction data, transforming the residual data as a spatial domain component into a frequency domain component, quantizing the residual data transformed into the frequency domain component, entropy encoding the quantized residual data, and the like. The first encoding process 1520 may be implemented by an image compression method using frequency transformation (such as, for example, but not limited to, MPEG-2, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1, etc.).
[0224] The second image 1535 corresponding to the first image 1515 may be reconstructed by first decoding 1530 of the image data. The first decoding 1530 may include: entropy decoding the image data to generate quantized residual data; inverse quantizing the quantized residual data; transforming the residual data of the frequency domain component into the spatial domain component; generating prediction data; and reconstructing the second image 1535 using the prediction data and the residual data, etc. The first decoding 1530 may be implemented by an image reconstruction method corresponding to one of the image compression methods using frequency transformation used in the first encoding 1520 (such as, for example, but not limited to, MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, AV1, etc.).
[0225] The AI encoded data obtained through the AI encoding process may include image data obtained as a result of the first encoding 1520 of the first image 1515 and AI data related to the AI downscaling 1515 of the image 1505 .
[0226] The image data may be transmitted in the form of a bitstream. The image data may include data obtained based on pixel values in the first image 1515, for example, residual data as a difference between the first image 1515 and prediction data of the first image 1515. In addition, the image data includes information used in the first encoding 1520 of the first image 1515. For example, the image data may include prediction mode information and motion information for the first encoding 1520 of the first image 1515, information related to a quantization parameter used in the first encoding 1520, and the like. The image data may be generated according to the rules of the image compression method used in the first encoding 1520 (for example, the syntax of an image compression method using frequency transformation (such as, for example, but not limited to MPEG-2, AVC, H.264AVC, MPEG-4, HEVC, VC-1, VP8, VP9, AV1, etc.)).
[0227] The AI data may be used for AI upscaling 1540 based on the second DNN. As described above, the first DNN and the second DNN are jointly trained, and the AI data includes information that enables AI upscaling 1540 of the second image 1535 by the second DNN to be performed. In the AI decoding process, the second image 1535 may be AI upscaled 1540 to a target resolution and / or quality based on the AI data.
[0228] The AI data may be transmitted in a bitstream format together with the image data. Alternatively, according to an embodiment, the AI data may be transmitted in a frame or data packet separated from the image data. The image data and the AI data obtained as a result of the AI encoding may be transmitted over the same network or different networks.
[0229] Fig.16 is a block diagram illustrating an example configuration of an example AI decoding device 1600 according to an embodiment.
[0230] Reference Fig.16 , the AI decoding device 1600 according to an embodiment may include a receiver 1610 and an AI decoder 1630. The receiver 1610 may include a communicator (e.g., including a communication circuit) 1612, a parser (e.g., including a processing circuit and / or an executable program element) 1614, and an outputter (e.g., including an output circuit) 1616. The AI decoder 1630 may include a first decoder (e.g., including a processing circuit and / or an executable program element) 1632 and an AI upscaler (e.g., including a processing circuit and / or an executable program element) 1634.
[0231] The receiver 1610 may include various receiving circuits, and receives and parses AI encoded data obtained as a result of AI encoding, divides image data and AI data, and outputs the data to the AI decoder 1630 .
[0232] For example, the communicator 1612 may include various communication circuits and receive AI-encoded data obtained as a result of AI encoding through a network. The AI-encoded data obtained as a result of AI encoding includes image data and AI data. The image data and AI data may be received through the same network or different networks.
[0233] The parser 1614 may include various processing circuits and / or executable program elements, and parse the AI encoded data received by the communicator 1612, and divide the data into image data and AI data. For example, the data is divided into image data or AI data by reading the header of the data obtained from the communicator 1612. In one example, the parser 1614 divides the image data and AI data by the header of the data received via the communicator 1612, and passes the divided image data and AI data to the outputter 1616, and the outputter 1616 (which may include various output circuits) passes the respective divided data to the first decoder 1632 and the AI upper scaler 1634. It can be determined that the image data included in the AI encoded data is image data obtained by a predetermined codec (e.g., MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9 or AV1). In this example, the corresponding information can be sent to the first decoder 1632 by the outputter 1616, so that the image data can be processed using the identified codec.
[0234] In an embodiment, the AI encoded data parsed by the parser 1614 can be obtained from data storage media including magnetic media (such as hard disks, floppy disks, magnetic tapes), optical recordable media (such as CD-ROMs and DVDs), magneto-optical media (such as magneto-optical floppy disks), etc.
[0235] The first decoder 1632 may include various processing circuits and / or executable program elements, and reconstructs a second image 1535 corresponding to the first image 1515 based on the image data. The second image 1535 obtained by the first decoder 1632 is provided to the AI upscaler 1634. According to an embodiment, first decoding related information (such as prediction mode information, motion information, quantization parameter information, etc.) included in the image data may also be provided to the AI upscaler 1634.
[0236] The AI upscaler 1634 receiving the AI data may include various processing circuits and / or executable program elements, and performs AI upscaling on the second image 1535 based on the AI data. According to an embodiment, the AI upscaling may be performed by further using information related to the first encoding (such as prediction mode information and quantization parameter information included in the image data, etc.).
[0237] The receiver 1610 and the AI decoder 1630 according to the embodiment are described as separate devices, but may be implemented by a single processor. In this case, the receiver 1610 and the AI decoder 1630 may be implemented as a dedicated processor, and the receiver 1610 and the AI decoder 1630 may be implemented by a combination of a general-purpose processor (such as, for example, but not limited to, an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), and S / W). In addition, in the case of a dedicated processor, the dedicated processor may be implemented by including a memory for implementing the embodiment or a memory processor for using an external memory.
[0238] The receiver 1610 and the AI decoder 1630 may include a plurality of processors, may be implemented as a combination of dedicated processors, or may be implemented by a combination of a plurality of general-purpose processors (such as AP, CPU, GPU, and S / W). Similarly, the AI upscaler 1634 and the first decoder 1632 may be implemented as different processors.
[0239] The AI data provided to the AI upscaler 1634 includes information for performing AI upscaling on the second image 1535. The upscaling target should correspond to the downscaling of the first DNN. Therefore, the AI data needs to include information for identifying the downscaling target of the first DNN.
[0240] As examples of information included in the AI data, there is information about a difference between a resolution of the image 1505 and a resolution of the first image 1515 and information related to the first image 1515 .
[0241] The difference information may be represented as information about the degree of resolution conversion of the first image 1515 relative to the image 1505 (e.g., resolution conversion rate information), for example. Since the resolution of the first image 1515 is known by the resolution of the reconstructed second image 1535, and the degree of resolution conversion can be confirmed by the known resolution, the difference information may be represented by only the resolution information of the image 1505. The resolution information may be represented as a screen size in the horizontal / vertical direction, and may be represented as a ratio (16:9, 4:3, etc.) and a size on one axis. In addition, the predetermined resolution information may be represented in the form of an index or a flag.
[0242] The information related to the first image 1515 may include information about at least one of a bit rate of image data obtained as a result of first encoding the first image 1515 and a codec type used for the first encoding of the first image 1515 .
[0243] The AI upscaler 1634 may determine an upscaling target of the second image 1535 based on at least one of the difference information included in the AI data and the information related to the first image 1515. The upscaling target may indicate, for example, whether to upscale the second image 1535 to which resolution. Once the upscaling target has been determined, the AI upscaler 1634 performs AI upscaling on the second image 1535 via the second DNN to obtain a third image 1545 corresponding to the upscaling target.
[0244] Before describing a method of performing AI upscaling on the second image 1535 to correspond to an upscaling target by the AI upscaler 1634, reference will be made to Figure 3 and Figure 4 Describe the process of AI upscaling through a second DNN.
[0245] Fig.17 is a diagram showing a second DNN 300 for AI upscaling of a second image 1535, and Fig.18 Shown by Fig.17 An example convolution operation performed by the first convolutional layer 310 is shown in FIG.
[0246] Reference Fig.17 , the second image 1535 is input to the first convolutional layer 310. Fig.17 The 3×3×4 shown in the first convolution layer 310 shown in FIG. 1 shows that a convolution process is performed on one input image using four filter kernels of size 3×3. As a result of the convolution process, four feature maps are generated by the four filter kernels. Each feature map represents a unique feature of the second image 1535. For example, each feature map may indicate a vertical direction feature, a horizontal direction feature, or an edge feature of the second image 1535.
[0247] Reference Fig.18 , the convolution operation in the first convolutional layer 310 will be described in more detail.
[0248] One feature map 450 may be generated by multiplication and addition operations between parameters of the filter kernel 430 having a size of 3×3 used in the first convolution layer 310 and corresponding pixel values in the second image 1535. Since four filter kernels are used in the first convolution layer 310, four feature maps may be generated by convolution operations using the four filter kernels.
[0249] exist Fig.18 , I1 to I49 displayed in the second image 1535 represent pixels of the second image 1535, and F1 to F9 displayed in the filter kernel 430 represent parameters of the filter kernel 430. M1 to M9 displayed in the feature map 450 represent samples of the feature map 450.
[0250] exist Fig.18 , the second image 1535 includes 49 pixels, but this is merely an illustrative, non-limiting example, and when the resolution of the second image 1535 is 4K, for example, it may include 3840×2160 pixels.
[0251] In the convolution operation, each of the pixel values of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the second image 1535 and each of F1, F2, F3, F4, F5, F6, F7, F8, and F9 of the filter kernel 430 may be respectively performed, and a value combining (e.g., adding) the result values of the multiplication operations may be assigned as the value of M1 of the feature map 450. If the stride of the convolution operation is 2, each of the pixel values of I3, I4, I5, I10, I11, I12, I17, I18, and I19 of the second image 1535 and each of F1, F2, F3, F4, F5, F6, F7, F8, and F9 of the filter kernel 430 may be performed, and a value combining the result values of the multiplications may be assigned as the value of M2 of the feature map 450.
[0252] When the filter kernel 430 moves along the stride until reaching the last pixel of the second image 1535, a convolution operation may be performed between the pixel values of the second image 1535 and the parameters of the filter kernel 430, and a feature map 450 having a predetermined size may be obtained.
[0253] According to an embodiment, by jointly training the first DNN and the second DNN, the parameters of the second DNN, for example, the parameters of the filter kernel used in the convolution layer of the second DNN (for example, the values of F1, F2, F3, F4, F5, F6, F7, F8, and F9 of the filter kernel 430) may be optimized. As described above, the AI up-scaler 1634 may determine an up-scaling target corresponding to the down-scaling target of the first DNN based on the AI data, and determine the parameters corresponding to the determined up-scaling target as the parameters of the filter kernel used in the convolution layer of the second DNN.
[0254] The convolutional layers included in the first DNN and the second DNN can be based on the reference Fig.18 The convolution operation described above is used to perform the processing, but Fig.18 The convolution operation processing is only an example and is not limited to this.
[0255] Reference Fig.17 , the feature map output from the first convolution layer 310 is input to the first activation layer 320. The first activation layer 320 may assign nonlinear features to each feature map. The first activation layer 320 may include, but is not limited to, a sigmoid function, a Tanh mathematical function, a ReLU function, etc., but is not limited thereto.
[0256] Assigning a nonlinear feature in the first activation layer 320 means changing some sample values of the feature map and outputting the changed sample values. The change is performed by applying the nonlinear feature.
[0257] The first activation layer 320 determines whether to pass sample values of the feature map output from the first convolution layer 310 to the second convolution layer 330. For example, some of the sample values of the feature map are activated by the first activation layer 320 and passed to the second convolution layer 330, and some of the sample values are deactivated by the first activation layer 320 and not passed to the second convolution layer 330. The inherent features of the second image 1535 represented by the feature map are highlighted by the first activation layer 320.
[0258] The feature map 325 output from the first activation layer 320 is input to the second convolutional layer 330 . Fig.17 One of the feature graphs 325 shown in FIG. 1 is obtained by comparing Fig.18 The result obtained by processing the feature map 450 in the first activation layer 320.
[0259] The 3×3×4 indicated in the second convolution layer 330 shows that the input feature map 325 is convolved using four filter kernels of size 3×3. The output of the second convolution layer 330 is input to the second activation layer 340. The second activation layer 340 can emit nonlinear features to the input data.
[0260] The feature map 345 output from the second activation layer 340 is input to the third convolutional layer 350 . Fig.17 The 3×3×1 shown in the third convolution layer 350 shown in the figure shows that a convolution process is performed using a filter kernel of size 3×3 to generate an output image. The third convolution layer 350 and the layer for outputting the final image use a filter kernel to generate an output. According to the example, the third convolution layer 350 can output the third image 1545 via the convolution operation result.
[0261] As described below, there may be multiple DNN configuration information indicating the number of filter cores, parameters of the filter cores, etc. in the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300, but the multiple DNN configuration information should be associated with the multiple DNN configuration information of the first DNN. The association between the multiple DNN configuration information of the second DNN and the multiple DNN configuration information of the first DNN can be achieved by jointly training the first DNN and the second DNN.
[0262] although Fig.17The second DNN 300 is shown to include three convolutional layers 310, 330, 350 and two activation layers 320, 340, but this is only an example, and according to embodiments, the number of convolutional layers and activation layers may vary in a wide range. In addition, according to embodiments, the second DNN 300 may be implemented via RNN. In this example, according to embodiments, the CNN structure of the second DNN 300 is changed to an RNN structure.
[0263] In an example embodiment, the AI upper scaler 1634 may include at least one arithmetic logic unit (ALU) for the above-mentioned convolution operation and the operation of the activation layer. The ALU algorithm can be implemented using a processor. For the convolution operation, the ALU may include a multiplier and an adder, wherein the multiplier performs a multiplication operation between the sample value of the filter kernel and the sample value of the feature map output from the second image 1535 or the previous layer, and the adder adds the result value of the multiplication. For the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier is used to multiply the input sample value by the weight used in a predetermined S-shaped function, Tanh function, or ReLU function, and the comparator is used to compare the result of the multiplication with a predetermined value to determine whether to pass the input sample value to the next layer.
[0264] Hereinafter, a method of AI upscaling the second image 1535 to correspond to an upscaling target by the AI upscaler 1634 will be described.
[0265] In an embodiment, the AI upscaler 1634 may store a plurality of DNN configuration information that may be set in the second DNN.
[0266] The DNN configuration information may include information about at least one of the number of convolutional layers included in the second DNN, the number of filter kernels for each convolutional layer, and the parameters of each filter kernel. A plurality of DNN configuration information may correspond to various upscaling targets, respectively, and the second DNN may operate based on the DNN configuration information corresponding to a specific upscaling target. According to the DNN configuration information, the second DNN may have a different structure. For example, the second DNN may include three convolutional layers according to any DNN configuration information, and the second DNN may include four convolutional layers according to other DNN configuration information.
[0267] In an embodiment, the DNN configuration information may include only parameters of the filter kernel for the second DNN. The structure of the second DNN may not change, but only the parameters of the internal filter kernel may change according to the DNN configuration information.
[0268] The AI upscaler 1634 may obtain DNN configuration information from a plurality of DNN configuration information for performing AI upscaling on the second image 1535. Each of the plurality of DNN configuration information is used to obtain a third image 1545 of a predetermined resolution and / or a predetermined quality, and is trained together with the first DNN.
[0269] For example, any one of the multiple DNN configuration information may include information for obtaining a third image 1545 of 4K (4096*2160) having a resolution twice that of the second image 1535 of 2K (2048*1080), and another DNN configuration information may include information for obtaining a third image 1545 of 8K (8192*4320) having a resolution four times that of the second image 1535 of 2K (2048*1080).
[0270] Each of the plurality of DNN configuration information is generated in association with the DNN configuration information of the first DNN of the AI encoding device 600, and the AI upper scaler 1634 obtains one of the plurality of DNN configuration information according to the enlargement ratio corresponding to the reduction ratio of the DNN configuration information of the first DNN. To this end, the AI upper scaler 1634 should check the information of the first DNN. In order for the AI upper scaler 1634 to check the information of the first DNN, the AI decoding device 1600 according to the embodiment receives AI data including the information of the first DNN from the AI encoding device 600.
[0271] In other words, the AI up-scaler 1634 may identify information targeted by the DNN configuration information of the first DNN used to obtain the first image 1515 using information received from the AI encoding device 600, and obtain the DNN configuration information of the second DNN that is jointly trained.
[0272] When DNN configuration information is obtained from a plurality of DNN configuration information or AI upscaling is performed on the second image 1535, input data may be processed based on the second DNN that operates according to the obtained DNN configuration information.
[0273] For example, if a DNN configuration information is obtained, then for Fig.17 Each of the first convolution layer 310, the second convolution layer 330, and the third convolution layer 350 of the second DNN 300 shown in FIG. 3 may set the number of filter kernels included in each layer and the parameters of the filter kernels to the values included in the obtained DNN configuration information.
[0274] For example, if you Fig.17The parameters of the 3×3 filter kernel used in any one of the convolutional layers of the second DNN shown in are set to {1, 1, 1, 1, 1, 1, 1, 1}, and then the DNN configuration information is changed, the parameters can be replaced by {2, 2, 2, 2, 2, 2, 2, 2} included in the changed DNN configuration information.
[0275] The AI upscaler 1634 may obtain, based on information included in the AI data, DNN configuration information for upscaling the second image 1535 from among a plurality of DNN configuration information. The AI data for obtaining the DNN configuration information will be described in more detail.
[0276] In an embodiment, the AI upscaler 1634 may obtain DNN configuration information for upscaling the second image 1535 based on the difference information included in the AI data of the plurality of DNN configuration information. For example, based on the difference information, if it is identified that the resolution of the image 1505 (e.g., 4K (4096*2160)) is twice the resolution of the first image 1515 (e.g., 2K (2048*1080)), the AI upscaler 1634 may obtain DNN configuration information that can increase the resolution of the second image 1535 to twice.
[0277] In another embodiment, the AI upscaler 1634 may obtain DNN configuration information for performing AI upscaling on the second image 1535 from among a plurality of DNN configuration information based on information related to the first image 1515 included in the AI data. The AI upscaler 1634 may predetermine a mapping relationship between image-related information and DNN configuration information, and obtain DNN configuration information mapped to information related to the first image 1515.
[0278] Fig.19 is a diagram showing an example mapping relationship between various image-related information and DNN configuration information.
[0279] Reference Fig.19 As can be seen from the example, AI encoding / AI decoding processing does not only consider the change of resolution. Fig.19 As shown in , DNN configuration information can be selected by considering resolution (such as SD, HD, full HD), bit rate (such as 10Mbps, 15Mbps, 20Mbps), and codec information (such as AV1, H.264, HEVC), respectively. To this end, the training of each element in the AI training process must be performed together with the encoding and decoding processes (see Fig.23 ).
[0280] Depending on the content of the training, Fig.19When multiple DNN configuration information is provided based on image-related information including codec type, image resolution, etc. as shown in , DNN configuration information for AI upscaling of the second image 1535 can be obtained based on information about the first image 1515 received in the AI decoding process.
[0281] AI upper scaler 1634 will Fig.19 The image-related information shown on the left side and the DNN configuration information on the right side are matched, and the DNN configuration information based on the image-related information can be used.
[0282] like Fig.19 As shown in , if it is identified that the resolution of the first image 1515 is SD, the bit rate of the image data obtained by first encoding the first image 1515 is 10Mbps, and the first image 1515 is first encoded by the AV1 codec, the AI upper scaler 1634 can use A DNN configuration information among multiple DNN configuration information.
[0283] If it is identified from information related to the first image 1515 that the resolution of the first image 1515 is HD, the bit rate of the image data obtained as a result of the first encoding is 15Mbps, and the first image 1515 is first encoded through the H.264 codec, the AI upper scaler 1634 can use B DNN configuration information among multiple DNN configuration information.
[0284] In addition, if it is identified from the information related to the first image 1515 that the resolution of the first image 1515 is full HD, the bit rate of the image data obtained as a result of first encoding the first image 1515 is 20 Mbps, and the first image 1515 is first encoded by the HEVC codec, the AI upper scaler 1634 may use the C DNN configuration information among the plurality of DNN configuration information, and if it is identified that the resolution of the first image 1515 is full HD, the bit rate obtained as a result of first encoding the first image 1515 is 15 Mbps, and the first image 1515 is first encoded by the HEVC codec, the AI upper scaler 1634 may use the D DNN configuration information among the plurality of DNN configuration information. One of the C DNN configuration information and the D DNN configuration information is selected according to whether the bit rate of the image data obtained as a result of first encoding the first image 1515 is 20 Mbps or 15 Mbps. When the first image 1515 having the same resolution is first encoded by the same codec, the bit rates of the image data being different from each other may mean, for example, that the image qualities of the restored images are different from each other. Therefore, the first DNN and the second DNN may be jointly trained based on a predetermined image quality, and thus, the AI upscaler 1634 may obtain DNN configuration information based on a bit rate of image data indicating an image quality of the second image 1535 .
[0285] In another embodiment, the AI upscaler 1634 may obtain DNN configuration information for AI upscaling the second image 1535 from among a plurality of DNN configuration information, taking into account information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 1632 and information related to the first image 1515 included in the AI data. For example, the AI upscaler 1634 may receive quantization parameter information used in a first encoding process of the first image 1515 from the first decoder 1632, identify a bit rate of image data obtained as a result of encoding the first image 1515 from the AI data, and obtain DNN configuration information corresponding to the quantization parameter and the bit rate. Depending on the complexity of the image, the same bit rate may allow the reconstructed image to have different image qualities, and in the first image 1515, the bit rate of each frame may be different as a value representing the entire first image 1515 in which the first encoded image 1515 is encoded. Therefore, considering the prediction mode information, motion information and / or quantization parameters for each frame that can be obtained from the first decoder 1632, DNN configuration information suitable for the second image 1535 can be obtained compared to using only AI data.
[0286] In addition, according to an embodiment, the AI data may include an identifier of mutually agreed DNN configuration information. The identifier of the DNN configuration information is information for identifying a pair of DNN configuration information that are jointly trained between the first DNN and the second DNN, so that the second image 1535 is AI scaled up to an upper scaling target corresponding to the lower scaling target of the first DNN. After obtaining the identifier of the DNN configuration information included in the AI data, the AI upscaler 1634 may AI scale up the second image 1535 using the DNN configuration information corresponding to the identifier of the DNN configuration information. For example, an identifier indicating each of the multiple DNN configuration information configurable in the first DNN and an identifier indicating each of the multiple DNN configuration information configurable in the second DNN may be predetermined. In this case, the same identifier may be specified for each pair of DNN configuration information configurable for each of the first DNN and the second DNN. The AI data may include an identifier of the DNN configuration information set in the first DNN for AI downscaling the image 1505. The AI upscaler 1634 receiving the AI data may perform AI upscaling on the second image 1535 using DNN configuration information of a plurality of DNN configuration information indicated by an identifier included in the AI data.
[0287] According to an embodiment, the AI data may include DNN configuration information. The AI upscaler 1634 may obtain the DNN configuration information included in the AI data, and then perform AI upscaling on the second image 1535 using the corresponding DNN configuration information.
[0288] According to an embodiment, if information of DNN configuration information (e.g., the number of convolutional layers, the number of filter kernels used for each convolutional layer, parameters of each filter kernel, etc.) is stored in the form of a lookup table, the AI upscaler 1634 may combine selected portions of the lookup table values based on information included in the AI data to obtain the DNN configuration information, and perform AI upscaling on the second image 1535 using the obtained DNN configuration information.
[0289] According to an embodiment, when a DNN structure corresponding to an upscaling target is determined, the AI upscaler 1634 may obtain DNN configuration information corresponding to the determined DNN structure, for example, parameters of a filter kernel.
[0290] The AI up-scaler 1634 obtains DNN configuration information of the second DNN through AI data including information related to the first DNN, and performs AI up-scaling on the second image 1535 by using the second DNN set using the obtained DNN configuration information, wherein the above operation can reduce memory usage and computational complexity compared to up-scaling by directly analyzing features of the second image 1535.
[0291] In an embodiment, when the second image 1535 includes a plurality of frames, the AI upscaler 1634 may independently obtain DNN configuration information according to a predetermined number of frames, or obtain common DNN configuration information for the entire frames.
[0292] Fig. 20 is a diagram illustrating a second image 1535 including a plurality of frames.
[0293] like Fig. 20 As shown in , the second image 1535 may include frames corresponding to t0 to tn.
[0294] In an example, the AI upscaler 1634 may obtain the DNN configuration information of the second DNN via the AI data, and perform AI upscaling on the frames corresponding to t0 to tn based on the obtained DNN configuration information. That is, the frames corresponding to t0 to tn may be AI upscaled based on the common DNN configuration information.
[0295] In another example, for each group including a predetermined number of frames, the AI up-scaler 1634 may perform AI up-scaling on some of the frames corresponding to t0 to tn (e.g., frames corresponding to t0 to ta) using the "A" DNN configuration information obtained from the AI data, and perform AI up-scaling on the frames corresponding to ta+1 to tb using the "B" DNN configuration obtained from the AI data. The AI up-scaler 1634 may perform AI up-scaling on the frames corresponding to tb+1 to tn using the "C" DNN configuration information. In other words, the AI up-scaler 1634 may independently obtain DNN configuration information for each group including a predetermined number of frames among a plurality of frames, and may perform AI up-scaling on the frames included in each group according to the independently obtained DNN configuration information.
[0296] In another embodiment, the AI upscaler 1634 may independently obtain DNN configuration information for each frame of the second image 1535. That is, when the second image 1535 includes three frames, the AI upscaler 1634 may AI upscale the first frame using the DNN configuration information obtained for the first frame, AI upscale the second frame using the DNN configuration information obtained for the second frame, and AI upscale the third frame using the DNN configuration information obtained for the third frame. Since the mode information, quantization parameter information, etc. can be independently determined for each frame of the second image 1535, the DNN configuration information can be independently obtained according to each frame of the second image 1535 based on the information (prediction mode information, motion information, quantization parameter information, etc.) provided from the first decoder 1632 and the relevant information of the first image 1515 included in the AI data according to the manner of obtaining the DNN configuration information.
[0297] In another example, the AI data may include information indicating until which frame the DNN configuration information obtained based on the AI data is valid. For example, if the AI data includes information that the DNN configuration information is valid until ta frame, the AI upscaler 1634 may perform AI upscaling on t0 to ta frames using the DNN configuration information obtained based on the AI data. If another AI data includes information that the DNN configuration information is valid until tn frame, the AI upscaler 1634 may perform AI upscaling on ta+1 to tn frames based on the DNN configuration information obtained according to another AI data.
[0298] Reference Fig.21 , an AI encoding device 600 for performing AI encoding on an image 1505 will be described.
[0299] Fig.21 is a block diagram illustrating an example configuration of an AI encoding device 600 according to an embodiment.
[0300] Reference Fig.21 , the AI encoding device 600 may include an AI encoder (e.g., including a processing circuit and / or an executable program element) 610 and a transmitter (e.g., including a transmission circuit) 630. The AI encoder 610 may include an AI downscaler (e.g., including a processing circuit and / or an executable program element) 612 and a first encoder (e.g., including a processing circuit and / or an executable program element) 614. The transmitter 630 may include a data processor (e.g., including a processing circuit and / or an executable program element) 632 and a communicator (e.g., including a communication circuit) 634.
[0301] although Fig.21 The AI encoder 610 and the transmitter 630 are shown as separate devices, but the AI encoder 610 and the transmitter 630 may be implemented by a single processor. In this case, a dedicated processor may be implemented, or a combination of a general-purpose processor (such as an AP, a CPU or a GPU and S / W) may be implemented. In addition, in the case of a dedicated processor, a memory processor including a memory for implementing the embodiment or a memory processor for using an external memory may implement the dedicated processor.
[0302] The AI encoder 610 and the transmitter 630 may be composed of a plurality of processors. In this case, a combination of dedicated processors may be implemented, or a combination of a plurality of general-purpose processors (such as an AP, a CPU or a GPU and S / W) may be implemented. The AI downscaler 612 and the first encoder 614 may also be implemented with different processors.
[0303] The AI encoder 610 may include various processing circuits and / or executable program elements, and perform AI downscaling of the image 1505 and first encoding of the first image 1515 , and transmit the AI data and the image data to the transmitter 630 .
[0304] The image data includes data obtained as a result of first encoding the first image 1515. The image data may include data obtained based on pixel values in the first image 1515, for example, residual data that is a difference between the first image 1515 and prediction data of the first image 1515. The image data includes information used in the first encoding process of the first image 1515. For example, the image data may include prediction mode information used for first encoding the first image 1515, motion information, and quantization parameter related information used for first encoding the first image 1515.
[0305] The AI data includes information that enables the AI up-scaler 1634 to AI up-scale the second image 1535 to an up-scaling target corresponding to the down-scaling target of the first DNN. In one example, the AI data may include difference information between the image 1505 and the first image 1515. The AI data may also include information related to the first image 1515. The information related to the first image 1515 may include information about at least one of a resolution of the first image 1515, a bit rate of image data obtained as a result of first encoding the first image 1515, or a codec type used in the first encoding of the first image 1515.
[0306] In an embodiment, the AI data may include an identifier of mutually agreed DNN configuration information such that the second image 1535 is upscaled by the AI to an upscaling target corresponding to the downscaling of the first DNN.
[0307] In an embodiment, the AI data may include DNN configuration information that may be set as the second DNN.
[0308] The AI downscaler 612 may include various processing circuits and / or executable program elements, and obtains the first image 1515 that is AI downscaled by the first DNN from the image 1505. The AI downscaler 612 may determine a downscaling target of the image 1505 based on a predetermined criterion.
[0309] In order to obtain the first image 1515 corresponding to the downscaling target, the AI downscaler 612 may store a plurality of DNN configuration information that may be set as the first DNN. The AI downscaler 612 may obtain DNN configuration information corresponding to the downscaling target from among the plurality of DNN configuration information, and perform AI downscaling on the image 1505 by using the first DNN set using the obtained DNN configuration information.
[0310] Each of the plurality of DNN configuration information may be trained to obtain a predetermined resolution and / or a predetermined image of the first image 1515. For example, any one of the plurality of DNN configuration information may include information for obtaining a first image 1515 having a resolution 1 / 2 times that of the image 1505, for example, a first image 1515 having a resolution of 2K (2048*1080) which is 1 / 2 times that of the image 1505 having 4K (4096*2160), and other DNN configuration information may include information about a first image 1515 having a resolution 1 / 4 times that of the image 1505, for example, a first image 1515 having a resolution of 2K (2048*1080) which is 1 / 4 times that of the image 1505 having 8K (8192*4320).
[0311] According to an embodiment, if the information included in the DNN configuration information (e.g., the number of convolutional layers, the number of filter kernels used for each convolutional layer, the parameters of each filter kernel, etc.) is stored in the form of a lookup table, the AI downscaler 612 may combine selected portions of the lookup table values according to the downscaling target to obtain the DNN configuration information, and may perform AI downscaling on the image 1505 using the obtained DNN configuration information.
[0312] According to an embodiment, the AI downscaler 612 may determine the structure of a DNN corresponding to a downscaling target, and obtain DNN configuration information corresponding to the determined structure of the DNN, for example, parameters of a filter kernel.
[0313] By jointly training the first DNN and the second DNN, a plurality of DNN configuration information for performing AI downscaling on the image 1505 may have an optimal value. Each DNN configuration information includes at least one of the number of convolutional layers included in the first DNN, the number of filter kernels per convolutional layer, or a parameter of each filter kernel.
[0314] The AI downscaler 612 may set the first DNN using the DNN configuration information determined for AI downscaling the image 1505 to obtain a predetermined resolution and / or a predetermined image quality of the first image 1515 via the first DNN. When the DNN configuration information for AI downscaling the image 1505 among the plurality of DNN configuration information is obtained, each layer in the first DNN may process input data based on information included in the DNN configuration information.
[0315] The method of determining the downscaling target by the AI downscaler 612 will be described below. The downscaling target may indicate how much the first image 1515 with a resolution reduced from the image 1505 needs to be obtained.
[0316] In an embodiment, the AI downscaler 612 may determine a downscaling target based on at least one of a compression rate (e.g., a difference in resolution between image 1505 and first image 1515 , a target bitrate), a compression quality (e.g., a bitrate type), compression history information, or a type of image 1505 .
[0317] In an example, the AI downscaler 612 may determine a downscaling target based on a preset compression rate or compression quality input from a user.
[0318] As another example, the AI downscaler 612 may determine a downscaling target using compression history information stored in the AI encoding device 600. For example, based on the compression history information available to the AI encoding device 600, a user's preferred encoding quality or compression rate may be determined, and the downscaling target may be determined based on the encoding quality determined based on the compression history information. For example, the resolution, image quality, etc. of the first image 1515 may be determined based on the encoding quality most commonly used based on the compression history information.
[0319] As another example, the AI downscaler 612 may determine a downscaling target based on the encoding quality that has been used for more than a predetermined threshold according to the compression history information (eg, the average quality of the encoding quality that has been used for more than a predetermined threshold).
[0320] As another example, the AI downscaler 612 may determine a downscaling target based on the resolution, type (eg, file format), etc. of the image 1505 .
[0321] In an embodiment, when the image 1505 includes a plurality of frames, the AI downscaler 612 may determine downscaling targets independently according to a predetermined number of frames, or determine a common downscaling target for the entire frames.
[0322] In an example, the AI downscaler 612 may divide the frames of the image 1505 into a predetermined number of groups and independently determine a downscaling target for each group. The same or different downscaling targets may be determined for each group. For each group, the number of frames included in the group may be the same or different.
[0323] In another example, the AI downscaler 612 may determine the downscaling target independently per frame of the image 1505. For each frame, the same or different downscaling target may be determined.
[0324] Hereinafter, an example structure of the first DNN 700 as an example basis for scaling under AI will be described.
[0325] Fig. 22 is a diagram illustrating a first example DNN 700 for AI downscaling of an image according to an embodiment.
[0326] like Fig. 22 As shown in , the image 1505 is input to the first convolution layer 710. The first convolution layer 710 performs a convolution process on the image 1505 using 32 filter kernels of size 5×5. The 32 feature maps generated as a result of the convolution process are input to the first activation layer 720. The first activation layer 720 may assign nonlinear features to the 32 feature maps.
[0327] The first activation layer 720 determines whether to pass sample values of the feature map output from the first convolution layer 710 to the second convolution layer 730. For example, some sample values of the feature map are activated by the first activation layer 720 and passed to the second convolution layer 730, and some sample values are deactivated by the first activation layer 720 and are not passed to the second convolution layer 730. Information represented by the feature map output from the first convolution layer 710 is highlighted by the first activation layer 720.
[0328] The output 725 of the first activation layer 720 is input to the second convolution layer 730. The second convolution layer 730 performs a convolution process on the input data using 32 filter kernels of a size of 5×5. The 32 feature maps generated as a result of the convolution process are input to the second activation layer 740, and the second activation layer 740 may assign nonlinear features to the 32 feature maps.
[0329] The output 745 of the second activation layer 740 is input to the third convolution layer 750. The third convolution layer 750 performs a convolution process on the input data using one filter kernel of size 5×5. As a result of the convolution process, one image may be output from the third convolution layer 750. The third convolution layer 750 is a layer for outputting a final image and obtains one output using one filter kernel. According to an example, the third convolution layer 750 may output the first image 1515 via the convolution operation result.
[0330] There may be multiple DNN configuration information indicating the number of filter cores, parameters of the filter cores, etc. in the first convolution layer 710, the second convolution layer 730, and the third convolution layer 750 of the first DNN 700, wherein the multiple DNN configuration information should be associated with the multiple DNN configuration information of the second DNN. The association between the multiple DNN configuration information of the first DNN and the multiple DNN configuration information of the second DNN can be achieved by jointly training the first DNN and the second DNN.
[0331] Fig. 22 The first DNN 700 is shown to include three convolutional layers 710, 730, 750 and two activation layers 720, 740, but this is only an example, and according to embodiments, the number of convolutional layers and activation layers may vary widely. In addition, according to embodiments, the first DNN 700 may be implemented via RNN. This case may refer to, for example, changing the CNN structure of the first DNN 700 to RNN according to an example.
[0332] In an example embodiment, the AI downscaler 612 may include at least one ALU for convolution operations and operations of activation layers. The ALU may be implemented with a processor. For convolution operations, the ALU may include a multiplier and an adder, wherein the multiplier is used to perform a multiplication operation between the sample value of the filter kernel and the sample value of the feature map output from the image 1505 or the previous layer, and the adder adds the result values of the multiplication. In addition, for the operation of the activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier is used to multiply the input sample value by the weight used in a predetermined S-shaped function, Tanh function, or ReLU function, and the comparator is used to compare the result of the multiplication with a predetermined value to determine whether to pass the input sample value to the next layer.
[0333] Reference Fig. 22 , the first encoder 614 receiving the first image 1515 from the AI downscaler 612 may perform first encoding on the first image 1515 to reduce the amount of information of the first image 1515. As a result of the first encoding by the first encoder 614, image data corresponding to the first image 1515 may be obtained.
[0334] The data processor 632 may include various processing circuits and / or executable program elements, and performs processing so that at least one of the AI data or the image data can be sent in a predetermined format. For example, if the AI data and the image data are to be sent in the form of a bit stream, the data processor 632 processes the AI data so that the AI data is represented in the form of a bit stream, and sends the AI data and the image data in the form of one bit stream through the communicator 634. As another example, the data processor 632 processes the AI data so that the AI data is represented in the form of a bit stream, and sends each of the bit streams corresponding to the AI data and the bit streams corresponding to the image data through the communicator 634. As another example, the data processor 632 processes the AI data so that the AI data is represented as a frame or a data packet, and sends the image data in the form of a bit stream through the communicator 634 and sends the AI data in the form of a frame or a data packet.
[0335] The transmitter 630 may include various circuits and transmit the AI-encoded data obtained as a result of the AI encoding through a network. The AI-encoded data obtained as a result of the AI encoding includes image data and AI data. The image data and AI data may be transmitted through the same network or different networks.
[0336] In an embodiment, the AI encoded data obtained as a result of processing by the data processor 632 may be stored in a data storage medium including a magnetic medium (such as a hard disk, a floppy disk, a magnetic tape), an optical recordable medium (such as a CD-ROM and a DVD), a magneto-optical medium (such as a magneto-optical floppy disk), and the like.
[0337] In the following, reference will be made to Fig.23 A method of jointly training a first DNN 700 and a second DNN 300 is described.
[0338] Fig.23 is a diagram illustrating an example method for training the first DNN 700 and the second DNN 300 according to an embodiment.
[0339] In an exemplary embodiment, the AI-encoded image 1505 is reconstructed into the third image 1545 through the AI decoding process, and the correlation of the AI encoding process and the AI decoding process is required in order to maintain the similarity between the third image 1545 obtained as a result of the AI decoding and the image 1505. That is, the information lost in the AI encoding process must be reconstructed in the AI decoding process, which requires the joint training of the first DNN 700 and the second DNN 300.
[0340] For accurate AI decoding, it is necessary to reduce Fig.23 The quality loss information 2330 corresponding to the comparison result between the third training image 804 and the original training image 801 shown in FIG. The quality loss information 2330 is used to train both the first DNN 700 and the second DNN 300.
[0341] Will describe Fig.23 The training process shown in .
[0342] exist Fig.23 In the example, the original training image 801 is an image that has not been downscaled by AI, and the first training image 802 is an image that has been downscaled by AI from the original training image 801. The third training image 804 is an image that has been upscaled by AI from the first training image 802.
[0343] The original training image 801 includes a still image or a moving image including a plurality of frames. In an example embodiment, the original training image 801 may include a luminance image extracted from a still image or a moving image including a plurality of frames. In an example embodiment, the original training image 801 may include a patch image extracted from a still image or a moving image consisting of a plurality of frames. When the original training image 801 includes a plurality of frames, the first training image 802, the second training image, and the third training image 804 also include a plurality of frames. When the plurality of frames of the original training image 801 are sequentially input to the first DNN 700, the plurality of frames of the first training image 802, the second training image, and the third training image 804 may be sequentially acquired by the first DNN 700 and the second DNN 300.
[0344] In order to jointly train the first DNN 700 and the second DNN 300, an original training image 801 is input to the first DNN 700. The original training image 801 input to the first DNN 700 is AI down-scaled and output as a first training image 802, and the first training image 802 is input to the second DNN 300. As a result of AI up-scaling for the first training image 802, a third training image 804 is output.
[0345] Reference Fig.23 According to an embodiment, the first training image 802 is input to the second DNN 300, and the second training image obtained by performing the first encoding and the first decoding process on the first training image 802 may be input to the second DNN 300. Any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1 may be used to input the second training image into the second DNN. For example, in the first encoding of the first training image 802 and the first decoding of the image data corresponding to the first training image 802, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1 may be used.
[0346] Reference Fig.23 In addition to outputting the first training image 802 via the first DNN 700, a conventional down-scaling reduced training image 803 is obtained from the original training image 801. The conventional down-scaling may include at least one of bilinear scaling, bicubic scaling, lanzos scaling, or step scaling.
[0347] In order to prevent and / or reduce the structural features of the first image 1515 from significantly deviating from the structural features of the image 1505 , a reduced training image 803 is obtained that retains the structural features of the original training image 801 .
[0348] Before training, the first DNN 700 and the second DNN 300 may be set using predetermined DNN configuration information. As training proceeds, structure loss information 2310, complexity loss information 2320, and quality loss information 2330 may be determined.
[0349] The structural loss information 2310 may be determined based on the comparison result of the reduced training image 803 and the first training image 802. In one example, the structural loss information 2310 may correspond to the difference between the structural information of the reduced training image 803 and the structural information of the first training image 802. The structural information may include various features of the image that may be extracted from the image, such as brightness, contrast, histogram, etc. The structural loss information 2310 indicates the degree to which the structural information of the original training image 801 is preserved in the first training image 802. The smaller the structural loss information 2310 is, the more similar the structural information of the first training image 802 is to the structural information of the original training image 801.
[0350] The complexity loss information 2320 may be determined based on the spatial complexity of the first training image 802. In one example, the total variance value of the first training image 802 may be used as the spatial complexity. The complexity loss information 2320 is related to the bit rate of the image data obtained by first encoding the first training image 802. It is defined that the smaller the complexity loss information 2320 is, the smaller the bit rate of the image is.
[0351] The quality loss information 2330 may be determined based on the comparison result of the original training image 801 and the third training image 804. The quality loss information 2330 may include at least one of the following values: an L1-norm value, an L2-norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance interpolation factor (VIF) value, and a video quality multi-method assessment fusion (VMAF) value for the difference between the original training image 801 and the third training image 804. The quality loss information 2330 indicates the degree to which the third training image 804 is similar to the original training image 801. The smaller the quality loss information 2330 is, the more similar the third training image 804 is to the original training image 801.
[0352] Reference Fig.23 , the structure loss information 2310, the complexity loss information 2320, and the quality loss information 2330 are used to train the first DNN 700, and the quality loss information 2330 is used to train the second DNN 300. That is, the quality loss information 2330 is used to train both the first DNN 700 and the second DNN 300.
[0353] The first DNN 700 may update parameters so that the final loss information determined based on the structure loss information 2310, the complexity loss information 2320, and the quality loss information 2330 is reduced or minimized. The second DNN 300 may update parameters so that the quality loss information 2330 is reduced or minimized.
[0354] The final loss information for training the first DNN 700 and the second DNN 300 may be determined as Equation 3 below.
[0355] [Equation 3]
[0356] LossDS = a*structural loss information + b*complexity loss information + c*quality loss information
[0357] LossUS=d*quality loss information
[0358] In Equation 3, LossDS represents final loss information that should be reduced or minimized for training of the first DNN 700, and LossUS represents final loss information that should be reduced or minimized for training of the second DNN 300. In addition, a, b, c, and d may correspond to predetermined weights.
[0359] The first DNN 700 updates parameters in a direction in which the LossDS of Equation 3 is reduced, and the second DNN 300 updates parameters in a direction in which the LossUS is reduced. When the parameters of the first DNN 700 are updated according to the LossDS derived from the training process, the first training image 802 obtained based on the updated parameters is different from the first training image 802 in the previous training process, so that the third training image 804 is also different from the third training image 804 in the previous training process. When the third training image 804 is different from the third training image 804 in the previous training process, the quality loss information 2330 is also newly determined, and thus, the second DNN 300 updates the parameters. When the quality loss information 2330 is newly determined, the LossDS is also newly determined, and thus, the first DNN 700 updates the parameters according to the newly determined LossDS. That is, the parameter update of the first DNN 700 leads to the parameter update of the second DNN 300, and the parameter update of the second DNN 300 leads to the parameter update of the first DNN 700. In other words, since the first DNN 700 and the second DNN 300 are jointly trained by sharing the quality loss information 2330, the parameters of the first DNN 700 and the parameters of the second DNN 300 can be optimized in relation to each other.
[0360] Referring to Equation 3, it is known that LossUS is determined according to the quality loss information 2330 , but this is only an illustrative example, and LossUS may be determined based on at least one of the structure loss information 2310 or the complexity loss information 2320 and the quality loss information 2330 .
[0361] It has been described that the AI upscaler 1634 of the AI decoding device 1600 and the AI downscaler 612 of the AI encoding device 600 store a plurality of DNN configuration information. A method of training each of the plurality of DNN configuration information stored in the AI upscaler 1634 and the AI downscaler 612 will be described.
[0362] As described in conjunction with Equation 3, in the case of the first DNN 700, the following items are considered to update the parameters: the degree of similarity between the structural information of the first training image 802 and the structural information of the original training image 801 (structural loss information 2310), the bit rate of the image data obtained by the first encoding result of the first training image 802 (complexity loss information 2320), and the difference between the third training image 804 and the original training image 801 (quality loss information 2330).
[0363] The parameters of the first DNN 700 can be updated so that a first training image 802 having structural information similar to that of the original training image 801 can be obtained, and the bit rate of the image obtained from the first encoding is similar to that of the original training image 801, and the second DNN 300 used to perform AI upscaling on the first training image 802 obtains a third training image 804 similar to the original training image 801.
[0364] By adjusting the weights of a, b, and c of Equation 3, the direction in which the parameters of the first DNN 700 are optimized is different. For example, if the weight of b is determined to be high, the parameters of the first DNN 700 may be updated so that importance is given to a bit rate lower than the quality of the third training image 804. In addition, when the weight of c is determined to be high, the parameters of the first DNN 700 may be updated so that importance is given to improving the quality of the third training image 804 or maintaining the structural information of the original training image 801 compared to a higher bit rate.
[0365] In addition, the parameters of the first DNN 700 may differ depending on the type of codec used to first encode the first training image 802, and the second training image to be input to the second DNN 300 may differ depending on the type of codec.
[0366] The parameters of the first DNN 700 and the parameters of the second DNN 300 may be jointly updated based on the weight a, the weight b, the weight c, and the type of the codec used to first encode the first training image 802. Therefore, the parameters of the first DNN 700 and the parameters of the second DNN 300 that are jointly optimized may be determined by determining the weight a, the weight b, and the weight c as predetermined values, determining the type of the codec as a predetermined type, and then training the first DNN 700 and the second DNN 300.
[0367] After changing the weight a, weight b, weight c, and the type of codec, the parameters of the first DNN 700 and the parameters of the second DNN 300 that are optimized in relation to each other can be determined by training the first DNN 700 and the second DNN 300. In other words, when the first DNN 700 and the second DNN 300 are trained while changing the values of the weight a, weight b, weight c, and the type of codec, respectively, a plurality of DNN configuration information that are trained together can be determined by the first DNN 700 and the second DNN 300.
[0368] As mentioned above Fig.19 As described, a plurality of DNN configuration information of the first DNN 700 and the second DNN 300 may be mapped to the first image related information. In order to establish the mapping relationship, the first training image 802 output from the first DNN 700 may be first encoded using a specific codec according to a specific bit rate, and the second training image obtained by first decoding the bit stream obtained by the first encoding may be input to the second DNN 300. That is, after setting an environment so that the first training image 802 of a specific resolution is first encoded at a specific bit rate by a specific codec, by training the first DNN 700 and the second DNN 300, the resolution of the first training image 802, the type of codec used for first encoding the first training image 802, and the DNN configuration information pair mapped to the bit rate of the bit stream obtained as a result of first encoding the first training image 802 may be determined. By changing the resolution of the first training image 802, the type of codec used for the first encoding of the first training image 802, the bit rate of the bit stream obtained by the first encoding of the first training image 802, and the mapping relationship between the first image related information of the first DNN 700 and the second DNN 300 and the multiple DNN configuration information can be determined.
[0369] Fig.24 is a signal flow chart illustrating an example training process of the first DNN 700 and the second DNN 300 by the training apparatus 1000 according to an embodiment.
[0370] Reference Fig.24, the training of the first DNN 700 and the second DNN 300 may be performed by the training device 1000. The training device 1000 may include the first DNN 700 and the second DNN 300. The training device 1000 may be, for example, the AI encoding device 600 or a separate server. The DNN configuration information of the second DNN 300 obtained through the training is stored in the AI decoding device 1600.
[0371] Reference Fig.24 , in operations S840 and S845, the training apparatus 1000 initially sets the DNN configuration information of the first DNN 700 and the second DNN 300. Therefore, the first DNN 700 and the second DNN 300 may operate according to the predetermined DNN configuration information. The DNN configuration information may include information about at least one of the number of convolutional layers included in the first DNN 700 and the second DNN 300, the number of filter kernels for each convolutional layer, the size of the filter kernel for each convolutional layer, and the parameters of each filter kernel.
[0372] In operation S850, the training apparatus 1000 inputs the original training image 801 to the first DNN 700. The original training image 801 may include at least one frame of a still image or a moving image.
[0373] In operation S855, the first DNN 700 processes the original training image 801 according to the initially set DNN configuration information, and outputs a first training image 802 that is AI downscaled from the original training image 801. Fig.23 The first training image 802 output from the first DNN 700 is shown to be directly input to the second DNN 300, but the first training image 802 output from the first DNN 700 may be input to the second DNN 300 through the training device 1000. In addition, the training device 1000 may first encode and first decode the first training image 802 using a predetermined codec, and then input the second training image into the second DNN 300.
[0374] In operation S860, the second DNN 300 processes the first training image 802 or the second training image according to the initially set DNN setting information, and outputs a third training image 804 that is AI upscaled from the first training image 802 or the second training image.
[0375] In operation S865, the training apparatus 1000 calculates complexity loss information 2320 based on the first training image.
[0376] In operation S870 , the training apparatus 1000 calculates the structural loss information 2310 by comparing the reduced training image 803 and the first training image 802 .
[0377] In operation S875 , the training apparatus 1000 calculates quality loss information 2330 by comparing the original training image 801 and the third training image 804 .
[0378] In operation S880, the first DNN 700 updates the initially set DNN configuration information through a back propagation process based on the final loss information. The training apparatus 1000 may calculate the final loss information for training the first DNN 700 based on the complexity loss information 2320, the structure loss information 2310, and the quality loss information 2330.
[0379] In operation S885, the second DNN 300 updates the initially set DNN configuration information through a back propagation process based on the quality loss information or the final loss information. The training apparatus 1000 may calculate the final loss information for training the second DNN 300 based on the quality loss information 2330.
[0380] Then, the training apparatus 1000, the first DNN 700, and the second DNN 300 update the DNN configuration information by repeating operations S850 to S885 until the final loss information is minimized and / or reduced. During each repeated process, the first DNN 700 and the second DNN 300 operate according to the DNN configuration information updated in the previous process.
[0381] In Table 1 below, effects of a case where AI encoding and AI decoding are performed on the image 1505 and a case where the image 1505 is encoded and decoded using HEVC are shown according to an embodiment.
[0382] Table 1
[0383]
[0384] As shown in Table 1, it can be seen that even though the subjective image quality of AI encoding and AI decoding of content including 300 frames of 8K resolution is higher than that of encoding and decoding by HEVC, the bit rate is reduced by 50% or more.
[0385] Fig.25 is a diagram illustrating an apparatus 10 ′ for AI downscaling an image 1505 and an apparatus 20 ′ for AI upscaling a second image 1535 according to an embodiment.
[0386] Device 10' receives image 1505 and provides image data 22 and AI data 23 to device 20' using AI downscaler 2524 and transform-based encoder 2526. In an example embodiment, image data Fig. 22 and Fig.15 The image data corresponds to the AI data 23. Fig.15 In addition, in an exemplary embodiment, the transform-based encoder 2526 is Fig.21 The first encoder 614 corresponds to the AI downscaler 2524 and Fig.21 The AI lower scaler 612 responds accordingly.
[0387] The device 20' receives the AI data 23 and the image data 22, and obtains the third image 1545 using the transform-based decoder 2546 and the AI upscaler 2544. In an example embodiment, the transform-based decoder 2546 and the AI upscaler 2544 are used to obtain the third image 1545. Fig.16 The first decoder 1632 corresponds to the AI upper scaler 2544 and Fig.16 The AI upper scaler 1634 corresponds.
[0388] In an exemplary embodiment, the device 10' comprises a computer program CPU and a memory, wherein the computer program comprises instructions. The computer program is stored in the memory. In an exemplary embodiment, when the computer program is executed by the CPU, the device 10' performs a combination of Fig.25 In an exemplary embodiment, the combination of Fig.25 Describes the functionality.
[0389] In an exemplary embodiment, the device 20' comprises a computer program CPU and a memory, wherein the computer program comprises instructions. The computer program is stored in the memory. In an exemplary embodiment, when the computer program is executed by the CPU, the device 20' performs a combination of Fig.25 In an exemplary embodiment, the combination of Fig.25 Describes the functionality.
[0390] exist Fig.25 , the configuration controller 2522 receives one or more input values 21. In an example embodiment, the one or more input values 21 may include at least one of a target resolution difference for the AI downscaler 2524 and the AI upscaler 2544, a bit rate of the image data 22, a bit rate type of the image data 22 (e.g., a variable bit rate type, a constant bit rate type, an average bit rate type, etc.), or a codec type for the transform-based encoder 2526. The one or more input values 21 may be pre-stored in the device 10', or may include a value received from a user.
[0391] The configuration controller 2522 controls the operation of the AI downscaler 2524 and the transform-based encoder 2526 based on the received input value 21. In an example embodiment, the configuration controller 2522 obtains DNN configuration information for the AI downscaler 2524 according to the received input value 21, and sets the AI downscaler 2524 using the obtained DNN configuration information. In an example embodiment, the configuration controller 2522 may pass the received input value 21 to the AI downscaler 2524, and the AI downscaler 2524 may obtain DNN configuration information for AI downscaling the image 1505 based on the received input value 21. In an example embodiment, the configuration controller 2522 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component, etc.) information to which AI downscaling is applied, tone mapping information of high dynamic range (HDR), etc.) to the AI downscaler 2524 together with the input value 21, and the AI downscaler 2524 may obtain DNN configuration information considering the input value 21 and the additional information. In an example embodiment, the configuration controller 2522 passes at least a portion of the received input value 21 to the transform-based encoder 2526 to cause the transform-based encoder 2526 to first encode the first image 1515 using a specific bit rate value, a specific type of bit rate, and a specific codec.
[0392] AI downscaler 2524 receives image 1505 and performs a combination Fig.15 , 21 , 22, 23 or 24 to obtain a first image 1515.
[0393] In an example embodiment, the AI data 23 is provided to the device 20'. The AI data 23 may include at least one of resolution difference information between the image 1505 and the first image 1515 or information related to the first image 1515. The resolution difference information may be determined based on a target resolution difference of the input value 21, and the information related to the first image 1515 may be determined based on at least one of a target bit rate, a bit rate type, and a codec type. In an example embodiment, the AI data 23 may include parameters used in an AI upscaling process. The AI metric data may be provided to the device 20' from the AI downscaler 2524.
[0394] The first image 1505 is processed by a transform-based encoder 2526 to obtain image data 22, and the image data 22 is sent to the device 20'. The transform-based encoder 2526 can process the first image 2515 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9 or AV1.
[0395] The configuration controller 2542 controls the operation of the AI upper scaler 2544 based on the AI data 23. In an example embodiment, the configuration controller 2542 obtains DNN configuration information for the AI upper scaler 2544 according to the received AI data 23, and sets the AI upper scaler 2544 using the obtained DNN configuration information. In an example embodiment, the configuration controller 2542 passes the received AI data 23 to the AI upper scaler 2544, and the AI upper scaler 2544 may obtain DNN configuration information for AI upper scaling of the second image 1535 based on the AI data 23. In an example embodiment, the configuration controller 2542 may provide additional information (e.g., color format (luminance component, chrominance component, red component, green component, or blue component, etc.) information to which AI upper scaling is applied, tone mapping information of HDR, etc.) to the AI upper scaler 2544 together with the AI data 23, and the AI upper scaler 2544 may obtain the DNN configuration information considering the AI data 23 and the additional information. In an example embodiment, the AI upscaler 2544 may receive AI data 23 from the configuration controller 2542, receive at least one of prediction mode information, motion information, and quantization parameter information from the transform-based decoder 2546, and obtain DNN configuration information based on at least one of the prediction mode information, motion information, and quantization parameter information and the AI data 23.
[0396] The transform-based decoder 2546 processes the image data 22 to reconstruct the second image 1535. The transform-based decoder 2546 may process the image data according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, or AV1.
[0397] The AI up-scaler 2544 may obtain a third image 1545 by performing AI up-scaling on the second image 1535 provided from the transform-based decoder 2546 based on the set DNN setting information.
[0398] The AI down-scaler 2524 includes a first DNN, and the AI up-scaler 2544 may include a second DNN, wherein the ... Fig.23 and Fig.24 The described training method trains DNN configuration information for a first DNN and a second DNN.
[0399] In addition to being applicable to image processing devices (such as set-top boxes), various embodiments can also be applied to all electronic devices capable of image processing, such as servers. In addition, the above various embodiments can be performed by an embedded server set in an electronic device or an external server of the image processing device.
[0400] The methods according to various embodiments may be implemented in the format of software or application installable to existing electronic devices.
[0401] The methods according to various embodiments may be implemented through software upgrade or only hardware upgrade of existing electronic devices.
[0402] At the same time, various embodiments of the present disclosure can be implemented in software, wherein the software includes instructions stored on a machine-readable storage medium that can be read by a machine (e.g., a computer). A device including an image processing device (e.g., image processing device A) according to the disclosed embodiment can call instructions from a storage medium and execute the called instructions. When the instructions are executed by a processor, the processor can directly or use other components under the control of the processor to perform functions corresponding to the instructions. The instructions may include code generated by a compiler or code executable by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a "non-transitory" storage medium may not include a signal, but is tangible, and does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored in a storage medium.
[0403] According to an embodiment, the method according to the above-described embodiment may be included in a computer program product. The computer program product may be traded between a seller and a consumer as a product. The computer program product may be in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or through an application store (e.g., Play Store TM ) to publish the computer program product online, or directly publish the computer program product online. In the case of online publishing, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a server of the manufacturer, a server of the application store, or a machine-readable storage medium (such as a memory of a relay server).
[0404] According to various example embodiments, each element (e.g., module or program) in the above-mentioned elements may include a single entity or multiple entities. According to an embodiment, at least one element or operation in the above-mentioned corresponding elements may be omitted, or at least one other element or operation may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be combined to form a single entity. In this case, the integrated entity may perform the function of at least one function of the element of each element in the multiple elements in the same or similar manner as the function performed by the corresponding element in the multiple elements before integration. The modules, program modules or operations performed by other elements according to various embodiments may be performed continuously, in parallel, repeatedly or heuristically, or at least some operations may be performed according to different orders, at least some operations may be omitted, or other operations may be added thereto.
[0405] Although various embodiments have been shown and described with reference to various example embodiments, the present disclosure is not limited to the specific embodiments or drawings, and those skilled in the art will understand that various changes in form and details may be made thereto without departing from the spirit and scope of the present disclosure, including the claims and their equivalents.
Claims
1. An electronic device, include: The processor is configured as: Using the trained second artificial intelligence AI model to obtain quality information of the image; determining a downscaling ratio of the image and a bit rate of the downscaled image based on the quality information of the image; Downscaling the image based on the determined downscaling ratio using the trained first AI model; and encoding the downscaled image based on the determined bit rate, wherein the second AI model is trained using feature information of the image obtained from the first AI model, The processor is further configured to control downscaling of the current frame based on quality information obtained from the current frame of the image, or control downscaling of the current frame based on quality information predicted from at least one previous frame of the image.
2. The electronic device according to claim 1, in, The quality information of the image includes a plurality of quality identifiers corresponding to different combinations of resolution information and bit rate information related to the image, The processor is configured as follows: determining a downscaling ratio of the image and a bit rate of the downscaled image based on resolution information and bit rate information corresponding to any one of the plurality of quality identifiers, and The image is down-scaled based on the determined down-scaling ratio control, and the down-scaled image is encoded based on the determined bit rate control.
3. The electronic device according to claim 1, further comprising: include: A communication interface, including a communication circuit, The processor is configured to obtain status information of a network to which the communication interface is connected, and control downscaling of the image based on the status information of the network and quality information of the image.
4. The electronic device according to claim 3, in, The image quality information includes a plurality of quality identifiers corresponding to different combinations of information on a plurality of resolutions and information on a plurality of bit rates associated with the image, The processor is configured as follows: determining at least one quality identifier among the plurality of quality identifiers based on network status information; determining a quality identifier among the at least one quality identifier based on a target quality identifier; Down-scaling of the image is controlled based on the resolution information corresponding to the determined one quality identifier, and encoding of the down-scaled image is controlled based on the information on the bit rate corresponding to the determined one quality identifier.
5. The electronic device according to claim 1, in, The second AI model includes at least one neural network layer, and The parameters of the at least one neural network layer are trained together with the parameters of at least some layers included in the first AI model.
6. The electronic device according to claim 1, in, The second AI model includes at least one recurrent neural network (RNN) layer trained to predict quality information of a current frame based on feature information corresponding to at least one previous frame.
7. The electronic device according to claim 1, in, The quality information of the image includes a plurality of quality identifiers corresponding to different combinations of information on a plurality of resolutions and information on a plurality of bit rates associated with the image, and In which, the second AI model is trained based on supervised training using the difference between multiple quality identifiers corresponding to different combinations of information about multiple resolutions and information about multiple bit rates related to the training image and multiple quality identifiers output by inputting the training image into the second AI model as a loss function.
8. The electronic device according to claim 1, in, The processor is configured as: generating a data packet based on the encoded image and transmitting the generated data packet, and The resolution information and the bit rate information are included in the header of the data packet.
9. A system comprising a sending device and a receiving device, the system include: A sending device configured to obtain quality information of an image using a trained second artificial intelligence (AI) model, determine a downscaling ratio of the image and a bit rate of the downscaled image based on the quality information of the image, downscale the image based on the determined downscaling ratio using the trained first AI model, and encode the downscaled image based on the determined bit rate; as well as a receiving device configured to decode the encoded image received from the sending device and obtain an output image by upscaling the decoded image, wherein the second AI model is configured to be trained using feature information of the image obtained from the first AI model, The sending device is configured to control downscaling of the current frame based on quality information obtained from the current frame of the image, or to control downscaling of the current frame based on quality information predicted from at least one previous frame of the image.
10. The system of claim 9, in, The received encoded image includes at least one of bit rate information or resolution information, and The receiving device is configured to decode the encoded image based on the bit rate information included in the received encoded image, and up-scale the decoded image based on the resolution information included in the received encoded image.
11. A method for controlling an electronic device, the method include: Using the trained second artificial intelligence AI model to obtain quality information of the image; determining a downscaling ratio of the image and a bit rate of the downscaled image based on the quality information of the image; Downscaling the image based on the determined downscaling ratio using the trained first AI model; and encoding the downscaled image based on the determined bit rate, wherein the second AI model is trained using feature information of the image obtained from the first AI model, The method further comprises: controlling downscaling of the current frame based on quality information obtained from the current frame of the image, or controlling downscaling of the current frame based on quality information predicted from at least one previous frame of the image.
