Image processing device and operation method thereof

Abstract

An image processing device comprises a memory, a first processor, a second processor, and a video processor, wherein the first processor: acquires an input image and information of the input image; determines a resource allocation amount of the second processor, for executing at least one neural network; controls the second processor to generate a first image quality processed image by performing first image quality processing on an image input into the second processor through the at least one neural network; controls the video processor to generate a second image quality processed image by performing, on the basis of hardware, second image quality processing on an image input into the video processor; and generates an output image through at least one of the first image quality processing and the second image quality processing.

Description

Image processing device and its operating method

[0001] The present disclosure relates to the field of image processing, and more specifically, to a device and method for processing an image to improve the image quality.

[0002] General image processing algorithms can be implemented and executed on various hardware devices equipped with image processing devices. For example, image processing algorithms can be implemented as image processing circuits executed by hardware devices. These image processing circuits each have their own logic tailored to their specific purposes, so they must be implemented separately. It is difficult to perform operations other than those specifically designed for them. Furthermore, the order of operations among image processing circuits can be difficult to change once designed.

[0003] Meanwhile, a variety of image processing algorithms utilizing neural networks are being developed. By implementing the algorithms used in image processing as neural network-based algorithms, the performance of each algorithm can be improved.

[0004] An image processing device according to one embodiment of the present disclosure includes a memory storing one or more instructions, a first processor executing the one or more instructions stored in the memory, a second processor, and a video processor.

[0005] According to one embodiment of the present disclosure, the image processing device obtains an input image and information about the input image by having the first processor individually or collectively execute one or more instructions.

[0006] According to one embodiment of the present disclosure, the first processor individually or collectively executes one or more instructions, thereby determining, based on information of the input image, a resource allocation amount of the second processor for executing at least one neural network among a plurality of neural networks executable by the second processor.

[0007] According to one embodiment of the present disclosure, the first processor individually or collectively executes one or more instructions, thereby controlling the second processor to generate a first image quality processing image by performing first image quality processing on an image input to the second processor through the at least one neural network based on the determined resource allocation amount.

[0008] According to one embodiment of the present disclosure, the first processor individually or collectively executes one or more of the instructions, thereby controlling the video processor to generate a second image processing image by performing second image processing based on hardware on an image input to the video processor.

[0009] According to one embodiment of the present disclosure, the first processor individually or collectively executes one or more of the instructions, thereby generating an output image through at least one of the first image quality processing or the second image quality processing.

[0010] A method of operating an image processing device according to one embodiment includes the steps of: obtaining an input image and information about the input image; determining, based on the information about the input image, a resource allocation amount of the second processor for executing at least one neural network among a plurality of neural networks executable by the second processor; controlling the second processor to generate a first image quality processing image by performing first image quality processing on an image input to the second processor through the at least one neural network based on the determined resource allocation amount; controlling the video processor to generate a second image quality processing image by performing second image quality processing based on hardware on an image input to the video processor; and generating an output image through at least one of the first image quality processing and the second image quality processing.

[0011] According to one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing at least one of the operating methods of an image processing device on a computer may be provided.

[0012] An embodiment of the present disclosure will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0013] FIG. 1 is a diagram illustrating an image processing process of an image processing device according to one embodiment of the present disclosure.

[0014] FIG. 2 is a block diagram illustrating a configuration of an image processing device according to one embodiment of the present disclosure.

[0015] FIG. 3 is a block diagram illustrating the configuration of an image processing unit according to one embodiment of the present disclosure.

[0016] FIG. 4 is a flowchart illustrating the operation of an image processing device according to one embodiment of the present disclosure.

[0017] FIG. 5 is a flowchart illustrating an operation of an image processing device according to one embodiment of the present disclosure to determine a resource allocation amount for a neural network model.

[0018] FIG. 6 is an example of an image processing unit allocated resources to a neural network model based on information of an input image according to one embodiment of the present disclosure.

[0019] FIG. 7 is an example of an image processing unit allocated resources to a neural network model based on information of an input image according to one embodiment of the present disclosure.

[0020] FIG. 8 is an example of an image processing unit allocated resources to a neural network model based on information of an input image according to one embodiment of the present disclosure.

[0021] FIG. 9a, FIG. 9b, FIG. 9c, and FIG. 9d are diagrams showing an operation of an image processing unit performing image processing on an input image according to one embodiment of the present disclosure.

[0022] FIG. 10 is a flowchart illustrating an operation of an image processing device according to one embodiment of the present disclosure to determine a resource allocation amount for a neural network based on characteristic information of an input image.

[0023] FIG. 11 is an example of an image processing unit allocated resources to a neural network based on the amount of motion of an input image according to one embodiment of the present disclosure.

[0024] FIG. 12 is an example of an image processing unit allocated resources to a neural network based on the amount of motion of an input image according to one embodiment of the present disclosure.

[0025] FIG. 13 is an example of an image processing unit allocated resources to a neural network based on the quality of an input image according to one embodiment of the present disclosure.

[0026] FIG. 14 is an example of an image processing unit allocated resources to a neural network based on the quality of an input image according to one embodiment of the present disclosure.

[0027] FIG. 15 is a flowchart illustrating an operation of an image processing device according to one embodiment of the present disclosure to perform image processing based on an activation command.

[0028] FIG. 16 is an example of an image processing unit allocated resources to a neural network based on a deactivation command according to one embodiment of the present disclosure.

[0029] FIG. 17 is an example of an image processing unit allocated resources to a neural network based on a low power mode activation command according to one embodiment of the present disclosure.

[0030] FIG. 18 is a detailed block diagram of an image processing device according to one embodiment of the present disclosure.

[0031] The present disclosure may be subject to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments of the present disclosure, and it should be understood that the present disclosure encompasses all modifications, equivalents, and alternatives falling within the spirit and technical scope of the various embodiments.

[0032] In describing the embodiments, detailed descriptions of related known technologies will be omitted if they are deemed to unnecessarily obscure the gist of the present disclosure. Furthermore, numbers (e.g., "first," "second," etc.) used in the description of the embodiments are merely identifiers used to distinguish one component from another.

[0033] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0034] Additionally, in the present disclosure, when a component is referred to as being “connected” or “connected” to another component, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless there is a specific description to the contrary.

[0035] In addition, components expressed as 'unit', 'module', etc. in the present disclosure may be two or more components combined into one component, or one component may be divided into two or more components with more detailed functions. In addition, each component described below may additionally perform some or all of the functions performed by other components in addition to its own main function, and of course, some of the main functions performed by each component may be exclusively performed by other components.

[0036] Additionally, in the present disclosure, 'image' may refer to a still image, a frame, a moving image composed of a plurality of consecutive still images, or a video.

[0037] Additionally, in this disclosure, "neural network" refers to a representative example of an artificial neural network model that mimics brain neurons, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network (DNN).

[0038] Additionally, in this disclosure, "first image quality processing" refers to AI (artificial intelligence) image quality processing using a neural network. "First image quality processing image" refers to an AI image quality processing image generated through AI image quality processing. Furthermore, "second image quality processing" refers to image quality processing performed on a hardware basis via a video processor. "Second image quality processing image" refers to an image quality processing image generated through a video processor.

[0039] Additionally, in the present disclosure, functions related to AI are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU (Central Processing Unit), an AP (Application Processor), a DSP (Digital Signal Processor), a graphics-only processor such as a GPU (Graphics Processing Unit), a VPU (Vision Processing Unit), or an AI-only processor such as an NPU (Neural Processing Unit). One or more processors control the processing of input data according to predefined operation rules or AI models stored in memory. Alternatively, if one or more processors are AI-only processors, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.

[0040] In the present disclosure, a processor may include various processing circuits and / or multiple processors. For example, the term “processor” as used in the present disclosure, including in the claims, may include various processing circuits including at least one processor, wherein one or more of the at least one processor in the present disclosure may be individually and / or collectively configured to perform various functions described herein. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform various functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. The at least one processor may execute program instructions to achieve or perform various functions.

[0041] FIG. 1 is a diagram illustrating an image processing process of an image processing device according to one embodiment of the present disclosure.

[0042] Referring to FIG. 1, the image processing device (100) of the present disclosure may be an electronic device capable of processing and outputting an image. The image processing device (100) may be implemented in various forms including a display. For example, the image processing device (100) may be implemented in various electronic devices such as a mobile phone, a tablet PC, a digital camera, a camcorder, a laptop computer, a tablet PC, a desktop, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a wearable device, and the like.

[0043] An image processing device (100) according to one embodiment of the present disclosure can perform image processing on an input image (110) to generate an output image (120).

[0044] For example, the image processing device (100) can generate an output image (120) by applying a noise removal algorithm, an upscaling algorithm, a sharpness enhancement algorithm, a contrast enhancement algorithm, a color correction algorithm, and a frame rate up conversion (FRC) algorithm to the input image (110). However, the image processing process performed by the image processing device (100) is not limited to the described example.

[0045] In one embodiment of the present disclosure, the algorithm used in the image processing process may be implemented as an algorithm based on an image processing circuit (20) or as an algorithm based on an image processing neural network (10).

[0046] In one embodiment of the present disclosure, the image processing circuit (20) may include various image processing circuits that perform operations corresponding to various image processing algorithms. Each of the image processing circuits is specialized to perform a single operation, thereby accelerating or efficiently processing the desired image processing. The various image processing circuits included in the image processing circuit (20) may include hardware configurations, circuits, and / or logic required for each operation. For example, the image processing circuit (20) may be designed as a video processor specialized for image processing, but is not limited thereto.

[0047] When an image processing algorithm is implemented as an image processing circuit (20), image processing is accelerated and precise calculations are possible. However, since the image processing circuits each have their own logic suited to their specific purpose, they must be implemented separately, making it difficult to perform any operations other than those specialized for them. Furthermore, the operation order between the image processing circuits, once designed, can be difficult to change.

[0048] Meanwhile, when an image processing algorithm is implemented as an image processing neural network (10), it can learn and adapt to images with various properties and characteristics. Furthermore, the image processing neural network (10) can be updated by training the neural network with new data, or a new image quality processing algorithm can be applied. However, the image processing neural network (10) has difficulty performing precise calculations compared to the image processing circuit (20), and the calculation time may be longer.

[0049] In one embodiment of the present disclosure, the image processing device (100) may include an image processing unit (not shown) in which an image processing algorithm is implemented as an image processing circuit (20) or an image processing neural network (10), taking into account the performance, function, purpose, need for precise control, suitability for AI operations (e.g., convolution operations), etc. of the image processing algorithm. The manufacturer of the image processing device (100) may implement various image processing algorithms as either the image processing circuit (20) or the image processing neural network (10), taking into account the above-described criteria.

[0050] For example, it is assumed that a picture quality processing algorithm for an 8k resolution display is implemented only by an image processing circuit (20) of an image processing device (100). For example, the image processing circuit (20) may include first image processing circuits for processing images having a resolution of 2k or less, a first upscaler for upscaling images having a resolution of 2k or less into images having a resolution of 4k or less, second image processing circuits for processing images having a resolution of 4k, a second upscaler for upscaling images having a resolution of 4k or less into images having a resolution of 8k, and third image processing circuits for processing images having a resolution of 8k.

[0051] In this case, if the resolution of the input image input to the image processing device (100) is 2k or less, the image processing circuit (20) can process the image quality of the input image using the first image processing circuits, upscale it to 4k resolution using the first upscaler, process the image quality using the second image processing circuits, upscale it to 8k resolution using the second upscaler, and process the image quality using the third image processing circuits to generate an output image.

[0052] In contrast, if the resolution of the input image input to the image processing device (100) is 4k, the first image processing circuits and the first upscaler included in the image processing circuit (20) are designed not to be able to process images with a 4k resolution, so the image processing circuit (20) applies the second image processing circuits to the input image.

[0053] In this way, if some of the image processing circuits (e.g., first image processing circuits, first upscaler, etc.) included in the image processing circuit (20) are not used in the image quality processing process according to the information (e.g., resolution information) of the input image input to the image processing device (100), a temporal redundancy problem occurs. Temporal redundancy is a phenomenon of resource waste due to image quality processing in a state where some resources are not used.

[0054] In addition, for example, even if a frame rate conversion (hereinafter, FRC) algorithm for a display supporting a frame rate (or refresh rate) of 120 Hz is implemented only with an image processing circuit (20) of an image processing device (100), a temporal duplication problem occurs.

[0055] For example, if the input image is of a low frame rate (e.g., 30 Hz, 60 Hz), the image processing circuit (20) can generate a high frame rate (e.g., 120 Hz) image from the low frame rate image using the FRC circuit. The FRC algorithm is a technology that improves the frame rate of the video image by predicting a new frame to be added between frames of the video image and compensating for the new frame.

[0056] In contrast, if the input image input to the image processing device (100) is a high frame rate (e.g., 120 Hz), temporal overlap occurs because the FRC circuit is not used in the image quality processing process even though it is designed.

[0057] In this way, if all of the image processing algorithms performed by the image processing device (100) are implemented as image processing circuits (20), it will result in a decrease in image processing performance due to resource waste and time duplication.

[0058] An image processing device (100) according to one embodiment of the present disclosure may implement a part of an image processing algorithm as an image processing neural network (10) and another part as an image processing circuit (20).

[0059] In an image processing device (100) according to one embodiment of the present disclosure, an image processing algorithm, the use of which may vary depending on the properties and characteristics of an input image, may be implemented as an image processing neural network (10), and an image processing algorithm that is more efficient in performing image processing using an image processing circuit (20) may be implemented as an image processing circuit (20). Accordingly, the image processing device (100) may perform various image processing according to input images having various properties and characteristics through the image processing neural network (10) and the image processing circuit (20), and minimize waste of resources for the image processing circuit (20), thereby improving image processing performance.

[0060] For example, an image processing algorithm whose use may vary depending on the properties and characteristics of an input image may include an image processing algorithm whose application may vary depending on the resolution size or frame rate size of the input image. This is described in detail in FIGS. 2 and 3.

[0061] For example, an image processing algorithm that is more efficient in performing image processing using an image processing circuit (20) may include an upscaling algorithm that simply adjusts the image size, or an image processing algorithm that requires minimal computational errors. This will be described in detail in FIGS. 2 and 3.

[0062] An image processing device (100) according to an embodiment of the present disclosure can flexibly adjust whether to allocate computational resources for executing an image processing neural network (10) and the allocation amount in response to input images having various properties and characteristics. For example, the image processing device (100) can determine whether to allocate resources and the allocation amount for an image processing neural network (10) that performs image quality processing related to information of the input image based on information of the input image. For example, when there are two or more pieces of information of an input image, the image processing device (100) can determine a resource allocation ratio for two or more image processing neural networks (10) that perform image quality processing related to each piece of information. Accordingly, the image processing device (100) can perform image quality processing by allocating limited computational resources to the image processing neural network (10) required for each property and characteristic of the input image, thereby maximizing the image quality processing performance of the image processing device (100). This will be described with reference to FIGS. 4 to 14 .

[0063] An image processing device (100) according to one embodiment of the present disclosure may change whether or not to allocate resources to an image processing neural network (10) and the amount of resource allocation according to consumer or system requirements, etc., and this is described in FIGS. 15 to 17.

[0064] Referring to the drawings below, an image processing device (100) according to one embodiment of the present disclosure, in which an image processing algorithm is implemented as an image processing neural network (10) and / or an image processing circuit (20), will be described. In addition, a method in which the image processing device (100) selectively uses an image processing neural network (10) and / or an image processing circuit (20) to perform image processing according to an input image having various properties and characteristics will be described.

[0065] FIG. 2 is a block diagram illustrating the configuration of an image processing device according to one embodiment of the present disclosure. FIG. 3 is a block diagram illustrating the configuration of an image processing unit according to one embodiment of the present disclosure.

[0066] Referring to FIG. 2, an image processing device (100) according to one embodiment of the present disclosure may include a first processor (210), an image processing unit (220), and a memory (230).

[0067] The first processor (210) can control the image processing device (100) as a whole. According to one embodiment, the first processor (210) can execute one or more programs stored in the memory (230). According to one embodiment, the first processor (210) can be composed of one or more processors.

[0068] According to one embodiment, the memory (230) may store various data, programs, or applications for driving and controlling the image processing device (100). The program stored in the memory (230) may include one or more instructions. The program (one or more instructions) or application stored in the memory (230) may be executed by the first processor (210).

[0069] The memory (230) is a configuration for storing various programs or data, and may be configured as a storage medium such as a ROM, a RAM, a hard disk, a CD-ROM, and a DVD, or a combination of storage media. The memory (230) may not exist separately and may be configured to be included in the first processor (210). The memory (230) may be configured as a volatile memory, a non-volatile memory, or a combination of volatile memory and non-volatile memory. A program or at least one instruction for performing operations according to embodiments described below may be stored in the memory (230). The memory (230) may also provide stored data to the first processor (210) at the request of the first processor (210).

[0070] One or more processors included in the first processor (210) according to one embodiment may be a general-purpose processor such as a Central Processing Unit (CPU), an Application Processor (AP), a Digital Signal Processor (DSP), a graphics-only processor such as a Graphics Processing Unit (GPU), a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as a Neural Processing Unit (NPU). Alternatively, according to one embodiment, the first processor (210) may be a circuit device (circuitry) implemented in the form of a System On Chip (SoC) or an Integrated Circuit (IC) that integrates at least one of a CPU, a GPU, a VPU, and an NPU.

[0071] An image processing unit (220) according to one embodiment can perform image processing on video data. An input image can be transmitted to the image processing unit (220) under the control of the first processor (210). The image processing unit (220) decodes and processes the input image signal, and can scale the decoded image signal so that it is sized to a frame to be output on a display. The image processing unit (220) can apply various image processing algorithms to the image to generate an output image that has undergone image processing.

[0072] An image processing unit (220) according to one embodiment may include a second processor (240) and a video processor (250).

[0073] The second processor (240) may include one or more processors for executing an image processing neural network (e.g., 10 in FIG. 1). The second processor (240) may be manufactured in the form of a dedicated hardware chip for artificial intelligence (e.g., NPU). Alternatively, the second processor (240) may be manufactured as part of an existing general-purpose processor (e.g., CPU or AP) or a graphics-only processor (e.g., GPU).

[0074] The video processor (250) may be a processor for executing an image processing circuit (e.g., 20 of FIG. 1). The video processor (250) is a processor specialized in image processing and may include hardware configurations, circuits, logic, etc. required for image processing. For example, the video processor (250) may include at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array), but is not limited thereto.

[0075] The second processor (240) and the video processor (250) can exchange video data corresponding to an image using a frame memory or a line buffer. One or more frames can be stored in the frame memory. The lines constituting the frame can be stored in real time in the line buffer.

[0076] Referring to FIG. 3, a second processor (240) according to one embodiment can execute multiple neural networks under the control of the first processor (210). For example, the multiple neural networks can include the first neural network to the n-th neural network (311, 312, 319). The second processor (240) can perform various image processing algorithms using the first neural network to the n-th neural network (311, 312, 319).

[0077] For example, the plurality of neural networks may include an upscaling model (or super-resolution model) that can convert a low-resolution image into a high-resolution image. The upscaling model is a neural network that learns the differences between a low-resolution image and a high-resolution image and obtains a clearer and more detailed high-resolution image from the low-resolution image. For example, the upscaling model may include various neural networks such as a 2k-to-4k upscaling model, a 4k-to-8k upscaling model, a 4k-to-4k upscaling model, and an 8k-to-8k upscaling model, depending on the input resolution and output resolution.

[0078] For example, the resolution may include SD (Standard Definition), HD (High Definition), FHD (Full High Definition), QHD (Quad High Definition), 4K UHD (Ultra High Definition), 8K UHD (Ultra High Definition), or higher resolution.

[0079] Here, the 2k-to-4k upscaling model may be a neural network that executes a 2k-to-4k upscaling algorithm to generate an output image with a resolution of 4k from an input image with a resolution of 2k (or lower).

[0080] Here, the 4k-to-4k upscaling model could be a neural network that enhances the quality of the image, such as by creating textures or enhancing sharpness for 4k resolution images, instead of resizing the image.

[0081] For example, the plurality of neural networks may include a neural network capable of implementing the FRC algorithm. For example, the plurality of neural networks may include a frame rate conversion (FRC) model, a motion estimation model, a motion compensation model, and the like. For example, a motion estimation model is a neural network that estimates motion between frames of an image and extracts the motion in the form of a motion vector. For example, a motion compensation model is a neural network that obtains a high frame rate image from a low frame rate image by compensating for a new frame using the extracted motion vector.

[0082] For example, multiple neural networks may include a contrast enhancement (or contrast extension) model. The image processing unit (220) may analyze an image using the contrast enhancement model to infer a pixel-to-pixel mapping function (or mapping curve), and implement a contrast enhancement (or contrast extension) algorithm that applies the inferred mapping function to the image to obtain a three-dimensional image.

[0083] For example, multiple neural networks may include a picture quality analysis model that analyzes the picture quality or quality of an input image. For example, the image processing unit (220) may obtain image characteristic information, such as the degree of compression degradation, blurring, degradation, sharpness, noise level, and image resolution, through the picture quality analysis model.

[0084] For example, multiple neural networks may include a classification model capable of identifying the genre to which an input image belongs. For example, the image processing unit (220) may use the classification model to identify and classify the genre of an input image, such as a movie, documentary, news, sports, or animation.

[0085] A video processor (250) according to one embodiment may include a plurality of image processing circuits. The plurality of image processing circuits may include a first image processing circuit to an n-th image processing circuit (331, 332, 339). The video processor (250) may perform various image processing algorithms using the first image processing circuit to an n-th image processing circuit (331, 332, 339). The video processor (250) may perform image processing under the control of the first processor (210).

[0086] For example, the plurality of image processing circuits may include an upscaler. The upscaler may be configured as a circuit for an upscaling algorithm. The upscaler may include one or more upscalers that support processing of images of different resolutions. For example, the upscaler may include at least one of a first upscaler implementing a 2k-to-4k upscaling algorithm and a second upscaler implementing a 4k-to-8k upscaling algorithm.

[0087] For example, the plurality of image processing circuits may include a color correction circuit. The color correction circuit may correspond to an image processing circuit that performs an image color enhancement algorithm, a display dispersion correction algorithm (e.g., color, brightness, color temperature correction, etc.), a tone mapping algorithm, an HDR (High Dynamic Range) image processing algorithm, etc.

[0088] In an image processing device (100) according to one embodiment, depending on the performance, function, purpose, need for precision control, suitability for AI operations (e.g., convolution operations) of the image processing algorithm, the image processing algorithm may be implemented as an image processing neural network executed in a second processor (240), or as an image processing circuit executed in a video processor (250).

[0089] In an image processing device (100) according to one embodiment, an image processing algorithm may be implemented as an image processing circuit or an image processing neural network depending on the purpose of use of the image processing algorithm.

[0090] For example, an upscaling algorithm may be implemented as an upscaler to simply resize an image. Alternatively, for example, an upscaling algorithm may be implemented as an upscaling model to perform upscaling in response to an input image with various properties or characteristics, or to process an image, such as creating texture or enhancing sharpness, in addition to resizing the image.

[0091] In an image processing device (100) according to one embodiment, when the computational error of an image processing algorithm needs to be minimized, the image processing algorithm may be implemented as an image processing circuit.

[0092] For example, in the case of an image color improvement algorithm, a display dispersion correction (calibration) algorithm, a tone mapping algorithm, an HDR (High Dynamic Range) image processing algorithm, etc., it is necessary to precisely adjust the inherent deviation of the display to a certain error or less. Accordingly, the above-described algorithm may be implemented as an image processing circuit (e.g., a color correction circuit) instead of an image processing neural network. However, the present invention is not limited thereto, and depending on the implementation, some of the above-described algorithms may be implemented as an image processing neural network taking into account the performance of the image processing device (100).

[0093] In an image processing device (100) according to one embodiment, if the image processing algorithm is suitable for AI operation, the image processing algorithm may be implemented as an image processing neural network.

[0094] For example, while existing image processing algorithms each required their own unique logic, AI-based image processing technologies typically utilize the same operations even when their purposes are different. For example, an upscaling algorithm and a noise removal algorithm, although serving different purposes, can be executed on the second processor (240) through the same operations, such as convolution operations. In this way, image processing algorithms can be implemented on the second processor (240) depending on whether they can be executed by the same processor.

[0095] According to one or more of the above-described criteria, the image processing device (100) including the image processing unit (220) can allocate resources of the second processor (240) for executing a neural network according to information of the input image, characteristic information of the input image, consumer or system requirements, etc., and generate an output image with image quality processing by controlling at least one of the second processor (240) or the video processor (250).

[0096] According to one embodiment, a first processor (210) can obtain an input image and information about the input image.

[0097] According to one embodiment, the first processor (210) may determine a resource allocation amount of the second processor (240) for executing at least one neural network among a plurality of neural networks executable by the second processor (240) based on information of the input image. According to one embodiment, the first processor (210) may also determine a resource allocation amount of the second processor (240) for executing at least one neural network based on at least one of information of the input image, characteristic information, or an activation command.

[0098] According to one embodiment, the first processor (210) may control the second processor (240) to generate a first image quality processing image by performing first image quality processing on an image input to the second processor (240) through at least one neural network in response to a determined resource allocation amount. The first image quality processing may correspond to AI image quality processing. The first image quality processing image may correspond to an AI image quality processing image generated through AI image quality processing.

[0099] For example, the first processor (210) can control the second processor (240) to generate an upscaled image through an upscaling model.

[0100] For example, the first processor (210) can control the second processor (240) to generate an image with an improved frame rate through a neural network that can implement the FRC algorithm.

[0101] According to one embodiment, the first processor (210) may control the video processor (250) to generate a second image quality processing image by performing second image quality processing through at least one image processing circuit on an image input to the video processor (250). The second image quality processing may correspond to hardware-based image quality processing performed through an image quality processing circuit included in the video processor (250). The second image quality processing image may correspond to an image quality processing image generated through hardware-based image quality processing.

[0102] For example, the first processor (210) can control the video processor (250) to generate an upscaled image through an upscaler.

[0103] For example, the first processor (210) can control the video processor (250) to generate a color-corrected image through a color correction circuit.

[0104] An image processing device (100) according to one embodiment can improve image quality processing performance by adaptively changing whether to allocate computational resources to a neural network and the amount of computational resources allocated based on information of an input image, characteristic information of the input image, consumer or system requirements, etc.

[0105] FIG. 4 is a flowchart illustrating the operation of an image processing device according to one embodiment of the present disclosure.

[0106] Referring to FIG. 4, in operation 410, an image processing device (100) according to one embodiment of the present disclosure can obtain an input image and information about the input image.

[0107] The image processing device (100) can receive input images through an external device connected via wired or wireless means. For example, the image processing device (100) can be connected to an external device via an input / output interface such as HDMI and receive various images. Alternatively, for example, the image processing device (100) can be connected to an external device via a communication module such as Wi-Fi or WLAN and receive images.

[0108] The image processing device (100) can obtain information about the input image through metadata of the input image. For example, the information about the input image can include bitrate information (e.g., 40 Mbps), codec information (e.g., H.264, HEVC), resolution information (e.g., 4k, 8k), frame rate information (e.g., 60 Hz, 120 Hz), etc. of the input image. The frame rate can be referred to as frames per second or a refresh rate.

[0109] In operation 420, the image processing device (100) according to one embodiment of the present disclosure may determine a resource allocation amount of the second processor (240) for executing at least one neural network among a plurality of neural networks executable by the second processor (240) based on information of the input image.

[0110] For example, in the image processing device (100), an upscaling algorithm, a color correction algorithm, etc. may be implemented as a hardware circuit that runs in the video processor (250). For example, the image processing device (100) may include an upscaler (e.g., 650 of FIG. 6) in which an upscaling algorithm is implemented, and a color correction circuit (e.g., 670 of FIG. 6) in which a color correction algorithm is implemented. The image processing device (100) may further include a motion compensation circuit (e.g., 660 of FIG. 7) in which a frame rate conversion algorithm is implemented.

[0111] For example, in the image processing device (100), an upscaling algorithm, a frame rate conversion algorithm, a contrast enhancement algorithm, etc. may be implemented as a neural network running in the second processor (240). For example, the image processing device (100) may include an upscaling model in which an upscaling algorithm is implemented (e.g., 610 of FIG. 6 and 810 of FIG. 8), and a contrast enhancement model in which a contrast enhancement algorithm is implemented (e.g., 620 of FIG. 6). The image processing device (100) may further include a model in which a frame rate conversion algorithm is implemented, for example, a motion estimation model (e.g., 710 of FIG. 7 and 820 of FIG. 8), a motion compensation model, and an FRC model.

[0112] An image processing device (100) according to one embodiment of the present disclosure can perform optimal resource allocation for various neural networks executed in a second processor (240) so as to have optimized image quality processing performance in response to an input image having various information. The first processor (210) can control the second processor (240) to perform optimal resource allocation for the neural network in response to various image scenarios (e.g., various properties and characteristics of an input image).

[0113] An image processing device (100) according to one embodiment of the present disclosure can determine whether to allocate resources to a neural network and the amount of resource allocation based on information about an input image. When an input image is received, the image processing device (100) can allocate resources to a neural network related to the input image in real time. Here, real-time resource allocation may mean not only resource allocation simultaneously with the reception of the input image, but also resource allocation after a predetermined period of time has elapsed since the input image was received.

[0114] An image processing device (100) according to one embodiment of the present disclosure may determine whether to allocate resources to each of two or more neural networks based on two or more pieces of input image information. For example, the image processing device (100) may determine the ratio of resource allocation amounts of the second processor (240) to two or more neural networks based on values ​​representing each of two or more pieces of input image information.

[0115] For example, the image processing device (100) may determine the resource allocation of the second processor (240) for the first neural network that performs image processing on the first information among the plurality of neural networks based on the first information of the input image. Here, the first information may be resolution information such as 2k, 4k, 8k, etc., and the first neural network may be an upscaling model.

[0116] For example, the image processing device (100) may determine the resource allocation of the second processor (240) for a second neural network that performs image processing on the second information among a plurality of neural networks based on the second information of the input image. Here, the second information may be frame rate information such as 60 Hz, 120 Hz, etc., and the second neural network may be a motion estimation model.

[0117] For example, the image processing device (100) can determine the ratio of resource allocation for the first neural network and resource allocation for the second neural network based on a value representing first information of the input image (e.g., resolution size) and a value representing second information of the input image (e.g., frame rate size).

[0118] For example, when the resolution of the input image is low, the image processing device (100) can increase the resource allocation ratio for the first neural network. For example, when the frame rate of the input image is low, the image processing device (100) can increase the resource allocation ratio for the second neural network.

[0119] For example, the image processing device (100) may not allocate resources to the first neural network when the resolution of the input image is high. For example, the image processing device (100) may not allocate resources to the second neural network when the frame rate of the input image is high.

[0120] This is explained in detail in Fig. 5.

[0121] In operation 430, the image processing device (100) according to one embodiment of the present disclosure may generate a first image quality processing image by performing first image quality processing on an image input to the second processor (240) through at least one neural network in response to the determined resource allocation amount. For example, the first processor (210) may control the second processor (240) to generate the first image quality processing image.

[0122] Here, the first image quality processing can correspond to AI image quality processing. The first image quality processing image can correspond to an AI image quality processing image generated through AI image quality processing.

[0123] For example, the image input to the second processor (240) may be an input image (or original image) received by the image processing device (100) or an image subjected to second quality processing through the video processor (250). Depending on the order of the first quality processing and the second quality processing, the image input to the second processor (240) may vary. For example, if the first quality processing is performed before the second quality processing, the image input to the second processor (240) may be an original image received by the image processing device (100). In another example, if the second quality processing is performed before the first quality processing, the image input to the second processor (240) may be an image subjected to second quality processing through the video processor (250).

[0124] The image processing device (100) can perform first image quality processing on an image input to the second processor (240) through at least one neural network, as the resources of the second processor (240) are allocated to at least one neural network. The image processing device (100) can generate an AI-processed image by performing AI image quality processing on the image through at least one neural network.

[0125] For example, when the resolution of an input image is low, the image processing device (100) can generate an image with improved resolution by performing upscaling on the input image through the first neural network.

[0126] For example, when the frame rate of the input image is low, the image processing device (100) can obtain motion vector information for the input image through the second neural network. Based on the motion vector information and the input image, the image processing device (100) can generate an image with an improved frame rate by performing motion compensation processing using a motion compensation circuit within the video processor (250) to be described later.

[0127] The image processing device (100) may generate an AI-processed image by sequentially or in parallel executing multiple neural networks as the resources of the second processor (240) are allocated to multiple neural networks (e.g., see FIGS. 9a, 9b, 9c, and 9d).

[0128] In operation 440, the image processing device (100) according to one embodiment of the present disclosure may generate a second image quality processing image by performing second image quality processing based on hardware on an image input to the video processor (250). For example, the first processor (210) may control the video processor (250) to generate the second image quality processing image.

[0129] Here, the second image quality processing may correspond to hardware-based image quality processing performed through an image quality processing circuit implemented in the video processor (250). The second image quality processing image may correspond to an image quality processing image generated through hardware-based image quality processing.

[0130] For example, the image input to the video processor (250) may be an input image (or original image) received by the image processing device (100) or an image subjected to first quality processing through the second processor (240). Depending on the order of the first quality processing and the second quality processing, the image input to the video processor (250) may vary. For example, if the first quality processing is performed before the second quality processing, the image input to the video processor (250) may be an image subjected to first quality processing through the second processor (240). In another example, if the second quality processing is performed before the first quality processing, the image input to the video processor (250) may be an original image received by the image processing device (100).

[0131] For example, when the resolution of the input image is low and the resources of the second processor (240) are not allocated to the first neural network, the image processing device (100) can generate an image with improved resolution by performing upscaling on the input image through an upscaler.

[0132] For example, the image processing device (100) can generate an image with an improved frame rate by performing motion compensation processing based on motion vector information obtained from the second neural network and an input image.

[0133] In operation 450, an image processing device (100) according to one embodiment of the present disclosure can generate an output image through first image quality processing and second image quality processing.

[0134] The image processing device (100) can generate an image with image quality processing by performing only the first image quality processing, only the second image quality processing, or both the first image quality processing and the second image quality processing.

[0135] The image processing device (100) can efficiently generate a first image with image quality processing by changing a neural network to allocate computational resources based on information of an input image and performing first image quality processing using various neural networks.

[0136] For example, since image quality processing can be performed on each image using the limited computational resources of the second processor (240), the image quality processing performance of the image processing device (100) can be maximized. For example, the image processing device (100) can generate a clearer and more detailed high-resolution image by increasing the execution time of the upscaling model for a low-resolution and high-frame-rate image. For example, the image processing device (100) can improve the accuracy of motion amount estimation by increasing the execution time of the motion estimation model for a high-resolution and low-frame-rate image.

[0137] In addition, since the image processing device (100) implements the image processing algorithm as a neural network instead of an image processing circuit, temporal overlap that occurs due to the image processing circuit designed in the image processing device (100) not being used can be minimized.

[0138] In addition, in the image processing device (100), the image processing algorithm that requires minimal computational error can be precisely processed into second image quality using the image processing circuit, so that the image quality processing performance can be improved.

[0139] FIG. 5 is a flowchart illustrating an operation of an image processing device according to one embodiment of the present disclosure to determine a resource allocation amount for a neural network.

[0140] In Fig. 5, the first information is resolution information, and the second information is frame rate information.

[0141] In operation 510, an image processing device (100) according to one embodiment of the present disclosure may determine a resource allocation amount for a first neural network based on first information of an input image, for example, resolution information.

[0142] In operation 520, an image processing device (100) according to one embodiment of the present disclosure may determine a resource allocation amount of a second processor (240) for a second neural network based on second information of an input image, for example, frame rate information.

[0143] For example, if the input image is low-resolution, the image processing device (100) may determine to allocate resources to the first neural network (operation 530). For example, if the input image is high-resolution, the image processing device (100) may determine not to allocate resources to the first neural network (operation 540). For example, low resolution may refer to 2k or 4k resolution, and high resolution may refer to 8k resolution.

[0144] For example, when the resolution of the input image is 2k, the first neural network to which the resources of the second processor (240) are allocated may include an upscaling model capable of processing image quality beyond 2k resolution, for example, a 2k-to-4k upscaling model, a 4k-to-8k upscaling model, etc. For example, when the resolution of the input image is 4k, the first neural network to which the resources of the second processor (240) are allocated may include an upscaling model capable of processing image quality beyond 4k resolution, for example, a 4k-to-8k upscaling model, etc.

[0145] For example, if the input image has a low frame rate, the image processing device (100) may determine to allocate resources to the second neural network (operation 550). For example, if the input image has a high frame rate, the image processing device (100) may determine not to allocate resources to the second neural network (operation 560). For example, the low frame rate may represent 30 Hz or 60 Hz, and the high frame rate may represent 120 Hz.

[0146] Here, resource allocation refers to the act of selecting one of the processes ready in memory and allocating processor resources to it. For example, the resource allocation amount may represent the time allocated for the second processor (240) to perform operations on the selected process, i.e., resource allocation time.

[0147] In one embodiment of the present disclosure, when the image processing device (100) acquires two or more pieces of information about an input image, it can determine the ratio of resource allocations for two or more neural networks based on values ​​representing each piece of information.

[0148] For example, when the input image has a low resolution and a high frame rate, the image processing device (100) may determine that the resource allocation for the first neural network is greater than the resource allocation for the second neural network (see FIG. 6).

[0149] For example, when the input image has a high resolution and a low frame rate, the image processing device (100) may determine that the resource allocation for the first neural network is smaller than the resource allocation for the second neural network (see FIG. 7).

[0150] For example, if the input image has a low resolution and low frame rate, the image processing device (100) may determine that the resource allocation for the first neural network corresponds to the resource allocation for the second neural network. In this case, the resource allocation for the first neural network and the resource allocation for the second neural network may be the same (or similar). For example, the ratio of the resource allocation for the first neural network to the resource allocation for the second neural network may be 1:1 or similar to 1:1 (see FIG. 8).

[0151] For example, when the input image has high resolution and a high frame rate, the image processing device (100) may allocate resources to neural networks other than the first neural network and the second neural network. For example, when the input image has high resolution and a high frame rate, there is no need to use the first neural network and the second neural network, so the image processing device (100) may not allocate the second processor (240) resources to the first neural network and the second neural network.

[0152] FIG. 6 is an example of an image processing unit allocated resources to a neural network based on information of an input image according to one embodiment of the present disclosure.

[0153] In Fig. 6, neural networks to which resources of the second processor (240) are allocated are exemplarily illustrated when the resolution information of the image input to the image processing device (100) is 2k and the frame rate information is 120Hz. In Fig. 6, the area of ​​each neural network block is illustrated as corresponding to the allocated amount of resources of the second processor (240).

[0154] In one embodiment, the first processor (210) may allocate resources of the second processor (240) to the upscaling model (610) based on resolution information of the input image. The first processor (210) may control the second processor (240) to perform upscaling through the resource-allocated upscaling model (610). In some cases, the first processor (210) may also control the video processor (250) to perform upscaling through an upscaler (650) implemented in the video processor (250).

[0155] The upscaling model (610) may include upscaling models having various input resolutions and output resolutions.

[0156] In one embodiment, the first processor (210) may allocate resources of the second processor (240) to an upscaling model (610) having various input resolutions and output resolutions depending on the resolution of the input image and the target resolution.

[0157] For example, if the resolution of the input image is 2k, the first processor (210) can allocate the resources of the second processor (240) to an upscaling model capable of processing image quality beyond 2k resolution. For example, the first processor (210) can allocate the resources of the second processor (240) to a 2k-to-4k upscaling model and a 4k-to-8k upscaling model. In this case, the image processing unit (220) can operate as shown in FIG. 9d, which will be described later.

[0158] Alternatively, in one embodiment, the first processor (210) may allocate resources of the second processor (240) to various types of upscaling models (610) by further considering the type of upscaler (650) included in the video processor (250).

[0159] For example, if the upscaler (650) implemented in the video processor (250) includes a first upscaler that implements a 2k-to-4k upscale algorithm, the first processor (210) may not allocate resources of the second processor (240) to the 2k-to-4k upscale model, but may allocate resources only to the 4k-to-8k upscale model. Or, for example, if the upscaler (650) implemented in the video processor (250) includes a second upscaler that implements a 4k-to-8k upscale algorithm, the first processor (210) may allocate resources only to the 2k-to-4k upscale model.

[0160] Alternatively, in one embodiment, the first processor (210) may allocate the resources of the second processor (240) to various types of upscaling models (610) by further considering the remaining computational load of the second processor (240).

[0161] For example, if the remaining computational amount of the second processor (240) is small, the first processor (210) may not allocate the resources of the second processor (240) to the upscaling model (610). In this case, the image processing unit (220) may perform upscaling on the input image using the upscaler (650). Alternatively, for example, if the remaining computational amount of the second processor (240) is large, the first processor (210) may allocate the resources of the second processor (240) to the upscaling model (610).

[0162] The image processing unit (220) can generate a 4k resolution image through either the first upscaler or the 2k-to-4k upscaling model, and can generate an 8k resolution image through either the second upscaler or the 4k-to-8k upscaling model.

[0163] For example, the image processing unit (220) can generate an 8k resolution image through a 2k-to-4k upscaling model and a 4k-to-8k upscaling model.

[0164] Or, for example, the image processing unit (220) can generate a 4k resolution image from an input image through a first upscaler, and generate an 8k resolution image from the 4k resolution image through a 4k-to-8k upscaling model, thereby generating a high-quality image including high-frequency components in units of 8k pixels.

[0165] Alternatively, for example, the image processing unit (220) can reduce the amount of computation by generating a 4k resolution image through a 2k-to-4k upscaling model and generating an 8k resolution image through a second upscaler. The image processing device (100) can determine whether to allocate resources to the neural network depending on whether the purpose of upscaling is to enhance performance or reduce the amount of computation.

[0166] Meanwhile, since the frame rate is 120 Hz, the first processor (210) may not allocate the resources of the second processor (240) to the model implementing the FRC algorithm. The first processor (210) may further allocate the resources of the second processor (240) to the contrast enhancement model (620) to perform a contrast enhancement algorithm for the input image. Here, the amount of resources allocated to the upscaling model (610) is exemplified as being greater than the amount of resources allocated to the contrast enhancement model (620), but is not limited thereto.

[0167] The image processing unit (220) can generate a color-corrected image by performing color correction on an image input to the video processor (250) through a color correction circuit (670).

[0168] The image processing unit (220) can generate an output image using at least one of the second processor (240) and the video processor (250). For example, the output image can be an image that has been upscaled, contrast-enhanced, and color-corrected from an input image.

[0169] An image processing device (100) according to one embodiment can generate a clearer and more detailed high-resolution image by increasing the execution time of an upscaling model for a low-resolution and high-frame-rate image.

[0170] FIG. 7 is an example of an image processing unit allocated resources to a neural network based on information of an input image according to one embodiment of the present disclosure.

[0171] In Fig. 7, neural networks to which resources of the second processor (240) are allocated are exemplarily illustrated when the resolution information of the image input to the image processing device (100) is 8k and the frame rate information is 60Hz. In Fig. 7, the area of ​​each neural network block is illustrated as corresponding to the allocated amount of resources of the second processor (240).

[0172] For example, the first processor (210) may allocate the resources of the second processor (240) to a model implementing the FRC algorithm based on the frame rate of the input image being 60 Hz. For example, the model implementing the FRC algorithm may include a motion estimation model (710). In FIG. 7, since the motion compensation circuit (660) is implemented in the video processor (250), the resources of the second processor (240) may not be allocated to the motion compensation model or the FRC model. However, the present invention is not limited thereto, and if the motion compensation circuit (660) is omitted, the resources of the second processor (240) may also be allocated to the motion compensation model or the FRC model.

[0173] Meanwhile, since the resolution is 8k, the first processor (210) may not allocate resources of the second processor (240) to the upscaling model. In order to perform a contrast enhancement algorithm for the input image, the first processor (210) may further allocate resources of the second processor (240) to the contrast enhancement model (620). Here, the amount of resources allocated to the motion estimation model (710) may be greater than the amount of resources allocated to the contrast enhancement model (620), but is not limited thereto.

[0174] The first processor (210) can control the second processor (240) to obtain motion vector information of an image input to the second processor (240) through the motion estimation model (710) as the resources of the second processor (240) are allocated to the motion estimation model (710). The second processor (240) can extract motion vector information of an input image through the motion estimation model (710).

[0175] The first processor (210) can control the video processor (250) to generate a high frame rate image by performing motion compensation processing based on motion vector information and an input image. The video processor (250) can compensate for frames for the input image based on motion vector information through a motion compensation circuit (660).

[0176] The image processing unit (220) can generate an output image using at least one of the second processor (240) and the video processor (250). For example, the output image can be an image with a frame rate enhanced, a contrast ratio enhanced, and a color corrected image from the input image.

[0177] According to one embodiment, an image processing device (100) can improve the accuracy of motion amount estimation by increasing the execution time of a motion estimation model for high-resolution and low-frame-rate images.

[0178] FIG. 8 is an example of an image processing unit allocated resources to a neural network based on information of an input image according to one embodiment of the present disclosure.

[0179] In Fig. 8, neural networks to which resources of the second processor (240) are allocated are exemplarily illustrated when the resolution information of the image input to the image processing device (100) is 4k and the frame rate information is 60Hz. In Fig. 8, the area of ​​each neural network block is illustrated as corresponding to the allocated amount of resources of the second processor (240).

[0180] For example, the first processor (210) may allocate the resources of the second processor (240) to the upscaling model (810) based on the resolution of the input image being 4k. The upscaling model (810) may include an upscaling model capable of processing image quality beyond 4k resolution, for example, a 4k-to-8k upscaling model.

[0181] For example, the first processor (210) may allocate resources of the second processor (240) to the motion estimation model (820) as the frame rate is 60 Hz.

[0182] For example, the first processor (210) may further allocate resources of the second processor (240) to a contrast enhancement model (620) to perform a contrast enhancement algorithm for the input image.

[0183] For example, the amount of resources allocated to the upscaling model (810) may be the same as or similar to the amount of resources allocated to the motion estimation model (820).

[0184] The image processing unit (220) can generate an 8k resolution image from a 4k resolution image through an upscaling model (810) that implements a 4k-to-8k upscaling algorithm. If the resources of the second processor (240) are allocated to the upscaling model (810), the upscaler (650) (e.g., the second upscaler) implemented in the video processor (250) may not be used.

[0185] The image processing unit (220) can generate an output image using at least one of the second processor (240) and the video processor (250). For example, the output image can be an image that has been upscaled, frame rate enhanced, contrast ratio enhanced, and color corrected from an input image.

[0186] FIG. 9a, FIG. 9b, FIG. 9c, and FIG. 9d are diagrams showing an operation of an image processing unit performing image processing on an input image according to one embodiment of the present disclosure.

[0187] Referring to FIGS. 9a to 9c, various sequences in which the image processing unit (220a, 220b, 220c, 220d) executes multiple neural networks to which the resources of the second processor (240) are allocated are described.

[0188] Referring to FIG. 9A, an image processing unit (220a) according to one embodiment may sequentially perform first image quality processing on an input image in the order of an upscaling model (910), a contrast enhancement model (920), and a motion estimation model (930). For example, the upscaling model (910) may process an input image to generate a first image having a higher resolution than the input image. The first image output from the upscaling model (910) may be input to the contrast enhancement model (920). The contrast enhancement model (920) may perform image quality processing on the input first image to generate a second image having an enhanced contrast ratio than the first image. The second image output from the contrast enhancement model (920) may be input to the motion estimation model (930). The motion estimation model (930) may output motion vector information indicating the amount of motion of the input second image. Additional information output from the motion estimation model (930), for example, motion vector information, may be input to the video processor (250) along with the second image. The additional information and the second image input to the video processor (250) may be image quality processed through the motion compensation circuit (660) and the color correction circuit (670). The video processor (250) may generate an output image.

[0189] Referring to FIG. 9b, an image processing unit (220b) according to one embodiment can independently execute an upscaling model (910) and a motion estimation model (930) to perform first image quality processing on an input image, and sequentially execute the upscaling model (910) and the contrast enhancement model (920).

[0190] Referring to FIG. 9c, an image processing unit (220c) according to one embodiment may sequentially perform first image quality processing on an input image in the order of an upscaling model (910), a contrast enhancement model (920), and an FRC model (930). The FRC model (930) may perform image quality processing on a second image to generate a third image having an improved frame rate than the second image.

[0191] However, the execution order of the neural network illustrated in FIGS. 9a, 9b, and 9c is not limited, and the execution order of the upscaling model (910), the contrast enhancement model (920), and the FRC model (930) may be different.

[0192] Referring to FIG. 9d, an image processing unit (220d) according to one embodiment can improve the resolution of an input image of 2k or less by using a 2k-to-4k upscaling model (911) and a 4k-to-8k upscaling model (912). For example, the 2k-to-4k upscaling model (911) can process an input image to generate a first image having a higher resolution than the input image. The first image output from the 2k-to-4k upscaling model (911) can be input to the 4k-to-8k upscaling model (912). The 4k-to-8k upscaling model (912) can process the image quality of the input first image to generate a second image having a higher resolution than the first image. The second image can be image quality-processed through a color correction circuit (670).

[0193] FIG. 10 is a flowchart illustrating an operation of an image processing device according to one embodiment of the present disclosure to determine a resource allocation amount for a neural network based on characteristic information of an input image.

[0194] Referring to FIG. 10, in operation 1010, an image processing device (100) according to an embodiment of the present disclosure may obtain an input image and information about the input image. Operation 1010 may correspond to operation 410 of FIG. 4.

[0195] In operation 1020, an image processing device (100) according to one embodiment of the present disclosure can obtain characteristic information of an input image by analyzing the input image.

[0196] In one embodiment, the characteristic information of the input image may include at least one of the amount of motion of the input image, quality characteristics of the input image, noise information of the input image, brightness level of the input image, and genre of the input image.

[0197] For example, the image processing device (100) may perform a feature analysis algorithm to analyze the features of an input image. The feature analysis algorithm may extract feature information by analyzing at least one of the amount of motion of the input image, quality features of the input image, noise information of the input image, brightness level of the input image, and genre of the input image. The feature analysis algorithm may be implemented as a feature analysis model executed through the second processor (240). Alternatively, the feature analysis algorithm may be implemented as a feature analysis circuit executed through the video processor (250). For example, the feature analysis model may correspond to the image quality analysis model or classification model described above in FIG. 3.

[0198] For example, the image processing device (100) can obtain brightness information of an input image by histogramizing the brightness distribution of pixels constituting the input image.

[0199] In operation 1030, an image processing device (100) according to one embodiment of the present disclosure may determine a resource allocation amount of a second processor (240) for executing at least one neural network based on information of an input image and characteristic information of the input image.

[0200] For example, the image processing device (100) can obtain the amount of motion of the input image by analyzing the input image. If the input image corresponds to a low resolution and low frame rate, the image processing device (100) can determine the resource allocation ratio of the upscaling model and the motion estimation model based on the amount of motion of the input image. This will be described with reference to FIGS. 11 and 12.

[0201] For example, the image processing device (100) can obtain quality information of the input image by analyzing the input image. If the input image corresponds to a low resolution and low frame rate, the image processing device (100) can determine whether to allocate resources to a specific upscaling model based on the quality information of the input image. The upscaling model may vary depending on the input resolution and output resolution. This will be described with reference to FIGS. 13 and 14.

[0202] For example, the image processing device (100) can obtain noise information of the input image by analyzing the input image. If the input image has a lot of noise, the image processing device (100) can allocate more resources to the noise removal model.

[0203] For example, the image processing device (100) may determine the resource allocation based on the genre of the input image. For example, if the input image is a sports image, the image processing device (100) may allocate more resources to the motion estimation model because the image contains a lot of movement.

[0204] In operation 1040, the image processing device (100) according to one embodiment of the present disclosure may control the second processor (240) to generate a first image quality processing image by performing first image quality processing on an image input to the second processor (240) through at least one neural network in response to the determined resource allocation amount. Operation 1040 may correspond to operation 430 of FIG. 4.

[0205] In operation 1050, the image processing device (100) according to one embodiment of the present disclosure may control the video processor (250) to generate a second image quality processing image by performing second image quality processing based on hardware on an image input to the video processor (250). Operation 1050 may correspond to operation 440 of FIG. 4.

[0206] In operation 1060, an image processing device (100) according to an embodiment of the present disclosure may generate an output image through first image quality processing and second image quality processing. Operation 1060 may correspond to operation 450 of FIG. 4.

[0207] FIG. 11 is an example of an image processing unit allocated resources to a neural network based on the amount of motion of an input image according to one embodiment of the present disclosure.

[0208] In Fig. 11, neural networks to which resources of the second processor (240) are allocated are exemplarily illustrated when the resolution information of an image input to an image processing device (100) is 4k, the frame rate information is 60Hz, and the movement of the input image is small. The resource allocation for the contrast ratio enhancement model (620) may correspond to Fig. 8.

[0209] Referring to FIG. 11, the image processing device (100) can obtain motion amount information of the input image by analyzing the input image. For example, if the input image has a first motion amount or less, the image processing device (100) can determine that the input image is an image with little motion.

[0210] Even if the input image is a 4k, 60Hz image as in FIG. 8, the image processing device (100) can allocate more resources to the upscaling model (1110) if the image has little movement.

[0211] When the first processor (210) determines that the input image is an image with little motion, it may allocate more resources of the second processor (240) to the upscaling model (1110) and less resources of the second processor (240) to the motion estimation model (1120). For example, the resource allocation for the upscaling model (1110) and the resource allocation for the motion estimation model (1120) may be 6:4, but is not limited thereto.

[0212] FIG. 12 is an example of an image processing unit allocated resources to a neural network based on the amount of motion of an input image according to one embodiment of the present disclosure.

[0213] In Fig. 12, neural networks to which resources of the second processor (240) are allocated are exemplarily illustrated when the resolution information of an image input to an image processing device (100) is 4k, the frame rate information is 60Hz, and the input image has a lot of movement. The resource allocation for the contrast ratio enhancement model (620) may correspond to Fig. 8.

[0214] Referring to FIG. 12, the image processing device (100) can obtain motion amount information of the input image by analyzing the input image. For example, if the input image has a second motion amount, the image processing device (100) can determine that the input image is an image with a lot of motion.

[0215] Even if the input image is a 4k, 60Hz image as in FIG. 8, the image processing device (100) can allocate more resources to the motion estimation model (1220) if the image has a lot of movement.

[0216] When the first processor (210) determines that the input image is an image with a lot of motion, it can allocate more resources of the second processor (240) to the motion estimation model (1120) and less resources of the second processor (240) to the upscaling model (1110). For example, the resource allocation for the upscaling model (1110) and the resource allocation for the motion estimation model (1120) may be 4:6, but is not limited thereto.

[0217] In one embodiment, for images with a lot of motion, users may not perceive differences in fine texture or sharpness, so increasing the frame rate can provide maximum image quality. Accordingly, the image processing device (100) may allocate more resources to the frame rate-enhancing algorithm to efficiently provide maximum image quality.

[0218] Meanwhile, in one embodiment, the image processing device (100) may determine the resource allocation based on the genre of the input image. For example, if the input image is a sports image, the image processing device (100) may allocate more resources to the motion estimation model (1220) because the image contains a lot of movement.

[0219] FIG. 13 is an example of an image processing unit allocated resources to a neural network based on the quality of an input image according to one embodiment of the present disclosure.

[0220] In Fig. 13, an upscaling model (1310) to which resources of a second processor (240) are allocated is exemplarily illustrated when the resolution information of an image input to an image processing device (100) is 4k, the frame rate information is 60Hz, and the input image is a low-quality image. The resource allocation for the motion estimation model and the contrast ratio enhancement model can correspond to Fig. 8, and is omitted and illustrated in Fig. 13.

[0221] Referring to FIG. 13, the image processing device (100) can obtain quality information of the input image by analyzing the input image. For example, even if the input image has the same resolution and frame rate, the quality of the image may vary depending on the transmission speed of the network, the degree of compression, etc. For example, the image processing device (100) can determine that the input image is a low-quality image by analyzing the degree of compression degradation, degree of blur, degree of degradation, sharpness, degree of noise, and resolution of the input image.

[0222] If the input image is a low-quality image with the same 4k, 60Hz resolution, the image processing device (100) may allocate the resources of the second processor (240) to an upscaling model (1310) having the same input resolution and output resolution as the resolution of the input image in order to generate texture or enhance sharpness of the image using the upscaling model (1310). For example, the upscaling model (1310) may be a 4k-to-4k upscaling model. The upscaling model (1310) having the same input resolution and output resolution can improve the quality of the image instead of adjusting the size of the image.

[0223] The image processing device (100) can control the second processor (240) to generate a high-quality first image by inputting an input image to the upscaling model (1310). The high-quality first image can be input to the second upscaler (652) of the video processor (250). The image processing device (100) can control the video processor (250) to generate an upscaled image by inputting the high-quality first image to the second upscaler (652). The second upscaler (652) can be an upscaling circuit that receives a 4k image, performs upscaling, and outputs an 8k image.

[0224] The image processing device (100) can perform quality enhancement on the input image using the upscaling model (1310) and can perform resolution enhancement on the input image using the second upscaler (652). For example, the amount of computation required for the upscaling model increases in proportion to the output resolution, but the image processing device (100) can minimize the amount of computation by using the second upscaler (652) implemented in hardware to adjust the image size.

[0225] Meanwhile, in FIG. 13, the first upscaler (651) is exemplified as being implemented in the video processor (250). In this case, since the input image has a 4k resolution, the first upscaler (651) may not be used. However, this is not limited thereto, and in some cases, the first upscaler (651) may not be implemented in the video processor (250).

[0226] FIG. 14 is an example of an image processing unit allocated resources to a neural network based on the quality of an input image according to one embodiment of the present disclosure.

[0227] In Fig. 14, an upscaling model (1410) to which resources of a second processor (240) are allocated is exemplarily illustrated when the resolution information of an image input to an image processing device (100) is 4k, the frame rate information is 60Hz, and the input image is a high-quality image. The resource allocation for the motion estimation model and the contrast ratio enhancement model can correspond to Fig. 8, and is omitted and illustrated in Fig. 14.

[0228] Referring to FIG. 14, the image processing device (100) can obtain quality information of the input image by analyzing the input image. For example, the image processing device (100) can determine that the input image is a high-quality image by analyzing the input image. For example, the image processing device (100) can identify that the input image is a high-quality image when the image is received from a Blu-ray disc player or input through a high-performance network, etc. For example, a high-quality image may contain high-frequency information (e.g., sharpness, detail, etc.) at the pixel level.

[0229] The image processing device (100) can allocate the resources of the second processor (240) to the upscaling model (1410) when the input image is the same 4k, 60Hz, and is a high-quality image. The upscaling model (1410) can be an upscaling model in which the input resolution corresponds to the resolution of the input image and the output resolution corresponds to the target resolution, for example, a 4k-to-8k upscaling model.

[0230] The image processing device (100) can control the second processor (240) to generate an upscaled image by inputting the input image into the upscaling model (1410).

[0231] Since the input image contains high-frequency information in units of 4k pixels, there is no need to obtain additional image quality improvement by using an upscaling model (e.g., 4k-to-4k) with the same input resolution and output resolution. Instead, the image processing device (100) can generate high-frequency information in units of pixels at the target resolution by allocating the resources of the second processor (240) to an upscaling model (e.g., 4k-to-8k) with the same output resolution as the target resolution (e.g., 8k). Accordingly, when executing the upscaling model (1410), the required computational amount is greater than when executing the second upscaler (652) implemented in hardware, but a higher quality output image can be generated than when executing the second upscaler (652), so that maximum image quality processing performance can be implemented.

[0232] Meanwhile, in FIG. 14, the first upscaler (651) and the second upscaler (652) are exemplified as being implemented in the video processor (250). In this case, since the input image has a 4k resolution, the first upscaler (651) may not be used. In addition, although the input resolution of the second upscaler (652) corresponds to the resolution of the input image, the second upscaler (652) may not be used for maximum performance of the image processing device (100).

[0233] However, this is not limited thereto, and in some cases, the first upscaler (651) and the second upscaler (652) may not be implemented in the video processor (250).

[0234] FIG. 15 is a flowchart illustrating an operation of an image processing device according to one embodiment of the present disclosure to perform image processing based on an activation command.

[0235] Referring to FIG. 15, in operation 1510, an image processing device (100) according to an embodiment of the present disclosure may obtain an input image and information about the input image. Operation 1510 may correspond to operation 410 of FIG. 4.

[0236] In operation 1520, an image processing device (100) according to one embodiment of the present disclosure may receive a command to activate a picture quality processing function.

[0237] For example, the image processing device (100) may receive a deactivation command for the operation of a neural network that performs image processing corresponding to an image processing function. For example, the image processing function may correspond to the above-described image processing algorithm, such as an upscaling algorithm, a FRC algorithm, a contrast ratio enhancement algorithm, etc.

[0238] Alternatively, for example, the image processing device (100) may receive an activation command for a subtitle provision function. For example, the subtitle provision function may correspond to an algorithm for generating subtitles by analyzing an input image. The subtitle provision function may be performed by a neural network implemented in the second processor (240), but is not limited thereto.

[0239] Alternatively, for example, the image processing device (100) may receive an activation command for a low power mode.

[0240] An activation command according to one embodiment of the present disclosure may be received via an input interface, such as a touch screen, microphone, keyboard, etc., but is not limited thereto. For example, the image processing device (100) may receive an activation command from a user via an input interface.

[0241] In operation 1530, an image processing device (100) according to one embodiment of the present disclosure may determine a resource allocation amount of a processor for at least one neural network based on information of an input image and an activation command.

[0242] For example, the image processing device (100) may allocate the resources of the second processor (240) to the remaining neural networks excluding one or more neural networks performing the image processing function based on a command to deactivate the image processing function.

[0243] For example, the image processing device (100) may recover all resources allocated to a neural network performing an image processing function and reallocate them to another neural network. Here, the operation of recovering and reallocating resources allocated to a neural network may correspond to the operation of turning off a specific program and turning on another program.

[0244] For example, the image processing device (100) may reduce the resource allocation of the second processor (240) for each of the plurality of neural networks based on a low-power mode activation command. For example, the image processing device (100) may reallocate resources so that only a portion of the total resource amount of the second processor (240) is used.

[0245] For example, the image processing device (100) may additionally allocate resources of a second processor (240) to a neural network that performs the subtitle providing function based on an activation command for the subtitle providing function.

[0246] In operation 1540, the image processing device (100) according to one embodiment of the present disclosure may control the second processor (240) to generate a first image quality processing image by performing first image quality processing on an image input to the second processor (240) through at least one neural network in response to the determined resource allocation amount. Operation 1540 may correspond to operation 430 of FIG. 4.

[0247] In operation 1550, the image processing device (100) according to one embodiment of the present disclosure may control the video processor (250) to generate a second image quality processing image by performing second image quality processing based on hardware on an image input to the video processor (250). Operation 1550 may correspond to operation 440 of FIG. 4.

[0248] In operation 1560, an image processing device (100) according to an embodiment of the present disclosure may generate an output image through first image quality processing and second image quality processing. Operation 1560 may correspond to operation 450 of FIG. 4.

[0249] FIG. 16 is an example of an image processing unit allocated resources to a neural network based on a deactivation command according to one embodiment of the present disclosure.

[0250] In Fig. 16, neural networks to which resources of the second processor (240) are allocated are exemplarily illustrated when the resolution information of the image input to the image processing device (100) is 4k, the frame rate information is 60Hz, and the image processing device (100) receives an FRC function deactivation command. The resource allocation for the contrast ratio enhancement model (620) may correspond to Fig. 8.

[0251] Referring to FIG. 16, the first processor (210) may receive a deactivation command for the FRC function. Based on the deactivation command, the first processor (210) may reclaim resource allocation for the motion estimation model and reallocate it to another neural network, for example, an upscaling model (1610).

[0252] The image processing device (100) can quickly perform operations on other image quality processing neural networks by making maximum use of the remaining second processor (240) resources according to the deactivation command.

[0253] FIG. 17 is an example of an image processing unit allocated resources to a neural network based on a low power mode activation command according to one embodiment of the present disclosure.

[0254] In Fig. 17, when the resolution information of an image input to an image processing device (100) is 4k, the frame rate information is 60Hz, and the image processing device (100) receives a low power mode activation command, neural networks to which resources of a second processor (240) are allocated are exemplarily illustrated. The ratio of resource allocations for an upscaling model (1710), a motion estimation model (1720), and a contrast ratio enhancement model (1730) may correspond to Fig. 8.

[0255] Referring to FIG. 17, the first processor (210) can receive an activation command for a low-power mode. The activation command for the low-power mode can be received through an input interface or can be set in the system.

[0256] The first processor (210) may change the resource allocation for the upscaling model (1710), the motion estimation model (1720), and the contrast enhancement model (1730) based on the activation command. For example, the resource allocation for each of the upscaling model (1710), the motion estimation model (1720), and the contrast enhancement model (1730) may be reduced while maintaining the resource allocation ratios for the upscaling model (1710), the motion estimation model (1720), and the contrast enhancement model (1730). The first processor (210) may reclaim a portion, for example, 50%, of the resources for the upscaling model (1710), the motion estimation model (1720), and the contrast enhancement model (1730).

[0257] The image processing device (100) can increase power consumption efficiency by using the remaining second processor (240) resources to a minimum according to a low power mode activation command.

[0258] FIG. 18 is a detailed block diagram of an image processing device according to one embodiment of the present disclosure.

[0259] Referring to FIG. 18, the image processing device (1800) may include a tuner unit (1840), a processor (1801), a display (1820), a communication unit (1850), a detection unit (1830), an input / output unit (1870), a video processing unit (1880), an audio processing unit (1885), an audio output unit (1860), a memory (1802), and a power supply unit (1895).

[0260] A tuner unit (1840) according to one embodiment can select and tune only the frequency of a channel to be received by an image processing device (1800) among many radio wave components through amplification, mixing, resonance, etc. of a broadcast signal received wired or wirelessly. The broadcast signal includes audio, video, and additional information (e.g., EPG (Electronic Program Guide)).

[0261] The tuner unit (1840) can receive broadcast signals from various sources, such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and internet broadcasting. The tuner unit (1840) can also receive broadcast signals from sources, such as analog broadcasting or digital broadcasting.

[0262] The communication unit (1850) can transmit and receive data or signals with an external device or server. For example, the communication unit (1850) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, a LAN module, an Ethernet module, a wired communication module, etc. In this case, each communication module may be implemented in the form of at least one hardware chip.

[0263] The Wi-Fi module and Bluetooth module perform communication in the Wi-Fi and Bluetooth modes, respectively. When using the Wi-Fi module or Bluetooth module, various connection information such as the SSID and session key are first transmitted and received, and after establishing a communication connection using this, various information can be transmitted and received. The wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0264] A sensing unit (1830) according to one embodiment detects a user's voice, a user's image, or a user's interaction, and may include a microphone (1831), a camera unit (1832), and a light receiving unit (1833).

[0265] The microphone (1831) receives the user's spoken voice. The microphone (1831) can convert the received voice into an electrical signal and output it to the processor (1801).

[0266] The optical receiver (1833) receives an optical signal (including a control signal) from an external control device through an optical window (not shown) of a bezel of the display (1820), etc. The optical receiver (1833) can receive an optical signal corresponding to a user input (e.g., touch, pressing, touch gesture, voice, or motion) from the control device. A control signal can be extracted from the received optical signal under the control of the processor (1801).

[0267] The input / output unit (1870) according to one embodiment can receive video (e.g., moving picture, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.) from the outside of the image processing device (1800). The input / output unit (1870) can include any one of a High-Definition Multimedia Interface (HDMI), a Mobile High-Definition Link (MHL), a Universal Serial Bus (USB), a Display Port (DP), a Thunderbolt, a Video Graphics Array (VGA) port, an RGB port, a D-subminiature (D-SUB), a Digital Visual Interface (DVI), a component jack, and a PC port.

[0268] A video processing unit (1880) according to one embodiment performs processing on video data received by an image processing device (1800). The video processing unit (1880) may perform various image processing operations, such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on the video data. The video processing unit (1880) may correspond to the image processing unit (220) of FIG. 2.

[0269] Additionally, the processor (1801) may include at least one of a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), and a Video Processing Unit (VPU). Alternatively, according to an embodiment, the processor (1801) may be implemented in the form of a System On Chip (SoC) that integrates at least one of a CPU, a GPU, and a VPU. Alternatively, the processor (1801) may further include an NPU (Neural Processing Unit). Alternatively, the processor (1801) may further include an ASIC.

[0270] A processor (1801) according to one embodiment may include at least one of a first processor (210) and a second processor (240). In one embodiment, the first processor (210) and the second processor (240) may be implemented as a single integrated chip.

[0271] According to one embodiment, the memory (1802) can store various data, programs or applications for driving and controlling the image processing device (1800).

[0272] Additionally, the program stored in the memory (1802) may include one or more instructions. The program (one or more instructions) or application stored in the memory (1802) may be executed by the processor (1801).

[0273] According to one embodiment, a processor (1801) may acquire an input image by executing one or more instructions stored in the memory (1802). The input image may be an image previously stored in the memory (1802) or an image received from an external device via a tuner unit (1840) or a communication unit (1850). In addition, the input image may be an image on which various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion are performed in a video processing unit (1880).

[0274] A display (1820) according to one embodiment converts image signals, data signals, OSD signals, control signals, etc. processed by a processor (1801) to generate a driving signal. The display (1820) may be implemented as a plasma display panel (PDP), a liquid crystal display (LCD), an organic light-emitting diode (OLED), a flexible display, etc., and may also be implemented as a three-dimensional display (3D display). In addition, the display (1820) may be configured as a touch screen and may be used as an input device in addition to an output device.

[0275] The audio processing unit (1885) processes audio data. The audio processing unit (1885) may perform various processing operations, such as decoding, amplification, and noise filtering, on audio data. Meanwhile, the audio processing unit (1885) may include multiple audio processing modules to process audio corresponding to multiple contents.

[0276] The audio output unit (1860) outputs audio included in a broadcast signal received through the tuner unit (1840) under the control of the processor (1801). The audio output unit (1860) can output audio (e.g., voice, sound) input through the communication unit (1850) or the input / output unit (1870). In addition, the audio output unit (1860) can output audio stored in the memory (1802) under the control of the processor (1801). The audio output unit (1860) can include at least one of a speaker, a headphone output terminal, or an S / PDIF (Sony / Philips Digital Interface:) output terminal.

[0277] The power supply unit (1895) supplies power input from an external power source to components inside the image processing device (1800) under the control of the processor (1801). In addition, the power supply unit (1895) can supply power output from one or more batteries (not shown) located inside the image processing device (1800) to the internal components under the control of the processor (1801).

[0278] The memory (1802) can store various data, programs or applications for driving and controlling the image processing device (1800) under the control of the processor (1801).

[0279] A processor (1801) according to one embodiment can obtain an input image and information about the input image. A processor (1801) according to one embodiment can determine a resource allocation of the processor (1801) for executing at least one neural network among a plurality of neural networks based on the information about the input image. A processor (1801) according to one embodiment can generate a first image quality processing image by performing a first image quality processing on the image through at least one neural network in response to the determined resource allocation. A processor (1801) according to one embodiment can generate a second image quality processing image by performing a second image quality processing based on hardware. A processor (1801) according to one embodiment can generate an output image through the first image quality processing and the second image quality processing.

[0280] According to one embodiment, the first processor (210) obtains an input image and information about the input image by executing one or more instructions.

[0281] According to one embodiment, the first processor (210) determines a resource allocation amount of the second processor (240) for executing at least one neural network among a plurality of neural networks executable by the second processor (240) based on information of the input image.

[0282] According to one embodiment, the first processor (210) controls the second processor (240) to generate a first image quality processing image by performing first image quality processing on an image input to the second processor (240) through the at least one neural network in response to the determined resource allocation amount.

[0283] According to one embodiment, the first processor (210) controls the video processor (250) to generate a second image quality processing image by performing second image quality processing based on hardware on an image input to the video processor (250).

[0284] According to one embodiment, the first processor (210) generates an output image through the first image quality processing and the second image quality processing.

[0285] According to one embodiment, the first processor (210) may determine the resource allocation of the second processor (240) for a first neural network that performs image quality processing on the first information among the plurality of neural networks based on the first information of the input image by executing one or more of the instructions.

[0286] Based on the second information of the input image, the resource allocation of the second processor (240) for the second neural network that performs image quality processing on the second information among the plurality of neural networks can be determined.

[0287] The ratio of the resource allocation for the first neural network and the resource allocation for the second neural network can be determined according to a value representing the first information of the input image and a value representing the second information of the input image.

[0288] The first information may include resolution information and the second information may include frame rate information.

[0289] According to one embodiment, the first processor (210) may, by executing one or more of the instructions, determine that the resource allocation for the first neural network (610) is greater than the resource allocation for the second neural network when the input image is of low resolution and high frame rate.

[0290] According to one embodiment, the first processor (210) may, by executing one or more of the instructions, determine that the resource allocation for the first neural network is smaller than the resource allocation for the second neural network (710) when the input image is of high resolution and low frame rate.

[0291] According to one embodiment, the first processor (210) may, by executing one or more of the instructions, determine that the resource allocation for the first neural network (810) corresponds to the resource allocation for the second neural network (820) when the input image is of low resolution and low frame rate.

[0292] According to one embodiment, the first processor (210) may, by executing one or more of the instructions, determine whether to allocate resources of the second processor (240) to the upscaling model (610) based on at least one of the resolution of the input image, whether the video processor (250) includes an upscaler (650), and the remaining computational amount of the second processor (240).

[0293] According to one embodiment, the first processor (210) can control the second processor (240) to generate an upscaled image corresponding to the first image quality processing image by upscaling an image input to the second processor (240) through the upscaling model (610) by executing the one or more instructions, as the resources of the second processor (240) are allocated to the upscaling model (610).

[0294] According to one embodiment, the first processor (210) can control the video processor (250) to generate an upscaled image corresponding to the second image quality processing image by upscaling an image input to the video processor (250) by executing one or more of the instructions, as the upscaler (650) is implemented in the video processor (250).

[0295] According to one embodiment, the first processor (210) may determine the resource allocation amount of the second processor (240) for the motion estimation model (930) among the plurality of neural networks by executing the one or more instructions, as the input image corresponds to a low frame rate.

[0296] According to one embodiment, the first processor (210) may control the second processor (240) to obtain motion vector information of an image input to the second processor (240) through the motion estimation model (930) based on the resource allocation of the second processor (240) for the motion estimation model (930) by executing one or more of the instructions.

[0297] According to one embodiment, the first processor (210) can control the video processor (250) to generate a high frame rate image corresponding to the second image quality processing image by performing motion compensation processing based on the motion vector information and the input image by executing the one or more instructions.

[0298] The second processor (240) may be configured to perform first image quality processing on the input image through a plurality of calculation units.

[0299] The above video processor (250) may be configured to perform second image quality processing on the input image through a single operator for each image processing circuit.

[0300] The above video processor (250) may include at least one of an upscaler, a scatter correction circuit, a color difference correction circuit, a high-quality image quality processing circuit, and a motion compensation circuit.

[0301] According to one embodiment, the first processor (210) can obtain characteristic information of the input image by analyzing the input image by executing one or more instructions.

[0302] According to one embodiment, the first processor (210) may determine the resource allocation amount of the second processor (240) to the at least one neural network based on characteristic information of the input image by executing the one or more instructions.

[0303] The characteristic information of the input image may include at least one of the amount of motion of the input image, quality characteristics of the input image, noise information of the input image, brightness level of the input image, and genre of the input image.

[0304] According to one embodiment, the first processor (210) can obtain the amount of motion of the input image by executing one or more of the instructions.

[0305] According to one embodiment, the first processor (210) may, by executing one or more of the instructions, allocate more of the second processor (240) resources to the first neural network as the amount of motion in the input image is small. As the amount of motion in the input image is large, the second processor (240) may allocate more of the second processor (240) resources to the second neural network.

[0306] According to one embodiment, the first processor (210) can obtain quality characteristics of the input image by executing one or more of the instructions.

[0307] According to one embodiment, the first processor (210) may, by executing one or more instructions, allocate the resources of the second processor (240) to a first upscaling model in which the input resolution and the output resolution correspond to the resolution of the input image, if the input image is of low quality. If the input image is of high quality, the resources of the second processor (240) may be allocated to a second upscaling model in which the input resolution corresponds to the resolution of the input image and the output resolution corresponds to the target resolution.

[0308] According to one embodiment, the first processor (210) can receive a deactivation command for an image processing function by executing one or more of the instructions.

[0309] According to one embodiment, the first processor (210) may, by executing one or more of the instructions, allocate the resources of the second processor (240) to the remaining neural networks except for one neural network that performs the image processing function based on the deactivation command.

[0310] According to one embodiment, the first processor (210) can receive an activation command for a low power mode by executing one or more of the instructions.

[0311] According to one embodiment, the first processor (210) may reduce the resource allocation of the second processor (240) for each of the plurality of neural networks based on the activation command by executing the one or more instructions.

[0312] A method of operating an image processing device (100) according to one embodiment includes the steps of: acquiring an input image and information about the input image; determining a resource allocation of the second processor (240) for executing at least one neural network among a plurality of neural networks executable by the second processor (240) based on the information about the input image; controlling the second processor (240) to generate a first image quality processing image by performing first image quality processing on an image input to the second processor (240) through the at least one neural network in response to the determined resource allocation; controlling the video processor (250) to generate a second image quality processing image by performing second image quality processing based on hardware on an image input to the video processor (250); and generating an output image through the first image quality processing and / or the second image quality processing.

[0313] The step of determining the resource allocation of the second processor (240) may include the step of determining the resource allocation of the second processor (240) for a first neural network among the plurality of neural networks that performs image quality processing on the first information of the input image, based on the first information of the input image, and the step of determining the resource allocation of the second processor (240) for the second neural network among the plurality of neural networks that performs image quality processing on the second information, based on the second information of the input image. The ratio of the resource allocation for the first neural network and the resource allocation for the second neural network may be determined according to a value representing the first information of the input image and a value representing the second information of the input image.

[0314] The first information includes resolution information and the second information includes frame rate information,

[0315] The step of determining the resource allocation of the second processor (240) may include a step of determining that the resource allocation for the first neural network is greater than the resource allocation for the second neural network when the input image is low-resolution and high-frame rate, a step of determining that the resource allocation for the first neural network is less than the resource allocation for the second neural network when the input image is high-resolution and low-frame rate, and a step of determining that the resource allocation for the first neural network corresponds to the resource allocation for the second neural network when the input image is low-resolution and low-frame rate.

[0316] The method may further include a step of determining a resource allocation of the second processor (240) for a motion estimation model among the plurality of neural networks when the input image corresponds to a low frame rate, a step of controlling the second processor (240) to obtain motion vector information of an image input to the second processor (240) through the motion estimation model based on the resource allocation of the second processor (240) for the motion estimation model, and a step of controlling the video processor (250) to generate a high frame rate image corresponding to the second quality-processed image by performing motion compensation processing based on the motion vector information and the input image.

[0317] The method may further include a step (1020) of obtaining characteristic information of the input image by analyzing the input image, and a step (1030) of determining a resource allocation amount of the second processor (240) to the at least one neural network based on the characteristic information of the input image. The characteristic information of the input image may include at least one of the amount of motion of the input image, quality characteristics of the input image, noise information of the input image, brightness level of the input image, and genre of the input image.

[0318] The step of acquiring characteristic information of the input image may include a step of acquiring the amount of motion of the input image. The step of determining the amount of resource allocation of the second processor (240) may further include a step of allocating more resources of the second processor (240) to the first neural network as the amount of motion of the input image is small, and further allocating more resources of the second processor (240) to the second neural network as the amount of motion of the input image is large.

[0319] The step of acquiring characteristic information of the input image may include a step of acquiring quality characteristics of the input image. The step of determining the resource allocation of the second processor (240) may include a step of allocating the second processor (240) resources to a first upscaling model in which the input resolution and the output resolution correspond to the resolution of the input image when the input image is of low quality, and allocating the second processor (240) resources to a second upscaling model in which the input resolution corresponds to the resolution of the input image and the output resolution corresponds to the target resolution when the input image is of high quality.

[0320] According to one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing at least one of the operating methods of an image processing device on a computer may be provided.

[0321] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0322] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In the image processing device (100), A memory (230) storing one or more instructions; A first processor (210) that executes one or more instructions stored in the memory (230); a second processor (240); and Includes a video processor (250), The image processing device (100) executes the one or more instructions individually or collectively by the first processor (210). Obtaining an input image and information about the input image, Based on the information of the input image, determine the resource allocation of the second processor (240) for executing at least one neural network among a plurality of neural networks executable by the second processor (240), Controlling the second processor (240) to generate a first image quality processing image by performing first image quality processing on an image input to the second processor (240) through the at least one neural network based on the determined resource allocation amount, Controlling the video processor (250) to generate a second image quality processing image by performing second image quality processing based on hardware on the image input to the video processor (250), An image processing device (100) that generates an output image through at least one of the first image quality processing or the second image quality processing.

2. In paragraph 1, The image processing device (100) executes the one or more instructions individually or collectively by the first processor (210). Based on the first information of the input image, the resource allocation of the second processor (240) for the first neural network that performs image quality processing on the first information among the plurality of neural networks is determined, Based on the second information of the input image, the resource allocation of the second processor (240) for the second neural network that performs image quality processing on the second information among the plurality of neural networks is determined, An image processing device (100), wherein the ratio of the resource allocation for the first neural network and the resource allocation for the second neural network is determined according to a value representing the first information of the input image and a value representing the second information of the input image.

3. In paragraph 2, The first information includes resolution information and the second information includes frame rate information, The image processing device (100) executes the one or more instructions individually or collectively by the first processor (210). When the input image is of low resolution and high frame rate, the resource allocation for the first neural network (610) is determined to be greater than the resource allocation for the second neural network, When the input image is of high resolution and low frame rate, the resource allocation for the first neural network is determined to be smaller than the resource allocation for the second neural network (710), An image processing device (100) that determines, when the input image is of low resolution and low frame rate, the resource allocation for the first neural network (810) to correspond to the resource allocation for the second neural network (820).

4. In any one of paragraphs 1 to 3, The image processing device (100) executes the one or more instructions individually or collectively by the first processor (210). Based on at least one of the resolution of the input image, whether the video processor (250) includes an upscaler (650), and the remaining computational amount of the second processor (240), it is determined whether to allocate resources of the second processor (240) to the upscaling model (610). As the resources of the second processor (240) are allocated to the upscaling model (610), the second processor (240) is controlled to generate an upscaled image corresponding to the first image quality processing image by upscaling the image input to the second processor (240) through the upscaling model (610). An image processing device (100) that controls the video processor (250) to generate an upscaled image corresponding to the second image quality processing image by upscaling an image input to the video processor (250), as the video processor (250) includes an upscaler (650).

5. In any one of paragraphs 1 to 4, The image processing device (100) executes the one or more instructions individually or collectively by the first processor (210). As the above input image corresponds to a low frame rate, the resource allocation of the second processor (240) for the motion estimation model (930) among the plurality of neural networks is determined, Based on the resource allocation of the second processor (240) for the motion estimation model (930), the second processor (240) is controlled to obtain motion vector information of an image input to the second processor (240) through the motion estimation model (930). An image processing device (100) that controls the video processor (250) to generate a high frame rate image corresponding to the second image quality processing image by performing motion compensation processing based on the motion vector information and the input image.

6. In any one of paragraphs 1 to 5, The second processor (240) is configured to perform first image quality processing on the input image through a plurality of calculation units, The above video processor (250) is an image processing device (100) configured to perform second image quality processing on the input image through a single operator for each image processing circuit.

7. In any one of paragraphs 1 to 6, The video processor (250) is an image processing device (100) including at least one of an upscaler, a scatter correction circuit, a color difference correction circuit, a high-quality image quality processing circuit, and a motion compensation circuit.

8. In any one of paragraphs 1 to 7, The image processing device (100) executes the one or more instructions individually or collectively by the first processor (210), thereby executing the one or more instructions. By analyzing the above input image, characteristic information of the input image is obtained, Based on the characteristic information of the input image, the resource allocation amount of the second processor (240) is determined for the at least one neural network, An image processing device (100), wherein the characteristic information of the input image includes at least one of the amount of motion of the input image, quality characteristics of the input image, noise information of the input image, brightness level of the input image, and genre of the input image.

9. In paragraph 8, The image processing device (100) executes the one or more instructions individually or collectively by the first processor (210). Obtain the amount of motion of the above input image, An image processing device (100) that allocates more of the second processor (240) resources to the first neural network as the amount of motion of the input image is small, and allocates more of the second processor (240) resources to the second neural network as the amount of motion of the input image is large.

10. In paragraph 8 or 9, The image processing device (100) executes the one or more instructions individually or collectively by the first processor (210). Obtain the quality characteristics of the above input image, An image processing device (100) that allocates the second processor (240) resources to a first upscaling model in which the input resolution and the output resolution correspond to the resolution of the input image, as the input image is of low quality, and allocates the second processor (240) resources to a second upscaling model in which the input resolution corresponds to the resolution of the input image and the output resolution corresponds to the target resolution, as the input image is of high quality.

11. In any one of paragraphs 1 to 10, The image processing device (100) executes the one or more instructions individually or collectively by the first processor (210). Receive a command to disable the image processing function, An image processing device (100) that allocates the second processor (240) resources to the remaining neural networks excluding one or more neural networks performing the image processing function based on the deactivation command.

12. In any one of paragraphs 1 to 11, The image processing device (100) executes the one or more instructions individually or collectively by the first processor (210). Receives an activation command for low power mode, An image processing device (100) that reduces the resource allocation of the second processor (240) for each of the plurality of neural networks based on the activation command.

13. In the operating method of the image processing device (100), Step (410) of acquiring an input image and information of the input image; A step (420) of determining a resource allocation of the second processor (240) for executing at least one neural network among a plurality of neural networks executable by the second processor (240) based on information of the input image; A step (430) of controlling the second processor (240) to generate a first image quality processing image by performing first image quality processing on an image input to the second processor (240) through the at least one neural network based on the determined resource allocation; A step (440) of controlling the video processor (250) to generate a second image quality processing image by performing a second image quality processing based on hardware on an image input to the video processor (250); and A method comprising a step (450) of generating an output image through at least one of the first image quality processing or the second image quality processing.

14. In paragraph 13, The step of determining the resource allocation of the second processor (240) is: A step of determining a resource allocation amount of the second processor (240) for a first neural network among the plurality of neural networks that performs image quality processing on the first information based on the first information of the input image; and A step of determining a resource allocation of the second processor (240) for the second neural network that performs image quality processing on the second information among the plurality of neural networks based on the second information of the input image, A method wherein the ratio of resource allocation for the first neural network and resource allocation for the second neural network is determined according to a value representing first information of the input image and a value representing second information of the input image.

15. Step of acquiring an input image and information of the input image (410); A step (420) of determining a resource allocation of the second processor (240) for executing at least one neural network among a plurality of neural networks executable by the second processor (240) based on information of the input image; A step (430) of controlling the second processor (240) to generate a first image quality processing image by performing first image quality processing on an image input to the second processor (240) through the at least one neural network based on the determined resource allocation; A step (440) of controlling the video processor (250) to generate a second image quality processing image by performing a second image quality processing based on hardware on an image input to the video processor (250); and A computer-readable recording medium having recorded thereon a program for performing a method on a computer, the method comprising the step (450) of generating an output image through at least one of the first image quality processing or the second image quality processing.

Images