Image display device

Abstract

The present disclosure relates to an image display device. The image display device according to an embodiment of the present disclosure comprises: a display; and a signal processing device for signal-processing an input image and outputting an output image, wherein the signal processing device performs a first super-resolution process and a second super-resolution process in parallel on the basis of the input image, and outputs, as the output image, an image synthesized on the basis of result data of the first super-resolution process and result data of the second super-resolution process. Accordingly, it is possible to efficiently perform a plurality of super-resolution processes.

Description

Video display device

[0001] The present disclosure relates to an image display device, and more particularly, to an image display device capable of efficiently performing multiple super resolution processing.

[0002] A video display device is a device that displays images.

[0003] The video display device can display images stored internally or images received from outside.

[0004] Meanwhile, when a low-resolution image is input to a video display device, it is desirable to perform super-resolution processing to display a high-resolution image.

[0005] Prior art document 1, Korean Patent Publication No. 10-2019-0131205, relates to a super resolution network processing method and system, which discloses upscaling an input image based on convolutional neural network (CNN) processing and restoring the upscaled image to output a high-resolution image (HR).

[0006] However, prior art document 1 has a disadvantage in that learning or tuning is limited because the networks in the subsequent stages are dependent on the result data of the previous stage due to the serial structure during super-resolution processing.

[0007] Prior art document 2, Korean Patent Registration No. 10-2467092, relates to a method and system for processing super-resolution images that are robust to coding noise using multiple neural networks, and discloses performing super-resolution image processing that is robust to coding noise using multiple neural networks capable of generating super-resolution images according to the level of coding noise generated in the process of compressing a low-resolution image.

[0008] However, prior literature 2 has a disadvantage in that it utilizes multiple neural networks only to reduce image coding noise.

[0009] The problem of the present disclosure is to provide an image display device capable of efficiently performing multiple super-resolution processing.

[0010] Another problem of the present disclosure is to provide an image display device capable of efficiently improving the sharpness of edges and textures of an image.

[0011] An image display device according to one embodiment of the present disclosure for solving the above problem includes a display and a signal processing device that processes an input image and outputs an output image, wherein the signal processing device performs first super-resolution processing and second super-resolution processing in parallel based on the input image, and outputs a synthesized image based on result data of the first super-resolution processing and result data of the second super-resolution processing as an output image.

[0012] Meanwhile, the signal processing device can increase the sharpness or resolution of edges and textures of objects in an input image based on parallel-based first super-resolution processing and second super-resolution processing.

[0013] Meanwhile, the signal processing device can perform learning based on a first learning database during the first super resolution processing, and can perform learning based on a second learning database different from the first learning database during the second super resolution processing.

[0014] Meanwhile, the signal processing device includes a neural network processor for learning processing, and the neural network processor can perform learning based on a first learning database during first super-resolution processing, and can perform learning based on a second learning database different from the first learning database during second super-resolution processing.

[0015] Meanwhile, the signal processing device can perform learning based on a first resolution image and a second resolution image during the first super resolution processing, and can perform learning based on a third resolution image having a lower resolution than the first resolution image and a pre-processed image during the second super resolution processing.

[0016] Meanwhile, the signal processing device can extract a first feature map based on result data of the first super-resolution processing, extract a second feature map based on result data of the second super-resolution processing, calculate a synthesis ratio of the result data of the first super-resolution processing and the result data of the second super-resolution processing based on the first feature map and the second feature map, and output a synthesized image as an output image based on the calculated synthesis ratio.

[0017] Meanwhile, the signal processing device can vary the synthesis ratio on a pixel-by-pixel basis.

[0018] Meanwhile, the signal processing device can vary the synthesis ratio based on the type or resolution of the content.

[0019] Meanwhile, the signal processing device may include a first super-resolution processing unit that performs first super-resolution processing based on an input image, and a second super-resolution processing unit that performs second super-resolution processing in parallel with the first super-resolution processing unit based on the input image.

[0020] Meanwhile, the signal processing device may further include a first feature map extraction unit that extracts a first feature map based on result data of the first super resolution processing, a second feature map extraction unit that extracts a second feature map based on result data of the second super resolution processing, a feature map analysis unit that calculates a synthesis ratio of the result data of the first super resolution processing and the result data of the second super resolution processing based on the first feature map and the second feature map, and an image synthesis unit that outputs a synthesized image as an output image based on the synthesis ratio calculated by the feature map analysis unit.

[0021] Meanwhile, the signal processing device can perform first super-resolution processing and second super-resolution processing in parallel based on an input image according to a super-resolution mode, and output a synthesized image based on result data of the first super-resolution processing and result data of the second super-resolution processing as an output image.

[0022] Meanwhile, the signal processing device, when set to the sports mode, can perform first super-resolution processing and second super-resolution processing in parallel based on the input image, and output a synthesized image based on the result data of the first super-resolution processing and the result data of the second super-resolution processing as an output image.

[0023] Meanwhile, the signal processing device, when the input image is a video, can perform first super-resolution processing and second super-resolution processing in parallel based on the input video, and output a video synthesized based on the result data of the first super-resolution processing and the result data of the second super-resolution processing as an output video.

[0024] Meanwhile, the signal processing device can further perform a third super-resolution process in parallel based on the input image, and output a synthesized image based on the result data of the first super-resolution process, the result data of the second super-resolution process, and the result data of the first super-resolution process as an output image.

[0025] Meanwhile, the signal processing device can increase the sharpness or resolution of edges and textures of objects in the input image based on the first super-resolution processing and the second super-resolution processing based on the parallel basis, and can increase the sharpness or resolution of flat areas in objects in the input image based on the third super-resolution processing based on the parallel basis.

[0026] Meanwhile, the signal processing device can extract a first feature map based on result data of the first super-resolution processing, extract a second feature map based on result data of the second super-resolution processing, extract a third feature map based on result data of the third super-resolution processing, calculate a synthesis ratio of the result data of the first super-resolution processing, the result data of the second super-resolution processing, and the result data of the third super-resolution processing based on the first feature map, the second feature map, and the third feature map, and output a synthesized image as an output image based on the calculated synthesis ratio.

[0027] Meanwhile, the signal processing device can vary the synthesis ratio on a pixel-by-pixel basis.

[0028] Meanwhile, the signal processing device may include a first super-resolution processing unit that performs first super-resolution processing based on an input image, a second super-resolution processing unit that performs second super-resolution processing in parallel with the first super-resolution processing unit based on the input image, and a third super-resolution processing unit that performs third super-resolution processing in parallel with the first super-resolution processing unit and the first super-resolution processing unit based on the input image.

[0029] Meanwhile, the signal processing device may further include a first feature map extraction unit that extracts a first feature map based on result data of the first super resolution processing, a second feature map extraction unit that extracts a second feature map based on result data of the second super resolution processing, a third feature map extraction unit that extracts a third feature map based on result data of the third super resolution processing, a feature map analysis unit that calculates a synthesis ratio of the result data of the first super resolution processing, the result data of the second super resolution processing, and the result data of the third super resolution processing based on the first feature map, the second feature map, and the third feature map, and an image synthesis unit that outputs a synthesized image as an output image based on the synthesis ratio calculated by the feature map analysis unit.

[0030] According to another embodiment of the present disclosure, an image display device includes a display and a signal processing device that processes an input image and outputs an output image, wherein the signal processing device performs a first super-resolution process, a second super-resolution process, and a third super-resolution process in parallel based on the input image, and outputs a synthesized image based on result data of the first super-resolution process, result data of the second super-resolution process, and result data of the third super-resolution process as an output image.

[0031] An image display device according to one embodiment of the present disclosure includes a display and a signal processing device that performs signal processing on an input image and outputs an output image, wherein the signal processing device performs first super-resolution processing and second super-resolution processing in parallel based on the input image, and outputs a synthesized image based on result data of the first super-resolution processing and result data of the second super-resolution processing as an output image. Accordingly, a plurality of super-resolution processing operations can be efficiently performed. Furthermore, the sharpness of edges and textures of an image can be efficiently improved.

[0032] Meanwhile, the signal processing device can increase the sharpness or resolution of edges and textures of objects within an input image based on parallel-based first and second super-resolution processing. Accordingly, the sharpness or resolution of edges and textures within an image can be efficiently improved.

[0033] Meanwhile, the signal processing device can perform learning based on a first learning database during the first super-resolution processing, and perform learning based on a second learning database different from the first learning database during the second super-resolution processing. Accordingly, it is possible to efficiently perform multiple super-resolution processing operations while efficiently improving the sharpness of edges and textures in the image.

[0034] Meanwhile, the signal processing device includes a neural network processor for learning processing, wherein the neural network processor can perform learning based on a first learning database during the first super-resolution processing, and can perform learning based on a second learning database different from the first learning database during the second super-resolution processing. Accordingly, while efficiently performing multiple super-resolution processing operations, the sharpness of edges and textures of an image can be efficiently improved.

[0035] Meanwhile, the signal processing device can perform learning based on the first resolution image and the second resolution image during the first super-resolution processing, and can perform learning based on the third resolution image and the previously processed image, which has a lower resolution than the first resolution image, based on the second super-resolution processing. Accordingly, it is possible to efficiently perform multiple super-resolution processing operations while efficiently improving the sharpness of the edges and textures of the image.

[0036] Meanwhile, the signal processing device can extract a first feature map based on result data of the first super-resolution processing, extract a second feature map based on result data of the second super-resolution processing, calculate a synthesis ratio of the result data of the first super-resolution processing and the result data of the second super-resolution processing based on the first feature map and the second feature map, and output a synthesized image as an output image based on the calculated synthesis ratio. Accordingly, it is possible to efficiently perform a plurality of super-resolution processes while efficiently improving the sharpness of edges and textures of the image.

[0037] Meanwhile, the signal processing device can vary the synthesis ratio on a pixel-by-pixel basis. This allows for efficient performance of multiple super-resolution processes, effectively improving the clarity of image edges and textures.

[0038] Meanwhile, the signal processing device can vary the synthesis ratio based on the content type or resolution. This allows for efficient performance of multiple super-resolution processes, while effectively improving the clarity of image edges and textures.

[0039] Meanwhile, the signal processing device may include a first super-resolution processing unit that performs first super-resolution processing based on an input image, and a second super-resolution processing unit that performs second super-resolution processing in parallel with the first super-resolution processing unit based on the input image. Accordingly, it is possible to efficiently perform multiple super-resolution processing operations while efficiently improving the sharpness of edges and textures of the image.

[0040] Meanwhile, the signal processing device may further include a first feature map extraction unit that extracts a first feature map based on result data of the first super-resolution processing, a second feature map extraction unit that extracts a second feature map based on result data of the second super-resolution processing, a feature map analysis unit that calculates a synthesis ratio of the result data of the first super-resolution processing and the result data of the second super-resolution processing based on the first feature map and the second feature map, and an image synthesis unit that outputs a synthesized image as an output image based on the synthesis ratio calculated by the feature map analysis unit. Accordingly, it is possible to efficiently perform a plurality of super-resolution processes while efficiently improving the sharpness of edges and textures of the image.

[0041] Meanwhile, the signal processing device can perform first super-resolution processing and second super-resolution processing in parallel based on the input image according to the super-resolution mode, and output a synthesized image based on the result data of the first super-resolution processing and the result data of the second super-resolution processing as an output image. Accordingly, it is possible to efficiently perform multiple super-resolution processing operations while efficiently improving the sharpness of edges and textures of the image.

[0042] Meanwhile, the signal processing device, when set to sports mode, can perform first super-resolution processing and second super-resolution processing in parallel based on an input image, and output a synthesized image based on the result data of the first super-resolution processing and the result data of the second super-resolution processing as an output image. Accordingly, it is possible to efficiently perform multiple super-resolution processing operations while efficiently improving the sharpness of edges and textures of the image.

[0043] Meanwhile, if the input image is a video, the signal processing device can perform first super-resolution processing and second super-resolution processing in parallel based on the input video, and output a video synthesized based on the result data of the first super-resolution processing and the result data of the second super-resolution processing as an output video. Accordingly, it is possible to efficiently perform multiple super-resolution processing operations while efficiently improving the sharpness of edges and textures of the image.

[0044] Meanwhile, the signal processing device can further perform a third super-resolution process in parallel based on the input image, and output an image synthesized based on the result data of the first super-resolution process, the result data of the second super-resolution process, and the result data of the first super-resolution process as an output image. Accordingly, while efficiently performing multiple super-resolution processes, it is possible to efficiently improve the sharpness of the edges and textures of the image.

[0045] Meanwhile, the signal processing device can increase the sharpness or resolution of edges and textures of objects within an input image based on the first and second super-resolution processing in parallel, and can increase the sharpness or resolution of flat areas within objects within the input image based on the third super-resolution processing in parallel. Accordingly, it is possible to efficiently perform multiple super-resolution processing operations while efficiently improving the sharpness of edges and textures of an image.

[0046] Meanwhile, the signal processing device can extract a first feature map based on result data of the first super-resolution processing, extract a second feature map based on result data of the second super-resolution processing, extract a third feature map based on result data of the third super-resolution processing, calculate a synthesis ratio of the result data of the first super-resolution processing, the result data of the second super-resolution processing, and the result data of the third super-resolution processing based on the first feature map, the second feature map, and the third feature map, and output a synthesized image as an output image based on the calculated synthesis ratio. Accordingly, it is possible to efficiently perform a plurality of super-resolution processes while efficiently improving the sharpness of edges and textures of the image.

[0047] Meanwhile, the signal processing device can vary the synthesis ratio on a pixel-by-pixel basis. This allows for efficient performance of multiple super-resolution processes, effectively improving the clarity of image edges and textures.

[0048] Meanwhile, the signal processing device may include a first super-resolution processing unit that performs first super-resolution processing based on an input image, a second super-resolution processing unit that performs second super-resolution processing in parallel with the first super-resolution processing unit based on the input image, and a third super-resolution processing unit that performs third super-resolution processing in parallel with the first super-resolution processing unit and the first super-resolution processing unit based on the input image. Accordingly, it is possible to efficiently perform a plurality of super-resolution processing operations while efficiently improving the sharpness of edges and textures of the image.

[0049] Meanwhile, the signal processing device may further include a first feature map extraction unit that extracts a first feature map based on result data of the first super-resolution processing, a second feature map extraction unit that extracts a second feature map based on result data of the second super-resolution processing, a third feature map extraction unit that extracts a third feature map based on result data of the third super-resolution processing, a feature map analysis unit that calculates a synthesis ratio of the result data of the first super-resolution processing, the result data of the second super-resolution processing, and the result data of the third super-resolution processing based on the first feature map, the second feature map, and the third feature map, and an image synthesis unit that outputs a synthesized image as an output image based on the synthesis ratio calculated by the feature map analysis unit. Accordingly, it is possible to efficiently perform a plurality of super-resolution processes while efficiently improving the sharpness of edges and textures of the image.

[0050] According to another embodiment of the present disclosure, an image display device includes a display and a signal processing device that processes an input image and outputs an output image, wherein the signal processing device performs a first super-resolution process, a second super-resolution process, and a third super-resolution process in parallel based on the input image, and outputs a synthesized image based on result data of the first super-resolution process, result data of the second super-resolution process, and result data of the third super-resolution process as an output image. Accordingly, a plurality of super-resolution processes can be efficiently performed. Furthermore, the sharpness of edges and textures of the image can be efficiently improved.

[0051] FIG. 1 is a drawing illustrating an image display device according to one embodiment of the present disclosure.

[0052] Figure 2 is an example of an internal block diagram of the video display device of Figure 1.

[0053] Figure 3 is an example of an internal block diagram of the signal processing device of Figure 2.

[0054] Figure 4a is a drawing illustrating a control method of the remote control device of Figure 2.

[0055] Figure 4b is an internal block diagram of the remote control device of Figure 2.

[0056] FIG. 5 is a flowchart showing the operation of a video display device according to one embodiment of the present disclosure.

[0057] FIG. 6 is an example of an internal block diagram of a signal processing device according to one embodiment of the present disclosure.

[0058] Figures 7 to 10c are drawings referenced in the description of Figure 5 or Figure 6.

[0059] FIG. 11 is a flowchart showing the operation of a video display device according to another embodiment of the present disclosure.

[0060] FIG. 12 is an example of an internal block diagram of a signal processing device according to another embodiment of the present disclosure.

[0061] Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

[0062] The suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification and do not impart any particularly significant meaning or role to the components themselves. Therefore, the terms "module" and "part" may be used interchangeably.

[0063] FIG. 1 is a drawing illustrating an image display device according to one embodiment of the present disclosure.

[0064] Referring to the drawing, the image display device (100) may include a display (180).

[0065] Meanwhile, the display (180) may be implemented as any one of various panels. For example, the display (180) may be any one of a liquid crystal display panel (LCD panel), an organic light-emitting panel (OLED panel), an inorganic light-emitting panel (LED panel), etc.

[0066] Liquid crystal display panels may require a separate backlight in addition to the panel for displaying images.

[0067] Meanwhile, organic light-emitting panels or inorganic light-emitting panels do not require a separate backlight for image display.

[0068] The video display device (100) can receive broadcast signals through an internal tuner unit (105 in FIG. 2).

[0069] Alternatively, the video display device (100) can be connected to a set-top box (STB) and can receive a broadcast signal or a video signal through the set-top box (STB).

[0070] Meanwhile, the video display device (100) can receive a video signal from outside via a wired or wireless network.

[0071] Meanwhile, the video display device (100) can receive a video signal from outside via a wired or wireless network.

[0072] Meanwhile, the video display device (100) can receive a video signal from a connected external device.

[0073] The external device at this time may be a set-top box (STB), a mobile terminal, a USB type storage device or tablet, or a set-top box or laptop connected via HDMI.

[0074] Meanwhile, the video display device (100) of Fig. 1 can be a TV, monitor, tablet PC, mobile terminal, etc.

[0075] Figure 2 is an example of an internal block diagram of the video display device of Figure 1.

[0076] Referring to FIG. 2, an image display device (100) according to an embodiment of the present disclosure may include an image receiving unit (105), an external device interface unit (130), a storage unit (140), a user input interface unit (150), a sensor unit (not shown), a signal processing unit (170), a display (180), and an audio output unit (185).

[0077] An image display device (100) according to one embodiment of the present disclosure may further include a power supply unit (190) and a microcomputer (173).

[0078] The video receiving unit (105) may include a tuner unit (110), a demodulation unit (120), a network interface unit (130), and an external device interface unit (130).

[0079] Meanwhile, unlike the drawing, the video receiving unit (105) may include only a tuner unit (110), a demodulator unit (120), and an external device interface unit (130). That is, it may not include a network interface unit (130).

[0080] The tuner unit (110) selects an RF broadcast signal corresponding to a channel selected by the user or all pre-stored channels among RF (Radio Frequency) broadcast signals received through an antenna (not shown). In addition, it converts the selected RF broadcast signal into an intermediate frequency signal or a baseband video or audio signal.

[0081] For example, if the selected RF broadcast signal is a digital broadcast signal, it is converted into a digital IF signal (DIF), and if it is an analog broadcast signal, it is converted into an analog baseband video or audio signal (CVBS / SIF). That is, the tuner unit (110) can process a digital broadcast signal or an analog broadcast signal. The analog baseband video or audio signal (CVBS / SIF) output from the tuner unit (110) can be directly input to the signal processing device (170).

[0082] Meanwhile, the tuner unit (110) may be equipped with multiple tuners to receive broadcast signals of multiple channels. Alternatively, a single tuner that simultaneously receives broadcast signals of multiple channels is also possible.

[0083] The demodulation unit (120) receives the digital IF signal (DIF) converted from the tuner unit (110) and performs a demodulation operation.

[0084] The demodulator (120) can output a stream signal (TS) after performing demodulation and channel decoding. At this time, the stream signal may be a signal in which a video signal, an audio signal, or a data signal is multiplexed.

[0085] The stream signal output from the demodulator (120) can be input to the signal processing device (170). The signal processing device (170) performs demultiplexing, image / audio signal processing, etc., and then outputs an image to the display (180) and outputs an audio to the audio output device (185).

[0086] The external device interface unit (130) can transmit or receive data to or from a connected external device (not shown), for example, a set-top box (50). To this end, the external device interface unit (130) may include an A / V input / output unit (not shown).

[0087] The external device interface unit (130) can be connected to external devices such as a DVD (Digital Versatile Disk), Blu-ray, game device, camera, camcorder, computer (laptop), set-top box, etc., via wired / wireless connection, and can also perform input / output operations with the external devices.

[0088] The A / V input / output unit can receive video and audio signals from an external device. Meanwhile, the wireless communication unit (not shown) can perform short-range wireless communication with other electronic devices.

[0089] Through this wireless communication unit (not shown), the external device interface unit (130) can exchange data with an adjacent mobile terminal (600). In particular, the external device interface unit (130) can receive device information, running application information, application images, etc. from the mobile terminal (600) in mirroring mode.

[0090] The network interface unit (135) provides an interface for connecting the video display device (100) to a wired / wireless network, including the Internet. For example, the network interface unit (135) can receive content or data provided by the Internet, a content provider, or a network operator via a network.

[0091] Meanwhile, the network interface unit (135) may include a wireless communication unit (not shown).

[0092] The storage unit (140) may store programs for each signal processing and control within the signal processing device (170), and may also store signal-processed image, voice, or data signals.

[0093] In addition, the storage unit (140) may also perform a function for temporary storage of video, audio, or data signals input to the external device interface unit (130). In addition, the storage unit (140) may store information regarding a specific broadcast channel through a channel memory function such as a channel map.

[0094] Although the storage unit (140) of FIG. 2 illustrates an embodiment in which the storage unit (140) is provided separately from the signal processing device (170), the scope of the present disclosure is not limited thereto. The storage unit (140) may be included within the signal processing device (170).

[0095] The user input interface unit (150) transmits a signal input by the user to the signal processing device (170) or transmits a signal from the signal processing device (170) to the user.

[0096] For example, a user input signal such as power on / off, channel selection, screen setting, etc. may be transmitted / received from a remote control device (200), a user input signal input from a local key (not shown) such as a power key, a channel key, a volume key, a setting value, etc. may be transmitted to a signal processing device (170), a user input signal input from a sensor unit (not shown) that senses a user's gesture may be transmitted to the signal processing device (170), or a signal from the signal processing device (170) may be transmitted to a sensor unit (not shown).

[0097] The signal processing device (170) can demultiplex an input stream or process demultiplexed signals through a tuner unit (110), a demodulator unit (120), a network interface unit (135), or an external device interface unit (130) to generate and output a signal for video or audio output.

[0098] For example, the signal processing device (170) can receive a broadcast signal or an HDMI signal received from the image receiving unit (105), perform signal processing based on the received broadcast signal or HDMI signal, and output a signal-processed image signal.

[0099] An image signal processed by a signal processing device (170) may be input to a display (180) and displayed as an image corresponding to the image signal. In addition, an image signal processed by a signal processing device (170) may be input to an external output device through an external device interface unit (130).

[0100] The voice signal processed in the signal processing device (170) can be output as sound to the audio output unit (185). In addition, the voice signal processed in the signal processing device (170) can be input to an external output device through the external device interface unit (130).

[0101] Although not illustrated in FIG. 2, the signal processing device (170) may include a demultiplexing unit, an image processing unit, etc. That is, the signal processing device (170) may perform various signal processing operations and, accordingly, may be implemented in the form of a system on chip (SOC). This will be described later with reference to FIG. 3.

[0102] In addition, the signal processing device (170) can control the overall operation within the video display device (100). For example, the signal processing device (170) can control the tuner unit (110) to select (tune) an RF broadcast corresponding to a channel selected by a user or a pre-stored channel.

[0103] In addition, the signal processing device (170) can control the image display device (100) by a user command or internal program input through the user input interface unit (150).

[0104] Meanwhile, the signal processing device (170) can control the display (180) to display an image. At this time, the image displayed on the display (180) may be a still image or a moving image, and may be a 2D image or a 3D image.

[0105] Meanwhile, the signal processing device (170) can cause a predetermined object to be displayed within an image displayed on the display (180). For example, the object can be at least one of a connected web screen (newspaper, magazine, etc.), an EPG (Electronic Program Guide), various menus, widgets, icons, still images, videos, and text.

[0106] Meanwhile, the signal processing device (170) can recognize the user's location based on an image captured from a camera (not shown). For example, the distance (z-axis coordinate) between the user and the image display device (100) can be determined. In addition, the x-axis coordinate and y-axis coordinate within the display (180) corresponding to the user's location can be determined.

[0107] The display (180) generates a driving signal by converting a video signal, data signal, OSD signal, control signal, etc. processed by the signal processing device (170) or a video signal, data signal, control signal, etc. received from the external device interface unit (130).

[0108] Meanwhile, the display (180) is configured as a touch screen and can be used as an input device in addition to an output device.

[0109] The audio output unit (185) receives a signal processed by the signal processing device (170) and outputs it as voice.

[0110] A camera unit (not shown) photographs a user. The camera unit (not shown) may be implemented with a single camera, but is not limited thereto, and may also be implemented with multiple cameras. Image information captured by the camera unit (not shown) may be input to a signal processing device (170).

[0111] The signal processing device (170) can detect the user's gesture based on an image captured from a shooting unit (not shown) or a signal detected from a sensor unit (not shown), or a combination thereof.

[0112] The power supply unit (190) supplies power to the entire image display device (100). In particular, the power supply unit (190) can supply power to a signal processing device (170) that can be implemented in the form of a system on chip (SOC), a display (180) for image display, and an audio output unit (185) for audio output.

[0113] Specifically, the power supply unit (190) may be equipped with a DC / DC converter that converts AC voltage into DC voltage and a DC / DC converter that converts the level of the DC voltage.

[0114] The remote control device (200) transmits user input to the user input interface unit (150). To this end, the remote control device (200) may use Bluetooth, RF (Radio Frequency) communication, IR (Infrared) communication, UWB (Ultra Wideband), ZigBee, etc. In addition, the remote control device (200) may receive video, audio, or data signals output from the user input interface unit (150) and display or output the same as audio on the remote control device (200).

[0115] Meanwhile, the above-described video display device (100) may be a digital broadcast receiver capable of receiving fixed or mobile digital broadcasts.

[0116] Meanwhile, the block diagram of the image display device (100) illustrated in FIG. 2 is a block diagram for one embodiment of the present disclosure. Each component of the block diagram may be integrated, added, or omitted depending on the specifications of the image display device (100) actually implemented. That is, two or more components may be combined into one component, or one component may be subdivided into two or more components, as needed. In addition, the functions performed by each block are intended to explain the embodiment of the present disclosure, and the specific operations or devices thereof do not limit the scope of the present disclosure.

[0117] Figure 3 is an example of an internal block diagram of the signal processing device of Figure 2.

[0118] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure may include a demultiplexing unit (310), an image processing unit (320), a processor (330), and an audio processing unit (370). In addition, a data processing unit (not shown) may be further included.

[0119] The demultiplexer (310) demultiplexes the input stream. For example, when MPEG-2 TS is input, it can be demultiplexed to separate it into video, audio, and data signals, respectively. Here, the stream signal input to the demultiplexer (310) may be a stream signal output from the tuner (110), the demodulator (120), or the external device interface (130).

[0120] The image processing unit (320) can perform signal processing on an input image. For example, the image processing unit (320) can perform image processing on an image signal demultiplexed from the demultiplexing unit (310).

[0121] To this end, the image processing unit (320) may include an image decoder (325), a scaler (335), an image quality processing unit (635), an image encoder (not shown), a graphics processing unit (340), a frame rate conversion unit (350), and a formatter (360).

[0122] The video decoder (325) decodes the demultiplexed video signal, and the scaler (335) scales the resolution of the decoded video signal so that it can be output on the display (180).

[0123] The video decoder (325) can be equipped with decoders of various standards. For example, it can be equipped with an MPEG-2, H.264 decoder, a 3D video decoder for color images and depth images, a decoder for multi-view images, etc.

[0124] The scaler (335) can scale an input video signal that has been decoded by a video decoder (325), etc.

[0125] For example, the scaler (335) can upscale when the size or resolution of the input image signal is small, and downscale when the size or resolution of the input image signal is large.

[0126] The image quality processing unit (635) can perform image quality processing on an input image signal for which image decoding has been completed in the image decoder (325), etc.

[0127] For example, the image quality processing unit (635) may perform noise removal processing of an input image signal, expand the resolution of the gradation of an input image signal, perform image resolution enhancement, perform signal processing based on high dynamic range (HDR), vary the frame rate, or perform image quality processing corresponding to panel characteristics, particularly the panel.

[0128] The graphic processing unit (340) generates an OSD signal based on user input or on its own. For example, based on a user input signal, a signal for displaying various information in the form of graphics or text on the screen of the display (180) may be generated. The generated OSD signal may include various data such as the user interface screen of the image display device (100), various menu screens, widgets, and icons. In addition, the generated OSD signal may include a 2D object or a 3D object.

[0129] In addition, the graphic processing unit (340) can generate a pointer that can be displayed on the display based on a pointing signal input from the remote control device (200). In particular, such a pointer can be generated by the pointing signal processing unit, and the graphic processing unit (240) can include such a pointing signal processing unit (not shown). Of course, the pointing signal processing unit (not shown) can also be provided separately rather than being included within the graphic processing unit (240).

[0130] The frame rate converter (FRC) (350) can convert the frame rate of an input video. Meanwhile, the frame rate converter (350) can also output the video as is without a separate frame rate conversion.

[0131] Meanwhile, the formatter (360) can change the format of an input video signal into a video signal for display on a display and output it.

[0132] In particular, the formatter (360) can change the format of the video signal to correspond to the display panel.

[0133] Meanwhile, the formatter (360) can also change the format of the video signal.

[0134] The processor (330) can control the overall operation within the image display device (100) or the signal processing device (170).

[0135] For example, the processor (330) can control the tuner (110) to select (tuning) an RF broadcast corresponding to a channel selected by the user or a pre-stored channel.

[0136] In addition, the processor (330) can control the image display device (100) by a user command or internal program input through the user input interface unit (150).

[0137] Additionally, the processor (330) can perform data transmission control with the network interface unit (135) or the external device interface unit (130).

[0138] Additionally, the processor (330) can control the operation of the demultiplexing unit (310), the image processing unit (320), etc., within the signal processing device (170).

[0139] Meanwhile, the audio processing unit (370) within the signal processing device (170) can perform audio processing of the demultiplexed audio signal. To this end, the audio processing unit (370) can be equipped with various decoders.

[0140] Additionally, the audio processing unit (370) within the signal processing device (170) can process bass, treble, volume control, etc.

[0141] A data processing unit (not shown) within a signal processing device (170) can perform data processing on a demultiplexed data signal. For example, if the demultiplexed data signal is an encoded data signal, it can be decoded. The encoded data signal may be electronic program guide information (EPG) information that includes broadcast information such as the start time and end time of a broadcast program broadcast on each channel.

[0142] Meanwhile, the block diagram of the signal processing device (170) illustrated in FIG. 3 is a block diagram for one embodiment of the present disclosure. Each component of the block diagram may be integrated, added, or omitted depending on the specifications of the signal processing device (170) actually implemented.

[0143] In particular, the frame rate conversion unit (350) and formatter (360) may be provided separately from the image processing unit (320).

[0144] Meanwhile, a signal processing device (170) according to an embodiment of the present disclosure may further include a neural network processor (333) for learning processing, etc.

[0145] Figure 4a is a drawing illustrating a control method of the remote control device of Figure 2.

[0146] As shown in (a) of FIG. 4a, a pointer (205) corresponding to a remote control device (200) is displayed on the display (180).

[0147] The user can move or rotate the remote control device (200) up and down, left and right ((b) of FIG. 4a), and forward and backward ((c) of FIG. 4a). The pointer (205) displayed on the display (180) of the video display device corresponds to the movement of the remote control device (200). As shown in the drawing, the pointer (205) moves and is displayed according to the movement in 3D space, so the remote control device (200) can be called a space remote control or a 3D pointing device.

[0148] Figure 4a (b) illustrates that when a user moves the remote control device (200) to the left, the pointer (205) displayed on the display (180) of the video display device also moves to the left in response.

[0149] Information about the movement of the remote control device (200) detected by the sensor of the remote control device (200) is transmitted to the image display device. The image display device can calculate the coordinates of the pointer (205) from the information about the movement of the remote control device (200). The image display device can display the pointer (205) to correspond to the calculated coordinates.

[0150] FIG. 4A (c) illustrates a case where, while pressing a specific button within the remote control device (200), the user moves the remote control device (200) away from the display (180). As a result, the selection area within the display (180) corresponding to the pointer (205) may be zoomed in and displayed in an enlarged manner. Conversely, when the user moves the remote control device (200) closer to the display (180), the selection area within the display (180) corresponding to the pointer (205) may be zoomed out and displayed in a reduced manner. Meanwhile, when the remote control device (200) moves away from the display (180), the selection area may be zoomed out, and when the remote control device (200) moves closer to the display (180), the selection area may be zoomed in.

[0151] Meanwhile, when a specific button within the remote control device (200) is pressed, recognition of up, down, left, and right movements may be excluded. That is, when the remote control device (200) moves away from or toward the display (180), up, down, left, and right movements may not be recognized, and only forward and backward movements may be recognized. When a specific button within the remote control device (200) is not pressed, only the pointer (205) moves in accordance with the up, down, left, and right movements of the remote control device (200).

[0152] Meanwhile, the movement speed or movement direction of the pointer (205) can correspond to the movement speed or movement direction of the remote control device (200).

[0153] Figure 4b is an internal block diagram of the remote control device of Figure 2.

[0154] Referring to the drawing, the remote control device (200) may include a wireless communication unit (425), a user input unit (435), a sensor unit (440), an output unit (450), a power supply unit (460), a storage unit (470), and a control unit (480).

[0155] The wireless communication unit (425) transmits and receives signals with any one of the image display devices according to the embodiments of the present disclosure described above. Among the image display devices according to the embodiments of the present disclosure, one image display device (100) will be described as an example.

[0156] In this embodiment, the remote control device (200) may be equipped with an RF module (421) capable of transmitting and receiving signals with the image display device (100) in accordance with RF communication standards. In addition, the remote control device (200) may be equipped with an IR module (423) capable of transmitting and receiving signals with the image display device (100) in accordance with IR communication standards.

[0157] In this embodiment, the remote control device (200) transmits a signal containing information about the movement of the remote control device (200) to the image display device (100) through the RF module (421).

[0158] In addition, the remote control device (200) can receive a signal transmitted by the image display device (100) through the RF module (421). In addition, the remote control device (200) can transmit commands for power on / off, channel change, volume change, etc. to the image display device (100) through the IR module (423) as needed.

[0159] The user input unit (435) may be configured as a keypad, a button, a touch pad, or a touch screen. The user can input a command related to the image display device (100) to the remote control device (200) by operating the user input unit (435). If the user input unit (435) has a hard key button, the user can input a command related to the image display device (100) to the remote control device (200) by pushing the hard key button. If the user input unit (435) has a touch screen, the user can input a command related to the image display device (100) to the remote control device (200) by touching a soft key of the touch screen. In addition, the user input unit (435) may be equipped with various types of input means that the user can operate, such as a scroll key or a jog key, and the present embodiment does not limit the scope of the present disclosure.

[0160] The sensor unit (440) may be equipped with a gyro sensor (441) or an acceleration sensor (443). The gyro sensor (441) may sense information regarding the movement of the remote control device (200).

[0161] For example, a gyro sensor (441) can sense information about the operation of a remote control device (200) based on the x, y, and z axes. An acceleration sensor (443) can sense information about the movement speed of the remote control device (200). Meanwhile, a distance measuring sensor can be further provided, thereby sensing the distance to the display (180).

[0162] The output unit (450) can output a video or audio signal corresponding to the operation of the user input unit (435) or to a signal transmitted from the video display device (100). Through the output unit (450), the user can recognize whether the user input unit (435) is being operated or whether the video display device (100) is being controlled.

[0163] For example, the output unit (450) may be equipped with an LED module (451) that lights up when the user input unit (435) is operated or a signal is transmitted and received with the image display device (100) through the wireless communication unit (425), a vibration module (453) that generates vibration, an audio output module (455) that outputs audio, or a display module (457) that outputs audio.

[0164] The power supply unit (460) supplies power to the remote control device (200). The power supply unit (460) can reduce power waste by stopping the power supply when the remote control device (200) is not moved for a predetermined period of time. The power supply unit (460) can resume the power supply when a predetermined key provided on the remote control device (200) is operated.

[0165] The storage unit (470) can store various types of programs, application data, etc. required for the control or operation of the remote control device (200). If the remote control device (200) wirelessly transmits and receives signals through the image display device (100) and the RF module (421), the remote control device (200) and the image display device (100) transmit and receive signals through a predetermined frequency band. The control unit (480) of the remote control device (200) can store and refer to information regarding the frequency band through which signals can be wirelessly transmitted and received between the remote control device (200) and the paired image display device (100), etc., in the storage unit (470).

[0166] The control unit (480) controls all matters related to the control of the remote control device (200). The control unit (480) can transmit a signal corresponding to a predetermined key operation of the user input unit (435) or a signal corresponding to the movement of the remote control device (200) sensed by the sensor unit (440) to the image display device (100) via the wireless communication unit (425).

[0167] The user input interface unit (150) of the video display device (100) may be equipped with a wireless communication unit (151) capable of wirelessly transmitting and receiving signals with a remote control device (200), and a coordinate value calculation unit (415) capable of calculating the coordinate value of a pointer corresponding to the operation of the remote control device (200).

[0168] The user input interface unit (150) can wirelessly transmit and receive signals to and from the remote control device (200) via the RF module (412). In addition, the user input interface unit (150) can receive signals transmitted by the remote control device (200) according to the IR communication standard via the IR module (413).

[0169] The coordinate value calculation unit (415) can calculate the coordinate values ​​(x, y) of the pointer (205) to be displayed on the display (170) by correcting hand shake or error from a signal corresponding to the operation of the remote control device (200) received through the wireless communication unit (151).

[0170] A transmission signal of a remote control device (200) input to a video display device (100) through a user input interface unit (150) is transmitted to a signal processing device (170) of the video display device (100). The signal processing device (170) can determine information about the operation and key operation of the remote control device (200) from the signal transmitted from the remote control device (200) and control the video display device (100) in response thereto.

[0171] As another example, the remote control device (200) can calculate pointer coordinate values ​​corresponding to the operation and output them to the user input interface unit (150) of the image display device (100). In this case, the user input interface unit (150) of the image display device (100) can transmit information about the received pointer coordinate values ​​to the signal processing device (170) without a separate hand shake or error correction process.

[0172] In addition, as another example, the coordinate value calculation unit (415) may be provided inside the signal processing device (170) rather than the user input interface unit (150), unlike in the drawing.

[0173] FIG. 5 is a flowchart showing the operation of a video display device according to one embodiment of the present disclosure.

[0174] Referring to the drawing, a signal processing device (170) in an image display device (100) according to one embodiment of the present disclosure determines whether performance of a super resolution mode for a received input image is required (S5310).

[0175] For example, the signal processing device (170) within the image display device (100) may determine that it is necessary to perform super resolution mode for the input image when the resolution of the input image being received is a first resolution and the resolution of the display (180) is a second resolution that is higher than the first resolution.

[0176] As another example, the signal processing device (170) within the image display device (100) may determine that the super resolution mode for the input image needs to be performed regardless of the resolution of the display (180) when the resolution of the input image being received is lower than the reference resolution.

[0177] As another example, the signal processing device (170) within the image display device (100) may determine that it is necessary to perform a super resolution mode for the input image when the first resolution of the input image being received is lower than the reference resolution and the resolution of the display (180) is a second resolution that is higher than the first resolution.

[0178] Next, the signal processing device (170) according to one embodiment of the present disclosure enters the super resolution mode when performance in the super resolution mode is required.

[0179] Next, a signal processing device (170) according to one embodiment of the present disclosure performs first super-resolution processing and second super-resolution processing in parallel based on an input image according to a super-resolution mode (S5320), and outputs a synthesized image based on the result data of the first super-resolution processing and the result data of the second super-resolution processing as an output image (S5330).

[0180] Meanwhile, the signal processing device (170) can perform first super-resolution processing and second super-resolution processing in parallel based on the input image according to the super-resolution mode, and output a synthesized image based on the result data of the first super-resolution processing and the result data of the second super-resolution processing as an output image.

[0181] This allows for efficient multi-superresolution processing. Furthermore, it effectively improves the sharpness of image edges and textures.

[0182] Meanwhile, the signal processing device (170) can increase the sharpness or resolution of the edge and texture of an object in an input image based on the first super resolution processing and the second super resolution processing on a parallel basis.

[0183] Meanwhile, the signal processing device (170) can perform learning based on a first learning database during the first super-resolution processing, and perform learning based on a second learning database different from the first learning database during the second super-resolution processing in parallel. Accordingly, it is possible to efficiently perform multiple super-resolution processing operations while efficiently improving the sharpness of edges and textures of an image.

[0184] For example, the signal processing device (170) can perform first super-resolution processing centered on the edge of an object in an input image, and perform second super-resolution processing centered on the texture in an object in the input image in parallel.

[0185] Accordingly, it is possible to efficiently perform multiple super-resolution processing while efficiently improving the sharpness of edges and textures in the image.

[0186] Meanwhile, the signal processing device (170) includes a neural network processor (333) for learning processing, and the neural network processor (333) can perform learning based on a first learning database during the first super-resolution processing, and can perform learning based on a second learning database different from the first learning database during the second super-resolution processing. Accordingly, while efficiently performing multiple super-resolution processing operations, it is possible to efficiently improve the sharpness of edges and textures of an image.

[0187] FIG. 6 is an example of an internal block diagram of a signal processing device according to one embodiment of the present disclosure.

[0188] Referring to the drawings, a signal processing device (170) according to one embodiment of the present disclosure may include a first super-resolution processing unit (610) that performs first super-resolution processing based on an input image (IMa), and a second super-resolution processing unit (620) that performs second super-resolution processing in parallel with the first super-resolution processing unit (610) based on the input image (IMa).

[0189] For example, in the case of super resolution mode, the signal processing device (170) can control the first super resolution processing unit (610) and the second super resolution processing unit (620) to operate in parallel.

[0190] Meanwhile, the first super resolution processing unit (610) can perform learning based on the first learning database during the first super resolution processing.

[0191] For example, the first super resolution processing unit (610) can perform learning based on the first learning database using the neural network processor (333) during the first super resolution processing.

[0192] Specifically, the first super resolution processing unit (610) within the signal processing device (170) can perform learning based on the first resolution image and the second resolution image during the first super resolution processing.

[0193] Meanwhile, the second super resolution processing unit (620) can perform learning based on the second learning database during the second super resolution processing.

[0194] For example, the second super resolution processing unit (620) can perform learning based on a second learning database that is different from the first learning database by using the neural network processor (333) during the second super resolution processing.

[0195] Meanwhile, the second super resolution processing unit (620) within the signal processing device (170) can perform learning based on a third resolution image having a lower resolution than the first resolution image and a pre-processed image based on the second super resolution processing.

[0196] Meanwhile, the first super resolution processing unit (610) and the second super resolution processing unit (620) can individually perform learning or adjust learning.

[0197] Meanwhile, the first super resolution processing unit (610) mainly processes edges within the input image, and the second super resolution processing unit (620) can process, for example, textures, as an area excluding edges within the input image.

[0198] Accordingly, it is possible to efficiently perform multiple super-resolution processing while efficiently improving the sharpness of edges and textures in the image.

[0199] Meanwhile, the signal processing device (170) can extract a first feature map (MPa) based on the result data (IMb) of the first super resolution processing, and can extract a second feature map (MPb) based on the result data (IMc) of the second super resolution processing.

[0200] At this time, the result data (IMb) of the first super resolution processing may be a result image of the first super resolution processing, and the result data (IMc) of the second super resolution processing may be a result image of the second super resolution processing.

[0201] Meanwhile, the signal processing device (170) can calculate a synthesis ratio (RTa) of the result data (IMb) of the first super-resolution processing and the result data (IMc) of the second super-resolution processing based on the first feature map (MPa) and the second feature map (MPb), and output a synthesized image as an output image (IMm) based on the calculated synthesis ratio (RTa).

[0202] To this end, the signal processing device (170) may further include a first feature map extraction unit (612) that extracts a first feature map (Mpa) based on the result data (IMb) of the first super resolution processing, and a second feature map extraction unit (622) that extracts a second feature map (MPb) based on the result data (IMc) of the second super resolution processing.

[0203] Meanwhile, the first feature map extraction unit (612) and the second feature map extraction unit (622) can extract the degree of improvement in resolution and clarity as feature values ​​or feature maps (MPa, MPb) based on the result data (IMb, IMc) of super resolution processing on a pixel basis.

[0204] For example, the feature values ​​or feature maps (MPa, MPb) output from the first feature map extraction unit (612) and the second feature map extraction unit (622) may include the level or energy of the middle and high frequency components of the spatial frequency of the result data (IMb, IMc) of each super resolution processing.

[0205] Meanwhile, the first feature map extraction unit (612) and the second feature map extraction unit (622) may utilize different feature values ​​or feature maps (MPa, MPb) as feature values ​​or feature maps that represent the degree of improvement in resolution and clarity.

[0206] Meanwhile, the signal processing device (170) may further include a feature map analysis unit (630) that calculates a synthesis ratio (RTa) of the result data (IMb) of the first super resolution processing and the result data (IMc) of the second super resolution processing based on the first feature map (MPa) and the second feature map (MPb), and an image synthesis unit (640) that outputs a synthesized image as an output image (IMm) based on the synthesis ratio (RTa) calculated by the feature map analysis unit (630).

[0207] Meanwhile, the image synthesis unit (640) can determine the synthesis ratio (RTa) of the result data (IMb) of the first super resolution processing and the result data (IMc) of the second super resolution processing based on each feature value or feature map (Mpa, MPb) in pixel units.

[0208] For example, the image synthesis unit (640) can determine a synthesis ratio (RTa) at which resolution and clarity are optimally improved based on each feature value or feature map (Mpa, MPb).

[0209] As another example, the image synthesis unit (640) can vary the synthesis ratio (RTa) for each image component for which resolution and clarity are to be enhanced.

[0210] Meanwhile, the feature map analysis unit (630) or image synthesis unit (640) within the signal processing device (170) can vary the synthesis ratio (RTa) on a pixel-by-pixel basis.

[0211] For example, the feature map analysis unit (630) or the image synthesis unit (640) within the signal processing device (170) can control the ratio of the result data (IMb) of the first super resolution processing to increase near the edge area within the input image (IMa).

[0212] As another example, the feature map analysis unit (630) or the image synthesis unit (640) within the signal processing device (170) can control the ratio of the result data (IMc) of the second super resolution processing to increase near the texture area within the input image (IMa).

[0213] Accordingly, it is possible to efficiently perform multiple super-resolution processing while efficiently improving the sharpness of edges and textures in the image.

[0214] Meanwhile, the feature map analysis unit (630) or image synthesis unit (640) within the signal processing device (170) can vary the synthesis ratio (RTa) based on the type or resolution of the content.

[0215] For example, the feature map analysis unit (630) or the image synthesis unit (640) can control the ratio of the result data (IMc) of the second super resolution processing among the result data (IMb) of the first super resolution processing and the result data (IMc) of the second super resolution processing to increase as the resolution increases.

[0216] Accordingly, the sharpness of the edges and textures of the image can be efficiently improved in response to the resolution of the image.

[0217] As another example, the feature map analysis unit (630) or the image synthesis unit (640) can control the ratio of the result data (IMb) of the first super resolution processing among the result data (IMb) of the first super resolution processing and the result data (IMc) of the second super resolution processing to increase as the motion or motion vector of the content increases.

[0218] Accordingly, the sharpness of the edges and textures of the image can be efficiently improved in response to the movement of the content.

[0219] In this way, according to the first super resolution processing unit (610) and the second super resolution processing unit (620) in the signal processing device (170) of FIG. 6, super resolution processing is implemented in parallel, thereby enabling learning or tuning for each purpose.

[0220] In addition, according to the first super resolution processing unit (610) and the second super resolution processing unit (620) in the signal processing device (170) of FIG. 6, there is no dependency in each super resolution processing, so a relatively simple implementation is possible.

[0221] Meanwhile, the feature map analysis unit (630) in the signal processing device (170) of FIG. 6 can precisely adjust the feature map analysis so that the resolution or clarity of the result data (IMb, IMc) of each super resolution processing does not deteriorate.

[0222] Meanwhile, the signal processing device (170), when the input image is a video, can perform first super-resolution processing and second super-resolution processing in parallel based on the input video, and output a video synthesized based on the result data (IMb) of the first super-resolution processing and the result data (IMc) of the second super-resolution processing as an output video.

[0223] Accordingly, it is possible to efficiently improve the sharpness of edges and textures in videos while efficiently performing multiple super-resolution processing.

[0224] Figures 7 to 10c are drawings referenced in the description of Figure 5 or Figure 6.

[0225] Figure 7 is a diagram illustrating super-resolution processing for a first input image.

[0226] Referring to the drawing, (a) of FIG. 7 illustrates a first input image (710).

[0227] When the first input image (710) includes branches and leaves, some areas (Arx) including the leaves become unclear, like areas (Arxa) in the enlarged image (712).

[0228] Figure 7 (b) illustrates an output image (720) processed based on the operations of the first super resolution processing unit (610) and the second super resolution processing unit (620) of Figure 6 based on the first input image.

[0229] The first super resolution processing unit (610) performs edge region processing, and the second super resolution processing unit (620) performs texture region processing in parallel. Some areas (Ara) including leaves become clear, like the areas (Araa) in the enlarged image (722). In addition, the edge areas of the tree branches also become clear.

[0230] Accordingly, it is possible to efficiently perform multiple super-resolution processing while efficiently improving the sharpness of edges and textures in the image.

[0231] Figure 8 is a diagram illustrating super-resolution processing for a second input image.

[0232] Referring to the drawing, (a) of FIG. 8 illustrates a second input image (810).

[0233] When the second input image (810) includes various fruits, the edge area (RGx) of the fruits is clear, but some areas (OBx) containing the texture of the fruits are not clear, like the area (TXx) in the enlarged image (812).

[0234] Figure 8 (b) illustrates an output image (820) processed based on the operations of the first super resolution processing unit (610) and the second super resolution processing unit (620) of Figure 6 for the second input image.

[0235] The first super resolution processing unit (610) performs edge region processing, and the second super resolution processing unit (620) performs texture region processing in parallel. As a result, some areas (OBa) including the fruit become clear, like the areas (Txa) in the enlarged image (822). In addition, the edge area (EGa) of the fruit also becomes clear.

[0236] Accordingly, it is possible to efficiently perform multiple super-resolution processing while efficiently improving the sharpness of edges and textures in the image.

[0237] Figure 9 is a diagram illustrating super-resolution processing for a third input image.

[0238] Referring to the drawing, (a) of FIG. 9 illustrates a third input image (910).

[0239] When the third input image (910) includes a rock wall and the sky, some areas (OBy) including the rock wall become unclear, like areas (Txy) in the enlarged image (912).

[0240] Figure 9 (b) illustrates an output image (920) processed based on the operations of the first super resolution processing unit (610) and the second super resolution processing unit (620) of Figure 6 for the third input image.

[0241] Meanwhile, the first super resolution processing unit (610) performs edge region processing, and in parallel, the second super resolution processing unit (620) performs texture region processing. Some areas (OBb) including the rock wall become clear, like the area (Txb) in the enlarged image (922). In addition, the edge area of ​​the rock wall also becomes clear.

[0242] Accordingly, it is possible to efficiently perform multiple super-resolution processing while efficiently improving the sharpness of edges and textures in the image.

[0243] Meanwhile, the signal processing device (170), when set to the sports mode or super resolution mode, can perform first super resolution processing and second super resolution processing in parallel based on the input image, and output a synthesized image based on the result data (IMb) of the first super resolution processing and the result data (IMc) of the second super resolution processing as an output image (IMm). This will be described later with reference to FIG. 10a and below.

[0244] Figures 10a to 10c are diagrams illustrating various setting screens for parallel processing of multiple super resolution processes.

[0245] Figure 10a illustrates an example of a settings screen.

[0246] Referring to the drawing, the signal processing device (170) can control the display of a setting screen (1010) based on an input signal.

[0247] The setting screen (1010) at this time may include a sports mode item (1012) for sports mode setting and a normal mode item (1014) for normal mode setting.

[0248] Meanwhile, the signal processing device (170) can control the display of a setting screen (1010b) including an on item (1022) and an off item (1024) of the super resolution mode (1020), as shown in FIG. 10b, when the sports mode item (1012) is selected based on the input signal.

[0249] Meanwhile, the signal processing device (170) can control the display of a setting screen (1010c) including an edge and texture item (1032) and an edge, texture, and flat item (1034), as shown in FIG. 10c, when the on item (1022) of the super resolution mode (1020) is selected based on the input signal.

[0250] For example, when an edge and texture item (1032) is selected, the signal processing device (170) can control two super-resolution processing units to operate in parallel, as shown in FIG. 6, so that the first super-resolution processing unit (610) performs processing on the edge area of ​​the image, and the second super-resolution processing unit (620) performs processing on the texture area of ​​the image.

[0251] Accordingly, it is possible to efficiently perform multiple super-resolution processing while efficiently improving the sharpness of edges and textures in the image.

[0252] As another example, the signal processing device (170) can control three super-resolution processing units to operate in parallel, as shown in FIG. 12, when edge, texture, and flat items (1034) are selected, such that the first super-resolution processing unit (610) performs processing on the edge area of ​​the image, the second super-resolution processing unit (620) performs processing on the texture area of ​​the image, and the third super-resolution processing unit (625) performs processing on the flat area of ​​the image.

[0253] That is, when an edge, texture, and flat item (1034) are selected, the signal processing device (170) can further perform a third super-resolution process in parallel with the first and second super-resolution processes of FIG. 6, and output an image synthesized based on the result data (IMb) of the first super-resolution process, the result data (IMc) of the second super-resolution process, and the result data (IMb) of the first super-resolution process as an output image (IMm). Accordingly, while efficiently performing a plurality of super-resolution processes, it is possible to efficiently improve the sharpness of the edges and textures of the image.

[0254] Accordingly, it is possible to efficiently perform multiple super-resolution processing while efficiently improving the sharpness of edges, textures, and flatness of the image.

[0255] Meanwhile, the signal processing device (170) can increase the sharpness or resolution of the edge and texture of an object in the input image based on the first super-resolution processing and the second super-resolution processing based on the parallel basis when the edge, texture, and flat items (1034) are selected, and can increase the sharpness or resolution of a flat area in an object in the input image based on the third super-resolution processing based on the parallel basis. Accordingly, it is possible to efficiently improve the sharpness of the edge and texture of the image while efficiently performing a plurality of super-resolution processing.

[0256] FIG. 11 is a flowchart showing the operation of a video display device according to another embodiment of the present disclosure.

[0257] Referring to the drawing, a signal processing device (170b) in an image display device (100) according to another embodiment of the present disclosure determines whether performance of a super resolution mode for a received input image is required (S1110).

[0258] For example, a signal processing device (170b) within an image display device (100) may determine that super resolution mode for the input image is required when the resolution of the input image being received is a first resolution and the resolution of the display (180) is a second resolution that is higher than the first resolution.

[0259] As another example, the signal processing device (170b) in the image display device (100) may determine that it is necessary to perform super resolution mode on the input image when the resolution of the input image being received is lower than the reference resolution.

[0260] Next, a signal processing device (170b) according to another embodiment of the present disclosure enters the super resolution mode when performance of the super resolution mode is required.

[0261] Next, a signal processing device (170b) according to another embodiment of the present disclosure performs first super-resolution processing, second super-resolution processing, and third super-resolution processing in parallel based on an input image according to a super-resolution mode (S1120), and outputs a synthesized image based on the result data of the first super-resolution processing, the result data of the second super-resolution processing, and the result data of the third super-resolution processing as an output image (S1130).

[0262] Meanwhile, the signal processing device (170b) can perform first super-resolution processing, second super-resolution processing, and third super-resolution processing in parallel based on the input image according to the super-resolution mode, and output a synthesized image based on the result data of the first super-resolution processing, the result data of the second super-resolution processing, and the result data of the third super-resolution processing as an output image. Accordingly, it is possible to efficiently perform multiple super-resolution processing.

[0263] Meanwhile, the signal processing device (170b) can increase the sharpness or resolution of edges, textures, and flats of objects in the input image based on the first super resolution processing, the second super resolution processing, the second super resolution processing, and the third super resolution processing on a parallel basis.

[0264] Meanwhile, the signal processing device (170b) can perform learning based on a first learning database during the first super-resolution processing, perform learning based on a second learning database different from the first learning database during the second super-resolution processing in parallel, and perform learning based on a third learning database different from the second learning database during the third super-resolution processing in parallel. Accordingly, multiple super-resolution processing can be efficiently performed.

[0265] For example, the signal processing device (170b) may perform first super-resolution processing centered on the edge of an object in an input image, perform second super-resolution processing centered on the texture in an object in the input image in parallel, and perform third super-resolution processing centered on a flat in an object in the input image in parallel.

[0266] Accordingly, it is possible to efficiently improve the sharpness of edges, textures, and flats of an image while efficiently performing multiple super-resolution processing.

[0267] Meanwhile, the signal processing device (170b) includes a neural network processor (333) for learning processing, and the neural network processor (333) can perform learning based on a first learning database during the first super-resolution processing, perform learning based on a second learning database different from the first learning database during the second super-resolution processing, and perform learning based on a third learning database different from the second learning database during the third super-resolution processing. Accordingly, while efficiently performing multiple super-resolution processing operations, it is possible to efficiently improve the sharpness of edges, textures, and flats of an image.

[0268] Meanwhile, a signal processing device (170b) according to another embodiment of the present disclosure performs first super-resolution processing, second super-resolution processing, and third super-resolution processing in parallel based on an input image, and outputs a synthesized image based on the result data (IMb) of the first super-resolution processing, the result data (IMc) of the second super-resolution processing, and the result data (IMd) of the third super-resolution processing as an output image (IMm). This will be described with reference to FIG. 12.

[0269] FIG. 12 is an example of an internal block diagram of a signal processing device according to another embodiment of the present disclosure.

[0270] Referring to the drawings, a signal processing device (170b) according to another embodiment of the present disclosure includes a first super-resolution processing unit (610) that performs first super-resolution processing based on an input image (IMa), a second super-resolution processing unit (620) that performs second super-resolution processing in parallel with the first super-resolution processing unit (610) based on the input image (IMa), and a third super-resolution processing unit (625) that performs third super-resolution processing in parallel with the first super-resolution processing unit (610) and the second super-resolution processing unit (620) based on the input image (IMa).

[0271] For example, in the case of super resolution mode, the signal processing device (170b) can control the first super resolution processing unit (610), the second super resolution processing unit (620), and the third super resolution processing unit (625) to operate in parallel.

[0272] Meanwhile, the first super resolution processing unit (610) can perform learning based on the first learning database during the first super resolution processing.

[0273] For example, the first super resolution processing unit (610) can perform learning based on the first learning database using the neural network processor (333) during the first super resolution processing.

[0274] Specifically, the first super resolution processing unit (610) within the signal processing device (170b) can perform learning based on the first resolution image and the second resolution image during the first super resolution processing.

[0275] Meanwhile, the second super resolution processing unit (620) can perform learning based on the second learning database during the second super resolution processing.

[0276] For example, the second super resolution processing unit (620) can perform learning based on a second learning database that is different from the first learning database by using the neural network processor (333) during the second super resolution processing.

[0277] Meanwhile, the second super resolution processing unit (620) within the signal processing device (170b) can perform learning based on a third resolution image having a lower resolution than the first resolution image and a pre-processed image based on the second super resolution processing.

[0278] For example, the third super resolution processing unit (625) can perform learning based on a third learning database that is different from the second learning database by using the neural network processor (333) during third super resolution processing.

[0279] Meanwhile, the first super resolution processing unit (610), the second super resolution processing unit (620), and the third super resolution processing unit (625) can each individually perform learning or adjust learning.

[0280] Meanwhile, the first super resolution processing unit (610) processes mainly edges within the input image, the second super resolution processing unit (620) processes textures, for example, in an area excluding edges within the input image, and the third super resolution processing unit (620) can process flat areas within the input image.

[0281] Accordingly, it is possible to efficiently perform multiple super-resolution processing while efficiently improving the sharpness of edges, textures, and flats in the image.

[0282] Meanwhile, the signal processing device (170b) can extract a first feature map (MPa) based on the result data (IMb) of the first super-resolution processing, extract a second feature map (MPb) based on the result data (IMc) of the second super-resolution processing, and extract a third feature map (MPc) based on the result data (IMd) of the third super-resolution processing.

[0283] Meanwhile, the signal processing device (170b) calculates a synthesis ratio (RTa) of the result data (IMb) of the first super-resolution processing, the result data (IMc) of the second super-resolution processing, and the result data (IMd) of the third super-resolution processing based on the first feature map (MPa), the second feature map (MPb), and the third feature map (MPc), and outputs a synthesized image as an output image (IMm) based on the calculated synthesis ratio (RTb).

[0284] To this end, the signal processing device (170b) may further include a first feature map extraction unit (612) that extracts a first feature map (Mpa) based on the result data (IMb) of the first super resolution processing, a second feature map extraction unit (622) that extracts a second feature map (MPb) based on the result data (IMc) of the second super resolution processing, and a third feature map extraction unit (627) that extracts a third feature map (MPc) based on the result data (IMd) of the third super resolution processing.

[0285] Meanwhile, the first feature map extraction unit (612), the second feature map extraction unit (622), and the third feature map extraction unit (627) can extract the degree of improvement in resolution and clarity as feature values ​​or feature maps (MPa, MPb, MPc) based on the result data (IMb, IMc, IMd) of super resolution processing on a pixel-by-pixel basis.

[0286] For example, the feature values ​​or feature maps (MPa, MPb, MPc) output from the first feature map extraction unit (612), the second feature map extraction unit (622), and the third feature map extraction unit (627) may include the levels or energies of the low, middle, and high frequency components of the spatial frequencies of the result data (IMb, IMc, IMd) of each super resolution processing.

[0287] Meanwhile, the first feature map extraction unit (612), the second feature map extraction unit (622), and the third feature map extraction unit (627) may utilize different feature values ​​or feature maps (MPa, MPb, MPc) as feature values ​​or feature maps that indicate the degree of improvement in resolution and clarity.

[0288] Meanwhile, the signal processing device (170b) may further include a feature map analysis unit (630) that calculates a synthesis ratio (RTb) of the result data (IMb) of the first super resolution processing, the result data (IMc) of the second super resolution processing, and the result data (IMd) of the third super resolution processing based on the first feature map (MPa), the second feature map (MPb), and the third feature map (MPc), and an image synthesis unit (640) that outputs a synthesized image as an output image (IMm) based on the synthesis ratio (RTb) calculated by the feature map analysis unit (630).

[0289] Meanwhile, the image synthesis unit (640) can determine the synthesis ratio (RTb) of the result data (IMb) of the first super resolution processing, the result data (IMc) of the second super resolution processing, and the result data (IMd) of the third super resolution processing based on each feature value or feature map (MPa, MPb, MPc) in units of pixels.

[0290] For example, the image synthesis unit (640) can determine a synthesis ratio (RTb) at which resolution and clarity are optimally improved based on each feature value or feature map (MPa, MPb, MPc).

[0291] As another example, the image synthesis unit (640) can vary the synthesis ratio (RTb) for each image component for which resolution and clarity are to be enhanced.

[0292] Meanwhile, the feature map analysis unit (630) or image synthesis unit (640) within the signal processing device (170b) can vary the synthesis ratio (RTb) on a pixel-by-pixel basis.

[0293] For example, the feature map analysis unit (630) or image synthesis unit (640) within the signal processing device (170b) can control the ratio of the result data (IMb) of the first super resolution processing to increase near the edge area within the input image (IMa).

[0294] As another example, the feature map analysis unit (630) or the image synthesis unit (640) within the signal processing device (170b) can control the ratio of the result data (IMc) of the second super resolution processing to increase near the texture area within the input image (IMa).

[0295] As another example, the feature map analysis unit (630) or image synthesis unit (640) within the signal processing device (170b) can control the ratio of the result data (IMd) of the third super resolution processing to increase near the flat area within the input image (IMa).

[0296] Accordingly, it is possible to efficiently perform multiple super-resolution processing while efficiently improving the sharpness of edges, textures, and flats in the image.

[0297] Meanwhile, the feature map analysis unit (630) or image synthesis unit (640) within the signal processing device (170b) can vary the synthesis ratio (RTb) based on the type or resolution of the content.

[0298] For example, the feature map analysis unit (630) or the image synthesis unit (640) can control the ratio of the result data (IMc) of the second super resolution processing among the result data (IMb) of the first super resolution processing, the result data (IMc) of the second super resolution processing, and the result data (IMd) of the third super resolution processing to increase as the resolution increases.

[0299] Accordingly, the sharpness of edges, textures, and flatness of the image can be efficiently improved in response to the resolution of the image.

[0300] As another example, the feature map analysis unit (630) or the image synthesis unit (640) can control the ratio of the result data (IMb) of the first super resolution processing among the result data (IMc) of the second super resolution processing and the result data (IMd) of the third super resolution processing to increase as the motion or motion vector of the content increases.

[0301] Accordingly, the sharpness of edges, textures, and flatness of the image can be efficiently improved in response to the movement of the content.

[0302] In this way, according to the first super resolution processing unit (610), the second super resolution processing unit (620), and the third super resolution processing unit (625) in the signal processing device (170b) of FIG. 12, super resolution processing is implemented in parallel, thereby enabling learning or tuning for each purpose.

[0303] In addition, according to the first super resolution processing unit (610), the second super resolution processing unit (620), and the third super resolution processing unit (625) in the signal processing device (170b) of FIG. 12, there is no dependency in each super resolution processing, so a relatively simple implementation is possible.

[0304] Meanwhile, the feature map analysis unit (630) in the signal processing device (170b) of FIG. 12 can precisely adjust the feature map analysis so that the resolution or clarity of the result data (IMb, IMc, IMd) of each super resolution processing does not deteriorate.

[0305] Meanwhile, the signal processing device (170b), when the input image is a video, can perform first super-resolution processing, second super-resolution processing, and third super-resolution processing in parallel based on the input video, and output a synthesized video as an output video based on the result data (IMb) of the first super-resolution processing, the result data (IMc) of the second super-resolution processing, and the result data (IMd) of the third super-resolution processing.

[0306] Accordingly, it is possible to efficiently improve the sharpness of edges, textures, and flats in a video while efficiently performing multiple super-resolution processing.

[0307] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present invention pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. Display; A signal processing device that processes an input image and outputs an output image; The above signal processing device, An image display device that performs first super-resolution processing and second super-resolution processing in parallel based on the input image, and outputs a synthesized image based on the result data of the first super-resolution processing and the result data of the second super-resolution processing as the output image.

2. In paragraph 1, The above signal processing device, An image display device that increases the sharpness or resolution of edges and textures of objects in the input image based on the first super resolution processing and the second super resolution processing based on the parallel basis.

3. In paragraph 1, The above signal processing device, In the above first super resolution processing, learning is performed based on the first learning database, An image display device that performs learning based on a second learning database different from the first learning database during the second super resolution processing.

4. In paragraph 3, The above signal processing device, a neural network processor for learning processing; The above neural network processor, In the above first super resolution processing, learning is performed based on the first learning database, An image display device that performs learning based on a second learning database different from the first learning database based on the second super resolution processing.

5. In paragraph 1, The above signal processing device, In the above first super resolution processing, learning is performed based on the first resolution image and the second resolution image, An image display device that performs learning based on a third resolution image having a lower resolution than the first resolution image and a pre-processed image based on the second super resolution processing.

6. In paragraph 1, The above signal processing device, Based on the result data of the above first super resolution processing, a first feature map is extracted, Based on the result data of the above second super resolution processing, a second feature map is extracted, Based on the first feature map and the second feature map, a synthesis ratio of the result data of the first super resolution processing and the result data of the second super resolution processing is calculated, An image display device that outputs the synthesized image as the output image based on the above-described calculated synthesis ratio.

7. In paragraph 6, The above signal processing device, A video display device that varies the above synthesis ratio on a pixel basis.

8. In paragraph 6, The above signal processing device, A video display device that varies the synthesis ratio based on the type or resolution of the content.

9. In paragraph 1, The above signal processing device, A first super resolution processing unit that performs first super resolution processing based on the above input image; An image display device comprising a second super resolution processing unit that performs the second super resolution processing in parallel with the first super resolution processing unit based on the input image.

10. In paragraph 9, The above signal processing device, A first feature map extraction unit that extracts a first feature map based on the result data of the first super resolution processing; A second feature map extraction unit that extracts a second feature map based on the result data of the second super resolution processing; A feature map analysis unit that calculates a synthesis ratio of the result data of the first super resolution processing and the result data of the second super resolution processing based on the first feature map and the second feature map; An image display device further comprising an image synthesis unit that outputs the synthesized image as the output image based on the synthesis ratio calculated in the feature map analysis unit.

11. In paragraph 1, The above signal processing device, An image display device that, according to a super resolution mode, performs the first super resolution processing and the second super resolution processing in parallel based on the input image, and outputs a synthesized image based on the result data of the first super resolution processing and the result data of the second super resolution processing as the output image.

12. In paragraph 1, The above signal processing device, An image display device that, when set to sports mode, performs the first super resolution processing and the second super resolution processing in parallel based on the input image, and outputs a synthesized image based on the result data of the first super resolution processing and the result data of the second super resolution processing as the output image.

13. In paragraph 1, The above signal processing device, An image display device that, when the input image is a video, performs the first super-resolution processing and the second super-resolution processing in parallel based on the input image, and outputs a video synthesized based on the result data of the first super-resolution processing and the result data of the second super-resolution processing as an output video.

14. In paragraph 1, The above signal processing device, An image display device that performs a third super-resolution process in parallel based on the input image, and outputs a synthesized image based on the result data of the first super-resolution process, the result data of the second super-resolution process, and the result data of the first super-resolution process as the output image.

15. In paragraph 14, The above signal processing device, Based on the first super resolution processing and the second super resolution processing based on the above parallel basis, the sharpness or resolution of the edge and texture of the object in the input image is increased, An image display device that increases the clarity or resolution of a flat area within an object within an input image based on the third super resolution processing based on the parallel basis.

16. In paragraph 13, The above signal processing device, Based on the result data of the above first super resolution processing, a first feature map is extracted, Based on the result data of the above second super resolution processing, a second feature map is extracted, Based on the result data of the third super resolution processing, a third feature map is extracted, Based on the first feature map, the second feature map, and the third feature map, a synthesis ratio of the result data of the first super resolution processing, the result data of the second super resolution processing, and the result data of the third super resolution processing is calculated, An image display device that outputs the synthesized image as the output image based on the above-described calculated synthesis ratio.

17. In paragraph 16, The above signal processing device, A video display device that varies the above synthesis ratio on a pixel basis.

18. In paragraph 13, The above signal processing device, A first super resolution processing unit that performs first super resolution processing based on the above input image; A second super resolution processing unit that performs the second super resolution processing in parallel with the first super resolution processing unit based on the input image; An image display device comprising: a first super-resolution processing unit and a third super-resolution processing unit that performs the third super-resolution processing in parallel with the first super-resolution processing unit based on the input image; 19. In paragraph 18, The above signal processing device, A first feature map extraction unit that extracts a first feature map based on the result data of the first super resolution processing; A second feature map extraction unit that extracts a second feature map based on the result data of the second super resolution processing; A third feature map extraction unit that extracts a third feature map based on the result data of the third super resolution processing; A feature map analysis unit that calculates a synthesis ratio of the result data of the first super resolution processing, the result data of the second super resolution processing, and the result data of the third super resolution processing based on the first feature map, the second feature map, and the third feature map; An image display device further comprising an image synthesis unit that outputs the synthesized image as the output image based on the synthesis ratio calculated in the feature map analysis unit.

20. Display; A signal processing device that processes an input image and outputs an output image; The above signal processing device, An image display device that performs first super-resolution processing, second super-resolution processing, and third super-resolution processing in parallel based on the input image, and outputs a synthesized image based on the result data of the first super-resolution processing, the result data of the second super-resolution processing, and the result data of the third super-resolution processing as the output image.

Images