Display device and control method thereof

Abstract

The present invention relates to a display device and a control method thereof, wherein the display device includes: a display; a video processing unit for processing a video signal; a graphics processing unit for processing a graphics signal; a mixing unit for mixing an image corresponding to the video signal processed by the video processing unit and a graphics corresponding to the graphics signal processed by the graphics processing unit, so that the video and the image are displayed together on the display; and a main processing unit for identifying video frames and graphics frames assigned with identification information corresponding to each other based on identification information assigned according to the order of corresponding frames for multiple video frames of the video signal and multiple graphics frames of the graphics signal, and controlling the video processing unit and the graphics processing unit so that at least one of the identified video frames and the identified graphics frames is output with a delay, thereby displaying the image of the identified video frame and the graphics of the identified graphics frame together on the display.

Description

Technical Field

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based upon and claims the benefit of priority from Korean Patent Application No. 10-2020-0099484 filed on August 7, 2020, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.

[0003] Certain example embodiments relate to a display apparatus and a control method thereof, and more particularly, to a display apparatus capable of processing an image and a graphic to be displayed together and a control method thereof. Background Art

[0004] An electronic device having a display, such as a television (TV), receives various contents from an external source and displays an image based on the contents on the display.

[0005] With the recent spread of various video services using a network environment, broadcast services or video services that provide content including additional images such as graphics are also increasing.

[0006] An electronic device such as a television may include separate processors for separately processing a video signal and a graphics signal to enhance image quality and display an image based on a mix of the two processed signals.

[0007] In an electronic device including such an independent processor, when one of two signals processed through different paths (e.g., a video signal) is delayed, the video and graphics may be mixed and mismatched with each other, or the boundary between the video and graphics may be distorted, so that the image cannot be displayed normally on the screen, thereby causing inconvenience to the user in viewing the image. Summary of the Invention

[0008] [Technical Issues]

[0009] Certain example embodiments provide a display apparatus and a control method thereof, in which a video signal and a graphic signal output through separate processors are controlled to be synchronized, thereby preventing viewing inconvenience due to mismatch or distortion.

[0010] [Technical solution]

[0011] According to an exemplary embodiment, a display device may be provided, comprising: a display; a video processor configured to process a video signal; a graphics processor configured to process a graphics signal; a mixer configured to mix a video corresponding to the video signal processed by the video processor and a graphics corresponding to the graphics signal processed by the graphics processor to display them together on the display; and a main processor configured to: identify video frames and graphics frames assigned with matching identification information based on identification information of multiple video frames assigned to the video signal and identification information of multiple graphics frames assigned to the graphics signal in the order of corresponding frames, and control the video processor and the graphics processor to delay and output at least one of the identified video frames and the identified graphics frames so that the video of the identified video frame and the graphics of the identified graphics frame are displayed together on the display.

[0012] The main processor may control the video processor and the graphic processor to synchronize and output the identified video frame and the identified graphic frame at a reference time point set based on a predetermined synchronization signal.

[0013] The main processor may sequentially assign identification information to a plurality of video frames, sequentially assign identification information to a plurality of graphic frames, and identify video frames and graphic frames assigned with matching identification information.

[0014] The main processor can control the data of video frames to be sequentially stored in the first queue by assigning identification information to multiple video frames, control the data of graphic frames to be sequentially stored in the second queue by assigning identification information to multiple graphic frames, and control the data of video frames and graphic frames assigned with matching identification information to be output from the first queue and the second queue respectively at a reference time point set based on a predetermined synchronization signal.

[0015] The display device may further include a frame rate converter configured to convert the frame rate of the video signal, and the main processor may identify a delay time occurring during the conversion of the frame rate and control the output of the data of the identified graphics frame to be delayed by the identified delay time compared to the data of the identified video frame between the video frame assigned the matching identification information and the graphics frame.

[0016] The display device may further include a frame rate converter configured to convert a frame rate of a video signal; and a storage unit including a buffer configured to store a graphic signal processed by the graphic processor based on a delay time occurring during conversion of the frame rate.

[0017] The video processor may include a video scaler, and the main processor may set geometry information by executing an application to display a video corresponding to a video signal, and may provide the set geometry information to the video scaler.

[0018] The graphic processor may include a graphic scaler, and the main processor may render graphics corresponding to a graphic signal by executing an application, set a pointer to output the rendered graphics, and provide information about the set pointer to the graphic scaler.

[0019] The display device may further include a storage unit including a first buffer configured to store a video signal processed by the video processor and a second buffer configured to store a graphic signal processed by the graphics processor, and the main processor may control the synchronization and output of the video of the video frame of the video signal stored in the first buffer and the graphic of the graphic frame of the graphic signal stored in the second buffer.

[0020] The synchronization signal may include a vertical synchronization signal for the display.

[0021] According to an exemplary embodiment, a method for controlling a display device may be provided, wherein the method may include: identifying video frames and graphic frames assigned with matching identification information based on identification information of multiple video frames assigned to a video signal and identification information of multiple graphic frames assigned to a graphic signal in the order of corresponding frames; controlling at least one of the identified video frames and the identified graphic frames to be delayed and output so that the video of the identified video frame and the graphics of the identified graphic frame are displayed together on the display; and mixing the video of the output video frame and the graphics of the output graphic frame.

[0022] The method may further include controlling the identified video frame and the identified graphic frame to be synchronized and output at a reference time point set based on a predetermined synchronization signal.

[0023] The method may further include sequentially assigning identification information to a plurality of video frames, and sequentially assigning identification information to a plurality of graphic frames.

[0024] The method may also include: sequentially storing data of video frames in a first queue by assigning identification information to multiple video frames; sequentially storing data of graphic frames in a second queue by assigning identification information to multiple graphic frames; and controlling the data of video frames and graphic frames assigned with matching identification information to be output from the first queue and the second queue, respectively, at a reference time point set based on a predetermined synchronization signal.

[0025] The method may also include: identifying a delay time that occurs during frame rate conversion of a video signal; and controlling the output of data of the identified graphics frame to be delayed by the identified delay time compared to the data of the identified video frame between a video frame and a graphics frame assigned with matching identification information.

[0026] The method may further include storing the graphic signal in a buffer corresponding to a delay time occurring during the frame rate conversion of the video signal.

[0027] The method may further include setting geometry information by executing an application to display a video corresponding to the video signal, and providing the set geometry information to a video scaler.

[0028] The method may further include rendering graphics corresponding to the graphic signal by executing the application, setting a pointer to output the rendered graphics, and providing information about the set pointer to the graphics scaler.

[0029] The display device may further include a first buffer configured to store a video signal processed by the video processor and a second buffer configured to store a graphic signal processed by the graphics processor, and the method may further include controlling the synchronization and output of the video of the video frame of the video signal stored in the first buffer and the graphic of the graphic frame of the graphic signal stored in the second buffer.

[0030] According to an example embodiment, a computer-readable non-volatile recording medium may be provided, in which a program of a method executable by a processor of a display device is recorded, wherein the method may include: identifying video frames and graphic frames assigned with matching identification information based on identification information of multiple video frames assigned to a video signal and identification information of multiple graphic frames assigned to a graphic signal in the order of corresponding frames; and controlling at least one of the identified video frame and the identified graphic frame to be delayed and output so that the video of the identified video frame and the graphics of the identified graphic frame are displayed together on a display; and mixing the video of the output video frame and the graphics of the output graphic frame.

[0031] [Beneficial Effects]

[0032] In an example display device and a control method thereof, a pair of frames corresponding to two images to be displayed together can be controlled to be synchronized and output by an electronic device so that the two images can be matched and displayed normally without distortion, thereby reducing the user's visual fatigue and solving the problem of user viewing inconvenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 An example of an electronic device according to example embodiments is shown.

[0034] Figure 2 is a block diagram of an electronic device according to example embodiments.

[0035] Figure 3 is a block diagram illustrating paths through which video signals and graphic signals are processed in an electronic device according to example embodiments.

[0036] Figure 4 is a flowchart illustrating a control operation for displaying an image in an electronic device according to an example embodiment.

[0037] Figure 5 is a diagram schematically illustrating an operation of a configuration for displaying a first image and a second image together in an electronic device according to an exemplary embodiment.

[0038] Figure 6 is a diagram for explaining a case where image distortion occurs when video and graphics are displayed together in the related art.

[0039] Figure 7 is a diagram showing an example of image distortion occurring in the related art.

[0040] Figure 8 is a flowchart illustrating a control operation for synchronizing and displaying images in an electronic device according to an example embodiment.

[0041] Figure 9 It shows that Figure 8 FIG. 1 is a diagram of the operation of a configuration for synchronizing and displaying images in an embodiment.

[0042] Figure 10 It shows that Figure 8 FIG. 1 is a diagram of operations for synchronizing and displaying frame rate converted images in an example embodiment of FIG.

[0043] Figure 11 Is used to illustrate Figure 8 FIG. 1 is a diagram of an example of synchronizing and outputting video and graphics in an embodiment.

[0044] Figure 12 is a flowchart illustrating a control operation for synchronizing and displaying images in an electronic device according to another example embodiment.

[0045] Figure 13 Shown in Figure 12

[0046] Operations of a configuration for synchronizing and displaying images in an example embodiment of the present invention.

[0046] Figure 14 It shows that Figure 12 FIG. 1 is a diagram of an example of delaying and outputting a graphic signal in an example embodiment of the present invention. DETAILED DESCRIPTION

[0047] Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. In the accompanying drawings, similar reference numerals or symbols refer to similar elements having substantially the same functions, and for the sake of clarity and ease of description, the size of each element may be exaggerated. However, the configurations and functions shown in the following example embodiments should not be interpreted as limiting and key configurations and functions. In the following description, if it is recognized that well-known functions or features will obscure the main idea of ​​the present invention, details about them will be omitted.

[0048] In the example embodiments below, the terms "first", "second", etc. are only used to distinguish one element from another, and unless the context indicates otherwise, the singular form is intended to include the plural form. In the example embodiments below, it will be understood that the terms "comprise", "include", "have", etc. do not exclude the presence or addition of one or more other features, numbers, steps, operations, elements, components, or combinations thereof. In addition, a "module" or "part" can perform at least one function or operation, is implemented by hardware, software, or a combination of hardware and software, and is integrated into at least one module. In the present disclosure, at least one element in a plurality of elements refers not only to all elements in a plurality of elements, but also to each element in a plurality of elements except other elements, and combinations thereof.

[0049] Figure 1 An example of an electronic device according to example embodiments is shown.

[0050] According to an example embodiment, the electronic device 10 may be implemented as a system including: Figure 1 The display 110 is shown as a display device.

[0051] The electronic device 10 according to an exemplary embodiment receives a signal from an external signal source (eg, data regarding content) and processes the received content data according to a preset procedure to be displayed as an image on the display 110 .

[0052] According to an embodiment, the electronic device 10 implemented as a display device may include a TV that processes a broadcast image based on at least one of a broadcast signal, broadcast information, or broadcast data received from a transmitter of a broadcast station. In this case, the electronic device 10 may include a tuner tuned to a channel corresponding to the broadcast signal.

[0053] However, the present disclosure is not limited to the embodiment of the electronic device 10. Alternatively, the electronic device 10 may be implemented as an image processing device, such as a set-top box, that transmits a signal to an external display via a wired or wireless connection. Alternatively, the electronic device 10 may be implemented as a terminal device (hereinafter referred to as a user terminal or user equipment) having a display, such as a smartphone, a tablet computer, or a smart tablet. Alternatively, the electronic device 10 may be applied to a monitor of a desktop or laptop computer (or a personal computer (PC)).

[0054] When the electronic device 10 is a TV, the electronic device 10 can receive broadcast content based on at least one of a broadcast signal, broadcast information, or broadcast data from a transmitter of a broadcast station directly or through an additional device that can be connected to the electronic device 10 via a cable (for example, through a set-top box (STB), an on-device connection box (OC box), a media box, etc.). Here, the connection between the electronic device 10 and the additional device is not limited to a cable, and various wired / wireless interfaces can be used.

[0055] The electronic device 10 may wirelessly receive radio frequency (RF) signals, such as broadcast content transmitted from a broadcast station, for example. To this end, the electronic device 10 may include an antenna for receiving the broadcast signal.

[0056] In the electronic device 10, broadcast content can be received through ground waves, cables, satellites, etc., and the signal source is not limited to a broadcast station. In other words, according to the present disclosure, any device or station capable of transmitting and receiving data can be included in the source.

[0057] The standard of the signal received in the electronic device 10 may vary depending on the type of the device, and the electronic device 10 may be configured according to the interface 120 (see Figure 2 ) receives signals as image content through cables based on standards such as High-Definition Multimedia Interface (HDMI), HDMI-Consumer Electronics Control (CEC), DisplayPort (DP), Digital Visual Interface (DVI), composite video, component video, super video, DVI, Thunderbolt, RGB cable, SCART, and Universal Serial Bus (USB).

[0058] Depending on the embodiment, the electronic device 10 may be implemented as a smart TV or an Internet Protocol (IP) TV. A smart TV is a TV that can receive and display broadcast signals in real time, has web browsing functionality for searching and consuming various content over the internet while simultaneously displaying the broadcast signal in real time, and provides a convenient user environment for this purpose. Furthermore, a smart TV can provide interactive services to users because it includes an open software platform. Therefore, a smart TV can provide users with a variety of content, such as content from applications for pre-defined services, through this open software platform. Such applications include applications for various services, such as social networking services (SNS), finance, news, weather, maps, music, movies, games, e-books, and other services.

[0059] The electronic device 10 may process signals to display moving images, still images, applications, an on-screen display (OSD), a user interface (UI) for controlling various operations, etc. on a screen based on data / signals stored in an internal / external storage medium.

[0060] The electronic device 10 may receive content from various external devices including the server 20 and terminal devices as sources for providing content using wired or wireless network communication, but there is no limitation on the kind of communication.

[0061] Specifically, the electronic device 10 may use wireless network communication to receive signals corresponding to standards such as Wi-Fi, Wi-Fi Direct, Bluetooth, Bluetooth Low Energy, Zigbee, UWB, and NFC as image content corresponding to the type of interface 120 (described later). Furthermore, the electronic device 10 may use wired network communication such as Ethernet to receive content signals.

[0062] According to an embodiment, the external device may be provided as a content provider (e.g., a content server that can transmit content to various devices such as the electronic device 10 via a wired or wireless network). For example, the external device may provide a media file based on a video on demand (VOD) service, web content, etc. in real time via a streaming method.

[0063] According to an exemplary embodiment, there may be a plurality of external devices, for example, servers. In this case, the electronic device 10 may be implemented to be connected to each of the plurality of external devices and receive various contents from each connected external device.

[0064] The electronic device 10 may receive media content or video content based on a VOD service from, for example, an OTT server capable of providing an over-the-top (OTT) service such as Netflix or a web server such as YouTube.

[0065] The electronic device 10 may execute an application for reproducing content (e.g., a VOD application) to receive content from an external device provided for providing content, and process the received content to output (e.g., display) an image corresponding to the content through the display 110. Here, the electronic device 10 may receive content from a server (e.g., an external device) based on a user account corresponding to the executed application.

[0066] According to the embodiment, Figure 1 As shown, the electronic device 10 may display the first image 21 and the second image 22 together on the display 110 .

[0067] Specifically, the electronic device 10 may receive a first signal corresponding to the first image 21 (hereinafter referred to as the first image signal) and a second image signal corresponding to the second image 22 (hereinafter referred to as the second image signal), and process each of the first signal and the second signal. The electronic device 10 may mix the first signal and the second signal processed through the separate paths as described above so that the two images 21 and 22 are displayed together on the display 110.

[0068] Here, the first signal may correspond to a video signal, and the second signal may correspond to a graphic signal.

[0069] The graphic signal may be, for example, a signal for displaying a sub-picture, subtitles, teletext, an on-screen display (OSD), or a user interface (hereinafter referred to as a graphical user interface (GUI)) for providing information (e.g., channel number, program name, etc.) to a user or controlling various operations, but is not limited thereto.

[0070] The graphic signal may be included in the content provided from an external device such as a server, or may be provided from an external device as a separate signal from the content. Here, the external device providing the content may be different from or the same as the external device providing the graphic signal.

[0071] Furthermore, the graphics signal may already be embedded in the electronic apparatus 10 or in an additional device such as a set-top box.

[0072] According to an embodiment, the graphic signal may have a plurality of layers.

[0073] According to an embodiment, the electronic device 10 may display an interactive graphic (IG) or a presentation graphic (PG) generated by processing a graphic signal on the display 110 as the second image.

[0074] According to an embodiment, the second image (ie, graphics) based on the graphic signal may be displayed on the display 110 while being overlaid on the first image based on the video signal, or may be displayed in a separate area from the area where the first image based on the video signal is located.

[0075] Hereinafter, a configuration of an electronic device according to an example embodiment will be described with reference to the accompanying drawings.

[0076] Figure 2 is a block diagram of an electronic device according to example embodiments.

[0077] However, Figure 2 Only exemplary elements of the electronic device 10 according to the example embodiment are shown, and a first electronic device according to an alternative embodiment may include Figure 2 In other words, the electronic device 10 of the present disclosure may also include Figure 2 Other elements than those shown, or may be Figure 2 At least one element is excluded from the elements shown. In addition, the electronic device 10 of the present disclosure can be changed Figure 2 Some of the components shown are implemented.

[0078] like Figure 2 As shown, the electronic device 10 according to example embodiments may include a display 110 .

[0079] The display 110 may display images.

[0080] The display 110 may be implemented by, but is not limited to, various display types such as liquid crystal, plasma, light emitting diode, organic light emitting diode, surface conduction electron emitter, carbon nanotube, nanocrystal, etc. Depending on the embodiment, the display 110 may include a panel for displaying an image thereon, and may also include additional elements (e.g., a driver) depending on its type.

[0081] According to an embodiment, the display 110 may display an image of content received from a source (ie, an external device such as a server).

[0082] According to an embodiment, the display 110 may display a first image based on the first signal and a second image based on the second signal together on the display 110 .

[0083] According to an embodiment, the second image may be displayed on the display 110 while being overlaid on the first image, or may be displayed on a separate area separate from the area where the first image is displayed.

[0084] The electronic device 10 may include an interface 120 .

[0085] The interface 120 allows the electronic device 10 to communicate with various external devices such as a server.

[0086] The interface 120 may include a wired interface 121. The wired interface 121 may include a connector for transmitting / receiving signals / data based on standards such as HDMI, HDMI-CEC, USB, component, DP, DVI, Thunderbolt, RGB cable, etc. Here, the wired interface 121 may include at least one connector, terminal, or port corresponding to each of these standards.

[0087] The wired interface 121 is embodied to include an input port for receiving a signal from a source or the like, and also includes an output port for interactively transmitting and receiving signals as needed.

[0088] The wired interface 121 may include a connector, port, or the like based on a video and / or audio transmission standard (such as an HDMI port, a display port, a DVI port, Thunderbolt, composite video, component video, super video, and SCART) to connect to an antenna for receiving broadcast signals based on a broadcast standard (such as terrestrial / satellite broadcasting) or a cable for receiving broadcast signals based on a cable broadcast standard. Alternatively, the electronic device 10 may include a built-in antenna for receiving broadcast signals.

[0089] When the video / audio signal received through the interface 120 is a broadcast signal, the electronic device 10 may further include a tuner that is tuned to a channel corresponding to the received broadcast signal. The tuner may include a demodulator that demodulates the broadcast signal of the specific tuned channel and outputs the signal in the form of a transport stream (TS). In other words, the tuner and demodulator may be designed as a single integrated chip, or may be designed as two separate chips.

[0090] The wired interface 121 may include a connector or port based on a universal data transmission standard, such as a USB port. The wired interface 121 may include a connector or port to which an optical cable based on an optical transmission standard can be connected. The wired interface 121 may include a connector or port to which an external microphone or an external audio device having a microphone is connected for receiving or inputting audio signals from the microphone or audio device. The wired interface 121 may include a connector or port to which an audio device such as a headset, earphones, or external speakers is connected for sending or outputting audio signals to the audio device. The wired interface 121 may include a connector or port based on a network transmission standard, such as Ethernet. For example, the wired interface 121 may be implemented as a local area network (LAN) connected to a router or gateway via a wire.

[0091] The wired interface 121 is connected to an external device such as a set-top box, an optical media playback device, an external display device, a speaker, a server, etc. via a connector or port in a 1:1 or 1:N (where N is a natural number) configuration, thereby receiving or transmitting video / audio signals from or to the external device. The wired interface 121 may include a connector or port for transmitting video / audio signals separately.

[0092] The wired interface 121 may be implemented by a communication circuit including a wireless communication module (eg, a S / W module, a chip, etc.) corresponding to various communication protocols.

[0093] According to an embodiment, the wired interface 121 may be built in the electronic device 10 , or implemented as a dongle or a module and detachably connected to a connector of the electronic device 10 .

[0094] The interface 120 may include a wireless interface 122 .

[0095] The wireless interface 122 may be implemented in different ways depending on the implementation of the electronic device 100. For example, the wireless interface 122 may adopt wireless communication methods such as radio frequency, Zigbee, Bluetooth, Wi-Fi, ultra-wideband (UWB), near field communication (NFC), etc.

[0096] The wireless interface 122 may be implemented by a communication circuit including a wired or wireless communication module (eg, a S / W module, a chip, etc.) corresponding to various communication protocols.

[0097] According to an embodiment, the wireless interface 122 includes a wireless local area network (WLAN) unit. The WLAN unit may be wirelessly connected to an external device through an access point (AP) under the control of the main processor 180. The WLAN unit includes a Wi-Fi communication module.

[0098] According to an embodiment, the wireless interface 122 includes a wireless communication module that supports one-to-one direct wireless communication between the electronic device 10 and an external device without requiring an access point (AP). The wireless communication module may be implemented to support communication methods such as Wi-Fi Direct, BT, and BLE. When the electronic device 10 performs direct communication with an external device, the storage device 140 may be configured to store identification information (e.g., a Media Access Control (MAC) address or an Internet Protocol (IP) address) regarding the external device with which communication is to be performed.

[0099] In the electronic device 10 according to an exemplary embodiment, the wireless interface 122 is configured to perform wireless communication with an external device through at least one of a WLAN unit and a wireless communication module according to its performance.

[0100] According to alternative embodiments, the wireless interface 122 may also include a communication module based on various communication methods such as mobile communication such as Long Term Evolution (LTE), electromagnetic (EM) communication including magnetic fields, visible light communication (VLC), etc.

[0101] The wireless interface 122 may wirelessly communicate with an external device such as a server on a network, thereby transmitting and receiving data packets to and from the external device.

[0102] The wireless interface 122 may include an infrared (IR) transmitter for transmitting infrared (IR) signals according to an IR communication standard and / or an IR receiver for receiving infrared (IR) signals according to an IR communication standard. The wireless interface 122 may receive or input remote control signals from a remote control or other external device, or transmit or output remote control signals to other external devices via the IR transmitter and / or IR receiver. Alternatively, the electronic device 10 may exchange remote control signals with a remote control or other external device via the wireless interface 122 based on another method (such as Wi-Fi, BT, etc.).

[0103] According to an embodiment, the wireless interface 122 may transmit predetermined data as information about the user's voice received through a voice input terminal such as a microphone to an external device such as a server. Here, there is no limitation on the format / type of the data to be transmitted, and the data may include, for example, an audio signal corresponding to the voice uttered by the user, a voice feature extracted from the audio signal, etc.

[0104] In addition, the wireless interface 122 can receive data based on the processing result of the corresponding user voice from an external device such as a server. Based on the received data, the electronic device 10 can output a sound corresponding to the voice processing result through an internal or external speaker.

[0105] However, the above-described embodiment is merely an example, and the user voice may be processed by the electronic device 10 without being sent to a server. In other words, according to an alternative embodiment, the electronic device 10 may be implemented to function as a speech-to-text (STT) server.

[0106] The electronic device 10 can communicate with an input device such as a remote controller through the wireless interface 122 and receive a sound signal corresponding to the user's voice from the input device.

[0107] In the electronic device 10 according to the embodiment, the communication module for communicating with an external device such as a server and the communication module for communicating with a remote controller may be different from each other. For example, the electronic device 10 may use an Ethernet modem or a Wi-Fi module to communicate with the external device and use a Bluetooth module to communicate with the remote controller.

[0108] In the electronic device 10 according to an alternative embodiment, the communication module for communicating with an external device such as a server and the communication module for communicating with a remote controller may be the same. For example, the electronic device 10 may use a Bluetooth module to communicate with the external device and the remote controller.

[0109] According to an embodiment, the wireless interface 122 may be built in the electronic device 10 , or implemented as a dongle or a module and detachably connected to a connector of the electronic device 10 .

[0110] According to an embodiment, the electronic device 10 may receive a broadcast signal through the interface 120. The electronic device 10 may extract or generate a first signal (or video signal) corresponding to a first image and a second signal (or graphic signal) corresponding to a second image based on data embedded in the broadcast signal.

[0111] According to an embodiment, the electronic device 10 may receive a content signal from an external device such as a server through the interface 120 in a real-time streaming method. The electronic device 10 may extract or generate a first signal (or video signal) corresponding to the first image and a second signal (or graphic signal) corresponding to the second image based on the content signal.

[0112] The electronic device 10 may include a user input receiver 130 .

[0113] The user input receiver 130 transmits various preset control instructions or unlimited information to the main processor 180 (described later) in response to the user input.

[0114] The user input receiver 130 may include various input means for receiving user input.

[0115] According to an embodiment, the user input receiver 130 may include a keypad (or input panel) including buttons such as a power key, number keys, and a menu key provided in the electronic device 10 .

[0116] According to an embodiment, the user input receiver 130 includes an input device that generates a previously set command / data / information / signal for remotely controlling the electronic device 10 and transmits it to the electronic device 10. The input device may include, for example, a remote controller, a game console, a keyboard, a mouse, etc., and receives user input separately from the electronic device 10.

[0117] The remote control may include at least one button for receiving user input. Depending on the embodiment, the remote control may include a touch sensor for receiving user touch input and / or a motion sensor for detecting user-induced motion of the remote control itself. Depending on the embodiment, the input device may include a terminal, such as a smartphone, on which a remote control application is installed. In this case, the input device may receive user touch input via a touchscreen.

[0118] The input device serves as an external device that performs wireless communication with the main body of the electronic apparatus 10 , wherein the wireless communication is based on Bluetooth, IrDA, RF communication, WLAN, or Wi-Fi Direct.

[0119] According to an embodiment, the user input receiver 130 may include a voice input unit for receiving a voice / sound uttered by a user. The voice input unit may be implemented as a microphone capable of receiving a user's voice, and the microphone may be provided in the electronic device 10, separately from the electronic device 10, or in another device (e.g., a remote controller separate from the electronic device 10).

[0120] According to an embodiment, the user input receiver 130 may include a motion detector that detects a user's hand motion (e.g., a hand gesture (hereinafter referred to as a "gesture"). The motion detector of the electronic device 10 may output data by detecting a moving distance, a moving speed, an area of ​​a moving region, etc. of the hand.

[0121] The electronic apparatus 10 may include a storage device 140 .

[0122] The storage device 140 may be configured to store various data of the electronic apparatus 10 .

[0123] The storage device 140 may be implemented as a non-volatile memory (or a writable read-only memory (ROM)) that can retain data and mirror changes even when the electronic device 10 is powered off. Specifically, the storage device 140 may include a flash memory, an HDD, an erasable programmable ROM (EPROM), or an electrically erasable programmable ROM (EEPROM). The storage device 140 may also include a volatile memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). In this case, the read or write speed of the electronic device 10 is faster than that of the non-volatile memory.

[0124] The data stored in the storage device 140 may include, for example, not only an OS for driving the electronic apparatus 10 but also various programs, applications, image data, additional data, etc. that can be executed on the OS.

[0125] Specifically, the storage device 140 may be configured to store signals or data input / output corresponding to operations of elements under the control of the main processor 180. The storage device 140 may be configured to store a control program for controlling the electronic apparatus 10, applications provided by a manufacturer or downloaded from the outside, a related UI, graphics or images for providing a UI, user information, documents, a database, or related data.

[0126] According to an embodiment, the storage device 140 may be configured to store a TV application or a TV client as a program for using the electronic apparatus 10 as a TV, and a VOD application as a program for reproducing content received from an external device such as a server.

[0127] According to an embodiment, images displayed on the electronic device 10 (e.g., a first image (video) and a second image (graphics)) may be based on data stored in a non-volatile storage device 140, such as a flash memory or a hard disk. The storage device 140 may be provided inside or outside the electronic device 10, and an external storage device 140 may be connected to the electronic device 10 via the wired interface 121.

[0128] According to an example embodiment, the term "storage device" is defined to include the storage device 140, a ROM (not shown) in the main processor 180, a RAM (not shown) that can be installed to the electronic device 10, or a memory card (not shown, for example, a micro SD card, a memory stick, etc.).

[0129] The electronic device 10 may include a first processor 150 .

[0130] The first processor 150 (hereinafter referred to as a video processor) may process a first signal (ie, a video signal) so that a corresponding first image may be displayed on the display 110 .

[0131] The electronic device 10 according to an exemplary embodiment may process a video signal through a first path (eg, a video path using the first processor 150 ) distinguished from a second path (eg, a graphics path processing a graphic signal using the second processor 160 (described later)).

[0132] Figure 3 is a block diagram illustrating paths through which video signals and graphic signals are processed in an electronic device according to example embodiments.

[0133] The first processor 150 is configured to perform various previously set processes on the first image signal (eg, video signal), and as shown in FIG. Figure 3 As shown, a video decoder 151 for decoding an image signal to match an image format of the electronic device 10 may be included.

[0134] According to an embodiment, the video decoder may be implemented, for example, by an H.264 / AVC decoder, but is not limited thereto. In other words, the video decoder in this embodiment may be implemented by a decoder such as a Moving Picture Experts Group (MPEG) decoder or a High Efficiency Video Codec (HEVC) decoder corresponding to various compression standards.

[0135] The video decoder can be implemented as a hardware decoder or a software decoder.

[0136] According to an embodiment, the electronic device 10 may include a plurality of video decoders. Here, the plurality of video decoders provided in the electronic device 10 may be implemented as hardware decoders or software decoders, or in the form of a combination of hardware decoders and software decoders.

[0137] The first processor 150 may include a video scaler (hereinafter referred to as a V scaler) 152 configured to adjust a first signal (eg, a video signal) to match an output format (eg, panel specifications of the display 110 ).

[0138] According to an embodiment, the video scaler 152 may process the video signal according to frames in response to a synchronization signal. Here, the synchronization signal may be, for example, a vertical synchronization signal Vsync for the display 110.

[0139] Specifically, the video scaler 152 may process (ie, scale) a plurality of video frames of a video signal based on a synchronization signal (eg, a vertical synchronization signal for the display 110 ).

[0140] According to an embodiment, the video scaler 152 may be implemented by hardware (e.g., a chip) and outputs a plurality of scaled video frames to be stored in a buffer or a frame buffer (hereinafter referred to as a video buffer). Here, the buffer may be provided in the storage device 140 implemented as a memory such as DRAM, for example.

[0141] The video scaler 152 may scale and output a video frame based on geometric information recognized or set by software (eg, the application 181 installed in the electronic device 10 ).

[0142] Here, the geometric information may include size information and position information of the first image (eg, video corresponding to the video signal). In other words, the geometric information may include coordinate values ​​(x, y, w, h) as geometric parameters for representing video and graphics.

[0143] In addition, the geometric information may include information about a start point and an end point of the second image (eg, graphics corresponding to the graphic signal), and each frame of the graphic signal may be controlled to be partially output between the start point and the end point.

[0144] According to the embodiment, Figure 3 As shown, geometric information may be transmitted from a main processor 180 (described later) (e.g., a central processing unit (CPU)) to the video scaler 152. The main processor 180 (e.g., the CPU) may execute an application 181 installed in the electronic device 10, obtain geometric information or data from the executed application, and provide the obtained geometric information or data to the video scaler 152. Here, the application 181 may generate an alpha value representing transparency information of the graphic together with the geometric information, and provide the generated alpha value to the main processor 180 (e.g., the CPU).

[0145] The video scaler 152 may adjust the size and position of an image (ie, video) corresponding to a video signal based on the geometric information received from the CPU 180 and generate an output image (ie, video).

[0146] According to an embodiment, the first processor 150 may include a frame rate converter (FRC) 153 that performs frame rate conversion (FRC) on the first signal (ie, the video signal output from the video scaler 152 ).

[0147] The frame rate converter 153 may be implemented by a hardware chip separate from the video scaler 152 or may be designed as a single chip in which the video scaler 152 and the frame rate converter 153 are integrated.

[0148] Through frame rate conversion (FRC), the number of frames per second of the first image output to the display 110 may be converted. In this process, video frame delay, in which the first signal (eg, video signal) is delayed, may be generated in the process.

[0149] Such video frame delay may occur in the FRC regardless of the synchronization of the video signal based on the aforementioned synchronization signal.

[0150] Furthermore, the video frame delay due to FRC may depend on the type of video signal. For example, the video frame delay due to FRC may be generated by a predetermined number of frames (e.g., three frames in movie video mode), but may not be generated in game mode. In this case, the electronic device 10 can identify the video frame delay due to FRC based on its operating mode.

[0151] At the same time, the first processor 150 may perform, for example, at least one of various processes such as deinterlacing for converting an interlaced broadcast signal into a progressive broadcast signal, noise reduction for improving image quality, detail enhancement, frame refresh rate conversion, and line scanning for processing a video signal. However, this is merely an example, and a configuration for performing the aforementioned processes may be additionally provided.

[0152] The electronic device 10 may include a second processor 160 .

[0153] The second processor 160 (hereinafter referred to as a graphic processor) may process a second signal (eg, a graphic signal) so that a second image corresponding to the processed second signal may be displayed on the display 110 .

[0154] The second processor 160 processes the graphic signal through a second path (eg, a graphic path) different from the first path for the video signal processed by the first processor 150 .

[0155] The second processor 160 is configured to perform various processes on the second image signal (eg, graphic signal), and as shown in FIG. Figure 3 As shown, a graphics processing unit (GPU) 161 that performs calculations for processing graphics may be included.

[0156] According to an embodiment, the GPU 161 may perform at least one of various preset processing operations (e.g., various processing operations such as animation processing, color conversion, gamma conversion, and acceleration processing) on ​​a second image signal (e.g., a graphics signal) received from an external device such as a server or embedded in the electronic device 10. However, the present disclosure is not limited to this embodiment, and the GPU 161 may also perform various graphics processing operations in addition to the aforementioned processing. Furthermore, the electronic device 10 may further include a configuration for processing graphics signals as needed.

[0157] The GPU 161 may obtain geometry information and alpha values ​​from the main processor 180 (eg, CPU), and send the geometry information and the alpha values ​​to the graphics scaler 162 and the graphics quality block 163 .

[0158] The second processor 160 may include a graphics scaler (hereinafter referred to as a G scaler) 162 to output graphics corresponding to the second signal (eg, graphics signal) to the display 110. Figure 3 As shown, the graphics scaler 162 may be provided in a graphics plane (GP) block controlled by the GPU 161 .

[0159] According to an embodiment, the graphics scaler 162 may process the graphics signal according to the frame in response to a synchronization signal. Here, the synchronization signal may be, for example, a vertical synchronization signal Vsync for the display 110. The vertical synchronization signal may be provided from the main processor 180 (e.g., CPU) to the graphics scaler 162 of the GPU block via the GPU 161.

[0160] According to an embodiment, the graphics scaler 162 may be implemented by hardware (e.g., a chip) so that the graphics frames may be sequentially stored and output to a buffer or frame buffer (hereinafter referred to as a graphics buffer). Here, the buffer may be provided in the storage device 140 implemented as a DRAM, for example.

[0161] The graphics scaler 162 may scale the graphics frame based on geometry information recognized or set by software (eg, the application 181 installed in the electronic device 10 ).

[0162] Specifically, the graphics scaler 162 may output a graphics frame based on a preset pointer (or rendering pointer, for example, the start and end points included in the geometry information after the application 181 renders the graphics in the buffer). The pointer is variable and may be set or applied, for example, relative to the vertical synchronization signal Vsync for the display 110. According to an embodiment, the application 181 may be executed by the main processor 180 and perform rendering for the graphics signal.

[0163] According to an embodiment, the vertical synchronization signal of the display 110 may be provided to the graphics scaler 162 through the video scaler 152 , so that the graphics scaler 162 may sequentially output the graphics frames rendered in the buffer.

[0164] The second processor 160 may include a graphics quality block 163 for processing graphics quality. Figure 3 As shown, the graphics quality block 163 can be set in the GP block controlled by the GPU 161.

[0165] According to an embodiment, the GP block may start operating in response to a vertical synchronization signal for the display 110 received from the video scaler 152. The GPU 161 may receive an alpha value generated in the application 181 via the host processor 180 (e.g., CPU) and transmit the received alpha value to the GP block. In addition, the GPU 161 may transmit geometric information obtained from the host processor 180 (e.g., CPU) to the GP block.

[0166] The graphics signal is adjusted for size and position of graphics based on geometric information by the graphics scaler 162 forming the second path (ie, graphics path), and is also subjected to graphics quality processing by the graphics quality block 163 .

[0167] The electronic device 10 may include a mixer 170 .

[0168] The mixer 170 can mix the first signal and the second signal so that a first image (e.g., video) corresponding to the first signal (e.g., video signal) processed by the first processor 150 and a second image (e.g., graphics) corresponding to the second signal (e.g., graphics signal) processed by the second processor 160 can be displayed together on the display 110.

[0169] The mixer 170 may combine the first signal (eg, a video signal) and the second signal (eg, a graphic signal) and output the combined signal to the display 110 .

[0170] According to an embodiment, the mixer 170 may be embodied by hardware (eg, a chip) and ultimately output synchronized video frames and graphic frames to the display 110 .

[0171] like Figure 3 As shown, the mixer 170 may receive a graphics signal (e.g., graphics data) output via the GP block (e.g., graphics sealer 162 and graphics quality block 163) and a video signal (e.g., video data) output from the video sealer 152 or the FRC 153. The mixer 170 may mix the received video data and graphics data.

[0172] According to an embodiment, the mixer 170 may perform alpha blending to synthesize an image based on transparency information regarding the transparency of an image to be superimposed (e.g., a second image (e.g., graphics)). Transparency information is generally referred to as an alpha value, and the alpha value may be, for example, 8-bit data for distinguishing transparency from '0' to '255'.

[0173] The mixer 170 may combine (e.g., blend) the video signal and the graphics signal based on a predetermined algorithm with reference to an alpha value indicating transparency information. An image including the combined video and graphics (e.g., alpha-blended in the mixer 170 based on the alpha value) may be output via the display 110. The mixer 170 may use various known algorithms, but is not limited to a specific method.

[0174] According to an embodiment, the mixer 170 may merge a lower layer corresponding to a video signal such as broadcasting and an upper layer corresponding to a graphic signal such as sub-picture, subtitle, teletext, and OSD.

[0175] As described above, the first signal (e.g., video signal) and the second signal (e.g., graphic signal) are mixed by the mixer 170 and output to the display 110, so that two images (in other words, the first image (e.g., video) and the second image (e.g., graphic)) can be displayed together on one screen of the display 110.

[0176] The electronic device 10 may include a main processor 180 .

[0177] The main processor 180 performs control to operate general components of the electronic device 10. The main processor 180 may include a control program (or instruction) for performing such control operations, a non-volatile memory in which the control program is installed, a volatile memory (such as a dynamic RAM (DRAM)) in which at least a portion of the installed control program is loaded, and at least one processor (e.g., a microprocessor, an application processor, or a CPU) for executing the loaded control program.

[0178] The processor of the main processor 180 may include a multi-core processor such as a single-core processor, a dual-core processor, a triple-core processor, a quad-core processor, etc. In addition, the processor, ROM, and RAM may be connected to each other through an internal bus.

[0179] According to an embodiment, the processor may include a plurality of processors. For example, the electronic device 10 may separately include a sub-processor that operates only with standby power in a sleep mode in which the electronic device 10 is not fully operational.

[0180] According to an embodiment, the main processor 180 may execute the application 181 installed in the electronic device 10 and identify geometric information for displaying a first image (e.g., video) corresponding to a video signal and a second image (e.g., graphics) corresponding to a graphic signal. The identified geometric information (e.g., information about the size and position of the video) may be provided to the video scaler 152.

[0181] In addition, the main processor 180 can execute the application 161 to perform rendering based on the graphics signal in the buffer. The graphics scaler 162 can be controlled to output such a rendered graphics image based on a pointer including the identified geometric information (eg, information about the start and end points of the graphics).

[0182] According to an embodiment, the main processor 180 may be implemented to be included in a main SoC mounted to a PCB provided inside the electronic device 10 .

[0183] The control program may include a program implemented by at least one of a BIOS, a device driver, an OS, firmware, a platform, or an application. According to an exemplary embodiment, the application program may be pre-installed or stored in the electronic device 10 when the electronic device 10 is manufactured, or may be installed in the electronic device 10 in the future when needed based on application data received from the outside. The application data may be downloaded to the electronic device 10 from an external server such as an application market, for example. Such an external server is merely an example of a computer program product according to the present disclosure, but is not limited thereto.

[0184] The control program can be recorded on a storage medium that is readable by a machine, such as a computer. The machine-readable storage medium can be provided in the form of a non-transitory storage medium or a non-volatile storage medium. Here, the term "non-transitory storage medium" refers to a tangible device and does not include signals (e.g., electromagnetic waves). The term does not distinguish between semi-permanent storage of data in a storage medium and temporary storage of data. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data.

[0185] Figure 4 is a flowchart illustrating a control operation of displaying an image in an electronic device according to an exemplary embodiment.

[0186] According to an embodiment, the first processor 150 and the second processor 160 of the electronic device 10 may process and output an image for each frame in response to a synchronization signal. Specifically, the first processor 150 may process a first signal (e.g., a video signal) in response to the synchronization signal so that the first image (e.g., video) can be output in units of frames. Furthermore, the second processor 160 may process a second signal (e.g., a graphics signal) in response to the synchronization signal so that the second image (e.g., graphics) can be output in units of frames.

[0187] Two images (eg, a first image (eg, video) and a second image (eg, graphics)) output from the first processor 150 and the second processor 160 may be mixed (eg, combined) by the mixer 170 to be displayed together on the display 110 .

[0188] like Figure 4 As shown, the main processor 180 may identify video frames and graphic frames assigned with matching identification information with respect to a plurality of frames of a first signal (eg, a video signal) and a plurality of frames of a second signal (eg, a graphic signal) ( 301 ).

[0189] Here, the identified video frames and graphics frames may form pairs corresponding to video and graphics, respectively, to be displayed together on the display 110. For example, one frame V1 among multiple frames of a video signal and one frame G1 among multiple frames of a graphics signal to be displayed together with frame V1 may be identified as a pair of frames.

[0190] The main processor 180 may control the video processor 150 and the graphics processor 160 to display images corresponding to the pair of frames identified in operation 301 (e.g., the image (video) of the video frame V1 and the graphics of the graphics frame G1) together on one screen (302). During the control process of operation 302, the identified video frame V1 may be controlled to be delayed and output from the video processor 150, or the graphics frame G1 may be controlled to be delayed and output from the graphics processor 160, so that the image of the identified video frame V1 and the graphics of the identified graphics frame G1 can be displayed together on the display 110.

[0191] According to an embodiment, the aforementioned operations of the main processor 180 may be implemented by a computer program stored in a computer program product (not shown) provided separately from the electronic device 10 .

[0192] In this case, the computer program product includes a non-transitory or non-volatile memory storing instructions corresponding to the computer program, and a processor. The instructions are executable by the processor to identify a video frame and a graphics frame assigned with matching identification numbers among a plurality of video frames and a plurality of graphics frames, and to delay and output at least one of the identified video frame and the identified graphics frame so that an image of the identified video frame and a graphic of the identified graphics frame can be displayed together.

[0193] Therefore, the electronic device 10 may download and execute a computer program stored in a separate computer program product and perform the operations of the main processor 180 .

[0194] Hereinafter, embodiments of synchronizing and outputting images in an electronic device according to the present disclosure will be described with reference to the accompanying drawings.

[0195] Figure 5 is a diagram schematically illustrating an operation of a configuration for displaying a first image and a second image together in an electronic device according to an exemplary embodiment.

[0196] In the electronic device 10 according to the embodiment, as Figure 5 As shown, the video driver 182 and the graphics driver 183 may be executed by the main processor 180 in a software layer of the application 181 and may send information to the video scaler (V scaler) 152 and the graphics scaler (G scaler) 162 .

[0197] The video driver 182 may, for example, set video geometry information for displaying the first image (video) and transmit the set geometry information to the video scaler 152. Here, the geometry information may include information on the size and position of the video.

[0198] The graphics driver 183 may, for example, perform graphics rendering (or image rendering) for the second image (graphics) in the buffer of the display memory 184, set a buffer pointer, and transmit the set pointer information to the graphics scaler 162. Here, the pointer information includes information on the start and end points of the graphics, so that each frame of the rendered graphics can be output within the portion between the start and end points.

[0199] The video scaler 152 may perform scaling on each frame (video frame) of the first signal (eg, the video signal processed by the video decoder 151 based on the geometric information) and output the scaled video frame to the mixer 170 .

[0200] According to an embodiment, the scaled video frames in the video scaler 152 are frame rate converted in the frame rate converter 153 and then output to the mixer 170. In other words, the video frames output from the video scaler 152 are sent to the frame rate converter (FRC) 153, are frame rate converted in the frame rate converter 153, and are provided to the mixer 170 to be mixed with the graphic frames.

[0201] The graphics scaler 162 may output each frame (graphics frame) of the second signal (ie, the graphics signal for which image rendering has been performed in the buffer) to the mixer 170 .

[0202] The mixer 170 can mix the video frames provided from the video scaler 152 or the frame rate converter 153 and the graphics frames provided from the graphics scaler 162, and output the mixed frames to the display 110 so that the first image (video) and the second image (graphics) that match each other can be displayed together on the display 110.

[0203] Even if you will refer to it later Figure 8 and Figure 12 In the described embodiments, the aforementioned processing and mixing of video and graphics and the aforementioned transmission of information may also be performed similarly.

[0204] However, contrary to the present disclosure, image distortion may occur in the related art in which video and graphics are displayed together.

[0205] Figure 6 is a diagram for explaining a case where image distortion occurs when video and graphics are displayed together in the related art, and Figure 7 is a diagram showing an example of image distortion occurring in the related art.

[0206] exist Figure 6 In the related art shown, a video signal and a graphic signal may be processed for each frame in response to a predetermined synchronization signal. Here, the synchronization signal may use a vertical synchronization signal Vsync of the display 110 for displaying an image thereon.

[0207] However, even when geometry information is set for both the video signal and the graphics signal in the application layer, either signal may be delayed. For example, the data of the video signal may arrive at the video scaler 152 later than the first vertical synchronization signal of the display 110.

[0208] Reference Figure 6 , when the data of the graphic signal reaches the time point ta, the graphic scaler 162 may control the first graphic frame G1 to be output at the time point of the first pulse of the vertical synchronization signal (ie, timing T1 ).

[0209] However, since the data of the video signal arrives at the time point tb later than the data of the graphic signal by a predetermined time Δt, the video scaler 152 controls the first video frame V1 to be output at the timing T2 of the second pulse of the vertical synchronization signal, as shown in FIG. Figure 6 In addition, the graphics scaler 162 may control the second graphics frame G2 to be output at the same timing T2 of the second pulse.

[0210] In other words, in Figure 6 In the related art, it is impossible to identify whether a video frame and a graphic frame output simultaneously match each other, and thus a video frame V1 and a graphic frame G1 to be displayed on one screen are output at different time points.

[0211] Therefore, the video frame V1 and the graphic frame G2 that do not match each other are output at the same time, and thus the two images are not normally synchronized, resulting in image distortion.

[0212] For example, such synchronization failures may occur during the Out-of-Box Experience (OOBE) process (i.e., the initialization process after a consumer purchases a TV), or during the authentication process of a YouTube application (App) installed on the TV.

[0213] Specifically, in the final stage of OOBE, Figure 7 As shown, when the video 61 and the graphic 62 are displayed on the screen at the same time, image distortion may occur, so that a black border portion 63 (ie, a black border between the video 61 and the graphic 62) is displayed while the area of ​​the video 61 is gradually enlarged.

[0214] In the electronic device 10 according to the exemplary embodiment, a synchronization process is performed to prevent the aforementioned image distortion.

[0215] Figure 8 is a flowchart illustrating a control operation for synchronizing and displaying images in an electronic device according to an example embodiment, and Figure 9 It shows that Figure 8 FIG. 1 is a diagram of the operation of a configuration for synchronizing and displaying images in an embodiment.

[0216] and Figure 4 Similar to the embodiment, Figure 8 An embodiment is characterized in that at least one of the video frames and the graphics frames is controlled to be delayed by operating to identify a pair of frames to be displayed together based on identification information (for example, a serial number assigned according to the order of the frames), assigning identification numbers to the video frames and the graphics frames, storing data of the video frames and the graphics frames assigned the identification numbers in a queue, and then popping the stored data.

[0217] According to an embodiment, the electronic device 10 may operate in a mode for synchronizing video and graphics, and control a synchronization process (described later) to be performed.

[0218] like Figure 8 As shown, the main processor 180 of the electronic device 10 may assign or match information such as numbers (e.g., serial numbers) to frames (video frames) of a first signal (e.g., a video signal) and frames (graphic frames) of a second signal (e.g., a graphic signal), and store data of the video signal and data of the graphic signal assigned with the numbers of each frame in each queue (701). Here, the queue has a format for storing data in a first-in-first-out (FIFO) method, so that data of the video frame and the graphic frame can be sequentially stored in the queue and output from the queue.

[0219] In operation 701, the main processor 180 may assign identification information for each frame to the video signal and the graphics signal based on execution (e.g., the start of a synchronization mode) so that data of the video frame and the graphics frame (e.g., information about the size and position of the video as geometric information, and information about a rendering pointer of the graphics) may be stored (e.g., enqueued) in sequence along with their assigned numbers.

[0220] Reference Figure 9The video driver 182, executed by the main processor 180, assigns serial numbers (such as 1, 2, 3, 4, ...) as identification information to frames of the video signal (video frames) and stores the associated data or information in a first queue 801, which is provided as a video geometry information queue. Similarly, the graphics driver 183, executed by the main processor 180, assigns serial numbers (such as 1, 2, 3, 4, ...) as identification information to frames of the graphics signal (graphics frames) and stores the associated data or information in a second queue 802, which is provided as a graphics pointer queue. Therefore, video frames and graphics frames assigned the same identification information (or serial numbers) are displayed together on the display 110 as a pair of matching frames.

[0221] The video driver 182 and the graphics driver 183 executed by the main processor 180 may control data to be simultaneously ejected at a predetermined reference time point (e.g., outputted from the queues 801 and 802, respectively) (702). Here, the reference time point may be set based on a predetermined synchronization signal, and may be, for example, the timing T1 of the first pulse of the vertical synchronization signal Vsync for the display 110 (as an interrupt service routine (IRS)).

[0222] That is, in Figure 9 In the illustrated embodiment, for example, the data of the video frame V1 stored in the first queue 801 and the data of the graphic frame G1 stored in the second queue 802 can be controlled to be popped out (e.g., dequeued) from the queues 801 and 802 respectively as a pair of frames assigned with matching identification information at a reference time point.

[0223] Therefore, the data of the video frame and the data of the graphic frame are popped out from the first queue 801 and the second queue 802 at the reference time point at the same time and sent to the video scaler 152 and the graphic scaler 162 respectively.

[0224] Here, the video driver 182 may provide the video scaler 152 with geometry information (or video geometry information) set for the popped video frame, and the graphics driver 183 may perform graphics rendering for the popped graphics frame and provide the graphics scaler 162 with information about the buffer pointer as geometry information.

[0225] According to an embodiment, the main processor 180 can compare multiple pieces of identification information (i.e., serial numbers respectively assigned to video frames and graphic frames popped up at a reference time point) and control data to be sent to each of the video scaler 152 and the graphic scaler 162 based on recognizing that two pieces of identification information (numbers) correspond to each other (e.g., match).

[0226] In other words, the main processor 180 can control the information about the buffer pointer and the video geometry information to be sent as geometry information to each scaler in the video scaler 152 and the graphics scaler 162, and set it when it is identified based on the comparison result between the serial numbers that the video frames and graphics frames popped out simultaneously from the queues 801 and 802 correspond to each other.

[0227] In addition, the main processor 180 may control the video processor 150 and the graphic processor 160 to synchronize and output the image of the video frame V1 and the image of the graphic frame G1 assigned with the pieces of identification information (serial numbers) transmitted in operation 702 and corresponding to each other ( 703 ).

[0228] Here, the main processor 180 may, for example, control the video frame V1 and the graphic frame G1 to be synchronized at the timing T2 of the next pulse of the vertical synchronization signal Vsync and output from the video scaler 152 and the graphic scaler 162 , respectively, and provided to the mixer 170 .

[0229] In the above-described embodiment, in a state where data is stored in both the first queue 801 and the second queue 802 , the data is popped out at the same time, and thus the video frame and the graphic frame must be controlled to be synchronized and output as a pair.

[0230] During this process, at least one of the video frame V1 and the graphics frame G1 can be controlled to be delayed. For example, when the video decoder 151 of the video processor 150 forming the video path delays outputting the video frame V1 to the video scaler 152, the graphics scaler 162 of the graphics processor 160 forming the graphics path can delay and output the graphics frame G1. In other words, the graphics scaler 162 can receive geometry information (or rendering pointer information) from the queue 802 as data for the graphics frame G1 and output the possibly delayed graphics frame G1 based on the received geometry information.

[0231] Therefore, if Figure 6 As shown, even if one frame (e.g., graphics frame G1) arrives earlier, graphics frame G1 is controlled to be delayed so that video frame V1 and graphics frame G1 can be synchronized and output together after the matching video frame V1 arrives, thereby avoiding image distortion when video and graphics that do not match each other are displayed together.

[0232] According to an embodiment, the video frame scaled by the video scaler 152 may be output to the mixer 170 through a frame rate converter (FRC) 153 .

[0233] Figure 10 It shows that Figure 8 A diagram of operations for synchronizing and displaying frame rate converted images in an embodiment of the present invention.

[0234] according to Figure 10The electronic device 10 of the illustrated embodiment may be implemented to further include a graphics buffer for additional buffering corresponding to a delay time of a video frame (in other words, FRC delay) occurring in frame rate conversion of a video signal.

[0235] In the electronic device 10 according to an example embodiment, the memory 184 may include a video buffer 901 for sequentially storing video frames output by the video scaler 152 and a graphic buffer 902 for sequentially storing graphic frames output by the graphic scaler 162 .

[0236] In addition, if Figure 10 As shown, the memory 184 may further include an FRC buffer 903 as a graphics buffer for storing a graphics frame corresponding to an FRC delay of a video frame.

[0237] For example, when three frames are delayed while the frame rate converter 153 converts the frame rate of the video signal, the FRC buffer 903 may be configured to have a size corresponding to three frames.

[0238] Therefore, even if the video frame output from the video scaler 152 is delayed and sent to the mixer 170 during the FRC period of the frame rate converter 153, the graphic frame is delayed and provided to the mixer 170 to offset the FRC delay through the FRC buffer 903, so that the video frame and the graphic frame are synchronized and output.

[0239] Figure 11 Is used to illustrate Figure 8 FIG. 1 is a diagram of an example of synchronizing and outputting video and graphics in an embodiment.

[0240] exist Figure 11 In the illustrated embodiment, the video signal and the graphic signal may be synchronized and output for each frame in response to a predetermined synchronization signal. Here, the synchronization signal may be a vertical synchronization signal Vsync of the display 110 for displaying an image thereon.

[0241] like Figure 11 As shown, in the electronic device according to the exemplary embodiment, the data or information of the video frame V1 and the graphic frame G1 respectively stored in the first queue 801 and the second queue 802 are popped out from the queues 801 and 802 corresponding to a pair of frames at the timing T1 of the synchronization signal, and then the video frame V1 and the graphic frame G1 are controlled to be synchronized and output by the video scaler 152 and the graphic scaler 162 at the timing T2 of the synchronization signal.

[0242] In the same manner, data or information of the video frame V2 and the graphic frame G2 are popped out from the queues 801 and 802 corresponding to a pair of frames at timing T2, and then the video frame V2 and the graphic frame G2 are controlled to be synchronized and output by the video scaler 152 and the graphic scaler 162 at timing T3 of the synchronization signal.

[0243] Therefore, according to Figure 8 In the electronic device 10 of the embodiment, the data of the video frame and the graphic frame (whose identification information, i.e., the numbers (sequence numbers) must correspond to each other) are popped out from the queues 801 and 802 at the same time as a pair, and therefore, even if one of the video signal and the graphic signal is delayed due to a different processing path, the same situation will not occur. Figure 6 The problem of outputting video frames and graphic frames matching each other at different time points as described in the related art of FIG. Figure 7 The image shown is mismatched or distorted due to a failure in synchronization between the video and graphics.

[0244] Figure 12 is a flowchart illustrating a control operation for synchronizing and displaying images in an electronic device according to an alternative exemplary embodiment, Figure 13 Shown Figure 12 The operation of the configuration for synchronizing and displaying an image in an embodiment, and Figure 14 It shows that Figure 12 FIG. 1 is a diagram of an example of delaying and outputting a graphic signal in an embodiment.

[0245] according to Figure 12 The electronic device of an embodiment recognizes a delay time (e.g., FRC delay) occurring during frame rate conversion of a first signal (e.g., a video signal) and pre-delays a second signal (e.g., a graphic signal) by as much as the FRC delay, thereby controlling the first signal and the second signal to be synchronized and output.

[0246] and Figure 3 and Figure 6 Similar to the embodiment, Figure 12 An embodiment is characterized in that a pair of frames to be displayed together are identified based on identification information (e.g., serial numbers assigned according to the order of frames), and data of the graphics frame assigned the identification number is delayed more than data of the video frame due to FRC delay and pops up.

[0247] Specifically, if Figure 12As shown, the main processor 180 of the electronic device 10 may assign or match a number (e.g., a serial number as an identification number) to a frame of a first signal (e.g., a video signal) and a frame of a second signal (e.g., a graphic signal), and store the data of the video signal and the data of the graphic signal assigned with the number according to the frame in a queue (1101). Here, the queue is configured to store data in a FIFO method, and thus the data of the video frame and the graphic frame can be sequentially stored in the queue and output.

[0248] In operation 1101, the main processor 180 may assign identification information of each frame to the video signal and the graphic signal based on execution (e.g., the start of a synchronization mode) so that data (e.g., geometric information of the video frame and the graphic frame) are sequentially stored (e.g., enqueued) along with their assigned numbers.

[0249] Reference Figure 9 The video driver 182 may be executed by the main processor 180 to assign serial numbers (such as 1, 2, 3, 4, ...) as identification information to frames of the video signal (video frames) and store them in the first queue 1201. Similarly, the graphics driver 183 may be executed by the main processor 180 to match serial numbers (such as 1, 2, 3, 4, ...) as identification information to frames of the graphics signal (graphics frames) and store them in the second queue 1202. Therefore, the video frame and the graphics frame assigned the same identification information (e.g., serial number) will be displayed together on the display 110 as a pair of matching frames.

[0250] The graphics driver 183 executed by the main processor 180 may identify an FRC delay occurring during frame rate conversion of a first signal (eg, a video signal) ( 1102 ). Here, the main processor 180 may obtain information about the FRC delay occurring due to FRC from the frame rate converter 153 .

[0251] Graphics driver 183 may control data of a second signal (e.g., a graphics frame) (stored in second queue 1202 and matching the sequence number in operation 1101) to be popped (e.g., dequeued) from second queue 1202, delayed further than the first signal (e.g., a video frame) (stored in first queue 1201 and matching the same sequence number in operation 1101) corresponding to the FRC delay identified in operation 1102 (1103). In this process, the graphics rendering timing of graphics driver 183 may be delayed corresponding to the FRC delay.

[0252] For example, when it is identified in operation 1102 that the FRC delay corresponds to two frames, as shown in FIG. Figure 14As shown, the data of the video frame V1 stored in the first queue 1201 can be controlled to be popped out from the first queue 1201 at the timing T1 of the synchronization signal (1301), but the data of the graphic frame G1 stored in the second queue 1202 and corresponding to the video frame V1 can be controlled to be popped out from the second queue 1202 at the timing T3 delayed by two frames (1302).

[0253] In the same manner, the data controlling the video frame V2 pops up at the T2 timing, while the matching data controlling the graphic frame G2 pops up at the T4 timing delayed by two frames.

[0254] In other words, in Figure 12 In an embodiment, data of one frame in a pair of frames assigned with matching identification information may be controlled to be popped out (eg, out of the pair) with a delay as much as the FRC delay relative to data of the other frame delayed due to FRC.

[0255] Therefore, a time point at which the data of the graphic frame G1 is transmitted to the graphics scaler 162 is later than a time point at which the data of the video frame V1 is transmitted to the video scaler 152 by the FRC delay time.

[0256] In addition, the main processor 180 may control the first processor 150 and the second processor 160 to synchronize and output images of a pair of frames (eg, video frame V1 and graphic frame G1) received in operation 1102 and assigned matching identification information (or sequence numbers) (1104).

[0257] Specifically, in operation 1103, the video frame V1 is scaled in the video scaler 152 based on data (e.g., video geometry information output at timing T1), sent to the frame rate converter 153 to be delayed by two frames during FRC, and thus output to the mixer 170 together with the graphics frame G1, synchronized with the graphics frame G1 output from the graphics scaler 162 based on data (e.g., pointer information output at timing T3).

[0258] In the aforementioned embodiment, one of the video frame and the graphics frame to be displayed together (i.e., the graphics frame) is pre-delayed corresponding to the RFC delay of the video and then output, and thus two images (in other words, the first image (video) and the second image (graphics)) are synchronized and displayed together on the display 110 without the need for a Figure 10 An FRC buffer 903 is shown additionally provided for the graphics frame in memory 184.

[0259] According to embodiments, the methods according to various embodiments of the present disclosure may be provided as involving a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) between two user devices (e.g., smartphones) directly or through an app store (e.g., PlayStore™). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable application) may be temporarily stored or temporarily generated in a machine-readable storage medium (e.g., a memory of a manufacturer's server, an app store server, or a relay server).

[0260] Although several example embodiments of the present disclosure have been described in detail, various changes may be made to these example embodiments without departing from the scope defined in the appended claims.

Claims

1. A display device comprising: monitor; a video processor configured to process and output a plurality of video frames of a video signal and comprising a video scaler; a graphics processor configured to process and output a plurality of graphics frames of a graphics signal; Video scaler; Graphics scaler; frame rate converter; a mixer configured to mix videos corresponding to the plurality of video frames of the video signal processed and output by the video processor and graphics corresponding to the plurality of graphics frames of the graphics signal processed and output by the graphics processor to be displayed together on the display; as well as The main processor is configured to: assigning matching identification information to the plurality of video frames in the order of the corresponding frames, assigning matching identification information to the plurality of graphic frames in the order of the corresponding frames, Setting geometric information for the plurality of video frames and the plurality of graphic frames assigned with the matching identification information, identifying video frames and graphics frames based on the matching identification information assigned to the plurality of video frames and the matching identification information assigned to the plurality of graphics frames, controlling the video processor and the graphics processor to synchronize at a reference time point set based on a predetermined synchronization signal and output the identified video frame and the identified graphics frame, respectively, providing the set geometric information to the video scaler and the graphics scaler so that the video scaler and the graphics scaler scale the identified video frame and the identified graphics frame respectively based on the set geometric information, controlling the frame rate converter to convert the frame rate of the scaled video frames, and Based on a delay time occurring between the frame rate converted video frame and the scaled graphics frame during the frame rate conversion, the video scaler is controlled to output the frame rate converted video frame to the mixer, and the graphics scaler is controlled to delay and output the scaled graphics frame to the mixer, so that the mixer mixes the video of the frame rate converted video frame with the graphics of the scaled graphics frame to display them together on the display.

2. The display device according to claim 1, wherein The main processor is configured to: control the multiple video frames to be sequentially stored in a first queue by assigning the matching identification information to the multiple video frames, control the multiple graphic frames to be sequentially stored in a second queue by assigning the matching identification information to the multiple graphic frames, and control the stored video frames and the stored graphic frames assigned with the matching identification information to be output from the first queue and the second queue respectively at the reference time point set based on the predetermined synchronization signal.

3. The display device according to claim 2, wherein: The main processor is configured to identify the delay time occurring during the conversion of the frame rate, and control the stored graphic frame to be output to be delayed by the identified delay time compared to the stored video frame.

4. The display device according to claim 1, further comprising: A storage unit includes a buffer configured to store the scaled graphic frame based on the delay time occurring during the conversion of the frame rate.

5. The display device according to claim 1, wherein The main processor is configured to render graphics corresponding to the graphic signal by executing an application, set a pointer to output the rendered graphics, and provide information about the set pointer to the graphics scaler.

6. The display device according to claim 1 , further comprising a storage unit including a first buffer configured to store video frames processed by the video processor and a second buffer configured to store graphic frames processed by the graphic processor. in, The main processor is configured to control the video frames stored in the first buffer and the graphic frames stored in the second buffer to be synchronized and output.

7. The display device according to claim 1, wherein The synchronization signal includes a vertical synchronization signal for the display.

8. A method for controlling a display device, the display device comprising a video scaler, a graphics scaler, a frame rate converter, a video processor, and a graphics processor, the video processor being configured to process and output a plurality of video frames of a video signal, and the graphics processor being configured to process and output a plurality of graphics frames of a graphics signal, the method comprising: assigning matching identification information to the plurality of video frames according to an order of corresponding frames; assigning matching identification information to the plurality of graphic frames according to the order of the corresponding frames; Setting geometric information for the plurality of video frames and the plurality of graphic frames assigned with the matching identification information; identifying video frames and graphics frames based on the matching identification information assigned to the plurality of video frames and the matching identification information assigned to the plurality of graphics frames; controlling the video processor and the graphics processor to synchronize at a reference time point set based on a predetermined synchronization signal and output the identified video frame and the identified graphics frame, respectively; providing the set geometric information to the video scaler and the graphics scaler, so that the video scaler and the graphics scaler scale the identified video frame and the identified graphics frame respectively based on the set geometric information; controlling the frame rate converter to convert the frame rate of the scaled video frame; as well as Based on a delay time occurring between the frame rate converted video frame and the scaled graphics frame during the frame rate conversion, the video scaler is controlled to output the frame rate converted video frame to the mixer, and the graphics scaler is controlled to delay and output the scaled graphics frame to the mixer, so that the mixer mixes the video of the frame rate converted video frame with the graphics of the scaled graphics frame to display them together on a display.

9. The method according to claim 8, further comprising: sequentially storing the plurality of video frames in a first queue by assigning the matching identification information to the plurality of video frames; sequentially storing the plurality of graphics frames in a second queue by assigning the matching identification information to the plurality of graphics frames; as well as The stored video frame and the stored graphic frame assigned with the matching identification information are controlled to be output from the first queue and the second queue, respectively, at the reference time point set based on the predetermined synchronization signal.

10. The method according to claim 9, further comprising: identifying the delay time occurring during the conversion of the frame rate; as well as The stored graphics frame is controlled to be outputted to be delayed by the identified delay time compared to the stored video frame.

Images