Display device and its operating method

Abstract

This invention provides a display device and a method for operating the same that simultaneously provides live content video and streamer broadcast video retrieved based on the content video. [Solution] The operation method by the controller of the display device involves searching for one or more streamer broadcasts based on the information of the content video and the display that plays the content video, displaying one or more of the searched streamer broadcasts on the display, and playing the streamer video corresponding to the selected streamer broadcast together with the content video according to a command to select one of the one or more streamer broadcasts.

Description

Technical Field

[0001] The present disclosure relates to a display device, and more particularly, to a display device that provides both content video and a streamer relay video.

Background Art

[0002] Digital TV services using wired or wireless communication networks have become widespread. Digital TV services can provide various services that could not be provided by conventional analog broadcast services.

[0003] For example, in the case of IPTV (Internet Protocol Television), which is a type of digital TV service, and smart TV services, a bidirectionality is provided that enables users to actively select the type of program they watch, viewing time, etc. IPTV and smart TV services can also provide various additional services based on such bidirectionality, such as Internet search, home shopping, online games, etc.

[0004] Recently, it is not just about unilaterally watching or listening to a TV sports relay, but the fun of viewing is enhanced through communication while watching with many fans at the same time.

[0005] Generally, on a PC, a sports relay application and a streaming application are played together for viewing, but there is an inconvenience in that the user has to directly search for and execute each application and directly synchronize the sinks, and particularly, there is a problem that such an operation is impossible on a TV.

Summary of the Invention

[0007] The purpose of this disclosure is to synchronize the content video and the streamer broadcast video.

[0008] The purpose of this disclosure is to balance the audio between the content video and the audio of the streamed video.

[0009] The purpose of this disclosure is to automatically adjust the aspect ratio of each video according to the type of content video or the size of the text included in the streamed video.

[0010] The purpose of this disclosure is to identify objects within the content video by analyzing the audio spoken by the streamer.

[0011] The purpose of this disclosure is to provide summary information, which summarizes the content of the streamer's speech, during the intervals of the competition.

[0012] The purpose of this disclosure is to easily change the relative positions of content video and streamer broadcast video. [Means for solving the problem]

[0013] The display device according to the embodiments of the present disclosure may include a display for playing content video and a controller for searching for one or more streamer broadcasts based on information of the content video, displaying the one or more streamer broadcasts found on the display, and playing a streamer video corresponding to the selected streamer broadcast together with the content video according to a command to select one of the one or more streamer broadcasts.

[0014] The operation method of the display device according to the embodiment of the present disclosure may include the steps of: playing content video; searching for one or more streamer broadcasts based on information of the content video; displaying one or more streamer broadcasts that have been found; and playing a streamer video corresponding to a selected streamer broadcast in accordance with a command to select one of the one or more streamer broadcasts, together with the content video. [Effects of the Invention]

[0015] According to embodiments of this disclosure, a user can select one of the streamer broadcasts relaying the video of the channel they are watching and listen to commentary that suits their preferences. This allows users to not only passively watch or listen to the relay of content video, but also maximize their viewing experience by watching their preferred streamer channel while simultaneously communicating with many other users.

[0016] According to embodiments of this disclosure, the synchronization between content video and streamer-broadcasted video enhances user immersion and facilitates real-time interaction with other users.

[0017] According to embodiments of this disclosure, the balance between the audio of the content video and the audio of the streamed video can be automatically adjusted according to the situation. This can significantly improve the user's viewing experience.

[0018] According to embodiments of this disclosure, the aspect ratio of each video can be automatically adjusted according to the type of content video or the size of the text contained in the streamed video. This allows the user to be provided with an optimal viewing aspect ratio.

[0019] According to an embodiment of the present disclosure, the name of the player spoken by the streamer can be recognized and the corresponding player can be marked in the sports video. As a result, the user can immediately recognize the player mentioned by the streamer on the screen, and an immersive viewing experience can be provided.

[0020] According to an embodiment of the present disclosure, summary information summarizing the speech content of the streamer can be provided in the interval section of the competition. As a result, even if the user cannot listen to the entire speech content of the streamer, information related to the competition can be efficiently provided by the summary information.

[0021] According to an embodiment of the present disclosure, the user can easily change the mutual position between the content video and the streamer relay video. As a result, the user can configure the layout in the desired manner, and the viewing satisfaction can be increased.

Brief Description of Drawings

[0022] [Figure 1] It is a diagram showing the configuration of a display device according to an embodiment of the present invention in a block diagram. [Figure 2] It is a block diagram of a remote control device according to an embodiment of the present invention. [Figure 3] An actual configuration example of a remote control device according to an embodiment of the present invention is shown. [Figure 4] An example of utilizing a remote control device according to an embodiment of the present invention is shown. [Figure 5] It is a flowchart for explaining an operation method of a display device according to an embodiment of the present disclosure. [Figures 6a-6c] It is a diagram for explaining the process of providing a streamer relay video together during viewing of a content video according to an embodiment of the present disclosure. [Figure 7] It is a diagram for explaining a method for synchronizing a content video and a streamer relay video according to an embodiment of the present disclosure. [Figures 8a-8d]FIG. 0 is a diagram for explaining an example of adjusting the timing of content video or stream relay video by a sink button according to an embodiment of the present disclosure. [Figure 9] FIG. 3 is a diagram for explaining an example of adjusting the audio of content video and the audio of stream relay video according to an embodiment of the present disclosure. [Figures 10a-10d] FIG. 6 is a diagram for explaining an example of adaptively switching the audio between content video and stream relay video according to an embodiment of the present disclosure. [Figures 11a-11b] FIG. 9 is a diagram for explaining an example of adjusting the screen size of content video and the screen size of stream relay video according to various embodiments of the present disclosure. [Figures 12a-12d] FIG. 12 is a diagram for explaining an example of adjusting the screen ratio of content video or stream relay video. [Figures 13a-13b] FIG. 15 is a diagram for explaining an example of identifying an object included in content video based on the audio of stream relay video according to an embodiment of the present disclosure. [Figures 14a-14c] FIG. 18 is a diagram for explaining an example of summarizing and providing the speech content of a streamer according to an embodiment of the present disclosure. [Figures 15a-15b] FIG. 21 is a diagram for explaining an example of changing the display position between content video and stream relay video according to an embodiment of the present disclosure. [Figure 16] FIG. 24 is a diagram for explaining an example of providing information regarding an advertised product of a streamer according to an embodiment of the present disclosure.

BEST MODE FOR CARRYING OUT THE INVENTION

[0023] Hereinafter, embodiments according to the present invention will be described in more detail with reference to the drawings. The suffixes "module" and "section" for components used in the following description are given or used only for ease of specification writing, and do not have meanings or roles that are distinguishable from each other.

[0024] An embodiment of the present invention is, for example, an intelligent display device that adds computer support functionality to a broadcast reception function, enhancing the broadcast reception function while also adding internet functionality and other features, and providing a more user-friendly interface such as a handwritten input device, a touchscreen, or a spatial remote control. Furthermore, by supporting wired or wireless internet functionality, it can connect to the internet and a computer, enabling the execution of functions such as email, web browsing, banking, or games. A standardized general-purpose OS may be used for such diverse functions.

[0025] Therefore, the display device described in the present invention can perform a variety of user-friendly functions, for example, by allowing various applications to be freely added or removed on a general-purpose OS kernel. More specifically, the display device may be, for example, a network TV, HBB TV, smart TV, LED TV, OLED TV, etc., and may also be applicable to smartphones.

[0026] Figure 1 is a block diagram showing the configuration of a display device according to one embodiment of the present invention.

[0027] Referring to Figure 1, the display device 100 may include a broadcast receiver 130, an external device interface 135, a memory 140, a user input interface 150, a controller 170, a wireless communication interface 173, a display 180, a speaker 185, and a power supply circuit 190.

[0028] The broadcast receiver 130 may include a tuner 131, a demodulator 132, and a network interface 133.

[0029] The tuner 131 can select a specific broadcast channel in accordance with a channel selection command. The tuner 131 can receive broadcast signals for the selected specific broadcast channel.

[0030] The demodulator 132 can separate the received broadcast signal into a video signal, an audio signal, and a data signal associated with the broadcast program, and can restore the separated video signal, audio signal, and data signal into an output-ready form.

[0031] The external device interface 135 can receive applications or application lists from adjacent external devices and transmit them to the controller 170 or memory 140.

[0032] The external device interface 135 provides a connection path between the display device 100 and the external device. The external device interface 135 can receive one or more video and audio outputs from the external device connected to the display device 100 wirelessly or via a wired connection and transmit them to the controller 170. The external device interface 135 may include multiple external input terminals. The multiple external input terminals may include RGB terminals, one or more HDMI (High Definition Multimedia Interface) terminals, and component terminals.

[0033] Video signals from external devices input via the external device interface 135 are output via the display 180. Audio signals from external devices input via the external device interface 135 can be output via the speaker 185.

[0034] External devices that can be connected to the external device interface 135 may include any one of the following: a set-top box, a Blu-ray (registered trademark; hereinafter the same), a DVD player, a game console, a soundbar, a smartphone, a PC, a USB memory stick, or a home theater system; however, these are merely examples.

[0035] The network interface 133 provides an interface for connecting the display device 100 to a wired / wireless network, including the Internet. The network interface 133 can transmit or receive data with other users or other electronic devices via the connected network or other networks linked to the connected network.

[0036] Furthermore, the display device 100 can transmit some of the content data stored in the display device 100 to other users or other electronic devices that have been pre-registered in the display device 100, or to selected users or selected electronic devices.

[0037] The network interface 133 can connect to a designated web page via the connected network or another network linked to the connected network. That is, it can connect to a designated web page via the network and send or receive data with the corresponding server.

[0038] Furthermore, the network interface 133 can receive content or data provided by content providers or network operators. In other words, the network interface 133 can receive content such as movies, advertisements, games, VOD, and broadcast signals, as well as related information, provided by content providers or network providers via the network.

[0039] Furthermore, the network interface 133 can receive firmware update information and update files provided by the network operator, and can transmit data to the internet, content providers, or network operators.

[0040] The network interface 133 can select and receive desired applications from among those publicly available (open) applications via the network.

[0041] The memory 140 stores programs for each signal processing and control within the controller 170, and can store processed video, audio, or data signals.

[0042] Furthermore, the memory 140 can also function for the temporary storage of video, audio, or data signals input from the external device interface 135 or the network interface 133, and can also store information related to a predetermined image through its channel memory function.

[0043] Memory 140 can store applications or application lists input from the external device interface 135 or the network interface 133.

[0044] The display device 100 can play and provide to the user content files (video files, still image files, music files, document files, application files, etc.) stored in the memory 140.

[0045] The user input interface 150 can transmit signals input by the user to the controller 170, or transmit signals from the controller 170 to the user. For example, the user input interface 150 can receive and process control signals such as power on / off, channel selection, and screen settings from the remote control device 200 using various communication methods such as Bluetooth (registered trademark; hereinafter the same), WB (Ultra Wideband), ZigBee, RF (Radio Frequency) communication, or infrared (IR) communication, or it can process control signals from the controller 170 to transmit them to the remote control device 200.

[0046] Furthermore, the user input interface 150 can transmit control signals input from local keys (not shown), such as the power key, channel key, volume key, and setting key, to the controller 170.

[0047] The video signal processed by the controller 170 can be input to the display 180 and displayed as the corresponding video signal. Furthermore, the video signal processed by the controller 170 can be input to an external output device via the external device interface 135.

[0048] The audio signal processed by the controller 170 can be output to the speaker 185. Alternatively, the audio signal processed by the controller 170 can be input to an external output device via the external device interface 135.

[0049] In addition, the controller 170 can control the overall operation of the display device 100.

[0050] Furthermore, the controller 170 can control the display device 100 by user commands or internal programs input via the user input interface 150, and can connect to a network to allow the user to download desired applications or application lists into the display device 100.

[0051] The controller 170 ensures that the channel information selected by the user, along with the processed video or audio signal, is output via the display 180 or speaker 185.

[0052] Furthermore, the controller 170, in accordance with an external device video playback command received via the user input interface 150, ensures that video signals or audio signals from an external device, such as a camera or video camera, input via the external device interface 135 are output via the display 180 or speaker 185.

[0053] On the other hand, the controller 170 can control the display 180 to display video, for example, broadcast video input via the tuner 131, external input video input via the external device interface 135, video input via the network interface unit, or video stored in the memory 140, so that it is displayed on the display 180. In this case, the video displayed on the display 180 may be a still image or a moving image, and may be 2D video or 3D video.

[0054] Furthermore, the controller 170 can control the playback of content stored in the display device 100, received broadcast content, or external input content input from an external source. The content may take various forms, such as broadcast video, external input video, audio files, still images, connected web pages, and document files.

[0055] The wireless communication interface 173 can communicate with external devices via wired or wireless connection. The wireless communication interface 173 can perform short-range communication with external devices. For this purpose, the wireless communication interface 173 uses Bluetooth. TMThe wireless communication interface 173 can support short-range communication using at least one of the following technologies: RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, or Wireless USB (Wireless Universal Serial Bus). Such a wireless communication interface 173 can support wireless communication between the display device 100 and a wireless communication system, between the display device 100 and other display devices 100, or between the display device 100 and a network where the display device 100 (or an external server) is located, via a Wireless Area Network. The Near Field Network may also be a Wireless Personal Area Network.

[0056] Here, the other display device 100 may be a wearable device (e.g., a smart watch, smart glasses, HMD (head-mounted display)), a mobile terminal such as a smartphone, that can exchange (or interact with) data with the display device 100 according to the present invention. The wireless communication interface 173 can sense (or recognize) a communicable wearable device in the vicinity of the display device 100.

[0057] Furthermore, if the sensed wearable device is authenticated to communicate with the display device 100 according to the present invention, the controller 170 can transmit at least a portion of the data processed in the display device 100 to the wearable device via the wireless communication interface 173. Therefore, the user of the wearable device can utilize the data processed in the display device 100 through the wearable device.

[0058] The display 180 can generate drive signals by converting video signals, data signals, OSD signals processed by the controller 170, or video signals, data signals, etc., received at the external device interface 135 into R, G, and B signals, respectively.

[0059] On the other hand, the display device 100 shown in Figure 1 is merely one embodiment of the present invention, so some of the illustrated components can be integrated, added, or omitted depending on the specifications of the display device 100 that is actually implemented.

[0060] In other words, two or more components may be combined into one component as needed, or one component may be subdivided into two or more components. Furthermore, the functions performed in each block are for the purpose of illustrating embodiments of the present invention, and their specific operations and devices do not limit the scope of the present invention.

[0061] According to yet another embodiment of the present invention, the display device 100 can also receive and play video via a network interface 133 or an external device interface 135 without having a tuner 131 and a demodulator 132, unlike the one shown in Figure 1.

[0062] For example, the display device 100 may be implemented by separating it into a video processing device, such as a set-top box, for receiving broadcast signals or content from various network services, and a content playback device for playing the content input from the video processing device.

[0063] In this case, the operation method of the display device according to the embodiment of the present invention described below may be performed not only by the display device 100 as described with reference to Figure 1, but also by any one of the following: an image processing device such as the separated set-top box or a content playback device equipped with a display 180 and a speaker 185.

[0064] Next, with reference to Figures 2 and 3, a remote control device according to one embodiment of the present invention will be described.

[0065] Figure 2 is a block diagram of a remote control device according to one embodiment of the present invention, and Figure 3 shows an actual configuration example of the remote control device 200 according to one embodiment of the present invention.

[0066] First, referring to Figure 2, the remote control device 200 may include a fingerprint recognition device 210, a wireless communication circuit 220, a user input interface 230, a sensor 240, an output interface 250, a power supply circuit 260, a memory 270, a controller 280, and a microphone 290.

[0067] Referring to Figure 2, the wireless communication circuit 220 transmits and receives signals to any one of the display devices according to the embodiments of the present invention described above.

[0068] The remote control device 200 may include an RF circuit 221 capable of sending and receiving signals with the display device 100 according to the RF communication standard, and an IR circuit 223 capable of sending and receiving signals with the display device 100 according to the IR communication standard. The remote control device 200 may also include a Bluetooth circuit 225 capable of sending and receiving signals with the display device 100 according to the Bluetooth communication standard. The remote control device 200 may also include an NFC (Near Field Communication) circuit 227 capable of sending and receiving signals with the display device 100 according to the NFC communication standard, and a WLAN (Wireless LAN) circuit 229 capable of sending and receiving signals with the display device 100 according to the WLAN (Wireless LAN) communication standard.

[0069] Furthermore, the remote control device 200 transmits a signal containing information about the movement of the remote control device 200 to the display device 100 via the wireless communication circuit 220.

[0070] Meanwhile, the remote control device 200 receives signals transmitted by the display device 100 via the RF circuit 221 and, if necessary, can transmit commands to the display device 100 via the IR circuit 223 regarding power on / off, channel change, volume change, etc.

[0071] The user input interface 230 may consist of a keypad, buttons, a touchpad, or a touchscreen. The user can input commands related to the display device 100 to the remote control device 200 by operating the user input interface 230. If the user input interface 230 includes hard key buttons, the user can input commands related to the display device 100 to the remote control device 200 by pushing the hard key buttons. This will be explained with reference to Figure 3.

[0072] Referring to Figure 3, the remote control device 200 may include multiple buttons. These multiple buttons may include a fingerprint recognition button 212, a power button 231, a home button 232, a live button 233, an external input button 234, a volume control button 235, a voice recognition button 236, a channel change button 237, a confirmation button 238, and a back button 239.

[0073] The fingerprint recognition button 212 may be a button for recognizing the user's fingerprint. In one embodiment, the fingerprint recognition button 212 is capable of being pushed and can also receive push operations and fingerprint recognition operations.

[0074] The power button 231 may also be a button for turning the power of the display device 100 on or off.

[0075] The home button 232 may also be a button for moving to the home screen of the display device 100.

[0076] The live button 233 may be a button for displaying a real-time broadcast program.

[0077] The external input button 234 may also be a button for receiving an external input connected to the display device 100.

[0078] The volume control button 235 may also be a button for adjusting the volume level output by the display device 100.

[0079] The voice recognition button 236 may also be a button for receiving and recognizing the user's voice.

[0080] The channel change button 237 may also be a button for receiving the broadcast signal of a specific broadcast channel.

[0081] The confirmation button 238 may be a button for selecting a specific function, and the back button 239 may be a button for returning to the previous screen.

[0082] Let's explain Figure 2 again.

[0083] If the user input interface 230 includes a touchscreen, the user can input commands related to the display device 100 to the remote control device 200 by touching the soft keys on the touchscreen. Furthermore, the user input interface 230 can be equipped with various types of input means that the user can operate, such as scroll keys and jog keys, and this embodiment does not limit the scope of the present invention.

[0084] The sensor 240 may include a gyro sensor 241 or an accelerometer 243, and the gyro sensor 241 can sense information regarding the movement of the remote control device 200.

[0085] For example, the gyro sensor 241 can sense information about the operation of the remote control device 200 based on the x, y, and z axes, and the acceleration sensor 243 can sense information about the movement speed of the remote control device 200. On the other hand, the remote control device 200 may also be equipped with a distance measuring sensor so that it can sense the distance to the display 180 of the display device 100.

[0086] The output interface 250 can output video or audio signals that correspond to the operation of the user input interface 230 or the signals transmitted from the display device 100.

[0087] The user can use the output interface 250 to recognize whether the user input interface 230 is being operated or whether the display device 100 is being controlled.

[0088] For example, the output interface 250 may include an LED 251 that lights up when the user input interface 230 is operated or when signals are sent and received with the display device 100 via the wireless communication circuit 220, a vibrator 253 that generates vibrations, a speaker 255 that outputs sound, or a display 257 that outputs images.

[0089] Furthermore, the power supply circuit 260 supplies power to the remote control device 200, and if the remote control device 200 does not operate for a predetermined period of time, the power supply is interrupted, thereby reducing power waste.

[0090] The power supply circuit 260 can resume power supply when a predetermined key provided on the remote control device 200 is operated.

[0091] The memory 270 can store various types of programs, application data, etc., necessary for controlling or operating the remote control device 200.

[0092] When the remote control device 200 transmits and receives signals wirelessly with the display device 100 via the RF circuit 221, the remote control device 200 and the display device 100 transmit and receive signals via a predetermined frequency band.

[0093] The controller 280 of the remote control device 200 can store and refer to information in its memory 270, such as the frequency band that can wirelessly transmit and receive signals with the display device 100 paired with the remote control device 200.

[0094] The controller 280 controls various matters related to the control of the remote control device 200. The controller 280 can transmit signals corresponding to predetermined key operations of the user input interface 230 or signals corresponding to the movement of the remote control device 200 sensed by the sensor 240 to the display device 100 via the wireless communication circuit 220.

[0095] Furthermore, the microphone 290 of the remote control device 200 can acquire sound.

[0096] Multiple microphones 290 may be provided.

[0097] Next, we will explain Figure 4.

[0098] Figure 4 shows an example of utilizing a remote control device according to an embodiment of the present invention.

[0099] Figure 4(a) illustrates that a pointer 205 corresponding to the remote control device 200 is displayed on the display 180.

[0100] The user can move or rotate the remote control device 200 up, down, left, and right. The pointer 205 displayed on the display 180 of the display device 100 corresponds to the movement of the remote control device 200. Since the pointer 205 moves and is displayed in accordance with the movement in 3D space as shown in the diagram, such a remote control device 200 may be named a spatial remote control.

[0101] Figure 4(b) illustrates that when the user moves the remote control device 200 to the left, the pointer 205 displayed on the display 180 of the display device 100 also moves to the left in response.

[0102] Information regarding the movement of the remote control device 200, as detected by the sensor of the remote control device 200, is transmitted to the display device 100. The display device 100 can calculate the coordinates of the pointer 205 from the information regarding the movement of the remote control device 200. The display device 100 can then display the pointer 205 in accordance with the calculated coordinates.

[0103] Figure 4(c) illustrates a case where a specific button on the remote control device 200 is pressed, and the user moves the remote control device 200 away from the display 180. This allows the selected area on the display 180 corresponding to the pointer 205 to be zoomed in and enlarged.

[0104] Conversely, if the user moves the remote control device 200 closer to the display 180, the selected area on the display 180 corresponding to the pointer 205 can be zoomed out and displayed in a reduced size.

[0105] On the other hand, if the remote control device 200 moves away from the display 180, the selection area can be zoomed out, and if the remote control device 200 moves closer to the display 180, the selection area can be zoomed in.

[0106] Furthermore, when a specific button in the remote control device 200 is pressed, the recognition of up, down, left, and right movement can be excluded. That is, when the remote control device 200 moves away from or closer to the display 180, up, down, left, and right movements are not recognized, and only forward and backward movements are recognized. When no specific button in the remote control device 200 is pressed, only the pointer 205 moves in accordance with the up, down, left, and right movements of the remote control device 200.

[0107] On the other hand, the movement speed and direction of the pointer 205 can correspond to the movement speed and direction of the remote control device 200.

[0108] On the other hand, in this specification, a pointer refers to an object displayed on the display 180 in response to the operation of the remote control device 200. Therefore, in addition to the arrow shape illustrated in the drawing as the pointer 205, objects of various shapes are possible. For example, it could be a concept that includes a point, cursor, prompt, or thick outline. Furthermore, the pointer 205 can be displayed corresponding to a single point on either the horizontal or vertical axis of the display 180, or it can be displayed corresponding to multiple points such as a line or surface.

[0109] Figure 5 is a flowchart illustrating the operation method of a display device according to one embodiment of the present disclosure.

[0110] In the following, the audio corresponding to the content video may be the audio that is transmitted (or played) together with the content video, and the audio corresponding to the streamer broadcast video may be the audio that is transmitted (or played) together with the streamer broadcast video.

[0111] Audio corresponding to the content video may be called the first audio, and audio corresponding to the streamer broadcast video may be called the second audio.

[0112] Referring to Figure 5, the controller 170 of the display device 100 plays the content video on the display 180 (S501).

[0113] In one embodiment, the genre of the video content may be either sports or games, but this is merely an example.

[0114] The content video may be video received from a broadcasting station server via tuner 131, video received from an external server via network interface 133, video received from an external device via external device interface 135, or video received from an external terminal via wireless communication interface 173.

[0115] The controller 170 searches for one or more streamer broadcasts by the application based on the content video information (S503).

[0116] In one embodiment, the content video information may include metadata for the content video. If the content video is a sports broadcast (or sports video), the metadata for the content video may include at least one of the following: the title of the content video, the channel or broadcast time of the content video, the opposing teams, event information, or the competition schedule.

[0117] The controller 170 can use the content video information to search for one or more streamer broadcasts using either an application installed on the display device 100 or an application installed on a set-top box connected to the display device 100.

[0118] The application may be a streaming application that provides a video streaming service.

[0119] In one embodiment, the controller 170 can search for one or more streamer broadcasts by matching content video information with streamer broadcast information. The streamer broadcast information may include at least one of the streamer broadcast title or broadcast time. For example, the controller 170 can search for one or more streamer broadcasts that match the competing teams and match time by comparing the metadata of the content video with the streamer broadcast information.

[0120] When multiple streamer relay broadcasts are found, the controller 170 can determine a recommendation order based on the reliability of the streamer relaying each streamer relay broadcast, and can list the multiple streamer relay broadcasts according to the recommendation order. The controller 170 may also prioritize listing streamer relay broadcasts with higher streamer reliability.

[0121] The reliability of a streamer may be determined based on at least one of the following: the number of live attendees, the number of subscribers, or the existing viewing history of the channel in question.

[0122] The more people who attend the live stream or the more subscribers there are, the higher the reliability. The reliability also increases if the user has a prior viewing history.

[0123] In another embodiment, the controller 170 can search for streamer broadcasts using information about streamers subscribed to by a user account logged into the streaming application.

[0124] The controller 170 displays one or more found streamer broadcasts on the display 180 (S505).

[0125] The controller 170 can overlay and display one or more searched streamer broadcasts onto the content video.

[0126] The controller 170 displays the streamer broadcast video corresponding to the selected streamer broadcast from among the one or more displayed streamer broadcasts on the display 180 along with the content video (S507).

[0127] The source of the content video and the source of the streamer broadcast video may be the same or different. For example, the sources of the content video and the streamer broadcast video may be set-top boxes connected via the external device interface 135.

[0128] As another example, the source of the content video may be a broadcasting station connected via the tuner 131, and the source of the streamer relay video may be a set-top box connected via the external device interface 135.

[0129] The controller 170 can display not only content video and streamer video, but also a chat window and advertising video on the display 180.

[0130] The controller 170 can receive commands to select two or more of several streamer broadcasts, and according to the received commands, it can display the streamer broadcast video corresponding to the selected streamer broadcasts on the display 180 along with the content video.

[0131] Figures 6a to 6c illustrate the process of providing streamer-broadcast video simultaneously with content video viewing according to embodiments of the present disclosure.

[0132] Referring to Figure 6a, the controller 170 of the display device 100 can play the content video 600 on the display 180. While the content video 600 is playing, the controller 170 can display a pop-up window 610 on the display 180 for searching for streamer broadcasts.

[0133] In one embodiment, the controller 170 can automatically expose the pop-up window 610 if the genre of the content video 600 is sports.

[0134] In another embodiment, the controller 170 can expose the pop-up window 610 by receiving user input to select a specific button provided on the remote control device 200.

[0135] In another embodiment, the controller 170 can expose the pop-up window 610 by receiving an audio command for searching for a streamer broadcast.

[0136] The controller 170 can display the searched multiple streamer broadcasts 621 to 624 on the display 180, as shown in Figure 6b, in accordance with the command to select the pop-up window 610. The controller 170 can also reduce the size of the content video 600 in accordance with the command to select the pop-up window 610, and display the searched multiple streamer broadcasts 621 to 624 on one side of the reduced content video 600.

[0137] Multiple streamer broadcasts 621 to 624 may also be called a streamer channel list.

[0138] The controller 170 can display content video 600 on a first area of ​​the screen and display multiple streamer relay broadcasts 621 to 624 on a second area of ​​the screen.

[0139] The controller 170 can use the information of the content video 600 to search for multiple streamer broadcasts by an application, in accordance with a command to select a popup window 610. The application may be a streaming application that provides a video streaming service. In one embodiment, the controller 170 can search for multiple streamer broadcasts 621 to 624 by matching the information of the content video with the information of the streamer broadcast.

[0140] The controller 170 receives a command to select one of the multiple streamer broadcasts 621 to 624, and according to the received command, can display the streamer broadcast video 631 corresponding to the selected streamer broadcast 622 on the display 180, as shown in Figure 6c.

[0141] For example, if the content video is sports footage, a streamer broadcast (or streamer channel) associated with the sports footage being broadcast can be automatically generated.

[0142] The controller 170 can switch the streamer broadcast list to streamer broadcast video 631 in response to the selection of streamer broadcast 622.

[0143] The streamer relay video 631 may be video that the streamer is relaying the content video 600 to.

[0144] The controller 170 may further display a chat window 632 and an advertisement video 633 in addition to the streamer broadcast video 631. The chat window 632 may include a real-time chat breakdown in the window for sharing reactions to the streamer broadcast in real time. The chat window 632 may further include an emoji window for transmitting emojis and a voice input window for chatting by voice input.

[0145] The advertisement video 633 may show an advertisement provided by the application that provides the streamer broadcast video 631.

[0146] The controller 170 can further display a first volume control bar 641 for adjusting the volume of the audio corresponding to the content video 600 and a second volume control bar 641 for adjusting the volume of the audio corresponding to the streamer relay video 631.

[0147] The user can select which audio they want to emphasize using the first volume control bar 641 and the second volume control bar 641.

[0148] Thus, according to the embodiments of this disclosure, a user can select one of the streamer broadcasts that relay the video of the channel they are watching and listen to commentary that suits their preferences.

[0149] This allows users to not only passively watch or listen to live content broadcasts, but also maximize their viewing experience by watching their preferred streamer channels while simultaneously communicating with numerous other users.

[0150] Figure 7 illustrates a method for syncing content video and streamer broadcast video according to one embodiment of the present disclosure.

[0151] Figure 7 may represent an operation that is included in step S507 of Figure 5, or an operation that takes place after step S507.

[0152] Since content video and streamer broadcast video can be played back via different sources, their synchronization may not always be correct. Therefore, this disclosure presents a method for synchronizing the synchronization between content video and streamer broadcast video.

[0153] Referring to Figure 7, the controller 170 senses the first text contained in the content video being played and stores the first time information indicating the event time (or event progress time) in the memory 140 (S701).

[0154] If the content video is a sports broadcast, the event duration can be the real-time competition duration. If the content video is a game broadcast, the event duration can be the real-time game duration.

[0155] In the case of live sports broadcasts, a number indicating the current time elapsed in the competition may be displayed in the upper left or upper right corner of the screen.

[0156] Controller 170 can detect a first text from the content video using optical character recognition (OCR) and extract numbers from the detected text. If the extracted numbers change over time, Controller 170 can determine the corresponding numbers as the event progress time.

[0157] The controller 170 detects the second text contained in the streamer broadcast video being played back and stores the second time information indicating the event time (or event progress time) in the memory 140 (S703).

[0158] Controller 170 can detect a second text from the streamer video using optical character recognition (OCR) and extract numbers from the detected second text. If the extracted numbers change over time, Controller 170 can determine the corresponding numbers as the event progress time.

[0159] The controller 170 checks the delay time between the content video and the streamer relay video based on the stored information for the first and second time periods (S705).

[0160] The controller 170 can obtain the difference between the first time information and the second time information as the delay time.

[0161] If the controller 170 determines that there is a playback delay between videos based on the confirmed delay time (S707), it stores the video to be played first in the buffer of the memory 140, and then plays the stored video with a delay equal to the confirmed delay time (S709).

[0162] For example, if the controller 170 has a delay of 2 seconds and the content video is played first, it can store the content video in a buffer and play the stored content video 2 seconds later to synchronize it with the streamer broadcast video.

[0163] The controller 170 can periodically (for example, every N minutes) check the delay time between the content video and the streamer relay video and readjust the playback timing of the video output. To this end, after performing step S709, the controller 170 can perform step S701 and subsequent steps again.

[0164] The controller 170 can check the delay time if buffering occurs in at least one of the content video and streamer relay video, or if the reception quality of at least one video changes, and adjust the synchronization between the content video and the streamer relay video based on the checked delay time. A change in the reception quality of the video may occur when adaptive streaming occurs, in which the image quality or transmission speed of the streamer relay video is adjusted according to the network conditions.

[0165] Thus, according to the embodiments of this disclosure, the synchronization between the content video and the streamer broadcast video enhances the user's viewing immersion and facilitates real-time interaction with other users.

[0166] In another embodiment, if the controller 170 is unable to extract time information from text contained in either the content video or the streamer broadcast video, it can synchronize the content video and the streamer broadcast video based on a first peak point of the first audio played with the content video and a second peak point of the second audio played with the streamer broadcast video. For example, if the content video is a soccer match video, loud cheers from the crowd may commonly occur during scoring moments.

[0167] The controller 170 compares the peak point of the first audio output from the scoring scene of the soccer match video with the peak point of the second audio output from the streamer broadcast video, and can determine the delay time between the soccer match video and the streamer broadcast video based on the comparison result. After storing the video that will be played first based on the determined delay time in a buffer, the controller 170 can play the stored video with a delay of the amount of the delay time.

[0168] In another embodiment, the controller 170 can match the scene analysis results of the content video with the audio corresponding to the streamer broadcast video to confirm the delay time. For example, the controller 170 can identify scoring scenes in soccer match video through video analysis, and can identify scoring scenes by analyzing the streamer's audio included in the audio of the streamer broadcast video.

[0169] Controller 170 can compare the timing of the goal in the soccer match video with the timing of the goal as heard in the streamer's audio to check the delay time, and can synchronize the two video streams based on the checked delay time.

[0170] In another embodiment, if the controller 170 is unable to extract time information from the text contained in either the content video or the streamer broadcast video, it can match the scene analysis results of the content video with the audio of the streamer broadcast video to determine the delay time.

[0171] Figures 8a to 8d illustrate an example of adjusting the timing of content video or streamer relay video using a sync button according to an embodiment of the present disclosure.

[0172] Referring to Figure 8a, the display device 100 can play content video 810 and streamer relay video 820 on the display 180.

[0173] The display device 100 can display a sync notification pop-up window 800 on the display 180 to inform the user that the sync of the two videos will be automatically synchronized if there is a delay between the content video 810 and the streamer relay video 820. The sync notification pop-up window 800 may include the delay confirmed in step S705 of Figure 7.

[0174] Referring to Figure 8b, the display device 100 can display a sync button for precisely adjusting the sync between two video streams when the pointer 205 is positioned at the lower left or lower right of the content video stream 810 or the streamer relay video stream 820.

[0175] The display device 100 can display a first sync button 801 and a second sync button 802 on the display 180, which can adjust the playback timing of the content video 810.

[0176] The first sync button 801 may be a button for adjusting the playback of the content video 810 to be slower for a certain period of time, and the second sync button 802 may be a button for adjusting the playback of the content video 810 to be faster for a certain period of time. The first sync button 801 and the second sync button 802 may be displayed superimposed on the content video 810, but they may also be displayed below the content video 810, and the display position is not limited to this.

[0177] The display device 100 can display a third sync button 803 and a fourth sync button 804 on the display 180, which can adjust the playback timing of the streamer relay video 820. The third sync button 803 may be a button for adjusting the playback of the streamer relay video 820 to be slower for a certain period of time, and the fourth sync button 804 may be a button for adjusting the playback of the streamer relay video 820 to be faster for a certain period of time. The third sync button 803 and the fourth sync button 804 may be displayed superimposed on the streamer relay video 820, but the display position is not limited to this.

[0178] The user can manually adjust the playback timing of each video by clicking the sync button, using the pointer 205 which moves with the movement of the remote control device 200.

[0179] When the output of the content video 810 is adjusted by selecting the first sync button 801, the display device 100 can display a delayed pop-up window 830 on the display 180, as shown in Figure 8c, indicating that the content video 810 is being played back with a 2-second delay.

[0180] If the display device 100 is unable to store frames of the content video 810 due to insufficient memory capacity 140, it may display a non-delay pop-up window 840 on the display 180, as shown in Figure 8d, indicating that the content video 810 cannot be played back any slower even if the user clicks the sync button.

[0181] According to the embodiments of this disclosure, sync adjustment in seconds may be performed by clicking a sync button, as shown in Figure 8a, and sync adjustment in units of less than seconds may be performed according to the embodiment in Figure 7.

[0182] Figure 9 illustrates an example of adjusting the audio of content video and the audio of streamer-broadcast video according to one embodiment of the present disclosure.

[0183] Figure 9 shows an embodiment that takes place from step S507 onwards in Figure 5.

[0184] Let's assume the content video is footage of a soccer match.

[0185] Referring to Figure 9, the controller 170 separates the first audio corresponding to the content video into multiple audio channels (S901).

[0186] The first audio can be audio output along with the content video. The controller 170 can separate the first audio into multiple audio channels depending on the type of sound. For example, the multiple audio channels may include commentary from the content video, player voices from the players, coach voices from the coach, and audience voices from the spectators.

[0187] In one embodiment, the controller 170 can separate the first audio into multiple audio channels using multiple audio sources that provide the first audio.

[0188] In another embodiment, the controller 170 can separate the first audio into multiple audio channels by utilizing the positions of the sounds that constitute the first audio.

[0189] Furthermore, the controller 170 can separate the second audio corresponding to the streamer broadcast video into multiple audio channels according to the type of sound.

[0190] The controller 170 determines whether an event has occurred based on the video content being played (S903).

[0191] In one embodiment, the event may be any one of the following: a goal in a soccer match video, playback of a highlight video, halftime, player substitution, tactical timeout, advertisement playback, or change of possession.

[0192] In one embodiment, the controller 170 can identify objects included in the content video using an object sensing model, and can recognize the actions of the identified objects using an action recognition model.

[0193] The object detection model may be either a YOLO (You Only Look Once) model, which can predict the objects present in an image and their locations by viewing the image only once, or a Faster R-CNN (Regions with Convolutional Neural Networks) model, which extracts feature maps from the video and identifies regions of interest based on the extracted feature maps.

[0194] The behavior recognition model may also be a 3D CNN (3D Convolutional Neural Network) model that recognizes object behavior by simultaneously processing spatial and temporal information using a 3D filter on video and the video's temporal dimension.

[0195] The controller 170 can determine whether or not an event has occurred by using the behavior of an object inferred by the behavior recognition model.

[0196] In another embodiment, the controller 170 can convert audio corresponding to the content video or audio corresponding to the streamer relay video into text and determine whether or not an event has occurred based on the converted text.

[0197] If the controller 170 determines that an event has occurred, it adjusts the volume of each audio channel of the first audio and the volume of the second audio for the streamer relay video according to the characteristics of the event (S905).

[0198] For example, when an event such as a score situation occurs, controller 170 can decrease the volume of the broadcast audio for the content video and the second audio for the streamer broadcast video, and increase the volume of the audience audio and player audio for the content video.

[0199] As another example, when an event such as a tactical timeout occurs, the controller 170 can decrease the volume of the audience audio in the content video and increase the volume of the coach's and players' audio. The controller 170 can control the output of the speaker 185 to adjust the volume of each audio.

[0200] Thus, according to the embodiments of this disclosure, the sense of presence can be enhanced by adjusting the volume of the separated audio channels to be optimized according to the circumstances of the event.

[0201] If the controller 170 determines that no event has occurred, it outputs a first audio track corresponding to the content video and a second audio track for the streamer relay video (S907).

[0202] Figures 10a to 10d illustrate an example of adaptively switching audio between content video and streamer-relayed video according to an embodiment of the present disclosure.

[0203] Referring to Figure 10a, the display device 100 can display content video 1010, advertising video 1020, first streamer relay video 1030, and second streamer relay video 1040 on the display 180.

[0204] Initially, the volume of the relay audio included in the audio of the content video 1010 may be reduced, and the volume of the relay audio included in the audio of the streamer relay video selected by the user may be set to sound louder.

[0205] When the pointer 205 is positioned over the first streamer relay video 1030 (i.e., hovering over it), the display device 100 can increase the volume of the relay audio for the first streamer relay video 1030 and decrease the volume of the relay audio for the content video 1010 and the relay audio for the second streamer relay video 1040, respectively.

[0206] When the display device 100 receives a command to select the first streamer relay video 1030, it can mute the relay audio of the content video 1010 and the relay audio of the second streamer relay video 1040, respectively. Furthermore, when the display device 100 receives a command to select the first streamer relay video 1030, it can reduce the screen size of the content video 1010 and the second streamer relay video 1040, and increase the screen size of the selected first streamer relay video 1030. Additionally, when the display device 100 receives a command to select the first streamer relay video 1030, it can display subtitles 1041 on the second streamer relay video 1040 that match the audio of the second streamer relay video.

[0207] Referring to Figure 10b, when the pointer 205 is positioned over the second streamer relay video 1040 in the state shown in Figure 10a, the display device 100 can increase the volume of the relay audio for the second streamer relay video 1040 and decrease the volumes of the relay audio for the content video 1010 and the relay audio for the first streamer relay video 1030, respectively.

[0208] Subsequently, when the display device 100 receives a command to select the second streamer relay video 1040, it can mute the relay audio of the content video 1010 and the relay audio of the first streamer relay video 1030, respectively. Furthermore, when the display device 100 receives a command to select the second streamer relay video 1040, it can reduce the screen size of the content video 1010 and the first streamer relay video 1030, and increase the screen size of the selected second streamer relay video 1040. Also, when the display device 100 receives a command to select the second streamer relay video 1040, it can display subtitles 1031 on the first streamer relay video 1030 that match the audio of the first streamer relay video, as shown in Figure 10c.

[0209] If the audio corresponding to the second streamer relay video 1040, which is set as the main channel, is not output for a certain period of time, and the audio corresponding to the first streamer relay video 1030 is being output, the display device 100 may display a switching pop-up window 1050, as shown in Figure 10c, on the display 180. The switching pop-up window 1050 may be a window that notifies the user to automatically increase the audio of the other streamer relay video and switch to the main channel.

[0210] When the display device 100 accepts a switch to another streamer relay video via the switching pop-up window 1050, it can mute the relay audio of the content video 1010 and the relay audio of the second streamer relay video 1040, respectively, and output the relay audio of the first streamer relay video 1030, as shown in Figure 10a.

[0211] Referring to Figure 10d, the display device 100 can display content video 1010, advertising video 1020, first streamer relay video 1030, and second streamer relay video 1040 on the display 180.

[0212] The display device 100 can decrease or increase the volume of the audio corresponding to the first streamer relay video 1030 when the pointer 205 is positioned over the first streamer relay video 1030 and the remote control device 200 receives a signal from the remote control device 200 to select either the left or right button after selecting a volume button on the remote control device 200. The volume change of the audio corresponding to the first streamer relay video 1030 can be displayed by the volume adjustment bar 1060.

[0213] Thus, according to the embodiments of this disclosure, the balance between the audio of the content video and the audio of the streamed video can be automatically adjusted according to the situation. This can significantly improve the user's viewing experience.

[0214] Figures 11a and 11b illustrate examples of adjusting the screen size of content video and streamer video according to various embodiments of the present disclosure.

[0215] Referring to Figure 11a, the display device 100 can display content video 1110, advertising video 1120, chat window 1130, and streamer broadcast video 1140 on the display 180.

[0216] The display device 100 can enter a screen aspect ratio adjustment mode to change the screen aspect ratio according to a command to select a menu for adjusting the screen size or a command to select a hotkey provided on the remote control device 200. Upon entering the screen aspect ratio adjustment mode, the display device 100 can display a split bar handler 1151, a left indicator 1152, and a right indicator 1153 on the display 180.

[0217] The split bar handler 1151 may also be a handler for adjusting the size of each UI element on the screen. The content video 1110, the advertisement video 1120, the chat window 1130, and the streamer broadcast video 1140 may each be a single UI element. The split bar handler 1151 can identify the boundaries of UI elements. In Figure 11a, only one split bar handler 1151 is shown, but it is not limited to this and may be provided at the boundaries between UI elements.

[0218] The left indicator 1152 is an indicator that signals the split bar handler 1151 to move to the left, and the right indicator 1153 may be an indicator that signals the split bar handler 1151 to move to the right.

[0219] The display device 100 can move the split bar handler 1151 to the left or right side by receiving a command from the remote control device 200 to select either the left or right button. This allows the size of each UI element to be reduced or increased.

[0220] In another embodiment, the display device 100 can move the split bar handler 1151 to the left or right by receiving a command from the remote control device 200 to move the pointer 205 located on the split bar handler 1151 to the left or right. This allows the size of each UI element to be reduced or increased.

[0221] For example, if the split bar handler 1151 moves to the left, the size of the content video 1110 decreases, while the sizes of the advertisement video 1120, the chat window 1130, and the streamer broadcast video 1140 increase.

[0222] As another example, if the split bar handler 1151 moves to the left, the size of the content video 1110 decreases, the sizes of the advertisement video 1120, the chat window 1130, and the streamer broadcast video 1140 increase, and the layout of the advertisement video 1120 and the chat window 1130 may change as shown in Figure 11b.

[0223] Figures 12a to 12d illustrate examples of adjusting the aspect ratio of content video or streamer broadcast video.

[0224] Figures 12a and 12b illustrate examples of adjusting the aspect ratio of content video depending on the sport of sports.

[0225] Referring to Figure 12a, the display device 100 can display sports video 1210, advertising video 1220, chat window 1230, and streamer broadcast video 1240 on the display 180. The sports video 1210 may be content video representing a football sport.

[0226] In one embodiment, the display device 100 can calculate the minimum aspect ratio of the sports video 1210 based on the sport of the sports video 1210. The display device 100 may store multiple minimum aspect ratios corresponding to multiple sports in the memory 140. The minimum aspect ratio may be the minimum ratio of the horizontal length to the vertical length of the sports video 1210. The display device 100 can identify balls included in the sports video 1210 using an object sensing model and recognize the sport based on the identified balls.

[0227] In another embodiment, the display device 100 can calculate the minimum screen aspect ratio of the sports video 1210 based on the size of the ball included in the sports video 1210.

[0228] The display device 100 can enlarge the aspect ratio of the sports video 1210 by a pre-set minimum aspect ratio if the ratio between the size of the ball and the size of the sports video 1210 is less than a certain ratio. The display device 100 can display an auto-adjustment pop-up window 1250 on the display 180 indicating that the aspect ratio has been adjusted for optimal viewing of the sports video 1210.

[0229] The display device 100 can enter a screen aspect ratio adjustment mode in accordance with a command to select a menu for adjusting the screen size or a command to select a hotkey provided on the remote control device 200 to change the screen aspect ratio. Upon entering the screen aspect ratio adjustment mode, the display device 100 can display a split bar handler 1151, a left indicator 1152, and a right indicator 1153 on the display 180.

[0230] When the display device 100 receives a command from the remote control device 200 to select the left button, it can display an adjustment limit pop-up window 1260 on the display 180 indicating that the adjustment of the aspect ratio is limited for optimal viewing. In addition, the display device 100 can deactivate the left indicator 1152 corresponding to the left button when the aspect ratio of the sports video 1210 is at the minimum aspect ratio.

[0231] Subsequently, the display device 100 continuously receives commands from the remote control device 200 to select the right button, and when the aspect ratio of the sports video 1210 reaches the maximum aspect ratio, it can display an adjustment limit pop-up window 1260 on the display 180, as shown in Figure 12b, indicating that the adjustment of the aspect ratio is limited for optimal viewing. In addition, when the aspect ratio of the sports video 1210 is at the maximum aspect ratio, the display device 100 can deactivate the right indicator 1153 corresponding to the right button.

[0232] Next, we will explain Figures 12c and 12d.

[0233] Figures 12c and 12d illustrate an example of adjusting the aspect ratio of a streamer broadcast video according to the size of the text contained in the streamer broadcast video.

[0234] In one embodiment, the display device 100 can determine the minimum aspect ratio of the streamer video 1240 based on the size of the text contained in the streamer video 1240. The display device 100 may store multiple minimum aspect ratios corresponding to each of multiple text sizes in the memory 140. The minimum aspect ratio may also be the minimum ratio of the width to the height of the streamer video 1240. The display device 100 can recognize the text contained in the streamer video 1240 by optical character recognition and calculate the size of the recognized text.

[0235] The display device 100 can enlarge the aspect ratio of the streamer video 1240, which matches the calculated text size, by the amount of a previously set minimum aspect ratio. The display device 100 can display an auto-adjustment pop-up window 1250 on the display 180 indicating that the aspect ratio has been adjusted for optimal viewing of the streamer video 1240.

[0236] The display device 100 can enter a screen aspect ratio adjustment mode in accordance with a command to select a menu for adjusting the screen size or a command to select a hotkey provided on the remote control device 200 to change the screen aspect ratio. Upon entering the screen aspect ratio adjustment mode, the display device 100 can display a split bar handler 1151, a left indicator 1152, and a right indicator 1153 on the display 180.

[0237] When the display device 100 receives a command from the remote control device 200 to select the right button, it can display an adjustment limit pop-up window 1260 on the display 180 indicating that the adjustment of the aspect ratio is limited for optimal viewing. In addition, if the aspect ratio of the streamer relay video 1240 is at the minimum aspect ratio, the display device 100 can deactivate the right indicator 1153 corresponding to the right button.

[0238] Subsequently, the display device 100 continuously receives commands from the remote control device 200 to select the left button, and when the aspect ratio of the streamer relay video 1240 reaches the maximum aspect ratio, it can display an adjustment limit pop-up window 1260 on the display 180, as shown in Figure 12d, indicating that the adjustment of the aspect ratio is limited for optimal viewing. In addition, when the aspect ratio of the streamer relay video 1240 is at the maximum aspect ratio, the display device 100 can deactivate the left indicator 1152 corresponding to the left button.

[0239] Thus, according to the embodiments of this disclosure, the aspect ratio of each video can be automatically adjusted according to the type of content video or the size of the text included in the streamed video. This allows the user to be provided with an optimal viewing aspect ratio.

[0240] Figures 13a and 13b illustrate an example of identifying objects contained in content video based on audio from streamer-broadcast video according to an embodiment of the present disclosure.

[0241] Referring to Figure 13a, the controller 170 extracts a specific player based on the audio of the streamer relay video while playing the content video and the streamer relay video (S1301).

[0242] In one embodiment, the controller 170 can convert audio corresponding to the streamer broadcast video into text using an STT (Speech To Text) engine, and obtain the analysis results of the converted text using an NLP (Natural Language Processing) engine. The controller 170 can identify the player's name from the text and obtain the search results for the identified name as the analysis result.

[0243] The analysis results may include at least one of the following: the name of a specific player, the jersey number of a specific player, or the facial image of a specific player. Extracting a specific player may involve extracting at least one of the following: the name of a specific player, the jersey number of a specific player, or the facial image of a specific player.

[0244] The controller 170 marks a specific player extracted from among multiple players included in the content video (S1303).

[0245] The controller 170 can identify multiple players from the content video using an object-sensing model. The controller 170 can identify each player by recognizing their face, name, or jersey number using the object-sensing model.

[0246] In one embodiment, the controller 170 can identify multiple players within the content video after a specific player has been extracted based on the audio of the streamer-relayed video.

[0247] In another embodiment, the controller 170 can identify multiple players within the content video, regardless of whether a specific player is extracted based on the audio of the streamer-broadcast video.

[0248] As shown in Figure 13b, the controller 170 can display an indicator 1320 on one side of the player that can identify a specific player extracted from among multiple players included in the content video.

[0249] As shown in Figure 13c, the controller 170 can display the extracted information 1310 about a specific player together with the sports video 1210 and the streamer broadcast video 1240. The information 1310 about a specific player can be displayed in an empty area of ​​the screen.

[0250] Information about a specific player 1310 may include one or more of the player's real-time records, the player's name, the player's jersey number, the player's age, and the player's position.

[0251] The controller 170, upon receiving a command to select the indicator 1320 in Figure 13b, can display information 1310 related to a specific player on one side of the player or on an empty area of ​​the screen.

[0252] Thus, according to the embodiments of this disclosure, the streamer can recognize the name of the player it speaks to and mark that player within the sports video. This allows the user to immediately recognize the player mentioned by the streamer on the screen, providing an immersive viewing experience.

[0253] Figures 14a to 14c illustrate an example of summarizing and providing the content of a streamer's speech according to an embodiment of the present disclosure.

[0254] Referring to Figure 14a, the controller 170 collects the content of the streamer's speech from the audio corresponding to the streamer's relayed video, and can obtain a summary from the content of the streamer's speech using a Large Language Model (LLM) (S1401).

[0255] In one embodiment, the controller 170 can continuously collect the streamer's audio from the audio corresponding to the streamer's broadcast video. The controller 170 can convert the collected audio into text using an STT engine and acquire the converted text as the spoken content.

[0256] The controller 170 can obtain a summary from the utterance content acquired using the LLM stored in memory 140. The LLM is an artificial intelligence-based model trained via a deep learning algorithm, and may be a model that understands the linguistic context from text prompts and outputs a response according to the result of that understanding.

[0257] The controller 170 determines the interval section based on the audio of the streamer relay video (S1403) and outputs the summary content acquired in the determined interval section (S1405).

[0258] The controller 170 can convert the streamer's audio contained in the streamer broadcast video into text and determine the interval section based on the text analysis results. The interval section may be any one of the following: the player changeover section, the section between the first and second halves of the match, or the highlight broadcast section.

[0259] Referring to Figure 14b, the display device 100 can display sports video 1410, advertising video 1420, chat window 1430, and streamer broadcast video 1440 on the display 180. The display device 100 can generate summary information from the streamer's speech content using LLM. The display device 100 can generate summary information from the streamer's speech content using LLM at regular time intervals. The display device 100 can determine the interval section of the sports based on the streamer's audio.

[0260] The display device 100 can display summary information 1450 generated during the determined interval on the display 180.

[0261] The display device 100 can display a summary time setting menu 1460 on the display 180 in accordance with a command to select a summary content 1450. The summary time setting menu 1460 may be a menu for setting the summary time unit for the summary content. For example, if 5 minutes is selected on the summary time setting menu 1460, the display device 100 can output a summary content in which the streamer's speech content is summarized to 5 minutes.

[0262] As another example, the summary time setting menu 1460 has different attributes depending on the sport of the sports video 1410. For example, if the sport is baseball, the summary time setting menu 1460 may include multiple inning units, and if the sport is boxing, the summary time setting menu 1460 may include multiple round units.

[0263] After the summary content 1450 is selected, the display device 100 can display subtitles summarizing the streamer's speech from the present moment up to 5 minutes prior, if it receives a command to select the left button on the remote control device 200.

[0264] Thus, according to the embodiments of this disclosure, summary information, which summarizes the content of the streamer's speech, can be provided during the intervals of the competition. This allows users to efficiently receive information about the competition from the summary information, even if they cannot hear the entire content of the streamer's speech.

[0265] Figures 15a and 15b illustrate an example of changing the display position between content video and streamer relay video according to an embodiment of the present disclosure.

[0266] Referring to Figure 15a, the display device 100 can display sports video 1510, advertising video 1520, chat window 1530, and streamer broadcast video 1540 on the display 180.

[0267] The display device 100 can display a switch button 1550 on the display 180 if it determines that the pointer 205 is positioned between the sports video 1510 and the streamer broadcast video 1540, or if a specific button on the remote control device 200 is selected. The switch button 1550 may be a button for swapping positions between the sports video 1510 and the streamer broadcast video 1540. The display device 100 can swap the positions of the sports video 1510 and the streamer broadcast video 1540 in response to a command to select the switch button 1550.

[0268] In this state, if the display device 100 receives a command to re-select the switch button 1550, it can swap the positions of the sports video 1510 and the streamer broadcast video 1540 again.

[0269] The display device 100 can display a volume adjustment pop-up window 1560 on the display 180 for adjusting the volume of the audio corresponding to each video by changing the position between the two videos. When the display device 100 receives a command to accept volume adjustment via the volume adjustment pop-up window 1560, it can increase the volume of the audio corresponding to the streamer video 1540 and decrease the volume of the audio corresponding to the sports video 1510.

[0270] Thus, according to the embodiments of this disclosure, users can easily change the relative positions of content video and streamer broadcast video. This allows users to configure the layout in their preferred manner, thereby increasing viewing satisfaction.

[0271] Referring to Figure 15b, the display device 100 can sense events such as halftime, tactical time, or advertising time based on the sports video 1510 or the audio corresponding to the sports video 1510.

[0272] When an event is detected, the display device 100 can display an automatic switching pop-up window 1570 on the display 180 to enlarge the streamer broadcast video 1540. When the display device 100 receives a command to accept automatic switching via the automatic switching pop-up window 1570, it can exchange the positions of the sports video 1510 and the streamer broadcast video 1540.

[0273] After the event ends, the display device 100 can swap the positions of the sports video 1510 and the streamer broadcast video 1540. Simultaneously with or after the position swap, the display device 100 can display a swap pop-up window 1580 on the display 180 indicating that it has returned to the previous screen.

[0274] Thus, according to the embodiments of this disclosure, when an event such as advertisement playback occurs, the relative positions of the content video and the streamer broadcast video can be automatically changed. This ensures that the user's viewing flow is not interrupted and reduces the burden of additional operations.

[0275] Figure 16 illustrates an example of providing information about a streamer advertising product according to one embodiment of the present disclosure.

[0276] Referring to Figure 16, sports video 1610, advertising video 1620, chat window 1630, and streamer broadcast video 1640 can be displayed on the display 180.

[0277] In one embodiment, the display device 100 can acquire advertising product information based on advertising information contained in the streamer broadcast video 1640. The advertising product information may include either a link or a QR code (registered trademark) that provides access to information about the advertising product (such as the product name, product manufacturer, and product price). The display device 100 can extract advertising products from the streamer broadcast video 1640 using an object detection model and retrieve information about the extracted advertising products.

[0278] In another embodiment, the display device 100 can acquire advertising product information based on the metadata of the streamer broadcast video 1640. The metadata of the streamer broadcast video 1640 may include one or more of the following: streamer-related information, the broadcast title set by the streamer, and indirect advertising information (Product Placement, PPL). When the display device 100 receives the streamer broadcast video 1640, it can also receive the metadata of the streamer broadcast video 1640. The display device 100 can parse the metadata of the streamer broadcast video 1640 to acquire indirect advertising information. The indirect advertising information may include information about the advertising products promoted by the streamer. The display device 100 can use the acquired information about advertising products to search for advertising product information and acquire advertising product information based on the search results.

[0279] As another example, the advertising product information in the streamer broadcast video 1640 may include one or more links or QR codes that provide access to information about the advertised product (such as the product name, manufacturer, and price).

[0280] The display device 100 can display advertising product items 1651 and 1652 promoted by the streamer on the display 180 when a sports event corresponding to the sports video 1610 occurs, when a specific button on the remote control device 200 is selected, or when the streamer broadcast video 1640 is selected. Each advertising product item may include a link providing information about the advertising product.

[0281] When an advertising product item 1651 is selected, the display device 100 can display a QR code 1653 on the display 180 for accessing a website that provides information about the relevant advertising product.

[0282] Users can scan a QR code with their user terminal 1600 to access a website that provides information about advertised products.

[0283] Thus, according to the embodiments of the present disclosure, by providing access to advertising products promoted by the streamer during viewing of the content video and the streamer relay video, the user can easily obtain information about the relevant advertising products.

[0284] The display device 100 according to the embodiments of the present disclosure may include a display 180 that plays a content video, and a controller 170 that searches for one or more streamer relay broadcasts based on the information of the content video, displays the one or more searched streamer relay broadcasts on the display, and plays a streamer relay video corresponding to the selected streamer relay broadcast together with the content video according to an instruction to select any one of the one or more streamer relay broadcasts.

[0285] The controller 170 obtains first time information indicating the competition progress time based on the first text included in the content video, obtains second time information indicating the competition progress time based on the second text included in the streamer relay video, obtains the difference between the first and second time information as the delay time, and can align the sync between the content video and the streamer relay video based on the delay time.

[0286] The controller 170 can temporarily store the video that is being played first among the content video and the streamer relay video in the buffer based on the first and second time information, and delay the output of the video that is played first by the delay time. [[ID=??]]

[0287] When buffering occurs in at least one of the content video and the streamer relay video, or when the reception quality of at least one of the videos is changed, the controller 170 can align the sync between the content video and the streamer relay video based on the delay time.

[0288] The controller 170 can match the scene analysis results of the content video with the audio corresponding to the streamer relay video to synchronize the synchronization between the content video and the streamer relay video.

[0289] The controller 170 can display one or more sync buttons on the display for syncing the content video and the streamer relay video.

[0290] The information of the content video includes metadata such as the competing teams and the competition schedule. The controller 170 compares the metadata with the information of the streamer broadcast being transmitted by the streaming application to search for one or more streamer broadcasts. The information of the streamer broadcast may include the title and broadcast time.

[0291] When multiple streamer broadcasts are found, the controller 170 determines a recommendation order for the multiple streamer broadcasts based on the reliability of the streamers, lists the multiple streamer broadcasts according to the determined recommendation order, and the reliability can be determined by one or more of the following: the number of live participants, the number of subscribers, or the existing viewing history of the channel in question.

[0292] The controller 170 can further display a first volume adjustment bar on the display for adjusting the volume of the first audio corresponding to the content video and a second volume adjustment bar for adjusting the volume of the second audio corresponding to the streamer relay video.

[0293] The controller 170 separates the first audio corresponding to the content video into multiple audio channels, and when it is determined that an event has occurred based on the content video, it can adjust the volume of each of the multiple audio channels and the volume of the second audio corresponding to the streamer relay video according to the characteristics of the event.

[0294] The multiple audio channels include commentary from the content video, player voices from the players, coach voices from the coach, and spectator voices from the audience. The event indicates the score, and the controller 170 can decrease the volume of the commentary from the content video and the second audio from the streamer video, while increasing the volume of the spectator voices and player voices in the content video.

[0295] The controller 170 can adjust the size of the content video and the streamer relay video in accordance with commands received from the remote control device.

[0296] The controller 170 can determine the minimum aspect ratio of the content video according to the sport indicated by the content video, and can play the content video at the determined minimum aspect ratio.

[0297] The controller 170 can determine the minimum aspect ratio of the streamer video based on the size of the text included in the streamer video, and can play the streamer video at the determined minimum aspect ratio.

[0298] The controller 170 can extract objects from the audio corresponding to the streamer relay video and display an indicator on the display that identifies the objects extracted from the content video.

[0299] The controller 170 can display information about the object on the display in response to a command to select the indicator.

[0300] The controller 170 collects the speech content of the streamer from the audio corresponding to the streamer relay video, obtains a summary from the speech content using a large language model, and can display the summary on the display during the interval.

[0301] The controller 170 further displays a switch button on the display, and can exchange the display positions between the content video and the streamer relay video in response to an instruction to select the switch button.

[0302] The controller 170 can further display a chat window for sharing reactions to the streamer relay on the display.

[0303] According to an embodiment of the present disclosure, the above-described method can be realized as processor-readable code on a medium on which a program is recorded. Examples of processor-readable media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage devices, and the like.

[0304] The display device described as above is not limited to the configuration and method of the described embodiment, and the embodiment can also be configured by selectively combining all or part of each embodiment so that various modifications can be made.

[0305] 〔One aspect of the present invention〕 〔Claim 1〕 A display device, comprising: A display for playing a content video; and A controller; The controller is configured to: Search for one or more streamer relay broadcasts based on the information of the content video, Display the one or more searched streamer relay broadcasts on the display, Play a streamer relay video corresponding to the selected streamer relay broadcast together with the content video according to an instruction to select any one of the one or more streamer relay broadcasts. 〔Claim 2〕 The controller is configured to: Based on the first text contained in the aforementioned content video, first hour information indicating the progress of the competition is obtained, Based on the second text contained in the aforementioned streamer broadcast video, second time information indicating the progress of the competition is obtained. The display device according to claim 1, further configured to obtain the difference between the first time information and the second time information as a delay time, and to synchronize the synchronization between the content video and the streamer relay video based on the delay time. [Claim 3] The aforementioned controller, Based on the first time information and the second time information, the video that is currently being played among the content video and the streamer relay video is stored in a buffer. The display device according to claim 2, further configured to delay the output of the previously played video by the aforementioned delay time. [Claim 4] The aforementioned controller, Buffering occurs in at least one of the content video and the streamer relay video, or The display device according to claim 2, further configured to synchronize the sync between the content video and the streamer relay video based on the delay time if the reception quality of at least one video is changed. [Claim 5] The display device according to claim 1, wherein the controller is further configured to match the scene analysis results of the content video and the audio corresponding to the streamer relay video to synchronize the synchronization between the content video and the streamer relay video. [Claim 6] The display device according to claim 1, wherein the controller is further configured to display one or more sync buttons on the display for syncing the content video and the streamer relay video. [Claim 7] The aforementioned video content information includes metadata such as the opposing teams and the competition schedule. The controller is further configured to search for one or more streamer broadcasts by comparing the metadata with information of a streamer broadcast being relayed by a streaming application. The display device according to claim 1, wherein the information of the streamer broadcast includes the title and broadcast time. [Claim 8] The aforementioned controller, If multiple streamer broadcasts are found, the recommendation order for these multiple streamer broadcasts is determined based on the reliability of the streamers. It is further configured to list the aforementioned multiple streamer broadcasts according to the determined recommendation order, The apparatus according to claim 7, wherein the reliability is determined by one or more of the following: the number of live participants, the number of subscribers, or the existing viewing history of the channel. [Claim 9] The aforementioned controller, The display device according to claim 1, further configured to display on the display a first volume control bar for adjusting the volume of a first audio corresponding to the content video and a second volume control bar for adjusting the volume of a second audio corresponding to the streamer relay video. [Claim 10] The aforementioned controller, The first audio corresponding to the aforementioned content video is separated into multiple audio channels. The display device according to claim 1, further configured to adjust the volume of each of the multiple audio channels and the volume of the second audio corresponding to the streamer relay video according to the characteristics of the event when it is determined that an event has occurred based on the content video. [Claim 11] The aforementioned multiple audio channels include commentary from the content video, player voices from the players, coach voices from the coach, and audience voices from the audience. The aforementioned event shows the scoring situation, The display device according to claim 10, wherein the controller is configured to decrease the volume of the relay audio of the content video and the second audio of the streamer relay video, and to increase the volume of the audience audio and the player audio of the content video. [Claim 12] The display device according to claim 1, wherein the controller is further configured to adjust the size of the content video and the size of the streamer relay video in accordance with commands received from a remote control device. [Claim 13] The aforementioned controller, The minimum aspect ratio of the content video is determined according to the sport shown in the content video. The display device according to claim 1, further configured to play the content video at a determined minimum aspect ratio. [Claim 14] The aforementioned controller, The minimum aspect ratio of the streamer broadcast video is determined based on the size of the text included in the streamer broadcast video. The display device according to claim 1, further configured to play back the streamer broadcast video at a determined minimum aspect ratio. [Claim 15] The aforementioned controller, Objects are extracted from the audio corresponding to the streamer video, The display device according to claim 1, further configured to display an indicator on the display that identifies an object extracted from the content video. [Claim 16] The display device according to claim 15, wherein the controller is further configured to display information about the object on the display in response to a command to select the indicator. [Claim 17] The aforementioned controller, The streamer's speech content is collected from the audio corresponding to the streamer's broadcast video, and a large language model is used to obtain a summary from the speech content. The display device according to claim 1, further configured to display the summary content on the display during interval intervals. [Claim 18] The aforementioned controller, A switch button is further displayed on the aforementioned display, The display device according to claim 1, further configured to swap the display positions between the content video and the streamer relay video in response to a command to select the switch button. [Claim 19] The display device according to claim 1, wherein the controller is further configured to display a chat window on the display for sharing reactions to the streamer broadcast. [Claim 20] A method for operating a display device, The stage of playing the content video; A step of searching for one or more streamer broadcasts based on the information of the aforementioned content video; The stage of displaying one or more streamer broadcasts that were found; and A method for operating a display device, comprising the step of playing back a streamer broadcast video corresponding to a streamer broadcast selected in accordance with a command to select one of the one or more streamer broadcasts, together with the content video.

Claims

1. A display device, A display for playing video content; and A controller is included; The aforementioned controller, Based on the information of the aforementioned content video, search for one or more streamer broadcasts, The one or more streamer broadcasts found are displayed on the display. A display device configured to play back a streamer broadcast video corresponding to a selected streamer broadcast in accordance with a command to select one of the one or more streamer broadcasts, together with the content video.

2. The aforementioned controller, Based on the first text contained in the aforementioned content video, first hour information indicating the progress of the competition is obtained, Based on the second text contained in the aforementioned streamer broadcast video, second time information indicating the progress of the competition is obtained. The display device according to claim 1, further configured to obtain the difference between the first time information and the second time information as a delay time, and to synchronize the synchronization between the content video and the streamer relay video based on the delay time.

3. The aforementioned controller, Based on the first time information and the second time information, the video that is currently being played among the content video and the streamer relay video is stored in a buffer. The display device according to claim 2, further configured to delay the output of the previously played video by the aforementioned delay time.

4. The aforementioned controller, Buffering occurs in at least one of the content video and the streamer relay video, or The display device according to claim 2, further configured to synchronize the sync between the content video and the streamer relay video based on the delay time if the reception quality of at least one video is changed.

5. The display device according to claim 1, wherein the controller is further configured to match the scene analysis results of the content video and the audio corresponding to the streamer relay video to synchronize the synchronization between the content video and the streamer relay video.

6. The display device according to claim 1, wherein the controller is further configured to display one or more sync buttons on the display for syncing the content video and the streamer relay video.

7. The aforementioned video content information includes metadata such as the opposing teams and the competition schedule. The controller is further configured to search for one or more streamer broadcasts by comparing the metadata with information of a streamer broadcast being relayed by a streaming application. The display device according to claim 1, wherein the information of the streamer broadcast includes the title and broadcast time.

8. The aforementioned controller, If multiple streamer broadcasts are found, the recommendation order for these multiple streamer broadcasts is determined based on the reliability of the streamers. It is further configured to list the aforementioned multiple streamer broadcasts according to the determined recommendation order, The apparatus according to claim 7, wherein the reliability is determined by one or more of the following: the number of live participants, the number of subscribers, or the existing viewing history of the channel.

9. The aforementioned controller, The display device according to claim 1, further configured to display on the display a first volume control bar for adjusting the volume of a first audio corresponding to the content video and a second volume control bar for adjusting the volume of a second audio corresponding to the streamer relay video.

10. The aforementioned controller, The first audio corresponding to the aforementioned content video is separated into multiple audio channels. The display device according to claim 1, further configured to adjust the volume of each of the multiple audio channels and the volume of the second audio corresponding to the streamer relay video according to the characteristics of the event when it is determined that an event has occurred based on the content video.

11. The aforementioned multiple audio channels include commentary from the content video, player voices from the players, coach voices from the coach, and audience voices from the audience. The aforementioned event shows the scoring situation, The display device according to claim 10, wherein the controller is configured to decrease the volume of the relay audio of the content video and the second audio of the streamer relay video, and to increase the volume of the audience audio and the player audio of the content video.

12. The display device according to claim 1, wherein the controller is further configured to adjust the size of the content video and the size of the streamer relay video in accordance with commands received from a remote control device.

13. The aforementioned controller, The minimum aspect ratio of the content video is determined according to the sport shown in the content video. The display device according to claim 1, further configured to play the content video at a determined minimum aspect ratio.

14. The aforementioned controller, The minimum aspect ratio of the streamer broadcast video is determined based on the size of the text included in the streamer broadcast video. The display device according to claim 1, further configured to play back the streamer broadcast video at a determined minimum aspect ratio.

15. The aforementioned controller, Objects are extracted from the audio corresponding to the streamer video, The display device according to claim 1, further configured to display an indicator on the display that identifies an object extracted from the content video.

16. The display device according to claim 15, wherein the controller is further configured to display information about the object on the display in response to a command to select the indicator.

17. The aforementioned controller, The streamer's speech content is collected from the audio corresponding to the streamer's broadcast video, and a large language model is used to obtain a summary from the speech content. The display device according to claim 1, further configured to display the summary content on the display during interval intervals.

18. The aforementioned controller, A switch button is further displayed on the aforementioned display, The display device according to claim 1, further configured to exchange the display positions between the content video and the streamer relay video in response to a command to select the switch button.

19. The display device according to claim 1, wherein the controller is further configured to display a chat window on the display for sharing responses to the streamer broadcast.

20. A method for operating a display device, The stage of playing the content video; A step in which one or more streamer broadcasts are searched based on the information of the aforementioned content video; The stage of displaying one or more streamer broadcasts that have been found; and A method for operating a display device, comprising the step of playing back a streamer broadcast video corresponding to a streamer broadcast selected in accordance with a command to select one of the one or more streamer broadcasts, together with the content video.

Images