Display device for immersive screen, and control method therefor

Abstract

A display device is disclosed. The display device comprises: a communication interface; a display; a memory for storing one or more instructions; and one or more processors, wherein the one or more processors execute the one or more instructions so as to control the display such that an image is displayed on the basis of content received from a source device, transmit first image data corresponding to the content to the external device through the communication interface when a network state for communication with an external device is a first state, and, when the network state is a second state, transmit feature information of the first image data and / or the second image data, which is converted on the basis of the first image data, to the external device through the communication interface.

Description

Display device for immersive screen and control method thereof

[0001] The present disclosure relates to a display device and a control method thereof, and more particularly, to a display device that can implement an immersive screen by being connected to a video output device, and a control method thereof.

[0002] Recently, there has been an increase in technology that allows users to become more immersed in the image by connecting a display device to a lighting device or projector and outputting lighting effects or images corresponding to the image being displayed on the display device to the periphery of the display device.

[0003] However, for a projector to generate graphic effects corresponding to real-time streaming video content outside the display device, it must receive a large video signal, analyze it in real time, and identify the corresponding graphic effects. Therefore, in cases of poor network conditions (e.g., limited available bandwidth), receiving and analyzing the original video signal in real time can be extremely challenging.

[0004] Therefore, the need for a technology that can change the device that analyzes the video signal by monitoring the network status in real time has arisen.

[0005] A display device according to at least one embodiment of the present disclosure includes a communication interface, a display, a memory, and one or more processors, wherein the one or more processors control the display to display an image based on content received from a source device by executing the one or more commands, and when a network state for communication with an external device is a first state, transmit first image data corresponding to the content to the external device through the communication interface, and when the network state is a second state, transmit at least one of feature information of the first image data and second image data converted based on the first image data to the external device through the communication interface.

[0006] A method for controlling a display device according to at least one embodiment of the present disclosure may include a step of displaying an image based on content received from a source device, a step of transmitting first image data corresponding to the content to the external device when a network state for communication with the external device is a first state, and a step of transmitting at least one of characteristic information of the first image data and second image data converted based on the first image data to the external device when the network state is a second state.

[0007] A computer-readable recording medium including a program for executing a method for controlling a display device according to at least one embodiment of the present disclosure, wherein the method for controlling the display device may include a step of displaying an image based on content received from a source device, a step of transmitting first image data corresponding to the content to the external device when a network state for communication with an external device is a first state, and a step of transmitting at least one of characteristic information of the first image data and second image data converted based on the first image data to the external device when the network state is a second state.

[0008] FIG. 1 is a diagram for explaining the operation of a display device and an image output device according to one or more embodiments of the present disclosure.

[0009] FIG. 2 is a block diagram illustrating a configuration of a display device according to one or more embodiments of the present disclosure.

[0010] FIGS. 3A and 3B are diagrams for explaining the operation of a display device and an image output device according to one or more embodiments of the present disclosure.

[0011] FIGS. 4 and 5 are drawings for explaining the operation of a display device according to one or more embodiments of the present disclosure.

[0012] FIG. 6 is a block diagram illustrating a configuration of an image output device according to one or more embodiments of the present disclosure.

[0013] FIGS. 7 and 8 are drawings for explaining a method for generating a graphic image of an image output device according to one or more embodiments of the present disclosure.

[0014] FIGS. 9 and 10 are drawings for explaining an image output method of an image output device according to one or more embodiments of the present disclosure.

[0015] FIG. 11 is a flowchart illustrating a method for controlling a display device according to one or more embodiments of the present disclosure.

[0016] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0017] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0018] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0019] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0020] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0021] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0022] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0023] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0024] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0025] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0026] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0027] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0028] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concepts of the present disclosure are not limited by the relative sizes or spacings drawn in the attached drawings.

[0029] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0030] FIG. 1 is a diagram for explaining the operation of a display device and an image output device according to one or more embodiments of the present disclosure.

[0031] According to FIG. 1, the display device (100) can be connected to an image output device (200), and the image output device (200) can output a graphic image to an external area of ​​the display device (100).

[0032] The display device (100) may be a device that displays an image based on content received from an external source device. Although the display device (100) is illustrated as a TV in FIG. 1, this is merely an example and may be implemented as a device with an image display function, such as a tablet PC, a video wall, a large format display (LFD), a digital signage, or a digital information display (DID).

[0033] Here, an external source device can transmit content to the display device (100), and the external source device can be implemented as various devices such as a PC, a laptop PC, a smartphone, a tablet PC, a set-top box, a media player, a game console, etc.

[0034] The video output device (200) can output a graphic image corresponding to the content received by the display device (100) to an external area of ​​the display device (100). In FIG. 1, the video output device (200) is illustrated as a projector (or projector display) that projects a graphic image, but is not limited thereto, and can be implemented as a device having a function of outputting an image, such as a TV, a video wall, a large format display (LFD), a digital signage, a digital information display (DID), etc.

[0035] In addition, although the image output device (200) is illustrated as a small projector in FIG. 1, this is merely an example, and the image output device (200) may be implemented as a fixed, stationary, or mobile projector. Here, a mobile projector refers to a projector having a movable member that allows the user to freely move the projector to a desired location.

[0036] Additionally, the video output device (200) may be located at a distance from the display device (100) as shown in FIG. 1, but this is only one example, and may be integrally attached to the back of the display device (100), located on a table or wall located at a short distance from the display device (100), or located on an accessory (e.g., speaker, etc.) of the display device (100).

[0037] According to one embodiment of the present disclosure, the display device (100) and the image output device (200) can provide an immersive screen to the user. Here, the immersive screen may refer to an image displayed so that the user can be more deeply immersed in the content. For example, the immersive screen may include an image that is a combination of an image output to the display device (100) and a graphic image output to an external area of ​​the display device (100) by the image output device (200). The immersive screen may be replaced with the terms 'Multi-Screen', 'Expanded Screen', 'Comprehensive Screen', and 'Integrated Screen', but in this description, it will be collectively referred to as an immersive screen.

[0038] As illustrated in FIG. 1, the image output device (200) can receive an image signal from the display device (100) and output a graphic image corresponding to the image signal. At this time, the display device (100) can identify a network status and determine whether to transmit the image signal and audio signal as is, transmit a downscaled image signal and audio signal, or transmit only characteristic information of the image signal and audio signal to the image output device (200). Since the display device (100) according to the present disclosure identifies the network status and transmits a signal corresponding to the identified network status to the image output device (200), even when the network status is not good (for example, when the available bandwidth of the network is limited), the image output device (200) can receive only the information essential for outputting the graphic image and output a graphic image corresponding to the content in real time.

[0039] Below, the operation of the display device (100) according to various embodiments of the present disclosure will be described.

[0040] FIG. 2 is a block diagram illustrating a configuration of a display device according to one or more embodiments of the present disclosure.

[0041] According to FIG. 2, the display device (100) may include a memory (110), a communication interface (120), a display (130), and a processor (140).

[0042] The memory (110) can store various programs, data, commands, etc. used in the display device (100). In addition, the memory (110) can store various information according to various embodiments of the present disclosure.

[0043] A memory (110) according to an example of the present disclosure may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or a RAM included in one or more processors (140), or may be implemented as a separate memory from one or more processors (140). In this case, the memory (110) may be implemented as a memory embedded in the display device (100) or as a memory detachable from the display device (100) depending on the purpose of data storage. For example, data for driving the display device (100) may be stored in a memory embedded in the display device (100), and data for an extended function of the display device (100) may be stored in a memory detachable from the display device (100).

[0044] Meanwhile, in the case of the memory embedded in the display device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)), and in the case of the memory that can be detachably attached to the display device (100), it may be implemented as a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. there is.

[0045] The communication interface (120) is a configuration for the display device (100) to communicate with external devices such as an external source device or an image output device (200).

[0046] The communication interface (120) may include at least one wireless communication module, at least one wired communication module, etc. Each communication module may be implemented in the form of at least one hardware chip. The wireless communication module may include at least one module among a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules. In addition, the communication interface may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc. The wired communication module may include at least one among a LAN (Local Area Network) module, an Ethernet module, a pair cable, a coaxial cable, a fiber optic cable, or a UWB (Ultra Wide-Band) module, for example. The communication interface may be implemented in various forms as described above, and may transmit and receive various signals by communicating with external devices.

[0047] The display (130) refers to a configuration for displaying various contents.

[0048] The display (130) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (130) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (130) may be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc.

[0049] One or more processors (140) control the overall operation of the electronic device (100). Specifically, one or more processors (140) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (140) may be operatively connected to a memory (110), a communication interface (120), and a display (130). The processor (130) may be composed of one or more processors.

[0050] One or more processors (140) may perform operations of the electronic device (100) according to various embodiments by executing at least one instruction stored in the memory (110).

[0051] The one or more processors (140) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (140) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors (140) may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0052] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0053] One or more processors (140) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (130) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0054] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0055] In the embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. Hereinafter, for the convenience of description, one or more processors (140) will be referred to as a processor (140).

[0056] The processor (140) can control the display (130) to display an image based on content received from an external source device. Here, the content received from the external source device can include a video signal and an audio signal. The processor (140) can be connected to the external source device through a communication interface (120) and can receive content through the communication interface (120).

[0057] For example, the processor (140) can be connected to an external source device via Bluetooth communication using a Bluetooth module, and can receive video signals and audio signals from the external source device via a Bluetooth communication channel. The connection method via Bluetooth communication is merely an example, and the processor (140) can of course be connected to an external source device via various types of communication interfaces (120) described above.

[0058] For example, the communication method with the external source device and the external image output device may be the same or different, but for convenience of explanation, in the following description, communication with the external source device and the external image output device is performed through the communication interface (120).

[0059] According to one embodiment, when a network state connected through a communication interface (120) that performs communication with an external image output device (e.g., FIG. 1, 200) is identified as a first state, the processor (140) can transmit first image data corresponding to the received content to the image output device through the communication interface (120).

[0060] Here, the "network status" may be determined based on the available bandwidth of the network, packet loss rate, round-trip time (RTT) of packets, delay-gradient of packets, RSSI information, communication channel information, link speed information, channel interference information, or retry rate information. In addition, the "network status" may include a "first status" and a "second status." The "first status" may mean a status in which the network status is good and the processor (140) can transmit a large-capacity video signal or audio signal to the video output device (200). Conversely, the "second status" may mean a status in which the network status is worse than the "first status" and is therefore not suitable for transmitting a large-capacity video signal or audio signal. For example, the processor (140) may identify that the network is in the "first status" when the available bandwidth of the network is 30 Mbps or more, and may identify that the network is in the "second status" when the available bandwidth of the network is less than 30 Mbps. Additionally, the processor (140) may identify that the network is in a “first state” when the RTT of the packet is less than 30 ms, and may identify that the network is in a “second state” when the RTT of the packet is 30 ms or more.

[0061] In the above explanation, arbitrary numbers such as 30Mbps and 30ms were used as examples, but this is only an example and it is obvious that arbitrary values ​​can be used depending on the network requirements.

[0062] The above description only describes a method for identifying the network status based on the available bandwidth of the network and the RTT of the packet, but this is only one example, and it is of course possible to identify the network status based on various information such as the packet's Delay-Gradient, RSSI information, communication channel information, link speed information, channel interference information, or retry rate information.

[0063] For example, “first image data” may refer to an image signal received by the processor (140) from an external source device. For example, “the processor (140) transmits the first image data to the image output device (200)” may refer to the processor (140) providing the image signal received from the external source device to the image output device (200) while maintaining the details and quality thereof. “First image data” may be replaced by various terms of the same / similar concept, such as “first image signal.”

[0064] For example, "second image data" may mean image data converted by the processor (140) based on the first image data. For example, the second image data may mean an image signal obtained by downscaling an image signal received by the processor (140) from an external source device, or, "second image data" may include an image signal generated by converting an original image signal received from an external source device to a relatively low resolution or low quality in order to reduce the data capacity.

[0065] For example, “the processor (140) transmits second image data to the image output device (200)” may mean that the processor (140) does not maintain the quality of the image signal received from the external source device, but transmits it with relatively low quality. “Second image data” may be replaced by various terms of the same / similar concept, such as “second image signal”.

[0066] According to one embodiment, when the network state connected through the communication interface (120) that performs communication with the image output device (200) is identified as the second state, the processor (140) may transmit at least one of the feature information of the first image data and the second image data to the image output device (200) through the communication interface (120). For example, when the network state is identified as the second state, the processor (140) may transmit only the feature information of the first image data to the image output device (200), transmit only the second image data to the image output device (200), or transmit both the feature information of the first image data and the second image data to the image output device (200).

[0067] Here, "characteristic information of the first image data" may mean information generated by the processor (140) by extracting characteristics for each scene based on the first image data. For example, a scene may be a plurality of video frame sections corresponding to the same space in a scenario distinguished by a content producer such as a movie. However, the present invention is not limited thereto, and even a plurality of video frame sections corresponding to the same space may be distinguished as different scenes depending on the brightness, color, etc. of the image.

[0068] For example, "feature information of the first image data" may include feature information corresponding to each of the multiple scenes included in the first image data. Specifically, the feature information of the first image data may include various information, such as color information, main object information, contrast information, brightness information, edge information, and texture information, for each of the multiple scenes.

[0069] As an example, the processor (140) may perform image analysis on each of the plurality of image frames constituting ‘Scene A’ to extract feature information from ‘Scene A’.

[0070] For example, in order to perform image analysis on the first frame among a plurality of video frames, the processor (140) may identify pixel values ​​(e.g., grayscale values) of pixels included in the first frame and identify a representative pixel value of the corresponding frame based on the identified pixel values. For example, the representative pixel value may be determined as a main pixel value of the corresponding frame, such as an average value of pixels, a pixel value that occupies the largest proportion, etc.

[0071]

[0072] For example, the processor (140) can identify an object by detecting an edge in the first frame. For example, the processor (140) can detect an edge portion by connecting pixels with pixel values ​​in a similar range that are consecutively connected, and when an area surrounded by an edge is identified, the processor (140) can identify an object represented by the edge based on the shape of the edge.

[0073] For example, the processor (140) can obtain information about the main color and the main object constituting 'Scene A' by the method described above. The scene analysis method of the processor (140) described above is only one example, and the processor (140) can extract feature information by various analysis methods for the image signal, such as extracting feature information of the image signal by using various artificial intelligence models. For example, the processor (140) can identify an object included in a frame based on at least one of object recognition, object detection, object tracking, or image segmentation. For example, the processor (140) can detect objects using techniques such as semantic segmentation, which classifies and extracts objects included in a frame by type as needed, instance segmentation, which recognizes objects by classifying them by object even if they are the same type, and a bounding box in the shape of a rectangle that includes the detected object when detecting an object included in an image.

[0074] According to one embodiment, when the processor (140) identifies that the available bandwidth of the network of the communication interface (120) that performs communication with the image output device (200) is greater than a preset value, the processor (140) may transmit the first image signal to the image output device (200) through the communication interface (120).

[0075] According to one embodiment, when the processor (140) determines that the available bandwidth of the network is less than a preset value, the processor (140) may transmit at least one of the feature information of the first image data and the second image data to the image output device (200) through the communication interface (120).

[0076] Here, the "preset value" may refer to an available bandwidth value required to transmit the first image data to the image output device (200) without delay. The preset value may have different values ​​depending on the type of the first image data. For example, if the first image data is a video with a 4K UHD resolution, the preset value may be 25 Mbps, and if the video is a video with a resolution of HD (1080P), the preset value may be 5 Mbps. In addition, the preset value may be set to various values ​​depending on what the first image data recommends, such as a larger value being set as the preset value as the bit rate or frame rate required by the first image data increases.

[0077] According to one embodiment, the processor (140) may directly monitor the network status to determine whether the available bandwidth of the network of the communication interface (120) is greater than a preset value, thereby obtaining network status information, or may obtain network status information from an external source device. A description of a method for identifying the network status will be described in detail in the description of FIG. 3 below.

[0078] FIGS. 3A and 3B are diagrams for explaining the operation of a display device and an image output device according to one or more embodiments of the present disclosure.

[0079] According to FIGS. 3A and 3B, the display device (100) or the external source device (300) can monitor the network status to obtain status information of the network, and the display device (100), the image output device (200), and the external source device (300) can change the device for transmitting and receiving the image signal and / or the audio signal depending on the status of the network.

[0080] Here, the "network status information" may include at least one of the network's available bandwidth, packet loss rate, packet round-trip time, packet delay-gradient, RSSI (Received signal strength indicator) information, communication channel information, link speed information, channel interference information, or retry rate information. In other words, the network status information may be various network-related information that is affected by network congestion, distance, and transmission speed.

[0081] For example, the display device (100), the image output device (200), and the external source device (300) may all be connected using the same communication method (e.g., a Wi-Fi communication method). The external source device (300) may transmit image signals and audio signals to the display device (100) via a Wi-Fi network, and the processor (140) may also transmit image signals and audio signals to the image output device (200) via the Wi-Fi network. The processor (140) may monitor the status of the Wi-Fi network while transmitting and receiving the image signals and audio signals. For example, the processor (140) may obtain status information of the network, such as the available bandwidth of the Wi-Fi network, packet loss rate, round-trip time of packets, delay-gradient of packets, and RSSI (Received signal strength indicator). Similarly, the external source device (300) may also monitor the Wi-Fi network and transmit the status information of the network as described above to the display device (100).

[0082] For example, the display device (100), the image output device (200), and the external source device (300) may be connected using different communication methods. For example, the external source device (300) and the display device (100) may be connected via a Bluetooth communication method to transmit and receive image signals, and the display device (100) and the image output device (200) may be connected via a Wi-Fi communication method to transmit and receive image signals. In this case, the display device (100) is connected to the image output device (200) via a Wi-Fi network, so the processor (140) can monitor the Wi-Fi network to obtain network status information, whereas the external source device (300) is connected via a Bluetooth network, so it cannot directly monitor the Wi-Fi network. Therefore, the processor (140) can obtain network status information only by directly monitoring the network status.

[0083] By the above-described method, the processor (140) can identify whether the network status is the first state or the second state based on the network status information obtained by the processor (140) directly monitoring the network status or the network status information obtained from the external source device (300).

[0084] As illustrated in FIG. 3A, an external source device (300) transmits a video signal to the display device (100) through a network (e.g., a Wi-Fi network) configured with the display device (100), and the display device (100) can transmit a video signal or characteristic information of the video signal to the image output device (200) through a network configured with the image output device (200). By the above-described method, the display device (100) can identify the state of the network configured with the image output device (200), and when the identified network state is the first state, the display device (100) can transmit a control signal to the image output device (200) for outputting a first graphic image based on the first image data to an external area of ​​the display device (100). In addition, when the identified network state is the second state, the display device (100) may transmit a control signal to the image output device (200) for outputting a second graphic image based on at least one of the characteristic information of the first image data and the second image data in an external area of ​​the display device (100). However, in some cases, even if the control signal is not transmitted to the image output device (200), an event in which at least one of the first image data, the characteristic information of the first image signal, and the second image data is transmitted to the image output device (200) may replace the function of the control signal.

[0085] As illustrated in FIG. 3B, the external source device (300) may configure a network with each of the display device (100) and the image output device (200). In one example, the available bandwidth of the network configured between the display device (100) and the image output device (200) may be very limited, making it difficult to transmit an image signal or feature information. In this case, the external source device (300) may directly configure a network with the image output device (200) and transmit the image signal or feature information to the image output device (200) through the network. By the above-described method, the external source device (300) can identify the state of the network, and when the state of the network configured between the external source device (300) and the image output device (200) is the first state, the external source device (300) can transmit a control signal to the image output device (200) for outputting a first graphic image based on the first image data to the image output device (200). In addition, when the identified network state is the second state, the external source device (300) may transmit a control signal to the image output device (200) for outputting a second graphic image based on at least one of the characteristic information of the first image data and the second image data to the external area of ​​the display device (100). However, in some cases, even if the control signal is not transmitted to the image output device (200), an event in which at least one of the first image data, the characteristic information of the first image signal, and the second image data is transmitted to the image output device (200) may replace the function of the control signal.

[0086] In the above description, only a Wi-Fi network and a Bluetooth network are exemplified, but this is only one example, and the display device (100), the video output device (200), and the external source device (300) can be connected through a network configured according to various communication methods such as Zigbee, 3G, 3GPP, LTE, LTE-A, 4G, and 5G.

[0087]

[0088] In the above description, it has been described that the display device (100) or the external source device (300) can identify the status of the network and perform an operation corresponding to the identified network status. However, the present invention is not limited thereto, and the display device (100), the image output device (200), and the external source device (300) may analyze the performance of each device and perform an operation corresponding to the performance of each device. For example, the occupancy rate of the CPU built into each device may be analyzed and the device with the lowest CPU occupancy rate may be controlled to extract feature information from the image signal. Specifically, if the CPU occupancy rate of the display device (100) is 70%, the CPU occupancy rate of the image output device (200) is 20%, and the CPU occupancy rate of the external source device (300) is 50%, the image output device (200) may be controlled to extract feature information from the received image signal.

[0089] The “first graphic image” and the “second graphic image” may refer to images that the image output device (200) outputs to an external area of ​​the display device (100) to provide an immersive screen effect to the user, which will be described in detail later in FIGS. 4 and 5.

[0090] FIGS. 4 and 5 are drawings for explaining the operation of a display device according to one or more embodiments of the present disclosure.

[0091] According to FIG. 4, in operation 401, the display device (100) can perform network monitoring to identify the network status of the communication interface (120) that performs communication with the image output device (200). For example, the display device (100) can monitor the network status by obtaining network status information such as the available bandwidth of the network configured for communication with the image output device (200), the packet loss rate of the network, and the Round-Trip Time of packets.

[0092] In operation 402, the display device (100) can identify the network status as the first status, and in operation 403, can transmit the first image data to the image output device (200). For example, the display device (100) can identify the network status as the first status when the available bandwidth of the network is equal to or greater than a preset value.

[0093] In operation 403, the display device (100) may transmit information corresponding to a first graphic image output based on the first image data to the external area of ​​the display (130) to the image output device (200). Here, “information corresponding to the first graphic image output” may refer to a signal transmitted to an external device such as the image output device (200) to output the first graphic image to the external area of ​​the display (130). For example, “information corresponding to the first graphic image output” may include a control signal to output the first graphic image to the external area of ​​the display (130).

[0094] In operation 404, when the first image data and the control signal are received, the image output device (200) may output the first graphic image to an external area of ​​the display device (100) in operation 405. However, in some cases, the control signal may be included in the first image data and transmitted to the image output device (200), or may not be transmitted to the image output device (200). For example, the image output device (200) may output the first graphic image to an external area of ​​the display device (100) based on an event in which the first image data is received.

[0095] Here, the "first graphic image" may mean a graphic image generated by the image output device (200) based on the first image data. In one embodiment, the first graphic image may include a graphic image generated by an artificial intelligence model trained to input feature information of an image signal and output a graphic effect (e.g., a blur effect) and / or a graphic object (e.g., a water drop) corresponding to the feature information. In another embodiment, the first graphic image may include a graphic image generated by an artificial intelligence model trained to input a video signal and feature information of the video signal together and output a graphic effect and / or a graphic object corresponding to the video signal and the feature information.

[0096] For example, the external area of ​​the display device (100) may include at least one of the peripheral area, the upper area, the lower area, the left area, or the right area of ​​the display device (100). Here, the peripheral area of ​​the display device (100), returning to FIG. 1, may mean an area formed in the same shape as the display device (100) while surrounding the external area of ​​the bezel provided in the display device (100).

[0097] According to one example, the first graphic image and / or the second graphic image may be displayed on an external area of ​​the display (130) provided in the display device (100). In this case, the processor (240) may generate a projection image having a size (or resolution) that is larger than the size of the display (130). For example, the processor (240) may perform alpha blending processing on an area corresponding to the display (130) and generate a projection image by including the first graphic image and / or the second graphic image in the remaining area. Here, alpha blending refers to a method of mixing the background RGB value and the RGB value above it by assigning a new value called A (Alpha) to the color value RGB in order to produce a transparent effect when overlaying another image on top of the image.

[0098] For example, the Alpha value is divided into 0~255 values ​​or 0.0~1.0 values, and 0 can mean completely transparent, and the opposite, 255 (or the highest value such as 1.0) can mean fully opaque. Or, 0 can mean completely opaque, and the opposite, 255 (or the highest value such as 1.0) can mean fully transparent. For example, when 8 bits are allocated to the Alpha value and can express values ​​from 0 to 255, the larger the value, the higher the ratio of the corresponding pixel, and the lower the value, the lower the ratio. For example, by increasing the transparency of the area corresponding to the display (130), the image displayed on the display (130) provided at the rear end is maintained, and the first graphic image and / or the second graphic image can be provided only in the external area of ​​the display (130).

[0099] According to another embodiment, the first graphic image and / or the second graphic image may be displayed on both the display (130) area and the external area of ​​the display (130) provided in the display device (100). In this case, the processor (240) may provide the first graphic image and / or the second graphic image including a graphic effect (e.g., a blur effect) and / or a graphic object (e.g., a water droplet) corresponding to the content being displayed on the display (130). In this case, the processor (240) may output the first graphic image and / or the second graphic image to an area including an area surrounding the display (130) area and the external area of ​​the bezel with a predetermined thickness, thereby overlaying the image of the content being displayed on the display (130) with the first graphic image and / or the second graphic image. The first graphic image and / or the second graphic image may be overlaid with the image of the content being displayed on the display (130), so that the user may play the content in a more immersive manner than when the first graphic image and / or the second graphic image is provided only in an external area of ​​the display (130).

[0100] According to FIG. 5, in operation 406, the display device (100) can perform network monitoring to identify the network status of the communication interface that performs communication with the image output device (200).

[0101] In operation 407, if the network status is identified as the second status, the display device (100) may acquire feature information from the first image data or acquire a second image signal in operation 408. For example, if the available bandwidth of the network is less than a preset value, the display device (100) may identify the network status as the second status.

[0102] As described in FIG. 2, feature information may mean feature information corresponding to each of a plurality of scenes included in the first image data, and second image data may mean an image signal generated by downscaling the first image data to a lower resolution or lower quality.

[0103] In operation 409, the display device (100) may transmit at least one of the feature information and the second image data acquired from the first image data to the image output device (200), and in operation 410, transmit information corresponding to a second graphic image output based on at least one of the feature information and the second image data. Here, “information corresponding to a second graphic image output based on at least one of the feature information and the second image data” may mean a signal transmitted to an external device such as the image output device (200) to output the second graphic image to an external area of ​​the display (130). For example, “information corresponding to a second graphic image output based on at least one of the feature information and the second image data” may include a control signal to output the second graphic image to an external area of ​​the display (130).

[0104] In operation 411, the image output device (200) can output the second graphic image to an external area of ​​the display device (100).

[0105] Here, the "second graphic image" may mean a graphic image generated by the image output device (200) based on at least one of the feature information and the second graphic image. In one embodiment, the second graphic image may include a graphic image generated by an artificial intelligence model trained to receive feature information of an image signal and output a graphic effect (e.g., a blur effect) and / or a graphic object (e.g., a water droplet) corresponding to the feature information. In another embodiment, the second graphic image may include a graphic image generated by an artificial intelligence model trained to receive a downscaled image signal and the feature information of the image signal together and output a graphic effect and / or a graphic object corresponding to the downscaled image signal and the feature information.

[0106] In the above description, it has been explained that both the first graphic image and the second graphic image are generated by a learned artificial intelligence model, but this is only one example, and graphic effects and / or graphic objects corresponding to the video signal or the characteristic information of the video signal may of course also be generated by a predefined filter. Here, the predefined filter may predefine and store graphic effects that are suitable for the characteristics of the display device (100), the viewing mode selected by the user, and the type of content before the video signal or the characteristic information is received by the video output device (200). For example, if the type of content is a 'soccer game', the predefined filter may generate a resizing effect to output the received image as an enlarged image in order to implement an immersive screen similar to watching a soccer game in an actual soccer field. In addition, the predefined filter may detect the size of the display device (100) to generate a blur effect in the periphery of the display, or generate graphic objects such as water droplets, and may generate various graphic effects.

[0107] In addition, although the above description describes that the processor (140) transmits only the video signal, feature information, or control signal to the video output device (200), the processor (140) can output corresponding video and audio at the same timing by synchronizing the time information of the display device (100) and the video output device (200) to share the same time standard. For example, the processor (140) can also transmit a packet including a Timestamp for synchronizing the output of each video frame and audio frame with the output of the video output device (200), and the video output device (200) can transmit a packet including a Timestamp to the display device (100). For example, a Timestamp can be included in a region of each video frame and audio frame and transmitted.

[0108] For example, the display device (100) can be a Network time protocol (NTP) server device and the image output device (200) can be an NTP client device. The display device (100) can be connected to an upper NTP server that provides very precise time information. The display device (100) can exchange Request packets and Reply packets with the image output device (200) for synchronization with each other. Specifically, the image output device (200) can transmit a Request packet including a Reference Timestamp, an Origin Timestamp, a Receive Timestamp, etc. to the display device (100), and the display device (100) can transmit a Reply packet including the above-described Timestamp to the image output device (200) in response thereto. The image output device (200) can perform synchronization with the display device (100) based on the Reply packet received from the display device (100), the time taken for the packet to be received, etc.

[0109] The above-described method of synchronizing two devices using NTP is only one example, and the time information of the display device (100) and the image output device (200) can be synchronized through various methods, such as a method of synchronizing two devices using Precision Time Protocol (PTP), a method of obtaining time information by receiving GPS signals through a GPS receiver, etc.

[0110] Additionally, although FIGS. 4 and 5 only illustrate that the processor (140) can transmit first image data or second image data, the processor (140) may also transmit first audio data or second audio data to the image output device (200).

[0111] Specifically, returning to FIG. 4, if the network status is identified as the first status, the display device (100) may transmit first audio data to the image output device (200). Here, “first audio data” may refer to an audio signal received by the processor (140) from the external source device (300). That is, “the processor (140) transmits the first audio data to the image output device (200)” may refer to the processor (140) maintaining the details and quality of the audio signal received from the external source device (300) and providing it to the image output device (200). The term “first audio data” may be replaced by various terms of the same / similar concept, such as the term “first audio signal.”

[0112] Also, returning to FIG. 5, the display device (100) can transmit at least one of the characteristic information of the first audio data and the second audio data to the video output device (200) when the network state is identified as the second state.

[0113] Here, "second audio data" may refer to audio data converted by the processor (140) based on the first audio data. For example, the second audio data may refer to an audio signal obtained by downscaling an audio signal received by the processor (140) from an external source device. In other words, the "second audio data" may include an audio signal generated by downscaling an original audio signal received from an external source device to a lower sampling rate or lower sound quality.

[0114] Therefore, "the processor (140) transmits second audio data to the image output device (200)" may mean that the processor (140) transmits a reduced-quality audio signal that retains only the essential information without maintaining the details and quality of the image signal received from the external source device. "Second audio data" may be replaced by various terms of the same / similar concept, such as "second audio signal".

[0115] Here, "characteristic information of the first audio data" may refer to information generated by the processor (140) by extracting characteristics for each scene based on the first audio data. Specifically, the characteristic information of the first audio data may include various information such as frequency, volume, and bit of audio signals constituting each of the plurality of scenes.

[0116] The image output device (200) can output a first graphic image or a second graphic image based on an audio signal and characteristic information of the audio signal, and this will be described in detail in the description of FIGS. 6 to 9 described below.

[0117] Meanwhile, in FIGS. 4 and 5, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0118] FIG. 6 is a block diagram illustrating a configuration of an image output device according to one or more embodiments of the present disclosure.

[0119] According to FIG. 6, the image output device (200) may include a memory (210), a communication interface (220), an image projection unit (230), and a processor (240).

[0120] The memory (210) can store various programs, data, commands, etc. used in the image output device (200). In addition, the memory (210) can also store various information and pre-trained artificial intelligence models according to various embodiments of the present disclosure.

[0121] The communication interface (220) is configured for communication with other electronic devices, including the display device (100). A detailed description of the communication interface (220) overlaps with the description of FIG. 2, so the overlapping portion will be omitted.

[0122] The image projection unit (230) can perform a function of projecting light to the outside to express an image and outputting the image on a projection surface. Here, the projection surface can be a part of the physical space where the image is output or a separate projection surface. The image projection unit (230) can include various detailed configurations such as a light source of at least one of a lamp, an LED, and a laser, a projection lens, and a reflector.

[0123] The image projection unit (230) can project an image using one of various projection methods (e.g., CRT (cathode-ray tube) method, LCD (Liquid Crystal Display) method, DLP (Digital Light Processing) method, laser method, etc.). The image projection unit (230) can include at least one light source.

[0124] The video projection unit (230) can output images in a 4:3 screen ratio, a 5:4 screen ratio, a 16:9 wide screen ratio, etc., depending on the user's settings, and can also project images in various resolutions such as WVGA (854*480), SVGA (800*600), XGA (1024*768), WXGA (1280*720), WXGA (1280*800), SXGA (1280*1024), UXGA (1600*1200), Full HD (1920*1080), etc., depending on the screen ratio.

[0125] The image projection unit (230) can perform various functions for adjusting the projection image under the control of the processor (240). For example, the image projection unit (230) can also perform a zoom function, a lens shift function, etc.

[0126] The processor (240) controls the overall operation of the image output device (200). Specifically, the processor (240) is connected to the memory (210), the communication interface (220), and the image projection unit (230), and can control the overall operation of the image output device (200) by executing at least one command or program stored in the memory (210).

[0127] The various operations described as the operation of the image output device (200) in the above-described section can be performed through the control of the processor (240).

[0128] For example, the operation of the image output device (200) outputting the first graphic image to an external area of ​​the display device (100) may be performed by the processor (240) as the subject of the operation. Specifically, the processor (240) may control the image projection unit (230) to output the first graphic image to an external area of ​​the display device (100).

[0129] When the processor (240) receives first image data from the display device (100) through the communication interface (220), it analyzes the first image data to obtain first feature information of the first image data, and inputs the obtained first feature information into a learned artificial intelligence model to obtain a first graphic image.

[0130] Alternatively, when the processor (240) receives at least one of the second feature information of the first image data and the second image data from the display device (100) through the communication interface (220), the processor (240) may input at least one of the second feature information or the second image data into the learned artificial intelligence model to obtain a second graphic image.

[0131] Here, "first feature information of the first image data" may mean information obtained by the processor (240) by extracting feature information corresponding to each of a plurality of scenes included in the image based on the first image data. Conversely, "second feature information of the first image data" may mean feature information of the first image data transmitted by the display device (100) to the processor (240) of the image output device (200). In addition, a method of obtaining a first graphic image or a second graphic image using a learned artificial intelligence model will be described in detail in the description of FIGS. 7 and 8 described below.

[0132] FIGS. 7 and 8 are drawings for explaining a method for generating a graphic image of an image output device according to one or more embodiments of the present disclosure.

[0133] According to FIG. 7, when the processor (240) receives first image data and / or first audio data from the display device (100), the processor (240) may analyze the first image data and / or the first audio data to extract first feature information of the first image data and / or first feature information of the first audio data (501). The “first feature information of the first image data” may include various information such as color information, main object information, and contrast information corresponding to each of a plurality of scenes constituting the first image data, and the “first feature information of the first audio data” may include various information such as frequency, volume, and bit of the first audio data.

[0134] Next, the processor (240) can input the first feature information of the first image data and / or the first feature information of the first audio data into at least one of a predefined filter (502) and a learned artificial intelligence model (503) to obtain a first graphic image (504).

[0135] The learned artificial intelligence model (503) may mean an artificial intelligence model that is learned to input image feature information and / or audio feature information and output at least one graphic effect or graphic object corresponding to the input data.

[0136] For example, if the content that the user is viewing through the display device (100) is a soccer game, the processor (240) can receive video signals and audio signals for the soccer game, and input feature information extracted from the received video signals and audio signals into the learned artificial intelligence model (503).

[0137] The learned artificial intelligence model (503) can identify key events (e.g., goal scenes) based on the extracted feature information. For example, the learned artificial intelligence model (503) can detect a sudden increase in the volume of a commentator's voice or recognize the commentator's voice based on volume information and frequency information among the feature information of an audio signal, thereby detecting the word "goal." Furthermore, the learned artificial intelligence model (503) can detect that a "goal net" is shaking based on the feature information of a video signal, and can identify a goal scene by detecting such changes.

[0138] The learned artificial intelligence model (503) can identify graphic effects corresponding to the detected key events based on the characteristic information of the video and audio signals when a key event is detected. For example, when a goal scene is detected, a graphic effect can be generated to output an image of the audience cheering on the external area of ​​the display device (100). Similarly, when a gunshot scene in a movie is detected, a graphic effect can be generated to output an image similar to blood splattering around the external area of ​​the display device (100).

[0139] As described above, an artificial intelligence model that receives the first feature information of the first image data extracted by the processor (240) and / or the first feature information of the first audio data and generates a corresponding graphic effect is defined as a “learned first artificial intelligence model.”

[0140] The processor (240) can generate simple graphic effects not only by the learned first artificial intelligence model but also by a predefined filter (502). However, the description of the predefined filter (502) has already been mentioned in the description of FIGS. 4 and 5 described above, and thus any overlapping portions will be omitted.

[0141] According to FIG. 8, when the processor (240) receives second feature information of the first image data and / or second feature information of the first audio data from the display device (100), the processor (240) may input the second feature information of the first image data and / or the second feature information of the first audio data into at least one of a predefined filter (502) and a learned artificial intelligence model (503) to obtain a second graphic image (505).

[0142] “Second characteristic information of the first image data” may mean characteristic information extracted by the display device (100) based on the first image data, and may include various information such as color information, main object information, and contrast information corresponding to each of a plurality of scenes constituting the first image data, and “second characteristic information of the first audio data” may also mean characteristic information extracted by the display device (100) based on the first audio data, and may include various information such as frequency, volume, and bit of the first audio data.

[0143] As illustrated in FIG. 8, the processor (240) does not need to extract feature information because it receives feature information extracted from the display device (100).

[0144] The learned artificial intelligence model (503) can obtain a second graphic image (505) corresponding thereto based on the second feature information of the first image data and / or the second feature information of the first audio data. As described above, an artificial intelligence model that receives the second feature information of the first image data and / or the second feature information of the first audio data and outputs a second graphic image (505) corresponding thereto is defined as a “learned second artificial intelligence model.” The method by which the learned second artificial intelligence model obtains the second graphic image (505) is the same as the method by which the learned first artificial intelligence model obtains the first graphic image (504), and therefore, a detailed description thereof will be omitted.

[0145] Additionally, when the processor (240) receives second image data and / or second audio data from the display device (100), the processor (240) may input the second image data and / or second audio data into the learned artificial intelligence model (503) to obtain a second graphic image (505).

[0146] An artificial intelligence model that inputs second image data and / or second audio data and outputs a corresponding second graphic image (505) is defined as a “learned third artificial intelligence model.”

[0147] The first graphic image (504) may be an image generated by an artificial intelligence model based on feature information extracted by the processor (240) from the original video signal and / or the original audio signal, while the second graphic image (505) may be an image generated by an artificial intelligence model based on small-capacity feature information or downscaled video signal and / or audio signal generated by extracting only necessary features from the original video signal and / or the original audio signal by the display device (100) when the network status is in the second status (bad). Accordingly, the first graphic image may provide a richer graphic effect than the second graphic image, and may provide a more immersive screen experience to the user.

[0148] In the above description, for the convenience of explanation, the types of data input to the learned artificial intelligence model are defined as 'first, second, and third artificial intelligence models' respectively, but it is of course possible for a single artificial intelligence model to receive all of the following inputs: image / audio signals, image / audio feature information, and downscaled image / audio signals.

[0149] The above-described artificial intelligence model (503) can be stored in the memory (210) of the image output device (200), and the operation of the artificial intelligence model (503) can also be performed by an external server.

[0150] In the above description, for the convenience of explanation, an embodiment in which the processor (240) inputs the first image data, the second image data, and the feature information of the first image data into the artificial intelligence model has been individually described, but this is only one example, and the processor (240) may perform an operation that combines the above-described embodiments, such as simultaneously inputting the feature information of the second image data and the first image data into the artificial intelligence model.

[0151] The above-described artificial intelligence model (503) may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs neural network operations through operations between the operation results of the previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized based on the learning results of the artificial intelligence model (503). For example, the plurality of weights may be updated so that the loss value or cost value acquired by the artificial intelligence model (503) is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks, but is not limited to the examples described above.

[0152] FIGS. 9 and 10 are drawings for explaining an image output method of an image output device according to one or more embodiments of the present disclosure.

[0153] According to FIG. 9, the processor (240) can output a graphic image (610) including a graphic effect corresponding to the content being played by the user to an external area of ​​the display device (100).

[0154] For example, as illustrated in FIG. 9, the processor (240) can analyze the video and audio of the content to identify a major event occurring within the content. Specifically, the processor (240) can detect the occurrence of a major event, such as a bullet being fired, by identifying the movement of a major object within the video, such as a gun being shaken or a bullet casing being identified around the gun, using an artificial intelligence model. Alternatively, the processor (240) can detect the occurrence of a major event, such as a bullet being fired, based on analyzing the frequency and volume of an audio signal to detect that an audio signal of a certain frequency continues and then an audio signal of a different frequency appears or an audio signal of a suddenly high volume appears.

[0155] When an event that a bullet has been fired is detected, the processor (240) can control the image projector (230) to output a graphic image (610) including a graphic effect such as blood splatter on an external area of ​​the display device (100) so that the user can become more immersed in the content.

[0156] Although FIG. 9 only shows that a graphic image (610) including a corresponding graphic effect is output by detecting the occurrence of a major event, this is only one example, and it is of course possible to output a graphic image including various graphic effects, such as generating a lighting effect of a color corresponding to the content in an external area of ​​the display device (100) based on the characteristic information of the content.

[0157] According to FIG. 10, the image output device (200) can output different graphic images (620, 630, 640) to different areas according to output mode information.

[0158] Here, the output mode information may refer to information on how the processor (240) pre-stored in the image output device (200) outputs an image. For example, the output mode information may include a watching mode, an immersive ambient mode, and a 3D mode. In addition, the term “output mode” may be replaced by the term “output state.” Accordingly, the above-described watching mode, immersive ambient mode, and 3D mode may be replaced by the terms watching state, immersive ambient state, and 3D state, respectively. For convenience of explanation, the term “output mode” will be used instead of the term “output state” in this description.

[0159] The processor (240) can receive user input and determine an output mode, and can also determine a corresponding output mode depending on the type of content received.

[0160] In one embodiment, if the content to be displayed by the display device (100) is movie content with a resolution having a ratio of 16:9, the processor (240) may determine the viewing mode as the output mode based on the information that it is movie content. The processor (240) may input the video signal of the movie content and the output mode information called the viewing mode into an artificial intelligence model to obtain a corresponding graphic image (620). The graphic image (620) corresponding to the viewing mode may be a graphic image that allows an image with a resolution having a ratio of 21:9 to be displayed through the display device (100) and an external area of ​​the display device (100), as illustrated in FIG. 10. However, this is merely an example, and the graphic image corresponding to the viewing mode is not limited to an image with a resolution having a specific ratio, but may include various images that provide a wider screen than the original image so that the user can enjoy the content with a more immersive experience.

[0161] In another embodiment, when the content to be displayed by the display device (100) is music content, the processor (240) may determine the immersive ambient mode as an output mode based on information that the content is music content. The processor (240) may input the video signal, audio signal, and output mode information of the music content, called the immersive ambient mode, into an artificial intelligence model to obtain a corresponding graphic image (630). The graphic image (630) corresponding to the immersive ambient mode may be implemented by outputting a graphic effect corresponding to the characteristics of the video signal or audio signal to an external area of ​​the display device (100), as illustrated in FIG. 10.

[0162] In another embodiment, when the content to be displayed by the display device (100) is a 3D game, the processor (240) may determine the 3D mode as an output mode based on information that the content is 3D game content. The processor (240) may input the video signal, audio signal, and output mode information of the 3D mode of the game content into an artificial intelligence model to obtain a corresponding graphic image (640). The graphic image (640) corresponding to the 3D mode may be implemented by outputting a graphic effect corresponding to the characteristics of the video signal and audio signal of the game content to the front bottom surface of the display device (100), as illustrated in FIG. 10.

[0163] In the description of FIGS. 1 to 10 described above, only an embodiment in which the display device (100) and the image output device (200) directly communicate with each other has been described, but the display device (100) and the image output device (200) may be linked with an external server to perform the operations described in the description of FIGS. 1 to 10 described above.

[0164] For example, an external server can control the display device (100) to output video and audio by providing video signals and audio signals, and at the same time, the external server can control the image output device (200) to output a corresponding graphic image toward the display device (100) by providing video signals and audio signals.

[0165] FIG. 11 is a flowchart illustrating a method for controlling a display device according to one or more embodiments of the present disclosure.

[0166] According to FIG. 11, in operation 1110, the display device (100) can display an image based on content received from an external source device.

[0167] In operation 1120, the display device (100) can identify a network status. Here, the network status may refer to the status of a network configured for the display device (100) to perform communication with the image output device (200). The network status may be determined based on the available bandwidth of the network, packet loss rate, round-trip time (RTT) of a packet, delay-gradient of a packet, RSSI information, communication channel information, link speed information, channel interference information, or retry rate information.

[0168] In operation 1130, the display device (100) may transmit the first image data to an external image output device if the network status is identified as the first status. For example, the display device (100) may transmit the first image data to the external image output device if the available bandwidth of the network is identified as being greater than a preset value.

[0169] In operation 1140, the display device (100) may transmit a control signal for outputting a first graphic image to an external image output device (200). For example, the display device (100) may transmit a control signal for outputting a first graphic image based on first image data to an external area of ​​the display device (100).

[0170] In operation 1150, if the network status is identified as the second status, the display device (100) may transmit the characteristic information of the first image data or the second image data to the external image output device (200). For example, if the available bandwidth of the network is identified as being less than a preset value, the display device (100) may transmit the characteristic information of the first image data or the second image data to the external image output device (200).

[0171] In operation 1160, the display device (100) may transmit a control signal for outputting a second graphic image to an external image output device (200). For example, the display device (100) may transmit a control signal for outputting a second graphic image based on at least one of the characteristic information of the first image data and the second image data to an external area of ​​the display device (100).

[0172] The various methods described in FIG. 11 can be performed by a display device having the configuration shown in FIG. 2, but are not necessarily limited thereto, and can be performed by display devices having various configurations.

[0173] Meanwhile, in Fig. 11, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0174] The various embodiments of the present disclosure can be applied and implemented not only to a display device but also to all types of electronic devices including a display function, and the embodiments can be combined in whole or in part to be applied to a single device.

[0175] Meanwhile, the various embodiments described above may be implemented in a computer-readable recording medium or similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. In a software implementation, embodiments, such as the procedures and functions described herein, may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0176] Meanwhile, computer instructions for performing processing operations of the electronic device (100) according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium.

[0177] Computer instructions stored on such non-transitory computer-readable media, when executed by a processor of a specific device, cause the specific device to perform the setting methods according to the various embodiments described above.

[0178] A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0179] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the display device, communication interface; display; A memory that stores one or more instructions; and comprising one or more processors; The one or more processors, by executing the one or more instructions, Controlling the display to display an image based on content received from a source device; If the network status for communication with an external device is the first state, control is provided to transmit the first image data corresponding to the content to the external device through the communication interface, A display device that controls to transmit at least one of the characteristic information of the first image data and the second image data converted based on the first image data to the external device through the communication interface when the network state is the second state.

2. In paragraph 1, The one or more processors, by executing the one or more instructions, If the available bandwidth of the above network is greater than a preset value, the first image data is controlled to be transmitted to the external device through the communication interface, A display device that controls to transmit at least one of the characteristic information of the first image data and the second image data to the external device through the communication interface if the available bandwidth of the network is less than the preset value.

3. In paragraph 1, The one or more processors, by executing the one or more instructions, A display device that identifies whether the network status is the first state or the second state based on at least one of network status information acquired from the display device and network status information received from the source device.

4. In paragraph 1, The one or more processors, by executing the one or more instructions, A display device that controls the acquisition of feature information corresponding to each scene included in the first image data and transmits it to the external device through the communication interface.

5. In paragraph 1, The one or more processors, by executing the one or more instructions, If the above network state is the first state, control is provided to transmit information corresponding to the first graphic image output based on the first image data to the external area of ​​the display to the external device through the communication interface, A display device that controls, when the network state is the second state, to transmit information corresponding to a second graphic image output based on at least one of the characteristic information of the first image data and the second image data to the external area of ​​the display to the external device through the communication interface.

6. In paragraph 5, The above first graphic image and the above second graphic image, A display device, which is a graphic image for providing an immersive screen effect for images displayed on the display.

7. In paragraph 1, The one or more processors, by executing the one or more instructions, If the above network state is the first state, control is made to transmit the first audio data included in the content to the external device through the communication interface, A display device that controls to transmit at least one of the characteristic information of the first audio data and the second audio data converted based on the first audio data to the external device through the communication interface when the network state is the second state.

8. In paragraph 1, The one or more processors, by executing the one or more instructions, A display device that controls the first image data to be transmitted to the external device through the communication interface so that the external device obtains a first graphic image from first feature information obtained based on the first image data through an artificial intelligence model and outputs the obtained first graphic image.

9. In paragraph 8, The one or more processors, by executing the one or more instructions, A display device that controls to transmit at least one of the second feature information of the first image data and the second image data converted based on the first image data to the external device through the communication interface so that the external device obtains and outputs a second graphic image from at least one of the second feature information of the first image data and the second image data through the artificial intelligence model.

10. In paragraph 8, The one or more processors, by executing the one or more instructions, Controlling to transmit at least one of the first image data, the feature information of the first image data, and the second image data to the external device through the communication interface so as to obtain and output a graphic image corresponding to the output state of the external device from the artificial intelligence model based on at least one of the first image data, the feature information of the first image data, and the second image data and the output state of the external device; The output status of the above external device is: A display device comprising at least one of a viewing state, an immersive ambient state, and a 3D state.

11. In a method for controlling a display device, A step of displaying an image based on content received from a source device; If the network status for communication with an external device is the first state, a step of transmitting first image data corresponding to the content to the external device; and A method for controlling a display device, comprising: a step of transmitting at least one of characteristic information of the first image data and second image data converted based on the first image data to the external device when the network state is the second state.

12. In paragraph 11, If the available bandwidth of the above network is greater than or equal to a preset value, a step of transmitting the first image data to the external device; and A method for controlling a display device, comprising: a step of transmitting at least one of characteristic information of the first image data and the second image data to the external device if the available bandwidth of the network is less than the preset value; 13. In paragraph 11, A method for controlling a display device, comprising: a step of identifying whether the network status is the first state or the second state based on at least one of network status information acquired from the display device and network status information received from the source device.

14. In paragraph 11, If the above network state is the first state, information corresponding to outputting the first graphic image based on the first image data to the external area of ​​the display device is transmitted to the external device, or A method for controlling a display device, further comprising: a step of transmitting, to an external device, information corresponding to a second graphic image output based on at least one of characteristic information of the first image data and the second image data, if the network state is the second state.

15. A non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of a display device, cause the display device to perform an operation, wherein the operation is: A step of displaying an image based on content received from a source device; If the network status for communication with an external device is the first state, a step of transmitting first image data corresponding to the content to the external device; and A non-transitory computer-readable recording medium comprising: a step of transmitting at least one of characteristic information of the first image data and second image data converted based on the first image data to the external device when the network state is the second state.

Images