Display device and control method thereof

Abstract

A display apparatus includes a processor configured to execute instructions to: select a category from a plurality of categories related to characteristics of a plurality of source devices connectable to an interface of the display apparatus based on an input from a user; receive a plurality of pieces of content from at least one source device connected to the interface and corresponding to the selected category; and display a plurality of images on a display of the display apparatus based on at least two of the plurality of pieces of content.

Description

Technical Field

[0001] The present disclosure relates to a display apparatus capable of displaying a multi-view image based on contents received from a plurality of source devices and a method of controlling the display apparatus. Background Art

[0002] In order to calculate and process predetermined information according to a specific process, the electronic device includes electronic components such as a central processing unit (CPU), a chipset, a memory, etc. Such an electronic device can be classified according to what information is to be processed and what it is used for. For example, the electronic device includes: an information processing device for processing general information, such as a personal computer (PC), a server, etc.; an image processing device for processing image data; an audio device for audio processing; a home appliance for miscellaneous household chores, etc. In the image processing device, a display device refers to a device including a display panel for displaying an image.

[0003] Due to various factors such as the development of technology, the needs of users, and the diversification of use, the screen size of the display device is constantly increasing. As a method of increasing the screen size, a modular display device has been proposed, in which a display panel is modularized in a certain size and a plurality of display modules are arranged like tiles to realize a large-size screen. The modular display device is gradually expanding in its application range because it can be harmoniously installed in any space by being assembled with a screen size and a screen pattern desired by the user. In the case of an average family, the display device can be implemented to completely occupy one side wall of the living room.

[0004] Such a display device may adopt various display methods to utilize a large-sized screen. For example, a display device with a small screen has a practical limit on the size of a displayed image, and thus adopts a method of displaying an image based on only one piece of content on the screen and displaying multiple images based on two pieces of content through a picture-in-picture (PIP). On the other hand, a display device with a large screen may display images based on multiple pieces of content together by dividing the screen so as not to overlap each other, without using a PIP or overlay method. For convenience, such a display method will be referred to as a multi-view method.

[0005] In terms of displaying multi-view images, the display device may receive content of multiple images to be displayed on the screen from one source device, or may receive multiple pieces of content separately from multiple source devices. When connected to multiple source devices, the display device is assigned source devices and content through manual control of the user, thereby displaying the multi-view image. However, this method is inconvenient for the user because the user needs to select multiple pieces of content to be displayed as multi-view images one by one. In this regard, the display device may be required to display a multi-view image reflecting the user's intention with minimal input from the user. Summary of the invention

[0006] According to one aspect of the present disclosure, a display device includes: a display; an interface configured to connect to at least one source device among a plurality of source devices that can be connected to the interface, and receive at least one content from the at least one source device; a user input interface configured to receive user input; a memory storing instructions; and a processor. The processor is configured to execute instructions so as to: select a category from a plurality of categories related to characteristics of the plurality of source devices based on the user input; receive a plurality of content from at least one source device connected to the interface, the at least one source device corresponding to the selected category; and display a plurality of images on the display based on at least two of the plurality of content.

[0007] The multiple categories may correspond to multiple types associated with the multiple source devices.

[0008] The processor may be further configured to execute the instructions to identify characteristics of the plurality of source devices based on device information received from the plurality of source devices.

[0009] The processor may be further configured to execute instructions to: based on identifying that at least two source devices are connected to the interface, display a user interface UI on the display to select the category from the plurality of categories.

[0010] The processor is further configured to execute instructions to: not display the UI based on identifying that only one source device is connected to the interface.

[0011] The processor may be further configured to execute instructions to: identify a selectable category from the plurality of categories on the UI based on characteristics of content being displayed on the display.

[0012] The processor may be further configured to execute instructions to: store user setting options including at least a first source device and a second source device selected by a user based on identifying that at least two source devices are connected to the interface; and identify the user setting options as a selectable category among the plurality of categories.

[0013] The processor may be further configured to execute instructions to: identify the at least two pieces of content from the plurality of pieces of content based on the descending order of the content-relevance rankings of the user.

[0014] The content-related ranking of a user may be related to the user's content preference.

[0015] The processor may be further configured to execute the instructions to: identify content-relevance rankings based on the user's content usage history.

[0016] The processor may be further configured to execute the instructions to: identify the at least two pieces of content from the plurality of pieces of content based on an audience rating of the user.

[0017] The processor may be further configured to execute the instructions to: identify an audience level based on user account information of the display device.

[0018] The processor may be further configured to execute the instructions to: identify a layout corresponding to the selected category from a plurality of layouts associated with a mode for displaying the multiple images; and display the multiple images based on the identified layout.

[0019] The processor may be further configured to execute the instructions to: display the multi-image including a main image and a sub-image smaller than the main image based on the at least two pieces of content.

[0020] According to one aspect of the present disclosure, a method for controlling a display device includes: based on user input, selecting a category from a plurality of categories related to characteristics of a plurality of source devices that can be connected to an interface of the display device, receiving a plurality of pieces of content from at least one source device connected to the interface, the at least one source device corresponding to the selected category; and displaying a plurality of images on a display of the display device based on at least two of the plurality of pieces of content.

[0021] The method may further include displaying a user interface (UI) on a display to select a category from a plurality of categories based on identifying that at least two source devices are connected to the interface.

[0022] The method may further include storing user setting options including at least a first source device and a second source device selected by a user based on identifying that at least two source devices are connected to the interface, and identifying the user setting options as selectable categories among the plurality of categories.

[0023] The method may also include identifying at least two pieces of content from the plurality of pieces of content based on at least one of an order of the highest content-relevance rankings of the users and an audience rating of the users.

[0024] The method may further include: identifying a layout corresponding to the selected category from a plurality of layouts associated with a mode for displaying the multiple images; and displaying the multiple images on a display based on the identified layout.

[0025] According to one aspect of the present disclosure, a non-transitory computer-readable medium stores computer-readable program code or instructions that can be executed by a processor to perform a method for controlling a display device. The method includes: based on user input, selecting a category from a plurality of categories related to characteristics of a plurality of source devices that can be connected to an interface of the display device, receiving a plurality of pieces of content from at least one source device connected to the interface, the at least one source device corresponding to the selected category; and displaying a plurality of images on a display of the display device based on at least two of the plurality of pieces of content. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A display device according to an embodiment is shown;

[0028] Figure 2 is a block diagram of a display device according to an embodiment;

[0029] Figure 3 is a flowchart showing a method of controlling a display device according to an embodiment;

[0030] Figure 4 shows a plurality of source devices classified into a plurality of categories corresponding to characteristics according to an embodiment;

[0031] Figure 5 A display device that displays a user interface (UI) used to select a category on a screen according to an embodiment is shown;

[0032] Figure 6 A display apparatus for selecting content for multi-view from a plurality of pieces of content received from two or more source devices according to an embodiment is shown;

[0033] Figure 7 A display device for displaying a multi-view image having a layout in the form of a main image and sub-images according to an embodiment is shown;

[0034] Figure 8 A display device for displaying a multi-view image having a layout in the form of a balanced image according to an embodiment is shown;

[0035] Fig. 9 is a flowchart showing a display device selecting a layout for a multi-view image according to an embodiment;

[0036] Fig.10 , Fig.11 , Fig.12 and Fig.13shows a multi-view image displayed by a display device connected to two game consoles;

[0037] Fig.14 shows a user selecting a source device from a plurality of predefined categories in a display apparatus according to an embodiment;

[0038] Fig.15 illustrates a user setting icon included in a UI for selection of a multi-view image according to an embodiment;

[0039] Fig.16 A display device that changes category selection options provided in a UI according to preset conditions according to an embodiment is shown. DETAILED DESCRIPTION

[0040] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. In addition, unless otherwise mentioned, the embodiments described with reference to the accompanying drawings are not mutually exclusive, and multiple embodiments can be selectively combined in one device. A person of ordinary skill in the art can arbitrarily select and apply the combination of these multiple embodiments to realize various aspects of the present disclosure.

[0041] In the description of the embodiments, ordinal numbers used in terms such as first element, second element, etc. are used to describe various elements, and these terms are used to distinguish one element from another element. Therefore, the meaning of the element is not limited by these terms, and these terms are only used to explain the corresponding embodiments, but not to limit the present disclosure.

[0042] Furthermore, “at least one” among a plurality of elements in the present disclosure refers not only to all the elements but also to each of these elements, excluding other elements or all combinations of these elements.

[0043] Figure 1 A display device according to an embodiment is shown.

[0044] like Figure 1 As shown, the display device 1 according to the embodiment can be implemented by a TV, for example. In addition to the TV, the display device 1 can also be implemented by various types of devices, such as a monitor capable of displaying images on the screen 201, a digital sign, an electronic blackboard, an electronic photo frame, etc. Here, the screen 201 of the display device 1 is large enough for multiple users to watch together.

[0045] The display apparatus 1 includes an image processing device 100 for processing an image signal based on image content received from the outside or stored in itself, and a display 200 for displaying an image based on the image signal output from the image processing device 100. According to the embodiment, the image processing device 100 in which the display apparatus 1 is provided in the form of a media box separate from the display 200 will be described, but the structure of the display apparatus 1 is not limited to this embodiment. The display apparatus 1 may be provided as a single entity structure in which the image processing device 100 and the display 200 are integrated into one frame or accommodated in one housing.

[0046] The image processing device 100 receives the image signal through various communication methods such as: performing short-distance communication with various external devices such as a personal computer (PC), a set-top box, a game console, and a multimedia player; performing wide area network (WAN) communication with a server; or receiving a radio frequency (RF) broadcast signal. Alternatively, the image processing device 100 can obtain image content data stored in a built-in storage device. The image processing device 100 performs image-related processing such as decoding and scaling on the obtained image signal, and outputs the processed image signal to the display 200. Both the wired method and the wireless method can be used for signal transmission between the image processing device 100 and the display 200.

[0047] The display 200 forms a screen 201 to display an image based on an image signal from the image processing device 100. The display 200 includes a display panel, and various design methods can be applied to the structure of the display panel. For example, the display 200 may include a single display panel and a backlight unit for emitting light to the display panel. Alternatively, the display 200 may have a structure in which a plurality of micro light emitting diode (LED) modules are combined in a tile form to form a large screen 201.

[0048] Next, the configuration of the display device 1 will be described.

[0049] Figure 2 is a block diagram of a display device according to an embodiment.

[0050] like Figure 2 As shown, the image processing device 100 of the display device 1 includes various hardware elements for operation. The display device 1 according to the embodiment has a structure in which the image processing device 100 is separated from the display 200, but this is only an example. Alternatively, the display device 1 may have a structure in which the image processing device 100 is not separated from the display 200.

[0051] The image processing apparatus 100 may include an interface 110. The interface 110 includes an interface circuit through which the image processing apparatus 100 performs communication with various kinds of external devices and transmits and receives data. The interface 110 may receive image content from an external device (e.g., a plurality of source devices 10, 20, and 30) and output an image content signal processed by the processor 150 to the display 200.

[0052] Depending on the connection type, the interface 110 may include one or more wired interfaces 111 for wired communication and one or more wireless interfaces 112 for wireless communication.

[0053] The wired interface 111 includes a connector or port connected to a cable with a predefined transmission standard. For example, the wired interface 111 includes a port connected to a terrestrial or satellite antenna to receive a broadcast signal or connected to a cable for wired broadcasting. In addition, the wired interface 111 includes ports with various wired transmission standards (e.g., high-definition multimedia interface (HDMI), display port (DP), digital video interaction (DVI), component, composite, S video, thunderbolt, etc.) to connect with various image processing devices. In addition, the wired interface 111 includes a port with a universal serial bus (USB) standard to connect with a USB device. In addition, the wired interface 111 includes an optical port connected to an optical cable. In addition, the wired interface 111 includes an audio input port to which an external microphone is connected, and an audio output port to which headphones, earplugs, speakers, etc. are connected. In addition, the wired interface 111 includes an Ethernet port connected to a gateway, a router, a hub, etc., for connecting to a WAN.

[0054] The wireless interface 112 includes an interactive communication circuit, which includes at least one of the elements corresponding to various types of wireless communication protocols (e.g., communication module, communication chip, etc.). For example, the wireless interface 112 includes: a Wi-Fi communication chip for wireless communication with an AP based on Wi-Fi; a communication chip for wireless communication based on Bluetooth, Zigbee, Z-wave, Wireless HD, Wireless Gigabit (WiGig), Near Field Communication (NFC), etc.; an infrared (IR) module for IR communication; a mobile communication chip for mobile communication with a mobile device, etc.

[0055] The image processing device 100 may include a user input interface 120. The user input interface 120 includes circuits related to various types of user input interfaces so as to be controlled by a user to receive user input. The user input interface 120 may be configured differently according to the type of the image processing device 100, and may include, for example, a mechanical or electronic button of the image processing device 100, a touch panel, a sensor, a camera, a touch screen installed in the display 200, a remote controller separated from the image processing device 100, and the like.

[0056] The image processing device 100 may include a storage device 130. The storage device 130 is configured to store digitized data. The storage device 130 includes a non-volatile memory that retains data regardless of whether power is supplied, and a volatile memory that retains data loaded to be processed by the processor 150 only when power is supplied. The storage device includes a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a read-only memory (ROM), etc., and the memory includes a buffer, a random access memory (RAM), etc.

[0057] The image processing apparatus 100 may include a speaker 140. The speaker 140 outputs the audio signal processed by the processor 150 as sound. The speaker 140 may be installed in the image processing apparatus 100, may be installed in the display 200, or may be provided as a separate device. When the speaker 140 is provided as a separate device, the speaker 140 is connected to the image processing apparatus 100 through the interface 110.

[0058] The image processing device 100 may include a processor 150. The processor 150 includes one or more hardware processors implemented by a central processing unit (CPU), a chipset, a buffer, a circuit, etc. mounted on a printed circuit board (PCB). Alternatively, the processor 150 may be designed as a system on chip (SoC). The processor 150 includes modules corresponding to various processes of a demultiplexer, a decoder, a scaler, an audio digital signal processor (DSP), an amplifier, etc., to display an image based on image content. Here, some or all of such modules may be implemented by an SoC. For example, modules such as a demultiplexer, a decoder, a scaler, etc. related to image processing may be implemented as an image processing SoC, and an audio DSP may be implemented as a chipset separated from the SoC.

[0059] According to an embodiment, a plurality of categories related to characteristics of valid source devices 10, 20, 30, 40, 50, 60, and 70 connectable to the interface 110 are predefined. When one of the plurality of predefined categories is selected, the processor 150 performs displaying a multi-view image on the display 200 based on the selected category. Hereinafter, a method of controlling the display apparatus 1 will be described in detail.

[0060] Figure 3 is a flowchart illustrating a method of controlling a display device according to an embodiment.

[0061] like Figure 2 and Figure 3 As shown, the operation of the display device 1 according to the embodiment is performed by the processor 150 .

[0062] At operation 310 , the display apparatus 1 receives a selection of one of a plurality of categories related to characteristics of the source devices 10 , 20 , 30 , 40 , 50 , 60 , and 70 connectable to the interface 110 .

[0063] At operation 320 , the display apparatus 1 identifies two or more source devices 10 , 20 , 30 , 40 , 50 , 60 , and 70 corresponding to the selected category among the plurality of source devices 10 , 20 , 30 , 40 , 50 , 60 , and 70 connected to the interface 110 .

[0064] At operation 330 , the display apparatus 1 receives a plurality of pieces of content from two or more identified source devices 10 , 20 , 30 , 40 , 50 , 60 , and 70 through the interface 110 .

[0065] At operation 340 , the display apparatus 1 displays a multi-image, in other words, a multi-view image, based on two or more pieces of received content on the display 200 .

[0066] In this manner, the display apparatus 1 recognizes a plurality of connected source devices 10, 20, 30, 40, 50, 60, and 70, and displays a multi-view image based on two or more source devices 10, 20, 30, 40, 50, 60, and 70 belonging to the same category for each characteristic. Therefore, the display apparatus 1 easily and automatically selects two or more source devices 10, 20, 30, 40, 50, 60, and 70 having characteristics suitable for the user's intention among a plurality of currently connected source devices 10, 20, 30, 40, 50, 60, and 70, and provides a multi-view image based on content suitable for the user.

[0067] Hereinafter, it will be described by way of example that the plurality of source devices 10 , 20 , 30 , 40 , 50 , 60 , and 70 are classified into a plurality of categories corresponding to characteristics.

[0068] Figure 4 A plurality of source devices classified into a plurality of categories corresponding to characteristics according to an embodiment are shown.

[0069] like Figure 2 and Figure 4As shown, when there are a plurality of source devices 10, 20, 30, 40, 50, 60, and 70 that can be connected to the interface 110, a plurality of categories may be set according to the characteristics of the source devices 10, 20, 30, 40, 50, 60, and 70. The characteristics of the source devices 10, 20, 30, 40, 50, 60, and 70 used to classify the categories may be defined differently, and may include, for example, the device type of the source devices 10, 20, 30, 40, 50, 60, and 70 (e.g., set-top boxes, game consoles, audio / video (AV) devices, etc.), conversion standards of the source devices 10, 20, 30, 40, 50, 60, and 70 (e.g., HDMI, DP, Wi-Fi, etc.), methods of providing content by the source devices 10, 20, 30, 40, 50, 60, and 70 (e.g., live channels, video on demand (VOD), etc.). In addition, various characteristics may be defined. According to an embodiment, device types of the source devices 10 , 20 , 30 , 40 , 50 , 60 , and 70 among the above-mentioned various characteristics will be described by way of example for classifying the categories.

[0070] A plurality of source devices 10, 20, 30, 40, 50, 60, and 70 may be classified into a plurality of categories 410, 420, and 430 according to device types. For example, a first category 410 related to a set-top box, a second category 420 related to a game console, and a third category 430 related to an A / V device may be defined, and the plurality of source devices 10, 20, 30, 40, 50, 60, and 70 belong to any one of the categories 410, 420, and 430 according to device types. For example, the first source device 10, the second source device 20, and the third source device 30 as a set-top box belong to the first category 410. The fourth source device 40 and the fifth source device 50 as a game console belong to the second category 420. The sixth source device 60 and the seventh source device 70 as an A / V device belong to the third category 430. According to an embodiment, one source device 10, 20, 30, 40, 50, 60, or 70 belongs to only one category 410, 420, or 430. Alternatively, one source device 10, 20, 30, 40, 50, 60 or 70 may belong to a plurality of categories 410, 420 and 430. For example, when a predetermined source device functions as both a game console and an A / V device, the source device may belong to both the second category 420 and the third category 430.

[0071] The display apparatus 1 identifies which source devices 10, 20, 30, 40, 50, 60, and 70 are connected thereto at a current point in time from among the plurality of source devices 10, 20, 30, 40, 50, 60, and 70 classified into the plurality of categories as described above. The display apparatus 1 receives device information from the plurality of currently connected source devices 10, 20, 30, 40, 50, 60, and 70, identifies device IDs and characteristics of the corresponding source devices 10, 20, 30, 40, 50, 60, and 70 based on the received device information, and identifies which categories 410, 420, and 430 the source devices 10, 20, 30, 40, 50, 60, and 70 belong to based on the identified characteristics. For example, when it is identified that the first source device 10 among the plurality of source devices 10, 20, 30, 40, 50, 60, and 70 is connected and the first source device 10 is identified as a set-top box, the display apparatus 1 identifies that the first source device 10 belongs to the first category 410.

[0072] The settings of the plurality of categories 410, 420, and 430 corresponding to the characteristics may be stored in various forms such as a table and a list in the storage device 130 of the display apparatus 1. When one of the plurality of preset categories 410, 420, and 430 is selected for displaying a multi-view image, the display apparatus 1 identifies two or more source devices 10, 20, 30, 40, 50, 60, and 70 corresponding to the selected category 410, 420, or 430. The display apparatus 1 may display a UI for allowing a user to select a predetermined category 410, 420, or 430, or may automatically select a category 410, 420, or 430 based on a predetermined condition.

[0073] Hereinafter, the display apparatus 1 receiving the user's selection of the predetermined category 410 , 420 or 430 through the UI will be described by way of example.

[0074] Figure 5 A display device displaying a UI for selecting a category on a screen according to an embodiment is shown.

[0075] like Figure 5 As shown, when a preset input of a user is detected during operation, the display apparatus 1 may display a UI 510 for selecting a plurality of categories according to predefined characteristics of a source device. There is no restriction on the point in time at which the display apparatus 1 may display the UI 510 during operation. The display apparatus 1 may display the UI 510 on the image 520 based on the specific content being displayed. Alternatively, the display apparatus 1 may display the UI 510 on the standby screen when the standby screen is displayed, or may display the UI 510 instead of the standby screen.

[0076] There is no need to classify the plurality of source devices according to categories when only one source device is connected to the display apparatus 1. Therefore, the display apparatus 1 does not display the UI 510 even if a user's input is received.

[0077] The display device 1 displays the options corresponding to multiple categories as objects such as icons 511, 512 and 513 on UI 510. According to an embodiment, UI 510 includes an icon 511 corresponding to a set-top box category, an icon 512 corresponding to a game console category, and an icon 513 corresponding to an A / V device category. Alternatively, icons 511, 512 and 513 may vary depending on the category. According to an embodiment, UI 510 includes icons 511, 512 and 513, but the object corresponding to the category is not limited to icons 511, 512 or 513. Such an object can be realized differently by text, thumbnails, etc.

[0078] When the user selects one of the plurality of icons 511, 512, and 513 through the UI 510, the display apparatus 1 identifies two or more source devices corresponding to the category of the selected icons 511, 512, and 513 from among the plurality of currently connected source devices. For example, when the icon 511 of the set-top box category is selected, the display apparatus 1 identifies two or more source devices corresponding to the set-top box category from among the currently connected source devices.

[0079] The display device 1 receives content from two or more source devices corresponding to the selected category and displays a multi-view image based on the received content. Here, when the number of pieces of content received from the corresponding source device is large, the display device 1 selects a plurality of pieces of content to be displayed as a multi-view image according to a predefined standard. In this regard, an embodiment will be described below.

[0080] Figure 6 A display apparatus for selecting content for multi-view from a plurality of pieces of content received from two or more source devices according to an embodiment is illustrated.

[0081] like Figure 6 As shown, when two or more source devices 10, 20, and 30 corresponding to the selected category are identified, the display apparatus 1 receives a plurality of pieces of content from these source devices 10, 20, and 30. The number of pieces of content received from one of the source devices 10, 20, and 30 may be singular or plural. For example, when the source devices 10, 20, and 30 are multimedia players, the source devices 10, 20, and 30 only output the content currently being played, so the number of pieces of content received from each of the source devices 10, 20, and 30 is singular. On the other hand, when the source devices 10, 20, and 30 are set-top boxes, the source devices 10, 20, and 30 output broadcast streams corresponding to a plurality of broadcast channels, so the number of pieces of content received from each of the source devices 10, 20, and 30 is plural.

[0082] When the number of pieces of content received from two or more source devices 10, 20, and 30 is greater than the number of images to be displayed as the multi-view image 640 by the display apparatus 1, the display apparatus 1 selects content to be displayed as the multi-view image 640 from among the received pieces of content. In this process, the preset content selection criterion 610 is applied.

[0083] The content selection criteria 610 may be specified differently. For example, when the source devices 10, 20, and 30 are set-top boxes that provide broadcast channels, the display apparatus 1 may select a channel or program that is highly preferred by the user of the corresponding display apparatus 1. In this case, the display apparatus 1 may check the preference ranking of the channels or programs based on the usage history 620 of the user having viewed content using the display apparatus 1, and select the channels and programs in the order of the highest preference ranking. The highest ranking may be based on various parameters, such as how many times a particular channel or program has been selected, and how long a particular channel or program has been viewed.

[0084] Alternatively, the display device 1 can select live content from a given plurality of content. The content may include live type content that is unilaterally sent to the display device 1 at a time specified by the content provider, and VOD type content that is obtained and viewed by the user at any time. When the user misses the content broadcast time, the live type content is not viewable, but the VOD type content can be viewed at any time desired by the user. Therefore, the live type content may be selected in consideration of the aforementioned points.

[0085] Alternatively, the display device 1 may select content based on restrictions of audience ratings. The audience ratings are based on various criteria, such as classification of adults and minors, classification of ratings that allow viewing of paid content and ratings that do not allow viewing of paid content, etc. The display device 1 may select content based on the audience rating of the user recorded in the user account information 630.

[0086] Alternatively, the display device 1 may select content based on a preset user recommended channel. Such a user recommended channel may include, for example, a channel recommended by a content provider based on the user's characteristics (e.g., age, gender, nationality, place of residence, work, etc.). The display device 1 may select content suitable for the user's characteristics specified by the content provider based on the user's characteristics recorded in the user account information 630. In addition to the above examples, the display device 1 may select content based on various content selection criteria 610.

[0087] The processor 150 (see Figure 2) may perform at least a portion of data analysis, processing, and result information generation based on at least one of a machine learning, a neural network, or a deep learning algorithm, as a rule-based or artificial intelligence (AI) algorithm, so as to set the content selection criteria 610 as described above, or select content based on the content selection criteria 610. Here, AI is not applied to the aforementioned operations, but may be applied to general operations described in the embodiments of the present disclosure (e.g., an operation of automatically selecting one of a plurality of categories through user input).

[0088] The processor of the display device 1 can be used as a learner and a recognizer. The learner can perform the function of generating a trained neural network, and the recognizer can perform the function of recognizing (or inferring, predicting, estimating and identifying) data based on the trained neural network. The learner can generate or update the neural network. The learner can obtain learning data to generate the neural network. For example, the learner can obtain learning data from a storage device of the display device 1 or from the outside. The learning data can be data for training the neural network, and the data subjected to the aforementioned operation can be used as learning data for training the neural network.

[0089] Before training a neural network based on learning data, the learner may perform a preprocessing operation on the obtained learning data or select data to be used for training from a plurality of learning data. For example, the learner may process the learning data to have a preset format, apply filtering to the learning data, or process the learning data to be suitable for training by adding noise to / removing noise from the learning data. The learner may use the preprocessed learning data to generate a neural network that is configured to perform an operation.

[0090] The learned neural network may include multiple neural networks (or layers). The nodes of multiple neural networks have weight values, and multiple neural networks may be connected to each other so that the output value of a neural network may be used as the input value of another neural network. As examples of neural networks, there are convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), and deep Q networks.

[0091] The recognizer may obtain target data to perform the aforementioned operations. The target data may be obtained from a storage device of the display apparatus 1 or from the outside. The target data may be target data to be recognized by a neural network. Before applying the target data to a trained neural network, the recognizer may perform a preprocessing operation on the obtained target data, or select data to be used for recognition from a plurality of target data. For example, the recognizer may process the target data to have a preset format, filter the target data, or process the target data to be suitable for recognition by adding / removing noise. The recognizer may obtain an output value output from the neural network by applying the preprocessed target data to the neural network. In addition, the recognizer may obtain a random value or a reliability value together with the output value.

[0092] The display apparatus 1 displays a multi-view image 640 based on two or more pieces of content selected according to the content selection criterion 610. The display apparatus 1 may display the multi-view image 640 in various forms. Next, an example of displaying the multi-view image 640 will be described.

[0093] Figure 7 A display device displaying a multi-view image having a layout in the form of a main image and a sub-image according to an embodiment is shown.

[0094] like Figure 7 As shown, the display device 1 displays a multi-view image 700 in which images 710 and 720 based on two or more pieces of content that are recognized are displayed together. According to an embodiment, since multiple pieces of content are recognized, the multi-view image 700 includes four images 710 and 720. However, the number of pieces of content that are recognized may be any natural number not less than 2, so the number of images 710 and 720 included in the multi-view image 700 may vary.

[0095] According to an embodiment, the multi-view image 700 has a layout in the form of a main image and sub-images, and includes a relatively large main image 710 and a plurality of relatively small sub-images 720. There may be various methods of selecting content to be displayed as the main image 710 from among the identified plurality of pieces of content. For example, when the UI 510 (see Figure 5 ) when the image 520 (see Figure 5 ) is identified as a plurality of pieces of content, the display device 1 displays the corresponding identified content as the main image 710. Alternatively, when identifying content based on ranking, the display device 1 may display the top-ranked content as the main image 710.

[0096] The multiple sub-images 720 may have the same size in an embodiment, but may be designed to be different in size from each other. For example, the sizes of the multiple sub-images 720 may correspond to the ranking of the identified content. When the content from the first position to the fourth position in the user preference ranking is identified, the display device 1 may display the content corresponding to the first position in the ranking as the main image 710, and set the size of the content image corresponding to the second position in the multiple sub-images 720 to be larger than the size of the content image corresponding to the third position or the fourth position.

[0097] In the multi-view image 700, the positions of the main image 710 and the sub-images 720 may be designated differently. In the multi-view image 700, the display device 1 according to the embodiment may display the main image 710 on the left, and display a plurality of sub-images 720 on the right. The display device 1 may display the main image 710 at various positions such as right, top, left, and center positions in the multi-view image 700, and may appropriately position the sub-images 720 according to the position of the main image 710.

[0098] The display device 1 may play back all of the multiple pieces of content as images 710 and 720 within the multi-view image 700, or may play back multiple pieces of content as part of the image, for example, only the content corresponding to the main image 710. The display device 1 displays a mark indicating playback on the images 710 and 720 currently being played back. In addition, when content with audio is played back at the same time, the user may be confused. Therefore, the display device 1 plays back the audio of the content corresponding to any one of the images in the multi-view image 700, for example, the audio of the content corresponding to the main image 710, and displays a mark 711 indicating audio playback on the main image 710. According to an embodiment, the mark 711 can notify the user that not only the main image 710 is being played back but also the sound of the main image 710 is output.

[0099] Figure 8 A display device displaying a multi-view image having a layout in the form of a balanced image according to an embodiment is shown.

[0100] like Figure 8 As shown, the display device 1 displays a multi-view image 800 in which images 810 are displayed together based on the two or more pieces of content identified. According to an embodiment, the multi-view image 800 has a layout in the form of a balanced image in which all images 810 have the same size. According to an embodiment, four images 810 are displayed within the multi-view image 800, and these images 810 have the same size and are symmetrically arranged within the multi-view image 800. Of course, the number, size, and position of the images 810 within the multi-view image 800 are merely examples and are not limited thereto.

[0101] The display apparatus 1 may display a mark 811 indicating playback or sound output on an image 810 that is currently being played back or currently outputting sound from among the plurality of images 810 within the multi-view image 800 .

[0102] because Figure 7 The layout of the multi-view image 700 shown is different from Figure 8 The layout of the multi-view image 800 shown in FIG. 8 is a layout of the multi-view image 800, so a more appropriate layout can be applied. Hereinafter, the selection of the layout by the display device 1 will be described by way of example.

[0103] Fig. 9 is a flowchart illustrating that a display device selects a layout for a multi-view image according to an embodiment.

[0104] like Figure 2 and Fig. 9 As shown, the following operations are performed by the processor 150 of the display device 1.

[0105] At operation 910 , the display apparatus 1 receives a selection of one of a plurality of categories related to characteristics of the source devices 10 , 20 , 30 , 40 , 50 , 60 , and 70 .

[0106] At operation 920 , the display apparatus 1 identifies two or more source devices 10 , 20 , 30 , 40 , 50 , 60 , and 70 corresponding to the selected category among the plurality of source devices connected to the interface 110 .

[0107] At operation 930 , the display apparatus 1 identifies a layout of a multi-view image corresponding to the selected category.

[0108] At operation 940 , the display apparatus 1 receives content from two or more source devices 10 , 20 , 30 , 40 , 50 , 60 , and 70 .

[0109] At operation 950 , the display apparatus 1 displays a multi-view image based on the two or more pieces of received content and the recognized layout.

[0110] For example, the aforementioned operation can be implemented as follows. Figure 7 In the layout of the multi-view image 700 shown in FIG. 1 , the main image 710 is relatively large, so it is relatively easy to notice the main image 710 even when several people simultaneously view the multi-view image 700. Therefore, this layout is suitable for motion picture content or broadcast content. Figure 8In the layout of the multi-view image 800 shown, the multiple images 810 have the same size. Therefore, this layout is suitable for a situation where multiple users pay attention to different images 810, for example, for multi-player game content. When multiple game consoles are connected to the display device 1, the display device 1 can display the images 810 corresponding to the game consoles in the multi-view image 800. Therefore, the users of the game consoles can play games while watching the images 810 corresponding to their game consoles.

[0111] For example, when the category selected from the plurality of categories is related to the characteristics of the set-top box, the display device 1 selects Figure 7 On the other hand, when the category selected from the plurality of categories is related to the characteristics of the game console, the display device 1 selects the category as shown in FIG. Figure 8 The layout of the multi-view image 800 is shown. In this way, the display device 1 uses a layout suitable for the selected category, thereby providing a more optimized multi-view image.

[0112] When the source devices 10 , 20 , 30 , 40 , 50 , 60 , and 70 are game consoles that require cooperation between servers, the display apparatus 1 may perform the following operations.

[0113] Fig.10 , Fig.11 , Fig.12 and Fig.13 A multi-view image displayed by a display device connected to two game consoles according to an embodiment is shown.

[0114] like Fig.10 As shown, when the category of game console is selected, the display apparatus 1 identifies game console A 1001 and game console B 1002 among a plurality of source devices connected thereto. According to an embodiment, two users of game console A 1001 and game console B 1002 intend to play a multiplayer game by cooperating with server 1003.

[0115] When game console A 1001 and game console B 1002 are connected to display device 1, display device 1 first displays a multi-view image including a standby image 1010 of game console A 1001 and a standby image 1020 of game console B 1002. At this time, a game is not yet executed in either of game console A 1001 and game console B 1002.

[0116] The display device 1 obtains user account information from each of the game console A 1001 and the game console B 1002, and based on the information, connects to the server 1003. The game console A 1001 and the game console B 1002 respectively log in to the server 1003 using their own accounts. Therefore, the server 1003 recognizes that the game console A 1001 and the game console B 1002 are connected thereto.

[0117] Next, a case where a user of the game console A 1001 executes a predetermined game in the game console A 1001 will be described.

[0118] like Fig.11 As shown, when a predetermined game is played in the game console A 1001, the game console A 1001 transmits game data of the corresponding game to the display device 1. The display device 1 displays a game image 1030 of the game console A in a multi-view image based on the game data received from the game console A 1001. On the other hand, no game data is received from the game console B 1002, and the display device 1 keeps displaying the standby image of the game console B.

[0119] The display device 1 sends predetermined information based on the game data received from the game console A 1001 to the server 1003. Here, the game data received by the display device 1 from the game console A 1001 may be the same as or different from the information sent by the display device 1 to the server 1003, but the information is at least based on the game data. For example, the game data sent by the game console A 1001 to the display device 1 may include information to be sent to the server 1003 to play the game, and information required for the display device 1 to display the game play image 1030 but not to be sent to the server 1003. In this case, the display device 1 can identify some information designated to be sent to the server 1003 from the received game data, and send the identified information to the server 1003. This information may include, for example, the ID of the game running in the game console A 1001.

[0120] Therefore, the server 1003 can identify which game is currently running in the game console A 1001.

[0121] In terms of managing user accounts and games, the server 1003 may identify multiple user accounts that are set to multiplayer games for a particular game. For example, the server 1003 may identify that multiplayer of a game currently running in game console A 1001 is set to be allowed in game console B 1002. In this regard, the operation of the server 1003 will be described below.

[0122] like Fig.12As shown, when it is identified that a predetermined game is being run in game console A 1001, server 1003 identifies whether a multiplayer game of the corresponding game is allowed in game console B 1002. Alternatively, when game console A 1001 is logged in using the account of user A and game console B 1002 is logged in using the account of user B, server 1003 may identify whether user B can play a multiplayer game of the corresponding game with user A. Alternatively, server 1003 may search for user B in a list of user accounts that are set to play a multiplayer game of the corresponding game with user A.

[0123] When it is recognized that a multiplayer game in which the corresponding game is not allowed to be played in the game console B 1002, the Fig.11 On the other hand, when it is recognized that a multi-player game of the corresponding game is allowed in the game console B 1002, the server 1003 sends a message to the display device 1 to guide the user B 1002 of the game console B to play the multi-player game of the corresponding game with the user A. Therefore, the display device 1 displays the multi-view image including the message 1040 together with the game image 1030 of the game console A. Alternatively, the server 1003 may not send a message to the display device 1, but may send an instruction for executing the corresponding game to the display device 1. In this case, the display device 1 may send an instruction for executing the game to the game console B 1002, and the game console B 1002 may automatically execute the game in response to the received instruction.

[0124] In this way, user B can recognize that user A is playing the corresponding game through the message 1040 displayed on the display device 1.

[0125] like Fig.13 As shown, the user B can recognize that the user A is playing a predetermined game through the guidance on the multi-view image displayed on the display device 1, and execute the guided game in the game console B 1002.

[0126] The game console B 1002 transmits game data of the corresponding game to the display device 1, and the display device 1 displays a multi-view image including an image 1030 of the game running in the game console A 1001 and an image 1050 of the game running in the game console B 1002. In addition, the display device 1 transmits information based on the game data obtained from each of the game consoles A 1001 and the game console B 1002 to the server 1003. Therefore, the server 1003 can relay and control a multi-player game of the user A and the user B for the game running in each of the game consoles A 1001 and the game console B 1002 based on the received information.

[0127] According to the aforementioned embodiment, specifically, Figure 5 The display device 1 is shown to display a UI 510, which includes icons 511, 512, and 513 corresponding to a plurality of predefined categories and provides an option for selecting one of the categories. When the user selects one of the icons 511, 512, and 513, two or more source devices corresponding to the category of the selected icon 511, 512, or 513 are identified. Alternatively, the user can personally select the source device of each category and store the result of the selection as a setting, and the display device 1 can provide the setting to the user as an option. In this regard, an embodiment will be described below.

[0128] Fig.14 It is shown that a user selects a source device among a plurality of predefined categories in a display apparatus according to an embodiment.

[0129] like Fig.14 As shown, a plurality of source devices 10, 20, 30, 40, 50, 60, and 70 are pre-set to correspond to a plurality of categories 410, 420, and 430. For example, the first source device 10, the second source device 20, and the third source device 30 belong to the first category 410, the fourth source device 40 and the fifth source device 50 belong to the second category 420, and the sixth source device 60 and the seventh source device 70 belong to the third category 430.

[0130] The user may randomly select two or more source devices 10, 20, 30, 40, 50, 60, and 70 from among the source devices 10, 20, 30, 40, 50, 60, and 70 classified according to the categories as described above. For example, the user may select the second source device 20 corresponding to the first category 410, the fourth source device 40 corresponding to the second category 420, and the seventh source device 70 corresponding to the third category 430 from among the source devices 10, 20, 30, 40, 50, 60, and 70 connected to the display apparatus 1. The result of this selection will be referred to as the user setting category 1400.

[0131] The display device 1 stores this newly generated user setting category 1400 as user setting category information 1410 .

[0132] Fig.15 A user setting icon included in a UI for selection of a multi-view image according to an embodiment is illustrated.

[0133] like Fig.14 and Fig.15 As shown, the display apparatus 1 displays a UI 510 provided to select a category. The UI 510 includes a plurality of icons 511, 512, and 513 corresponding to categories related to characteristics of the source device. Figure 5 The configurations in the aforementioned embodiments shown are basically the same.

[0134] According to an embodiment, the UI 510 further includes a user setting icon 1500. The user setting icon 1500 corresponds to the user setting category 1400 indicating two or more source devices selected by the user, and is displayed based on the user setting category information 1410. The display apparatus 1 refers to the user setting category information 1410 when displaying the UI 510, and does not display the user setting icon 1500 on the UI 510 when the user setting category information 1410 is not stored.

[0135] When the user setting icon 1500 is selected on the UI 510, the display apparatus 1 identifies two or more source devices corresponding to the user setting category 1400 indicated by the user setting category information 1410, and displays a multi-view image based on the content from the identified two or more source devices. For example, when the user setting category 1400 corresponds to the second source device 20, the fourth source device 40, and the seventh source device 70, the display apparatus 1 may display a multi-view image based on the content received from the second source device 20, the fourth source device 40, and the seventh source device 70.

[0136] The display apparatus 1 may change the options of the categories selectable by the user on the UI 510 according to various conditions. In this regard, an embodiment will be described below.

[0137] Fig.16 A display device that changes category selection options provided in a UI according to preset conditions according to an embodiment is shown.

[0138] like Fig.16 As shown, the display device 1 can select the categories of multiple icons 1601, 1602, 1603, 1604 and 1605 based on the conditions of the current use environment when multiple icons 1601, 1602, 1603, 1604 and 1605 are displayed on the UI 1600. Here, various conditions are applicable. As an example, the display device 1 can select a category based on the characteristics of the content currently being played back (i.e., the content of the image 1610 currently being displayed). The characteristics of the content can be achieved, for example, by various factors such as content types (sports, dramas, news, movies, etc.) and content types (live broadcast, VOD, etc.). The characteristics of the content can be obtained from the metadata provided with the content or the content-related information (electronic program guide (EPG) etc.) provided through various paths.

[0139] This method may be applied when the number of icons 1601, 1602, 1603, 1604, and 1605 displayed on the UI 1600 is less than the number of predefined categories 1620. For example, when five icons 1601, 1602, 1603, 1604, and 1605 are displayable on the UI 1600 and ten categories 1620 from the first category to the tenth category are predefined, the display device 1 selects five categories 1630 based on the characteristics of the content of the image 520 currently being displayed from the ten categories 1620, and displays the icons 1601, 1602, 1603, 1604, and 1605 corresponding to the selected five categories 1630 on the UI 1600.

[0140] As an example according to the embodiment, the case where the content of the image 1610 currently being displayed is a sports game will be considered. The display device 1 can select the category of the set-top box for providing sports broadcasts in the same time period as the content of the image 1610 and the category of the game console for playing games based on the characteristics of the content of the image 1610 (i.e., two factors of "sports" and "games").

[0141] According to an alternative embodiment, when multiple pieces of content are received from two or more source devices corresponding to the category selected through UI 1600, display apparatus 1 may select two or more pieces of content from the received multiple pieces of content based on characteristics of the content of image 1610 currently being displayed to display as a multi-view image.

[0142] For example, it is assumed that the display apparatus 1 selects the category of the set-top box through the UI 1600, and the image 1610 currently being displayed is a live sports broadcast. In this case, the display apparatus 1 can select a channel corresponding to the live sports broadcast in the current period from the broadcast receiving channels of the two or more source devices corresponding to the set-top box, thereby displaying the content image of the selected channel as a multi-view image.

[0143] The operation of the device described in the foregoing embodiments can be performed by artificial intelligence installed in the device. Artificial intelligence can be applied to various systems based on machine learning algorithms. An artificial intelligence system refers to a computer system that achieves human-level intelligence or close to human-level intelligence, in which the machine, device or system learns and makes decisions autonomously, and improves the recognition rate and decision accuracy based on accumulated usage experience. Artificial intelligence technology is based on basic technology, using machine learning technology and algorithms, and using algorithms that autonomously classify / learn features of input data to replicate the perception, judgment and other functions of the human brain.

[0144] The basic technologies may include, for example, language understanding technology for recognizing human language / text, visual understanding technology for recognizing objects like human vision, deduction / prediction technology for recognizing information and logically deducing and predicting, knowledge representation technology for processing human experience information into knowledge data, and at least one of motion control technology for controlling vehicle autonomous driving or robot movement.

[0145] Here, language understanding refers to the technology of recognizing, applying and processing human language or text, and includes natural language processing, machine translation, conversational systems, question and answer, speech recognition and synthesis, etc.

[0146] Inference / prediction refers to the technology of identifying information and logically making predictions, and includes knowledge and probability-based inferences, optimized predictions, preference-based plans, recommendations, etc.

[0147] Knowledge representation refers to a technology that automates human experience information into knowledge data, and includes knowledge construction such as data creation and classification, knowledge management such as data utilization, and the like.

[0148] The method according to the foregoing embodiment can be implemented in the form of program instructions, which can be implemented in various computers and recorded in a computer-readable medium. Such a computer-readable medium may include program instructions, data files, data structures, etc. or a combination thereof. For example, a computer-readable medium may be stored in a non-volatile storage unit such as a universal serial bus (USB) memory, regardless of whether it is deletable or rewritable, such as a RAM, ROM, flash memory, memory chip, integrated circuit (IC) and other memory, or may be stored in an optical or magnetic recordable or machine (e.g., computer) readable storage unit medium, such as a compact disk (CD), a digital versatile disk (DVD), a disk, a magnetic tape, etc. It should be understood that the memory that may be included in the mobile terminal is an example of a machine-readable storage unit medium suitable for storing a program, and the program has instructions for implementing the embodiment. The program instructions recorded in the storage unit medium may be specially designed and configured according to the embodiment, or may be known and available to a technician in the field of computer software. In addition, the computer program instructions may be implemented by a computer program product.

Claims

1. A display device, comprising: monitor; interface; User input; as well as A processor configured to execute instructions to: selecting, based on the user input, a category from a plurality of categories related to characteristics of a plurality of source devices capable of connecting to the interface; identifying two or more source devices corresponding to the selected category among a plurality of source devices connected to the interface; receiving, via the interface, a plurality of pieces of content from the identified two or more source devices; as well as Based on the selected category, a multi-image is displayed on the display based on two or more of the plurality of pieces of content.

2. The display device according to claim 1, wherein: The plurality of categories corresponds to a plurality of types associated with the plurality of source devices.

3. The display device according to claim 1, wherein: The processor is further configured to execute the instructions to identify characteristics of the plurality of source devices based on device information received from the plurality of source devices.

4. The display device according to claim 1, wherein: The processor is further configured to execute the instructions to: based on identifying that at least two source devices are connected to the interface, display a user interface (UI) on the display to select the category from the plurality of categories.

5. The display device according to claim 4, wherein: The processor is further configured to execute the instructions to not display the UI based on identifying that only one source device is connected to the interface.

6. The display device according to claim 4, wherein: The processor is further configured to execute the instructions to identify a selectable category from the plurality of categories on the UI based on characteristics of content being displayed on the display.

7. The display device according to claim 1, wherein: The processor is further configured to execute instructions to: storing user setting options including at least a first source device and a second source device selected by a user based on identifying that at least two source devices are connected to the interface; as well as The user setting option is identified as a selectable category among the plurality of categories.

8. The display device according to claim 1, wherein: The processor is further configured to execute the instructions to identify the two or more pieces of content from the plurality of pieces of content based on a descending order of the content relevance rankings of the user.

9. The display device according to claim 8, wherein: The content-related ranking of the user is related to the content preference of the user.

10. The display device according to claim 9, wherein: The processor is further configured to execute the instructions to identify the content relevance ranking based on the content usage history of the user.

11. The display device according to claim 1, wherein: The processor is further configured to execute the instructions to identify the two or more pieces of content from the plurality of pieces of content based on an audience rating of a user.

12. The display device according to claim 11, wherein: The processor is further configured to execute the instructions to: identify the audience level based on user account information of the display device.

13. The display device according to claim 1, wherein: The processor is further configured to execute instructions to: identifying a layout corresponding to the selected category from a plurality of layouts associated with a mode for displaying the multiple images; as well as The multiple images are displayed on the display based on the identified layout.

14. The display device according to claim 13, wherein: The processor is further configured to execute the instructions to display, on the display, the multi-image including a main image and a sub-image smaller than the main image based on the two or more pieces of content.

15. A method for controlling a display device, the method comprising: selecting, based on user input, a category from a plurality of categories associated with characteristics of a plurality of source devices connectable to an interface of the display apparatus; identifying two or more source devices corresponding to the selected category among a plurality of source devices connected to the interface; receiving, via the interface, a plurality of pieces of content from the identified two or more source devices; as well as Based on the selected category, a multi-image is displayed on a display of the display device based on two or more pieces of content among the plurality of pieces of content.

Images