Display device and operating method thereof

Abstract

A display device according to an embodiment of the present disclosure may comprise: a memory; a display that plays a video; and a controller that extracts one or more objects from the video being played, stores, in the memory, one or more object items corresponding to the extracted one or more objects, displays the one or more object items on the display according to an exposure command, receives a selection command for selecting any one of the one or more object items, acquires a search result for an object item selected according to the received selection command, and outputs the acquired search result.

Description

Display device and method of operation thereof

[0001] The present disclosure relates to a display device, and more particularly, to a display device that extracts an object during playback of a video and provides a search result of the extracted object.

[0002] Digital TV services utilizing wired or wireless networks are becoming more widespread. Digital TV services can offer a variety of services not available with existing analog broadcasting services.

[0003] For example, IPTV (Internet Protocol Television), a type of digital TV service, and smart TV services offer interactivity, allowing users to actively choose the type of program they want to watch and when. Building on this interactivity, IPTV and smart TV services can also offer a variety of additional services, such as internet search, home shopping, and online games.

[0004] When viewers become curious about characters, props, or locations featured in TV content, they often use a separate device to search for them. This process, rather than directly finding what they want, forces viewers to resort to indirect searches based on various background information, such as the title or broadcast date of the current program, which creates the inconvenience of having to rely on such information.

[0005] Additionally, text-based searches present a challenge in that the search itself is impossible if the desired information is not disclosed in text.

[0006] Furthermore, existing information retrieval services rely on pre-acquired metadata for each video scene, making integration impossible in the absence of metadata. In this case, searches must be conducted solely based on the viewer's voice, making it difficult to determine when the viewer is referring to an object and what exactly the object is.

[0007] The purpose of the present disclosure may be to extract and store appropriate objects in advance from a screen being viewed using AI technology, and to provide search results for objects at any time according to a viewer's request.

[0008] The purpose of the present disclosure may be to perform object extraction in real time on-device, thereby avoiding unnecessary server costs in the remaining processes except for the final web search.

[0009] The purpose of the present disclosure may be to eliminate inconvenience to consumers and provide a more easy and accurate search service by enabling the viewer to access the necessary information by performing an image search when selecting a desired object or information on the screen.

[0010] A display device according to an embodiment of the present disclosure may include a memory, a display for playing a video, and a controller for extracting one or more objects from the playing video, storing one or more object items corresponding to the extracted one or more objects in the memory, displaying the one or more object items on the display according to a display command, receiving a selection command for selecting one of the one or more object items, obtaining a search result of an object item selected according to the received selection command, and outputting the obtained search result.

[0011] A method of operating a display device according to an embodiment of the present disclosure may include a step of playing a video, a step of extracting one or more objects from the video being played, a step of storing one or more object items corresponding to the one or more extracted objects, a step of displaying the one or more object items according to a display command, a step of receiving a selection command for selecting one of the one or more object items, a step of obtaining a search result of an object item selected according to the received selection command, and a step of outputting the obtained search result.

[0012] According to an embodiment of the present disclosure, a user can quickly receive search results for an object he or she wants to search for while watching a video, thereby greatly improving user convenience.

[0013] Additionally, according to the embodiments of the present disclosure, the usability of the TV system as a platform operator can be increased.

[0014] Additionally, according to an embodiment of the present disclosure, since images are extracted directly from the on-device (display device, 100) instead of accessing a continuous server, the server cost used for extracting objects can be reduced.

[0015] Additionally, according to an embodiment of the present disclosure, issues such as copyright can be alleviated by handling images on-device.

[0016] In addition, according to the embodiment of the present disclosure, there is an advantage in that search linkage is possible in any content without inserting metadata.

[0017] Figure 1 is a block diagram illustrating the configuration of a display device according to one embodiment of the present invention.

[0018] Figure 2 is a block diagram of a remote control device according to one embodiment of the present invention.

[0019] Figure 3 shows an example of an actual configuration of a remote control device according to one embodiment of the present invention.

[0020] Figure 4 shows an example of utilizing a remote control device according to an embodiment of the present invention.

[0021] FIG. 5 is a diagram for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0022] FIG. 6 is a flowchart illustrating an operating method of a display device according to an embodiment of the present disclosure.

[0023] FIG. 7 is a flowchart illustrating an operating method of a display device according to an embodiment of the present disclosure.

[0024] FIGS. 8A to 8F are diagrams illustrating a process for providing a search result for a desired object while a user is watching a video according to an embodiment of the present disclosure.

[0025] FIGS. 9A to 9D are drawings illustrating an embodiment in which a viewer selects an object while watching a video and search results for the selected object are provided.

[0026] Hereinafter, embodiments related to the present invention will be described in more detail with reference to the drawings. The suffixes "module" and "part" used in the following description for components are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles.

[0027] A display device according to an embodiment of the present invention is, for example, an intelligent display device that adds computer-assisted functionality to its broadcast reception function. While faithfully performing the broadcast reception function, it can also be equipped with Internet functionality and other features, providing a more user-friendly interface, such as a manual input device, touch screen, or space remote control. Furthermore, with support for wired or wireless Internet functionality, it can connect to the Internet and a computer, enabling functions such as email, web browsing, banking, or gaming. A standardized, general-purpose operating system can be used for these various functions.

[0028] Accordingly, the display device described in the present invention can perform various user-friendly functions, for example, by allowing various applications to be freely added or deleted on a general-purpose operating system kernel. More specifically, the display device can be a network TV, HBB TV, smart TV, LED TV, OLED TV, etc., and in some cases, it can also be applied to smartphones.

[0029] FIG. 1 is a block diagram illustrating the configuration of a display device according to one embodiment of the present invention.

[0030] Referring to FIG. 1, the display device (100) may include a broadcast receiving unit (130), an external device interface (135), a memory (140), a user input interface (150), a controller (170), a wireless communication interface (173), a display (180), a speaker (185), and a power supply circuit (190).

[0031] The broadcast receiving unit (130) may include a tuner (131), a demodulator (132), and a network interface (133).

[0032] The tuner (131) can select a specific broadcast channel according to a channel selection command. The tuner (131) can receive a broadcast signal for the selected specific broadcast channel.

[0033] The demodulator (132) can separate the received broadcast signal into a video signal, an audio signal, and a data signal related to the broadcast program, and can restore the separated video signal, audio signal, and data signal into a form that can be output.

[0034] The external device interface (135) can receive an application or a list of applications within an adjacent external device and transmit it to the controller (170) or memory (140).

[0035] The external device interface (135) can provide a connection path between the display device (100) and the external device. The external device interface (135) can receive one or more of images and audio output from an external device connected wirelessly or wiredly to the display device (100) and transmit them to the controller (170). The external device interface (135) can include a plurality of external input terminals. The plurality of external input terminals can include an RGB terminal, one or more HDMI (High Definition Multimedia Interface) terminals, and a component terminal.

[0036] A video signal of an external device input through an external device interface (135) can be output through a display (180). A voice signal of an external device input through an external device interface (135) can be output through a speaker (185).

[0037] An external device that can be connected to the external device interface (135) may be any one of a set-top box, a Blu-ray player, a DVD player, a game console, a sound bar, a smartphone, a PC, a USB memory, and a home theater, but this is only an example.

[0038] The network interface (133) may provide an interface for connecting the display device (100) to a wired / wireless network including the Internet. The network interface (133) may transmit or receive data to or from other users or other electronic devices via the connected network or another network linked to the connected network.

[0039] Additionally, some content data stored in the display device (100) can be transmitted to a selected user or electronic device among other users or other electronic devices pre-registered in the display device (100).

[0040] The network interface (133) can access a predetermined web page through a connected network or another network linked to the connected network. That is, by accessing a predetermined web page through a network, data can be transmitted or received with the corresponding server.

[0041] In addition, the network interface (133) can receive content or data provided by a content provider or network operator. That is, the network interface (133) can receive content such as movies, advertisements, games, VOD, broadcast signals, etc. and information related thereto provided from a content provider or network provider via a network.

[0042] Additionally, the network interface (133) can receive firmware update information and update files provided by the network operator, and can transmit data to the Internet or content provider or network operator.

[0043] The network interface (133) can select and receive a desired application from among applications open to the public via a network.

[0044] The memory (140) stores a program for each signal processing and control within the controller (170), and can store signal-processed image, voice, or data signals.

[0045] In addition, the memory (140) may perform a function for temporary storage of video, audio, or data signals input from an external device interface (135) or a network interface (133), and may store information about a specific image through a channel memory function.

[0046] The memory (140) can store an application or a list of applications input from an external device interface (135) or a network interface (133).

[0047] The display device (100) can play content files (video files, still image files, music files, document files, application files, etc.) stored in the memory (140) and provide them to the user.

[0048] The user input interface (150) can transmit a signal input by the user to the controller (170) or transmit a signal from the controller (170) to the user. For example, the user input interface (150) can receive and process control signals such as power on / off, channel selection, and screen setting from the remote control device (200) according to various communication methods such as Bluetooth, Ultra Wideband (WB), ZigBee, Radio Frequency (RF) communication, or infrared (IR) communication, or process control signals from the controller (170) to be transmitted to the remote control device (200).

[0049] In addition, the user input interface (150) can transmit control signals input from local keys (not shown) such as a power key, channel key, volume key, and setting value to the controller (170).

[0050] An image signal processed by the controller (170) can be input to the display (180) and displayed as an image corresponding to the image signal. In addition, an image signal processed by the controller (170) can be input to an external output device through an external device interface (135).

[0051] The voice signal processed by the controller (170) can be output as audio to the speaker (185). In addition, the voice signal processed by the controller (170) can be input to an external output device through the external device interface (135).

[0052] In addition, the controller (170) can control the overall operation within the display device (100).

[0053] In addition, the controller (170) can control the display device (100) by a user command or internal program input through the user input interface (150), and can connect to a network to enable the user to download a desired application or application list into the display device (100).

[0054] The controller (170) enables the user-selected channel information, etc. to be output through a display (180) or speaker (185) together with processed video or audio signals.

[0055] In addition, the controller (170) allows a video signal or audio signal from an external device, for example, a camera or camcorder, input through the external device interface (135) to be output through the display (180) or speaker (185) in accordance with an external device video playback command received through the user input interface (150).

[0056] Meanwhile, the controller (170) can control the display (180) to display an image, for example, a broadcast image input through a tuner (131), an external input image input through an external device interface (135), an image input through a network interface, or an image stored in a memory (140) can be controlled to be displayed on the display (180). In this case, the image displayed on the display (180) can be a still image or a moving image, and can be a 2D image or a 3D image.

[0057] In addition, the controller (170) can control the playback of content stored in the display device (100), received broadcast content, or external input content input from outside, and the content can be in various forms such as broadcast video, external input video, audio file, still image, connected web screen, and document file.

[0058] The wireless communication interface (173) can communicate with an external device through wired or wireless communication. The wireless communication interface (173) can perform short-range communication with the external device. To this end, the wireless communication interface (173) can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies. This wireless communication interface (173) can support wireless communication between the display device (100) and a wireless communication system, between the display device (100) and another display device (100), or between the display device (100) and a network in which the display device (100, or an external server) is located via a short-range wireless communication network (Wireless Area Network). The short-range wireless communication network can be a short-range wireless personal area network (Wireless Personal Area Network).

[0059] Here, the other display device (100) may be a wearable device (e.g., a smartwatch, smart glasses, a head-mounted display (HMD)) or a mobile terminal such as a smart phone that can exchange data with (or be linked to) the display device (100) according to the present invention. The wireless communication interface (173) may detect (or recognize) a wearable device capable of communication around the display device (100).

[0060] Furthermore, if the detected wearable device is a device certified to communicate with the display device (100) according to the present invention, the controller (170) can transmit at least a portion of the data processed in the display device (100) to the wearable device via the wireless communication interface (173). Accordingly, a user of the wearable device can utilize the data processed in the display device (100) via the wearable device.

[0061] The display (180) can generate a driving signal by converting a video signal, data signal, OSD signal processed by the controller (170) or a video signal, data signal, etc. received from an external device interface (135) into R, G, and B signals, respectively.

[0062] Meanwhile, since the display device (100) illustrated in FIG. 1 is merely an embodiment of the present invention, some of the illustrated components may be integrated, added, or omitted depending on the specifications of the display device (100) actually implemented.

[0063] That is, two or more components may be combined into a single component, or a single component may be subdivided into two or more components, as needed. Furthermore, the functions performed by each block are intended to illustrate embodiments of the present invention, and their specific operations or devices do not limit the scope of the present invention.

[0064] According to another embodiment of the present invention, the display device (100) may receive and play back an image through a network interface (133) or an external device interface (135) without having a tuner (131) and a demodulator (132), unlike that shown in FIG. 1.

[0065] For example, the display device (100) may be implemented separately as an image processing device, such as a set-top box, for receiving broadcast signals or contents according to various network services, and a content playback device for playing contents input from the image processing device.

[0066] In this case, the operating method of the display device according to the embodiment of the present invention to be described below may be performed by any one of the display device (100) described with reference to FIG. 1, as well as an image processing device such as the separated set-top box, or a content playback device having a display (180) and a speaker (185).

[0067] Next, a remote control device according to an embodiment of the present invention will be described with reference to FIGS. 2 and 3.

[0068] FIG. 2 is a block diagram of a remote control device according to an embodiment of the present invention, and FIG. 3 shows an example of an actual configuration of a remote control device (200) according to an embodiment of the present invention.

[0069] First, referring to FIG. 2, the remote control device (200) may include a fingerprint recognition device (210), a wireless communication circuit (220), a user input interface (230), a sensor (240), an output interface (250), a power supply circuit (260), a memory (270), a controller (280), and a microphone (290).

[0070] Referring to FIG. 2, the wireless communication circuit (220) transmits and receives signals with any one of the display devices according to the embodiments of the present invention described above.

[0071] The remote control device (200) may be equipped with an RF circuit (221) capable of transmitting and receiving signals with the display device (100) according to RF communication standards, and an IR circuit (223) capable of transmitting and receiving signals with the display device (100) according to IR communication standards. In addition, the remote control device (200) may be equipped with a Bluetooth circuit (225) capable of transmitting and receiving signals with the display device (100) according to Bluetooth communication standards. In addition, the remote control device (200) may be equipped with an NFC circuit (227) capable of transmitting and receiving signals with the display device (100) according to NFC (Near Field Communication) communication standards, and a WLAN circuit (229) capable of transmitting and receiving signals with the display device (100) according to WLAN (Wireless LAN) communication standards.

[0072] In addition, the remote control device (200) transmits a signal containing information about the movement of the remote control device (200) to the display device (100) through a wireless communication circuit (220).

[0073] Meanwhile, the remote control device (200) can receive a signal transmitted by the display device (100) through the RF circuit (221), and, if necessary, can transmit commands for turning the power on / off, changing the channel, changing the volume, etc. to the display device (100) through the IR circuit (223).

[0074] The user input interface (230) may be configured as a keypad, buttons, a touchpad, or a touch screen. The user can input commands related to the display device (100) to the remote control device (200) by operating the user input interface (230). If the user input interface (230) includes a hard key button, the user can input commands related to the display device (100) to the remote control device (200) by pushing the hard key button. This will be described with reference to FIG. 3.

[0075] Referring to FIG. 3, the remote control device (200) may include a plurality of buttons. The plurality of buttons may include a fingerprint recognition button (212), a power button (231), a home button (232), a live button (233), an external input button (234), a volume control button (235), a voice recognition button (236), a channel change button (237), a confirmation button (238), and a back button (239).

[0076] The fingerprint recognition button (212) may be a button for recognizing a user's fingerprint. In one embodiment, the fingerprint recognition button (212) may be capable of a push operation, and may receive a push operation and a fingerprint recognition operation.

[0077] The power button (231) may be a button for turning the power of the display device (100) on / off.

[0078] The home button (232) may be a button for moving to the home screen of the display device (100).

[0079] The live button (233) may be a button for displaying a real-time broadcast program.

[0080] The external input button (234) may be a button for receiving an external input connected to the display device (100).

[0081] The volume control button (235) may be a button for adjusting the size of the volume output by the display device (100).

[0082] The voice recognition button (236) may be a button for receiving a user's voice and recognizing the received voice.

[0083] The channel change button (237) may be a button for receiving a broadcast signal of a specific broadcast channel.

[0084] The confirmation button (238) may be a button for selecting a specific function, and the back button (239) may be a button for returning to the previous screen.

[0085] Let's explain Figure 2 again.

[0086] When the user input interface (230) has a touch screen, the user can input commands related to the display device (100) using the remote control device (200) by touching the soft keys of the touch screen. In addition, the user input interface (230) may have various types of input means that can be operated by the user, such as a scroll key or a jog key, and this embodiment does not limit the scope of the present invention.

[0087] The sensor (240) may include a gyro sensor (241) or an acceleration sensor (243), and the gyro sensor (241) may sense information about the movement of the remote control device (200).

[0088] For example, the gyro sensor (241) can sense information about the operation of the remote control device (200) based on the x, y, and z axes, and the acceleration sensor (243) can sense information about the movement speed of the remote control device (200). Meanwhile, the remote control device (200) can further include a distance measuring sensor, so as to sense the distance to the display (180) of the display device (100).

[0089] The output interface (250) can output a video or audio signal corresponding to an operation of the user input interface (230) or a signal transmitted from the display device (100).

[0090] The user can recognize whether the output interface (250) is manipulating the user input interface (230) or controlling the display device (100).

[0091] For example, the output interface (250) may include an LED (251) that lights up when the user input interface (230) is operated or a signal is transmitted and received with the display device (100) via the wireless communication unit (225), a vibrator (253) that generates vibrations, a speaker (255) that outputs sound, or a display (257) that outputs images.

[0092] In addition, the power supply circuit (260) supplies power to the remote control device (200), and power waste can be reduced by stopping the power supply when the remote control device (200) does not move for a predetermined period of time.

[0093] The power supply circuit (260) can resume power supply when a predetermined key provided in the remote control device (200) is operated.

[0094] The memory (270) can store various types of programs, application data, etc. required for the control or operation of the remote control device (200).

[0095] When the remote control device (200) wirelessly transmits and receives signals through the display device (100) and the RF circuit (221), the remote control device (200) and the display device (100) transmit and receive signals through a predetermined frequency band.

[0096] The controller (280) of the remote control device (200) can store and reference information regarding the frequency band that can wirelessly transmit and receive signals with the display device (100) paired with the remote control device (200) in the memory (270).

[0097] The controller (280) controls all matters related to the control of the remote control device (200). The controller (280) can transmit a signal corresponding to a predetermined key operation of the user input interface (230) or a signal corresponding to the movement of the remote control device (200) sensed by the sensor (240) to the display device (100) via the wireless communication unit (225).

[0098] Additionally, the microphone (290) of the remote control device (200) can acquire voice.

[0099] A plurality of microphones (290) may be provided.

[0100] Next, Figure 4 is described.

[0101] Figure 4 shows an example of utilizing a remote control device according to an embodiment of the present invention.

[0102] Figure 4 (a) illustrates that a pointer (205) corresponding to a remote control device (200) is displayed on a display (180).

[0103] The user can move or rotate the remote control device (200) up and down, left and right. The pointer (205) displayed on the display (180) of the display device (100) corresponds to the movement of the remote control device (200). This remote control device (200) can be called a space remote control because, as shown in the drawing, the pointer (205) moves and is displayed according to the movement in 3D space.

[0104] Figure 4 (b) illustrates that when a user moves the remote control device (200) to the left, the pointer (205) displayed on the display (180) of the display device (100) also moves to the left correspondingly.

[0105] Information about the movement of the remote control device (200) detected through the sensor of the remote control device (200) is transmitted to the display device (100). The display device (100) can calculate the coordinates of the pointer (205) from the information about the movement of the remote control device (200). The display device (100) can display the pointer (205) to correspond to the calculated coordinates.

[0106] Figure 4 (c) illustrates a case where a user moves the remote control device (200) away from the display (180) while pressing a specific button within the remote control device (200). As a result, a selection area within the display (180) corresponding to the pointer (205) can be zoomed in and displayed in an enlarged manner.

[0107] Conversely, when the user moves the remote control device (200) closer to the display (180), the selection area within the display (180) corresponding to the pointer (205) may be zoomed out and displayed in a reduced size.

[0108] Meanwhile, when the remote control device (200) moves away from the display (180), the selection area may be zoomed out, and when the remote control device (200) moves closer to the display (180), the selection area may be zoomed in.

[0109] Additionally, when a specific button within the remote control device (200) is pressed, recognition of up, down, left, and right movements may be excluded. That is, when the remote control device (200) moves away from or toward the display (180), up, down, left, and right movements may not be recognized, and only forward and backward movements may be recognized. When a specific button within the remote control device (200) is not pressed, only the pointer (205) moves in accordance with the up, down, left, and right movements of the remote control device (200).

[0110] Meanwhile, the movement speed or movement direction of the pointer (205) can correspond to the movement speed or movement direction of the remote control device (200).

[0111] Meanwhile, the pointer in this specification refers to an object displayed on the display (180) in response to the operation of the remote control device (200). Accordingly, objects of various shapes other than the arrow shape illustrated in the drawing can be used as the pointer (205). For example, the pointer may be a concept including a point, a cursor, a prompt, a thick outline, etc. In addition, the pointer (205) may be displayed corresponding to one point on the horizontal or vertical axis on the display (180), or may be displayed corresponding to multiple points such as lines or surfaces.

[0112] FIG. 5 is a diagram for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0113] Referring to FIG. 5, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network.

[0114] The AI ​​server (200) may be composed of multiple servers to perform distributed processing, and may be defined as a 5G network.

[0115] The AI ​​server (200) may include a communication interface (210), memory (230), a learning processor (240), and a processor (260).

[0116] The communication interface (210) can transmit and receive data with an external device such as a display device (100).

[0117] The memory (230) may include a model memory (231). The model memory (231) may store a model (or artificial neural network, 231a) being learned or learned through the learning processor (240).

[0118] A learning processor (240) can train an artificial neural network (231a) using learning data. The learning model can be used while mounted on the AI ​​server (200) of the artificial neural network, or can be mounted on an external device such as a display device (100).

[0119] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0120] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0121] FIG. 6 is a flowchart illustrating an operating method of a display device according to an embodiment of the present disclosure.

[0122] The controller (170) of the display device (100) can play a video through the display (180) (S601).

[0123] The video may be either a moving image or an image. The video may be a broadcast image, a content image received from a content provider via a network interface (133), or an external input image received from an external device connected via an external device interface (135).

[0124] In the following, the video is explained assuming that it is a moving picture.

[0125] The controller (170) can extract one or more objects from the video being played and store information about the extracted objects in the memory (140) (S603).

[0126] An object can be any of the following: a person, the person's face, an accessory, clothing, a pillow, or a sofa.

[0127] Extracting objects can mean identifying objects contained within an image.

[0128] In one embodiment, the controller (170) may extract one or more objects from a video via an on-device neural processing unit (ON-device NPU). The on-device neural processing unit may be referred to as a neural processing unit or an artificial intelligence chip (171).

[0129] The artificial intelligence chip (171) may be included in the controller (170) or may be a component configured separately from the controller (170). The artificial intelligence chip (171) may be composed of one or more chips.

[0130] The artificial intelligence chip (171) can extract one or more objects from a video using an artificial intelligence model. The artificial intelligence chip (171) can identify the object and obtain identification information about the object. The identification information about the object may be the type or class of the object.

[0131] The artificial intelligence model may be stored in memory (140) or in a storage provided by the artificial intelligence chip (171).

[0132] The artificial intelligence model can be any of the following: YOLO (You Only Look Once) model, Convolution Neural Network (CNN) model, or Faster R-CNN (Regions with Convolution Neural Network) model trained using deep learning algorithms or machine learning algorithms.

[0133] The YOLO (You Only Look Once) model is a model that can predict objects and their locations within an image at once.

[0134] The YOLO (You Only Look Once) model divides the original image into grids of the same size, and for each grid, the number of bounding boxes specified in a predefined shape centered around the center of the grid is predicted, and the confidence is calculated based on this.

[0135] Afterwards, a process of determining whether the image contains an object or is just a background is performed, and locations with high object confidence are selected so that the object category can be confirmed.

[0136] The Faster R-CNN (Regions with Convolution Neural Network) model is a model that can detect objects faster than the CNN model.

[0137] This section describes the Faster R-CNN (Regions with Convolution Neural Network) model in detail. First, feature maps are extracted from an image using a CNN (Convolution Neural Network) model. Based on the extracted feature maps, multiple regions of interest (RoIs) are extracted, and RoI pooling is performed for each ROI.

[0138] RoI pooling is a process of extracting a feature map of size H x W by setting a grid that matches a predetermined size H x W on which the region of interest is projected, and extracting the largest value for each cell included in each grid to extract a feature map of size H x W. After that, a feature vector is extracted from the feature map of size H x W, and an object can be recognized from the feature vector.

[0139] In another embodiment, the controller (170) can transmit an image of a video to an AI server (200) via a network interface (133) and receive information about one or more objects included in the image from the AI ​​server (200).

[0140] The AI ​​server (200) can extract one or more objects using an artificial intelligence model stored in a memory (231) and an image received from a display device (100). The artificial intelligence model may be any one of the YOLO model or the Faster R-CNN model described above.

[0141] Information about an object may include one or more of the name of the object or an object image containing the object. The object image may be a cropped image of the object.

[0142] In one embodiment, the controller (170) or the artificial intelligence chip (171) can extract an object when an event occurs.

[0143] In one embodiment, the controller (170) or the artificial intelligence chip (171) may determine that an event has occurred when the scene of the video is switched.

[0144] In another embodiment, the controller (170) or the artificial intelligence chip (171) may determine that an event has occurred when the playback time of a broadcast program for which viewing has been scheduled has been reached.

[0145] In another embodiment, the controller (170) or artificial intelligence chip (171) may determine that an event has occurred when a specific action of the viewer is detected.

[0146] A specific action by the viewer can be either a voice command indicating the extraction of an object spoken by the user, or a specific gesture made by the user via the camera.

[0147] In another embodiment, the controller (170) or artificial intelligence chip (171) may determine that an event has occurred when a viewer's preferred character appears in the video.

[0148] In another embodiment, the controller (170) or the artificial intelligence chip (171) may determine that an event has occurred when the viewer receives a command to select a specific object via the pointer (205).

[0149] The artificial intelligence chip (171) can extract objects in real time and automatically from a playing video. Information about the extracted objects can be stored in the memory (140) for a certain period of time.

[0150] When objects are extracted through an artificial intelligence chip (171), a separate server for object extraction is not required, which provides cost advantages and allows objects to be extracted more quickly.

[0151] The controller (170) can provide search results for a selected object among one or more extracted objects (S605).

[0152] The controller (170) may receive a selection command to select one of the extracted objects. In one embodiment, the controller (170) may receive the selection command from a remote control device (200) via a user input interface (150). In another embodiment, the controller (170) may receive the selection command via a voice command uttered by a user.

[0153] The controller (170) can obtain search results for objects using information about the selected object.

[0154] The controller (170) can transmit an image of an object to an AI server (200) or a search server (not shown), and can receive search results for the object from the AI ​​server (200) or the search server. The search server may be a server of a website that provides search results for text or images.

[0155] Search results for an object may include multiple search items. Each search item may include one or more of the following: the object's price, its attributes, its brand, or its seller. Each search item may be clickable and, when clicked, may provide access to a linked site.

[0156] While watching a video, a user can select any of the objects included in the video and quickly receive search results for the selected object.

[0157] FIG. 7 is a flowchart illustrating an operating method of a display device according to an embodiment of the present disclosure.

[0158] Figure 7 is a drawing that explains the embodiment of Figure 6 in more detail.

[0159] Referring to FIG. 7, the controller (170) of the display device (100) can play a video through the display (180) (S701).

[0160] The controller (170) can play the first scene of the video. The scene of the video can be played during the playback period. The playback period for each scene can be different. The scene of the video can be composed of one or more video frames.

[0161] The controller (170) can extract one or more objects from the first scene of the video being played and store information about the one or more extracted objects in the memory (140) (S703).

[0162] The controller (170) can extract one or more objects from the first scene using the method described in step S603 of FIG. 6. The controller (170) can obtain information about the objects identified from the first scene and store the information about the obtained objects in the memory (140).

[0163] The controller (170) can determine whether the first scene of the video is switched to the second scene (S705).

[0164] In one embodiment, the controller (170) may determine that a scene has transitioned when a different object appears in the scene. Specifically, the controller (170) may determine that a scene has transitioned when an object not identified in the first scene appears in the second scene.

[0165] In another embodiment, the controller (170) can compare color histograms between two video scenes, and if the difference in color histograms is greater than a certain value, it can determine that the scenes have been switched.

[0166] If the controller (170) determines that the first scene has been switched to the second scene, it can extract one or more objects from the second scene and store information about the one or more extracted objects in the memory (140) (S707).

[0167] The controller (170) can obtain information about an object identified from the second scene and store the information about the obtained object in the memory (140).

[0168] The process of extracting one or more objects from the second scene is replaced with the description of step S603 of FIG. 6.

[0169] The controller (170) can obtain an exposure command of an object (S709), and can display a plurality of object items on the display (180) based on the obtained exposure command (S711).

[0170] In one embodiment, the object exposure command may be a command that exposes information about each object stored in each scene.

[0171] In one embodiment, the controller (170) can obtain an exposure command by receiving an IR signal generated by selecting a preset button provided on a remote control device (200).

[0172] In another embodiment, the controller (170) can obtain an exposure command based on a voice command spoken by a user through a microphone (not shown) provided in a user input interface (150) of the display device (100).

[0173] In another embodiment, the controller (170) can obtain an exposure command by receiving a voice signal corresponding to a voice command received by the remote control device (200).

[0174] The controller (170) can display a plurality of object items stored during playback of a second scene of a video on the display (180) according to an exposure command. The controller (170) can display a second scene of a video and a plurality of object items on the display (180) according to an exposure command.

[0175] Each of the plurality of object items may be an item generated based on information about the extracted object. For example, the plurality of object items may include a plurality of object images corresponding to objects extracted from a first scene and a plurality of object images corresponding to objects extracted from a second scene.

[0176] The controller (170) can receive a selection command to select one object item among a plurality of displayed object items (S713), and can output a search result for the selected object according to the received selection command (S715).

[0177] In one embodiment, the controller (170) may receive a selection command via a pointer or highlight box that moves in accordance with the movement of the remote control device (200). For example, the controller (170) may receive a selection command from the remote control device (200) to select an object item after the pointer is positioned on one of the plurality of object items.

[0178] In another embodiment, the controller (170) may select one of a plurality of object items based on receiving a voice command spoken by the user.

[0179] The controller (170) may transmit a search result request requesting search results for selected object items to the AI ​​server (200) or search server via the network interface (133). The search result request may include the selected object items. The object items may include one or more of the object's name, type, or cropped object image.

[0180] The controller (170) can receive search results for selected object items from the AI ​​server (200) or search server.

[0181] In one embodiment, the controller (170) can display the received search results on the display (180).

[0182] In another embodiment, the controller (170) may transmit the received search results to a user's smart device connected to the display device (100).

[0183] Below, an actual usage scenario of the embodiment of Fig. 7 is described.

[0184] FIGS. 8A to 8F are diagrams illustrating a process for providing a search result for a desired object while a user is watching a video according to an embodiment of the present disclosure.

[0185] Referring to FIG. 8a, the display device (100) is playing a first scene (810) of a video on the display (180).

[0186] The display device (100) can extract a plurality of objects from the first scene (810) and store a plurality of object items corresponding to each of the extracted plurality of objects in the memory (140).

[0187] The display device (100) can identify a plurality of objects from the first scene (810) using its own artificial intelligence chip (171). The artificial intelligence chip (171) can identify a plurality of objects from the first scene (810) using the artificial intelligence model described above, and can obtain an object image corresponding to each identified object.

[0188] For example, the artificial intelligence chip (171) can identify a plurality of objects from the first scene (810) and obtain a plurality of object images (811 to 815) corresponding to each of the identified plurality of objects. Each of the first object image (811), the second object image (812), and the third object image (813) may be an image that identifies a person. Each of the fourth object image (814) and the fifth object image (815) may be an image that identifies a pillow.

[0189] The image of each object may be cropped into a square shape, but this is only an example and the image may be cropped into other shapes such as circles or ovals.

[0190] Referring to FIG. 8b, the display device (100) is playing back a second scene (830) of the video on the display (180).

[0191] The display device (100) can determine that the scene of the video has been switched to the second scene (830) if objects extracted from the first scene (810) are not detected.

[0192] The artificial intelligence chip (171) can identify a plurality of objects from the second scene (830) and obtain a plurality of object images (831 to 834) corresponding to each of the identified plurality of objects. Each of the sixth object image (831) and the seventh object image (832) may be an image identifying a person. Each of the eighth object image (833) and the ninth object image (834) may be an image identifying a bag.

[0193] Meanwhile, the controller (170) can classify a plurality of acquired object images according to the class (or type) of the object and store them in the memory (140). The class of the object may be any one of a person, clothing, or accessory.

[0194] Referring to FIG. 8c, multiple object images stored in memory (140) are illustrated.

[0195] The controller (170) can store object images in the memory (140) by object class, such as a person, a pillow, or a bag. For example, the controller (170) can classify the first object image (811), the second object image (812), the third object image (813), the sixth image (831), and the seventh object image (832) into the person class.

[0196] The controller (170) can classify the fourth object image (814) and the fifth object image (815) into the pillow class. The controller (170) can classify the eighth object image (833) and the ninth object image (834) into the bag class.

[0197] Meanwhile, the controller (170) can convert the properties of the object image and store the object image having the converted properties in the memory (140). For example, the controller (170) can lower the resolution of the object image or reduce the size of the object image and store it in the memory (140). Accordingly, the storage capacity of the object image can be reduced. As another example, the controller (170) can store an image that includes only the object by removing the background area excluding the object from the object image.

[0198] Upon receiving an exposure command for exposure of the extracted object, the controller (170) can display an object item list (870) on the display (180), as illustrated in FIG. 8d.

[0199] The controller (170) can display the third scene (850) of the video played according to the scene transition and the object item list (870) together on the display (180).

[0200] The controller (170) can display the third scene (850) on the first area and the object item list (870) on the second area, respectively. As another example, the controller (170) can display additional viewing information of the video in addition to the third scene (850) and the object item list (870). The viewing information can include the title of the video, characters, plot, and narration.

[0201] The object item list (870) may be a window for selecting one of a plurality of object images (811 to 815, 831 to 834). Each of the plurality of object images (811 to 815, 831 to 834) may be a clickable image.

[0202] In one embodiment, the controller (170) can determine which object items are to be displayed in the object item list (870) based on priorities. The factor determining the priorities may be either the viewer's preference or past search history (or search frequency).

[0203] For example, the controller (170) may first display, in the object item list (870), only object items that include the type of object preferred by the viewer among the multiple object items stored in the memory (140). The type of object preferred by the viewer may be registered in advance through a user account. The type of object preferred by the viewer may be any one of a person, a car, clothes, accessories, and props.

[0204] The controller (170) can preferentially display object items representing clothes when the search frequency of clothes is higher than a certain frequency based on the search history of past objects.

[0205] In another embodiment, the controller (170) may expose object items according to a priority order of object types preset by the viewer.

[0206] The controller (170) can provide object items hierarchically. For example, the object item list (700) can include a person image, clothes worn by the person included in the person image, and a brooch included in the clothes.

[0207] When a person image included in the object item list (700) is selected, the controller (170) may obtain search results for the person image, search results for clothes worn by the person included in the person image, and search results for a brooch on the clothes.

[0208] The controller (170) can display an object item list (700) based on the genre of the video being played. For example, when playing a home shopping broadcast, the controller (170) can display product images according to a display command. As another example, when playing a drama, the controller (170) can display person images according to a display command. The controller (170) can obtain the video genre using an electronic program guide or from an external server.

[0209] The controller (170) can receive a selection command from the remote control device (200) to select the ninth object image (834) included in the object item list (870). The controller (170) can generate a search result request when the pointer (205) is positioned on the ninth object image (834) for a preset time or when the ninth object image (834) is selected through the pointer (205).

[0210] The controller (170) can transmit a search result request including the ninth object image (834) to the search server.

[0211] Meanwhile, when the ninth object image (834) is selected, the controller (170) may further display command guides (881, 883, 885) on the display (180), as illustrated in FIG. 8e. Each command guide may include text and may be a clickable element.

[0212] The first command guide (881) may be a guide indicating that the search results of the selected ninth object image (834) can be provided through a search engine. When the first command guide (881) is selected, the controller (170) may request the search results of the ninth object image (834) from a search server equipped with a search engine and receive the search results from the search server.

[0213] The second command guide (883) may be a guide indicating that the search results for the selected ninth object image (834) can be provided through the shopping site. When the second command guide (883) is selected, the controller (170) may request the search results for the ninth object image (834) from the shopping server operating the shopping site and receive the search results from the shopping server.

[0214] The third command guide (885) may be a guide indicating that the search results of the selected ninth object image (834) can be transmitted to the viewer's smartphone. When the third command guide (885) is selected, the controller (170) can transmit the search results of the ninth object image (834) acquired through the search server to the viewer's smartphone through the wireless communication interface (173).

[0215] In one embodiment, the controller (170) may receive a guide selection command from the remote control device (200) to select any one of the command guides (881, 883, 885).

[0216] In another embodiment, the controller (170) may receive a guide selection command to select any one of the command guides (881, 883, 885) via a voice command that utters the text included in each guide command.

[0217] Referring to FIG. 8F, the controller (170) can display the search results (890) of the selected ninth object image (834) on the display (180). The controller (170) can display the third scene (850) in the first area, the object list items (870) in the second area, and the search results (890) in the third area.

[0218] In another embodiment, the controller (170) may display a screen of a website containing search results (890) of the selected ninth object image (834) on the entire screen of the display (180). The website may be a site that provides search results (890).

[0219] In this way, according to an embodiment of the present disclosure, a user can quickly receive search results for an object he or she wants to search for while watching a video, thereby greatly improving user convenience.

[0220] Additionally, according to the embodiments of the present disclosure, the usability of the TV system as a platform operator can be increased.

[0221] Additionally, according to an embodiment of the present disclosure, since images are extracted directly from the on-device (display device, 100) instead of accessing a continuous server, the server cost used for extracting objects can be reduced.

[0222] Additionally, according to an embodiment of the present disclosure, issues such as copyright can be alleviated by handling images on-device.

[0223] In addition, according to the embodiment of the present disclosure, there is an advantage in that search linkage is possible in any content without inserting metadata.

[0224] FIGS. 9A to 9D are drawings illustrating an embodiment in which a viewer selects an object while watching a video and search results for the selected object are provided.

[0225] Describes Fig. 9a and Fig. 9b.

[0226] Referring to FIG. 9A, the display device (100) can play a video (900) on the display (180). The display device (100) can receive a selection command to click a first object (901) through a pointer (205) controlled by the movement of the remote control device (200).

[0227] The artificial intelligence chip (171) can extract a first object (901) upon receiving a selection command. The first object (901) may be clothes worn by a person.

[0228] The artificial intelligence chip (171) can obtain a first crop image (910) including a first object (901).

[0229] The controller (170) can obtain a first search result for the first crop image (910) obtained using a search server providing an image search function. After selecting the first object (901), the controller (170) can also display a screen (920) of the website of the search server providing an image search function, as illustrated in FIG. 9b.

[0230] The controller (170) can display a first crop image (910) and a first search result (930) corresponding to the first object (901) on the display (180).

[0231] The first search result (930) may include a plurality of clothing items corresponding to the first object (901). The plurality of clothing items may be clickable and provide access to different websites selling the extracted clothing.

[0232] Next, Figs. 9c and 9d are described.

[0233] Referring to FIG. 9c, the display device (100) can play a video (900) on the display (180). The display device (100) can receive a selection command to click a second object (903) through a pointer (205) controlled by the movement of the remote control device (200).

[0234] The artificial intelligence chip (171) can extract a second object (903) upon receiving a selection command. The second object (903) may be a pillow.

[0235] The artificial intelligence chip (171) can obtain a second crop image (940) including a second object (903).

[0236] The controller (170) can obtain a second search result for the second crop image (940) obtained using a search server providing an image search function. After selecting the second object (903), the controller (170) can also display a screen (950) of the website of the search server providing an image search function, as illustrated in FIG. 9d.

[0237] The controller (170) can display a second crop image (940) and a second search result (960) corresponding to the second object (903) on the display (180).

[0238] The second search result (960) may include a plurality of pillow items corresponding to the second object (903). The plurality of pillow items may be clickable and provide access to different websites selling the extracted pillows.

[0239] In this way, according to an embodiment of the present disclosure, a viewer can quickly search for a desired product while watching a video and can also shop for the searched product.

[0240] A display device (100) according to an embodiment of the present disclosure may include a memory (140), a display (180) that plays a video, and a controller (170) that extracts one or more objects from the video being played, stores one or more object items corresponding to the one or more extracted objects in the memory, displays the one or more object items on the display according to a display command, receives a selection command for selecting one of the one or more object items, obtains a search result of an object item selected according to the received selection command, and outputs the obtained search result.

[0241] The above controller (170) can determine the object items to be displayed on the display based on the viewer's preferences or past search history.

[0242] The above controller (170) can extract one or more objects from the video when the occurrence of an event is detected.

[0243] The above event may be any of a scene change in the video, reaching a certain time, or a certain action by the viewer.

[0244] The above controller (170) can determine the object items to be displayed on the display based on the genre of the video.

[0245] The above object item is an object image including an object, and the controller (170) can transmit the object image to a search server, obtain the search result from the search server, and display the obtained search result on the display. The controller (170) can lower the resolution of the object image including the object or reduce the size of the object and store it in the memory (140).

[0246] The above controller (170) may include an artificial intelligence chip (171) that extracts one or more objects from the video using an artificial intelligence model learned through a deep learning algorithm or a machine learning algorithm.

[0247] According to one embodiment of the present invention, the above-described method can be implemented as processor-readable code on a medium in which a program is recorded. Examples of processor-readable media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices.

[0248] The display device described above is not limited to the configuration and method of the embodiments described above, and the embodiments may be configured by selectively combining all or part of each embodiment so that various modifications can be made.

Claims

1. In the display device, memory; A display that plays videos; and A controller that extracts one or more objects from the video being played, stores one or more object items corresponding to the extracted one or more objects in the memory, displays the one or more object items on the display according to an exposure command, receives a selection command for selecting one of the one or more object items, obtains a search result of the object item selected according to the received selection command, and outputs the obtained search result. Display device.

2. In paragraph 1, The above controller Determines which object items will be displayed on the display based on the viewer's preferences or past search history. Display device.

3. In paragraph 1, The above controller When the occurrence of an event is detected, extracting one or more objects from the video. Display device.

4. In paragraph 3, The above event is Any of the above video's scene changes, reaching a certain time, or a certain action by the viewer. Display device.

5. In paragraph 1, The above controller Determine the object items to be displayed on the display based on the genre of the video. Display device.

6. In paragraph 1, The above object item is an object image containing an object, The above controller Transmitting the object image to a search server, obtaining the search result from the search server, and displaying the obtained search result on the display. Display device.

7. In paragraph 6, The above controller Lowering the resolution of the object image including the object or reducing the size of the object and storing it in the memory Display device.

8. In paragraph 1, The above controller An artificial intelligence chip that extracts one or more objects from the video using an artificial intelligence model learned through a deep learning algorithm or a machine learning algorithm. Display device.

9. In the method of operating the display device, Steps to play the video; A step of extracting one or more objects from the video being played; A step of storing one or more object items corresponding to one or more extracted objects; A step of displaying one or more object items according to an exposure command; A step of receiving a selection command for selecting any one of the one or more object items; A step of obtaining a search result of a selected object item according to a received selection command; and A step of outputting the obtained search results How the display device operates.

10. In paragraph 9, The above controller Determines which object items will be displayed on the display based on the viewer's preferences or past search history. How the display device operates.

11. In paragraph 9, The above controller When the occurrence of an event is detected, extracting one or more objects from the video. How the display device operates.

12. In paragraph 11, The above event is Any of the above video's scene changes, reaching a certain time, or a certain action by the viewer. How the display device operates.

13. In paragraph 9, The above controller Determine the object items to be displayed on the display based on the genre of the video. How the display device operates.

14. In paragraph 9, The above object item is an object image containing an object, The above controller Transmitting the object image to a search server, obtaining the search result from the search server, and displaying the obtained search result on the display. How the display device operates.

15. In paragraph 14, The above controller Lowering the resolution of the object image including the object or reducing the size of the object and storing it in the memory How the display device operates.

Images