Display device and method for controlling display device

Abstract

To provide a display device or the like capable of presenting related information of an object intended to be retrieved by a user.SOLUTION: A display device includes a display unit, a first display control unit configured to, in response to receiving a first operation from a user, display, on the display unit, a plurality of object specifying figures that specify a plurality of types of objects included in a still image of video displayed on the display unit, the plurality of types being different from each other, the plurality of object specifying figures being superimposed on the still image, and a second display control unit configured to, in response to receiving a second operation of selecting a target object specifying figure from among the plurality of object specifying figures, display, on the display unit, related information related to an object corresponding to the target object specifying figure.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a display device and a method for controlling the display device. [Background technology]

[0002] For example, Patent Document 1 proposes a technology in which a program playback device cuts out a face image from a video signal and displays a profile based on the cut-out face image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-016030 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, in recent display devices such as televisions, a technology has been adopted that allows a user to specify an object, such as a person, included in a displayed image and search for information about the specified object. Here, the displayed image often includes various types of objects. In this case, since the image also includes multiple objects that the user does not intend to search for, it is difficult to easily perform an information search targeting the object the user intends to search for. Patent Document 1 does not solve this problem.

[0005] In one aspect, an object of the present disclosure is to provide a display device or the like that can present information related to an object that a user intends to search for. [Means for solving the problem]

[0006] A display device according to one embodiment of the present invention comprises a display unit, a first display control unit that, in response to receiving a first operation from a user, causes the display unit to superimpose a plurality of object identification figures that identify a plurality of different types of objects contained in a still image of a video displayed on the display unit on the still image, and a second display control unit that, in response to receiving a second operation to select a target object identification figure from the plurality of object identification figures, causes the display unit to display related information relating to an object that corresponds to the target object identification figure. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of a system. [Figure 2] FIG. 1 is a diagram illustrating an example of a television. [Figure 3] FIG. 2 is a diagram illustrating an example of a remote control. [Figure 4] FIG. 2 is a diagram illustrating an example of a screen transition in the first embodiment. [Figure 5] FIG. 5 is a diagram showing an example of an image transitioned from the image in FIG. 4. [Figure 6] 4 is a flowchart showing an example of a processing flow according to the first embodiment. [Figure 7] This is a flowchart continuing from FIG. [Figure 8] FIG. 10 is a diagram showing an example of a screen including a person's name as related information. [Figure 9] FIG. 10 is a diagram showing an example of a screen including the name of a landscape and a reduced map as related information. [Figure 10] FIG. 10 is a diagram illustrating an example of a screen transition in the second embodiment. [Figure 11] 11 is a flowchart showing an example of a processing flow according to a third embodiment. [Figure 12] 13 is a flowchart showing an example of a processing flow according to a fourth embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a screen transition in the fifth embodiment. [Figure 14] FIG. 14 is a diagram showing an example of a screen transition from the image in FIG. 13. [Figure 15] 10 is an example of a table showing a first example of search content. [Figure 16] FIG. 20 is a diagram showing a first example of a screen transition in the sixth embodiment. [Figure 17] 10 is an example table showing a second example of search content. [Figure 18] FIG. 23 is a diagram showing a second example of a screen transition in the sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] First Embodiment The first embodiment will be described below. FIG. 1 is a diagram showing an example of a system 100. The system 100 includes a television 101, an external server 102, a communication terminal 103, and a remote control 104. The television 101 is, for example, a display device that displays video of a program broadcast by a television station. The television 101 has a display unit 101A, and displays the video of the program on the display unit 101A. The television 101 may also display, for example, video distributed over the Internet on the display unit 101A.

[0009] The external server 102 is an external device capable of communicating with the television 101 via an Internet line. For example, the external server 102 is a cloud server. The communication terminal 103 is a terminal capable of short-range wireless communication with the television 101. The communication terminal 103 is, for example, a smartphone, a tablet terminal, or a mobile phone. The remote control 104 is a remote controller for operating the television 101.

[0010] Next, an example of the television 101 will be described. Fig. 2 is a diagram showing an example of the television 101. The television 101 has a control unit 121, a broadcast wave receiving unit 122, a storage unit 123, a remote control signal receiving unit 124, a first communication unit 125, a second communication unit 126, and a display unit 101A. The television 101 may have other configurations. For example, the television 101 has a speaker that outputs sound.

[0011] The control unit 121 executes various processes of the television 101. The control unit 121 has a processor and a memory. The processor of the control unit 121 executes a plurality of instruction sets (programs) stored in the memory of the control unit 121, thereby realizing various controls of the control unit 121. Any processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field Programmable Gate Array) may be used as the processor.

[0012] The broadcast wave receiving unit 122 receives broadcast waves from multiple television stations. These broadcast waves are broadcast waves for video of programs broadcast by the multiple television stations. The storage unit 123 stores predetermined information. The remote control signal receiving unit 124 is an operation accepting unit that receives a remote control signal from the remote control 104 based on a user operation.

[0013] First communication unit 125 is connected to, for example, an Internet line and communicates with external server 102 via the Internet line. Second communication unit 126 performs short-range wireless communication with communication terminal 103. Display unit 101A displays video of a program received by broadcast wave receiving unit 122. Display unit 101A also displays other information.

[0014] The control unit 121 has an extraction unit 201 , an object classification unit 202 , a related information acquisition unit 203 , a search content setting unit 204 , a notification control unit 205 , an access control unit 206 , a first display control unit 207 , and a second display control unit 208 .

[0015] The extraction unit 201 acquires a still image (picture) of the video of the program being received by the broadcast wave receiving unit 122 at the time of receiving a user operation using the remote control 104. The extraction unit 201 performs predetermined image processing on the acquired still image to extract images of multiple types of objects included in the still image.

[0016] The images of the objects to be extracted are, for example, images of people, objects, characters, landscapes, etc. The extraction unit 201 cuts out and extracts images of each object from the still image by applying an image recognition method such as pattern matching to the still image.

[0017] The object classification unit 202 classifies the images of the multiple types of objects extracted by the extraction unit 201. As described above, the object types include, for example, people, objects, characters, landscapes, and the like.

[0018] For example, the object classification unit 202 may use AI (Artificial Intelligence) to classify the multiple types of objects extracted by the extraction unit 201 into each type. In this case, the object classification unit 202 may use a trained model that is machine-learned using training object images and ground truth data that indicates the type of the training object images. The object classification unit 202 may input an image of an object into the trained model and recognize the type of the object. The object classification unit 202 may classify objects using a predetermined image classification method (e.g., pattern matching) other than the trained model.

[0019] The related information acquisition unit 203 acquires, from the external server 102 (storage device), related information related to an object designated by the user as a search target among the images of multiple types of objects extracted by the extraction unit 201. The operation by which the user designates an object as a search target will be described in detail later. The external server 102 stores first related information associated with each of the images of the multiple objects.

[0020] The related information acquisition unit 203 transmits an acquisition request to acquire related information of the object specified as the search target to the external server 102, for example, via the first communication unit 125. At this time, the related information acquisition unit 203 includes an image of the object specified by the user in the acquisition request.

[0021] In response to the acquisition request, the external server 102 acquires first related information corresponding to an image of an object that is closest to the received image of the object from among the images of multiple objects stored therein. For example, the external server 102 may acquire first related information corresponding to an image of an object that has the highest degree of match with the received image of the object from among the images of multiple objects stored therein.

[0022] The external server 102 transmits the acquired first related information to the television 101. The first communication unit 125 of the television 101 receives the first related information from the external server 102. The related information acquisition unit 203 acquires the first related information received from the external server 102 as related information. As a result, the related information acquisition unit 203 acquires related information related to the object specified by the user from among the multiple types of objects extracted by the extraction unit 201.

[0023] For example, if the object type is a person, the related information is information about the person's name. For example, if the object type is an item, the related information is information about the item's price. For example, if the object type is a landscape, the related information is information about the name of the landscape. For example, if the object type is a character, the related information is information indicating the translation result of the character.

[0024] The related information acquisition unit 203 may acquire multiple pieces of related information with different contents for the same type of object. For example, if the type of object is a landscape, the related information acquisition unit 203 may acquire information on the landscape and information on a reduced map. This point will be described in detail later.

[0025] The control unit 121 may acquire the above-mentioned first related information from the storage unit 123 (storage device). In this case, the storage unit 123 stores the first related information in association with each of the images of the plurality of objects, similar to the information stored in the external server 102.

[0026] As described above, the external server 102 stores images of a plurality of objects in association with first related information. For example, the external server 102 periodically collects images of objects and first related information associated with the objects from various servers connected to the external server 102. The external server 102 then stores the collected images of objects in association with the first related information. The external server 102 may collect and store images of objects and first related information associated with the objects using a method other than the above method.

[0027] The search content setting unit 204 sets in advance the search content to be searched for for each of the plurality of types of objects based on the user's operation. For example, the user can use the remote control 104 to set in advance the search content for each of the plurality of types of objects.

[0028] The search content is different for each object of the same type to be searched. For example, when a user specifies a search target of an object of the type "person," the search content indicates which of the name, gender, and age to search for for the object of the type "person." The same applies to other types of objects. Details will be described later.

[0029] When a character type object is designated by the user, notification control unit 205 notifies communication terminal 103 of character information of the designated character via second communication unit 126. The character information is assumed to be information on an access destination to be accessed by communication terminal 103. The character information indicating the access destination is information such as a telephone number or a URL (Uniform Resource Locator). Details will be described later.

[0030] When a character-type object is designated by the user, the access control unit 206 transmits character information indicating the access destination to the communication terminal 103 via the second communication unit 126, and controls the communication terminal 103 to access the access destination. Details will be described later.

[0031] First display control unit 207 sets a rectangular frame around each object extracted from the still image by extraction unit 201. The frame is an object-specific graphic. At this time, first display control unit 207 sets a frame around the object in a different display mode for each type of object.

[0032] For example, first display control unit 207 sets frames of different colors for a person object, an item object, a landscape object, and a character object. First display control unit 207 superimposes the frame set for each object on a still image and displays it on display unit 101A.

[0033] The object-specific graphic does not have to be a rectangular frame of a different color. For example, the object-specific graphic may be a circular or oval frame of a different color for each type of object. The object-specific graphic may also be a graphic other than a frame.

[0034] Second display control unit 208 causes display unit 101A to display information related to an object specified by the user, based on the user's operation using remote control 104. As described above, first display control unit 207 causes display unit 101A to display a still image in which frames of different colors are superimposed around each of the objects extracted from the still image by extraction unit 201, depending on the type of object.

[0035] For example, suppose that the user performs an operation to specify a blue frame indicating a person object using remote control 104. Based on the operation, second display control unit 208 controls related information acquisition unit 203 to acquire related information related to the person object. Based on the control, related information acquisition unit 203 acquires related information of the specified object from external server 102. Then, second display control unit 208 causes display unit 101A to display an image on which the acquired related information is superimposed.

[0036] The colors of the frames described above correspond to the color buttons (first buttons) provided on the remote control 104. Fig. 3 is a diagram showing an example of the remote control 104. The remote control 104 has a power button 301, numeric buttons 302, a search button 303, a blue button 304B, a red button 304R, a green button 304G, and a yellow button 304Y. The remote control 104 may have other buttons.

[0037] The power button 301 is a button for turning on and off the power of the television 101. The number buttons 302 are buttons (second buttons) for selecting a channel. The search button 303 is a button for searching for related information.

[0038] Red button 304R, green button 304G, and yellow button 304Y are color buttons, each of which is colored with a corresponding color. The color buttons can be used to execute any of the multiple functions of television 101. In each embodiment, the color buttons are used when the user specifies the type of object to be searched for.

[0039] For example, suppose that a still image displayed on display unit 101A includes a person object, an item object, a landscape object, and a text object. As described above, first display control unit 207 sets a blue frame around the person object, a red frame around the item object, a green frame around the landscape object, and a yellow frame around the text object. A frame of a different color is superimposed around each object included in the still image displayed on display unit 101A.

[0040] For example, when the user presses blue button 304B on remote control 104, second display control unit 208 recognizes an object surrounded by a blue frame. Related information acquisition unit 203 acquires related information of the recognized object from external server 102. Second display control unit 208 displays the acquired related information on display unit 101A.

[0041] This allows the user to recognize related information of a specified object by pressing a color button that matches the color of the frame of the object that the user intends to search for among the multiple types of objects displayed on display unit 101A.

[0042] The storage unit 123 of the television 101 stores in advance information about the colors of the color buttons of the remote control 104. This allows the first display control unit 207 to assign the colors of the color buttons to the types of objects.

[0043] Next, transitions of screens displayed on the display unit 101A will be described. Fig. 4 is a diagram showing an example of screen transitions in the first embodiment. The display unit 101A displays video of the program currently being selected. The video is a moving image.

[0044] For example, suppose that the user presses search button 303 on remote control 104 while image 400A in FIG. 4 is displayed on display unit 101A. This operation is a first operation. Remote control signal receiving unit 124 receives a signal indicating the first operation. First display control unit 207 acquires a still image of the video at the time the signal indicating the first operation is received.

[0045] 4 includes a person object 401, a first article object 402A, a second article object 402B, a first character object 403A, and a second character object 403B.

[0046] The extraction unit 201 extracts images of each of the above objects by performing image processing using an image recognition technique such as pattern matching on the image 400A. The object classification unit 202 classifies each object into one of a person, an object, a character, and a landscape, for example, using the trained model described above.

[0047] In the case of image 400A, the object classification unit 202 classifies the extracted person object 401 into a type called "person." The object classification unit 202 classifies the extracted first article object 402A and second article object 402B into a type called "article." The object classification unit 202 classifies the extracted first character object 403A and second character object 403B into a type called "character."

[0048] Image 400B is an image in which a frame is superimposed on image 400A. First display control unit 207 stores in advance colors corresponding to each type of person, object, landscape, and text. As described above, the colors for each type are assigned to the respective colors of the color buttons on remote control 104.

[0049] The first display control unit 207 superimposes a rectangular blue frame 411 on the image 400A around the image of the person object 401 of the person type. In the image 400B of Fig. 4, the blue frame 411 is represented by a dashed line.

[0050] The first display control unit 207 superimposes rectangular red frames 412A and 412B around the images of the first and second item objects 402A and 402B of the item type on the image 400A. In the image 400B of Figure 4, the red frames are represented by dotted lines.

[0051] The first display control unit 207 superimposes rectangular yellow frames 413A and 413B around the images of the first character object 403A and the second character object 403B of the character type on the image 400A. In the image 400B in Fig. 4, the yellow frames are represented by dashed lines.

[0052] As shown in Fig. 4, image 400B includes three types of objects. The total number of objects is five. If related information for each of the five objects is simultaneously displayed on display unit 101A, it is difficult for the user to easily identify the related information for the object that the user is interested in searching for.

[0053] Therefore, before the related information is displayed, first display control unit 207 causes display unit 101A to display image 400B with frames of each color superimposed thereon. Here, there are two object types: an object of the item type and an object of the character type.

[0054] In this case, first display control unit 207 adds different number information to the vicinity of each of the images of multiple objects of the same type. For example, first display control unit 207 adds numbers to multiple objects of the same type in order of size, starting from the largest object.

[0055] 4, the size of the image of the first item object 402A is larger than the size of the image of the second item object 402B. In this case, the first display control unit 207 adds "1" as number information near the first item object 402A and adds "2" as number information near the second item object 402B.

[0056] 4, the image size of the first character object 403A is larger than the image size of the second character object 403B. In this case, the first display control unit 207 adds "1" as number information near the first character object 403A and adds "2" as number information near the second character object 403B.

[0057] The numbers indicated by the number information correspond to the numeric buttons 302 on the remote control 104. For example, assume that the user intends to search for information related to the first item object 402A. In this case, the user presses the red button 304R and then presses "1" on the numeric button 302. In other words, the user performs a two-step operation to specify an object.

[0058] In response to receiving the above operation (second operation), the first display control unit 207 recognizes that the first item object 402A is the search target. The related information acquisition unit 203 transmits an image of the first item object 402A to the external server 102 via the first communication unit 125.

[0059] The external server 102 searches for the first related information corresponding to the image of the object that is closest to the image of the first item object 402A received from the television 101 (the image with the highest matching rate) among the images of the objects stored therein. Assume that the first related information obtained as a result of the search is "6000 yen."

[0060] The external server 102 transmits the first related information "6000 yen" obtained as a result of the search to the television 101. The second display control unit 208 adds information indicating the first related information "6000 yen" to the image 400B as related information. At this time, the second display control unit 208 may delete the frame and number information other than that of the specified first item object 402A from the image 400B.

[0061] The second display control unit 208 causes the display unit 101A to display image 400C of Fig. 5, which has undergone the above processing. Fig. 5 is a diagram showing an example of an image transitioned from image 400B of Fig. 4. Image 400C of Fig. 5 includes related information 420 ("6000 yen") of the first item object 402A that the user intended to search for, but does not include related information of the other four objects.

[0062] Therefore, the display unit 101A displays the related information of the first item object 402A that the user intends to search for, but does not display the related information of the other four objects, allowing the user to easily recognize the related information of the object that the user intends to search for.

[0063] In the above example, the second operation is an operation of first pressing a color button and then pressing a numeric button 302. The second operation may also be an operation of first pressing a numeric button 302 and then pressing a color button.

[0064] Furthermore, when one of the color buttons is pressed, first display control unit 207 may delete from image 400B any frames of a color other than the color corresponding to the pressed color button. For example, when the user presses red button 304R, first display control unit 207 may delete blue frame 411 and yellow frames 413A and 413B from image 400B in FIG. 4. As a result, an image including red frames 412A and 412B is displayed on display unit 101A. This image includes number information.

[0065] Then, suppose the user performs an operation of pressing "1" on the numeric button 302. In response to receiving this operation, the first display control unit 207 may recognize that the first item object 402A has been designated. This prevents frames of object types that the user does not intend to search for from being displayed, making it easier for the user to identify the object to be searched for.

[0066] 4 and 5, first display control unit 207 superimposes on the image classification correspondence information 409 indicating which classification each color of each frame corresponds to. In the examples of FIGS. 4 and 5, the classification correspondence information indicates that blue is the classification of people, red is the classification of items, green is the classification of landscapes, and yellow is the classification of characters. By displaying an image including classification correspondence information 409, the user can easily visually recognize the color of the color button on remote control 104 that corresponds to the classification of the item they are attempting to search for.

[0067] Next, the processing flow of the first embodiment will be described. Fig. 6 is a flowchart showing an example of the processing flow of the first embodiment. The display unit 101A displays video of a program of a broadcast wave received by the broadcast wave receiving unit 122 (step S101).

[0068] The control unit 121 determines whether the search button 303 has been pressed (step S102). The control unit 121 may make the determination in step S102 based on whether the remote control signal receiving unit 124 has received, from the remote control 104, a signal indicating that a first operation of pressing the search button 303 has been performed.

[0069] If the search button 303 has not been pressed, the control unit 121 determines No in step S102 and returns the process to step S102. If the search button 303 has been pressed, the control unit 121 determines Yes in step S102 and advances the process to step S103.

[0070] The extraction unit 201 acquires a still image at the time when the search button 303 is pressed from the video of the program being received by the broadcast wave receiving unit 122 (step S103). For example, the extraction unit 201 acquires the still image in response to the remote control signal receiving unit 124 receiving a signal indicating that a first operation has been performed.

[0071] The extraction unit 201 performs image processing on the acquired image using an image recognition method such as pattern matching, thereby extracting each of the multiple types of objects included in the still image (step S104).

[0072] The object classification unit 202 classifies each of the extracted multiple types of objects into each type (step S105). The object classification unit 202 may classify each of the extracted multiple types of objects into each type, for example, by using the trained model described above.

[0073] First display control unit 207 superimposes a frame of a corresponding color on each object for each classified type on the still image acquired in step S103 (step S106). In step S106, if there are multiple objects of the same type in the still image, first display control unit 207 performs processing to add different numbers of number information to each of the multiple objects of the same type (step S107).

[0074] First display control unit 207 causes display unit 101A to display the image that has been subjected to the process of step S107 (step S108). At this time, the image displayed on display unit 101A is, for example, image 400B in FIG.

[0075] The control unit 121 determines whether a button operation has been received (step S109). If there is only one corresponding object for all of the multiple types of objects included in the still image, the button operation is an operation of pressing a color button of any color.

[0076] When there are two or more objects of a certain type included in a still image, the button operation is an operation of pressing the color button and number button 302 of the corresponding color.

[0077] If no button operation has been received, the control unit 121 determines No in step S109 and returns the process to step S109. If a button operation has been received, the control unit 121 determines Yes in step S109 and advances the process from "A" to step S110 in FIG.

[0078] Fig. 7 is a flowchart continuing from Fig. 6. Based on the button operation, the second display control unit 208 identifies one object from among the multiple types of objects included in the still image (step S110). For example, in the example described above, when the red button 304R is pressed and the "1" button of the number buttons 302 is pressed, the second display control unit 208 identifies the first item object 402A.

[0079] The second display control unit 208 transmits the image of the object identified in step S110 as a search target to the external server 102 via the first communication unit 125 (step S111).

[0080] The external server 102 acquires first related information corresponding to the image of the object that most closely matches the received image of the object from among the images of the plurality of objects stored therein. The external server 102 transmits the acquired first related information to the television 101. The first communication unit 125 of the television 101 receives the first related information.

[0081] The second display control unit 208 acquires the first related information received by the first communication unit 125 (step S112). The second display control unit 208 superimposes the acquired first related information on the image as related information (step S113). Then, the second display control unit 208 causes the display unit 101A to display the image on which the related information is superimposed (step S114). As a result, an image such as that shown in FIG. 5 is displayed on the display unit 101A. This completes the processing of the first embodiment.

[0082] 8 is a diagram showing an example of a screen including a person's name as related information. In image 400B of FIG. 4, it is assumed that the user presses blue button 304B on remote control 104 with the intention of searching for the name of person object 401.

[0083] The extraction unit 201 extracts each object from the image 400D. The object classification unit 202 classifies each extracted object. The image 400D in Fig. 4 contains one person object. In other words, the image 400D contains one object of the type "person."

[0084] Therefore, the object to be searched for by the user can be specified to one object simply by pressing blue button 304B, without having to press any of number buttons 302 on remote control 104. Therefore, pressing number buttons 302 is not necessary.

[0085] The related information acquisition unit 203 transmits the image of the person object 401 to the external server 102 via the first communication unit 125. The external server 102 transmits first related information related to the person object 401 to the television 101. In the example of FIG. 8, the first related information is information called "Oyama A-ko."

[0086] Second display control unit 208 superimposes the acquired information "Oyama A-ko" on the image as related information 430, and generates image 400D shown in FIG. 8. Second display control unit 208 causes image 400D to be displayed on display unit 101A. As a result, for the object that is a person specified by the user, the name of the person is displayed on display unit 101A. At this time, related information of objects other than the object (person object 401) specified by the user using remote control 104 is not displayed.

[0087] 8, the first display control unit 207 does not include in the image 400D the blue frame 411 around the person object 401. Since there is no blue frame 411 around the person object 401, good visibility can be ensured.

[0088] 9 is a diagram showing an example of a screen including a name of a landscape and a reduced map as related information. For example, assume that search button 303 on remote control 104 is pressed while video of a program including a waterfall landscape and people is displayed on display unit 101A. Extraction unit 201 acquires a still image from the video. Extraction unit 201 extracts a person object and a landscape object included in the acquired still image.

[0089] The object classification unit 202 classifies the extracted person objects and scenery objects. The first display control unit 207 superimposes a blue frame (not shown) around the person object 411 to indicate that it is a person object. The first display control unit 207 also superimposes a green frame 460 around the scenery object (waterfall object) to indicate that it is a scenery object. The green frame 460 is represented by a solid line.

[0090] For example, if the user has previously performed an operation to specify a name and a reduced map as search content for a landscape object, the search content setting unit 204 sets the name and the reduced map in association with the landscape object based on the operation.

[0091] 9, it is assumed that the user intends to search for the name and reduced map of the scenery object and presses the green button 304G on the remote control 104. The related information acquisition unit 203 transmits, via the first communication unit 125, to the external server 102, together with the image of the scenery object, information indicating a request for related information on the name and reduced map.

[0092] The external server 102 acquires first related information corresponding to the image of the object that most closely matches the image of the scenery object received from the television 101, among the images of the plurality of objects stored therein. At this time, based on the information received from the television 101 indicating a request for the above-mentioned name and reduced map related information, the external server 102 acquires the name and reduced map information of the image of the object that most closely matches as the first related information. The external server 102 transmits the acquired name and reduced map information (first related information) to the television 101.

[0093] The external server 102 transmits first related information related to the person object 401 to the television 101. The first related information is the name of the scenery indicated by the scenery object and information on a reduced map of the location of the scenery.

[0094] Second display control unit 208 superimposes acquired landscape name 471 and reduced map 470 of the location of the landscape as related information on the image to generate image 400E shown in Fig. 9. Second display control unit 208 causes image 400E to be displayed on display unit 101A. As a result, for objects classified as landscapes, landscape name 471 and reduced map 470 are displayed on display unit 101A.

[0095] In the above, if the number of types of objects extracted by the extraction unit 201 from the still image of the video is equal to or less than the number of color buttons, the process of adding number information may not be performed. For example, assume that the image 400A in FIG. 4B does not include the second article object 402B and the second character object 403B.

[0096] In this case, the number of objects corresponding to each of the four types of objects included in image 400A in FIG. 4 is one. In this case, the object to be searched for by the user is specified as one. Therefore, when a color button is pressed, related information acquisition unit 203 transmits an image of the object corresponding to the pressed color button to external server 102 via first communication unit 125, without requiring the operation of pressing number button 302. This omits the operation of pressing number button 302.

[0097] In the above example, the object types are assigned to the four color buttons, but the object types may be assigned to each of the number buttons 302. In the case of the remote control 104 in Fig. 3, up to 12 object types can be designated.

[0098] Second Embodiment Next, a second embodiment will be described. In the second embodiment, the first display control unit 207 divides the display unit 101A into a plurality of areas. Fig. 10 is a diagram showing an example of a screen transition in the second embodiment.

[0099] 10, first display control unit 207 divides display unit 101A into four regions. For example, the number of divided regions may be the same as the number of types of objects. First display control unit 207 assigns the four regions to regions for different types of objects. First display control unit 207 adds classification correspondence information 409A to 409D, which indicates the classification of the objects assigned to each region, to the corresponding region.

[0100] For example, in response to pressing of search button 303 on remote control 104, extraction unit 201 acquires image 400A of Fig. 4 from the video of a program being received by broadcast wave receiving unit 122. Extraction unit 201 extracts multiple types of objects from image 400A. Object classification unit 202 classifies each of the multiple types of objects.

[0101] First display control unit 207 generates an image by duplicating and reducing image 400A in Fig. 4. Then, first display control unit 207 displays the image on display unit 101A with a frame of the corresponding color superimposed on each of the four regions.

[0102] In the example of Figure 10, a blue frame 411A is displayed superimposed on the area of ​​the person object. Red frames 412A and 412B are displayed superimposed on the area of ​​the item object. Yellow frames 413A and 413B are displayed superimposed on the area of ​​the character object. Since image 400A in Figure 4 does not include a landscape image, no frames identifying the objects are superimposed on the areas of the landscape objects.

[0103] For example, suppose that the user presses red button 304R. In this case, in response to receiving the operation of pressing red button 304R, first display control unit 207 restores image 400F to its original size and displays image 400G with number information added on display unit 101A.

[0104] For example, suppose the user presses "1" among the numeric buttons 302 on the remote control 104. In this case, the target of the user's search is identified as the first item object 402A. The related information acquisition unit 203 acquires related information of the first item object 402A from the external server 102. Then, the second display control unit 208 causes the display unit 101A to display, for example, an image 400C on which related information 420 is superimposed, as shown in the example of FIG. 5.

[0105] 10, first display control unit 207 divides the area into areas for each type of object, displays an image acquired from the video in each area, and superimposes the frame of the object corresponding to the area on the image in the area. This allows the user to easily recognize the object to be searched for by visually checking the area assigned to each type of object.

[0106] <Third embodiment> Next, a third embodiment will be described. After image 400B in Fig. 4 is displayed on display unit 101A, the user presses yellow button 304Y on remote control 104 and then presses "2" on numeric button 302. In this case, the object that the user intends to search for is identified as second character object 403B, whose object type is character.

[0107] In this case, the notification control unit 205 performs short-range wireless communication with the communication terminal 103 via the second communication unit 126, and notifies the communication terminal 103 of the character information of the second character object 403B. For example, the notification control unit 205 determines whether the second character object 403B is a phone number.

[0108] When the second character object 403B is a phone number, the notification control unit 205 notifies the communication terminal 103 of the character information of the second character object 403B as a phone number.

[0109] 11 is a flowchart showing an example of the processing flow of the third embodiment. The processing of the notification control unit 205 in steps S101 to S109 before "A" is the same as in the flowchart of FIG.

[0110] The notification control unit 205 determines whether a character type object has been designated (step S210). For example, the notification control unit 205 may determine whether a character type object has been designated based on whether the yellow button 304Y of the color buttons has been pressed.

[0111] If a character type object is specified, the notification control unit 205 determines Yes in step S210 and proceeds to step S211. In this case, the notification control unit 205 determines whether the specified object is a phone number (step S211).

[0112] For example, the notification control unit 205 may determine whether the specified object is a telephone number based on whether the specified character object is composed only of numbers and whether the number of digits in the numbers matches the number of digits specific to the telephone number.

[0113] If the specified object is a phone number, the notification control unit 205 determines Yes in step S211 and proceeds to step S212. In this case, the notification control unit 205 transmits information about the phone number to the communication terminal 103 via the second communication unit 126 (step S212).

[0114] In step S212, the notification control unit 205 may, for example, start a telephone application stored in the communication terminal 103 and prepare the telephone application to make a call to the notified telephone number. In this case, for example, the user can automatically make a call to the above telephone number by simply pressing the call button on the communication terminal 103.

[0115] If the determination in step S210 is No, or if the determination in step S211 is No, notification control unit 205 proceeds to step S110 without executing the process in step S212. In this case, the processes from step S110 onwards are executed.

[0116] <Fourth embodiment> Next, a fourth embodiment will be described. Fig. 12 is a flowchart showing an example of the processing flow of the fourth embodiment. In step S111, the related information acquisition unit 203 transmits an image of the identified object together with a request to the external server 102 to acquire an address on the Internet that lists information about the object. The address is, for example, a URL.

[0117] External server 102 stores an image of an object, first related information, and the address information in association with each other, and transmits the first related information and address information associated with an image that is closest to the received image of the object to television 101. As a result, notification control unit 205 obtains an address that lists information about the object being searched for.

[0118] Notification control unit 205 transmits the acquired address information to communication terminal 103 via second communication unit 126 (step S301). This allows communication terminal 103 to access an address on the Internet that lists information about the object to be searched for.

[0119] For example, in step S301, notification control unit 205 may transmit the above address information and may also control the activation of a browser application on communication terminal 103. Then, notification control unit 205 may set the above address in the browser application on communication terminal 103 to prepare for access.

[0120] In this case, for example, the user simply presses an access button or the like on the touch panel of communication terminal 103, and the browser application of communication terminal 103 accesses the address. This allows the user to view detailed information on the Internet about the object being searched for, simply by pressing the access button.

[0121] Fifth Embodiment Next, a fifth embodiment will be described. Fig. 13 is a diagram showing an example of screen transitions in the fifth embodiment. An image 600 displayed on a screen 600S of the display unit 101A is one of the images of a program. The first display control unit 207 causes the display unit 101A to display the image 600 of the program in one large area 600A of the screen 600S without dividing the screen. The image displayed in area 600A changes over time.

[0122] For example, suppose the user presses search button 303 on remote control 104. In this case, in response to remote control signal receiving unit 124 accepting the operation, first display control unit 207 divides the area of ​​display unit 101A into two. Screen 601S, which is transitioned from screen 600S, represents a screen obtained by dividing screen 600S into two.

[0123] The first display control unit 207 displays a video 601 of the program in area 601A (first area) of the two divided areas of the display unit 101A, and displays an image 602 of a still image acquired at the time the remote control signal receiving unit 124 received the above operation in area 602A (second area).

[0124] Video 601 is video from a program, and therefore changes over time. For example, assume that image 602 (still image) includes a landscape object of a shrine. Extraction unit 201 extracts the landscape object from the still image. Object classification unit 202 classifies the landscape object into landscape-type objects. Then, first display control unit 207 superimposes a green frame 605 (a frame representing a landscape-type object) around the landscape object. As a result, image 602 as shown in FIG. 13 is displayed in area 602A.

[0125] Thereafter, the video of the program changes over time. Fig. 14 is a diagram showing an example of a screen transition from the image in Fig. 13. Screen 611S in Fig. 14 is a screen transitioned to from screen 601S in Fig. 13. For example, suppose the user presses search button 303 on remote control 104 again (presses search button 303 for the second time).

[0126] First display control unit 207 divides the area of ​​display unit 101A into four areas and allocates images to three of the four areas. First display control unit 207 displays video 611 of a program in area 611A (first area) in Fig. 14. Video 611 changes over time.

[0127] In area 612A, first display control unit 207 displays image 612, which is the same as image 602 in area 602A in Fig. 13. However, image 612 has been reduced in size from image 602 in Fig. 13. In addition, first display control unit 207 displays image 613, a still image acquired at the time when the operation of pressing search button 303 for the second time is accepted, in area 613A. Areas 612A and 613A correspond to the second area.

[0128] For example, image 613 (still image) includes a person object. Extraction unit 201 extracts the person object from the still image. Object classification unit 202 classifies the person object into a person-type object. Then, first display control unit 207 superimposes a blue frame 606 (a frame representing a person-type object) around the landscape object. As a result, image 613 as shown in FIG. 14 is displayed in area 613A.

[0129] Thereafter, the video of the program changes over time. Screen 621S is a screen transitioned to from screen 611S. For example, suppose the user presses search button 303 on remote control 104 again (presses search button 303 for the third time).

[0130] First display control unit 207 allocates four images to the four divided areas of display unit 101A. Then, first display control unit 207 displays video 621 of a program in area 621A (first area) in Fig. 14. This video 621 changes over time.

[0131] In area 622A, first display control unit 207 displays image 612, which is the same as image 602 in area 602A in Fig. 13. In area 603A, first display control unit 207 displays image 623, which is the same as the image in area 613A.

[0132] Furthermore, first display control unit 207 displays image 624, which is a still image acquired at the time when first display control unit 207 receives the operation of pressing search button 303 for the third time, in area 624A. Areas 622A, 623A, and 624A correspond to the second area.

[0133] For example, image 624 (still image) includes a cat object. Extraction unit 201 extracts the cat object from the still image. Object classification unit 202 classifies the cat object into an object of the item type. Then, first display control unit 207 superimposes a red frame 607 (a frame representing an object of the item type) around the cat object. As a result, image 624 as shown in FIG. 14 is displayed in area 624A.

[0134] Here, it is assumed that the user presses the red button 304R on the remote control 104. The related information acquisition unit 203 extracts an image of a cat object from the image 624 including the object of the item type. The related information acquisition unit 203 transmits the extracted image of the cat object to the external server 102 via the second communication unit 126.

[0135] External server 102 acquires first related information of an image that is closest to the received image of the cat object, and transmits the first related information to television 101. Assume that the first related information is, for example, "Persian cat." Second display control unit 208 superimposes the received first related information on image 624 as related information 630. As a result, the related information of the cat object, "Persian cat," is displayed, as shown in area 624A of screen 621S in FIG. 14 .

[0136] As described above, display unit 101A displays the video of the program and the still image acquired at the time of pressing search button 303. This allows the user to view both the video of the program, the image of which changes over time, and the still image containing the object that the user is attempting to search for.

[0137] Display unit 101A also displays the video of the program and a plurality of still images acquired when the operation of pressing search button 303 is performed. The plurality of still images are images acquired when the operation of pressing search button 303 is performed at different times.

[0138] This allows the user to reserve a plurality of objects displayed at different times in the video of the program as search candidates, and later select one of the plurality of objects to recognize related information.

[0139] 14 shows an example in which display unit 101A is divided into four regions, but the number of divided regions may be any number. Furthermore, if the image of a region contains multiple objects of the same type, first display control unit 207 may superimpose frames around the objects and add different number information around each of the multiple objects of the same type. This allows the object that the user intends to search for to be identified as a single object.

[0140] Sixth Embodiment Next, a sixth embodiment will be described. The sixth embodiment is an example in which search content for an object to be searched is set in advance. Fig. 15 is an example table showing a first example of search content. For example, the table in the example of Fig. 15 is stored in the storage unit 123.

[0141] For example, the user presses a predetermined button (menu button) on remote control 104. Control unit 121 causes display unit 101A to display a menu screen for setting search contents.

[0142] For example, on the menu screen, it is possible to specify the type of object for which related information is to be searched from among a plurality of types of objects, and also to specify search content for the specified object.

[0143] 15, the main category represents each type of a plurality of types of objects, and the subcategory represents the search content of which content of each type of a plurality of objects is to be searched.

[0144] For example, suppose that the user performs an operation on the menu screen to specify "people" as the search target (main category) and "name" as the search content (subcategory). In response to receiving this operation, the search content setting unit 204 sets "SET" to "people" in the main category and sets "SET" to "name" in the subcategory.

[0145] This allows the television 101 to recognize that the type of object that the user intends to search for is a "person" and that the user intends to search for the "name" of the person to be searched for.

[0146] For example, suppose that while a video of a program is being displayed on display unit 101A, the user presses search button 303 without pressing any of the color buttons or number buttons 302 on remote control 104. In response to receiving this operation, related information acquisition unit 203 refers to the table in Fig. 15. In the table in Fig. 15, the type of object to be searched for is set to "person."

[0147] 15, the related information acquisition unit 203 acquires an image of the person object 401 from among the multiple types of objects extracted by the extraction unit 201. The related information acquisition unit 203 transmits information indicating that the name is the search content to the external server 102 together with the image of the person object 401.

[0148] The external server 102 transmits, as the first related information, name information (for example, "A-ko Oyama") from the first related information corresponding to the image closest to the image of the person object 401 to the television 101. The second display control unit 208 superimposes the first related information as related information 430 (information "A-ko Oyama") near the person object 401, as shown in the example of FIG.

[0149] In the sixth embodiment, as described above, the type of object to be searched and the search content are set in advance by the search content setting unit 204. Therefore, the type of object to be searched and the search content can be specified without the user having to press any of the color buttons or the number buttons 302. This reduces the amount of user operation.

[0150] FIG. 16 is a diagram showing a first example of a screen transition in the sixth embodiment. In FIG. 16, the second display control unit 208 may display an image 400H on which related information is superimposed for a certain period of time. This allows the user to recognize related information of the object that the user intends to search for for a certain period of time. Then, after the certain period of time has elapsed, the second display control unit 208 may switch from the image 400H to a video of the program. This causes a video on which no related information is superimposed to be displayed on the display unit 101A. This is the same for each embodiment.

[0151] In the example table of Figure 15, an example is shown in which one object type and one search content are set, but the search content setting unit 204 may set two or more types of objects out of multiple types of objects as search targets based on user operation.

[0152] For example, suppose that two or more types of objects can be specified on the above-mentioned menu screen. On the menu screen, the user performs an operation using remote control 104 to specify a person, an object, and text as the types of objects to be searched. Also, suppose that on the menu screen, the user performs an operation using remote control 104 to specify a name for a person, a price for an object, and a translation for text.

[0153] Fig. 17 is an example of a table showing a second example of search content. The table of the example of Fig. 17 is set in the table of the example of Fig. 17 by the search content setting unit 204 based on the above operation. For example, the table of the example of Fig. 17 is stored in the storage unit 123.

[0154] Fig. 18 is a diagram showing a second example of a screen transition in the sixth embodiment. As shown in Fig. 16, it is assumed that the search button 303 is pressed when a video of a program is displayed on the display unit 101A.

[0155] 17, the object to be searched is set to "person," and the search content is set to "name." Therefore, as described above, second display control unit 208 causes display unit 101A to display image 400H in which name information acquired from external server 102 is superimposed as related information 430.

[0156] Thereafter, it is assumed that the user presses the search button 303 again (presses the search button 303 for the second time). The related information acquisition unit 203 refers to the table in Fig. 17 and recognizes that the type of object next to person in the main category, "item", and the search content, "price", are set.

[0157] 17, the related information acquisition unit 203 acquires an image of the first item object 402A from among the multiple types of objects extracted by the extraction unit 201. The related information acquisition unit 203 transmits information indicating that the search content is price as the search target to the external server 102 together with the image of the first item object 402A.

[0158] The external server 102 transmits name information (for example, "6000 yen") from the first related information corresponding to the image closest to the image of the first item object 402A to the television 101 as the first related information. The second display control unit 208 superimposes the first related information near the first item object 402A as related information 420 (the information "6000 yen"). As a result, the image 400I in FIG. 18 is displayed on the display unit 101A.

[0159] Thereafter, it is assumed that the user presses the search button 303 again (presses the search button 303 for the third time). The related information acquisition unit 203 refers to the table in Fig. 17 and recognizes that the type of object next to the item in the main category, "Landscape", has not been set.

[0160] In this case, the related information acquisition unit 203 recognizes that the object type "text" next to landscape in the main category is set. Based on the settings in the table of Fig. 17, the related information acquisition unit 203 acquires an image of the first text object 403A from among the multiple types of objects extracted by the extraction unit 201. The related information acquisition unit 203 transmits information indicating that translation is the search target to the external server 102 together with the image of the first text object 403A.

[0161] For example, the external server 102 performs character recognition processing on the image of the first character object 403A to extract the English characters "SHOPPING." The external server 102 translates the English characters "SHOPPING" into Japanese and generates character information "Kaimon."

[0162] The external server 102 transmits the translated text information (first related information) to the television 101. The second display control unit 208 superimposes the first related information near the first text object 403A as related information 440 ("Shopping"). As a result, the image 400J in FIG. 18 is displayed on the display unit 101A.

[0163] As described above, when search content setting unit 204 sets two or more types of objects and their respective search contents based on user operation, the related information displayed on display unit 101A is switched each time the user presses search button 303. That is, second display control unit 208 switches the search contents set by search content setting unit 204 and displays them on display unit 101A. As a result, when the user has specified multiple search targets in advance, the related information of the object intended for the search can be switched and displayed simply by pressing search button 303.

[0164] <Other> In each of the above embodiments, an example has been described in which the related information acquisition unit 203 acquires related information from the external server 102. However, the related information acquisition unit 203 may acquire related information from the storage unit 123. In this case, the storage unit 123 stores images of a plurality of objects stored in the external server 102 and first related information for each of the images of the plurality of objects. Even in this case, the effects of each embodiment can be obtained.

[0165] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment. Furthermore, the order of the processes in the flowcharts described in the above-described embodiments can be changed as much as possible.

[0166] The program that realizes the functions of this embodiment is stored in a non-transitory recording medium such as a semiconductor medium, an optical recording medium, or a magneto-optical recording medium. For example, a non-volatile memory card can be used as the semiconductor medium. For example, a CD (Compact Disc) or a DVD (Digital Versatile Disc) can be used as the optical recording medium or magneto-optical recording medium. The program may also be supplied to a computer via any transmission medium that can transmit the program. [Explanation of symbols]

[0167] 100 System, 101 Television, 101A Display Unit, 102 External Server, 103 Communication Terminal, 104 Remote Control, 121 Control Unit, 122 Broadcast Wave Receiving Unit, 123 Memory Unit, 124 Remote Control Signal Receiving Unit, 125 First Communication Unit, 126 Second Communication Unit, 201 Extraction Unit, 202 Object Classification Unit, 203 Related Information Acquisition Unit, 204 Search Content Setting Unit, 205 Notification Control Unit, 206 Access Control Unit, 207 First Display Control Unit, 208 Second Display Control Unit, 302 Number Buttons, 303 Search Button, 304B Blue Button, 304R Red Button, 304G Green Button, 304Y Yellow Button

Claims

1. A display device, A display unit; a first display control unit that, in response to receiving a first operation from a user, causes a plurality of object specifying figures that specify a plurality of different types of objects included in a still image of the video displayed on the display unit to be superimposed on the still image; a second display control unit that, in response to receiving a second operation for selecting a target object identification figure from the plurality of object identification figures, causes the display unit to display related information relating to the object corresponding to the target object identification figure; A display device comprising:

2. an extraction unit that performs image processing on a still image of the video to extract objects, and extracts the plurality of types of objects from the still image of the video; The display device according to claim 1 , wherein the first display control unit superimposes an object-specifying figure around each of the extracted types of objects on a still image of the video.

3. The display device according to claim 2 , further comprising an object classification unit that classifies the types of the plurality of types of objects by applying a predetermined image classification method to each image of the plurality of types of objects that have been extracted.

4. a related information acquisition unit that acquires, in response to receiving the second operation, first related information associated with an object corresponding to the target object specifying figure from a storage device that stores a plurality of objects including the plurality of types of objects and first related information associated with each of the plurality of objects, The display device according to claim 1 , wherein the second display control unit causes the display unit to display the acquired first related information as the related information.

5. The display device according to claim 1 , wherein the first display control unit superimposes the plurality of types of object-specific figures on the still image in different display modes.

6. An operation receiving unit that receives an operation signal from an operation device that performs an operation on the display device, the different display modes are different colors, The display device according to claim 5 , wherein the colors of the plurality of object-specifying figures correspond to the colors of a plurality of first buttons among a plurality of buttons of the operation device.

7. 2. The display device according to claim 1, wherein the first display control unit divides the display unit into a plurality of areas, displays still images in each of the plurality of areas, assigns the plurality of object-specific figures to the plurality of areas, and superimposes the assigned object-specific figures on the still images displayed in the plurality of areas.

8. The display device according to claim 7 , wherein the first display control unit displays information indicating the plurality of types of objects corresponding to the plurality of object-identifying figures in corresponding areas of the plurality of areas.

9. the first display control unit adds number information indicating numbers assigned to a plurality of second buttons provided on the operation device to a plurality of first object specifying figures of the same type among the plurality of object specifying figures; The display device according to claim 6 , wherein the second display control unit causes the display unit to display the related information corresponding to a number of a second button for which a selection operation has been accepted from among the plurality of second buttons.

10. a communication unit that communicates with the terminal; a notification control unit that notifies the terminal of the character information via the communication unit when the object corresponding to the target object identification figure selected by the second operation is character information indicating a predetermined access destination; The display device according to claim 1 , further comprising:

11. a communication unit that communicates with the terminal; an access control unit that controls the terminal to access the access destination via the communication unit when the object corresponding to the target object identification figure selected by the second operation is text information indicating a predetermined access destination; The display device according to claim 1 , further comprising:

12. 2. The display device of claim 1, wherein the first display control unit divides the display unit into a plurality of regions, displays the video in a first region of the plurality of regions, displays the still image in a second region of the plurality of regions, assigns the plurality of object-identifying figures to the second region, and superimposes the plurality of object-identifying figures on the still image displayed in the plurality of second regions.

13. a storage unit configured to store a plurality of first still images based on a plurality of first operations received at different times, 2. The display device of claim 1, wherein the first display control unit divides the display unit into a plurality of regions, displays the video in a first region among the plurality of regions, displays the plurality of first still images in a plurality of second regions among the plurality of regions, and assigns the plurality of object-identifying figures to the plurality of second regions and superimposes the assigned plurality of object-identifying figures on the still images displayed in the plurality of second regions.

14. a search content setting unit that sets search content to be searched for for each of the plurality of types of objects; The display device according to claim 1, wherein the second display control unit, in response to receiving the second operation, causes the display unit to display search results based on the search content for an object corresponding to the target object-identifying figure as the related information.

15. a search content setting unit that sets search content to be searched for for each of the plurality of types of objects; the second operation is an operation of switching and selecting a target object-identifying graphic from the plurality of object-identifying graphics, The display device according to claim 1, wherein the second display control unit, in response to receiving the second operation, switches search results based on the search content for objects corresponding to the target object-specific figures of each of the plurality of types of objects and displays them on the display unit.

16. A method for controlling a display device having a display unit, comprising: In response to receiving a first operation from a user, displaying on the display unit a plurality of object identifying figures that identify a plurality of different types of objects included in a still image of the video displayed on the display unit, superimposed on the still image; displaying, on the display unit, related information relating to the object corresponding to the target object identification graphic in response to receiving a second operation for selecting the target object identification graphic from the plurality of object identification graphics; A method for controlling a display device.

Images