Control device, display system, and program

Abstract

To enable display content to be more easily grasped, as compared with the case where alteration of display is made for all images being displayed on a display unit when there is a change of position of an operation terminal relative to the display unit as seen through the display unit by a user.SOLUTION: In cases when a terminal device 300 is positioned behind a display unit 101, a CPU changes display of some of the images displayed on the display unit 101 in accordance with a change of the position of the terminal device 300 relative to a display device 100. Furthermore, even when there is a change of the position of the terminal device 300, the CPU ensures that the other of the images displayed on the display unit 101 are maintained so as to be displayed at a predetermined place, irrespective of the position of the terminal device 300.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a control device, a display system, and a program. [Background technology]

[0002] Patent Document 1 discloses a process in which an application management and communication system manages and starts applications in a wearable display device via a user interface of a portable display device. Patent Document 2 discloses a multi-display system including an information terminal that displays a real screen and an augmented reality display device that displays a virtual screen, which is an augmented reality display screen different from the real screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-507970 A [Patent Document 2] JP 2014-174507 A Summary of the Invention [Problem to be solved by the invention]

[0004] When a terminal device referenced by a user is displayed on a display unit referenced by the user, changing the display content of the display unit in response to the movement of the terminal device makes it possible, for example, to change the display position of an image displayed on the display unit in response to the position of the terminal device. However, when the operating terminal is moved, if the display of all images displayed on the display unit is changed and the layout of the display unit is significantly changed, it becomes difficult to grasp the display contents of the display unit. The object of the present invention is to make it easier for a user to understand the displayed content when there is a change in the position of the operating terminal relative to the display unit as viewed by the user through the display unit, compared to when the display of all images displayed on the display unit is changed. [Means for solving the problem]

[0005] The invention described in claim 1 is a control device including a processor and controlling a display device including a display unit, the control device controlling the display device allowing a user to view the back of the display unit through the display unit when the user views the display unit, the processor changes the display of a portion of an image displayed on the display unit in response to a change in the position of a terminal device located behind the display unit and viewed by the user through the display unit relative to the display device, and the control device maintains a state in which the other portion of an image displayed on the display unit is displayed at a predetermined location regardless of the change in the position of the terminal device. and the processor controls the control device so that, among images displayed in sequence on the display unit by a user operating the terminal device, an image displayed at a predetermined location on the display unit is maintained in the predetermined location regardless of a change in the position of the terminal device. It is. The invention described in claim 2 is The processor is a control device as described in claim 1, which causes an image displayed on the upper or lower side of the display unit, among images displayed in sequence on the display unit by a user operating the terminal device, to remain displayed on the upper or lower side regardless of a change in the position of the terminal device. In the invention described in claim 3, the processor performs the following in response to a change in the position of the terminal device: Among the images sequentially displayed on the display unit, an image displayed at a location other than the predetermined location, A display change is performed for an image having a first positional relationship with the terminal device that is seen through the display unit. cormorant, The control device according to claim 1. The invention described in claim 4 further comprises: an image displayed in a portion other than the predetermined portion among the images displayed in sequence on the display unit in response to a change in the position of the terminal device, The display of the image displayed next to the terminal device that is visible through the display unit is changed. cormorant 3 is a control device according to claim 2. The invention described in claim 5 is the control device described in claim 3, wherein the processor changes a display position of an image having the first positional relationship in response to a change in the position of the terminal device. The invention described in claim 6 is the control device described in claim 3, wherein the processor moves the image having the first positional relationship in the direction in which the terminal device is moving in response to a change in the position of the terminal device. The invention recited in claim 7 is the control device recited in claim 1, wherein the processor displays, on the display unit, information about a selection item being displayed on the terminal device. The invention described in claim 8 is the control device described in claim 7, wherein the processor displays the information about the selection item on the display unit when a user's finger approaches the selection item. The invention described in claim 9 is a control device including a processor and controlling a display device including a display unit, the control device controlling the display device allowing a user to view the back of the display unit through the display unit when the user views the display unit, the processor changes the display of a part of an image displayed on the display unit in response to a change in the position of a terminal device located behind the display unit and viewed by the user through the display unit relative to the display device, and the control device maintains a state in which the other part of the image displayed on the display unit is displayed at a predetermined location regardless of the change in the position of the terminal device. and the processor changes the display of a higher-ranking image, which is an image that is higher in the display order among images displayed in order on the display unit by a user operating the terminal device, and a lower-ranking image, which is an image that is lower in the display order than the higher-ranking image, in response to a change in the position of the terminal device, and maintains the lower-ranking image in a state where it is displayed at a predetermined position regardless of the change in the position of the terminal device. It is. The invention recited in claim 10 is the control device recited in claim 9, wherein the processor changes a display position of the higher-level image in response to a change in the position of the terminal device. The invention described in claim 11 is the control device described in claim 10, wherein the processor changes the display position of the higher-level image to a position that does not overlap with the terminal device whose position is changing. The invention described in claim 12 is the control device described in claim 9, in which the processor moves the upper image displayed on the display unit so that it accompanies the terminal device that is visible through the display unit. The invention described in claim 13 is the control device described in claim 9, wherein the processor maintains the sub-image in a state in which the sub-image is displayed near either the upper or lower end of the display unit regardless of changes in the position of the terminal device. The invention described in claim 14 is the control device described in claim 9, wherein the processor causes the sub-image to be displayed at a specific location behind the display unit and visible through the display unit. Claim 15 The present invention relates to a display device having a display unit and a control device that controls the display device, the display device being a display device that allows a user to view the back of the display unit through the display unit when the user views the display unit, and the control device being a display device that is capable of displaying a display image of a user's face in a manner as described above. 4 A display system including the control device according to any one of the above. Claim 16 The invention described in the item (1) changes the display of a part of an image displayed on a display unit in response to a change in the position of a terminal device, which is located behind a display unit provided on a display device and which is visually recognized by a user through the display unit, with respect to the display device. change The function and other part of the image displayed on the display unit are maintained in a state where they are displayed at a predetermined location regardless of the change in the position of the terminal device. maintain A function and a program to realize it on a computer the maintaining function is a program for maintaining a state in which an image displayed at a predetermined position on the display unit, among images displayed in sequence on the display unit by a user operating the terminal device, is displayed at the predetermined position regardless of a change in the position of the terminal device. It is. The invention described in claim 17 changes the display of a part of an image displayed on a display unit in response to a change in the position of a terminal device located behind a display unit provided on a display device and visually recognized by a user through the display unit, relative to the display device. change The function and other part of the image displayed on the display unit are maintained in a state where they are displayed at a predetermined location regardless of the change in the position of the terminal device. maintain A function and a program to realize it on a computer the change function changes the display of a higher-ranking image, which is an image with a higher display rank among images displayed in sequence on the display unit by a user operating the terminal device, and a lower-ranking image, which is an image with a lower display rank than the higher-ranking image, in response to a change in the position of the terminal device; and the maintenance function maintains the lower-ranking image in a state where it is displayed at a predetermined position regardless of the change in the position of the terminal device. It is. Effect of the Invention

[0006] Claim 1 、2According to the invention, when the position of the operating terminal relative to the display unit as viewed by the user through the display unit changes, it becomes easier for the user to understand the displayed content compared to when the display of all images displayed on the display unit is changed. According to the invention of claim 3, the display of some of the images displayed on the display unit can be changed, while the display of the other images can be maintained in a predetermined location. According to the invention of claim 4, it is possible to maintain the state in which an image displayed on the display unit, which is located above or below the terminal device visible through the display unit, is displayed at a predetermined location. According to the invention of claim 5, the display position of the image can be changed in response to a change in the position of the terminal device. According to the invention of claim 6, the image can be moved in the direction in which the terminal device moves in response to a change in the position of the terminal device. According to the invention of claim 7, it becomes possible to notify the user of information about the selection items displayed on the terminal device via the display unit. According to the invention of claim 8, it is possible to notify the user of information about a selection item that is displayed on the terminal device and that the user's finger has approached. According to the invention of claim 9, it is possible to maintain a state in which a lower-ranked image, which is an image that is lower in the display order among images displayed in sequence on the display unit, is displayed in a predetermined location. According to the invention of claim 10, the display position of a higher-ranked image, which is an image that is displayed at the top of the display order among images displayed in sequence on the display unit, can be changed in response to a change in the position of the terminal device. According to the invention of claim 11, the upper image can be displayed at a position that does not overlap with the terminal device whose position changes. According to the invention of claim 12, the upper image can be moved so as to accompany the terminal device seen through the display unit. According to the invention of claim 13, the state in which the lower image is brought close to either the upper end or the lower end of the display section and displayed can be maintained. According to the invention of claim 14, a lower image can be displayed at a specific location behind the display unit, which can be seen through the display unit. Claim 15 According to the invention, when the position of the operating terminal relative to the display unit as viewed by the user through the display unit changes, it becomes easier for the user to understand the displayed content compared to when the display of all images displayed on the display unit is changed. Claim 16 According to the invention, when the position of the operating terminal relative to the display unit as viewed by the user through the display unit changes, it becomes easier for the user to understand the displayed content compared to when the display of all images displayed on the display unit is changed. Claim 17 According to the invention, when the position of the operating terminal relative to the display unit as viewed by the user through the display unit changes, it becomes easier for the user to understand the displayed content compared to when the display of all images displayed on the display unit is changed. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of a display system. [Diagram 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of a control device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a terminal device. [Figure 4] FIG. 4 is a diagram showing a display example of a display unit. [Diagram 5] 13A and 13B are diagrams showing other display examples on the display unit. [Figure 6] 13A and 13B are diagrams showing other display examples on the display unit. [Figure 7] 13A and 13B are diagrams showing other display examples on the display unit. [Figure 8] 13A and 13B are diagrams showing other display examples on the display unit. [Figure 9] 13A and 13B are diagrams showing other display examples on the display unit. [Figure 10] 13A and 13B are diagrams showing other display examples on the display unit. [Figure 11] 13A and 13B are diagrams showing other display examples on the display unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. FIG. 1 is a diagram showing an example of a display system 1 according to the present embodiment. The display system 1 of this embodiment is provided with a display device 100 and a control device 700 that controls the display device 100. The display device 100 is a so-called AR (Augmented Reality) display device that displays an image in a real space visually recognized by a user. The display device 100 is provided with a display unit 101 that is visually recognized by a user. The display unit 101 is formed in a rectangular shape. The display unit 101 is not particularly limited as long as it can display an image, and is configured, for example, by a liquid crystal display or an organic EL display.

[0009] The display device 100 is also provided with a housing 102 that supports the display unit 101. The display device 100 is also provided with a speaker (not shown) that outputs sound, and operation buttons 103 that accept instructions from a user. Furthermore, the display device 100 may be a so-called head-mounted or glasses-type display device, in which case a mounting part that is mounted on the user's head is provided.

[0010] The display device 100 may also be a tablet-type display device 100 as shown in FIG. 1, in which case the display unit 101 is provided on one surface side of a plate-shaped housing 102. Hereinafter, in this specification, a head-mounted or eyeglass-type display device 100 will be referred to as a "wearable display device 100," and a display device 100 that is supported by the user's hand, such as a tablet-type display device 100, will be referred to as a "supported display device 100."

[0011] Furthermore, the display device 100 of this embodiment is provided with a camera 104 that captures an image behind the display unit 101 . Here, "behind the display unit 101" refers to the back side of the image display surface 101A of the planar display unit 101. In other words, "behind the display unit 101" refers to a location that is located ahead of the line of sight of a user viewing the display unit 101 from the front, and is located on the back side of the image display surface 101A of the display unit 101. A camera 104 provided on the display device 100 captures an image of the rear of the display unit 101 from the upstream side in the direction of the line of sight of a user viewing the display unit 101, and obtains an image of this rear.

[0012] In this embodiment, a case in which a control device 700 is provided separately from the display device 100 will be described as an example. However, the present invention is not limited to this, and the control device 700 may be provided as a part of the display device 100. In other words, the control device 700 may be provided inside the housing 102 provided in the display device 100. As described above, the display device 100 is a so-called AR display device, and in this embodiment, when a user views the display unit 101, the user views the objects and situations behind the display unit 101 through the display unit 101.

[0013] In order to enable the user to visually recognize the area behind the display unit 101 through the display unit 101, for example, an image captured by the camera 104 that captures an image behind the display unit 101 is displayed on the display unit 101. Alternatively, for example, the display unit 101 may be configured to transmit light. In this case, the display unit 101 is see-through, and the user can view what is behind the display unit 101 through the see-through display unit 101.

[0014] Thus, there are two types of display devices 100: a display device 100 equipped with a display section 101 that transmits light, and a display device 100 equipped with a display section 101 that does not transmit light and displays images obtained by a camera 104. In the following description, where necessary, a display device 100 having a display section 101 through which light passes will be referred to as a "transmissive display device 100," and a display device 100 having a display section 101 through which light does not pass and which displays an image obtained by a camera 104 will be referred to as a "non-transmissive display device 100."

[0015] In other words, in this specification, a display device 100 in which the user can see behind the display unit 101 through the transparent display unit 101 will be referred to as a "transparent display device 100," and a display device 100 in which an image captured by the camera 104 is displayed on the display unit 101 will be referred to as a "non-transparent display device 100." In this embodiment, images other than the video captured by the camera 104, such as content to be described later, are displayed on the display unit 101. As a result, the image displayed on the display unit 101 is superimposed on the real space RW that is located behind the display unit 101 and is visually recognized by the user.

[0016] [Explanation of the control device 700] FIG. 2 is a diagram showing an example of the hardware configuration of the control device 700. As shown in FIG. The control device 700 includes a control unit 101, an information storage device 102 for storing information, and a network interface 103 for realizing communication via a LAN (Local Area Network) cable or the like.

[0017] The control unit 101 has a CPU (=Central Processing Unit) 111 as an example of a processor, a ROM (=Read Only Memory) 112 in which basic software, a BIOS (=Basic Input Output System), etc. are stored, and a RAM (=Random Access Memory) 113 used as a work area. The CPU 111 may be a multi-core processor. The ROM 112 may be a rewritable non-volatile semiconductor memory. The control unit 101 is a so-called computer.

[0018] The information storage device 102 is, for example, a hard disk drive, although the information storage device 102 may also be a semiconductor memory or a magnetic tape. The control unit 101, the information storage device 102, and the network interface 103 are connected via a bus 104 and a signal line (not shown).

[0019] Here, the program executed by CPU 111 can be provided to the control device 700 in a state where it is stored in a computer-readable recording medium such as a magnetic recording medium (magnetic tape, magnetic disk, etc.), an optical recording medium (optical disk, etc.), a magneto-optical recording medium, or a semiconductor memory. Furthermore, the programs executed by the CPU 111 may be provided to the control device 700 using a communication means such as the Internet.

[0020] In this embodiment, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.). In addition, the operations of the processor may not only be performed by one processor, but may be performed by multiple processors in physically separate locations working together. The order of the operations of the processor is not limited to the order described in this embodiment, and may be changed.

[0021] [Explanation of display processing] In this embodiment, a CPU 111 as an example of a processor controls a display unit 101 provided in a display device 100, and changes the display on the display unit 101. Specifically, when a terminal device 300 (described later) is located behind the display unit 101, the CPU 111 changes the display of some of the images displayed on the display unit 101 in response to a change in the position of the terminal device 300 relative to the display device 100.

[0022] More specifically, the CPU 111 changes the display of some of the images displayed on the display unit 101 in response to a change in the position of the terminal device 300, which is located behind the display unit 101 and viewed by the user through the display unit 101, relative to the display device 100. "The user views through the display unit 101" includes a situation in which the user directly views an object that is visible through the display unit 101, and a situation in which the user views an image captured by the above-mentioned camera 104 provided on the display device 100 by looking at the display unit 101. In other words, "the user views through the display unit 101" includes a situation in which an object behind the display unit 101 is viewed through the display unit 101 provided in the transmissive display device 100, and a situation in which the user views an image of the object reflected on the display unit 101 provided in the non-transmissive display device 100.

[0023] On the other hand, even if a terminal device 300 is positioned behind the display unit 101 and the position of the terminal device 300 changes, the CPU 111 ensures that other images displayed on the display unit 101 remain displayed in predetermined locations regardless of the change in the position of the terminal device 300. In this embodiment, the positions of all images displayed on the display unit 101 are not changed, but some images are maintained in a state where they are displayed in a predetermined position.

[0024] [Description of Terminal Device 300] The "terminal device 300" refers to a device that can be carried by a user. Examples of the terminal device 300 include a smartphone and a tablet. Other examples of the terminal device 300 include a game console, a smart watch, and a mobile notebook PC.

[0025] Fig. 3 is a diagram showing an example of a hardware configuration of the terminal device 300. The configuration shown in Fig. 3 assumes that the terminal device 300 is a smartphone. The terminal device 300 has a control unit 201 that controls the operation of the entire device, a memory card 202 that stores various data, various communication interfaces 203 that comply with wireless communication standards, and an input device 204 such as a touch sensor. The terminal device 300 also includes a terminal display unit 205, which is a display unit configured with a liquid crystal display, an organic EL (Electro Luminescence) display, or the like, and a GPS (Global Positioning System) sensor 206.

[0026] The control unit 201 includes a CPU 211, a ROM 212 in which firmware, BIOS, and the like are stored, and a RAM 213 used as a work area. The CPU 211 may be a multi-core processor. The ROM 212 may be a rewritable non-volatile semiconductor memory.

[0027] The communication interface 203 is, for example, an interface used for connection to a mobile communication system, or an interface used for connection to a wireless LAN. The GPS sensor 206 is a sensor that receives radio waves from GPS satellites and measures the position of the terminal device 300. Information of latitude, longitude, and altitude output from the GPS sensor 206 provides the current position of the terminal device 300. The GPS sensor 206 may be compatible with an indoor positioning system.

[0028] In this embodiment, when the user moves the terminal device 300, the CPU 111 changes the display of some of the images displayed on the display unit 101 as described above. Furthermore, the CPU 111 controls other images displayed on the display unit 101 so that they are maintained in a predetermined location even if the user moves the terminal device 300.

[0029] [Specific examples of displays on the display unit 101] FIG. 4 is a diagram showing a display example of the display unit 101. As shown in FIG. In this display example, a first virtual screen 510 and a third virtual screen 530 are displayed on the display unit 101. The second virtual screen 520 will be described later. In this display example, the terminal device 300 is located behind the display unit 101 and is held by the user in his / her left hand (not shown). Here, the term "virtual screen" refers to an image displayed on the display unit 101 under the control of the control device 700, and which can be viewed by the user together with the real space RW existing behind the display unit 101.

[0030] In this embodiment, the first virtual screen 510 is located above the terminal device 300 that is visible through the display unit 101. The CPU 111 maintains the first virtual screen 510 in a state where it is displayed at a predetermined position even if the position of the terminal device 300 changes. Specifically, the first virtual screen 510 is displayed at a predetermined location, that is, the upper side of the display unit 101, and the CPU 111 causes this first virtual screen 510 to remain displayed at the upper side of the display unit 101 regardless of any change in the position of the terminal device 300. More specifically, the CPU 111 outputs a control signal for displaying the first virtual screen 510 on the upper side of the display unit 101 even if the terminal device 300 is moved.

[0031] In this example, a case will be described in which first virtual screen 510 is positioned on the upper side of display unit 101 and the state in which first virtual screen 510 is displayed on this upper side is maintained. However, the present invention is not limited to this, and first virtual screen 510 may be displayed on the lower side of display unit 101, and the state in which first virtual screen 510 is displayed on the lower side of display unit 101 may be maintained. Furthermore, without being limited to this, for example, first virtual screen 510 may be displayed on the right or left side of display unit 101, and the state in which first virtual screen 510 is displayed on this right or left side may be maintained.

[0032] The third virtual screen 530, which is the target of the display change, will be described. The CPU 111 changes the display position of the third virtual screen 530 in response to a change in the position of the terminal device 300. Specifically, the CPU 111 moves the third virtual screen 530 in the direction in which the terminal device 300 moves in response to a change in the position of the terminal device 300. Specifically, for example, when the terminal device 300 moves leftward from the state shown in FIG. 4 to the state shown in FIG. 5 (a diagram showing another display example on the display unit 101), the CPU 111 moves the third virtual screen 530 so that the third virtual screen 530 moves along with the terminal device 300.

[0033] Specifically, when performing this process, the CPU 111 recognizes the position of the terminal device 300, and determines the display position of the third virtual screen 530 so that the third virtual screen 530 is displayed next to the recognized position of the terminal device 300. Then, the CPU 111 outputs a control signal to display the third virtual screen 530 at this determined position. As a result, the third virtual screen 530 moves in conjunction with the terminal device 300 in this embodiment.

[0034] The CPU 111 recognizes the position of the terminal device 300, for example, by analyzing an image captured by the camera 104 provided on the display device 100. The recognition of the terminal device 300 may be performed by recognizing the terminal device 300 itself, or, for example, by displaying an identification image for indicating the terminal device 300 on the terminal display unit 205 provided on the terminal device 300, and recognizing this image, thereby recognizing the terminal device 300.

[0035] In this embodiment, a case where the third virtual screen 530 moves will be described as an example of a change in the display of the third virtual screen 530. However, other than the movement, other examples of the change in the display include enlargement or reduction of the third virtual screen 530. For example, when the terminal device 300 moves from the state shown in FIG. 4 in a direction away from the display unit 101 and toward the user's line of sight (hereinafter referred to as the "line of sight direction"), the third virtual screen 530 may be reduced.

[0036] Also, for example, when the terminal device 300 moves from the state shown in FIG. 4 in a direction approaching the display unit 101 and opposite to the line of sight, the third virtual screen 530 may be enlarged. Alternatively, the third virtual screen 530 may be made to move in conjunction with the terminal device 300 moving in the left-right or up-down direction, and the third virtual screen 530 may be enlarged or reduced in accordance with the movement of the terminal device 300 in the line of sight or in the direction opposite to the line of sight.

[0037] Furthermore, for example, when third virtual screen 530 moves from the state shown in FIG. 4 to right end portion 101C of display unit 101, the display area of ​​third virtual screen 530 disappears. In this case, the display position of the third virtual screen 530 with respect to the terminal device 300 may be changed as shown in FIG. 6 (a diagram showing another display example on the display unit 101). In the example shown in FIG. 6, the left side of the terminal device 300 becomes the new display position of the third virtual screen 530, and the third virtual screen 530 is displayed at this new display position.

[0038] In addition, for example, it is also conceivable that terminal device 300 moves upward in the figure from the state shown in FIG. 4, and as a result, first virtual screen 510 and third virtual screen 530 overlap each other. In addition to the terminal device 300 moving upward in the figure, the first virtual screen 510 and the third virtual screen 530 can also overlap when, for example, a user wearing the wearable display device 100 moves his or her head from the state shown in Figure 4, or when the user moves the supportable display device 100 downward. In this case, for example, the portion of third virtual screen 530 that overlaps with first virtual screen 510 is erased, and first virtual screen 510 is displayed with priority.

[0039] In this case, the terminal device 300 is located behind the first virtual screen 510, but if the display device 100 is a transparent display device 100, the terminal device 300 may or may not be visible depending on the density of the first virtual screen 510. In addition, if the display device 100 is a non-transparent display device 100 and is configured to display an image of the terminal device 300 acquired by the camera 104 on the display unit 101, image processing may be performed to erase the portion of the image representing the terminal device 300 that overlaps with the first virtual screen 510.

[0040] The first virtual screen 510 and the third virtual screen 530 will be further described. In this embodiment, a first virtual screen 510 and a third virtual screen 530 are images that are displayed in sequence on the display unit 101 as a result of the user operating the terminal device 300. In this embodiment, when a user operates the terminal device 300 to sequentially select selection items 310 (described later) such as thumbnail images displayed on the terminal device 300, a first virtual screen 510 and a third virtual screen 530 are displayed in sequence on the display unit 101.

[0041] In this embodiment, the first virtual screen 510 and the third virtual screen 530 are images representing each of the contents, and when the user operates the terminal device 300 to select a selection item 310 such as a thumbnail image, the content corresponding to this selection item 310 is displayed on the display unit 101. The user operates the terminal device 300 present in the real space RW to select the selection item 310. In other words, the user selects the selection item 310 by actually touching the terminal device 300 present in the real space RW.

[0042] In response to this, in this embodiment, an image (content) corresponding to the selection item 310 selected by the user is displayed on the display unit 101 as a virtual screen. More specifically, in this embodiment, information on the selection item 310 selected by the user is transmitted from the terminal device 300 to the control device 700 through a communication line. In this case, the CPU 111 of the control device 700 performs a process of downloading the content corresponding to the selection item 310 selected by the user from an external server, and further displays the downloaded content on the display unit 101. As a result, the content corresponding to the selection item 310 selected by the user is displayed on the display unit 101 of the display device 100.

[0043] The third virtual screen 530 (see FIG. 4) is an upper image, which is an image that is ranked higher on the display unit 101, and the first virtual screen 510 is a lower image, which is an image that is ranked lower than the upper image, the third virtual screen 530. In other words, the third virtual screen 530 is a screen corresponding to the previously selected item, which is the selection item 310 selected first by the user, and the first virtual screen 510 is an image corresponding to the selection item 310 selected after the previously selected item.

[0044] In this embodiment, it is assumed that the user operates the terminal device 300 and first selects the selection item 310 corresponding to the third virtual screen 530. In this case, the display on display unit 101 first becomes the state shown in FIG. 7 (a diagram showing another display example on display unit 101), and third virtual screen 530 is displayed on the upper side of display unit 101. At this time, depending on the position of the terminal device 300, the terminal device 300 and the third virtual screen 530 may overlap, as indicated by reference symbol 7A in FIG.

[0045] In this case, in order to make it easier to see the terminal display unit 205 of the terminal device 300, the user moves the terminal device 300 downward, or moves the head on which the wearable display device 100 is worn, or moves the supportable display device 100, as shown by the arrow 7B in the figure. As a result, the terminal device 300 is positioned at the location indicated by reference symbol 8A in Fig. 8 (a diagram showing another display example on the display unit 101). In this case, the user can easily view the terminal display unit 205 of the terminal device 300.

[0046] Next, in this example, the user further performs an operation on the terminal device 300 and selects the selection item 310 corresponding to the first virtual screen 510 (the selection item 310 indicated by reference symbol 8C in FIG. 8). As a result, as shown in FIG. 4, a first virtual screen 510 is newly displayed on the upper side of the display unit 101, and the third virtual screen 530 that had been displayed until then is displayed to the side of the terminal device 300.

[0047] In this embodiment, the virtual screen corresponding to the most recent selection item 310 selected by the user is displayed at the upper side of the display unit 101, which is a predetermined location. In this embodiment, the virtual screens corresponding to the selection items 310 selected by the user prior to the latest selection item 310 are displayed to the side of the terminal device 300.

[0048] In the display system 1 of this embodiment, it is possible to achieve both ease of operation for the user and ease of viewing of images. When an image is displayed on the display unit 101 that can be placed in front of the user's face, as in the display device 100 of this embodiment, the user can view the image on a large screen, making it easier for the user to view the image. On the other hand, if the user operates the display device 100 to select an option or the like without using the terminal device 300, the operability for the user will be reduced.

[0049] When the user operates the display device 100 to select the selection item 310, for example, a touch panel that accepts the selection from the user is displayed as a virtual screen on the display device 100. Furthermore, in this case, the operation of the user selecting the selection item 310 in the air is grasped by recognizing the position of the user's fingertip, etc. In this case, operability is reduced compared to when the user touches an actual screen to select the selection item 310. In addition, in this case, the accuracy of the selection of the selection item 310 is likely to decrease.

[0050] On the other hand, when the user selects the selection item 310 by performing an operation on the terminal device 300, the user performs the operation on the actual screen, so that the user can easily select the selection item 310. On the other hand, when the user visually views the terminal device 300 to view the content or the like, the image visually perceived by the user becomes small, making it difficult for the user to visually recognize the content or the like.

[0051] That is, when a user uses only the display device 100 or only the terminal device 300, this leads to a decrease in either visibility or operability. In contrast to this, in the present embodiment, the selection of the selection item 310 is accepted by the terminal device 300, and the content associated with the selection of the selection item 310 is displayed on the display unit 101 of the display device 100, thereby achieving both ease of operation for the user and ease of viewing of the image.

[0052] In this embodiment, when the third virtual screen 530 is displayed next to the terminal device 300, if the display device 100 is a non-transparent display device 100, the portion of the third virtual screen 530 that overlaps the terminal device 300 is not displayed, making it easier to view the terminal device 300. Moreover, in the case where the display device 100 is a transmissive display device 100, the portion of the third virtual screen 530 that overlaps with the terminal device 300 is similarly prevented from being displayed. Furthermore, in this embodiment, as described above, the display on the third virtual screen 530 is changed in response to a change in the position of the terminal device 300.

[0053] Specifically, in this embodiment, the CPU 111 changes the display position of the third virtual screen 530, which is the upper image, in response to a change in the position of the terminal device 300, as described above. Specifically, the CPU 111 determines, as the destination of the change in the display position of the third virtual screen 530, a position that does not overlap with the terminal device 300, whose position is changed. If the display position of the third virtual screen 530 is changed to a position that overlaps with the terminal device 300, it will be difficult to view the terminal device 300, so the display position of the third virtual screen 530 is changed to a position that does not overlap with the terminal device 300, whose position is changing.

[0054] In this specification, "a position that does not overlap with the terminal device 300" refers to a position other than a position where all of the third virtual screen 530 is contained within the inner peripheral edge 380 (see FIG. 5) of the terminal device 300 as viewed through the display unit 101. As shown in FIG. 5, when at least a part of the third virtual screen 530 is located outside the outer peripheral edge 380 of the terminal device 300, it can be said that the display position of the third virtual screen 530 is changed to a position that does not overlap with the terminal device 300.

[0055] In this embodiment, the CPU 111 moves the third virtual screen 530 displayed on the display unit 101 so as to accompany the terminal device 300 that is visible through the display unit 101 . During this movement, the third virtual screen 530 is moved so that the terminal device 300 and the third virtual screen 530 do not overlap each other. In other words, the third virtual screen 530 is moved so that the entire third virtual screen 530 is not contained inside the outer periphery 380 of the terminal device 300 . In other words, in this embodiment, the third virtual screen 530 is moved while the initial positional relationship between the terminal device 300 and the third virtual screen 530 is maintained.

[0056] 5, which is the lower image, is maintained in a predetermined position, regardless of changes in the position of the terminal device 300. That is, the CPU 111 maintains the first virtual screen 510 displayed on the upper side of the display unit 101. In other words, in this embodiment, the first virtual screen 510 is displayed in a state where it is close to the upper end 101E of the display unit 101, and the CPU 111 causes the first virtual screen 510 to remain in a state where it is displayed in a state where it is close to the upper end 101E of the display unit 101, regardless of any change in the position of the terminal device 300.

[0057] In addition, the first virtual screen 510 may be displayed close to other ends of the display unit 101, such as the bottom end 101F, the right end 101C, or the left end 101G. In this case, the state in which the first virtual screen 510 is displayed close to this end of the display unit 101 is maintained regardless of changes in the position of the terminal device 300. When the first virtual screen 510 is displayed close to the edge as in this embodiment, it becomes easier for the user to see the terminal device 300 and easier for the user to operate the terminal device 300 compared to when the first virtual screen 510 is displayed in the center of the display unit 101.

[0058] In the present embodiment, it can also be understood that the display of an image having a first positional relationship with the terminal device 300 visible through the display unit 101, among images displayed on the display unit 101, is changed. Specifically, in this embodiment, the third virtual screen 530 (see FIG. 4) can be regarded as an image having a first positional relationship with the terminal device 300, and in this embodiment, a process is performed to change the display position of this third virtual screen 530 having the first positional relationship with the terminal device 300.

[0059] In other words, in this embodiment, among the images displayed on the display unit 101, the third virtual screen 530 displayed next to the terminal device 300 can be regarded as an image having a first positional relationship with the terminal device 300. In this embodiment, a process for changing the display position of the third virtual screen 530 displayed to the side of the terminal device 300 is performed.

[0060] Furthermore, in this embodiment, it can be understood that, among the images displayed on the display unit 101, images that have a second positional relationship with the terminal device 300 visible through the display unit 101 are maintained in a state of being displayed at a predetermined location. Specifically, in this embodiment, the first virtual screen 510 located above or below the terminal device 300 can be regarded as an image having a second positional relationship with the terminal device 300. In this embodiment, the first virtual screen 510 having the second positional relationship is maintained in a state where it is displayed at a predetermined location, such as the upper side of the display unit 101, as described above.

[0061] Next, a process to be performed when the user further selects a selection item 310 while first virtual screen 510 and third virtual screen 530 are displayed on display unit 101 will be described. Specifically, for example, a process will be described in which the user further operates the terminal device 300 in the state shown in FIG. 4 and selects a new selection item 310.

[0062] In this embodiment, when the user selects a new selection item 310, a virtual screen 560 corresponding to this new selection item 310 is displayed on the upper side of the display unit 101, as shown in FIG. 9 (a diagram showing another display example on the display unit 101). In this case, the first virtual screen 510, which had been displayed on the upper side of the display unit 101 until then, is displayed to the side of the terminal device 300, as shown by reference symbol 9A in Fig. 9. In other words, the first virtual screen 510 is displayed to the side of the terminal device 300 and to the side of the third virtual screen 530. In this case, when the terminal device 300 moves, the first virtual screen 510 as well as the third virtual screen 530 moves together with the terminal device 300 .

[0063] In addition, in this embodiment, the user can select a virtual screen displayed beside the terminal device 300. Specifically, in this example, a first virtual screen 510 and a third virtual screen 530 are displayed beside the terminal device 300, and the user can select one virtual screen from the first virtual screen 510 and the third virtual screen 530. Then, when the user selects one virtual screen, the selected virtual screen is displayed on the upper side of the display unit 101. In this case, the virtual screen that was previously displayed on the upper side of the display unit 101 is displayed on the side of the terminal device 300.

[0064] Selection of one virtual screen from among the multiple virtual screens displayed to the side of the terminal device 300 is performed, for example, by the user pointing with his / her finger at the one virtual screen to be selected. When a user points a finger at the virtual screen, an image of the finger is captured by camera 104. CPU 111 analyzes the image captured by camera 104 to grasp the virtual screen located at the tip of the finger, and sets the grasped virtual screen as the virtual screen selected by the user. Then, the CPU 111 displays this virtual screen selected by the user on the upper side of the display unit 101.

[0065] The display process in the display unit 101 will be further described. In this embodiment, for example, when the user's finger PF moves outside the outer peripheral edge 101X of the display unit 101 from the display state shown in FIG. 9, the display object on the display unit 101 is determined to be the virtual screen 560 displayed on the upper side of the display unit 101. In other words, in this embodiment, when a state is reached in which it is considered that the user has stopped operating the terminal device 300, the display target of the display unit 101 is determined to be the virtual screen (content) displayed on the upper side of the display unit 101 at that time. In this case, CPU 111 switches the display content of display unit 101, and displays virtual screen 560 in an enlarged state, as shown in FIG. 10 (a diagram showing another display example on display unit 101).

[0066] Note that, here, a case has been described as an example in which the display target is finalized when it is considered that the user has stopped operating the terminal device 300. However, the present invention is not limited to this. For example, the display target of the display unit 101 may be confirmed when the user presses a confirmation button (not shown) displayed on the terminal device 300. Alternatively, for example, if the state in which the user's finger points at the virtual screen continues for a predetermined period of time, this virtual image may be determined as the display target of the display unit 101.

[0067] Additionally, CPU 111 may maintain first virtual screen 510 in a state in which first virtual screen 510 is displayed on display unit 101 when the user looks in a specific direction. Here, the first virtual screen 510 shown in FIG. 4 corresponds to the virtual screen most recently selected by the user, and this first virtual screen 510 may be maintained in a state in which this first virtual screen 510 is displayed on the display unit 101 when the user looks in a specific direction. In other words, although the above describes a case in which the first virtual screen 510 is maintained in a state in which it is displayed on the upper side of the display unit 101, the first virtual screen 510 may be maintained in a state in which the first virtual screen 510 is displayed on the display unit 101 when the user looks in a specific direction. In other words, the virtual screen showing the latest content selected by the user and displayed on the upper side of the display unit 101 may be maintained in a state in which the virtual screen is displayed when the user looks in a particular direction.

[0068] 4, a case has been described in which the first virtual screen 510 is displayed on the upper side of the display unit 101 regardless of the direction in which the user faces. In this case, for example, no matter in which direction the user wearing the wearable display device 100 faces, the state in which the first virtual screen 510 is displayed on the upper side of the display unit 101 is maintained. In contrast, if the state in which the first virtual screen 510 is displayed on the display unit 101 is maintained when the user looks in a specific direction as described above, the first virtual screen 510 will not be displayed on the display unit 101 when the user wearing the wearable display device 100 faces in a direction other than this specific direction. On the other hand, when the user wearing the wearable display device 100 looks in a specific direction, the first virtual screen 510 is displayed. In this way, when first virtual screen 510 is displayed when the user looks in a specific direction, first virtual screen 510 is essentially fixed to a specific location of the background displayed on display unit 101. In other words, when first virtual screen 510 is displayed when the user looks in a specific direction, first virtual screen 510 is displayed at a specific location behind display unit 101 that is visible through display unit 101.

[0069] In addition, the specific location where first virtual screen 510 is displayed may be a location behind display unit 101 that satisfies predetermined conditions, and first virtual screen 510 may be displayed at this location that satisfies the predetermined conditions. For example, the first virtual screen 510 may be displayed at a location in the real space RW that exists behind the display unit 101 where a flat wall is provided. More specifically, when this process is performed, the image captured by camera 104 is analyzed to identify a location that satisfies the above-mentioned predetermined condition. Then, when the location that satisfies the predetermined condition appears on display unit 101, first virtual screen 510 is displayed at this specific location.

[0070] As described above, the third virtual screen 530 moves in conjunction with the terminal device 300. Therefore, even if a user wearing the wearable display device 100 is facing a direction other than the specific direction mentioned above, if the terminal device 300 is located behind the display unit 101, the user can view the third virtual screen 530 through the display unit 101. Also, for example, when a user wearing the wearable display device 100 is facing a particular direction, if the terminal device 300 is not located behind the display unit 101, the third virtual screen 530 will not be displayed on the display unit 101.

[0071] [Other display examples on the display unit 101] FIG. 11 is a diagram showing another display example on the display unit 101. In FIG. The second virtual screen 520 will be described with reference to FIG. The CPU 111 of this embodiment further performs a process of displaying, on the display unit 101, information about the selection item 310 displayed on the terminal device 300. Specifically, when the user's finger PF approaches a selection item 310, the CPU 111 displays, on the display unit 101, information about the selection item 310 that the user's finger PF has approached, as shown in FIG.

[0072] In other words, when the user's finger PF approaches the selection item 310, the CPU 111 displays on the display unit 101 a second virtual screen 520, which is a screen for displaying information about the selection item 310, as shown in FIG. 11. In this embodiment, the image captured by the camera 104 is analyzed to determine whether or not the user's finger PF has approached a selection item 310, and if the finger PF has approached a selection item 310, the selection item 310 to which the user's finger PF has approached is determined.

[0073] In this embodiment, the CPU 111 displays the second virtual screen 520 so that a part of the terminal device 300 and a part of the second virtual screen 520 overlap with each other, as shown in FIG. More specifically, in this embodiment, for example, when the second virtual screen 520 is displayed on the transparent display device 100, the second virtual screen 520 is displayed so that the second virtual screen 520 partially overlaps with the terminal device 300 existing in real space (the terminal device 300 located behind the display unit 101). Furthermore, for example, when the second virtual screen 520 is displayed on the non-transparent display device 100, the second virtual screen 520 is displayed so that the terminal device 300 displayed on the display unit 101 partially overlaps with the second virtual screen 520. In this embodiment, the second virtual screen 520 is displayed so that the second virtual screen 520 does not overlap the selection item 310.

[0074] In this example, second virtual screen 520 displays the title of the content and information about the creator of the content. Note that the information included in second virtual screen 520 is not limited to this, and second virtual screen 520 may include other types of information. Specifically, for example, second virtual screen 520 may include information such as the time required to view the content, and the results of ratings of the content by other users.

[0075] Note that, here, a case has been described in which, when the user's finger PF approaches a selection item 310, second virtual screen 520 for displaying information about this selection item 310 is displayed. However, the present invention is not limited to this. For example, the direction of the user's line of sight may be grasped, and information about the selection item 310 located in the direction of the user's line of sight may be displayed as the second virtual screen 520. Also, for example, when the user touches a selection item 310 on the terminal display unit 205, information about the selection item 310 that the user touched may be displayed as the second virtual screen 520. [Explanation of symbols]

[0076] Reference Signs List 1: display system, 100: display device, 101: display unit, 111: CPU, 300: terminal device, 310: selection item, 510: first virtual screen, 520: second virtual screen, 530: third virtual screen, 700: control device

Claims

1. A control device including a processor and controlling a display device including a display unit, the control device controlling the display device allowing a user to view a portion behind the display unit through the display unit when the user views the display unit, The processor, changing a display of a part of the image displayed on the display unit in response to a change in a position of a terminal device located behind the display unit and viewed by a user through the display unit with respect to the display device; Regarding the other part of the image displayed on the display unit, a state in which the image is displayed at a predetermined position is maintained regardless of a change in the position of the terminal device. A control device, The processor, among images displayed in sequence on the display unit by a user operating the terminal device, an image displayed at a predetermined location on the display unit is maintained in the predetermined location regardless of a change in the position of the terminal device; Control device.

2. The processor, Among the images displayed in sequence on the display unit by the user operating the terminal device, an image displayed on the upper side or the lower side of the display unit is maintained in the upper side or the lower side regardless of a change in the position of the terminal device. The control device according to claim 1 .

3. The processor, In response to the change in the position of the terminal device, a display change is performed for an image that is displayed in a position other than the predetermined position among the images displayed in sequence on the display unit and has a first positional relationship with the terminal device that is visible through the display unit. The control device according to claim 1 .

4. The processor, In response to the change in the position of the terminal device, a display change is performed for an image that is displayed in a position other than the predetermined position among the images displayed in sequence on the display unit and is displayed next to the terminal device that is visible through the display unit. The control device according to claim 3.

5. The processor, The control device according to claim 3 , wherein a display position of the image having the first positional relationship is changed in response to a change in the position of the terminal device.

6. The processor, The control device according to claim 3 , wherein the image having the first positional relationship is moved in a direction in which the terminal device moves in response to a change in the position of the terminal device.

7. The processor, The control device according to claim 1, wherein information about a selection item displayed on the terminal device is displayed on the display unit.

8. The processor, The control device according to claim 7 , wherein when a user's finger approaches the selection item, the information about the selection item is displayed on the display unit.

9. A control device including a processor and controlling a display device including a display unit, the control device controlling the display device allowing a user to view a portion behind the display unit through the display unit when the user views the display unit, The processor, changing a display of a part of the image displayed on the display unit in response to a change in a position of a terminal device located behind the display unit and viewed by a user through the display unit with respect to the display device; Regarding the other part of the image displayed on the display unit, a state in which the image is displayed at a predetermined position is maintained regardless of a change in the position of the terminal device. A control device, The processor, a display of a top image, which is an image that is ranked higher among images displayed in order on the display unit by a user operating the terminal device, and a display of a bottom image, which is an image that is ranked lower than the top image, is changed in response to a change in the position of the terminal device; The lower-level image is maintained in a state of being displayed at a predetermined location regardless of a change in the position of the terminal device. Control device.

10. The processor, The control device according to claim 9 , wherein a display position of the higher-level image is changed in response to a change in the position of the terminal device.

11. The processor, The control device according to claim 10 , wherein the change destination of the display position of the upper image is a position that does not overlap with the terminal device whose position is changed.

12. The processor, The control device according to claim 9 , wherein the upper image displayed on the display unit is moved so as to accompany the terminal device that is visible through the display unit.

13. The processor, A control device as described in claim 9, wherein the lower-level image is maintained in a displayed state shifted toward either the upper end or the lower end of the display unit regardless of changes in the position of the terminal device.

14. The processor, The control device according to claim 9 , wherein the lower image is displayed at a specific location behind the display unit and is visible through the display unit.

15. A display device including a display unit and a control device that controls the display device, the display device is a display device that allows a user to visually recognize a portion behind the display unit through the display unit when the user visually recognizes the display unit, The control device is configured to include a control device according to any one of claims 1 to 14. Display system.

16. a change function for changing the display of a part of an image displayed on the display unit in response to a change in the position of a terminal device located behind a display unit provided on the display device and viewed by a user through the display unit, relative to the display device; and a maintenance function for maintaining the other part of the image displayed on the display unit in a predetermined position regardless of a change in the position of the terminal device; It is a program to realize the above on a computer. The maintenance function is among images displayed in sequence on the display unit by a user operating the terminal device, an image displayed at a predetermined location on the display unit is maintained in the predetermined location regardless of a change in the position of the terminal device; program.

17. a change function for changing the display of a part of an image displayed on the display unit in response to a change in the position of a terminal device located behind a display unit provided on the display device and viewed by a user through the display unit, relative to the display device; and a maintenance function for maintaining the other part of the image displayed on the display unit in a predetermined position regardless of a change in the position of the terminal device; It is a program to realize the above on a computer. the change function changes the display of a top image, which is an image that is ranked higher among images displayed in sequence on the display unit by a user operating the terminal device, and a bottom image, which is an image that is ranked lower than the top image, in response to a change in the position of the terminal device; the maintaining function maintains the state in which the lower-level image is displayed at a predetermined location regardless of a change in the position of the terminal device; program.

Images