Display system, display device, and control method and program thereof

Abstract

To provide a display system capable of obtaining high operability while maintaining visibility when displaying a virtual object.SOLUTION: A display system configured by communicably connecting a smartphone 100 and a smart glass 110 having an imaging unit 203 has detection means for detecting information of the smartphone 100 from an image photographed by the imaging unit 203. The smart glass 110 includes: a display unit 204 capable of displaying a virtual object 600; and the display control unit 304 for displaying, on the basis of information of the detected smartphone 100, the virtual object 600 on the display unit 204 by being arranged adjacently to or partly superimposed on the smartphone 100.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a display system, a display device, and a control method and program thereof. [Background technology]

[0002] There is known a technology for effectively displaying an augmented reality image on a display unit of a head-mounted display device. For example, Patent Document 1 describes a technology for displaying an augmented reality image on a display unit of a head-mounted display device as if it were added to a real object, thereby achieving a high sense of realism. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-32131 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, while the technology described in Patent Document 1 above can display a virtual object as if it were attached to a real object, it has problems such as reduced visibility and reduced operability of the virtual object due to the reduced visibility.

[0005] An object of the present invention is to provide a display system that is capable of obtaining high operability while maintaining visibility when displaying a virtual object. [Means for solving the problem]

[0006] A display system according to the present invention includes an information processing device and a head-mounted display device capable of communicating with the information processing device and having an imaging means, the display system including a detection means for detecting information about the information processing device from an image captured by the imaging means, and a display control means for displaying a virtual object on the display device at a position adjacent to the information processing device or by partially superimposing the virtual object on the information processing device based on the information, Hagu A display means for displaying a graphical user interface, the display device has a requesting means for requesting, via communication, that the graphical user interface be displayed on the display means of the information processing device; The display control means controls the display of the virtual object in accordance with a signal representing an operation on the graphical user interface. Effect of the Invention

[0007] According to the present invention, it is possible to realize a display system that is capable of obtaining high operability while maintaining visibility when displaying a virtual object. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing a schematic configuration of a display system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is an external perspective view of smart glasses. [Diagram 3] FIG. 1 is a block diagram showing a schematic configuration of smart glasses and a smartphone that constitute a display system according to a first embodiment. [Figure 4] 1 is a diagram showing an example of a state in which the smartphone in the first embodiment is in a position visible through a right display unit. FIG. [Diagram 5] 4A to 4C are schematic diagrams showing a method for detecting the tilt of the smartphone in the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of how a virtual object is displayed in the first embodiment. [Figure 7] 3A to 3C are schematic diagrams illustrating a method of displaying a virtual object in the first embodiment. [Figure 8]4 is a flowchart illustrating control of the display system in the first embodiment. [Figure 9] FIG. 11 is a block diagram showing a schematic configuration of smart glasses constituting a display system according to a second embodiment. [Figure 10] FIG. 11 is a schematic diagram illustrating the relationship between an example screen display of a smartphone in the second embodiment and a virtual object displayed on a display unit of smart glasses. [Figure 11] 10 is a flowchart illustrating control of a display system in a second embodiment. [Figure 12] FIG. 11 is a block diagram showing a schematic configuration of smart glasses constituting a display system according to a third embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of how a virtual object is displayed in the third embodiment. [Figure 14] 13 is a flowchart illustrating control of smart glasses in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0010] <Outline of display system configuration> Fig. 1 is a diagram showing a schematic configuration of a display system according to an embodiment of the present invention. The display system in Fig. 1 is configured by connecting a smartphone 100 as an example of an information processing device and smart glasses 110 as an example of a head-mounted display device via a wireless communication path 120 so as to be able to communicate with each other.

[0011] The smartphone 100 has a display unit 101 that can display various icons, photo information, etc., and has a touch panel arranged in a superimposed manner to enable touch operation. A more detailed configuration of the smartphone 100 will be described later.

[0012] Since the smartphone 100 may be a well-known one, detailed description will be omitted. In addition, the information processing device constituting the display system according to the present invention is not limited to the smartphone 100, and may be other electronic devices such as a tablet PC or a portable game machine. The present invention assumes that the information processing device is a portable one that the user holds in his / her hand and operates.

[0013] The wireless communication path 120 connects the smartphone 100 and the smart glasses 110 so that they can communicate with each other. Examples of the wireless communication path 120 include direct communication using Bluetooth (registered trademark) or Wi-Fi, and wireless LAN communication via an access point device (not shown), and are not particularly limited. The communication between the smartphone 100 and the smart glasses 110 includes a pairing request from the smart glasses 110 to the smartphone 100, and a pairing permission from the smartphone 100 to the smart glasses 110 in response to the pairing request.

[0014] Next, the smart glasses 110 will be described with reference to Fig. 2. Fig. 2 is an external perspective view of the smart glasses 110. Note that details of the smart glasses 110 are omitted in Fig. 1. The smart glasses 110 include a wireless communication unit 201, a control unit 202, an imaging unit 203, and a display unit 204. The smart glasses 110 also include an operation unit (not shown) such as a power switch.

[0015] The smart glasses 110 are configured as a glasses-type head mounted display (HMD). The smart glasses 110 are configured as an optically transmissive HMD that allows the user to view an image (display image) displayed on the display unit 204 and simultaneously view an outside scene directly. Note that the head-mounted display device constituting the display system according to the present invention is not limited to a glasses-type, and may be of other types such as a goggle-type that covers the area around the eyes or a helmet-type that covers the entire head.

[0016] The wireless communication unit 201 communicates bidirectionally with the smartphone 100 via the wireless communication path 120. The wireless communication unit 201 is shown as an antenna symbol in FIG. 2, but may be built into the frame of the smart glasses 110. The imaging unit 203 is a camera having a lens and an image sensor, but is not limited to this. For example, the imaging unit may include a LIDAR (Laser Imaging Detection and Ranging). In other words, the imaging unit may be of any type as long as it can acquire an image (distance image) of an object. The control unit 202 controls the overall operation of the smart glasses 110, and details thereof will be described later with reference to FIG. 3.

[0017] As described above, the smart glasses 110 are configured as an optically transmissive HMD. That is, the display unit 204, which corresponds to the lens of general glasses, is made of a material with high light transmittance (high light transmittance), and is a transmissive display that does not block the user's field of vision when no image is displayed. The smart glasses 110 are configured to display a virtual object 600 (see FIG. 6 and FIG. 7, etc.) on the display unit 204, which is a transmissive display, and to allow the user to view a real object through the display unit 204. The display unit 204 is configured of a right display unit 204R arranged in front of the user's right eye and a left display unit 204L arranged in front of the user's left eye.

[0018] First Embodiment Fig. 3(a) is a block diagram showing a schematic configuration of smart glasses 110 according to the first embodiment. The smart glasses 110 include a wireless communication unit 201, a control unit 202, an imaging unit 203, and a display unit 204. The wireless communication unit 201, the imaging unit 203, and the display unit 204 have been described with reference to Fig. 2, and therefore will not be described here.

[0019] The control unit 202 includes a so-called microcomputer including a CPU, a ROM, a RAM, a flash memory, and the like. The CPU loads a predetermined program stored in the ROM into the RAM and executes it, so that the control unit 202 functions as a detection unit and a display control unit 304. The detection unit includes a distance detection unit 301, a position detection unit 302, and an inclination detection unit 303, and detects information about the smartphone 100 from an image captured by the imaging unit 203. The CPU loads an OS program stored in the ROM into the RAM, so as to control various operations that can be executed by the smart glasses 110.

[0020] The distance detection unit 301 detects distance information from the smart glasses 110 (display unit 204) to the smartphone 100 from an image captured by the imaging unit 203 as one piece of information of the smartphone 100. Here, in the description of the first embodiment, it is assumed that the wireless communication unit 311 (see FIG. 3(b)) of the smartphone 100 to be imaged by the imaging unit 203 and the wireless communication unit 201 of the smart glasses 110 have a wireless connection established (paired) for performing wireless communication via the wireless communication path 120. The distance detection unit 301 detects the distance L between the smart glasses 110 (display unit 204) and the smartphone 100 as distance information, for example, 'L=50 cm', by calculation (image processing) from the image of the smartphone 100 captured by the imaging unit 203.

[0021] The position detection unit 302 detects position information of the smartphone 100 by calculation (image processing) from an image of the smartphone 100 captured by the imaging unit 203, as one piece of information of the smartphone 100. The position information of the smartphone 100 is information indicating the position (direction) of the smartphone 100 in the captured image relative to the display unit 204. A method of detecting the position information will be described later with reference to FIG. 4.

[0022] The tilt detection unit 303 detects the tilt of the smartphone 100 captured by the imaging unit 203 through calculation (image processing) from an image of the smartphone 100 captured by the imaging unit 203 as one piece of information about the smartphone 100. Specifically, the tilt detection unit 303 detects the tilt in the vertical rotation direction, the tilt in the horizontal rotation direction, and the tilt in the rotation axis direction. A method of detecting the tilt information of the smartphone 100 will be described later with reference to FIG. 5.

[0023] The display control unit 304 performs control for displaying a virtual object on the display unit 204 based on the information on the distance, position, and tilt detected by the distance detection unit 301 , position detection unit 302 , and tilt detection unit 303 .

[0024] 3B is a block diagram of the smartphone 100. The smartphone 100 includes a wireless communication unit 311, a control unit 312, and a display unit 101. The wireless communication unit 311, for example, grants permission to a communication establishment request, such as a pairing request, from the smart glasses 110 via the wireless communication path 120. . system The control unit 312 is composed of a CPU, a ROM, a RAM, an EEPROM, etc., and executes a predetermined program stored in the ROM or EEPROM to control the overall operation of the smartphone 100. The display unit 101 is composed of a liquid crystal display or an organic EL display that displays various information, and a touch panel as an operation means superimposed on the liquid crystal display or the organic EL display.

[0025] Next, a method for detecting the smartphone 100 by the smart glasses 110 will be described. First, a method for detecting the position of the smartphone 100 by the position detection unit 302 of the smart glasses 110 will be described.

[0026] FIG. 4 shows an example of a state in which the smartphone 100 is viewed as a real object through the right display unit 204R. As described above, the position information of the smartphone 100 is obtained using an image captured by the imaging unit 203 when the smartphone 100 is within the shooting angle of view of the imaging unit 203. The coordinates 402 of the smartphone 100 are defined as the position of the upper right corner of the smartphone 100 with respect to the origin 401 of the right display unit 204R. The right display unit 204R has a horizontal length of 'A' and a vertical length of 'B'. For example, when the horizontal length A of the right display unit 204R is 3 cm and the vertical length B is 2 cm, the position detection unit 302 detects the coordinates 402 of the smartphone 100 as [vertical, horizontal] = [1.5 cm, 0.5 cm].

[0027] Next, a description will be given of a method for detecting the tilt of the smartphone 100 by the tilt detection unit 303. The tilt detection unit 303 detects the tilt in the vertical rotation direction, the tilt in the horizontal rotation direction, and the tilt in the direction of the rotation axis.

[0028] FIG. 5(a) is a schematic diagram showing a method for detecting the tilt of the smartphone 100 in the vertical rotation direction. Note that FIG. 5(a) and FIGS. 5(b) and (c) shown later show a simplified view of the smartphone 100. The vertical rotation direction of the smartphone 100 refers to the rotation direction around an axis perpendicular to the vertical direction. The tilt of the smartphone 100 in the vertical rotation direction is calculated by the ratio of the length 'M' of the lower side of the smartphone 100 captured by the imaging unit 203 to the length 'N' of the upper side. For example, when M=5 and N=4 are detected, the tilt of the smartphone 100 in the vertical rotation direction is calculated as M:N=5:4. Note that the smartphone 100 has a substantially rectangular shape when viewed from a direction perpendicular to the display unit 101.

[0029] 5(b) is a schematic diagram showing a method for detecting the inclination of the horizontal rotation direction of the smartphone 100. The horizontal rotation direction of the smartphone 100 refers to the rotation direction around an axis perpendicular to the horizontal plane. The inclination of the horizontal rotation direction of the smartphone 100 is calculated from the ratio of the length 'K' of the left side of the smartphone to the length 'L' of the right side of the smartphone. For example, when it is detected that K=7 and L=6, the inclination of the horizontal rotation direction is calculated as K:L=7:6.

[0030] 5(c) is a schematic diagram showing a method for detecting the inclination of the smartphone 100 in the direction of the rotation axis. The inclination of the smartphone 100 in the direction of the rotation axis is determined by the angle 'α' at which the top side of the smartphone 100 is inclined relative to the horizontal plane. For example, if the inclination of the top side of the smartphone 100 relative to the horizontal plane is detected to be 30 degrees, α is determined to be 30 degrees.

[0031] In the first embodiment, the aspect of the virtual object displayed on the right display unit 204R is controlled based on information on the distance, position, and tilt of the smartphone 100 relative to the smart glasses 110 detected from an image captured by the imaging unit 203 of the smartphone 100.

[0032] 6(a) and 6(b) are diagrams showing a display example of a virtual object 600 on the right display unit 204R of the smart glasses 110 for the smartphone 100 viewed through the right display unit 204R. The virtual object 600 is displayed on the right display unit 204R so as to be in contact with or close to the right side based on the coordinates 402 of the smartphone 100 viewed directly through the right display unit 204R made of a highly translucent material.

[0033] The distance detection unit 301, the position detection unit 302, and the tilt detection unit 303 continuously acquire and detect information on the distance, position, and tilt of the smartphone 100 relative to the smart glasses 110 from images captured by the imaging unit 203. Then, as shown in Figs. 6(a) and (b), the display mode of the virtual object 600 displayed on the right display unit 204R is controlled in accordance with changes in the distance, position, and attitude of the smartphone 100 relative to the smart glasses 110. Details of the control will be described below with reference to Figs. 7 and 8.

[0034] 7 is a schematic diagram illustrating a method for displaying a virtual object 600 on the right display unit 204R of the smart glasses 110 by the display control unit 304. The display control unit 304 controls the display mode of the virtual object 600 to be displayed on the right display unit 204R based on information on the distance, position, and tilt of the smartphone 100 relative to the smart glasses 110.

[0035] For example, when the distance information of the smartphone 100 is 50 cm, the display control unit 304 controls to place the virtual object 600 so that it appears at a position of 50 cm from the viewpoint of the smart glasses 110. Then, the display control unit 304 performs affine transformation on the virtual object 600 so that the tilt of the vertical rotation direction and the tilt of the horizontal rotation direction of the virtual object 600 become the same as the tilt of the vertical rotation direction and the tilt of the horizontal rotation direction of the smartphone 100, respectively. For example, if the tilt of the vertical rotation direction of the smartphone 100 is 'M:N=5:4', the display control unit 304 performs affine transformation on the virtual object 600 so that the ratio of the lower side to the upper side becomes 'lower side:upper side=5:4'. Also, if the tilt of the horizontal rotation direction of the smartphone 100 is 'K:L=7:6', the display control unit 304 performs affine transformation on the virtual object 600 so that the ratio of the left side to the right side becomes 'left side:right side=7:6'. Furthermore, the display control unit 304 performs a rotation process on the virtual object 600 after the affine transformation so that the inclination of the rotation axis direction becomes the same as the inclination of the rotation axis direction of the smartphone 100. For example, if the angle of the rotation axis direction of the smartphone 100 is 30 degrees, the display control unit 304 performs a vector rotation on the virtual object 600 after the affine transformation by 30 degrees.

[0036] The display control unit 304 performs control so that the virtual object 600 is placed on the right display unit 204R by making the coordinates of the upper left corner of the virtual object 600 coincident with or close to the coordinates 402 of the smartphone 100. At this time, the virtual object 600 is displayed on the right display unit 204R so that its upper and lower sides are approximately aligned with the upper and lower sides of the smartphone 100, respectively. In other words, the length of the left side of the virtual object 600 is set to be approximately equal to the length of the right side of the smartphone 100. In this way, the display state shown in FIG. 7 is obtained.

[0037] Fig. 8 is a flowchart of processing executed by the smartphone 100 and the smart glasses 110 when controlling the display mode of the virtual object 600 by operating the smartphone 100. In the flowchart of Fig. 8, each process (step) indicated by an S number for the smart glasses 110 is realized by the CPU of the smart glasses 110 expanding a predetermined program stored in the ROM into the RAM and controlling the operation of each part of the smart glasses 110.

[0038] The power of the smart glasses 110 is turned on by a user's operation on an operation unit provided on the smart glasses 110, which enables the display of a virtual object on the display unit 204 and starts this process. Note that when the process in the smart glasses 110 starts, the power of the smartphone 100 is assumed to be on.

[0039] In S801, the CPU requests pairing to the smartphone 100 in response to a user operation. The user operation here may be an operation on an operation unit, or a gesture of the user captured by the imaging unit 203. In S811, the smartphone 100 permits pairing in response to the pairing request in S801. Although a block diagram of the smartphone 100 is not shown, the processing in the smartphone 100 is realized by the CPU of the smartphone 100 executing a predetermined program and controlling the operation of each unit of the smartphone 100. Note that the smartphone 100 and the smart glasses 110 are paired because, for example, when there are multiple smartphones in an image captured by the imaging unit 203, the pairing can be used as a means for identifying the paired smartphone.

[0040] In S802, the distance detection unit 301 detects distance information of the smartphone 100 from an image of the smartphone 100 captured by the imaging unit 203. In the first embodiment, since the purpose is to operate the virtual object 600 by the smartphone 100, it is assumed that the user directly views the smartphone 100 through the display unit 204. It is also assumed that the smartphone 100 is reflected in the image captured by the imaging unit 203. Furthermore, the captured image may be analyzed (image processed) by the control unit 202 before S802, and if it is detected as a result that the smartphone 100 (display unit 101) is not reflected in the captured image, the processing of this flowchart may be terminated.

[0041] In S803, the position detection unit 302 detects position information of the smartphone 100, that is, the coordinates 402 of the upper right corner of the smartphone 100, from an image of the smartphone 100 captured by the imaging unit 203. In S804, the tilt detection unit 303 detects the tilt in the vertical rotation direction, the tilt in the horizontal rotation direction, and the tilt in the rotation axis direction as tilt information of the smartphone 100, from the image of the smartphone 100 captured by the imaging unit 203. In S805, the display control unit 304 displays a virtual object 600 on the display unit 204 (right display unit 204R) based on the distance, position, and tilt information of the smartphone 100 calculated in S802 to S804.

[0042] The display of the virtual object 600 continues until, for example, the user operates the smart glasses 110 to turn off the power. During that time, the process of obtaining information about the position and orientation of the smartphone 100 and displaying the virtual object 600 (S802 to S805) is repeated. That is, the display control unit 304 continues to display the virtual object adjacent to the smartphone 100 while deforming the virtual object in accordance with changes in the position and orientation of the smartphone 100.

[0043] As described above, in the first embodiment, the control unit 202 continues to detect information on the distance, position, and tilt of the smartphone 100, and based on this information, continues to display a virtual object on the display unit 204. This allows the user to view the virtual object by changing the display mode of the virtual object by changing the attitude or position of the smartphone 100 while the smartphone 100 is directly visible to the user.

[0044] In the first embodiment, the smart glasses 110 detect the distance, position, and tilt information of the smartphone 100, but the present invention is not limited to such a configuration. For example, if the smartphone 100 has various tilt information using a gyro sensor or the like, the smartphone 100 may provide the tilt information to the smart glasses 110. In addition, regarding the distance information and position information of the smartphone 100, if the smartphone 100 has a distance measuring function such as a lidar, the smartphone 100 may provide the distance information and position information to the smart glasses 110.

[0045] In the first embodiment, the virtual object is displayed adjacent to the right side of the smartphone 100 which is directly visible through the display unit 204, but the display position of the virtual object is not limited to this. For example, the virtual object may be displayed adjacent to the left side of the smartphone 100. Also, a part or the entirety of the virtual object 600 may be displayed so as to overlap the display unit 101 of the smartphone 100.

[0046] <Second embodiment> The display system according to the second embodiment of the present invention has the same basic configuration as the display system according to the first embodiment, but differs in the configuration of the control unit 202 included in the smart glasses 110. Therefore, hereinafter, the configuration and control of the display system according to the second embodiment will be described, focusing on the configuration and functions of the control unit 202. Note that, among the components of the display system according to the second embodiment, those that are substantially the same as the components of the display system according to the first embodiment will be given the same names and the same reference numerals, and descriptions of matters that have already been described will be omitted.

[0047] 9 is a block diagram showing a schematic configuration of smart glasses 110 constituting a display system according to the second embodiment. The smart glasses 110 include a wireless communication unit 201, a control unit 202, an imaging unit 203, and a display unit 204. The control unit 202 includes functions as a distance detection unit 301, a position detection unit 302, a tilt detection unit 303, an icon display unit 901, and a display control unit 902. The functions of the distance detection unit 301, the position detection unit 302, and the tilt detection unit 303 are equivalent to those in the first embodiment, and therefore will not be described here.

[0048] The icon display unit 901 performs control for arranging icons used for operating virtual objects displayed on the display unit 204 on the smartphone 100. That is, the icon display unit 901 requests the smartphone 100 to display icons for operating virtual objects on the display unit 101. Upon receiving this request, the smartphone 100 returns a notification to the icon display unit 901 permitting the icon display request, and displays icons for operating virtual objects on the screen of the smartphone 100.

[0049] An application (software) for displaying icons for operating virtual objects on the display unit 101 is stored in advance in the smartphone 100, or is acquired from an external device via the wireless communication path 120. For example, the user can move a cursor within the virtual object by performing a flick operation on the icon, and can partially enlarge the virtual object 600 by performing a double tap operation.

[0050] An operation on an icon arranged on the display unit 101 of the smartphone 100 is converted into a signal and transmitted to a display control unit 902 of the smart glasses 110. The display control unit 902 receives a signal representing a user operation on an icon displayed on the display unit 101 of the smartphone 100, and controls the movement of a cursor displayed within a virtual object, the enlargement / reduction of the virtual object, and the like, according to the received signal.

[0051] 10(a) and (b) are schematic diagrams illustrating the relationship between a screen display example on the smartphone 100 and a virtual object 600 displayed on the display unit 204 of the smart glasses 110. Note that the display on the display unit 204 is omitted in FIG.

[0052] 10(a), an icon 1002 for operating a virtual object 600 is arranged (displayed) at the bottom left of a display unit 101 of a smartphone 100. An arrow 1003 shown within the icon 1002 diagrammatically indicates the operating direction of a user's finger (the arrow 1003 is not actually displayed), and here, a state in which a flick operation is being performed on the icon 1002 from the left side to the right side is shown.

[0053] 10(a), based on a flick operation from left to right on an icon 1002, the display control unit 902 moves a cursor 1004 on the virtual object 600 from left to right as indicated by an arrow 1005. As a result, although not shown, the cursor 1004 moves onto a person 1006 on the right side.

[0054] In Fig. 10(b), a circle 1007 shown within the icon 1002 typically indicates a double tap operation of the user's finger. After a flick operation is performed on the icon 1002 in Fig. 10(a), the user performs a double tap operation on the icon 1002 as shown in Fig. 10(b). As a result, the cursor 1004 is moved onto the person 1006 by the flick operation in Fig. 10(a), and therefore the area on the person 1006 where the cursor 1004 is located is enlarged and displayed. Thus, in Fig. 10(b), only the upper body part 1008 of the person 1006 is displayed as the virtual object 600.

[0055] Fig. 11 is a flowchart of a process executed by the smartphone 100 and the smart glasses 110 to enable the operation of the virtual object 600 by an operation on the icon 1002. In the flowchart of Fig. 10, each process (step) indicated by an S number for the smart glasses 110 is realized by the CPU of the smart glasses 110 expanding a predetermined program stored in the ROM into the RAM and controlling the operation of each part of the smart glasses 110. Also, each process indicated by an S number for the smartphone 100 is realized by the CPU of the smartphone 100 executing a predetermined program and controlling the operation of each part of the smartphone 100.

[0056] The power of the smart glasses 110 is turned on by a user's operation on an operation unit provided on the smart glasses 110, which enables display of a virtual object on the display unit 204, and this process is started. It is assumed that the power of the smartphone 100 is on when the process in the smart glasses 110 starts. Also, the processes of S1101 to S1105 and S1111 are the same as the processes of S801 to S805 and S811 in the flowchart of FIG. 8, and therefore description thereof will be omitted.

[0057] In S1106 following S1105, the control unit 202 (CPU) determines whether or not an icon display request has been received through a user operation. The user operation here may be an operation on the operation unit, or may be a user gesture captured by the imaging unit 203. When the control unit 202 determines that an icon display request has not been received (NO in S1106), the control unit 202 continues to display the virtual object 600, for example, until the smart glasses 110 are powered off through a user operation, in the same manner as in the first embodiment. During this time, the process of obtaining information about the position and orientation of the smartphone 100 and displaying the virtual object 600 (S1102 to S1105) is repeated.

[0058] When the control unit 202 determines that the icon display request has been received (YES in S1106), the process proceeds to S1107. In S1107, the icon display unit 901 makes an icon display request to the smartphone 100. In S1112, the CPU of the smartphone 100 permits icon display in response to the icon display request in S1107, and returns permission to display the icon to the icon display unit. Then, in S1113, the CPU of the smartphone 100 displays an icon 1002 on the display unit 101. This allows the user to operate the icon 1002.

[0059] In the smart glasses 110, in response to the icon display permission to the icon display request in S1107, the display control unit 902 displays the virtual object 600 on the display unit 204 in S1108. At that time, as shown in Fig. 10, the cursor 1004 is displayed superimposed on the virtual object 600. The processing after S1108 is the same as the processing when the determination in S1106 is 'NO', and therefore description thereof will be omitted.

[0060] In this way, in the second embodiment, an icon for operating a virtual object displayed on the display unit 204 is displayed on the display unit 101 of the smartphone 100, and the virtual object can be operated by a user's operation on the icon. This makes it possible to intuitively operate a virtual object on the smartphone 100 that is viewed directly through the display unit 204, without touching the display unit 204 on which the virtual object is displayed. In other words, it becomes possible to easily operate a virtual object while ensuring visibility of the smartphone 100 and the virtual object.

[0061] The display position of the icon 1002 is not limited to the bottom left of the display unit 101, and the size of the icon 1002 is not limited as long as the user can easily operate it. The icon displayed on the display unit 101 is not limited to one, and multiple icons (for example, icons dedicated to performing a predetermined operation) may be displayed. Operations that can be performed on the icon 1002 are not limited to flicks and double taps, and these are not necessarily required, and may be single taps, swipes, pinches in, pinches out, etc.

[0062] <Third embodiment> The display system according to the third embodiment of the present invention has the same basic configuration as the display system according to the first embodiment, but differs in the configuration of the control unit 202 included in the smart glasses 110. Therefore, the configuration and control of the display system according to the third embodiment will be described below, focusing on the configuration and functions of the control unit 202. Note that, among the components of the display system according to the third embodiment, those that are substantially the same as the components of the display system according to the first embodiment will be given the same names and the same reference numerals, and descriptions of matters that have already been described will be omitted.

[0063] 12 is a block diagram showing a schematic configuration of smart glasses 110 constituting a display system according to the third embodiment. The smart glasses 110 include a control unit 202, an imaging unit 203, and a display unit 204. The control unit 202 includes functions as an object recognition unit 1201, a distance detection unit 301, a position detection unit 302, a tilt detection unit 303, and a display control unit 1202. The functions of the distance detection unit 301, the position detection unit 302, and the tilt detection unit 303 are equivalent to those in the first embodiment, and therefore description thereof will be omitted here.

[0064] In the third embodiment, the power supply of the smartphone 100 is turned off, and pairing with the smart glasses 110 is not performed. On the other hand, the object recognition unit 1201 recognizes the smartphone 100 by detecting a specific operation such as a double tap on the smartphone 100 by the user from an image acquired through the imaging unit 203.

[0065] Here, it is expected that a situation may occur in which the tilt in the vertical rotation direction, the tilt in the horizontal rotation direction, and the tilt in the rotation axis direction of the smartphone 100 detected by the tilt detection unit 303 are each greater than a preset tilt threshold. In that case, the display control unit 1202 invalidates the tilt information detected by the tilt detection unit 303 without touching the smartphone 100, and normally displays the virtual object on the display unit 204.

[0066] Displaying a virtual object normally refers to displaying the virtual object on the display unit 204 in a positional relationship in which a line connecting the right eye and the left eye of a user wearing the smart glasses 110 is approximately parallel to the upper side or the lower side of the virtual object. For example, when the angle α of the rotation axis direction of the smartphone 100 detected by the tilt detection unit 303 is greater than a preset tilt threshold, the virtual object is displayed on the display unit 204 without vector-rotating the virtual object in the rotation axis direction. Note that the preset tilt threshold may be a value set as a default value in the smart glasses 110, or may be a value set by the user.

[0067] FIG. 13 is a diagram showing an example of display of the virtual object 600 by the display control unit 1202, and shows a state in which the inclination of the rotation axis direction of the smartphone 100 is 60 degrees. For example, when the inclination threshold is 30 degrees, the inclination of the rotation axis direction of the smartphone 100 is tilted from 20 degrees to 60 degrees. In this case, until the inclination of the rotation axis direction becomes 20 degrees to 30 degrees, the virtual object 600 is rotated in the rotation axis direction according to the inclination of the rotation axis direction. Then, when the inclination of the rotation axis direction exceeds 30 degrees, the virtual object 600 is not further rotated in the vector direction, and the display in the state in which the inclination of the rotation axis direction is 30 degrees is maintained. Therefore, while the inclination of the rotation axis direction exceeds 30 degrees and reaches 60 degrees, the display of the virtual object 600 does not change.

[0068] This makes it possible to prevent the virtual object 600 from being displayed on the display unit 24 in a state in which it is difficult for the user to recognize the contents of the virtual object 600. Note that similar display control is performed for the tilt in the vertical rotation direction and the tilt in the horizontal rotation direction by setting threshold values ​​(threshold value for the ratio of the upper side to the lower side, and threshold value for the ratio of the left side to the right side).

[0069] Fig. 14 is a flowchart explaining display control for regulating the inclination of the virtual object 600. In the flowchart in Fig. 14, each process (step) indicated by an S number is realized by the CPU of the smart glasses 110 expanding a predetermined program stored in the ROM into the RAM and controlling the operation of each part of the smart glasses 110.

[0070] The smart glasses 110 are powered on by a user's operation on an operation unit provided on the smart glasses 110, which enables a virtual object to be displayed on the display unit 204, and this process is started. In S1401, the object recognition unit 1201 recognizes the smartphone 100 as an object. The processes of S1102 to S1104 are the same as the processes of S802 to S804 in the flowchart of FIG. 8, and therefore will not be described.

[0071] In step 1405, the display control unit 1202 determines whether the tilts in the vertical rotation direction, horizontal rotation direction, and rotation axis direction detected for the smartphone 100 recognized in step S1401 are greater than the tilt threshold values ​​set for each tilt.

[0072] When the display control unit 1202 determines that each detected tilt is equal to or smaller than the corresponding tilt threshold (NO in S1405), the process proceeds to S1406. The process of S1406 and thereafter is the same as the process of S805 and thereafter in the flowchart of FIG. 8, and description thereof will be omitted here. On the other hand, when the display control unit 1202 determines that each detected tilt is greater than the corresponding tilt threshold (YES in S1405), the process proceeds to S1407. In S1407, the display control unit 1202 cancels (invalidates) information on the tilt determined to exceed the tilt threshold, and displays the virtual object 600 on the display unit 204. The process after S1407 is the same as the process after S1406, and description thereof will be omitted.

[0073] In this way, in the third embodiment, an unpaired smartphone 100 is recognized, and if the tilt detected for the recognized smartphone 100 is greater than the tilt threshold, the detected tilt is determined to be invalid. This makes it possible to control the display mode of a virtual object by operating the smartphone 100 visually recognized through the display unit 204. At that time, the virtual object can be displayed on the display unit 204 in a state in which the user can always easily recognize the content.

[0074] Note that the restriction on the transformation of a virtual object in the third embodiment can also be applied to the display control of a virtual object in a state in which the smartphone 100 and the smart glasses 110 are paired as in the configuration of the first embodiment.

[0075] Although the present invention has been described in detail above based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0076] For example, in the above embodiment, smart glasses 110, which are an example of an optically transmissive (optically see-through) HMD, are used as the display device, but instead, a video see-through HMD may be used. In a video see-through HMD, the display unit, which corresponds to the lens of general glasses, is not transparent. Therefore, when a video see-through HMD is used, a display image in which an image of a virtual object is superimposed on a real image captured by an imaging unit is displayed on the display unit so that the user can see it.

[0077] In addition, in the above embodiment, the detection of the smartphone 100 from the captured image captured by the imaging unit 203 is performed by the control unit 202 of the smart glasses 110, but the detection may be performed by the control unit 312 of the smartphone 100. In other words, the smartphone 100 may receive (image data of) the captured image captured by the imaging unit 203 at the wireless communication unit 311, the control unit 312 may detect the image, and the detection result may be transmitted from the wireless communication unit 311 to the smart glasses 110. This method is useful when the processing capacity of the control unit 312 of the smartphone 100 is superior to the processing capacity of the control unit 202 of the smart glasses 110.

[0078] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0079] 100 Smartphones 110 Smart Glasses 120 Wireless communication paths 202 Control Unit 203 Imaging unit 204 Display section 301 Distance detection unit 302 Position detection unit 303 Tilt detection unit 304,902,1202 Display control unit 901 Icon display section 1201 Object recognition unit

Claims

1. A display system including an information processing device and a head-mounted display device capable of communicating with the information processing device and having an imaging means, a detection means for detecting information of the information processing device from an image captured by the imaging means; a display control means for displaying a virtual object on the display device at a position adjacent to the information processing device or by partially superimposing the virtual object on the information processing device based on the information, the information processing device has a display means for displaying a graphical user interface; the display device has a requesting means for requesting, via communication, that the graphical user interface be displayed on the display means of the information processing device; The display system according to claim 1, wherein the display control means controls display of the virtual object in accordance with a signal representing an operation on the graphical user interface.

2. 2. A display system according to claim 1, wherein said display device comprises said detection means.

3. The detection means includes: a distance detection unit for detecting distance information from the display device to the information processing device; a position detection means for detecting position information of the information processing device relative to the display device; 3. The display system according to claim 1, further comprising: a tilt detection unit for detecting tilt information of the information processing device.

4. The distance detection means detects the distance information from an image captured by the imaging means, and the position detection means detects the position information from the image.

4. The display system according to claim 3, wherein said tilt detection means detects said tilt information from said image.

5. the distance detection means acquires the distance information detected by the information processing device from the information processing device through communication; The position detection means acquires the position information detected by the information processing device from the information processing device through communication; 4. The display system according to claim 3, wherein the tilt detection unit acquires the tilt information detected by the information processing device from the information processing device through communication.

6. The display system according to any one of claims 3 to 5, wherein the display control means, when the tilt of the information processing device is greater than a predetermined threshold, cancels the tilt information and displays the virtual object on the display device.

7. The display system according to any one of claims 1 to 6, characterized in that the display control means continues to display the virtual object on the display device at a position adjacent to the information processing device or by partially superimposing the virtual object on the information processing device in accordance with changes in the information.

8. the information processing device has a substantially rectangular shape when viewed from a predetermined direction; The display system according to any one of claims 1 to 7, characterized in that the display control means deforms the virtual object and displays it on the display device based on the information so that a ratio of an upper side to a lower side and a ratio of a left side to a right side of the virtual object are equivalent to a ratio of an upper side to a lower side and a ratio of a left side to a right side of the information processing device, respectively.

9. A display system having an information processing device and a head-mounted display device capable of communicating with the information processing device and having an imaging means, a detection means for detecting information of the information processing device from an image captured by the imaging means; a display control means for displaying a virtual object on the display device at a position adjacent to the information processing device or by partially superimposing the virtual object on the information processing device based on the information, the information processing device having a display means for displaying a graphical user interface, the display device has an object recognition means for, when the display device and the information processing device cannot be communicatively connected and a specific gesture has been made on the information processing device is detected from an image captured by the imaging means, identifying the information processing device on which the gesture has been made as an information processing device that displays the virtual object adjacent to the display device or displays the virtual object partially superimposed on the display device; The display system according to claim 1, wherein the display control means controls display of the virtual object in accordance with a signal representing an operation on the graphical user interface displayed on a display means of the information processing device.

10. A head-mounted display device capable of communicating with an information processing device and having an imaging means and a display means, a display control means for displaying a virtual object on a display means of the information processing device, the virtual object being adjacent to the information processing device or being partially superimposed on the information processing device, based on information about the information processing device detected from an image captured by the imaging means; a requesting means for requesting, via communication, that a graphical user interface be displayed on a display means of the information processing device; The display control means receives a signal representing an operation on a graphical user interface displayed on a display means of the information processing device from the information processing device, and controls the display of the virtual object in accordance with the signal.

11. A detection means for detecting the information, The detection means includes: a distance detection unit for detecting distance information from the display device to the information processing device; a position detection means for detecting position information of the information processing device relative to the display device; 11. The head-mounted display device according to claim 10, further comprising a tilt detection unit for detecting tilt information of the information processing device.

12. A method for controlling a display system having an information processing device having a display means, and a head-mounted display device capable of communicating with the information processing device and having an imaging means, comprising: detecting information of the information processing device from an image captured by the imaging means; displaying a virtual object on the display device adjacent to the information processing device or partially superimposed on the information processing device based on the information; a step of the display device requesting the information processing device through communication to display a graphical user interface on a display means of the information processing device; a step of displaying the graphical user interface on a display means of the information processing device; and controlling display of the virtual object in accordance with a signal representing an operation on the graphical user interface.

13. A method for controlling a display system as described in Claim 12, characterized in that the information processing device has a step of notifying the display device that the graphical user interface will be displayed on the display means of the information processing device in response to the request.

14. A method for controlling a head-mounted display device capable of communicating with an information processing device and having an imaging means and a display means, comprising: displaying a virtual object on a display means of the display device at a position adjacent to the information processing device or by partially superimposing the virtual object on the information processing device based on information of the information processing device detected from an image captured by the imaging means; requesting, via communication, that a graphical user interface be displayed on a display means of the information processing device; receiving, from the information processing device, a signal representing an operation on the graphical user interface displayed on a display means of the information processing device, and controlling display of the virtual object in accordance with the signal.

15. A program for causing a computer to execute each step of the method for controlling a head-mounted display device according to claim 14.

Images