Head-mounted display, data display method for head-mounted display, and head-mounted display system

Abstract

The present invention solves the problem that, in a head-mounted display capable of interacting with an information apparatus such as a PC, a user interface in a position-free mode cannot necessarily be said to be appropriate.　Provided is a head-mounted display comprising a transmissive screen that is capable of interacting with an information processing apparatus, the head-mounted display comprising a storage unit that is capable of storing data stored in the memory of an information processing apparatus within a visually recognizable angle of view, a recognition unit that recognizes the information processing apparatus within the visually recognizable viewing of view, a communication unit that receives the data from the information processing apparatus recognized by the recognition unit, and a display unit that displays the received data. A user interface is improved by the aforementioned head-mounted display.

Description

Head-mounted display, data display method for head-mounted display, and head-mounted display system

[0001] The present invention relates to data display in a head-mounted display.

[0002] A head-mounted display (hereinafter referred to as HMD) displays virtual images and allows a user wearing the HMD to experience a virtual reality space. The virtual space can be experienced by obtaining information from the cloud and using the HMD in a location-free manner, or by connecting the HMD to an information device such as a personal computer (hereinafter referred to as PC).

[0003] Prior art document Patent Document 1 discloses a head-mounted display for displaying computer-generated information in conjunction with a physical display of a computer system, the head-mounted display comprising generation logic, a see-through display, and a presentation unit, the generation logic being used to generate at least one virtual display, the presentation unit being used to present the at least one virtual display on the see-through display, the see-through display being configured to enable a user of the head-mounted display to view the at least one virtual display and a physical display in a real-world scene, the at least one virtual display being presented on the see-through display as an additional object in the real-world scene, and the at least one virtual display being used in conjunction with the physical display to present computer-generated information within an extended desktop from the user's visual viewpoint.

[0004] Special table 2017-510106 publication

[0005] In Patent Document 1, an HMD can be used as an extended display for a PC, but when the HMD is used in a position-free mode, its use is limited. That is, Patent Document 1 describes that when the HMD is not capturing the display of the PC main body, the display frame of the extended display (a virtual display in Patent Document 1) is maintained and information stored in the HMD is displayed in the display frame, but no consideration is given to the reduction in operability of the HMD screen in position-free mode.

[0006] In an HMD that can be linked to an information device such as a PC, it is necessary to realize a function that allows the HMD to be used in a position-free mode that does not require the HMD to be used in a specific location.

[0007] The present invention has been made in consideration of the above points, and its purpose is to achieve suitable operation in an HMD that can be linked to an information device by switching the operation interface when the HMD is switched to position-free mode.

[0008] The above problem is solved by a head-mounted display equipped with a transparent screen that can be linked with an information processing device of the present invention, the head-mounted display comprising: a memory unit capable of storing data stored in the memory of an information processing device within the visual field of view; a recognition unit that recognizes an information processing device within the visual field of view; a communication unit that receives the data from the information processing device recognized by the recognition unit; and a display unit that displays the received data.

[0009] According to the present invention, a suitable user interface can be provided in an HMD that can be linked to an information device.

[0010] FIG. 1 is an external view of an HMD according to an embodiment of the present invention; FIG. 2 is a functional block diagram of an HMD according to an embodiment of the present invention; FIG. 3 is an explanatory diagram of a case where an HMD according to an embodiment of the present invention is used as an extended display of a PC; FIG. 4 is an explanatory diagram of a case where an HMD according to an embodiment of the present invention is used as an extended display of a PC; FIG. 5 is an example of a screen for selecting and copying information according to an embodiment of the present invention; FIG. 6 is an example of a screen for reading out copied information according to an embodiment of the present invention; FIG. 7 is an explanatory diagram of another example of a method for copying information according to an embodiment of the present invention; FIG. 8 is an explanatory diagram of a case where an HMD according to an embodiment of the present invention is used in position-free mode; FIG. 9 is an example of a screen for explaining a change in user interface accompanying a mode change according to an embodiment of the present invention; FIG. 10 is an example of a screen for explaining a change in user interface accompanying a mode change according to an embodiment of the present invention;

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. First Embodiment of the Present Invention A first embodiment of an HMD that can be linked to an information device will be described with reference to FIGS.

[0012] Fig. 1 is an external view of an HMD according to an embodiment of the present invention. In Fig. 1, 10 denotes an HMD, 11 denotes a camera, 12 denotes a distance measurement unit, 13a and 13b denote a pair of left and right image projection units, 14 denotes a screen, 15 denotes a group of position and movement sensors, 16 denotes a control unit, 17a and 17b denote a pair of left and right speakers, 18 denotes a microphone, and 19a and 19b denote wearable units.

[0013] An HMD user (a person experiencing the virtual space) wears the HMD 10 on their head using mounting units 19 a and 19 b. Mounting unit 19 a supports the HMD at the nose of the face, and mounting unit 19 b secures the HMD around the head. The HMD 10 may be a glasses-type HMD.

[0014] The camera 11 captures an image in front of the HMD. The control unit 16 captures the image from the camera, recognizes real objects from the image, and assigns depth data obtained from the distance measurement unit 12 to the real objects to recognize the real objects in three dimensions. Furthermore, the projection units 13a and 13b generate virtual images as two-dimensional or three-dimensional images to be projected onto a screen 14. The control unit 16 also generates sounds to be amplified by the speakers 17a and 17b.

[0015] The projection units 13a and 13b and the screen 14 constitute the display unit 19 of the HMD. An image of a virtual object viewed with the left eye is projected onto the screen 14 by the projection unit 13a, and an image viewed with the right eye is projected onto the screen 14 by the projection unit 13b, so that the virtual object is displayed as if it were at a predetermined distance in real space. If the screen 14 is a semi-transparent screen, the virtual image appears to be mixed into the real space viewed through the screen 14.

[0016] 2 is a functional block diagram of the HMD 10 according to an embodiment of the present invention, showing the details of the internal configuration of the control unit 16. Note that the same blocks as those in FIG. 1 are assigned the same numbers. The projection units 13a and 13b in FIG. 1 are collectively referred to as 13, and together with the screen 14, they form the display unit 19.

[0017] 2 has a storage unit (e.g., FROM) 43, and includes an operation recognition unit 20, a connection and communication unit 21, a virtual image control unit 22, a position and movement processing unit 23, a virtual image generation unit 24, an information control unit 25, a display processing unit 26, and an information storage unit 27. In this embodiment, an example will be described in which the control unit 16 is realized by a software module stored in the storage unit, but it may also be realized by using hardware.

[0018] The operation recognition unit 20 receives the camera image from the camera 11 and the depth data from the distance measurement unit 12, recognizes the user's fingers, arms, and other real objects captured in the camera image, and assigns depth data to the feature points of the real objects. Furthermore, it recognizes the user's intended operation from the movements of the user's fingers and hands.

[0019] The connection and communication unit 21 connects to a PC or the like and receives virtual images and copy information from the PC. It also connects to the Internet, etc., in accordance with communication protocols such as LAN. The position and movement processing unit 23 determines the HMD's viewpoint based on the position and the HMD's line of sight based on the sensor signals of the GPS, orientation, and gyro sensor output by the position and movement sensor 15.

[0020] The virtual image control unit 22 obtains a virtual image via the connection communication unit 21 and sends it to the display processing unit 26. The information control unit 25 obtains copy information via the connection communication unit 21 and stores it in the information storage unit 27. It also reads out the copy information stored in the information storage unit 27 and sends it to the virtual image generation unit 24. The virtual image generation unit 24 generates a virtual image from the copy information to be sent to the display processing unit 26. As an example of the relationship between copy information and virtual images, the copy information is document data or encoded image data, and the virtual image is two-dimensional or three-dimensional image data for display that is generated from this data. The virtual image is generated according to a display shape that deforms to reflect the line of sight of the HMD, etc., and further includes attributes related to the display position.

[0021] The display processing unit 26 sends the virtual image from the virtual image control unit 22 or the virtual image generated by the virtual image generation unit 24 to the display unit 19 .

[0022] FIG. 3 is a diagram illustrating a case where an HMD and a PC cooperate with each other in an embodiment of the present invention and the HMD is used as an extended display of the PC.

[0023] 3, reference numeral 10 denotes the HMD, 30 denotes the user, 31a denotes a PC, 31b denotes the physical display of the PC, 32b denotes a display frame (also referred to as a virtual display) that displays a virtual image, and 32 denotes the viewing angle of the HMD. The PC 31a and the physical display 31b may be integrated as shown in the figure, or may be separate. The PC 31a and the HMD 10 are connected by a wired connection such as a USB or a display port, or by a wireless LAN.

[0024] The PC 31a uses the display frame 32b displayed on the display of the HMD 10 as an extended display for the PC 31a. The PC 31a determines the position, size, and line-of-sight angle of the display frame 32b in a PC coordinate system with the PC 31a or the physical display 31b of the PC as the origin. The PC coordinate system is a coordinate system in which, when the user moves the HMD, the virtual image maintains its positional relationship with the PC without following the movement of the HMD. The PC 31a further transmits the virtual image to the HMD 10 and displays it on the display frame 32b. The display frame 32b is recognized by the PC 31a as a virtual display, thereby realizing a multi-display system for the PC 31a. The PC 31a can display information on the display frame 32b of the HMD 10 in addition to the physical display 31b, allowing the user to view and view more information at a glance.

[0025] FIG. 4A shows an example of a case where multiple display frames are displayed on the HMD 10 according to an embodiment of the present invention. Because the visual field of the HMD 10, i.e., the range that can be displayed on the display of the HMD 10, is limited, the user only sees the virtual display frames displayed within the visual field of the HMD 10 at that time. In the example of FIG. 4A , the user is facing forward, and display frames 32b and 32c within the visual field of the HMD 32 are displayed on the display of the HMD 10. When the user turns their head, for example, to the left, the direction of the visual field of the HMD 32 changes to the left, and display frame 32a is displayed on the display of the HMD 10. Similarly, when the user turns to the right, display frame 32d is displayed on the display of the HMD 10. In this way, the multiple display frames of the HMD 10 are displayed in the aforementioned PC coordinate system, thereby achieving the same display as when multiple physical displays are arranged in real space.

[0026] 4B and 4C are diagrams illustrating the operations of selecting, saving, and reading information. In FIGS. 4B and 4C, 32a to 32c are the same display frames as in FIG. 4A. 33a and 33b are virtual images displayed in display frame 32b, 34a is an operation button for operating information saving, and 34b is a read button that indicates the state in which information is saved and reads it out.

[0027] FIG. 4B is an explanatory diagram of selecting and copying information in an embodiment of the present invention.

[0028] 4B , when the user 30 attempts to copy and save the virtual image 33a to the HMD 10, the user 30 operates, for example, a pointing device on the PC 31a to select the virtual image 33a with a left click and then right-click to display the operation button 34a. Further left-clicking the operation button 34a copies the virtual image 33a from the PC 31a to the HMD 10. When the copy is executed, the operation button 34a switches to a read operation button 34b, as shown in FIG. 4C . This copy operation is not limited to one piece of information; multiple pieces of information may be selected and pressed simultaneously, or multiple pieces of information may be copied by repeatedly copying each piece of information.

[0029] 4C is an explanatory diagram illustrating how copy information is read out in an embodiment of the present invention. To switch the HMD 10 from the linked mode with the PC 31a to the position-free mode and view information at a location away from the PC, the user operates the HMD 10 in the state shown in FIG. 4C and presses the read operation button 34b. At this time, the HMD automatically switches to the position-free mode.

[0030] For example, when a maintenance worker performs maintenance on a plant, the worker displays instruction information for the plant's equipment. In PC-linked mode, a huge amount of information is displayed on the PC's display and the HMD's extended display. The user operates the PC to select the necessary equipment instructions while viewing the displayed images, and copies the selected instruction information to the HMD. When switching to position-free mode, the HMD is directly operated to generate and display a virtual image of the copied information. This allows the user wearing the HMD to move to the vicinity of plant equipment located away from the PC and operate the equipment while viewing the virtual image.

[0031] FIG. 5 is a diagram illustrating another example of copying and saving information in an embodiment of the present invention. This example shows a case in which the user 30 displays a virtual image 33a on a display frame 32b displayed on the HMD 10. By operating a pointing device or the like on the PC 31a, the user moves the virtual image 33a displayed on the physical display 31b onto the display frame 32b by left-clicking and dragging the virtual image. At this time, the virtual image 33a is simultaneously displayed on the display frame 32b and copied and saved to the HMD 10, and subsequent display is performed on the HMD side. Simultaneously moving the information display relative to the display frame and copying and saving the information has the advantage of making it easier to understand the location of the information compared to instructions using button operations.

[0032] Figure 6 is a diagram illustrating a case where the HMD 10 according to an embodiment of the present invention is used in position-free mode. In Figure 6, the HMD 10, user 30, and HMD viewing angle 32 are the same as those in Figure 4, but 36 indicates the range of virtual space visible to the HMD (the range is not actually displayed), and a virtual image generated from the virtual image 33a read and operated in Figure 4C is displayed within the range 36 of the visible virtual space. The HMD 10 is in position-free mode, and the virtual image is displayed in a local coordinate system originating from the HMD 10. Hereinafter, the local coordinate system originating from the HMD 10 will be referred to as the HMD coordinate system.

[0033] At this time, the display position and size can be set by user operation. In the HMD coordinate system, when the user moves the HMD, the virtual image tries to maintain its positional relationship with the HMD, so the position of the virtual image in the virtual space does not change. It is also possible to switch between saved copy information and display it by pressing the information selection buttons 35a, 35b, and 35c that represent saved copy information. Operations other than information selection are also possible, such as simultaneously displaying multiple pieces of information, controlling the overlap of the displays, and turning pages of information, all of which can be performed by user operation of the HMD 10.

[0034] When switching to position-free mode and moving away from the PC, the HMD cannot use the PC's pointing device, so operations are performed using gesture input, etc., using the HMD's built-in camera. Compared to a PC's pointing device, gesture input makes it difficult to specify precise positions on the screen. Also, pointing devices connected to PCs do not generally have multiple physical buttons, so there are fewer operational variations. Therefore, input operations are difficult if the display remains the same as in PC-connected mode. Therefore, position-free mode provides an on-screen operation interface that is suitable for gesture input, etc.

[0035] The change of interface may be determined by the operation recognition unit 20 receiving a user instruction in step S69 of Figure 13, or may be determined based on whether the PC is within the visual field of the image obtained from the camera.

[0036] The interface may also be changed based on the selected display coordinate system. For example, if the world coordinate system is selected, an interface displaying buttons and the like of the minimum size necessary for PC operation is provided, and if another local coordinate system or trunk coordinate system is selected, an interface displaying large buttons and the like suitable for HMD operation is provided, resulting in a system with good operability that requires less manual interface switching.

[0037] 7A and 7B are diagrams illustrating changes in the user interface when a mode is changed in an embodiment of the present invention. For example, as shown in Fig. 7A, in PC link mode, priority is given to the visibility of the virtual image 33a, and buttons and the like are displayed small. In position-free mode, buttons are displayed large as shown in Fig. 7B, or buttons are displayed with more space between them to make them easier to press with gestures.

[0038] The interface may be switched simultaneously with the switch to the location-free mode. Alternatively, after the switch to the location-free mode, the display may be switched when various sensors detect that the user has moved a certain distance or more away from the PC. A switching button or the like may be displayed, and the user may manually switch the display by pressing the button. The interface is switched by the operation recognition unit 20 receiving a change instruction, similar to the processing of step S69 in the flowchart of FIG. 13 , and the virtual image generation unit 24 generates a virtual image with the changed interface.

[0039] As described above, according to the first embodiment of the present invention, in an HMD that can be linked with an information device, a function for using the HMD in a position-free manner can be provided. The position-free mode provided allows for the display of virtual images that are not restricted by the display frame when linked with an information device, and has the characteristic that by switching to a screen suitable for input operations using the HMD alone in accordance with mode switching, it becomes easier to operate the HMD alone. [Second embodiment of the present invention] A second embodiment will be described with reference to Figures 8 and 9.

[0040] 8 is a block diagram illustrating an HMD configured using an information processing device according to an embodiment of the present invention. In the first embodiment, the HMD 10 is configured as a standalone HMD, but in the second embodiment, the HMD 10 is configured as an HMD and an information processing device separate from the HMD. This differs from the functional block diagram of FIG. 2 in that the control unit 16 is configured as an information processing device, which has the advantage that a general-purpose device such as a portable terminal like a smartphone can be used as the information processing device.

[0041] In FIG. 8, the same components as those in FIGS. 1 and 2 are assigned the same numbers. In FIG. 8, the control unit 16 includes a communication unit 40, a CPU (Central Processing Unit) 41, a RAM (Random Access Memory) 42, a FROM (Flash Read Only Memory) 43, and an interface unit 47a. The interface unit 47a is connected to an interface unit 47b in the HMD main body. It also serves as an external connection and can be used to connect to a PC. The interfaces realized by the interface units 47a and 47b may be wired or wireless, such as USB. Although not shown, the control unit 16 may also include an operation interface unit such as a display unit or touch panel, as in a typical smartphone.

[0042] The communication unit 40 of the control unit 16 can select an appropriate process from several communication processes, such as mobile communications such as 4G and 5G, wireless LAN (Local Area Network), etc., to connect the HMD 10 to a network or a PC. Furthermore, the FROM 43 stores a basic HMD program 44 and a PC-linked HMD program 45 as processing programs, and also includes an information storage unit 46. The processing programs 44 and 45 are deployed in the RAM 42 and executed by the CPU 41. Furthermore, the information storage unit 46 temporarily stores intermediate data required to execute the processing programs, as well as copy information obtained from the PC. The FROM 43 may be a single memory medium or may be composed of multiple memory media. It may also be a non-volatile memory medium other than the FROM.

[0043] 9 is a flowchart showing an example of a cooperative HMD program according to an embodiment of the present invention. When the multi-display system of the PC is executed by the basic HMD program, the PC cooperative HMD program 45 is started and processing starts from S10.

[0044] In S11, a virtual image is received from the PC. Corresponding display frame information is assigned to the virtual image, and in S12, the virtual image is displayed. In S13, the program's termination condition (such as the PC stopping the multi-display system) is checked, and if it is to be terminated (YES), the process ends in S23. If it is to be continued (NO), the process checks whether information copy is waiting to be executed in S14, and if a copy operation is not waiting to be executed (NO), the process returns to step S11, and if a copy operation is waiting to be executed, the copy information is received and saved in S15. Steps S11 to S16 constitute the PC linkage mode.

[0045] In S16, it is confirmed whether or not the copy information has been reproduced by operating the HMD. If no reproduction operation has been performed (NO), the process returns to step S11. If a reproduction operation has been performed (YES), the process transitions to position-free mode, and the copy information is read out in S17. A virtual image of the copy information is generated in S18, and the virtual image is displayed in S19. There are no restrictions on the display of the virtual image as long as it is within the visible range of the HMD, and the image to be generated can be specified by user operation.

[0046] In S20, an instruction to change the copy information to be played is accepted, and if the change is to be made (YES), the process returns to step S17. If the change is not to be made (NO), the process checks whether or not a PC operation has occurred in S21. If the PC operation has occurred (YES), the process returns to step S11 to transition from the location-free mode to the PC-linked mode. If the PC operation has not occurred (NO), the process checks the termination conditions in S22, and if the termination is to be made (YES), the process ends in S23. If the operation is to continue (NO), the process returns to step S20.

[0047] 10 is a diagram illustrating the transition of information display to a mobile information terminal in an embodiment of the present invention, in which information displayed on a PC in location-free mode is displayed on the display device of a mobile information terminal.

[0048] When the device is switched to location-free mode, the display and operation that were previously performed on the PC may be performed on a smartphone, which is a mobile mobile information terminal. Specifically, the information displayed on the PC screen is copied to the smartphone screen and displayed on the smartphone screen. When displaying information that was displayed on the PC on the smartphone, an application used to display the information is launched on the smartphone as necessary before the information is displayed.

[0049] Furthermore, operations that were previously performed using the input operation interface of a PC can be performed using the touch panel of a smartphone, etc. As in PC connection mode, the touch panel of the smartphone can be used to perform operations such as moving the cursor or pressing buttons within the virtual frame displayed on the HMD. Using the touch panel of the smartphone as a pointing device allows for more precise operations than operations using gesture input on the HMD alone. While a smartphone and its touch panel have been given as examples, the type of device and the type of input interface are not important as long as using a mobile device and its input interface makes operation easier than using the HMD alone.

[0050] As described above, the second embodiment of the present invention has the same effects as the first embodiment, and also has the advantage that by carrying a versatile information processing device instead of a PC, it is possible to provide convenience similar to that of PC-linked mode even when traveling. [Third Embodiment of the Present Invention] Next, a third embodiment will be described with reference to Figure 11 and Figures 12A to 12D.

[0051] The processing flow of the third embodiment will be described with reference to an example of a flowchart of FIG. 13 showing a method for displaying a virtual image in an embodiment of the present invention.

[0052] In the third embodiment of the present invention, in the position-free mode, a trunk coordinate system is used, with the origin being the user's body equipped with a body sensor. The virtual image placed in the trunk coordinate system has a fixed position relative to the position and orientation of the user's body, and the user can look around the virtual image by moving the HMD separately from the user's body, such as by shaking their head.

[0053] FIG. 11 is a block diagram of an HMD corresponding to a trunk coordinate system in an embodiment of the present invention. The only difference from the block diagram of FIG. 8 is that a body sensor input 50 is input to an interface unit 47b. The body sensor input 50 calculates the body orientation from the difference in output between a sensor worn on the body and a sensor mounted on the HMD, and inputs the calculated information to the HMD. Alternatively, the user's body orientation may be determined using a camera attached to the HMD and used as body sensor input information. FIGS. 12A-D are diagrams illustrating a virtual image of an HMD placed in a trunk coordinate system.

[0054] Fig. 12A is a diagram illustrating a virtual image of an HMD based on a trunk coordinate system in an embodiment of the present invention. The HMD 10, user 30, HMD viewing angle 32, and range 36 of virtual space visible from the HMD are the same as those already described. Virtual images 33c to 33f are placed in the trunk coordinate system and displayed in range 36 of virtual space visible from the HMD. Fig. 12A shows the state immediately after the HMD 10 has transitioned from PC-linked mode to position-free mode, but the transition to position-free mode is performed using the example screen shown in Fig. 12B.

[0055] In the process of transitioning from the PC-linked mode to the position-free mode, the operation recognition unit 20 judges the image received from the camera 11, and determines whether the display coordinate system should be changed based on whether the PC is within the visual field of view in step S65. This determination may be made by pressing a button that instructs a coordinate system change displayed in the virtual image of the HMD. The coordinate system to be transitioned to may be a world coordinate system, a local coordinate system, a trunk coordinate system, or the like.

[0056] FIG. 12B is a diagram illustrating the transition to the trunk coordinate system in an embodiment of the present invention. For example, by operating a pointing device or the like on the PC 31a, a save-all operation button 34c is displayed within the visible range 36 of the virtual space. In FIG. 12B, the save-all operation button 34c is displayed superimposed on the virtual image 33d. Furthermore, by operating the HMD 10 to press the save-all operation button 34c, information corresponding to the virtual images 33c to 33f is copied from the PC 31a to the HMD 10 and saved. At the same time, the HMD 10 transitions to the position-free mode. In the position-free mode, as shown in FIG. 12A, the virtual images 33c to 33f are displayed even when the PC's physical display 31b is outside the visible range of the HMD 10. This is because the virtual image data for displaying the virtual images 33c to 33f is stored within the HMD 10. The virtual images 33c to 33f are held throughout the trunk coordinate system and are maintained even when the display is outside the visible range of the virtual space.

[0057] 12C is a diagram illustrating a case where the angle of view is narrow in the trunk coordinate system according to an embodiment of the present invention. Virtual images 33d and 33e are displayed within a visible range 36 of the virtual space corresponding to the narrowed visible angle of view 32, and user 30 can view enlarged virtual images 33d and 33e.

[0058] The selection of the narrow angle of view is accepted by the operation recognition unit 20 together with the change of the coordinate system in the process of step S65, thereby making it possible to provide a virtual image that meets the user's request.

[0059] FIG. 12D is a diagram illustrating the case where the HMD 10 is rotated in the trunk coordinate system in an embodiment of the present invention. A case where the HMD 10 is rotated from the state of FIG. 12C is described. Virtual images 33c and 33d are displayed within a visible range 36 of the virtual space corresponding to the narrowed and rotated viewing angle 32. Note that virtual images displaced in the trunk coordinate system in FIGS. 12C and 12D are regenerated using corresponding stored information. By moving the head wearing the HMD 10, the user can look around at the regenerated virtual image fixed in world coordinates ahead of the line of sight, allowing the user to view more information relative to the display range of the actual display of the HMD 10.

[0060] When the HMD is rotated, the virtual image display is rotated based on the origin that determines the position at which the virtual image is to be displayed by specifying the display position in step S71.

[0061] In addition, by copying the information displayed on the physical display in PC link mode to the virtual display, you can perform the same tasks as viewing on a PC in one location in another. The position of each virtual display does not change relative to your body orientation even in another location, so you can work efficiently without having to re-learn the placement.

[0062] As described above, the third embodiment has the characteristics of the first and second embodiments, and also has the effect of preventing a decrease in the user's work efficiency even when switching from PC link mode to location-free mode.

[0063] 1 to 13 are not limited to the embodiments of the present invention, and it is possible to replace part of the configuration of one embodiment with another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. These all fall within the scope of the present invention, and furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the use of different ones does not impair the effects of the present invention.

[0064] Furthermore, some or all of the functions of the invention may be implemented in hardware, for example, by designing an integrated circuit. They may also be implemented in software, where a microprocessor unit, CPU, etc. interprets and executes an operating program. Furthermore, the scope of software implementation is not limited, and both hardware and software may be used.

[0065] 10: Head-mounted display (HMD), 11: Camera, 12: Distance measurement unit, 13a, 13b: Projection unit, 14: Screen, 15: Sensor group, 16: Control unit, 20: Operation recognition unit, 21: Connection and communication unit, 22: Virtual image control unit, 23: Position and movement processing unit, 24: Virtual image generation unit, 25: Information control unit, 26: Display processing unit, 27: Information storage unit, 30: User, 31a: PC, 31b: Physical display, 32: HMD viewing angle, 32a to 32d: Display frames, 33a to 33f: Virtual images, 34a to 34c: Operation buttons, 35a to 35c: Information selection buttons, 36: Range of visible virtual space, 41: CPU, 43: FROM, 44: Basic HMD program, 45: PC-linked HMD program, 47a, 47b: Interface unit, 50: Body sensor input.

Claims

1. A head-mounted display equipped with a transmissive screen capable of interlocking with an information processing device, comprising: a recognition unit that recognizes an information processing device within the visual field of the head-mounted display; a communication unit that receives data from the information processing device recognized by the recognition unit; a memory unit that stores the data received by the communication unit; and a display unit that displays the data stored in the memory unit.

2. A head mounted display as described in claim 1, comprising an interface processing unit which, when the recognition unit determines that the information processing device is within the visual angle of the head mounted display, accepts the processing of the data through a first interface, and, when the information processing device is not within the visual angle of the head mounted display, accepts the processing of the data through a second interface.

3. A head mounted display as described in claim 1, wherein, when the recognition unit determines that the information processing device is within the visual angle of the head mounted display, the display unit displays the data in a coordinate system associated with the information processing device, and, when the recognition unit determines that the information processing device is not within the visual angle of the head mounted display, the display unit displays the data in a coordinate system associated with the head mounted display.

4. A head-mounted display as described in claim 3, further comprising an image generation unit which, when the recognition unit receives an instruction to change the coordinate system, generates a virtual image of the data stored in the memory unit based on the received coordinate system, and displays the virtual image generated by the display unit.

5. A head mounted display as described in claim 3, wherein the recognition unit receives an arbitrary position within a virtual space visible to the head mounted display, and an image generation unit generates a virtual image of the data stored in the memory unit based on the received arbitrary position, and the head mounted display displays the virtual image generated by the display unit.

6. A head-mounted display as claimed in claim 1, wherein the display unit displays the data in a trunk coordinate system with the user's body as the origin if the information processing device is not within the visual field angle of the head-mounted display.

7. A head mounted display according to claim 6, wherein the display unit outputs the data stored in the memory unit when a predetermined movement of the head mounted display is detected.

8. A data display method for a head-mounted display equipped with a transmissive screen capable of interlocking with an information processing device, comprising: a recognition unit that recognizes an information processing device within the visual field angle of the head-mounted display; a communication unit that receives data from the recognized information processing device; a memory unit that stores the received data; and a display unit that displays the stored data.

9. A head mounted display system having a transparent screen capable of interfacing with an information processing device, comprising: a portable terminal having a recognition unit that recognizes an information processing device within the visual field of the head mounted display, a communication unit that receives data from the information processing device recognized by the recognition unit, and a memory unit that stores the data received by the communication unit; and a head mounted display having a display unit that displays the data stored in the memory unit.