Virtual input device, virtual input system, virtual input method, and program

Abstract

To provide a virtual input device capable of realizing continuous operation to a virtual terminal displayed in a virtual space.SOLUTION: A virtual input device comprises: an input / output unit that acquires image data captured by a camera of a virtual reality display device, and outputs display information to be displayed on a display unit of the virtual reality display device to the virtual reality display device; an image analysis unit that analyzes the image data to identify a position at which a virtual terminal is to be displayed; and a display information generation unit that generates display information including the virtual terminal according to user information of a user wearing the virtual reality display device.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a virtual input device that receives an input operation in a virtual space and the like.

Background Art

[0002] A virtual reality display device having a display wearable on a human body, such as a head-mounted display (HMD), has been developed. By wearing such a device, visual information displayed on the display can be visually recognized.

[0003] Patent Document 1 discloses a method of displaying a virtual reality environment on an HMD. In the method of Patent Document 1, it is determined whether there is an opportunity in a virtual reality environment for user interaction with real-world objects. In the method of Patent Document 1, in response to the determined opportunity, a real-world object located within the real-world view point of the user of the head-mounted display is detected. In the method of Patent Document 1, an image of the detected real-world physical object is displayed on the head-mounted display.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By using the method of Patent Document 1, a user interface that receives operations in a virtual space displayed on a display can be realized. For example, a user wearing an HMD can perform an operation on visual information constructed in a virtual space, thereby realizing a user interface in which predetermined input processing is executed. In such a method, in order to continuously view the visual information displayed on the display of the HMD, it is necessary to continuously pay attention and concentration. Therefore, due to the fatigue caused by the continuous operation using the virtual reality display device, there is a possibility that a touch panel, a switch, etc. constructed in the virtual space may be overlooked and an incorrect operation may be performed.

[0006] An object of the present disclosure is to provide a virtual input device or the like that can realize continuous operations on a virtual terminal displayed in a virtual space.

Means for Solving the Problems

[0007] A virtual input device according to an aspect of the present disclosure includes an input / output unit that acquires image data captured by a camera of a virtual reality display device and outputs display information displayed on a display unit of the virtual reality display device to the virtual reality display device, a video analysis unit that analyzes the image data to identify a position where a virtual terminal is to be displayed, and a display information generation unit that generates display information including a virtual terminal according to user information of a user who uses the virtual reality display device.

[0008] In a virtual input method according to an aspect of the present disclosure, image data captured by a camera of a virtual reality display device is acquired, the image data is analyzed to identify a position where a virtual terminal is to be displayed, display information including a virtual terminal according to user information of a user who uses the virtual reality display device is generated, and the generated display information is output to the virtual reality display device to be displayed on a display unit of the virtual reality display device.

[0009] A program according to an aspect of the present disclosure causes a computer to execute a process of acquiring image data captured by a camera of a virtual reality display device, a process of analyzing the image data to identify a position where a virtual terminal is to be displayed, a process of generating display information including a virtual terminal according to user information of a user using the virtual reality display device, and a process of outputting the generated display information to the virtual reality display device to be displayed on a display unit of the virtual reality display device.

Effect of the Invention

[0010] According to the present disclosure, it becomes possible to provide a virtual input device or the like that can realize continuous operations on a virtual terminal displayed in a virtual space.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. However, although the embodiments described below have technically preferable limitations for carrying out the present invention, the scope of the invention is not limited thereto. In all the drawings used in the following description of the embodiments, the same reference numerals are given to the same parts unless otherwise specified. Also, in the following embodiments, repeated explanations regarding the same configuration and operation may be omitted.

[0013] (First Embodiment) First, the virtual input system according to the first embodiment will be described with reference to the drawings. The virtual input system of this embodiment includes a virtual reality display device having a display unit wearable on the human body, such as a head-mounted display (HMD). Hereinafter, an example in which the virtual reality display device is realized by a non-transmissive HMD will be described assuming a usage scene of virtual reality (VR). The virtual input system of this embodiment is also applicable to usage scenes such as augmented reality (AR) and mixed reality (MR). Therefore, depending on the usage scene, the virtual reality display device may be a transmissive HMD. In this embodiment, an example in which the virtual input system includes a virtual reality display device and a virtual input device will be given. Hereinafter, an example in which the virtual reality display device and the virtual input device are configured by different devices will be described. The virtual input device may be built into the virtual reality display device.

[0014] The virtual reality display device of this embodiment includes a pair of cameras that capture the real world corresponding to the range of the user's field of view, and a display unit on which virtual reality viewed by the user is displayed. In the following, in order to distinguish between the pair of cameras, the camera on the right-eye side is assigned the letter R (Right) of the alphabet, and the camera on the left-eye side is assigned the letter L (Left) of the alphabet. Note that when not distinguishing between the right-eye side and the left-eye side cameras, the R or L at the end of the reference sign may be omitted.

[0015] (Configuration) FIG. 1 is a conceptual diagram showing an example of the configuration of a virtual input system 1 according to this embodiment. The virtual input system 1 of this embodiment includes a virtual input device 10, a virtual reality display device 100, and an operation terminal 170. The virtual input device 10 is connected to the virtual reality display device 100 and the operation terminal 170 by wire or wirelessly. The virtual input device 10 may be built into the virtual reality display device 100. There is no particular limitation on the connection method between the virtual input device 10, the virtual reality display device 100, and the operation terminal 170. As will be described later, the operation terminal 170 may be realized by a mock-up having no input function.

[0016] The virtual reality display device 100 includes a control device 110, a camera 120R, a camera 120L, and a display unit 130. Further, the virtual reality display device 100 includes a driver described later. The control device 110 may be arranged corresponding to at least one of the camera 120R and the camera 120L. The camera 120R and the camera 120L have the same configuration. In the following, there are cases where the camera 120R and the camera 120L are not distinguished and are described as the camera 120. Further, the virtual reality display device 100 includes a cover 140 and a band 160. The display unit 130 is fixed to the human body side of the cover 140. The band 160 is a band for fixing the virtual reality display device 100 to the head.

[0017] The virtual reality display device 100 uses the camera 120 to capture images and videos in front of the virtual reality display device 100. The virtual reality display device 100 transmits the image captured by the camera 120 to the virtual input device 10. In the example of FIG. 1, the virtual reality display device 100 has two cameras 120. The virtual reality display device 100 may be configured with a single camera 120 or three cameras 120. The position where the camera 120 is disposed may not be on both sides of the cover 140. For example, the camera 120 may be disposed above, below, or in front of the cover 140. For example, if the camera 120 is disposed at a position corresponding to a person's eyes, a video close to the image viewed by the person can be captured.

[0018] The camera 120L and the camera 120R are a pair of cameras for photographing an outdoor scene. The camera 120L and the camera 120R constitute a stereo camera. The camera 120L and the camera 120R generate image data corresponding to the photographed outdoor scene. From the positions in the image data captured by each of the camera 120L and the camera 120R, information on the depth direction of the outdoor scene in the real world on the other side of the cover 140 (display unit 130) for the user wearing the virtual reality display device 100 can be measured. For example, the virtual reality display device 100 can calculate the distance between the virtual reality display device 100 and the object from the difference in pixel positions of the image data captured by the camera 120L and the camera 120R. The camera 120L and the camera 120R can be realized by a general digital camera. Also, the camera 120L and the camera 120R may be realized by an infrared camera or an ultraviolet camera.

[0019] In addition, the virtual reality display device 100 acquires display information from the virtual input device 10. The display information includes the image displayed on the display unit 130 and the coordinates at which the image is displayed. The virtual reality display device 100 displays an image corresponding to the acquired display information. The image displayed on the display unit 130 of the virtual reality display device 100 corresponds to the real world on the other side of the cover 140 (display unit 130) for the user. Therefore, an image at the position in the real world is displayed on the display unit 130 of the virtual reality display device 100. In the present embodiment, an image (virtual keyboard) corresponding to the operation terminal 170 at the position in the real world is displayed on the display unit 130 of the virtual reality display device 100.

[0020] FIG. 2 is a conceptual diagram showing an example of an image displayed on the display unit 130 of the virtual reality display device 100. In the example of FIG. 2, a virtual monitor VM1 and a virtual monitor VM2 are displayed on the display unit 130. In addition, FIG. 2 shows the operation terminal 170 arranged in the real world on the other side of the display unit 130 and the position of the user's hand by a dashed line. Information used by the user is displayed on the virtual monitor VM1 and the virtual monitor VM2 (hereinafter, virtual monitor VM). For example, instructions and notifications for the user are displayed on the virtual monitor VM. The user can use the operation terminal 170 to perform an input corresponding to the instructions and notifications displayed on the virtual monitor VM.

[0021] FIG. 3 is a conceptual diagram showing an example of an image displayed on the display unit 130 of the virtual reality display device 100. In the example of FIG. 3, a virtual monitor VM1 and a virtual monitor VM2 (virtual monitor VM) are displayed on the display unit 130. Also, in the example of FIG. 3, a virtual terminal VT is displayed on the display unit 130 in association with the position of the operation terminal 170 arranged in the real world on the other side of the display unit 130. Further, an image of a hand captured by the camera 120 is displayed on the display unit 130 in association with the position of the user's hand in the real world. FIG. 3 shows a display state in which the next selection target key K0 is highlighted. The displayed hand image may be an image of an actual hand or a picture imitating a hand. For example, the virtual terminal VT and the hand are displayed on the display unit 130 in accordance with the timing at which the operation terminal 170 and the hand are photographed by the camera 120.

[0022] The key arrangement and key binding of the virtual terminal VT may be set according to user information. For example, the key arrangement and key binding of the virtual terminal VT are set according to the organization to which the user belongs. If the key arrangement and key binding of the virtual terminal VT are set for each organization to which the user belongs, the work can be streamlined according to the needs of each organization. For example, the key arrangement and key binding of the virtual terminal VT are set for each user. If the key arrangement and key binding of the virtual terminal VT are set for each user, the work can be streamlined according to the needs of each user.

[0023] FIG. 4 is a conceptual diagram showing another example of an image displayed on the display unit 130 of the virtual reality display device 100. FIG. 4 shows a state in which the user has performed an input operation on the highlighted key K0. The user wearing the virtual reality display device 100 performs an input operation on the highlighted key K0 in the virtual terminal VT displayed on the display unit 130. As a result, the user presses the key K0 of the operation terminal 170 arranged in the real world.

[0024] FIG. 5 is a conceptual diagram showing an example of an input operation on the operation terminal 170 arranged in the real world. When the user attempts to perform an input operation on the virtual terminal VT displayed on the display unit 130 as shown in FIG. 4, an input operation on the operation terminal 170 arranged in the real world is executed. An input to the virtual reality display device 100 is executed by the input operation on the operation terminal 170 arranged in the real world. The user operates the operation terminal 170 as if operating the virtual terminal VT. Therefore, the user can operate the virtual terminal VT while feeling the touch of operating the operation terminal 170.

[0025] The operation terminal 170 is a terminal device that receives an input to the virtual reality display device 100. The operation terminal 170 is not limited as long as it can execute an input operation such as a key input. For example, the operation terminal 170 is a general-purpose keyboard. For example, the operation terminal 170 may be a dedicated terminal customized for an organization or business.

[0026] At least one mark for detecting the position and orientation of the operation terminal 170 is attached to the operation terminal 170. The mark may be a mark that can be detected by visible light or a mark that can be detected by infrared light. Also, the mark may not be an intentionally attached mark, but a feature extracted from the surface of the operation terminal 170. For example, the mark may be a scratch, a character, etc. that can be recognized and identified by a machine reading technology such as an object fingerprint authentication technology. If the mark attached to the operation terminal 170 is aligned with the reference position of the virtual terminal VT virtual, the real world and the virtual space are displayed in association with each other. By using a plurality of reference points, even if the operation terminal 170 arranged in the real world rotates at an arbitrary angle, the operation terminal 170 and the virtual terminal VT can be superimposed and displayed. Also, the inclination of the virtual terminal VT may be calculated in conjunction with a sensor that detects the inclination of the operation terminal 170. For example, when the operation terminal 170 arranged in the real world is detected by image recognition and configured to align the detected operation terminal 170 with the reference position of the virtual terminal VT, it is not necessary to attach a mark to the operation terminal 170. For example, by detecting feature points such as the vertices of the operation terminal 170, the position and angle of the operation terminal 170 can be calibrated.

[0027] At least one key K for receiving input to the virtual reality display device 100 is arranged on the operation terminal 170. For example, the operation terminal 170 is realized by a general keyboard. For example, symbols such as alphabets and numbers, and specific functions are assigned to the key K. A specific function may be assigned to the key K in advance, or a specific function may be assigned according to the user's settings. For example, a specific function may be assigned to the key K according to the user who uses the virtual reality display device 100, the group to which the user belongs, or the role of the user. For example, the operation terminal 170 may be a dedicated terminal specialized for input of a specific function.

[0028] For example, the operation terminal 170 can be applied to a dedicated terminal used in a fire command center. The dedicated terminals used in the fire command center have different frequently used keys for each organization. Therefore, even for the same dedicated terminal, the dedicated terminal is customized for each organization in order to make the frequently used keys prominent. Such customization is made for each fire department or each section within the fire department. In the actual usage scenario, it is required to arrange the functions of the keys for each individual. By using the virtual input system 1 of the present embodiment, since the display of the virtual terminal VT displayed in the virtual space can be changed, there is no need to customize the dedicated terminal for each organization.

[0029] For example, the operation terminal 170 may not be arranged in the real world. In that case, it may be configured such that an input operation on the virtual terminal VT is performed according to the position and movement of the user's finger or hand in the real world photographed by the camera 120. If the position and movement of the key of the virtual terminal VT and the user's finger or hand are associated with each other, the input to the virtual terminal VT can be realized.

[0030] For example, instead of the operation terminal 170, a mockup that provides feedback on the tactile sensation of key operations may be placed in the real world. The mockup may have the same design as an actual terminal device or may be plain. In that case, a user who touches the mockup placed in the real world can perform an input operation on the virtual terminal VT as if actually operating the operation terminal placed in the real world. The mockup is not particularly limited as long as it provides feedback on the tactile sensation corresponding to the key operation. For example, it may be configured such that virtual operation inputs are made using a general-purpose mockup that provides feedback on the tactile sensation of key operations. When configured in this way, the mockup and the virtual terminal VT may have different key arrangements. Also, the key arrangement of the virtual terminal VT may be rearranged and displayed on the display unit 130 in association with the key arrangement of the mockup.

[0031] In the examples of FIGS. 2 to 5, examples of input using the virtual terminal VT displayed on the display unit 130 were given. The virtual input system 1 of the present embodiment may be configured such that input is made in response to an operation on the virtual monitor VM displayed on the display unit 130 without using the virtual terminal VT. If configured in this way, intuitive operations on the display information displayed on the virtual monitor VM can execute inputs according to the purpose.

[0032] 〔Virtual Reality Display Device〕 Next, the detailed configuration of the virtual reality display device 100 will be described with reference to the drawings. FIG. 6 is a block diagram showing an example of the configuration of the virtual reality display device 100. The virtual reality display device 100 includes a control device 110, a camera 120R, a camera 120L, a display unit 130, and a driver 150. The control device 110 may be arranged corresponding to at least one of the camera 120R and the camera 120L. The camera 120R and the camera 120L have the same configuration. In the following, there may be cases where the camera 120R and the camera 120L are not distinguished and are described as the camera 120. Further, the virtual reality display device 100 includes a cover 140 and a band 160. The display unit 130 is fixed to the human body side of the cover 140. The band 160 is a band for fixing the virtual reality display device 100 to the head.

[0033] The control device 110 includes a communication unit 111, a display controller 113, a memory 115, and a power supply 117. For example, the control device 110 is arranged corresponding to at least one of the camera 120L and the camera 120R. The control device 110 is configured inside the housing. The control device 110 may be arranged on the cover 140 or the band 160. For example, the control device 110 is realized by hardware having a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, and the like.

[0034] The communication unit 111 (communication means) is a communication interface that communicates with the virtual input device 10. The communication unit 111 transmits and receives data to and from the virtual input device 10. For example, the communication unit 111 is connected to the virtual input device 10 by wireless communication such as Bluetooth (registered trademark), WiFi (registered trademark), or wireless LAN (Local Area Network). The communication unit 111 may be connected to the virtual input device 10 by a wireless method other than Bluetooth (registered trademark), WiFi (registered trademark), or wireless LAN. Further, the communication unit 111 may be connected to the virtual input device 10 via a wired cable as long as it does not affect the user's workability.

[0035] The communication unit 111 is connected to the cameras 120L and 120R. The communication unit 111 acquires image data from the cameras 120L and 120R. The communication unit 111 transmits the acquired image data to the virtual input device 10. The communication unit 111 may be configured to transmit video data composed of a plurality of temporally continuous image data (frames) to the virtual input device 10.

[0036] Also, the communication unit 111 is connected to the display controller 113. The communication unit 111 receives display information to be displayed on the display unit 130 from the virtual input device 10. The display information includes an image to be displayed on the display unit 130 and coordinates at which the image is to be displayed. The communication unit 111 outputs the received display information to the display controller 113.

[0037] The display controller 113 is connected to the communication unit 111, the memory 115, and the driver 150. The display controller 113 acquires the display information transmitted from the virtual input device 10 from the communication unit 111. The display controller 113 acquires image data corresponding to the acquired display information from the memory 115. The display controller 113 controls the driver 150 so that an image corresponding to the acquired display information and the image data is displayed. For example, the display controller 113 controls the driver 150 so that the image included in the display information transmitted from the virtual input device 10 and the image corresponding to the image data acquired from the memory 115 are superimposed.

[0038] The memory 115 is connected to the display controller 113. Images to be displayed on the display unit 130 are stored in the memory 115. For example, image data corresponding to the images to be displayed on the display unit 130 is stored in the memory 115. For example, image data corresponding to the shapes and characters to be displayed on the display unit 130 is stored in the memory 115.

[0039] The power source 117 is a secondary battery for driving the virtual input device 10. For example, the power source 117 is a secondary battery that can be powered via a power cable. For example, the power source 117 may be a secondary battery that can be charged by wireless power supply.

[0040] The display unit 130 is connected to the driver 150. The display unit 130 is a display on which information visually recognized by the user is displayed. A plurality of pixels are arranged in an array on the display unit 130 so as to be able to represent the shape and color of the real world. Information visually recognized by the user is displayed on the display unit 130 according to the control of the driver 150.

[0041] For example, the display unit 130 is realized by a liquid crystal display or an organic EL (Electro-Luminescence) display. For example, the display unit 130 may be realized by a micro LED (Light Emitting Diode) display or an inorganic EL display. The form of the display unit 130 is not particularly limited as long as it is a display visible to the virtual reality display device 100.

[0042] The driver 150 is a driving unit that causes the display unit 130 to display an image. For example, the driver 150 includes a first driving circuit (not shown) that drives the horizontal display and a second driving circuit (not shown) that drives the vertical display. For example, the first driving circuit is disposed above the display unit 130. The first driving circuit sets the vertical display position. For example, the second driving circuit is disposed beside the display unit 130. The second driving circuit sets the horizontal display position. By driving the first driving circuit and the second driving circuit, pixels forming an image can be displayed at arbitrary coordinates of the display unit 130.

[0043] 〔Virtual Input Device〕 Next, the detailed configuration of the virtual input device 10 will be described with reference to the drawings. FIG. 7 is a block diagram showing an example of the configuration of the virtual input device 10. The virtual input device 10 includes a user information acquisition unit 11, a video acquisition unit 12, a display information output unit 13, a video analysis unit 15, an operation recognition unit 16, and a display information generation unit 17. The video acquisition unit 12 and the display information output unit 13 constitute the input / output unit 14.

[0044] The user information acquisition unit 11 (user information acquisition means) acquires user information regarding the user wearing the virtual reality display device 100. The user information acquisition unit 11 outputs the acquired user information to the display information generation unit 17. The user information includes information such as the user's personal information, attributes, skills, and affiliation. For example, the user information is stored in a storage unit (not shown). For example, the user information acquisition unit 11 acquires preset user information from the storage unit according to the user using the virtual reality display device 100. For example, the user information acquisition unit 11 selects any one of a plurality of pieces of user information stored in the storage unit. For example, the user information acquisition unit 11 acquires user information regarding the user using the virtual input device 10 from an authentication device (not shown) connected to the virtual input device 10.

[0045] The video acquisition unit 12 (video acquisition means) acquires image data from the virtual reality display device 100. The image data is image data captured by the cameras 120 (camera 120L, camera 120R) of the virtual reality display device 100. The video acquisition unit 12 outputs the acquired image data to the video analysis unit 15. The video acquisition unit 12 may be configured to acquire video data composed of a plurality of temporally continuous image data (frames). In such a configuration, the video acquisition unit 12 outputs the acquired video data to the video analysis unit 15.

[0046] The display information output unit 13 (display information acquisition means) outputs the display information generated by the display information generation unit 17 to the virtual reality display device 100. The display information includes an image displayed on the display unit 130 of the virtual reality display device 100 and the coordinates at which the image is displayed.

[0047] The video acquisition unit 12 and the display information output unit 13 constitute the input / output unit 14 (input / output means). For example, the input / output unit 14 is connected to the virtual reality display device 100 by wireless communication such as Bluetooth (registered trademark), WiFi (registered trademark), or wireless LAN. The input / output unit 14 may be connected to the virtual reality display device 100 by a wireless method other than Bluetooth (registered trademark), WiFi (registered trademark), or wireless LAN. Also, the input / output unit 14 may be connected to the virtual reality display device 100 via a wired cable as long as it does not affect the user's workability.

[0048] The video analysis unit 15 (video analysis means) acquires video data from the video acquisition unit 12. The video analysis unit 15 analyzes the acquired video data to detect the operation terminal 170. The video analysis unit 15 detects the position of at least one landmark attached to the operation terminal 170. The video analysis unit 15 may set a landmark on the operation terminal 170 according to the features detected from the operation terminal 170. The video analysis unit 15 outputs the position of the detected at least one landmark to the display information generation unit 17.

[0049] Also, the video analysis unit 15 detects the position of the user's hand or finger operating the operation terminal 170. The video analysis unit 15 outputs the position of the user's hand or finger and the position of at least one landmark attached to the operation terminal 170 to the operation recognition unit 16. For example, the landmark is physically attached to the surface of the operation terminal 170. The landmark may be specified based on the outer shape or features of the operation terminal 170.

[0050] The operation recognition unit 16 (image recognition means) acquires the position of the user's hand or finger and the position of the mark attached to the operation terminal 170 from the video analysis unit 15. Based on the position of the mark attached to the operation terminal 170, the operation recognition unit 16 identifies the position of the keys of the operation terminal 170. For example, if the relative position of the keys of the operation terminal 170 with respect to the position of the mark attached to the operation terminal 170 is stored in a storage unit (not shown), the position of the keys of the operation terminal 170 can be identified based on the position of the mark. Further, the operation recognition unit 16 associates the position of the user's hand or finger with the position of the keys of the operation terminal 170 to identify the key input by the user. The operation recognition unit 16 outputs information regarding the identified key to the display information generation unit 17.

[0051] The display information generation unit 17 (display information generation means) acquires user information from the user information acquisition unit 11. Further, the display information generation unit 17 acquires the position of at least one mark attached to the operation terminal 170 from the video analysis unit 15. The display information generation unit 17 generates an image of the virtual terminal VT according to the user information in accordance with the acquired position of at least one mark.

[0052] Also, the display information generation unit 17 acquires information regarding the identified key from the operation recognition unit 16. The display information generation unit 17 generates an image of the virtual monitor VM to be displayed on the display unit 130 of the virtual reality display device 100 according to the identified key. The display information generation unit 17 outputs the generated image (display information) to the display information output unit 13.

[0053] (Operation) Next, an example of the operation of the virtual input system 1 according to the present embodiment will be described with reference to the drawings. In the following, an example of the operation of the virtual input device 10 included in the virtual input system 1 will be described.

[0054] FIG. 8 is a flowchart for explaining an example of the operation of the virtual input device 10. The flowchart of FIG. 8 relates to the stage of displaying the virtual terminal VT according to the user information. In the process along the flowchart of FIG. 8, the virtual input device 10 is the operating entity.

[0055] In FIG. 8, first, the virtual input device 10 acquires user information of a user who uses the virtual reality display device 100 (step S111).

[0056] Next, the virtual input device 10 acquires video data (image data) captured by the camera 120 of the virtual reality display device 100 (step S112).

[0057] Next, the virtual input device 10 analyzes the acquired video data (image data) to detect the operation terminal 170 (step S113).

[0058] When the operation terminal 170 is detected from the video data (image data) (Yes in step S113), the virtual input device 10 specifies the position of the operation terminal 170 in the real world (step S114). If the operation terminal 170 is not detected (No in step S113), the virtual input device 10 acquires new video data (image data) (step S112).

[0059] Next to step S114, the virtual input device 10 generates display information including the virtual terminal VT according to the user information in accordance with the position of the operation terminal 170 in the real world (step S115).

[0060] Next, the virtual input device 10 outputs the generated display information to the virtual reality display device 100 (step S116). On the display unit 130 of the virtual reality display device 100, a video (image) corresponding to the display information output from the virtual input device 10 is displayed. Next to step S116, the process proceeds to the flowchart process of FIG. 9.

[0061] FIG. 9 is a flowchart for explaining an example of the operation of the virtual input device 10. The flowchart of FIG. 9 relates to the stage of changing the display according to the user's operation on the virtual terminal VT displayed according to the user information. In the process along the flowchart of FIG. 9, the virtual input device 10 is the operating entity.

[0062] In FIG. 9, first, the virtual input device 10 acquires video data (image data) captured by the camera 120 of the virtual reality display device 100 (step S121).

[0063] Next, the virtual input device 10 analyzes the acquired video data (image data) to detect a change in the position of the operation terminal 170 (step S122).

[0064] When a change in the position of the operation terminal 170 is detected from the video data (image data) (Yes in step S122), the virtual input device 10 changes the display information including the virtual terminal VT in accordance with the change in the position of the operation terminal 170 in the real world (step S123). If no change in the position of the operation terminal 170 is detected (No in step S122), the process proceeds to step S124.

[0065] Next to step S123, or when No in step S122, the virtual input device 10 recognizes the user's operation on the operation terminal 170 based on the video data (image data) (step S124).

[0066] When the user's operation on the operation terminal 170 is recognized (Yes in step S124), the virtual input device 10 generates display information corresponding to the user's operation (step S125). If the user's operation on the operation terminal 170 is not recognized (No in step S124), the process proceeds to step S127.

[0067] Next to step S125, the virtual input device 10 generates display information corresponding to the user's operation (step S125).

[0068] Next, the virtual input device 10 outputs the generated display information to the virtual reality display device 100 (step S126). On the display unit 130 of the virtual reality display device 100, a video (image) corresponding to the display information output from the virtual input device 10 is displayed.

[0069] Next to step S126, or if the answer is No in step S124, return to step S121. If the series of processes is to be terminated (No in step S127), the processes along the flowchart of FIG. 9 are terminated.

[0070] As described above, the virtual input system of this embodiment includes a virtual reality display device, a virtual input device, and an operation terminal. The virtual reality display device has a camera and a display unit. The virtual reality display device outputs the image data captured by the camera to the virtual input device. The virtual reality display device acquires the display information output from the virtual input device. The virtual reality display device causes the display unit to display an image corresponding to the acquired display information. The virtual input device includes a user information acquisition unit, a video acquisition unit, a video analysis unit, an operation recognition unit, a display information generation unit, and a display information output unit. The video acquisition unit and the display information output unit constitute an input / output unit. The user information acquisition unit acquires the user information of the user who uses the virtual reality display device. The video acquisition unit acquires the image data captured by the camera of the virtual reality display device. The video analysis unit analyzes the image data to identify the position where the virtual terminal is to be displayed. The display information generation unit generates display information including a virtual terminal according to the user information of the user wearing the virtual reality display device. The display information output unit outputs the display information to be displayed on the display unit of the virtual reality display device to the virtual reality display device. The operation terminal is arranged in the real world. The operation terminal is operated by the user wearing the virtual input device.

[0071] The virtual input device included in the virtual input system of this embodiment identifies the position where the virtual terminal is to be displayed based on the real-world image data captured by the camera of the virtual reality display device. The virtual input device of this embodiment outputs display information for displaying the virtual terminal at the identified position to the virtual reality display device. On the display unit of the virtual reality display device worn by the user, a virtual terminal corresponding to the user information of the user is displayed at a position where the real-world situation is reflected. The user wearing the virtual reality display device visually recognizes the virtual terminal displayed at the position where the real-world situation is reflected. The user can operate the virtual terminal customized for the user in a natural way as if the operation terminal is placed in the real world without any discomfort. Therefore, according to the virtual input system of this embodiment, continuous operation on the virtual terminal displayed in the virtual space can be realized.

[0072] For example, the virtual input system of this embodiment can be applied as an alternative to the dedicated terminal used in the fire command center. The dedicated terminals used in the fire command center have different frequently used keys for each organization. Therefore, even for the same dedicated terminal, the dedicated terminal is customized for each organization in order to make the frequently used keys prominent. By using the virtual input system of this embodiment, the virtual terminal can be customized for each organization by changing the display on the display unit.

[0073] The virtual input system of this embodiment can also be applied to uses other than the fire command center. For example, the virtual input system of this embodiment can be applied to the communication command center of a police station. For example, the virtual input system of this embodiment can be applied to a call center that conducts communication through calls. For example, the virtual input system of this embodiment can be applied to a POS (Point Of Sale) system. The virtual input system of this embodiment can be applied to any system as long as it uses a virtual reality display device such as an HMD.

[0074] The virtual input system of this embodiment can also be applied to usage scenarios such as augmented reality (AR) and mixed reality (MR). In usage scenarios such as AR and MR, a transmissive virtual reality display device is used. By using a transmissive virtual reality display device, an interactive operation with the real world can be realized while viewing the real world through the display unit. In that case, without displaying the virtual terminal, it is only necessary to superimpose on the keys of the operation terminal in the real world and highlight those keys.

[0075] Also, the virtual input system of this embodiment may be applied to a usage scenario of substitutional reality (SR). In the usage scenario of SR, the virtual input system of this embodiment realizes an interactive operation not with the real-time real world but with a past real world or an imaginary virtual world. For example, an interactive operation with a past real world is suitable for training to deal with an actually occurred fire or accident. For example, an interactive operation with an imaginary virtual world is suitable for games and the like.

[0076] In one aspect of this embodiment, the display information generation unit generates display information in which a key to be operated by the user among a plurality of keys included in the virtual terminal is highlighted according to the user information. Therefore, according to this aspect, a user wearing a virtual reality display device can intuitively grasp the key to be operated without overlooking it by visually recognizing the highlighted key.

[0077] In one aspect of this embodiment, the video analysis unit detects an operation terminal arranged in the real world from image data. The video analysis unit specifies the position of the operation terminal on the display unit of the virtual reality display device. The display information generation unit generates display information in which a virtual terminal is displayed at the specified position of the operation terminal. In this aspect, the virtual terminal is displayed on the display unit of the virtual reality display device so as to be superimposed on the operation terminal arranged in the real world. According to this aspect, the operation terminal arranged in the real world can be operated while visually recognizing the virtual terminal displayed on the display unit of the virtual reality display device. According to this aspect, since the virtual terminal can be operated while obtaining the tactile sensation of the operation terminal arranged in the real world, the virtual terminal can be continuously operated with a natural operation feeling.

[0078] The virtual input device according to one aspect of this embodiment includes an operation recognition unit that analyzes image data to recognize a user's operation. The display information generation unit generates display information according to the recognized user's operation. According to this aspect, by recognizing the user's operation on the operation terminal arranged in the real world, the user's operation input for working while visually recognizing the virtual terminal superimposed on the operation terminal can be realized. The user can operate the virtual terminal superimposed on the operation terminal arranged in the real world without discomfort while touching the operation terminal arranged in the real world.

[0079] (Second Embodiment) Next, the virtual input device according to the second embodiment will be described with reference to the drawings. In the virtual input device of this embodiment, the display of the virtual terminal is set according to the content of a list registered in advance. The virtual input device of this embodiment is incorporated into the virtual input system of the first embodiment instead of the virtual input device according to the first embodiment.

[0080] (Configuration) FIG. 10 is a block diagram showing an example of the configuration of the virtual input device 20. The virtual input device 20 includes a user information acquisition unit 21, a video acquisition unit 22, a display information output unit 23, a video analysis unit 25, an operation recognition unit 26, a display information generation unit 27, and a related information acquisition unit 28. The video acquisition unit 22 and the display information output unit 23 constitute an input / output unit 24. A related list L corresponding to the user information is stored in the virtual input device 20.

[0081] The user information acquisition unit 21 (user information acquisition means) has the same configuration as the user information acquisition unit 11 in the first embodiment. The user information acquisition unit 21 acquires user information regarding the user wearing the virtual reality display device.

[0082] The video acquisition unit 22 (video acquisition means) has the same configuration as the video acquisition unit 12 in the first embodiment. The video acquisition unit 22 acquires image data from the virtual reality display device. The image data is the image data captured by the camera of the virtual reality display device. The video acquisition unit 22 outputs the acquired image data to the video analysis unit 25. The video acquisition unit 22 may be configured to acquire video data composed of a plurality of temporally continuous image data (frames). When configured in this way, the video acquisition unit 22 outputs the acquired video data to the video analysis unit 25.

[0083] The display information output unit 23 (display information output means) has the same configuration as the display information output unit 13 in the first embodiment. The display information output unit 23 outputs the display information generated by the display information generation unit 27 to the virtual reality display device. The display information includes an image to be displayed on the display unit of the virtual reality display device and the coordinates at which the image is to be displayed.

[0084] Similar to the first embodiment, the video acquisition unit 22 and the display information output unit 23 constitute an input / output unit 24 (input / output means). For example, the input / output unit 24 is connected to the virtual reality display device by wireless communication such as Bluetooth (registered trademark), WiFi (registered trademark), or wireless LAN. The input / output unit 24 may be connected to the virtual reality display device by a wireless method other than Bluetooth (registered trademark), WiFi (registered trademark), or wireless LAN. Also, the input / output unit 24 may be connected to the virtual reality display device via a wired cable as long as it does not affect the user's workability.

[0085] The image analysis unit 25 (image analysis means) has the same configuration as the image analysis unit 15 in the first embodiment. The image analysis unit 25 acquires video data from the video acquisition unit 22. The image analysis unit 25 analyzes the acquired video data to detect the operation terminal. The image analysis unit 25 detects the position of at least one landmark attached to the operation terminal. The image analysis unit 25 outputs the position of the at least one detected landmark to the display information generation unit 27.

[0086] In addition, the image analysis unit 25 detects the position of the hand or finger of the user operating the operation terminal. The image analysis unit 25 may set a landmark on the operation terminal according to the features detected from the operation terminal. The image analysis unit 25 outputs the position of the user's hand or finger and the position of at least one landmark attached to the operation terminal to the operation recognition unit 26.

[0087] The operation recognition unit 26 (operation recognition means) acquires the position of the user's hand or finger and the position of the landmark attached to the operation terminal from the image analysis unit 25. The operation recognition unit 26 identifies the position of the key of the operation terminal based on the position of the landmark attached to the operation terminal. In addition, the operation recognition unit 26 associates the position of the user's hand or finger with the position of the key of the operation terminal to identify the key input by the user. The operation recognition unit 26 outputs information regarding the identified key to the display information generation unit 27.

[0088] The related information acquisition unit 28 (related information acquisition means) acquires user information of the user using the virtual reality display device from the user information acquisition unit 21. The related information acquisition unit 28 acquires related information corresponding to the user information from the related list L stored in a storage unit (not shown). The related list L is a list in which related information according to the intention of the system is summarized. For example, the related list L includes related information regarding the user's job. The related list L includes related information regarding the scenario scene of the user's job. In the present embodiment, the related information is information regarding the key that the user should operate according to the scenario scene. The related information acquisition unit 28 outputs the acquired related information to the display information generation unit 27.

[0089] The related information acquisition unit 28 may be configured to acquire related information from the outside. For example, the related information is information regarding a scenario acquired according to the user's job. For example, when the user is a staff member of a fire command center, the related information is information regarding scenarios such as fire-related and emergency-related scenarios.

[0090] The display information generation unit 27 (display information generation means) acquires user information from the user information acquisition unit 21. Further, the display information generation unit 27 acquires the position of at least one mark attached to the operation terminal from the video analysis unit 25. Furthermore, the display information generation unit 27 acquires related information from the related information acquisition unit 28. The display information generation unit 27 generates an image of the virtual terminal VT according to the user information and the related information in accordance with the position of the at least one mark acquired.

[0091] Also, the display information generation unit 27 acquires information regarding the specified key from the operation recognition unit 26. The display information generation unit 27 generates an image of the virtual monitor VM to be displayed on the display unit of the virtual reality display device according to the specified key. The display information generation unit 27 outputs the generated image (display information) to the display information output unit 23.

[0092] FIG. 11 shows an example in which a virtual terminal VT whose display state is changed according to the related information included in the related list L is displayed on the display unit 230 of the virtual reality display device. In the example of FIG. 11, virtual monitors VM1 and VM2 (virtual monitor VM) are displayed on the display unit 230. Also, in the example of FIG. 11, the virtual terminal VT is displayed in association with the position of the operation terminal arranged in the real world on the other side of the display unit 230. In the area R1 of the virtual terminal VT, a key whose display state is changed according to the scenario is displayed. The display state of the key in the area R1 is displayed in a prominent color or brightness compared to the display states of the keys arranged in other areas. In the area R2 of the virtual terminal VT, character information regarding the key is displayed in association with the key. A user wearing the virtual reality display device can pay attention to the key related to the user's job or the like by visually recognizing the key whose display state is changed.

[0093] For example, when the display of the virtual terminal VT is changed according to related information, the virtual reality display device may be configured to emit sound or vibration. If configured in this way, the user wearing the virtual reality display device can recognize that the display of the virtual terminal VT has been changed by the sound or vibration emitted from the virtual reality display device. When speakers are arranged on the left and right of the virtual reality display device, the volume of the left and right speakers can be changed according to the position where the display of the virtual terminal VT is changed, so as to notify which key around the virtual terminal VT has been changed. Even if the display of the virtual terminal VT has not been changed, the virtual reality display device may be configured to emit sound or vibration at the timing when the virtual terminal VT should be noticed. If configured in this way, the user's attention can be directed to the virtual terminal VT at an appropriate timing.

[0094] 〔Related List〕 Next, an example of the related list L stored in the virtual input device 20 will be described with reference to the drawings. The related list L given below is an example and does not limit the related list L stored in the virtual input device 20.

[0095] FIG. 12 is an example (related list L1) of the related list L stored in the virtual input device 20. The related list L1 stores information about keys whose display states are changed according to scenarios. In the example of the related list L1, the keys (K1, K2, K3) are associated with the scenario S1.

[0096] For example, the related information acquisition unit 28 selects the scenario S1 registered in the related list L1 according to the user information. The related information acquisition unit 28 outputs information about the keys (K1, K2, K3) whose display states are changed according to the acquired scenario S1 to the display information generation unit 27. The display information generation unit 27 acquires information about the keys (K1, K2, K3) whose display states are changed from the related information acquisition unit 28. The display information generation unit 27 changes the display states of the keys (K1, K2, K3) according to the information acquired from the related information acquisition unit 28.

[0097] FIG. 13 shows an example in which a virtual terminal VT whose display state has been changed according to an associated list L1 is displayed on a display unit 230 of a virtual reality display device. The virtual terminal VT is displayed on the display unit 230 in association with the position of an operation terminal arranged in the real world on the other side of the display unit 230. The display states of the keys K1, K2, and K3 are displayed in a prominent color or brightness compared to the display states of the other keys. A user wearing the virtual reality display device can pay attention to the keys related to the user's duties or the like by visually recognizing the keys whose display states have been changed.

[0098] For example, in a scenario scene where a selection of "Yes" or "No" is made, the keys for selecting "Yes" or "No" are highlighted. A user wearing the virtual reality display device can respond according to the scenario scene by selecting any one of the highlighted keys for selecting "Yes" or "No".

[0099] For example, in a scenario scene where vehicles are arranged in response to the occurrence of an accident, a fire, or the like, the keys for arranging vehicles are highlighted. In such a scenario scene, for example, the keys for selecting "ambulance" or "fire truck" are highlighted. A user wearing the virtual reality display device can arrange a vehicle according to the scenario scene by selecting at least any one of the highlighted keys for selecting "ambulance" or "fire truck". Also, in the scenario scene of an accident or a fire, a map of the area near the site where the fire or accident occurred, the situation of the fire station near the site, the video of the site, etc. may be displayed on the virtual monitor VM.

[0100] For example, in an emergency scenario scene, the key for arranging an ambulance is highlighted. A user wearing the virtual reality display device can arrange an ambulance by selecting the highlighted key for selecting "ambulance". Also, in the emergency scenario scene, the keys for selecting and inputting accompanying information such as the address of the destination of the ambulance, the attributes and status of the person to be transported, etc. may be highlighted.

[0101] FIG. 14 shows an example (associated list L2) of the associated list L stored in the virtual input device 20. In the associated list L2, the degree of interest is stored for each key of the virtual terminal VT.

[0102] FIG. 15 shows an example in which the virtual terminal VT whose display state has been changed according to the associated list L2 is displayed on the display unit 230 of the virtual reality display device. On the display unit 230, the virtual terminal VT is displayed in association with the position of the operation terminal arranged in the real world on the other side of the display unit 230. On the virtual terminal VT, keys whose display states have been changed according to the degree of interest are displayed. In the example of FIG. 15, the keys with higher degrees of interest are displayed larger. The user wearing the virtual reality display device can focus on the keys suitable for his / her own scenario or situation by visually recognizing the keys whose display states have been changed according to the degree of interest.

[0103] FIG. 16 shows an example (associated list L3) of the associated list L stored in the virtual input device 20. In the associated list L3, keys that should be ignored and should not be pressed by mistake are stored. For example, in a fire-related scenario scene, keys of a scenario scene that has little relation to the fire system (for example, the emergency system) are stored in the associated list L3 as keys to be ignored.

[0104] FIG. 17 shows an example in which the virtual terminal VT whose display state has been changed according to the associated list L3 is displayed on the display unit 230 of the virtual reality display device. On the display unit 230, the virtual terminal VT is displayed in association with the position of the operation terminal arranged in the real world on the other side of the display unit 230. On the virtual terminal VT, keys that may be ignored are not displayed, and only key K1, key K2, and key 3 are displayed. The user wearing the virtual reality display device can focus on the keys suitable for his / her own scenario or situation by visually recognizing the keys whose display states have been changed.

[0105] FIG. 18 is an example in which guidance regarding the next operation is displayed on the display unit 230 of the virtual reality display device in response to the selection of a specific key. On the display unit 230, a virtual terminal VT is displayed in association with the position of an operation terminal arranged in the real world on the other side of the display unit 230. In the example of FIG. 18, in response to the selection of key K0 of virtual terminal VT, guidance saying "Next, please press key K1" is displayed on virtual monitor VM1. Further, in area R1 of virtual terminal VT displayed on display unit 230, keys whose display states have been changed according to the scenario are displayed. The display state of the keys in area R1 is displayed in a conspicuous color or brightness as compared with the display states of the keys arranged in other areas. In particular, key K1, which is the next operation target, is displayed in an emphasized manner as compared with other keys. For example, the displays of virtual monitor VM1 and virtual terminal VT are changed according to the scenario of the related list. The user wearing the virtual reality display device can select key K1 to be operated next by visually recognizing the guidance displayed on virtual monitor VM1 and key K1 displayed in an emphasized manner on virtual terminal VT. The guidance regarding the next operation may be made not only by the display of virtual monitor VM but also by voice.

[0106] (Operation) Next, an example of the operation of the virtual input device 20 according to the present embodiment will be described with reference to the drawings. In the following, the stage of displaying the virtual terminal VT according to the user information and the related information will be described. The stage of changing the display according to the user's operation on the virtual terminal VT displayed according to the user information is the same as that in the first embodiment (FIG. 9), and thus the description thereof will be omitted.

[0107] FIG. 19 is a flowchart for explaining an example of the operation of the virtual input device 20. In the process according to the flowchart of FIG. 19, the virtual input device 20 is the main body of operation.

[0108] In FIG. 19, first, the virtual input device 20 acquires the user information of the user who uses the virtual reality display device (step S211).

[0109] Next, the virtual input device 20 acquires related information corresponding to the user information from the related list L stored in a storage unit (not shown) (step S212).

[0110] Next, the virtual input device 20 acquires video data (image data) captured by the camera of the virtual reality display device (step S213).

[0111] Next, the virtual input device 20 analyzes the acquired video data (image data) to detect the operation terminal (step S214).

[0112] When the operation terminal is detected from the video data (image data) (Yes in step S214), the virtual input device 20 specifies the position of the operation terminal in the real world (step S215). If the operation terminal is not detected (No in step S214), the virtual input device 20 acquires new video data (image data) (step S213).

[0113] Next to step S215, the virtual input device 20 generates display information including the virtual terminal VT according to the user information and the related information in accordance with the position of the operation terminal in the real world (step S216).

[0114] Next, the virtual input device 20 outputs the generated display information to the virtual reality display device (step S217). On the display unit 230 of the virtual reality display device, a video (image) corresponding to the display information output from the virtual input device 20 is displayed. After step S217, the process proceeds to a process according to the process (flowchart in FIG. 9) according to the first embodiment.

[0115] As described above, the virtual input device of the present embodiment includes a user information acquisition unit, a related information acquisition unit, a video acquisition unit, a video analysis unit, an operation recognition unit, a display information generation unit, and a display information output unit. The video acquisition unit and the display information output unit constitute an input / output unit. The user information acquisition unit acquires user information of a user who uses the virtual reality display device. The related information acquisition unit acquires related information corresponding to the user information. The video acquisition unit acquires image data captured by the camera of the virtual reality display device. The video analysis unit analyzes the image data to identify the position where the virtual terminal is to be displayed. The display information generation unit generates display information including a virtual terminal corresponding to the user information of the user wearing the virtual reality display device. Further, the display information generation unit highlights a key according to the related information. The display information output unit outputs the display information to be displayed on the display unit of the virtual reality display device to the virtual reality display device.

[0116] The virtual input device of the present embodiment identifies the position where the virtual terminal is to be displayed based on the real-world image data captured by the camera of the virtual reality display device. The virtual input device of the present embodiment outputs, to the virtual reality display device, display information in which the virtual terminal is displayed at the identified position. On the display unit of the virtual reality display device worn by the user, a virtual terminal corresponding to the user information of the user is displayed at a position where the real-world situation is reflected. In the virtual terminal, a key corresponding to the related information is highlighted. The user wearing the virtual reality display device visually recognizes the virtual terminal in which the key corresponding to the related information is highlighted. By visually recognizing the highlighted key, the user can recognize the key to be operated. Therefore, according to the virtual input device of the present embodiment, an intuitive operation on the virtual terminal displayed in the virtual space can be realized.

[0117] (Third Embodiment) Next, a virtual input device according to a third embodiment will be described with reference to the drawings. The virtual input device of this embodiment authenticates a user using an authentication device. The virtual input device of this embodiment is incorporated into the virtual input system of the first embodiment instead of the virtual input device according to the first embodiment. The virtual input device of this embodiment may be combined with the virtual input device according to the second embodiment.

[0118] (Configuration) FIG. 20 is a block diagram showing an example of the configuration of the virtual input device 30. The virtual input device 30 includes a user information acquisition unit 31, a video acquisition unit 32, a display information output unit 33, a video analysis unit 35, an operation recognition unit 36, and a display information generation unit 37. The video acquisition unit 32 and the display information output unit 33 constitute the input / output unit 34. The virtual input device 30 is connected to the authentication device 310.

[0119] The authentication device 310 is a device that authenticates a user wearing a virtual reality display device. The authentication device 310 outputs the authentication information of the authenticated user to the virtual input device 30. For example, the authentication device 310 outputs the identifier of the user as the authentication information of the authenticated user to the virtual input device 30. The authentication device 310 does not output the authentication information of the unauthenticated user to the virtual input device 30.

[0120] For example, the authentication device 310 accepts the input of the user's ID (Identifier) and password and authenticates the user. For example, the authentication device 310 is a biometric authentication device that authenticates the user using biometric information such as the user's fingerprint, palmprint, finger vein, iris, face, and voice. There is no limitation on the authentication method performed by the authentication device 310 as long as the user can be authenticated. The authentication device 310 may be provided separately from the virtual reality display device connected to the virtual input device 30, or may be provided in the virtual reality display device. For example, it may be configured to authenticate the user's face or iris by a camera built inside the virtual reality display device. For example, it may be configured to authenticate the user by ear authentication using a hearable device provided in the virtual reality display device.

[0121] The user information acquisition unit 31 (user information acquisition means) acquires the user's authentication information from the authentication device 310. The user information acquisition unit 31 authenticates the user according to the acquired user authentication information. When the user is authenticated, the user information acquisition unit 31 acquires the user information regarding the user. The user information acquisition unit 31 outputs the acquired user information to the display information generation unit 37. Also, when the user is not authenticated, the user information acquisition unit 31 outputs non-authentication information indicating that the user has not been authenticated to the display information generation unit 37.

[0122] The video acquisition unit 32 (video acquisition means) has the same configuration as the video acquisition unit 12 of the first embodiment. The video acquisition unit 32 acquires image data from the virtual reality display device. The image data is the image data captured by the camera of the virtual reality display device. The video acquisition unit 32 outputs the acquired image data to the video analysis unit 35. The video acquisition unit 32 may be configured to acquire video data composed of a plurality of temporally continuous image data (frames). When configured in such a way, the video acquisition unit 32 outputs the acquired video data to the video analysis unit 35.

[0123] The display information output unit 33 (display information output means) has the same configuration as the display information output unit 13 of the first embodiment. The display information output unit 33 outputs the display information generated by the display information generation unit 37 to the virtual reality display device. The display information includes the image displayed on the display unit of the virtual reality display device and the coordinates at which the image is displayed.

[0124] Similar to the first embodiment, the video acquisition unit 32 and the display information output unit 33 constitute the input / output unit 34. For example, the input / output unit 34 is connected to the virtual reality display device by wireless communication such as Bluetooth (registered trademark), WiFi (registered trademark), or wireless LAN. The input / output unit 34 may be connected to the virtual reality display device by a wireless method other than Bluetooth (registered trademark), WiFi (registered trademark), or wireless LAN. Also, the input / output unit 34 may be connected to the virtual reality display device via a wired cable as long as it does not affect the user's workability.

[0125] The video analysis unit 35 (video analysis means) has the same configuration as the video analysis unit 15 of the first embodiment. The video analysis unit 35 acquires video data from the video acquisition unit 32. The video analysis unit 35 analyzes the acquired video data to detect the operation terminal. The video analysis unit 35 detects the position of at least one landmark attached to the operation terminal. The video analysis unit 35 outputs the position of the at least one detected landmark to the display information generation unit 37.

[0126] Also, the video analysis unit 35 detects the position of the user's hand or finger operating the operation terminal. The video analysis unit 35 may set a landmark on the operation terminal according to the features detected from the operation terminal. The video analysis unit 35 outputs the position of the user's hand or finger and the position of at least one landmark attached to the operation terminal to the operation recognition unit 36.

[0127] The operation recognition unit 36 (operation recognition means) acquires the position of the user's hand or finger and the position of the landmark attached to the operation terminal from the video analysis unit 35. The operation recognition unit 36 specifies the position of the key of the operation terminal based on the position of the landmark attached to the operation terminal. Also, the operation recognition unit 36 associates the position of the user's hand or finger with the position of the key of the operation terminal to specify the key input by the user. The operation recognition unit 36 outputs information regarding the specified key to the display information generation unit 37.

[0128] The display information generation unit 37 (display information generation means) acquires user information from the user information acquisition unit 31. Further, the display information generation unit 37 acquires the position of at least one landmark attached to the operation terminal from the video analysis unit 35. The display information generation unit 37 generates an image of the virtual terminal VT according to the acquired position of at least one landmark, the user information, and the related information.

[0129] In addition, the display information generation unit 37 acquires information regarding the specified key from the operation recognition unit 36. The display information generation unit 37 generates an image of the virtual monitor VM to be displayed on the display unit of the virtual reality display device according to the specified key. The display information generation unit 37 outputs the generated image (display information) to the display information output unit 33.

[0130] Also, when the user information acquisition unit 31 acquires non-authentication information, the display information generation unit 37 generates display information indicating that authentication is impossible. The display information generation unit 37 outputs the generated display information to the display information output unit 33. When non-authentication information is acquired, the display information generation unit 37 may be configured not to generate display information.

[0131] (Operation) Next, an example of the operation of the virtual input device 30 according to the present embodiment will be described with reference to the drawings. In the following, the stage of displaying the virtual terminal VT according to the user information will be described. The stage of changing the display according to the user's operation on the virtual terminal VT displayed according to the user information is the same as that of the first embodiment (FIG. 9), and thus the description will be omitted.

[0132] FIG. 21 is a flowchart for explaining an example of the operation of the virtual input device 30. In the process along the flowchart of FIG. 21, the virtual input device 30 is the operating entity.

[0133] In FIG. 21, first, the virtual input device 30 acquires authentication information of the user wearing the virtual reality display device (step S311).

[0134] Next, the virtual input device 30 authenticates the user using the acquired authentication information (step S312). If the user is not authenticated (No in step S312), the virtual input device 30 generates display information indicating authentication failure (step S313). After step S313, the process proceeds to step S318.

[0135] If the user is authenticated (Yes in step S312), the virtual input device 30 acquires video data (image data) captured by the camera of the virtual reality display device (step S314).

[0136] Next, the virtual input device 30 analyzes the acquired video data (image data) to detect the operation terminal (step S315).

[0137] When the operation terminal is detected from the video data (image data) (Yes in step S315), the virtual input device 30 specifies the position of the operation terminal in the real world (step S316). If the operation terminal is not detected (No in step S315), the virtual input device 30 acquires new video data (image data) (step S314).

[0138] After step S316, the virtual input device 30 generates display information including a virtual terminal according to the user information in accordance with the position of the operation terminal in the real world (step S317).

[0139] After step S313 or step S317, the virtual input device 30 outputs the generated display information to the virtual reality display device (step S318). On the display unit of the virtual reality display device, a video (image) corresponding to the display information output from the virtual input device 30 is displayed. After step S318, the process proceeds to a process according to the process (flowchart in FIG. 9) according to the first embodiment.

[0140] As described above, the virtual input device of the present embodiment includes a user information acquisition unit, an image acquisition unit, an image analysis unit, an operation recognition unit, a display information generation unit, and a display information output unit. The image acquisition unit and the display information output unit constitute an input / output unit. The user information acquisition unit acquires user information of a user who uses the virtual reality display device. Further, the user information acquisition unit acquires authentication information of a user who uses the virtual reality display device. The image acquisition unit acquires image data captured by the camera of the virtual reality display device. The image analysis unit analyzes the image data to identify the position where the virtual terminal is to be displayed. The display information generation unit generates display information including a virtual terminal corresponding to the user information of the authenticated user for the virtual reality display device used by the authenticated user. The display information output unit outputs the display information to be displayed on the display unit of the virtual reality display device to the virtual reality display device.

[0141] The virtual input device of the present embodiment identifies the position where the virtual terminal is to be displayed based on the real-world image data captured by the camera of the virtual reality display device. The virtual input device of the present embodiment outputs display information in which the virtual terminal is displayed at the identified position for the virtual reality display device used by the authenticated user. On the display unit of the virtual reality display device worn by the authenticated user, a virtual terminal corresponding to the user information of the user is displayed at a position where the real-world situation is reflected. The virtual terminal displayed on the display unit of the virtual reality display device can be visually recognized only by the authenticated user. Therefore, according to the virtual input system of the present embodiment, the security of operations on the virtual terminal displayed in the virtual space can be improved.

[0142] (Fourth Embodiment) Next, an example of the configuration of the virtual input device according to the fourth embodiment will be described with reference to the drawings. The virtual input device of the present embodiment has a simplified configuration of the virtual input devices of the first to third embodiments.

[0143] FIG. 22 is a block diagram showing an example of the configuration of the virtual input device 40 according to the present embodiment. The virtual input device 40 of the present embodiment includes an input / output unit 44, a video analysis unit 45, and a display information generation unit 47. The input / output unit 44 (input / output means) acquires image data captured by a camera of the virtual reality display device. The input / output unit 44 outputs display information displayed on the display unit of the virtual reality display device to the virtual reality display device. The video analysis unit 45 (video analysis means) analyzes the image data to identify the position where the virtual terminal is to be displayed. The display information generation unit 47 (display information generation means) generates display information including a virtual terminal according to the user information of the user wearing the virtual reality display device.

[0144] The virtual input device of the present embodiment identifies the position where the virtual terminal is to be displayed based on the real-world image data captured by the camera of the virtual reality display device. The virtual input device of the present embodiment outputs display information in which the virtual terminal is displayed at the identified position to the virtual reality display device. On the display unit of the virtual reality display device worn by the user, a virtual terminal corresponding to the user information of the user is displayed at a position where the real-world situation is reflected. The user wearing the virtual reality display device visually recognizes the virtual terminal displayed at the position where the real-world situation is reflected. The user can operate the virtual terminal customized for the user in a natural way as if the operation terminal is placed in the real world without any discomfort. Therefore, according to the virtual input device of the present embodiment, continuous operation on the virtual terminal displayed in the virtual space can be realized.

[0145] (Hardware) Next, the hardware configuration for executing the control and processing according to each embodiment of the present disclosure will be described with reference to the drawings. Here, as an example of such a hardware configuration, the information processing device 90 (computer) in FIG. 23 is given. The information processing device 90 in FIG. 23 is an example of a configuration for executing the control and processing of each embodiment, and does not limit the scope of the present disclosure.

[0146] As shown in FIG. 23, the information processing apparatus 90 includes a processor 91, a main memory device 92, an auxiliary storage device 93, an input / output interface 95, and a communication interface 96. In FIG. 23, the interface is abbreviated as I / F (Interface). The processor 91, the main memory device 92, the auxiliary storage device 93, the input / output interface 95, and the communication interface 96 are connected to each other via a bus 98 so as to be capable of data communication. Further, the processor 91, the main memory device 92, the auxiliary storage device 93, and the input / output interface 95 are connected to a network such as the Internet or an intranet via the communication interface 96.

[0147] The processor 91 expands a program (instruction) stored in the auxiliary storage device 93 or the like into the main memory device 92. For example, the program is a software program for executing the control and processing of each embodiment. The processor 91 executes the program expanded in the main memory device 92. By executing the program, the processor 91 executes the control and processing according to each embodiment.

[0148] The main memory device 92 has an area where a program is expanded. In the main memory device 92, a program stored in the auxiliary storage device 93 or the like is expanded by the processor 91. The main memory device 92 is realized by a volatile memory such as a DRAM (Dynamic Random Access Memory), for example. Further, a non-volatile memory such as an MRAM (Magneto resistive Random Access Memory) may be configured / added as the main memory device 92.

[0149] The auxiliary storage device 93 stores various data such as programs. The auxiliary storage device 93 is realized by a local disk such as a hard disk or a flash memory. Note that it is also possible to configure to store various data in the main memory device 92 and omit the auxiliary storage device 93.

[0150] The input / output interface 95 is an interface for connecting the information processing apparatus 90 and peripheral devices based on standards and specifications. The communication interface 96 is an interface for connecting to external systems and devices through networks such as the Internet and intranet based on standards and specifications. As interfaces for connecting to external devices, the input / output interface 95 and the communication interface 96 may be shared.

[0151] Input devices such as a keyboard, a mouse, and a touch panel may be connected to the information processing apparatus 90 as necessary. Those input devices are used for inputting information and settings. When a touch panel is used as the input device, a screen having the function of the touch panel becomes the interface. The processor 91 and the input devices are connected via the input / output interface 95.

[0152] The information processing apparatus 90 may be provided with a display device for displaying information. When the display device is provided, the information processing apparatus 90 is provided with a display control device (not shown) for controlling the display of the display device. The information processing apparatus 90 and the display device are connected via the input / output interface 95.

[0153] The information processing apparatus 90 may be provided with a drive device. The drive device mediates the reading of data and programs stored in the recording medium and the writing of the processing results of the information processing apparatus 90 to the recording medium between the processor 91 and the recording medium (program recording medium). The information processing apparatus 90 and the drive device are connected via the input / output interface 95.

[0154] The above is an example of the hardware configuration for enabling the control and processing according to each embodiment of the present disclosure. The hardware configuration of FIG. 23 is an example of the hardware configuration for executing the control and processing according to each embodiment and does not limit the scope of the present disclosure. Programs for causing a computer to execute the control and processing according to each embodiment are also included in the scope of the present disclosure.

[0155] A program recording medium storing a program according to each embodiment is also included in the scope of the present disclosure. The recording medium can be realized by, for example, an optical recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc). The recording medium may be realized by a semiconductor recording medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) card. Further, the recording medium may be realized by a magnetic recording medium such as a flexible disk or other recording media. When a program executed by a processor is recorded on the recording medium, the recording medium corresponds to a program recording medium.

[0156] The components of each embodiment may be arbitrarily combined. The components of each embodiment may be realized by software. The components of each embodiment may be realized by a circuit.

[0157] Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

Description of Reference Numerals

[0158] 1 Virtual input system 10, 20, 30, 40 Virtual input devices 11, 21, 31 User information acquisition unit 12, 22, 32 Video acquisition unit 13, 23, 33 Display information output unit 14, 24, 34, 44 Input / output unit 15, 25, 35, 45 Video analysis unit 16, 26, 36 Operation recognition unit 17, 27, 37, 47 Display information generation unit 28 Related information acquisition unit 100 Virtual reality display device 110 Control device 111 Communication unit 113 indicates the controller 115 Memory 117 Power supply 120R, 120L Cameras 130, 230 Display units 140 Cover 150 Driver 170 Operation terminal 310 Authentication device

Claims

1. Input / output means for acquiring image data captured by a camera of a virtual reality display device and outputting display information to be displayed on a display unit of the virtual reality display device to the virtual reality display device; Video analysis means for analyzing the image data to identify a position for displaying a virtual terminal; Display information generation means for generating the display information including the virtual terminal according to user information of a user wearing the virtual reality display device, comprising: The display information generation means: A virtual input device that generates the display information in which a key to be operated by the user among a plurality of keys included in the virtual terminal is highlighted according to the user information.

2. The video analysis means: Detect an operation terminal arranged in the real world from the image data; Identify a position of the operation terminal on the display unit of the virtual reality display device; The display information generation means: The virtual input device according to claim 1, which generates the display information in which the virtual terminal is displayed at the identified position of the operation terminal.

3. Comprising operation recognition means for analyzing the image data to recognize an operation of the user; The display information generation means: The virtual input device according to claim 2, which generates the display information according to the recognized operation of the user.

4. Comprising related information acquisition means for acquiring related information according to the user information; The display information generation means: The virtual input device according to claim 2, which generates the display information in which a key according to the related information is highlighted.

5. Comprising user information acquisition means for acquiring authentication information of the user using the virtual reality display device and authenticating the user based on the acquired authentication information; The display information generation means: The virtual input device according to claim 1, which generates the display information for the virtual reality display device used by the authenticated user.

6. The virtual input device according to any one of claims 1 to 5; and A virtual reality display device having a camera and a display unit, outputting image data captured by the camera to the virtual input device, acquiring display information output from the virtual input device, and displaying an image according to the acquired display information on the display unit, comprising a virtual input system.

7. Comprising an operation terminal arranged in the real world and operated by a user wearing the virtual input device; The virtual input device: Generates the display information in which a virtual terminal is displayed at the position of the operation terminal. Output the generated display information to the virtual reality display device, The virtual reality display device, The virtual input system according to claim 6, which causes the display unit to display an image according to the display information.

8. A computer, Obtain image data captured by a camera of a virtual reality display device, Analyze the image data to identify a position for displaying a virtual terminal, Generate display information including the virtual terminal according to user information of a user wearing the virtual reality display device, Output the generated display information to the virtual reality display device, In the generation of the display information, A virtual input method for generating the display information in which a key to be operated by the user among a plurality of keys included in the virtual terminal is highlighted according to the user information.

9. A process of obtaining image data captured by a camera of a virtual reality display device, A process of analyzing the image data to identify a position for displaying a virtual terminal, A process of generating display information including the virtual terminal according to user information of a user wearing the virtual reality display device, A process of outputting the generated display information to the virtual reality display device, In the process of generating the display information, A program that causes a computer to execute a process of generating the display information in which a key to be operated by the user among a plurality of keys included in the virtual terminal is highlighted according to the user information.

Images