Head-mounted display and information processing terminal

Abstract

A head-mounted display according to the present invention includes a display, an image processing unit that generates a generated image to be displayed on the display, a connection unit that performs connection to an information processing terminal, and a control unit that controls input processing for performing text input. Upon starting an HMD text input mode for performing text input by the head-mounted display, the control unit transmits, to the information processing terminal, an instruction for enabling an information processing terminal text input mode for performing text input by the information processing terminal, stops the HMD text input mode upon receiving information that startup of the information processing terminal text input mode is completed from the information processing terminal, and receives input information in the information processing terminal text input mode from the information processing terminal. The image processing unit generates a superimposed image in which the received input information is superimposed on the generated image, and displays the superimposed image on the display.

Description

Head-mounted display and information processing terminal

[0001] The present invention relates to character input when using a head mounted display (hereinafter referred to as HMD).

[0002] Wearable devices are a general term for devices worn on the human body. They come in a variety of forms, including wristbands, watches, and glasses. Because they are so convenient, they are used in a variety of applications, including games, education, tourism, and system maintenance.

[0003] HMDs, a type of wearable device, are becoming more common and are beginning to spread, with applications that utilize virtual reality (VR) and augmented reality (AR).

[0004] HMDs are devices worn on the head, and display virtual reality (VR) worlds or augmented reality (AR) worlds on the HMD display, allowing the various applications mentioned above to be used.

[0005] When using an HMD, the method of character input has become one of the issues, and various techniques have been proposed.

[0006] Input methods include, for example, using the HMD camera to recognize the movement of a finger as characters and inputting the characters, displaying a keyboard on the VR or AR screen and inputting using a controller with stick buttons or the like, and installing an input device itself in the HMD and inputting the characters.

[0007] Patent Document 1 discloses that a mobile phone and an HMD can be connected via a short-range wireless communication network, and information on the network can be viewed by inputting text into the browser screen of the HMD using key operations on the mobile phone.

[0008] JP 2003-087363

[0009] Depending on the situation, character input using an HMD such as that described in Patent Document 1 can be inconvenient. When attempting to operate the HMD in conjunction with key operations on a smartphone, the HMD operator has difficulty recognizing that the smartphone is linked to the HMD, and is unable to determine whether the smartphone is in a state where it can be used for input, making it difficult to input characters reliably.

[0010] The present invention aims to ensure that information input from a smartphone is reliably performed and provided to the HMD when operating the HMD.

[0011] In order to solve the above-mentioned problems, the present invention provides a head-mounted display comprising a display, an image processing unit that generates a generated image to be displayed on the display, a connection unit that connects to an information processing terminal, and a control unit that controls input processing for character input, wherein when the control unit activates an HMD character input mode for character input using the head-mounted display, it sends to the information processing terminal an instruction to enable an information processing terminal character input mode for character input using the information processing terminal, and when it receives information from the information processing terminal that the activation of the information processing terminal character input mode has been completed, it stops the HMD character input mode and receives input information from the information processing terminal using the information processing terminal character input mode, and the image processing unit generates a superimposed image that superimposes the received input information on the generated image and displays the superimposed image on the display.

[0012] According to the present invention, when operating an HMD, information can be reliably input from a smartphone and provided to the HMD.

[0013] FIG. 1 is a schematic diagram of a configuration according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of a hardware configuration of an HMD according to the present invention. FIG. 2 is a diagram illustrating an example of a hardware configuration of a smartphone according to the present invention. FIG. 3 is a diagram illustrating a character input screen of a smartphone according to an embodiment of the present invention. FIG. 4 is a diagram illustrating a character input method for a smartphone according to an embodiment of the present invention. FIG. 5 is a diagram illustrating an image of an HMD according to an embodiment of the present invention. FIG. 6 is a diagram illustrating character input by an HMD according to an embodiment of the present invention. FIG. 7 is a diagram illustrating character input from a smartphone to an HMD according to an embodiment of the present invention. FIG. 8 is a diagram illustrating another character input screen of a smartphone according to an embodiment of the present invention. FIG. 9 is a diagram illustrating another character input method for a smartphone according to an embodiment of the present invention. FIG. 10 is a diagram illustrating another image of an HMD according to an embodiment of the present invention. FIG. 11 is a diagram illustrating another character input by an HMD according to an embodiment of the present invention. FIG. 12 is a diagram illustrating character input from another smartphone to an HMD according to an embodiment of the present invention. FIG. 13 is a flowchart illustrating an example of operation of Example 1 of the present invention. FIG. 14 is a flowchart illustrating an example of operation of Example 2 of the present invention. FIG. 15 is a diagram illustrating an example of operation of character input of the present invention. FIG. 16 is a diagram illustrating an example of character code of the present invention. FIG. 17 is a flowchart illustrating operation of Example 3 of the present invention. FIG. 18 is a flowchart illustrating operation of Example 4 of the present invention. FIG. 19 is a schematic diagram of a configuration according to an embodiment of the present invention. FIG. 11 is a diagram illustrating an image of an HMD according to Example 5 of the present invention. FIG. 12 is a diagram illustrating character input to an HMD according to Example 5 of the present invention. FIG. 13 is a

[0014] 1 is a schematic diagram of a configuration according to an embodiment of the present invention. A VR-type HMD 1 is connected to a PC 100 via a network 99, and operates by receiving information and commands from a VR application on the PC 100. A smartphone 2 is connected to the HMD 1 via communication such as near-field wireless communication. The smartphone 2 is an information processing terminal, and the information processing terminal is an externally portable information terminal device including a mobile device such as a smartphone, tablet, laptop PC, or mobile phone.

[0015] The operator inputs characters on the smartphone 2 while viewing the image of the smartphone 2 displayed on the display of the HMD 1, and the input characters are transmitted to the PC 100 as input characters for the HMD 1.

[0016] The HMD 1 operates using a VR application on the PC 100, but may also operate using a VR built-in application built into the HMD 1. Alternatively, the HMD 1 may operate using a VR application on a server or the like via a server or the like connected to the network 99.

[0017] 2 is an example of a hardware configuration diagram of the HMD 1. The HMD 1 is composed of an image processing unit 102, a control circuit 103, a data bus 104, a sensor unit 105, a communication processing unit 106, an audio processing unit 107, and a battery 108.

[0018] The control circuit 103 includes a main processor 110, a RAM (random access memory) 111, a ROM (read only memory) 112, a flash memory 113, and an input processing unit 114. The ROM 112 stores an operation program for the HMD 1.

[0019] The RAM 111 is a memory for performing program processing. The flash memory 113 stores applications for the HMD 1, application data, various setting information for the HMD 1, etc. The input processing 114 is a process for inputting characters, such as a process for inputting an HMD character input mode in which a camera of the HMD recognizes a moving finger as a character and obtains the input character as a character code. In this way, the control circuit 103 is a control unit that controls each process and operation.

[0020] The sensor unit 105 includes a GPS receiver 150, a geomagnetic sensor 151, a distance sensor 152, an acceleration sensor 153, a gyro sensor 154, a temperature sensor 155, and a wearing sensor 156. These sensors detect the operation of the HMD 1 (rotation, impact, movement, etc.), measure the position of the HMD 1, the distance to an object photographed by the camera 120, temperature, etc., and detect whether the HMD is being worn.

[0021] The image processing unit 102 includes a camera 120, an image processing unit 122, a display 123, etc. The camera 120 can capture still images, videos, etc. The image processing unit 122 processes the captured images and performs image generation processing to convert application images into images to be displayed on the display 123 using the main processor 110 or another processor.

[0022] The audio processing unit 107 includes a microphone 170, a codec 171, and a speaker 172. The microphone 170 receives audio input, and the codec 171 processes the audio in the main processor 110 or another processor.

[0023] The communication processing unit 106 is composed of a wireless LAN interface 160 and a short-range communication interface 161, and performs communication processing with the main processor 110 or another processor. The short-range communication interface 161 is a connection unit for wireless connection, and establishes a short-range wireless connection with the smartphone 2, transmits instructions such as activation of a smartphone character input mode for inputting characters to the smartphone 2, and receives input information from the smartphone 2.

[0024] FIG. 3 is an example of a hardware configuration diagram of the smartphone 2.

[0025] The smartphone 2 comprises a control circuit 201, an image processing unit 202, a data bus 204, a sensor unit 205, a communication processing unit 206, an audio processing unit 207, a battery 208, and an I / O interface 209 including volume and power ON / OFF buttons for the user to operate the smartphone.

[0026] The control circuit 201 includes a main processor 210, a RAM (random access memory) 211, a ROM (read-only memory) 212, a flash memory 213, and the like. The ROM 212 stores an operating program for the smartphone 2. The RAM 211 is a memory for performing program processing, such as selecting a character input method and prioritizing the display of frequently used characters or characters that have been used when inputting characters. The flash memory 213 stores applications for the smartphone 2, application data, and various setting information for the smartphone 2. The control circuit 201 operates the smartphone 2 based on the programs stored in the ROM 212 and the flash memory 213, the output of the sensor unit 205, and information obtained through the communication processing unit 206.

[0027] The sensor unit 205 includes a GPS receiver 250, a geomagnetic sensor 251, a distance sensor 252, an acceleration sensor 253, a gyro sensor 254, a temperature sensor 255, and a screen proximity sensor 262. These sensors detect the operation of the smartphone 2 (rotation, impact, movement, etc.), measure the position of the smartphone 2, the distance to an object photographed by the camera 220, temperature, etc., and detect the proximity of a face, finger, etc. to the screen. The screen proximity sensor 262 detects the proximity of a finger on the display 223. Therefore, the screen proximity sensor 262 can be used to detect the position of the finger on the display 223, i.e., the proximity position. This allows the position to be detected even when the finger is not in contact with the display 223. Therefore, pointing position information indicating the position of a living body such as a finger can be obtained even when the finger is in proximity but not in contact with the panel-shaped display 223 with a touch input function, which serves as a character input unit in the smartphone 2.

[0028] The image processing unit 202 includes a camera 220, image processing 222, and display 223. The camera 220 can capture still images, videos, and the like. The image processing 222 processes the captured images and processes application images to display them on the display 223. The touch panel 224 is a panel that allows an operator to input using a finger or the like, and functions integrally with the display 223 that displays images, etc., although each may be provided separately.

[0029] The audio processing unit 207 includes a microphone 270, a codec 271, and a speaker 272. The microphone 270 performs audio communication and audio input, and the codec 271 processes audio.

[0030] The communication processing unit 206 includes a wireless LAN interface 260, a short-range communication interface 261, and a mobile communication interface 263. The short-range communication interface 261 is a wireless connection unit that establishes a short-range wireless connection with the HMD 1, and receives instructions from the HMD 1, such as activation of a smartphone character input mode, and transmits input information to the HMD 1.

[0031] The smartphone character input mode for inputting characters on the smartphone 2 of Fig. 1 will now be described. Fig. 4A is an example of a screen displaying the smartphone character input mode for inputting characters on the display 223 of the smartphone 2. Specifically, this is an example of a Japanese character input screen called flick input, which is used on smartphones 2 in Japan. The display 223 of Fig. 4A displays a keyboard 21 for inputting characters and a display area 20 for displaying input characters.

[0032] 4B shows a state where a finger 22, which selects the next character to be input on the keyboard 21 (on the flick input screen), touches "a" and immediately before swiping to "u." Then, the finger 22 swipes to the next character "u" and the character is input when the finger 22 is released.

[0033] The display area 20 displays the characters that have already been input (here, the characters "Ohayo"), and displays a cursor waiting for the next character to be input. At this time, the character input by the previous input operation on the keyboard 21 is displayed in place of the cursor.

[0034] When the finger 22 touches the character "a" on the keyboard 21 of the display 223, the character "a" is displayed in reverse.

[0035] 5A to 5C illustrate examples of screens on which characters are displayed on the display 123 of the HMD 1. FIG.

[0036] Fig. 5A is an example of a VR image displayed as a stereoscopic image on the display 123 of the HMD 1. Fig. 5B is a screen in HMD character input mode when character input is required during operation of the VR-type HMD 1, for example, when sending a greeting message to another operator in the VR space. Fig. 5B shows a character input area 10 displayed on the VR image, which is the generated image displayed in Fig. 5A.

[0037] FIG. 5C shows a superimposed image in a superimposed input mode in which the keyboard 11 is superimposed on the VR image on the display 123 using information from the smartphone 2 on the screen in the HMD character input mode of FIG. 5B.

[0038] The character input area 10 here is displayed in the same position as in the HMD character input mode of Fig. 5B and the superimposed input mode of Fig. 5C, but it may be displayed in a different position. As shown in Fig. 5C, by displaying the character input area 10 and keyboard 11 on the display 123, part of the VR image is made invisible.

[0039] The operations of the HMD 1 and the smartphone 2 will be explained in detail below, but the screens of the HMD 1 and the smartphone 2 are linked as follows.

[0040] When the HMD 1 switches from the HMD character input mode of FIG. 5B to the superimposed input mode of FIG. 5C, the HMD 1 transmits to the smartphone 2 an instruction code for enabling the smartphone character input mode.

[0041] When the smartphone 2 receives the instruction code, it enables the smartphone character input mode of the smartphone 2 and displays a smartphone character input mode screen as shown in Fig. 4A . Then, as shown in Fig. 4B , when a finger 22 presses down on the keyboard 21 displayed on the smartphone 2 and then releases it, the input characters are displayed in the display area 20. Then, screen information of the keyboard 21 and character information of the display area 20 are transmitted to the HMD 1.

[0042] 5C, the HMD 1 displays a superimposed image on the VR image, with the received image of the keyboard 21 with the finger 22 in contact as the keyboard 11 and the input character as the character input area 10. When a new character is input on the smartphone 2 (for example, when the user swipes toward the character "u" and then releases the finger as shown in FIG. 4B), the newly input character is displayed in the character input area 10 on the display 123 of the VR-type HMD 1 shown in FIG. 5C.

[0043] By displaying the VR image superimposed in this way, the operator can focus more closely on the keyboard 11 on which the input characters and finger images are superimposed in the character input area 10. Also, a finger mark 12 of a finger image for selecting the next character is displayed on the keyboard 11, and the operator can display new characters inputted sequentially on the smartphone 2 in the character input area 10 of the HMD 1 while checking the position and movement of the finger mark 12 on the keyboard 11 of the HMD 1.

[0044] Note that the characters input here refer not only to simple characters but also to code information such as symbols, line feed keys, operation keys, etc. Furthermore, the screen information of the smartphone 2 may be transmitted to the HMD 1 as a cutout image that integrates the display area 20 including the input characters of the smartphone 2 and the keyboard 21, and the character input area 10 and keyboard 11 of the HMD 1 may be displayed integrally superimposed on the VR image.

[0045] 6A and 6B are examples of other screens in which the smartphone character input mode is displayed on the display 223 of the smartphone 2 in Fig. 1. Specifically, Fig. 6A is an example of an input screen using an English keyboard used for alphabet input. Also, Figs. 7A to 7C are diagrams illustrating other HMD images.

[0046] FIG. 6A shows a display 223 having a keyboard 21 for inputting characters and a display area 20 for displaying input characters.

[0047] 6B shows that the input character (here, the character "air") is displayed in the display area 20, and a cursor is displayed waiting for the next character to be input. The keyboard 21 shows that the finger is in contact with the keyboard 21 to select the next character to be input, "e." The next character, "e," is then input by removing the finger from the character, and the character is displayed in the display area 20.

[0048] Fig. 7A is an example of an English keyboard input screen. Fig. 7A is an example of a VR image displayed on the display of the HMD 1, similar to Fig. 5A. Fig. 7B is an example of a character input area 10, similar to Fig. 5B.

[0049] As shown in Fig. 6B, input characters in smartphone character input mode are displayed on the smartphone 2. A keyboard 21 including input characters and position information of a finger 22 is transmitted to the HMD 1 in the display area 20, and the keyboard 11 and character input area 10 are displayed superimposed on the VR image as shown in Fig. 7C.

[0050] When characters are input on the smartphone 2, the input characters are displayed in the character input area 10 on the display 123 of the VR-type HMD 1 as shown in FIG. 7C.

[0051] In FIG. 7C, the characters up to "air" have already been input in the character input area 10, and the cursor is displayed at the position where the next character should be input.

[0052] Furthermore, the next character "e" is displayed on the keyboard 11 with a finger mark 13 of a finger image superimposed thereon. Then, in conjunction with the separation of the finger on the smartphone 2, a new input character "e" is displayed in the character input area 10. In this way, the displayed screen can be changed from the flick input screen to the English keyboard input screen, and this selection information is set in advance and stored in the flash memory 113 or the like.

[0053] 8 is a flowchart showing an example of the operation of the HMD 1 and the smartphone 2 according to the first embodiment. When the power button of each of the HMD 1 and the smartphone 2 is pressed, the respective operations start (S1, S31). In FIG. 8, the processing on the HMD 1 side (S1-S12) is controlled and executed by the control circuit 103 of the HMD 1. The processing on the smartphone 2 side (S31-40) is controlled and executed by the control circuit 201 of the smartphone 2.

[0054] The HMD 1 and the smartphone 2 automatically establish a wireless connection using the short-range communication interface 161 of the HMD 1 and the short-range communication interface 261 of the smartphone 2 (S2, S32).

[0055] Here, a situation occurs in which character input is required while using the HMD 1, such as when the operator of the HMD 1 sends a greeting message to another operator in the VR space. This situation occurs when the operator operates a button or other device on the HMD 1 that inputs character. Then, the input processing 114 of the HMD 1 enables an HMD character input mode in which the HMD operator directly operates the HMD 1, and a screen such as that shown in Figure 5B is displayed. This HMD character input mode is a mode in which, for example, the movement of a finger is recognized as a character by the HMD camera and input, and the recognized character can be displayed in the character input area 10.

[0056] When the HMD character input mode is activated, it is determined from the character input device information whether or not to input characters to the smartphone 2 (S3 YES). The character input device information is information obtained when the operator operates a button or the like on the HMD 1 to input characters to the smartphone 2. Alternatively, if the character input device information is stored in advance in a state table such as the flash memory 113 and "1" is set in the character input device column, this information can also be used to achieve this.

[0057] If S3 is YES, an instruction (instruction code) is transmitted (S4) from the HMD 1 to the smartphone 2, which is the information processing terminal, by wireless connection via the short-range communication interface 161. If character input is not required in S3 (No in S3), character input is not performed on the smartphone 2 and the process transitions to character input end (S12).

[0058] Meanwhile, the smartphone 2 receives the instruction code transmitted by the HMD 1 in S4 via a wireless connection (S33). Upon receiving the instruction code in S33, the smartphone 2 sets the preparation flag of the smartphone 2 to an on state (preparation state) so that the smartphone character input mode is enabled. The preparation flag can be realized, for example, by setting "1" in the preparation flag column of the state table stored in the flash memory 213 of the control circuit 201 in FIG. 3.

[0059] Then, when the smartphone 2 determines that the acceleration sensor 253 of the sensor unit 205 or the like has detected a movement of the smartphone 2, such as the operator wearing the HMD 1 taking out (picking up) the smartphone 2, within a predetermined time period while the preparation flag for the smartphone character input mode is in an on state (YES in S34), the smartphone 2 proceeds to S35. Note that the detection of the smartphone 2 may also be detection that the operator is picking up the smartphone 2.

[0060] Furthermore, before the predetermined time, a message prompting the user to move the smartphone 2 or activate the smartphone character input mode may be displayed on the display 123 of the HMD 1 or an alert warning may be issued through the speaker 172. Furthermore, an alert warning may be issued through the speaker 272 of the smartphone 2.

[0061] Even if the operator moves the smartphone 2 while the preparation flag is in the OFF state (the preparation flag column in the state table is "0"), the process does not proceed to S35.

[0062] When the smartphone 2 receives the instruction code and obtains information that the preparation flag is on and information that the smartphone 2 has moved (YES in S34), the smartphone 2 starts and transitions to a smartphone character input mode in which characters can be input from the smartphone 2.

[0063] 4A, the display area 20 for character input and the keyboard 21 are displayed on the display 223 of the smartphone 2, and information such as a startup completion code indicating that startup of the smartphone character input mode has been completed (the smartphone character input mode is enabled) is transmitted to the HMD 1 via a wireless connection (S35). The transmitted information here includes the startup completion code, the screen of the keyboard 21, etc.

[0064] If movement of the smartphone 2 cannot be detected (No in S34), the process proceeds to S40, where the smartphone character input mode is ended or preparation for starting the smartphone character input mode is ended (S40).

[0065] The HMD 2 receives information such as the startup completion code transmitted by the smartphone 2 in S35. The received information includes the startup completion code, the screen of the keyboard 21, etc. When the startup completion code is received, the startup flag is set to ON (S5). When the instruction code is transmitted in S4 and the startup flag is set to ON in S5 (YES in S6), the process proceeds to S7.

[0066] If the activation flag is not set to on (No in S6), that is, if the movement of the smartphone 2 is not detected in S34 of the smartphone 2 or the smartphone character input mode is not activated in S35, activation of the smartphone character input mode of the smartphone 2 is not completed. Therefore, the determination in S6 is No. The HMD 1 then waits to receive a code indicating the activation completion of the smartphone character input mode of the smartphone 2.

[0067] If S6 becomes YES, the HMD character input mode of the input processing 114 of the HMD 1 is stopped. That is, the HMD 1 switches from the HMD character input mode to a superimposed input mode in which the screen of the keyboard 21 received in S35 is superimposed on the VR image and the keyboard 11, etc. are displayed (S7). Note that the superimposed screen as the superimposed input mode at this point is the superimposed screen before characters, etc. are received, and character information, etc. is received and displayed in the subsequent step (S8).

[0068] 5B , a message urging the user to switch to the HMD character input mode screen or a message indicating that the startup completion code has been received may be displayed temporarily on the HMD character input mode screen on the display 123 until the switch is completed, so that the user can reliably switch from the HMD character input mode screen to the superimposed input mode screen and subsequently input data. In addition to the keyboard 11 displayed on the superimposed input mode screen after the switch, a message indicating that character input from the smartphone 2 is possible or a character input enabled mark may be displayed.

[0069] The smartphone character input mode enabled in S35 is a state in which character input is possible, and character input is performed in S36. This input operation is performed by the operator operating the smartphone 2 while looking at the screen of the keyboard 21 on the display 123 of the HMD 1 to input characters.

[0070] Then, the character information of the input characters displayed in the display area 20 is transmitted to the HMD 1 as input information (S37).

[0071] This input character is recognized as predetermined character information (character code) within the smartphone 2. For example, the character code is a code defined by the UTF-16 standard as shown in Fig. 11. The character code for the character "a" is "3042".

[0072] Also, the keyboard 21 screen when the finger 22 is in contact with the keyboard 21 and key pointing position information showing the position of the key pointed to by the finger 22, or the keyboard 21 screen when the finger 22 is not in contact with the keyboard 21, are transmitted to the HMD 1 as input information (S37).

[0073] In S8, the HMD 1 receives the input information transmitted by the smartphone 2 in S37. Then, a superimposition process is performed to superimpose the screen of the keyboard 21 of the smartphone 2 and the pointing position information on the VR image so that the screen and pointing position information can be displayed as the keyboard 11 with finger marks superimposed thereon, and the superimposed image is displayed on the display 123.

[0074] Specifically, a pointing image is generated from the received screen of the keyboard 21 and pointing position information of the key indicated by the finger 22 to superimpose the finger mark 12 as a finger image onto the position of the key on the keyboard 11. Then, the keyboard 11 with the finger image superimposed thereon (pointing image) is displayed superimposed on the VR image of the HMD 1.

[0075] Furthermore, when pointing position information is not available, a superimposition process is performed to superimpose an image of the keyboard 21 on the VR image so that the keyboard 21 of the smartphone 2 can be displayed as the keyboard 11, and the superimposed image is displayed on the display 123.

[0076] Furthermore, the received character information is superimposed on the character input area 10 and displayed on the display 123 (S36).

[0077] The display 123 displays a character input area 10 and a keyboard 11 on the top surface to make the operator more aware of the character input, and the VR image is partially hidden and superimposed.

[0078] The keyboard 11 may also be displayed in an empty space where the VR image of the HMD 1 can be viewed. This can be done by setting the display position in advance in a setting mode and storing the display position information in the flash memory 113 or the like so that the keyboard 11 can be displayed in any free space, such as an empty space above, below, left, or right of the display 123, without affecting the VR image. The HMD 1 may also have a setting mode for displaying the keyboard 11 at the line of sight using a line of sight position determination function or the like of the operator.

[0079] A predetermined end character code, such as a line feed key, is entered (S36), and character input ends (S39). The end character code is sent to the HMD 1 (S37), and the HMD 1 determines that input is complete (S10), and character input to the HMD 1 ends (S12).

[0080] On the other hand, when the MDH 1 receives a predetermined number of characters (S8), it determines that input is complete (S10), and character input to the HMD 1 ends (S12). An input end code is then sent to the smartphone 2 (S11). The smartphone 2 determines that character input has ended based on the received input end code (S39), and the smartphone character input mode ends (S40).

[0081] For the sake of explanation, the smartphone 2 transmits the image in S37 after new character input. However, the smartphone 2 may also transmit the image even if no new character input is made, for example, every predetermined time of one second or less. In other words, if the end of character input is not determined in S10 (if S10 does not result in YES), the smartphone 2 repeatedly executes S8 to S10. The smartphone 2 also repeatedly executes S36 and S37. This allows the smartphone 2 to superimpose the same image as the immediately preceding image.

[0082] Therefore, the processing operations of the smartphone 2 and the HMD 1 are repeated, which can solve the problem that the HMD 1 becomes unable to input characters or the like from the smartphone if the state of waiting for the reception of the characters or the like to be input from the smartphone continues for a long time.

[0083] In this way, in Example 1, the operator of HMD 1 can reliably link smartphone 2 when operating HMD 1, and can reliably send input characters to HMD 1 using the smartphone character input mode of smartphone 2.

[0084] In addition, by transmitting input information such as text information, screen information, and pointing position information from the smartphone 2 and superimposing it on the VR image, it is possible to easily input text while checking the familiar display screen of the smartphone 2 on the display 123 of the HMD 1.

[0085] The smartphone 2 originally had a variety of applications, including phone, email, chat, information search, and route guidance, and its interface allowed for easy character input. However, even with its ease of operation, some skilled operators can input characters without looking at the screen. However, while typical operators cannot directly see the keys on a flat display like a smartphone when wearing a VR-type HMD, the operator of the HMD 1 can easily input characters by displaying the familiar input screen of the smartphone 2 on the HMD 1 display. Furthermore, the smartphone 2 has a support function that memorizes frequently used characters and characters that have been used before, making character input even easier. Example 2 Example 2 will be described using the operational flowchart of Figure 9. Functions identical to those in Figure 8 are assigned the same numbers and will not be described again.

[0086] In Example 1, the HMD 1 transmits an instruction code to the smartphone 2 to prepare for activation of the smartphone character input mode. The smartphone 2, upon receiving the instruction code, transitions to a state in which the smartphone character input mode is operable, and activates the smartphone character input mode upon detecting a movement such as picking up the smartphone 2. The smartphone 2 then transmits the character information and character input screen input by the smartphone 2 to the HMD 1, and the HMD 1 displays the received character input screen and character information superimposed on each other.

[0087] In the second embodiment, the camera of the HMD 1 captures an image of the finger typing on the smartphone 2 and the character input screen, and the image is used as the character input screen on the HMD 1. When it is difficult to detect the movement of taking out the smartphone 2 or when the HMD 1 cannot capture an image of the character input screen on the smartphone 2, the audio output unit 107, which serves as an alarm output unit, issues an alarm warning.

[0088] 9 , in a situation where the HMD 1 requires character input, the smartphone 2 is instructed by the HMD 1 to prepare for activation of the smartphone character input mode (S33). The smartphone 2 detects that it is being picked up (YES in S34), and the smartphone character input mode is activated (S35). As a result, a character input screen is displayed on the display of the smartphone 2. Furthermore, the smartphone 2 transmits an activation completion code to the HMD 1 indicating that the character input screen has been displayed.

[0089] The HMD 1 receives the startup completion code (S5), and when it determines that it has been received (YES in S6), it stops the HMD character input mode of the HMD 1 and switches to a superimposed input mode in which character information entered in the smartphone character input mode of the smartphone 2 and sent to the HMD 1 is displayed on the display 123 of the HMD 1 (S7).

[0090] After switching the character input of the HMD 1 to the character input from the smartphone 2 in S7, the camera 120 mounted on the HMD 1 captures an image of the screen of the smartphone 2 (S21). When this switching is performed, a message prompting the user to capture an image of the screen of the smartphone 2 is displayed on the display 123 of the HMD 1.

[0091] Next, it is detected whether the image captured in S21 is a character input screen for the keyboard 21 of the smartphone 2 (S22). If the character input screen of the smartphone 2 is detected (YES in S22), the display of the message prompting the user to take a photo in S21 is stopped, and the keyboard 21 portion of the captured screen is cut out and superimposed on the VR image of the HMD as the keyboard 11, which is a character input screen (S23).

[0092] If the captured image in S22 is not the character input screen of the smartphone 2 or if the smartphone 2 cannot be photographed (S22 is No), the smartphone 2 is instructed to display an alarm mark or emit an alarm sound so that the screen of the smartphone 2 can be photographed with the camera 120 of the HMD 1 (S15).

[0093] Upon receiving the instruction, the audio processing unit 207 serving as the alarm output unit of the smartphone 2 issues an alarm warning (S41). After that, when a command to not display the alarm (S18) is received from the HMD 1, the alarm is stopped (S42).

[0094] The HMD 1 itself also issues an alarm for a certain period of time and then stops, which is repeated a predetermined number of times (S15 to S17). The alarm may be a superimposed image on the VR screen of the HMD 1 warning the user to take a picture of the screen of the smartphone 2, or a flash on the display. The smartphone 2 may also flash, vibrate, or ring.

[0095] If the condition does not improve after the alarm has been sounded a certain number of times (or for a certain period of time), an alarm end notification is sent to the smartphone 2 (S18). Then, the character input of the HMD 1 is switched from character input from the smartphone 2 to the HMD 1's own input method, which is the default of the HMD 1 (S19).

[0096] In this way, the timing of alarm generation is controlled in steps S15 to S18 of the HMD 1. Note that the timing of alarm generation may also be controlled in the smartphone 2. Example 3 In FIG. 6A, input is performed by pressing an English keyboard used for alphabet input. As another example of character input operation, there is a character input method known as the glide method, in which characters on the display are touched and continuously slid with the finger, as shown in FIG. 10, and the character (word) to be input is inferred and displayed from the characters traced by the finger. Input using this glide method is also possible.

[0097] This method has the advantage that characters can be input by sliding a finger over the characters, so that input can be done without staring at the character input screen and the input speed is fast.

[0098] Even when glide character input is performed, the input character information (character code) can be transmitted to the HMD 1 as a code as shown in Fig. 11, similar to pressing a key on a keyboard. There are various character input methods called keyboard applications, but any method can be used as long as it can transmit a character code corresponding to the selected character to the HMD 1.

[0099] Example 3 will be described using the operation flowchart of Fig. 12. The same functions as those in Fig. 8 and Fig. 9 are assigned the same numbers, and descriptions thereof will be omitted. In Fig. 8 of Example 1, when the smartphone 2 detects a movement, such as being picked up by the operator of the HMD 1, using the acceleration sensor 253 of the sensor unit 205 (YES in S34), the smartphone 2 activates the smartphone character input mode.

[0100] In Example 3, an instruction code is sent from HMD 1 to smartphone 2 instructing it to activate smartphone text input mode, and when smartphone 2 receives the instruction code, smartphone 2 enables smartphone text input mode regardless of whether it is lifted or not (S33, S35).

[0101] When the smartphone character input mode is enabled in S35, the smartphone character input mode is transmitted to the HMD 1 (S5), and character input switching is performed to switch the character input of the HMD 1 (S7). The alarm processing of the smartphone 2 is skipped, and character input is performed using the glide method (S36). When characters are input, character information (character code) of the string of characters that has been traced is transmitted to the HMD 1 (S37). As a result, the HMD 1 receives the transmitted character information (character code) (S13).

[0102] From the received character code, the HMD 1 detects the position of the finger on the character input screen (S14). In S14, for example, in Fig. 10, the tracing direction changes between the characters "S" and "P" on the screen, and the finger stops at "R". These three characters can be estimated as character information. The character code of these characters is linked to the UTF-16 character code in Fig. 11 and transmitted to the HMD 1. "S" and "P" are the input characters. It can be estimated that "R" is the character where the finger is stopping, and the position of the finger is detected based on this estimation.

[0103] Furthermore, the smartphone 2 maintains the mode for continuous character input until character input is no longer required on the HMD 1. If the HMD 1 cannot receive character information within a predetermined time after the smartphone 2 displays the character input screen (No in S13), it instructs the smartphone 2 to issue an alarm such as a ringing sound. The alarm continues for a certain period of time and up to a certain number of times (S15 to S17).

[0104] As described above, in Example 3, character input and finger position are estimated from character information received by the HMD 1, so that input by gazing at the character input screen of the smartphone 2 is not necessary. Example 4 Example 4 will be described using the operation flowchart of FIG. 13. Functions that are the same as those in FIGS. 8, 9, and 12 are assigned the same numbers, and descriptions thereof will be omitted. In the previous examples, the finger position during character input was obtained from the character input image displayed on the smartphone 2 and character information (character code) from the smartphone 2. In Example 4 of FIG. 13, the screen proximity sensor 262 provided in the smartphone 2 is utilized to estimate the finger position.

[0105] The screen proximity sensor 262 is mainly used to prevent erroneous operation when the smartphone 2 is in contact with the body, for example, when making a call on the smartphone 2, and this screen proximity sensor 262 detects the position of the finger. The screen proximity sensor 262 uses infrared rays to detect the distance to an object from the intensity of the infrared rays reflected from the object. This is used to detect the position of the finger.

[0106] When the HMD 1 requires character input, the smartphone 2 activates the smartphone character input mode in response to an instruction from the HMD 1, and a character input screen is displayed on the display 223 (S35).

[0107] Next, characters are input (S36), and the distance to the finger is detected based on the output from the screen proximity sensor 262. Position information of the finger is estimated from this distance (S47).

[0108] The finger position information is transmitted as input information together with the character information of the smartphone 2 to the HMD 1 (S37). The transmitted information is received by the HMD 1 in S8, and then a finger mark 12 of a finger image is displayed at a location corresponding to the finger position on the character input screen of the keyboard 11 shown in Fig. 5C, and is displayed in the character input area 10 together with the input character superimposed on the VR image.

[0109] Furthermore, by combining a method for estimating the position of an approaching finger using a screen proximity sensor 262 with a method for determining the position of a touching finger on a display 223 in which the touch panel 224 and the display are integrated, the position of the finger can be estimated more accurately. Furthermore, by increasing the number of screen proximity sensors 262 and using a triangulation technique, the detection accuracy of the finger position can be further improved. Example 5 As Example 5, FIG. 14 shows an example of an AR image captured by a camera mounted on the AR-type HMD 1. As shown in the conceptual diagram of the configuration in FIG. 14, an AR-type HMD 1001 is connected to a PC 100 via a network 99, and the HMD 1001 operates by receiving information and commands from an AR application on the PC 100. A smartphone 2 is connected to the HMD 1001 via communication such as near-field wireless communication. A real-world image is captured by a camera mounted on the HMD, and the image is displayed as an AR image.

[0110] Figure 15 shows an example of pipe system maintenance in which an AR-type HMD is applied to maintain pipe pressure at a specified value. The real image shows three pipes: pipe A 1125, pipe B 1126, and pipe C 1127, each with a control panel 1124 attached. Figure 15A shows this real image. The realistic AR image is viewable through a see-through display 1123, but the control panel 1124 cannot be fully seen. The control panel 1124 in this pipe system is used to adjust the pressure of each pipe to a specified value.

[0111] During maintenance, the actual pipe system is viewed through a translucent display 1123. When the control panel application is started, a control panel screen 1128 is displayed superimposed on the screen, showing the standard and current pressure values ​​for each pipe, as shown in Fig. 15B.

[0112] Here, when the pressure adjustment mode is enabled, a keyboard 1129 is superimposed on the real image as shown in FIG. 15C. When adjustment is necessary, the pressure input area on the control panel screen 1128 for inputting a correction value becomes editable. To input a correction value for adjusting the pressure value, the smartphone character input mode, which can be used to input values ​​including numerical values ​​on the smartphone 2, is used. The smartphone character input mode on the smartphone is enabled, and the pressure adjustment value is input from the smartphone. It is also possible to enable the comment mode and input comments other than numerical values.

[0113] Character input using the smartphone 2 is performed when performing correction input for the AR type HMD, that is, when performing numerical input or character input.

[0114] Specifically, an instruction code is sent from the HMD 1011 to the smartphone 2. The smartphone 2 prepares to activate the smartphone character input mode, detects movement such as when the HMD operator takes out the smartphone 2, enables the smartphone character input mode, and displays a character input screen on the smartphone 2. When characters are input, character information is sent from the smartphone 2 to the HMD 1001. The sent character information is superimposed on the AR image of FIG. 15C , and a pressure correction value is input in a mode for adjusting pressure. In this way, the AR-type HMD can be applied in the same way as the VR-type HMD.

[0115] 1: HMD 2: Smartphone 10: Character input area 11: Keyboard 12, 13: Finger mark 20: Display area 21: Keyboard 22: Finger 99: Network 100: PC 102: Image processing unit 103: Control circuit 104: Data bus 105: Sensor unit 106: Communication processing unit 107: Audio processing unit 114: Input processing 123: Display 161: Short-range communication interface 172: Speaker 201: Control circuit 202: Image processing unit 204: Data bus 205: Sensor unit 206: Communication processing unit 207: Audio processing unit 223: Display 262: Screen proximity sensor 261: Short-range communication interface 1001: HMD 1123: Display 1124: Control panel 1128: Control panel screen 1129: Keyboard

Claims

1. A head-mounted display, comprising: a display; an image processing unit that generates a generated image to be displayed on the display; a connection unit that connects to an information processing terminal; and a control unit that controls input processing for character input, wherein when the control unit activates an HMD character input mode for performing character input using the head-mounted display, it transmits an instruction to enable an information processing terminal character input mode for performing character input using the information processing terminal to the information processing terminal, and when receiving information that the information processing terminal character input mode has been activated from the information processing terminal, it stops the HMD character input mode and receives input information in the information processing terminal character input mode from the information processing terminal, and the image processing unit generates a superimposed image that superimposes the received input information on the generated image and displays the superimposed image on the display.

2. The head-mounted display according to claim 1, wherein the received input information includes character information.

3. The head-mounted display according to claim 1, wherein the received input information includes screen information of a character input unit in the information processing terminal.

4. The head-mounted display according to claim 1, wherein the received input information includes screen information of a character input unit in the information processing terminal and pointing position information of a living body that contacts or is close to the character input unit, and the image processing unit generates a pointing image from the screen information and the pointing position information and displays a superimposed image obtained by superimposing the pointing image on the generated image on the display.

5. The head-mounted display according to claim 1, wherein the control unit performs transmission to prompt activation of the information processing terminal character input mode when not receiving information indicating completion of activation of the information processing terminal character input mode.

6. An information processing terminal that operates in conjunction with a head-mounted display, comprising: a display; a connection unit that connects to the head-mounted display; a character input unit for performing character input; and a control unit that controls the character input unit, wherein when an HMD character input mode for performing character input by the head-mounted display is activated, the control unit receives an instruction to enable an information processing terminal character input mode for performing character input at the character input unit of the information processing terminal, transmits information indicating that the information processing terminal character input mode has been activated to the head-mounted display, and transmits input information in the information processing terminal character input mode to the head-mounted display.

7. The information processing terminal according to claim 6, further comprising a sensor that detects movement of the information processing terminal, wherein activation of the information processing terminal character input mode is performed when the sensor detects movement of the information processing terminal.

8. The information processing terminal according to claim 6, wherein the input information includes screen information of the character input unit.

9. The information processing terminal according to claim 6, further comprising a position sensor that detects pointing position information of a living body that contacts or is close to the character input unit, wherein the input information includes the pointing position information.

10. The information processing terminal according to claim 6, further comprising an alarm output unit that issues an alarm, wherein when information indicating completion of activation of the information processing terminal character input mode is not transmitted, an instruction to prompt activation of the information processing terminal character input mode is received, and the alarm output unit issues an alarm.

Images