Program, storage medium storing program, character input system and character input method

Abstract

To provide a program for character input suitable for a wearable display terminal.SOLUTION: A program (150) causes a computer to create video data of a character input interface (320) including at least one or more keys to receive a character input operation, transmit the video data to a wearable display terminal (10), transit a key in which a pointer linked with the movement of a remote controller (20) overlaps, of the key(s) included in the character input interface, to a character selection operation standby mode, and, when an operation signal of the remote controller is received during the character selection operation standby mode, display the character(s) assigned to the key(s) on the wearable display terminal based on the operation signal.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a program, a storage medium storing the program, a character input system, and a character input method. [Background technology]

[0002] Gesture operations are known as input operations to a head mounted display. For example, paragraphs 0048 and 0049 of the specification of Patent Document 1 state that "The CPU detects the input operation of the user to the virtual input device displayed on the HMD. Such detection can be performed, for example, by using a technique for tracking the movement of the user's hands and fingers captured by a camera. As a technique for tracking the movement of the user's hands and fingers, various techniques that apply image processing can be applied. For example, the technique described in the following document is an example of a technique that can be suitably applied as a technique for tracking the movement of the user's hands and fingers. Document: Sasaki et al., "Tenohira Daiyaru: Input Interface for Wearable Computers", IEICE Technical Report PRMU 2000-157 (2001-1) Alternatively, a contact sensor is attached to the user's fingertip, and the movement of the user's finger is tracked based on the output signal from the contact sensor (partial excerpt from the specification of Patent Document 1)." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2002-318652 A Summary of the Invention [Problem to be solved by the invention]

[0004] Keyboards are widely known as input devices for computers. It is conceivable to use such keyboards as an input device for a head-mounted display, for example, by displaying a virtual keyboard on the head-mounted display and inputting characters using the gesture movements described in Patent Document 1.

[0005] However, because a head-mounted display needs to be equipped with a display large enough to hold it in front of the user's eyes, there is a restriction on the size of the screen area. Therefore, when a virtual keyboard is displayed on a head-mounted display, the size of each key is inevitably small, and there is an issue that accurate key input cannot be performed by gesture movements using both hands, and character input operation cannot be performed smoothly. This issue is not limited to head-mounted displays, but is common to wearable display terminals such as smart glasses and AR glasses (virtual reality display glasses).

[0006] The present invention has been made in consideration of the above circumstances, and its object is to provide a program suitable for character input on a wearable display terminal, a storage medium storing the program, a character input system, and a character input method. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention has the configuration described in the claims. As an example, the present invention is a program for causing a computer to realize the following functions: generating video data of a character input interface including at least one key for receiving a character input operation and transmitting the video data to a wearable display terminal; transitioning a key included in the character input interface on which a pointer linked to the movement of a remote controller is superimposed to a character selection operation standby mode; and, when an operation signal of the remote controller is received during the character selection operation standby mode, displaying a character assigned to the key based on the operation signal on the wearable display terminal.

[0008] The present invention also relates to a storage medium storing the above program.

[0009] The present invention also provides a character input system including a computer storing the above program, a wearable display terminal, and a remote controller, wherein the computer receives the operation signal from the remote controller and transmits from the computer to the wearable display terminal video data of the character input interface and video data of a character assigned to the key selected in accordance with the operation signal of the remote controller, and the wearable display terminal displays the character assigned to the key selected in accordance with the character input interface and the operation signal of the remote controller.

[0010] The present invention also provides a character input method, in which a computer executes the steps of transmitting image data of a character input interface including at least one key for receiving character input operations to a wearable display terminal, transitioning a key included in the character input interface that is superimposed with a pointer linked to the movement of a remote controller to a character selection operation standby mode, and, upon receiving an operation signal from the remote controller during the character selection operation standby mode, displaying on the wearable display terminal a character assigned to the key based on the operation signal. Effect of the Invention

[0011] According to the present invention, it is possible to provide a program suitable for character input to a wearable display terminal, a storage medium storing the program, a character input system, and a character input method. Objects, configurations, and effects other than those described above will be made clear in the following embodiments. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram showing an overview of a character input system according to an embodiment of the present invention; [Diagram 2]FIG. 2 is a diagram illustrating the hardware configuration of an HMD. [Diagram 3] FIG. 2 is a hardware configuration diagram of a remote controller. [Figure 4] FIG. 2 is a diagram illustrating a hardware configuration of a personal computer. [Diagram 5] FIG. 2 is a functional block diagram of the character input system. [Figure 6] FIG. 1 is a diagram conceptually illustrating one aspect of a virtual space. [Figure 7] 10 is a flowchart showing a process flow in a first input mode in the character input system 1 according to the present embodiment. [Figure 8] 10 is a flowchart showing a process flow in a first input mode in the character input system 1 according to the present embodiment. [Figure 9] FIG. 13 is a diagram showing an example of a screen display displayed on an HMD. [Figure 10] FIG. 13 is a diagram showing an example of a screen display displayed on an HMD. [Figure 11] 11A and 11B are diagrams illustrating a process of determining a direction of a similar flick operation. [Figure 12] 4 is a diagram showing an example of button assignment to operators on a remote controller. FIG. [Figure 13] 4 is a diagram showing an example of button assignment to operators on a remote controller. FIG. [Figure 14] 4 is a diagram showing an example of button assignment to operators on a remote controller. FIG. [Figure 15] 13 is a flowchart showing a process flow in a second input mode in the character input system. [Figure 16] 13 is a flowchart showing a process flow in a second input mode in the character input system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment of the present invention described below is an example of how the present invention is embodied, and the scope of the present invention is not limited to the scope of the embodiment. Therefore, the present invention can be implemented by adding various modifications to the embodiment.

[0014] This embodiment relates to a character input system in which a wearable display terminal and a remote controller configured separately are linked. Types of wearable display terminals include, for example, head mounted displays (hereinafter referred to as "HMD"), smart glasses, and AR glasses (virtual reality display glasses), and the HMD may be an immersive type or a see-through type. The following describes an example of a character input system using an immersive HMD.

[0015] [Overview of Character Input System 1] Fig. 1 is a diagram showing an overview of a character input system 1 according to this embodiment. As shown in Fig. 1, the character input system 1 includes an HMD 10, a remote controller 20, and a personal computer (hereinafter abbreviated as "PC") 30, which are communicatively connected.

[0016] The HMD 10 includes a display 11, a control device 12, an HMD-side short-range wireless communication device 13, and a holder 19 for holding the display 11 in front of the user's eyes. As an example, the HMD-side short-range wireless communication device 13 may be a Bluetooth (registered trademark, hereinafter abbreviated as "BT") communication device.

[0017] The remote controller 20 includes a remote control side short-range wireless communication device 21 in a housing 29, and an operation button 22 on the back surface of the housing 29. A BT communication device is used as the remote control side short-range wireless communication device 21, which is communicatively connected to the HMD side short-range wireless communication device 13 or the PC 30. When the HMD 10 performs the image generation process, the HMD 10 may directly receive the operation signal of the remote controller 20 and reflect it in the image displayed on the HMD 10. Also, if the PC 30 performs the image generation process and the HMD 10 receives and displays the image signal, the operation signal of the remote controller 20 may be configured to be received by the PC 30. In this embodiment, the latter example will be described below.

[0018] The PC 30 is configured using, for example, a desktop personal computer. The PC 30 may be communicatively connected to a virtual reality providing server 50 via a communication network 40 such as the Internet. The PC 30 receives virtual reality rendering data from the virtual reality providing server 50, generates virtual reality video data, and transmits the data to the HMD 10 from the PC 30. At that time, the PC 30 accepts character input according to an operation signal received from the remote controller 20, and generates a virtual reality video in which the inputted characters are superimposed on the virtual reality video. Hereinafter, virtual reality will be referred to as VR (Virtual Reality).

[0019] [HMD10 Configuration] 2 is a hardware configuration diagram of the HMD 10. The HMD 10 includes a processor 101, a RAM 102, a display 103, a storage 104, an input interface 105, a motion sensor 106, an outer camera 107, a communication interface 108, a microphone 109, and a speaker 110, which are connected to each other via a communication bus 111.

[0020] The processor 101 realizes the processes described below by executing a series of instructions included in a terminal program 104P stored in the storage 104. The processor 101 is realized as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), a field-programmable gate array (FPGA), or other device.

[0021] The RAM (Random Access Memory) 102 is a volatile memory.

[0022] As an example, the display 103 is realized as a non-transmissive display device. The display 103 is disposed on the main body of the HMD 10 so as to be located in front of both eyes of the user. The non-transmissive display 103 is realized as, for example, a liquid crystal monitor or an organic EL (Electro Luminescence) monitor.

[0023] As another example, the display 103 is realized as a transmissive display device. In this case, the HMD 10 is not a closed type that covers the user's eyes, but an open type such as glasses. The display 103 may include a configuration that simultaneously displays a part of an image that constitutes a virtual space and a real space. As an example, the transmissive display 103 may display an image of the real space captured by a camera mounted on the HMD 10. As another example, the transmissive display 103 may be configured to have an adjustable transmittance. The transmissive display 103 may set the transmittance of a part of the display area high so that the real space can be directly viewed.

[0024] Furthermore, the display 103 may employ the following configuration to allow the user to view a three-dimensional image. As an example, the display 103 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. As another example, the display 103 may be configured to integrally display an image for the right eye and an image for the left eye. In this case, the display 103 includes a high-speed shutter. The high-speed shutter operates to alternately display an image for the right eye and an image for the left eye so that the image is recognized by only one of the eyes.

[0025] The storage 104 is a storage medium that permanently holds data, and is realized as, for example, a ROM (Read-Only Memory), a flash memory, or other non-volatile storage medium. The storage 104 may also be realized as a removable storage medium such as a memory card. The storage 104 stores a terminal program 104P required for controlling the HMD 10.

[0026] The input interface 105 is, for example, a USB (Universal Serial Bus) terminal.

[0027] The motion sensor 106 detects the movement of the HMD 10, for example, the movement of the user's head when the HMD 10 is worn on the user's head, because the HMD 10 and the user's head move almost as one unit. The motion sensor 106 is realized, for example, by using a three-axis acceleration sensor.

[0028] The outer camera 107 is a camera that captures an external world image. In the immersive HMD 10, a through image of the outer camera 107 is displayed on the display 103, so that the user can visually recognize the external world image while wearing the HMD 10 and understand the situation around him / her.

[0029] The communication interface 108 is an interface for communicatively connecting the HMD 10 to an external device. As an example, the communication interface 108 may be configured as a USB terminal, and a BT dongle may be used as the HMD-side short-range wireless communication device 13 to perform short-range wireless communication with the remote controller 20 or the PC 30. Simultaneous connection between the remote controller 20 and the PC 30 may be realized by using a BT communication device compatible with multi-pairing.

[0030] The microphone 109 is a device that collects sound around the HMD 10. When the user inputs voice, the microphone 109 collects the user's voice and inputs it to the HMD 10, where voice recognition processing is executed, thereby enabling voice input.

[0031] The speaker 110 is an audio output device that outputs the audio of the content being displayed on the display 103 and other audio.

[0032] [Configuration of remote controller 20] 3 is a diagram showing the hardware configuration of the remote controller 20. The remote controller 20 includes a processor 201, a RAM 202, a ROM 203, an operation switch 204, a motion sensor 205, and a BT communication device 206, which are connected to each other via a communication bus 209.

[0033] The RAM 202 is a volatile memory and is used as a temporary storage area for processing programs and data.

[0034] The ROM 203 stores the processing program for the remote controller 20 .

[0035] The operation switch 204 is a switch that the user presses down when performing a flick-like operation for selecting and confirming a character type. The "flick-like operation" is an operation for inputting instructions by the acceleration when the remote controller 20 is swung. A flick operation performed on a conventional smartphone or tablet terminal is an operation in which a finger is moved in any direction, up, down, left or right, while pressing a touch panel laminated on the surface of the smartphone or tablet terminal, but even if a similar operation is attempted using the remote controller 20, the flick operation itself cannot be performed because there is no touch panel. Therefore, the HMD 10 equipped with the character input UI program 150 according to this embodiment executes a unique control process that is completely different from a conventional flick operation, but since the operation is intended to give the user a feeling of operation similar to a pseudo flick operation, it is referred to as a "flick-like operation" in this embodiment. The flick-like operation corresponds to a predetermined operation of the remote controller for performing a character selection operation.

[0036] The motion sensor 205 is a sensor that detects the movement of the remote controller 20, and is realized using, for example, a three-axis acceleration sensor.

[0037] The processor 201 is configured using, for example, an MPU (Micro Processor Unit), and reads out a control program for the remote controller 20 from a ROM 203, loads it into a RAM 202, and executes it. The processor 201 transmits operation information of an operation switch 204, the movement of the remote controller 20 detected by a motion sensor 205, and the like to the HMD 10 via a BT communication device 206. Therefore, the BT communication device 206 corresponds to a short-range wireless communication device on the remote control side.

[0038] [PC30 Configuration] 4 is a hardware configuration diagram of the PC 30. The PC 30 includes a processor 301, a RAM 302, a ROM 303, a storage 304, an input interface 305, and a communication interface 306, which are connected to each other via a communication bus 309. Details of each part overlap with those of the HMD 10 and the remote controller 20, and will not be described here.

[0039] The processor 301 is realized, for example, by using a GPU (Graphics Processing Unit).

[0040] The RAM 302 is a volatile memory and is used as a temporary storage area for processing programs and data.

[0041] The ROM 303 is a non-volatile memory that stores the operating system program of the PC and the like.

[0042] The storage 304 is realized by a non-volatile storage medium such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores the character input UI program 150 (see Figure 5) and the virtual reality display program 160 (see Figure 5) related to this embodiment.

[0043] [Function block diagram of character input system 1] Fig. 5 is a functional block diagram of the character input system 1. As shown in Fig. 5, the PC 30 includes a character input UI program 150 and a virtual reality display program 160. The virtual reality display program 160 is optional, but in the following description, in order to explain an example in which characters are input while a virtual reality image in which an avatar is placed in a virtual space is being displayed, the PC 30 is described as including the virtual reality display program 160. The PC 30 also includes a communication control unit 170 that transmits display data such as VR images to the HMD 10.

[0044] The character input UI program 150 includes a pointer control unit 151, a character input UI display control unit 152, and a character input control unit 153. The functions of these units are realized by the processor 301 reading out the character input UI program 150 stored in the storage 304 and loading it into the RAM 302. The functions of each unit will be described later along with the flow of processing in the character input system 1 according to this embodiment.

[0045] [Overview of Virtual Space 90] FIG. 6 is a diagram conceptually illustrating one aspect of the virtual space 90. As shown in FIG. 6, the virtual space 90 has a spherical structure covering the entire 360-degree direction from the center C. In FIG. 6, in order to avoid complicating the explanation, the celestial sphere in the upper half of the virtual space 90 is illustrated as an example. Meshes are defined in the virtual space 90. The position of each mesh is defined in advance as a coordinate value in an XYZ coordinate system, which is a global coordinate system defined in the virtual space 90. Each partial image constituting a panoramic image 91 (still image, video, etc.) that can be deployed in the virtual space 90 is associated with a corresponding mesh in the virtual space 90.

[0046] For example, in the virtual space 90, an XYZ coordinate system is defined with the center C as the origin. The XYZ coordinate system is, for example, parallel to the real coordinate system. The horizontal direction, vertical direction (up-down direction), and front-back direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the real coordinate system, and the Z-axis (front-back direction) of the XYZ coordinate system is parallel to the z-axis of the real coordinate system.

[0047] The viewpoint of the user of the HMD 10 is placed at the center C of the virtual space 90, and a field of view 94 in the virtual space 90 is defined from the user's viewpoint to the user's line of sight (reference line of sight 93) in the virtual space 90 as the center.

[0048] Processor 301 generates (extracts) a partial image included in field of view 94 of panoramic image 91 deployed in virtual space 90 as virtual reality image 97, and displays it on HMD 10. That is, field of view 94 corresponds to the user's field of view within virtual space 90. Furthermore, field of view 94 moves in response to changes in the position and orientation of virtual camera 92 within virtual space 90, and virtual reality image 97 displayed on HMD 10 is updated. That is, the user's field of view moves.

[0049] The character input UI 320 (character input interface) displayed in the character input UI program 150 is disposed in front of the field of view 94 in the virtual space 90. Therefore, when the character input UI 320 and the virtual reality image 97 overlap from the user's viewpoint of the HMD 10, occlusion processing is performed to display the character input UI 320 so that it appears to be in front of the virtual reality image 97. In the occlusion processing, the transparency of the character input UI 320 may be set to 0% to completely prevent the virtual reality image 97 overlapping with the character input UI 320 from being seen, thereby improving the visibility of the keys, or the transparency may be set to, for example, 50% to perform display control so that the virtual reality image 97 can be seen while inputting characters. The transparency can be set arbitrarily between 0% and 100%. In the occlusion processing, the character input UI 320 is basically displayed at the forefront, but the display order of the character input UI 320 may be appropriately set by the user, for example, by displaying it behind an avatar registration screen described later.

[0050] Furthermore, the processor 301 selects and inputs characters in conjunction with a user's operation received by the remote controller 20. Specifically, the processor 301 moves a pointer 330 in the virtual space 90 in response to the movement of the remote controller 20 detected by the motion sensor 205 of the remote controller 20. The processor 101 also determines which character has been selected in the character input UI 320 based on the acceleration detected by the motion sensor 205 and an operation signal indicating that the operation button 22 has been pressed, and displays a virtual reality image 97 including the selected character on the HMD 10.

[0051] The character input system according to this embodiment is suitable for inputting characters in the Japanese 50 sounds. The following description will be given taking the input of Japanese as an example. The character input system 1 according to this embodiment has a first input mode in which the movement of the index finger and the movement of the wrist are used in combination for input operation, and a second input mode in which input operation can be performed only by the movement of the fingers. The first input mode or the second input mode is set by selecting one of the modes on a setting screen (not shown) displayed on the HMD 10.

[0052] [First input mode] 7 and 8 are flowcharts showing the flow of processing in the first input mode in the character input system 1. FIGS. 9 and 10 are diagrams showing examples of screen displays displayed on the HMD 10. FIG.

[0053] It is assumed that a short-distance wireless communication connection has been established between the PC 30 and the HMD 10, and between the remote controller 20 and the PC 30, prior to the start of processing by the character input system 1 according to this embodiment.

[0054] The PC 30 executes the virtual reality display program 160, generates a virtual reality image including an avatar 300 and an avatar registration screen 310 shown in Fig. 9 as a virtual reality image, and transmits the generated image to the HMD 10. The HMD 10 displays the image on the display 103 (S01).

[0055] When the display condition for the character input UI 320 is met (S02: Yes), the character input UI display control unit 152 generates the character input UI 320, which is transmitted from the PC 30 to the HMD 10 and displayed on the display 103 (S03).

[0056] As an example of a display condition for the character input UI 320, when the user operates the remote controller 20 to point the pointer 330 at the character input field 311 of the avatar registration screen 310 in Fig. 9, the virtual reality display program 160 displays the cursor 312 in the character input field 311 and transitions to an input waiting state. This transition to the input waiting state may be set as a display condition for the character input UI 320.

[0057] The character input UI display control unit 152 places the character input UI 320 in front of the virtual reality image 97 as described above, and executes the occlusion process (see FIG. 6). In this embodiment, the character input UI display control unit 152 displays the character input UI 320 including 12 keys arranged in a 4-row x 3-column matrix, with 5 characters of the 50 hiragana or katakana sounds assigned to 10 of the keys, and other functions, such as changing the input mode or inputting special symbols such as punctuation marks, question marks, and exclamation marks, assigned to the remaining 2 keys.

[0058] Each key is assigned a character from the A, Ka, Sa, Ta, Na, Ha, Ma, Ya, or Wa row. In the 12 keys included in the character input UI 320 in FIG. 9 and FIG. 10, only the first character of each row (the first sound, for example, corresponding to "A" in the A row) is displayed on each key, and the second to fifth sounds of the A row (for example, corresponding to "I, U, E, O" in the A row) are omitted. This improves the visibility of the row assigned to each key, and even if the display of the second to fifth sounds is omitted, a person who can understand Japanese can easily guess that the second to fifth sounds are assigned to the left, upper, right, and lower in the order of the first sound on the 12 keys of a smartphone or tablet terminal, so operability is not impaired. Note that the first sound may be displayed in the center on each key, and the second to fifth sounds may be displayed around it in the order of the left, upper, right, and lower.

[0059] Furthermore, in addition to the 12 keys, the character input UI 320 according to this embodiment also includes 9 function keys such as a delete key, a line feed key, and a space key, but this is optional.

[0060] The pointer control unit 151 judges whether the pointer indicated by the remote controller 20 touches any of the keys (S04). The pointer control unit 151 calculates the position of the pointer 330 in the virtual space 90 (three-dimensional coordinates in the virtual space). There are several calculation methods, but as an example, a reference point (starting point) of the pointer 330 in the virtual space 90 is automatically determined at the time when the remote controller 20 is turned on. For example, the intersection of the reference line of sight 93 and the panoramic image 91 in the virtual space 90 of FIG. 6 is set as the reference point of the pointer 330. In this way, the pointer 330 is displayed on the reference line of sight 93 that coincides with the user's viewpoint direction, so that the pointer 330 is displayed in the center of the user's field of view 94 from the time the remote controller 20 is turned on, and a problem such as the user losing sight of the pointer 330 can be avoided.

[0061] Thereafter, the motion sensor 205 of the remote controller 20 detects the acceleration in a three-dimensional real space consisting of the X, Y and Z axes of the remote controller 20. Here, the ZX plane is two axes in a horizontal plane, and the Y axis is a vertical axis perpendicular to the ZX plane. The X axis is the left-right axis of the user holding the remote controller 20, and the Z axis direction is the depth direction axis as seen from the user. The motion sensor 205 detects the acceleration (a xtn ,a ytn ,a ztn ) is detected as sensor information. The remote controller 20 transmits the acceleration (a xtn ,a ytn ,a ztn ) is sent.

[0062] The pointer control unit 151 controls the acceleration (a xtn ,a ytn ,a ztn) and adding it to a reference point (starting point), the direction (or coordinates) in the virtual space 90 pointed by the remote controller 20 is calculated, and the pointer 330 is superimposed on the virtual reality image 97 of the display 103. When the pointer control unit 151 determines that the calculated pointer 330 is touching any key of the character input UI 320 (S04: Yes), the character input control unit 153 transitions the key touched by the pointer to a character selection operation standby mode (S04). Hereinafter, a state in the character selection operation standby mode is called "activated," and a state not in the character selection operation standby mode is called "inactivated." The character selection operation is performed by pressing the operation button 22 without moving the remote controller 20 substantially when selecting the first sound of an activated key, and by performing a flick-like operation with the operation button 22 of the remote controller 20 pressed down when selecting the second to fifth sounds of an activated key.

[0063] In the example of Fig. 9, pointer 330 is assigned a character in the "ma" row and is superimposed on key 321, so key 321 is activated. The distinction between activated keys and inactivated keys may be realized by differentiating the display mode. The change in the display mode may be a change in the display color, a blinking display, or the transparency of inactivated keys may be high (or low) and that of activated keys may be low (or high). The differentiation of the display mode is performed by character input UI display control unit 152.

[0064] The PC 30 waits to receive sensor information of the motion sensor 205 and a pressing signal of the operation button 22 from the remote controller 20 (S06: No). If the PC 30 further receives sensor information of the motion sensor 205 while receiving a pressing signal of the operation button 22 from the remote controller 20 (S06: Yes), the pointer control unit 151 calculates the movement direction of the pointer based on the sensor information, and classifies the movement direction of the pointer into either the 0 direction or the up, down, left, or right directions with the current position of the pointer as the starting point (S07).

[0065] FIG. 11 is an explanatory diagram of the process of determining the direction of a flick-like operation. The pointer control unit 151 determines whether the user has performed a flick-like operation based on the acceleration of the remote controller 20 in the XY plane. xtn ,a ytn ,a ztn ) among which a ztn is not used in the process of determining the direction of the flick-like operation. ·|Acceleration a xtn |≦ma x and |acceleration a ytn |≦ma y If so, then 0 direction (S08:0 direction), ·Acceleration a xtn <-ma x and |acceleration a ytn |≦|acceleration a xtn |If there is, go to the left (S08: left), ·Acceleration a ytn >ma y and |acceleration a xtn |≦|acceleration a ytn |, then upward (S08: upward), ·Acceleration a xtn >ma x and |acceleration a ytn |≦|acceleration a xtn |, then right direction (S08: right direction), ·Acceleration a ytn <-ma y and |acceleration a xtn |≦|acceleration a ytn |, then downward (S08: downward); It is determined that (S08). x , ma yare static margin accelerations in the left / right and up / down directions provided to prevent erroneous detection of a flick-like operation. Providing the static margin accelerations can improve operability by preventing the acceleration detected due to the user's hand moving even though the user does not intend to perform a flick-like operation from being recognized as a flick-like operation. Note that, in this embodiment, it is determined whether a flick-like operation has been performed based on the acceleration, but it is also possible to determine whether a flick-like operation has been performed by integrating the acceleration over time to find the speed.

[0066] The character input control unit 153 selects and confirms a character according to the movement direction of the pointer. More specifically, if the movement direction is the 0 direction (S08: 0 direction), the character input control unit 153 confirms the selection of the first sound assigned to the activated key (S09). As an example, the first sound may be determined as the central character written at the center of the key, or "a" for the "a row". Similarly, if the movement direction is the left direction (S08: left direction), the character input control unit 153 confirms the selection of the second sound assigned to the activated key (S10), if the movement direction is the up direction (S08: direction), the character input control unit 153 confirms the selection of the third sound assigned to the activated key (S11), if the movement direction is the right direction (S08: right direction), the character input control unit 153 confirms the selection of the fourth sound assigned to the activated key (S12), and if the movement direction is the down direction (S08: down direction), the character input control unit 153 confirms the selection of the fifth sound assigned to the activated key (S13).

[0067] The character input control unit 153 inputs and displays the characters selected and confirmed in steps S09 to S12 in the character input area (S14). In the example of FIG. 10, when the pointer 330 is on the activated key of the "ma" row, the operation button 22 of the remote controller 20 is pressed down, and the remote controller 20 is rotated with an acceleration a xtn <-ma x and |acceleration a ytn |≦|acceleration a xtn10, the pointer 330 moves in the direction indicated by the arrow 331 in Fig. 10, and the pointer control unit 151 detects a leftward flick-like operation, and the character input control unit 153 inputs and displays "mi" in the character input field 311. Note that the arrow 331 in Fig. 10 is shown for the sake of convenience, and on the display 103, only the pointer 330 moves and is displayed, and the arrow 331 is not displayed.

[0068] Until the condition for ending the display of the character input UI 320 is satisfied in the PC 30 (S15: No), the process returns to step S04, and character input is continued using the character input UI 320. When the condition for ending the display of the character input UI 320 is satisfied (S15: Yes), the character input UI display control unit 152 changes the display of the character input UI 320 displayed on the display 103 to a hidden state, and ends the character input process.

[0069] The display of the character input UI 320 may be ended, for example, when the virtual reality display program 160 stops displaying the avatar registration screen 310 or when the cursor 312 in the character input field 311 is switched to hidden.

[0070] According to this embodiment, character input can be performed by using a combination of finger button operation and wrist snapping (flick-like operation). Therefore, character input can be performed even if the remote controller 20 has buttons and an acceleration sensor but does not have an operator capable of performing the four types of input used in the second mode.

[0071] [Second input mode] 12, 13 and 14 are diagrams showing examples of button assignment to the controls of a remote controller.

[0072] The second input mode uses a first operator and a second operator of the remote controller 20. The first operator is an operator capable of switching input on and off, and the second operator is an operator capable of four types of input.

[0073] In the remote controller 20A of Fig. 12, a back button 240 is used as the first operator, and a stick 241 is used as the second operator. The first sound of each row of Japanese vowels "a" and consonants "ka, sa, ta, na, ha, ma, ya, ra, wa", that is, "a" in the "a" row, is assigned to the back button 240. The movement direction of the stick 241, which is the second operator, is classified into four directions, and the second sound of each row, that is, "i" in the "a" row, is assigned to the first direction (e.g., leftward) of the four directions, the third sound of each row, that is, "u" in the "a" row, is assigned to the second direction (e.g., upward) of the four directions of the stick 241, the fourth sound of each row, that is, "e" in the "a" row, is assigned to the third direction (e.g., rightward) of the four directions of the stick 241, and the fifth sound of each row, that is, "o" in the "a" row, is assigned to the fourth direction (e.g., downward) of the four directions of the stick 241. The same is true for each row of consonants.

[0074] In the remote controller 20B of FIG. 13, the back button 240 is used as the first operator, and the track pad 242 is used as the second operator. The touch surface of the track pad 242 is divided into four regions, and the second sound of each row, i.e., "i" in the "a" row, is assigned to the first divided region 242a, the third sound of each row, i.e., "u" in the "a" row, is assigned to the second divided region 242b, the fourth sound of each row, i.e., "e" in the "a" row, is assigned to the third divided region 242c, and the fifth sound of each row, i.e., "o" in the "a" row, is assigned to the fourth divided region 242d. The same is true for each row of consonants. The first sound of each row of Japanese vowels "a" row and consonants "ka, sa, ta, na, ha, ma, ya, ra, wa", i.e., "a" in the "a" row, is assigned to the back button 240, as in the example of FIG. 12.

[0075] In the remote controller 20C in Fig. 14, five buttons are used to realize the first and second operators. That is, the right shoulder button 250 is used as the first operator, and the left button 251, the up button 252, the right button 253, and the down button 254 are used as the second operator. The right shoulder button 250 is assigned the first sound of the Japanese vowel "a" row and the consonants "ka, sa, ta, na, ha, ma, ya, ra, wa" row, that is, "a" in the "a" row, the left button 251 is assigned the second sound of each row, that is, "i" in the "a" row, the up button 252 is assigned the third sound of each row, that is, "u" in the "a" row, the right button 253 is assigned the fourth sound of each row, that is, "e" in the "a" row, and the down button 254 is assigned the fifth sound of each row, that is, "o" in the "a" row. The same is true for each row of consonants. The pressing operations of the left button 251, the up button 252, the right button 253, and the down button 254 correspond to a first operation, a second operation, a third operation, and a fourth operation, respectively.

[0076] 15 and 16 are flowcharts showing the flow of processing in the second input mode in the character input system 1. In Fig. 15 and Fig. 16, duplicated explanations will be omitted for steps common to the flow of processing in the first input mode shown in Fig. 7 and Fig. 8.

[0077] When the PC 30 receives a press signal of the back button 240 or the right shoulder button 250 from the remote controller 20 (S21: Yes) while an activated key is present (S05), it confirms the selection of the first sound of a vowel or consonant (S22) and proceeds to step S14.

[0078] On the other hand, when the PC 30 receives a directional operation signal of the stick 241, a touch signal of the trackpad 242, or a pressing signal of the up, down, left, right buttons 251 to 254 from the remote controller 20 (S23: Yes) while an activated key is present (S05), and the back button 240 is not pressed (S21: No), the PC 30 confirms the selection of any of the second to fifth sounds according to the directional operation signal, touch signal, and pressing signal (S10 to S13), and proceeds to step S14. If the PC 30 does not receive any of the pressing signal of the back button 240, the directional operation signal of the stick 241, the touch signal of the trackpad 242, or the pressing signal of the up, down, left, right buttons 251 to 254 (S23: No), the PC 30 proceeds to step S15.

[0079] According to the second input mode, characters can be input by simply moving the fingers, without using a snapping motion of the wrist as in a flick-like operation.

[0080] As a modified example of the second input mode, if a directional operation signal of the stick 241, a touch signal of the trackpad 242, or a press signal of the up, down, left, right buttons 251 to 254 is received (S23: Yes) while a press signal of the back button 249 is still received in step S21, the selection of any one of the second to fifth sounds is confirmed (S10 to S13) according to the directional operation signal, touch signal, or press signal, and the process proceeds to step S14. In this case, for example, if only the press signal of the back button 249 continues for a predetermined time, for example, one second or more, it may be configured to determine that the first sound has been selected.

[0081] As another modification of the second input mode, a remote controller not provided with the back button 240, or even if provided with the back button 240, input may be possible only with the stick 241 or only with the trackpad 242. For example, a configuration may be made in which the first sound is selected when the stick 241 is pressed without tilting it up, down, left, or right (corresponding to the 0 direction input in FIG. 11). Also, a configuration may be made in which the touch surface of the trackpad 242 is divided into five areas shown in FIG. 11, and the first sound is selected when the central area (the 0 direction area in FIG. 11) is touched. In this modification, the determination of S21 in FIG. 15 does not need to be performed.

[0082] [Automatic switching mode] In the above description, only one of the first input mode and the second input mode is valid, but the first input mode and the second input mode may be automatically switched to accept character input. In the case of automatic detection, the input mode is switched to the first input mode when a snap operation (a flick-like operation) is detected, and to the second input mode when a stick operation is detected.

[0083] According to this embodiment, when wearing a wearable display terminal such as an HMD, character input can be performed smoothly by using a remote controller, and operability is improved. For example, compared with gesture operations, gesture operations require the out-camera to recognize fingers. Here, since it is necessary to display keys at a size that does not press adjacent keys even with a finger thicker than the pointer, the display area of ​​the character input UI becomes larger than when a remote controller is used, and there is a restriction that it is difficult to display other virtual reality objects, such as avatars and input fields, in the limited display area of ​​the display 103. In contrast, in this embodiment, a flick-like operation is performed by aligning the pointer with the key using a remote controller, so the key only needs to be large enough to overlap with the pointer, and the display area of ​​the display 103 can be effectively utilized.

[0084] Also, in this embodiment, character selection and input display can be performed simply by changing the orientation of the remote controller so that the pointer is superimposed on the key, and performing a flick-like operation of swinging the remote controller in the desired direction while pressing an operation button on the remote controller, making the operations required for character input simpler than gesture actions and improving the operability of character input.

[0085] The above embodiment merely illustrates one embodiment of the present invention, and the present invention is not limited to the above embodiment. Furthermore, the program according to the present invention is not limited to a single program, but may be a collection of multiple programs. Furthermore, the program according to the present invention is not limited to one executed by a single device, but may be shared and executed by multiple devices.

[0086] For example, the remote controller 20 may be equipped with an infrared emitter, and a receiver may be provided in which a plurality of infrared receivers configured separately from the HMD 10 are arranged. Then, the relative position of the remote controller 20 as viewed from the receiver (referred to as a remote control relative position) is detected based on which infrared receiver mounted on the receiver receives the infrared light emitted from the remote controller 20. Similarly, infrared communication is performed between the receiver and the HMD 10, and the HMD 10 detects the relative position of the receiver. The HMD 10 may be configured to receive the remote control relative position from the receiver, and further add the remote control relative position to the position of the receiver as viewed from the HMD 10, thereby detecting the relative position of the remote controller 20 to the HMD 10 via the receiver, and displaying the relative position with the pointer 330. In this case, the sensor information of the motion sensor 205 of the remote controller 20 may be used only for detecting a flick-like operation.

[0087] Also, although a 12-key character input UI has been given as an example, even in a "101 keyboard" with 101 keys, characters may be input by a flick-like operation in which the remote controller 20 is moved so that the pointer is superimposed on a key, and the operation button 22 is moved at a predetermined acceleration or higher while being pressed down, to input characters. In the case where one alphabetical character is assigned to each key in the 101 keyboard, it is not necessary to detect the direction of movement, and it is sufficient to determine that the character selection operation has been confirmed when a flick-like operation is detected in any direction.

[0088] Furthermore, a part or all of the means realized by the program can be realized by hardware such as an integrated circuit. Furthermore, the program may be provided by being recorded on a non-transitory recording medium that can be read by a computer. Examples of the recording medium include a hard disk, an SD card, a DVD, and a server on the Internet. [Explanation of symbols]

[0089] 1: Character input system, 10: HMD, 11: Display, 12: Control device, 13: HMD side short-range wireless communication device, 19: Holder, 20: Remote controller, 20A: Remote controller, 20B: Remote controller, 20C: Remote controller, 21: Remote control side short-range wireless communication device, 22: Operation button, 29: Housing, 30: PC, 40: Communication network, 50: Virtual reality providing server, 90: Virtual space, 91: Panoramic image, 92: Virtual camera, 93: Reference line of sight, 94: Field of view area, 97: Virtual reality image, 101: Processor, 102: RAM, 103: Display, 104: Storage, 104P: Terminal program, 105: Input interface, 106: Motion sensor, 107: Out camera, 108: Communication interface, 109: Microphone, 110: Speaker, 111: Communication bus, 150: Character input UI program, 151: Pointer control unit, 1 52: Character input UI display control unit, 153: Character input control unit, 160: Virtual reality display program, 170: Communication control unit, 201: Processor, 202: RAM, 203: ROM, 204: Operation switch, 205: Motion sensor, 206: BT communication device, 209: Communication bus, 240: Back button, 241: Stick, 242: Track pad, 242a: First division area, 242b: Second division area, 242c: Third division area, 242d: Fourth division area Division area, 249: back button, 250: right shoulder button, 251: left button, 252: up button, 253: right button, 254: down button, 300: avatar, 301: processor, 302: RAM, 303: ROM, 304: storage, 305: input interface, 306: communication interface, 309: communication bus, 310: avatar registration screen, 311: character input field, 312: cursor, 321: key, 330: pointer, 331: arrow

Claims

1. On the computer, A function of transmitting image data of a character input interface including at least one key for receiving a character input operation to a wearable display terminal; A function of transitioning a key included in the character input interface, on which a pointer linked to the movement of a remote controller is superimposed, to a character selection operation standby mode; and a program for realizing a function of, when an operation signal of the remote controller is received during the character selection operation standby mode, displaying the character assigned to the key on the wearable display terminal based on the operation signal.

2. The program according to claim 1, the remote controller includes a first operator and a second operator; the first operator is an operator that accepts an input operation of on and off, the second operator is an operator that accepts four types of input operations, The character input interface includes at least ten keys; Each key is assigned the first sound of each of the 50 vowel and consonant rows that make up the Japanese alphabet. an input operation of the first operator is an operation of selecting a first sound of a vowel row and a first sound of a consonant row assigned to each key, and a first operation of the four types of input operations of the second operator is an operation of selecting a second sound of a vowel row and a second sound of a consonant row assigned to each key, A second operation of the four types of input operations is an operation of selecting the third sound of the vowel row and the third sound of the consonant row assigned to each key, A third operation among the four types of input operations is an operation for selecting the fourth sound of the vowel row and the fourth sound of the consonant row assigned to each key, A program, wherein a fourth operation of the four types of input operations is an operation of selecting the fifth sound of a vowel row and a fifth sound of a consonant row assigned to each key.

3. The program according to claim 1, the remote controller includes a first operator and a motion sensor that detects a movement of the remote controller; the first operator is an operator that accepts an input operation of on and off, The character input interface includes at least ten keys; Each key is assigned the first sound of each of the 50 vowel and consonant rows that make up the Japanese alphabet. the input operation of the first operator is a selection operation of the first sound of a vowel row and a first sound of a consonant row assigned to each key, and a movement direction of the remote controller is classified into four directions based on the input operation of the first operator and sensor information detected by the motion sensor; The motion operation in the first direction among the four directions is a selection operation of the second sound of the vowel row and the consonant row assigned to each key, the motion operation in the second direction among the four directions is a selection operation of the third sound of the vowel row and the consonant row assigned to each key; the motion operation in the third direction among the four directions is a selection operation of the fourth sound of the vowel row and the consonant row assigned to each key; A program in which the movement operation in the fourth direction among the four directions is a selection operation of the fifth sound of the vowel row and the consonant row assigned to each key.

4. A storage medium storing the program according to claim 1, 2 or 3.

5. A character input system, comprising: A computer storing the program according to claim 1, 2 or 3; A wearable display terminal; A remote controller, The computer receives the operation signal from the remote controller, Transmitting image data of the character input interface and image data of the character assigned to the key selected in accordance with an operation signal of the remote controller from the computer to the wearable display terminal; The wearable display terminal displays a character assigned to the key selected in accordance with an operation signal of the character input interface and the remote controller.

6. The computer generating image data of a character input interface including at least one key for receiving a character input operation, and transmitting the image data to a wearable display terminal; transitioning a key on which a pointer linked to the movement of a remote controller is superimposed, among the keys included in the character input interface, to a character selection operation standby mode; when an operation signal of the remote controller is received during the character selection operation standby mode, the character assigned to the key is displayed on the wearable display terminal based on the operation signal.

Images