Information processing device

Abstract

To provide an information processing device which appropriately arranges a virtual key according to physical features of a user U.SOLUTION: In a wearable device comprising an AR glass and a terminal device, a terminal device 20 which is an information processing device comprises: an acquisition unit 214 which acquires body information indicating features of a user's entire body and measurement information measured by a first imaging device etc. mounted in the AR glass worn by the user; a determination unit 216 which determines key information indicating the area of a virtual key virtually arranged around the user, on the basis of the body information and the measurement information; and a detection unit 218 which detects that the virtual key was operated, on the basis of the key information and the measurement information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] XR glasses that apply XR (Cross Reality) technologies, such as AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality), have become widespread. A virtual keyboard is used as an input interface for XR glasses. For example, Patent Document 1 below discloses a virtual keyboard input method that uses a camera to accurately detect key input actions of both hands, even if the position of the user's hands moves. Specifically, an information processing device thins images of the user's left and right hands and determines the fingertip positions from the endpoints of the images. The information processing device acquires the branch point positions and fingertip positions as feature points by tracking changes in the images of the left and right hands, calculating the length and inclination angle from the branch point positions to the fingertip positions, and drawing rectangular frames along each finger. The information processing device draws each rectangular frame so that at least a portion of the fingertip extends outside the frame when the hand is not bent. The information processing device detects whether the fingertip position of each finger is inside or outside each rectangular frame based on the presence or absence of skin-color pixels of the finger outside the short side of the fingertip side drawn in the approximate finger width direction of each rectangular frame. The information processing device estimates the input key from the fingertip position of the finger whose fingertip position is within the rectangular frame and each input key area of ​​the virtual keyboard. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-165660 Summary of the Invention [Problem to be solved by the invention]

[0004] The aforementioned Patent Document 1 improves the accuracy of keyboard input by changing the key layout of a virtual keyboard using skeletal information of a user's hands. Meanwhile, various forms of virtual input interfaces other than a virtual keyboard can be adopted in XR glasses. Therefore, it is desirable to realize a virtual key layout suitable for each user in various forms of virtual input interfaces.

[0005] An object of the present invention is to appropriately arrange virtual keys in accordance with the physical characteristics of a user. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present invention includes an acquisition unit that acquires physical information indicating the characteristics of a user's entire body and measurement information measured by a sensor worn by the user, a determination unit that determines key information indicating an area of ​​virtual keys virtually arranged around the user based on the physical information and the measurement information, and a detection unit that detects that the virtual keys have been operated based on the key information and the measurement information. [Effects of the Invention]

[0007] According to one aspect of the present invention, virtual keys can be appropriately arranged to suit the physical characteristics of a user. [Brief explanation of the drawings]

[0008] [Figure 1A] 1 is a diagram schematically showing a user U using a wearable device 1 according to an embodiment. [Figure 1B] FIG. 1 is a block diagram showing the configuration of a wearable device 1. [Figure 2] 1 is an explanatory diagram showing the appearance of AR glasses 10. FIG. [Figure 3] FIG. 2 is a block diagram showing the configuration of the AR glasses 10. [Figure 4] FIG. 2 is a block diagram showing the configuration of a terminal device 20. [Figure 5]FIG. 2 is a diagram showing an example of a physical information input screen PC1. [Figure 6] FIG. 10 is a diagram illustrating a method for setting an initial region S0[i] based on physical information. [Figure 7] FIG. 10 is a diagram showing an example of a virtual key layout image PC2-1. [Figure 8] FIG. 10 is a diagram showing an example of a virtual key layout image PC2-2. [Figure 9] FIG. 10 is a diagram showing an input method in the virtual input interface VI[1]. [Figure 10] FIG. 10 is a diagram showing an input method in the virtual input interface VI[1]. [Figure 11] FIG. 10 is a diagram showing an example of a virtual key layout image PC2-3. [Figure 12] 10 is a diagram schematically illustrating the relationship between the posture of a user U and the range in which a virtual key VK[i] is arranged. FIG. [Figure 13A] 10 is a diagram schematically illustrating the relationship between the environment around the user U and the range in which the virtual keys VK[i] are arranged. FIG. [Figure 13B] 1 is a diagram schematically illustrating the visual field range of a user U. FIG. [Figure 14] FIG. 10 is a diagram showing a virtual key VK[i] reduced in size. [Figure 15] FIG. 10 is an explanatory diagram showing another example of a method for correcting key information IK[i]. [Figure 16] 10 is a flowchart showing the operation of the processing unit 206 at the time of initializing the virtual input interface VI[m]. [Figure 17] 10 is a flowchart showing the operation of the processing unit 206 when using the virtual input interface VI[m]. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. Embodiment The configuration of a wearable device 1 including an information processing device according to an embodiment of the present invention will be described below.

[0010] A-1. System Configuration Fig. 1A is a diagram schematically showing a user U using a wearable device 1 according to an embodiment. Fig. 1B is a block diagram showing the configuration of the wearable device 1. The wearable device 1 includes AR glasses 10 and a terminal device 20. The terminal device 20 is an example of an information processing device.

[0011] In this embodiment, the wearable device 1 is a system that uses AR technology to present various types of information to a user U wearing AR glasses 10. Here, AR technology refers to a technology that allows a user U wearing a device such as a see-through head-mounted display to visually recognize information indicated by a virtual object by superimposing the virtual object on real space. Note that the AR glasses 10 may display virtual objects and also output other types of information, such as audio information.

[0012] The AR glasses 10 and the terminal device 20 are connected to each other so that they can communicate with each other. The terminal device 20 is connected to, for example, an information providing server (not shown) so that they can communicate with each other, and acquires information to be output by the AR glasses 10 from the information providing server. The terminal device 20 also outputs the information acquired from the information providing server from the AR glasses 10. The terminal device 20 is preferably a mobile terminal device such as a smartphone or a tablet.

[0013] A-2.AR Glasses 10 The AR glasses 10 are a see-through head-mounted display worn on the head of a user U. The AR glasses 10 display virtual objects on display panels provided on both lenses 110A and 110B for the eyes, based on the control of the terminal device 20. Note that, for example, VR glasses that block vision from the outside world may also be used as the display device.

[0014] 2 is an explanatory diagram showing the external appearance of the AR glasses 10. The AR glasses 10 have temples 102A and 102B, a bridge 103, frames 104A and 104B, rims 108A and 108B, lenses 110A and 110B, an arm 112, imaging lenses 120A and 120B, and light transmitting and receiving units 122A and 122B that can be seen from the outside.

[0015] 3, an imaging lens 120A constituting the first imaging device 134A and a light transmitting / receiving unit 122A constituting the first scanner 135A are arranged on the bridge 103. An image of an area in front of the user U wearing the AR glasses 10 is captured from the imaging lens 120A arranged on the bridge 103. Information about an object located in front of the user U wearing the AR glasses 10 is acquired from the light transmitting / receiving unit 122A arranged on the bridge 103.

[0016] The arm 112 is provided with an imaging lens 120B constituting the second imaging device 134B shown in FIG. 3 and a light transmitting / receiving unit 122B constituting the second scanner 135B. More specifically, the arm 112 has a rectangular parallelepiped shape having long and short sides, with one end of the long side connected to the rim 108B and the other end of the long side protruding in front of the lens 110B. The imaging lens 120B and the light transmitting / receiving unit 122B are located on a lower surface 112N of the arm 112. Therefore, the imaging lens 120B located on the arm 112 captures an image of an area below the eyes of a user U wearing the AR glasses 10. Furthermore, the light transmitting / receiving unit 122B located on the arm 112 acquires information about an object located in an area below the eyes of a user U wearing the AR glasses 10.

[0017] In this embodiment, the arm 112 is disposed on the right eye side of the user U, but it may also be disposed on the left eye side, or in the center between the right and left eyes. The arm 112 may also be disposed on both the right and left eye sides of the user U. In other words, the arm 112 may be disposed in any position as long as the movement of the user U's hand H, the presence or absence of objects around the user U, etc. can be detected using the second imaging device 134B and the second scanner 135B described below.

[0018] The frame 104A is provided with a display panel for the left eye and an optical member for the left eye. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The display panel for the left eye displays an image based on control from, for example, the terminal device 20. The optical member for the left eye is an optical member that guides light emitted from the display panel for the left eye to the lens 110A. The frame 104A is also provided with a speaker 132, which will be described later.

[0019] The frame 104B is provided with a display panel for the right eye and an optical member for the right eye. The display panel for the right eye displays an image, for example, under control of the terminal device 20. The optical member for the right eye is an optical member that guides light emitted from the display panel for the right eye to the lens 110B. The frame 104B is also provided with a speaker 132, which will be described later.

[0020] The rim 108A holds the lens 110A, and the rim 108B holds the lens 110B. Note that the optical members may be provided on the rims 108A and 108B instead of the frames 104A and 104B.

[0021] Each of lenses 110A and 110B has a half mirror. The half mirror of lens 110A transmits light representing real space, thereby guiding the light representing real space to the left eye of user U. The half mirror of lens 110A also reflects light guided by an optical member for the left eye to the left eye of user U. The half mirror of lens 110B transmits light representing real space, thereby guiding the light representing real space to the right eye of user U. The half mirror of lens 110B also reflects light guided by an optical member for the right eye to the right eye of user U.

[0022] 3 is a block diagram showing the configuration of the AR glasses 10. In addition to the temples 102A and 102B, bridge 103, frames 104A and 104B, rims 108A and 108B, lenses 110A and 110B, imaging lenses 120A and 120B, and light transmitting and receiving units 122A and 122B described above, the AR glasses 10 also include a projection device 131, a speaker 132, a communication device 133, a first imaging device 134A, a second imaging device 134B, a first scanner 135A, a second scanner 135B, a storage device 140, a processing device 142, and a bus 150. The first imaging device 134A, the second imaging device 134B, the first scanner 135A, and the second scanner 135B are examples of sensors worn by the user U.

[0023] 3 are stored in frames 104A and 104B, for example. The projection device 131, speaker 132, communication device 133, first image capture device 134A, second image capture device 134B, first scanner 135A, second scanner 135B, storage device 140, and processing device 142 are connected to one another by a bus 150 for communicating information. The bus 150 may be configured using a single bus, or may be configured using different buses between each element of the device, etc.

[0024] The projection device 131 includes a lens 110A, a display panel for the left eye, an optical member for the left eye, a lens 110B, a display panel for the right eye, and an optical member for the right eye. Light representing real space passes through the projection device 131. The projection device 131 displays an image based on control from the terminal device 20. In this embodiment, the image displayed by the projection device 131 is, for example, a moving image based on a video viewing application, an image for initial setting of the virtual input interface VI, etc.

[0025] The virtual input interface VI is an interface that accepts operations by a user U in a virtual space. The virtual input interface VI has virtual keys VK arranged in the virtual space, as shown in Fig. 7, for example. When the hand H of the user U is positioned in an area in the real space that corresponds to the area in the virtual space where the virtual keys VK are arranged, the operation is accepted as an operation on the virtual keys VK.

[0026] When the user U wears the AR glasses 10, the projection device 131 is located in front of the left and right eyes of the user U. The projection device 131 is an example of a display device placed in front of the eyeballs of the user U. The user U wearing the AR glasses 10 can visually recognize the real space represented by the light transmitted through the projection device 131 and the image projected by the projection device 131 superimposed on each other.

[0027] The speaker 132 is located on each of the frames 104A and 104B. The speaker 132 may not be located on each of the frames 104A and 104B, but may be located on, for example, one of the frames 104A and 104B, at least one of the temples 102A and 102B, or the bridge 103. The speaker 132 is controlled by the terminal device 20 directly or via a processing device 142 of the AR glasses 10. The speaker 132 may not be included in the AR glasses 10 and may be separate from the AR glasses 10.

[0028] The communication device 133 communicates with the terminal device 20 using wireless communication or wired communication. In this embodiment, the communication device 133 communicates with the communication device 202 (see FIG. 4) of the terminal device 20 using short-range wireless communication such as Bluetooth (registered trademark).

[0029] The first imaging device 134A and the second imaging device 134B capture images of a subject and output captured image information indicating the captured images. The captured images generated by the first imaging device 134A and the second imaging device 134B are transmitted as captured image information to the terminal device 20 via the communication device 133. Hereinafter, the image captured by the first imaging device 134A will be referred to as a first image, and the captured image information indicating the first image will be referred to as first image information. Also, the image captured by the second imaging device 134B will be referred to as a second image, and the captured image information indicating the second image will be referred to as second image information. The first imaging device 134A and the second imaging device 134B repeat imaging at a predetermined imaging interval, and transmit the generated first image information and second image information to the terminal device 20 each time an image is captured.

[0030] The first imaging device 134A and the second imaging device 134B each have, for example, an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens (imaging lens 120A in the first imaging device 134A, and imaging lens 120B in the second imaging device 134B (see FIG. 2)). For example, the imaging optical system may have various optical elements such as a prism, or may have a zoom lens or a focus lens. The imaging element is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

[0031] The imaging range of the first imaging device 134A includes, for example, the field of view of the user U. Therefore, the first image information includes information on the appearance of objects present in the field of view of the user U. For example, when the user U is on a train, images of other passengers, seats, train doors, etc. positioned in front of the user U are captured.

[0032] The imaging range of the second imaging device 134B includes, for example, the body of the user U (more specifically, the part of the body of the user U below the eyes) and an area adjacent to the body of the user U outside the field of view of the user U. Therefore, the second image information includes information on the movement of the body of the user U and the appearance of objects located near the body of the user U. For example, when the user U is riding on a train, an image is captured of the part of the body of the user U, mainly from the torso below the neck to the feet, as well as other passengers adjacent to the user U in the horizontal direction, seats, train doors, etc.

[0033] The first scanner 135A and the second scanner 135B output shape information indicating the results of measuring the three-dimensional shape of an object near the user U. The first scanner 135A and the second scanner 135B may be, for example, LiDAR scanners. The shape information measured by the first scanner 135A is referred to as first shape information, and the shape information measured by the second scanner 135B is referred to as second shape information. The first scanner 135A and the second scanner 135B transmit the first shape information and the second shape information to the terminal device 20 via the communication device 133. The first scanner 135A and the second scanner 135B repeat measurements at predetermined measurement intervals and transmit the generated first shape information and second shape information to the terminal device 20 each time a measurement is performed.

[0034] In this embodiment, the first scanner 135A and the second scanner 135B irradiate the measurement range with infrared light from a light source provided in the light-transmitting and receiving units 122A and 122B. The infrared light irradiated from the light source is pattern light having a pattern such as a stripe pattern. The first scanner 135A and the second scanner 135B receive the reflected light using a light-receiving unit provided in the light-transmitting and receiving units 122A and 122B. The time from irradiating the light to receiving the reflected light is proportional to the distance to the object from which the light is reflected. The first scanner 135A and the second scanner 135B measure the time from receiving the light for each point within the measurement range and calculate the distance to each point within the measurement range. The unevenness of the object is also recognized based on the distortion of the pattern reflected on the object's surface. In this way, the first scanner 135A and the second scanner 135B can obtain the three-dimensional coordinates of the object's surface and generate shape information indicating the three-dimensional shape of the object. In this embodiment, the shape information includes distance information indicating the distance to the object.

[0035] The measurement range of the first scanner 135A is, for example, a range that includes the field of view of the user U. Therefore, the first shape information includes information on the shape of objects present in the field of view of the user U. For example, when the user U is on a train, the shapes of other passengers, seats, train doors, etc. positioned in front of the user U are measured.

[0036] The measurement range of the second scanner 135B includes, for example, the body of the user U (more specifically, the part of the body of the user U below the eyes) and an area adjacent to the body of the user U outside the field of view of the user U. Therefore, the second shape information includes information on the shape of the body of the user U and the shape of objects located near the body of the user U. For example, when the user U is riding on a train, the shapes of the part of the body of the user U, mainly from the torso below the neck to the feet, other passengers adjacent to the user U in the horizontal direction, seats, train doors, etc. are measured.

[0037] The storage device 140 is a recording medium that can be read by the processing device 142. The storage device 140 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM). The storage device 140 stores a program PG1. The program PG1 is a program for operating the AR glasses 10.

[0038] The processing device 142 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.

[0039] The processing device 142 reads the program PG1 from the storage device 140. The processing device 142 executes the program PG1 to function as the operation control unit 160. The operation control unit 160 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0040] The operation control unit 160 controls the operation of the AR glasses 10. For example, the operation control unit 160 provides the projection device 131 with a control signal for image display that the communication device 133 has received from the terminal device 20. The projection device 131 displays an image indicated by the control signal for image display. The operation control unit 160 also provides the speaker 132 with a control signal for audio output that the communication device 133 has received from the terminal device 20. The speaker 132 emits a sound indicated by the control signal for audio output. The operation control unit 160 also transmits, to the terminal device 20, first image information captured by the first imaging device 134A, second image information captured by the second imaging device 134B, first shape information measured by the first scanner 135A, and second shape information measured by the second scanner 135B using the communication device 133.

[0041] A-3. Terminal device 20 4 is a block diagram showing the configuration of terminal device 20. Terminal device 20 includes a touch panel 201, a communication device 202, a storage device 204, a processing device 206, and a bus 208. Touch panel 201, communication device 202, storage device 204, and processing device 206 are interconnected by bus 208 for communicating information. Bus 208 may be configured using a single bus, or may be configured using different buses for each device.

[0042] The touch panel 201 displays various information to the user U and detects touch operations by the user U. The touch panel 201 serves as both an input device and an output device. For example, the touch panel 201 is configured by bonding a touch sensor unit capable of detecting touch operations between a cover glass and a display panel such as a liquid crystal display panel or an organic EL display panel. For example, when the user U's finger is in contact with the touch panel 201, the touch panel 201 periodically detects the contact position of the user U's finger on the touch panel 201 and transmits touch information indicating the detected contact position to the processing device 206.

[0043] The communication device 202 communicates with the AR glasses 10 using wireless communication or wired communication. In this embodiment, the communication device 202 communicates with the communication device 133 (see FIG. 3 ) using short-range wireless communication of the same type as that used by the communication device 133 of the AR glasses 10. The communication device 202 also communicates with an information providing server (not shown) using wireless communication or wired communication.

[0044] The storage device 204 is a recording medium readable by the processing device 206. The storage device 204 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 204 stores a program PG2, reference information IS[X], and key information IK[X]. The program PG2 is a program for operating the terminal device 20. The reference information IS and key information IK are information regarding the placement of the virtual key VK. The reference information IS[X] includes reference information IS[i] (i is any integer satisfying 1≦i≦K). Furthermore, the key information IK[X] includes key information IK[i].

[0045] The processing unit 206 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

[0046] The processing device 206 reads the program PG2 from the storage device 204. By executing the program PG2, the processing device 206 functions as an application processing unit 210, a display control unit 212, an acquisition unit 214, a determination unit 216, and a detection unit 218. At least one of the application processing unit 210, the display control unit 212, the acquisition unit 214, the determination unit 216, and the detection unit 218 may be configured by a circuit such as a DSP, an ASIC, a PLD, or an FPGA.

[0047] The application processing unit 210 processes applications executed by the terminal device 20. Examples of applications executed by the terminal device 20 include a video viewing application, a social network service application, a message sending / receiving application, a voice call application, and an internet browser. In this embodiment, the application processing unit 210 outputs information obtained as a result of executing the application to the user U using the AR glasses 10 (in many cases, as a display output and an audio output).

[0048] The application processing unit 210 may communicate with the information providing server described above and send and receive information as the application is executed. For example, if the application is a video viewing application, the information providing server is a video storage server. For example, when the user U launches the video viewing application, the application processing unit 210 requests initial screen data of the video viewing application from the video storage server. Upon receiving the initial screen data from the video storage server, the application processing unit 210 controls the projection device 131 of the AR glasses 10 using a display control unit 212 (described later) to display the initial screen for the user U. When the user U inputs a predetermined keyword on the initial screen, the application processing unit 210 transmits the input keyword to the video storage server. The keyword is input using, for example, a virtual input interface VI (described later). The video storage server creates a list of videos corresponding to the transmitted keyword and transmits the list of videos to the terminal device 20 as screen data showing the list of videos. In this way, the application processing unit 210 processes the application while transmitting and receiving information to and from the video storage server.

[0049] The display control unit 212 causes the projection device 131, which is placed in front of the eyeballs of the user U, to display an image showing a virtual space. For example, the display control unit 212 causes the projection device 131 to display an image related to an application acquired by the application processing unit 210. Taking the above-mentioned video viewing application as an example, the display control unit 212 causes the projection device 131 to display an initial screen of a video storage server. Furthermore, the display control unit 212 causes the projection device 131 to display a screen showing a list of videos corresponding to a keyword. Furthermore, the display control unit 212 causes the projection device 131 to display an image showing a setting screen of a virtual input interface VI, which will be described later.

[0050] Here, when an application is executed, input may be made by the user U. For example, in the case of the video viewing application described above, input of keywords for searching for videos, instructions to start and stop video playback, input of ratings for the videos, etc. may be performed. When making these inputs, a method is available in which a virtual keyboard or virtual input buttons are displayed in the user U's field of view. On the other hand, for example, for a user U who is accustomed to touch typing, displaying a virtual keyboard may be unnecessary because it narrows the field of view. Furthermore, for example, there are only a few types of virtual input buttons for starting and stopping video playback, and their placement is generally the same. Therefore, it may be more convenient for the user U to operate them at hand rather than displaying virtual input buttons in the field of view.

[0051] For this reason, in this embodiment, when receiving input from the user U during execution of an application, a virtual input interface VI is set at a position close to the user U's hand. The virtual input interface VI includes K (K is any integer equal to or greater than 1) virtual keys VK. In this embodiment, in order to distinguish between the K virtual keys VK, the i-th virtual key VK is referred to as virtual key VK[i], where i is any integer satisfying 1≦i≦K. Furthermore, there may be multiple virtual input interfaces VI depending on the application. In the following description, in order to distinguish between multiple virtual input interfaces VI, one type of virtual input interface VI is referred to as virtual input interface VI[m], where m is any integer equal to or greater than 1.

[0052] The acquiring unit 214, the determining unit 216, and the detecting unit 218 are functional units related to the virtual input interface VI[m]. Below, the acquiring unit 214, the determining unit 216, and the detecting unit 218 will be described, and the virtual input interface VI[m] will be described in detail.

[0053] A-4. Initial setting of virtual input interface VI[m] First, the initial setting of the virtual input interface VI[m] will be described. The user U performs settings related to the virtual input interface VI[m] when the AR glasses 10 are initially set, for example.

[0054] The acquisition unit 214 acquires physical information indicating the characteristics of the entire body of the user U, and measurement information measured by the first imaging device 134A, the second imaging device 134B, the first scanner 135A, and the second scanner 135B mounted on the AR glasses 10 worn by the user U. In this embodiment, the physical information does not indicate the characteristics of parts of the user U's body, but indicates information about the entire body.

[0055] The physical information is, for example, the height and weight values ​​of the user U. The acquisition unit 214 acquires the physical information by, for example, receiving input of the height and weight values ​​from the user U. The physical information may also be keywords that indicate the overall physical characteristics of the user U, such as "stocky (muscular)," "plump (chubby)," or "slim (slim)."

[0056] The measurement information includes at least one of the first image captured by the first imaging device 134A or the second image captured by the second imaging device 134B, the first shape information detected by the first scanner 135A, and the second shape information detected by the second scanner 135B. During initial setting, of the measurement information acquired by the acquisition unit 214, at least one of the first image captured by the first imaging device 134A or the second image captured by the second imaging device 134B is used.

[0057] 5 is a diagram showing an example of a physical information input screen PC1. The acquisition unit 214 causes the projection device 131 to display the input screen PC1. The input screen PC1 displays a height input field 501, a weight input field 502, and a characteristic input field 503. The height input field 501, the weight input field 502, and the characteristic input field 503 are associated with positions in real space, and when the hand H of the user U moves to a position corresponding to each input field, it is detected as an operation on that field. The position of the hand H of the user U is detected based on the first image or the second image, which is measurement information.

[0058] When the user U taps a position corresponding to the height input field 501 or the weight input field 502, a virtual keyboard 504 for entering numbers appears. The user U enters a value corresponding to their height or weight in each field. In addition, keywords indicating overall body characteristics are displayed in the characteristic input field 503. If there is a keyword that matches their body type, the user U taps to select that keyword. When the input is complete, the user U presses the OK button 505 to end the input on the input screen PC1.

[0059] Additionally, the acquiring unit 214 may input, as the physical information, for example, a photograph showing the entire body of the user U. By using a photograph, the characteristics of the entire body of the user U can be grasped in more detail and more reliably.

[0060] The determination unit 216 generates reference information IS[i] indicating a reference area, which is an area in the virtual space where the virtual key VK[i] is placed. In this embodiment, the virtual key VK[i] is defined as an area having an area or volume. The determination unit 216 determines the coordinates of the reference area occupied by the virtual key VK[i] in a coordinate system in the virtual space, for example, and records the coordinates as reference information IS[i].

[0061] More specifically, the determination unit 216 calculates initial information indicating an initial area S0[i] of the virtual key VK[i] based on the physical information. The initial area S0[i] is an initial value of an area in which the virtual key VK[i] is arranged in the virtual space. The determination unit 216 generates an image of the virtual space in which the virtual key VK[i] is arranged in the initial area S0[i] indicated by the initial information (hereinafter referred to as a "virtual key arrangement image") PC2, and displays the virtual key arrangement image PC2 to the user U using the projection device 131. The user U can adjust the arrangement of the virtual key VK[i] by performing a predetermined operation on the virtual key arrangement image PC2. The determination unit 216 detects an adjustment operation of the arrangement of the virtual key VK[i] by the user U based on measurement information such as the second image. The determination unit 216 generates reference information IS[i] indicating a reference area, which is the area of ​​the virtual key VK[i] after the arrangement has been adjusted.

[0062] Fig. 6 is a diagram showing a method for setting the initial region S0[i] based on physical information. Fig. 6 shows a first user U1 and a second user U2 as examples of users U. The first user U1 is relatively taller and heavier than the second user U2. The second user U2 is relatively shorter and lighter than the first user U1.

[0063] The initial region S0[i]-1, which is the initial region S0[i] corresponding to the first user U1, is set on a plane F1 at a distance R1 below the elbow position N1 of the first user U1 (more specifically, the predicted value of the elbow position). The reason for this is that when the first user U1 operates the virtual key VK[i], it is assumed that the first user U1 will bend their elbow, extend their forearm forward, and make an input by touching the virtual key VK[i] with their hand H. The distance R1 is determined based on the height of the first user U1.

[0064] More specifically, the distance R1 is set to be longer as the height of the first user U1 increases. It is also predicted that the distance between the position of the AR glasses 10 (the position of the eyes of the first user U1) and the elbow position N1 increases as the height increases. Therefore, the distance L1 between the AR glasses 10 and the plane F1 is set to be longer as the height increases.

[0065] Furthermore, the width W1 of the initial region S0[i]-1 (the distance along the extension direction of the shoulder width of the first user U1) is set to be longer in proportion to the weight of the first user U1. The reason for this is that the heavier the weight of the first user U1, the wider the shoulder width and the wider the range of motion of the hand H are predicted to be.

[0066] The initial region S0[i]-2 corresponding to the second user U2 can be calculated in the same way as the initial region S0[i]-1. Comparing the distance L1 of the first user U1 with the distance L2 of the second user U2, distance L1 > distance L2. Furthermore, comparing the width W1 of the first user U1 with the width W2 of the second user U2, width W1 > width W2.

[0067] In this embodiment, the virtual key VK[i] is placed in a position that is not visible to the user U when the user U's head is facing forward. In other words, the virtual key VK[i] is placed outside the field of view SN of the user U. FIG. 13B is a diagram schematically showing the field of view range SI of the user U. The field of view range SI is the range that the user U can see. The field of view range SI includes, for example, the central field of view, the effective field of view, and the peripheral field of view. In this case, the area outside the central field of view, the effective field of view, and the peripheral field of view is the outside field of view SN. Furthermore, for example, an area of ​​the peripheral field of view where the user U's discrimination ability is significantly reduced (generally an area far from the central field of view) may be included in the outside field of view SN.

[0068] The visual field range SI has a spread centered on the line of sight LI of the user U. The visual field range SI becomes wider the farther away from the user U. On the other hand, for example, the user U's hands are outside the visual field SN when the user U's head is facing forward. The virtual key VK[i] is located outside the visual field SN.

[0069] The reason for this is to place the virtual key VK[i] in a position that does not overlap the display area of ​​the virtual object. The projection device 131 displays the virtual object in the field of view SI of the user U. If the virtual key VK[i] is placed in a position that overlaps the display area of ​​the virtual object, the user U will need to move his / her hand H into the display area to operate it, which will reduce the visibility of the virtual object. In this embodiment, to ensure the visibility of the virtual object, the virtual key VK[i] is placed outside the field of view SN of the user U. Note that when the user U's head is facing forward, this means, for example, a state in which the user U is not shaking their head up and down or left and right. "When the user U's head is facing forward" refers to a state that is suitable for viewing content, for example.

[0070] In this embodiment, the number of virtual keys VK[i] in one virtual input interface VI[m] is smaller than that of a keyboard attached to a personal computer, for example. Therefore, even if the user U is not skilled in keyboard operation, there is a high possibility that the user U can operate the virtual keys VK[i] without looking at them.

[0071] Furthermore, in this embodiment, the virtual input interface VI[m] is not generally displayed in the virtual space. This is because it is assumed that the user U will not generally direct his or her gaze toward the virtual keys VK[i] when inputting data into the virtual input interface VI[m]. For example, the virtual input interface VI[m] may be configured to display or hide the virtual keys VK[i]. For example, if the user U's face changes from facing forward to facing toward his or her hands while inputting data into the virtual input interface VI[m], the projection device 113 may display a display corresponding to the virtual keys VK[i]. This is because it is expected that if the user U's face changes toward his or her hands while inputting data into the virtual input interface VI[m], he or she will want to check the arrangement of the virtual keys VK[i].

[0072] FIG. 7 is a diagram showing an example of a virtual key layout image PC2-1. In the following figures, parentheses indicating subscripts are omitted for ease of visual recognition. The virtual key layout image PC2-1 is an example of a virtual key layout image PC2. As described above, the virtual input interface VI[m] is not generally displayed in the virtual space, but when checking the layout of the virtual keys VK[i], it is displayed in the virtual space as the virtual key layout image PC2.

[0073] The virtual key layout image PC2 is displayed superimposed on the real space. For example, the hand H and foot M of the user U are objects located in the real space. The virtual key layout image PC2-1 displays a virtual input interface VI[1] for character input. The virtual input interface VI[1] displays virtual keys VK[1] to VK

[10] corresponding to the hiragana vowel "a," a virtual key VK

[11] corresponding to the operation of deleting the input character, and a virtual key VK

[12] corresponding to the operation of confirming the input character.

[0074] The area occupied by the virtual keys VK[1] to VK

[12] in the virtual space shown in FIG. 7 corresponds to the initial area S0[1] to S0

[12] . The user U visually checks the virtual key layout image PC2-1 to confirm the layout of the virtual keys VK[1] to VK

[12] in the virtual space. If the layout is appropriate, the user U touches the OK button B1. When the OK button B1 is pressed, the area in the virtual space occupied by the virtual key VK[i] at that time becomes the reference area and is stored as reference information IS[i].

[0075] Furthermore, if the user U wishes to adjust the arrangement of the virtual keys VK[1] to VK

[12] , the user U touches the edit button B2. Touching the edit button B2 transitions to an adjustment screen where the positions of the virtual keys VK[1] to VK

[12] on the virtual key arrangement image PC2-1 can be adjusted. Any adjustment method can be used on the adjustment screen. For example, if the user U touches the virtual key VK[i] that the user U wants to move for a predetermined period of time or longer, the virtual key VK[i] is selected. The user U can move the selected virtual key VK[i] three-dimensionally in a desired direction in the virtual space. After the user U moves the virtual key VK[i] to the desired position, the user U touches the OK button B1 to confirm the reference area corresponding to the virtual key VK[i] and store it as reference information IS[i].

[0076] For example, the virtual key layout image PC2-2 shown in FIG. 8 is an example in which the positions of the virtual keys VK[1] to VK

[12] have been changed relative to the virtual key layout image PC2-1 shown in FIG. 7. In FIG. 8, the positions of the virtual keys VK[1] to VK

[12] of the virtual input interface VI[1] have been changed from the initial areas S0[1] to S0

[12] . Specifically, in the virtual key layout image PC2-2 shown in FIG. 8, the virtual keys VK

[11] and VK

[12] are positioned closer to the hand H of the user U. When the OK button B1 is pressed, the changed areas of the virtual keys VK[1] to VK

[12] are confirmed as the reference area and recorded as reference information IS[1] to IS

[12] .

[0077] Note that the number of virtual input interfaces VI[m] may not be limited to one, but may be multiple. FIG. 11 shows an example of a virtual key layout image PC2-3. The virtual key layout image PC2-3 displays a virtual input interface VI[2] for watching videos. The virtual input interface VI[2] displays a virtual key VK

[21] for starting or pausing video playback, a virtual key VK

[22] for fast-forwarding the video being watched by 10 seconds, a virtual key VK

[23] for continuously fast-forwarding the video being watched, a virtual key VK

[24] for rewinding the video being watched by 10 seconds, and a virtual key VK

[25] for continuously rewinding the video being watched. The arrangement of the virtual keys VK[i] in the virtual input interface VI[2] can also be adjusted based on the user U's operation.

[0078] By using the virtual input interface VI[2], input can be easily performed while watching videos. In other words, since the virtual input interface VI[m] is changed according to the application, input according to each application can be performed quickly, improving the operability of the wearable device 1.

[0079] The virtual keys VK[1] to VK

[12] that make up the virtual input interface VI[1] are an example of a plurality of first virtual keys, and the virtual keys VK

[21] to VK

[25] that make up the virtual input interface VI[2] are an example of a plurality of second virtual keys.

[0080] A-5. Use of virtual input interface VI[m] (1) Normal usage Next, the use of the virtual input interface VI[m] will be described in detail. When the application processing unit 210 executes an application, the processing unit 206 puts the virtual input interface VI[m] into a standby state. The determination unit 216 determines key information IK[i] indicating the area of ​​the virtual key VK[i] virtually placed around the user U based on physical information and measurement information. More specifically, the determination unit 216 determines the key information IK[i] based on reference information IS[i] generated based on the physical information and measurement information. The area of ​​the virtual key VK[i] refers to the area SR[i] in real space corresponding to the area in virtual space (reference area) where the virtual key VK[i] is placed. More specifically, when the virtual key VK[i] is placed in virtual space based on the reference information IS[i], the determination unit 216 identifies the area SR[i] in real space that overlaps with the virtual key VK[i] (reference area). Information indicating the area SR[i] in real space becomes the key information IK[i].

[0081] The detection unit 218 detects that the virtual key VK[i] has been operated based on the key information IK[i] and the measurement information. The detection unit 218 detects that the virtual key VK[i] has been operated when the position of the user U's hand H calculated based on the measurement information comes into contact with the area of ​​the virtual key VK[i] indicated by the key information IK[i] (the area SR[i] in real space). As described above, the position of the user U's hand H is detected based on the first image or the second image, which is the measurement information.

[0082] For example, for the virtual input interface VI[2], when the position of the user U's hand H overlaps one of the virtual keys VK

[21] to VK

[25] , the detection unit 218 detects that one of the virtual keys VK

[21] to VK

[25] has been operated.

[0083] The virtual input interface VI[1] is an interface for inputting characters in hiragana, but there are only 10 virtual keys VK[1] to VK

[10] corresponding to the approximately 50 hiragana characters. Therefore, the input method of the virtual input interface VI[1] is different from that of the virtual input interface VI[2].

[0084] Specifically, for example, when the user U inputs the character "ki," as shown in FIG. 9, the user U moves his / her hand H to the virtual key VK[2] corresponding to "ka." The reason for keeping the hand H in a clenched state is to prevent erroneous detection of an operation. For example, to select one of the virtual keys VK[6] to VK

[10] located at the back side from the user U's perspective, the user U needs to extend his / her hand H beyond the virtual keys VK[1] to VK[5] located at the front side. At this time, the detection unit 218 may erroneously detect that the virtual keys VK[1] to VK[5] have been operated. To prevent this, when operating the virtual keys VK[1] to VK

[10] , the user U moves his / her hand H to one of the virtual keys VK[1] to VK

[10] while keeping his / her hand H clenched.

[0085] When an operation on the virtual key VK[2] corresponding to "ka" is detected, the display control unit 212 displays the virtual key VK

[31] corresponding to "a", the virtual key VK

[32] corresponding to "i", the virtual key VK

[33] corresponding to "u", the virtual key VK

[34] corresponding to "e", and the virtual key VK

[35] corresponding to "o" around the virtual key VK[2]. At this time, the virtual keys VK[1], VK[3] to VK

[10] do not need to be displayed. The user U points their fingertip in the direction of the virtual keys VK

[31] to VK

[35] corresponding to the vowel of the character they wish to input. When they wish to input "ki", the user U points their fingertip to the virtual key VK

[32] corresponding to "i", the vowel of "ki". Then, the character "ki" is input.

[0086] Note that the movement of the hand H accepted as an operation on the virtual key VK[i] may be changed in the virtual key layout image PC2, etc. Also, in this embodiment, the operation on the virtual input interface VI[m] is performed with the hand H, but it may also be possible to perform the operation with another part of the body, such as the foot M.

[0087] (2) Relationship between multiple applications and the virtual input interface VI[m] As described above, in this embodiment, multiple types of virtual input interfaces VI[m] are provided. Each of the multiple virtual input interfaces VI[m] is associated with an application status. For example, in a video viewing application, when a screen for searching for videos using keywords is active, the virtual input interface VI[1] for character input is in a standby state. Also, in a video viewing application, when a video viewing screen is active, the virtual input interface VI[2] for video viewing is in a standby state. Also, in a message sending / receiving application, when a message sending / receiving screen is active, the virtual input interface VI[1] for character input is in a standby state.

[0088] For example, when a message sending / receiving application and a video viewing application are running at the same time, a message sending / receiving screen of the message sending / receiving application and a video viewing screen of the video viewing application may be simultaneously displayed on the AR glasses 10. The message sending / receiving screen is an example of a first virtual object, and the video viewing screen is an example of a second virtual object.

[0089] In a first state in which the message sending / receiving screen is active and the video watching screen is inactive, the determination unit 216 sets the virtual input interface VI[1] to a standby state. Setting the virtual input interface VI[1] to a standby state means determining the key information IK[1] to IK

[12] indicating the positions of the virtual keys VK[1] to VK

[12] included in the virtual input interface VI[1] as the key information IK[i] to be used for detection by the detection unit 218. The key information IK[1] to IK

[12] is an example of first key information.

[0090] Furthermore, in a second state in which the message sending / receiving screen is inactive and the video watching screen is active, the determination unit 216 sets the virtual input interface VI[2] to a standby state. Setting the virtual input interface VI[2] to a standby state means determining the key information IK

[21] to IK

[25] indicating the positions of the virtual keys VK

[21] to VK

[25] included in the virtual input interface VI[2] as the key information IK[i] to be used for detection by the detection unit 218. The key information IK

[21] to IK

[25] is an example of second key information.

[0091] The detection unit 218 detects, in the first state, that the virtual keys VK[1] to VK

[12] included in the virtual input interface VI[1] have been operated based on the key information IK[1] to IK

[12] and the second image. Also, in the second state, the detection unit 218 detects, in the second state, that the virtual keys VK

[21] to VK

[25] included in the virtual input interface VI[2] have been operated based on the virtual keys VK

[21] to VK

[25] and the second image.

[0092] In this way, the determination unit 216 switches the virtual input interface VI[m] depending on the active application, and can arrange the virtual keys VK[i] according to the application's functions. For example, although the virtual input interfaces VI[1] and VI[2] have different functions, they can be arranged based on the same physical information and measurement information, thereby reducing the processing load on the processing device 206.

[0093] (3) Relationship between the user U's surrounding environment and the virtual input interface VI[m] For example, in the virtual key layout image PC2-1 shown in FIG. 7, the user U performed the initial setup while standing on the floor G. In this case, the layout of the virtual keys VK[i] indicated by the reference information IS[i] is optimized for the user U standing, as shown in FIG. 12A. In FIG. 12A, the plane on which the virtual keys VK[i] are laid out based on the reference information IS[i] is indicated by the symbol F. The distance between the plane F and the AR glasses 10 is indicated by the symbol L. On the other hand, the user U is not necessarily in a standing state. For example, as shown in FIG. 12B, the user U may be sitting in a chair 62 facing a desk 60, or as shown in FIG. 12C, the user U may be lying on the floor G. In such cases, if the virtual keys VK[i] are laid out based on the reference information IS[i], it may be difficult to operate the virtual keys VK[i], or it may be impossible to operate the virtual keys VK[i].

[0094] 7, the initial setting was performed when there were no other people or objects around the user U. On the other hand, as shown in FIG. 13A, there are cases where there is another person T or an object around the user U. In this case, when the virtual key VK[i] is arranged based on the reference information IS[i], the area of ​​the virtual key VK[i] overlaps with the area of ​​the other person T or the object, making it difficult to input using the virtual key VK[i], or it may be impossible to input using the virtual key VK[i].

[0095] Therefore, the determination unit 216 may correct the key information IK[i] based on the measurement information. In this case, the measurement information includes environmental information related to the environment around the user U. The environmental information may be, for example, information related to surrounding objects (hereinafter, "objects" includes other people) shown in at least one of the first image and the second image, or distance information to objects around the user U and shape information of the objects obtained based on at least one of the first shape information and the second shape information. In other words, the environmental information indicates the positions of objects around the user U.

[0096] The determination unit 216 generates, based on the environmental information, placement information indicating an area SP in which the virtual key VK[i] can be placed. Taking Fig. 13A as an example, for a user U who is flanked by other person T, the area SP in which the virtual key VK[i] can be placed is a range that does not overlap with other person T, for example, a range approximately the width of the user U's shoulders.

[0097] When the virtual key VK[i] is placed in the virtual space based on the reference information IS[i], the determination unit 216 generates key information IK[i] that identifies the area SR[i] in real space corresponding to the area of ​​the virtual key VK[i]. Using FIG. 13A as an example, the detection unit 218 generates key information IK[1] to IK

[12] when placing the virtual input interface VI[1] based on the reference information IS[1] to IS

[12] . The areas SR[1] to SR

[12] identified based on the key information IK[1] to IK

[12] are assumed to be distributed over a wider range than the area SP. In FIG. 13A, the areas SR[1] to SR

[12] are indicated by thick dotted lines.

[0098] When the region SR[i] of the virtual key VK[i] indicated by the key information IK[i] is outside the region SP indicated by the placement information, the determination unit 216 determines corrected key information that corrects the key information IK[i] to bring the region SR[i] of the virtual key VK[i] into the region SP indicated by the placement information. That is, the determination unit 216 corrects the key information IK[i] so that the virtual key VK[i] falls within the region SP. In this case, the detection unit 218 detects that the virtual key VK[i] has been operated, based on the corrected key information and the second image.

[0099] Specifically, the determination unit 216 determines the corrected key information by, for example, reducing the size of the virtual key VK[i]. FIG. 14 is a diagram showing the virtual key VK[i] after reducing its size. FIG. 14 also shows the area SR[1] to SR

[12] indicating the distribution range of the virtual keys VK[1] to VK

[12] in FIG. 7, and the area SP in which the virtual key VK[i] can be arranged. The area SP is narrower than the area SR[1] to SR

[12] , i.e., the areas SR[1] to SR

[12] are outside the area SP. On the other hand, in FIG. 14, the virtual keys VK[1] to VK

[12] are arranged within the area SP by reducing the size of the virtual keys VK[1] to VK

[12] compared to FIG. 7.

[0100] Fig. 15 is an explanatory diagram showing another example of a method for correcting key information IK[i]. In Fig. 15, the sizes of virtual keys VK[1] to VK

[12] are the same as those in Fig. 7. However, by overlapping virtual keys VK[6] to VK

[10] with virtual keys VK[1] to VK[5], the areas SR[1] to SR

[12] are made narrower than those in Fig. 7. In the virtual input interface VI[m], it is possible to arrange virtual keys VK[i] three-dimensionally, and therefore various modes can be adopted for correcting key information IK[i].

[0101] 12(B), the region SR[i] of the virtual key VK[i] based on the reference information IS[i] is located below the surface 64 of the desk 60. In this case, the determination unit 216 determines the corrected key information so that the virtual key VK[i] is located above the surface 64 of the desk 60. The plane on which the virtual keys VK[i] are distributed based on the corrected key information is indicated by the symbol Fc. The distance between the plane Fc and the AR glasses 10 is indicated by the symbol Lc.

[0102] 12(C), the region SR[i] of the virtual key VK[i] based on the reference information IS[i] is located at a position different from the hand of the user U. In this case, the determination unit 216 determines corrected key information so that the virtual key VK[i] is located above the floor G and at the hand of the user U. The plane on which the virtual keys VK[i] are distributed based on the corrected key information is indicated by the symbol Fd. The distance between the plane Fd and the AR glasses 10 is indicated by the symbol Ld.

[0103] A-6. Operation of the processing unit 206 Next, the operation of the processing device 206 will be described. Below, the operation at the time of initial setup of the virtual input interface VI[m] will be described using Fig. 16, and the operation at the time of use of the virtual input interface VI[m] will be described using Fig. 17.

[0104] 16 is a flowchart showing the operation of the processing device 206 during the initial setting of the virtual input interface VI[m]. The processing device 206 waits until an instruction to initialize the virtual input interface VI[m] is given (step S100: NO). The instruction to initialize the virtual input interface VI[m] is given, for example, by the user U specifying the initial setting of the virtual input interface VI[m] on a setting screen of the AR glasses 10 or the terminal device 20.

[0105] When an instruction to initialize the virtual input interface VI[m] is issued (step S100: YES), the processing device 206 functions as the display control unit 212 and causes the projection device 131 to display an input screen PC1 for physical information (step S102). The input screen PC1 is exemplified in FIG. 5. The processing device 206 waits until physical information is input into the input screen PC1 (step S104: NO). When physical information is input (step S104: YES), the processing device 206 functions as the determination unit 216 and calculates initial information indicating the initial region S0[i] of the virtual key VK[i] (step S106).

[0106] Next, the processing device 206 functions as the display control unit 212 and causes the projection device 131 to display a virtual key layout image PC2 in which the virtual keys VK[i] are laid out in the virtual space based on the initial information (step S108). If the user U adjusts the layout of the virtual keys VK[i] in the virtual key layout image PC2, i.e., the layout of the virtual keys VK[i] based on the initial information (step S110: YES), the processing device 206 functions as the determination unit 216 and changes the initial information based on the operation of the user U (step S112). If the user U does not adjust the layout of the virtual keys VK[i] in the virtual key layout image PC2 (step S110: NO), the processing device 206 proceeds to step S114.

[0107] The determination unit 216 generates reference information IS[i] indicating the reference area where the virtual key VK[i] is placed in the virtual space based on the initial information or the changed initial information (step S114), and then ends the processing according to this flowchart. The reference information IS[i] is stored in the storage device 204.

[0108] 17 is a flowchart showing the operation of the processing device 206 when using the virtual input interface VI[m]. The processing device 206 waits until an application is started on the wearable device 1 (step S200: NO). The application is started, for example, by the user U selecting a desired application on an application selection screen of the AR glasses 10 or the terminal device 20.

[0109] When the application is started (step S200: YES), the processing device 206 reads the reference information IS[i] from the storage device 204 (step S201). The processing device 206 functions as the determination unit 216 and determines the key information IK[i] based on the reference information IS[i] (step S202). The processing device 206 functions as the acquisition unit 214 and acquires environmental information around the user U (step S204). The processing device 206 functions as the determination unit 216 and determines whether or not the key information IK[i] needs to be corrected based on the environmental information (step S206). If correction is necessary (step S206: YES), the processing device 206 corrects the key information IK[i] to generate corrected key information (step S208). If correction is not necessary (step S206: NO), the processing device 206 proceeds to step S210.

[0110] The processing device 206 functions as the detection unit 218 and determines whether the hand H of the user U is located at the position of the virtual key VK[i] (step S210). The position of the hand H of the user U is detected based on the second image acquired by the processing device 206 functioning as the acquisition unit 214. If the hand H of the user U is not located at the position of the virtual key VK[i] (step S210: NO), the processing device 206 proceeds to step S214. If the hand H of the user U is located at the position of the virtual key VK[i] (step S210: YES), the processing device 206 functions as the application processing unit 210 and performs processing corresponding to the virtual key VK[i] on the application (step S212).

[0111] The processing unit 206 returns the process to step S204 and repeats the subsequent processes until the application is terminated (step S214: NO). When the application is terminated (step S214: YES), the processing unit 206 terminates the process according to this flowchart.

[0112] A-7. Summary of embodiments As described above, in this embodiment, the terminal device 20 uses physical information indicating the overall physical characteristics of the user U to determine the key information IK[i] indicating the area of ​​the virtual keys VK[i] virtually arranged around the user U. Therefore, the virtual keys VK[i] are arranged taking into consideration the overall physical characteristics of the user U, making it easier for the user U to operate them and improving the operability of the wearable device 1.

[0113] In addition, in this embodiment, when the position of the hand H of the user U comes into contact with the area of ​​the virtual key VK[i] indicated by the key information IK[i], the terminal device 20 detects that the virtual key VK[i] has been operated. Therefore, when viewing an image displayed in a virtual space using the AR glasses 10, the operation can be performed without any discomfort.

[0114] Furthermore, in this embodiment, the terminal device 20 places the virtual key VK[i] in a position that is not visible to the user U. Therefore, the virtual key VK[i] can be placed without interfering with the viewing of content by the user U. Furthermore, since the virtual key VK[i] does not need to be displayed, the processing load on the AR glasses 10 and the terminal device 20 can be reduced.

[0115] In this embodiment, the terminal device 20 switches the virtual input interface VI[m] depending on the active application, and can therefore arrange the virtual keys VK[i] according to the application's functions. For example, the virtual input interfaces VI[1] and VI[2] have different functions, but can be arranged based on the same physical information and measurement information, thereby reducing the processing load on the processing device 206.

[0116] Furthermore, in this embodiment, the terminal device 20 accepts adjustments from the user U to the arrangement of the virtual keys VK[i] based on physical information, so that the arrangement of the virtual keys VK[i] can be set to suit the user U's preferences.

[0117] Furthermore, in this embodiment, the environmental information indicates the positions of objects around the user U, so the virtual key VK[i] can be positioned so that the area where the object exists does not overlap with the area of ​​the virtual key VK[i].

[0118] Furthermore, in this embodiment, the size of the virtual key VK[i] can be reduced when the virtual key VK[i] cannot be placed in the area SP around the user U where the virtual key VK[i] can be placed. This makes it possible to operate the virtual key VK[i] even in a place where an object or the like is present and the user U cannot move his / her hand H.

[0119] B: Modified example The following are variations of the above-described embodiment. Two or more variations arbitrarily selected from the following variations may be combined as appropriate within the scope of not mutually contradicting each other.

[0120] B1: First modified example In the above-described embodiment, the AR glasses 10 and the terminal device 20 are separate entities. However, this is not limiting, and for example, the AR glasses 10 may have the functions of the terminal device 20. That is, the processing device 142 of the AR glasses 10 may function as the application processing unit 210, the display control unit 212, the acquisition unit 214, the determination unit 216, and the detection unit 218. Note that if the AR glasses 10 have the functions of the terminal device 20, another user input / output device, such as a controller or hand tracking, or another user input / output device function may be provided instead of the touch panel 201.

[0121] According to the first modification, for example, the virtual input interface VI[m] can be realized without using the terminal device 20.

[0122] B2: Second variant In the above-described embodiment, the processing device 206 of the terminal device 20 functions as the application processing unit 210, the display control unit 212, the acquisition unit 214, the determination unit 216, and the detection unit 218. However, this is not limiting, and at least one function of the application processing unit 210, the display control unit 212, the acquisition unit 214, the determination unit 216, and the detection unit 218 may be executed in a processing server connected to the terminal device 20 or the AR glasses 10 via a communication network, for example.

[0123] According to the second modification, the processing load on the terminal device 20 can be reduced.

[0124] B3: Third variant In the above-described embodiment, the number of virtual keys VK[i] of the virtual input interface VI[m] is set to be smaller than, for example, the number of virtual keys VK[i] on the virtual input interface VI[m] is set to be smaller than that of, for example, a keyboard attached to a personal computer. This is not a limitation, and the virtual keys VK[i] on the virtual input interface VI[m] may be arranged in the same manner as, for example, a keyboard attached to a personal computer. In this case, as in the above-described embodiment, the user U may be able to customize the arrangement of the virtual keys VK[i].

[0125] According to the third modification, a user U who is accustomed to keyboard input can input data easily and quickly.

[0126] B4: Fourth variant In the above-described embodiment, the AR glasses 10 include the first imaging device 134A, the second imaging device 134B, the first scanner 135A, and the second scanner 135B. However, the present invention is not limited to this, and the AR glasses 10 may include, for example, at least one of the second imaging device 134B or the second scanner 135B. For example, if the imaging range of the first imaging device 134A is wide enough to cover both the front and the hand, the second imaging device 134B may not be provided. Alternatively, if the detection range of the first scanner 135A is wide enough to cover both the front and the hand, the second scanner 135B may not be provided.

[0127] According to the fourth modification, the configuration of the AR glasses 10 can be simplified and the processing load on the AR glasses 10 can be reduced.

[0128] C:Other (1) Each function illustrated in Figures 3 and 4 is realized by any combination of hardware and software. There are no particular limitations on how each function is realized. Each function may be realized using a single device that is physically or logically coupled, or may be realized using a device that is configured by connecting two or more physically or logically separated devices directly or indirectly (for example, using wires, wirelessly, etc.). Each function may be realized by combining software with the single device or multiple devices.

[0129] (2) In this specification, the term "apparatus" may be replaced with other terms such as circuit, device, or unit.

[0130] (3) In each of the embodiment and the first to fourth modifications, storage device 140 and storage device 204 may be configured by at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Also, the program may be transmitted from a network via a telecommunications line.

[0131] (4) Each of the embodiment and the first to fourth modifications may be implemented using any of the following standards: LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (where x is, for example, an integer or a decimal point), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE The present invention may be applied to at least one of systems using 802.20, UWB (Ultra-Wideband), Bluetooth (registered trademark), or other appropriate systems, and next-generation systems that are extended, modified, created, or defined based on these systems. The present invention may also be applied to a combination of multiple systems (e.g., a combination of LTE and / or LTE-A with 5G).

[0132] (5) The order of the exemplary procedures, sequences, or flowcharts shown in the embodiment and each of the first to fourth modifications may be changed as long as there is no contradiction. For example, the methods described herein present various step elements in an exemplary order and are not limited to the particular order presented.

[0133] (6) In each of the embodiment and the first to fourth modifications, input and output information, etc. may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information, etc. may be overwritten, updated, or added. Output information, etc. may be deleted. Input information, etc. may be transmitted to another device.

[0134] (7) In each of the embodiments and the first to fourth variants, the determination may be based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values ​​(e.g., a comparison with a predetermined value).

[0135] (8) The programs exemplified in the embodiments and the first to fourth modifications should be broadly construed to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, or functions, regardless of whether they are called software, firmware, middleware, microcode, hardware description languages, or by other names. Software, instructions, or the like, may also be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, optical fiber cable, twisted pair, and digital subscriber line (DSL)) and wireless technology (such as infrared and microwave), at least one of these wired and wireless technologies is included within the definition of a transmission medium.

[0136] (9) The information described in each of the embodiments and the first to fourth modifications may be represented using any of a variety of different technologies. For example, data, information, and the like that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Note that terms described in this specification and terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0137] (10) In each of the embodiment and the first to fourth modifications, the terms "system" and "network" are used interchangeably.

[0138] (11) In each of the embodiment and the first to fourth modifications, the terminal device 20 may be a mobile station, which may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate term.

[0139] (12) A mobile station may be referred to as a transmitting device, a receiving device, a communication device, or the like. A mobile station may be a device mounted on a mobile object, or the mobile object itself. A mobile object refers to an object that can move. A mobile object can move at any speed. A mobile object can be stopped. Examples of mobile objects include, but are not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcars, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and objects mounted thereon. A mobile object may also be an object that moves autonomously based on an operation command. A mobile object may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile object (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). A mobile station also includes devices that do not necessarily move during communication operations. For example, the mobile station may be an IoT (Internet of Things) device such as a sensor.

[0140] (13) In each of the embodiment and the first to fourth modifications, the term "determining" or "determining" may encompass a wide variety of actions. "Determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching a table, database, or other data structure), ascertaining, and regarding that as a "determination." Also, "determining" may include regarding receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and regarding that as a "judging" or "determining." Furthermore, "decision" can include the act of considering something to be "decided" such as resolving, selecting, choosing, establishing, or comparing. In other words, "decision" can include the act of considering something to be "decided" to be an action. "Decision" can also be interpreted as "assuming," "expecting," or "considering," among others.

[0141] (14) In each of the embodiments and the first to fourth modifications, the term "connected," or any variation thereof, refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more electric wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0142] (15) In each of the embodiment and the first to fourth modifications, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0143] (16) As used herein, any reference to elements using designations such as "first" and "second" does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0144] (17) When the words "include," "including," and variations thereof are used in the present specification or claims in each of the embodiments and the first to fourth modifications, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in the present specification or claims is not intended to mean an exclusive logical OR.

[0145] (18) Throughout this application, where articles are added by translation, such as a, an, and the in English, the disclosure may include the plural form of the noun following these articles.

[0146] (19) It is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended as an illustrative explanation and does not have any limiting meaning on the present invention. Furthermore, multiple embodiments selected from the embodiments exemplified in this specification may be combined. [Explanation of symbols]

[0147] 1...wearable device, 10...AR glasses, 20...terminal device, 131...projection device, 132...speaker, 133...communication device, 134A...first imaging device, 134B...second imaging device, 135A...first scanner, 135B...second scanner, 140...storage device, 142...processing device, 160...operation control unit, 201...touch panel, 202...communication device, 204...storage device, 206...processing device, 210...application processing unit, 212...display control unit, 214...acquisition unit, 216...determination unit, 218...detection unit, ...physical information, IK[i]...key information, ...measurement information, IS[i]...reference information, ...reference area, U[j]...user, VI[m]...virtual input interface, VK[i] virtual key.

Claims

1. an acquisition unit that acquires physical information indicating characteristics of the entire body of a user and measurement information measured by a sensor worn by the user; a determination unit that determines key information indicating an area of ​​virtual keys virtually arranged around the user based on the physical information and the measurement information; a detection unit that detects that the virtual key has been operated based on the key information and the measurement information; a display control unit that displays an image showing a virtual space on a display device that is placed in front of the user's eyeball, the virtual keys are arranged at positions that are not visible to the user when the user's head faces forward; Information processing device.

2. a first virtual object and a second virtual object are placed in the virtual space; the virtual keys include a plurality of first virtual keys associated with the first virtual object and a plurality of second virtual keys associated with the second virtual object; The determination unit in a first state in which the first virtual object is active and the second virtual object is inactive, determining, as the key information, first key information indicating positions of the plurality of first virtual keys based on the physical information and the measurement information; in a second state in which the first virtual object is inactive and the second virtual object is active, determining, as the key information, second key information indicating positions of the second virtual keys, based on the physical information and the measurement information; The detection unit In the first state, detecting that the first virtual key has been operated based on the first key information and the measurement information; In the second state, detecting that the second virtual key has been operated based on the second key information and the measurement information.

2. The information processing device according to claim 1.

3. the detection unit detects that the virtual key has been operated when the position of the user's hand calculated based on the measurement information comes into contact with an area of ​​the virtual key indicated by the key information. The information processing device according to claim 1 .

4. The determination unit Calculating initial information indicating an initial area of ​​the virtual key based on the physical information; generating an image of a virtual space in which the virtual key is arranged in the initial area indicated by the initial information; detecting an operation by the user to adjust the arrangement based on the measurement information; generating reference information indicating a reference area that is an area of ​​the virtual key after the arrangement has been adjusted; determining the key information based on the reference information; 2. The information processing device according to claim 1.

5. the measurement information includes environmental information regarding an environment surrounding the user; The determination unit generating layout information indicating an area in which the virtual keys can be arranged based on the environment information; If the area of ​​the virtual key indicated by the key information is outside the area indicated by the layout information, corrected key information is determined so that the area of ​​the virtual key is within the area indicated by the layout information by correcting the key information; the detection unit detects that the virtual key has been operated based on the correction key information and the measurement information.

2. The information processing device according to claim 1.

6. The environmental information indicates the positions of objects present around the user. The information processing device according to claim 5 .

7. The information processing apparatus according to claim 5 , wherein the determination unit determines the corrected key information by reducing the size of the virtual key.

Images