User Interface System

Abstract

We provide a user interface system that can enhance user convenience. [Solution] The user interface system 1 includes a virtual operation unit setting means (server 6) that sets up virtual operation units (virtual operation unit 131 for mouse cursor operation, virtual operation unit 141 for flick / toggle input, virtual operation unit 151 for full key input) in space to be operated by the user, an operation detection means (server 6) that detects operations performed by the user on the virtual operation units based on the image of the user captured by the imaging means, and a display means (AR glasses 2) that displays an image (spatial display image 10) whose content has been changed according to the operation detected by the operation detection means. The virtual operation unit setting means determines the position of the user's fingers based on the image of the user captured by the imaging means, and sets up the virtual operation units in space based on this determination result.

Description

Technical Field

[0001] The present invention relates to a user interface system that detects a user's operation and displays an image. Background Art

[0002] In recent years, XR (Cross Reality or Extended Reality) such as VR information (Virtual Reality information), AR information (Augmented Reality information), and MR information (Mixed Reality information) that create virtual spaces and worlds on computers has become widespread. Among these, various techniques have been proposed for displaying an augmented reality space, which is combined with the real space as AR or MR, on a head-mounted display device or the like.

[0003] For example, there is an aerial operation technology that displays, on a head-mounted display device, an augmented reality space in which a computer-generated keyboard image is combined with the real space as AR information, and allows a user to reach out and operate the displayed virtual keyboard (see, for example, Patent Documents 1 and 2). Prior Art Documents Patent Documents

[0004] Patent Document 1 Japanese Unexamined Patent Application Publication No. 07-078055 Patent Document 2 Japanese Unexamined Patent Application Publication No. 2009-146333 Summary of Invention Problems to be Solved by the Invention

[0005] In the aerial operation techniques described in Patent Documents 1 and 2, since the user reaches out and operates the virtual keyboard displayed in the augmented reality space, there are limits to the range and diversity of operations and displays.

[0006] Furthermore, in recent years, advancements in the handling of technological information have led to the emergence of virtual spaces like the metaverse, in addition to VR, AR, and MR. Within this context, it is believed that the density of information in virtual spaces will increase, and objects in both the real world and virtual spaces will increase in information density, transforming them from mere facades into objects with substance and information. In this regard, conventional two-dimensional user interfaces that ignore distance cannot provide more realistic and intuitive operation that takes distance into account, thus failing to utilize the best aspects of real-world user interfaces.

[0007] The present invention aims to provide a user interface system that can enhance user convenience. [Means for solving the problem]

[0008] A user interface system in a first aspect of the present invention comprises: an imaging means for capturing images of a user; a virtual operation unit setting means for setting up a virtual operation unit operated by the user in space; an operation detection means for detecting operations performed by the user on the virtual operation unit based on the images of the user captured by the imaging means; and a display means for displaying images whose content has been modified according to the operations detected by the operation detection means.

[0009] Preferably, the virtual operation unit setting means determines the position of the user's fingers based on the image of the user captured by the imaging means, and sets the virtual operation unit in space based on this determination result.

[0010] A user interface system in a second aspect of the present invention is characterized in that the virtual operation unit setting means sets a plurality of virtual operation units to different positions based on the result of determining the position of the user's finger.

[0011] A third aspect of the present invention provides a user interface system in which the plurality of virtual operation units correspond to mouse input, flick input, toggle input, or full keyboard input, and the operation detection means enables user input by detecting an operation performed by the user's finger on any of the plurality of virtual operation units when the user's finger is away from the video whose content has been changed. [Effects of the Invention]

[0012] The user interface system of the present invention makes it possible to improve user convenience. [Brief explanation of the drawing]

[0013] [Figure 1] This is a side view of a user interface system according to a first embodiment of the present invention. [Figure 2] This is a perspective view showing a video camera stand for a user interface system according to a first embodiment of the present invention. [Figure 3] This is a block diagram showing the overall configuration of a user interface system according to a first embodiment of the present invention. [Figure 4] This is a flowchart showing the operation of a user interface system according to the first embodiment of the present invention. [Figure 5] This is an explanatory diagram showing the video processing of a user interface system according to a first embodiment of the present invention. [Figure 6] This is a first explanatory diagram showing detailed image processing for AR glasses in a user interface system according to a first embodiment of the present invention. [Figure 7] This is a second explanatory diagram showing detailed image processing for AR glasses of a user interface system according to a first embodiment of the present invention. [Figure 8] This is a third explanatory diagram showing detailed image processing for AR glasses in a user interface system according to the first embodiment of the present invention. [Figure 9]FIG. 4 is an explanatory diagram showing detailed image processing for AR glasses of the user interface system according to the first embodiment of the present invention. [Figure 10] FIG. is an explanatory diagram showing setting processing of a virtual operation unit of the user interface system according to the first embodiment of the present invention. [Figure 11] FIG. 5 is an explanatory diagram showing detailed image processing for AR glasses of the user interface system according to the first embodiment of the present invention. [Figure 12] FIG. is an explanatory diagram showing an operation on a virtual operation unit for mouse cursor operation of the user interface system according to the first embodiment of the present invention. [Figure 13] FIG. 6 is an explanatory diagram showing detailed image processing for AR glasses of the user interface system according to the first embodiment of the present invention. [Figure 14] FIG. 7 is an explanatory diagram showing detailed image processing for AR glasses of the user interface system according to the first embodiment of the present invention. [Figure 15] FIG. is an explanatory diagram showing an operation on a virtual operation unit for flick toggle input of the user interface system according to the first embodiment of the present invention. [Figure 16] FIG. 8 is an explanatory diagram showing detailed image processing for AR glasses of the user interface system according to the first embodiment of the present invention. [Figure 17] FIG. 9 is an explanatory diagram showing detailed image processing for AR glasses of the user interface system according to the first embodiment of the present invention. [Figure 18] FIG. is an explanatory diagram showing an operation on a virtual operation unit for full-key input of the user interface system according to the first embodiment of the present invention. [Figure 19] FIG. is an explanatory diagram showing an image captured by a first video camera of a video camera stand of the user interface system according to the first embodiment of the present invention. [Figure 20] FIG. 10 is an explanatory diagram showing detailed image processing for AR glasses of the user interface system according to the first embodiment of the present invention. [Modes for carrying out the invention]

[0014] <First Embodiment of the Invention> A user interface system according to a first embodiment of the present invention will be described below with reference to the drawings.

[0015] <Configuration of User Interface System> First, the configuration of the user interface system according to the first embodiment of the present invention will be described. Figure 1 is a side view of the user interface system according to the first embodiment of the present invention. Figure 2 is a perspective view showing the video camera stand of the user interface system according to the first embodiment of the present invention. Figure 3 is a block diagram showing the overall configuration of the user interface system according to the first embodiment of the present invention.

[0016] In Figure 1, the user interface system 1 according to the first embodiment of the present invention comprises AR glasses (augmented reality glasses) 2 worn by the injured or ill person 14, a projector 3 that displays an image of an operation screen viewable by the injured or ill person 14 lying in bed 13 in a hospital room 12, a video camera stand 4 that captures images of the injured or ill person 14 lying in bed 13, a wireless LAN 5, and a server 6 installed in the management room 17 of the hospital 11.

[0017] AR Glasses 2 uses a holographic reflector to display augmented reality (AR) in a form that is indistinguishable from regular glasses, allowing the human eye to see images similar to those displayed with AR goggles.

[0018] The projector 3 and video camera stand 4 are positioned to the right of the patient 14 in the hospital room 12. The curtain 15 is for the privacy of the patient 14 and is positioned at a predetermined distance from the patient's feet.

[0019] Furthermore, Projector 3 initially needs to either manually input or calculate the distance from the display surface (curtain 15). Projector 3 also needs to know or set the distance to the display surface and potential projection surfaces (curtain 15, wall, monitor), as well as the room shape and spatial shape.

[0020] The video camera stand 4 may include a first video camera 41 as a first imaging means for imaging the user (injured or ill person 14) from above, and a second video camera 42 as a second imaging means for imaging the user (injured or ill person 14) from the right side.

[0021] The video camera stand 4 will be described in detail below using Figure 2. The video camera stand 4 consists of a first video camera 41, a second video camera 42, a flexible arm 43, and a base 44.

[0022] A first video camera 41 is attached to one end of the flexible arm 43. A base 44 is attached to the other end of the flexible arm 43. A second video camera 42 is attached to the middle section of the flexible arm 43, near the other end.

[0023] The first and second video cameras 41 and 42 initially need to perform a process such as setting or calculating the distance from the injured person's hand 91 (see Figure 5(a)) to the injured person's hand 14. The first and second video cameras 41 and 42 may also have their positions and the center point of the hand's fist pre-set.

[0024] As shown in Figure 1, the wireless LAN 5 includes a wireless LAN access point 51 and wiring. The wireless LAN access point 51 is mounted on the wall 16 of the patient room 12 and connected to the server 6 in the control room 17 via wiring. The wireless LAN 5 communicates wirelessly with the AR glasses 2, the projector 3, and the first and second video cameras 41 and 42.

[0025] As shown in Figure 3, the server 6 is connected to a keyboard 65, a mouse 66, a display device 67, a wireless LAN 5, and various devices 7.

[0026] The various devices 7 include the intercom in the patient room 12, the light switch in the patient room 12, and the call bell installed in the control room 17.

[0027] Server 6 comprises a control unit 61, a storage unit 62, an interface 63, and a communication unit 64.

[0028] The memory unit 62 stores the program of the user interface system 1, images of the operation screen and AR operation space display, video data of images captured by the projector 3, and judgment results for images captured by the projector 3.

[0029] The control unit 61 is connected to the keyboard 65, mouse 66, and display device 67 via the interface 63. Various settings are made by the user using the keyboard 65 and mouse 66, and the display device 67 displays the information based on these settings.

[0030] Furthermore, the control unit 61 is connected to the AR glasses 2, projector 3, and first and second video cameras 41 and 42 via the communication unit 64 and wireless LAN 5, and performs judgment processing based on input signals from the first and second video cameras 41 and 42, and outputs signals such as 3D projection mapping to the AR glasses 2 and projector 3.

[0031] <How the user interface system works> Figure 4 is a flowchart showing the operation of a user interface system according to the first embodiment of the present invention.

[0032] Figure 5 is an explanatory diagram showing the video processing of a user interface system according to a first embodiment of the present invention.

[0033] In Figure 4, when the user interface system 1 is set up, first the injured person 14 puts on the AR glasses 2, the projector 3 performs processing such as maintaining the distance from the display surface (curtain 15) and the shape of the room by setting it, or calculating the distance and the shape of the room, and the first and second video cameras 41 and 42 perform processing such as maintaining the distance from the injured person 14's hand 91 (see Figure 5(a)) by setting it, or calculating it.

[0034] Subsequently, in step S1, the first and second video cameras 41 and 42 capture images of the upper body of the injured person 14 lying in bed 13 from above and to the right, respectively, and transmit the video data to the server 6. The image 9 shown in Figure 5(a) is the video data from the first video camera 41.

[0035] As a result, in step S2, server 6 receives video data from the first and second video cameras 41 and 42 (in the case of the first video camera 41, video data of image 9).

[0036] Next, in step S3, the server 6 performs image analysis on the received video data to determine, for example, whether the hand 91 of the injured person 14 is clenched in the image shown by the video data (in the case of the first video camera 41, the image 9 shown in Figure 5(a)). If the hand 91 of the injured person 14 is clenched, in step S4, the server 6 generates video data of various images within the AR operation space 71 shown in Figure 5(b) and video data of the image 81 shown in Figure 5(c) as images of the operation space and operation screen that can be viewed by the user.

[0037] Furthermore, although it was stated above that this applies when hand 91 is in a clenched fist, other methods may also be used to detect various inputs, such as when the index finger is extended or when using fingers to perform flick input, toggle input, or full key input, and then execute the operation process.

[0038] Next, in step S5, server 6 transmits the video data generated in step S4 to AR glasses 2 and projector 3.

[0039] As a result, in step S6, the AR glasses 2 receive video data from the server 6, and in step S7, the video light from the video data is reflected by a holographic reflector and made visible to the eyes of the injured person 14, thereby performing the initial operation space display shown in Figure 5(b).

[0040] The AR operating space 71 shown in Figure 5(b) is an image of the augmented reality space that is visible to the injured person 14 through the AR glasses 2. The spatial display image 10 of the operation display panel is superimposed on images of the ceiling 18 that is transparent to the AR glasses 2, the first and second video cameras 41 and 42 on the video camera stand 4, the flexible arm 43, and the injured person 14's hand 91.

[0041] The AR glasses 2 display a spatial display image 10 of the operation display panel on a virtual surface 8 (see Figures 1 and 2) located slightly below the first video camera 41, using stereoscopic 3D image display.

[0042] While the AR glasses 2 allow the viewer to see the actual video camera stand 4 and the hand 91 of the injured person 14, the spatial display image 10 is a virtual representation that does not exist in the actual location, using video data generated by the server 6 in step S4.

[0043] Meanwhile, in step S8, projector 3 receives video data from server 6, and in step S9, projects the image from the video data onto curtain 15 (see Figure 1). In this case, the image projected onto curtain 15 is the image 81 shown in Figure 5(c).

[0044] The video 81 shown in Figure 5(c) is a video of the operation screen, in which the upper body of the injured person 14 lying on the bed 13, captured by the first video camera 41, is horizontally flipped, and the horizontally flipped spatial display image 10 of the operation display panel is superimposed on the horizontally flipped video 82, which is then displayed as video 83.

[0045] In addition, the video 81 shown in Figure 5(c) displays a video 82 which is a horizontally flipped image of the upper body of the injured person 14. However, it is also possible to basically hide the screen that is operated by a person, that is, the operator themselves.

[0046] If the operator is not shown, the image 82 in Figure 5(c) does not show the operator, but instead displays an image 83 of the operation display panel on the curtain, and a red dot-like object (any bright spot will do) on the curtain moves in the same way as the movement of the operator's fingers or the tip of the virtual rod being held.

[0047] Next, in step S10, the first and second video cameras 41 and 42 continuously capture images of the upper body of the injured or ill person 14 lying in bed 13 and transmit the video data to the server 6.

[0048] As a result, in step S11, server 6 receives new video data of the upper body of the injured person 14 from the first and second video cameras 41 and 42.

[0049] Next, in step S12, the server 6 performs image analysis on the received video data and automatically calculates the tip X1, Y1 of the index finger and the grip and hand center X2, Y2 of the injured person 14 in the new video shown in the video data. Based on the coordinates X1, Y1, X2, Y2, the server 6 detects an aerial operation, which is an operation performed by the injured person 14 in mid-air with their fingers or a virtual rod, corresponding to the spatial display image 10 of the operation display panel displayed by the AR glasses 2.

[0050] Next, in step S13, the server 6 generates video data of the augmented reality space displayed by the AR glasses 2 and the virtual operation screen displayed on the curtain 15, according to the aerial operation detected in step S12.

[0051] Next, in step S14, server 6 transmits the video data generated in step S13 to AR glasses 2 and projector 3.

[0052] Next, in step S15, the AR glasses 2 receive video data from the server 6, and in step S16, the video light based on the video data is reflected by the holographic reflector to be visually recognized by the eyes of the patient 14.

[0053] In this case, the image of the augmented reality space visually recognized by the eyes of the patient 14 via the AR glasses 2 is displayed by superimposing the virtual light stick 92 and the spatial display image 10 of the operation display panel on the images of the ceiling 18 (see FIG. 1) transmitted through the AR glasses 2, the first and second video cameras 41, 42 of the video camera stand 4, the flexible arm 43, and the hand 91 of the patient 14, and effect processing such as light emission is performed on the images of buttons and keys corresponding to the aerial operation detected in step S12 in the spatial display image 10 of the operation display panel.

[0054] Next, in step S17, the projector 3 receives the video data from the server 6, and in step S18, projects and displays the video of the virtual operation screen based on the video data onto the curtain 15.

[0055] In this case, the video of the virtual operation screen projected and displayed on the curtain 15 is obtained by horizontally inverting the video of the upper body of the patient 14 captured by the first video camera 41 in step S10, superimposing and displaying the virtual light stick 92 and the image 83 of the operation display panel on the horizontally inverted video, and performing effect processing such as light emission on the images of buttons and keys corresponding to the aerial operation detected in step S12 among the images of buttons and keys displayed on the image 83 of the operation display panel.

[0056] Thereafter, in step S19, the server 6 performs switching of the input screen displayed as the spatial display image 10 of the operation display panel, and controls various devices 7 such as the intercom in the hospital room 12, the lighting switch, and the call bell installed in the management room 17, in accordance with the detected aerial operation.

[0057] <Detailed Video Processing for AR Glasses> Hereinafter, detailed video processing for AR glasses will be described.

[0058] Figure 6 is a first explanatory diagram showing detailed image processing for AR glasses in a user interface system according to a first embodiment of the present invention.

[0059] In the user interface system according to the first embodiment, the server 6 (see Figures 1 and 3) performs image analysis on the video data received from the first and second video cameras 41 and 42 (see Figures 1 to 3). Basically, it determines the direction of the injured person's hand 91 and the degree to which the hand 91 is gripping the image 9 shown in Figure 5(a) from the first video camera 41. If it is difficult to determine this from the image 9 from the first video camera 41 alone, it also refers to the image from the second video camera 42. As an image in augmented reality space, it generates an image of a virtual rod of light 92 being gripped by the hand 91, according to the direction of the hand 91 and the degree to which the hand 91 is gripping, and transmits it as video data to the AR glasses 2. The AR glasses 2 receive the video data from the server 6 and reflect the video light from the video data onto a holographic reflector so that it can be seen by the injured person's eyes.

[0060] As shown in Figure 6(a), if a real injured person 14 forms a large cavity 93 between the fingers of the hand 91, the image of a large virtual rod of light 92 is used as the image of the virtual rod of light that the hand 91 grasps in the augmented reality space of the AR glasses 2.

[0061] The tip 92a of the large virtual rod of light 92 serves as an operating point, and a dot image 92b is displayed at its location and around it.

[0062] When the injured person 14 views the image of the AR operating space 71 in Figure 5(b) through the AR glasses 2, and aligns the tip of the virtual rod of light 92 held in the injured person 14's hand 91 with each of the operation unit images on the spatial display image 10 of the operation display panel (for example, the light operation button 111, the on operation button 112, the call button 113, the intercom button 114, the key input mode selection button 115, the cancel button 116, and the OK button 117 in Figure 8), the operation unit image lights up and the control corresponding to the operation unit image is performed.

[0063] As shown in Figure 6(b), if the actual injured person 14 forms a moderate cavity 94 between the fingers of the hand 91, a medium-sized virtual rod of light 95 is used as the image of the virtual rod of light that the hand 91 grasps in the augmented reality space shown by the AR glasses 2.

[0064] The tip 95a of the medium-sized virtual rod of light 95 serves as the control point, and a dot image 95b is displayed at its position and around it.

[0065] When the injured person 14 aligns the tip 95a of the virtual rod of light 95 with each operation part of the spatial display image 10 on the operation display panel, the operation part image lights up and control corresponding to the operation part image is performed.

[0066] As mentioned above, the system could be configured such that when the spatial display image 10 of the operation display panel is aligned with each operation unit image (for example, the light operation button 111, the on operation button 112, the call button 113, the intercom button 114, the key input mode selection button 115, the cancel button 116, and the OK button 117 in Figure 8), the operation unit image lights up and the control corresponding to the operation unit image is performed. However, considering practical use, it may be configured so that even if there is slight movement for a set number of seconds, if the area on the button is occupied for a set number of seconds or longer, it is determined that the button has been pressed. Therefore, even if the input selection button is not pressed, the system may be configured so that the action is executed if the bright spot is on that button.

[0067] To explain in more detail, the server 6 (see Figures 1 and 3) automatically calculates the direction of the hand 91 from the video data received from the first and second video cameras 41 and 42 (see Figures 1 and 3), determining the X1, Y1 points of the tip of the index finger of the injured person's hand 91 and the X2, Y2 points of the grip and the center of the hand. The line connecting these two points X2, Y2 to X1, Y1 is used as the central axis of the virtual rod, which is represented as the virtual rod of light 92 in Figure 6(a) and the virtual rod of light 95 in Figure 6(b).

[0068] In the case of a grip on hand 91, the virtual light rods 92 and 95 in Figure 6 may be assumed based on the shape of the grip, but in order to create the virtual light rods 92 and 95 in Figure 6, X3 and Y3 may be automatically calculated and set near the upper part of the grip on hand 91, and X4 and Y4 may be set near the lower part of the grip on hand 91.

[0069] The grip of hand 91 is usually as shown in Figure 6, but if the hand can only grip in the reverse position, that is, if it can only be operated with the little finger side facing upwards, the virtual rod of light may extend from behind the hand's grip. In that case, it will extend from X3,Y3 to X4,Y4 and reach the contact surface of the ceiling 18.

[0070] In addition to this configuration, to guide demonstrations and operations, the shape of the hand 91 (showing how it moves) can be displayed on the AR glasses 2 (see Figure 1) as a dotted line (or edge image (an image extracted using edge detection, a type of image processing technology)) at the position of the hand in the AR space (showing how it moves when done). Alternatively, the video from the first and second video cameras 41 and 42 that are filming the hand 91 can also be shown, displaying the hand's position and other details as a dotted line (or edge image) rather than the actual video, making it easier for first-time users to understand. Furthermore, overlaying images such as a rod of fire (a virtual rod of fire) in addition to the image of a rod of light (a virtual rod of light) can also make it easier to guide operations. For example, if you think of it as a rod of fire, displaying it in the augmented reality space of the AR glasses 2, along with the dotted line shape of the hand 91, as if it were burning a wall and penetrating it, would be easy to understand. It also becomes possible to convey the principles and mechanisms.

[0071] Furthermore, when the first and second video cameras 41 and 42 are capturing the upper body (or when observed using a motion sensor, etc.), the operation may be signaled to begin when the arms are made into a certain shape or when an infrequently performed operation is used as a signal. Examples of such actions include bending the elbow to form a V shape, performing several shapes in a set sequence, or making a circle with the thumb and index finger of the right hand and extending three other fingers, which may signal the start or end of the operation of the user interface of the present invention and the beginning or end of hand observation, or crossing the index and middle fingers of the right hand to signal the end or start of the operation.

[0072] The system may be designed so that raising the little finger or middle finger in a set procedure, or raising a certain number of fingers in a specific order, signifies completion or start. Such simple signals or operations may be used to initiate the user interface of the present invention.

[0073] Furthermore, when a virtual rod (the user interface of the present invention) created by the index finger or grip is pointed towards the left or right wall, a spatial display image of the operation display panel or button group set on that surface appears. Alternatively, a spatial display image of the set operation display panel or button group may appear even if it is not a button on that surface. During operation with fingers extended or with a grip that creates a hollow space, extending and then lowering the thumb may be considered a click, and consecutive clicks may be considered a double-click. In this case, even if the finger extension or grip shape is deformed, if it is recognized that an operation is in progress, processing may continue even with the distorted shape.

[0074] As shown in Figure 6(c), when the real injured person 14 transitions to a state where no gaps are formed between the fingers of the hand 91, the virtual rod of light held by the hand 91 in the augmented reality space by the AR glasses 2 disappears.

[0075] Figure 7 is a second explanatory diagram showing detailed image processing for AR glasses in a user interface system according to the first embodiment of the present invention.

[0076] To illustrate the operation of this user interface system 1 with a specific example, when the injured person 14 views the image of the AR operating space 71 in Figure 5(b) through the AR glasses 2, and swings the tip of the virtual rod of light 92 held in the injured person 14's hand 91 from the lower left to the upper right in a trajectory 101, as shown in Figure 7, the positions of the lower left corner 10a and the upper right corner 10b of the spatial display image 10 on the operation display panel are adjusted accordingly. When the tip of the virtual rod 92 is swung from the lower right to the upper left in a trajectory 102, the positions of the lower right corner 10c and the upper left corner 10d of the spatial display image 10 on the operation display panel are adjusted accordingly. As a result of adjusting the position of the spatial display image 10, the position of the virtual surface 8 is also adjusted.

[0077] Figure 8 is a third explanatory diagram showing detailed image processing for AR glasses of a user interface system according to the first embodiment of the present invention, and shows a selection input operation screen 110 that is displayed as the initial screen of the spatial display image 10 of the operation display panel.

[0078] In Figure 8, the spatial display image 10 of the operation display panel shows the selection input operation screen 110 as the initial screen.

[0079] The selection input operation screen 110 displays the following buttons: a light operation button 111, a light operation button 112, a call button 113, an intercom button 114, a key input mode selection button 115, a cancel button 116, and an OK button 117.

[0080] When the injured person 14 views the image of the AR operating space 71 in Figure 5(b) through the AR glasses 2, the tip 92a or the image 92b of the virtual rod of light 92 held in the injured person's hand 91 is selected on the light operation button on the selection input operation screen 110 shown in Figure 8. When you press button 111, the operation button for turning on the light 112, the call button 113, the intercom button 114, the key input mode selection button 115, the cancel button 116, or the OK button 117, the corresponding control will be executed.

[0081] To illustrate the operation of this user interface system 1 with a specific example, when the tip 92a or point image 92b of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the light operation button 111 on the selection input operation screen 110 shown in Figure 8, the light off operation button 111 lights up, and the light on the bed 13 is turned off.

[0082] When the tip 92a or the image 92b of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the illumination operation button 112 on the selection input operation screen 110 shown in Figure 8, the illumination operation button 112 lights up, and the light on the bed 13 turns on.

[0083] When the tip 92a or the image 92b of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the call button 113 on the selection input screen 110 shown in Figure 8, the call button 113 lights up, the call bell installed in the control room 17 rings, and a nurse goes to the injured person 14.

[0084] When the tip 92a or the image 92b of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the intercom button 114 on the selection input operation screen 110 shown in Figure 8, the intercom button 114 lights up. After this, when the tip of the virtual rod of light 92 is aligned with the OK button 117, the intercom between the patient room 12 and the management room 17 is connected.

[0085] When the tip 92a or point image 92b of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the key input mode selection button 115 on the selection input operation screen 110 shown in Figure 8, the key input mode selection button 115 lights up. After this, the user interface system 1 moves to the key input mode setting stage, and the virtual operation unit setting screen 120 shown in Figure 9(a) is displayed on the spatial display image 10 of the operation display panel.

[0086] When the tip 92a or the image 92b of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the cancel button 116 shown in Figure 8, the previous operation is invalidated and the state returns to what it was before the operation.

[0087] Figure 9 is a fourth explanatory diagram showing detailed image processing for AR glasses of a user interface system according to the first embodiment of the present invention, where Figure 9(a) shows the virtual operation unit setting screen in the preparation stage for key input mode, and Figure 9(b) shows the injured person 14 performing the virtual operation unit settings as viewed from the right side.

[0088] In the preparation phase for key input mode, the user interface system 1 first displays the virtual operation unit setting screen 120 shown in Figure 9(a) on the spatial display image 10 of the operation display panel.

[0089] The virtual control unit settings screen 120 displays the text image 121 that reads "Please position your right hand at the key input position" and the OK button 122.

[0090] When the injured person 14 looks at the image of the virtual control unit setting screen 120 shown in Figure 9(a) using the AR glasses 2, and as shown in Figure 9(b), the injured person 14 bends the elbows of their left and right arms 94L and 94R into a V shape, stops the movement of their right hand 91R at the desired operating position, and aligns the tip 92a (see Figure 6) or the dot image 92b (see Figure 6) of the virtual rod of light 92 held in the injured person 14's left hand 91L with the confirmation button 122 on the virtual control unit setting screen 120 shown in Figure 9(a), the virtual control unit is set to match the position of the fingers of the right hand 91R, and the user interface system 1 switches to key input mode.

[0091] Figure 10 is an explanatory diagram showing the setting process of the virtual operation unit of a user interface system according to the first embodiment of the present invention.

[0092] In Figure 10, the server 6 of the user interface system 1 determines the position of the fingers of the right hand 91R that the injured person 14 has stopped moving at the desired operating position shown in Figure 9(b), based on the image of the injured person 14 captured by the imaging means (first and second video cameras 41 and 42) that capture images of the injured person 14. The server 6 sets a virtual operation unit 131 for mouse cursor operation at a height through which the thumb 96R of the right hand 91R can pass, sets a virtual operation unit 141 for flick / toggle input between the thumb 96R and the index finger 97R, and sets a virtual operation unit 151 for full key input below the little finger 98R (or below the hypothenar eminence or ulnar head).

[0093] The virtual control unit 131 for mouse cursor operation is a small virtual control unit, with the left and right control units 132L and 132R being thin plates, and the central control unit 132M being thicker than the left and right control units 132L and 132R.

[0094] The virtual operation unit 141 for flick / toggle input is a flat plate the same size as a commercially available smartphone, with its long side facing diagonally upward relative to the injured person 14.

[0095] The virtual control unit 151 for full key input is a flat plate the same size as a commercially available laptop keyboard, with its long side facing laterally relative to the injured person 14.

[0096] The virtual operation unit 131 for mouse cursor operation, the virtual operation unit 141 for flick / toggle input, and the virtual operation unit 151 for full keyboard input are set up in a multi-layered configuration arranged vertically around the injured person 14.

[0097] Furthermore, when switching to key input mode, the virtual operation unit 131 for mouse cursor operation, the virtual operation unit 141 for flick / toggle input, and the virtual operation unit 151 for full key input are temporarily fixed at that point.

[0098] <Image Processing of Mouse Cursor Input> Figure 11 is a fifth explanatory diagram showing detailed image processing for AR glasses of a user interface system according to the first embodiment of the present invention, and shows a mouse cursor operation screen that is displayed as a spatial display image on the operation display panel by operating the virtual operation unit 131 for mouse cursor operation.

[0099] Figure 12 is an explanatory diagram showing the operation of the virtual operation unit for mouse cursor operation in a user interface system according to the first embodiment of the present invention.

[0100] The user interface system 1 shown in Figure 1 detects the operation of the injured person 14's right hand 91R on the virtual operation unit 131 for mouse cursor operation based on the image of the injured person 14 captured by the imaging means (first and second video cameras 41 and 42) that capture images of the injured person 14. Based on this detection result, the system displays the mouse cursor input operation screen 160 shown in Figure 11 on the AR glasses 2. In this case, the AR glasses 2 displays the mouse cursor input operation screen 160 as a spatial display image 10-1 of the operation display panel on a vertically elongated virtual surface 8-1 shown in Figure 11, located slightly below the first video camera 41, using stereoscopic 3D image display.

[0101] In Figure 11, the mouse cursor input screen 160 consists of an upper screen display unit 161 corresponding to the screen of a laptop computer and a lower operation unit display unit 162 showing a virtual operation unit 131 for mouse cursor operation.

[0102] The screen display unit 161 displays the following: a light operation button 171, a light operation button 172, a call button 173, an intercom button 174, a display unit 175 that displays entered characters and numbers, and a mouse cursor (also called a mouse pointer) 176.

[0103] In Figure 11, the operation unit display unit 162 displays a virtual operation unit image 181 for mouse cursor operation, which corresponds to the virtual operation unit 131 for mouse cursor operation shown in Figure 10.

[0104] In the virtual control unit image 181 for mouse cursor operation, the left and right control unit images 182L and 182R, and the central control unit image 182M correspond to the left and right control units 132L and 132R, and the central control unit 132M shown in Figure 10.

[0105] Furthermore, server 6 (see Figures 1 and 3) performs image analysis on the video data received from the first and second video cameras 41 and 42 (see Figures 1 to 3) to determine the position and angle of the thumb 93R of the right hand 91R, and simultaneously displays a semi-transparent hand image 91Ra corresponding to the right hand 91R shown in Figure 10 on the operation display unit 162 of the mouse cursor input operation screen 160 shown in Figure 11.

[0106] The injured person 14 is positioned slightly below the first video camera 41, away from the spatial display image 10-1 (virtual plane 8-1) of the operation display panel shown in Figure 11. For example, the virtual operation unit 131 for mouse cursor operation, which does not actually exist as shown in Figure 10, can be automatically set or set to match the range of motion of the thumb.

[0107] To perform a left mouse click, insert your thumb (93R) through the left control panel (131L) in the left area and immediately pull it out. To perform a right mouse click, insert your thumb (93R) through the right control panel (132R) in the right area and immediately pull it out.

[0108] Mouse movement is performed by moving the thumb 93R forward, backward, left, and right while the thumb 93R is touching the central operating part 132M of the thick area. Mouse dragging is performed by moving the thumb 93R forward, backward, left, and right while the thumb 93R is stuck in the central operating part 132M. The amount of movement of the mouse cursor 176 shown in Figure 11 is imagined as the virtual mouse ball rotating when the thumb 93R shown in Figure 10 is touching the virtual surface of the central operating part 132M. If the hand is lifted, the virtual mouse ball is away from the virtual surface, and even if the virtual mouse (hand) is moved, the mouse cursor 176 shown in Figure 11 will not move. In the central operating part 132M of the thick area shown in Figure 10, it is also possible to set it so that contact is detected when a finger or hand touches its thickness, and in that case, only the mouse cursor or mouse pointer moves instead of clicking or dragging. In that case, it would be set so that a click occurs when the finger penetrates the plate with the thickness of the central operating part 132M. For dragging, it would be when the finger that penetrated moves while the finger is penetrating. It is also possible to assign other functions to the operation.

[0109] The movements of the thumb 93R in relation to these virtual operation units 131 for mouse cursor control are shown in the thumb image 93Ra and the virtual operation unit image 181 for mouse cursor control shown in Figure 11.

[0110] Alternatively, the mouse cursor may be positioned directly on the surface of the mouse cursor input screen 160 shown in Figure 11, with the mouse cursor following and moving under the tip of the index finger of a dotted hand-like object such as hand image 91Ra or hand image 91La (see Figure 17).

[0111] The mouse cursor 176 on the screen display unit 161 performs left-click, right-click, move, and drag operations by the thumb 93R on the virtual operation unit 131 for mouse cursor operation, and various input operations are performed on the light operation button 171, the light operation button 172, the call button 173, the intercom button 174, and the display unit 175.

[0112] <Image Processing for Flick Input> Figure 13 is a sixth explanatory diagram showing detailed image processing for AR glasses of a user interface system according to the first embodiment of the present invention, and shows a flick input operation screen that is displayed as a spatial display image on the operation display panel by flicking the virtual operation unit 141 for flick toggle input.

[0113] Figure 14 is a seventh explanatory diagram showing detailed image processing for AR glasses of a user interface system according to a first embodiment of the present invention, where Figure 14(a) shows flick input by the right hand 91R of the injured person 14, and Figure 14(b) shows the image displayed for flick input.

[0114] The user interface system 1 shown in Figure 1 detects the operation of the injured person 14's right hand 91R on the flick / toggle input virtual operation unit 141 based on the image of the injured person 14 captured by the imaging means (first and second video cameras 41 and 42) that capture images of the injured person 14. By flicking the flick / toggle input virtual operation unit 141, the system displays the flick input operation screen 190 shown in Figure 13 on the AR glasses 2. In this case, the AR glasses 2 displays the flick input operation screen 190 as a spatial display image 10-1 of the operation display panel on a vertically elongated virtual surface 8-1 shown in Figure 13, located slightly below the first video camera 41, using stereoscopic 3D image display.

[0115] In Figure 14, the flick input screen 190 corresponds to the upper part of the smartphone screen. It consists of a screen display unit 191 and a lower operation unit display unit 192 that shows a virtual operation unit 141 for flick / toggle input.

[0116] The screen display unit 191 shows the operation buttons for the light 201, the operation button for turning on the light 202, the call button 203, the intercom button 204, and the display unit 205 which displays the entered characters and numbers.

[0117] In Figure 13, the operation unit display unit 192 displays a simplified keyboard display 211 for smartphones, corresponding to the flick / toggle input virtual operation unit 141 shown in Figure 10.

[0118] Furthermore, the server 6 (see Figures 1 and 3) performs image analysis on the video data received from the first and second video cameras 41 and 42 (see Figures 1 to 3) to determine the position and angle of the thumb 93 of the hand 91, and simultaneously displays a semi-transparent thumb image 93Ra corresponding to the thumb 93R of the right hand 91R shown in Figure 14(a) on the operation unit display unit 192 shown in Figure 13.

[0119] When the injured person 14, at a position below the spatial display image 10-1 (virtual surface 8-1) of the operation display panel, presses the part corresponding to key 212 of the keyboard display 211 shown in Figure 13 with the thumb 93R of the right hand 91R on the operation surface 142 of the virtual operation unit 141 for flick / toggle input, which corresponds to a virtual smartphone that does not actually exist as shown in Figure 14(a), the operation unit display unit 192 displays the image 193 for flick input shown in Figure 14(b), and a semi-transparent thumb image 93Ra is displayed superimposed on the image 193. Image 194 within image 193 shows five keys arranged in a cross shape, and by sliding the thumb 93R of the right hand 91R up, down, left, and right, the semi-transparent thumb image 93Ra moves up, down, left, and right, allowing the desired key to be selected. By continuously selecting the desired key, commands are sent to server 6, allowing for the operation of devices such as the intercom in patient room 12, the light switches in patient room 12, and the call bell installed in the control room 17, as well as the display of messages.

[0120] For example, when the word "intercom" is entered using flick input, the word "intercom" is displayed on the display unit 205 of the upper screen display unit 191, the intercom between the patient room 12 and the management room 17 is connected, and the intercom becomes usable.

[0121] <Image Processing for Toggle Input> Figure 15 is an explanatory diagram showing operation to the virtual operation unit for flick toggle input of the user interface system according to the first embodiment of the present invention, and shows toggle input by the right hand 91R of the injured person 14. Figure 16 is an eighth explanatory diagram showing detailed image processing for AR glasses of the user interface system according to the first embodiment of the invention, and shows the flick input operation screen displayed as a spatial display image on the operation display panel by a flick operation of the virtual operation unit 141 for flick toggle input. Operation to the virtual operation unit for flick toggle input is usually in the form shown in Figure 14, but it can also be in the form shown in Figure 15.

[0122] The user interface system 1 shown in Figure 1 detects the operation of the injured person 14's right hand 91R on the flick / toggle input virtual operation unit 141 based on the image of the injured person 14 captured by the imaging means (first and second video cameras 41 and 42) that capture images of the injured person 14. The toggle operation on the flick / toggle input virtual operation unit 141 causes the AR glasses 2 to display the toggle input operation screen 220 shown in Figure 16. In this case, the AR glasses 2 displays the toggle input operation screen 220 as a spatial display image 10-1 of the operation display panel on a vertically elongated virtual surface 8-1 shown in Figure 16, located slightly below the first video camera 41, using stereoscopic 3D image display.

[0123] In Figure 16, the toggle input operation screen 220 consists of an upper screen display section 221, which corresponds to the upper part of the smartphone screen, and a lower operation section display section 222, which shows a virtual operation section 141 for flick / toggle input.

[0124] The screen display unit 221 shows the operation buttons for the lights 231, the operation button for turning on the lights 232, the call button 233, the intercom button 234, and the display unit 235 which displays the entered characters and numbers.

[0125] In Figure 16, the operation unit display unit 222 shows a simplified keyboard display 241 for smartphones, corresponding to the flick / toggle input virtual operation unit 141 shown in Figure 10.

[0126] Furthermore, the server 6 (see Figures 1 and 3) performs image analysis on the video data received from the first and second video cameras 41 and 42 (see Figures 1 to 3) to determine the position and angle of the thumb 93 of the hand 91, and simultaneously displays a semi-transparent thumb image 93Ra corresponding to the thumb 93R of the right hand 91R shown in Figure 15 on the operation unit display unit 222 shown in Figure 16.

[0127] When the injured person 14, at a position below the spatial display image 10-1 (virtual surface 8-1) of the operation display panel, presses once with the thumb 93R of the right hand 91R on the operation surface 142 of the virtual operation unit 141 for flick-toggle input corresponding to a virtual smartphone that does not actually exist as shown in Figure 15, the semi-transparent thumb image 93Ra superimposed on the operation unit display unit 222 performs the pressing action, and the character "あ" is displayed on the display unit 235. When the injured person 14 presses the part corresponding to key 242 again, the semi-transparent thumb image 93Ra performs the pressing action, and the character "い" is displayed on the display unit 235. When the injured person 14 presses the part corresponding to key 242 yet again, the semi-transparent thumb image 93Ra performs the pressing action, and the character "う" is displayed on the display unit 235. On the toggle input screen 220, the user can select desired characters or numbers by using the thumb 93R of the right hand 91R to toggle input. By continuously selecting desired characters or numbers, a command is sent to the server 6, which can then be used to operate devices such as the intercom in the patient room 12, the light switches in the patient room 12, and the call bell installed in the control room 17, as well as to display messages.

[0128] For example, if the text "light switch on" is entered via toggle input, the text "light switch on" will be displayed on the display unit 142, and the lights in the patient room 12 will be turned on.

[0129] <Video Processing for Full Key Input> Figure 17 is a ninth explanatory diagram showing detailed video processing for AR glasses of a user interface system according to the first embodiment of the present invention, and shows a full key input operation screen that is displayed as a spatial display image on the operation display panel by operating the virtual operation unit 151 for full key input.

[0130] Figure 18 is an explanatory diagram showing the operation of the virtual operation unit for full key input of a user interface system according to the first embodiment of the present invention, and shows a view from the right side of a patient 14 performing full key input in a first posture.

[0131] Figure 19 is an explanatory diagram showing images captured by a first video camera 41 of a video camera stand 4 of a user interface system according to a first embodiment of the present invention. Figure 19(a) shows a view from above of a patient 14 performing full keyboard input in a first posture. Figure 19(b) shows a view from above of a patient 14 performing full keyboard input in a second posture.

[0132] The user interface system 1 shown in Figure 1 detects the operation of the injured person 14's right hand 91R on the virtual operation unit 151 for full key input based on the image of the injured person 14 captured by the imaging means (first and second video cameras 41 and 42) that capture images of the injured person 14. Upon operation on the virtual operation unit 151 for full key input, the system switches to full key input mode and displays the full key input operation screen 250 shown in Figure 17 on the AR glasses 2. In this case, the AR glasses 2 displays the full key input operation screen 250 as a spatial display image 10-2 of the operation display panel on a vertically elongated virtual surface 8-2 shown in Figure 17, located slightly below the first video camera 41, using stereoscopic 3D image display.

[0133] In Figure 17, the full key input operation screen 250 consists of an upper screen display section 251, which corresponds to the upper part of the smartphone screen, and a lower operation section display section 252, which shows the virtual operation section 151 for full key input.

[0134] The screen display unit 251 displays the operation buttons for the lights 261, the operation button for turning on the lights 262, the call button 263, the intercom button 264, and the display unit 265 which displays the entered characters and numbers. The operation unit display unit 252 displays the full-key input type keyboard display 271 and the touchpad display unit 272 which displays the touchpad.

[0135] Furthermore, the server 6 (see Figures 1 and 3) performs image analysis on the video data received from the first and second video cameras 41 and 42 (see Figures 1 to 3) to determine the position and angle of the palms and fingers of the left hand 91L and right hand 91R shown in Figure 18, and simultaneously displays semi-transparent left-hand image 91La and right-hand image 91Ra corresponding to the left hand 91L and right hand 91R shown in Figure 18 on the operation unit display unit 252 shown in Figure 17.

[0136] Furthermore, in the first embodiment of the present invention, even if the left hand 91L and the right hand 91R are separated, input can be performed with a display on the keyboard display 271 that shows the virtual left-hand image 91La and the right-hand image 91Ra appearing close together.

[0137] When performing full key input in the first posture, as shown in Figure 18, the injured person 14 bends the elbows of both arms 94L and 94R into a V-shape, and with the palms of the left hand 91L and right hand 91R facing towards the feet, at a position below the spatial display image 10-2 of the operation display panel, as shown in Figures 18 and 19(a), the injured person 14 presses the corresponding parts of the full key input type keyboard display 271 shown in Figure 17 with the fingers of the left hand 91L and right hand 91R on the virtual operation unit 151 for full key input, which corresponds to a virtual full key input type keyboard that does not actually exist. The fingers of the translucent left hand image 91La and right hand image 91Ra, which are superimposed on the operation unit display unit 252, perform the pressing action. For example, when the index finger of the right hand 91R presses key 273, the letter "U" is displayed on the display unit 265. By continuously selecting the desired key, commands are sent to server 6, allowing for the operation of devices such as the intercom in patient room 12, the light switches in patient room 12, and the call bell installed in the control room 17, as well as the display of messages.

[0138] For example, when the characters "OHAYOU" are entered using flick input, the characters "OHAYOU" are displayed on the display unit 265, and the characters "OHAYOU" are also displayed on the curtain 27 via the projector 3.

[0139] When performing full key input in the second posture, as shown in Figure 19(b), the patient 14 extends both arms 94L and 94R downwards, and with the palms of their left hand 91L and right hand 91R facing downwards, at a position diagonally downwards from the spatial display image 10-2 of the operation display panel, they press the corresponding parts of the full key input type keyboard display 271 shown in Figure 17 with the fingers of their left hand 91L and right hand 91R against the virtual operation unit 151 for full key input. The fingers of the translucent left hand image 91La and right hand image 91Ra, which are superimposed on the operation unit display unit 252, perform the pressing action. For example, if the index finger of the right hand 91R presses key 273, the letter "U" is displayed on the display unit 265. By continuing to select the desired key, a command is sent to the server 6, allowing the operation of equipment such as the intercom in the patient room 12, the light switch in the patient room 12, and the call bell installed in the management room 17, as well as the display of messages.

[0140] Furthermore, by operating the virtual operation unit 151 for full key input that corresponds to the touchpad display unit 272, touchpad input operations become possible, enabling operations such as moving the mouse cursor, left-clicking, right-clicking, and dropping.

[0141] Figure 20 is a tenth explanatory diagram showing detailed image processing for AR glasses of a user interface system according to the first embodiment of the present invention, and displays a full key input operation screen 280 which is displayed as a spatial display image 10-3 of the operation display panel displayed on the virtual surface 8-3 by selection and adjustment using the left hand 91L and the right hand 91R.

[0142] As shown in Figure 20, the spatial display image 10-3 is adjusted to be wider than the spatial display image 10-2 shown in Figure 17 by adjustment using the virtual rod of light 92 held in the hand 91 of the injured person 14 shown in Figure 7. It is being done.

[0143] In Figure 20, the full key input operation screen 280 consists of an upper screen display section 281 corresponding to the upper part of the smartphone screen and a lower operation section display section 282 showing the virtual operation section 151 for full key input.

[0144] The screen display unit 281 shows the operation buttons for the lights 291, the operation button for turning on the lights 292, the call button 293, the intercom button 294, and the display unit 295 which displays the entered characters and numbers.

[0145] The control panel display unit 282 displays the left and right keyboards 301L and 301R, which are full-key input type keyboards, and the touchpad display unit 302, which displays the touchpad.

[0146] Furthermore, server 6 (see Figures 1 and 3) performs image analysis on the video data received from the first and second video cameras 41 and 42 (see Figures 1 to 3) to determine the position and angle of the palms and fingers of the left hand 91L and the right hand 91R (see Figure 18), and simultaneously displays semi-transparent left-hand image 91La and right-hand image 91Ra corresponding to the left hand 91L and right hand 91R on the operation unit display unit 282 of the full key input screen 280 shown in Figure 20.

[0147] The left keyboard display 301L and the right keyboard display 301R are separated horizontally. Accordingly, the left-hand image 91La and the right-hand image 91Ra are also separated horizontally. The left keyboard display 301L and the left-hand image 91La correspond to the operation of the left and right hands 91L of the injured person 14, and the left keyboard display 301L and the right-hand image 91Ra correspond to the operation of the right hand 91R of the injured person 14.

[0148] The size and position of each key on the left and right keyboards 301L and 301R are adjusted based on the habits of the injured person 14, as determined by the video detection results from the first and second video cameras 41 and 42, as well as customization and calibration by the injured person 14, their family, medical personnel, etc.

[0149] By operating the virtual operation unit 151 for full key input, which corresponds to the touchpad display unit 302, touchpad input becomes possible, enabling operations such as moving the mouse cursor, left-clicking, right-clicking, and dropping.

[0150] In the first embodiment, if there is something, such as a mousepad, under the little finger, hypothenar eminence, or ulnar head of the injured person 14, the virtual operation unit 151 for full key input is set taking that object into consideration and its surface into account.

[0151] Furthermore, in the first embodiment, when inputting keys, the user opens their hand from a gripping position and places it on the virtual surface of the virtual operation unit 151 for full key input. From this point onward, the position of this virtual surface remains unchanged. To change its position, the user performs a gesture in setting mode, that is, returns their hand to a gripping position. This causes the area around the little finger, hypothenar eminence, and the base of the ulnar head to become the virtual surface, and the position of this virtual surface remains unchanged from the moment the hand is opened from the gripping position. Full key input and touchpad operation become possible on this virtual surface.

[0152] When switching mouse cursor operation, touching the virtual surface of the virtual operation unit 151 for full keyboard input and moving the entire unit to a certain extent will cause the mouse cursor to move properly, allowing you to locate it. From this point, you enter mouse cursor operation mode. When switching to full keyboard input, the system switches to full keyboard input mode when the hand position is set to the input position (home position) relative to this surface, and when the index finger, little finger, or specific designated fingers appear in the expected positions.

[0153] Furthermore, in the user interface system 1 of the first embodiment shown in Figures 1 to 20, the touchpad area can be specified and set, or it may be automatically set from the center of the hand, the range of motion of the fingers, or any set point or position.

[0154] Each individual can configure each virtual face and key according to their preferences, or they can be automatically adjusted based on the shape of the hand and the range of motion of the hand and fingers. When determining the range of motion, individuals can set the pivot point of the hand and finger movements, or it can be automatically recognized and calculated by AI or other means. In addition to the three, new faces can be set near the hand, or parts of each virtual face can be set as virtual face areas that can be used to instruct different movements.

[0155] The interface of the virtual operation unit 131 for mouse cursor operation, the virtual operation unit 141 for flick / toggle input, and the virtual operation unit 151 for full key input can be visualized as not being limited to a flat plane. Touching a virtual surface indicates physical contact with something. For example, when operating the mouse wheel, rotation occurs because the hand is touching the virtual surface. Actions that do not involve contact will not cause the mouse wheel to rotate. Also, when the hand, which has become a mouse in mouse mode, moves while touching the virtual surface, that amount (the amount varies depending on the settings) corresponds to the movement of the mouse cursor. If the hand moves while away from the virtual surface, the mouse cursor does not move because the mouse is in a state of being away from the surface. This is similar to a mouse ball floating in the air after leaving the mousepad. The same applies to key input. The virtual surface has two purposes: for mouse cursor operation and for key input.

[0156] Furthermore, in the first embodiment of the user interface system 1 shown in Figures 1 to 20, the first video camera 41 on the video camera stand 4 captures images of the user (injured or ill person 14) from above, and the second video camera 42 on the video camera stand 4 captures images of the user (injured or ill person 14) from the right side. However, there may be two video cameras, one for the right hand and one for the left hand, for capturing images of the user (injured or ill person 14) from the side.

[0157] The virtual control unit 131 for mouse cursor operation, the virtual control unit 141 for flick / toggle input, and the virtual control unit 151 for full keyboard input are designed to allow operation even while looking at the ceiling, as the grip of the hand may be detected from below, and operation is possible if recognition is possible.

[0158] In the first embodiment of the user interface system 1 shown in Figures 1 to 20, when entering key input mode, the virtual surfaces of the virtual operation unit 131 for mouse cursor operation, the virtual operation unit 141 for flick / toggle input, and the virtual operation unit 151 for full key input are temporarily fixed at that point.

[0159] Specifically, the idea is to use a smartphone, which has no weight or shape, by attaching it to the top of a device, or to hold it in your hand, or to place a keyboard, mouse, or touchpad under your hand.

[0160] Furthermore, the virtual surfaces have shapes and angles that make them easy to operate. There are also three sets of virtual surfaces. One virtual surface is above the grip for flick input, toggle input, and thumb mouse cursor operation—in short, the upper smartphone virtual surface; the smartphone virtual surface is closer to the hand; and there are two sets of full key input and finger mouse cursor operation on the virtual surfaces for the fingers, hypothenar eminence, and ulnar head. The first upper smartphone virtual surface is for mouse cursor operation by gripping, and also by piercing with the thumb, which allows for more direct and precise movements. Of course, flick input and toggle input can also be performed on the upper surface. In that sense, it is like the upper smartphone surface. This third virtual surface can also be divided into four. The distance from the hand can be changed and set. When operating by gripping, if the virtual surface is too close, the unit of operation becomes coarse. Conversely, piercing upwards with the thumb may be easier if it is closer.

[0161] Furthermore, the shape of the virtual operation unit 151 for full key input before entering full key input mode may be set to a position where the fingers do not overlap with each other, are spread within a certain range, three or more fingers are slightly bent, and there is a little time to adjust the hand shape, or to tap all five fingers in order, or to tap several fingers in succession, or to set it to a unique shape.

[0162] Furthermore, the left and right hands can be in different modes. If you open your hand wide and move the entire hand, it will enter mouse cursor mode. Even if you hold a mouse or make a fist, moving the entire hand will enter mouse mode.

[0163] It guides the user to where their fingers are. In other words, it displays the key layout screen for full key input or flick input, and also shows it at the bottom of the screen, overlaid on the shape of the hand. If the key layout is automatically hidden and the user is making input errors, a function to automatically reappear may be added.

[0164] You could create a three-dimensional guide using a 3D printer, or you could make something like a mousepad to help people understand it by touch.

[0165] For users who input using only one finger, the system won't switch between mouse mode and full keyboard input mode, so a button for mode switching will be created. This button could be a key that is already configured, or it could be placed in an area that is not touched, allowing the user to switch modes manually.

[0166] When moving the mouse cursor with your thumb, at normal speed, the cursor moves precisely from one end to the other. However, if you move it slowly, it moves only a little, allowing for more precise and accurate movement. As with any hand shape, if they are similar enough, AI can automatically detect them.

[0167] Furthermore, in the user interface system 1 of the first embodiment, point (pointer) images 92b and 95b were displayed at the tips 92a and 95a of the virtual rods 92 and 95 held in the hand 91 of the injured person 14. However, the tips 92a and 95a of the virtual rods 92 and 95 may be configured to display a mouse cursor image. These point (pointer) and mouse cursor images are operated by the virtual rods 92 and 95, but are operated by a virtual surface 8 that is located slightly below the first video camera 41, which is the upper camera, and displays the spatial display image 10 of the operation display panel. In the user interface system 1, the positions of the points (pointers) and mouse cursors at the tips 92a and 95a of the virtual rods 92 and 95 become smoother due to the individual range of motion of each person.

[0168] Furthermore, in the user interface system 1 of the first embodiment, operation display panels for flick input and toggle input (operation display section 192 of the flick input operation screen 190, operation display section 222 of the flick input operation screen 220) are displayed to enable key input operations by using the thumb 93R. However, key input operations may also be enabled using fingers other than the thumb 93R, such as the index finger.

[0169] Furthermore, in the user interface system 1 of the first embodiment, an operation display panel for full key input (operation display section 252 of the full key input operation screen 250) is displayed, and key input operations are made possible by movements of the fingers of the right and left hands. However, key input operations may also be made possible by movements of the fingers of one hand.

[0170] Furthermore, in the user interface system 1 of the first embodiment, finger key press operations are basically detected by the first video camera 41 of the upper camera, but they may also be detected by the second video camera 42 of the lower camera.

[0171] Furthermore, in the user interface system 1 of the first embodiment, the system detects finger operations on a virtual stick and finger input operations on virtual operation units (virtual operation unit 131 for mouse cursor operation, virtual operation unit 141 for flick / toggle input, virtual operation unit 151 for full key input) corresponding to operation display panels for flick input, toggle input, or full key input (operation unit display unit 192 of the flick input operation screen 190, operation unit display unit 222 of the flick input operation screen 220, operation unit display unit 252 of the full key input operation screen 250). However, it is also possible to detect finger operations on a virtual mouse and perform various inputs and controls. In this case, calibration of the mouse operation range and the center position of operation may be performed on virtual surfaces 8, 8-1, 8-2, and 8-3. Calibration may be performed only once initially or before operation.

[0172] Furthermore, in the user interface system 1 of the first embodiment, the operation display panel (operation display unit 192 of the flick input operation screen 190, operation display unit 222 of the flick input operation screen 220, and operation display unit 252 of the full key input operation screen 250) is a virtual keyboard, so it is possible to set the split keys for the right hand and left hand, as well as the arrangement and size of each key. In addition, automatic adjustment may be made based on the point that moves the least during operation.

[0173] Furthermore, in the user interface system 1 of the first embodiment, it may be possible to determine keys such as "a", "i", "u", "e", and "o" in stages at a new certain depth or range. For example, from the normal position of the thumb, 0 to -5 mm in the depth direction would be "a", -5 mm to -10 mm would be "i", -10 mm to -15 mm would be "u", -15 mm to -20 mm would be "e", and -20 mm to -25 mm would be "o".

[0174] Furthermore, in the user interface system 1 of the first embodiment, functions such as pointer control, right-click, drag, left-click, drag, right-double-click, and left-double-click can be implemented by changing the grip shape, right-hand or left-hand operation, and mode changes.

[0175] Furthermore, in the user interface system 1 of the first embodiment, as described above, when the injured person 14 looks at the image of the AR operating space 71 in Figure 5(b) using the AR glasses 2 and aligns the tip 92a or the image 92b of the virtual rod of light 92 held in the injured person 14's hand 91 with one of the light operation buttons 111, the light operation button 112, the call button 113, the intercom button 114, or the key input mode selection button 115 on the selection input operation screen 110 shown in Figure 8, the corresponding input screen may be automatically displayed. However, considering practical use, it may also be configured so that even if there is slight movement within a set number of seconds, if it is determined that the area on the button has been occupied for more than the set number of seconds, it is judged that the button has been pressed.

[0176] Thus, the system may be configured so that the operation is performed even if the light control button 111, the on / off control button 112, the call button 113, the intercom button 114, or the key input mode selection button 115 is not pressed, as long as the bright spot remains on the button for a set number of seconds or longer.

[0177] Related to this, instead of a set number of seconds, it's also possible to consider a method where the pressure is applied by, for example, reducing the size of the grip, that is, by squeezing the grip, or by pushing.

[0178] In this way, the injured person 14 can adjust the operating points of the operation buttons in the augmented reality space provided by the AR glasses 2 by changing the grip of their hand 91. These buttons include the light control button 111, the power-on control button 112, the call button 113, the intercom button 114, and the key input mode selection button 115.

[0179] Furthermore, before starting operations using the light operation button 111, the light operation button 112, the call button 113, the intercom button 114, and the key input mode selection button 115, the center position of the hand 91 may be temporarily placed at the location where the hand 91 was, or at the location where the observation device (the first and second video cameras 41 and 42, and the circuit and software related to aerial operation detection of the server 6) determined that the hand was not moving for a set period of time, or at the location where the observation device determined that the hand was not moving to a certain extent during operation. Alternatively, the center position of the hand 91 may be placed, and a sphere may be calculated and created from that center using the radius of the sphere and the set radius. From that center position or the center position of the sphere, if the hand is thrust forward beyond a set distance toward the bright spot display surface (display surface of bright spots (92b, 95b)) of the user interface of the present invention (virtual rods of light 92, 95), it may be clicked. If the hand is thrust further forward and moved up, down, left, or right, it may be dragged up, down, left, or right. To release the drag state, the drag may be considered to have ended when the position approaches the center of the original hand 91 or the center of the sphere, relative to the drag reference point. Similarly, a click may be considered to have ended when the position approaches the center of the original hand 91 or the center of the sphere, relative to the click reference point.

[0180] Furthermore, the observation device captures the position of the injured person's face 14 and the hand 91 moves more As the hand 91 moves closer to the face within a set distance range, the speed of the glowing point may be adjusted to move slowly and precisely according to the set distance range. In this case, the hand 91 may be positioned in front of the face.

[0181] Furthermore, the bright spot may be the extension of a line connecting the center of the injured person's (14) head to their fingertips, or the extension of a straight line connecting the center of the head to a certain position on a virtual rod. The point at the center of the head may be set to indicate how far away it is in a certain direction. Also, to avoid capturing the vibrations of the person (injured person 14) in fine detail, a certain degree of shaking does not need to be reflected in the movement of the bright spot as it is considered error or noise.

[0182] Furthermore, the injured person 14 may place their hand 91 on their leg or on a table to stabilize the movement of the bright spot.

[0183] Instead of the light operation button 111, the on operation button 112, the call button 113, the intercom button 114, and the key input mode selection button 115 on the selection input operation screen 110 displayed in the spatial display image 10 of the operation display panel, other input selection buttons may be displayed as images of the first layer. Also, if there is an interface beyond the ceiling 18, a group of interfaces of another layer will be displayed.

[0184] For example, when the interface of the virtual rods of light 92 and 95 is directed towards the wall (ceiling 18) directly above the injured person (patient 14), the nurse call button, the switches for the patient's reading lamp and television receiver, etc., will appear as images in the second layer.

[0185] Furthermore, when the interface formed by the virtual light rods 92 and 95 is directed towards the room's entrance, the switches for the air conditioning, air conditioner, and the room's sailing lights appear as images of the third layer.

[0186] Furthermore, in the first embodiment, it is also possible to display a group of switches in another room. Similar to tabs in a web browser, it is possible to display, for example, five layers in tabs.

[0187] Furthermore, in the first embodiment, each layer may be configured such that when the destination of the virtual light rods 92, 95 changes (such as the top of a wall, the east or north side, or the east or west side of a curtain, or the first monitor surface, the second monitor surface, the second monitor surface, etc.), the first to fifth layers appear according to the direction.

[0188] Furthermore, in the first embodiment, instead of displaying tabs or similar information under normal circumstances, special settings may be provided for administrators or other personnel to operate. For example, in a small hospital, the status of cameras in each room, the on / off status of the air conditioning and sailing lights could be displayed at the entrance, and these could be turned on / off and adjusted, eliminating the need for manual operation. This would also reduce the number of monitors required. In a larger hospital, similar special settings could be provided at the nurses' station on each floor.

[0189] Furthermore, in the first embodiment, image analysis was performed on the video data received from the first and second video cameras 41 and 42 (see Figures 1 and 3) to calculate the degree to which the hand 91 of the injured person 14 was gripped in the AR operating space 71 (see Figure 5(a)). However, in addition to calculating the hand void, the system may also be configured to observe changes in the basic area and volume of the hand, such as an increase in area or volume from the side or length, and to perform a specified action if it exceeds a baseline value.

[0190] Furthermore, when using multiple first and second video cameras 41 and 42, if the operating speed becomes slow or the change in size becomes small in order to speed up calculations, the multiple first and second video cameras 41 and 42 may be combined into one to perform calculations only for area and edge shape. However, when one of the first and second video cameras 41 and 42 is used to perform calculations only for area and edge shape, the settings should be configured to automatically set the error to ±3, centimeters, square centimeters, or 3 cubic centimeters based on factors such as gender, date of birth, height, weight, body fat percentage, arm circumference, hand size, and finger circumference, to account for tremors during human movement and errors in shape recognition during movement, or to be set arbitrarily. Therefore, this can be applied to one or multiple video cameras as imaging means.

[0191] Furthermore, the direction and point of the virtual rod of light in the first embodiment shown in Figure 6 may be determined by the surface shape of the hand 91 of the injured person 14.

[0192] Furthermore, in the first embodiment, the approach to error in the hand 91 is similar; if the movement is reproduced too accurately, the point at the end of the virtual rod of light will waver, so it is undesirable to reproduce the movement too accurately. Therefore, it is conceivable to implement noise countermeasures, such as not detecting movement above a certain speed or not detecting movement above a certain frequency. It is also conceivable to perform interpolation processing on the trajectory of the hand 91.

[0193] The thickness of the images 92b and 95b at the ends of the interfaces (virtual rods of light 92 and 95) can be narrowed, or made thinner, in accordance with the movement of the hand 91. This can be done by pressing a switch.

[0194] After extending the interface (virtual rods of light 92, 95), it is possible to move it parallel to the projection surface: the contact surface of the virtual rod (curtain, wall, ceiling, monitor) while maintaining the same distance from that surface (in this case, a certain degree of error is acceptable, and this error can be handled in the settings), which produces the same effect as dragging with a mouse (the trajectory of the movement is processed as if it is being pressed and traced, the strength of which can be set, or if the distance of the pushing motion can be measured, it is possible to reflect that in the degree of pressure applied (the depth to the contact surface can also be measured, in which case the normal contact surface is imagined as the tip of the virtual rod touching it, and in some cases it is better to think of it this way). As mentioned above, a certain degree of noise and error processing is performed.

[0195] In cases where the operation has the same effect as dragging with a mouse, it becomes possible to flip through the operation screen.

[0196] Furthermore, actions that thicken the interface (virtual light bars 92, 95) or extend it forward can be used to simulate pressing a switch. Similar to mouse clicks, performing these actions consecutively can be used to simulate a double-click.

[0197] The movement speed of the light points displayed on the interface (virtual light bars 92, 95) and the AR glasses 2 or contact surfaces (curtains, walls, ceilings, monitors) requires adjustment, and calculations are needed to suppress and control vibrations that occur when a person operates the system. To this end, measures such as not capturing input movements above a certain frequency, or not capturing small movements, that is, not capturing movements below a certain distance as movement and not representing them as light point movements, will be implemented. For example, the vibration of a person's hand could be corrected by the camera capturing that movement, that is, the movement of the captured object could be extracted, and the vibration of the hand itself could be removed. Alternatively, assistive devices could be attached to the hand to prevent vibration, if necessary. Something like a hand cast or a stand to rest the hand on could be provided.

[0198] Furthermore, the processing for the same interface shape may be modified. For example, while the virtual light bars 92 and 95 are displayed, clicking with two fingers held up (like making a peace sign with your fingers together) may be treated as a right-click on a mouse. Different modes may be set depending on the hand shape.

[0199] Furthermore, the first embodiment can be modified in various ways. For example, the display unit can be not only an AR space or curtain displayed on AR glasses, but also walls, LCD monitor screens, and other similar elements.

[0200] Furthermore, in the first embodiment, the direction of the injured person's face and gaze may be assumed, and the screen displaying the projection target and buttons of the projector 3 may be limited to the direction in which the person is thought to be looking. In other words, normally, the screen displaying the projection target and buttons of the projector 3 is the direction pointed to by a finger, or a curtain or wall into which a virtual rod of light is pierced, or a monitor (for example, an LCD monitor as a display means), but it is also possible to ignore this and project in the direction the person is looking. This is suitable when the injured person 14 has an extremely different orientation of their face and body.

[0201] Furthermore, in the first embodiment, the initial image is that the hand is clenched and operated as shown in Figure 5. However, assuming two fingers as the default, if three fingers are used for operation, the virtual rod of light can be made thicker as shown in Figure 6(a), and if one finger is used for operation, the virtual rod of light can be made thinner as shown in Figure 6(b). Alternatively, this mechanism can be considered not as adjusting the thickness, but as representing the amount pushed forward during the poking motion (which can also be considered as the depth from the display surface by the projector 3). In other words, the difference in the number of fingers from the default two-finger operation can be considered as the amount pushed forward (depth) during the poking motion. Alternatively, if the virtual rod of light is considered as a virtual rod of flame, it can be considered as a process that changes the output or temperature of that virtual rod.

[0202] Furthermore, in the first embodiment, the shape of the virtual bar of light dragged onto the control button can also be given meaning. For example, by dragging and drawing a circle (or any shape such as a triangle or an X, as long as it is set), it is possible to set various actions, such as turning on the air conditioner switch in the current room or turning off the light switch.

[0203] Furthermore, in the first embodiment, if the force of the drag operation or the amount pushed forward during the thrusting operation (which can also be considered as the depth from the display surface by projector 3) is insufficient, it is possible to represent the physical properties of the wall or the properties of the virtual flame rod (of course, it is also possible to set it as a weak wall, or usually not assume a wall at all). For example, if a virtual rod of light is assumed, the amount (value) that must be pushed forward during the thrusting operation for the light to penetrate to another room is determined, and if this value is exceeded, the virtual rod of light reaches the adjacent room. Alternatively, the amount of pushing may be determined to show that if the amount pushed is insufficient, it will not reach, or the wall will be damaged.

[0204] This level of pressure may be reproduced if it can be replicated when operating buttons or other controls on the control surface.

[0205] When operating buttons or other controls equipped with pressure sensors, the operation simulation device can simulate the gradual pressing of these controls, allowing for gradual increases in pressure (within the trajectory of the drag operation) or a gradual decrease in pressure from a strong press (within the trajectory of the drag operation).

[0206] To give you a concrete example, it can even recreate the feeling of tracing your finger from the center of a fairly large button to the left.

[0207] To summarize, the images on the operation screens of the AR glasses 2 and the curtain 15 have an operation reproduction device that reproduces operations such as gradually pressing by the user. When the user operates a button with a pressure sensor, the operation reproduction device can reproduce operations such as gradually pressing harder, and operations that gradually decrease in pressure from a strong press.

[0208] Furthermore, in the first embodiment, it becomes possible to press adjacent buttons in succession, or to operate adjustments such as adjusting the brightness by pressing down on a bar with appropriate force and controlling its shape in directions such as left, right, up, and down. In this application, it would be possible to create an image of carving into a wall, or to draw pictures or letters with depth on a wall. This information is reflected in the display using AR glasses 2 or projector 3. Of course, it could also be physically reproduced on the wall by a robot. This information can also be used for building wall maintenance, etc.

[0209] By combining this information with settings, it becomes possible to virtually destroy the wall of an adjacent room, which will then display and allow operation of the buttons in that room. Alternatively, by destroying a wall, it may be possible to display and allow operation of a control panel (level) for only the air conditioners in the entire building, or a control panel (level) for only the light switches, rather than a room wall.

[0210] As another example, depending on the amount pushed forward during the pushing motion (which could be considered the depth from the ceiling 18 visible through the AR glasses 2 or the display surface projected by the projector 3), buttons on a certain floor (e.g., the 5th floor) could be displayed. If the amount pushed falls outside a certain range, buttons on a different floor (e.g., the 4th floor) could become operable.

[0211] Furthermore, in the first embodiment, if a virtual rod of light is directed towards a certain point on an operation button and dragged to draw a circle or the like (it would be sufficient if an area or enclosed space could be specified), then it is conceivable that all the buttons contained within that circle, or in other words, the cylinder of the building, would be arranged on the operation surface, ignoring the idea of ​​separate rooms. This arrangement is thought to be useful for checking and locating malfunctions during maintenance. For example, if the operating status of not only the equipment buttons in the rooms in that direction, but also the buttons of other devices such as refrigerators and televisions can be seen, then the operation of individual devices (refrigerators, televisions, etc.), as well as the building and room equipment, can be checked, potentially leading to an efficient determination of the fault location. Here, refrigerators and televisions are included, but of course, refrigerators and televisions are also considered to be targets of operation during normal use. Also, when a maintenance worker arrives at the building, such a human-readable interface would likely be more efficient and effective.

[0212] Furthermore, in the first embodiment, for maintenance purposes, cameras and buttons (including buttons for refrigerators and televisions) are placed in each room, and communication is conducted to obtain information such as the status of each piece of equipment and device. This information is then classified, and only switches of the same classification, such as only air conditioner switches (remote control information), are grouped together and arranged on the control panel, regardless of the room. (Of course, there can be two or three types if they can be efficiently arranged on the control panel.) The control panel may also be a single tab (hierarchy). In this case, to indicate the status of the switches, they may be displayed as such even if they are not actually fluorescent switches. In addition, the operating status of complex equipment such as air conditioners may be displayed, for example, next to the respective air conditioner switches, or on the control panel itself. In this way, the operating status of the air conditioners, etc., can be easily seen at a glance by displaying information from cameras and switches in each room, or from devices acting as information gathering devices (which also monitor temperature and electricity usage, etc.), which are not limited in form, alongside the relevant switches. Of course, by compiling relevant information, maintenance and the actual current situation can be understood without having to go to the site, and appropriate adjustments can be made efficiently.

[0213] Furthermore, in the first embodiment, larger-than-usual buttons may be displayed in order to accurately identify the buttons.

[0214] The size, magnification, and reduction ratio of the buttons may be maintained by the settings of the user interface system components (e.g., server 6), linked to the thickness of the virtual light bar, or configured to be set by automatically detecting the user's circumstances (age, whether or not they wear glasses, prescription of glasses, i.e., a numerical representation of the corrective strength of the lenses, etc., which are actually input and set), or the conditions of the contact surface between the buttons or virtual light bar and the wall (distance from the person to the wall, the darkness of the environment, the brightness of the display surface, etc., which may be input in the settings).

[0215] Furthermore, in the first embodiment, the bright point (pointer) that serves as an indicator for operation, the tip of the operator's finger, and the point at the end of the virtual rod of light being held virtually (calculated as the point of contact between the curtain and the fingertip) can sometimes be an obstruction to the operator. Therefore, the bright point is made transparent and can be made transparent as needed.

[0216] In the case of a transparent bright spot, it can be difficult to tell which part you are operating, so in that case, buttons or other controls may light up or blink to indicate that the transparent bright spot is currently over a button.

[0217] Alternatively, a system can be configured to show the user that there are short lines at the edges of the projector's display frame (the vertical frame (either the left or right edge, or both), the horizontal frame (either the top or bottom edge, or both)) or at the edges of the AR glasses' field of view (the vertical and horizontal frame edges), and that there are bright spots where the lines extending virtually from the edge of the horizontal frame intersect with the lines extending virtually from the edge of the vertical frame.

[0218] (A) When the user's gaze is directed towards the area near the bright spot (pointer), or when the user's face is facing the display surface (pointer), the amount of movement is reduced compared to normal, and the pointer moves slowly, allowing for finer movements.

[0219] (B) The pointer's movement may also be made slower when the grip is tightened, allowing for delicate and precise movements. In other words, even when moving the hand holding the fingers or virtual light rod the same distance in normal mode, the pointer will move less than usual. This type of control may also be implemented.

[0220] Here, noise (small, gradual movements of the body) can lead to pointer wobble and make operation difficult, so it's best to remove it as much as possible. However, in cases (A) and (B) above, if the amount of noise is larger than usual or the frequency is high, it will be discarded as noise, depending on the type of movement. In other words, unless the movement is larger and slower than usual, or the frequency is low, the movement information will be removed as noise.

[0221] Alternatively, by observing biological information such as body sway and heart rate, the observed body sway can be incorporated into calculations and removed as a subtle vibration (noise) distinct from the main movements of the fingers or the hand virtually holding the virtual rod of light.

[0222] Furthermore, in the first embodiment, the imaging means for observing the hand 91 of the injured person 14 is not limited to the general-purpose first and second video cameras 41 and 42, but may also be a night vision camera, thermal camera, infrared camera, etc., which can be used even in the dark to capture the movement of the hand 91.

[0223] Furthermore, in the first embodiment, supplementary processing, calculations, or settings may be performed to enable similar operation even when holding a virtual wooden stick, a metal tube that can represent a cylinder instead of the cavity created by the hand grip, or a virtual plastic stick (a luminous virtual stick considering operation in dark places). Of course, the light may be shone only on the hand when taking a picture with the camera.

[0224] <Configuration and Effects of the First Embodiment> The configuration and effects of the first embodiment of the present invention will be summarized below. The user interface system 1 includes imaging means (first and second video cameras 41 and 42) for capturing images of a user (injured or ill person 14), virtual operation unit setting means (circuits and software related to setting the virtual operation units of the server 6) for setting virtual operation units (a virtual operation unit 131 for mouse cursor operation, a virtual operation unit 141 for flick / toggle input, and a virtual operation unit 151 for full key input) in space, operation detection means (circuits and software related to operation detection of the server 6) for detecting operations performed by the user on the virtual operation units based on the images of the user captured by the imaging means, and display means (AR glasses 2) for displaying images (spatial display images 10) with modified content according to the operations detected by the operation detection means. The virtual operation unit setting means determines the position of the user's fingers based on the images of the user captured by the imaging means, and sets the virtual operation units in space based on this determination result.

[0225] In the user interface system 1, the virtual operation unit setting means sets multiple virtual operation units to different positions based on the result of determining the position of the user's finger.

[0226] In the user interface system 1, the plurality of virtual operation units correspond to mouse input, flick input, toggle input, or full keyboard input, and the operation detection means enables user input by detecting an operation performed by the user's finger on one of the plurality of virtual operation units when the user's finger is away from the video whose content has been changed.

[0227] According to this first embodiment of the present invention, the virtual operation unit setting means determines the position of the user's fingers based on the image of the user captured by the imaging means, and sets the virtual operation unit in space based on this determination result. This allows for realistic and intuitive operation, and makes use of the best aspects of real-world user interfaces. Therefore, according to the first embodiment, user convenience can be enhanced.

[0228] Furthermore, according to the first embodiment of the present invention, the virtual rod 91 extending from the user's hand expands the range and versatility of operation and display beyond the conventional range of operation by reaching out, enabling more realistic and intuitive operation that takes distance into account, and making use of the best aspects of real-world user interfaces. Therefore, according to the first embodiment, user convenience can be enhanced.

[0229] Furthermore, according to the first embodiment of the present invention, the display means displays an operation display panel for flick input, toggle input, or full key input (operation display unit 192 for the flick input operation screen 190, operation display unit 222 for the flick input operation screen 220, and operation display unit 252 for the full key input operation screen 250), and the operation detection means detects an operation performed by the user's finger when the user's finger 91 is away from the image whose content has been changed, thereby enabling input by the user. This eliminates the need for the user to extend their finger to the operation display panel, thereby improving user convenience.

[0230] Furthermore, in addition to the methods described above, various other methods can be applied to the virtual bar display method shown in the first embodiment of the present invention. For example, at the start of the virtual bar display shown in Figure 6(a), it is also possible to use display methods such as setting the length and shape of the virtual bar, or creating a virtual bar through normal operation and then holding it.

[0231] Furthermore, the virtual rod's softness (flexibility) can also be adjusted. For example, it can be set to be soft and thread-like, or soft and pliable.

[0232] <Second Embodiment of the Present Invention> The following describes a user interface system according to a second embodiment of the present invention.

[0233] In the user interface system according to the second embodiment, VR goggles are used instead of AR glasses. The images displayed in the VR goggles are images of the VR background and the VR hand of the injured person 14, instead of the ceiling 18 (see Figure 1) that is visible through the AR glasses 2 in the AR operating space 71 shown in Figure 5(b) or the image of the injured person's hand 91.

[0234] The configuration and operation of the second embodiment of the present invention will be summarized below. The user interface system of the second embodiment includes: imaging means (first and second video cameras 41 and 42) for capturing images of a user (injured or ill person 14); virtual operation unit setting means (circuits and software related to setting the virtual operation units of the server 6) for setting virtual operation units (a virtual operation unit for mouse cursor operation 131, a virtual operation unit for flick / toggle input 141, and a virtual operation unit for full key input 151) in space for operation by the user; operation detection means (circuits and software related to operation detection of the server 6) for detecting operations performed by the user on the virtual operation units based on the images of the user captured by the imaging means; and display means (VR goggles) for displaying images (spatial display images 10) with modified content according to the operations detected by the operation detection means. The virtual operation unit setting means determines the position of the user's fingers based on the images of the user captured by the imaging means, and sets the virtual operation units in space based on this determination result.

[0235] In the user interface system of the second embodiment, the virtual operation unit setting means sets a plurality of virtual operation units to different positions based on the result of determining the position of the user's finger.

[0236] In the user interface system of the second embodiment, the plurality of virtual operation units correspond to mouse input, flick input, toggle input, or full keyboard input, and the operation detection means enables user input by detecting an operation performed by the user's finger on any of the plurality of virtual operation units when the user's finger is away from the video whose content has been changed.

[0237] According to this second embodiment of the present invention, the same effects as in the first embodiment can be obtained even when VR goggles are used instead of AR glasses 2.

[0238] In the first embodiment, AR glasses 2 were used to display the image in augmented reality space as a display means for displaying the image whose content has been modified by an operation performed by the user on the virtual control unit, and in the second embodiment, VR goggles were used as the display means for displaying the image in a virtual reality space. However, other display means, such as a projector or monitor, may also be used as a display means for displaying the image whose content has been modified.

[0239] This invention may apply not only to VR and AR, but also to the real world and XR spaces such as MR.

[0240] Therefore, the systems, means, methods, etc. of the present invention can be modified in various ways without altering the essence of the invention.

[0241] For example, the imaging means (first and second video cameras 41 and 42) for capturing images of the user (injured or ill person 14) can be installed in various ways, such as on the ceiling or on a desk. Also, when typing, a hiragana chart is displayed one by one, and when selecting a character or word, the next candidate is arranged in an easy-to-use direction, such as up and down in a straight line, left and right, or diagonally, making it easier to select a candidate and assisting with input. When a circle or similar object is selected, candidates appear as a vertical or horizontal list. The user is not limited to the injured or ill person 14, but can be a general public member of society, a worker, an educator, a civil servant, etc., and the system can be applied to a wide range of people.

[0242] For example, it is possible to combine two or more systems into one, or conversely, to configure and connect one system with two or more other systems.

[0243] Furthermore, the first and second embodiments described above are merely two of the best possible forms or forms that come close to them at present. [Industrial applicability]

[0244] The user interface system of the present invention can be effectively used in businesses and corporations operating medical facilities and accommodation facilities, as well as in private residences. [Explanation of symbols]

[0245] 1: User Interface System 2: AR glasses 3: Projector 4: Video camera stand 5: Wireless LAN 6: Server 7: Various devices 8: Virtual surface 9: Video 10: Spatial display image 11: Hospital 12: Patient room 13: Bed 14: Patient 15: Curtain 16: Wall 17: Control room 41: First video camera 42: Second video camera 61: Control unit 62: Memory unit 63: Interface 64: Communication unit 65: Keyboard 66: Mouse 67: Display device 91: Hand 92: Virtual stick 131: Virtual control unit for mouse cursor operation 141: Virtual control unit for flick / toggle input 151: Virtual control unit for full key input 190: Flick input operation screen 192: Operation section display section 220: Flick input operation screen 222: Operation section display section 250: Full keyboard input screen

Claims

1. A user interface system comprising: an imaging means for capturing images of a user; a virtual operation unit setting means for setting a virtual operation unit operated by the user in space; an operation detection means for detecting an operation performed by the user on the virtual operation unit based on the images of the user captured by the imaging means; and a display means for displaying an image with modified content according to the operation detected by the operation detection means, wherein the virtual operation unit setting means determines the position of the user's fingers based on the images of the user captured by the imaging means and sets the virtual operation unit in space based on this determination result.

2. The user interface system according to claim 1, wherein the virtual operation unit setting means sets a plurality of virtual operation units to different positions based on the result of determining the position of the user's finger.

3. The user interface system according to claim 2, wherein the plurality of virtual operation units correspond to mouse input, flick input, toggle input, or full keyboard input, and the operation detection means enables user input by detecting an operation performed by the user's finger on any of the plurality of virtual operation units when the user's finger is away from the video whose content has been changed.

Images