User interface system

Abstract

To provide a user interface system that can improve user convenience.SOLUTION: A user interface system 1 includes: imaging means (video camera 4) for capturing a video of a user (sick patient 14); aerial operation detection means (circuit and software related to aerial operation detection of server 6) for detecting an operation performed in the air by the user, the operation being detected on the basis of the video of the user captured by the imaging means; and display means (AR glasses 2) for displaying a video of which content has been modified, in an augmented reality space according to the operation detected by the aerial operation detection means. The display means uses a video of a virtual rod 91, extending from a hand of the user, as at least a part of the video to be displayed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a user interface system that detects a user's operation and displays an image. [Background technology]

[0002] In recent years, XR (cross reality or extended reality) such as VR (virtual reality), AR (augmented reality), and MR (mixed reality) that create virtual spaces and worlds on computers have become widespread. Among these, various technologies have been proposed for displaying augmented reality spaces synthesized into real spaces as AR or MR (AR (Augmented Reality) information) on head-mounted display devices.

[0003] For example, there is a technology for mid-air operation that displays an augmented reality space on a head-mounted display device in which an image of a computer-created keyboard is synthesized into real space as AR information, allowing the user to reach out and operate the displayed virtual keyboard (see, for example, Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 07-078055 [Patent Document 2] Japanese Patent Application Laid-Open No. 2009-146333 Summary of the Invention [Problem to be solved by the invention]

[0005] In the mid-air operation technology described in Patent Documents 1 and 2, the user reaches out to operate a virtual keyboard displayed in an augmented reality space, so the range of operation is limited to the range that can be operated by reaching out to the virtual keyboard, which means there are limitations to the range and diversity of operation and display.

[0006] Furthermore, the aerial operation technologies described in Patent Documents 1 and 2 allow for fusion with virtual space and the real world, and create a world without a sense of distance, but in many cases require direct contact to operate, which is a cumbersome process.

[0007] Furthermore, in recent years, advances in the handling of technological information have led to the birth of virtual spaces such as the Metaverse in addition to VR, AR, and MR. It is expected that the density of information in virtual spaces will increase, and the information density of objects in both the real world and virtual spaces will increase, transforming them into objects with substance and information that are not merely decoys. In this context, the two-dimensional user interfaces that have traditionally ignored distances have been unable to provide realistic, intuitive operation that takes distance into account, and have not been able to take advantage of the advantages of real-world user interfaces.

[0008] An object of the present invention is to provide a user interface system that can improve user convenience. [Means for solving the problem]

[0009] The user interface system of the invention described in claim 1 comprises an imaging means for capturing an image of a user, an aerial operation detection means for detecting an operation performed by the user in the air based on the image of the user captured by the imaging means, and a display means for displaying an image with modified content in an augmented reality space or a virtual reality space in accordance with the operation detected by the aerial operation detection means, wherein the display means uses an image of a virtual rod extending from the user's hand as at least part of the image to be displayed.

[0010] The user interface system of the invention described in claim 2 is the user interface system described in claim 1, characterized in that the length or speed of extension of the virtual rod can be adjusted in accordance with the hand operation detected by the mid-air operation detection means.

[0011] The user interface system of the invention described in claim 3 is the user interface system described in claim 1 or 2, characterized in that it further comprises an equipment control means for controlling a specified equipment in accordance with the operation detected by the aerial operation detection means. [Effects of the Invention]

[0012] The user interface system of the present invention can improve user convenience. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a side view of a user interface system according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing the overall configuration of a user interface system according to a first embodiment of the present invention. [Figure 3] 3 is a flowchart showing the operation of the user interface system according to the first embodiment of the present invention. [Figure 4] FIG. 2 is an explanatory diagram showing video processing of the user interface system according to the first embodiment of the present invention. [Figure 5] FIG. 2 is a first explanatory diagram showing detailed image processing for AR glasses of the user interface system according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a second explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. [Figure 7] FIG. 10 is a third explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. [Figure 8] FIG. 4 is a fourth explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. [Figure 9]FIG. 5 is a fifth explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. [Figure 10] FIG. 6 is a sixth explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. [Figure 11] FIG. 7 is a seventh explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] First Embodiment of the Present Invention A user interface system according to a first embodiment of the present invention will now be described with reference to the drawings.

[0015] <Configuration of User Interface System> First, the configuration of a user interface system according to a first embodiment of the present invention will be described. Fig. 1 is a side view of the user interface system according to the first embodiment. Fig. 2 is a block diagram showing the overall configuration of the user interface system according to the first embodiment of the present invention.

[0016] In FIG. 1, a user interface system 1 according to a first embodiment of the present invention includes AR glasses (augmented reality glasses) 2 worn by an injured or sick person 14, a projector 3 that displays on a curtain 15 an image of an operation screen that can be viewed by the injured or sick person 14 lying in bed 13 in a hospital room 12 in a hospital 11, a video camera 4 that captures an image of the upper body of the injured or sick person 14 lying in bed 13, a wireless LAN 5, and a server 6 installed in a control room 17 of the hospital 11.

[0017] AR Glasses 2 uses a holographic reflector to display augmented reality (AR) in a form that looks natural like regular glasses, and can display images similar to those seen with AR goggles to the human eye.

[0018] The projector 3 and video camera 4 are attached to the ceiling 18 of the hospital room 12. The curtain 15 is for protecting the privacy of the injured person 14 and is placed at a predetermined distance from the feet of the injured person 14.

[0019] The projector 3 must first hold the distance from the display surface (curtain 15) by manually inputting the setting, or must first calculate the distance. The projector 3 must also know or set the distance to the display surface and candidate projection surfaces (curtain 15, wall, monitor), as well as the shape of the room and the shape of the space.

[0020] The video camera 4 must first set and maintain the distance from the hand 91 (see FIG. 4(a)) of the injured person 14, or perform processing such as calculation. The video camera 4 may also set in advance the position of the person (injured person 14) and the center point of the fist.

[0021] The wireless LAN 5 includes a wireless LAN base station 51 and wiring. The wireless LAN base station 51 is attached to a wall 16 of the patient room 12 and is connected to a server 6 in a management room 17 via wiring. The wireless LAN 5 communicates wirelessly with the AR glasses 2, the projector 3, and the video camera 4.

[0022] As shown in FIG. 2, the server 6 is connected to a keyboard 65, a mouse 66, a display device 67, a wireless LAN 5, and various devices 7.

[0023] The various devices 7 are equipment such as an intercom in the patient room 12, a light switch in the patient room 12, and a call bell installed in the management room 17.

[0024] The server 6 includes a control unit 61, a storage unit 62, an interface 63, and a communication unit 64.

[0025] The storage unit 62 stores the program of the user interface system 1, images of the operation screen and AR operation space display, video data of images captured by the projector 3, judgment results for the images captured by the projector 3, and the like.

[0026] The control unit 61 is connected to a keyboard 65, a mouse 66, and a display device 67 via an interface 63, and various settings are made by the relevant person using the keyboard 65 and mouse 66, and displays based on the various settings are displayed on the display device 67.

[0027] In addition, the control unit 61 is connected to the projector 3 and the video camera 4 via the communication unit 64 and the wireless LAN 5, and performs determination processing based on input signals from the video camera 4, and outputs signals to the AR glasses 2 and the projector 3 using 3D projection mapping, etc.

[0028] <User interface system operation> FIG. 3 is a flowchart showing the operation of the user interface system according to the first embodiment of the present invention. FIG. 4 is an explanatory diagram showing the video processing of the user interface system according to the first embodiment of the present invention.

[0029] In Figure 3, when the user interface system 1 is set up, first the injured person 14 puts on the AR glasses 2, the projector 3 stores the distance from the display surface (curtain 15) and the room shape as settings, or performs processing such as calculating the distance and room shape, and the video camera 4 stores the distance from the hand 91 of the injured person 14 (see Figure 4(a)) as settings, or performs processing such as calculating the distance.

[0030] Thereafter, in step S1, the video camera 4 captures an image of the upper body of the sick or injured person 14 lying on the bed 13, and transmits the video data of the image 9 shown in FIG.

[0031] As a result, the server 6 receives the video data of the image 9 from the video camera 4 in step S2.

[0032] Next, in step S3, the server 6 performs image analysis on the received video data and determines whether the hand 91 of the injured person 14 in the image 9 shown in Figure 4(a) is in a clenched state.If the hand 91 of the injured person 14 is in a clenched state, in step S4, video data of various images in the AR operation space 71 shown in Figure 4(b) and video data of the image 81 shown in Figure 4(c) are generated as images of the operation space and operation screen that can be viewed by the user.

[0033] In the above description, the hand 91 is in a clenched state, but other methods are also possible, for example, by detecting a state in which the index finger is extended and executing the operation process.

[0034] Next, the server 6 transmits the video data generated in step S4 to the AR glasses 2 and the projector 3 in step S5.

[0035] As a result, in step S6, the AR glasses 2 receive video data from the server 6, and in step S7, the image light from the video data is reflected by the holographic reflector so that it can be seen by the eyes of the injured person 14, thereby displaying the initial operation space shown in Figure 4(b).

[0036] The AR operation space 71 shown in Figure 4(b) is an image of an augmented reality space that is visible to the eyes of the injured person 14 through the AR glasses 2, and an image of the ceiling 18 (see Figure 1) and the hand 91 of the injured person 14 that are transmitted through the AR glasses 2 are superimposed with a spatial display image 73 of the light operation button, a spatial display image 74 that is virtual information for the lighting operation button, a spatial display image 75 of the call button, a spatial display image 76 of the intercom button, a spatial display image 77 of the cancel button, and a spatial display image 78 of the OK button.

[0037] With the AR glasses 2, the actual ceiling 18 and the hand 91 of the injured person 14 are visible, whereas the spatial display images 73 to 78 are virtual displays that do not exist in the actual location, using video data generated by the server 6 in step S4.

[0038] Meanwhile, in step S8, projector 3 receives the video data from server 6, and in step S9 projects and displays an image based on the video data onto curtain 15 (see FIG. 1). In this case, the image projected and displayed on curtain 15 is image 81 shown in FIG. 4(c).

[0039] Image 81 shown in Figure 4(c) is an image of the operation screen, in which an image of the upper body of injured person 14 lying in bed 13 captured by video camera 4 has been flipped left to right, and an image of a light-off operation button 83, an image of a light-on operation button 84, an image of a call button 85, an image of an intercom button 86, an image of a cancel button 87, and an image of an OK button 88 are superimposed on the flipped image 82.

[0040] In addition, the image 81 shown in Figure 4(c) displays an image 82 in which the upper body of the injured person 14 is flipped left and right, but it is also possible to basically not show the screen that is operated by a person, i.e., the operator himself.

[0041] When the operator himself is not shown in the image 82 of Figure 4(c), the operator himself is not shown, and images of buttons 83 to 88 are displayed on the curtain, and a red dot like a laser pointer (or any bright dot) moves on the curtain in the same way as the operator's finger or the tip of the virtual stick in his hand.

[0042] Next, in step S10, the projector 3 continuously captures images of the upper body of the sick or injured person 14 lying on the bed 13, and transmits the video data to the server 6.

[0043] As a result, the server 6 receives video data of a new image of the upper body of the injured person 14 from the video camera 4 in step S11.

[0044] Next, in step S12, the server 6 performs image analysis on the received video data, automatically calculating the X1, Y1 tip of the index finger of the hand of the injured person 14 and the X2, Y2 center of the grip or hand in the new image shown in the video data, and based on the coordinates X1, Y1, X2, Y2, detects the aerial operations, which are operations performed in the air by the injured person 14 with the virtual stick, corresponding to the spatial display images 73 to 78 of each button displayed by the AR glasses 2.

[0045] Next, in step S13, the server 6 generates video data of the augmented reality space created by the AR glasses 2 and the image of the virtual operation screen displayed on the curtain 15 in accordance with the mid-air operation detected in step S12.

[0046] Next, the server 6 transmits the video data generated in step S13 to the AR glasses 2 and the projector 3 in step S14.

[0047] Next, in step S15, the AR glasses 2 receive the video data from the server 6, and in step S16, reflects the image light of the video data onto the holographic reflector so that the image light is visible to the eyes of the injured person 14.

[0048] In this case, the image of the augmented reality space that is visible to the eyes of the injured person 14 through the AR glasses 2 is displayed by superimposing a virtual rod of light 92, a spatial display image 73 of the light operation button, a spatial display image 74 of the turn-on operation button, a spatial display image 75 of the call button, a spatial display image 76 of the intercom button, a spatial display image 77 of the cancel button, and a spatial display image 78 of the OK button on the image of the spatial display image 73 to 78 of each button that corresponds to the aerial operation detected in step S12, and effect processing such as illumination is performed on the image of the button.

[0049] Next, in step S17, the projector 3 receives the video data from the server 6, and in step S18, projects and displays an image of the virtual operation screen based on the video data onto the curtain 15.

[0050] In this case, the image of the virtual operation screen projected onto the curtain 15 is obtained by flipping the image of the upper body of the injured person 14 captured by the video camera 4 in step S10, and superimposing a virtual rod of light 92, an image 83 of the light-off operation button, an image 84 of the light-on operation button, an image 85 of the call button, an image 86 of the intercom button, an image 87 of the cancel button, and an image 88 of the OK button on the flipped image, and applying effects such as lighting to the image of the button corresponding to the aerial operation detected in step S12 among the images 83 to 88 of each button.

[0051] Thereafter, in step S19, the server 6 controls various devices 7 such as an intercom in the hospital room 12, a light switch, and a call bell installed in the management room 17, in accordance with the detected mid-air operation.

[0052] To explain the operation of such a user interface system 1 using a specific example, when the injured person 14 looks at the image of the AR operation space 71 in Figure 4(b) through the AR glasses 2 and aligns the tip of the virtual rod of light 92 held in the hand 91 of the injured person 14 with the spatial display image 73 of the light-off operation button, the spatial display image 73 of the light-off operation button lights up and the lights on the bed 13 are turned off.

[0053] When the tip of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the spatial display image 74 of the lighting operation button, the spatial display image 74 of the lighting operation button lights up, and the light on the bed 13 is turned on.

[0054] When the tip of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the spatial display image 75 of the call button, the spatial display image 75 of the call button lights up, the call bell installed in the control room 17 rings, and a nurse heads to the injured person 14.

[0055] When the tip of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the spatial display image 76 of the intercom button, the spatial display image 76 of the intercom button lights up, and then when the tip of the virtual rod of light 92 is aligned with the spatial display image 78 of the OK button, the intercom between the hospital room 12 and the management room 17 is connected.

[0056] When the tip of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the spatial display image 77 of the cancel button, the previous operation is invalidated and the state returns to the state before the operation.

[0057] Furthermore, when the injured person 14 looks at the image 81 of Figure 4(c) projected onto the curtain 15 by the projector 3 and aligns the tip of the virtual rod of light 92 held in the hand 91 of the injured person 14 with the image 83 of the light-off operation button, the image 83 of the light-off operation button lights up and the lights on the bed 13 are turned off.

[0058] When the tip of the imaginary rod of light 92 held in the hand 91 of the injured person 14 is aligned with the image 84 of the lighting operation button, the image 84 of the lighting operation button lights up, and the light on the bed 13 is turned on.

[0059] When the tip of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the image of a call button 85, the image of the call button 85 lights up, a call bell installed in the control room 17 rings, and a nurse heads to the injured person 14.

[0060] When the tip of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the image 86 of the intercom button, the image 86 of the intercom button lights up, and then when the tip of the virtual rod of light 92 is aligned with the image 88 of the OK button, the intercom between the hospital room 12 and the management room 17 is connected.

[0061] When the tip of the virtual rod of light 92 held in the hand 91 of the injured person 14 is aligned with the image 87 of the cancel button, the previous operation is invalidated and the state before the operation is restored.

[0062] <Detailed Video Processing for AR Glasses> The following describes the detailed video processing for AR glasses. FIG. 5 is a first explanatory diagram showing the detailed video processing for AR glasses of the user interface system according to the first embodiment of the present invention.

[0063] In the user interface system according to the first embodiment, the server 6 (see FIGS. 1 and 2) performs image analysis on the video data received from the video camera 4 (see FIGS. 1 and 2), determines the direction of the hand 91 of the injured person 14 and the state of how much the hand 91 is grasped in the video 9 shown in FIG. 4(a), and as an image of the augmented reality space, according to the direction of the hand 91 and the degree of grasping of the hand 91, generates a video of the virtual bar 92 of light grasped by the hand 91 and transmits it to the AR glasses 2 as video data. The AR glasses 2 receive the video data from the server 6, reflect the video light by the video data on the holographic reflector, and make it visible to the eyes of the injured person 14.

[0064] [[ID=??]]As shown in FIG. 5(a), when the actual injured person 14 forms a large cavity 93 between the fingers of the hand 91, as an image of the virtual bar of light grasped by the hand 91 in the augmented reality space by the AR glasses 2, a video of a large virtual bar ********************* of light is used.

[0065] The tip 92a of the large virtual bar 92 of light serves as an operation point, and dot images 92b are displayed at its position and around it.

[0066] As shown in FIG. 5(b), when the actual injured person 14 forms a medium-sized cavity 94 between the fingers of the hand 91, as a video of the virtual bar of light grasped by the hand 91 in the augmented reality space by the AR glasses 2, a medium-sized virtual bar 95 of light is used.

[0067] The tip 95a of the medium-sized virtual bar 95 of light serves as an operation point, and dot images 95b are displayed at its position and around it.

[0068] When the injured person 14 aligns the tip 95a of the virtual bar 95 of light with the spatially displayed image 74 of the lighting operation button, the spatially displayed image 74 of the lighting operation button lights up, and the lighting of the bed 13 lights up. It should be noted that there seems to be an incomplete or incorrect expression in the original text at line ID=8, where "大型の光の仮想棒92の映像を用いる" is followed by some garbled characters "*********************". This has been marked as "??" in the translation for now. If you can provide the correct and complete original text, the translation can be further refined.

[0069] As mentioned above, the lighting of the bed 13 may be turned on when the lighting operation button is aligned with the spatial display image 74, but in practical use, it may be configured so that it is determined that the button has been pressed if it is found that the number of seconds in the area above the button is equal to or exceeds the set number of seconds, even if the button is slightly moved. Therefore, it may be configured so that the lighting operation button is executed as long as a bright point is on the button, even if it is not pressed.

[0070] To explain in more detail, the server 6 (see Figures 1 and 2) automatically calculates the hand 91 and the tip of the index finger X1, Y1 and the center of the grip or hand X2, Y2 of the injured person 14 from the video data received from the video camera 4 (see Figures 1 and 2) as the direction of the hand 91, and the line connecting these two points X2, Y2 to X1, Y1 is used as the central axis of the virtual rod, which is used as the virtual rod of light 92 in Figure 5(a) or the virtual rod of light 95 in Figure 5(b).

[0071] In the case of the grip of the hand 91, virtual rods of light 92 and 95 in Figure 5 may be assumed based on the shape of the grip, but in order to create the virtual rods of light 92 and 95 in Figure 5, X3 and Y3 may be automatically calculated and set around the top of the grip of the hand 91, and X4 and Y4 may be automatically calculated and set around the bottom of the grip of the hand.

[0072] The usual grip shape of hand 91 is shown in Figure 5, but if the hand can only be gripped in the reverse direction, that is, if operation can only be performed with the little finger side facing up, the virtual rod of light may extend from behind the grip of the hand. In that case, it will extend from X3, Y3 to X4, Y4 and reach the contact surface of ceiling 18.

[0073] In addition to this configuration, to guide demonstrations and operations, the shape of the hand 91 (showing how to move) as a dotted line (which can also be an edge image (an image obtained by edge detection (extraction) using an image processing technique)) can be displayed on the AR glasses 2 (see Figure 1) at the position of the hand in the AR space using the AR glasses 2 (or an image from the video camera 4 capturing the hand 91 can be shown, and of course the position of the hand can be displayed as a dotted line (an edge image) rather than an actual image), making it easier for first-time users to understand. Furthermore, superimposing an image such as a fiery rod (a virtual fiery rod) in addition to an image of a fiery rod can make operation easier to guide. For example, if you think of it as a fiery rod, you can demonstrate it burning a wall and penetrating it, and displaying the dotted shape of the hand 91 in the augmented reality space using the AR glasses 2 makes it easier to understand. It also makes it possible to convey the principles and mechanisms.

[0074] Furthermore, when the video camera 4 is capturing an image of the upper body (observation may be performed using a motion sensor or the like), the start of an operation may be signaled by forming the arm in a certain shape or by performing an operation that is not often performed. Examples of such signals include bending the elbow to form a V, performing several shapes in a set order, or forming a circle with the thumb and index finger of the right hand and extending three other fingers, which starts the operation of the user interface of the present invention and starts or ends hand observation, or crossing the index finger and middle finger of the right hand to indicate the end or start of an operation.

[0075] The user interface of the present invention may be initiated by simple signals or actions such as extending the pinky or middle finger in a set sequence, or extending certain fingers in a certain order.

[0076] Furthermore, when a virtual stick (the user interface of the present invention) created by the index finger or grip is pointed at the left or right wall, a group of buttons set on that surface will appear. Also, the set group of buttons may appear even if they are not buttons in the direction of that surface. While operating with the fingers extended or in a hollow grip, raising and lowering the thumb may be considered a click, and successive clicks may be considered a double-click. Even if the shape of the fingers extended or the grip is distorted, as long as it can be recognized that an operation is in progress, processing may continue as is even if the shape is distorted.

[0077] As shown in FIG. 5(c), when the real injured person 14 transitions to a state where no cavity is formed between the fingers of the hand 91, the virtual rod of light held by the hand 91 in the augmented reality space created by the AR glasses 2 disappears.

[0078] Furthermore, when the injured person 14 aligns the tips 92a and 95a of the virtual light rods 92 and 95 with the spatial display image 76 of the intercom button, the spatial display image of the intercom button The intercom display image 76 lights up, and then when the tips 92a and 95a are aligned with the spatial display image 78 of the OK button, the intercom between the patient room 12 and the management room 17 is connected.

[0079] As mentioned above, when the OK button spatial display image 78 is aligned, the intercom between the patient room 12 and the management room 17 is connected, and at that time, a bright spot (images 92b, 95b) appears on the button, and the OK button is automatically pressed. However, assuming practical use, the system may be configured so that the button is determined to have been pressed if it is found that the number of seconds in the area above the button is equal to or greater than the set number of seconds, even if the button moves slightly.

[0080] In this way, the OK, off, on, or button can be configured to be executed as long as the bright spot remains on the button for a set number of seconds or more, without the need to directly press the button.

[0081] In relation to this, instead of a set number of seconds, it is also possible to make the grip tube smaller, that is, to squeeze the grip or to press it down with a thrusting motion.

[0082] In this way, the injured person 14 can adjust the operation points of the spatial display images 73 to 78 of the operation buttons in the augmented reality space by the AR glasses 2, by changing the way the hand 91 is held.

[0083] Alternatively, the center position of the hand 91 may be temporarily placed at the location where the hand 91 was before starting a button operation, at the location where the observation device (the video camera 4 and the circuitry and software related to mid-air operation detection in the server 6) determines that the hand has not moved for a certain period of time, or at the location where the observation device determines that the hand has not moved to a certain extent during the operation. Alternatively, the center position of the hand 91 may be placed and a sphere may be calculated from the center with a radius set as the radius of the sphere. If the hand 91 is pushed forward from that center position or the center position of the sphere toward the bright spot display surface (the display surface of the bright spots (92b, 95b)) of the user interface (virtual light rods 92, 95) of the present invention by a predetermined amount, a click occurs. If the hand 91 is pushed forward further than that, it is considered to be a drag. The drag state may be terminated when the hand 91 approaches the center position of the original hand 91 or the center position of the sphere from the drag reference. The click may also be terminated when the hand 91 approaches the center position of the original hand 91 or the center position of the sphere from the click reference.

[0084] Furthermore, when the observation device captures the position of the face of the injured person 14 and the hand 91 comes within a set distance of multiple stages, the speed of the bright spot may be slowed down and moved accurately according to the set distance of multiple stages as the hand 91 approaches the face. In this case, the hand 91 may be held in front of the face.

[0085] Furthermore, the bright point may be an extension of a line connecting the center of the head of the injured person 14 to the fingertips, or a line connecting the center of the head to the position of a virtual rod. The point at the center of the head may be set to indicate the distance in a certain direction from the point. Also, to avoid capturing the vibrations of the person (injured person 14) in detail, a certain degree of shaking may not be reflected in the movement of the bright point as error or noise.

[0086] Furthermore, the injured person 14 may place his / her hand 91 on his / her leg or on a table to stabilize the movement of the bright spot.

[0087] Furthermore, if there are interfaces beyond the ceiling 18, a group of interfaces in another hierarchy will be displayed.

[0088] For example, when the interface consisting of the virtual light rods 92 and 95 is aimed at the wall (ceiling 18) directly above the patient (patient 14), the nurse call button, the patient's reading light, the television receiver switch, etc. appear as images on the second layer.

[0089] Furthermore, when the interface consisting of virtual rods of light 92, 95 is pointed in the direction of the room entrance, the switches for the air conditioning, air conditioner, and sailing lights in the room appear as images on the third layer.

[0090] Furthermore, in the first embodiment, it is also possible to display a group of switches in another room. It is also possible to display, for example, five floors in tabs, imagining the tabs of a web browser.

[0091] Furthermore, in the first embodiment, each layer may be an I / F in which the first to fifth layers appear depending on the direction in which the virtual rods of light 92, 95 point (top of the wall, east, north, etc., or east or west face of a curtain, first monitor surface, second monitor surface, second monitor surface, etc.) change.

[0092] Furthermore, in the first embodiment, a tab-like display is not normally used, and special settings may be provided for administrators or other personnel to operate. For example, if the hospital is small, the camera status of each room, the on / off status of the air conditioning and sailing lights can be displayed at the entrance and exit, and these can be turned on / off and adjusted, eliminating the need for manual operations. The number of monitors can also be reduced. If the hospital is large, similar special settings may be provided in the nurse center on each floor.

[0093] Furthermore, in the first embodiment, image analysis was performed on the video data received from the video camera 4 (see Figures 1 and 2) and the degree to which the hand 91 of the injured person 14 in the AR operation space 71 (see Figure 4(a)) was determined by calculating the cavity of the hand, but it is also possible to configure it so that in addition to calculating the cavity of the hand, changes in basic area and volume, such as an increase in the area or volume of the hand, are observed from the horizontal or vertical directions, and if a reference value is exceeded, a specified action is performed.

[0094] Furthermore, when multiple video cameras 4 are used and the movement speed becomes slow or the change in size is small in order to speed up calculations, multiple video cameras 4 may be combined into one to perform calculations only for area and edge shape. However, when one video camera 4 is used to perform calculations only for area and edge shape, the setting can be such that the error is automatically set to + / -3 square centimeters or 3 cubic centimeters based on the gender, date of birth, height, weight, body fat percentage, arm thickness, hand size, finger thickness, etc., to account for tremors during human movement and shape recognition error during movement, or can be set arbitrarily.

[0095] Furthermore, the direction and point of the virtual rod of light in the first embodiment shown in FIG. 5 may be determined based on the surface shape of the hand 91 of the injured person 14.

[0096] Furthermore, in the first embodiment, the error of the hand 91 can be considered in the same way; if the movement is reproduced too much, the point at the tip of the virtual rod of light will shake, so it is not preferable to reproduce the movement too much. Therefore, it is possible to take measures against noise, such as not detecting movements above a certain speed or not detecting movements above a certain frequency. It is also possible to perform interpolation processing of the trajectory of the hand 91.

[0097] The thickness of the images 92b, 95b of the points at the ends of the interfaces (virtual rods of light 92, 95) can be narrowed, that is, made thinner, according to the movement of the hand 91, which can also be achieved by pressing a switch.

[0098] After thrusting the interface (virtual light rod 92, 95), it is possible to move it from the thrust position parallel to the projection surface: the contact surface of the virtual rod (curtain, wall, ceiling, monitor), while maintaining the same distance from that surface (in this case, a certain degree of error is allowed, and that error can be set in the settings), which produces the same effect as dragging a mouse (the movement trajectory, in terms of processing, is traced while holding down the trajectory, the strength can be set, or if the distance of the thrusting movement can be measured, it can be reflected in the degree of pressure (the depth into the contact surface can be measured; if you think of it this way, the normal contact surface becomes an image of the tip of the virtual rod touching, so it is sometimes better to think of it that way). As mentioned above, a certain degree of noise and error processing is performed).

[0099] Here, in the case of a process that has the same effect as dragging a mouse, it is also possible to turn over the operation screen.

[0100] Furthermore, the action of making the interface (virtual rods of light 92, 95) thicker or pushing it forward may be regarded as the action of pressing a switch. As with mouse clicks, if done repeatedly, it may be regarded as a double click.

[0101] The movement speed of the interface (virtual light rods 92, 95), AR glasses 2, and light dots displayed on contact surfaces (curtains, walls, ceilings, monitors) will need to be adjusted, and calculations will likely be required to suppress and control as much as possible the vibrations that enter during human operation. To achieve this, measures such as not capturing movements above a certain frequency of input motion or not capturing small movements, i.e., movements below a certain distance will not be recognized as movement and will not be considered as light dot movement, will be taken. For example, the vibration of a person's hand can be corrected using a camera capturing the movement, or processing can be performed to extract only the movement of the captured object and remove the vibration of the hand itself. Alternatively, assistive devices can be worn to prevent hand vibration, as needed. A hand cast or a support for the hand can be provided.

[0102] Furthermore, the processing for interfaces with the same shape may be changed. For example, while the virtual rods of light 92, 95 are displayed, clicking with two fingers extended (with the index finger and middle finger extended in a clenched hand, or in a scissors-like shape) may be equivalent to right-clicking on a mouse. Various modes may be set depending on the shape of the hand.

[0103] The first embodiment can be modified in various ways. For example, not only the AR space displayed on the AR glasses or the curtain, but also various walls, LCD monitor screens, etc. can be applied as the display unit.

[0104] Furthermore, in the first embodiment, the face and line of sight of the sick or injured person 14 may be assumed, and the screen displaying the projection destination and buttons of the projector 3 may be displayed only in the direction that the person is likely looking. In other words, the screen displaying the projection destination and buttons of the projector 3 is usually the target of a finger, a curtain or wall with a virtual light stick stuck in it, or a monitor (for example, a liquid crystal monitor as display means), but it is also possible to ignore this and project the image in the direction the person is looking. This is suitable for cases where there is a sick or injured person 14 whose face and body are facing in extremely different directions.

[0105] Furthermore, in the first embodiment, as shown in FIG. 4, the image is that the operation is performed by first fisting the hand, but if the default is two fingers and the operation is performed with three fingers upright, the virtual rod of light can be made thick as shown in FIG. 5(a), and if the operation is performed with one finger upright, the virtual rod of light can be made thin as shown in FIG. 5(b). This mechanism can also be considered to represent the amount of forward thrust (which can also be considered as the depth from the display surface of the projector 3) during a thrusting motion, rather than adjusting the thickness. In other words, the difference in the number of fingers from the default operation with two fingers can be used as the amount of forward thrust (depth) during a thrusting motion. If the virtual rod of light is considered to be a virtual rod of fire, this can also be considered to be a process that changes the output or temperature of the virtual rod.

[0106] In addition, in the first embodiment, it is also possible to assign meaning to the shape of the virtual light rod that is dragged onto the operation button. For example, by drawing a circle (or any other shape such as a triangle or an x, as long as you set it) while dragging, it is possible to set various operations such as turning on the air conditioner in the current room or turning off the lights.

[0107] Furthermore, in the first embodiment, if the strength of the drag action or the amount of push-out before the poke action (which can also be thought of as the depth from the display surface of the projector 3) is insufficient, it is not possible to penetrate the wall to the next room, and only the surface of the wall is damaged, and it is also possible to express the properties of a physical wall or the properties of a virtual rod of fire (of course, a weak wall may be set, or no wall is normally assumed). For example, assuming a virtual rod of light, the amount of push-out before the poke action is determined so that the light penetrates to another room, and if that value is exceeded, the virtual rod of light will reach the next room. It is also possible to determine the amount of impression that the light will not reach if the push-out amount is insufficient, or the amount of impression that the wall is damaged.

[0108] This strength may be reproduced if it is possible to reproduce the strength when operating a button on the operation surface.

[0109] When operating a button or the like that has a pressure sensor, an operation reproduction device that gradually presses the button can be used to reproduce gradually increasing pressure (within the dragging trajectory), or gradually decreasing pressure from a strong pressure (within the dragging trajectory).

[0110] To give a concrete example, it can be used to simulate tracing your finger from the center of a fairly large button to the left.

[0111] To summarize, the image on the operation screen of the AR glasses 2 or the curtain 15 has an operation reproduction device that reproduces an operation such as a gradual pressing by the user, and when the user operates a button having a pressure sensor, the operation reproduction device can reproduce an operation such as a gradual stronger pressing, and an operation in which the pressure gradually decreases from a strong pressing.

[0112] Furthermore, in the first embodiment, it is possible to turn on adjacent buttons in succession, or to operate something like an illumination adjustment bar by pressing it with an appropriate force and adjusting the shape of the adjustment bar, such as left and right, up and down. When applied, this can give the impression of carving into a wall, making it possible to draw pictures or letters with depth on the wall. This information is reflected in the display by the AR glasses 2 and the projector 3. Of course, it can also be physically reproduced on the wall by a robot. This information can also be used for maintenance of building walls.

[0113] By combining this information with the settings, when you virtually destroy the wall of an adjacent room, the buttons in the adjacent room will be displayed and made operable. Also, by destroying a wall, it is possible to display and operate an operation surface (level) for only the air conditioners in the entire building, or only the light switches, rather than the condition of a room wall.

[0114] As another example, depending on the amount of pushing before the poking action (which can be thought of as the depth from the ceiling 18 visible from the AR glasses 2 or the display surface of the projector 3), buttons for a certain floor (for example, the 5th floor) may be shown, and when the amount of pushing falls within a different range, buttons for a certain floor (for example, the 4th floor) may become operable.

[0115] Furthermore, in the first embodiment, if a virtual light rod on the operation button is aimed at a certain location and dragged to draw a circle or the like (as long as an area can be specified, a closed space can be specified), it is possible to consider a shape in which all buttons contained in that circle, in other words, the cylinder of the building, are lined up on the operation surface, ignoring the idea of separate rooms, etc. This shape is thought to be useful for checking and searching for maintenance malfunctions. For example, if the operating status of not only the equipment buttons in the room in that direction but also the buttons of devices such as refrigerators and televisions can be seen, it is possible to check the operation of not only the building and room equipment but also individual devices (refrigerators, televisions, etc.), which may be used to efficiently determine the location of a malfunction. Here, refrigerators and televisions are included, but of course, refrigerators and televisions are also considered to be targets of operation in normal operations. Furthermore, when a maintenance person arrives at the building, etc., such a proposed interface that humans can understand is thought to be more efficient and effective.

[0116] Furthermore, in the first embodiment, cameras and buttons (including buttons for refrigerators, televisions, etc.) are placed in each room for maintenance purposes, and communication is conducted to obtain information on the status of each facility and device. The information is then categorized, and only switches of the same category, such as air conditioner switches (remote control information), are collected and arranged on the operation screen without regard to room. (Of course, two or three types are acceptable as long as they can be arranged efficiently on the operation screen.) The operation screen may be organized into a single tab (hierarchy). In this case, switches may not actually be displayed as light-emitting switches, but may be displayed as such to indicate their status. Furthermore, the operating status of complex devices such as air conditioners may be displayed, for example, next to each related air conditioner switch or by itself on the operation screen. In this case, information on the operating status of air conditioners can be displayed alongside the related switches, of course, by incorporating information from cameras and switches in each room, or from any type of information-gathering device (monitoring temperature, electricity usage, etc.) alongside the related switches, making the operating status clear at a glance. Of course, by compiling related information, maintenance and the actual current situation can be understood without going to the site, and appropriate adjustments can be made efficiently.

[0117] Furthermore, in the first embodiment, buttons may be displayed larger than normal in order to accurately capture the buttons.

[0118] The size, magnification, and reduction of the button may be maintained in the settings of a component of the user interface system (for example, server 6), or may be linked to the thickness of the virtual light rod, or may be configured to be automatically determined and set based on the situation of the person operating the device (age, whether or not they wear glasses, and the prescription of their glasses, i.e., a numerical value of the correction strength of the lenses, etc., which are actually input and set as settings) and the situation of the contact surface between the button or virtual light rod and the wall (the distance from the person to the wall, the darkness of the environment, the brightness of the display surface, etc., may be input in the settings).

[0119] Furthermore, in the first embodiment, the bright spot (pointer) that serves as a guide for operation, the tip of the operator's finger, and the tip of the virtual rod of light that is being held (calculated as the contact point between the curtain and the tip of the finger) can sometimes be a nuisance to the operator, so they are made into transparent bright spots and can be made transparent as needed.

[0120] In the case of a transparent bright dot, it is difficult to tell what you are operating, so in that case, buttons etc. will light up or flash to indicate that the transparent bright dot is currently over the button.

[0121] Alternatively, a system can be configured in which there are short lines at the edge of the projector's display frame (the position of the vertical frame (which could be either the left or right frame line, or both), or the edge of the horizontal frame (which could be either the top or bottom frame line, or both)) or at the edge of the field of view of the AR glasses (the position of the vertical or horizontal frame edge), and the operator can see that there is a bright spot where a virtual line extending from the line at the edge of the horizontal frame intersects with a virtual line extending from the line at the edge of the vertical frame.

[0122] (A) When the system detects that your gaze is directed toward the area near the bright spot (pointer) or that your face is looking at the display surface (pointer), it moves less than usual, and the pointer moves only slowly, allowing for finer movements.

[0123] (B) The movement of the pointer may also be slowed down when the grip is tightened, allowing for more delicate and precise movements. In other words, even if the user moves a finger or a hand holding a virtual light stick the same distance as in normal mode, the pointer will move a smaller distance than normal. This type of control may be used.

[0124] Here, noise (small trembling of the living body) can lead to shaking of the pointer and make operation difficult, so it is best to remove it as much as possible, but in the cases of (A) and (B) above, if the amount of noise is larger than usual or the frequency is high in accordance with small movements, it will be discarded as noise. In other words, unless the movement is larger, slower, or has a low frequency than usual, the movement information will be deleted as noise.

[0125] In addition, by observing biometric information, such as body sway and heart rate, the observed body sway can be taken into account and removed as fine vibrations (noise) that are different from the main movements of the fingers or the hand virtually holding the virtual light rod.

[0126] Furthermore, in the first embodiment, the imaging means for observing the hand 91 of the injured person 14 is not limited to the general-purpose video camera 4, but may also be a night-vision camera, a thermal camera, an infrared camera, or the like, which can capture the movement of the hand 91 even in the dark.

[0127] Furthermore, in the first embodiment, supplementary processing, calculations, and settings may be performed so that the same operation is possible even if a virtual wooden stick, a metal tube that can represent a cylinder instead of the hollow space created by the grip of the hand, or a virtual plastic stick (a virtual stick that emits light when considering operation in a dark place) is held. Of course, light may be shone only on the hand when taking a picture with the camera.

[0128] <Other Image Processing for AR Glasses> Various types of image processing for AR glasses will be described below with reference to FIGS. 6 to 11. FIG. 6 is a second explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. FIG. 7 is a third explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. FIG. 8 is a fourth explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. FIG. 9 is a fifth explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. FIG. 10 is a sixth explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. FIG. 11 is a seventh explanatory diagram showing detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention.

[0129] In a first display method other than those shown in Figures 1 to 5 (hereinafter referred to as "other"), as shown in Figure 6(a), a virtual rod of light 101 held in a hand 91, a spatial display image 102 of a meter indicating the length of the virtual rod of light 101, and a spatial display image 103 of a meter indicating the speed at which the virtual rod of light 101 is extending are displayed in the AR space displayed by the AR glasses 2.

[0130] An operator (including an injured person 14 and a general operator) wearing the AR glasses 2 adjusts the length of the virtual rod of light 101 by holding the spatial display image 102 with the hand 91, and adjusts the speed at which the virtual rod of light 101 extends by holding the spatial display image 103 with the hand 91.

[0131] This display method makes it easy to control a real aerial drone with a virtual light stick 101, as shown in FIG. 10(c) described below, for example.

[0132] In the second display method, as shown in FIG. 6(b), a virtual rod of light 111 held in a hand 91 is displayed in the AR space displayed by the AR glasses 2, and a spatial display image 112 of attribute information at the tip of the virtual rod of light 111 is displayed.

[0133] If we imagine the operator wearing the AR glasses 2 as a worker demolishing a building, the spatial display image 112 at the tip of the virtual rod of light 111 penetrates into the interior of the building's wall and displays the internal structure of the building's wall (reinforced concrete, cavities, concrete, etc.).

[0134] In a third display method, as shown in Fig. 6(c), a virtual rod of light 121 held in a hand 91 is displayed in the AR space displayed by the AR glasses 2. If the operator wearing the AR glasses 2 is assumed to be a worker demolishing a building, the virtual rod of light 121 can be inserted into a pipe 122 to display attribute information about the inside of the pipe 122.

[0135] In a fourth display method, as shown in Fig. 6(c), a virtual rod of light 123 is made soft and flexible like rubber or thread, so that a real or virtual kite or drone can be guided by the tip of the virtual rod of light 123. The virtual kite or drone can be guided underground or inside a building by the virtual rod of light 123.

[0136] The virtual rods of light 121, 123 can also be used as penetrating laser pointers.

[0137] In a fifth display method, as shown in FIG. 7(a), a virtual rod of light 131 held by a hand 91 is displayed in the AR space displayed by the AR glasses 2. The virtual rod of light 131 reflects an actual liquid crystal television receiver 132 like a mirror and bends, and a spatial display image 134 of the back surface of the liquid crystal television receiver 132 is displayed at the tip of the bend 133. The spatial display image 134 displays the state of the wiring and switches on the back surface of the liquid crystal television receiver 132. The virtual rod of light 131 may be imagined as a bendable Luminite rod.

[0138] In a sixth display method, as shown in Fig. 7(b), a virtual rod of light 141 held by a hand 91 is displayed in the AR space displayed by the AR glasses 2. The object at the tip of the virtual rod of light 141 can be replaced, and can be replaced with a camera 143 that views the back side of an object 142, a drill 144 that drills a virtual hole in the object 142, a brush, a magic pen, a lantern, a light, or the like. Various virtual displays 145 and the like are displayed in accordance with the replaced object at the tip.

[0139] In a seventh display method, as shown in FIG. 7(c), a virtual rod of light 151 held by a hand 91 is displayed in the AR space displayed by the AR glasses 2. The hand 91 and the virtual rod of light 151 can be copied as spatial display images 91a, 91b, 91c, 151a, 151b, and 151c. Copying can display surfaces and lines, move them simultaneously, or skip surfaces and lines. AI can also move the spatial display images 91a, 91b, 91c, 151a, 151b, and 151c to operate various switches. AI can also recognize and analyze the movements of the hand 91 and the virtual rod of light 151 and copy the movements.

[0140] In the eighth display method, as shown in FIG. 8(a), virtual rods of light 161, 162, 163, and 164 held by a hand 91 are displayed in sequence in the AR space displayed by the AR glasses 2.

[0141] A graph-like space display image 165 is displayed in the AR space, and a plurality of vertical lines 166... and a plurality of horizontal lines 167... are set in the space display image 165, and the image is corrected so that the tips of the virtual rods 161, 162, 163, 164 are aligned with the plurality of vertical lines 166... and the plurality of horizontal lines 167..., thereby correcting camera shake at the tips of the virtual rods 161, 162, 163, 164. This makes it convenient to use the virtual rods to draw a space display image that includes many straight lines, such as buildings, in the AR space.

[0142] In a ninth display method, as shown in Fig. 8(b), a virtual rod of light 171 held by a hand 91 is displayed in the AR space displayed by the AR glasses 2, and the virtual rod of light 171 is thrust into a virtual object 172, and an operation of extending the virtual rod of light 171 (see Fig. 6(a)) is performed. This moves the virtual object 172 farther away.

[0143] In a tenth display method, as shown in Fig. 8(c), a virtual rod of light 181 held by a hand 91 is displayed in the AR space displayed by the AR glasses 2, and the virtual rod of light 181 is thrust into a virtual table 182, and an operation is performed to extend the virtual rod of light 181 (see Fig. 6(a)). As a result, the virtual table 182 is moved far away and attached to a wall or the like.

[0144] In an eleventh display method, as shown in Fig. 9(a), a virtual rod of light 191 held in a hand 91 is displayed in the AR space displayed by the AR glasses 2, and by piercing the virtual rod of light 191 into the screen of a real monitor 192 (an image of choppy water in the case of Fig. 9(a)), a tunnel 194 is formed on a screen 193 of the monitor 192, and the underwater world is displayed in the tunnel 194. Depending on the situation of the screen where the virtual rod of light 191 is pierced, information under the rubble (for example, an image of the inside, the location of life-saving equipment, etc.) may be set to be displayed.

[0145] In another twelfth display method, as shown in FIG. 9(b), a virtual rod of light 201 held in a hand 91 is displayed in the AR space displayed by the AR glasses 2, and the virtual rod of light 201 is extended far away, with the intermediate length omitted.

[0146] In a thirteenth display method, as shown in FIG. 9( c), a virtual rod of light 211 held by the hand 91 is displayed in the AR space displayed by the AR glasses 2, and the virtual rod of light 211 is extended far away, with the intermediate length omitted. By pointing the tip of the virtual rod of light 211 at a distant point area 212, a spatial display image 213 in which the area 212 is enlarged can be displayed. Furthermore, by pointing the tip of the virtual rod of light 211 at the spatial display image 213, various operations can be performed on the distant point area 212. Because the spatial display image 213 is enlarged, the action may be delayed when pointing with the tip of the virtual rod of light 211.

[0147] In another 14th display method, as shown in FIG. 10(a), when core drilling 222, 223 is performed on a concrete object 221, the information about the core drilling 222, 223 may be displayed when necessary (for example, during maintenance) by operating a virtual light rod, or a virtual camera may be embedded in the core drilling 222, 223 and images captured by the virtual camera may be displayed.

[0148] In another 14th display method, as shown in Figure 10(b), when using a camera 237 moving behind a concrete object 236 or a camera 238 moving behind the concrete object 236, a virtual rod of light 231 or a virtual thread 234 held by a hand 91 is displayed in the AR space displayed by the AR glasses 2, and the camera 237 moving behind the object 236 or the camera 238 moving behind the object 236 can be operated by moving the virtual rod of light 231 to positions 232, 233, etc., or the camera 237 moving behind the object 236 or the camera 238 moving behind the object 236 can be operated by moving the position for pulling the virtual thread 234 to position 235, etc.

[0149] In the other fifteenth display method, as shown in FIG. 10(c), a virtual rod of light 241 held by the hand 91 is displayed in the AR space displayed by the AR glasses 2.

[0150] An operator (including an injured person 14 and a general operator) wearing the AR glasses 2 adjusts the length of the virtual rod of light 241 by holding the spatial display image 102 (see Figure 6(a)) with the hand 91, adjusts the speed at which the virtual rod of light 241 extends by holding the spatial display image 103 (see Figure 6(a)) with the hand 91, and waves the tip of the virtual rod of light 241 by waving the hand 91 up and down and left and right.

[0151] This display method makes it easy to control the real aerial drone 242 with the virtual rod of light 101. In this case, the aerial drone 242 is guided to a position corresponding to the tip point of the virtual rod of light 241.

[0152] In another 16th display method, the planned trajectory of the virtual light rod is notified to the operator (including the injured person 14 and general operators) by voice, light, color, etc.

[0153] In the other 17th display method, as shown in FIG. 11(a), a virtual rod of light 251 held in a hand 91 is displayed in the AR space displayed by the AR glasses 2, and a spatial display image 253 of a light switch is initially displayed near the right side of the area 252.

[0154] When the operator (including the sick or injured person 14 or a general operator) aligns the tip of the virtual rod of light 251 with the spatial display image 253 of the light switch, the spatial display image 253 of the light switch lights up, and the lights in the room turn on. When the spatial display image 253 of the light turns on, a spatial display image 254 of an air conditioner switch is displayed near the upper left corner of the area 252 based on the action prediction of the operator by the server 6. Furthermore, on the operator's birthday, a spatial display image 254 celebrating the birthday is displayed near the bottom side of the area 252.

[0155] When the operator aligns the tip of the virtual rod of light 251 with the spatial display image 254 of the air conditioner switch, the spatial display image 254 of the air conditioner switch lights up and the air conditioner in the room starts up. In the 17th display method, specific displays such as the spatial display image 254 celebrating a birthday are also possible.

[0156] In the other 18th display method, as shown in FIG. 11(b), a virtual rod of light 261 held in a hand 91 is displayed in the AR space displayed by the AR glasses 2, and spatial display images 263-1, 263-2, and 263-3 of the first to third selection candidates are displayed in an area 263 near the left side of the area 262.

[0157] In the AR space displayed by the AR glasses 2, a spatial display image 263-1 is followed by a past operation history 264 of the tip of the virtual rod of light 261 and a spatial display image 265 of the operated switch, and a spatial display image 263-2 is followed by a past operation history 266 of the tip of the virtual rod of light 261 and a spatial display image 265 of the operated switch. A display image 267 is displayed, followed by spatial display image 263-3, followed by a past operation history 268 of the tip of virtual rod of light 261, a spatial display image 269 of a meter showing the length of virtual rod of light 261 corresponding to past operation history 268, a spatial display image 270 of a meter showing the speed at which virtual rod of light 261 is extending corresponding to operation history 268, and a spatial display image 271 showing that the operation corresponding to past operation history 268 has been canceled.

[0158] When the tip of the virtual rod of light 261 is aligned with the spatial display image 263-1 of the first selection candidate, the virtual rod 261 is automatically operated in the same manner as in the operation history 264, and the spatial display image 265 of the switch is operated.

[0159] When the tip of the virtual rod of light 261 is aligned with the spatial display image 263-2 of the second selection candidate, the virtual rod 261 is automatically operated in the same manner as in the operation history 266, and the spatial display image 267 of the switch is operated.

[0160] When the tip of the virtual rod of light 261 is aligned with the spatial display image 263-3 of the second selection candidate, the same operation of the virtual rod 261 as in the operation history 268 is automatically performed, and this operation is then canceled.

[0161] The features of the first embodiment will be described below in order. (Feature 1) In the AR space displayed by the AR glasses 2, a virtual stick (which can be extended or retracted depending on the previous hand operation) not only presses a switch at the location where it is touched, but can also become transparent, as if there is a hole penetrating through the location, or conversely, allow the back side to be seen.

[0162] (Feature 2) In the AR space displayed by the AR Glasses 2, it is possible to set up a virtual mirror and reflect the virtual stick to see the back side.

[0163] (Feature 3) In the AR space displayed by the AR Glasses 2, buildings or pieces of paper attached to the end of a virtual stick can be moved in the same way as the virtual stick itself extends.

[0164] (Feature 4) In the AR space displayed by the AR Glasses 2, you can attach something like a virtual table to the tip of a virtual stick instead of an object and move it around. Once you place the virtual table, it can be used as a guide. When the tip of the virtual stick is far away, operation is inaccurate, so the lines on the table can be used as a guide to correct this.

[0165] (Feature 5) In the AR space displayed by the AR Glasses 2, it is possible to display an image of a virtual rod with a camera at the tip. Because the tip is at a distance, it can be seen even when it is penetrating a thick wall, and what would be seen at the tip after penetrating can be seen.

[0166] (Feature 6) In the AR space displayed by the AR glasses 2, an action occurs when the virtual rod penetrates, but this can also be made visible using a projector (monitor, screen) or the like.

[0167] (Feature 7) In the AR space displayed by the AR Glasses 2, attributes are assigned to the places that are touched and the inside of thick walls that are penetrated, so it is possible to take these attributes into account and set actions to be implemented in those places. For example, you can show the outside when you are on a massage chair, in a toilet, or inside a wall (in an emergency, surrounded by rubble). It is also possible to show the inside in reverse.

[0168] (Feature 8) In the AR space displayed by the AR Glasses 2, it is possible to make a virtual stick (which has a calculated distance extending from the hand, and can be copied including the hand, and the copy itself can be fine-tuned or moved) transparent wherever it touches.

[0169] (Feature 9) In the AR space displayed by the AR Glasses 2, you can see the other side (like a mirror). You can also press switches on the other side. You can also see the middle of the transparent wall. You can also press switches on the middle of the transparent wall.

[0170] (Feature 10) In the AR space displayed by the AR Glasses 2, you can see the middle of the wall's thickness (it can be underwater instead of a wall). You can touch the middle of the wall's thickness (this is intended for VR, but it can also be done in real life if there is data or a switch).

[0171] (Feature 11) In the AR space displayed by the AR Glasses 2, the virtual rod can pass through multiple walls. The tip of the virtual rod can repeat the above process (passing through a wall, passing through it, and operating the next wall).

[0172] (Feature 12) In the AR space displayed by AR Glasses 2, calculations are made so that the tip of a point extending from a certain starting point or fulcrum will touch anything that matches the calculated length and position. It is possible to touch various objects. However, if the tip is set to sharp mode, it will be able to poke the object in front of it. If the sharpness and hardness are increased, it will be able to poke even hard objects. Also, if you move forward, it will be stuck like a thumbtack. However, because it is electronic data, it can be touched in a way that simply overlaps the data, without damaging the actual object. In other words, the behavior may change depending on the combination of attribute information of the touching and touched parties (touching situation). There may also be an overlapping mode. In that case, the behavior also changes depending on the attribute.

[0173] (Feature 13) In the AR space displayed by AR Glasses 2, if there is a length in the 3D space, you can adjust the distance to that point and measure the distance. In this case, the accuracy can be low. Initially, the default length is fixed. At the end of the point, you can attach paper, boards, concrete slabs, the tops of buildings, etc. It is also possible to extend the rod and place huge structures on it. You can also attach virtual paper to an LED display.

[0174] (Feature 14) In the AR space displayed by the AR glasses 2, the virtual rod may be like a rod extending from a hand, or may be operable by manipulating the fingers of that hand. It may also extend in both directions, not just in one direction.

[0175] (Feature 15) In the AR space displayed by the AR Glasses 2, a GUI may be displayed that shortens the distance of the virtual rod.

[0176] (Feature 16) In the AR space displayed by the AR Glasses 2, the virtual rod penetrates, overlaps, or touches, and its behavior changes depending on the attributes of the penetrated area and the action record. In the virtual space, the behavior depends on the action of the contacting side, but in the real world, the behavior is displayed using 3D mapping, etc. Also, it can be set so that the wall in front can be penetrated, but the next wall can be touched. The behavior can also be changed depending on the information of the contacting side.

[0177] (Feature 17) In the AR space displayed by AR Glasses 2, the story begins with specifying how to operate a virtual rod (which has length, can be extended (basically a straight line, but it can also be curved, but this becomes less intuitive), the virtual surface it hits, the principle of mirrors, and the ability to align light with a curved tube that can be bent, allowing the flow to be shown).

[0178] (Feature 18) In the AR space displayed by AR Glasses 2, a virtual rod can be used to draw a drone's trajectory. This trajectory can be drawn with variable thickness. The tip of the rod is equipped with a pen that can be used to draw in the air. While the rod is purpose-specific, the tip can have different purposes depending on the attribute information of the object at the tip, what the tip touches, and the purpose. For example, coloring the air. Creating a curved, solid (iron) rail-like line allows the drone to fly along it, or to grab and move it with a robotic arm, or even a camera. The tip must vary depending on the purpose and target (of course, multiple tips can be attached at the same time). In other words, the virtual rod (which has length and tip information), the tip of the virtual rod, and the space or object it touches are recognized, and attribute information (object or space can also be used; in this case, the basic information is air, or even vacuum, but the object space is considered vacuum) is organized, and the attribute information of the object is matched with the attribute information of the tip of the virtual rod according to the purpose, thereby achieving the goal. In reality, it recognizes real spaces and objects, and when acquiring real data (humans and AI analyze information from cameras around it and at the tip), it also recognizes situations where it has information about the space and objects in advance (assuming it is in a created VR space from the start), and then constructs and understands that situation.

[0179] (Feature 19) In the AR space displayed by the AR Glasses 2, the operation may vary depending on the attributes of the rod itself, such as controlling it as if it were a hard rod or a soft thread. If the target object is on a hard, curved rail, a hard rod will expand and contract, and even if it overlaps with the trajectory, it will become a transparent, passing rod. Of course, this does not have to be the case in the case of a hard rod or a realistic thread.

[0180] (Feature 20) In the AR space displayed by AR Glasses 2, the drone's trajectory is automatically calculated based on the position of walls to prevent collisions; it can be a line or a tube-like shape. This becomes the tip that performs the calculations and the tip that depicts the space. If it is not a line, movement within the space can be directed with a virtual rod. The rod can be used as a reel like a dog or as a rigid rod to move drones or move supported objects (this can move independently or be operated by a surrounding support system). The position of drones can be adjusted. It is possible to imagine the drones connected to each other with a string, and move only the first drone with a fixed rod.

[0181] (Feature 21) In the AR space displayed by AR Glasses 2, a cylindrical rod with a camera inside for close-range operation of drones, power lines and their rail cameras, a rod for checking movement by air, or a rod moved by string or wire, plastic gears for rotation, or gears next to the electric wires are used as interfaces to control drones, etc.

[0182] (Feature 22) In the AR space displayed by the AR Glasses 2, a virtual handle is attached to the stick, and the angle of the virtual mirror in the direction of the handle, the direction of reflection, and how to move the handle are automatically calculated once the aim is determined. The angle is calculated from the data when a straight line is formed based on the position of the stick in the person's hand. Alternatively, a button or marker can be set to fix the angle and operate the interface of the present invention.

[0183] (Feature 23) In the AR space displayed by the AR Glasses 2, there is a mode to show or hide the survey results of the 6 nearby. Also, it is possible to switch whether or not to show them automatically or by setting based on the information at the tip of the stick.

[0184] (Feature 24) In the AR space displayed by the AR Glasses 2, the speed of the rod can be adjusted. It is not only the operator who moves the rod, but the system and the copied rod itself move at a set speed. Such adjustments can be made, such as adjusting the numbers, moving slowly, moving off the rails, adjusting the rod or string, and changing the attribute information of the object being stuck (in the real world, this is the case in a construction site, and in the virtual world, such as VR).

[0185] (Feature 25) In the AR space displayed by the AR Glasses 2, the virtual rod can enter anywhere, so it is possible to automatically adjust the depth inside the wall. It is also possible to see the center of the board based on information such as the front and back surfaces of the target wall.

[0186] (Feature 26) In the AR space displayed by the AR Glasses 2, if the virtual rod penetrates the back side, the back side may be visible. Also, information about the TV and the wiring on the back side may be remembered and displayed / explained.

[0187] (Feature 27) In the AR space displayed by the AR Glasses 2, you can use your finger (any finger is fine, you can also set a specific part of a finger) to move forward and backward like a mouse wheel, or a virtual plane, and the movement will move back and forth like a wheel. You can also observe the maximum speed of your finger's movement and set that value to determine the speed at which the rod extends. This means that the extension speed will be the set speed. It is also possible to switch to a delicate mode. For example, if your finger moves slowly in the opposite direction, the image will be enlarged at that position, allowing for delicate movements. Depending on the speed, the image will be enlarged or will extend at high speed.

[0188] (Feature 28) In the AR space displayed by the AR Glasses 2, a projector displays mirrors and shadows. By displaying them in 3D, the operator can see where the reflections occur.

[0189] (Feature 29) In the AR space displayed by the AR Glasses 2, while the virtual rod is extended, the situation beyond it and a graphical interface that can be seen even from a slightly behind eye are used, making operation easy.

[0190] (Feature 30) In the AR space displayed by AR Glasses 2, a virtual mirror and a bird's-eye view of the person and goal point are presented, and the goal point is shifted and automatically calculated from the position of the hand, and guide information is displayed.

[0191] (Feature 31) In the AR space displayed by the AR Glasses 2, a virtual surface is created and a virtual rod is pushed out as if poking it, or the virtual rod is extended, and if there is a virtual surface, it is pushed out.

[0192] (Feature 32) In the AR space displayed by AR Glasses 2, the object the virtual rod is aiming at is perceived as three-dimensional by a group of cameras located in front, above, behind, and on the left and right, allowing distance to be read. A depth camera may also be used. This information is saved for the present and past times. If current information is not available, past information can of course be used. If spatial information is available, the rod continues to extend within it. A camera is placed at the end of the virtual rod. It is a virtual rod, but it is a ghost-like rod. Some parts are transparent, but the end may take on a real form (attribute). A meter or numbers are visible to the person controlling it so that the length can be determined. The extension speed can also be adjusted. It may also be possible to touch the distance, speed, and acceleration meters to set distance, speed, and acceleration.

[0193] (Feature 33) In the AR space displayed by AR Glasses 2, you can see the back side of the object by aligning it with a virtual rod (transparent rod). You can also see what's beyond that. You can see the view from the camera at the end of the rod. It can also be used for maintenance. Also, if you use a VR space instead of an AR space, all information is available within the VR space, making it easier to destroy, cut, or repair. The time the attribute information was saved is also displayed. This is not necessary in the real world, though.

[0194] (Feature 34) In the AR space displayed by the AR Glasses 2, at the appropriate time (time interval can be set), the shutter will automatically activate at the set location or after a wall or thick wall layer changes at the intruded location, and data will be collected automatically and simultaneously, collecting time and location as attribute information. In the real world, time and location will also be collected as attribute information when information is collected through penetration such as ultrasound.

[0195] (Feature 35) In the AR space displayed by AR Glasses 2, there is a camera at the end of a virtual pole like a lantern. It reflects off a virtual wall (such as a mirror, which can be installed multiple times), or it has a virtual eye (in the case of the previous example, a virtual eye is installed somewhere along the pole, and a drone, for example, is controlled to be at that eye position) that determines the angle at which it should bend, etc.

[0196] (Feature 36) In the AR space displayed by AR Glasses 2, if a person controls it, the limit is two mirrors. Anything more than that would require a computer with recommendations, such as AI. It will be possible to ask what the person is aiming for and how they are operating.

[0197] (Feature 37) In the AR space displayed by AR Glasses 2, multiple virtual mirrors can be placed, moving around the viewpoint. They also move parallel to the viewer's movements. It's also possible to inflate lava anywhere. The distance and direction of expansion can be set. Hands and other objects can be duplicated, allowing for control with multiple hands. Hands can be thrown sideways. They can also be thrown with a virtual stick, the speed of which can be adjusted. There's also an acceleration function. However, instead of adjusting this, it's also possible to set the speed of the virtual stick, table, or trajectory. Such trajectories can also be created. If a VR space is used instead of an AR space, it's possible to draw in space with a long stick. It's also possible to assign speed to the trajectory as attribute information. The sticks are connected, and the speed can be set on them. Certain sections are set to that speed. It's also possible to move them off the line. The movement is also dependent on the attribute information of that space. Attribute information can be written into a certain space with this stick. It's possible to embed rules for each trajectory.

[0198] (Feature 38) In the AR space displayed by the AR Glasses 2, if something cannot be pressed directly and linearly, it can be made into a right-angled rod or bent multiple times. This can sometimes avoid the need for a linear press.

[0199] (Feature 39) In the AR space displayed by AR Glasses 2, this technology allows the user to see the back side of the object. For example, it is possible to show the wall or monitor behind the virtual rod. Furthermore, since the operation may be inaccurate, the back side can be shown by moving the guideline on the front side or other surface. It can be superimposed on the front side, or the back side can be displayed below the point indicated by the virtual rod while the point is moved sideways, and projected (followed) there. It has the technology to show the back side information of non-destructive testing on the front side. It can show the inside of a wall, or the inside or back side of concrete. This allows the desired cutting location and a marker on the back side to be displayed on the front side, allowing the user to cut there. Projections for human surgery require precision. The position of the back side information can also be indicated, and the angle can be changed to show a more than pure back side. It can be presented like a mirror or a reversible mirror.

[0200] (Feature 40) In the AR space displayed by AR Glasses 2, for example, a virtual rod can be used to connect and operate something inside a pipe, such as a camera (which can move autonomously). In order to operate it, the absolute position of the camera itself must be determined (using the camera's own GPS, ultrasound, non-destructive testing, etc.). Once the virtual rod and camera are connected, it becomes possible to not only operate the camera but also rotate it. Of course, it is also possible to display images of the interior of the object or the interior including the camera on a projector.

[0201] (Feature 41) The AR space displayed by the AR Glasses 2 is equipped with technology that allows you to see the end of a pierced object with a virtual rod, and technology that allows you to see the back side as the front side. This can be real, but it also allows you to see the data that was originally saved (past data) by moving it according to the position of the virtual rod. (In the case of demolition, for example, you can see the back side, which improves work efficiency.) It is determined by the location and attribute information.

[0202] (Feature 42) In the AR space displayed by the AR Glasses 2, the surface touched by the virtual stick appears as a tunnel, and operations can be performed after passing through the tunnel. For example, you can pick up the receiver of a communication device from the bathroom. You can also operate the TV in the next room with a remote control located here.

[0203] (Feature 43) The AR space displayed by the AR Glasses 2 can respond to emergencies by displaying swimming pools, rubble, toilets, etc. In the case of swimming pools, cameras and swim rings can be placed there.

[0204] (Feature 44) In the AR space displayed by AR Glasses 2, the virtual rod treats things that people cannot touch, such as air or mist, as nothing, as in an X-ray, and allows things that can be recognized as objects to be touched on a certain plane. It can recognize 3D objects and allow you to touch one side of them, like flipping through pages in a medical image viewer, or it can even flip through hybrid displays such as MPR cross sections and 3D volume data. It can also be made into something like an album and flipped through pages.

[0205] (Feature 45) In the AR space displayed by AR Glasses 2, it is possible to unfold the penetrating part using a virtual rod like a radar chart, unfold a tube seen from the side, unfold the unfolded image, or turn from the outside of the tube. To make it even more realistic, you can use core data, or even if it is an invisible line, with this virtual rod, you can force information on the front or guide line when driving nearby and display the exact position with a projector, making it possible to specify the back side, which was previously impossible to specify. In this case, the penetrating property is utilized.

[0206] (Feature 46) In the AR space displayed by AR Glasses 2, it is believed that there will be errors due to hand movements when operating the virtual rod, and even if high-frequency movements are ignored, errors will still occur, so virtual guide lines are added. For example, if a VR space is used instead of an AR space, a large table can be set in the VR space, and when the light of the virtual rod reaches a nearby location, it can be corrected to that table.

[0207] (Feature 47) In the AR space displayed by AR Glasses 2, guide tables and the like can be placed in distant locations by actually extending a virtual rod (attaching a guide table or the like to the end of the virtual rod and flying it).

[0208] (Feature 48) In the AR space displayed by AR Glasses 2, touching a recognized object with the virtual stick actually specifies that object, and the information about that object is deleted. After deletion, the object disappears and an empty space is created. Alternatively, information about penetration and overlapping information is used. This causes objects to disappear one by one.

[0209] (Feature 49) In the AR space displayed by AR Glasses 2, it would be nice to have a mode where paper or cloth can be turned over slightly, like wind blowing through a book.

[0210] (Feature 50) In the AR space displayed by AR Glasses 2, a curved light line will appear behind the physical switch monitor, and sounds will also be emitted. When you write, text will be read, and audio and music will be played.

[0211] (Feature 51) In the AR space displayed by AR Glasses 2, a transparent tube moves through the air, and the camera rotates along the center line of the tube (you can virtually change the size of the tube and the size of the eyes), or you can choose to rotate around the center of the eyes. The rotation is achieved through wireless communication, and a guide wire is attached for the rotation; the wire is attracted by a magnet, or something like a guide pulley follows the guide. This allows you to know that it is below. Since power is a challenge for video, a flash is used with a small battery. Extending the above, a larger object is docked and completed, and then it comes into play, acting like an endoscope, performing resections, and then disassembles and returns.

[0212] (Feature 52) In the AR space displayed by AR Glasses 2, there is a tool that allows you to see the part of a tube-like object that passes through when collecting information, and it is possible to show the contents of the tube. A small camera moves and rotates the tube, and then the contents are extracted and can be operated as usual with a mouse or VR lens. This equipment collects information at certain intervals, and by setting the time, it is possible to make it easier to see the changes inside the material. Although a certain amount of information is required, it can be used as a reference for actual deterioration over time.

[0213] (Feature 53) In the AR space displayed by the AR Glasses 2, for example, if there is a button on the back seat of a car that you want to touch, you can intuitively press it if you have a virtual mirror. The seat or something similar can be used as a virtual mirror. There can also be a mechanism that shows the position of the button, or a mechanism that shows the bar with an LED, such as a Luminite bar that shows the bar as it passes by.

[0214] (Feature 54) In the AR space displayed by the AR Glasses 2, there is a function to omit the intermediate process, and it is possible to simultaneously display the beginning part, the operation part, and the base part in the display area.

[0215] (Feature 55) In the AR space displayed by AR Glasses 2, it is possible to point optical fibers in all directions and illuminate one of them to make the whole space visible.

[0216] (Feature 56) In the AR space displayed by the AR Glasses 2, it may be possible to make the object being touched appear as if it were a mirror projecting the back of the object, or to make it appear as if the other person can see it (reversal mirror).

[0217] (Feature 57) In the AR space displayed by the AR Glasses 2, it is possible to change the angle of view at the position of the piercing point. It is also possible to move forward, backward, and rotate.

[0218] (Feature 58) In the AR space displayed by AR Glasses 2, you can create an L with your left and right hands, connect the vertices, and create a surface with the size and angle of the surface.

[0219] (Feature 59) In the AR space displayed by AR Glasses 2, a thermal sensor produces a noisy image, but DL image recognition corrects the error.

[0220] (Feature 60) The AR space displayed by the AR Glasses 2 can be a restroom where you cannot move or a closed room. It would also be good to be able to show your own situation, so that the image of someone seeking emergency help can be shown from the outside and in that position. When moving on the bed (in the back seat of a car), a virtual mirror is set in a fixed position, and by pointing the interface at this mirror, it turns and reaches the target location. It would also be easier to understand if a lighted bar or something similar was used to indicate the direction. The direction is indicated where the target switch or object is located.

[0221] (Feature 61) The AR space displayed by the AR glasses 2 may use a transparent tube (with embedded wires), optical fiber, multiple cameras, embedded cameras, or holes for cameras.

[0222] (Feature 62) The AR space displayed by AR Glasses 2 can also be used to open car doors, etc. For example, someone with a key can open a door or window from inside or outside, provided they aim at the switch position and operate it. This also allows them to close a door that is far away.

[0223] (Feature 63) In the AR space displayed by the AR Glasses 2, it is possible to easily change the mode of movement relative to a mirror with a fixed center or parallel movement from the operator. This allows for angle changes through rotation, and for the operator to move by translating. Furthermore, by making it possible to copy the operator or the operator's hand and have them fly or move, the operator can freely translate by moving their own copy. Furthermore, by enlarging a set portion of the tube or a portion determined by operation, it is possible to create a space in which one can actually move inside the tube, or by joining disassembled parts together to form a larger object.

[0224] (Feature 64) In the AR space displayed by AR Glasses 2, the story begins with specifying how to operate a virtual rod (which has length, can be extended (basically a straight line, but it can also be curved, but this becomes less intuitive), the virtual surface it hits, the principle of mirrors, and the ability to align light with a curved tube that can be bent to show a flow).

[0225] (Feature 64) Virtual rod (from the point of extension, it can extend from the end, or it can extend in both directions) (it can bend itself (a curved object, or it can be made flexible like a thread. Furthermore, it can be made to break when a certain amount of force is applied, like konjac, calculated or estimated. It can pass through objects: it can be transparent, or it can be attached (it can control moving objects such as drones (can be virtual): the tip (thickness of the tip and the middle can be changed) can be attached, and the middle can be controlled with a hard rod, and it can be developed from there, and the tip rod attached to the drone can be used as a wall. It operates while checking the distance to its surroundings to avoid bumping into things. It calculates for independent support: in the case of the real world), virtual board (attached to the tip of a virtual stick and moved. Move buildings. Tables can also be moved. It can be used as a guide. When a virtual stick is created from the hand, the farther it is, the more difficult it can be to perform precise control operations. It is also possible to display an enlarged view of only the area near the tip. There are limitations to operating only at the tip of the hand. For this reason, a detailed table-like guide is used to set it accurately near the target. Furthermore, an enlarged view of the position of that point is also available. This is not a hand operation. It is placed like an image that copies the hand operation. If necessary, it can be placed over the image and take over the operation again.), virtual guide lines (guide rails: can be made of iron or plastic, and can be set to move faster in certain sections), virtual tables, etc. VR goggles, hand camera, front camera (virtual), projector (can move) (shows the back side as it is not actually visible in the real world, and a camera that can be shown to the rescuers is also needed), camera that moves the back side (can move), CPU that integrates this information and attribute information of the touch point Storage that stores information and behavior information (reaction information). It can be a monitor or PC. It is also possible to show a virtual rod or a GUI that shrinks midway. The behavior changes depending on the reaction record and the attributes of the penetrated point and the behavior record when it penetrates, overlaps, or is touched. In virtual space, it depends on the behavior of the contacting side, but in the real world, the behavior is shown using 3D mapping, etc. It is also possible to set it so that the wall in front is penetrated, but the next wall is touched. The behavior can also be changed depending on the information on the touching side. It is possible to know where the tip of the rod is. The situation at the tip is also photographed like an eye, using a camera.At the same time, it can also be displayed on a monitor or projector like a picture-in-picture.

[0226] (Feature 65) In the AR space displayed by the AR Glasses 2, the area that can be seen is analyzed in advance (image analysis) and the location to point to is displayed as a candidate point. The system shows the next operation candidate, point, line, or action candidate based on the current situation. For anniversaries, it also suggests words of gratitude.

[0227] (Feature 66) In the AR space displayed by AR Glasses 2, the operation history, operation results, and the reproduction and trajectory of the movement can be displayed on the screen and selected. Each operation history and movement can be edited like a PowerPoint animation. Alternatively, you can simulate and adjust the copied hand.

[0228] (Feature 67) In the AR space displayed by AR Glasses 2, candidates can appear as surfaces, objects, or volumes depending on the touching side and surface, regardless of predictions or operation history. For example, in an LED ceiling light, the shining object is the switch, the surface in the middle is the mode change switch, and when the tip of the light enters the interior of the surface, the interior is revealed. Furthermore, when the virtual stick is advanced, the second floor is revealed. For example, touching any switch on a remote control will mimic the operation of that remote control. In other words, when you touch something, it can mimic the function of the part you are touching.

[0229] The configuration and operation of the first embodiment of the present invention will be summarized below. The user interface system 1 comprises an imaging means (video camera 4) that captures an image of the user (injured person 14), an aerial operation detection means (circuits and software related to aerial operation detection in the server 6) that detects operations performed by the user in the air based on the image of the user captured by the imaging means, and a display means (AR glasses 2) that displays an image in an augmented reality space with modified content in accordance with the operation detected by the aerial operation detection means, and the display means uses an image of a virtual rod 91 extending from the user's hand as at least part of the image to be displayed (see Figure 4).

[0230] Furthermore, the user interface system 1 is configured to be able to adjust the length or speed of extension of the virtual stick 91 in accordance with the operation of the hand detected by the mid-air operation detection means (see FIG. 6(a)).

[0231] Furthermore, the user interface system 1 further includes a device control means (circuits and software related to device control of the server 6) that controls predetermined devices (various devices 7) in accordance with the operation detected by the aerial operation detection means.

[0232] Furthermore, the user interface system 1 comprises an imaging means (video camera 4) for capturing an image of the user (injured person 14), an aerial operation detection means (circuits and software related to aerial operation detection in the server 6) for detecting operations performed by the user in the air based on the image of the user captured by the imaging means, and a display means (AR glasses 2) for displaying an image with modified content in an augmented reality space in accordance with the operation detected by the aerial operation detection means. The display means uses an image of a virtual rod 91 extending from the user's hand as at least part of the image to be displayed, sets multiple lines in the augmented reality space or the virtual reality space, and corrects the image so that the tip of the virtual rod 91 is aligned with the multiple lines, thereby correcting hand shake at the tip of the virtual rod 91 (see Figure 8(a)).

[0233] Furthermore, the user interface system 1 comprises an imaging means (video camera 4) that captures an image of the user (injured person 14), an aerial operation detection means (circuits and software related to aerial operation detection in the server 6) that detects operations performed by the user in the air based on the image of the user captured by the imaging means, and a display means (AR glasses 2) that displays an image in an augmented reality space with modified content in accordance with the operation detected by the aerial operation detection means, and the display means uses an image of a virtual rod 91 extending from the user's hand as at least part of the image to be displayed, and displays the inside or back side of the real object or the virtual object by inserting the tip of the virtual rod into a real object in the augmented reality space or a virtual object in the virtual reality space (see Figure 7(b) and Figure 9).

[0234] According to the first embodiment of the present invention, the virtual stick 91 extending from the user's hand can expand the range and variety of operations and displays beyond the conventional range that can be operated by extending the hand, allowing for more realistic and intuitive operations that take distance into consideration, and making use of the best aspects of user interfaces in the real world. Therefore, the first embodiment can improve user convenience.

[0235] In addition, various methods other than those described above can be applied as the method of displaying the virtual rod shown in the first embodiment of the present invention. For example, at the start of the virtual rod display shown in Figure 5(a), it is possible to use display methods such as setting the length and shape of the virtual rod, or creating a virtual rod through normal operations and holding it.

[0236] The softness of the virtual stick can also be set. For example, it can be set to a soft, string-like shape, a soft, floppy shape, etc.

[0237] Furthermore, in the ninth display method shown in FIG. 8(b), a virtual rod of light 171 is stabbed into a virtual object 172, and the virtual object 172 is moved far away by extending the virtual rod of light 171 (see FIG. 6(a)). However, in consideration of cases where an object is stabbed and moved, it may be possible to set the softness of the side to be stabbed and the sharpness of the virtual rod, to set the part that can be stabbed, and to make this part that can be stabbed visible to the person holding the virtual rod (injured person 14).

[0238] Furthermore, as an application of the virtual mirror described in (Feature 22), there is a method in which the display or object projected by the virtual mirror can be later touched with another virtual stick to give it movement.

[0239] Furthermore, information about the operator of the virtual stick, such as the birthday of the operator, can also be set in the storage unit 62 of the server 6 shown in FIG.

[0240] Furthermore, the attribute information at the tip position of the virtual rod 111 shown by the spatial display image 112 shown in Figure 6(b), the attribute information inside the pipe 122 shown in Figure 6(c), and the attribute information shown in (Feature 12), (Feature 18), (Feature 24), etc. (which may be an object or space, in which case it is basically air, or even a vacuum, which is considered to be an object space called a vacuum) can also be arbitrarily set in the space, for example, like a metaverse space.

[0241] <Second embodiment of the present invention> A user interface system according to the second embodiment of the present invention will now be described.

[0242] In the user interface system according to the second embodiment, VR goggles are used instead of AR glasses. The images of the VR goggles display a VR background image and a VR hand image of the injured person 14 instead of the image of the ceiling 18 (see FIG. 1) and the hand 91 of the injured person 14 that are transmitted through the AR glasses 2 in the AR operation space 71 shown in FIG. 4(b).

[0243] The configuration and operation of the second embodiment of the present invention will be summarized below. The user interface system 1 comprises an imaging means (video camera 4) that captures an image of the user (injured person 14), an aerial operation detection means (circuits and software related to aerial operation detection in the server 6) that detects operations performed by the user in the air based on the image of the user captured by the imaging means, and a display means (VR goggles) that displays an image in a virtual reality space with modified content in accordance with the operation detected by the aerial operation detection means, and the display means uses an image of a virtual rod 91 extending from the user's hand as at least part of the image to be displayed (see Figure 4).

[0244] Furthermore, the user interface system 1 is configured to be able to adjust the length or speed of extension of the virtual stick 91 in accordance with the operation of the hand detected by the mid-air operation detection means (see FIG. 6(a)).

[0245] Furthermore, the user interface system 1 further includes a device control means (circuits and software related to device control of the server 6) that controls predetermined devices (various devices 7) in accordance with the operation detected by the aerial operation detection means.

[0246] Furthermore, the user interface system 1 comprises an imaging means (video camera 4) for capturing an image of the user (injured person 14), an aerial operation detection means (circuits and software related to aerial operation detection in the server 6) for detecting operations performed by the user in the air based on the image of the user captured by the imaging means, and a display means (VR goggles) for displaying an image with modified content in a virtual reality space in accordance with the operation detected by the aerial operation detection means, and the display means uses an image of a virtual rod 91 extending from the user's hand as at least part of the image to be displayed, sets multiple lines in the virtual reality space, and corrects the image so that the tip of the virtual rod 91 is aligned with the multiple lines, thereby correcting hand shake at the tip of the virtual rod 91 (see Figure 8(a)).

[0247] Furthermore, the user interface system 1 comprises an imaging means (video camera 4) that captures an image of the user (injured person 14), an aerial operation detection means (circuits and software related to aerial operation detection in the server 6) that detects operations performed by the user in the air based on the image of the user captured by the imaging means, and a display means (VR goggles) that displays an image with modified content in a virtual reality space in accordance with the operation detected by the aerial operation detection means, and the display means uses an image of a virtual rod 91 extending from the user's hand as at least part of the image to be displayed, and displays the inside or back side of the real object or virtual object by inserting the tip of the virtual rod into a real object in the virtual reality space or a virtual object in the virtual reality space (see Figure 7(b) and Figure 9).

[0248] According to the second embodiment of the present invention, the virtual stick 91 extending from the user's hand can expand the range and variety of operations and displays beyond the conventional range that can be operated by extending the hand, allowing for more realistic and intuitive operations that take distance into consideration, and making use of the advantages of user interfaces in the real world. Therefore, according to the first embodiment, it is possible to improve user convenience.

[0249] The present invention may be applicable not only to VR and AR but also to XR spaces such as the real world and MR.

[0250] Therefore, the system, means, method, etc. of the present invention can be modified in various ways without departing from the spirit of the present invention.

[0251] For example, the imaging means (video camera 4) for capturing video of the user (injured person 14) can be installed not only on the ceiling but also on a desk, and various other applications are possible. Furthermore, when inputting characters, a hiragana table or the like is displayed one by one, and while selecting, the next candidate for the character or word is arranged in a straight line in an easy-to-operate direction, such as up and down, left and right, or diagonal, making it easier to select a candidate and facilitating input. When a circle or the like is specified, the candidates are displayed in a vertical or horizontal list. The user is not limited to the injured person 14, and various applications are possible, such as the general public, workers, educators, and civil servants.

[0252] For example, it is possible to combine two or more systems into one, or conversely, it is possible to configure one system from two or more separate systems and connect them together.

[0253] Furthermore, the first and second embodiments are merely one of the best modes or modes close to the best mode at present. [Industrial Applicability]

[0254] The user interface system of the present invention can be effectively used in businesses and corporations that operate medical facilities and accommodation facilities, private homes, and the like. [Explanation of symbols]

[0255] 1: User interface system 2: AR glasses 3: Projector 4: Video camera 5: Wireless LAN 6: Server 11: Hospital 12: Patient room 13: Bed 14: Injured person 15: Curtain 16: Wall 17: Management room 61: Control unit 62: Memory unit 63: Interface 64: Communication unit 65: Keyboard 66: Mouse 67: Display device 91: Virtual stick

Claims

1. A user interface system comprising: an imaging means for capturing an image of a user; an aerial operation detection means for detecting an operation performed by the user in the air based on the image of the user captured by the imaging means; and a display means for displaying an image in an augmented reality space or a virtual reality space, the content of the image having been modified in accordance with the operation detected by the aerial operation detection means, wherein the display means uses an image of a virtual rod extending from the user's hand as at least a part of the image to be displayed.

2. 2. The user interface system according to claim 1, wherein the extension length or speed of the virtual rod can be adjusted in accordance with the hand operation detected by the mid-air operation detection means.

3. 3. The user interface system according to claim 1, further comprising a device control unit that controls a predetermined device in accordance with the operation detected by the mid-air operation detection unit.

Images