User interface system

Abstract

The present invention relates to a user interface system that can improve user convenience. A user interface system (1) has an imaging device (camera 4) for taking an image of a user (patient 14); an air operation detection device (circuit and software related to air operation detection of a server 6) for detecting an operation performed by the user in the air based on the image of the user taken by the imaging device; and a display device (AR glasses 2) that displays an image of an altered content in an augmented reality space according to the operation detected by the air operation detection device, using an image of a virtual stick (91) that extends from the user's hand as at least a part of the displayed image.

Description

Technical Field

[0001] This invention relates to a user interface system that displays images by detecting user actions. Background Technology

[0002] In recent years, XR (Extended Reality) technologies, such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality), which construct virtual spaces and worlds on computers, have become increasingly popular. Furthermore, among these technologies, various solutions have been proposed for synthesizing augmented reality spaces (AR or MR information) into real space and displaying them through devices such as head-mounted displays.

[0003] For example, there is a spatial manipulation technology that synthesizes a computer-generated keyboard image as AR information into real space and displays the augmented reality space on a head-mounted display, thereby enabling the user to reach out and operate the displayed virtual keyboard (for example, see Japanese Patent Documents 1 and 2).

[0004] Existing technical documents

[0005] Japanese patent documents

[0006] Japanese Patent Document 1, Japanese Patent Application Publication No. 07-078055

[0007] Japanese Patent Document 2, JP 2009-146333 Summary of the Invention

[0008] In the aerial operation technology described in Japanese Patent Documents 1 and 2, since the user needs to reach out to operate the virtual keyboard displayed in the augmented reality space, the operation range is limited to the range of the virtual keyboard that can be operated by reaching out, which limits the range and diversity of operation and display.

[0009] Furthermore, although the aerial operation technology described in Japanese Patent Documents 1 and 2 involves the integration of virtual space and the real world and a world without a sense of distance, it still requires direct touch to operate in many cases, and the operation process is often quite cumbersome.

[0010] Furthermore, with the evolution of information processing technology in recent years, in addition to VR, AR, and MR, virtual spaces such as the Metaverse have emerged. Against this backdrop, the information density in virtual spaces is expected to continue to increase. Objects in both the real world and virtual spaces will transform from simple "paper models" into objects rich in substantial information and meaning, as information density increases. In this context, traditional two-dimensional user interfaces that disregard distance cannot achieve more distance-oriented, realistic, and intuitive operations, failing to fully leverage the advantages of real-world user interfaces.

[0011] The purpose of this invention is to provide a user interface system that improves user convenience.

[0012] The user interface system of the invention as claimed in claim 1 is characterized by: having an imaging device for capturing images of a user; an air operation detection device for detecting operations performed by the user in the air based on the images of the user captured by the imaging device; and a display device for displaying an image with modified content in an augmented reality space or a virtual reality space based on the operations detected by the air operation detection device, wherein the display device uses an image of a virtual stick extending from the user's hand as at least part of the displayed image.

[0013] The user interface system of the invention as described in claim 2 is the user interface system as described in claim 1, characterized in that: the adjustment of the extension length or speed of the virtual rod is made possible based on the hand operation detected by the air operation detection device.

[0014] The user interface system of the invention according to claim 3 is the user interface system according to claim 1 or 2, characterized in that: it further comprises a device control device that controls a predetermined device according to the operation detected by the air operation detection device.

[0015] The user interface system of this invention improves user convenience. Attached Figure Description

[0016] Figure 1 This is a side view of the user interface system according to the first embodiment of the present invention.

[0017] Figure 2 This is a block diagram illustrating the overall structure of the user interface system according to the first embodiment of the present invention.

[0018] Figure 3 This is a flowchart illustrating the operation of the user interface system according to the first embodiment of the present invention.

[0019] Figure 4This is an explanatory diagram illustrating the image processing of a user interface system according to the first embodiment of the present invention.

[0020] Figure 5 This is a first explanatory diagram showing the detailed image processing for AR glasses in a user interface system according to the first embodiment of the present invention.

[0021] Figure 6 This is a second explanatory diagram showing detailed image processing for AR glasses in a user interface system according to the first embodiment of the present invention.

[0022] Figure 7 This is a third explanatory diagram showing detailed image processing for AR glasses in a user interface system according to the first embodiment of the present invention.

[0023] Figure 8 This is the fourth explanatory diagram showing the detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention.

[0024] Figure 9 This is the fifth explanatory diagram showing the detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention.

[0025] Figure 10 This is the sixth explanatory figure, which shows the detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention.

[0026] Figure 11 This is the seventh explanatory diagram showing the detailed image processing for AR glasses in the user interface system according to the first embodiment of the present invention. Detailed Implementation

[0027] <First Embodiment of the Invention>

[0028] The user interface system according to the first embodiment of the present invention will now be described with reference to the accompanying drawings.

[0029] <The Composition of User Interface Systems>

[0030] First, the configuration of the user interface system according to the first embodiment of the present invention will be explained.

[0031] Figure 1 This is a side view of the user interface system involved in the first embodiment.

[0032] Figure 2 This is a block diagram illustrating the overall structure of the user interface system according to the first embodiment of the present invention.

[0033] exist Figure 1In the first embodiment of the present invention, the user interface system 1 includes: AR glasses (augmented reality glasses) 2 worn by the injured patient 14, a projector 3 that displays the image of the operation screen that the injured patient 14 can browse on the bed 13 in the ward 12 of the hospital 11 onto the curtain 15, a camera 4 that takes pictures of the upper body of the injured patient 14 lying on the bed 13, a wireless local area network 5, and a server 6 set up in the management room 17 of the hospital 11.

[0034] AR Glasses 2 is made by using a holographic reflector, making its shape indistinguishable from ordinary glasses, enabling the display of augmented reality (AR), and presenting the same image to the human eye as an AR head-mounted display.

[0035] Projector 3 and camera 4 are installed on the ceiling 18 of ward 12.

[0036] The curtain 15 is used to protect the privacy of the injured patient 14. It is set on the foot side of the injured patient 14 and at a predetermined distance from the foot.

[0037] In addition, the projector 3 needs to perform the following processing first: save the distance between itself and the display surface (curtain 15) through manual input or other settings, or calculate the distance. In addition, the projector 3 also needs to know or set information such as the distance to the display surface, projection candidate surface (curtain 15, wall, monitor), room shape, and space shape in advance.

[0038] Camera 4 needs to perform some processing first: by setting it to save the hand 91 of the injured patient 14 (see...) Figure 4 (a) The distance between them, or the calculation of that distance, etc. In addition, the camera 4 can also pre-set information such as the position of the person (patient 14) and the center point of the fist.

[0039] Wireless LAN 5 includes a wireless LAN host device 51 and a connecting cable. The wireless LAN host device 51 is installed on the wall 16 of the ward 12 and connected to the server 6 in the management room 17 via the connecting cable. Wireless LAN 5 communicates wirelessly with AR glasses 2, projector 3, and camera 4.

[0040] like Figure 2 As shown, server 6 is connected to keyboard 65, mouse 66, display device 67, wireless LAN 5 and various devices 7.

[0041] Various equipment 7 include intercoms in ward 12, lighting switches in ward 12, and call bells installed in management room 17.

[0042] Server 6 is equipped with a control unit 61, a storage unit 62, an interface 63, and a communication unit 64.

[0043] The storage unit 62 stores the program, operation screen and images displayed in the AR operation space of the user interface system 1, video data of the images captured by the projector 3, and the judgment results of the images captured by the projector 3.

[0044] The control unit 61 is connected to the keyboard 65, mouse 66 and display device 67 via interface 63, and various settings are made by relevant personnel through the operation of the keyboard 65 and mouse 66, and the display based on these various settings is displayed on the display device 67.

[0045] Furthermore, the control unit 61 is connected to the projector 3 and the camera 4 via the communication unit 64 and the wireless local area network 5. Based on the input signal from the camera 4, it performs judgment processing and outputs the signal generated by 3D projection mapping to the AR glasses 2 and the projector 3.

[0046] <The Role of User Interface Systems>

[0047] Figure 3 This is a flowchart illustrating the operation of the user interface system according to the first embodiment of the present invention.

[0048] Figure 4 This is an explanatory diagram illustrating the image processing of a user interface system according to the first embodiment of the present invention.

[0049] like Figure 3 As shown, when setting up the user interface system 1, firstly, the injured patient 14 wears AR glasses 2, and the projector 3 performs processing to save the distance and room shape between itself and the display surface (curtain 15), or performs processing to calculate the distance and room shape, etc., through settings. The camera 4 performs processing to save the hand 91 of the injured patient 14 (see...) through settings. Figure 4 (a) Distance processing, or calculation processing, etc.

[0050] Subsequently, in step S1, camera 4 films the upper body of the injured patient 14 lying on hospital bed 13, and... Figure 4 (a) The video data of image 9 shown is sent to server 6.

[0051] Therefore, in step S2, server 6 receives video data from image 9 of camera 4.

[0052] Next, in step S3, server 6 performs image analysis on the received video data to determine... Figure 4 (a) Whether the hand 91 of the injured patient 14 in the image 9 shown is in a grasping state; if the hand 91 of the injured patient 14 is in a grasping state, in step S4, an image of the user-browsable operating space and operating screen is generated. Figure 4 (b) The video data of various images within the AR operating space 71 shown, and Figure 4 (c) shows the video data of image 81.

[0053] In addition, although the above refers to the situation where hand 91 is in a clenched state, other methods can also be used, such as detecting the state of the index finger of the hand and performing operation processing.

[0054] Next, in step S5, server 6 sends the video data generated in step S4 to AR glasses 2 and projector 3.

[0055] Therefore, in step S6, the AR glasses 2 receive video data from the server 6, and in step S7, reflect the image light formed by the video data onto the holographic reflector, so that it is projected into the eyes of the injured patient 14, thereby performing [the desired action]. Figure 4 (b) shows the initial operating space.

[0056] Figure 4 (b) shows the AR operating space 71, an augmented reality spatial image presented to the injured patient 14 through AR glasses 2. This image is visible through the ceiling 18 (see also AR glasses 2). Figure 1 On the image of the hand 91 of the injured patient 14, a spatial display image 73 for the light operation button, a spatial display image 74 as virtual information for the light operation button, a spatial display image 75 for the call button, a spatial display image 76 for the walkie-talkie button, a spatial display image 77 for the cancel button, and a spatial display image 78 for the confirm (OK) button are superimposed.

[0057] In AR glasses 2, in contrast to physical objects such as the ceiling 18 and the hand 91 of the injured patient 14 that can be seen visually, the spatial display images 73-78 are virtual displays that do not exist on site, using video data generated by server 6 in step S4.

[0058] On the other hand, in step S8, projector 3 receives video data from server 6, and in step S9, projects an image based on the video data onto curtain 15 (see...). Figure 1 )superior.

[0059] At this time, the image projected onto the curtain 15 is Figure 4 Image 81 shown in (c).

[0060] Figure 4(c) The image 81 shown is the image of the operation interface. It flips the image of the upper body of the injured patient 14 lying on the hospital bed 13 captured by the camera 4 horizontally, and overlays the images 83, 84, 85, 86, 87, 88 of the operation button for turning off the lights, 85, 86, 87, and 88 of the operation button for calling, 88 of the walkie-talkie button, 89, and 80 of the operation button for confirming (OK) on the image 82.

[0061] In addition, Figure 4 Image 81 shown in (c) shows image 82 after the image of the upper body of the injured patient 14 is flipped left and right. In principle, the screen of human operation can also be omitted, that is, the operator is not photographed.

[0062] If the operator's image is not displayed, then Figure 4 In (c), the operator's image will not appear in the image 82. Instead, images of buttons 83-88 will be displayed on the curtain, and an object resembling a red laser pointer dot (any light-emitting point) will appear on the curtain. Its movement trajectory will be consistent with the movement of the operator's fingers or the front end of the virtual stick held.

[0063] Next, in step S10, the projector 3 continuously films the upper body of the injured patient 14 lying on the hospital bed 13 and sends the video data to the server 6.

[0064] Therefore, in step S11, server 6 receives video data of a new image of the upper body of the injured patient 14 from camera 4.

[0065] Next, in step S12, server 6 performs image analysis on the received video data, automatically calculates the coordinates of the position X1, Y1 of the tip of the index finger of the injured patient 14 and the position X2, Y2 of the grip or center of the hand in the new image shown in the video data, and based on the coordinates X1, Y1, X2, Y2, corresponding to the spatial display images 73-78 of each button displayed on AR glasses 2, detects the operation performed by the injured patient 14 in the air by using a virtual stick, i.e., the air operation.

[0066] Next, in step S13, server 6 generates video data of the augmented reality space presented by AR glasses 2 and the images displayed on the virtual operation interface of curtain 15, based on the air operation detected in step S12.

[0067] Next, in step S14, server 6 sends the video data generated in step S13 to AR glasses 2 and projector 3.

[0068] Next, in step S15, AR glasses 2 receives video data from server 6; in step S16, the image light formed by the video data is reflected onto the holographic reflector, so that it is presented in front of the injured patient 14.

[0069] In this case, the augmented reality image presented to the injured patient 14 through the AR glasses 2 is an overlay of the spatial display images 73 and 74 of the light virtual stick 92, the light operation button, the light-on operation button, the call button, the walkie-talkie button, the cancel button, and the OK button on the ceiling 18 seen through the AR glasses 2 (see [link]). Figure 1 The images of the hand 91 of the injured patient 14 are displayed on the screen; at the same time, in the spatial display images 73-78 of each button, the images of the buttons corresponding to the air operation detected in step S12 are processed with special effects such as light-up.

[0070] Next, in step S17, the projector 3 receives video data from the server 6, and in step S18, projects and displays the image of the virtual operation interface based on the video data onto the curtain 15.

[0071] In this case, the image projected onto the virtual operation interface of the curtain 15 is a left-right flip of the image of the upper body of the injured patient 14 captured by the camera 4 in step S10, and the images of the virtual light stick 92, the light-off operation button 83, the light-on operation button 84, the call button 85, the walkie-talkie button 86, the cancel button 87, and the confirm button 88 are superimposed on the flipped image. At the same time, in the images 83 to 88 of each button, the images of the buttons corresponding to the air operation detected in step S12 are treated with special effects such as light emission.

[0072] Subsequently, in step S19, the server 6 controls various devices 7, such as intercoms, lighting switches, and call bells in the management room 17, based on the detected over-the-air operations.

[0073] To illustrate the operation of this type of user interface system 1 with a specific example: when the injured patient 14 observes through AR glasses 2... Figure 4 (b) The image in the AR operation space 71 is shown. When the front end of the light virtual stick 92 held by the hand 91 of the injured patient 14 is aligned with the space display image 73 of the light-off operation button, the space display image 73 of the light-off operation button is lit up, and the lighting of the bed 13 is turned off.

[0074] When the front end of the virtual light stick 92 held by the patient 14's hand 91 is aligned with the space display image 74 of the light-lighting operation button, the space display image 74 of the light-lighting operation button is lit, and the lighting of the hospital bed 13 is lit accordingly.

[0075] When the tip of the virtual light stick 92 held by the injured patient 14's hand 91 is aligned with the spatial display image 75 of the call button, the spatial display image 75 of the call button lights up, and the call bell set up in the management room 17 rings, and the nurse goes to the location of the injured patient 14.

[0076] When the tip of the light virtual stick 92 held by the injured patient 14's hand 91 is aligned with the spatial display image 76 of the intercom button, the spatial display image 76 of the intercom button will light up; then, if the tip of the light virtual stick 92 is aligned with the spatial display image 78 of the confirmation (OK) button, the intercom between the ward 12 and the management room 17 will be connected.

[0077] When the front end of the light virtual stick 92 held by the injured patient 14's hand 91 is aligned with the space display image 77 of the cancel button, the previous operation will become invalid and the system will return to the state before the operation.

[0078] In addition, when the injured patient 14 watches the video projected onto the curtain 15 by the projector 3... Figure 4 (c) As shown in image 81, when the front end of the light virtual rod 92 held in the hand 91 of the injured patient 14 is aligned with the image 83 of the light-off operation button, the image 83 of the light-off operation button will light up and the lighting of the bed 13 will turn off.

[0079] When the front end of the virtual light stick 92 held by the injured patient 14's hand 91 is aligned with the image 84 of the light-lighting operation button, the image 84 of the light-lighting operation button is lit, and the lighting of the hospital bed 13 is turned on.

[0080] When the tip of the virtual light stick 92 held by the injured patient 14's hand 91 is aligned with the image 85 of the call button, the image 85 of the call button lights up, the call bell set in the management room 17 rings, and the nurse goes to the location of the injured patient 14.

[0081] When the tip of the light virtual stick 92 held by the injured patient 14's hand 91 is aligned with the image 86 of the intercom button, the image 86 of the intercom button lights up; then, if the tip of the light virtual stick 92 is aligned with the image 88 of the confirmation (OK) button, the intercom between the ward 12 and the management room 17 is established.

[0082] When the tip of the virtual light stick 92 held by the injured patient 14's hand 91 is aligned with the image 87 of the cancel button, the previous operation will become invalid, and the system will return to the state before the operation.

[0083] <Detailed Image Processing for AR Glasses>

[0084] The following explains the detailed image processing used in AR glasses.

[0085] Figure 5 This is a first explanatory diagram showing detailed image processing for AR glasses in a user interface system according to a first embodiment of the present invention.

[0086] In the user interface system described in the first embodiment, server 6 (see...) Figure 1 and Figure 2 ) from camera 4 (see Figure 1 and Figure 2 The received video data is analyzed to determine the image. Figure 4 (a) The direction and gripping degree of the hand 91 of the injured patient 14 in image 9 are used to generate an image of the light virtual rod 92 held by the hand 91 as an image of the augmented reality space, and this image is sent to AR glasses 2 as video data. AR glasses 2 receives video data from server 6 and reflects the image light formed by the video data onto the holographic reflector, so that it is presented in front of the injured patient 14.

[0087] like Figure 5 As shown in (a), when a real-world patient 14 forms a large hole 93 between the fingers of hand 91, the image of the light virtual stick held by hand 91 in the augmented reality space presented by AR glasses 2 will be the image of the large light virtual stick 92.

[0088] The front end 92a of the large light virtual rod 92 becomes the operation point, and dotted images 92b are displayed at its position and around it.

[0089] like Figure 5 As shown in (b), when a real-world patient 14 forms a medium-sized hole 94 between the fingers of hand 91, the image of the light virtual stick held by hand 91 in the augmented reality space presented by AR glasses 2 will use a medium-sized light virtual stick 95.

[0090] The front end 95a of the medium-sized light virtual rod 95 becomes the operation point, and dotted images 95b are displayed at its position and around it.

[0091] When the injured patient 14 aligns the front end 95a of the light virtual stick 95 with the space display image 74 of the lighting operation button, the space display image 74 of the lighting operation button will light up, and the lighting of the bed 13 will be turned on accordingly.

[0092] Furthermore, as mentioned earlier, although a configuration could be adopted where the illumination of the bed 13 is turned on when the spatial display image 74 is aligned with the light-lighting operation button, from a practical application perspective, it could also be designed such that, within a set number of seconds, even with slight movement, as long as the light spot remains in the area of ​​the button for a period equal to or exceeding the set number of seconds, it is determined that the button has been pressed. Therefore, even if the light-lighting operation button is not pressed, as long as the light spot is located on the button, the corresponding operation can be performed.

[0093] To elaborate further, in the direction of hand 91, server 6 (see...) Figure 1 and Figure 2 ) using camera 4 (see Figure 1 and Figure 2 The received video data is used to automatically calculate the position X1, Y1 of the tip of the index finger of the injured patient 14 and the position X2, Y2 of the center of the hand or grip posture. The straight line connecting X2, Y2 to X1, Y1 is used as the central axis of the virtual rod to form a... Figure 5 (a) The light virtual rod 92 or Figure 5 (b) The light virtual rod 95.

[0094] When holding the hand in a 91-inch grip, although sometimes the grip shape is considered... Figure 5 The virtual light sticks 92 and 95 in the image, but in order to generate... Figure 5 The virtual light rods 92 and 95 can sometimes automatically calculate and set X3 and Y3 near the upper part held by the hand 91, and automatically calculate and set X4 and Y4 near the lower part held by the hand.

[0095] The grip shape of the hand is usually as follows: Figure 5 The method shown can be modified if the hand grip can only be reversed, i.e., only with the little finger side facing upwards. In this case, the virtual light stick can also extend from behind the hand grip. In this case, the virtual light stick will extend from X3, Y3 to X4, Y4 and reach the contact surface of the ceiling 18.

[0096] Building upon the aforementioned structure, for demonstration or guidance, the shape of the hand 91 can be displayed on the AR glasses 2 using dotted lines (or edge images (images obtained through edge detection in image processing)) at the hand's location in the AR space presented by the AR glasses 2 (demonstrating what actions will occur when the hand is operated). (Alternatively, an image of the hand 91 captured by the camera 4 can be displayed, not the actual image, but rather as dotted lines (edge ​​images are also acceptable) to show the hand's position and other information, thus helping first-time users understand more easily. Furthermore, beyond just the image of a light stick (virtual light stick), overlaying a visual effect similar to a flame stick (virtual flame stick) can more effectively guide the operation. For example, if it is demonstrated as a flame stick, the dotted hand 91 shape in the augmented reality space of the AR glasses 2 can create an effect that appears as if the hand is burning and penetrating a wall, making it easier to understand. Additionally, this can also convey principles or mechanisms.

[0097] Furthermore, when camera 4 is filming the upper body (or observation can be performed via a human body sensor, etc.), the arms can be positioned in a specific posture, or an infrequently performed operation can be used as a signal to indicate the start of operation. For example, bending the elbow to form a V-shape, making several gestures in a preset sequence, or making a circle with the right thumb and index finger while raising the other three fingers will start the user interface of this invention and initiate or end hand observation; another example is crossing the right index and middle fingers to indicate end or start, etc.

[0098] Raising the little finger or middle finger according to set steps, or raising certain fingers in a specific order, can also be given the meaning of ending or beginning.

[0099] Such simplified illustrations or operations can also be used to launch the user interface of this invention.

[0100] Furthermore, when the virtual stick (the user interface of this invention) formed by the index finger or a grip is pointed towards the left or right wall, a set of buttons is displayed on that surface. Additionally, even buttons not facing that surface can display the set of buttons. During operations involving raising the finger or forming a hollow grip, raising the thumb first and then lowering it is considered a single click; continuous single clicks are considered a double click. Even if the finger raising or the hollow grip is deformed, as long as the operation can be recognized as ongoing, processing can continue even with slight distortion of the form.

[0101] As shown in Figure 5(c), when the hand 91 of the real-world patient 14 enters a state where no gap is formed between the fingers, the light virtual stick held by the hand 91 will disappear in the augmented reality space presented by the AR glasses 2.

[0102] Furthermore, when the injured patient 14 aligns the front ends 92a and 95a of the virtual light sticks 92 and 95 with the spatial display image 76 of the intercom button, the spatial display image 76 of the intercom button will light up; subsequently, if the front ends 92a and 95a are aligned with the spatial display image 78 of the confirmation (OK) button, the intercom between the ward 12 and the management room 17 will be connected.

[0103] In addition, as mentioned above, when the space display image 78 of the OK button is aligned with the button, the intercom between the ward 12 and the management room 17 is connected. At this time, after the bright spot (image 92b, 95b) is displayed on the button, the OK button is automatically pressed. However, considering practical applications, it can also be configured such that even if there is slight movement, as long as it can be confirmed that the area on the button has been held for a set number of seconds, it is determined that the button has been pressed.

[0104] In this way, it can also be configured so that there is no need to directly press the OK, turn off the light, turn on the light, or call button. As long as the light spot stays on the button for more than a set number of seconds, the corresponding operation will be performed.

[0105] In this regard, in addition to setting a few seconds, you can also consider pressing by reducing the grip size (i.e., tightening the grip) or by using a stabbing motion.

[0106] In this way, the injured patient 14 can adjust the operation points of the spatial display images 73-78 of the operation buttons in the augmented reality space displayed by the AR glasses 2 by holding the device with his / her hand 91.

[0107] Furthermore, before initiating the operation via the button, the center position of hand 91 can be temporarily placed at: its original position; the observation device (video camera 4, and the circuitry and software related to air operation detection in server 6) determines that it has not moved within a set time; or, during the operation, the observation device determines that it has not moved to a certain extent; or, using the center position of hand 91 as a reference, a sphere is generated with the center as the center and a set radius. When the sphere extends forward from the center position or the center position of the sphere towards the bright spot display surface (the display surface of bright spots (92b, 95b)) of the user interface (light virtual sticks 92, 95) of this invention, and the extension distance exceeds a set value, it is considered a click; if the sphere extends further than the above, and then moves up, down, left, or right, it can be considered a drag-like movement. Regarding the release of the drag state, when the sphere moves closer to the original center position of hand 91 or the center position of the sphere from the drag reference point, it can be considered the end of the drag. The same applies to click operations. When the click reference point moves closer to the center position of the original hand 91 or the center position of the sphere, it can be considered that the click has ended.

[0108] Furthermore, when the observation device captures the facial position of the injured patient 14 and the hand 91 enters the preset multi-level distance range, the movement speed of the light spot can be adjusted according to the preset multi-level distance as the hand 91 gradually approaches the face, making its movement slower and more precise. In this case, the hand 91 can also be placed in front of the face.

[0109] Alternatively, a light spot can be placed on the line connecting the center of the injured patient's head to the fingertips, or on the extension of the straight line connecting the center of the head to the location of the virtual rod. A parameter can be set to represent the distance of the center point of the head in a certain direction. Furthermore, to avoid overly capturing the vibrations of the human body (injured patient 14), a certain degree of shaking can be considered as error or noise and not reflected in the movement of the light spot.

[0110] In addition, to stabilize the movement of the light spot, the injured patient 14 can place his / her hand 91 on his / her leg or on a table for stability.

[0111] In addition, if there is an interface in front of the ceiling 18, other layers of interfaces will be displayed.

[0112] For example, when the interface composed of light virtual sticks 92 and 95 is oriented towards the wall (ceiling 18) directly above oneself (patient 14), the call nurse button, one's own reading light, and the TV switch will appear as images of the second layer.

[0113] Furthermore, if the interface formed by the virtual light sticks 92 and 95 is oriented towards the entrance of the room, the switches for the air conditioner, the air conditioner, and the ceiling lights will appear as images of the third layer.

[0114] Furthermore, in the first embodiment, switch groups for other rooms can also be displayed. One can imagine the concept of web browser tabs, for example, displaying five layers of content in tab format.

[0115] In addition, in the first embodiment, each layer can also be designed such that when the direction of the virtual light bars 92 and 95 (such as the top, east, or north of the wall, or the east or west of the curtain, the first display surface, the second display surface, the third display surface, etc.) changes, the first to fifth layers are displayed respectively according to their direction. Such a user interface I / F design is also possible.

[0116] Furthermore, in the first embodiment, label-based displays are typically not used; instead, special settings can be configured for administrators and other personnel to operate the system. For example, in a smaller hospital, the status of cameras in each room, the on / off status of air conditioners and overhead lights can be monitored at the entrances and exits, allowing for adjustments and reducing operational complexity. Additionally, the number of monitors and other equipment can be reduced. In larger hospitals, similar special settings can be implemented at nurse stations on each floor.

[0117] Furthermore, in the first embodiment, by adjusting the camera 4 (see...) Figure 1 and Figure 2 The received video data is analyzed to calculate the hole in the hand, thereby determining the location within the AR operating space 71 (see [link]). Figure 4 (a) The degree of gripping state of the hand 91 of the injured patient 14 is not limited to calculating the cavity of the hand, but can also be constituted as the basic area or volume change such as the increase of the area or volume of the hand observed from the horizontal, vertical and other angles, and the specified action is triggered when the reference value is exceeded.

[0118] Furthermore, when using multiple cameras 4, to improve calculation speed, when the movement speed slows down or the size change is small, the multiple cameras 4 can be integrated into one to perform only the area or edge shape calculation. However, in the case where only one camera 4 is used to perform the area or edge shape calculation, the error can be set to ±3 square centimeters or ±3 cubic centimeters, etc., or the tremor or shape recognition error during human movement can be automatically set based on factors such as gender, date of birth, height, weight, body fat percentage, arm thickness, hand size, and finger thickness. The setting can be arbitrary.

[0119] Furthermore, the direction and point of the light virtual rod in the first embodiment shown in Figure 5 can also be determined based on the surface shape of the hand 91 of the injured patient 14.

[0120] Furthermore, the same idea applies to the error of the opponent 91 in the first embodiment. Over-reproducing the motion would cause the point at the tip of the virtual light bar to jitter, making over-reproduction of the motion undesirable. Therefore, noise countermeasures can be considered, such as not detecting motions exceeding a certain speed or not detecting motions above a certain frequency. Additionally, interpolation processing of the trajectory of the opponent 91 can also be considered.

[0121] The thickness of the images 92b and 95b of the dots at the front of the interface (light virtual rods 92 and 95) can be reduced or thinned according to the movement of the hand 91, which can also be regarded as the operation of pressing a switch.

[0122] After the operation of the piercing interface (virtual stick 92, 95), if you start from the piercing position, keeping it parallel to the projection surface (i.e., the contact surface of the virtual stick, such as a curtain, wall, ceiling, or monitor), and keeping the distance from that surface constant (a certain degree of error is allowed in this case, which can be saved through settings, etc.), you can achieve the same effect as mouse dragging. (In terms of processing, the movement trajectory can be represented as a trajectory being pressed and drawn; the pressing force can be set, or it can be directly reflected as the pressing force by measuring the distance of the piercing action (or the depth into the contact surface can be measured; in this understanding, the usual contact surface can be regarded as the state of the front end of the virtual stick in contact, which is more appropriate in some cases). As mentioned above, a certain degree of noise and error needs to be processed.

[0123] In situations where the same effect as mouse dragging is achieved, page turning can also be implemented.

[0124] In addition, actions such as bolding the interface (virtual sticks 92 and 95) or thrusting forward can also be considered as pressing a switch. Similar to a mouse click, repeated actions can be considered as a double click.

[0125] The movement speed of the light dots displayed on the interface (virtual light sticks 92, 95) and AR glasses 2 or contact surfaces (curtains, walls, ceilings, displays) needs to be adjusted. Calculations are expected to be required to suppress and control vibrations generated during human operation as much as possible. To this end, the following measures can be taken: input actions above a certain frequency should not be captured, or minute movements should not be captured; that is, movements within a certain distance should not be considered actions and should not be converted into movement of light dots. For example, hand vibrations can be corrected using a camera that captures motion, i.e., only the movement of the captured object is extracted, and the vibration of the hand itself is removed. Alternatively, assistive devices can be worn on the hand to prevent hand vibrations, as needed. Braces similar to hand casts or surfaces for placing the hand can also be provided.

[0126] Furthermore, the handling of interfaces with the same shape can be modified. For example, when displaying the virtual light sticks 92 and 95, clicking with two fingers raised (index and middle fingers raised in a fist-like "scissor-hand" gesture) can be equated to a right-click on the mouse. Multiple operation modes can also be set based on different hand shapes.

[0127] Furthermore, the first embodiment can be modified in various ways. For example, as a display unit, it is not limited to an AR space or curtain displayed on AR glasses; it can also be used for walls or liquid crystal display screens.

[0128] Furthermore, in the first embodiment, considering the direction of the injured patient 14's face or gaze, the image of the projection target or display button on the projector 3 may sometimes only be displayed in the direction expected to be seen by the person. That is, usually the image of the projection target or display button on the projector 3 will be displayed in the direction the finger is pointing, or on a curtain or wall, or a display (such as an LCD display as a display device) that the virtual light stick is pointing at; however, these can also be ignored, and the image can be projected directly in the direction of the gaze. This method is particularly suitable for injured patients 14 whose facial orientation is very different from their body orientation.

[0129] Furthermore, in the first embodiment, such as Figure 4 As shown in the diagram, although the initial illustration depicts operation by holding the hand, it can also be configured so that the default setting is two fingers, and when operating with three fingers raised, it can be as follows: Figure 5 (a) shows the bolded virtual bar; when a finger is raised, it can be operated as follows: Figure 5 (b) shows the thinning of the light virtual rod, etc. Furthermore, this mechanism can also be understood as not adjusting the thickness, but rather representing the amount of forward thrust during a stabbing action (or, relatively speaking, the depth relative to the projection surface of the projector 3). In other words, the difference in the number of fingers between the default two-finger operation and the multi-finger operation can be considered as the amount (depth) of forward thrust during a stabbing action. Furthermore, if the light virtual rod is considered as a flame virtual rod, it can also be understood as a process that changes the output or temperature of the virtual rod.

[0130] Furthermore, in the first embodiment, the shape of the virtual light stick when dragged to the operation button can be given a specific meaning. For example, drawing a circle (or any shape such as a triangle or cross) during the dragging process can be used to set various operations, such as turning on the air conditioner switch or turning off the light switch in the current room.

[0131] Furthermore, in the first embodiment, if the force of the dragging motion and the forward thrust during the stabbing motion (which can also be considered as the depth relative to the projection surface of the projector 3) are insufficient, the light will not be able to penetrate the wall and enter the next room, but will only scratch the wall, thus representing the properties of the physical wall and the properties of the virtual flame rod (of course, it can also be set as a fragile wall, and the existence of a wall is usually not imagined). For example, assuming there is a virtual light rod, the forward thrust (value) required for the light to penetrate to another room during the stabbing motion is preset. When this value is exceeded, the virtual light rod will reach the adjacent room. In addition, the thrust force required to reach the room if the thrust force is too small, resulting in the appearance of scratches on the wall, can also be set.

[0132] If the buttons on the user interface can reproduce the force during operation, then it can be reproduced.

[0133] When operating buttons or other devices equipped with pressure sensors, an operation reproduction device that can simulate the gradual pressing of the button can reproduce effects such as gradually increasing pressure (in the dragging trajectory) or gradually decreasing pressure (in the dragging trajectory).

[0134] Specifically, it can reproduce actions similar to sliding a finger to the left from the center of a relatively large button.

[0135] In summary, the operation screen of AR glasses 2 or curtain 15 includes an operation reproduction device capable of reproducing the user's gradual pressing operation. When the user operates a button with a pressure sensor, the operation reproduction device can reproduce the gradually increasing pressing force and the gradually decreasing pressing force.

[0136] Furthermore, in the first embodiment, adjacent buttons can be clicked consecutively, or the user can operate the device by pressing a component resembling a brightness adjustment bar with appropriate force, while adjusting the shape of the left / right, up / down, etc. adjustment bars. In application, this creates the illusion of sculpting a wall, allowing for the creation of deep drawings or text on the wall. This information will be reflected in the display of AR glasses 2 or projector 3. Alternatively, it can be physically reproduced on the wall using a robot. This information can also be used for the maintenance of building exteriors, etc.

[0137] By combining this information with the settings, when the wall of the next room is broken in the virtual environment, various buttons in the next room can be displayed and made operable. In addition, by performing the operation of breaking the wall, it is not limited to the condition of a room wall, but can display and operate only the operation interface (hierarchy) containing only the air conditioning equipment of the entire building, or only the operation interface (hierarchy) containing only the lighting switch equipment.

[0138] Another example is that, based on the forward thrust during the stabbing action (which can also be considered as the depth measured from the ceiling 18 visible from the AR glasses 2 or the projection surface of the projector 3), a button class for a certain floor (e.g., the 5th floor) can be displayed; when the thrust moves from one range to another, the button class for a certain floor (e.g., the 4th floor) can become operable.

[0139] Furthermore, in the first embodiment, if a virtual pointer is pointed at a point on an operation button and dragged to draw a circle or similar shape (if an area or enclosed space can be specified), a form can be envisioned where all buttons within that circle—in other words, within the cylinder of the building—are arranged on the interface, regardless of their location in different rooms. This form is considered helpful for maintenance troubleshooting and other tasks. For example, if not only the button types of the rooms in that direction are visible, but also the operating status of buttons for various other devices, such as refrigerators and televisions, it becomes possible to check not only the equipment in the building and rooms, but also the operating status of individual devices (such as refrigerators and televisions), thus potentially allowing for efficient location of faults. While refrigerators and televisions are included here, they are, of course, also considered as operable objects in normal operation. Furthermore, when maintenance personnel arrive at the building or similar location, this human-readable interface design (I / F) is expected to be more efficient and effective.

[0140] Furthermore, in the first embodiment, for maintenance purposes, cameras and buttons (including buttons for refrigerators, televisions, etc.) can be installed in each room to obtain information such as the status of each device and machine through communication, and the content can be categorized. For example, only similar switches such as air conditioner switches (remote control information) can be collected, regardless of room affiliation, and arranged on the operation interface. (Of course, as long as they can be efficiently arranged on the operation interface, whether it is two or three types is fine.) The operation interface can also be set as a label (hierarchy). In this case, in order to display the status of switch-type devices, even if they are not actually fluorescent switches, they can be displayed in an appropriate form. In addition, the operating status of complex devices such as air conditioners can be displayed next to their respective air conditioner switches, or directly arranged on the operation interface. In this way, information such as the operating status of air conditioners can be obtained from the cameras in each room, the devices built into the switches, or devices that are not limited by form but operate as information collection devices (monitoring temperature, power consumption, etc.) and displayed side by side with the relevant switches, making the operating status clear at a glance. Of course, by summarizing relevant information, one can understand the maintenance status or current situation even without being on-site, and make appropriate adjustments efficiently.

[0141] Furthermore, in the first embodiment, a larger button than usual can be displayed in order to accurately identify the button.

[0142] The size, magnification, and reduction of the buttons can be saved by the settings of the components of the user interface system (such as server 6), or they can be linked to the thickness of the virtual light bar. Alternatively, they can be configured to be set and saved based on the operator's specific situation (actual age, whether glasses are worn, glasses prescription, i.e., numerical information such as lens correction intensity), and the contact surface conditions between the buttons and the virtual light bar and the wall (such as the distance between the person and the wall, ambient brightness, and display brightness, which can also be input through settings).

[0143] Furthermore, in the first embodiment, the light-emitting point (pointer) used as an operation mark, the point on the front of the operator's finger, or the point on the front of the virtual light stick held virtually (calculated as the contact point between the curtain and the front of the finger) may sometimes cause interference to the operator. Therefore, it is designed as a light-emitting point with transparency, and can be made transparent as needed.

[0144] In the case of a transparent light spot, it is difficult to determine the position of the operation. In this case, the button or other object can be lit up or flashed to indicate that the transparent light spot is currently on the button.

[0145] Alternatively, it can be configured such that short lines are set at the edges of the display frame of the projector (the vertical frame (which can be either the left or right frame line or both), the horizontal frame edge (which can be either the top or bottom frame line or both)) and the field of view edge of the AR glasses (the edges of the vertical and horizontal frames), and the operator can identify in a way that there is a bright spot at the intersection of the line virtually extending from the horizontal frame edge and the line virtually extending from the vertical frame edge.

[0146] (A) When the line of sight is detected to be directed toward the area near the light spot (cursor), or the face is directed toward the display surface (cursor), the amount of movement will be reduced compared to normal, and the cursor movement will become slower, which means that fine operations can be performed.

[0147] (B) In addition, the cursor movement can be slowed down when actions such as gripping or clenching are performed, thus enabling fine and subtle movements. That is to say, even in normal mode, when the finger or the hand holding the virtual light stick moves the same distance, the cursor moves a shorter distance than usual. This control method can also be used.

[0148] Here, noise (the subtle jitters of a living organism) can cause cursor jitter and other phenomena, thus increasing the difficulty of operation, and should therefore be eliminated as much as possible; however, in the cases of (A) and (B) above, combined with subtle movements, if the amplitude is greater than the usual level or the frequency is higher, it will be discarded as noise. That is to say, unless it is in a state with a larger amplitude, slower movement, and lower frequency than usual, the movement information will be deleted as noise.

[0149] Furthermore, by observing bio-information such as body swaying or heart rate, the observed body swaying can be incorporated into the calculation and eliminated as a subtle vibration (noise) that differs from the main movements of the fingers or the hand holding the virtual light stick.

[0150] Furthermore, in the first embodiment, the imaging device for observing the hand 91 of the injured patient 14 is not limited to a general camera 4, but can also use a night vision camera, thermal imaging camera, infrared camera, etc., so as to capture the movement of the hand 91 even in dark environments.

[0151] Furthermore, in the first embodiment, instead of a wooden virtual stick or a hole formed by hand grip, supplementary processing or calculations can be performed to enable similar operations even when using a metal tube or plastic virtual stick (which can be a glowing virtual stick to consider operation in a dark environment), and corresponding settings can be made accordingly. Of course, when the camera is shooting, only the hand can be illuminated.

[0152] <Image processing for other AR glasses>

[0153] The following will combine Figures 6 to 11 This explains the various image processing techniques used in AR glasses.

[0154] Figure 6 This is a second explanatory diagram showing detailed image processing for AR glasses, which is part of the user interface system according to the first embodiment of the present invention.

[0155] Figure 7 This is a third explanatory diagram showing detailed image processing for AR glasses, which is part of the user interface system according to the first embodiment of the present invention.

[0156] Figure 8 This is the fourth explanatory figure showing detailed image processing for AR glasses in a user interface system according to the first embodiment of the present invention.

[0157] Figure 9 This is the fifth explanatory diagram showing the detailed image processing of AR glasses in the user interface system according to the first embodiment of the present invention.

[0158] Figure 10 This is the sixth explanatory figure showing the detailed image processing of AR glasses in the user interface system according to the first embodiment of the present invention.

[0159] Figure 11 This is the seventh explanatory diagram showing the detailed image processing of AR glasses in the user interface system according to the first embodiment of the present invention.

[0160] exist Figures 1 to 5 In the first display method other than those shown (hereinafter referred to as "other"), such as Figure 6 As shown in (a), in the AR space displayed by AR glasses 2, there is a spatial display image 102 showing a light virtual rod 101 held by hand 91, a meter indicating the length of the light virtual rod 101, and a spatial display image 103 showing a meter indicating the elongation speed of the light virtual rod 101.

[0161] Operators wearing AR glasses 2 (including patients 14 and ordinary operators) adjust the length of the light virtual rod 101 by holding the spatial display image 102 with their hands 91, and adjust the extension speed of the light virtual rod 101 by holding the spatial display image 103 with their hands 91.

[0162] Through this display method, for example... Figure 10 As shown in (c), the physical aerial drone can be easily controlled using the light virtual stick 101.

[0163] In other second display methods, such as Figure 6 As shown in (b), in the AR space displayed by AR glasses 2, a light virtual rod 111 held by hand 91 is displayed, and a spatial display image 112 is displayed at the front end of the light virtual rod 111, showing the attribute information of the front end position.

[0164] Assuming the operator wearing AR glasses 2 is a worker carrying out building demolition work, the spatial display image 112 at the front end of the light virtual rod 111 will show the interior of the building wall, including the structure of the interior of the building wall (steel bars, cavities, concrete, etc.).

[0165] In other third display methods, such as Figure 6 As shown in (c), a virtual light rod 121 held by hand 91 is displayed in the AR space shown by AR glasses 2. Assuming the operator wearing AR glasses 2 is a worker in a building demolition operation, the virtual light rod 121 can be inserted into the pipe 122 to display the attribute information inside the pipe 122.

[0166] In other fourth display methods, such as Figure 6 As shown in (c), the optical virtual rod 123 is made into a soft and flexible shape, like rubber or thread, so that it can guide real or virtual kites or drones through the front end of the optical virtual rod 123. Virtual kites or drones can be guided underground or inside buildings through the optical virtual rod 123.

[0167] The light virtual rods 121 and 123 can also be used as penetrating laser pointers.

[0168] In other fifth display methods, such as Figure 7As shown in (a), in the AR space displayed by AR glasses 2, a light virtual stick 131 held in hand 91 is displayed. The light virtual stick 131 reflects the physical LCD TV 132 like a mirror and is bent, displaying a spatial display image 134 of the back of the LCD TV 132 at the front end of the bend 133. The wiring and on / off status of the back of the LCD TV 132 are displayed in the spatial display image 134. The light virtual stick 131 can be conceived as a foldable luminous stick.

[0169] In other sixth display methods, such as Figure 7 As shown in (b), in the AR space displayed by AR glasses 2, a light virtual stick 141 held by hand 91 is displayed. The object at the front end of the light virtual stick 141 can be replaced, and can be replaced with a camera 143 for observing the back of object 142, a drill bit 144 for drilling virtual holes in object 142, a paintbrush, a marker, a lantern, a flashlight, etc. Corresponding to the replaced front end object, various virtual displays 145 are implemented.

[0170] In other seventh display methods, such as Figure 7 As shown in (c), a light virtual stick 151 held by hand 91 is displayed in the AR space shown by AR glasses 2. Hand 91 and light virtual stick 151 can be copied as spatial display images 91a, 91b, 91c, 151a, 151b, 151c. By copying, surfaces or lines can be displayed, moved synchronously, or made to fly out. AI can also operate various switches by moving the spatial display images 91a, 91b, 91c, 151a, 151b, 151c. In addition, AI can also recognize and analyze the virtual stick movements of hand 91 and light virtual stick 151, and can also copy the movements.

[0171] In other eighth display methods, such as Figure 8 As shown in (a), in the AR space displayed by AR glasses 2, virtual light rods 161, 162, 163, and 164 held by hand 91 will be displayed in sequence.

[0172] A chart-like spatial display image 165 is displayed in AR space. The spatial display image 165 has multiple vertical lines 166... ​​and multiple horizontal lines 167.... The image is corrected so that the multiple vertical lines 166... ​​and multiple horizontal lines 167... are aligned with the front ends of the aforementioned virtual rods 161, 162, 163, and 164, thereby correcting the jitter of the front ends of the aforementioned virtual rods 161, 162, 163, and 164.

[0173] Therefore, when using virtual sticks to draw spatial display images containing a large number of straight lines, such as buildings, in AR space, the ease of operation can be improved.

[0174] In other ninth display methods, such as Figure 8As shown in (b), in the AR space displayed by AR glasses 2, a light virtual stick 171 held by hand 91 is displayed. The light virtual stick 171 is used to stab the virtual object 172 and to stretch the light virtual stick 171 (see [reference]). Figure 6 (a)). Thus, the virtual object 172 is moved to a distance.

[0175] In other 10th display methods, such as Figure 8 As shown in (c), in the AR space displayed by AR glasses 2, a light virtual stick 181 held by hand 91 is displayed. The light virtual stick 181 is used to strike the virtual watch 182, and the operation of stretching the light virtual stick 181 is performed (see...). Figure 6 (a) ). Thus, the virtual table 182 is moved to a distance and attached to a wall or other surface.

[0176] In other eleventh display methods, such as Figure 9 As shown in (a), in the AR space displayed by AR glasses 2, a light virtual stick 191 held by hand 91 is displayed, and the image of the light virtual stick 191 stabbing the physical display 192 is shown. Figure 9 (a) shows an image of a turbulent sea, which forms a tunnel 194 on screen 193 of display 192, and displays the underwater scene within the tunnel 194. It can also be set to display information under the rubble (such as internal images, the location of life-saving equipment, etc.) depending on the situation of the stabbing scene.

[0177] In other 12th display methods, such as Figure 9 As shown in (b), in the AR space displayed by AR glasses 2, a light virtual rod 201 held by hand 91 is displayed. The light virtual rod 201 is in a state of extending to the distance, and its length along the way can be omitted.

[0178] In other 13th display methods, such as Figure 9 As shown in (c), in the AR space displayed by AR glasses 2, a light virtual stick 211 held by hand 91 is displayed. The light virtual stick 211 is extended into the distance, and its length can be omitted. The light virtual stick 211, by pointing its tip at a distant point region 212, can display a magnified spatial display image 213 of region 212. Furthermore, by pointing the light virtual stick 211 at the spatial display image 213, various operations can be performed on the distant point region 212. Because the spatial display image 213 is magnified, actions can be delayed when the light virtual stick 211 points at this image.

[0179] In other 14th display methods, such as Figure 10As shown in (a), when the concrete object 221 is cored 222 and 223, its information can be displayed by operating the light virtual rod when needed (e.g., during maintenance), or a virtual camera can be embedded in the cored object 222 and 223 and the images captured by the virtual camera can be displayed.

[0180] In other 14th display methods, such as Figure 10 As shown in (b), when using a camera 237 that moves behind the concrete object 236 or a camera 238 that moves behind the object, a light virtual stick 231 or a virtual line 234 held by a hand 91 is displayed in the AR space displayed on the AR glasses 2; by moving the light virtual stick 231 to positions 232, 233, etc., the camera head 237 or the camera 238 that moves behind the object 236 can be operated; or by moving the pulling position of the virtual line 234 to position 235, etc., the camera 237 or the camera 238 that moves behind the object 236 can be operated.

[0181] In other 15th display methods, such as Figure 10 As shown in (c), in the AR space displayed by AR glasses 2, a light virtual rod 241 held by hand 91 is displayed.

[0182] Operators wearing AR glasses 2 (including injured patients 14 and ordinary operators) hold the spatial display image 102 with their hands 91 (see Figure 6 (a) to adjust the length of the light virtual rod 241; hold the spatial display image 103 by hand 91 (see Figure 6 (a) Adjust the elongation speed of the light virtual rod 241 by swinging the front end of the light virtual rod 241 by swinging the hand 91 up, down, left and right.

[0183] This display method allows for easy manipulation of a physical aerial drone 242 using the optical virtual stick 101. The aerial drone 242 will then be guided to a position corresponding to the point at the tip of the optical virtual stick 241.

[0184] In another 16th display method, the predetermined trajectory of the light virtual rod is prompted to the operator (including the injured patient 14 and ordinary operators) through sound, light, color and other means.

[0185] In other 17th display methods, such as Figure 11 As shown in (a), in the AR space displayed by AR glasses 2, a light virtual stick 251 held in hand 91 is displayed, and near the right side of area 252, a space display image 253 that would initially display a light switch is displayed.

[0186] When the operator (including the injured patient 14 and ordinary operators) aligns the front end of the light virtual stick 251 with the spatial display image 253 of the light switch, the spatial display image 253 of the light switch will light up, and the indoor lights will turn on.

[0187] When the spatial display image 253 is lit, based on the operator's predicted actions by server 6, a spatial display image 254 showing the air conditioner switch is displayed near the upper left corner of area 252. Furthermore, on the operator's birthday, a spatial display image 254 celebrating the birthday is displayed near the lower edge of area 252.

[0188] When the operator aligns the front end of the virtual light bar 251 with the spatial display image 254 of the air conditioner switch, the spatial display image 254 of the air conditioner switch lights up, and the indoor air conditioner starts.

[0189] In the 17th display method, specific content such as the space display image 254 for birthday wishes can also be displayed.

[0190] In other 18th display methods, such as Figure 11 As shown in (b), in the AR space displayed by AR glasses 2, a light virtual stick 261 held by hand 91 is displayed; in region 263 near the left side of region 262, spatial display images 263-1, 263-2, and 263-3 of the first to third selection candidates are displayed.

[0191] In the AR space displayed on AR glasses 2, the following spatial display images 263-1 show the operation history 264 of the previous light virtual stick 261 and the spatial display image 265 of the operated switch; the following spatial display images 263-2 show the operation history 266 of the previous light virtual stick 261 and the spatial display image 267 of the operated switch; the following spatial display images 263-3 show the operation history 268 of the previous light virtual stick 261, the spatial display image 269 showing a meter indicating the length of the light virtual stick 261 corresponding to the previous operation history 268, the spatial display image 270 showing a meter indicating the elongation speed of the light virtual stick 261 corresponding to the previous operation history 268, and the spatial display image 271 indicating that the operation corresponding to the previous operation history 268 has been cancelled.

[0192] When the front end of the light virtual rod 261 is aligned with the spatial display image 263-1 of the first selection candidate, the same operation of the virtual rod 261 as the operation history 264 will be automatically performed, and the spatial display image 265 of the switch will be operated.

[0193] When the front end of the light virtual rod 261 is aligned with the spatial display image 263-2 of the second selection candidate, the same operation of the virtual rod 261 as the operation history 266 will be automatically performed, and the spatial display image 267 of the switch will be operated.

[0194] When the front end of the light virtual rod 261 is aligned with the spatial display image 263-3 of the second selected candidate, the same operation of the virtual rod 261 as the operation history 268 will be automatically performed, and the operation will be cancelled.

[0195] The features of the first embodiment will be described in sequence below.

[0196] (Feature 1) In the AR space displayed by AR glasses 2, the virtual stick (which is operated by extending or shortening it by hand as before) can not only trigger the switch of the contact position, but also make the position appear transparent as if it has a hole through the center, or conversely, make the back visible.

[0197] (Feature 2) In the AR space displayed by AR glasses 2, the back can be observed by setting virtual mirrors or other means to make the virtual stick reflect.

[0198] (Feature 3) In the AR space displayed by AR glasses 2, buildings or papers attached to the front end of the virtual stick can move together as the virtual stick extends.

[0199] (Feature 4) In the AR space displayed by AR glasses 2, objects such as virtual tables can be attached to the front end of the virtual stick and moved. The virtual table can be used as a guide at the location where it is placed. When the front end of the virtual stick is far away, the operation may not be precise enough; in this case, the lines of the table can be used as an auxiliary guide for correction.

[0200] (Feature 5) In the AR space displayed by AR glasses 2, the image of the camera attached to the front end of the virtual stick can be displayed. Due to the distance of the front end, it can be seen even when it is penetrating a thick wall, and the things seen by the front end after penetration can be seen.

[0201] (Feature 6) In the AR space displayed by AR glasses 2, when the virtual stick passes through an object, it will trigger an action, and it can also be made visible through a projector (monitor, screen, etc.).

[0202] (Feature 7) In the AR space displayed by AR glasses 2, since the location of contact and the interior of a wall with a certain thickness that is penetrated are both assigned attributes, these attributes can also be considered together, or preset actions can be set for that location. For example, when located on a massage chair, in a toilet, or inside a wall (in an emergency, surrounded by rubble), the external view can be displayed; the internal view can also be displayed in reverse, etc.

[0203] (Feature 8) In the AR space displayed by AR glasses 2, the virtual stick (the distance extended from the hand can be calculated and determined, and can be copied including the hand, and the copy itself can be fine-tuned or moved) can also achieve a see-through effect at the position touched.

[0204] (Feature 9) In the AR space displayed on AR glasses 2, the other side can be seen (like a mirror). The switch on the other side can also be pressed. A portion of the wall seen through is also visible. The switch in the middle of the wall seen through can also be pressed.

[0205] (Feature 10) In the AR space displayed by AR glasses 2, a portion of the wall thickness can be seen (not limited to walls, but also in water). The portion of the wall thickness can be touched (although it is envisioned for VR, in reality, as long as there is data and a switch, it will suffice).

[0206] (Feature 11) In the AR space displayed by AR glasses 2, the virtual stick can penetrate multiple walls. The tip of the virtual stick can repeatedly perform this operation (passing through a wall, penetrating, and moving to the next wall).

[0207] (Feature 12) In the AR space displayed by AR glasses 2, a computational mechanism is implemented: when the tip extending from a certain starting point or fulcrum matches the calculated length or position, it can touch the object. Various objects can be touched. However, if the tip is set to a sharp mode, it can pierce the object in front. Increasing the sharpness or hardness allows it to pierce even hard objects. Furthermore, if it continues to move forward, it can be fixed like a thumbtack. However, because it involves electronic data, the touch is merely data superposition, and does not cause damage to the real object.

[0208] In other words, the action (i.e., the touch situation) can be changed based on the combination of attribute information of the toucher and the touched. An overlap mode can also be set. In this case, the action will also change according to the attributes.

[0209] (Feature 13) In the AR space displayed by AR glasses 2, if the 3D space has a length, the distance to that length can be adjusted and measured. The accuracy can be relatively low at this point. Initially, the default length is fixed. Paper, boards, concrete slabs, the top of buildings, etc., can be placed at its front end. Giant buildings can also be placed by extending the rod. Virtual paper can also be attached to the LED display screen.

[0210] (Feature 14) In the AR space displayed by AR glasses 2, the virtual stick can be designed as a stick extending from the hand, or it can be manipulated by the fingers. In addition, it can not only extend in one direction, but also be designed to extend in two directions.

[0211] (Feature 15) A GUI for shortening the distance of the virtual stick can be displayed in the AR space displayed on AR glasses 2.

[0212] (Feature 16) In the AR space displayed by AR glasses 2, the virtual stick can penetrate and overlap, and its movements will change based on the attributes of the contact or penetration position, action records, and reaction records. In the virtual space, its movements depend on the movement of the contact side; while in the real world, the movements are presented through 3D mapping and other methods. Furthermore, it can be set to penetrate the wall in front of it and touch the next wall, etc. The movements can also be changed based on information from the contact side.

[0213] (Feature 17) In the AR space displayed by AR glasses 2, there is a virtual rod (with length, which (usually a straight line, but can also be curved, but would reduce intuitiveness) can be extended, and its contact with a virtual surface, a mirror principle, can project light onto the curved tube to display flow), starting from its operation method, specifying the virtual rod to be operated.

[0214] (Feature 18) In the AR space displayed by AR glasses 2, a virtual stick can be used to draw the trajectory of a drone. The thickness of this trajectory can be adjusted. If the tip of the stick has a pen that can write in the air, it is a stick corresponding to a specific purpose. Depending on the object attribute information of its tip and the object it contacts, and corresponding to the purpose, it will become a stick with a different purpose at its tip. For example: coloring the air; drawing lines similar to curved, rigid (iron) tracks for the drone to fly along; grabbing and moving it with a robotic arm; or acting as a camera, etc. The tip needs to change according to the purpose or object (of course, multiple tips can be equipped at the same time). That is to say, here, it is necessary to identify the virtual stick (with length and tip information) and its tip, identify the space or object that the tip contacts, organize the attribute information (which can be the target object or space; in this case, air and vacuum are basically considered as objects or spaces called "vacuum"), and then match the attribute information of the target with the attribute information of the virtual stick tip that matches the purpose, thereby achieving the goal. In reality, it is possible to identify real space and target objects and acquire real-time data (through information from surrounding or front-end cameras analyzed by humans or AI), or to pre-acquire space and object information and identify the scene (assuming it exists from the beginning, like a constructed VR space), and to construct and master the situation.

[0215] (Feature 19) In the AR space displayed by AR glasses 2, operation can vary depending on the properties of the rod itself, such as control methods involving a rigid rod or control methods using a flexible line. When the target object is located on a rigid, curved track, if a rigid rod is used, the rod is extendable and can pass through or penetrate even if it overlaps with the track. Of course, even if it has properties such as a rigid rod or a real line, it is not necessarily required to do so.

[0216] (Feature 20) In the AR space displayed by AR glasses 2, to prevent drones from colliding, a trajectory can be automatically calculated based on factors such as the position of walls, sometimes appearing as a line, and sometimes as a cylindrical shape. This can serve as the front end for performing the calculations or as the front end for depicting the space. If it is not linear, a virtual rod can be used to indicate movement within the space. The rod-shaped object can be viewed as a leash for a dog, or used as a rigid rod to move drones, or to move a moving support object (which can move autonomously or be driven by a surrounding support system). The positions of the drones can be adjusted. Imagine connecting the drones with lines, moving only the initial drone using a fixed rod.

[0217] (Feature 21) In the AR space displayed by AR glasses 2, as an interface for short-range control of drones and other operations, a cylindrical rod with a built-in camera, a power cord and its track camera, an air-driven detection rod, or a rod driven by a wire or steel wire, a plastic gear for rotation, a gear near the wire to control drones, etc.

[0218] (Feature 22) In the AR space displayed on AR glasses 2, a virtual handle can be attached to the stick. Regarding how to move the handle, the angle and reflection direction of the virtual mirror in the direction are adjusted, and once the target is determined, the calculation is automatic. The angle is calculated based on the position of the person holding the stick and data from when a straight line is formed. Alternatively, buttons or markers can be set, or the angle can be fixed to activate the interface of this invention.

[0219] (Feature 23) In the AR space displayed on the AR glasses 2, there is also a mode that allows you to display or not display the survey results near 6. In addition, you can switch whether to display the results automatically or by setting the settings based on the information at the front of the stick.

[0220] (Feature 24) The speed of the stick can be adjusted within the AR space displayed on AR glasses 2. The stick can be moved not only by the operator but also at a speed set by the system or the copied stick itself. Such adjustments can be made, including numerical adjustments, such as setting it to move slowly, derailing, adjusting the stick or line, and, if possible, changing the property information of the object being pierced (in real-world scenarios such as a construction site, or in virtual scenarios such as VR spaces).

[0221] (Feature 25) In the AR space displayed by AR glasses 2, since the virtual rod can penetrate to any location, the depth inside the wall can also be automatically adjusted. By using information such as the back or front of the wall as the target wall, it is also possible to observe the center of the panel.

[0222] (Feature 26) In the AR space displayed on AR glasses 2, the back view can also be displayed if the virtual stick penetrates the back. In addition, information such as the television and the wiring on the back can be recorded and displayed or explained.

[0223] (Feature 27) In the AR space displayed on AR glasses 2, the user can use their finger (which can be set to any finger or a specific part of a specific finger) to slide and move back and forth on a virtual plane, such as a flat surface, similar to the movement of a mouse wheel. The maximum speed of the finger movement can be observed, and the extension speed of the stick can be set accordingly. That is, the extension speed is the set speed. A fine mode can also be switched. For example, conversely, when the finger movement speed slows down, the display at that position will be magnified, making fine movements possible. The magnification can be adjusted according to the speed, and high-speed extension can also be achieved.

[0224] (Feature 28) In the AR space displayed by AR glasses 2, mirrors or shadows can be projected through a projector. Through 3D presentation, the operator can clearly see the location where the reflection occurs.

[0225] (Feature 29) In the AR space displayed by AR glasses 2, when the virtual stick is extended, the operation is improved by using a graphical interface that displays the front end and is visible from a rearward perspective.

[0226] (Feature 30) In the AR space displayed by AR glasses 2, by presenting an overhead view of the virtual mirror, the person and the target point, the target point is adjusted and automatically calculated based on the hand position to display guidance information.

[0227] (Feature 31) In the AR space displayed by AR glasses 2, a virtual surface is generated and a virtual rod is launched as if stabbing the virtual surface, or the virtual rod is extended and launched if a virtual surface exists.

[0228] (Feature 32) In the AR space displayed by AR glasses 2, the target pointed to by the virtual stick is recognized as a three-dimensional object by reading the distance through a group of cameras arranged in front, above, behind, and to the left and right sides. Sometimes a depth camera is also used. This information saves current and past time data. If there is no current information, past information can of course be used. As long as there is spatial information, the stick will continue to extend inward. A camera is located at the front of the virtual stick. The virtual stick appears as a ghost-like stick. Part of it is penetrating, and its front can also appear as a real-world form (attribute). To allow the operator to understand the length, it should be displayed through a meter or numbers. The extension speed can also be adjusted. In addition, distance, speed, acceleration, etc. can be set by touching the meters for distance, speed, etc.

[0229] (Feature 33) In the AR space displayed by AR Glasses 2, the view behind the virtual stick (penetrating stick) can be seen. The view in front of the stick can also be seen. The viewpoint of the camera at the point in front of the stick can be seen. It can also be used for maintenance. Furthermore, if VR space is used instead of AR space, it is easier to perform operations such as damage, cutting, and repair because VR space contains all the information. The time when this attribute information is saved will also be displayed. However, this function is not needed in real-world scenarios.

[0230] (Feature 34) In the AR space displayed by AR glasses 2, at appropriate times (set time intervals), the set location or after switching intrusive locations such as walls or thick walls will automatically trigger the shutter, simultaneously and automatically collecting data, time, and location as attribute information. In real-world scenarios, if information is collected by penetrating objects using methods such as ultrasound, time and location will also be collected as attribute information.

[0231] (Feature 35) In the AR space displayed by AR glasses 2, a camera resembling a lantern is provided at the front end of the virtual rod. The system has virtual viewpoints set for reflection on virtual walls (such as mirrors, which can be set infinitely) or for determining bending at a certain length or angle (in the case of the front end, a virtual viewpoint is set at the middle position of the rod, for example, a drone will be controlled to move to that viewpoint position).

[0232] (Feature 36) In the AR space displayed by AR glasses 2, if controlled by a person, the maximum number of mirrors is approximately 2; if this number is exceeded, AI or other recommendation computers are required. The system can ask what to operate on and what operation to perform.

[0233] (Feature 37) In the AR space displayed by AR glasses 2, a large number of virtual mirrors can be set, which move around the viewpoint. If the user moves, the mirrors will also move horizontally. Lava can also be expanded at any position, and the distance and direction of expansion can be set. Hands can also be replicated, enabling multi-handed operation. Hands can fly out laterally or through virtual sticks, and their sensitivity can be adjusted. An acceleration function is also available. However, the above adjustments can be omitted, and speeds can be set for virtual sticks, tables, tracks, etc. Such tracks can also be created. If VR space is used instead of AR space, rods of length can be used to draw in the space in VR space. Speed ​​can also be attached to the track as attribute information. Rods are connected to each other, and speeds are set for them. A certain interval can be set to that speed. They can also be removed from the line. In addition, the motion will depend on the attribute information of the space. Attribute information can also be written using rods in a specific space. Rules can be embedded for each track.

[0234] (Feature 38) In the AR space displayed by AR glasses 2, objects that cannot be pressed directly in a straight line can be made into right-angled rods or bent multiple times. This can avoid straight-line pressing in certain situations.

[0235] (Feature 39) A technology for displaying the back side of a device within the AR space displayed on AR glasses 2. For example, it can display the back of a wall or display screen that a virtual rod is in contact with. Furthermore, due to operational deviations, the back side can be displayed by moving along a guide line from the front or other locations. The back side can be displayed either overlaid on the front or below the point the virtual rod is pointing to, while simultaneously moving the pointing point to the side and projecting (tracking) it to that position. It also possesses a technology for displaying back side information for non-destructive testing on the front. For example, it can display information about the interior of a wall, concrete, or the back side, thereby displaying the location to be cut and its back markings on the front for cutting purposes. This ensures accuracy in human surgical projection applications. Furthermore, it can also present a non-pure back view by changing the display format or angle while displaying the back side information. Display methods include mirror displays and displays similar to a reverse mirror.

[0236] (Feature 40) In the AR space displayed by AR glasses 2, for example, a virtual stick can be used to penetrate a pipe to connect to and manipulate objects or a camera (which can move autonomously) inside the pipe. To perform this operation, the absolute position of the camera itself needs to be determined (through the camera's own GPS, ultrasound, non-destructive testing, etc.). Once the virtual stick is connected to the camera, not only can the camera be operated, but it can also be rotated. Of course, internal images, or images including those of the camera, can also be displayed via a projector.

[0237] (Feature 41) In the AR space displayed on AR glasses 2, it has the technology to display targets penetrated by the virtual rod, and the technology to present the back view on the front. It can be a real scene, or it can move and display previously saved data (historical data) according to the position of the virtual rod. (For example, in demolition operations, since the back can be seen, work efficiency can be improved.) This depends on its position and attribute information.

[0238] (Feature 42) In the AR space displayed by AR glasses 2, the surface that the virtual stick touches will appear as a tunnel, allowing for operations behind the tunnel. For example, one can pick up the handset of a communication device from inside the bathroom. The remote control here can be used to operate the television in the next room.

[0239] (Feature 43) In the AR space displayed by AR glasses 2, scenes such as swimming pools, rubble, and toilets will be displayed to deal with emergencies. Taking the swimming pool as an example, cameras or items such as lifebuoys can be placed in advance in that location.

[0240] (Feature 44) In the AR space displayed by AR Glasses 2, the virtual stick, like an X-ray, treats objects inaccessible to humans, such as air or fog, as non-existent, while allowing touch on specific planes for objects that can be identified. By recognizing 3D objects and making specific faces touchable, page turning can be performed like the page-turning function in a medical image viewer, or a mixed display effect like MPR cross-section and 3D volume data display. It can also be designed to be browsed in a photo album style.

[0241] (Feature 45) In the AR space displayed by AR glasses 2, the portion penetrated by the virtual rod can be visualized like radar. Figure 1 The sample can be displayed unfolded, shown from the side, or as a side view of the cylinder, or as an image after unfolding, or observed from the outside of the cylinder. To further enhance realism, de-core data can be used, or even invisible lines can be forcibly displayed using tables or guide lines while the virtual rod is running nearby, displaying precise positions on a projector, etc., thus enabling the specification of previously unspecified back sides. In this case, the penetrating property is utilized.

[0242] (Feature 46) In the AR space displayed by AR glasses 2, the virtual stick may produce errors due to hand movements during operation. Even if high-frequency movements are ignored, errors will still occur, so virtual guide lines need to be added. For example, if VR space is used instead of AR space, a large table can be set up in VR space, and when the light from the virtual stick shines on the vicinity, it will make corrections at that point.

[0243] (Feature 47) In the AR space displayed by AR glasses 2, the guide table or the like can be set at a distant position by actually extending the virtual stick (attaching the guide table or the like to the front end of the virtual stick and launching it).

[0244] (Feature 48) In the AR space displayed by AR glasses 2, by actually specifying an object that is recognized upon being touched by the virtual stick, the information of that object is deleted. After deletion, the object disappears, forming an empty space. Alternatively, information superimposed with penetration information can be used. Thus, objects will disappear sequentially.

[0245] (Feature 49) In the AR space displayed by AR glasses 2, a mode can be set to make paper or cloth turn slightly like a book being blown by the wind.

[0246] (Feature 50) In the AR space displayed on AR glasses 2, luminous curved light can be displayed on the back of the physical switch display visible on the back, accompanied by sound effects. When writing text, it can be read aloud and audio / music can be played.

[0247] (Feature 51) In the AR space displayed by AR glasses 2, the transparent tube can move through the air, and the camera rotates around the center line of the tube (the size of the tube or the viewpoint can also be virtually changed), or it can be rotated around the center of the viewpoint. Rotation is achieved wirelessly, equipped with a guide line for rotation, which can be attracted by a magnet or run along a guide rail by a device similar to a guide wheel. This indicates that the camera is located below. Since video recording consumes a lot of power, a small battery is used for flash photography, etc. As an extension of the above functions, large devices can be assembled after docking, then put into use, becoming similar to an endoscope, allowing for operations such as resection, and then disassembled back to their original state.

[0248] (Feature 52) Within the AR space displayed by AR glasses 2, a tool exists that can observe the portion passing through a cylindrical object while collecting information, displaying the contents inside the cylinder. When a small camera moves or rotates the cylinder, the content can be temporarily extracted and manipulated as usual using a mouse or VR lens. The device collects information at regular intervals, and the time can be set to allow for a more intuitive presentation of changes within the material. While a certain level of information is required, it can also reference conditions such as degradation caused by actual time.

[0249] (Feature 53) In the AR space displayed by AR glasses 2, for example, if there is a button that you want to touch in the back seat of a car, you can press it intuitively if there is a virtual mirror. Seats, etc., can also serve as virtual mirrors. A mechanism similar to a luminous stick, which makes the plane indicating the position and the stick-shaped object passing through it visible, can also be used, using LEDs to make the stick-shaped object visible.

[0250] (Feature 54) In the AR space displayed by AR glasses 2, it also has the function of omitting intermediate processes, and can display the starting part, operation part and the fundamental part simultaneously in the display area.

[0251] (Feature 55) In the AR space displayed by AR glasses 2, the optical fiber can be directed in various directions, and one of them can be lit up to make the whole visible.

[0252] (Feature 56) In the AR space displayed by AR glasses 2, the back of the touched object can be projected like a mirror, or the object can be made to appear as if it is visible (reverse mirror).

[0253] (Feature 57) In the AR space displayed by AR glasses 2, the visible angle can also be changed by the position of the point that passes through it. Both forward and backward directions and rotation can be achieved.

[0254] (Feature 58) In the AR space displayed by AR glasses 2, an “L” shape can be made with the left and right hands, and the vertices can be connected to form a plane according to the size and angle of the surface.

[0255] (Feature 59) In the AR space displayed by AR glasses 2, the use of a thermal sensor may result in more image noise, but the error can be corrected by DL's image recognition technology.

[0256] (Feature 60) The AR space displayed in AR glasses 2 is applicable even in immovable spaces like bathrooms or enclosed rooms. It can also display one's own situation, presenting the location of the person seeking help in an emergency to the outside world. Through movements on a bed (or in the back seat of a car), a virtual mirror is positioned, allowing movement by projecting an interface onto the mirror to bend and reach the target location. Furthermore, the presence of auxiliary devices such as a light stick indicating the direction makes it easier to understand. The location and direction of the target switch or object are indicated.

[0257] (Feature 61) In the AR space displayed by AR glasses 2, a transparent tube (with embedded wires), optical fiber, multiple cameras, built-in camera and camera hole can be used.

[0258] (Feature 62) Within the AR space displayed on AR glasses 2, it can also be used for operations such as opening car doors. For example, a person holding the key can open a door or open a window from either the inside or outside. In this case, the target needs to be aligned with the switch position for operation. It can also realize functions such as closing car doors from a distance.

[0259] (Feature 63) Within the AR space displayed on the AR glasses 2, an operation mode is provided where the operator can move the object centrally or horizontally relative to a fixed mirror surface. In this way, objects can be moved either by rotating to change the angle or by the operator moving horizontally. Furthermore, if the operator or hand is allowed to be copied and launched or moved, the operator can freely move horizontally by moving their copy. Further, by enlarging a designated area or manipulating a defined area within the cylinder, a space can be constructed that allows actual movement inside the cylinder, or the reassembly of disassembled objects into a larger object.

[0260] (Feature 64) In the AR space displayed by AR glasses 2, there is a virtual rod (with length, which (usually a straight line, but can also be curved but would reduce intuitiveness) can be extended, and its contact with a virtual surface, a mirror principle, can project light onto the curved tube to show flow), starting from its operation method, specifying the virtual rod to be operated.

[0261] (Feature 64) A virtual rod (which can extend from one end or both ends) that can be bent (it can be a curved object or flexible and deformable like a line). Furthermore, it can break like a konjac vine when subjected to calculated or estimated forces. It can pass through, attach to, or manipulate movable objects (drones) (which can also be virtual). The front end (the thickness of the front end or the middle section can vary) can be attached, while the middle section acts as a rigid rod. Further development could allow drones or front-end rods attached to drones to be controlled based on their distance from the surrounding environment during movement, avoiding collisions with walls, etc., and performing autonomous assisted calculations (in the real world). There are also virtual boards (attached to the front end of the virtual rod and moved; they can move buildings, tables, etc., serving as guide tools). If the virtual rod is generated from the hand, fine control becomes increasingly difficult with greater distance. It's also possible to magnify the area near the front end. Hand operation alone has limitations, so precise settings can be made near the target using guide tools similar to fine tables. Furthermore, the display can be magnified at that point. This allows for more than just hand movements; an image replicating the hand's actions can be placed, and control can be regained when these images overlap. Virtual guide lines (rails: these can be made of metal or plastic and can be set to move faster within certain zones) and virtual tables are also included. The system comprises: a VR headset, hand cameras, a front-end camera (virtual), a projector (movable), a camera (used to display the back of objects, which is invisible in the real world; it also provides images to rescuers), a camera (movable) to capture images of the back, a CPU to integrate this information, storage devices to store touch point attribute information and action information (response information), and a monitor or PC. The virtual stick can also display a GUI for mid-course shortening. Penetration and overlap are possible, and actions can be changed based on the attributes of the contact or penetration location and action and reaction records. In virtual space, this depends on the action of the contacting object; in the real world, actions are presented through 3D modeling, etc. It can be set that the wall in front can be penetrated, while the wall below can only be touched, etc. It can also change its actions based on information about the object being touched. It can determine the position of the tip of the stick. It can also capture the condition of the tip like a camera or eye and display it on a monitor or projector via picture-in-picture.

[0262] (Feature 65) In the AR space displayed on AR glasses 2, as a result of the pre-visible area analyzed (image analysis), the location to be pointed to is marked as a candidate point. It will display candidate operations to be performed next in the current situation, as well as candidate points, lines, or actions to be pointed to. On occasions such as anniversaries, it will also suggest expressions of gratitude.

[0263] (Feature 66) Within the AR space displayed on AR glasses 2, operation history, action results, action reproduction, and trajectories can be displayed on the screen for the user to select. Each operation history or action can be edited, similar to editing a PowerPoint animation. Furthermore, a simulated demonstration effect can be achieved by copying a hand, and adjustments can be made.

[0264] (Feature 67) In the AR space displayed by AR glasses 2, the content to be displayed as candidates, whether based on prediction or operation history, will be presented as a plane, object, or volume depending on the side and surface touched. For example, for an LED ceiling light, the luminous object is the switch; a certain plane in the center is the switch for changing modes; when the front end enters the interior of that surface, the internal structure is displayed. In addition, if the virtual stick is moved further, the second floor, etc., will be displayed. For example, touching a switch on a remote control can simulate the operation of that remote control. That is to say, touching an object can also simulate the function corresponding to the touched part.

[0265] The following is a summary of the structure and function of the first embodiment of the present invention: The user interface system 1 has an imaging device (camera 4) for capturing images of a user (patient 14); an air operation detection device (circuit and software related to air operation detection in server 6) for detecting operations performed by the user in the air based on the images captured by the imaging device; and a display device (AR glasses 2) for displaying an image with altered content in augmented reality space based on the operations detected by the air operation detection device, wherein the display device uses at least a portion of the image of a virtual stick 91 extended from the user's hand as the displayed image (see [link]). Figure 4 ).

[0266] Furthermore, the user interface system 1 can adjust the extension length or speed of the virtual stick 91 based on the hand operation detected by the air operation detection device (see [link]). Figure 6 (a)).

[0267] In addition, the user interface system 1 further includes a device control device (circuit and software related to device control in the server 6), which controls the specified devices (various devices 7) according to the operation detected by the over-the-air operation detection device.

[0268] In addition, the user interface system 1 has an imaging device (camera 4) for capturing images of the user (patient 14); an air operation detection device (circuit and software related to air operation detection in server 6) for detecting operations performed by the user in the air based on the user images captured by the imaging device; and a display device (AR glasses 2) for displaying modified images in augmented reality space based on the operations detected by the air operation detection device. The display device uses the image of a virtual stick 91 extending from the user's hand as at least part of the displayed image, sets multiple lines in the augmented reality space or the virtual reality space, and corrects the image so that the tip of the virtual stick 91 is aligned with the multiple lines, thereby correcting hand tremors at the tip of the virtual stick 91 (see Figure 8(a)).

[0269] Furthermore, the user interface system 1 is characterized by: having an imaging device (camera 4) for capturing images of a user (patient 14); an air operation detection device (circuit and software related to air operation detection in server 6) for detecting operations performed by the user in the air based on the images captured by the imaging device; and a display device (AR glasses 2) for displaying an image with altered content in an augmented reality space based on the operations detected by the air operation detection device, wherein the display device uses an image of a virtual stick 91 extending from the user's hand as at least a part of the displayed image, and displays the interior or back of the real object or the virtual object by stabbing the front end of the virtual stick against a real object in the augmented reality space or a virtual object in the virtual reality space (see Figures 7(b) and 9).

[0270] According to this first embodiment of the invention, by using a virtual stick 91 extending from the user's hand, the range and diversity of operation and display can be expanded beyond the conventional reach of an outstretched hand, enabling more realistic and intuitive operations that take into account distance, and leveraging the advantages of a real-world user interface. Therefore, according to the first embodiment, user convenience can be improved.

[0271] Furthermore, in addition to the methods described above, various other methods can be employed as the method for displaying the virtual stick shown in the first embodiment of the present invention. For example, in Figure 5 (a) shows the start of the virtual stick display. The length and shape of the virtual stick can be set, or a display method such as generating the virtual stick through conventional operations and holding it can be used.

[0272] In addition, the flexibility (softness) of the virtual rod can be set. For example, it can be set to be a soft line, a soft and twisted state, etc.

[0273] In addition, Figure 8In the ninth display method shown in (b), the operation of extending the light virtual rod 171 by inserting it into the virtual object 172 (see...) Figure 6 (a) Move the virtual object 172 to a distance. Considering situations such as moving the object by inserting it, the softness of the object being inserted can be set, the sharpness of the virtual rod can be set, the piercing part can be set, and the person holding the virtual rod (patient 14) can see the piercing part.

[0274] In addition, as an application of the virtual mirror shown in (Feature 22), there is a method in which the displayed content or the object mapped by the virtual mirror is then touched by another virtual stick to make it move.

[0275] Furthermore, in the storage unit 62 of the server 6 shown in Figure 2, operator information such as the virtual stick operator's birthday can also be set.

[0276] also, Figure 6 (b) The spatial display image 112 shows the attribute information of the front position of the virtual rod 111. Figure 6 The attribute information inside the pipe 122 shown in (c), the attribute information shown in (feature 12), (feature 18), (feature 24), etc. (can be an object or space, in which case air and vacuum are basically regarded as objects or spaces called "vacuum"), can be arbitrarily set in the space, such as the metaverse space.

[0277] <Second Embodiment of the Invention>

[0278] The following describes the user interface system according to the second embodiment of the present invention.

[0279] In the user interface system of the second embodiment, a VR head-mounted display is used instead of AR glasses. Figure 4 (b) The ceiling 18 of the AR glasses 2 in the AR operating space 71 shown (see) Figure 1 The images include images of the hands of the injured patient 14 and the VR head-mounted display, which show the VR background and the VR hands of the injured patient 14.

[0280] The following is a summary of the structure and function of the second embodiment of the present invention. The user interface system 1 includes: an imaging device (camera 4) for capturing images of a user (patient 14); an air operation detection device (circuit and software related to air operation detection in server 6) for detecting operations performed by the user in the air based on the images captured by the imaging device; and a display device (VR head-mounted display) for displaying altered images in a virtual reality space based on the operations detected by the air operation detection device. The display device uses an image of a virtual stick 91 extending from the user's hand (see [link to VR head-mounted display]) as at least part of the displayed image. Figure 4 ).

[0281] Furthermore, the user interface system 1 can adjust the extension length or speed of the virtual stick 91 based on the hand operation detected by the air operation detection device (see [link]). Figure 6 (a)).

[0282] In addition, the user interface system 1 further includes a device control device (circuit and software related to device control in the server 6), which controls the predetermined devices (various devices 7) according to the operation detected by the air operation detection device.

[0283] Furthermore, the user interface system 1 includes: an imaging device (camera 4) for capturing images of a user (patient 14); an air operation detection device (circuit and software related to air operation detection in server 6) for detecting operations performed by the user in the air based on the images captured by the imaging device; and a display device (VR head-mounted display) for displaying modified images in a virtual reality space based on the operations detected by the air operation detection device. The display device includes an image of a virtual stick 91 extending from the user's hand in at least a portion of the displayed image. Multiple lines are set in the virtual reality space, and the image is corrected to align the tip of the virtual stick 91 with the multiple lines, thereby correcting hand tremors at the tip of the virtual stick 91 (see [link]). Figure 8 (a)).

[0284] Furthermore, the user interface system 1 is characterized by: having an imaging device (camera 4) for capturing images of a user (patient 14); an air operation detection device (circuit and software related to air operation detection in server 6) for detecting operations performed by the user in the air based on the images captured by the imaging device; and a display device (VR head-mounted display) for displaying images with altered content in a virtual reality space based on the operations detected by the air operation detection device, wherein the display device uses an image of a virtual stick 91 extending from the user's hand as at least a part of the displayed image, and displays the interior or back of the real object or the virtual object by stabbing the front end of the virtual stick into a real object or a virtual object in the virtual reality space (see Figures 7(b) and 9).

[0285] According to this second embodiment of the invention, by extending the virtual stick 91 from the user's hand, the range and diversity of operation and display can be expanded more widely than the conventional reachable range, enabling more realistic and intuitive operation that takes distance into account, and leveraging the advantages of a real-world user interface. Therefore, according to the first embodiment, user convenience can be improved.

[0286] Furthermore, the scope of this invention is not limited to VR and AR, but can also cover the real world and XR spaces such as MR.

[0287] Therefore, the systems, apparatus, methods, etc. of the present invention can be modified in various ways without changing the spirit of the present invention.

[0288] For example, the imaging device (camera 4) used to capture images of the user (patient 14) is not limited to being mounted on the ceiling; it can also be mounted on a table, etc., offering a variety of applications. Furthermore, during text input, when selecting from a list of hiragana characters, the next candidate words are arranged in a straight line, such as vertically, horizontally, or diagonally, facilitating selection and aiding input. When a circle or similar shape is recognized, the candidate words can also be displayed as a vertical or horizontal list. This system is not only suitable for patient 14 but also for ordinary people, workers, educators, civil servants, and other diverse groups.

[0289] For example, it is possible to merge two or more systems into one, and conversely, it is also possible to make one system consist of two or more different systems and connect them together.

[0290] Furthermore, the first and second embodiments described above are merely the best or near-best embodiments to date.

[0291] The user interface system of this invention can be effectively used in enterprises or legal entities operating medical facilities or accommodation facilities, as well as in private residences.

[0292] Symbol Explanation

[0293] 1: User interface system; 2: AR glasses; 3: Projector; 4: Camera; 5: Wireless LAN; 6: Server; 11: Hospital; 12: Ward; 13: Hospital bed; 14: Injured or sick patients; 15: Curtain; 16: Wall; 17: Management Office; 61: Control Department; 62: Storage Department; 63: Interface; 64: Ministry of Communications; 65: Keyboard; 66: Mouse; 67: Display device; 91: Virtual stick.

Claims

1. A user interface system, characterized in that: It has an imaging device for capturing images of the user; An air operation detection device that detects operations performed by the user in the air based on images captured by the imaging device. Based on the operation detected by the air operation detection device, the image with altered content is displayed on a display device in augmented reality or virtual reality space. The display device, as at least part of the displayed image, uses an image of a virtual stick extending from the user's hand.

2. The user interface system as described in claim 1, characterized in that: The extension length or speed of the virtual rod can be adjusted based on the hand operation detected by the air operation detection device.

3. The user interface system as described in claim 1 or 2, characterized in that: It also has an equipment control device that controls the predetermined equipment based on the operation detected by the air operation detection device.

Images