Information processing system and program

Abstract

To avoid collision with a surrounding object without stopping display of a virtual object.SOLUTION: An information processing system includes: a device which allows a user to visually recognize visual information in which a virtual object is superposed on information of a real space; an acquisition unit for acquiring a position of an object present in the real space; and a processor which moves the virtual object in a direction leaving from the object when the object is located in a prescribed range from the device.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an information processing system and a program. [Background technology]

[0002] Devices that allow users to view visual information in which virtual objects are superimposed on information from the real world include devices called AR (Augmented Reality) glasses and MR (Mixed Reality) glasses. In the following, these will be collectively referred to as "xR devices." Today, there are xR devices that can display virtual objects with high brightness and resolution. As the brightness and resolution of virtual objects increase, the sense of immersion for users wearing xR devices also increases. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-046366 Summary of the Invention [Problem to be solved by the invention]

[0004] As the sense of immersion increases, collisions with surrounding objects become a concern. For example, when touching or tapping a virtual object, there is a possibility that it may collide with and knock over an invisible drink. Also, since people around you cannot see the position of the virtual object, there is a possibility that you may collide with someone passing behind the virtual object. It is also possible to provide time to avoid a collision with a surrounding object by stopping the display of the virtual object when the possibility of a collision with a surrounding object arises, but this would result in an interruption to the operation.

[0005] An object of the present invention is to realize avoidance of collisions with surrounding objects without stopping the display of a virtual object. [Means for solving the problem]

[0006] The invention described in claim 1 includes a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space, an acquisition unit that acquires the position of an object existing in the real space, and a processor, wherein when the object is located within a predetermined range from the device, the processor moves the virtual object in a direction away from the object, The range is set through measurement of gesture operations related to registration of a range of motion, and the range is corrected according to the measurement result by the acquisition unit; The range is an information processing system that defines an area that the user uses to perform a gesture operation on the virtual object. The invention described in claim 2 is a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space; an acquisition unit that acquires the position of an object existing in the real space; and a processor, wherein when the object is located within a predetermined range from the device, the processor moves the virtual object in a direction away from the object, sets the range through measurement of gesture operations related to registration of a range of motion, and modifies the range according to a body part that is frequently used for gesture operations, and the range defines an area that the user uses for gesture operations on the virtual object; It is an information processing system. The invention described in claim 3 is a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space; an acquisition unit that acquires the position of an object existing in the real space; and a processor, wherein when the object is located within a predetermined range from the device, the processor moves the virtual object in a direction away from the object, and sets the range through measurement of a gesture operation related to registration of a range of motion, and widens the range corresponding to the user's line of sight, and the range defines an area used by the user for gesture operations on the virtual object; It is an information processing system. The invention described in claim 4 is a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space; an acquisition unit that acquires the position of an object existing in the real space; and a processor, wherein when the object is located within a predetermined range from the device, the processor moves the virtual object in a direction away from the object and sets the range through measurement of a gesture operation related to registration of a range of motion; and when an icon is placed as the virtual object, the processor widens the range for the direction of the icon, and the range defines an area used by the user for gesture operations on the virtual object; It is an information processing system. Claim 5 The invention described in the item (1) has a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space, and an acquisition unit that acquires the position of an object existing in the real space. Ko A function of moving the virtual object in a direction away from the object when the object is located within a predetermined range from the device. a function of setting the range through measurement of gesture operations related to registration of the range of motion; and a function of correcting the range according to the measurement result by the acquisition unit. The range is a program that defines an area that the user uses to perform a gesture operation on the virtual object. The invention described in claim 6 is a program for realizing, in a computer having a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space, and an acquisition unit that acquires the position of an object existing in the real space, a function of moving the virtual object in a direction away from the object when the object is located within a predetermined range from the device, a function of setting the range through measurement of gesture operations related to registration of a range of motion, and a function of modifying the range according to parts of the body that are frequently used for gesture operations, wherein the range specifies an area that the user will use for gesture operations on the virtual object. The invention described in claim 7 is a program for causing a computer having a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space, and an acquisition unit that acquires the position of an object existing in the real space, to realize the following functions in a computer: a function of moving the virtual object in a direction away from the object when the object is located within a predetermined range from the device; a function of setting the range through measurement of gesture operations related to registration of a range of motion; and a function of expanding the range corresponding to the user's line of sight, wherein the range specifies an area used by the user for gesture operations on the virtual object. The invention described in claim 8 is a program for realizing, in a computer having a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space, and an acquisition unit that acquires the position of an object existing in the real space, a function of moving the virtual object in a direction away from the object when the object is located within a predetermined range from the device, a function of setting the range through measurement of gesture operations related to registration of a range of motion, and a function of expanding the range in the direction of the icon when an icon is placed as the virtual object, wherein the range specifies an area used by the user for gesture operations on the virtual object. [Effects of the Invention]

[0007] Claim 1 According to the described invention, the registered ranges can be modified depending on the operational characteristics. Claim 2 According to the described invention, the registered range can be modified according to the tendency of frequency of use. Claim 3 The described invention allows for an expanded range of directions in which contact with an object may occur. Claim 4 The described invention allows for an expanded range of directions in which contact with an object may occur. According to the invention of claim 5, the registered range can be corrected according to operational characteristics. According to the invention of claim 6, the registered range can be corrected according to the tendency of frequency of use. According to the invention of claim 7, the range of directions in which contact with an object may occur can be expanded. According to the invention of claim 8, the range of directions in which contact with an object may occur can be expanded. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating an example of use of the information processing system assumed in the first embodiment. [Figure 2] FIG. 1 is a diagram illustrating an example of the appearance of an xR device when viewed from the front side. [Figure 3] FIG. 1 is a diagram illustrating an example of the hardware configuration of an xR device. [Figure 4] 1A and 1B are diagrams illustrating the principle by which a virtual screen is visually recognized by a user. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information terminal. [Figure 6] 1A and 1B are diagrams illustrating an example of setting a first range used for gesture operation and an example of setting a first distance to be secured between an object located within the first range and a virtual screen. (A) is a diagram of the user viewed from above, and (B) is a diagram of the user viewed from the side. [Figure 7] 10A and 10B are diagrams illustrating other examples of setting the first range used for gesture operations, where (A) shows an example of setting the first range when the user is lying on his / her back, and (B) shows an example of setting the first range when the user is lying on his / her side. [Figure 8] 10A and 10B are diagrams illustrating an example of setting a second range that is not used for gesture operations, in which (A) is a diagram of the user seen from above, and (B) is a diagram of the user seen from the side. [Figure 9] FIG. 10 is a diagram illustrating an example of a basic parameter setting screen. [Figure 10] FIG. 10 is a diagram illustrating an example of a setting screen used to set each threshold value. [Figure 11] 1A and 1B are diagrams illustrating the setting of a first range by a gesture. FIG. 1A is a diagram illustrating the setting of the first range as seen from behind the user. FIG. 1B is a diagram illustrating the setting of the first range as seen from the side of the user U. [Figure 12] 10A and 10B are diagrams illustrating examples of setting the first range. (A) is a basic example of the first range, (B) is an example in which the range on the dominant hand side is set wider than the other hand, (C) is an example in which the first range is expanded by a margin, and (D) is an example in which the range on the non-dominant hand side is narrowed. [Figure 13] 10 is a flowchart illustrating an example of a basic parameter setting operation. [Figure 14] 10 is a flowchart illustrating an example of a virtual screen saving operation in the first embodiment. [Figure 15] 10A and 10B are diagrams illustrating an example of retracting the display position of the virtual screen from an object toward the user. (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen and the user after retraction. [Figure 16] 10A and 10B are diagrams illustrating an example of retraction when an object is placed behind the virtual screen afterward. (A1) and (A2) show the positional relationship between the virtual screen and the user at its initial position, (B1) and (B2) show a state in which a drink as an object has been placed on a table, and (C1) and (C2) show the positional relationship between the virtual screen and the user after retraction. [Figure 17] 10A and 10B are diagrams illustrating an example of evacuation when a candidate evacuation position is closer to the user than a second distance, where (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen and the user after evacuation. [Figure 18]10A and 10B are diagrams illustrating an example of retraction when multiple virtual screens are arranged in the depth direction, where (A1) and (A2) show the positional relationship between the virtual screens and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screens and the user U after retraction. [Figure 19] 10A and 10B are diagrams illustrating an example of retracting the virtual screen upward, in which (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen and the user after retraction. [Figure 20] 10A and 10B are diagrams illustrating an example of horizontally retracting a virtual screen, where (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen and the user after retraction. [Figure 21] 10A and 10B are diagrams illustrating an example of a virtual screen retraction when multiple drinks are placed on a table in front of a user. (A1) and (A2) show the positional relationship between the virtual screen, the user, and the drinks at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen, the user, and the drinks after retraction. [Figure 22] 10A and 10B are diagrams illustrating an example in which the display position of the virtual screen is retracted toward the back of an object as viewed from the user. (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen and the user after retraction. [Figure 23] 1 is a diagram illustrating an example of retraction when a virtual screen overlaps with a surrounding object in space. (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, (B1) and (B2) show the positional relationship between the virtual screen and the user immediately after the user's face direction changes, and (C1) and (C2) show the positional relationship between the virtual screen and the user after retraction. [Figure 24] 10 is a flowchart illustrating an example of a virtual screen saving operation in the second embodiment. [Figure 25]10A and 10B are diagrams illustrating the processing operation when the virtual screen does not include icons for operation. (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen and the user when the virtual screen does not include icons for operation. [Figure 26] 13 is a flowchart illustrating an example of a virtual screen saving operation in the third embodiment. [Figure 27] 1A and 1B are diagrams illustrating an example of evacuation when a person is approaching the user. (A) shows a state in which the person is moving toward the user outside the first range, and (B) shows a state in which the person has entered the first range. [Figure 28] 1A and 1B are diagrams illustrating examples of retraction when two users are viewing virtual screens, where (A) shows a state in which the first range of a user and a part of the first range of a user overlap each other but the gesture operation of each user is not recognized as a movable object, and (B) shows a state in which the gesture operation of a user is recognized as a movable object within the first range of the user. [Figure 29] 1A and 1B are diagrams illustrating an example of evacuation when a movement schedule for a mobile robot can be acquired, where (A) shows a state in which the robot is outside a first range, and (B) shows a state in which the robot approaches the first range. [Figure 30] This figure explains exceptions to the evacuation rule. (A) shows the display position of the virtual screen when the window curtains are hanging down, and (B) shows the display position of the virtual screen when the window curtains are blown by the wind and spread close to the virtual screen. [Figure 31] 13 is a flowchart illustrating an example of a virtual screen saving operation in the fourth embodiment. [Figure 32] 10A and 10B are diagrams illustrating the retraction operation when two virtual screens are arranged in the rear direction as seen from the user. (A1) and (A2) show the positional relationship between the two virtual screens and the user in their initial positions, and (B1) and (B2) show the state in which only the front virtual screen has been retracted. [Figure 33] 13 is a flowchart illustrating an example of a virtual screen saving operation in the fifth embodiment. [Figure 34] An example of a cursor used to operate the virtual screen is shown below. (A) shows the cursor when the fingertips of the right hand are far from the virtual screen, (B) shows the cursor when the fingertips of the right hand are close to the virtual screen, and (C) shows the cursor when the fingertips of the right hand are in contact with the virtual screen. [Figure 35] 10A and 10B are diagrams illustrating an example in which the virtual screen located at the farthest of the two virtual screens is retracted in a direction away from the object toward the user. (A1) and (A2) show the positional relationship between the two virtual screens and the user at their initial positions, and (B1) and (B2) show the positional relationship between the two virtual screens and the user after the retraction process has been executed. [Figure 36] 20 is a flowchart illustrating an example of a virtual screen saving operation in the sixth embodiment. [Figure 37] 10A and 10B are diagrams illustrating an example of saving when a user instructs to add a third virtual screen while two virtual screens are displayed. (A1) and (A2) show the positional relationship between the two virtual screens and the user at their initial positions, and (B1) and (B2) show the positional relationship between the three virtual screens and the user after the saving process has been executed. [Figure 38] 13 is a flowchart illustrating an example of a virtual screen saving operation in the seventh embodiment. [Figure 39] 1 is a diagram illustrating a case where one virtual screen is shared by multiple users, where (A) shows the state before two users perform gesture operations, and (B) shows the state after the two users start performing gesture operations. [Figure 40] 1A and 1B are diagrams illustrating an example of a method for notifying a user viewing a virtual screen of the presence of an object, where (A) shows an example of notification by sound, and (B) shows an example of notification by synthetic image. [Figure 41] 10A and 10B are diagrams illustrating an example of another method for notifying a user viewing a virtual screen of the presence of an object. (A1) and (A2) show a method using deformation of the virtual screen, and (B1) and (B2) show a method using a composite image to draw attention to the virtual screen. [Figure 42]10A and 10B are diagrams illustrating adjustment of a threshold value that defines a first range based on a history of measurements of the range of motion of the right hand during gesture operations on a virtual screen. (A) shows the first range at the initial setting, and (B) shows the first range adjusted according to the range of motion of the user's gesture operations. [Figure 43] 13A and 13B are diagrams illustrating an example of retracting a virtual screen in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen and the user after retraction. [Figure 44] 13A and 13B are diagrams illustrating another example of retraction of the virtual screen in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen and the user after retraction. [Figure 45] 13A and 13B are diagrams illustrating another example of retraction of the virtual screen in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen and the user after retraction. [Figure 46] 13A and 13B are diagrams illustrating another example of retraction of the virtual screen in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen and the user at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen and the user after retraction. [Figure 47] 13A and 13B are diagrams illustrating another example of saving a virtual screen in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen and the user before reduction conversion, and (B1) and (B2) show the positional relationship between the virtual screen and the user after reduction conversion. [Figure 48] 13A and 13B are diagrams illustrating another example of saving a virtual screen in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen and the user before enlargement conversion, and (B1) and (B2) show the positional relationship between the virtual screen and the user after enlargement conversion. [Figure 49]This figure explains the case where the virtual screen viewed by the user is linked to the display screen of an actual monitor. (A) shows the virtual screen and the display screen of the monitor before linking, (B) shows the virtual screen and the display screen of the monitor immediately after linking, and (C) shows the state where the virtual screen and the monitor are arranged side by side after the virtual screen has been evacuated. [Figure 50] 10A and 10B are diagrams illustrating a function for expanding the shape of the first range in the direction of the icon included on the virtual screen. (A1) and (A2) show the first range when the virtual screen does not include an icon, and (B1) and (B2) show the first range when the virtual screen includes an icon. [Figure 51] 1 is a diagram illustrating a function for expanding a first range in relation to the direction of a user's line of sight. (A1) and (A2) show the shape of the first range when the user's line of sight is diagonally forward to the right, and (B1) and (B2) show the shape of the first range when the user's line of sight is diagonally forward to the left. [Figure 52] 13A and 13B are diagrams illustrating an example of the appearance of an xR device used in embodiment 13. (A) shows an example of the appearance of the xR device when observed from the front side, and (B) shows an example of the appearance of the xR device when observed from the user side. [Figure 53] FIG. 22 is a diagram illustrating an example of the hardware configuration of an xR device used in the thirteenth embodiment. [Figure 54] FIG. 2 is a diagram illustrating a configuration of a functional unit related to line-of-sight detection. [Figure 55] 14A and 14B are diagrams illustrating an example of the appearance of an xR device used in embodiment 14. (A) shows an example of the appearance of the xR device when viewed from the front side, (B1) and (B2) show examples of the configuration of earphone tips attached to the temples, and (C1) and (C2) show examples of the configuration of earphone tips attached to the temples. [Figure 56] FIG. 22 is a diagram illustrating an example of the hardware configuration of an xR device used in the fourteenth embodiment. [Figure 57] 10 is a diagram illustrating the configuration of a functional unit that detects a sign of a gesture operation using an electroencephalogram. FIG. [Figure 58]23 is a flowchart illustrating an example of a virtual screen saving operation in the fourteenth embodiment. [Figure 59] 10A and 10B are diagrams illustrating the difference between the saving process when a gesture operation is expected and when it is not expected, where (A1) and (A2) show the positional relationship between the virtual screen and the user when a gesture operation is not expected, and (B1) and (B2) show the positional relationship between the virtual screen and the user when a gesture operation is expected. [Figure 60] 15A and 15B are diagrams illustrating an example of how the information processing system is used in accordance with a fifteenth embodiment, in which (A) shows an example of the appearance of the information system, and (B) illustrates how the information system is worn. [Figure 61] 16A and 16B are diagrams illustrating an example of how the information processing system is used in accordance with the sixteenth embodiment. (A) shows an example of how the headset is worn, and (B) shows an example of the placement of electrodes placed inside the headset for measuring electrical signals corresponding to electroencephalograms. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. <First Embodiment> <Overall system configuration> FIG. 1 is a diagram illustrating an example of use of an information processing system 1 assumed in the first embodiment. The information processing system 1 shown in Fig. 1 is configured with an xR device 10 and an information terminal 20 that are connected to each other. Here, "connected" refers to a state in which communication is possible. The xR device 10 shown in FIG. 1 is a glasses-type device that a user U wears on his or her head.

[0010] The xR device 10 has a mode in which virtual objects are displayed and a mode in which virtual objects are not displayed. When the mode in which virtual objects are not displayed is enabled, the xR device 10 is the same as so-called glasses. In the case of Fig. 1, a mode for displaying virtual objects is active. Therefore, the user U can see a virtual screen (hereinafter referred to as "virtual screen") VD as a virtual object in front of the table 3. In Fig. 1, speech bubbles represent the state of space recognized by the user U wearing the xR device 10.

[0011] Although the user U cannot see it because it is hidden behind the virtual screen VD, there is a drink 5 placed on the real table 3. FIG. 1 shows a state in which a user U taps a virtual screen VD with his or her right hand 7. As mentioned above, modern xR devices 10 are capable of displaying a virtual screen VD with high brightness and high resolution. For this reason, there is a possibility that the user may tap the virtual screen VD without noticing a drink 5 on a table 3, which may result in the drink 5 being knocked over.

[0012] FIG. 1 shows the positional relationship between a right hand 7 and a drink 5 in real space, and the positional relationship between the right hand 7 and a virtual screen VD in the space visually recognized by a user U. In the case of FIG. 1, the virtual screen VD is a two-dimensional image, but it may be a three-dimensional image. Also, in the case of FIG. 1, there is one virtual screen VD, but there may be multiple virtual screens VD. Also, the virtual screen VD may be a still image or a moving image. The virtual screen VD is an example of a virtual object.

[0013] The information terminal 20 used in this embodiment provides, in cooperation with the xR device 10, a function of avoiding collisions with surrounding objects during gesture operations on the virtual screen VD (hereinafter referred to as a "collision avoidance function"). 1 depicts a smartphone as an example of information terminal 20. However, information terminal 20 is not limited to a smartphone and may be a notebook computer, desktop computer, tablet computer, or server on a network. Information terminal 20 may also be a wearable terminal such as a smartwatch. In the case of FIG. 1, one xR device 10 is connected to the information terminal 20, but a plurality of xR devices 10 may be connected. Also, one xR device 10 may be connected to a plurality of information terminals 20.

[0014] The xR device 10 and the information terminal 20 in the present embodiment are an example of an information processing system. In the present embodiment, the operation of the virtual screen VD by the user U is expressed as a "gesture operation". This is because the movement of the user U in the real space who operates the virtual screen VD is recognized as gestures such as body movements and hand gestures. In the present embodiment, it is called a "gesture operation" in terms of the meaning of the gesture used for the operation.

[0015] <Device configuration of each terminal> <Configuration of the xR device> FIG. 2 is a diagram for explaining an external appearance example when observing the xR device 10 from the front side. The spectacle frame is composed of left and right rims 11L and 11R into which lenses are fitted, and a bridge 12. On the left and right of the spectacle frame, temple pieces 13L and 13R are attached via hinges.

[0016] In the case of FIG. 2, a microphone 101 and a speaker 102 are attached near the hinge of the left rim 11L. On the other hand, an inertial sensor 103 and a positioning sensor 104 are attached near the hinge of the right rim 11R. Also, a camera 105 is attached to the bridge 12. In addition, an xR module 106 is attached to the left temple piece 13L, and a control unit composed of a processor 107, a RAM (=Random Access Memory) 108, and a flash memory 109, and a communication module 110 are attached to the right temple piece 13R.

[0017] FIG. 3 is a diagram for explaining an example of the hardware configuration of the xR device 10. The xR device 10 shown in Figure 3 has a processor 107 that constitutes a control unit, to which a microphone 101, a speaker 102, an inertial sensor 103, a positioning sensor 104, a camera 105, an xR module 106, a RAM 108, a flash memory 109, and a communication module 110 are connected. The microphone 101 is a device that converts the voice of the user U and surrounding sounds into electrical signals.

[0018] The speaker 102 is a device that converts an electrical signal into sound and outputs the sound. The speaker 102 may be a bone conduction speaker or a cartilage conduction speaker. The speaker 102 may be a device independent of the xR device 10, such as a wireless earphone. The inertial sensor 103 is configured, for example, by a six-axis sensor. The six-axis sensor is configured by a three-axis acceleration sensor and a three-axis angular velocity sensor. The movement of the head of the user U wearing the xR device 10 is estimated based on the output of the inertial sensor 103.

[0019] The positioning sensor 104 is, for example, a LiDAR (Light Detection and Ranging) module. The LiDAR module emits pulsed laser light and calculates the distance to an object based on the time it takes for the reflected light from the object to be detected. The positioning sensor 104 in this embodiment measures the surroundings of the user U, including the area in front of the xR device 10. A three-dimensional map of the surroundings of the user U is created using the positioning data of the positioning sensor 104.

[0020] The xR device 10 in this embodiment has the capability to measure the range of movement (i.e., the "range of motion") of at least the body parts used by the user U for gesture operations, as well as stationery such as writing implements and pointers, and assistive devices such as canes, prosthetic arms, and prosthetic legs. However, a plurality of positioning sensors 104 may be arranged to measure the distance to an object not only in front of the user U but also to an object behind or to the side of the user U, for example. In the present embodiment, the positioning sensor 104 is attached to the xR device 10, but it may also be attached to a necklace, pendant, brooch, hat, hair accessory, clothing, or the like. The positioning sensor 104 here is an example of an acquisition unit that acquires the position of an object existing in real space.

[0021] The camera 105 uses, for example, a CMOS (= Complementary Metal Oxide Semiconductor) image sensor or a CCD (= Charge Coupled Device) image sensor. By processing the image data captured by the camera 105, it becomes possible to identify the contents of objects located around the user U. In the case of FIG. 2, one camera 105 is attached to the eyeglass frame, but two or more cameras 105 may be attached.

[0022] Taking stereo images of the surroundings of the user U using two cameras 105 makes it possible to estimate the distance to the subject and the front-to-back relationship between the subjects. The two cameras 105 here are an example of an acquisition unit. Furthermore, when measuring distances using two cameras 105, a configuration without providing the positioning sensor 104 is also possible. The xR device 10 may also be configured without the camera 105. The xR device 10 with this configuration is also called smart glasses.

[0023] The xR module 106 is a module that integrates functions to allow the user U to visually recognize an xR image that does not exist in real space, and is composed of optical components and electronic components. Typical examples of the xR module 106 include a system in which a half mirror is placed in front of the user U's eyes, a system in which a volume hologram is placed in front of the user U's eyes, and a system in which a blazed diffraction grating is placed in front of the user U's eyes. In this case, the xR module 106 refers, in a narrow sense, to a device portion that allows a user to visually recognize visual information in which virtual objects are superimposed on information of real space. In a broad sense, the xR module 106 refers to a mechanism that allows a user to visually recognize visual information in which virtual objects are superimposed on information of real space.

[0024] The processor 107 is configured by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The processor 107 realizes various functions by executing programs. Both RAM (Random Access Memory) 108 and flash memory 109 are semiconductor memories. RAM 108 is a main storage device used for executing programs. Flash memory 109 is a non-volatile auxiliary storage device used for recording BIOS (Basic Input Output System), firmware, etc. In this embodiment, the control unit is configured by a processor 107, a RAM 108, and a flash memory 109, but may also be configured by an ASIC (=Application Specific Integrated Circuit) or an FPGA (=Field Programmable Gate Array), etc.

[0025] The communication module 110 is a device that realizes communication with the information terminal 20. In this embodiment, for example, Wifi (registered trademark) or Bluetooth (registered trademark) is used for the communication module 110. In this embodiment, processing with a large calculation load is executed by the information terminal 20, not by the processor 107. 2 and 3, a vibrator or an illuminance sensor may also be provided. A vibrator that generates continuous or intermittent vibrations can be used to notify the user U of an alarm or the like. An illuminance sensor can be placed inside the eyeglass frame or temples 13L, 13R and used to detect the wearing or removal of the xR device 10.

[0026] 4 is a diagram illustrating the principle by which the virtual screen VD is visually recognized by the user U. The xR device 10 shown in FIG. 4 corresponds to a method in which a blazed diffraction grating is placed in front of the user U's eyes. The xR device 10 shown in Figure 4 has a light guide plate 111 equivalent to a lens of glasses, a microdisplay 112, a diffraction grating 113A to which image light L12 is input, and a diffraction grating 113B from which image light L12 is output. The light guide plate 111 has a transmittance of, for example, 85% or more. Therefore, the user U can directly view the scenery ahead through the light guide plate 111. The external light L11 travels straight through the light guide plate 111 and the diffraction grating 113B and enters the eye E of the user U.

[0027] The microdisplay 112 is a small display on which a virtual screen VD is displayed, which is visually recognized by the user U. Light of an image displayed on the microdisplay 112 is projected onto the light guide plate 111 as image light L12. The image light L12 is refracted by the diffraction grating 113A, and reaches the diffraction grating 113B while being reflected inside the light guide plate 111. The diffraction grating 113B refracts the image light L12 in the direction of the eye E of the user U. As a result, external light L11 and image light L12 are simultaneously incident on the eye E of the user U. As a result, the user U visually perceives a virtual screen VD as if it were in front of the table 3, as shown in a balloon in FIG.

[0028] <Configuration of information terminal> FIG. 5 is a diagram illustrating an example of the hardware configuration of the information terminal 20. As shown in FIG. The information terminal 20 shown in Figure 5 has a processor 201, RAM 202, flash memory 203, a display 204, a capacitive touch sensor 205, a camera 206, a microphone 207, a speaker 208, a communication module 209, and a signal line 210 connecting these components.

[0029] The processor 201 is configured by, for example, a CPU or an MPU, and realizes various functions by executing programs. Both the RAM 202 and the flash memory 203 are semiconductor memories. The RAM 202 is used as a main storage device used for executing programs. The flash memory 203 is a non-volatile auxiliary storage device used for recording BIOS, firmware, etc. The processor 201 in this embodiment provides a function for avoiding collision between a surrounding object RO and the user U during a gesture operation on the virtual screen VD through the execution of a program.

[0030] The display 204 may be, for example, an organic EL (Electro Luminescent) display or a liquid crystal display. The capacitive touch sensor 205 is a film-like sensor attached to the surface of the display 204. The capacitive touch sensor 205 is translucent so as not to obstruct the view of the display 204, and detects the position touched by the fingertip or the like of the user U as a change in capacitance. A device in which the display 204 and the capacitive touch sensor 205 are integrated is called a touch panel.

[0031] The camera 206 is attached to the device body of the information terminal 20. If the information terminal 20 is a smartphone, multiple cameras 206 are provided on the same side as the display 204 and on the opposite side. The camera 206 may be, for example, a CMOS image sensor or a CCD image sensor. The microphone 207 is a device that converts sound into an electrical signal. The speaker 208 is a device that converts electronic signals into sound. The communication module 209 is, for example, Wifi (registered trademark) or Bluetooth (registered trademark), and is used for communication with the xR device 10.

[0032] <Basic parameters> Below, using Figures 6 to 12, we will explain the basic parameters that need to be set in advance to realize the function of avoiding or reducing the possibility of collision between the user U and surrounding objects during gesture operations on the virtual screen VD (see Figure 1). 6A and 6B are diagrams illustrating an example of setting a first range RA used for gesture operations and an example of setting a first distance LT1 to be secured between an object RO located within the first range RA and a virtual screen VD (see FIG. 1). (A) is a diagram of a user U viewed from above, and (B) is a diagram of the user U viewed from the side.

[0033] The first range RA used for gesture operations provides the range used by the user U for gesture operations on the virtual screen VD. Specifically, the first range RA provides the maximum range of the positioning range for the movement of body parts used by the user U for gesture operations, as well as stationery such as writing implements and pointers, and assistive devices such as canes, prosthetic arms, and prosthetic legs. However, the first range RA in this embodiment is a convenient range used to determine whether or not to execute the function of retracting the virtual screen VD from the object RO. Therefore, the set first range RA does not necessarily match the maximum range of movement of the body parts, etc., used by the user U for gesture operation.

[0034] However, when the user U uses his / her hand for the gesture operation, the first range RA may be set within the reach of the user U. This is because, even if an object RO exists within the reach of the user U, there is no possibility that the user U's hand will collide with the object RO, except in exceptional cases. However, there are also users U whose bodies move forward, etc., when performing gesture operations. In this case, it is meaningful to extend the first range RA defined in a stationary state.

[0035] In the case of FIG. 6(A), the first range RA is defined as a sector shape that is symmetrical on both sides of the Y axis, which is the front direction of the user U wearing the xR device 10. 6(A) is assumed to be standing in a real space. Therefore, the direction of the Z-axis arrow in the figure indicates a vertical upward direction. The direction of the Y-axis arrow indicates the front direction of the user U, and the direction of the X-axis arrow indicates the right direction of the user U. The actual range RA is determined by the three-dimensional shape. In Fig. 6(B), assuming the range of hand movement, the upper limit of the range RA is set to about eye height and the lower limit is set to about waist height. The first range RA here is an example of a range that is predetermined from the xR device 10.

[0036] 6(A) and 6(B), a first distance LT1 is also defined. The first distance LT1 here indicates the minimum distance to be secured between the object RO and the virtual screen VD. The first distance LT1 is given as an approximately spherical shape centered on the object RO. However, the positioning sensor 104 (see FIG. 3) can only grasp the surface shape of the object RO from the surface facing the positioning sensor 104, and the overall shape of the object RO is unknown. 6(A) and 6(B) is given based on the front side of the object RO that can be measured by the positioning sensor 104. In a simple system, it is also possible to set the system so that the first distance LT1 to the rear side of the object RO as seen from the user U is not managed.

[0037] Incidentally, the first distance LT1 is a guideline for the distance at which the user U does not collide with the object RO during the gesture operation. Therefore, the first distance LT1 does not guarantee that a collision between a body part or the like of the user U and the object RO will be avoided during the gesture operation. This first distance LT1 may be given as an initial value, or may be set or adjusted individually by the user U, for example.

[0038] 7A and 7B are diagrams illustrating another example of setting the first range RA used for gesture operations, in which (A) shows an example of setting the first range RA when the user U is lying on his / her back, and (B) shows an example of setting the first range RA when the user is lying on his / her side. Although not shown in FIGS. 7A and 7B, the first range RA seen from the top of the head of the user U has a sector shape, similar to that of FIG. 6A. In addition, in FIGS. 7(A) and (B), the surface on which the user U lies is defined by the X-axis and the Y-axis. In either case, the shape and direction of the first range RA are defined based on the xR device 10 worn by the user U.

[0039] 7A, the direction of the Z-axis arrow is the far side as seen by the user U. In addition, in FIG. 7B, the direction of the Y-axis arrow is the far side as seen by the user U. In the case of FIG. 7(B), the user U is lying on his / her side, so the first range RA is set to have an asymmetric shape facing the xR device 10. 6 and 7, the maximum range of the first range RA is defined by the range of motion of the body parts of the user U, but it may also be defined by the maximum viewing angle of the xR device 10. In the first place, the virtual screen VD cannot be displayed beyond the maximum viewing angle of the xR device 10. Therefore, an object RO that exists outside the maximum viewing angle will not be hidden behind the virtual screen VD. In other words, the user U can notice the object RO in advance without having to move the virtual screen VD away.

[0040] 8A and 8B are diagrams illustrating an example of setting a second range RB that is not used for gesture operations. (A) is a diagram of the user U viewed from above, and (B) is a diagram of the user U viewed from the side. In Fig. 8, parts corresponding to those in Fig. 6 are assigned the same reference numerals. 8 provides a range that the user U does not use for operations. In other words, it provides a range in which the virtual screen VD is not displayed. This is because if the virtual screen VD is too close to the user U, not only will the visibility of the virtual screen VD decrease, but gesture operations will also become difficult.

[0041] In FIG. 8, the second range RB is defined by a second distance LT2. The second distance LT2 is given as the distance from the xR device 10. The second distance LT2 defines the minimum distance to be secured between the xR device 10 and the virtual screen VD (see FIG. 1). Hereinafter, the second distance LT2 is also referred to as the "closest position." Hereinafter, the information defining the first range RA, the first distance LT1 defining the retraction distance, the second distance LT2 defining the second range RB, etc. will be collectively referred to as basic parameters.

[0042] <Setting basic parameters, etc.> FIG. 9 is a diagram illustrating an example of a basic parameter setting screen 300. As shown in FIG. The setting screen 300 shown in FIG. 9 is used to select a specific setting method for information about the user and information defining the first range RA. The set information is stored in the xR device 10 (see FIG. 1) or the information terminal 20 as an example of registration information.

[0043] In the case of the setting screen 300, a guide message 301 is placed below the title "Settings (1 / 2)." In Fig. 9, the guide message 301 states, "Used to avoid collisions with surrounding objects," and "Please enter information in the following fields," urging the user to set each field. Note that setting each field may be mandatory or optional.

[0044] In the case of the setting screen 300, the items to be input are age 302, gender 303, height 304, body part to be used for operation etc. 305, dominant hand etc. 306, and margin setting 307, and below that are a second guidance message 308 and selection buttons 309 to 313. 9, pull-down menus are provided for age 302, sex 303, height 304, body part used for operation etc. 305, and dominant hand etc. 306. Of these items, age and height may be provided with input fields for entering numerical values.

[0045] The pull-down menu for parts used for operation 305 lists selectable body parts such as fingers, hands, elbows, feet, knees, ankles, and also assistive tools such as writing implements and canes. In the setting of the dominant hand etc. 306, it is possible to select whether to use the right hand mainly, the left hand mainly, or both hands to operate the virtual screen VD. The margin setting 307 can be set to "yes" or "no" for extending the first range RA set by the first distance LT1.

[0046] When the margin setting 307 is "Yes," the first range RA is expanded further from the first distance LT1. The expanded distance may be predetermined, or may be specifically set or adjusted by the user using sub-settings. The expanded direction refers to the setting of the dominant hand, etc. 306. For example, if the setting of the dominant hand, etc. 306 is right-handed, the first range RA is expanded toward the right hand. When the margin setting 307 is "none", the first range RA is given by the first distance LT1.

[0047] The second guidance message 308 includes the following statements: "Set a threshold value for determination." and "Select one of the buttons below." Button 309 is labeled "Set on screen." When button 309 is operated, the screen switches to a setting screen for each threshold. The thresholds here are, for example, the distance that the user can reach, a first distance LT1, and a second distance LT2. The button 310 is labeled "Set by Gesture." When the button 310 is operated, each threshold value is set based on the measured values ​​of the range of motion of the limbs used for the operation.

[0048] The button 311 is labeled "Set by voice." When the button 311 is operated, the user sets each threshold by voice in response to a prompt. Button 312 is labeled "Set with estimated values." When button 312 is operated, each threshold value is estimated based on the age, sex, height, etc. selected from the pull-down menu, and each threshold value is set to the estimated value. During estimation, each threshold value is read from a table prepared for standard body types corresponding to age, sex, height, etc. When the button 313 is operated, the setting screen 300 switches to a home screen or the like.

[0049] FIG. 10 is a diagram illustrating an example of a setting screen 320 used to set each threshold value. In the case of the settings screen 320 shown in Fig. 10, guidance 320A is placed below the title "Settings (2 / 2)." In Fig. 10, guidance 320A reads, "Setting the function to avoid collisions with surrounding objects," and "Please enter information in the following fields." In FIG. 10, items to be set include "reachable distance" 321, "distance to retreat from surrounding objects" 322, and "closest position at which virtual screen is displayed" 323.

[0050] "Reachable distance" 321 is the distance required to set the first range RA. In Fig. 10, it is illustrated as the horizontal distance from the user in the depth direction. When the user U is standing on the ground or floor, the horizontal distance is the distance parallel to the ground. In addition, since the first range RA needs to be specified not only in the depth direction but also in the horizontal and vertical directions, it is also possible to provide fields for setting these values. In the case of Figure 10, the unit of distance is centimeters, but inches can also be used.

[0051] The "distance to retreat from surrounding objects" 322 is the distance that provides the first distance LT1. In Fig. 10, it is illustrated that this is the distance to be secured between the object RO and the virtual screen VD. In Fig. 10, the unit of distance is centimeters, but it may also be inches. The "closest position for displaying the virtual screen" 323 is the distance that gives the second distance LT2. In Fig. 10, the illustration shows that this is the distance to be secured between the user and the virtual screen VD. In Fig. 10, the unit of distance is centimeters, but it may also be inches.

[0052] Among these, "reachable distance" 321 is used to determine whether or not the virtual screen VD needs to be evacuated, and "evacuation distance from surrounding objects" 322 and "closest position for displaying virtual screen" 323 are used when evacuating the virtual screen VD from the object RO. If the distance between the virtual screen VD and the object RO is within a first distance LT1, the virtual screen VD is evacuated. However, the distance between the position after evacuating (hereinafter referred to as "evacuation position") and the user is set to be equal to or greater than a second distance LT2.

[0053] In addition, the setting screen 320 has a button 324 labeled "Back" and a button 325 labeled "Settings." When the button 324 is operated, the setting screen 320 switches to the setting screen 300 (see FIG. 9). In this case, the numerical value input on the setting screen 320 is not set. When the button 325 is operated, the numerical value input on the setting screen 320 is set.

[0054] 11A and 11B are diagrams illustrating how the first range RA is set by a gesture. (A) is a diagram illustrating how the first range is set as seen from behind the user U. (B) is a diagram illustrating how the first range is set as seen from the side of the user U. The setting of the first range RA by a gesture is executed when the button 310 (see FIG. 9) is operated. The setting by gesture is performed according to the guidance displayed on the virtual screen VD and the audio guide reproduced from the speaker 102 (see FIG. 2).

[0055] In the case of FIG. 11(A), the xR device 10 plays audio guidance such as "Register your moving area," "Keep facing forward and move your hands and feet widely to operate," and "If you need to use a tool to operate the device, please use the tool." 11(A), the user U draws a large circle in the air. The area enclosed by the circle is set as a first area RA. 11(B) shows how the first range RA is defined by the distance L(t) from the xR device 10. The tn in parentheses represents the distance L(t) measured at the time tn.

[0056] When using gestures, it is possible to accurately set the first range RA that the user U may actually use for operations. The first distance LT1 and the second distance LT2 can also be set by gesture. For example, they can be set as the distance between the left hand and the right hand. However, the first distance LT1 and the second distance LT2 can also be set by voice. For example, they can be set by saying, "Please set it to 5 cm."

[0057] <Example of setting the first range> 12 is a diagram illustrating examples of setting the first range RA. (A) is a basic example of the first range RA, (B) is an example in which the range on the dominant hand side is set wider than the other side, (C) is an example in which the first range RA is expanded by a margin, and (D) is an example in which the range on the non-dominant hand side is narrowed. In Fig. 12, parts corresponding to those in Fig. 6(A) are assigned the same reference numerals. That is, Fig. 12(A), (B), (C), and (D) all show a view of the user U as seen from above.

[0058] FIG. 12(A) shows the basic shape of the first range RA, which is symmetrical with respect to the front direction of the user U. 12(B) to 12(D) are examples of setting the first range RA when the right hand is set as the dominant hand. In the case of FIG. 12(B), the shape of the first range RA remains the basic shape shown in FIG. 12(A), but is rotated clockwise when viewed from above the user U.

[0059] The first range RA shown in Fig. 12(C) is the case when the right hand is set as the dominant hand and the margin setting is "ON." The first range RA shown in Fig. 12(C) is wider than the first range RA shown in Fig. 12(B) by the margin. The first range RA shown in FIG. 12(D) is obtained by removing the left front part of the user U that is difficult to reach with the right hand from the basic shape shown in FIG. 12(A). 12, the first range RA can take various shapes depending on whether a dominant hand is set or not and whether a margin is set or not. That is, the range may not only be widened on the dominant hand side, but also narrowed only on the non-dominant hand side.

[0060] <Processing operation> The following describes the processing operations executed by the information terminal 20 (see FIG. 1) that executes the saving function of the virtual screen VD. The processing operations and functions described below are realized through the execution of a program by the processor 201 (see FIG. 5).

[0061] <Basic parameter setting operation> 13 is a flowchart illustrating an example of the basic parameter setting operation, in which the symbol S indicates a step. First, the processor 201 determines whether or not the collision avoidance function is set (step 1). If a negative result is obtained in step 1, the processor 201 ends the process. If a positive result is obtained in step 1, the processor 201 accepts information about the user (step 2). This process corresponds to the input of information via the setting screen 300 (see FIG. 9).

[0062] Next, the processor 201 determines whether each threshold is set individually (step 3). If a positive result is obtained in step 3, processor 201 sets the threshold based on the method selected by user U (step 4). Step 4 is executed when setting by any of "Set on screen," "Set by gesture," and "Set by voice" is selected. If step 3 gives a negative result, the processor 201 estimates the threshold using information about the user U and sets the estimated threshold (step 5). By executing step 4 or step 5, the initial setting of the threshold value is completed.

[0063] <Virtual screen evacuation behavior> 14 is a flowchart illustrating an example of the operation of saving the virtual screen VD according to Embodiment 1. The symbol S in the drawing indicates a step. First, the processor 201 (see FIG. 5) determines whether the collision avoidance function is on or not (step 11). While a negative result is obtained in step 11, processor 201 repeats the determination in step 11. If a positive result is obtained in step 11, the processor 201 measures the distance to an object RO present around the user U (step 12).

[0064] When the measurement of the distance to the object RO around the user U is completed, the processor 201 determines whether the initial position of the virtual screen VD is within the first range RA (step 13). If a negative result is obtained in step 13, the processor 201 displays the virtual screen VD at the initial position (step 14). The initial position is defined as, for example, a position 50 cm in front of the xR device 10 (see FIG. 1). Of course, the initial position here is just an example, and the user U can also adjust it individually.

[0065] If a positive result is obtained in step 13, the processor 201 determines whether or not an object RO is present within the first range RA (step 15). If a negative result is obtained in step 15, collision avoidance of the virtual screen VD is not necessary, so the processor 201 returns to step 12. If a positive result is obtained in step 15, the processor 201 determines whether or not there is a single object within the first range RA (step 16).

[0066] If there is one target object RO, the processor 201 obtains a positive result in step 16. In this case, the processor 201 determines whether the candidate retraction position is closer than the closest position (step 17). The retraction position here is the position after the virtual screen VD is moved a first distance LT1 in a predetermined direction. The retraction direction is set, for example, to a direction approaching the user U. Note that the distance used to retract the virtual screen VD does not need to be the same as the first distance LT1. However, by setting the distance used to retract the virtual screen VD to the first distance LT1, the retract position is more likely to be the first distance LT1 or more away from the object RO.

[0067] Of course, even if the distance used to retract the virtual screen VD is shorter than the first distance LT1, retraction can be repeated until the retracted position is at least the first distance LT1. Furthermore, the distance of retraction may differ depending on the direction of retraction. For example, the distance may differ when retracting in a direction toward the user U and when retracting in a direction away from the user U. For example, the distance in the direction away from the user U may be shorter than the distance when retracting in a direction toward the user U. This is because if the retraction is made the same distance in the direction away from the user U as in the direction toward the user U, there is a possibility that parts of the user U's body may not reach the virtual screen VD. Furthermore, if the retraction is made to the far side of the object RO, the object RO will be located in front of the virtual screen VD, and the user U can avoid contact with the object RO.

[0068] If a positive result is obtained in step 17, the processor 201 displays the virtual screen VD at the closest position (step 18). That is, the virtual screen VD is displayed at the position at the smallest distance that allows it to approach the user U. In this case, although there remains a possibility that the distance between the retracted virtual screen VD and the object RO will not be the first distance LT1, the virtual screen VD is retracted in a direction closer to the user U than the initial position. Therefore, the possibility that a body part or the like will collide with the object RO during a gesture operation is reduced compared to the initial position.

[0069] If a negative result is obtained in step 17, the processor 201 determines the candidate evacuation position as the evacuation position and displays the virtual screen VD (step 19). This display reduces the possibility of a collision with an object RO present around the user U when performing a gesture operation on the virtual screen VD. Return to the explanation of step 16. If a negative result is obtained in step 16, the processor 201 determines whether or not there is at least one candidate for the evacuation position that is closer than the closest position (step 20).

[0070] Here, there are multiple candidates for the evacuation position for each of multiple objects RO present around the user U. For example, if the number of objects RO present within the first range RA is three, there are three candidates for the evacuation position. If any one of these three candidates is closer to the user U than the closest position, the processor 201 obtains a positive result in step 20. In this case, the processor 201 proceeds to step 18.

[0071] On the other hand, if a negative result is obtained in step 20, the processor 201 determines the candidate evacuation position closest to the user U as the evacuation position and displays the virtual screen VD (step 21). This evacuation makes it possible to evacuate the virtual screen VD as far as possible from the object RO closest to the user U, which is most likely to cause a collision. Of course, the possibility of collision with an object RO located further away is further reduced by evacuation of the virtual screen VD from its initial position.

[0072] 14, after executing step 18, step 19, or step 21, the processor 201 temporarily ends the series of processes shown in Fig. 14. In this case, the processor 201 again executes the determination of step 11. However, the processor 201 may return to step 12 after executing step 18, step 19, or step 21. Apart from the saving operation of the virtual screen VD shown in FIG. 14, the processor 201 executes a process of accepting a gesture operation on the virtual screen VD, and an execution of the accepted process.

[0073] Incidentally, the position where the processor 201 saves the virtual screen VD must satisfy the following conditions. One of the conditions is that the retracted position is included within the field of view of the user U. This is because it is a prerequisite that the visibility of the virtual screen VD is maintained. A similar condition is that the retracted position is included within the viewing angle of the xR device 10 (see FIG. 1). This is because the display of the virtual screen VD is restricted by the physical performance of the xR device 10. Even if the retracted position of the virtual screen VD is within the field of view of the user U, it is impossible for the user U to view the virtual screen VD if it is outside the field of view of the xR device 10.

[0074] Another condition is that the retreat position is within a range where gesture operation by the user U is possible. The range where gesture operation is possible here means a range where parts of the user U's body can reach. In other words, the retreat position is set within the first range RA. This is because in this embodiment, an operation of virtually touching the virtual screen VD is regarded as a gesture operation. For example, when the virtual screen VD is moved 2 m away from the user U, even if an assisting tool or the like is used, there is a high possibility that the surface of the virtual screen VD cannot be touched. 14 assumes the case where one virtual screen VD is displayed in the air by the xR device 10, but can also be applied to the case where multiple virtual screens VD are displayed. In this case, the saving of the virtual screen VD should be interpreted as being performed on the multiple virtual screens VD as a single unit.

[0075] <Example of virtual screen retreat> Hereinafter, specific examples of retreat of the virtual screen VD will be described using FIGS. 15 to 23. <Retreat Example 1> FIG. 15 is a diagram for explaining an example of retreating the display position of the virtual screen VD in a direction approaching from the object RO to the user U. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U after retreat.

[0076] In the case of FIG. 15(A1), the virtual screen VD is visible on the top plate of Table 3, but the drink 5 located behind it is not visible. According to FIG. 15(A2), the distance from the xR device 10 to the virtual screen VD is L0, and the distance from the xR device 10 to the drink 5 is L1. And the difference between the distance L0 and the distance L1 is L2. Incidentally, L2 <LT1. In this positional relationship, there is a high possibility of contact with the drink 5 when tapping a button arranged on the surface of the virtual screen VD or swiping the virtual screen VD. Also, in this example, the object RO hidden behind the virtual screen VD is the relatively lightweight and easily movable drink 5, but if it is a hard and fixed object, there is also a possibility that the user U may be injured by contact with the object RO.

[0077] On the other hand, in FIGS. 15(B1) and (B2), the display position of the virtual screen VD is retreated in the direction of the user U. In this example, the display position of the virtual screen VD is moved to a position separated from the drink 5 by the first distance LT1. Since the first distance LT1 is determined in consideration of the distance that a fingertip or the like used for a gesture operation moves on the back side of the virtual screen VD, the possibility of contact with the drink 5 is less than when the virtual screen VD is displayed at the initial position. In the case of FIG. 15(B2), although the virtual screen VD is moved by a distance of LT1 - L2 in the direction of the user U, it is also possible to move the virtual screen VD by the first distance LT1. In that case, the distance between the virtual screen VD and the drink 5 becomes LT1 + L2.

[0078] <Retreat Example 2> 16 is a diagram illustrating an example of retraction when an object RO is subsequently placed behind the virtual screen VD. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, (B1) and (B2) show a state in which a drink 5 as the object RO is placed on the table 3, and (C1) and (C2) show the positional relationship between the virtual screen VD and the user U after retraction. In FIG. 16, parts corresponding to those in FIG. 15 are assigned the same reference numerals.

[0079] In the case of evacuation example 1, it was assumed that a drink 5 was placed on the table 3 and the display position of the virtual screen VD subsequently moved closer to it, but in evacuation example 2, an object RO such as a drink 5 is placed at the position where the virtual screen VD is displayed. Even in this case, when a new object RO is detected, the virtual screen VD is retracted to a position where there is less possibility of a collision. Figures 16(C1) and 16(C2) show retraction after an object RO such as a drink 5 is detected. On the other hand, movement of the virtual screen VD that is not based on a gesture operation by the user U may confuse the user U.

[0080] Therefore, when the display position of the virtual screen VD is moved by the collision avoidance function while the virtual screen VD is being displayed, the user U may be notified of this. The notification here may be, for example, sound or voice played from the speaker 102 (see Figure 3), vibration from a vibrator, a warning message or the like may be displayed on the currently displayed virtual screen VD, or a warning virtual screen VD may be displayed separately from the currently displayed virtual screen VD.

[0081] <Evacuation Example 3> 17 is a diagram illustrating an example of evacuation when a candidate evacuation position is closer to the user U than the second distance LT2. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U after evacuation. In FIG. 17, parts corresponding to those in FIG. 15 are assigned the same reference numerals. Assume that the first distance LT1 used in evacuation example 3 is longer than the first distance LT1 used in evacuation example 1. In this case, a position closer to the user U by the first distance LT1 from the drink 5 is included in the second distance LT2 at which display of the virtual screen VD is prohibited. In other words, the candidate evacuation position is included in the second range RB.

[0082] Therefore, in retraction example 3, the retraction position of the virtual screen VD is set to a position that is a second distance LT2 from the user U. In this case, the distance between the virtual screen VD and the drink 5 after retraction is shorter than the first distance LT1. However, the distance between the virtual screen VD and the drink 5 is longer than the initial position. Therefore, the possibility of collision with the object RO such as the drink 5 during the gesture operation is higher than in the case of the evacuation example 1, but the possibility of collision is lower than in the case of the initial position.

[0083] <Evacuation Example 4> 18 is a diagram illustrating an example of retraction when multiple virtual screens VD1 and VD2 are arranged in the depth direction. (A1) and (A2) show the positional relationship between the virtual screens VD1 and VD2 and the user U at their initial positions, and (B1) and (B2) show the positional relationship between the virtual screens VD1 and VD2 and the user U after retraction. In FIG. 18, parts corresponding to those in FIG. 15 are assigned the same reference numerals.

[0084] In the case of the save example 4, two virtual screens VD1 and VD2 are displayed, and the display position of the virtual screen VD2 located behind the user U overlaps with the drink 5. However, since another virtual screen VD1 is displayed in front of the virtual screen VD2 located behind, the user U does not realize that the virtual screen VD2 located behind overlaps with the drink 5. However, the height at which the rear virtual screen VD2 of the two virtual screens VD1 and VD2 is displayed is offset upward from the height at which the front virtual screen VD1 is displayed, so the user U can recognize that another virtual screen VD2 exists behind the virtual screen VD1 displayed in the foreground.

[0085] In the case of retraction example 4, the target of the gesture operation is the virtual screen VD1 located in front of the user U. Therefore, the virtual screen VD1 located in front of the user U is retracted with respect to the drink 5. In other words, the virtual screen VD1 is retracted in a direction away from the drink 5 by a first distance LT1. This retraction reduces the possibility of contact with the drink 5 due to the gesture operation. 18(B1) and 18(B2), in conjunction with the retraction of the virtual screen VD1, the virtual screen VD2 located at the back is also retracted together with the virtual screen VD1, thereby eliminating the overlap between the virtual screen VD2 and the drink 5. However, since the target of the gesture operation is the virtual screen VD1 on the front side, only the display position of the virtual screen VD1 may be moved away, and the virtual screen VD2 may remain at the initial position.

[0086] Furthermore, when the display positions of the virtual screens VD1 and VD2 are retracted together, the distance between the virtual screen VD2 located on the rear side and the drink 5 may be increased by a first distance LT1. In this case, the distance between the virtual screen VD1 on the front side and the drink 5 becomes even greater than the first distance LT1. Therefore, the possibility that a body part or the like will collide with the drink 5 during a gesture operation on the virtual screen VD1 on the front side becomes even lower than in the cases of FIGS. 18(B1) and (B2). However, the virtual screen VD1 located on the closer side will be closer to the user U. Therefore, the possibility that the distance between the virtual screen VD1 located on the closer side and the user U will be shorter than the second distance LT2 is higher than when the virtual screen VD1 located on the closer side is moved away from the drink 5 by the first distance LT1.

[0087] <Evacuation Example 5> 19 is a diagram illustrating an example of retracting the virtual screen VD upward. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U after retraction. In FIG. 19, parts corresponding to those in FIG. 15 are assigned the same reference numerals. 19(B2), in the case of retraction example 5, the horizontal distance between the user U and the virtual screen VD does not change before and after the retraction of the virtual screen VD. However, since the virtual screen VD is spaced upward by a first distance LT1 from the upper end of the drink 5, the possibility that a part of the user U's body will collide with the drink 5 when performing a gesture operation on the virtual screen VD is reduced.

[0088] In the case of the save example 5, the virtual screen VD shown in FIG. 19(B1) is saved, and the drink 5 located behind it becomes visible to the user U. Although in Figure 19 (B2) the virtual screen VD is moved by a first distance LT1 based on the top end of the drink 5, it is also possible to simply move the virtual screen VD upward from the initial position by the first distance LT1.

[0089] However, when moving the virtual screen VD upward from its initial position, if the height dimension of an object RO, such as a drink 5, located behind the virtual screen VD is long, there is a possibility that the virtual screen VD and the object RO, such as a drink 5, will still overlap after the movement. In this case, the virtual screen VD may not only be moved upward as viewed from the xR device 10, but may also be moved forward by a predetermined distance, for example, a first distance LT1. However, in a situation where the upward movement of the virtual screen VD allows the drink 5 hidden behind it to be noticed, even if the virtual screen VD is close to the drink 5 after being evacuated, the possibility of a collision is reduced compared to when the virtual screen VD continues to be displayed in its initial position. The direction in which the virtual screen VD is retracted is not limited to upward, but may be downward.

[0090] <Evacuation Example 6> 20 is a diagram illustrating an example of retracting the virtual screen VD horizontally. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U after retraction. In FIG. 20, parts corresponding to those in FIG. 15 are assigned the same reference numerals. In the case of the evacuation example 6, as shown in Fig. 20(B1), the virtual screen VD is moved to the left of the drink 5 as seen from the user U. Therefore, as shown in Fig. 20(B2), the horizontal distance between the user U and the virtual screen VD does not change before and after the evacuation of the virtual screen VD.

[0091] However, not only is the virtual screen VD moved leftward by the first distance LT1 from the left edge of the drink 5, but the entire drink 5, which was initially hidden behind the virtual screen VD, becomes visible. Therefore, the user U can perform gesture operations on the virtual screen VD while being aware of the presence of the drink 5. That is, the possibility that a part of the user U's body or the like will collide with the drink 5 is reduced.

[0092] In Figure 20 (B2), the virtual screen VD is moved by a first distance LT1 based on the left end of the drink 5, but it is also possible to simply move the virtual screen VD by the first distance LT1 to the left from the initial position. However, when moving the virtual screen VD to the left from its initial position, if the left dimension of an object RO, such as a drink 5, located behind the virtual screen VD is long, there is a possibility that the virtual screen VD and the object RO, such as a drink 5, will still overlap after the movement. In this case, the virtual screen VD may not only be moved to the left as viewed from the xR device 10, but may also be moved forward by a predetermined distance, for example, a first distance LT1.

[0093] However, in a situation where the movement of the virtual screen VD to the left allows the drink 5 hidden behind it to be noticed, even if the distance between the virtual screen VD and the drink 5 after the retraction is close, the possibility of a collision is reduced compared to when the virtual screen VD continues to be displayed in its initial position. The direction in which the virtual screen VD is retracted is not limited to the left, but may be the right. In addition, the direction of evacuation may be any of the diagonally upper left, diagonally lower left, diagonally upper right, and diagonally lower right. Furthermore, by combining with other evacuation examples, movement in any direction is possible, such as diagonally forward left, diagonally backward left, diagonally upward front, diagonally upward back, diagonally forward right, diagonally backward back right, diagonally downward front, and diagonally downward back.

[0094] <Evacuation Example 7> 21 is a diagram illustrating an example of retraction of the virtual screen VD when multiple drinks 5A and 5B are placed on a table 3 in front of a user U. (A1) and (A2) show the positional relationship between the virtual screen VD, the user U, and the drinks 5A and 5B at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD, the user U, and the drinks 5A and 5B after retraction. In FIG. 21, parts corresponding to those in FIG. 15 are assigned the same reference numerals.

[0095] In the cases of FIGS. 21(A2) and (B2), the two drinks 5A and 5B are lined up in the rearward direction as seen from the user U. In this arrangement, the drink 5A located in the foreground is the drink most likely to collide with the drink 5A when a gesture operation is performed. Therefore, similar to the case of the evacuation example 1, when the distance L2 between the drink 5A located in the foreground and the virtual screen VD is equal to or less than the first distance LT1, the position where the virtual screen VD is displayed is evacuated. The evacuation of the virtual screen VD shown in FIGS. 21(B1) and (B2) is the same as the case of the evacuation example 1, except for the number of objects RO that cause the evacuation. When the distance L1 between the user U and the drink 5A is the same as the distance L1 between the user U and the drink 5B, it is determined whether the candidate evacuation position of the virtual screen VD satisfies the criteria for both the drink 5A and the drink 5B.

[0096] <Evacuation Example 8> 22 is a diagram illustrating an example in which the display position of the virtual screen VD is retracted toward the back of the object RO as viewed from the user U. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U after retraction. In FIG. 22, parts corresponding to those in FIG. 15 are assigned the same reference numerals. In the eighth evacuation example, as shown in Fig. 22(B2), the virtual screen VD is moved a first distance LT1 further in the rearward direction based on the distance L1 to the drink 5. Of course, as in the other evacuation examples, the virtual screen VD may be moved a first distance LT1 based on the initial position of the virtual screen VD.

[0097] 22(B1) and 22(B2), the entire drink 5 appears on the near side of the virtual screen VD. Therefore, the user U can perform a gesture operation while being aware of the presence of the drink 5. However, if the distance L2 between the display position of the virtual screen VD after retraction and the drink 5 is short or if the display position of the virtual screen VD after retraction overlaps with the drink 5, the visibility of the virtual screen VD for the user U may be reduced. In that case, however, the user U can change the direction of his or her face to change the position where the virtual screen VD is displayed, thereby eliminating the difficulty in viewing the virtual screen VD.

[0098] Furthermore, if the operation buttons, etc. on the retracted virtual screen VD are located immediately behind the drink 5 as viewed from the user U, it is difficult to operate the operation buttons, etc., even if the user U does not notice the drink 5 and bump into it when making a gesture operation. In this case, too, the user U can safely perform gesture operations on the operation buttons, etc. on the virtual screen VD by turning his or her face to change the position where the virtual screen VD is displayed. Depending on the physical limitations of the user U and the environment in which the xR device 10 is used, it may be difficult for the user U to freely change the direction of their face.

[0099] However, in this case, the display position of the virtual screen VD may be movable from the initial position by an instruction from the user U. For example, the moving distance of the virtual screen VD may be increased by voice, or a change in the display position may be instructed. Furthermore, the display position of the virtual screen VD may be changed from the initial position by a gesture operation such as pushing down in the depth direction a region of the virtual screen VD that does not overlap with the drink 5, a gesture operation of sweeping the side of the region that does not overlap with the drink 5 in the opposite direction, or a gesture operation of picking up and moving it to another location. Of course, the display position after the movement is moved within a range where the user U can perform gesture operations.

[0100] <Evacuation Example 9> 23 is a diagram illustrating an example of retraction when the virtual screen VD overlaps with a surrounding object RO in space. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, (B1) and (B2) show the positional relationship between the virtual screen VD and the user U immediately after the orientation of the user U's face changes, and (C1) and (C2) show the positional relationship between the virtual screen VD and the user U after retraction. In FIG. 23, parts corresponding to those in FIG. 15 are assigned the same reference numerals. 23(A2), (B2), and (C2) all show views of the user U as seen from above.

[0101] 23, it is assumed that the user U is located in a corner surrounded by two partitions 9A and 9B. However, the partitions 9A and 9B are just an example, and the same situation occurs near the wall of a room, etc. Furthermore, the case where part or all of the display position of the virtual screen VD overlaps with a surrounding object RO may also occur in the other evacuation examples described above. 23(A1) and 23(A2), the user U is facing the partition 9A, and the virtual screen VD is also located in front of the partition 9A. Note that another partition 9B exists to the right of the user U, but since the user U can see it, gesture operations on the virtual screen VD can be performed without any hindrance.

[0102] 23(B1) and (B2) show the case where the user U changes the orientation of his or her body clockwise when viewed from above. Even in this case, the display of the virtual screen VD is not restricted by the real space and is displayed in front of the user U. Therefore, in the example shown in FIG. 23(B2), a part of the virtual screen VD is in a position that penetrates the partition 9B. Although the arrangement shown in FIG. 23(B2) would not occur on a real display, the virtual screen VD is a virtual object recognized by the brain of the user U, and therefore the virtual screen VD is perceived as existing inside the partition 9B.

[0103] When the sense of immersion is high, the user U may forget the existence of the actual partition 9B and may try to perform gesture operations on the virtual screen 9B in the area that overlaps with the partition 9B, possibly causing a body part or the like to collide with the partition 9B. However, as shown in Figures 23(C1) and (C2), if the virtual screen VD is moved horizontally away from both partitions 9A and 9B by a first distance LT1, the user U can continue to view the virtual screen VD and perform gesture operations without any problems, without having to change the direction of his or her face.

[0104] However, in the cases of FIGS. 23(C1) and 23(C2), the display position of the virtual screen VD is required to be a first distance LT1 away from both the partitions 9A and 9B. As a result, there is a possibility that the candidate evacuation position will be located within the second range RB. In that case, as in the evacuation example 3, the virtual screen VD may be displayed with a position that is a second distance LT2 away from the user U as the evacuation position. In that case, although the distance between the virtual screen VD and either or both of the partitions 9A and 9B may be shorter than the first distance LT1, the possibility of a collision with a surrounding object RO during a gesture operation by the user U is reduced at least compared to the initial position.

[0105] Furthermore, even if the virtual screen VD is displayed at a position the second distance LT2 from the user U, there is a possibility that the overlap between a part of the virtual screen VD and either or both of the partitions 9A and 9B may not be resolved. In this case, it is desirable to recommend to the user U by voice or the like that they move their position. For example, by outputting voice such as "Could you move back just 5 cm more?" or "Could you turn your face a little more counterclockwise?", the possibility of collision with a surrounding object RO during a gesture operation can be reduced.

[0106] The brightness or color tone of the portion of the virtual screen VD that overlaps with the surrounding object RO may be changed to draw the user U's attention. For example, the brightness may be reduced to reduce visibility so that the portion is not recognized as a target for gesture operation. Also, the corresponding area may be displayed in a reddish color tone to notify the user U that the area is not suitable for gesture operation. However, if a suitable retreat position for displaying the virtual screen VD cannot be found, a function for stopping the display of the virtual screen VD may be provided. Note that if the user U changes their posture or position and it becomes possible to display the virtual screen VD, the display of the virtual screen VD is immediately resumed.

[0107] <Embodiment 2> In this embodiment, a case will be described in which a function is added that does not execute the saving of the virtual screen VD by the collision avoidance function. 24 is a flowchart illustrating an example of the operation of saving the virtual screen VD in embodiment 2. In Fig. 24, parts corresponding to those in Fig. 14 are assigned the same reference numerals. In this embodiment, if a positive result is obtained in step 11, the processor 201 (see FIG. 5) determines whether or not the virtual screen VD includes an icon for operation (step 31).

[0108] Examples of operation icons include slider bars, numeric keys, function keys, keyboard keys, operation panels, function buttons, pull-down menus, task bars, file icons, folder icons, thumbnails, avatars, and characters. An icon may be part of a virtual object, or it may be a virtual object on its own. The illustrated icons may be called by various names, but the operation icons include user interface displays that represent input / output to and from a computer, function calls, and differences in data files.

[0109] Therefore, icons that are not linked to operational functions, such as displayed patterns or parts of images, are not considered to be operational icons. In other words, the determination in step 31 can be regarded as a determination as to whether the virtual screen VD is for display only. For example, display only refers to when only video content is displayed on the virtual screen VD or when the editing function is turned off.

[0110] If a positive result is obtained in step 31, the processor 201 executes the processing from step 12 onwards in FIG. On the other hand, if a negative result is obtained in step 31, the processor 201 displays the virtual screen VD at the initial position (step 32). That is, if a negative result is obtained in step 31, the processor 201 does not save the virtual screen VD. As a result, the virtual screen VD continues to be displayed at a specific position in front of the xR device 10 (see FIG. 1). Although the display position is not saved, for example, when the direction of the user U's face changes, the virtual screen VD moves to a predetermined position relative to the xR device 10.

[0111] Fig. 25 is a diagram illustrating the processing operation when the virtual screen VD does not include icons for operation. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U when the virtual screen VD does not include icons for operation. In Fig. 25, parts corresponding to those in Fig. 15 are assigned the same reference numerals. In the cases of Figures 25(A1) and (A2), the distance L2 between the virtual screen VD and the drink 5 is equal to or less than the first distance LT1. However, the virtual screen VD does not include any icons for operation. Therefore, the display position of the virtual screen VD in Figures 25(B1) and (B2) is the same as that in Figures 25(A1) and (A2).

[0112] <Third Embodiment> In this embodiment, a function specific to the case where the object to be avoided by collision is a movable object will be described. 26 is a flowchart illustrating an example of the operation of saving the virtual screen VD in embodiment 3. In FIG. 26, parts corresponding to those in FIG. 14 are assigned the same reference numerals. In this embodiment as well, if the collision avoidance function is on (i.e., if the result in step 11 is positive), the processor 201 (see FIG. 5) measures the distance to an object present around the user (step 12).

[0113] Next, the processor 201 determines whether the initial position of the virtual screen VD is within the first range RA (step 13). If a positive result is obtained in step 13, the processor 201 determines whether or not an object RO is present within the first range RA (step 15). If step 15 gives a negative result, processor 201 returns to step 12. On the other hand, if a positive result is obtained in step 15, the processor 201 in this embodiment determines whether the detected object RO has been present in the same place for a predetermined period of time or longer (step 41).

[0114] The determination here roughly corresponds to determining whether or not the object is stationary. The time used as the threshold may be an initial value or may be set by the user U. Furthermore, the threshold time can be modified afterwards. If a positive result is obtained in step 41, for example, if the object RO is a drink 5 (see FIG. 1), the processor 201 executes the processing from step 16 onwards in FIG. 14. The processing here is for the case where the object RO is a stationary object, and is therefore the same as in the first embodiment.

[0115] If a negative result is obtained in step 41, processor 201 proceeds to step 42. A negative result is obtained in step 41 when an object RO present within first range RA is determined to be a movable object. Of course, the range of movable objects is wide, and for example, as in evacuation example 2 described in FIG. 16, a drink 5 immediately after being placed on table 3 is included in the movable object. However, a drink 5 that remains on table 3 and remains in the same place for a certain period of time or more is determined to be a stationary object. Additionally, moving objects also include cleaning robots, curtains blowing in the wind, tree branches, etc. In this way, the range of objects RO that are determined to be moving objects is wide.

[0116] In step 42, the processor 201 determines whether or not it is possible to acquire the movement schedule of the object RO. A travel schedule is information that specifies the travel route and the estimated time to pass each point. However, in a broad sense, a travel schedule may also be information that only provides the travel route. A case where a movement schedule can be obtained means a case where, assuming the existence of a movement schedule linked to an object RO determined to be a movable object, the movement schedule can be obtained through communication with the target object RO directly or via another terminal, or a case where the movement schedule can be obtained from a server or the like that manages the movement schedule.

[0117] Furthermore, when there are multiple movable objects with movement schedules, it is required to be able to identify the object RO determined to be a movable object in order to acquire the movement schedule. If any of these conditions is not met, the processor 201 obtains a negative result in step 42 . If a positive result is obtained in step 42, the processor 201 predicts the movement range of the object RO from the movement schedule (step 43).

[0118] Since this is a prediction, the position of the object RO is predicted as a range with a certain degree of spread. The same applies to the predictions in steps 44 and 47. On the other hand, if a negative result is obtained in step 42, the processor 201 predicts the movement range of the object RO from the movement of the object RO (step 44). For example, the movement range is predicted from the movement of a curtain or a tree branch.

[0119] When the movement range is predicted by executing step 43 or step 44, the processor 201 regards the predicted movement range as the position of the object RO and executes the processing from step 16 (see FIG. 14) onwards (step 45). If there are a plurality of objects RO that have been determined to be movable, the processing from step 42 onwards is executed for each of them.

[0120] Return to the explanation of step 13. If a negative result is obtained in step 13, the processor 201 determines whether there is an object RO for which a movement schedule can be obtained (step 46). The object RO here is only required to be located outside the first range RA, and it is irrelevant whether it is located at a position that can be measured by the positioning sensor 104 (see Figure 2) or at a position that can be imaged by the camera 105 (see Figure 2). If a positive result is obtained in step 46, the processor 201 predicts the movement range of the object RO from the movement schedule (step 47).

[0121] Thereafter, the processor 201 determines whether the initial position of the virtual screen VD is included in the predicted movement range (step 48). Here, the initial position is used, but the display position at the time of determination is also included. If a positive result is obtained in step 48, the processor 201 proceeds to step 45. That is, the processor 201 regards the predicted movement range as the position of the object and executes the processes from step 16 onwards. If a negative result is obtained in step 46 or step 48, the processor 201 displays the virtual screen VD at the initial position (step 14).

[0122] <Evacuation example 1> 27 is a diagram illustrating an example of evacuation when a person H is approaching the user U. (A) shows a state in which the person H is moving outside the first range RA toward the user U, and (B) shows a state in which the person H has entered the first range RA. In FIG. 27, parts corresponding to those in FIG. 15 are denoted by the same reference numerals. In the case of Fig. 27(A), there is no possibility of a collision with person H during the gesture operation. Therefore, the virtual screen VD is displayed at the initial position. Note that the case shown in Fig. 27(A) corresponds to the case where a negative result is obtained in step 46 (see Fig. 26) and step 14 (see Fig. 26) is executed.

[0123] On the other hand, in the case of FIG. 27(B), person H is inside the first range RA. In this case, processor 201 detects person H as a movable object. That is, a negative result is obtained in step 41 (see FIG. 26). Since there is no movement schedule for person H, the movement range is predicted from the movement of person H, and the virtual screen VD is evacuated. This case corresponds to the case where a negative result is obtained in step 42 (see FIG. 26). 27(B), the virtual screen VD is retracted in a direction away from the person H. In FIG. 27(B), as a result of the virtual screen VD retracting to the left side of the user U, the person H who was approaching from behind the virtual screen VD becomes visible. In the evacuation example 1, it is assumed that the person H approaches the user U, but the same evacuation process is also executed when the person H passes by the user U.

[0124] <Evacuation example 2> 28A and 28B are diagrams illustrating an example of retraction when two users U1 and U2 are viewing virtual screens VD1 and VD2, respectively. (A) shows a state in which the first range RA1 of user U1 and the first range RA2 of user U2 partially overlap each other, but the gesture operations of each user are not recognized as movable objects. (B) shows a state in which the gesture operation of user U2 is recognized as a movable object within the first range RA1 on the user U1 side. In FIG. 28, parts corresponding to those in FIG. 15 are assigned the same reference numerals.

[0125] As shown in Figure 28(A), user U1 and user U2 are wearing xR devices 10A and 10B, respectively, and are viewing different virtual screens VD1 and VD2. User U1 is viewing virtual screen VD1, and user U2 is viewing virtual screen VD2. In other words, user U1 is unaware of the existence of virtual screen VD2, and user U2 is unaware of the existence of virtual screen VD1. In the case of Fig. 28(A), neither user U1 nor user U2 performs a gesture operation. Therefore, in Fig. 28(A), virtual screens VD1 and VD2 remain displayed at their initial positions. This case corresponds to the case where a negative result is obtained in step 15 (see Fig. 26).

[0126] FIG. 28B shows a state in which the right hand of the user U2 is included in the first range RA1 on the user U1 side during the gesture operation, and as a result is recognized as a movable object, and a save process of the virtual screen VD1 is executed. In Figure 28(B), the position of the virtual screen VD1 viewed by user U1 before it is retracted is indicated by a dashed line. If the virtual screen VD1 is not retracted, the ranges of movement of the right hand of user U1 and the right hand of user U2 will overlap, which could result in a collision between their hands. However, by retracting the display position of the virtual screen VD1 from its initial position before user U1 starts a gesture operation, the collision between the right hand of user U1 and the right hand of user U2 can be prevented. This case corresponds to the case where a negative result is obtained in step 42 (see Figure 26). Of course, if the gesture operation of the user U1 falls within the first range RA2 on the user U2 side, the save process for the virtual screen VD2 is executed.

[0127] <Evacuation Example 3> Figure 29 is a diagram illustrating an example of evacuation when it is possible to obtain a movement schedule for the mobile robot 30. (A) shows a state in which the robot 30 is outside the first range RA, and (B) shows a state in which the robot 30 has approached the first range RA. In Figure 29, parts corresponding to those in Figure 15 are assigned the same reference numerals. The robot 30 shown in Figure 29(A) is a printing device with a mobile function. A plurality of wheels 31A are arranged on the bottom surface of the main body 31, and a drive motor is attached to at least one of the wheels. Inside the main body 31, a mechanical section that serves as a printing engine, as well as a control board and a communication device are arranged.

[0128] A display 32 used as a user interface and a printed material outlet 33 are arranged on the top surface of the main body 31. Also, a LiDAR unit 34 used for detecting obstacles on the movement path is installed on the top of the main body 31. The processor on the control board determines the movement route based on information from the LiDAR unit 34, and moves autonomously according to the movement schedule. Note that the main body 31 manages not only the movement schedule but also the printing schedule.

[0129] The robot 30 shown in FIG. 29(A) can communicate with an information terminal 20 that is linked to the xR device 10. Therefore, the information terminal 20 can acquire a movement schedule from the robot 30. The information terminal 20 can also communicate with a server 40 that manages the movement schedule of the robot 30. Therefore, the information terminal 20 can also acquire movement schedule data from the server 40. 29(A) is the position of the robot 30 as of 10:00. According to the movement schedule, the robot 30 is to move to the vicinity of the user U at 10:16.

[0130] 29(B) shows the position of the robot 30 and the state of the evacuation process of the virtual screen VD at 10:15. According to the movement schedule, the robot 30 will move to the vicinity of the user U at 10:15, and at that time, there is a possibility that it will come close to the display position of the virtual screen VD. 29(B), the approach of the robot 30 is predicted and the virtual screen VD is evacuated. If the robot 30 enters the first range RA of the user U earlier than expected, the movement range is predicted from the movement of the robot 30, and the virtual screen VD is evacuated. This case corresponds to the case where a negative result is obtained in step 42 (see FIG. 26).

[0131] <Evacuation Example 4> 30 is a diagram illustrating exceptions to the retraction. (A) shows the display position of the virtual screen VD when the window curtain 50 is hanging down, and (B) shows the display position of the virtual screen VD when the window curtain 50 is blown by the wind and spreads to the vicinity of the virtual screen VD. In FIG. 30, the parts corresponding to those in FIG. 15 are assigned the same reference numerals. In the processing operation described with reference to FIG. 26, when an object is determined to be a movable object, a retraction operation of the virtual screen VD is executed based on the movement range predicted from the movement.

[0132] However, even if the user U touches the curtain 50 during a gesture operation on the virtual screen VD, the user U will not be injured and the curtain 50 will not be damaged. The same applies if the user touches a thin tree branch or leaf outdoors. Therefore, if an object that is an exception to evacuation is registered in advance, and a recognition process of an image captured by camera 105 (see Figure 3) identifies a movable object as an exception, the evacuation function of the virtual screen VD is disabled. In FIG. 30(B), the movable object is the exceptional curtain 50, so even if the curtain 50 approaches the periphery of the virtual screen VD, the display position of the virtual screen VD remains the same.

[0133] <Fourth Embodiment> In this embodiment, a function specific to the case where a plurality of virtual screens VD are arranged in the depth direction as viewed from the user U will be described. 31 is a flowchart illustrating an example of the operation of saving the virtual screen VD in the embodiment 4. In FIG. 31, parts corresponding to those in FIG. 14 are denoted by the same reference numerals. In this embodiment as well, the processor 201 (see FIG. 5) determines whether the collision avoidance function is on (step 11).

[0134] If a positive result is obtained in step 11, the processor 201 determines whether or not a plurality of virtual screens VD are arranged in the depth direction as viewed from the user U (step 51). In this embodiment, "arranged in the depth direction" refers to a state in which the virtual screens VD are lined up in a line. Therefore, the virtual screens VD do not need to have the same shape, and the distance between the virtual screens VD does not need to be the same. If a negative result is obtained in step 51, the processor 201 executes the processing from step 12 (see FIG. 14) onwards. That is, it determines whether the initial position of the virtual screen VD is within the first range RA, whether an object RO is within the first range RA, etc., and saves the virtual screen VD as necessary. If a negative result is obtained in step 51, for example, there may be only one virtual screen VD or multiple virtual screens VD arranged in parallel.

[0135] If a positive result is obtained in step 51, the processor 201 sets the virtual screen VD located closest to the user as the target for collision avoidance (step 52), and executes the processing from step 12 (see FIG. 14) onwards for the set virtual screen VD. This is because the user U performs gesture operations on the virtual screen VD located at the front, and therefore there is no need to determine the positional relationship between the virtual screens VD in the second and subsequent rows and the surrounding objects RO. Furthermore, by not considering the positional relationship between the virtual screens VD arranged in the second and subsequent rows and the surrounding objects RO, a reduction in the calculation load is achieved.

[0136] <Example of evacuation> 32 is a diagram illustrating the retraction operation when two virtual screens VD1 and VD2 are arranged in the rearward direction as viewed from the user U. (A1) and (A2) show the positional relationship between the two virtual screens VD1 and VD2 and the user U at their initial positions, and (B1) and (B2) show the state in which only the front virtual screen VD1 has been retracted. In FIG. 32, parts corresponding to those in FIG. 15 are assigned the same reference numerals.

[0137] In the cases of FIGS. 32(A1) and (A2), the virtual screen VD1 is closer to the user U than the drink 5 by a distance L2, and the virtual screen VD2 overlaps with the drink 5. If the function of this embodiment is not provided, a process for saving the drink 5 would be required for each of the virtual screens VD1 and VD2. However, in this embodiment, when it is recognized that the two virtual screens VD1 and VD2 are aligned in a row, the target of the save is limited to only the virtual screen VD1 located on the front side.

[0138] 32(B1) and (B2), the virtual screen VD2 located at the back overlaps with the drink 5, while the virtual screen VD1 located at the front is retreated a first distance LT1 from the drink 5. This allows the user U to perform gesture operations on the virtual screen VD1 without worrying about colliding with the drink 5. The same applies when three or more virtual screens VD are arranged.

[0139] <Fifth Embodiment> In this embodiment, a function executed when a virtual screen VD located in the second or subsequent row is specified when a plurality of virtual screens VD are arranged in a row toward the rear as seen from the user U will be described. Fig. 33 is a flowchart illustrating an example of the operation of saving the virtual screen VD in embodiment 5. In Fig. 33, parts corresponding to those in Fig. 31 are assigned the same reference numerals. In this embodiment as well, the processor 201 (see FIG. 5) determines whether the collision avoidance function is on (step 11).

[0140] If a positive result is obtained in step 11, the processor 201 determines whether or not a plurality of virtual screens VD are arranged in the depth direction as viewed from the user U (step 51). If a negative result is obtained in step 51, processor 201 executes the processing from step 12 (see FIG. 14) onwards. On the other hand, if a positive result is obtained in step 51, the processor 201 determines whether or not the virtual screens VD in the second or subsequent columns have been designated as the target of operation (step 61). In this embodiment, the virtual screen VD is designated by a gesture operation using, for example, a cursor proposed by Microsoft (registered trademark) for the virtual screen VD.

[0141] 34 illustrates an example of a cursor 60 used to operate the virtual screen VD. (A) shows the cursor 60 when the fingertip of the right hand 7 is far from the virtual screen VD, (B) shows the cursor 60 when the fingertip of the right hand 7 is close to the virtual screen VD, and (C) shows the cursor 60 when the fingertip of the right hand 7 is in contact with the virtual screen VD. The distance between the virtual screen VD and the fingertips of the right hand 7 is measured using a positioning sensor 104 (see FIG. 3) and a camera 105 (see FIG. 3). By performing a recognition process on the image captured by the camera 105, the processor 201 (see FIG. 3) identifies the position of the fingertips of the user U from a three-dimensional map generated using the measurement results of the positioning sensor 104.

[0142] Of course, the display position of the virtual screen VD is not measured using the positioning sensor 104, nor is it captured using the camera 105. However, the processor 201 recognizes the display position of the virtual screen VD within the three-dimensional map, and therefore recognizes the positional relationship between the fingertip and the virtual screen VD, and reflects the recognition result in the display position and size of the cursor 60. In the case of Figure 34(A), the fingertips of the right hand 7 are farther from the virtual screen VD, so the diameter of the cursor 60 is larger than the other two. In Figure 34(A), an enlarged view of the vicinity of the fingertips is shown within a circle.

[0143] In FIG. 34(B), the state in which the fingertips of the right hand 7 approach the virtual screen VD is represented by the diameter of the cursor 60 being reduced and the brightness of the cursor 60 being increased. In FIG. 34(C), the state in which the fingertips of the right hand 7 are in contact with the virtual screen VD is represented by a further reduction in the diameter of the cursor 60 and an increase in the brightness of the cursor 60. The user U can recognize which virtual screen VD his / her gesture operation is associated with based on the display position of the cursor 60, the diameter of the cursor 60, and the like.

[0144] Returning to the explanation of Figure 33. If a positive result is obtained in step 61, the processor 201 replaces the virtual screen VD designated as the target of the operation with another virtual screen VD located in the foreground (step 62). After this, or if a negative result is obtained in step 61, the processor 201 sets the virtual screen VD located closest to the user as the target for collision avoidance (step 63), and executes the processing from step 12 (see FIG. 14) onwards.

[0145] <Example of evacuation> 35 is a diagram illustrating an example in which the virtual screen VD2 located at the far side of the two virtual screens VD1 and VD2 is retracted in a direction approaching the user U from the object RO. (A1) and (A2) show the positional relationship between the two virtual screens VD1 and VD2 and the user U at their initial positions, and (B1) and (B2) show the positional relationship between the two virtual screens VD1 and VD2 and the user U after the retraction process has been executed. In FIG. 35, parts corresponding to those in FIG. 15 are assigned the same reference numerals. 35(A1) and (A2), the user U brings the fingertips of the right hand 7 closer to the virtual screen VD2 located at the back. When the distance between the fingertips and the virtual screen VD2 becomes equal to or less than a predetermined distance, the processor 201 recognizes that the virtual screen VD2 located at the back has been designated.

[0146] In Figures 35(B1) and (B2), the virtual screen VD1, which was initially positioned at the front, has moved to the back, and the virtual screen VD2, which was initially positioned at the back, has moved to the front. In other words, the order of the screens has been swapped from the initial position. Then, a retraction process is performed so that only the virtual screen VD2 that has been moved to the front is moved a first distance LT1 away from the drink 5. Incidentally, the display position of the virtual screen VD1, which is positioned at the back after the swap, remains in its initial position. By employing the function described in this embodiment, the possibility of a collision with an object RO such as a drink 5 present in the vicinity during a gesture operation on the specified virtual screen VD2 is reduced. Also, the calculation load required for the save process is reduced.

[0147] <Sixth Embodiment> In this embodiment, a function executed when a newly displayed virtual screen VD is added to an existing virtual screen will be described. 36 is a flowchart illustrating an example of the operation of saving the virtual screen VD in the embodiment 6. In Fig. 36, the parts corresponding to those in Fig. 14 are assigned the same reference numerals. In this embodiment as well, the processor 201 (see FIG. 5) determines whether the collision avoidance function is on (step 11).

[0148] If a positive result is obtained in step 11, the processor 201 determines whether or not there is an instruction to add a virtual screen VD (step 71). An instruction to add includes, for example, an instruction to display a specific screen by voice, or the acceptance of an operation that involves the display of another window such as a pop-up screen or a dialog box. If a negative result is obtained in step 71, processor 201 executes the processing operations from step 12 (see FIG. 14) onwards. If a positive result is obtained in step 71, the processor 201 determines whether the total number of virtual screens VD after the addition is plural or not (step 72).

[0149] For example, if an instruction to display the virtual screen VD is issued when no virtual screen VD is currently being displayed, the processor 201 obtains a negative result in step 72. Even if a negative result is obtained in step 72, the processor 201 executes the processing operations from step 12 (see FIG. 14) onwards. If a positive result is obtained in step 72, the processor 201 places the virtual screen VD to be added in the forefront and sets it as a target for collision avoidance (step 73). In this embodiment, the virtual screen VD to be added is placed in front of the virtual screen VD currently being displayed. Therefore, the display position of the virtual screen VD currently being displayed is adjusted based on the display position of the newly added virtual screen VD. After step 73, the processor 201 executes the processing operations from step 12 (see FIG. 14) onwards.

[0150] <Example of evacuation> 37 is a diagram illustrating an example of saving when an instruction to add a third virtual screen VD3 is given while two virtual screens VD1 and VD2 are being displayed. (A1) and (A2) show the positional relationship between the two virtual screens VD1 and VD2 and the user U at their initial positions, and (B1) and (B2) show the positional relationship between the three virtual screens VD1, VD2, and VD3 and the user U after the saving process has been executed. In FIG. 37, parts corresponding to those in FIG. 15 are assigned the same reference numerals.

[0151] 37(A1) and 37(A2), two virtual screens VD1 and VD2 are displayed in front of the user U. Incidentally, the display position of the virtual screen VD1 located on the near side is retreated toward the user U by a first distance LT1 from the drink 5. 37(A1) and (A2), the user U commands the xR device 10 by voice, "Display the virtual screen VD3." The virtual screen VD3 here is different from either the virtual screen VD1 or VD2 currently being displayed. Therefore, the voice command here is determined to be a command to add a virtual screen VD.

[0152] Upon receiving the instruction, the xR device 10 places the third virtual screen VD3 in the virtual space. 37(B1) and (B2) show examples of layouts in the virtual space. In the cases of Fig. 37(B1) and (B2), the virtual screen VD3 located at the forefront is displayed at a position retreated from the drink 5 toward the user U by a first distance LT1. However, this position is the position where the virtual screen VD1 was displayed before the additional instruction was received.

[0153] Therefore, the display position of the existing virtual screen VD1 is adjusted to any position between the virtual screen VD1 at the initial position and the virtual screen VD2. Even if the distance between the display positions of the virtual screens VD1 and VD2 on the second and subsequent rows and the drink 5 is less than the first distance LT1, there is no risk of a collision during a gesture operation. This is because the user U makes the virtual screen VD3 located in the foreground the target of the gesture operation, and if the virtual screen VD1 or the virtual screen VD2 is the target of the operation, the display positions are swapped.

[0154] <Seventh Embodiment> In this embodiment, a case will be described in which a single virtual screen VD is shared by a plurality of people. 38 is a flowchart illustrating an example of the operation of saving the virtual screen VD in the seventh embodiment. In FIG. 38, the parts corresponding to those in FIG. 14 are denoted by the same reference numerals. In this embodiment as well, the processor 201 (see FIG. 5) determines whether the collision avoidance function is on (step 11).

[0155] If a positive result is obtained in step 11, the processor 201 determines whether the virtual screen VD is being shared with another device (step 81). In this embodiment, sharing the virtual screen VD does not mean that multiple people can view the same virtual screen VD, but rather means that the display positions of the virtual screen VD are linked as if a single display exists in real space. Therefore, if the common virtual screen VD is being viewed in a completely different location, a negative result will be obtained in step 81 .

[0156] If a negative result is obtained in step 81, processor 201 executes the processing from step 12 (see FIG. 14) onwards. If a positive result is obtained in step 81, the processor 201 temporarily turns off the save function while the virtual screen VD is being shared (step 82). That is, exception processing of the save processing is executed. 39 is a diagram illustrating a case where one virtual screen VD is shared by multiple people. (A) shows the state before two users U1 and U2 perform gesture operations, and (B) shows the state after two users U1 and U2 start performing gesture operations. In FIG. 39, parts corresponding to those in FIG. 1 are assigned the same reference numerals.

[0157] In Figure 39(A), users U1 and U2 share one virtual screen VD. However, neither of them performs a gesture operation on the virtual screen VD. Therefore, neither the xR device 10A worn by user U1 nor the xR device 10B worn by user U2 detects an object RO that should be removed from the periphery of the virtual screen VD. 39(B), both user U1 and user U2 have started gesture operations. In this case, the right hand 7 of user U2 seen from user U1's side is recognized as an object RO existing within the first range RA. Similarly, the right hand 7 of user U1 seen from user U2's side is recognized as an object RO existing within the first range RA.

[0158] In the other embodiments described above, in this case, the display position of the virtual screen VD viewed by user U1 would be retracted in a direction away from the right hand 7 of user U2, and the display position of the virtual screen VD viewed by user U2 would be retracted in a direction away from the right hand 7 of user U1. Retraction in different directions for each user is technically possible because the virtual screen VD does not actually exist. However, in the example of Figure 39, since the virtual screen VD is shared by users U1 and U2, the save function is forcibly controlled to be turned off. This allows users U1 and U2 to carry on a conversation or discussion while visually checking where the other is pointing. In other words, although there is no physical display in the real space, multiple people can share the same experience.

[0159] <Embodiment 8> In this embodiment, a method for notifying in advance the presence of an object RO that requires caution when performing a gesture operation will be described. 40A and 40B are diagrams illustrating an example of a method for notifying a user U viewing a virtual screen VD of the presence of an object RO. (A) shows an example of notification by sound, and (B) shows an example of notification by a synthetic image.

[0160] In the case of FIG. 40(A), the user U is notified by voice guidance such as "Pay attention to the glass behind the image" that a drink 5 is present behind the virtual screen VD. This voice guidance is executed regardless of the execution of the save process. For example, the voice guidance is output in combination with the save process of the virtual screen VD. However, even if the distance between the virtual screen VD and the drink 5 is less than the first distance LT1, the voice guidance may be output without executing the save process.

[0161] In the case of FIG. 40(B), an image of the drink 5 captured by the camera 105 (see FIG. 2) is superimposed on the virtual screen VD. When superimposing, it is possible to superimpose the image as it was captured by the camera 105, but it is also possible to superimpose an image that has been processed to the extent that the presence of the drink 5 is apparent. For example, an image in which the outline of the drink 5 has been extracted may be superimposed on the virtual screen VD, or the transmittance of the image of the drink 5 may be increased and then superimposed on the virtual screen VD. Of course, these are just examples of processing.

[0162] Fig. 41 is a diagram illustrating an example of another method for notifying a user U viewing a virtual screen VD of the presence of an object RO. (A1) and (A2) show a method using deformation of the virtual screen VD, and (B1) and (B2) show a method of synthesizing an image that draws attention to the virtual screen VD. In Fig. 41, parts corresponding to those in Fig. 15 are assigned the same reference numerals. In Fig. 41 (A1), the right hand 7 of the user U is away from the surface of the virtual screen VD. Therefore, the virtual screen VD remains in its initial state. On the other hand, in Fig. 41 (A2), the right hand 7 of the user U is approaching the surface of the virtual screen VD. At this time, when the distance between the right hand 7 and the virtual screen VD becomes equal to or less than a predetermined threshold, the virtual screen VD deforms from its initial state, causing the user U to feel something strange or uncomfortable.

[0163] 41(A2) is just one example, and it is also possible to deform the virtual screen VD into a shape that is split into left and right halves. In this case, if the object RO hidden behind the virtual screen VD can be seen through the split, the user U will not have to knock the object RO down with his or her right hand 7. Of course, if the user U becomes aware of the presence of the object RO behind the virtual screen VD in advance due to the deformation of the virtual screen VD, a collision with the object RO can be avoided. In the case of FIG. 41(B1), the right hand 7 of the user U is also away from the surface of the virtual screen VD. Therefore, the virtual screen VD remains in its initial state. On the other hand, in FIG. 41(B2), the right hand 7 of the user U is approaching the surface of the virtual screen VD. However, in the case of FIG. 41(B2), a picture or image representing danger is superimposed in the center of the virtual screen VD. In this case too, the user U can be made aware in advance of the danger of touching the virtual screen VD.

[0164] <Ninth Embodiment> In this embodiment, a function of changing or adjusting the threshold value or the like that defines the first range RA based on the measurement history of gesture operations on the virtual screen VD by the user U will be described. Fig. 42 is a diagram illustrating adjustment of a threshold value that defines the first range RA based on a history of measurements of the range of motion of the right hand 7 during gesture manipulation on the virtual screen VD. (A) shows the first range RA at the time of initial setting, and (B) shows the first range RA adjusted according to the range of motion of the gesture manipulation of the user U. In Fig. 42, parts corresponding to those in Fig. 6 are assigned the same reference numerals. Note that Figs. 42(A) and (B) are both views of the user U as seen from above.

[0165] In the case of FIG. 42(A), the first range RA is symmetrical with respect to the front direction of the user U. The xR device 10 (see FIG. 1) is equipped with a positioning sensor 104 and a camera 105, and constantly measures the distance to an object RO present around the user U. The objects of positioning at this time include body parts of the user U wearing the xR device 10 while performing gesture operations. Therefore, it is possible to continuously measure the range of motion of a body part or the like accompanying a gesture operation and accumulate the measurement history. Note that it is possible to identify the body part or the like of the user U by image processing using the camera 105.

[0166] The first range RA shown in Figure 42(B) is a first range RA that has been changed or adjusted using the results of analyzing the history of accumulated measurement data. In this example, the left front part and the right side part of the user U are reduced. As a result, the area of ​​the horizontal plane of the changed first range RA is reduced to about half of its initial state. When the area of ​​the horizontal plane of the first range RA, in other words, the three-dimensional volume, is reduced, the range of positioning by the positioning sensor 104 becomes narrower, which reduces the calculation load and is expected to extend the usage time. The change or adjustment of the first range RA for the measurement history is not limited to within a horizontal plane, but is performed for a range in the height direction in any direction. That is, the change or adjustment of the first range RA is performed three-dimensionally.

[0167] However, the change or adjustment of the first range RA is not limited to the case where the area in the horizontal plane is reduced compared to the initial state. For example, if the position on the virtual screen VD where a gesture operation is accepted is outside the initially set first range RA, the first range RA may be expanded. This expansion, change, or adjustment of the first range RA may be performed if acceptance of an out-of-range position is detected at least once, or may be performed on the condition that the same point is detected multiple times.

[0168] Furthermore, a collision between a body part or the like of the user U and the object RO may be detected during the gesture operation. In this case, since the distance to the object RO before the collision is measured, it is possible to estimate that the object RO that collided is included in the range of motion of the body part or the like of the user U. If the shape of the first range RA is changed or adjusted using information on the position where the collision with the object RO is detected, it becomes possible to subsequently move away from the virtual screen VD.

[0169] <Tenth Embodiment> In this embodiment, a function for retracting only the icons in the virtual screen VD from the surrounding object RO will be described. 43 is a diagram illustrating an example of saving the virtual screen VD in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U after saving. In FIG. 43, parts corresponding to those in FIG. 15 are assigned the same reference numerals.

[0170] 43(A1) and (A2) includes the icon 70 that is the target of the gesture operation. In the case of the above-described embodiment, the saving of the drink 5 is performed on the entire virtual screen VD. However, in the gesture operation, basically, no part other than the icon 70 is tapped. For this reason, in the present embodiment, only the icon 70 is separated from other areas of the virtual screen VD, or the virtual screen VD of the icon 70, which is a copy of the icon 70, is moved away from the drink 5 by a first distance LT1.

[0171] In the examples shown in FIGS. 43(B1) and (B2), the portion of the icon 70 is cut out from the virtual screen VD and moved away by a first distance LT1 in a direction approaching the user U. In this case, the user U performs a gesture operation on the icon 70 that appears on the virtual screen VD, so the possibility of a body part or the like colliding with the drink 5 is reduced, as in the other embodiments. Furthermore, moving the entire virtual screen VD can make the user U feel very uncomfortable, but moving only the icon 70 has the effect of making the gesture operation stand out, so the user U can feel less uncomfortable.

[0172] <Evacuation example 2> 44 is a diagram illustrating another example of saving the virtual screen VD in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U after saving. In FIG. 44, parts corresponding to those in FIG. 15 are assigned the same reference numerals. 44, the virtual screen VD is a diagram of the heart, and the target of the gesture operation by the user U is an artery.

[0173] Therefore, in Figures 44 (B1) and (B2), the portions corresponding to the aorta 75A, the left main coronary artery 75B, the left circumflex artery 75C, the left anterior descending artery 75D, and the right coronary artery 75E, which are the targets of the gesture operation, are cut out from other areas of the virtual screen VD representing the heart and are retracted by a first distance LT1 in a direction closer to the user U. In this case, too, the drink 5 behind the virtual screen VD can be prevented from being knocked over when performing the gesture operation. In addition, the artery portion is closer to the user U, making it easier to check the details that are the target of the gesture operation.

[0174] The arteries in this embodiment are an example of icons. Instead of retracting all of the arteries, only the part of the artery where the cursor 60 (see FIG. 34) is located may be retracted. This is because the artery where the cursor 60 is located is likely to be the target of a gesture operation, while other parts are likely not to be the target of a gesture operation.

[0175] <Evacuation Example 3> 45 is a diagram illustrating another example of saving the virtual screen VD in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U after saving. In FIG. 45, parts corresponding to those in FIG. 15 are assigned the same reference numerals. 45(A1) and (A2) show the virtual screen VD in the initial position, which is located immediately in front of the drink 5. If a gesture operation is performed in this state, there is a possibility that the drink 5 will be knocked over with a body part or the like.

[0176] However, on the virtual screen VD, a cursor 60 is displayed at a position pointed to by the fingertip of the right hand 7 of the user U. The distance between the virtual screen VD and the fingertip of the right hand 7 is expressed by a change in the dimensions of the cursor 60. 45(A1), the cursor 60 is positioned on the front wheel. When the diameter of the cursor 60 becomes smaller than a predetermined threshold, the processor 201 cuts out the front wheel as an independent object and sets it as a new virtual screen VD2 to be subject to the saving process.

[0177] As a result, as shown in Figures 45(B1) and (B2), the virtual screen VD2 corresponding to the front wheel portion is retracted to a position spaced a first distance LT1 from the drink 5. This is the same as the artery in Figure 44. However, in the case of Figure 45(B1), the object corresponding to the front wheel is displayed in an enlarged manner, which adds a visual effect that makes it easier to observe the retracted object, unlike simple retraction.

[0178] <Evacuation Example 4> 46 is a diagram illustrating another example of saving the virtual screen VD in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U at the initial position, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U after saving. In FIG. 46, parts corresponding to those in FIG. 15 are assigned the same reference numerals. In the case of Fig. 46, the virtual screen VD is a reduced shape of an airplane. That is, the virtual screen VD is a three-dimensional shape. For this reason, in Fig. 46(A2), the dashed line indicating the virtual screen VD is represented as a rectangle rather than a straight line.

[0179] In FIG. 46(A2), the portion of the three-dimensional virtual screen VD closest to the user U is separated from the drink 5 by a first distance LT1. This is based on the idea that the part closest to the user U is basically the target of the gesture operation. In FIG. 46(A1), the fingertips of the right hand 7 are touching the tip of the left wing of the airplane to instruct rotation within a horizontal plane. In this gesture operation, the fingertips of the user U move in a space separated from the drink 5 by the first distance LT1. Therefore, there is little chance that the drink 5 will be knocked over by the gesture operation.

[0180] 46 (B1) and (B2) show a state in which the rotation of the airplane has finished and the nose of the airplane is facing the user U. In this case, the user U gestures to the nose of the airplane, which is located directly in front of the user U and is closest to the user U, so that the first distance LT1 can be secured between the user U and the drink 5. As shown in Figure 46 (B2), the fuselage part of the airplane that constitutes the virtual screen VD includes the drink 5, but when performing gesture operations on the fuselage part or tail, it is possible to avoid collision with the drink 5 during gesture operations by changing the orientation of the airplane so that they are positioned in front of the user U.

[0181] <Evacuation Example 5> 47 is a diagram illustrating another example of saving the virtual screen VD in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen VD before reduction conversion and the user U, and (B1) and (B2) show the positional relationship between the virtual screen VD after reduction conversion and the user U. In FIG. 47, parts corresponding to those in FIG. 15 are assigned the same reference numerals. In the case of Fig. 47, the virtual screen VD is also a reduced shape of an aircraft. That is, the virtual screen VD is a three-dimensional shape. For this reason, in Fig. 47(A2), the dashed line indicating the virtual screen VD is represented as a rectangle rather than a straight line.

[0182] 47(A1) and (A2) show cases where the display dimensions of the virtual screen VD are too large. If the display dimensions are too large, it can actually make gesture operations inconvenient. Therefore, in FIG. 47(A1), the user U instructs the reduction of the display dimensions of the virtual screen VD by a gesture operation of bringing both hands closer together. In FIG. 47(A2), the virtual screen VD has not yet been reduced, so the range of the dashed line representing the virtual screen VD is large. Incidentally, the frontmost position on the virtual screen VD is separated from the drink 5 by a first distance LT1. Figure 47(B1) shows how the virtual screen VD looks after the reduction conversion. Because it has been reduced, the area of ​​the dashed line representing the virtual screen VD in Figure 47(B2) is smaller. In this case, too, the frontmost position of the virtual screen VD is separated from the drink 5 by the first distance LT1.

[0183] <Evacuation Example 6> 48 is a diagram illustrating another example of saving the virtual screen VD in embodiment 10. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U before enlargement conversion, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U after enlargement conversion. In FIG. 48, parts corresponding to those in FIG. 15 are assigned the same reference numerals. In the case of Fig. 48, the virtual screen VD is also a reduced shape of the aircraft. That is, the virtual screen VD is a three-dimensional shape. For this reason, in Fig. 48(A2), the dashed line indicating the virtual screen VD is represented as a rectangle rather than a straight line.

[0184] 48(A1) and (A2) show cases where the display dimensions of the virtual screen VD are too small. If the display dimensions are too small, it can be inconvenient to perform gesture operations. Therefore, in FIG. 48(A1), the user U instructs the enlargement of the display dimensions of the virtual screen VD by performing a gesture operation of spreading both arms. Note that in FIG. 48(A2), the virtual screen VD has not yet been enlarged, so the range of the dashed line representing the virtual screen VD is small. Incidentally, the frontmost position on the virtual screen VD is separated from the drink 5 by a first distance LT1. Figure 48 (B1) shows how the virtual screen VD looks after enlargement. Because it has been enlarged, the area of ​​the dashed line representing the virtual screen VD in Figure 48 (B2) is larger. In this case, too, the frontmost position of the virtual screen VD is separated from the drink 5 by the first distance LT1.

[0185] <Embodiment 11> In this embodiment, a case will be described in which a save function is added in association with the virtual screen VD and an actual monitor. 49 is a diagram illustrating a case where the virtual screen VD viewed by the user is linked to the display screen of the actual monitor 80. (A) shows the display screen of the virtual screen VD and monitor 80 before the linkage, (B) shows the display screen of the virtual screen VD and monitor 80 immediately after the linkage, and (C) shows the state in which the virtual screen VD and monitor 80 are arranged in parallel due to the retraction of the virtual screen VD. The monitor 80 here is an example of a display device.

[0186] In Figure 49(A), the distance between the user U and the monitor 80 is great. For this reason, there is no cooperation between the xR device 10 and the computer that displays the image on the monitor 80. For this reason, the contents of the virtual screen VD and the screen displayed on the monitor 80 are independent of each other. The monitor 80 in FIG. 49(A) is a display of a notebook computer. 49(B), the user U approaches the monitor 80 and is positioned substantially in front of the monitor 80. Therefore, the virtual screen VD is displayed on the front side of the monitor 80 as seen from the user U.

[0187] When the distance between the xR device 10 and the monitor 80 becomes less than a predetermined distance, image linkage is performed between the xR device 10 and the computer that displays the image on the monitor 80. Figure 49(B) shows the screen after image linkage, so the content displayed on the monitor 80 has switched to a linked screen with the virtual screen VD. As can be seen from FIG. 49(B), even if the screens are linked, when the two screens overlap, the visibility of the monitor 80 decreases and there is a concern that the monitor 80 may come into contact with the screen when performing a gesture operation.

[0188] In this embodiment, the direction of retraction, on the condition that the virtual screen VD is positioned in front of the monitor 80 by a first distance LT1, is set to the direction in which the monitors 80 are arranged. Therefore, the virtual screen VD in FIG. 49(C) is retracted to a position immediately to the left of the monitor 80 as seen from the user U. This allows the user U to work while viewing two screens. After image linkage begins, the position of the virtual screen VD is fixed at the retracted position, even if the user U turns his / her face toward the monitor 80, for example. This prevents the display position of the virtual screen VD from fluctuating every time the user U turns his / her face, realizing a stable work environment.

[0189] <Embodiment 12> 50 is a diagram illustrating a function for expanding the shape of the first range RA in the direction in which the icons included in the virtual screen VD are located. (A1) and (A2) show the first range RA when the virtual screen VD does not include the icon 70, and (B1) and (B2) show the first range RA when the virtual screen VD includes the icon 70. In FIG. 50, parts corresponding to those in FIG. 15 are assigned the same reference numerals. 50(A2) and (B2) are views of the user U seen from above.

[0190] The virtual screen VD shown in FIG. 50(A1) does not include the icon 70. Therefore, the first area RA shown in FIG. 50(A2) remains in its initial fan shape. In the case of FIG. 50(A2), the object RO is located outside the first area RA. In this embodiment, the retraction of the virtual screen VD is performed for the object RO included in the first area RA. Therefore, in the case of FIG. 50(A2), even if the virtual screen VD and the object RO are close to each other, the retraction process is not performed. As a result, there is a possibility that a body part or the like may collide with the object RO during a gesture operation.

[0191] The virtual screen VD shown in Fig. 50(B1) includes an icon 70, and a first range RA is expanded in the direction in which the icon 70 exists. In Fig. 50(B2), the object RO is included in the expanded first range RA. Therefore, the virtual screen VD is retracted by a first distance LT1 from the object RO. As a result of the virtual screen VD being retracted, it is possible to reduce the possibility of accidentally colliding a body part or the like with the object RO when performing a gesture operation on the icon 70.

[0192] <Embodiment 13> Fig. 51 is a diagram illustrating a function for expanding the first range RA in the direction of the line of sight of the user U. (A1) and (A2) show the shape of the first range RA when the line of sight of the user U is diagonally forward to the right, and (B1) and (B2) show the shape of the first range RA when the line of sight of the user U is diagonally forward to the left. In Fig. 51, parts corresponding to those in Fig. 15 are assigned the same reference numerals. 51(A2) and (B2) are views of the user U seen from above. This is because the gesture operation is likely to be performed in the area in the direction of the user U's line of sight.

[0193] In the present embodiment, the direction of the line of sight of the user U is estimated using the electrooculogram measured by the xR device 10C (see FIG. 52). Fig. 52 is a diagram illustrating an example of the appearance of the xR device 10C used in embodiment 13. (A) shows an example of the appearance when observing the xR device 10C from the front side, and (B) shows an example of the appearance when observing the xR device from the user U side. In Fig. 52, parts corresponding to those in Fig. 2 are assigned the same reference numerals.

[0194] The external configuration shown in FIG. 52(A) is the same as the external configuration shown in FIG. 2. In the case of FIG. 52(B), a left electrode 121 is provided on the left nose pad of the xR device 10C, a right electrode 122 is provided on the right nose pad, and a central electrode 123 is provided in contact with the skin between the eyebrows. The left electrode 121 contacts the left side of the nose of a user U wearing the xR device 10C. The right electrode 122 contacts the right side of the nose of a user U wearing the xR device 10C. The distance from the left eye of the user U to the left electrode 121 is substantially equal to the distance from the right eye of the user U to the right electrode 122. The central electrode 123 is provided near the center between the left electrode 121 and the right electrode 122.

[0195] Fig. 53 is a diagram illustrating an example of the hardware configuration of an xR device 10C used in embodiment 13. In Fig. 53, parts corresponding to those in Fig. 3 are assigned the same reference numerals. The xR device 10C shown in FIG. 53 differs from the hardware configuration described in FIG. 3 in that the left electrode 121, the right electrode 122, and the center electrode 123 are connected to the processor 107. The left electrode 121, the right electrode 122, and the center electrode 123 measure the electrooculogram corresponding to the left and right eye movements.

[0196] 54 is a diagram illustrating the configuration of a functional unit related to gaze detection. In this embodiment, the processor 201 of the information terminal 20 that cooperates with the xR device 10C executes gaze detection. The processor 201 realizes functions related to line of sight detection through the execution of a program. The electrooculography measuring unit 131 receives the potential value measured at the left electrode 121 , the potential value measured at the right electrode 122 , and the potential value measured at the center electrode 123 . For example, the electro-oculography measuring unit 131 acquires the potential value of the left electrode 121 based on the potential value of the central electrode 123 as the left electro-oculography, and acquires the potential value of the right electrode 122 based on the potential value of the central electrode 123 as the right electro-oculography. The electro-oculography measuring unit 131 provides these left and right electro-oculography as time-series data to the eye movement detecting unit 132 and the blink detecting unit 133.

[0197] The eye movement detection unit 132 detects the eye movement of the user based on the time series data of the left and right electro-oculograms acquired from the electro-oculogram measurement unit 131. The eye movement detection unit 132 identifies the direction of the eye, i.e., the direction of the gaze, based on a characteristic waveform that appears in the time series data of the left and right electro-oculograms. The blink detection unit 133 identifies the period of a blink based on the time-series data of the left and right electrooculograms. The calibration value calculation unit 134 calculates a calibration value of the eye movement based on the value of the electro-oculography accompanying the blink detected by the blink detection unit 133. The calculated calibration value is provided to the gaze direction determination unit 135. The gaze direction determination unit 135 calibrates the gaze direction detected by the eye movement detection unit 132 with a calibration value and outputs the calibrated gaze direction. This type of technology is described in, for example, JP 2018-75254 A.

[0198] <Embodiment 14> In this embodiment, a function will be described in which a gesture operation by the user U is predicted using an electroencephalogram signal measured in the ear canal of the user U, and this function is used to help move the virtual screen VD. 55 is a diagram illustrating an example of the appearance of the xR device 10D used in embodiment 14. (A) shows an example of the appearance when observing the xR device 10D from the front side, (B1) and (B2) show examples of the configuration of the earphone tip 14L attached to the temple 13L, and (C1) and (C2) show examples of the configuration of the earphone tip 14R attached to the temple 13R. In FIG. 55, parts corresponding to those in FIG. 2 are denoted by the same reference numerals.

[0199] The earphone tips 14L, 14R shown in Figure 55(A) are roughly cylindrical components that are inserted into the ear canal. A through-hole is formed near the center axis of the cylindrical earphone tips 14L, 14R. This allows the user to hear external sounds even when the earphone tips 14L, 14R are inserted into the ear canal. In other words, they are open-type components that do not block the ear canal. The earphone tip 14L shown in Figures 55(B1) and (B2) is equipped with a dome-shaped electrode 14L1 that contacts the inside of the external ear canal and a ring-shaped electrode 14L2 that contacts the cavity of the concha. Electrodes 14L1, 14L2, and 14R1 are all made of conductive rubber. This is to measure the electrical signals that appear on the skin. Electrodes 14L1 and 14L2 are electrically isolated by an insulator.

[0200] The electrode 14L1 is a terminal used to measure potential fluctuations caused by electroencephalograms (EEG) (hereinafter referred to as "EEG measurement terminal"), and the electrode 14L2 is a terminal used to measure ground potentials (hereinafter also referred to as "GND terminal"). Meanwhile, the earphone tip 14R is provided with a dome-shaped electrode 14R1 that contacts the inside of the ear canal. The electrode 14R1 is a terminal (hereinafter referred to as the "REF terminal") used to measure a reference potential (REF). In this embodiment, the electrodes 14R1 and 14L1 are electrically shorted.

[0201] The potential fluctuations caused by the electroencephalogram are measured as a differential signal between the electrical signals measured at electrode 14L1 and electrode 14L2. In the field of neuroscience, all potential fluctuations that originate other than brain waves are called artifacts. In the field of neuroscience, it is believed that electrical signals measuring brain waves always contain artifacts. In this embodiment, the source of the potential fluctuations is not distinguished, and the potential fluctuations measured by earphone tips 14L and 14R are referred to as electrical signals measuring brain waves.

[0202] The components contained in artifacts are classified into components derived from the living body, components derived from the measurement system such as electrodes, and components derived from external machines and the environment. Of these three components, components other than those derived from the living body can be measured as noise measured by earphone tips 14L and 14R. Noise can be measured as an electrical signal when electrodes 14R1 and 14L1 are electrically shorted. The fact that brain waves can be measured using earphone tips 14L and 14R is described, for example, in Patent Publication No. 2021-90136.

[0203] Fig. 56 is a diagram illustrating an example of the hardware configuration of an xR device 10D used in embodiment 14. In Fig. 56, parts corresponding to those in Fig. 3 are assigned the same reference numerals. In the case of the xR device 10D shown in FIG. 56, electrical signals measured by the electrodes 14L1, 14L2, and 14R1 provided on the earphone tips 14L and 14R are provided to the processor 107. In this embodiment, the electrical signals resulting from brain waves are measured using earphone tips 14L and 14R, but it is also possible to substitute earphone tips 14L and 14R by placing electrodes corresponding to electrodes 14L1, 14L2, and 14R1 on the inside of temples 13L and 13R that come into contact with the temporal region near the ear.

[0204] 57 is a diagram illustrating the configuration of a functional unit that detects a sign of a gesture operation using an electroencephalogram. In this embodiment, the processor 201 of the information terminal 20 that cooperates with the xR device 10D performs gaze detection. The processor 201 realizes a function of detecting a sign of a gesture operation through the execution of a program.

[0205] The electroencephalogram measuring section 141 receives as input the potential values ​​measured at the electrodes 14L1 and 14L2 of the earphone tip 14L and the potential value measured at the electrode 14R1 of the earphone tip 14R. The electroencephalogram measurement unit 141 measures an electrical signal corresponding to an electroencephalogram from the potential difference between the electrodes 14L1 and 14L2, while measuring a noise signal from the potential difference between the electrodes 14L1 and 14R1. The electroencephalogram analysis unit 142 removes the influence of noise signals from the electrical signals corresponding to electroencephalograms and predicts the likelihood of a gesture operation by the user. The prediction of the likelihood of a gesture operation is performed using, for example, artificial intelligence. The electroencephalogram analysis unit 142 uses a trained model that learns, as training data, the electrical signals or patterns thereof that appear immediately before a gesture operation is performed, and learns the relationship to output a numerical value representing the likelihood that a gesture operation will be performed immediately thereafter.

[0206] 58 is a flowchart illustrating an example of the operation of saving the virtual screen VD in the embodiment 14. In Fig. 58, parts corresponding to those in Fig. 14 are assigned the same reference numerals. In this embodiment as well, the processor 201 (see FIG. 5) determines whether the collision avoidance function is on (step 11). If a positive result is obtained in step 11, the processor 201 predicts the possibility of a gesture operation by the user based on the electrical signal corresponding to the electroencephalogram (step 91). Next, processor 201 determines whether a gesture operation is predicted (step 92).

[0207] If a positive result is obtained in step 92, processor 201 executes the processing from step 12 (see FIG. 14) onwards. On the other hand, if a negative result is obtained in step 92, the processor 201 displays the virtual screen VD at the initial position and ends the process (step 93). This processing operation eliminates the need to execute the process for determining whether to move the virtual screen VD while no gesture operation is predicted, thereby reducing the calculation load and extending the usage time.

[0208] 59 is a diagram illustrating the difference between the saving process when a gesture operation is expected and when it is not expected. (A1) and (A2) show the positional relationship between the virtual screen VD and the user U when a gesture operation is not expected, and (B1) and (B2) show the positional relationship between the virtual screen VD and the user U when a gesture operation is expected. In FIGS. 59(A1) and (A2), since a gesture operation is not expected, the virtual screen VD is not retracted even if the distance L2 between the virtual screen VD and the drink 5 is less than the first distance LT1.

[0209] On the other hand, in FIGS. 59(B1) and (B2), a gesture operation is expected, so the virtual screen VD is moved away based on the relationship between the distance L2 between the virtual screen VD and the drink 5 and the first distance LT1, etc. In this embodiment, electroencephalograms are measured using electrodes 14L1, 14L2, and 14R1, but electromyograms, heartbeats, electrocardiograms, pulses, pulse waves, etc. may also be measured. That is, electromyograms, heartbeats, electrocardiograms, pulses, pulse waves, etc. may also be measured on the head. Electromyograms, including electroencephalograms, are examples of biological data.

[0210] <Embodiment 15> In the first to fourteenth embodiments, a case has been described in which an eyeglass-type xR device 10 (see FIG. 1) that allows for transparent observation of real space is assumed, but in the present embodiment, a non-transparent xR device will be described. Fig. 60 is a diagram illustrating an example of how the information processing system 90 is used in accordance with the fifteenth embodiment. (A) shows an example of the appearance of the information processing system 90, and (B) is a diagram illustrating how it is worn. In Fig. 60, parts corresponding to those in Figs. 3, 4, and 5 are assigned the same reference numerals. 60(A) and (B), an information processing system 90 is used by attaching an information terminal 20A such as a smartphone to a head-mounted device 91. In this embodiment, a virtual screen VD is displayed on a display 204 by combining an image of the area in front of a user U captured by a camera 206.

[0211] 60(A) and (B) is equipped with a camera 206 and a positioning sensor 104. The positioning sensor 104 here is the same as the positioning sensor 104 used in the xR device 10 (see FIG. 1). The positioning sensor 104 measures the distance to an object RO located in the vicinity of the user U. That is, the information terminal 20A in this embodiment is a device that combines the hardware and functions of the xR device 10 and the information terminal 20 in the first embodiment. The information terminal 20A here is an example of an information processing system and an example of an information processing device.

[0212] <Embodiment 16> In this embodiment, another example of a head-mounted configuration will be described. FIG. 61 is a diagram illustrating an example of how to use a headset 95 assumed in the sixteenth embodiment. In FIG. 61, parts corresponding to those in FIGS. 3, 5, and 55 are assigned the same reference numerals. (A) shows an example of how the headset 95 is worn, and (B) shows an example of the arrangement of electrodes 14L1, 14R1, and 14R2 for measuring electrical signals corresponding to brain waves, which are placed inside the headset 95. The headset 95 shown in FIG. 61 is also a form of a see-through xR device.

[0213] 61 is an image of a configuration in which electrodes 14L1, 14R1, and 14R2 are attached to, for example, Microsoft's (registered trademark) Hololens (registered trademark). The electrodes 14L1, 14R1, and 14R2 are attached to portions of a ring-shaped member worn on the head of a user U that come into contact with the ears. The headset 95 is provided at its front with a positioning sensor 104 and a camera 206 . The headset 95 here is an example of an information processing system and an example of an information processing device.

[0214] <Other embodiments> (1) Although the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the scope of the above-described embodiments. It is clear from the claims that various modifications and improvements to the above-described embodiments are also included in the technical scope of the present invention.

[0215] (2) For example, in the above-described embodiment, a glasses-type device, a headset-type device worn on the head, and the like are given as specific examples of the xR device 10, but the xR device 10 may also be a contact lens-type device. In the case of a contact lens-type device, the positioning sensor 104 is worn on a necklace or the like of the user U. Note that a contact lens-type device is an example of an information processing system.

[0216] (3) In the above-described first embodiment, the information terminal 20 linked to the xR device 10 executes a function of retracting the virtual screen VD a predetermined distance from the real object RO, but the same function may be realized by the xR device 10 alone, or may be realized as a service by a server on a network. The server and the xR device 10 here are an example of an information processing system.

[0217] (4) In the above-described embodiments, the processor in each of the above-described embodiments refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs, etc.) as well as dedicated processors (e.g., GPUs (Graphical Processing Units), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), programmable logic devices, etc.).

[0218] Furthermore, the operations of the processor in each of the above-described embodiments may be performed by a single processor alone, or may be performed by multiple processors located in physically separate locations in cooperation with each other. Furthermore, the order in which the operations of the processors are performed is not limited to the order described in each of the above-described embodiments, and may be individually changed. [Explanation of symbols]

[0219] 1, 90...Information processing system, 10, 10A, 10B, 10C, 10D...xR device, 20, 20A...Information terminal, 30...Robot, 40...Server, 95...Headset

Claims

1. A device that allows users to visually recognize virtual objects overlaid on real-world spatial information; an acquisition unit that acquires the position of an object existing in real space; a processor; and The processor: If the object is located within a predetermined range from the device, moving the virtual object in a direction away from the object; setting said range through measurement of gesture operations related to registering the range of motion; modifying the range according to the result of the measurement by the acquisition unit; The range defines an area used by the user to perform a gesture operation on the virtual object. Information processing system.

2. A device that allows users to visually recognize virtual objects overlaid on real-world spatial information; an acquisition unit that acquires the position of an object existing in real space; a processor; and The processor: If the object is located within a predetermined range from the device, moving the virtual object in a direction away from the object; setting said range through measurement of gesture operations related to registering the range of motion; modifying the range according to the body part frequently used for gesture operation; The range defines an area used by the user to perform a gesture operation on the virtual object. Information processing system.

3. A device that allows users to visually recognize virtual objects overlaid on real-world spatial information; an acquisition unit that acquires the position of an object existing in real space; a processor; and The processor: If the object is located within a predetermined range from the device, moving the virtual object in a direction away from the object; setting said range through measurement of gesture operations related to registering the range of motion; widening the range corresponding to the user's line of sight; The range defines an area used by the user to perform a gesture operation on the virtual object. Information processing system.

4. A device that allows users to visually recognize virtual objects overlaid on real-world spatial information; an acquisition unit that acquires the position of an object existing in real space; a processor; and The processor: If the object is located within a predetermined range from the device, moving the virtual object in a direction away from the object; setting said range through measurement of gesture operations related to registering the range of motion; When an icon is arranged as the virtual object, the range of the direction of the icon is widened; The range defines an area used by the user to perform a gesture operation on the virtual object. Information processing system.

5. A computer having a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space, and an acquisition unit that acquires the position of an object existing in the real space, a function of moving the virtual object in a direction away from the object when the object is located within a predetermined range from the device; A function for setting the range of motion through measurement of gesture operations related to registration of the range of motion; a function of correcting the range according to the result of the measurement by the acquisition unit; It is a program to achieve The range defines an area used by the user to perform a gesture operation on the virtual object. program.

6. A computer having a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space, and an acquisition unit that acquires the position of an object existing in the real space, a function of moving the virtual object in a direction away from the object when the object is located within a predetermined range from the device; A function for setting the range of motion through measurement of gesture operations related to registration of the range of motion; a function of modifying the range according to a body part frequently used for gesture operation; It is a program to achieve The range defines an area used by the user to perform a gesture operation on the virtual object. program.

7. A computer having a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space, and an acquisition unit that acquires the position of an object existing in the real space, a function of moving the virtual object in a direction away from the object when the object is located within a predetermined range from the device; A function for setting the range of motion through measurement of gesture operations related to registration of the range of motion; a function of widening the range corresponding to the user's line of sight; It is a program to achieve The range defines an area used by the user to perform a gesture operation on the virtual object. program.

8. A computer having a device that allows a user to visually recognize visual information in which a virtual object is superimposed on information of a real space, and an acquisition unit that acquires the position of an object existing in the real space, a function of moving the virtual object in a direction away from the object when the object is located within a predetermined range from the device; A function for setting the range of motion through measurement of gesture operations related to registration of the range of motion; When an icon is arranged as the virtual object, a function of widening the range in the direction of the icon; It is a program to achieve The range defines an area used by the user to perform a gesture operation on the virtual object. program.

Images