Information processing device, information processing method, and augmented reality system

Abstract

Provided is an information processing device that processes information related to augmented reality.　This information processing device comprises: an acquisition unit acquiring the position of a hand of a user and the posture of a finger; and a control unit controlling a display operation of a display device that superimposes and displays a virtual object in actual space, wherein the control unit, on the basis of the position of the hand and the posture of the finger acquired by the acquisition unit, determines a hand interaction that the user is attempting to perform, further determines a virtual object that is the target of the hand interaction, and on the basis of a set attribute of the virtual object, controls the behavior of said virtual object according to the hand interaction.

Description

[Technical Field]

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to an information processing device and information processing method for processing information related to augmented reality, a computer program, and an augmented reality system. [Background technology]

[0002] Virtual reality (VR), augmented reality (AR), and mixed reality (MR) are becoming increasingly popular as technologies for realizing immersive experiences. VR is a technology that allows a user to perceive a virtual space as reality. AR is a technology that enhances the real space seen by a user by adding, emphasizing, attenuating, or deleting information from the real environment surrounding the user. MR is a technology that intertwines reality and virtuality by, for example, displaying virtual objects (hereinafter also referred to as "virtual objects") that replace real-space objects. AR and MR are realized, for example, using a see-through head-mounted display (hereinafter also referred to as "AR glasses"). AR technology allows virtual objects to be superimposed on the real-space scenery observed by a user through AR glasses, or specific real objects to be emphasized or attenuated, or specific real objects to be deleted so that they appear as if they do not exist. An information processing device that presents a user with contact between a real object (such as the user's finger) and a virtual object has also been proposed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-40226 [Non-patent literature]

[0004] [Non-Patent Document 1] J. Napier: "The prehensile movements of the human hand," J. Bone and Joint Surgery, 38B, 4, pp. 902-913 (1956). Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present disclosure is to provide an information processing device, an information processing method, a computer program, and an augmented reality system that process information related to augmented reality. [Means for solving the problem]

[0006] A first aspect of the present disclosure provides: an acquisition unit that acquires the position of the user's hand and the posture of the user's fingers; a control unit that controls a display operation of a display device that displays a virtual object superimposed on a real space; Equipped with the control unit determines a hand interaction that the user is about to perform and a virtual object that is a target of the hand interaction, based on the hand position and finger posture acquired by the acquisition unit, and controls a behavior of the virtual object in accordance with the hand interaction, based on attributes set for the virtual object. It is an information processing device.

[0007] In the present disclosure, hand interactions are defined, and attributes including whether each hand interaction is possible are set in advance for each virtual object. Then, the control unit controls the behavior of the virtual object in response to a user's hand interaction that is set as possible according to the attributes.

[0008] The control unit determines the hand interaction the user is about to perform from among multiple predefined types of hand interactions based on the hand position and finger posture acquired by the acquisition unit. For example, the control unit determines whether the grip operation is a precision grasp in which the user pinches with the thumb and index finger or a strength grasp in which the user grasps with three or more fingers, and further determines the hand interaction, such as "pushing," "crushing," "hitting," or "touching," based on the angle of change in the hand position relative to the virtual object and the speed of change in the finger position relative to the virtual object, and controls the behavior of the virtual object in response to various hand interactions.

[0009] Furthermore, a second aspect of the present disclosure is an acquisition step of acquiring the user's hand position and finger posture; a control step of controlling a display operation of a display device that displays a virtual object superimposed on a real space; and In the control step, a hand interaction that the user is about to perform and a virtual object that is a target of the hand interaction are determined based on the hand position and the posture of the fingers acquired in the acquisition step, and a behavior of the virtual object corresponding to the hand interaction is controlled based on attributes set for the virtual object. It is an information processing method.

[0010] Furthermore, a third aspect of the present disclosure is an acquisition unit that acquires the position of the user's hand and the posture of the user's fingers; a control unit that controls a display operation of a display device that displays a virtual object superimposed on a real space; and the control unit determines a hand interaction that the user is about to perform and a virtual object that is a target of the hand interaction, based on the hand position and finger posture acquired by the acquisition unit, and controls a behavior of the virtual object in accordance with the hand interaction, based on attributes set for the virtual object. It is a computer program.

[0011] A computer program according to a third aspect of the present disclosure defines a computer program written in a computer-readable format to execute predetermined processing on a computer. In other words, by installing the computer program according to the claims of the present application on a computer, a cooperative action is exerted on the computer, and the same effects as those of the information processing device according to the first aspect of the present disclosure can be obtained.

[0012] Furthermore, a fourth aspect of the present disclosure is a display device that displays a virtual object superimposed on a real space; a control unit that controls a display operation of a display device that displays a virtual object superimposed on a real space; Equipped with the control unit determines a hand interaction that the user is about to perform and a virtual object that is a target of the hand interaction, based on the hand position and finger posture acquired by the acquisition unit, and controls a behavior of the virtual object in accordance with the hand interaction, based on attributes set for the virtual object. It is an augmented reality system.

[0013] However, the term "system" used here refers to a logical collection of multiple devices (or functional modules that realize specific functions), and it does not matter whether each device or functional module is contained within a single housing. [Effects of the Invention]

[0014] According to the present disclosure, it is possible to provide an information processing device, an information processing method, a computer program, and an augmented reality system that realize hand interaction with a virtual object using a user's hands and fingers.

[0015] It should be noted that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited to these. Furthermore, the present disclosure may also bring about additional effects in addition to the effects described above.

[0016] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description based on the embodiments and accompanying drawings. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 shows the back of a user's hand on which a controller 10 is placed using a belt 11. [Figure 2] FIG. 2 is a diagram showing the palm of a user's hand on which the controller 10 is placed using a belt 11. [Figure 3] FIG. 3 is a diagram showing the side of a user's hand on which the controller 10 is placed using the belt 11. [Figure 4] FIG. 4 is a diagram showing a state in which a user wears AR glasses 41 on the head and controllers 42 and 43 on both hands, respectively. [Figure 5] FIG. 5 is a diagram illustrating an example of the functional configuration of the AR system 100. [Figure 6] FIG. 6 is a diagram showing a state in which the AR glasses are worn on the user's head. [Figure 7] FIG. 7 is a diagram showing an example of the configuration of an AR system 700 including AR glasses 701 and a controller 702. [Figure 8] FIG. 8 is a diagram showing an example of the configuration of an AR system 800 including AR glasses 801, a controller 802, and an information terminal 803. [Figure 9] FIG. 9 is a diagram showing a specific example of the configuration of the controller 110. As shown in FIG. [Figure 10] FIG. 10 is a diagram showing a specific example of the configuration of the controller 110. As shown in FIG. [Figure 11] FIG. 11 is a diagram showing a specific example of the configuration of the controller 110. As shown in FIG. [Figure 12] FIG. 12 is a diagram showing a specific example of the configuration of the controller 110. As shown in FIG. [Figure 13] FIG. 13 is a diagram showing a specific example of the configuration of the controller 110. As shown in FIG. [Figure 14] FIG. 14 is a diagram showing an example of the functional configuration of the control unit 140. [Figure 15] FIG. 15 is a diagram showing how virtual objects are arranged around the user. [Figure 16] FIG. 16 is a diagram illustrating a mechanism by which the AR glasses display a virtual object so as to follow the movement of the user's head. [Figure 17] FIG. 17 is a diagram showing a method for precisely gripping an object. [Figure 18] FIG. 18 is a diagram showing a method for gripping an object with a gripping force. [Figure 19] Figure 19 shows how to hold the sword. [Figure 20] Figure 20 shows how to grab the sword. [Figure 21] FIG. 21 is a diagram showing an example in which attributes relating to whether or not each gripping operation can be performed are set for each virtual object. [Figure 22] FIG. 22 is a diagram showing behavior of the virtual space in response to a user's gripping operation in an area where no virtual object exists. [Figure 23] FIG. 23 is a diagram showing behavior of the virtual space in response to a user's gripping operation in an area where no virtual object exists. [Figure 24] FIG. 24 is a flowchart showing a processing procedure for realizing the behavior of a virtual object in response to a gripping operation by a user. [Figure 25] FIG. 25 is a diagram showing an example of the configuration of a remote control system 2500. [Figure 26] FIG. 26 is a diagram showing the state in which an operator on the master device 2510 approaches a virtual object with his or her hand. [Figure 27]FIG. 27 is a diagram showing a state in which, on the slave device 2520 side, a robot 2521 approaches an object so as to follow the movement of the operator's hand. [Figure 28] FIG. 28 is a flowchart showing a processing procedure for the operator to grasp and operate a remote object in the remote operation system 2500. [Figure 29] FIG. 29 is a diagram showing an example in which attributes are set for each virtual object (an example including an attribute indicating whether or not a suction process (a process associated with a gripping operation) can be performed). [Figure 30] FIG. 30 is a diagram showing an example of the operation of the adsorption process. [Figure 31] FIG. 31 is a diagram showing an example in which attributes are set for each virtual object (an example in which attributes of a graspable area for each grasping operation are included). [Figure 32] FIG. 32 is a diagram showing a state according to the distance between the user's hand and the virtual object. [Figure 33] FIG. 33 is a diagram showing a gesture of a hand or fingers attempting to pinch a virtual object. [Figure 34] FIG. 34 is a diagram showing a gesture of a hand or fingers attempting to grasp a virtual object. [Figure 35] FIG. 35 is a diagram showing a gesture of a hand or fingers that is not attempting to grasp a virtual object. [Figure 36] FIG. 36 is a diagram showing UI behavior of a virtual object according to the position and orientation of the user's hand. [Figure 37] FIG. 37 is a diagram showing UI behavior of a virtual object according to the position and posture of the user's hand. [Figure 38] FIG. 38 is a diagram showing UI behavior of a virtual object according to the position and posture of the user's hand. [Figure 39] FIG. 39 is a diagram showing UI behavior of a virtual object according to the position and posture of the user's hand. [Figure 40] FIG. 40 is a diagram showing UI behavior of a virtual object according to the position and posture of the user's hand. [Figure 41]FIG. 41 is a diagram showing UI behavior of a virtual object according to the position and posture of the user's hand. [Figure 42] FIG. 42 is a diagram showing UI behavior of a virtual object according to the position and posture of the user's hand. [Figure 43] FIG. 43 is a diagram showing UI behavior of a virtual object according to the position and posture of the user's hand. [Figure 44] FIG. 44 is a flowchart showing a processing procedure for realizing the behavior of a virtual object in response to a gripping operation by a user (a processing procedure including a change in UI behavior of a grippable area of ​​a virtual object). [Figure 45] FIG. 45 is a diagram showing hand interaction for pushing a virtual object. [Figure 46] FIG. 46 shows hand interaction for crushing a virtual object. [Figure 47] FIG. 47 is a diagram showing an example in which attributes relating to whether or not each hand interaction (pushing, squashing) is possible are set for each virtual object. [Figure 48] FIG. 48 illustrates hand interaction for hitting a virtual object. [Figure 49] FIG. 49 illustrates hand interaction for touching a virtual object. [Figure 50] FIG. 50 is a diagram showing an example in which attributes relating to whether or not hand interactions (hitting, touching) are possible are set for each virtual object. [Figure 51] FIG. 51 is a diagram showing the characteristics of each hand interaction with a virtual object. [Figure 52] FIG. 52 is a flowchart showing a processing procedure for realizing the behavior of a virtual object in response to a hand interaction by a user. DETAILED DESCRIPTION OF THE INVENTION

[0018] In real space, objects can be grasped by pinching or grabbing, and the shape of the object changes depending on the force applied by the hand that pinches or grasps. In virtual space, however, objects do not exist in reality, and hands can slip through them, making it impossible to grasp them in the same manner as in real space. For example, an augmented reality system could provide a user interface (UI) that allows users to insert their fingers into an object in virtual space and pinch it with their fingertips, or pinch a frame around the object's perimeter. However, the way objects are grasped through a UI in virtual space is quite different from the way objects are grasped in real space, significantly compromising realism.

[0019] Furthermore, since objects in virtual space do not exist in reality, when grasping an object by pinching or grabbing it, the hand will slip through the object, and the user will not be able to get a realistic tactile sensation. For example, when grasping an object in virtual space by wearing an exoskeleton-type force feedback device on the hand, it is possible to lock the movement of the hand so that the hand does not slip through the object, thereby realizing a manner in virtual space similar to the manner of grasping an object in real space. However, the force feedback Awakening The high purchase cost of the haptic device and the need for space to install it mean that it can only be used by a limited number of users and in a limited environment. In addition, since a large device must be attached to the hand or finger, the feeling of wearing it can be a burden for the user, which can hinder the sense of immersion and realism.

[0020] Therefore, in the present disclosure, in order to improve the sense of immersion and realism when a user grasps a virtual object, a method is realized that allows a user to intuitively grasp a virtual object in a manner that does not deviate from the manner of grasping an object in real space, even when the user's fingers pass through the virtual object. The present disclosure will be described below in the following order with reference to the drawings.

[0021] A. System Configuration B. Controller Configuration C. Basic operation of AR glasses D. Grasping a virtual object D-1. Virtual object attributes D-2. How to set attributes of virtual objects D-3. Changes in behavior according to grasping operations E. Grasping operation trigger F. Attribute-based behavior control of virtual objects G. Remote System Applications H. Position and Orientation Control of Virtual Objects for Grasping Operations I. UI behavior according to grip operation J. Various hand interactions with virtual objects J-1. Attributes of hand interaction depending on the angle of position change J-2. Setting hand interaction attributes according to differences in angle of position change J-3. Triggering hand interaction according to the angle of position change J-4. Attributes of hand interaction depending on the speed of position change J-5. UI behavior according to different speeds of position change J-6. Setting hand interaction attributes according to differences in position change speed J-7. Triggering hand interactions according to differences in position change speed J-8. Attribute-Based Behavior Control of Virtual Objects

[0022] A. System Configuration In the fields of VR and AR, a known input method is for a user to wear a head-mounted display and explore a virtual space while holding and operating a controller in their hand. However, it is preferable for the user to be able to carry out daily activities in the real space, such as walking and grasping objects (including real and virtual objects), while looking around the real space through the AR glasses. For this reason, it is preferable for the input method to allow the user to use their fingers freely, without being restricted by holding a controller.

[0023] Another input method that does not restrict the user's fingers is to detect the user's hand movements from images captured by a camera. For example, the bones of the user's fingers can be extracted using an RGB camera or ToF (Time Of Flight) camera attached to the AR glasses facing outward, and the position, posture, and gestures of the fingers can be recognized. However, methods that detect the user's hand from camera images have problems such as occlusion and the inability to detect fingers outside the camera's field of view.

[0024] Therefore, in the present disclosure, an AR system is configured in which a controller used for detecting the position of the hand, recognizing the posture of the fingers, and recognizing the gestures of the fingers is attached to the user's hand and fingers, allowing the user to use the hands and fingers freely. Furthermore, in order to grasp real or virtual objects using the hand and fingers on which the controller is attached, or to place a virtual object on the palm of the hand, it is preferable to keep the palm free. Therefore, it is preferable to attach the controller to the back of the hand.

[0025] For example, the controller is attached to the back of the user's hand using a belt. FIGS. 1 to 3 show the back and palm of a user's hand, and the side of the hand, with the controller 10 attached using a belt 11. Of course, the fastening method is not necessarily limited to using a belt; for example, the controller may be attached to the back of the hand using an adhesive pad. Although FIGS. 1 to 3 show an example in which the controller 10 is worn on the user's left hand, a controller 10 with a symmetrical shape can also be worn on the right hand.

[0026] 4 shows a user wearing AR glasses 41 on their head and controllers 42 and 43 on their hands. As described above, each of the controllers 42 and 43 has the functions of detecting the position of the hand, recognizing the posture of the fingers, and recognizing the gestures of the fingers. The AR glasses 41 also have the function of superimposing and displaying virtual objects in real space.

[0027] Here, the AR glasses 41 can recognize the positions of the left and right hands, the postures of the fingers, and the gestures of the fingers through the controllers 42 and 43. The AR glasses 41 also have a function for detecting the position and posture of the user's head. Therefore, the AR glasses 41 can detect the relative positions of the user's head and the controllers 42 and 43, in other words, the relative positions of the user's left and right hands. Furthermore, because the AR glasses 41 know the coordinate positions in real space of virtual objects displayed by the AR glasses 41, they can detect the relative positions of the user's left and right hands and the virtual objects.

[0028] 5 shows an example of the functional configuration of an AR system 100 including AR glasses and a controller placed on the back of the user's hand. The illustrated AR system 100 includes a controller 110 placed on the back of the user's hand, a head sensor unit 120, a display unit 131 that displays virtual objects on the AR glasses, and a control unit 140 that comprehensively controls the operation of the entire AR system 100. The controller 110 includes a hand position detection unit 111, a hand posture recognition unit 112, a hand gesture recognition unit 113, and a haptic feedback unit 114. The head sensor unit 120 is mounted on the AR glasses and includes an outward-facing camera 121, an inward-facing camera 122, a microphone 123, a gyro sensor 124, an acceleration sensor 125, and a direction sensor 126. Although only one controller 110 is depicted in FIG. 5 for the sake of simplicity, the AR system 100 will have two controllers 110 if a controller 110 is placed on each of the user's left and right hands.

[0029] The AR system 100 may further include a speaker 132 that outputs audio signals such as sounds related to virtual objects, and a communication unit 133 that enables the AR system 100 to communicate with the outside. The control unit 140 may also include a large-scale storage unit 150 that includes an SSD (Solid State Drive) or the like.

[0030] The AR glasses are generally eyeglass- or goggle-type devices worn by the user on the head. They can superimpose digital information on the field of view of one or both of the user's eyes, emphasize or attenuate specific real objects, or remove specific real objects to make them appear non-existent. Figure 6 shows the AR glasses worn on a user's head. The illustrated AR glasses have a left-eye display unit 131 and a right-eye display unit 131 disposed in front of the user's left and right eyes, respectively. The display unit 131 is transparent or translucent and can superimpose virtual objects at specific positions in real space, emphasize or attenuate specific real objects, or remove specific real objects to make them appear non-existent. The left and right display units 131 may be independently driven to display parallax images, i.e., 3D virtual objects. An outward-facing camera 121 is located approximately in the center of the AR glasses and faces the user's line of sight.

[0031] The AR system 100 can be configured with two devices: AR glasses worn on the user's head, and a controller worn on the back of the user's hand. However, if the controllers are attached to the backs of the user's left and right hands, the AR system 100 is configured with three devices: the AR glasses and two controllers. FIG. 7 shows an example configuration of an AR system 700 including AR glasses 701 and a controller 110. The AR glasses 701 include a control unit 140, a storage unit 150, a head sensor unit 120, a display unit 131, a speaker 132, and a communication unit 133. The controller 110 also includes a hand position detection unit 111, a hand posture recognition unit 112, a hand gesture recognition unit 113, and a haptic feedback unit 114.

[0032] As another configuration example, an AR system 100 is composed of three devices: AR glasses worn on the user's head, a controller worn on the back of the user's hand, and an information terminal such as a smartphone or tablet. Fig. 8 shows a configuration example of an AR system 800 composed of AR glasses 801, a controller 110, and an information terminal 803. The AR glasses 801 include a display unit 131, a speaker 132, and a head sensor unit 120. The controller 110 includes a hand position detection unit 111, a hand posture recognition unit 112, a hand gesture recognition unit 113, and a haptic feedback unit 114. The information terminal 803 also includes a control unit 140, a storage unit 150, and a communication unit 133.

[0033] Note that the specific device configuration of the AR system 100 is not limited to that shown in Figures 7 and 8. Furthermore, the AR system 100 may further include components other than those shown in Figure 1.

[0034] Each component of the AR system 100 will be described with reference to FIG.

[0035] The controller 110 includes a hand position detection unit 111 , a hand and finger posture recognition unit 112 , a hand and finger gesture recognition unit 113 , and a tactile feedback unit 114 .

[0036] The hand position detection unit 111 detects the position of the user's hand. The hand posture recognition unit 112 recognizes the posture of the user's fingers. In this embodiment, the hand posture recognition unit 112 is not essential. The hand gesture recognition unit 113 recognizes a hand gesture, for example, whether the tips of the thumb and other fingers (such as the index finger) are in contact or apart.

[0037] The haptic feedback unit 114 is configured by arranging, for example, electromagnetic or piezoelectric vibrators in an array, and provides haptic feedback by presenting vibrations to the back of the user's hand. In this embodiment, the haptic feedback unit 114 is provided in the controller 110 that is placed on the back of the user's hand, but the haptic feedback unit 114 may also be attached to one or more parts of the user's body other than the back of the hand to provide vibrations.

[0038] Head sensor unit 120 is mounted on the AR glasses and includes outward-facing camera 121, inward-facing camera 122, microphone 123, gyro sensor 124, acceleration sensor 125, and orientation sensor 126.

[0039] The outward-facing camera 121 is, for example, an RGB camera and is installed to capture images of the outside of the AR glasses, i.e., the front direction of the user wearing the AR glasses. The outward-facing camera 121 can capture images of the user's finger movements, but cannot capture images of the user's finger movements when the user's fingers are hidden behind an obstacle, when the fingertips are hidden by the back of the hand, or when the user puts their hand behind their body. The outward-facing camera 121 may further include either an IR camera consisting of an IR emitter and an IR receiver, or a ToF camera. When an IR camera is used as the outward-facing camera 121, a retroreflective material is attached to the object to be captured, such as the back of the hand, and the IR camera emits infrared light and receives the infrared light reflected from the retroreflective material. The image signal captured by the outward-facing camera 121 is transferred to the control unit 140.

[0040] The microphone 123 may be a single sound pickup element or a microphone array consisting of multiple sound pickup elements. The microphone 123 picks up the voice of the user wearing the AR glasses and sounds around the user. The audio signal picked up by the microphone 123 is transferred to the control unit 140.

[0041] The gyro sensor 124, the acceleration sensor 125, and the orientation sensor 126 may be configured as an IMU (Inertial Measurement Unit). Sensor signals from the gyro sensor 124, the acceleration sensor 125, and the orientation sensor 126 are transferred to the control unit 140. The control unit 140 can detect the position and orientation of the head of the user wearing the AR glasses based on these sensor signals.

[0042] The display unit 131 is configured with a transmissive display (such as eyeglass lenses) placed in front of one or both eyes of a user wearing AR glasses, and is used to display a virtual space. Specifically, the display unit 131 displays information (virtual objects) and enhances, attenuates, or deletes real objects, thereby expanding the real space seen by the user. The display unit 131 performs display operations based on control signals from the control unit 140. Furthermore, the mechanism for see-through display of virtual objects on the display unit 131 is not particularly limited.

[0043] The speaker 132 is configured with a single sound-generating element or an array of multiple sound-generating elements, and is installed in, for example, AR glasses. The speaker 132 outputs, for example, sounds related to virtual objects displayed on the display unit 131, but may also output other audio signals.

[0044] The communication unit 133 has a wireless communication function such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), etc. The communication unit 133 mainly performs a communication operation to realize data exchange between the control unit 140 and an external system (not shown).

[0045] The control unit 140 is installed in the AR glasses, or is arranged in a device (such as a smartphone) separate from the AR glasses together with a storage unit 150 and a driving power source such as a battery. The control unit 140 executes various programs read from the storage unit 150 to perform various processes.

[0046] B. Controller Configuration The controller 110 is an input device for the AR system 100 according to this embodiment, and corresponds to a keyboard, mouse, touch panel, or the like in an OS (Operating System) such as Windows or Linux (registered trademark). As shown in FIGS. 1 to 3, the controller 110 is placed on the back of the user's hand. Therefore, the user's fingers are not restricted by the controller 110, and can use their fingers freely. For example, the user can use the hand on which the controller 110 is placed to grasp a real object or a virtual object, or place a virtual object on the palm of their hand.

[0047] The controller 110 is a device that performs input to the AR system 100 based on the position of the user's hand, the posture of the user's fingers, and the posture of the user's fingers. Therefore, as shown in FIGS. 5, 7, and 8, the controller 110 includes a hand position detection unit 111, a hand posture recognition unit 112, and a hand gesture recognition unit 113. The hand position detection unit 111 detects the position of the user's hand. The hand posture recognition unit 112 recognizes the posture of the user's fingers. The hand gesture recognition unit 113 recognizes a hand gesture, for example, whether the tips of the thumb and other fingers (such as the index finger) are in contact or apart. The controller 110 also includes a haptic feedback unit 114 that provides a tactile sensation to the back of the user's hand by presenting vibrations. Specific configuration examples of the hand position detection unit 111, the hand posture recognition unit 112, and the hand gesture recognition unit 113 will be described below.

[0048] Fig. 9 shows an example of the configuration of the hand position detection unit 111 and the hand gesture recognition unit 113. However, in the example shown in Fig. 9, the controller 110 is equipped with an optional hand posture recognition unit 112 or is not equipped with the hand posture recognition unit 112, and the hand posture recognition unit 112 will not be described here.

[0049] In the example shown in FIG. 9, the hand position detection unit 111 uses an IR detection method. That is, the hand position detection unit 111 is configured by a combination of a plurality of (four in the example shown in FIG. 9) IR reflective markers 901-904 attached to the housing 10 of the controller 110 and an IR camera (not shown) provided in the AR glasses (or the head sensor unit 120). The IR camera includes an IR transmitter and an IR receiver. An IR signal output from the IR transmitter is reflected by each of the IR reflective markers 901-904, and the reflected IR signal is received by the IR receiver, allowing the IR camera to detect bright spots on the IR reflective markers 901-904. Then, based on position information on these bright spots, the position and orientation of the main body 10 of the controller 110 (or the back of the user's hand) can be detected. The IR camera is preferably a stereo camera equipped with a plurality of IR receivers.

[0050] Furthermore, the finger gesture recognition unit 113 recognizes finger gestures, for example, whether the tips of the thumb and other fingers (such as the index finger) are in contact with or apart from each other. In the example shown in Fig. 9, the finger gesture recognition unit 113 uses an electrode detection method. Electrodes 911 and 912 are attached to the tips of the user's thumb and index finger, respectively. When the thumb and index finger come into contact with each other, a state of electrical continuity is established between the electrodes 911 and 912. Therefore, the gesture of the thumb and index finger can be recognized based on the state of electrical continuity between the electrodes 911 and 912.

[0051] FIG. 10 shows another example of the configuration of the hand position detection unit 111, the hand posture recognition unit 112, and the hand gesture recognition unit 113.

[0052] In the example shown in FIG. 10 , the hand position detection unit 111 detects the position of the user's hand by combining the IR detection method and the IMU detection method. In the IR detection method, IR reflection signals from multiple IR reflective markers 1001, 1002, 1003, ... attached to the housing 10 of the controller 110 are captured by an IR camera (not shown) provided in the AR glasses (or the head sensor unit 120), and the position and orientation of the user's hand are detected based on the bright spot position of each IR reflective marker. In the IMU detection method, the position and orientation of the user's hand are detected based on the detection signal of an IMU built into the main body 10 of the controller 110. The IMU includes a gyro sensor, an acceleration sensor, and a direction sensor. The IR detection method is used when the controller 110 is within the field of view of the IR camera, and the IMU method is used when the controller 110 is out of the field of view of the IR camera (including when occlusion occurs).

[0053] In the example shown in FIG. 10 , the hand posture recognition unit 112 is configured with IMUs attached to several locations on the user's fingers. Specifically, IMUs 1011, 1012, and 1013 are attached to three locations on the thumb and the proximal and middle phalanges of the index finger by bands 1021, 1022, and 1023, respectively. The control unit 140 can measure the posture of the thumb and the posture of the proximal and middle phalanges of the index finger (or the angle of the second joint of the index finger) based on the detection signals of the IMUs 1011, 1012, and 1013. However, other IMUs may be attached to other locations on the thumb and index finger, or IMUs may be attached to fingers other than the thumb and index finger. Furthermore, the method of fixing the IMUs to each finger is not limited to bands; for example, they may be attached to each finger using adhesive pads.

[0054] Furthermore, the finger gesture recognition unit 113 recognizes finger gestures, such as whether the tips of the thumb and other fingers (such as the index finger) are in contact with or separated from each other, separately from the finger joint angles recognized by the finger posture recognition unit 112. In the example shown in FIG. 10, the finger gesture recognition unit 113 uses a capacitance detection method. Although not shown in FIG. 10, electrodes for detecting capacitance are provided on the fingertips, mid-phalanges, and palm of each finger. For example, when the tips of the thumb and index finger are spread apart, the capacitance increases, and when the tips of the thumb and index finger are brought closer together, the capacitance decreases. Therefore, the finger posture recognition unit 112 can recognize the gesture of the thumb and index finger based on the change in capacitance between the tips of the thumb and index finger.

[0055] 11 shows yet another example of the configuration of the hand position detection unit 111, the hand and finger posture recognition unit 112, and the hand and finger gesture recognition unit 113. However, since the hand position detection unit 111 and the hand and finger gesture recognition unit 113 are configured in the same manner as in FIG. 10, illustration and detailed description thereof will be omitted here, and only the hand and finger posture recognition unit 112 will be described.

[0056] In the example shown in FIG. 11, the hand posture recognition unit 112 is configured with a ToF camera 1101 attached to the palm of the hand using a belt 11. The ToF camera 1101 has a wide angle and can capture five fingers by being attached, for example, near the base of the wrist. The control unit 140 can perform bone recognition of each finger based on the depth image from the ToF camera 1101 to obtain the finger posture. Bone recognition may also be used to recognize finger gestures such as touching of the tips of the thumb and index finger.

[0057] To further improve detection accuracy, it is preferable to recognize finger gestures using the capacitance detection method described above. For example, if a capacitance-type contact sensor 1201 is placed near the center of the palm using a belt 11 as shown in Fig. 12, it can recognize when the index finger, middle finger, ring finger, and little finger are bent and the fingertips approach or touch the palm, as well as a grasping (grasping) gesture using these four fingers, as shown in Fig. 13.

[0058] Note that the configurations of the hand position detection unit 111, hand posture recognition unit 112, and hand gesture recognition unit 113 included in the controller 110 are not necessarily limited to those described above. If it is possible to detect the position of the back of the hand with higher accuracy than the positions of the fingers, a configuration other than that described above can be applied to the controller 110. For example, if the controller 110 alone is capable of estimating its own position using SLAM (Simultaneous Localization and Mapping), or if it is configured to detect the position of the hand with high accuracy and robustness in hand and finger recognition using a camera (such as an RGB stereo camera or a ToF camera) of the head sensor unit 120, the controller 110 may have a configuration other than that described above.

[0059] C. Basic operation of AR glasses 14 schematically illustrates an example of the functional configuration of the control unit 140. In the illustrated example, the control unit 140 includes an application execution unit 1401, a head position / posture acquisition unit 1402, an output control unit 1403, a hand position acquisition unit 1404, a hand / finger posture acquisition unit 1405, and a hand / finger gesture acquisition unit 1406. These functional modules are realized by the control unit 140 executing various programs read from the storage unit 150. However, FIG. 14 illustrates only the minimum functional modules required to realize the present disclosure, and the control unit 140 may further include other functional modules.

[0060] The application execution unit 1401 executes application programs including AR applications under an execution environment provided by the OS. The application execution unit 1401 may execute multiple application programs simultaneously in parallel. The AR application is, for example, an application for playing video or a 3D object viewer, and displays virtual objects superimposed in the field of view of a user wearing AR glasses (see FIG. 6), highlights or attenuates specific real objects, or deletes specific real objects to make them appear as if they do not exist. The application execution unit 1401 also controls the display operation of the AR application (virtual objects) using the display unit 131.

[0061] The application execution unit 1401 also controls the behavior of virtual objects in response to a user's hand interaction with the virtual object, based on the finger operation acquired through the controller 110. In this embodiment, hand interactions such as a gripping operation with respect to a virtual object are defined, and attributes related to whether or not each hand interaction is possible are set in advance for each virtual object. When a user's hand interacts with a virtual object, the application execution unit 1401 controls the behavior of the virtual object based on the preset attributes; details of this point will be described later. The attributes of the virtual object may be internal data of the AR application or may be stored in the storage unit 150 as a data file independent of the AR application. The attributes related to whether or not each hand interaction is possible for each virtual object may be set based on real-world behaviors performed on a real object corresponding to the virtual object, or may be set by the creator of content such as an AR app. Alternatively, the attributes related to whether or not each hand interaction is possible for each virtual object may be set using a machine learning model pre-trained by deep learning.

[0062] Virtual objects generated by the AR application are arranged all around the user. Fig. 15 schematically shows how multiple virtual objects 1501, 1502, 1503, ... are arranged around a user 1500 wearing AR glasses on their head. The application execution unit 1401 arranges the virtual objects 1501, 1502, 1503, ... around the user based on the position of the user's head or the position of the center of gravity of the body estimated based on sensor information from the head sensor unit 120.

[0063] The head position and orientation detection unit 1402 detects the position and orientation of the user's head based on the sensor signals of the gyro sensor 124, acceleration sensor 125, and orientation sensor 126 included in the head sensor unit 120 mounted on the AR glasses, and further recognizes the user's line of sight or field of view.

[0064] The output control unit 1403 controls the output of the display unit 131, the speaker 132, and the haptic feedback unit 114 based on the execution result of an application program such as an AR application by the application execution unit 1401. For example, the output control unit 1403 identifies the user's field of view based on the detection result of the head position and orientation detection unit 1402, and controls the display operation of the virtual object by the display unit 131 so that the virtual object placed in the field of view can be observed by the user through the AR glasses, i.e., so that the virtual object follows the movement of the user's head.

[0065] A mechanism for displaying virtual objects so that the AR glasses follow the movement of the user's head will be described with reference to Fig. 16. In Fig. 16, when the depth direction of the user's line of sight is z w axis, horizontal is y w axis, vertical direction is x w axis, and the user's reference axis x w y w z w The origin of the roll θ is the user's viewpoint. z is the z axis of the user's head w Movement around the axis, tilt θ y is the y w Movement around the axis, pan θ z is the x in the user's head w The head position and orientation detection unit 1402 detects the roll, tilt, and pan movements of the user's head (θ z ,θ y ,θ z) and translational movement of the head. Then, the output control unit 1403 moves the display angle of view of the display unit 131 in the real space in which the virtual object is arranged (see, for example, FIG. 15 ) so as to follow the posture of the user's head, and displays an image of the virtual object present at the display angle of view on the display unit 131. Specifically, the display angle of view is moved so as to cancel out the movement of the user's head by rotating region 1602-1 in accordance with the roll component of the user's head movement, moving region 1602-2 in accordance with the tilt component of the user's head movement, or moving region 1602-3 in accordance with the pan component of the user's head movement. Therefore, the display unit 131 displays the virtual object arranged at the display angle of view that has moved in accordance with the position and posture of the user's head, and the user can observe the real space on which the virtual object is superimposed through the AR glasses.

[0066] Hand position acquisition unit 1404, finger posture acquisition unit 1405, and finger gesture acquisition unit 1406 cooperate with hand position detection unit 111, finger posture recognition unit 112, and finger gesture recognition unit 112 on controller 110, respectively, to acquire information on the user's hand position, finger posture, and gestures made by the fingers. When the user's hand is within the field of view of outward camera 121, information on the user's hand position, finger posture, and gestures made by the fingers can be acquired based on the image recognition results of the image captured by outward camera 121.

[0067] Here, the finger gestures acquired by the hand gesture acquisition unit 1406 include hand interactions that are being performed with the fingers on a virtual object. Examples of hand interactions include "pinch (precise grip)," "grab (strong grip)," "push," "crush," "hit," and "touch." The hand gesture acquisition unit 1406 can determine whether the gesture is "pinch (precise grip)" or "grab (strong grip)" based mainly on information about the position of the user's hand and the posture of the fingers. In contrast, "push" and "crush" involve almost the same posture of the fingers, so the hand gesture acquisition unit 1406 determines whether the gesture is "push" or "crush" based on the angle of change in hand position (or the direction of approach to the virtual object). Furthermore, since "tapping" and "touching" involve almost the same finger posture, the finger gesture acquiring unit 1406 determines whether it is "tapping" or "touching" based on the speed of the change in hand position (or the relative speed of the hand with respect to the virtual object).

[0068] D. Grasping a virtual object Grasping operations for objects in real space can be broadly divided into two types: precision grasp (see FIG. 17), in which the object is pinched with the thumb and index finger, and force grasp (see FIG. 18), in which the object is grasped using all fingers (or all fingers, or three or more fingers) (see Non-Patent Document 1). Of course, there are other methods of precision grasp for objects besides those shown in FIG. 17, and other methods of force grasp for objects besides those shown in FIG. 18. Furthermore, there are variations in grasping operations, such as intermediate grasp, which uses the sides of the fingers, and grasp without using the thumb. Furthermore, in order to stably grasp an object with only one hand, it is necessary to pinch the object between two or more opposing surfaces of the hand. In some cases, multiple fingers are used on one surface.

[0069] There are many types of gripping operations for objects in real space, including precision grip, force grip, and intermediate grip. It is difficult to realize all gripping operations in real space for virtual objects. Limiting the types of gripping operations possible for virtual objects and standardizing the behavior of the virtual objects in response to gripping operations simplifies processing and increases feasibility. However, if the virtual objects behave the same even when a user performs gripping operations on the same virtual object using both precision grip and force grip, this creates a significant discrepancy with the behavior in real space, hindering immersion and realism.

[0070] Therefore, in this disclosure, two or more types of gripping operations for virtual objects are defined, including a precision grip and a force grip. Also, attributes related to whether or not each gripping operation is possible are set in advance for each virtual object. Then, when a user performs a gripping operation on a virtual object, the application execution unit 1401 controls the behavior of the virtual object in accordance with the preset attributes.

[0071] Attributes for each virtual object regarding whether or not a gripping operation can be performed may be set based on real-world behaviors performed on corresponding real objects, or may be set by the creator of the content, such as an AR app. Alternatively, attributes regarding whether or not a gripping operation can be performed on each virtual object may be set using a machine learning model pre-trained by deep learning. For example, attributes such as "pinchable" or "grabable" for gripping operations on virtual objects may be set in advance based on the intentions of the creator of the AR content. Specifically, the creator of the content, such as an AR app, may set attributes that allow gripping operations similar to those performed on corresponding real objects. Then, when a user performs a gripping operation on a virtual object, the virtual object's behavior can be realized based on the creator's intention, such as "small objects can be pinched but not grabbed" or "a gun can be grabbed but not picked up (to prevent trigger operation)." Furthermore, users can perform appropriate gripping operations on virtual objects as intended by the creator, even without precise knowledge of the attributes of the AR content or virtual objects.

[0072] In the AR system 100 according to this embodiment, a controller 110 including a hand position detection unit 111, a hand posture recognition unit 112, and a hand gesture recognition unit 113 is placed on the back of a user's hand. Therefore, the application execution unit 1401 can identify a virtual object that the user is attempting to operate and the position at which the user is attempting to grasp the virtual object, based on hand position information detected by the hand position detection unit 111 and the recognition results by the hand posture recognition unit 112 and the hand gesture recognition unit 113, and can determine the type of grasping operation for the virtual object (whether it is a precision grasp or a force grasp). The application execution unit 1401 then controls the behavior of the virtual object based on attributes (whether it can be pinched or grabbed) that are set in advance for the virtual object. Furthermore, the output control unit 1403 displays and outputs an image of the virtual object being grasped on the display unit 131 (or AR glasses).

[0073] For example, for a virtual object called a "sword," the attributes "possible" are preset for both "pick up" and "grab." When picking up a sword, the sword can be picked up at any contact point between the hand and the blade (see Figure 19). On the other hand, when gripping a sword, the sword can be gripped along the handle, such as the hilt (see Figure 20).

[0074] Therefore, when the application execution unit 1401 recognizes a user's finger gesture to grasp the sword through the controller 110, it controls the behavior of grasping the sword at any contact position between the hand and the blade of the sword. Then, the application execution unit 1401 generates the movement of the sword after it is grasped by the user's fingers based on the position of the hand and the posture of the fingers, and displays it on the display unit 131 (AR glasses).

[0075] Furthermore, when the application execution unit 1401 recognizes a hand gesture in which the user grasps the sword with the thumb and index finger via the controller 110, it controls the behavior of grasping the sword by the handle. Then, the application execution unit 1401 generates the sword movement after the user grasps the sword by the handle with all of the fingers based on the position of the hand and the posture of the fingers, and displays it on the display unit 131 (AR glasses).

[0076] This behavior control of virtual objects makes it possible to realize the behavior of virtual objects in response to user hand interactions in a manner that is close to real-world behavior or a method that is close to the intention of the content creator. Furthermore, in content creation such as AR games, in scenes where virtual objects are directly manipulated, different behaviors can be realized for the same virtual object depending on the type of user gripping operation.

[0077] D-1. Virtual object attributes In the real world, there are many types of gripping operations, including precision grip, strength grip, and intermediate grip. The AR system 100 according to the present disclosure can also accommodate many types of gripping operations, allowing the user to realize the behavior of a virtual object in response to the user's hand interaction in a manner that is close to the real-world manner or the intention of the content creator.

[0078] In the AR system 100 according to the present disclosure, whether or not each gripping operation is possible is set in advance as an attribute of the virtual object. However, in the following, for the sake of simplicity of explanation, the gripping operations are limited to two types: precision grip (pinch) and strength grip (grasp), and the attributes of the virtual object are set as follows:

[0079] (1) Whether or not it can be picked up (2) Can you grab it?

[0080] FIG. 21 shows an example in which the above attributes are set in advance for each virtual object. In the figure, an "o" indicates that the corresponding grasping operation is "allowed (or permitted)," and an "x" indicates that the corresponding grasping operation is "not allowed (or prohibited)." According to the attribute settings shown in the figure, abstract virtual objects such as "icons" and "windows" can be pinched or grabbed. Objects that are small compared to the size of a hand, such as "playing cards," can be pinched but not grabbed. Objects for which the grasping method should be limited (for example, objects that a content creator wants users to grasp) such as "guns" and "rackets" cannot be pinched but can be grabbed. A "gun" virtual object has limited grasping operations (grasping the handle and then pulling the trigger with your finger), and it cannot be pinched but can be grabbed. Liquid virtual objects such as "water" cannot be pinched or grabbed. Therefore, according to the attribute settings shown in FIG. 21, the user can specify the gripping operation of these virtual objects in a manner that is close to the actual manner.

[0081] For a gripping operation for which the attribute "○" (i.e., "allowed") is set, the behavior of the virtual object when that gripping operation is performed can also be set. In Fig. 21, to simplify the drawing, the attributes of each gripping operation for each virtual object are indicated only with "○" and "×", and descriptions of the behavior are omitted.

[0082] Also, although not shown in FIG. 21 , there are virtual objects that can be grasped by both precision and force gripping, but whose behavior differs depending on the gripping operation (when grasped by precision gripping and when grasped by force gripping). This is because even with real objects, the intended action when pinching and when grabbing can differ. For example, if the virtual object to be grasped is a tissue box, when pinching, the behavior is to pinch a single tissue, but when grabbing, the behavior is to grasp the tissue box itself (or grasp and lift it up). Therefore, the AR system 100 may set, as an attribute of the virtual object, behavior for each possible gripping operation in addition to whether each gripping operation is possible. In this way, when the user grasps the virtual object with a certain gripping operation, the virtual object's behavior can be switched to one that is close to the real behavior and is as intended by the user.

[0083] In the AR system 100 according to the present disclosure, it is assumed that an attribute relating to whether each gripping operation is possible is set in advance for each virtual object. However, for a specific virtual object, the attribute value may be set to change before and after gripping. For example, for a virtual object of a "pistol," the trigger cannot be gripped before gripping the handle, but the trigger can be gripped after gripping the handle. In addition, by further setting an attribute that allows the behavior of "firing the gun" in response to the gripping operation of "gripping the trigger," the virtual object can be switched to behavior that is close to real behavior and is intended by the user. Furthermore, the attribute of the virtual object may be set so that the attribute value changes over time.

[0084] D-2. How to set attributes of virtual objects It is assumed that the attributes to be set for each virtual object will basically be manually determined by the designer of the AR system 100 or the creator of the AR content to be used in the AR system 100.

[0085] However, for virtual objects for which data such as the physical properties (e.g., shape and size) and purpose of the corresponding real-world object can be acquired, the attributes of the virtual object may be automatically set based on such data. This is because, when a large number of types of virtual objects are handled by AR system 100, it is cumbersome for system designers and content creators to manually set the attributes of all virtual objects, and this increases the costs of system design and content creation. In other words, automatically setting the attributes of virtual objects based on data such as physical properties and purpose can reduce the costs of system design and content creation.

[0086] The method for automatically setting the attributes of a virtual object may be a method for generating attributes according to an algorithm that analyzes data about the corresponding real-world object. For example, an algorithm may be used that automatically sets an attribute of a virtual object that is 3 cm wide or less as pinchable but not graspable.

[0087] Furthermore, the method for automatically setting the attributes of a virtual object may be a method using an algorithm that can change the rules for setting the attributes of a virtual object based on user information. For example, the rules for setting the attributes of a virtual object may be changed depending on the user's age (child, adult, elderly), hand size, race, physical injury, and the user's daily grasping operation. This is because the optimal grasping operation for the same object may differ for each user. For example, if the user has small hands, a grasping operation is more suitable than a pinching operation even for an object 3 cm wide, so the attribute for "grasping" can be automatically set to "allowed."

[0088] The method for automatically setting the attributes of a virtual object may be a method for estimating the attributes of a virtual object using a machine learning model pre-trained by deep learning. For example, a machine learning model pre-trained by deep learning is used to estimate optimal attributes related to a gripping operation for each virtual object from corresponding real-world data or user information.

[0089] D-3. Changes in behavior according to grasping operations There are virtual objects whose behavior differs depending on the gripping operation (as mentioned above). Changing the behavior due to the gripping operation may be effective even if there is no virtual object to be gripped. For example, the behavior of the virtual space may be changed depending on whether the user grips an area where no virtual object exists, using precision gripping or force gripping.

[0090] The application execution unit 1401 is aware of the positions of virtual objects placed in a virtual space. The application execution unit 1401 can also detect the position of the user's hand through the controller 110 placed on the back of the user's hand and recognize that no virtual object exists at that position. The application execution unit 1401 can also recognize through the controller 110 that the user's fingers have performed a gesture such as a precision grip or a force grip at that position. The application execution unit 1401 then executes different behaviors depending on whether the user has performed a grasping operation or a pinching operation in an area where no virtual object exists.

[0091] The application execution unit 1401 may switch the display of a UI (User Interface) in the virtual space based on a user's gripping operation on an area where no virtual objects exist. For example, when the user performs a gripping operation on an area where no virtual objects exist, the application execution unit 1401 displays a pointer from the fingertip (see FIG. 22). Subsequently, when the user performs a pinching operation on an area where no virtual objects exist (see FIG. 23), the application execution unit 1401 selects the destination pointed to by the fingertip pointer.

[0092] E. Grasping operation trigger In the AR system 100 according to the present disclosure, when a user's hand approaches a virtual object and the user performs a gripping operation permitted by the attributes of the virtual object, the behavior of the virtual object is activated in accordance with the gripping operation. However, the user may not have approached the virtual object with the intention of gripping it, but may have moved their fingers when their hand happened to be close to the virtual object. The behavior of the virtual object in accordance with the gripping operation is activated even though the user did not intend to grip the virtual object.

[0093] Therefore, when a user's hand approaches a virtual object, the user's looking at the target virtual object or the user's interest in the target virtual object may be added as a condition for activating a behavior of the virtual object in response to the gripping operation. It is extremely rare for a user to grip a real object without looking at it. Therefore, it is considered appropriate to require the user to be looking at the virtual object as a condition for activating a behavior, as this is closer to the real-world behavior. Furthermore, adding such a condition can prevent a behavior of the virtual object that differs from the user's intention from being activated when a gripping operation is recognized when the user is not looking at the virtual object.

[0094] The application execution unit 1401 can detect the user's gaze direction from an image captured by the inward-facing camera 122, for example, and determine whether the user is looking at a virtual object. Alternatively, the application execution unit 1401 may estimate the user's level of interest in a virtual object using a machine learning model pre-trained by deep learning from sensor information of the controller 110 or the head sensor unit 120. For this reason, the controller 110 or the head sensor unit 120 may be equipped with sensors other than those shown in FIG. 5 , such as a biosensor. Then, when the application execution unit 1401 recognizes that the user has performed a gripping operation permitted by the attributes of the virtual object when the condition that the user is looking at the target virtual object or is interested in the target virtual object is satisfied, the application execution unit 1401 activates a behavior corresponding to the gripping operation of the virtual object and displays the behavior of the virtual object on the display unit 131 (AR glasses).

[0095] F. Attribute-based behavior control of virtual objects 24 illustrates, in the form of a flowchart, a processing procedure for realizing the behavior of a virtual object in response to a gripping operation by a user, based on attributes of each gripping operation previously set for the virtual object, in the AR system 100 according to the present disclosure. This processing procedure is mainly performed by, for example, the application execution unit 1401.

[0096] First, the application execution unit 1401 acquires the detection result of the user's hand position and the recognition result of the posture and gesture of the fingers through the controller 110 (step S2401). The application execution unit 1401 constantly monitors the relative position of the virtual object being displayed on the display unit 131 and the user's hand that is trying to grasp this virtual object, as well as the posture and gesture of the fingers.

[0097] Then, based on the result acquired in step S2401, the application execution unit 1401 identifies the virtual object that the user is trying to grasp, and when it determines the grasping operation that the user is about to perform on the virtual object, it determines whether the grasping operation matches the attributes set in advance for the virtual object (step S2402). Note that in step S2402, in addition to determining whether the attributes are appropriate, it may also be determined whether the user is looking at the virtual object or is interested in the virtual object, that is, whether the condition for activating the behavior of the virtual object is met.

[0098] Here, if the operation performed by the user on the virtual object does not match the attributes of the virtual object (or does not satisfy the conditions for activating the behavior of the virtual object) (No in step S2402), the application execution unit 1401 determines that the user is not gripping the virtual object, returns to step S2401, and continues to acquire the position of the user's hand, the posture of the fingers, and the gesture of the fingers.

[0099] On the other hand, if the user's finger operation matches the attributes of the virtual object (Yes in step S2402), the application execution unit 1401 generates behavior when the virtual object is gripped, based on the attributes of the virtual object (step S2403).

[0100] Then, the application execution unit 1401 renders a virtual object that behaves according to the behavior generated in step S2403, and displays it in the virtual space, i.e., on the display unit 131 (AR glasses) (step S2404), thereby presenting to the user the behavior of the virtual object that the user is holding and operating.

[0101] G. Remote System Applications For example, a master-slave system is known in which an operator operates a controller on the master side to drive a robot at the output end on the slave side to perform remote work. Remote work has been introduced in various industries, such as remote surgery and remote construction.

[0102] In a master-slave system, it is assumed that an operator performs an operation such as pinching or grabbing an object that is not in hand using a remote robot. The remote operation performed by the operator in the master-slave system is equivalent to the operation of a user pinching or grabbing a virtual object with their fingers through AR glasses in the AR system 100, and the present disclosure can be applied.

[0103] In a master-slave system, the target of grasping is not a virtual object but a real object located at a remote location. The AR system 100 defines multiple types of grasping operations for virtual objects and can set attributes related to the availability of each grasping operation for each virtual object in advance. When a user performs a grasping operation for a virtual object for which an attribute set to "available" is set, the virtual object behaves in accordance with the grasping operation. On the other hand, in a master-slave system, the grasping operations that can be performed on the slave side are defined, and attributes related to the availability of each grasping operation are set in advance for each object to be grasped at a remote location. For example, an attribute that indicates a realistic grasping operation as "available" is set in advance for each object, such as "objects smaller than the size of a hand can be picked up but not grasped." When the master side issues a command to perform a grasping operation for an object located on the slave side with an attribute set to "available," the slave is remotely controlled to grasp the object to initiate the grasping operation. Thus, the user can use the slave to grasp a remote object in a realistic way, even though the user cannot touch the remote object directly but can only observe it through a camera image.

[0104] 25 shows an example of the configuration of a remote control system 2500 to which the present disclosure is applied. The remote control system 2500 shown in the figure is composed of a master device 2510 operated by an operator and a slave device 2520 including a robot 2521 that is the target of remote control.

[0105] The master device 2510 includes a controller 2511 , a display unit 2512 , a master control unit 2513 , and a communication unit 2514 .

[0106] The controller 2511 is used by the operator to input commands for remotely operating the robot 2521 on the slave device 2520 side. In this embodiment, the controller 2511 is assumed to be a device worn on the operator's hand as shown in Fig. 5 and inputs the position and posture of the operator's fingers and finger gestures as operation commands to the robot 2521. However, the controller 2511 may be a camera that photographs the operator's hand, and may be configured to perform image recognition of the position and posture of the operator's fingers and finger gestures from the photographed image of the hand.

[0107] The display unit 2512 is configured with, for example, AR glasses, but may also be a general display device such as a liquid crystal display. Under the control of the master control unit 2513, a virtual object is displayed in the real space where the operator's fingers are projected. The virtual object here is a virtual object that corresponds to a remote, real object that the remotely controlled robot 2521 is trying to grasp. The virtual object is displayed at a location where the relative position of the operator's hand matches the relative position of the robot 2521 and the object.

[0108] The master control unit 2513 receives an image captured by a camera 2522 showing the state of operation of a remote object by the robot 2521 from the slave device 2520 via the communication unit 2514. The master control unit 2513 then controls the display unit 2512 to display a virtual object in the real space where the operator's fingers are projected. The master control unit 2513 may also control the display unit 2512 to display the camera image received from the slave device 2520.

[0109] The virtual object referred to here is a virtual object that corresponds to a remote, real object that is being grasped by the remotely controlled robot 2521. The virtual object is placed at a location where the relative position of the virtual object with respect to the operator's hand coincides with the relative position of the robot 2521 with respect to the object.

[0110] Attributes relating to whether each gripping operation is possible are set in advance for the virtual object. For example, the attributes of the virtual object are set based on a realistic gripping operation of the corresponding remote object. Alternatively, the attributes relating to whether each gripping operation of the virtual object is possible are set based on the gripping operation of the object intended by the designer of the remote operation system 2500.

[0111] The master control unit 2513 acquires information on the position of the user's hand, the posture of the fingers, and the gestures made by the fingers when the operator grasps and manipulates a virtual object, through a controller 2511 installed on the back of the hand. If the operator's grasping manipulation matches the attributes set in advance for that virtual object, the master control unit 2513 converts the position and posture of the operator's fingers and the gestures of the fingers acquired through the controller 2511 into an operation command for remotely manipulating the robot 2521, and transmits the operation command to the slave device 2520 via a communication unit 2514.

[0112] The communication unit 2514 is a functional module for interconnecting with the slave device 2520. The communication medium between the master device 2510 and the slave device 2520 may be either wired or wireless, and is not limited to a specific communication standard.

[0113] The slave device 2520 includes a robot 2521, a camera 2522, a slave control unit 2523, and a communication unit 2524. The slave device 2520 is interconnected with the master device 2510 via the communication unit 2524, and receives operation commands for the robot 2521 from the master device 2510 and transmits images captured by the camera 2522 to the master device 2510.

[0114] The operation command sent from the master device 2510 is a command for driving the robot 2521 in accordance with the position and posture of the operator's fingers and the gestures of the fingers. The slave control unit 2523 interprets the operation command received from the master device 2520 and controls the driving of the robot 2521 so that the robot 2521 reproduces the position and posture of the operator's fingers and the gestures of the fingers (i.e., the grasping operation performed by the operator on the virtual object), thereby performing the object grasping operation. FIG. 26 shows a state on the master device 2510 side in which the operator approaches a virtual object with his / her hand and attempts to grasp it. FIG. 27 shows a state on the slave device 2520 side in which the robot 2521 approaches a real object and attempts to grasp it, so as to follow the movement of the operator's hand.

[0115] The camera 2522 captures images of the operation of an object by the robot 2521. The slave control unit 2523 encodes the images captured by the camera 2522 and controls the communication unit 2524 to transmit the encoded images to the master device 2510 in a predetermined transmission format. As described above, on the master device 2510 side, the display unit 2512 displays a virtual object corresponding to the object in the real space in which the operator's fingers are displayed. The virtual object is placed in a location where the relative position of the operator's hand matches the relative position of the robot 2521 and the object.

[0116] Therefore, according to the remote operation system 2500 to which the present disclosure is applied, the operator on the master device 2510 side cannot directly touch the remote object present on the slave device 2520 side, nor can he observe it in the image captured by the camera 2522, but he can still instruct the robot 2521 to operate by realistically grasping the remote object.

[0117] FIG. 28 shows, in the form of a flowchart, a processing procedure for an operator to grip and operate a remote object in a remote operation system 2500 to which the present disclosure is applied.

[0118] When the master device 2510 receives the video captured by the camera 2522 from the slave device 2520 (S2801), the display unit 2512 displays a virtual object corresponding to the remote object to be operated in the real space where the operator's fingers are displayed (step S2802).

[0119] The master control unit 2513 acquires the detection result of the operator's hand position and the recognition result of the posture and gesture of the fingers (step S2803) through the controller 2511. The master control unit 2513 constantly monitors the relative position of the virtual object being displayed on the display unit 2512 and the hand of the operator attempting to grasp this virtual object, as well as the posture and gesture of the fingers.

[0120] Then, when the master control unit 2513 recognizes the gripping operation that the operator is about to perform on the virtual object based on the result acquired in step S2803, it determines whether the gripping operation conforms to the attributes set in advance for the virtual object (step S2804). The attributes corresponding to realistic gripping operations that are allowed for remote objects are set in advance for the virtual object.

[0121] Here, if the operation performed by the operator on the virtual object does not match the attributes of the virtual object (No in step S2804), the master control unit 2513 determines that the operator is not gripping and operating the virtual object, and returns to step S2801, receives video captured by the camera from the slave device 2520, and updates the display of the virtual object on the display unit 2512.

[0122] On the other hand, if the operation of the operator's fingers matches the attributes of the virtual object (Yes in step S2804), the master control unit 2513 generates behavior of the virtual object when it is grasped, based on the attributes of the virtual object, and displays the virtual object that behaves according to the generated behavior on the display unit 2512 (step S2805).

[0123] Furthermore, the master control unit 2513 converts the positions and postures of the operator's fingers and the gestures of the fingers acquired through the controller 2511 into operation commands for remotely operating the robot 2521, and transmits the operation commands to the slave device 2520 via the communication unit 2514 (step S2806). On the slave device 2520 side, the slave control unit 2523 interprets the operation commands received from the master device 2520, and controls the driving of the robot 2521 so that the robot 2521 reproduces the positions and postures of the operator's fingers and the gestures of the fingers (i.e., the grasping operation performed by the operator on the virtual object), thereby performing the object grasping operation (step S2807).

[0124] In the remote operation system 2500, by executing the processing procedure shown in FIG. 28, the operator on the master device 2510 side cannot directly touch a remote object present on the slave device 2520 side, nor can he observe it in an image captured by the camera 2522. However, through a grasping operation on a virtual object displayed on the display unit 2512, he can instruct the robot 2521 to move in a realistic manner by grasping the remote object.

[0125] H. Position and Orientation Control of Virtual Objects for Grasping Operations In the present disclosure, two or more types of gripping operations for a virtual object, including a precision grip and a force grip, can be defined, and attributes relating to whether or not each gripping operation is possible can be set in advance for each virtual object. Then, when a user performs a gripping operation on a virtual object, the AR system 100 can control the behavior of the virtual object according to the attributes set for that virtual object.

[0126] When a virtual object that does not exist in reality is grasped, the user's fingers slip through it. This can make it difficult for the user to grasp the virtual object at the desired position. Therefore, for virtual objects with a fixed grip position, a behavior may be implemented in which the position and orientation of the virtual object are changed so that the grip position is aligned with the user's hand when the user performs a force grip, i.e., a gripping operation. This change in the position and orientation of the virtual object allows the virtual object to be grasped so that the grip position adheres to the user's hand. This process is also referred to as a "stiction process" or "snap process" in this specification. In addition to information regarding the availability of each grip operation, the availability of the snap process may also be set as an attribute of the virtual object if force gripping is possible.

[0127] FIG. 29 shows an example in which attributes are set in advance for each virtual object, including whether or not suction processing is possible. In the figure, "◯" represents the attribute "possible" and "×" represents the attribute "not possible." According to the attribute settings shown, for virtual objects such as "guns" and "rackets" for which it is desired to limit the grip operation to "strength grip," the attribute "force grip" is set to "possible," and the attribute "suction processing" is also set to "possible." On the other hand, for abstract virtual objects such as "icons" and "windows," there is no need to limit the grip operation, including the grip position, so the attribute "possible" is set for both precision grip and force grip, and the attribute "not possible" is set for suction processing.

[0128] The application execution unit 1401 grasps the positions of virtual objects placed in a virtual space. Furthermore, the application execution unit 1401 can detect the position of the user's hand and recognize the posture and gestures of the user's fingers through the controller 110 placed on the back of the user's hand. For example, when the application execution unit 1401 detects that the user is attempting to grip a virtual object based on the position, posture, and gestures of the user's hand, the application execution unit 1401 refers to the attributes set for the virtual object to confirm whether the object can be gripped, and if gripping is possible, further confirms whether adhesion processing is possible. If the virtual object is capable of adhesion processing, the application execution unit 1401 changes the position and posture of the virtual object so that the gripping position is aligned with the user's hand. Therefore, the user can easily grip the virtual object at a desired position.

[0129] 29, in addition to whether each grip operation is possible, attributes including whether or not suction processing is possible are set, so the application execution unit 1401 can realize virtual object behaviors that perform suction processing in accordance with the grip strength, such as "when you grip a gun, it is automatically positioned so that the trigger can be pulled with your index finger," or "when you grip a sword, it is automatically positioned so that the handle fits in your hand." This allows the user to realize natural and rational grip operations that are suited to the characteristics and purpose of each virtual object with little effort.

[0130] The attribute settings shown in Fig. 29 assume that suction processing will be performed when a "gripping force grasp (grab)" is performed on virtual objects with a fixed grip position, such as a "gun," "sword," or "racket." Fig. 30 shows an example of behavior in which, when the virtual object is a "sword," the position and orientation of the sword changes so that the handle fits the user's hand when the user approaches the handle to grab it.

[0131] Note that the above explanation and FIG. 29 omit the attribute setting of the suction process related to the gripping operation called "precise grip (pinch)." However, if you want to set the attributes of a virtual object so that it is gripped at a specific position, you may also set attributes related to behavior that changes the position and orientation of the virtual object, such as suction process, for the gripping operation called "precise grip (pinch)." This allows for precision gripping, that is, when the user performs a pinch operation on a virtual object whose gripping position is fixed, to perform suction processing so that the gripping position follows the user's hand as the behavior of the virtual object. This allows the user to grip a specific position on a virtual object with little effort.

[0132] In addition, although the above description cites the suction process as one of the behaviors (position and orientation control) for assisting or supporting the user's gripping operation of a virtual object, other processing may also be considered as processing associated with the gripping operation, such as displaying a UI that guides the user through a specific gripping operation of the virtual object. Therefore, whether or not behaviors (position and orientation control) associated with the gripping operation other than the suction process are possible may be set as an attribute of the virtual object.

[0133] I. UI behavior according to grip operation In the present disclosure, two or more types of gripping operations for a virtual object, including a precision grip and a force grip, are defined, and attributes relating to whether or not each gripping operation is possible can be set in advance for each virtual object. Then, when a user performs a gripping operation on a virtual object, the AR system 100 can control the behavior of the virtual object in accordance with the attributes set in advance for the virtual object.

[0134] Furthermore, when a user performs a gripping operation on a virtual object, the object slips through the user's hand, preventing the user from experiencing a realistic tactile sensation. Therefore, during a gripping operation, UI behaviors that stimulate the user's visual, auditory, and tactile senses may be implemented so that the user can intuitively understand that they are gripping a virtual object. Visual UI behaviors are implemented using the display unit 131 (AR glasses). Auditory UI behaviors are implemented using the speaker 132. Tactile UI behaviors are implemented using the haptic feedback unit 114 in the controller 110 placed on the back of the hand.

[0135] Comparing precision grip (pinching) and strength grip (grabbing) of virtual objects, precision grip is a gripping operation using only the fingertips, whereas strength grip is a gripping operation using the entire hand including the fingers. For example, tissues, marbles, and small insects are pinched using the fingertips, but plastic bottles, swords, guns, rackets, and balls are grasped using the entire hand. Therefore, as UI behavior using the haptic feedback unit 114, vibrations may be presented only to the fingertips during precision grip, and vibrations may be presented to the entire hand including the fingers during strength grip. UI behavior using haptic presentation to be performed for each gripping operation may be added and set to the attributes of each virtual object.

[0136] We will also compare the contact positions of a virtual object when using precision grasping (pinching) and force grasping (grabbing). When pinching, the user pinches a specific contact position on the virtual object with their fingers. In contrast, when grabbing, as explained in Section H above, the virtual object can be grasped through its behavior such that the handle or other gripping portion automatically fits into the user's hand as they attempt to grasp it. Even for the same virtual object, the graspable position may differ for each grasping operation. Therefore, when the user's hand approaches a virtual object, the UI behavior of the virtual object may be changed depending on the grasping operation the user is attempting to perform.

[0137] For example, in addition to whether each gripping operation is possible, information on grippable positions for possible gripping operations is set as an attribute of the virtual object. Of course, as explained in Section H above, whether the behavior of the virtual object accompanying the gripping operation (such as suction processing) is possible (or whether each gripping operation is possible for each part) may also be set as an attribute. In such a case, when the user's hand approaches a virtual object with a gripping operation for which the attribute is set to "possible," if the approaching location is set as a grippable position by the attribute, the UI behavior of the virtual object is changed, but if the user's hand approaches a location other than the grippable position, the UI behavior of the virtual object is not changed. This allows the user to intuitively understand the grippable area of ​​the virtual object.

[0138] FIG. 31 shows an example in which attributes, including a graspable area, are set in advance for each virtual object. In the figure, "○" represents an attribute of "possible" and "×" represents an attribute of "not possible." Furthermore, for a gripping operation for which the attribute "possible" is set, a graspable area can be further set. For a virtual object for which the attribute "possible" is set for multiple gripping operations, a different graspable area may be set for each gripping operation, or the same graspable area may be set for multiple gripping operations. Furthermore, when multiple graspable areas that differ for each gripping operation are set, a priority order may be set among the multiple graspable areas (or among multiple gripping operations). Furthermore, although not shown in FIG. 31, attributes may be set for each virtual object, including whether or not processing associated with a gripping operation, such as suction processing, is possible. For example, if attributes including priorities are set for precision grasping in the order of a first graspable area and a second graspable area, when a user makes a gesture to pinch a virtual object near the midpoint between the first graspable area and the second graspable area, the virtual object can be made to behave as if it is adhering to the user's finger in the first graspable area.

[0139] The application execution unit 1401 grasps the position of a virtual object placed in a virtual space. Furthermore, the application execution unit 1401 can detect the position of the user's hand and recognize the posture and gestures of the user's fingers through the controller 110 placed on the back of the user's hand. Therefore, when the application execution unit 1401 detects which gripping operation the user is attempting to use to grasp a virtual object based on the position of the user's hand, the posture, and the gestures of the user's fingers, the application execution unit 1401 refers to the attributes set for the virtual object to confirm whether the object can be grasped with the gripping operation. If the gripping operation is possible, the application execution unit 1401 further confirms whether a grippable position for the gripping operation is set. When the user approaches a grippable area of ​​the virtual object, the application execution unit 1401 changes the UI behavior of the virtual object. Therefore, the user can intuitively understand the grippable area of ​​the virtual object from the UI behavior of the virtual object observed through the AR glasses.

[0140] For ease of explanation, this section defines three states of the distance between the user's hand and the virtual object: "approach," "contact," and "sunken." Figure 32 shows the three states: "approach," "contact," and "sunken." "Approach" is a state in which the shortest distance between the user's hand and the virtual object is equal to or less than a predetermined value. "Contact" is a state in which the shortest distance between the user's hand and the virtual object is 0. "Sunken" is a state in which the user's hand interferes with the area of ​​the virtual object.

[0141] Furthermore, three types of finger gestures are assumed when a user's hand approaches a virtual object: "trying to pinch" (see FIG. 33), "trying to grasp" (see FIG. 34), and "not attempting to grasp" (see FIG. 35). The application execution unit 1401 can detect that the user's hand is approaching a virtual object through the controller 110 placed on the back of the user's hand and recognize which of the three types of finger gestures is being made at that time. When a user's hand approaches a virtual object with the intention of grasping, such as "grabbing" or "grabbing," the UI behavior of the virtual object is changed. On the other hand, when a user's hand approaches a virtual object without the intention of grasping, the UI behavior of the virtual object is not changed because it is expected that a grasping operation will not be performed.

[0142] The change in the UI behavior of a virtual object may be, for example, switching the display of the virtual object to a highlighted display when the user's hand approaches the virtual object in an attempt to grasp it. Figures 36 and 37 show how the display of a virtual object switches from normal display to a highlighted display when the user's hand approaches a cubic virtual object in an attempt to pick it up. However, in the examples shown in Figures 36 and 37, the user can pick up any part of the virtual object. That is, it is assumed that the entire virtual object is within the graspable area, and the entire virtual object is switched to a highlighted display when the user's hand approaches.

[0143] 38 and 39 show an example in which a user's hand approaches a virtual object without intending to grasp it. Even if a user's hand approaches a virtual object without intending to grasp it, it is expected that a grasping operation will not be performed, so the UI behavior of the virtual object is not changed.

[0144] 40 and 41 show examples of UI behavior when a user's hand approaches a virtual object of a "table tennis racket" and attempts to grasp it. The "table tennis racket" has a pre-set attribute of "allowed" for both precision grasp (pinch) and strength grasp (grasp). However, the attributes set are that the graspable area when grasping by pinching is the blade part of the racket, and the graspable area when grasping by grabbing is the grip part of the racket.

[0145] Therefore, as shown in FIG. 40, when the user's hand approaches the virtual object while attempting to grasp it, the blade portion of the racket, which is set in the graspable area for precision grasping (grasping), is highlighted. The user can intuitively understand that they can grasp the blade portion by observing the UI behavior of the blade portion of the racket through the AR glasses. On the other hand, even if the user's hand approaches the blade portion of the racket while attempting to grasp it (not shown), the blade portion is not highlighted because the attribute "allowed" is not set for grip strength grasping (grasping). Therefore, the user can intuitively understand that they cannot grasp the blade portion.

[0146] Also, as shown in FIG. 41, when the user's hand approaches the virtual object while attempting to grasp, the grip portion of the racket, which is set in the graspable area for grip strength grasp (grab), is highlighted. The user can intuitively understand that they can grasp the grip portion by observing the UI behavior of the grip portion of the racket through the AR glasses. On the other hand, even if the user's hand approaches the grip portion of the racket while attempting to grasp (not shown), the grip portion is not highlighted because the attribute "allowed" is not set for precision grasp (grab). Therefore, the user can intuitively understand that they cannot grasp the grip portion.

[0147] Also, as shown in FIG. 42, even if the user's hand approaches the virtual object of the table tennis racket without intending to grasp it, the UI behavior does not change.

[0148] Furthermore, when different graspable areas are set as attributes of a virtual object for each grip operation, a priority order may be set among the multiple graspable areas. In the case of a virtual object of a "table tennis racket," two graspable areas are set: the blade portion and the grip portion, as shown in FIGS. 40 and 41. Taking into account the usual manner in which a racket is used, a priority order can be set: grip portion > blade portion. When the user's hand approaches the "table tennis racket," the grip portion is highlighted more than the blade portion, as shown in FIG. 43 (in FIG. 43, the degree of emphasis is expressed by shading). By switching the change in UI behavior of each graspable area based on the priority order in this way, the user can intuitively understand that, although they can grasp both the blade portion and the grip portion, they should preferentially grasp the grip portion.

[0149] 36 to 43, it is assumed that a UI behavior indicating a gripping operation is performed before the user performs a gripping operation of a virtual object. In addition to the UI display before the gripping operation, a UI may be displayed after the gripping operation is completed to indicate to the user that the action of pinching or gripping the virtual object has been completed. This UI display allows the user to intuitively understand whether the action of pinching or gripping has been performed after the gripping operation is completed.

[0150] Furthermore, for example, when a user performs a gripping operation in a virtual space where a plurality of virtual objects are scattered, the following two methods can be given as a method for determining which virtual object is the target of the gripping operation.

[0151] (1) When a pinch gesture is recognized, a virtual object within a specific distance from the fingertip is targeted for grasping. (2) When a grasping gesture is recognized, a virtual object within a specific distance from the palm of the hand is targeted for grasping.

[0152] 44 shows, in the form of a flowchart, a processing procedure for realizing the behavior of a virtual object in response to a user's gripping operation based on attributes of each gripping operation previously set for the virtual object in the AR system 100 according to the present disclosure. However, the processing procedure shown in this figure differs from the processing procedure shown in FIG. 24 in that a graspable area is set for each gripping operation and the processing includes processing for changing the UI behavior of the graspable area of ​​the virtual object in accordance with the gripping operation that the user is about to perform. This processing procedure is mainly performed by, for example, the application execution unit 1401.

[0153] First, the application execution unit 1401 acquires the detection result of the position of the user's hand and the recognition result of the posture and gesture of the fingers through the controller 110 (step S4401). The application execution unit 1401 constantly monitors the relative position of the virtual object being displayed on the display unit 131 and the user's hand that is trying to grasp this virtual object, as well as the posture and gesture of the fingers.

[0154] Then, based on the result acquired in step S4401, the application execution unit 1401 identifies the virtual object that the user is trying to grasp, and upon recognizing the grasping operation that the user is about to perform, determines whether the grasping operation matches attributes set in advance for the virtual object (step S4402). Note that in step S4402, in addition to determining whether the attributes match, it may also be determined whether the user is looking at the virtual object or is interested in the virtual object (i.e., whether the condition for activating the behavior of the virtual object is met).

[0155] Here, if the operation performed by the user on the virtual object does not match the attributes of the virtual object (No in step S4402), the application execution unit 1401 determines that the user is not gripping the virtual object, and returns to step S4401 to continue acquiring the user's hand position, finger posture, and finger gesture.

[0156] On the other hand, if the user's finger operation matches the attributes of the virtual object (Yes in step S4402), the application execution unit 1401 further checks whether the attributes of the graspable area are set for the finger gesture, i.e., the grasping operation, acquired in step S4401 (step S4403).

[0157] If the attribute of the graspable area for the gripping operation being performed by the user is set (Yes in step S4403), the application execution unit 1401 performs processing to change the UI behavior of the virtual object in the graspable area (step S4404). As processing to change the UI behavior of the virtual object, for example, the graspable area of ​​the virtual object is switched to a highlighted display.

[0158] The user can intuitively understand the graspable area based on the change in the UI behavior of the virtual object and grasp the virtual object in the correct position. The application execution unit 1401 generates behavior when the virtual object is grasped based on the attributes of the virtual object (step S4405).

[0159] Next, the application execution unit 2401 renders a virtual object that behaves according to the behavior generated in step S4405 and displays it on the display unit 131 (AR glasses) (step S4406), thereby presenting to the user the behavior of the virtual object that the user is holding and operating.

[0160] Although not shown in FIG. 44, after the gripping operation is completed, the application execution unit 1401 may control a UI display to indicate to the user that the action of pinching or grabbing the virtual object has been completed.

[0161] J. Various hand interactions with virtual objects The above description has focused primarily on gripping operations as a user's hand interaction with a virtual object, and has described an embodiment in which multiple gripping operations (e.g., precision grip and strength grip) are defined for a virtual object, and attributes and behaviors of each gripping operation are preset for each virtual object. There are various other examples of hand interactions performed by a user with a virtual object besides gripping operations. According to the present disclosure, the present disclosure is not limited to gripping operations but is extended to all hand interactions with virtual objects. By presetting attributes related to whether each hand interaction is possible for each virtual object, when a user performs a hand interaction with a virtual object, it is possible to control the behavior of the virtual object based on the predefined attributes.

[0162] In the explanation so far, the application execution unit 1401 has been configured to recognize the grip operation (whether it is a precision grip or a force grip) that the user is about to perform on a virtual object, based on the position of the user's hand and the posture of the fingers acquired through the controller 110. In contrast to this, by further focusing on differences in the angle and speed at which the relative position of the user's hand changes with respect to the virtual object that is the target of hand interaction, it is possible to recognize hand interactions other than grip operations, such as "pushing," "crushing," "hitting," and "touching."

[0163] Therefore, the AR system 100 according to the present disclosure can also control the behavior of a virtual object in response to hand interaction based on the preset attribute when a user performs one of the hand interactions of "pushing," "crushing," "hitting," or "touching" on a virtual object, by presetting attributes related to whether or not the hand interaction can be performed, other than grasping operations such as "pushing," "crushing," "hitting," or "touching," which can be recognized based on differences in the angle and speed of change in the position of the fingers.

[0164] J-1. Attributes of hand interaction depending on the angle of position change Both "pushing" and "squishing" involve the hand being in an open position. For this reason, "pushing" and "squishing" are distinguished based not only on the position and posture of the fingers but also on the angle of the change in hand position (or the direction of approach to the virtual object). Specifically, the application execution unit 1401 calculates the angle of the change in hand position (or the direction of approach to the virtual object) based on the position information of the virtual object it has generated and the hand position and hand posture acquired from the controller 110 by the hand position acquisition unit 1404 and the hand posture acquisition unit 1405 from time to time, and determines whether the hand interaction is "pushing" or "squishing."

[0165] Figure 45 illustrates an example of a hand interaction for "pushing" a virtual object. Figure 46 illustrates an example of a hand interaction for "squashing" a virtual object. However, in Figures 45 and 46, the virtual objects are depicted as simple cubes to simplify the drawings.

[0166] As can be seen by comparing Figures 45 and 46, both "push" and "squish" are hand interactions in which the user approaches a virtual object with an open hand. However, in the "push" hand interaction shown in Figure 45, the angle of the change in hand position is horizontal with respect to the virtual object, whereas in the "squish" hand interaction shown in Figure 46, the angle of the change in hand position is vertically downward with respect to the virtual object. Therefore, when a gesture of approaching a virtual object with an open hand is performed, the application execution unit 1401 recognizes that a "push" hand interaction has been performed if the movement direction of the hand relative to the virtual object is horizontal (in a direction of ±10 degrees with respect to the horizontal plane), and recognizes that a "squish" hand interaction has been performed if the movement direction of the hand relative to the virtual object is vertically downward (in a direction of 80 to 100 degrees with respect to the horizontal plane).

[0167] In the AR system 100 according to the present disclosure, "pushing" and "squashing" are newly defined as hand interactions with virtual objects, and the following attributes are set in advance for each virtual object.

[0168] (1) Can you press it? (2) Can it be crushed?

[0169] FIG. 47 shows an example in which the above attributes are set in advance for each virtual object. In the figure, an "O" indicates that the corresponding hand interaction is "allowed (or permitted)," and an "X" indicates that the corresponding hand interaction is "not allowed (or prohibited)." According to the attribute settings shown in the figure, hard virtual objects such as a "plastic bottle filled with water" or a "virtual pet" can be pushed but not crushed. Soft virtual objects such as a "plastic bottle empty of water" or a "balloon" can be both pushed and crushed. Virtual objects whose operation methods should be limited (for example, objects that a content creator wants the user to crush), such as an "item box (a box containing items such as insects or small monsters)," cannot be pushed but can be crushed. Abstract virtual objects such as an "icon" or a "window" cannot be pushed or crushed.

[0170] According to the attribute settings shown in Fig. 47, the user can specify hand interactions such as pushing or squashing these virtual objects in a manner similar to real-life interactions. Then, when a hand interaction such as "pushing" or "squashing" is performed on a virtual object, the application execution unit 1401 controls the behavior of the virtual object based on the attributes set in advance as shown in Fig. 47.

[0171] For hand interactions for which the attribute "○" (i.e., "possible") is set, the behavior of the virtual object when that hand interaction is performed can also be set. However, in Fig. 47, for the sake of simplicity, the attributes relating to whether each virtual object can be "pushed" or "squished" are indicated by only "○" and "×," and descriptions of the behavior are omitted.

[0172] J-2. Setting hand interaction attributes according to differences in angle of position change 47, it is assumed that the attribute settings for each virtual object according to the angle of change in hand position will basically be manually determined by the designer of AR system 100 or the creator of the AR content used in AR system 100. However, for virtual objects for which data such as the physical characteristics, such as the shape and size, of the corresponding real-world object and its purpose can be obtained, the attributes related to "pushing" and "squishing" of the virtual object may be automatically set based on such data. By adopting an automatic setting technique, creators of content such as AR apps can be relieved from the hassle of manually setting the attributes of all virtual objects, thereby reducing the costs of system design and content creation.

[0173] The method for automatically setting the attributes of a virtual object may be a method for generating attributes according to an algorithm that analyzes data about a corresponding real-world object. For example, if the density is 1 g / cm 3 For the following virtual objects, an algorithm may be used that performs processing such as automatically setting the crushable attribute.

[0174] Furthermore, the method for automatically setting the attributes of a virtual object may be a method using an algorithm that can change the rules for setting the attributes of a virtual object based on user information. For example, the rules for setting the attributes of a virtual object may be changed based on the user's age (child, adult, elderly), hand size, race, physical injury, and the user's daily hand movements. This is because even objects with the same shape, size, and density may differ in whether or not they can be crushed by different users. For example, if the user is a child with small hands, the attributes related to "pushing" and "crushing" for each virtual object may be automatically set taking into account individual differences, such as the inability to crush an object with low density.

[0175] Furthermore, the method for automatically setting the attributes of a virtual object may be a method for estimating the attributes of a virtual object using a machine learning model pre-trained by deep learning. For example, a machine learning model pre-trained by deep learning is used to estimate optimal attributes related to "pushing" and "crushing" for each virtual object from corresponding real-world data and user information.

[0176] J-3. Triggering hand interaction according to the angle of position change In the above description, when a user approaches a virtual object with an open hand in a horizontal or vertical direction, if the hand interaction is permitted by the attributes of the virtual object, the virtual object behaves in accordance with the hand interaction (for example, the virtual object is pushed or crushed). However, there are cases where the user does not intend to approach a virtual object with their hand in an open position, but instead happens to approach the virtual object with their hand in an open position. It would be unnatural if the behavior of the virtual object were to be activated without the user's intention. Therefore, the conditions for activating the behavior of the virtual object in accordance with the hand interaction may include whether the user is looking at the target virtual object or is interested in the target virtual object.

[0177] When pushing or crushing a real object, it is extremely rare to do so without looking at the object. Therefore, requiring the user to look at a virtual object is considered appropriate, as it closely resembles real-world behavior. Adding such a condition also prevents the user from accidentally pushing or crushing a virtual object when the user is not looking at the virtual object, resulting in the user being unable to see the virtual object.

[0178] The application execution unit 1401 can determine whether the user is looking at a virtual object by detecting the user's line of sight from an image captured by the inward-facing camera 122, for example. Alternatively, the application execution unit 1401 may estimate the user's level of interest in a virtual object using a machine learning model pre-trained by deep learning from sensor information from the controller 110 or the head sensor unit 120. When the application execution unit 1401 recognizes that the user has performed a hand interaction permitted by the attributes of the virtual object when the condition that the user is looking at the target virtual object or is interested in the target virtual object is satisfied, the application execution unit 1401 activates a behavior of the virtual object according to the hand interaction, and displays the behavior of the virtual object being pushed or crushed on the display unit 131 (AR glasses).

[0179] J-4. Attributes of hand interaction depending on the speed of position change Both "tapping" and "touching" involve spreading the fingers apart from the thumb. Therefore, the determination of "tapping" or "touching" is based not only on the position and posture of the fingers but also on the speed of the change in the position of the hand (or the relative speed of the hand with respect to the virtual object). Specifically, the application execution unit 1401 calculates the speed of the change in the position of the hand (or the relative speed of the hand with respect to the virtual object) based on the position information of the virtual object it has generated and the hand position and finger posture acquired from the controller 110 by the hand position acquisition unit 1404 and the finger posture acquisition unit 1405 from time to time, and determines whether the hand interaction is "tapping" or "touching."

[0180] FIG. 48 illustrates an example of a hand interaction for "smacking" a virtual object. FIG. 49 illustrates an example of a hand interaction for "touching" a virtual object. However, in FIGS. 48 and 49, the virtual objects are depicted as simple cubes to simplify the drawings. As can be seen by comparing FIGS. 48 and 49, "smacking" and "touching" are performed with approximately the same hand and finger postures, in which the user approaches the virtual object with the fingers other than the thumb spread apart. However, the speed of the hand position change is fast in the case of "smacking" (e.g., 3 cm per second or more), while the speed of the hand position change is slow in the case of "touching" (e.g., less than 3 cm per second). Therefore, when a gesture is performed in which the user approaches the virtual object with the fingers other than the thumb spread apart, the application execution unit 1401 recognizes that a "smacking" hand interaction has been performed if the speed of the hand position change is fast, and recognizes that a "touching" hand interaction has been performed if the speed of the hand position change is slow.

[0181] Furthermore, if the virtual object itself has a velocity, the velocity of the change in the position of the hand may be calculated based on the relative velocity between the virtual object and the velocity of the fingers.

[0182] Furthermore, the timing for calculating the velocity of the position change is assumed to be the point in time when the virtual object's collider and the finger's collider come into contact with each other. However, the velocity of the position change may be calculated in advance, before contact, when it is detected that the user's hand is approaching the virtual object. Calculating the velocity of the position change in advance before contact can reduce delays that may occur when determining whether the hand interaction is a "tap" or a "touch."

[0183] In the AR system 100 according to the present disclosure, "hitting" and "touching" are newly defined as hand interactions with virtual objects, and the following attributes are set in advance for each virtual object.

[0184] (1) Can you hit it? (2) Whether or not it can be touched

[0185] FIG. 50 shows an example in which the above attributes are set in advance for each virtual object. In the figure, an "O" indicates that the corresponding hand interaction is "allowed (or permitted)," and an "X" indicates that the corresponding hand interaction is "not allowed (or prohibited)." According to the attribute settings shown in the figure, hard virtual objects such as a "desk," a "ball," or a "percussion instrument" can be hit but not touched. Soft virtual objects such as a "balloon" or a "water surface" can be both hit and touched. Virtual objects whose operation methods should be limited, such as an "animal" or a "human avatar," cannot be hit but can be touched. Abstract virtual objects such as an "icon" or a "window" cannot be hit or touched.

[0186] 50, the user can specify hand interactions, such as hitting or touching these virtual objects, in a manner similar to real interactions. When the user performs hand interaction with a virtual object, the application execution unit 1401 controls the behavior of the virtual object based on the preset attributes.

[0187] For hand interactions for which the attribute "○" (i.e., "possible") is set, the behavior of the virtual object when that hand interaction is performed can also be set. However, in Fig. 50, for the sake of simplicity, the attributes relating to whether or not each virtual object can be "hit" or "touched" are indicated by only "○" and "×," and descriptions of the behavior are omitted.

[0188] J-5. UI behavior according to different speeds of position change In this embodiment, in which hand interaction is performed according to differences in the speed of change in hand position, the UI behavior of the virtual object may be set according to the speed of change in hand position when the virtual object comes into contact with the fingers.

[0189] For example, if the virtual object is a "ball," the UI behavior of the virtual object is set so that if the ball is moving at a certain speed or above, it will stick to your hand and generate a sound with a volume and quality that corresponds to the relative speed (such as a "snap" sound when catching the ball).On the other hand, if the ball is stationary or its speed is below a certain level, the UI behavior of the virtual object is set so that no sticking process or sound effect is generated.

[0190] If the real-world object corresponding to the virtual object is a "drum" or "desk" that generates a loud impact sound when struck, a loud impact sound corresponding to the relative speed may be set as the UI behavior. Also, for virtual objects such as "monsters" that require sound effects for specific hand interactions, a loud impact sound corresponding to the relative speed may be set as the UI behavior for "hitting."

[0191] Additionally, in addition to (or instead of) a UI behavior that provides an auditory presentation such as a collision sound in response to a hand interaction of "hitting" a virtual object, a UI behavior that provides a tactile presentation using vibrations or the like may be set. For example, in response to a hand interaction of "hitting" a virtual object such as the above-mentioned "drum," "desk," or "monster," a UI behavior that provides high-frequency vibrations to the hand or fingers using the haptic feedback unit 114 may be set.

[0192] On the other hand, if the real object corresponding to the virtual object generates a small contact sound when touched, such as an "animal," "wood surface," or "water surface," a small contact sound corresponding to the relative speed may be set as the UI behavior. 。

[0193] Additionally, in addition to (or instead of) a UI behavior that provides an auditory presentation such as a contact sound in response to a hand interaction of "touching" a virtual object, a UI behavior that provides a tactile presentation using vibrations or the like may be set. For example, in response to a hand interaction of "touching" a virtual object such as the above-mentioned "animal," "wood surface," or "water surface," a UI behavior that provides low-frequency vibrations to the hand or fingers using the haptic feedback unit 114 may be set.

[0194] J-6. Setting hand interaction attributes according to differences in position change speed 50, it is assumed that the attribute setting for each virtual object according to the speed of the change in hand position will basically be determined manually by the designer of the AR system 100 or the creator of the AR content used in the AR system 100. However, for virtual objects for which data such as the shape, size, weight, hardness, and other physical characteristics and purpose of the corresponding real-world object can be obtained, the attributes related to "hitting" and "touching" the virtual object may be automatically set based on such data. By adopting an automatic setting technique, creators of content such as AR apps can be relieved from the hassle of manually setting the attributes of all virtual objects, thereby reducing the costs of system design and content creation.

[0195] The method for automatically setting the attributes of a virtual object may be a method for generating attributes according to an algorithm that analyzes data related to a corresponding real-world object. For example, an algorithm may be used that sets UI behavior for a "tapping" hand interaction based on the surface hardness of the corresponding real-world object. Using such an algorithm, if the surface hardness of the corresponding real-world object is equal to or greater than a predetermined value, the UI behavior may be automatically set such that "tapping" is "allowed" and a tapping sound is generated. Conversely, if the surface hardness is less than the predetermined value, the UI behavior may be automatically set such that "tapping" is "disallowed" and a tapping sound is not generated.

[0196] Furthermore, the method for automatically setting the attributes of a virtual object may be a method using an algorithm that can change the rules for setting the attributes of a virtual object based on user information. For example, the rules for setting the attributes of a virtual object may be changed depending on the user's age (child, adult, elderly), hand size, race, physical injury, and daily hand movements. If the user is elderly, taking into consideration individual differences such as difficulty in moving fingers quickly due to decreased muscle strength, the attribute may be automatically set by setting the threshold for the speed of position change that determines whether each virtual object is "tapped" or "touched" to 1 cm per second, slower than the usual 3 cm per second.

[0197] Furthermore, the method for automatically setting the attributes of a virtual object may be a method for estimating the attributes of a virtual object using a machine learning model pre-trained by deep learning. For example, a machine learning model pre-trained by deep learning is used to estimate optimal attributes related to "hitting" and "touching" for each virtual object from corresponding real-world data and user information.

[0198] J-7. Triggering hand interactions according to differences in position change speed In the above description, when a user approaches a virtual object quickly or slowly with fingers other than the thumb in an open position, if the hand interaction is permitted by the attributes of the virtual object, the virtual object behaves in accordance with the hand interaction (for example, the virtual object is hit or touched). However, there are cases where the user does not intend to approach a virtual object with their hand closer to the virtual object, but rather happens to approach the virtual object with fingers other than the thumb in an open position. It would be unnatural if the behavior of the virtual object were to be activated without the user's intention. Therefore, the conditions for activating the behavior of the virtual object in accordance with the hand interaction may include whether the user is looking at the target virtual object or is interested in the target virtual object.

[0199] When tapping or touching a real object, it is extremely rare to tap or touch the object without looking at it. Therefore, requiring the user to look at a virtual object is considered appropriate, as it closely resembles real-world behavior. Adding such a condition also prevents the user from accidentally tapping or touching a virtual object when the user is not looking at the virtual object, resulting in the user being unable to see the virtual object.

[0200] The application execution unit 1401 can determine whether the user is looking at a virtual object by detecting the user's line of sight from an image captured by the inward-facing camera 122, for example. Alternatively, the application execution unit 1401 may estimate the user's level of interest in a virtual object using a machine learning model pre-trained by deep learning from sensor information from the controller 110 or the head sensor unit 120. When the application execution unit 1401 recognizes that the user has performed a hand interaction permitted by the attributes of the virtual object when the condition that the user is looking at the target virtual object or is interested in the target virtual object is satisfied, the application execution unit 1401 activates a behavior corresponding to the hand interaction of the virtual object, and displays the behavior of the virtual object being hit or touched on the display unit 131 (AR glasses).

[0201] J-8. Attribute-Based Behavior Control of Virtual Objects In the above section F, the processing procedure for realizing the behavior of a virtual object when a user performs a gripping operation was explained with reference to the flowchart shown in Fig. 24. In this section J-8, we will explain the processing for realizing the behavior of a virtual object in response to various user hand interactions, not limited to gripping operations.

[0202] The characteristics of hand interactions discussed in Section J-8 are summarized in Figure 51. This figure shows examples of finger postures for six types of hand actions: precision grasp (pinching), strength grasp (grasping), pushing, crushing, hitting, and touching, as well as the criteria for judgment (angle of hand position change, speed of finger position change).

[0203] FIG. 52 shows, in the form of a flowchart, a processing procedure for realizing the behavior of a virtual object in response to a hand action by a user, based on the attributes of each hand action preset for the virtual object, in AR system 100 according to the present disclosure.

[0204] First, the application execution unit 1401 acquires the detection result of the user's hand position and the recognition result of the posture and gesture of the fingers through the controller 110 (step S5201). The application execution unit 1401 constantly monitors the relative position of the virtual object being displayed on the display unit 131 and the user's hand that is trying to grasp this virtual object, as well as the posture and gesture of the fingers.

[0205] Next, the application execution unit 1401 calculates the angle of the change in the position of the hand relative to the virtual object and the speed of the change in the position of the fingers relative to the virtual object (step S5202).

[0206] Then, based on the result acquired in step S5201 and the result calculated in step S5202, the application execution unit 1401 identifies a virtual object with which the user is to perform hand interaction, and upon determining the hand interaction the user is about to perform with the virtual object, determines whether the hand interaction matches attributes preset for the virtual object (step S5203). Note that in step S5203, in addition to determining whether the attributes are appropriate, it may also be determined whether the user is looking at or interested in the virtual object, that is, whether the condition for activating the behavior of the virtual object is met.

[0207] Here, if the hand interaction performed by the user with the virtual object does not match the attributes of the virtual object (or does not satisfy the conditions for activating the behavior of the virtual object) (No in step S5203), the application execution unit 1401 determines that the user has not performed a hand interaction with the virtual object, returns to step S5201, and continues to acquire the user's hand position, finger posture, and finger gesture.

[0208] On the other hand, if the user's hand interaction matches the attributes of the virtual object (Yes in step S5203), the application execution unit 1401 generates behavior when the hand interaction is performed on the virtual object based on the attributes of the virtual object (step S5204).

[0209] Then, the application execution unit 1401 renders a virtual object that behaves according to the behavior generated in step S5204, and displays it in the virtual space, i.e., on the display unit 131 (AR glasses) (step S5205), thereby presenting to the user the behavior of the virtual object with which the user performed hand interaction. [Industrial Applicability]

[0210] Although the present disclosure has been described in detail above with reference to specific embodiments, it is obvious that those skilled in the art can make modifications or substitutions to the embodiments without departing from the spirit and scope of the present disclosure.

[0211] Although the present specification has mainly described an embodiment in which hand interaction (such as grasping) of a virtual object according to the present disclosure is applied to an AR system, the gist of the present disclosure is not limited thereto. For example, the present disclosure can be similarly applied to a VR system that allows a virtual space to be perceived as reality, an MR system that combines reality and virtuality, and even a master-slave remote system.

[0212] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as limiting. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0213] The present disclosure may also be configured as follows.

[0214] (1) an acquisition unit for acquiring the position of a user's hand and the posture of his or her fingers; a control unit that controls a display operation of a display device that displays a virtual object superimposed on a real space; Equipped with the control unit determines a hand interaction that the user is about to perform and a virtual object that is a target of the hand interaction, based on the hand position and finger posture acquired by the acquisition unit, and controls a behavior of the virtual object in accordance with the hand interaction, based on attributes set for the virtual object. Information processing device.

[0215] (2) defining hand interactions and setting the attributes including whether each hand interaction is possible for each virtual object in advance; the control unit controls a behavior of the virtual object in response to the user's hand interaction that is set as possible by the attribute. The information processing device according to (1) above.

[0216] (3) The control unit determines the hand interaction that the user is about to perform from among multiple predefined types of hand interactions, based on the hand position and finger posture acquired by the acquisition unit. The information processing device according to any one of (1) and (2) above.

[0217] (4) The control unit determines whether the grip operation is a precision grip in which the object is pinched with the thumb and index finger or a force grip in which the object is gripped with three or more fingers, and controls the behavior of the virtual object in response to the determined grip operation based on the attribute that sets whether the object can be pinched with the precision grip or the force grip for each virtual object. An information processing device according to any one of (1) to (3) above.

[0218] (5) The control unit further determines a hand interaction that the user is about to perform based on an angle of a change in the position of the hand relative to the virtual object. An information processing device according to any one of (1) to (4) above.

[0219] (6) The control unit determines that when the hand is in an open hand posture, the hand interaction is a “push” of the virtual object when the hand approaches the virtual object from a horizontal direction, and determines that the hand interaction is a “crush” of the virtual object when the hand approaches the virtual object from a vertical direction, and controls the behavior of the virtual object in response to the determined “push” and “crush” hand interactions based on the attribute that sets whether or not the virtual object can be “pushed” or “crushed.” The information processing device according to any one of (1) to (5) above.

[0220] (7) The control unit further determines a hand interaction that the user is about to perform based on a speed of a change in the position of the hand or finger relative to the virtual object. The information processing device according to any one of (1) to (6) above.

[0221] (8) The control unit determines that the hand interaction is a “hit” of the virtual object when the hand approaches the virtual object at a speed equal to or greater than a predetermined speed while the fingers are in an open position, and determines that the hand interaction is a “touch” of the virtual object when the hand approaches the virtual object at a speed less than the predetermined speed, and controls the behavior of the virtual object in response to the determined “hit” and “touch” hand interactions based on the attribute that sets whether or not the “hit” and “touch” can be performed for each virtual object. The information processing device according to any one of (1) to (7) above.

[0222] (9) The control unit controls a behavior of the first virtual object when the user performs the second hand interaction following the first hand interaction, based on the attribute that further sets whether or not a second hand interaction is possible with the first virtual object before and after the first hand interaction is performed with the first virtual object. An information processing device according to any one of (1) to (8) above.

[0223] (10) The control unit controls a behavior of the second virtual object when a user performs a first hand interaction with the second virtual object, based on the attribute in which whether a first hand interaction is possible and a behavior of the second virtual object corresponding to the first hand interaction are set. The information processing device according to any one of (1) to (9) above.

[0224] (11) The control unit controls the position and orientation of the second virtual object when the user grips the second virtual object, based on the attribute that indicates whether a gripping operation on the second virtual object is possible and whether a behavior of changing the position and orientation of the second virtual object so that the position where the user grips the second virtual object is aligned with the user's hand. The information processing device according to any one of (1) to (10) above.

[0225] (12) The control unit controls the behavior of a virtual object when a user performs hand interaction with a certain location of the virtual object, based on the attribute that sets whether or not hand interaction is possible for each virtual object and a location where hand interaction is possible. The information processing device according to any one of (1) to (11) above.

[0226] (13) The control unit further controls the behavior of the user interface when the user performs hand interaction with a certain location of a certain virtual object, based on the attribute that sets whether or not hand interaction is possible for each virtual object and the behavior of the user interface in response to the hand interaction. The information processing device according to any one of (1) to (12) above.

[0227] (14) The control unit further controls the behavior of the user interface at a position where hand interaction is possible when a user performs hand interaction with a certain location of a certain virtual object, based on the attribute that sets whether or not hand interaction is possible for each virtual object and the behavior of the user interface at a position where hand interaction is possible. The information processing device according to any one of (1) to (13) above.

[0228] (15) acquiring the user's hand position and finger posture; a control step of controlling a display operation of a display device that displays a virtual object superimposed on a real space; and In the control step, a hand interaction that the user is about to perform and a virtual object that is a target of the hand interaction are determined based on the hand position and the posture of the fingers acquired in the acquisition step, and a behavior of the virtual object corresponding to the hand interaction is controlled based on attributes set for the virtual object. Information processing methods.

[0229] (16) an acquisition unit for acquiring the user's hand position and finger posture; a control unit that controls a display operation of a display device that displays a virtual object superimposed on a real space; and the control unit determines a hand interaction that the user is about to perform and a virtual object that is a target of the hand interaction, based on the hand position and finger posture acquired by the acquisition unit, and controls a behavior of the virtual object in accordance with the hand interaction, based on attributes set for the virtual object. Computer program.

[0230] (17) A display device that displays a virtual object superimposed on a real space; a control unit that controls a display operation of a display device that displays a virtual object superimposed on a real space; Equipped with the control unit determines a hand interaction that the user is about to perform and a virtual object that is a target of the hand interaction, based on the hand position and finger posture acquired by the acquisition unit, and controls a behavior of the virtual object in accordance with the hand interaction, based on attributes set for the virtual object. Augmented reality system.

[0231] (21) An acquisition unit for acquiring a user's hand position and finger gestures; a control unit that controls a display operation of a display device that displays a virtual object superimposed on a real space; Equipped with the control unit controls a behavior of the virtual object in accordance with the hand position and finger gesture acquired by the acquisition unit, based on attributes set for the virtual object. Information processing device.

[0232] (22) The control unit detects that the hand attempting to grasp the virtual object is approaching the virtual object based on the hand position and finger gesture acquired by the acquisition unit, and controls the behavior of the virtual object in response to the grasping operation by the hand. The information processing device according to (21) above.

[0233] (23) A plurality of gripping operations are defined, and the attribute regarding whether each gripping operation is possible for the virtual object is set; the control unit controls a behavior of the virtual object in response to a user's gripping operation that is set as possible based on the attribute. The information processing device according to any one of (21) and (22) above.

[0234] (24) The attribute including a behavior of the virtual object for each grasping operation is further set; the control unit controls a behavior of the virtual object in response to a gripping operation by the user based on the behavior set by the attribute. The information processing device according to (23) above.

[0235] (25) The attribute including whether or not each gripping operation on the virtual object is possible before and after gripping is further set, the control unit controls behavior of the virtual object in response to a gripping operation set as possible according to the attribute, before and after gripping. The information processing device according to any one of (23) and (24) above.

[0236] (26) The attribute including whether or not processing associated with a gripping operation is possible is further set; the control unit controls a behavior of the virtual object in response to a gripping operation by the user, the behavior including a process set as possible based on the attribute. The information processing device according to any one of (23) to (25) above.

[0237] (27) The processing associated with the grasping operation includes an adsorption process that changes the position and orientation of the virtual object so that the position where the user grasps it is aligned with the user's hand. The information processing device according to (26) above.

[0238] (28) The attribute relating to the user interface in each grasping operation on the virtual object is further set; the control unit controls the display device to perform a display operation of a user interface set by the attribute in response to a gripping operation by a user. The information processing device according to any one of (23) to (27) above.

[0239] (29) The attribute related to the user interface at the graspable position of the virtual object for each grasping operation is set; the control unit controls the display device to perform a display operation of a user interface at the grippable position set by the attribute in response to a gripping operation by a user. The information processing device according to (28) above.

[0240] (30) acquiring a user's hand position and hand gesture; a control step of controlling a display operation of a display device that displays a virtual object superimposed on a real space; and In the control step, a behavior of the virtual object is controlled in accordance with the hand position and the finger gesture acquired by the acquisition unit, based on attributes set for the virtual object. Information processing methods.

[0241] (31) an acquisition unit for acquiring the user's hand position and finger gestures; a control unit that controls a display operation of a display device that displays a virtual object superimposed on a real space; and the control unit controls a behavior of the virtual object in accordance with the hand position and finger gesture acquired by the acquisition unit, based on attributes set for the virtual object. Computer program.

[0242] (32) A display device that displays a virtual object superimposed on a real space; an acquisition unit that acquires a user's hand position and finger gestures; a control unit that controls a display operation of the display device; Equipped with the control unit controls a behavior of the virtual object in accordance with the hand position and finger gesture acquired by the acquisition unit, based on attributes set for the virtual object. Augmented reality system. [Explanation of symbols]

[0243] 10...Controller, 11...Belt 100...AR system, 110...controller, 111...hand position detection unit 112...Hand posture recognition unit, 113...Hand gesture recognition unit 114, tactile feedback unit, 120... head sensor unit 121...outward-facing camera, 122...inward-facing camera, 123...microphone 124... gyro sensor, 125... acceleration sensor, 126... orientation sensor 131...cover part, 132...speaker, 133...communication part, 140...control part 150...Storage section 300...AR system, 301...AR glasses, 302...controller 400...AR system, 401...AR glasses, 402...controller 403...Information terminal 500...controller, 501, 502, 503...IMU 511, 512, 513...Band 2500...Remote control system, 2510...Master device 2511...controller, 2512...display unit, 2513...master control unit 2514...communication unit, 2520...slave device, 2521...robot 2522...camera, 2523...slave control unit, 2524...communication unit

Claims

1. an acquisition unit that acquires the user's hand position, hand and finger posture, and gaze information; a control unit that controls a display operation of a display device that displays a virtual object superimposed on a real space; Equipped with when a plurality of areas in which the user's intended hand interaction is possible are set in the virtual object, a priority is set among the plurality of areas; The control unit determining the hand interaction and the virtual object that is a target of the hand interaction based on the hand position and finger posture acquired by the acquisition unit; controlling a behavior of the virtual object in response to the hand interaction based on a type of hand interaction that is set for the virtual object and that is possible with the virtual object; The system can be set to activate a behavior of the object in response to the hand interaction based on user's gaze information. Information processing device.

2. Define hand interactions and set whether or not each hand interaction is possible for each virtual object in advance. the control unit controls a behavior of the virtual object in response to the user's hand interaction that is set as possible. The information processing device according to claim 1 .

3. the control unit determines a hand interaction that the user is about to perform from among a plurality of predefined types of hand interactions, based on the hand position and finger posture acquired by the acquisition unit; The information processing device according to claim 1 .

4. the control unit determines whether the hand interaction is a precision grasp in which the user pinches the object with the thumb and index finger or a force grasp in which the user grasps the object with three or more fingers, and controls the behavior of the virtual object with respect to the determined hand interaction based on whether the virtual object is a precision grasp or a force grasp. The information processing device according to claim 1 .

5. The control unit further determines a hand interaction that the user is about to perform based on an angle of a change in the position of the hand relative to the virtual object. The information processing device according to claim 1.

6. the control unit determines, when the hand is in an open hand posture and the user approaches the virtual object from a horizontal direction, that the hand interaction is a "push" of the virtual object, and determines, when the user approaches the virtual object from a vertical direction, that the user interacts with the virtual object from a "crush" of the virtual object, and controls the behavior of the virtual object with respect to the determined "push" and "crush" hand interactions based on whether or not the user can "push" and "crush" each virtual object. The information processing device according to claim 1 .

7. The control unit further determines a hand interaction that the user is about to perform based on a speed of a change in position of a finger with respect to the virtual object. The information processing device according to claim 1.

8. the control unit determines that the hand interaction is a "hitting" of the virtual object when the hand approaches the virtual object at a speed equal to or greater than a predetermined speed while the fingers are in an open position, and determines that the hand interaction is a "touching" of the virtual object when the hand approaches the virtual object at a speed less than the predetermined speed, and controls the behavior of the virtual object in response to the determined "hitting" and "touching" hand interactions based on whether or not the "hitting" and "touching" are possible for each virtual object. The information processing device according to claim 1 .

9. The type of hand interaction that the user can perform can be changed before and after performing the hand interaction. The information processing device according to claim 1 .

10. the control unit controls a behavior of the second virtual object when the user performs a first hand interaction with the second virtual object, based on attributes set for the second virtual object as to whether a first hand interaction is possible and a behavior of the second virtual object corresponding to the first hand interaction. The information processing device according to claim 1 .

11. the control unit controls the position and orientation of the second virtual object when the user grips the second virtual object, based on whether a gripping operation can be performed on the second virtual object and whether a behavior of changing the position and orientation of the second virtual object so that the position gripped by the user is aligned with the user's hand is possible. The information processing device according to claim 1 .

12. the control unit controls the behavior of a virtual object when the user performs hand interaction with a certain location of the virtual object, based on attributes that set whether or not hand interaction is possible for each virtual object and positions where hand interaction is possible. The information processing device according to claim 1 .

13. The control unit further controls behavior on a user interface using at least one of visual, auditory, and tactile feedback in response to the hand interaction. The information processing device according to claim 1 .

14. The control unit can change the type of hand interaction available for the virtual object based on the characteristics of the user. The information processing device according to claim 1 .

15. the control unit determines whether a condition for invoking the behavior of the virtual object is satisfied, and then generates the behavior of the virtual object that is a target of the hand interaction. The information processing device according to claim 1 .

16. The types of hand interactions available for a specific virtual object change before and after a specific hand interaction. the control unit controls a behavior of the virtual object in accordance with a hand interaction that becomes possible after the specific hand interaction is performed with the specific virtual object. The information processing device according to claim 1 .

17. The virtual object is set to be capable of an adhesion process that changes the position and posture of the virtual object in accordance with the user's hand so that the virtual object snaps to the user's hand during hand interaction by the user; the control unit further controls a behavior of the virtual object in the hand interaction based on the set result of whether or not the adsorption process is enabled. The information processing device according to claim 1 .

18. an acquisition step of acquiring the user's hand position, hand and finger posture, and gaze information; a control step of controlling a display operation of a display device that displays a virtual object superimposed on a real space; and when a plurality of areas in which the user's intended hand interaction is possible are set in the virtual object, a priority is set among the plurality of areas; In the control step, determining the hand interaction and the virtual object that is a target of the hand interaction based on the hand position and the finger posture acquired in the acquiring step; controlling a behavior of the virtual object in accordance with the hand interaction based on a type of hand interaction that is set for the virtual object and that is possible with the virtual object; The system can be set to activate a behavior of the object in response to the hand interaction based on user's gaze information. Information processing methods.

19. an acquisition unit that acquires the user's hand position, hand and finger posture, and gaze information; a display device that displays a virtual object superimposed on a real space; a control unit that controls a display operation of a display device that displays a virtual object superimposed on a real space; Equipped with when a plurality of areas in which the user's intended hand interaction is possible are set in the virtual object, a priority is set among the plurality of areas; The control unit determining the hand interaction and the virtual object that is a target of the hand interaction based on the hand position and finger posture acquired by the acquisition unit; controlling a behavior of the virtual object in response to the hand interaction based on a type of hand interaction that is set for the virtual object and that is possible with the virtual object; The system can be set to activate a behavior of the object in response to the hand interaction based on user's gaze information. Augmented reality system.

Images