Information processing apparatus, information processing method, and program

Abstract

There is provided an exemplary information processing method, comprising: determining a plurality of states of an operation object related to a target object in a virtual space, the states including a first state in which the operation object takes a first action on the target object, the operation object keeps a second state of the first action on the target object, and the second state occurs in a third state after the first state and before the first state; determining the distance between the operation object in the third state and the target object in the virtual space, wherein the determined distance comprises a first distance and a second distance smaller than the first distance; transitioning from the third state to the first state in response to determining the second distance; transitioning from the first state to the second state; and generating a haptic signal corresponding to each of the first state, the second state, and the third state.

Description

[0001] Cross references to related applications

[0002] This application claims the benefit of Japanese Priority Patent Application JP 2022-133242 filed on August 24, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present technical content relates to an information processing device, an information processing method, and a program suitable for tactile presentation, etc. Background Art

[0004] Patent document 1 discloses a contact presentation device that determines the positional relationship between a human body position and a virtual object, stimulates the human body with a first intensity when the human body and the virtual object are not in contact with each other, and stimulates the human body with a second intensity that is stronger than the first intensity when the human body and the virtual object are in contact with each other. This allows the user to be stimulated without causing discomfort to the user (see paragraphs 0034 to 0047 of the specification of Patent document 1, Figure 2 wait).

[0005] Citation List

[0006] Patent Literature

[0007] PTL1: Japanese Patent Application No. 2008-123431 Summary of the invention

[0008] Technical issues

[0009] In this tactile presentation to the user, a technology that can improve usability is needed.

[0010] In view of the above-mentioned circumstances, an object of the present technical content is to provide an information processing device, an information processing method, and a program capable of improving usability.

[0011] Solutions to the problem

[0012] In order to achieve the above-mentioned object, an information processing apparatus according to an embodiment of the present technical content includes a processor communicating with a tracking device, a video display, and a haptic device.

[0013] For example, the information processing device includes at least one processor configured to:

[0014] Determine a plurality of states of an operation object in a virtual space related to a target object, the states comprising a first state, a second state, and a third state, in which the operation object takes a first action on the target object in the first state, and in which the operation object maintains the first action on the target object in the second state, the second state occurs after the first state, and the third state occurs before the first state;

[0015] determining a distance between the operation object and the target object in the virtual space in the third state, the determined distance comprising a first distance and a second distance smaller than the first distance;

[0016] in response to determining the second distance, transitioning from the third state to the first state;

[0017] transitioning from the first state to the second state; and

[0018] A haptic signal is generated corresponding to each of the first state, the second state, and the third state.

[0019] In an example, a system includes an information handling device, and a tracking device, a video display, and / or a haptic device.The information handling device may include a memory communicatively coupled to a processor.

[0020] As a result, the information processing apparatus can improve the usability of the system, in particular, improve the usability of the haptic device.

[0021] The first motion may include a motion in which the operating object touches or grasps the target object, or the first motion may be another other specific motion.

[0022] In an example, a system includes:

[0023] The information processing equipment discussed above;

[0024] a tracking device configured to track the movement of a user;

[0025] a video display configured to display the target object and the operation object in the virtual space; and

[0026] A haptic device is provided to present a haptic sensation based on the haptic signal in each of the plurality of states.

[0027] In an example, the system further comprises a glove comprising the tracking device and the haptic device, and a head mounted display comprising the video display.

[0028] In various exemplary embodiments, an information processing device and / or system may perform any one or more of the following methods or processes.

[0029] In an example, an information processing method includes:

[0030] Determine a plurality of states of an operation object in a virtual space related to a target object, the states comprising a first state, a second state, and a third state, in which the operation object takes a first action on the target object in the first state, and in which the operation object maintains the first action on the target object in the second state, the second state occurs after the first state, and the third state occurs before the first state;

[0031] determining a distance between the operation object and the target object in the virtual space in the third state, the determined distance comprising a first distance and a second distance smaller than the first distance;

[0032] in response to determining the second distance, transitioning from the third state to the first state;

[0033] transitioning from the first state to the second state; and

[0034] A haptic signal is generated corresponding to each of the first state, the second state, and the third state.

[0035] In an example, the method further comprises:

[0036] transitioning from the second state to a fourth state; and

[0037] A haptic signal corresponding to the fourth state is generated.

[0038] In the example, the haptic changes at each transition between states.

[0039] In an example, the operation object is a finger, a hand, or a glove.

[0040] In an example, the first action of the first state is touching the target object.

[0041] In an example, the operating object grasps the target object during the second state.

[0042] In an example, the operating object approaches the target object in the third state.

[0043] In an example, the operation object releases the target object in the fourth state.

[0044] In an example, the third state is entered when the operation object is at a third distance serving as a threshold, the third distance being equal to or greater than the first distance.

[0045] In an example, the second distance is zero or a negative value.

[0046] In an example, at least one of the third state or the fourth state is triggered by a gesture.

[0047] In an example, the gesture includes curling a finger.

[0048] In an example, the first distance varies based on a size of the target object.

[0049] In an example, the weight of the target object corresponds to a magnitude or amount of the tactile sensation of the second state.

[0050] In an example, the method further includes: when the operating object takes the first action on the target object, adding at least one effect of glowing, flickering, and color change to at least one of the target object or the operating object.

[0051] In an example, the method further includes presenting the tactile sensation of the operation object and a second tactile sensation of a second operation object in opposite directions.

[0052] In an example, the target object represents a real object, and the virtual space is presented to a user controlling a robot in a real space including the real object.

[0053] In an example, the tactile sensation caused by the tactile signal includes pressure, and at least one of vibration or temperature change.

[0054] In an example, the operation object takes a second action, and the second action is a different action type from the first action.

[0055] In an example, the operation object takes a second action, and the second action is to repeat the first action again.

[0056] In an example, a program stored on a computer readable medium, when executed by a processor, causes the processor to: perform any of the methods or processes discussed above, or to perform in or with any component of the system.

[0057] In an example, a program stored on a computer readable medium, when executed by a processor, causes the processor to:

[0058] Determine a plurality of states of an operation object in a virtual space related to a target object, the states comprising a first state, a second state, and a third state, in which the operation object takes a first action on the target object in the first state, and in which the operation object maintains the first action on the target object in the second state, the second state occurs after the first state, and the third state occurs before the first state;

[0059] determining a distance between the operation object and the target object in the virtual space in the third state, the determined distance comprising a first distance and a second distance smaller than the first distance;

[0060] in response to determining the second distance, transitioning from the third state to the first state;

[0061] transitioning from the first state to the second state; and

[0062] A haptic signal is generated corresponding to each of the first state, the second state, and the third state. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] [ Figure 1 ] Figure 1 is a view showing an overview of tactile feedback.

[0064] [ Figure 2 ] Figure 2 is a block diagram showing a configuration example of a tactile presentation system.

[0065] [ Figure 3 ] Figure 3 is a flow chart illustrating generating haptic data.

[0066] [ Figure 4 ] Figure 4 is a view schematically showing an example of generating tactile data based on threshold 1 and threshold 2.

[0067] [ Figure 5 ] Figure 5 is a flow chart illustrating another example of generating tactile data.

[0068] [ Figure 6 ] Figure 6 Each is a view schematically showing an example of generating tactile data in a region.

[0069] [ Figure 7 ] Figure 7 is a flow chart illustrating another example of generating tactile data.

[0070] [ Figure 8 ] Figure 8Each is a view schematically showing an example of generating tactile data according to the speed of an operation object.

[0071] [ Fig. 9 ] Fig. 9 Each is a view showing an example of grip control.

[0072] [ Fig.10 ] Fig.10 Each is a schematic diagram showing a control example of grip control.

[0073] [ Fig.11 ] Fig.11 is a view showing an example in which the operation object is not a hand.

[0074] [ Fig.12 ] Fig.12 is a view showing an example in which the operation object is not a hand. DETAILED DESCRIPTION

[0075] Hereinafter, embodiments according to the present technical content will be described with reference to the accompanying drawings.

[0076] Presentation control of tactile feedback

[0077] Figure 1 is a view showing an overview of haptic feedback according to an embodiment of the present technology.

[0078] like Figure 1 As shown, when the operating object 1 (hand) operated by the user 3 takes a specific action on the target object 2, the tactile feedback device 10 worn by the user 3 presents a tactile sensation.

[0079] In this embodiment, the tactile feedback device 10 is worn on the finger of the user 3. That is, by presenting a tactile sensation such as pressing to the finger of the user 3, feedback on whether the operating object 1 successfully takes a specific action on the target object 2 is presented.

[0080] Note that although the device worn by one finger of user 3 is Figure 1 The tactile feedback device 10 shown in the figure, but the present technical content is not limited to this, and a device worn by two or more fingers can be used. In addition, the tactile sense presented by the tactile feedback device 10 is not limited to pressing, and can be vibration or temperature change. Alternatively, sound, light, etc. other than tactile sense can be presented to the user.

[0081] Specific actions include grasping or touching. In this embodiment, grasping includes the target object 2 following the movement of the operation object 1. In the case where the operation object 1 is a hand, examples of grasping include holding the target object 2 with the palm and moving the target object 2, and the movement of the target object 2 according to the movement of the fingertips of the operation object 1.

[0082] In addition, in the case where the laser pointer is the operating object, grasping may include continuously irradiating the target object with a laser beam, and the target object moves according to the direction and movement of the laser point (movement of the laser beam irradiation point). That is, when the target object follows, the target object may contact or separate from the operating object.

[0083] like Figure 1 As shown, the specific actions taken by the operating object 1 are set as four states. In this embodiment, taking grasping as an example, the four states of the initial stage of grasping determination, touching or grasping, holding, and releasing are set as specific actions. In addition, the tactile feedback device 10 presents different tactile sensations to the user for each of the initial stage of grasping determination, grasping, and holding states.

[0084] The initial grasping determination stage indicates a state in which the operating object 1 is almost able to grasp the target object 2. For example, when the operating object 1 is located within a predetermined distance from the target object 2, it is determined to be the initial grasping determination stage. That is, the initial grasping determination stage is a state before transitioning to the "grasping" state, and when the tactile feedback device 10 presents tactile sensation when entering the initial grasping determination stage, the user 3 can understand that the target object 2 is a graspable object.

[0085] It should be noted that the condition for determining the preliminary stage of grasping determination is not limited to the distance, and various conditions may be combined, such as time and the case where a predetermined operation has been performed within a predetermined distance. For example, in the case where the operation object 1 needs to be in contact with the target object 2 and the button of the controller needs to be pressed for five seconds while grasping the target object 2, a time period from three seconds to five seconds may be determined as the preliminary stage of grasping determination.

[0086] Grasping is the state when the operating object 1 is in contact with the target object 2. In this embodiment, in the grasping stage, the tactile feedback device 10 presents a stronger sense of touch than the sense of touch presented in the preliminary stage of grasping determination. In addition, in this embodiment, the "grasping" state indicates the moment of grasping. Generally, the "grasping" state indicates the moment when the operating object 1 is in contact with the target object 2. It should be noted that the sense of touch presented in grasping is not limited and can be set arbitrarily. For example, a sense of touch that is equal to or weaker than the sense of touch presented in the preliminary stage of grasping determination can be presented. In addition, for example, the sense of touch can be presented instantaneously in the preliminary stage of grasping determination, and the sense of touch of the same intensity can be presented instantaneously in grasping.

[0087] The continuous grasping is a state in which the operating object 1 continuously grasps the target object 2. In this state, the tactile feedback device 10 presents a tactile sensation indicating that the grasping is continuing. For example, during the continuous grasping, squeezing is continuously presented. It should be noted that the tactile sensation presented during the continuous grasping may always have the same intensity and may vary according to the operation of the operating object during the continuous grasping.

[0088] As a result, since the tactile intensities presented are different, it is easy to identify each state and the change in the current state. However, in the case where the performance of the device is limited, the tactile sensation of the same intensity can be presented, for example, the tactile sensation that can be presented by the tactile feedback device 10 has only two intensities, namely weak and strong.

[0089] Note that the strength of the tactile sense is not limited and can be set arbitrarily. For example, the strength of the tactile sense can be set based on a discrimination threshold in the user's tactile sense, or can be set manually by the user.

[0090] Release is a state in which the operating object 1 does not grasp the target object 2. In this state, no tactile presentation is performed from the tactile feedback device 10. It should be noted that the tactile presentation can be performed at the moment of entering the release state (ie, the moment when the operating object 1 releases the target object 2).

[0091] It should be noted that the above four states are examples when the specific action is grasping. That is, when the specific action is not grasping, the above four states can be restated as the first state, the second state, the third state and the fourth state, where the first state is the state when the operation object takes a specific action on the target object, the second state is the state when the operation object continues to take a specific action on the target object, the third state is the state before the transition to the first state, and the fourth state is the state when no specific action is taken.

[0092] For example, when pressing a button (target object) is a specific action, the state of the operating object approaching the button corresponds to the third state. In addition, the moment when the operating object touches the button corresponds to the first state, and the state of pressing and holding the button key corresponds to the second state.

[0093] In addition, when the operation target is flame and the specific action is burning the target object, the state of the flame approaching the target object corresponds to the third state. In addition, the moment when the flame contacts the target object corresponds to the first state, and the state of keeping burning corresponds to the second state.

[0094] As described above, by setting the third status as a preliminary stage for various specific actions, the user can clearly understand whether the specific action is successfully taken.

[0095] In this embodiment, one of the above four states is determined based on the object information including information about the position of the operation object and the target object information including information about the position of the target object.

[0096] The operation object 1 includes a hand, a robotic arm, etc. displayed in a virtual space. For example, the operation object 1 can be operated by tracking the movement of the tactile feedback device 10 worn by the user 3 (the movement of the user), or the user 3 can operate the operation object 1 by operating a controller on which the tactile feedback device 10 has been installed.

[0097] The object information includes position information, motion information, and speed information of the operation object 1. The motion information includes motions different from specific motions, such as making gestures such as bending the hand and bending the fingers when the operation object 1 is a hand, and rotating the arm 360 degrees when the operation object 1 is a robotic arm.

[0098] The target object 2 includes a virtual object displayed in a virtual space, a real object, etc. In this embodiment, the target object information of the target object 2 includes the position information and physical parameters of the target object. For example, the physical parameters include various parameters such as the size, shape, weight, hardness, viscosity, center of mass position or temperature of the target object. It should be noted that the position information can be a coordinate relative to the operation object 1 or an absolute coordinate.

[0099] Figure 2 is a block diagram showing a configuration example of the haptic system 5 .

[0100] like Figure 2 As shown, the tactile presentation system 5 includes a tracking device 6 , a video display unit 7 , a tactile feedback device 10 and an information processing device 20 .

[0101] The tracking device 6 tracks the movement of the user. The tracking device 6 includes a sensor and a camera for detecting the movement of the user, and uses an inside-out method or an outside-in method.

[0102] The video display unit 7 is a device that displays the operation object and the target object for the user. For example, a mobile terminal such as a smartphone, a head mounted display (HMD), an optical perspective augmented reality (AR), etc. can be used. In this embodiment, the user can visually recognize the target object and the operation object via the video display unit 7. In addition, the operation object can be checked with the naked eye of the user.

[0103] The tactile feedback device 10 includes a data communication unit 11 and a tactile presentation unit 12. The data communication unit 11 receives tactile data transmitted by the tactile data processing unit 24 of the information processing device 20, and provides the received tactile data to the tactile presentation unit 12. The tactile presentation unit 12 presents a tactile sensation suitable for the user based on the tactile data. It should be noted that the method of presenting tactile sensation to the user by the tactile presentation unit 12 is not limited. The pressure sensation may be presented using air pressure, or the temperature change may be presented using a cooling device.

[0104] It should be noted that the tactile feedback device 10 is not limited and any device may be used, such as a mouse, a controller, a sensor glove worn on the user's hand, and a smartphone that can vibrate.

[0105] The information processing device 20 includes a tracking information processing unit 21 , an information processing unit 22 , a tactile data generating unit 23 , and a tactile data processing unit 24 .

[0106] The tracking information processing unit 21 receives information on the movement of the user's hand (fingers), the movement of the user's eyes, the movement of the user's head, or the user's gaze direction tracked by the tracking device 6 to use as tracking information.

[0107] The information processing unit 22 determines which of the above four states the state of the operation object 1 and the target object 2 is based on the target object information and the object information. In this embodiment, the information processing unit 22 determines one of the grasping determination preliminary stage, grasping, continuous grasping, and release based on the position information of the operation object 1 and the position information of the target object 2.

[0108] In addition, the information processing unit 22 outputs visual information in the specific action to the video display unit 7. For example, in the case where the operating subject grasps the target object, it can be visually recognized how the target object follows according to the movement of the operating subject. In addition, when the operating subject takes a specific action on the target object, effects such as luminescence, flashing, and color changes can be added to the target object and the operating object.

[0109] The tactile data generating unit 23 generates tactile data presented to the user. In this embodiment, the tactile data generating unit 23 generates tactile data based on the four states of the initial stage of grasping determination, grasping, continuous grasping and releasing, the target object information and the object information. For example, in the case where the operating object grasps a heavier target object, tactile data for presenting a stronger tactile sensation than the case where a lighter target object is grasped is generated.

[0110] The tactile data processing unit 24 outputs the tactile data generated by the tactile data generating unit 23 to the tactile feedback device 10. Note that, depending on the tactile feedback that can be presented by the tactile feedback device 10, the tactile data to be output can be changed by the tactile data processing unit 24. For example, in the case where tactile data for presenting the pressure sense when grasping a harder object and the tactile sense (temperature change) when grasping a hotter object to the user is generated, and the tactile feedback device 10 is a device that can present only the pressure sense, the tactile data can be changed to data excluding the tactile data of the above-mentioned temperature change.

[0111] In an exemplary embodiment, the device structure of the tactile presentation system 5 may be similar to Figure 2 The system shown is different. For example, it is not an information processing device 20 including a tracking information processing unit 21, an information processing unit 22, a tactile data generation unit 23, and a tactile data processing unit 24, and a tactile feedback device 10 including a data communication unit 11 and a tactile presentation unit 12. As will be understood by those skilled in the art, the present disclosure is not limited to the structure shown. In an exemplary embodiment (not shown), the tactile data generation unit 23 and the tactile data processing unit 24 can be located in the tactile feedback device 10 together with the data communication unit 11 and the tactile presentation unit 12, while the tracking information processing unit 21 and the information processing unit 22 can be located in the information processing device 20. Various other structures having components in one or more separate physical devices can be implemented, or in some cases a single physical device can include all of the above components, and in some cases can also include a tracking device 6 and / or a video display unit 7 as in a fully integrated system.

[0112] It should be noted that in this embodiment, the information processing unit 22 corresponds to a determination unit that determines the first state, the second state, and the third state based on object information including position information about the operation object and target object information including position information about the target object, the first state is a state when the operation object takes a specific action on the target object, the second state is a state when the operation object continues to take a specific action on the target object, and the third state is a state before transitioning to the first state.

[0113] Note that, in this embodiment, the tactile data generating unit 23 and the tactile data processing unit 24 function as a presenting unit that presents tactile feedback to the user who operates the operation object based on the determination result.

[0114] It should be noted that in this embodiment, grasping corresponds to the first state, and the first state is the state when the operating object takes a specific action on the target object.

[0115] It should be noted that, in this embodiment, the continuous grasping corresponds to the second state, and the second state is a state in which the operating object continues a specific action on the target object.

[0116] It should be noted that in this embodiment, the initial stage of grasping determination corresponds to the third state, and the third state is a state before transitioning to the first state.

[0117] Figure 3 is a flow chart illustrating generating haptic data. Figure 4 Each of them is a view schematically showing generation of tactile data based on threshold 1 and threshold 2.

[0118] exist Figure 3 In the present invention, it is assumed that a user takes a specific action on a target object (virtual object) presented on a virtual space such as AR and virtual reality (VR).

[0119] The tracking device 6 performs tracking, and the operation object operated by the user is displayed on the virtual space (step 101 ). In this embodiment, the displayed operation object 1 approaches the target object 2 to grasp the target object 2 .

[0120] The information processing unit 22 determines whether the distance between the operation object 1 and the target object 2 is a threshold value 1 or less (step 102 ).

[0121] like Figure 4 As shown in A, threshold 1 and threshold 2 are set for target object 2. Figure 4 In A, the surface of the target object 2 is used as a reference (distance = 0), Threshold 1 is greater than Threshold 2, and Threshold 2 is set so that the distance is zero or a negative value. Alternatively, Threshold 1 may be set so that the distance is zero or greater.

[0122] Note that the threshold 1 and the threshold 2 may each be a fixed value, and may be changed according to parameters such as the size and shape of the target object 2 or parameters such as the size and shape of the operation object 1 .

[0123] In a case where the distance between the operation object and the target object is the threshold value 1 or less (Yes in step 102 ), the information processing unit 22 extracts tactile information and physical parameters from the target object (step 103 ).

[0124] The tactile information is information about the tactile sensation corresponding to the target object. For example, the pressure sensation (tactile information) is set according to the hardness of the target object. In the case where the target object is soft, the pressure sensation is set so that the user is presented with a feeling that the operating object sinks into the target object. In addition, various tactile sensations are set according to physical parameters such as the temperature and weight of the target object. In addition, in the case of grasping a lighter target object and a heavier target object, the tactile sensation presented when grasping the heavier target object is stronger.

[0125] The tactile data generating unit 23 generates tactile data based on the tactile information and the physical parameter (step 104). The tactile data processing unit 24 provides the generated tactile data to the tactile feedback device 10, and the tactile feedback device 10 outputs the tactile data (step 105). As a result, the user can recognize the tactile sensation in the preliminary stage of grasping determination.

[0126] The information processing unit 22 determines whether the distance between the operation object 1 and the target object 2 is a threshold value 2 or less (step 106 ).

[0127] In the case where the distance between the operation object 1 and the target object 2 is the threshold value 2 or less (Yes in step 106 ), the tactile feedback device 10 outputs tactile data (step 107 ). As a result, the user can recognize the tactile sense in the grasping.

[0128] exist Figure 4 In B, the vertical axis represents the tactile intensity (such as pressure) to be presented, and the horizontal axis represents the distance of the operation object. Figure 4 As shown in A, the operating object 1 approaches the target object 2 in the direction opposite to the horizontal direction.

[0129] In this embodiment, when the operation object 1 is between the threshold 1 and the threshold 2, it is determined to be the preliminary stage of grasping determination. In addition, when the operation object 1 is at the threshold 2 or less, it is determined to be grasping. In this case, Figure 4 As shown in FIG. 2B , the tactile data generating unit 23 generates tactile data to be presented to the user in a region (region 30 ) between the threshold 1 and the threshold 2 as a preliminary stage of grip determination.

[0130] For example, in the graph 31, as the operation object 1 gets closer to the target object 2, the pressure to be presented increases. In addition, in the case where the operation object 1 has reached the threshold value 2, it is determined to be grasping, and the pressure increases instantaneously. In the case where the operation object 1 further gets closer to the threshold value 2 (the operation object 1 sinks into the target object 2), it is determined to be continuous grasping, and the pressure to be presented further increases.

[0131] In addition, for example, in the graph 32 , when the operation object 1 reaches the threshold 1 and the threshold 2 , the pressure increases instantaneously.

[0132] In addition, for example, in Chart 33, pressure is presented only when the operation object 1 reaches Threshold 1. In addition, pressure is not presented between Threshold 1 and Threshold 2, and pressure is presented instantaneously when the operation object 1 reaches Threshold 2. In this case, when the operation object 1 reaches Threshold 1, the state of the preliminary stage of grip determination can be clearly indicated to the user by vibration, sound, etc. instead of pressure. As a result, the power consumption of the tactile feedback device 10 can be reduced.

[0133] In addition, for example, in the diagram 34, as the operation object 1 is closer to the target object 2, the pressure to be presented is greater. In this example, when the operation object 1 reaches the threshold 1, the preliminary stage of grasping determination is clearly indicated to the user through vibration, sound, etc., and when the operation object 1 reaches the threshold 2, the grasping is clearly indicated to the user through vibration, sound, etc. That is, instead of the tactile sensation to be presented, the preliminary stage of grasping determination or the state of grasping may be presented to the user through another mode.

[0134] In addition to the above examples, various changes in pressure and pattern presentation can be used to clearly indicate to the user the initial stage of grasping determination or the state of grasping. Note that in order to clearly indicate the grasping state to the user, the pressure is expected to change nonlinearly at threshold 2.

[0135] The information processing unit 22 determines whether the target object has been released from the operation object (step 108). For example, when the operation object 1 is located at a distance of a threshold value 1 or more from the target object 2, it can be determined to be released. In addition, for example, when the operation object 1 makes a predetermined gesture, etc., it can be determined to be released.

[0136] In the case where it is determined that the target object has been released from the operating object (Yes in step 108 ), the tactile data processing unit 24 stops outputting tactile data and stops presenting the tactile sensation from the tactile feedback device 10 (step 109 ).

[0137] Figure 5 is a flow chart illustrating another example of generating tactile data. Figure 6 Each is a view schematically showing an example of generating tactile data in a region.

[0138] exist Figure 5 In the process, when the operating object 1 makes a gesture (action information) in a predetermined area 40 around the target object 2, it is determined to be the initial stage of grasping determination (see Figure 6 A).

[0139] like Figure 5 As shown, the operation object operated by the user is tracked by the tracking device 6 and displayed in the virtual space (step 201).

[0140] The information processing unit 22 determines whether the operating object makes a gesture in the area of ​​the target object (step 202). Figure 6 As shown in A, when a gesture of bending fingers is made in area 40, it is determined to be in the preliminary stage of grasping determination.

[0141] In the case where the operating object makes a gesture within the region of the target object (Yes in step 202 ), the information processing unit 22 extracts tactile information and physical parameters from the target object (step 203 ).

[0142] The tactile data generating unit 23 generates tactile data based on the tactile information and the physical parameter (step 204). The tactile data processing unit 24 provides the generated tactile data to the tactile feedback device 10, and the tactile feedback device 10 outputs the tactile data (step 205). As a result, the user can recognize the tactile sensation in the preliminary stage of grasping determination.

[0143] The information processing unit 22 determines whether the operation object is in contact with the target object (step 206). It should be noted that the gesture of the operation object only needs to be performed once in the area 40, and does not need to be performed continuously. That is, after bending the finger once, the finger can remain straight. In addition, a gesture for releasing the initial stage of grasping determination can be set.

[0144] In the case where the operation object 1 and the target object 2 are in contact with each other (Yes in step 206 ), the tactile feedback device 10 outputs tactile data (step 207 ). As a result, the user can recognize the tactile sense in the grasping.

[0145] exist Figure 6 In B, the vertical axis represents the tactile intensity to be presented, and the horizontal axis represents the distance of the operation object.

[0146] like Figure 6 As shown in FIG. 2B , the tactile data generating unit 23 determines the time from when the operation object 1 makes a gesture in the area 40 to when it contacts the target object 2 (distance=0) as the initial stage of grasping determination. In addition, the tactile data generating unit 23 determines the time when the operation object 1 contacts the target object 2 (distance=0) as grasping, and determines the state in which the operation object sinks into the target object (distance<0) as continuous grasping.

[0147] For example, in the graph 41, when the operating object 1 bends the finger and moves closer to the target object 2, the pressure to be presented increases. In addition, in the case where the operating object 1 is in contact with the target object 2, it is determined to be grasping, and the pressure increases instantaneously. In the case where the operating object 1 further moves closer to the threshold 2 (the operating object 1 sinks into the target object 2), it is determined to be continuous grasping, and the pressure to be presented further increases.

[0148] In addition, for example, in the graph 42, when the operating object 1 bends the finger in the area 40, the pressure increases instantaneously. In addition, when the operating object 1 comes into contact with the target object 2, the pressure increases instantaneously.

[0149] In addition, for example, in the graph 43, pressure is present only when the operating object 1 bends the finger in the area 40. In addition, before the operating object 1 contacts the target object 2, no pressure is present, and when the operating object 1 contacts the target object 2, pressure is present instantly.

[0150] In addition, for example, in the graph 44, as the operating object 1 is closer to the target object 2, the pressure to be presented is greater. In this example, when the operating object 1 enters the area 40, the user is notified by vibration, sound, etc. that the operating object 1 has entered the area 40. In addition, when the finger is bent in the area 40, the preliminary stage of grasping determination is clearly indicated to the user by presenting pressure in addition to vibration or sound, and when the operating object 1 is in contact with the target object 2, grasping is clearly indicated to the user by vibration, sound, etc.

[0151] The information processing unit 22 determines whether the target object has been released from the operation object (step 208 ). For example, when the operation object 1 is outside the area 40 of the target object 2 , it can be determined to be released.

[0152] In addition, when a gesture such as bending a finger is made during gripping or continuous gripping, it can be determined as a preliminary stage of release determination as a state before transition to release. For example, in the preliminary stage of release determination, similar to gripping, instantaneous pressure can be presented when transitioning to this state, or the transition to the preliminary stage of release determination can be clearly indicated by another mode using vibration, sound, etc.

[0153] The method of transitioning from the release determination preliminary stage to release is not limited, and the release determination preliminary stage can be transitioned to release when the operating object has moved outside the area 40, when the finger is bent again, or when a gesture different from the gesture for transitioning to the release determination preliminary stage is made.

[0154] If it is determined to be released (Yes in step 208 ), the tactile data processing unit 24 stops outputting tactile data and stops presenting the tactile sensation from the tactile feedback device 10 (step 209 ).

[0155] Figure 7 is a flow chart illustrating another example of generating tactile data.

[0156] Figure 7 1 is a flowchart showing another example of generating tactile data corresponding to the speed at which the operation object 1 comes into contact with the target object 2 . Figure 8 Each is a view schematically showing an example of generating tactile data according to the speed of an operation object.

[0157] exist Figure 7 In the embodiment, when the operating object 1 contacts the target object 2, different controls of grasping or contact are performed according to the speed.

[0158] like Figure 7 As shown, the tracking device 6 performs tracking and displays an operation object operated by a user in a virtual space (step 301 ).

[0159] The information processing unit 22 determines whether the speed at which the operation object comes into contact with the target object is a threshold value or more (step 302 ).

[0160] When the speed at the time of contact is the threshold or greater (yes in step 302), the information processing unit 22 determines it as grasping, and performs tactile presentation in grasping (step 303). Hereinafter, the control of tactile presentation in grasping is referred to as grasping control. Similarly, the control of tactile presentation in continuous grasping is referred to as continuous grasping control, and the control of tactile presentation in release is referred to as release control.

[0161] The information processing unit 22 extracts tactile information and physical parameters during grip control from the target object (step 304 ).

[0162] The tactile data generating unit 23 generates tactile data based on the tactile information and the physical parameter (step 305 ). The tactile data processing unit 24 provides the generated tactile data to the tactile feedback device 10 , and the tactile feedback device 10 outputs the tactile data (step 306 ).

[0163] exist Figure 8 In B, the vertical axis represents the tactile intensity to be presented, and the horizontal axis represents the push amount of the operation object 1. The push amount represents the degree to which the operation object 1 pushes the target object 2. In grasping control, the push amount can be said to be the amount of the moving distance of the target object 2 following the operation object 1 with the initial position of the target object 2 as the origin.

[0164] like Figure 8 As shown in FIG. 2B , when the operating object 1 contacts the target object 2, pressure is instantaneously presented. In this embodiment, in the continuous grasping, the tactile sensation is continuously presented without changing the pressure presented when contacting. This is intended to clearly indicate to the user that the target object 2 follows the operating object 1. That is, since the operating object 1 moves with the target object 2 while in contact with the surface of the target object 2, no matter how the operating object 1 pushes the target object 2, the operating object 1 does not sink into the target object 2.

[0165] When the speed at the time of contact is the threshold or less (No in step 302 ), the information processing unit 22 determines that it is contact, and performs tactile presentation in contact (step 307 ). Hereinafter, control of tactile presentation in contact is referred to as contact control.

[0166] The information processing unit 22 extracts tactile information and physical parameters at the time of contact from the target object (step 308 ).

[0167] The tactile data generating unit 23 generates tactile data based on the tactile information and the physical parameter (step 309 ). The tactile data processing unit 24 provides the generated tactile data to the tactile feedback device 10 , and the tactile feedback device 10 outputs the tactile data (step 310 ).

[0168] exist Figure 8 In C, the vertical axis represents the tactile intensity to be presented, and the horizontal axis represents the push amount of the operation object 1. The push amount in the contact control is represented as a value that takes the surface of the target object 2 as the origin (distance = 0) and increases as the distance between the operation object 1 and the target object 2 decreases from 0.

[0169] like Figure 8 As shown in FIG. 3 , when the operating object 1 contacts the target object 2, pressure is presented, and the pressure increases according to the pushing amount of the operating object. It should be noted that the pressure to be presented can be arbitrarily set according to the physical parameters of the target object. For example, in the case of a harder target object, the pressure can increase exponentially relative to the pushing amount.

[0170] The information processing unit 22 determines whether the target object 2 has been released from the operating object 1 (step 311). If it is determined to be released (yes in step 311), the tactile data processing unit 24 stops outputting tactile data and stops presenting the tactile sensation from the tactile feedback device 10 (step 312).

[0171] It should be noted that the threshold value of the speed in contact can be set arbitrarily. For example, the threshold value can be set based on the speed at which the user operates the operation object. In addition, in the case where the target object has physical parameters (such as fragility), the threshold value can be set so that when the operation object contacts the target object at a speed of the threshold value or greater, the state cannot be transitioned to grasping control. In addition, when the real object is fragile, and when the operation object 1 approaches at a speed of the threshold value or greater, a tactile sensation indicating that the operation object is stopped or a warning can be presented to the user.

[0172] In addition, Figure 7 and Figure 8 The method for determining the preliminary stage of grasping determination is not limited. For example, in the case where the operating object 1 contacts the target object 2 at a speed of threshold 1 or greater, it can be determined to be the preliminary stage of grasping determination. In this case, when the speed of the operating object 1 is lower than a threshold 2 equal to or less than threshold 1, it can be determined to be grasping. In addition, threshold 1 and threshold 2 can be set to the same value so that the state immediately transitions to grasping, thereby omitting the preliminary stage of grasping determination.

[0173] In addition, the threshold 1 and the threshold 2 may each be a fixed value, and may be changed according to parameters such as the size and shape of the target object 2 or the size and shape of the operation object 1 .

[0174] In addition, as a method of determining the preliminary stage of grasping determination, various conditions may be combined in addition to determining using only the speed. For example, when the speed of the operating object 1 is a threshold value or more, and the distance between the operating object 1 and the target object 2 is within a predetermined range, it may be determined to be the preliminary stage of grasping determination. In addition, for example, when the speed of the operating object 1 is a threshold value or more and a gesture has been made in an area set to cover the target object 2, it may be determined to be the preliminary stage of grasping determination.

[0175] in addition, Figure 8 D is a view showing an example of tactile feedback in the case where a threshold value 1 and a threshold value 2 related to the speed of the operation object 1 are set. Figure 8 In D, the vertical axis represents the tactile intensity (such as pressure) to be presented, and the horizontal axis represents the speed of the operation object.

[0176] like Figure 8 As shown in D, when contact is made at a speed of threshold 1, a tactile sensation is presented instantaneously so that the user can recognize the transition to the preliminary stage of grasping determination. Similarly, when the speed is lower than threshold 2, a tactile sensation indicating grasping is presented instantaneously. It should be noted that the method of presenting a tactile sensation is not limited, and the tactile sensation can be presented similarly to the above-mentioned charts 31, 32, 33, and 34. Generally, it is only necessary to present a tactile sensation for notifying the transition at the moments of threshold 1 and threshold 2 (i.e., at each of the preliminary stage of grasping determination and grasping). Obviously, sound, light, etc. can be presented instead of a tactile sensation.

[0177] Grip Control

[0178] Fig. 9 Each is a view showing an example of grip control. Fig. 9 A is a diagram showing a control example based on the speed at which the operating object comes into contact with the target object.

[0179] In the above, the tactile sense is presented in the grip (ie, when the operation object is in contact with the target object). A specific control example of the grip control will be described below.

[0180] For example, Fig. 9 As shown in A, the fade-in degree of the pressure value can be controlled according to the speed at which the operating object contacts the target object. Fig. 9 In A, the vertical axis represents the pressure to be presented, and the horizontal axis represents the time after the manipulation object contacts the target object.

[0181] Fig. 9 The left graph of A shows the case where the speed of the operating object at the time of contact is high. Fig. 9The right graph of A shows a case where the speed of the operating object at the time of contact is low. In the left graph, the time until the final pressure value to be presented is set to be shorter according to the speed of the operating object. In addition, in the right graph, although the final pressure value is the same as the final pressure value in the left graph, the time to reach the final pressure value is extended because the speed of the operating object is lower than that in the left graph.

[0182] In addition, in the case where the operating object comes into contact with the target object at a speed of a threshold value or more, the target object and the tactile sensation presented by approaching (grasp determination preliminary stage) and grasping the target object may be eliminated.

[0183] In addition, in a case where the operating object comes into contact with the target object at a speed of a threshold value or less, (ie, the operating object grasps the target object), tactile feedback indicating the degree of speed of the operating object may be presented.

[0184] In addition, in grip control, the distance and arrival time until the operation object placed in the virtual space comes into contact with the target object can be calculated.

[0185] For example, the output timing may be offset prior to contact so that the pressure sensation is presented at a precise timing.

[0186] In addition, for example, the tactile feedback can be controlled to become stronger as the distance between the operating object and the target object decreases. As a result, it can be clearly indicated to the user that the operating object is gradually approaching the target object.

[0187] In addition, in grip control, the intensity of tactile feedback and the like can be changed according to the physical parameters of the target object.

[0188] For example, the hardness of the target object can be presented using the strength of the pressing. In addition, for example, in the case where the target object is set to an adhesive material or a detachable material, when the operating object contacts the target object, a stronger tactile feedback is presented first, and the tactile feedback can be controlled to weaken over time. As a result, the adhesive feeling of the target object can be presented to the user.

[0189] Fig. 9 B is a schematic diagram showing the extent to which the operation object sinks into the target object.

[0190] The sinking degree indicates the difference between the actual sensed position of the user's hand 50 operating the operation object and the visually displayed position of the operation object (operation object 1). By calculating the sinking degree, tactile feedback can be controlled based on the calculated sinking degree.

[0191] Note that as a determination of grasping, a state other than the moment when the operating object contacts the target object may be determined as grasping. For example, when the operating object is a hand, and the index finger is bent while the tip of the index finger is closer to the target object than the threshold value, the information processing unit 22 may determine that the user is attempting to grasp the target object, and the state may transition to grasping.

[0192] Note that the portion of the operating object that contacts the target object and is used to grasp the target object is not limited. For example, multiple fingers, the back of the hand, or the wrist may contact the target object, or an object following the operating object may contact the target object.

[0193] Continuous grip control

[0194] Fig.10 Each is a schematic diagram showing a control example of continuous grip control. Fig.10 10A, 10B, and 10C are each a view showing a specific control example of the continuous grip control.

[0195] exist Fig.10 In A, in the continuous grasp, when the target object 2 follows according to the movement of the operating object 1, the following delay changes according to the weight of the target object 2. By changing the pressure to be presented according to the amount of following delay, the weight of the target object 2 and the strength of the inertial force are presented to the user.

[0196] For example, Fig.10 As shown in A, it is assumed that a pressure of 5 kPa is presented to the user when grasping a relatively light target object 2. When the target object 2 moves in the direction of the arrow, the target object 2 follows while continuously contacting the operating object 1, and presents an inertial force (such as 5 kPa) according to the moving direction to the user.

[0197] In addition, for example, it is assumed that a 5 kPa pressure is presented to the user when grasping a heavier target object 2. When the target object 2 moves in the direction of the arrow, the target object 2 follows the movement of the operating object 1 with a delay. In this embodiment, the following delay means that the target object 2 follows the movement of the operating object 1 while being separated from the operating object 1. In addition, how much following delay is caused (that is, the degree of delay in terms of how much distance, time, etc. the target object 2 is delayed relative to the movement of the operating object 1) is referred to as the following delay amount.

[0198] The following delay amount is set based on the weight of the target object 2. In this embodiment, as the pressure presented to the user when causing the following delay, a pressure greater than that presented in grasping (eg, 8 kPa) is presented.

[0199] The amount of follow-up delay and the pressure to be presented when causing the follow-up delay can be set arbitrarily. For example, when the target object is heavier, the follow-up delay can be increased and the pressure to be presented can be changed more. In addition, for example, the amount of follow-up delay can be changed according to the speed of the operating object.

[0200] Fig.10 B shows an example of continuous grip control in the case where the user wears two or more tactile feedback devices on his fingertips. Fig.10 In the example shown in B, the haptic feedback device is worn by the user's thumb and index finger, and presents a tactile sensation.

[0201] like Fig.10 As shown on the left side of B, when the operating subject 1 has grasped the target object 2, a tactile sensation is presented to the thumb 55 and the index finger 56. Fig.10 As shown on the right side of FIG. 1B, when the operation object 1 moves in the direction indicated by the arrow 57, the tactile feedback device worn from the index finger 56 presents a stronger tactile sensation in the direction opposite to the direction indicated by the arrow 57. As a result, the sense of movement, the sense of weight, the strength of the inertial force, etc. are clearly expressed to the user.

[0202] In addition, when each of two or more fingers wears a tactile feedback device, the direction of the presented feeling can be changed according to the inclination of the hand. As a result, the user is presented with a feeling as if the target object moves in the operation object.

[0203] In the above example, the tactile sensation has been continuously presented in the continuous grasping control. The present technology is not limited thereto, and the tactile sensation may be presented only when the target object is grasped and released. Fig.10 As shown in FIG. 3 , the tactile sensation may be presented only within a predetermined period of time after the operating object contacts the target object, and the tactile sensation may be presented only within a predetermined period of time after the target object is released. As a result, the contact between the operating object and the target object may be clearly presented, and power consumption may be reduced during the continuous grasping of the target object.

[0204] Please note that if Fig.10 Continuous grip control of the heavier target object shown in A or Fig.10 In the case where the intensity of the tactile sensation to be presented during the continued grasping is changed as in the example shown in B, the tactile sensation can be presented during the continued grasping.

[0205] In addition, when the grasped target object collides with another object (virtual object or real object) in the continuous grasp control, the tactile sensation presented to the user can be changed. For example, the tactile sensation presented at the time of collision can be presented to be stronger. Fig.10 In the example shown in C, when the target object collides with another object during a period in which the tactile sense is not presented, the tactile sense may be presented at the time of the collision.

[0206] Release control

[0207] In the above-mentioned grip control and continuous grip control, specific examples of how to present a tactile sensation to a user have been shown. The following example of release control refers to how to stop presenting a tactile sensation to a user. That is, the control of tactile presentation includes presenting a tactile sensation to a user and stopping presenting a tactile sensation.

[0208] For example, it is assumed that information indicating the release of a target object at an arbitrary position is given, and the target object is automatically released when the user moves the target object to that position. In this case, the tactile presentation to the user stops when the target object is released. That is, when transitioning to the "release" state, the transition to the "release" state can be determined based on the position information of the operating object and the position information of the target object, and the tactile presentation can be controlled.

[0209] In addition, it is assumed that the target object can be released by performing an arbitrary command (operation) such as a user pressing a button. In this case, the tactile sensation presented to the user stops when the target object is released. That is, when transitioning to the "released" state, the state can transition to the "released" state based on the target object information and the object information and the information about the user operation.

[0210] In addition, the intensity of the tactile feedback when released can be changed according to the physical parameters of the target object. For example, when the target object is set to an adhesive material or a detachable material, a strong tactile sense can be presented first when the target object is released, and when the target object and the operating object are a certain distance away from each other or more, the tactile sense can be stopped. As a result, when the sticky target object is separated, a sticky feeling like being stuck can be presented to the user.

[0211] Fig.11 and Fig.12 Each is a view showing an example in which the operation object is not a hand.

[0212] exist Fig.11 and Fig.12 In the embodiment, the operation object 62 includes a cursor and an indicator. For example, the user 3 performs various operations such as moving and dragging the operation object 62 via a mouse or a controller. Examples of tactile presentation in each of the grasping determination preliminary stage, grasping, continuous grasping, and release states will be described with reference to these operations of the operation object 60 as specific actions.

[0213] That is to say, Fig.11 and Fig.12 The grasping determination preliminary stage in the example of represents a state in which the operating object (eg, a cursor) approaches the target object 61 (eg, a window) within a predetermined distance.

[0214] In addition, the grip indicates the state when the user 3 performs the first drag-and-drop operation of moving the window, that is, the state when clicking the window. In addition, it is obvious that in the case of performing a specific action by various operations such as pressing a keyboard (such as a shortcut key) and double-clicking, the state when performing the operation corresponds to the grip.

[0215] In addition, the continuous grasping indicates a state in which the window 61 follows according to the movement of the cursor 62. A typical example thereof includes a state in which the user 3 holds the cursor over the window and holds the mouse button. The present technical content is not limited thereto, and in a case in which the window can be grasped by holding the cursor over the window and clicking once, a tactile sense can be presented even in a state in which the mouse button is not held.

[0216] In addition, release indicates a state where the cursor 62 does not grip the window 61. Generally, release indicates that the user stops continuously pressing the mouse button.

[0217] like Fig.11 As shown, when the cursor 62 approaches the window 61 within a predetermined distance, the information processing unit 22 determines the initial stage of grasping determination. The tactile data generated by the tactile data generating unit 23 is output to the tactile feedback device 10 (such as a mouse), and the tactile sense is presented to the user. Similarly, the tactile sense is presented to the user according to each stage of grasping, continuous grasping and releasing.

[0218] Note that in addition to the drag and drop of a window, in the case of dragging and dropping a file into a folder, a single tactile sensation may be presented upon release as to whether the file can be properly placed into the folder.

[0219] In addition, when the haptic feedback device 10 is not installed in a mouse or the like, or when the haptic sense cannot be presented to the fingers by using a device such as a mouse, the haptic feedback device capable of presenting the haptic sense may be worn by a body part other than the fingers of the user.

[0220] Fig.12 An example of tactile presentation in the operation of computer-aided design (CAD) is shown. Fig.12 In the example, a component, a part, or the like serving as a target object 66 is grasped by the operation object 65 .

[0221] For example, Fig.12 As shown, when the operation object 65 approaches the target object 66, the grasping determination preliminary stage is determined. In addition, when the operation object 65 clicks and drags the target object 66, grasping and continuous grasping are determined respectively.

[0222] In addition, when the part 66 during the continuous grasping is in contact with another part, a tactile sensation can be presented. As a result, by presenting tactile feedback when the part is in contact with another part during the continuous grasping, contact information about whether the parts can be merged can be clearly perceived.

[0223] In addition, for example, to clarify grasping and continued grasping, the target object 66 can be highlighted. In addition, in the case of a component, multiple parts can be highlighted. In addition, in the grasping determination preliminary stage, the target object can be highlighted with a different color from that in grasping and continued grasping.

[0224] In addition, for example, the intensity of the tactile sensation to be presented may be changed according to the number of parts grasped.

[0225] Please note that Figures 1 to 11 Similarly, the preliminary stage of grasping determination, each state of grasping, continuous grasping and releasing, and the presentation of tactile sensation in each state are determined.

[0226] As described above, in the information processing device 20 according to the embodiment, grasping, continuous grasping, and a preliminary grasping determination stage are determined based on object information including position information of the operation object 1 and target object information including position information of the target object 2, grasping being a state when the operation object 1 grasps the target object 2, continuous grasping being a state when the operation object 1 continuously grasps the target object 2, and the preliminary grasping determination stage being a state before entering grasping, and tactile feedback is presented to the user 3 who operates the operation object 1 based on the determination result. As a result, usability can be improved.

[0227] In the past, there have been haptic devices that are worn by two or more fingers and grasp virtual objects. However, the configuration is complicated and the cost is increased. Therefore, there is a device that cannot present haptics to only one finger according to the intended use. At the same time, in the case of one finger, there is a problem that it is difficult to clearly express states such as grasping a virtual object and releasing it. For example, in the case of a haptic with only vibration or when a finger approaches a virtual object and forcibly grasps the virtual object, it is unclear whether it is grasped or released.

[0228] In the present technical content, the initial stage of grasping determination, that is, the state of transitioning to grasping, is clearly indicated to the user through tactile presentation. As a result, the virtual object can be grasped easily. In addition, since tactile feedback is presented to the fingertips, it is only necessary to prepare a compact and lightweight device including a system for acquiring and generating target object information and object information for determining each state of the initial stage of grasping determination, grasping, continuous grasping, and release.

[0229] <Other Implementation Methods>

[0230] The present technical content is not limited to the above-mentioned embodiments, and other various embodiments can be implemented.

[0231] In the above-mentioned embodiments, a virtual object presented on a virtual space such as AR and VR has been used as a target object. The present technical content is not limited to this, and is applicable even when a real object is used as a target object and a robotic arm is used as an operation object. In this case, the tactile sense is presented by a tactile feedback device installed on a controller that operates the robotic arm. Obviously, the tactile feedback device can be worn by the user's body instead of the controller.

[0232] For example, it is assumed that a user remotely operates a robot arm and checks the state of a factory including the robot arm via a video. In this case, by presenting a sense of touch when the robot arm grasps an object, each of the grasping determination preliminary stage, grasping, continuous grasping, and release states can be perceived by the user. Although various senses of touch can be presented to the user according to the complex movement of the robot arm, a simple configuration in which at least only the sense of touch is presented to the user can be adopted.

[0233] In addition, when the gripping robot arm or target object collides with another target object during continuous gripping, a stronger tactile sensation than the presented tactile sensation or a different tactile sensation may be presented. In addition, the tactile intensity to be presented may be used to allow the user to recognize the hardness of the target object.

[0234] In addition, a control signal may be transmitted that performs image recognition of the type of target object and moves the robot arm to a defined target point to automatically release the target object.

[0235] As a result, the user can perceive that the robot arm does not grasp the target object because the tactile sensation is not presented. In addition, since the tactile sensation stops when the target object is released, the user can know that the robot arm is in a state where it does not grasp anything.

[0236] In the above embodiment, the operating object (hand) operated by the user grasps the target object. The technical content is not limited to this. The user may operate a character of an application (such as a game), and when the character grasps the target object, a tactile sensation may be presented. For example, when the character grasps the target object (continuous grasping), the tactile sensation is presented, and when the character releases the target object (release), the tactile sensation stops being presented.

[0237] In addition, for example, a sense of touch can be presented so that the difference in the weight or center of gravity of a target object such as a sword and an axe can be understood. In addition, a sense of touch can be presented so that the difference in the strength of the inertial force when the target object is shaken can be understood. In addition, in addition to the target object, a sense of touch can be presented even when touching another character. That is, when approaching another character within a predetermined range (invisible wall), the preliminary stage of grasping determination can be determined, and when contacting another character, grasping can be determined, and a sense of touch can be presented.

[0238] In the above-mentioned embodiment, one target object is grasped. The present technology is not limited thereto, and multiple target objects may be grasped. In this case, a tactile sensation may be presented based on the number, total weight, etc. of the target objects.

[0239] In the above-mentioned embodiments, pressure sensing is mainly used as an example of presenting tactile sensation to the user. The present technical content is not limited to this, and each of the grip determination preliminary stage, grip, maintained grip, and release states can be clearly represented to the user through tactile sensation (such as temperature change). For example, suppose that the target object is set to have heat generation as a physical parameter. In this case, in the grip determination preliminary stage, a first temperature is presented. In addition, in gripping, a second temperature higher than the first temperature is presented. In addition, in continuous gripping, a third temperature that increases according to the time period of holding is presented. In addition, in releasing, a fourth temperature that decreases from the third temperature over time when released is presented.

[0240] The various configurations of the information processing unit, the tactile data generating unit, the tactile data processing unit, etc. described with reference to the accompanying drawings, and the control flow performed by the communication system are merely implementation methods, and any modifications may be made without departing from the essence of the present technical content. In other words, any other configuration, algorithm, etc. may be adopted for implementing the present technical content.

[0241] It should be noted that the effects described herein are illustrative only and not restrictive, and other effects may be present. The above description of multiple effects does not necessarily mean that these effects are present at the same time. This means that at least one of the above effects can be achieved, and it is possible to present effects not described in the present disclosure.

[0242] Among the characteristic parts of each embodiment described above, at least two characteristic parts can be combined. That is, the various characteristic parts described in each embodiment can be arbitrarily combined regardless of the embodiment.

[0243] It should be noted that the present technical content can adopt the following configurations. (1)

[0245] An information processing method, comprising:

[0246] Determine a plurality of states of an operation object in a virtual space related to a target object, the states comprising a first state, a second state, and a third state, in which the operation object takes a first action on the target object in the first state, and in which the operation object maintains the first action on the target object in the second state, the second state occurs after the first state, and the third state occurs before the first state;

[0247] determining a distance between the operation object and the target object in the virtual space in the third state, the determined distance comprising a first distance and a second distance smaller than the first distance;

[0248] in response to determining the second distance, transitioning from the third state to the first state;

[0249] transitioning from the first state to the second state; and

[0250] A haptic signal is generated corresponding to each of the first state, the second state, and the third state. (2)

[0252] The method according to (1), further comprising:

[0253] transitioning from the second state to a fourth state; and

[0254] A haptic signal corresponding to the fourth state is generated. (3)

[0256] The method of (1) or (2), wherein the haptic sensation changes at each transition between states. (4)

[0258] The method according to (1) to (3), wherein the operation object is a finger, a hand, or a glove. (5)

[0260] The method according to (1) to (4), wherein the first action of the first state is touching the target object. (6)

[0262] The method according to (1) to (5), wherein the operating object grasps the target object during the second state. (7)

[0264] The method according to (1) to (6), wherein the operation object approaches the target object in the third state. (8)

[0266] The method according to (2) to (7), wherein the operation object releases the target object in the fourth state. (9)

[0268] The method according to (1) to (8), wherein the third state is entered when the operation object is at a third distance as a threshold, the third distance being equal to or greater than the first distance. (10)

[0270] The method according to (1) to (9), wherein the second distance is zero or a negative value. (11)

[0272] The method according to (2) to (10), wherein at least one of the third state or the fourth state is triggered by a gesture. (12)

[0274] The method of (11), wherein the gesture comprises bending a finger. (13)

[0276] The method according to (1) to (12), wherein the first distance varies based on a size of the target object. (14)

[0278] The method according to (1) to (13), wherein the weight of the target object corresponds to a magnitude or amount of the tactile sensation of the second state. (15)

[0280] The method according to (1) to (14), further comprising: when the operation object takes the first action on the target object, adding at least one effect of glowing, blinking, and color changing to at least one of the target object or the operation object. (16)

[0282] The method according to (1) to (15), further comprising presenting the tactile sensation of the operation object and a second tactile sensation of a second operation object in opposite directions. (17)

[0284] The method according to (1) to (16), wherein the target object represents a real object, and the virtual space is presented to a user who controls a robot in a real space including the real object. (18)

[0286] The method according to (1) to (17), wherein the tactile sensation caused by the tactile signal includes pressure, and at least one of vibration or temperature change. (19)

[0288] The method according to (1) to (18), wherein the operation object takes a second action, the second action being of a different action type than the first action. (20)

[0290] The method according to (1) to (18), wherein the operation object takes a second action, and the second action is to repeat the first action again. (twenty one)

[0292] An information processing device, comprising:

[0293] At least one processor configured to:

[0294] Determine a plurality of states of an operation object in a virtual space related to a target object, the states comprising a first state, a second state, and a third state, in which the operation object takes a first action on the target object in the first state, and in which the operation object maintains the first action on the target object in the second state, the second state occurs after the first state, and the third state occurs before the first state;

[0295] determining a distance between the operation object and the target object in the virtual space in the third state, the determined distance comprising a first distance and a second distance smaller than the first distance;

[0296] in response to determining the second distance, transitioning from the third state to the first state;

[0297] transitioning from the first state to the second state; and

[0298] A haptic signal is generated corresponding to each of the first state, the second state, and the third state. (twenty two)

[0300] A system comprising:

[0301] The information processing device according to (21);

[0302] a tracking device configured to track the movement of a user;

[0303] a video display configured to display the target object and the operation object in the virtual space; and

[0304] A haptic device is provided to present the haptic sensation based on the haptic signal in each of the plurality of states. (twenty three)

[0306] The system of (22), further comprising a glove comprising the tracking device and the haptic device, and a head mounted display comprising the video display. (twenty four)

[0308] A program stored on a computer readable medium, which, when executed by a processor, causes the processor to:

[0309] Determine a plurality of states of an operation object in a virtual space related to a target object, the states comprising a first state, a second state, and a third state, in which the operation object takes a first action on the target object in the first state, and in which the operation object maintains the first action on the target object in the second state, the second state occurs after the first state, and the third state occurs before the first state;

[0310] determining a distance between the operation object and the target object in the virtual space in the third state, the determined distance comprising a first distance and a second distance smaller than the first distance;

[0311] transitioning from the third state to the first state in response to determining the second distance;

[0312] transitioning from the first state to the second state; and

[0313] A haptic signal is generated corresponding to each of the first state, the second state, and the third state. (25)

[0315] An information processing device, comprising:

[0316] a determining unit, the determining unit determining a first state, a second state, and a third state based on object information including position information about an operation object and target object information including position information about a target object, the first state being a state when the operation object takes a specific action on the target object, the second state being a state when the operation object continuously performs the specific action on the target object, and the third state being a state before transitioning to the first state; and

[0317] A presenting unit presents tactile feedback to a user who operates the operation object based on the determination result. (26)

[0319] The information processing device according to (25), wherein

[0320] The specific action includes an action of the operating object grasping the target object. (27)

[0322] The information processing device according to (25) or (26), wherein

[0323] The determination unit determines one of the first state, the second state, and the third state based on a distance to the target object. (28)

[0325] The information processing device according to (25) to (27), wherein

[0326] The determination unit determines the third state when the distance between the operation object and the target object is between a first threshold and a second threshold greater than the first threshold, and determines the first state when the distance between the target object and the operation object is less than the first threshold. (29)

[0328] The information processing device according to (25) to (28), wherein

[0329] The determining unit determines the second state when the distance between the operation object and the target object decreases after the first state is determined. (30)

[0331] The information processing device according to (25) to (29), wherein

[0332] As the distance between the target object and the operating object decreases, the tactile feedback of the presentation unit becomes stronger. (31)

[0334] The information processing device according to (25) to (30), wherein

[0335] The object information includes action information related to the action of the operation object; and

[0336] The determination unit determines that the third state is a case where the action according to the action information is performed in the region referring to the target object. (32)

[0338] The information processing device according to (25) to (31), wherein

[0339] The determination unit determines the first state when the speed of the operation object is a threshold value or more and the operation object is in contact with the target object. (33)

[0341] The information processing device according to (25) to (32), wherein

[0342] The target object information includes physical parameters related to the target object; and

[0343] The physical parameter includes at least one of a size of the target object, a shape of the target object, a weight of the target object, a hardness of the target object, a viscosity of the target object, a center of mass position of the target object, or a temperature of the target object. (34)

[0345] The information processing device according to (33), wherein

[0346] The determining unit determines, based on the speed of the operating object and the physical parameter, a fourth state in which the operating object does not take the specific action on the target object. (35)

[0348] The information processing device according to (33) or (34), wherein

[0349] The presentation unit presents the tactile feedback based on the physical parameter. (36)

[0351] The information processing device according to (25) to (35), wherein

[0352] As the weight of the target object increases, the tactile feedback of the presentation unit becomes stronger. (37)

[0354] The information processing device according to (25) to (36), wherein

[0355] As the size of the target object increases, the tactile feedback of the presentation unit becomes stronger. (38)

[0357] The information processing device according to (25) to (37), wherein

[0358] When in the second state and the target object is in contact with another object, the tactile feedback of the presentation unit is stronger. (39)

[0360] The information processing device according to (25) to (38), wherein

[0361] The tactile feedback includes at least one of pressure, vibration, sound, temperature or light. (40)

[0363] The information processing device according to (25) to (39), wherein

[0364] When transitioning from the third state to the first state, the tactile feedback of the presentation unit becomes stronger instantaneously. (41)

[0366] The information processing device according to (33) to (40), wherein

[0367] The tactile feedback of the presentation unit is stronger based on the physical parameter in the second state and the moving position of the operation object. (42)

[0369] The information processing device according to (25) to (41), wherein

[0370] The presentation unit controls the tactile feedback based on a moving direction of the operation object and a moving position of the operation object in a second state. (43)

[0372] An information processing method executed by a computer system, comprising:

[0373] determining, based on object information including position information about an operation object and target object information including position information about a target object, a first state, a second state, and a third state, the first state being a state when the operation object takes a specific action on the target object, the second state being a state when the operation object continuously performs the specific action on the target object, and the third state being a state before transitioning to the first state; and

[0374] A tactile feedback is presented to a user operating the operation object based on the determination result. (44)

[0376] A method for causing a computer system to perform the following steps:

[0377] determining a first state, a second state, and a third state based on object information including position information about an operation object and target object information including position information about a target object, the first state being a state when the operation object takes a specific action on the target object, the second state being a state when the operation object continuously performs the specific action on the target object, and the third state being a state before transitioning to the first state; and

[0378] A tactile feedback is presented to a user operating the operation object based on the determination result.

[0379] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

[0380] Reference numerals

[0381] 1 Operation Object

[0382] 2 Objects

[0383] 3 Users

[0384] 10. Haptic feedback device

[0385] 20 Information processing equipment

[0386] 22 Information Processing Unit

[0387] 23. Tactile data generation unit

[0388] 24 Tactile Data Processing Unit

Claims

1. An information processing method, comprising: Determine a plurality of states of an operation object in a virtual space related to a target object, the states comprising a first state, a second state, and a third state, in which the operation object takes a first action on the target object in the first state, and in which the operation object maintains the first action on the target object in the second state, the second state occurs after the first state, and the third state occurs before the first state; determining a distance between the operation object and the target object in the virtual space in the third state, wherein the determined distance includes a first distance and a second distance smaller than the first distance; in response to determining the second distance, transitioning from the third state to the first state; transitioning from the first state to the second state; as well as A haptic signal is generated corresponding to each of the first state, the second state, and the third state.

2. The method according to claim 1, further comprising: transitioning from the second state to the fourth state; as well as A haptic signal corresponding to the fourth state is generated. The method of claim 1 , wherein the haptic sensation changes at each transition between states. The method according to claim 1 , wherein the operation object is a finger, a hand or a glove. The method of claim 1 , wherein the first action of the first state is touching the target object. The method according to claim 1 , wherein the operating object grasps the target object during the second state. The method according to claim 1 , wherein the operating object approaches the target object in the third state. The method according to claim 2 , wherein the operation object releases the target object in the fourth state. 9 . The method according to claim 1 , wherein the third state is entered when the operation object is at a third distance serving as a threshold, the third distance being equal to or greater than the first distance.

10. The method of claim 1, wherein the second distance is zero or a negative value. The method of claim 2 , wherein at least one of the third state or the fourth state is triggered by a gesture. The method of claim 11 , wherein the gesture comprises curling a finger.

13. The method of claim 1, wherein the first distance varies based on a size of the target object.

14. The method of claim 1, wherein a weight of the target object corresponds to a magnitude or amount of the tactile sensation of the second state.

15. The method according to claim 1, further comprising: When the operating object takes the first action on the target object, at least one effect of glowing, flickering, and color changing is added to at least one of the target object or the operating object.

16. The method according to claim 1, further comprising: The tactile sensation of the operation object and the second tactile sensation of the second operation object are presented in opposite directions.

17. The method of claim 1, wherein the target object represents a real object, and the virtual space is presented to a user controlling a robot in a real space including the real object.

18. The method of claim 1, wherein the tactile sensation caused by the tactile signal comprises pressure, and at least one of vibration or temperature change.

19. The method of claim 1, wherein the operation object takes a second action, the second action being of a different action type than the first action.

20. The method according to claim 1, wherein the operation object takes a second action, and the second action is to repeat the first action again.

21. An information processing device, comprising: At least one processor configured to: Determine a plurality of states of an operation object in a virtual space related to a target object, the states comprising a first state, a second state, and a third state, in which the operation object takes a first action on the target object in the first state, and in which the operation object maintains the first action on the target object in the second state, the second state occurs after the first state, and the third state occurs before the first state; determining a distance between the operation object and the target object in the virtual space in the third state, wherein the determined distance includes a first distance and a second distance smaller than the first distance; in response to determining the second distance, transitioning from the third state to the first state; transitioning from the first state to the second state; as well as A haptic signal is generated corresponding to each of the first state, the second state, and the third state.

22. A system comprising: The information processing device according to claim 21; a tracking device configured to track the movement of a user; a video display configured to display the target object and the operation object in the virtual space; as well as A haptic device is provided to present a haptic sensation based on the haptic signal in each of the plurality of states.

23. The system of claim 22, further comprising a glove comprising the tracking device and the haptic device, and a head mounted display comprising the video display.

24. A program stored on a computer readable medium which, when executed by a processor, causes the processor to: Determine a plurality of states of an operation object in a virtual space related to a target object, the states comprising a first state, a second state, and a third state, in which the operation object takes a first action on the target object in the first state, and in which the operation object maintains the first action on the target object in the second state, the second state occurs after the first state, and the third state occurs before the first state; determining a distance between the operation object and the target object in the virtual space in the third state, wherein the determined distance includes a first distance and a second distance smaller than the first distance; in response to determining the second distance, transitioning from the third state to the first state; transitioning from the first state to the second state; as well as A haptic signal is generated corresponding to each of the first state, the second state, and the third state.

Images