Haptic presentation device and virtual object haptic presentation system

Abstract

To provide a haptic presentation device and a virtual object haptic presentation system that can present a new and unprecedented tactile sensation to the operator of the haptic presentation device. [Solution] The system comprises an operating unit 37, a power output unit 31 that outputs power to move the operating unit 37, a power adjustment unit 32 that adjusts the power output by the power output unit 31, an operating position detection unit 33 that detects the operating position of the operating unit 37, and a control means 5 that controls the power output unit 31 and the power adjustment unit 32 based on the information detected by the operating position detection unit 33. The control means 5 vibrates the operating unit 37 by simultaneously operating the power output unit 31 and the power adjustment unit 32 when the operating position of the operating unit 37 is within a predetermined range, and controls the power output unit 31 and / or the power adjustment unit 32 to prevent the operating unit 37 from vibrating when the operating position of the operating unit 37 is outside the predetermined range.

Description

Technical Field

[0001] The present invention relates to a tactile presentation device that presents a sense of touch to an operator. The present invention also relates to a virtual object tactile presentation system that presents the sense of touch of a virtual object displayed on a display to an operator.

Background Art

[0002] Services that provide a virtual space called the metaverse using virtual reality (VR), augmented reality (AR), or mixed reality (MR) are known. In such services, an operator interface is provided that gives information to the operator's vision and hearing through video and sound. In recent years, in addition to vision and hearing, a tactile presentation device that presents a sense of touch to the operator's fingers or the like has been proposed (see, for example, Patent Documents 1 and 2).

[0003] According to the tactile presentation devices disclosed in Patent Documents 1 and 2, by variably controlling the strength of the magnetic field applied to the magnetorheological fluid, various tactile sensations can be presented to the operator.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The inventors of the present application have found that the origin of the sense of touch of the tactile presentation device differs in the load applied to the operator through the operation unit when the operator moves it by operating the operation unit and when it is moved by another external factor.

[0006] This invention is based on such a novel idea and aims to provide a haptic presentation device and a virtual object haptic presentation system that can present a new and unprecedented tactile sensation to an operator operating the haptic presentation device. [Means for solving the problem]

[0007] The tactile presentation device according to the present invention comprises an operating unit, a power output unit that outputs power for moving the operating unit, a power adjustment unit that adjusts the power output by the power output unit, an operating position detection unit that detects the operating position of the operating unit, and control means that controls the power output unit and the power adjustment unit based on information detected by the operating position detection unit. The power adjustment unit includes a first part that is rotatably held at a predetermined position relative to the operator's hand, a second part that is rotatable relative to the first part, and a power transmission adjustment means that adjusts the power transmitted between the first part and the second part. The power output from the power output unit is input to the second part, the power adjusted by the power transmission adjustment means is transmitted from the second part to the first part, and the adjusted power is output from the first part to the operating unit. The control means vibrates the operating unit by simultaneously operating the power output unit and the power transmission adjustment means of the power adjustment unit when the operating position of the operating unit is within a predetermined range, and controls the power output unit and / or the power adjustment unit to prevent the operating unit from vibrating when the operating position of the operating unit is outside the predetermined range.

[0008] The virtual object haptic presentation system according to the present invention comprises: a haptic presentation device that presents tactile sensations to an operator operating an operation unit; a display unit that displays an image including a virtual object; an operation position detection unit that detects the operation position of the operation unit; and control means that control the operation of the haptic presentation device and the image information output to the display unit. The haptic presentation device includes: an operation unit; a power output unit that outputs power to move the operation unit; and a power adjustment unit that adjusts the power output by the power output unit and supplies it to the operation unit. The power adjustment unit includes: a first part that is rotatably held at a predetermined position relative to the operator's hand; a second part that is rotatable relative to the first part; and a power transmission adjustment means that adjusts the power transmitted between the first part and the second part. The power output from the power output unit is input to the second part, the power adjusted by the power transmission adjustment means is transmitted from the second part to the first part, and the adjusted power is output from the first part to the operation unit. The control means controls the power output unit and the power adjustment unit based on the operating position of the operating unit detected by the operating position detection unit. When the operating position of the operating unit is within a predetermined range, the control means vibrates the operating unit by simultaneously operating the power output unit and the power transmission adjustment means of the power adjustment unit. When the operating position of the operating unit is outside the predetermined range, the control means controls the power output unit and / or the power adjustment unit to prevent the operating unit from vibrating. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a haptic presentation device and a virtual object haptic presentation system that can present a novel tactile sensation to an operator operating the haptic presentation device. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example configuration of the virtual object haptic presentation system according to this embodiment. [Figure 2] This is a functional block diagram of the virtual object haptic presentation system according to this embodiment. [Figure 3]This figure shows an example of an information table that stores tactile information of a virtual object according to this embodiment. [Figure 4] This is a perspective view of the haptic presentation device according to this embodiment. [Figure 5] This is a perspective view showing a part of the haptic presentation device according to this embodiment. [Figure 6] This is a cross-sectional view showing an example of a power adjustment unit. [Figure 7] This is a side view showing a part of the tactile presentation device according to this embodiment. [Figure 8] This is a partial cross-sectional view showing the power output section and the power adjustment section. [Figure 9] This is a plan view showing an example of AR marker installation. [Figure 10] This is an example of a video showing virtual objects of a hand and a gun displayed in a VR space. [Figure 11] This is an example of a video showing a virtual marker object and a real hand displayed in AR space. [Figure 12] This flowchart shows an example of processing operations performed in the virtual object haptic presentation system according to this embodiment. [Figure 13] This is an example of an image displayed in VR space showing a laser gun in the left hand and a machine gun in the right hand. [Figure 14] This figure shows the state of the first attachment part when the operator's hand is gripping a virtual gun. This figure only shows the first attachment part of the tactile presentation device, and other components of the tactile presentation device are not shown. [Figure 15] This graph shows the relationship between torque output from the power output unit via the power adjustment unit and time. (a) shows an example of periodic torque fluctuation used when presenting tactile sensations of a virtual object of a laser gun, and (b) shows an example of periodic torque fluctuation used when presenting tactile sensations of a virtual object of a machine gun. [Figure 16] This is an example of a video displayed in VR space showing the state when a virtual electric ball is grasped by a virtual operator's left hand. [Figure 17]This is a flowchart showing an example of the processing operations executed in the virtual object tactile presentation system according to this embodiment. [Figure 18] This is a flowchart showing an example of the processing operations executed in the virtual object tactile presentation system according to this embodiment.

Mode for Carrying Out the Invention

[0011] Hereinafter, a virtual object tactile presentation system according to an embodiment of the present invention will be described with reference to the drawings. As shown in FIGS. 1 and 2, the virtual object tactile presentation system 1 according to this embodiment includes a head-mounted display device 2, a tactile presentation device 3, and a positioning system 4.

[0012] <Configuration of the Head-Mounted Display Device 2> The head-mounted display device 2 includes a main body portion 21, a first control portion 22, a display portion 23, an input portion 24, an audio output portion 25, a camera 26, a storage portion 27, a first communication portion 28, and the like.

[0013] The main body portion 21 is composed of a housing or the like that houses or attaches the display portion 23, the input portion 24, the audio output portion 25, the camera 26, the storage portion 27, and the like. The main body portion 21 is formed in a shape and size that covers both eyes of the operator.

[0014] The first control portion 22 controls each part of the head-mounted display device 2. The first control portion 22 is configured, for example, by installing an OS (operation system) and an application program in a ultra-small computer. When the application program is executed, the first control portion 22 functions as a display control portion that controls the image information output to the display portion 23, an audio control portion that controls the audio data output to the audio output portion 25, a camera control portion that performs drive control of the camera 26, and the like.

[0015] The display unit 23 displays video based on video information input from the first control unit 22. The display unit 23 may be for VR, AR, or MR. In this embodiment, the display unit 23 functions as both a VR and an AR display unit by switching between VR and AR modes.

[0016] The input unit 24 accepts various types of information. Examples of inputs from the input unit 24 include input via physical keys, input via soft keys such as a touch panel, and voice input from a microphone. The input from the input unit 24 may also be input that receives the operator's selection information using a selection menu displayed in the virtual reality space. The input from the input unit 24 is not limited to the above, and various input methods are possible.

[0017] The audio output unit 25 generates sound based on the audio information input from the first control unit 22.

[0018] The camera 26 performs an imaging operation in accordance with the command of the first control unit 22 and acquires image information of the real world. The image information of the real world is used as video information displayed on the display unit 23 when in AR mode.

[0019] The memory unit 27 stores a tactile information database 271, a tactile information database for vibration processing 271A, an image information database 272, and an audio information database 273.

[0020] The tactile information database 271 contains tactile information for one or more virtual objects. As tactile information for virtual objects, information tables 271a to 271d can be used, for example, as shown in Figure 3, in which displacement, current value, frequency, and duty cycle are associated with each other. Information tables 271a to 271e are used as tactile information for each of the operator's five fingers, respectively. The vibration processing tactile information database 271A is read by the first control unit 22 when performing vibration processing, which will be described later, and stores current value, frequency, and duty cycle. Like the tactile information database 271, the vibration processing tactile information database 271A also stores information corresponding to each of the operator's fingers. The displacement corresponds to the rotation angle from the initial position of the arm 327a, which will be described later, and the current value, frequency, and duty cycle correspond to the current value, frequency, and duty cycle of the current supplied to the power adjustment unit 32, respectively. In addition, the displacement may not be a value corresponding to the rotation angle from the initial position, but rather a value corresponding to the rotation angle from the point where the finger contacts the virtual object.

[0021] The image information database 272 contains image information displayed on the display unit 23. The image information includes still image information and / or video information such as virtual spaces and virtual objects.

[0022] The audio information database 273 contains audio information output from the audio output unit 25. Much of the audio information is associated with still image information and moving image information displayed on the display unit 23 and used together with the image information as audiovisual information.

[0023] The first communication unit 28 transmits and receives various information with the second communication unit 40 provided in the haptic presentation device 3 and the communication unit of the fixed-position camera 41 of the positioning system 4, which will be described later. In this embodiment, the first communication unit 28 performs wireless communication with the second communication unit 40 and the communication unit of the fixed-position camera 41. Bluetooth® can be used as an example of wireless communication. Wired communication may be used instead of wireless communication. Wireless communication using other communication protocols may be used instead of Bluetooth®.

[0024] <Configuration of haptic presentation device 3> As shown in Figures 2 to 5, the tactile presentation device 3 includes a power output unit 31, a power adjustment unit 32, an operation position detection unit 33, a second control unit 34, a transmission mechanism 35, a holding member 36, a first mounting unit 37, a second mounting unit 38, a power supply unit 39, and the like. The tactile presentation device 3 of this embodiment is worn from the operator's hand to the wrist and presents tactile sensations to each of the operator's five fingers. Furthermore, the tactile presentation device 3 is equipped with five sets of power output units 31, power adjustment units 32, transmission mechanisms 35, and first mounting units 37 in order to individually present tactile sensations to each of the operator's five fingers.

[0025] The power output unit 31 outputs power to move the first mounting unit 37, which is operated by the operator. In this embodiment, the power output by the power output unit 31 is rotational power, and a motor is used as the power output unit 31. The rotational power is output from the motor's rotating shaft 311.

[0026] The power adjustment unit 32 adjusts the power output from the power output unit 31 and outputs the adjusted power to the first mounting unit 37 via the transmission mechanism 35. The power adjustment unit 32 is located on the power transmission path between the first mounting unit 37 and the power output unit 31.

[0027] Here, a specific example of the configuration of the power adjustment unit 32 will be explained with reference to Figure 6. The power adjustment unit 32 shown in Figure 6 consists of a rotating shaft 321, a disc 322, a first yoke 323, a second yoke 324, a coil 325 which is a magnetic field generating unit, a magnetorheological fluid 326, a case 327, and the like.

[0028] The disc 322 is an internal rotor that rotates inside the power adjustment unit 32 and is constructed using a magnetic material. A rotating shaft 321 made of a non-magnetic material is fixed to the center of the disc 322. The rotating shaft 321 is supported by a bearing 321b in a shaft hole 323a provided in the first yoke 323.

[0029] The first yoke 323 faces one surface 322a of the disc 322 with a small gap between them. This first yoke 323 is fitted into a cylindrical case 327.

[0030] The second yoke 324 faces the other surface 322b of the disc 322 with a small gap between them. The first yoke 323 and the second yoke 324 are arranged at a predetermined distance apart by a cylindrical casing 328.

[0031] The coil 325 is formed by winding coil wire onto a bobbin 325a. The coil 325 is disposed within an annular groove 323b formed in the first yoke 323.

[0032] The magnetorheological fluid 326 is sealed in the space enclosed by the first yoke 323, the second yoke 324, and the casing 328, in the gap between the disk 322 and the first and second yokes 323 and 324. This magnetorheological fluid 326 is a liquid obtained by dispersing magnetic particles in a dispersion medium. As magnetic particles, for example, nano-sized magnetizable metal particles (metal nanoparticles) can be used.

[0033] A bevel gear 321a is fixed to the tip of the rotating shaft 321. The bevel gear 321a meshes with a bevel gear 311a fixed to the rotating shaft 311 of the power output unit 31. Therefore, the power generated by the power output unit 31 is input to the rotating shaft 321 of the power adjustment unit 32 via the bevel gears 321a, 311a.

[0034] The power adjustment unit 32 can be divided into three parts: a first part 32A rotatably held by a holding member 36 (described later), a second part 32B rotatably mounted relative to the first part 32A, and a magnetorheological fluid 326 interposed between the first part 32A and the second part 32B. The force transmitted between the first part 32A and the second part 32B increases in proportion to the strength of the magnetic field applied to the magnetorheological fluid 326. The first part 32A includes a first yoke 323, a second yoke 324, and a case 327. The second part 32B includes a rotating shaft 321 and a disc 322. The second part 32B rotates relative to the first part 32A when power from the power output unit 31 is input via bevel gears 311a and 321a. In this embodiment, the magnetic field generating section is composed of a coil 325, a first yoke 323, and a second yoke 324, and these components also constitute the first section 32A.

[0035] The power generated by the power output unit 31 is adjusted by the force transmitted between the first part 32A and the second part 32B of the power adjustment unit 32. The power adjusted in the power adjustment unit 32 is output from the L-shaped arm 327a fixed to the outer circumference of the case 327 and transmitted to the first mounting unit 37 via the transmission mechanism 35.

[0036] The L-shaped arm 327a is rotatable within a predetermined rotation angle. Specifically, it is rotatable between a rotation position where the extension direction from the bent portion of the arm 327a toward the tip is approximately perpendicular to the back of the glove (second attachment portion 38) (the back of the operator's hand) (see Figure 5; however, the glove is not shown in the same figure) and a rotation position where the extension direction from the bent portion of the arm 327a toward the tip is approximately parallel to the back of the glove (second attachment portion 38) (see Figure 7). Hereinafter, as shown in Figure 5, the rotation position where the extension direction from the bent portion of the arm 327a toward the tip is approximately perpendicular to the back of the glove (the back of the operator's hand) is referred to as the "initial position". When the rotation position of the arm 327a is in the initial position, the operator's fingers, to which the first attachment portion 37 is attached at the fingertips, are extended (the state of the fingers when the hand is open).

[0037] In the power adjustment unit 32 having the above configuration, when power is supplied from the power supply unit 39 and current is applied to the coil 325 of the power adjustment unit 32, a magnetic field is applied to the magnetorheological fluid 326 inside the power adjustment unit 32, and the torque transmitted between the first part 32A and the second part 32B changes according to the strength of the current applied to the coil 325. The power adjustment unit 32 provides a transmission torque from the second part 32B to the first part 32A that is of a magnitude corresponding to the current value applied to the coil 325. As a result, the power adjustment unit 32 can adjust the power input from the power output unit 31 to the second part 32B, and can output the adjusted power to the first mounting part 37 via the arm 327a and the transmission mechanism 35.

[0038] If the power adjustment unit 32 is configured as shown in Figure 6, for example, when current is applied to the coil 325, a magnetic path is formed within the disk 322, the first yoke 323, and the second yoke 324 along the direction indicated by the dashed arrow P. This magnetic path penetrates the magnetorheological fluid 326 interposed in the gap between one surface 322a of the disk 322 and the first yoke 323, and the magnetorheological fluid 326 interposed in the gap between the other surface 322b of the disk 322 and the second yoke 324. As a result, viscosity (shear stress) is generated in the magnetorheological fluid 326 according to the strength of the magnetic field, and the torque transmitted between the disk 322 and the first yoke 323 and the second yoke 324 changes according to the strength of the magnetic field.

[0039] The operation position detection unit 33 detects the operation position of the first mounting unit 37, which is the operation unit. The operation position detection unit 33 detects the operation position of the first mounting unit 37 by detecting the rotational position of the arm 327a. In this embodiment, the operation position detection unit 33 is configured using an angle sensor. As shown in Figure 8, the angle sensor 3292 is positioned on the holding member 36 so as to face the holder 3293 which is attached to one axial side of the power adjustment unit 32. Further outside of this, a control board 3291 which functions as a second control unit 34, a second communication unit 40, etc., is positioned. The angle sensor 3292 detects the rotational position of the arm 327a by detecting the rotation angle of the holder 3293. The control board 3291 is equipped with elements for controlling the current supplied to the coil 325 of the power output unit 31 and the power adjustment unit 32, wireless communication elements, etc.

[0040] The second control unit 34 (the power output control unit 341 and MRFD control unit 342, described later) controls the power output unit 31 and the power adjustment unit 32, respectively, based on the operation position information detected by the operation position detection unit 33. The second control unit 34 can change the control parameters supplied to the power output unit 31 and / or the power adjustment unit 32 based on the operation position information. The control parameters are parameters that govern the operation of the power output unit 31 and the power adjustment unit 32, and examples include the current value of the supplied current, the fluctuation frequency of the current value, and the duty cycle of the current value.

[0041] The second control unit 34 functions as a power output control unit 341, an MRFD control unit 342, an angle acquisition unit 343, an inertial measurement information acquisition unit 344, and the like.

[0042] The power output control unit 341 controls the motor, which is the power output unit 31.

[0043] The MRFD control unit 342 controls the current supplied to the coil 325 of the power adjustment unit 32.

[0044] The angle acquisition unit 343 acquires the rotational position (rotation angle) of the arm 327a based on the information input from the angle sensor 3292.

[0045] The inertial measurement information acquisition unit 344 acquires inertial measurement information from the inertial measurement unit 43 (IMU) and transmits the acquired information to the head-mounted display device 2.

[0046] The transmission mechanism 35 (see Figure 5) transmits the power transmitted from the arm 327a to the first mounting part 37. In this embodiment, the transmission mechanism 35 includes a flexible member 351, a link mechanism 352, and a support member 353. The flexible member 351 and the link mechanism 352 are connected in series with each other. For example, a leaf spring is used as the flexible member 351.

[0047] One end of the flexible member 351 is rotatably connected to the tip of the arm 327a. The other end of the flexible member 351 is rotatably connected to one end of the link mechanism 352.

[0048] The support member 353 supports the flexible member 351 in a slidable manner while maintaining a distance from the operator's fingers. A portion of the support member 353 is fixed to the finger portion of the glove, which is the second mounting portion 38 described later (see Figure 4).

[0049] The link mechanism 352 (see Figure 5) receives power from one end of the flexible member 351 and transmits the input power to the first mounting part 37. The link mechanism 352 performs link operation by the power output of the power output unit 31 and / or by the operation of the first mounting part 37. The link mechanism 352 in this embodiment has a first link member 3521, a second link member 3522, a third link member 3523, and a fourth link member 3524. The first link member 3521 is a plate-shaped member having a substantially V-shape. One end of the first link member 3521 is rotatably connected to the other end of the second link member 3522.

[0050] A first mounting portion 37 is rotatably attached to the other end of the first link member 3521. One end of the third link member 3523 is rotatably connected to the middle portion of the first link member 3521. The middle portion of the first link member 3521 is located at the bent part of the first link member 3521. The other end of the third link member 3523 is rotatably attached to the fourth link member 3524. The fourth link member 3524 is fixed to the finger portion of the glove, which is the second mounting portion 38.

[0051] The other end of the flexible member 351 is fixed to one end of the second link member 3522. As a result, the flexible member 351 and the link mechanism 352 are rotatably connected.

[0052] The holding member 36 serves to hold the power output unit 31 and the power adjustment unit 32 in a predetermined position in the operator's hand. The holding member 36 has a bearing housing into which a bearing 360 is fitted, and the second yoke 324 of the power adjustment unit 32 is rotatably held within the bearing 360.

[0053] As previously described, the first mounting portion 37 is attached to the operator's fingertip, and is made of rubber, elastomer, or the like, to match the size of each fingertip. The first mounting portion 37 is rotatably attached to the other end of the first link member 3521. The first mounting portion 37 is directly attached to the operator's fingertip exposed from the glove. The power transmitted by the transmission mechanism 35 is applied to the operator's fingertip by the first mounting portion 37, causing the operator's finger to move. The first mounting portion 37 does not need to be limited to the specific form described above; it just needs to be a device (operating part) that operates in conjunction with the operator's finger and transmits vibrations etc. applied to the first mounting portion 37 as tactile sensation to the operator's finger.

[0054] In this embodiment, a glove worn on the operator's hand is used as the second mounting portion 38. A support member 353 and a fourth link member 3524 are fixed to the fingers of the glove. The glove (second mounting portion 38) illustrated in Figure 4 has an open fingertip, and the first mounting portion 37 is attached to the operator's fingertips exposed through the opening.

[0055] The power supply unit 39 comprises a battery 391, a case 392, a power line 393, and a mounting bracket 394. The power supply unit 39 supplies power to five power output units 31 and five power adjustment units 32 via the power line 393. The battery 391 is housed in the case 392. The mounting bracket 394 serves to attach the power supply unit 39 to the operator's arm. The mounting bracket 394 is a belt-shaped member with a hook-and-loop fastener, and can be wrapped around the operator's arm and held in place by the hook-and-loop fastener. The power supply unit 39 does not necessarily need to be attached to the operator's arm or body and may be installed in a location away from the operator.

[0056] In this embodiment, a power output unit 31, a power adjustment unit 32, a transmission mechanism 35, a first mounting unit 37, etc., are provided for each of the five fingers. However, the power output unit 31, power adjustment unit 32, transmission mechanism 35, first mounting unit 37, etc., may be provided for only 1 to 4 fingers.

[0057] When using the haptic presentation device 3, the operator puts gloves on their hands, attaches the power supply unit 39 to their arm, and attaches the first attachment part 37 to the tip of each finger. In this embodiment, when the arm 327a is in its initial position, the first attachment part 37 is positioned at the fingertip position when the operator extends their fingers, and when the arm 327a rotates approximately 90° in one direction (counterclockwise in Figure 7) from its initial position, the first attachment part 37 is positioned at the fingertip position when the operator lightly clenches their hand.

[0058] The positioning system 4 (see Figure 1) detects the position and orientation of the operator's hand in the world coordinate system (a coordinate system in virtual space) in real time. The positioning system 4 is composed of, for example, multiple fixed-position cameras 41, AR markers 42, an inertial measurement unit 43, a first control unit 22, a second control unit 34, and the like.

[0059] The fixed-position camera 41 captures the AR marker 42 (two-dimensional code) and transmits the captured information to the head-mounted display device 2. The fixed-position camera 41 is fixed to an immovable object such as a building or furniture. A webcam or the like can be used as the fixed-position camera 41. The first control unit 22 of the head-mounted display device 2 acquires (tracks) the position information of the operator's hand in the world coordinate system (position information of the AR marker 42) in real time based on the captured information of the AR marker 42 received from the fixed-position camera 41.

[0060] The inertial measurement unit 43 includes inertial sensors such as a 3-axis magnetic sensor, a 3-axis gyroscope, and a 3-axis accelerometer. The inertial measurement unit 43 is mounted on the back of the operator's hand (the back of the glove). The first control unit 22 of the head-mounted display device 2 acquires information such as the position, orientation, and acceleration of the operator's hand based on the measurement information from the inertial measurement unit 43 received from the inertial measurement information acquisition unit 344 of the haptic presentation device 3.

[0061] The AR markers 42 are placed in predetermined positions. For example, as shown in Figures 4 and 9, they are placed on each of the five sides of a cube-shaped base 44 fixed on the case 392 of the power supply unit 39 (hereinafter, the AR markers 42 placed on the five sides of the base 44 are also referred to as the "AR marker group 42"), but this is not limited to this arrangement. The fixed-position camera 41 photographs the group of AR markers 42 placed in predetermined positions, and the first control unit 22 tracks the position, orientation, and movement of the operator's hand by recognizing the state of each AR marker 42 from the captured information. By placing the AR markers 42 on the five sides, the operator's hand will remain within the field of view of the fixed-position camera 41 (position detection is possible) regardless of how it moves. On the other hand, if the AR marker 42 cannot be detected (for example, if the AR marker 42 is out of the field of view of the fixed-position camera 41, or if the operator's hand is moving at high speed and the AR marker 42 cannot be recognized), the first control unit 22 estimates the position of the hand using angle and acceleration data obtained from the inertial measurement unit 43 (gyro sensor + acceleration sensor). Specifically, using the hand position calculated from the last detected position of the AR marker 42 and the angle from the inertial measurement unit 43 as the origin, the hand position is updated by converting the acceleration data from the angle data obtained from the inertial measurement unit 43 into a world coordinate system. Then, when the AR marker 42 can be recognized, the information regarding the final hand position calculated using only the information obtained from the inertial measurement unit 43 is discarded, and the discrepancy is corrected so that the position is the one calculated using the AR marker 42 and the gyro sensor of the inertial measurement unit 43.

[0062] The first control unit 22, which functions as the control unit of the positioning system 4, obtains accurate hand position information by combining position information (coordinate system in virtual space: world coordinate system) obtained from the AR marker captured by the fixed-position camera 41 via the first communication unit 28 with the hand rotation measurement value (world coordinate system) obtained from the gyro sensor of the inertial measurement unit 43.

[0063] The reason for prioritizing the use of location information obtained from AR markers is that the location information obtained from AR markers is more accurate than the location information obtained from the inertial measurement unit 43. Because drift occurs in the location information obtained by the inertial measurement unit 43 (information calculated from the acceleration sensor), it is often difficult to estimate accurate location information.

[0064] As already described, the virtual object haptic presentation system 1 is controlled by a first control unit 22 and a second control unit 34. The first control unit 22 mainly controls various processes and operations in the head-mounted display device 2, and the second control unit 34 mainly controls various processes and operations in the haptic presentation device 3. In addition, the second control unit 34 may also control various processes and operations in the haptic presentation device 3 in cooperation with the first control unit 22. However, the control means 5 such as the first control unit 22 and the second control unit 34 only need to be capable of performing the above-mentioned control, and the number of control units and the location in which the control units are provided are not limited to those described above. For example, a separate computer capable of communicating with the head-mounted display device 2 and the haptic presentation device 3 may be provided, and this computer may perform some or all of the control performed by the first control unit 22 and the second control unit 34. Alternatively, for example, a storage device that stores a haptic information database 271, a vibration processing haptic information database 271A, etc., may be provided in the haptic presentation device 3, and some or all of the control performed by the first control unit 22 may be performed by the second control unit 34.

[0065] When using the virtual object haptic presentation system 1, a pairing process is performed between the head-mounted display device 2 and the haptic presentation device 3 in advance. In the case of the aforementioned positioning system 4, since it is equipped with a fixed-position camera 41 and an AR marker 42, the position of the operator's hand, etc., can be calculated in the world coordinate system. However, if the positioning system 4 consists only of the inertial measurement unit 43, settings are made to align the position and orientation of the operator's hand, calculated from the angle data, acceleration data, etc., detected by the inertial measurement unit 43, with the initial position in the world coordinate system. Note that if the direction of gravity in the virtual space is the same as the direction of gravity in the real space, the orientation of the hand will be the same in both the virtual and real spaces, so this can be omitted, and only the initial setting of the hand position needs to be performed.

[0066] To set the initial position, the operator first places the head-mounted display device 2 and haptic presentation device 3 on a horizontal platform with their palm facing downwards at a predetermined position. Markers, such as illustrations of palms, are placed at the positions on the horizontal platform where the palm should be placed. At this time, a predetermined head guidance means positions the head-mounted display device 2 in a predetermined position relative to the operator's hand (haptic presentation device 3). For example, by arranging a desk that serves as the horizontal platform and a chair with a large backrest (head guidance means) in a predetermined positional relationship, the operator can position the head-mounted display device 2 in a predetermined position relative to the operator's hand (haptic presentation device 3) by aligning their hand with the outline of a hand drawn on the desk and sitting with the back of their head lightly resting against the backrest of the chair.

[0067] Next, with the head-mounted display device 2 and the haptic feedback device 3 positioned in predetermined locations, the operator performs an "initial position determination operation." This "initial position determination operation" can be, for example, pressing a push button located on the head-mounted display device 2, the haptic feedback device 3, or another location while the head-mounted display device 2 is in VR mode. However, if the haptic feedback device 3 is worn on both hands, the "initial position determination operation" can also be performed by input from the haptic feedback device 3 (for example, bending a specific finger). This allows the "initial position determination operation" to be performed without moving the haptic feedback devices 3 on both hands from the horizontal platform. Once the "initial position determination operation" is performed, information indicating that the "initial position determination operation" has been performed is input to the head-mounted display device 2, and the first control unit 22 stores the initial position and orientation of the haptic feedback device 3 in the virtual space such as VR or AR, and based on the stored position and orientation information, reproduces a virtual hand object in the appropriate position in the VR space 231 (VR coordinates). When the "initial position determination operation" is performed, the display unit 23 displays a virtual object 232 resembling a hand in the VR space 231, as shown in Figure 10, tracing the hand movements of the operator in the real space. By setting in advance which hand the haptic presentation device 3 is for, if the haptic presentation device 3 is attached to the right hand, the virtual object 232 for the right hand is displayed; if the haptic presentation device 3 is attached to the left hand, the virtual object 232 for the left hand is displayed; and if the haptic presentation device 3 is attached to both hands, the virtual object 232 corresponding to each hand is displayed.

[0068] In addition, as another embodiment of the "initial position determination operation," the following operation may be performed. That is, with the head-mounted display device 2 in AR mode, the operator can perform a predetermined operation to display a marker virtual object 233 in the AR space 235, for example as shown in Figure 11. By placing the operator's real hand 234 over the marker virtual object 233 and maintaining that position, the operator can perform a predetermined hand action (such as making a fist) to set the initial position and orientation of the haptic presentation device 3 in the virtual space. The marker virtual object 233 is designed to guide the position and orientation of the operator's hand.

[0069] Next, the processing operations performed in the virtual object haptic presentation system 1 will be explained based on the flowchart in Figure 12. In the following explanation, it is assumed that the VR space 231 or AR space 235 is displayed on the display unit 23 when the operator performs a predetermined operation on the head-mounted display device 2. Furthermore, when the virtual hand object 232 displayed on the display unit 23 overlaps in position with another virtual object in the coordinate system of the virtual space, or when a virtual object grasped by the hand appears, the first control unit 22 reads the current value to be supplied to the coil 325 of the power adjustment unit 32 from the information of the angle sensor 3292 (information of the rotational position of the arm 327a) received from the haptic presentation device 3 and the haptic information of the other virtual object, and transmits it to the haptic presentation device 3. The second control unit 34 of the haptic presentation device 3 then supplies the current of the received current value to the coil 325 of the power adjustment unit 32.

[0070] In step S1, the first control unit 22 determines whether the operator's five fingers are in a position to hold the gun 236, for example, as shown in Figure 14. Specifically, it determines whether the operating positions of the five first attachment parts 37 attached to each of the operator's five fingers are in the "gun gripping position". To make this determination, the angle acquisition unit 343 of the second control unit 34 acquires information on the rotational position (rotation angle from the initial position) of the five arms 327a that indicate the operating positions of the five first attachment parts 37 and transmits it to the head-mounted display device 2. The first control unit 22 of the head-mounted display device 2 determines whether the received operating positions of the five first attachment parts 37 are in the preset "gun gripping position". Note that the direction in which the hand grips from the initial position of each arm 327a is defined as the forward rotation direction, and the opposite direction is defined as the reverse rotation direction.

[0071] The preset "gun gripping position" is determined by the first control unit 22 when, for example, the rotation angle from the initial position of the four arms 327a that rotate in response to the movements of the thumb, middle finger, ring finger, and little finger falls within a predetermined "first gripping angle" range (e.g., 70° to 80°), and the rotation angle from the initial position of the one arm 327a that rotates in response to the movement of the index finger falls within a predetermined "second gripping angle" range (e.g., 0° to 60°). In this step, the second control unit 34 controls the power output unit 31 and / or power adjustment unit 32 so that the four arms 327a corresponding to the first mounting unit 37 attached to those fingers do not rotate in the forward direction by more than a predetermined rotation angle, in order to prevent the fingers from bending beyond a predetermined position to match the shape of the gun. For example, if a servo motor is used as the power output unit 31, the power output units 31 corresponding to the four arms are turned off and locked, and the power adjustment unit 32 is controlled to maximize its torque. This prevents the rotation of these four arms 327a. In this case, rotation in the reverse direction is also prevented, so strain gauges that detect the force applied to each finger in the forward and reverse directions may be provided in the tactile presentation device 3. When a force applied in the reverse direction is detected, the power adjustment unit 32 is turned off, allowing rotation in the reverse direction.

[0072] If a positive result is obtained in step S1 (S1: YES), then in step S2, a virtual object showing the gun 237 being held in the hand 232 will be displayed on the display unit 23, for example, as shown in Figure 13. Figure 13 is an example where the operator's five fingers on both hands are in the position of holding a gun, with the laser gun 237a held in the left hand and the machine gun 237b held in the right hand, and a virtual object showing the gun being held in only that hand will be displayed.

[0073] On the other hand, if in step S1 the first control unit 22 determines that the operator's five fingers are not in a position to hold the gun 236 (S1: NO), the determination in step S1 is repeated.

[0074] In step S3, the first control unit 22 determines whether the operator's index finger is in the position to pull the trigger of the gun. Specifically, it determines whether the operating position of the first attachment unit 37 attached to the operator's index finger is in the "firing position". To make this determination, the angle acquisition unit 343 of the second control unit 34 acquires information on the rotational position (rotation angle from the initial position) of the arm 327a corresponding to the movement of the index finger and transmits it to the head-mounted display device 2. The first control unit 22 of the head-mounted display device 2 determines whether the operating position of the first attachment unit 37 that it has received is in the preset "firing position". If the display unit 23 shows the index finger of the virtual hand 232 and the trigger of the gun 237, the first control unit 22 changes the degree of curvature of the index finger of the virtual hand 232 displayed on the display unit 23 according to the rotational position information of the arm 327a received from the haptic presentation device 3, and also moves the trigger. The first control unit 22 may acquire information about the state of the operator's real hand's index finger through the camera 26, and based on this information, change the degree of curvature of the index finger of the virtual hand 232 displayed on the display unit 23, and may also move the trigger.

[0075] The first control unit 22 determines that the operating position of the first mounting unit 37 is the "firing position" when the rotation angle of the arm 327a, which rotates in response to the movement of the index finger, from its initial position falls within a predetermined angular range (for example, 70° to 80°). Note that the predetermined angular range (for example, 70° to 80°) used as a reference in this step S3 is a larger value than the "second gripping angle" used as a reference in step S1.

[0076] If a positive determination is made in step S3 (S3:YES), then in step S4, the first control unit 22 and the second control unit 34 perform vibration processing by simultaneously operating the power output unit 31 and the power adjustment unit 32 (supplying current to both the power output unit 31 and the power adjustment unit 32). The vibration processing includes a process that applies active vibration to the operator, which is generated when the disc 322 of the power adjustment unit 32 is moved by the power output unit 31.

[0077] During vibration processing, the first control unit 22 reads the information from the angle sensor 3292 (information on the rotational position of the arm 327a) received from the haptic presentation device 3 and the haptic information database 271A for vibration processing, and transmits the current values, frequencies, and duty cycles (hereinafter referred to as "current values, etc.") to be supplied to the coils 325 of the five power adjustment units 32 to the haptic presentation device 3. Meanwhile, the second control unit 34 of the haptic presentation device 3 supplies currents corresponding to the five types of current values, etc. received to the coils 325 of the five power adjustment units 32. At this time, the second control unit 34 controls the current output to the power output unit 31 so that the five arms 327a output rotational force in the reversal direction (the direction in which the hand is extended). In this embodiment, the current value supplied to the coil 325 of the power adjustment unit 32 corresponding to the movement of the index finger is a larger value than the current values, etc. supplied to the coils 325 of the power adjustment unit 32 corresponding to the movement of the other fingers. This applies a relatively strong vibration to the first attachment part 37 attached to the operator's index finger, and a relatively weak vibration to the first attachment parts 37 attached to the operator's fingers other than the index finger. It is also possible to apply vibration only to the first attachment part 37 attached to the operator's index finger, and not to the first attachment parts 37 attached to the other fingers. For example, this can be achieved by turning off the power output unit 31 and / or power adjustment unit corresponding to the finger to which vibration is not applied (stopping the sharing of current). Alternatively, the strongest vibration can be applied to the first attachment part 37 attached to the operator's index finger, and the vibration to the first attachment parts 37 attached to the other fingers becomes weaker the further away they are from the index finger. As for "vibration processing," instead of controlling the power adjustment unit 32 to apply vibration to the first mounting unit 37 as described above, or by controlling the current supplied to the power output unit 31 together with the power adjustment unit 32, the power output unit 31 may apply vibration to the first mounting unit 37. By controlling both the power output unit 31 and the power adjustment unit 32, the variations in vibration can be increased.

[0078] The vibration applied to the first attachment part 37, which is attached to the operator's index finger, is controlled by the MRFD control unit 342 of the second control unit 34 so that, for example, if the gun held in the hand is a laser gun 237a, the power (torque) of the power output unit 31 transmitted from the power adjustment unit 32 to the first attachment part 37 fluctuates periodically, as shown in Figure 15(a). Furthermore, if the gun held in the hand is a machine gun 237b, for example, as shown in Figure 15(b), the MRFD control unit 342 of the second control unit 34 controls the power supply to the coil 325 of the power adjustment unit 32 so that the power (torque) transmitted to the first attachment part 37 fluctuates with a longer period than in the case of the laser gun 237a. Note that the frequency in the case of the laser gun 237a can be, for example, 50 to 100 Hz, and the frequency in the case of the machine gun 237b can be, for example, 5 to 20 Hz. Regarding the duty cycle, both the laser gun 237a and the machine gun 237b can have a duty cycle of, for example, 80% to 90%.

[0079] On the other hand, if it is determined in step S3 that the operator's index finger is not in the position to pull the trigger of the gun (S3: NO), the determination in step S3 is repeated.

[0080] After starting the vibration processing in step S4, in step S5, the first control unit 22 determines whether the operator's index finger maintains the position of pulling the trigger of the gun for a predetermined time. Specifically, it determines whether the operating position of the first attachment part 37 attached to the operator's index finger maintains the "firing position" for a predetermined time.

[0081] If a positive determination is made in step S5 (S5:YES), then in step S6, the first control unit 22 and the second control unit 34 execute a "vibration enhancement process". As part of the "vibration enhancement process", the first control unit 22 and the second control unit 34 increase the output (torque and / or rotational speed) of the power output unit 31 by one step. Alternatively, instead of increasing the output of the power output unit 31 by one step as part of the "vibration enhancement process", the output of the power output unit 31 may be increased by one step, and the maximum value of the current supplied to the coil 325 of the power adjustment unit 32 (i.e., torque) may also be increased by one step.

[0082] On the other hand, if a negative determination is made in step S5 (S5: NO), the vibration processing is canceled in step S7. Specifically, the first control unit 22 and the second control unit 34 cancel the vibration processing by stopping the power supply to the power adjustment unit 32. It is also possible to cancel the vibration processing by stopping the power supply to the power output unit 31. After canceling the vibration processing, the processing procedure returns to step S3.

[0083] After executing the process in step S6, in step S8, the first control unit 22 determines whether the laser gun has run out of energy or the machine gun has run out of ammunition. Specifically, if a predetermined time has elapsed since the start of the vibration enhancement process in step S6, it determines that the laser gun has run out of energy or the machine gun has run out of ammunition. If the predetermined time has not elapsed since the start of the vibration enhancement process in step S6, it determines that the laser gun has not run out of energy or the machine gun has run out of ammunition.

[0084] If a positive result is obtained in step S8 (S8: YES), the vibration process is canceled in step S9. The cancellation of the vibration process in step S8 can be performed using the same method as the cancellation of the vibration process described in step S7.

[0085] By performing the series of processing actions described above, tactile sensations in the form of vibrations are imparted to the operator's index finger or all five fingers, only while the trigger of the gun is being held down.

[0086] Of the processing operations described above, steps S6 and S8 are optional. After making a positive determination in step S5, the vibration processing may be canceled (S9) and the series of processing operations may be terminated.

[0087] Furthermore, in the vibration processing of step S4 and the vibration enhancement processing of step S6, the rotational torque output by the power output unit 31 is greater than the rotational torque transmitted between the first and second parts of the power adjustment unit 32. However, if control is performed such that the rotational torque transmitted between the first and second parts of the power adjustment unit 32 is periodically switched on and off (if frequency is included in the parameters governing the operation of the power adjustment unit 32), vibration can be applied to the first mounting part 37 even if the rotational torque output by the power output unit 31 is less than the rotational torque transmitted between the first and second parts of the power adjustment unit 32.

[0088] Furthermore, in the vibration treatment of step S4 and the vibration enhancement treatment of step S6, vibration was applied to the first mounting part 37 by varying the current value supplied to the coil 325 of the power adjustment unit 32. However, the vibration applied to the first mounting part 37 may also be changed by varying the output torque of the power output unit 31.

[0089] Furthermore, in the vibration processing of step S4 and the vibration enhancement processing of step S6, the rotating shaft 311 of the power output unit 31 was rotating (also called "reverse rotation") to apply force in the direction in which the operator's fingers are opening (reverse direction), but the rotating shaft 311 may be made to rotate in the forward direction. Alternatively, the rotation may be periodically alternating between forward and reverse rotation depending on the position of the fingers (position of the first mounting unit 37) or the elapsed time.

[0090] Furthermore, in the vibration processing of step S4 and the vibration enhancement processing of step S6, the finger to which vibration is applied (first attachment part 37) may be changed depending on the size of the gun (laser gun, machine gun, etc.). In the example above, the strongest vibration was applied only to the index finger (only the first attachment part 37 attached to the index finger), but in the case of a larger gun, the same vibration may be applied to the index finger and the middle finger (the first attachment part 37 attached to the index finger and the first attachment part 37 attached to the middle finger).

[0091] Furthermore, in the vibration processing of step S4 and the vibration enhancement processing of step S6, if the gun is a machine gun, the vibration applied to the first attachment part 37 attached to the operator's finger may be changed according to the number of remaining bullets. For example, the cycle or number of times the power output unit 31 is turned on and off, or the cycle of turning the current supplied to the coil 325 of the power adjustment unit 32, may be changed according to the number of remaining bullets.

[0092] Furthermore, although the vibration processing was canceled in step S9, instead of canceling the vibration processing, a weaker vibration than the one applied to the first attachment part 37 in step S4 may be applied to the first attachment part 37 of each finger for a certain period of time, and then the vibration processing may be canceled. By doing so, the operator can more realistically experience running out of energy or ammunition.

[0093] Furthermore, in the above description, the virtual gun object 237 appeared when the operator made a predetermined hand shape. However, as shown in Figure 10, the virtual gun object 237 may be displayed on the display unit 23 in advance. In this case, the operator moves their hand to bring the virtual hand object 232 closer to the virtual gun object 237. Then, when the positions of the virtual hand object 232 and the virtual gun object 237 overlap and it is determined that the operator's hand is in the shape of holding a gun, the processing operations of steps S3 to S9 described above may be executed.

[0094] <Virtual objects other than laser guns and machine guns (e.g., electric balls)> In the embodiments described above, virtual objects of laser guns and machine guns were displayed on the display unit 23. However, other virtual objects may be displayed on the display unit 23 instead of laser guns and machine guns, and the tactile sensation of those virtual objects may be presented on the tactile presentation device 3.

[0095] For example, as shown in Figure 16, a virtual object 238 of an electric ball (charged ball) and a virtual object 232 of a hand may be displayed on the display unit 23, and vibration processing may be executed when the operator's hand is in a position to hold the virtual object. The specific processing operation will be explained based on the flowchart in Figure 17.

[0096] In other words, in step S11, the first control unit 22 determines whether the operator's fingers are in contact with the electric ball. Specifically, it determines whether the operating positions of the five first attachment parts 37, each attached to the operator's five fingers, are in a "virtual object contact position" where they would touch the electric ball. This determination is made for each finger. To perform this determination, the angle acquisition unit 343 of the second control unit 34 acquires information on the rotational position (rotation angle from the initial position) of the five arms 327a that indicate the operating positions of the five first attachment parts 37, and transmits this information to the head-mounted display device 2. The first control unit 22 of the head-mounted display device 2 determines whether the received operating positions of the five first attachment parts 37 are in a preset "virtual object contact position".

[0097] If the result in step S11 is positive (S11: YES), then in step S12, a virtual object is displayed showing the fingers of hand 232 touching the electric ball 238.

[0098] On the other hand, if in step S11 the first control unit 22 determines that the operator's finger is not in contact with the electric ball 238 (S1: NO), the determination in step S11 is repeated.

[0099] In step S12, after displaying a virtual object in which the fingers of hand 232 are touching the electric ball 238, in step S13, the first control unit 22 and the second control unit 34 perform vibration processing by simultaneously operating the power output unit 31 and the power adjustment unit 32 (supplying current to both the power output unit 31 and the power adjustment unit 32). However, this vibration processing is performed only for fingers that are determined to be touching the electric ball 238. The first control unit 22 changes the degree of bending of the fingers of the virtual hand 232 displayed on the display unit 23 according to the rotation position information of the arm 327a received from the haptic presentation device 3, and deforms the electric ball 238 according to the position and degree of bending of each finger. The first control unit 22 may also acquire information about the state of the operator's real hand fingers through the camera 26, and change the degree of bending of the fingers of the virtual hand 232 displayed on the display unit 23 based on this information, and deform the electric ball 238 according to the position and degree of bending of each finger.

[0100] During vibration processing, the first control unit 22 reads the information from the angle sensor 3292 (information on the rotational position of the arm 327a) received from the tactile presentation device 3 and the current value to be supplied to the coil 325 of the power adjustment unit 32 from the tactile presentation device 3 and the tactile information database 271A for vibration processing, and transmits it to the tactile presentation device 3. The second control unit 34 of the tactile presentation device 3 supplies the current of the received current value to the coil 325 of the power adjustment unit 32. Unlike the laser gun example, the second control unit 34 controls the current output to the power output unit 31 so that the arm 327a corresponding to the finger that is being touched outputs rotational force in the forward rotation direction (the direction in which the hand is gripping). This makes it possible to simultaneously provide a tactile sensation and vibration as if being sucked into an electric ball.

[0101] After performing the vibration processing in step S13, in step S14, the first control unit 22 determines whether the operator's fingers maintain a position of touching the electric ball for a predetermined time. Specifically, it determines whether the operating position of the first attachment part 37, which is attached to the operator's fingers that were determined to have touched the electric ball in step S12, maintains a "virtual object contact position" for a predetermined time.

[0102] If a positive determination is made in step S14 (S14: YES), in step S15, the first control unit 22 and the second control unit 34 perform a "vibration enhancement process". As part of the "vibration enhancement process", the first control unit 22 and the second control unit 34 increase the output (torque and / or rotational speed) of the power output unit 31 by one step. Alternatively, instead of increasing the output of the power output unit 31 by one step as part of the "vibration enhancement process", the maximum value of the current supplied to the coil 325 of the power adjustment unit 32 may be increased by one step in addition to increasing the output of the power output unit 31 by one step.

[0103] After executing the process in step S15, in step S16, the first control unit 22 determines whether or not the electric ball has run out of energy. Specifically, if a predetermined time has elapsed since the start of the vibration enhancement process in step S15, it is determined that the electric ball has run out of energy. If the predetermined time has not elapsed since the start of the vibration enhancement process in step S15, it is determined that the electric ball has not run out of energy.

[0104] If a positive determination is made in step S17 (S17: YES), and if a negative determination is made in step S14 (S14: NO), the vibration processing is canceled (step S18). Specifically, the first control unit 22 and the second control unit 34 cancel the vibration processing by stopping the power supply to the power adjustment unit 32. It is also possible to cancel the vibration processing by stopping the power supply to the power output unit 31.

[0105] Furthermore, in the vibration processing of step S13 and the vibration enhancement processing of step S15, various modifications of the vibration processing of step S4 and the vibration enhancement processing of step S6 can also be applied. In addition, in the vibration processing of step S13 and the vibration enhancement processing of step S15, the strength and type of vibration may be changed according to the number of times the electric ball is touched. The number of times the electric ball is touched can be determined, for example, by counting the number of times the operating position of the first mounting part 37 enters and exits a predetermined area (for example, an area where the rotation angle of the arm 327a is a predetermined angle (for example, 70°) or more) in the first control unit 22 or the second control unit 34.

[0106] Furthermore, in the vibration treatment of step S13 and the vibration enhancement treatment of step S15, the intensity and type of vibration may be changed according to the degree to which the electric ball is gripped. The degree to which the electric ball is gripped can be indicated by the rotation angle of the arm 327a, and the larger the rotation angle, the stronger the grip is considered to be.

[0107] Furthermore, in the vibration processing of step S13 and the vibration enhancement processing of step S15, it is preferable to periodically alternate between forward and reverse rotation of the rotating shaft 311 of the power output unit 31. At this time, it is even more preferable to synchronize the rotation of the five power output units 31, each corresponding to the movement of the five fingers. In this way, the operator's hand moves alternately in the gripping and opening directions without applying force, and the tactile sensation of the electric ball is presented more realistically.

[0108] <Virtual objects of guns other than laser guns and machine guns (handguns)> In the embodiments described above, virtual objects of laser guns and machine guns were displayed on the display unit 23. However, instead of laser guns and machine guns, virtual objects of other types of guns may be displayed on the display unit 23, and the tactile sensation of these virtual objects may be presented by the tactile presentation device 3.

[0109] For example, a virtual object of a handgun and a virtual object of a hand 232 may be displayed on the display unit 23. The specific processing operations will be explained based on the flowchart in Figure 18.

[0110] First, in step S21, the first control unit 22 determines whether the operator's five fingers are in the position of holding the pistol 236. This determination can be performed using the same method as described in step S1.

[0111] If the result in step S21 is positive (S21: YES), then in step S22, a virtual object is displayed that is holding a pistol in its hand.

[0112] On the other hand, if in step S21 the first control unit 22 determines that the operator's five fingers are not in the position to hold a pistol (S21: NO), the determination in step S21 is repeated.

[0113] In step S23, the first control unit 22 determines whether the operator's index finger is in the position to pull the trigger of the pistol. This determination can be performed using the same method as described in step S3. That is, the first control unit 22 determines that the operator's index finger is in the position to pull the trigger of the pistol when the rotation angle of the arm 327a, which rotates in response to the movement of the operator's index finger, from its initial position falls within a predetermined angular range (for example, 70° to 80°). When the display unit 23 displays a virtual index finger and the trigger of a gun, the first control unit 22 changes the degree of curvature of the virtual index finger displayed on the display unit 23 and moves the trigger according to the rotation position information of the arm 327a received from the tactile presentation device 3. The first control unit 22 may also acquire information about the state of the operator's real index finger through the camera 26 and change the degree of curvature of the virtual index finger displayed on the display unit 23 and move the trigger based on this information.

[0114] If a positive determination is made in step S23 (S23: YES), in step S24, the first control unit 22 and the second control unit 34 perform vibration processing by simultaneously operating the power output unit 31 and the power adjustment unit 32 (supplying current to both the power output unit 31 and the power adjustment unit 32).

[0115] The vibration processing in step S24 can be performed using the same method as described in step S4. However, in step S4, as long as the operating position of the first attachment part 37 attached to the index finger was within a predetermined range, the same vibration was applied to the first attachment part 37 even if the operating position of the first attachment part 37 moved within that predetermined range. In step S24, however, even while the operating position of the first attachment part 37 attached to the index finger is within a predetermined range, stronger vibration is applied as the first attachment part 37 moves in the direction of gripping the finger. That is, the first control unit 22 and the second control unit 34 control the power output unit 31 and the power adjustment unit 32 to achieve this. Furthermore, it is preferable to rotate the rotation axis 321 of the power output unit 31 in the direction of opening (reversing) the index finger. By doing so, the stronger the reaction force and vibration, the stronger the operator is presented with as the trigger is pulled, allowing the operator to experience a sensation similar to actually pulling the trigger of a pistol.

[0116] On the other hand, if it is determined in step S23 that the operator's index finger is not in the position to pull the trigger of the pistol (S23: NO), the determination in step S23 is repeated.

[0117] After starting the vibration processing in step S24, in step S25, the first control unit 22 determines whether the operator's index finger is maintaining the position of pulling the trigger of the gun. Specifically, it determines whether the operating position of the first attachment part 37 attached to the operator's index finger is maintaining the "firing position".

[0118] If a positive result is obtained in step S25, the process returns to step S24 and the vibration process continues.

[0119] On the other hand, if a negative determination is made in step S25, in step S26, the first control unit 22 determines whether the operator's index finger has pulled the trigger of the pistol all the way (whether it has been pulled beyond a predetermined amount). Specifically, it determines whether the operating position of the first attachment unit 37 attached to the operator's index finger has reached the "firing completion position". To make this determination, the angle acquisition unit 343 of the second control unit 34 acquires information on the rotational position (rotation angle from the initial position) of the arm 327a corresponding to the movement of the index finger and transmits it to the head-mounted display device 2. The first control unit 22 of the head-mounted display device 2 determines whether the operating position of the first attachment unit 37 that it has received has reached the preset "firing completion position".

[0120] The first control unit 22 determines that the operating position of the first mounting unit 37 has reached the "firing completion position" when the rotation angle of the arm 327a, which rotates in response to the movement of the index finger, from its initial position becomes greater than or equal to a predetermined angle (e.g., 80°). Note that the predetermined angle range used as a reference in step S26 (e.g., 80°) is a larger value than the predetermined angle range (e.g., 70° to 80°) that was used as the criterion for determining the "firing position" in step S23.

[0121] If the result in step S26 is positive (S26: YES), then in step S27, the vibration processing started in step S24 is released. The vibration processing is released using the same method as described in step S7.

[0122] On the other hand, if a negative result is obtained in step S26 (S26: NO), the vibration processing started in step S24 is canceled in step S28. The cancellation of the vibration processing in this step is also performed using the same method as described in step S7.

[0123] The above processing operations, particularly steps S23 to S27, can realistically reproduce the sensation of firing a pistol. After releasing the vibration processing in step S27, it is desirable that the tactile sensation from vibrations is not presented until the trigger position (the operating position of the first attachment part 37 attached to the index finger) is returned to its original position (the position when the index finger is extended). Alternatively, after releasing the vibration processing in step S27, the power output unit 31 and the power adjustment unit 32 may be controlled by the power of the power output unit 31 to return the operating position of the first attachment part 37 attached to the index finger to the position when the index finger is extended. In this case, different control parameters should be prepared for forward and reverse rotation, and these should be used to control the power output unit 31 and / or the power adjustment unit 32. Furthermore, after returning the trigger position (the operating position of the first attachment part 37 attached to the index finger) to its original position (the position when the index finger is extended), it is desirable that the power output unit 31 and the power adjustment unit 32 be controlled so that the tactile sensation of when the trigger was first pulled is restored.

[0124] <Variation> Next, a partial modification of the virtual object haptic presentation system 1 will be described.

[0125] In the embodiments described above, the operation position detection unit 33 detected the operation position of the first mounting unit 37, which is the operation unit, using the angle sensor 3292. However, it is also possible to provide the head-mounted display device 2 with a hand tracking function using a camera 26, and to detect the operation position of the first mounting unit 37, which is the operation unit, using this hand tracking function.

[0126] In the embodiments described above, the tactile information database 271 and the vibration processing tactile information database 271A were stored in the storage unit 27 of the head-mounted display device 2. However, these databases may also be stored in the storage unit (not shown) of the tactile presentation device 3. In this case, the second control unit 34 of the tactile presentation device 3 can read the necessary tactile information from the storage unit of the tactile presentation device 3 instead of receiving tactile information from the head-mounted display device.

[0127] In the embodiments described above, a tactile information database 271 and a tactile information database 271A for vibration processing were used, but it is also possible to omit the tactile information database 271A for vibration processing. In this case, instead of using the tactile information database 271A for vibration processing in steps S4, S13, and S24, the tactile information database 271 is used, and the values ​​stored in the tactile information database 271 are set so that the first mounting part 37 vibrates only when the arm 327a corresponding to the index finger is in the "firing position". For example, a non-zero value may be set for the "frequency" in the tactile information database 271 corresponding to the rotation angle of the arm 327a from the initial position corresponding to the "firing position" (the "displacement amount" in the tactile information database 271), and a zero value may be set for the "frequency" in the tactile information database 271 corresponding to positions other than the "firing position".

[0128] In the embodiments described above, the same tactile information database 271, or the same tactile information database 271 and vibration processing tactile information database 271A, was used regardless of the direction of finger movement (forward rotation or reverse rotation). However, different tactile information databases 271, or different tactile information databases 271 and vibration processing tactile information database 271A, may be used depending on the direction of finger movement. By doing so, different tactile sensations can be presented when gripping with the fingers and when releasing (opening) with the fingers. In this case, the second control unit 34 should determine forward rotation / reverse rotation based on the information acquired by the angle acquisition unit 343, and use the tactile information database 271 etc. for forward rotation in the case of forward rotation, and use the tactile information database 271 etc. for reverse rotation in the case of reverse rotation.

[0129] In the embodiments described above, the control means 5 changed the tactile sensation presented to the operator by controlling the power output unit 31 and / or power adjustment unit 32 based on information such as the tactile information database 271 and the vibration processing tactile information database 271A, and the rotation angle information of the arm 327a acquired by the angle acquisition unit 343 of the tactile presentation device 3. However, the control means 5 may also change the tactile sensation presented to the operator by controlling the power output unit 31 and / or power adjustment unit 32 according to the deformation state of the virtual object. For example, vibration processing can be performed when the virtual object is in a predetermined shape, and the vibration processing can be terminated when its shape changes.

[0130] <Note 1> The embodiments described above include the following technical concepts.

[0131] [First technical thought] An input system comprising an input device (e.g., a tactile presentation device 3) to be attached to a body part (e.g., hand, foot), a head-mounted display device 2 having a display unit 23, and control means 5 (which may be included in the input device, the head-mounted display device 2, or other external devices) for controlling the input device and the head-mounted display device 2, The control means 5 is The steps include obtaining the position and orientation of the input device, When the head-mounted display device 2 is in AR mode, the display unit 23 is made to display a virtual object 233 that indicates the initial position and orientation of the input device in the virtual space. The steps include detecting input from the input device (initial position determination operation), When the predetermined input (initial position determination operation) is detected, the following steps are performed: set the position and orientation of the input device at that time as the initial position and orientation in the virtual space. Input system.

[0132] [Second technical thought] In the input system described in [First Technical Concept] above, the control means 5 is an input system that displays the virtual objects corresponding to multiple body parts (right hand, left hand, right foot, left foot) on the display unit 23.

[0133] <Note 2> The embodiments described above also include the following technical concepts.

[0134] [First technical thought] Control panel and, A power output unit that outputs power to move the aforementioned operating unit, A power adjustment unit that adjusts the power output of the aforementioned power output unit, An operating position detection unit for detecting the operating position of the aforementioned operating unit, Control means for controlling the power output unit and the power adjustment unit based on information detected by the operation position detection unit, Equipped with, The aforementioned power adjustment unit is A first part is rotatably held in a predetermined position relative to the operator's hand, A second part that is rotatable relative to the first part, A power transmission adjustment means for adjusting the power transmitted between the first part and the second part, Includes, The power output from the power output unit is input to the second unit, the power adjusted by the power transmission adjustment means is transmitted from the second unit to the first unit, and the adjusted power is output from the first unit to the operation unit side. The control means vibrates the operating unit by simultaneously operating the power output unit and the power transmission adjustment means of the power adjustment unit when the operating position of the operating unit is within a predetermined range, and controls the power output unit and / or the power adjustment unit to prevent the operating unit from vibrating when the operating position of the operating unit is outside the predetermined range. A tactile presentation device characterized by the following features.

[0135] [Second technical thought] A tactile presentation device as described in [Technical Concept 1], The aforementioned power transmission adjustment means is A magnetorheological fluid interposed between the first part and the second part, A magnetic field generating unit that generates a magnetic field to be applied to the magnetoviscous fluid, It has, The power transmitted between the first and second parts is adjusted by adjusting the strength of the magnetic field that the magnetic field generating unit applies to the magnetoviscous fluid. A tactile presentation device characterized by the following features.

[0136] [Third technical thought] A tactile presentation device described in [First Technical Concept] or [Second Technical Concept], The aforementioned operating unit includes an attachment part that is attached to the operator's fingers, A tactile presentation device characterized by the following features.

[0137] [Fourth technical thought] A tactile presentation device described in any one of the following: [Technical Concept 1] to [Technical Concept 3], The control means vibrates the operating unit by periodically changing the control parameters of the power output unit and / or the control parameters of the power adjustment unit when the operating position of the operating unit is within a predetermined range. A tactile presentation device characterized by the following features.

[0138] [Fifth technical thought] A tactile presentation device described in any one of the following: [Technical Concept 1] to [Technical Concept 4], The control means vibrates the operating part by periodically changing the power transmitted by the power transmission adjustment means of the power adjustment unit when the operating position of the operating part is within a predetermined range. A tactile presentation device characterized by the following features.

[0139] [6th technical thought] A tactile presentation device described in any one of [Technical Concept 1] to [Technical Concept 5], The control means vibrates the operating unit by periodically changing the direction of the force output by the power output unit when the operating position of the operating unit is within a predetermined range. A tactile presentation device characterized by the following features.

[0140] [7th Technical Thought] A tactile presentation device described in any one of [Technical Concept 1] to [Technical Concept 6], The control means, when the operating position of the operating unit remains within a predetermined range for a predetermined time or longer, controls the power output unit and / or the power adjustment unit to change the amplitude or type of vibration applied to the operating unit. A tactile presentation device characterized by the following features.

[0141] [8th Technical Thought] A tactile presentation device that provides tactile feedback to the operator operating the control unit, A display unit that displays video including virtual objects, An operating position detection unit for detecting the operating position of the aforementioned operating unit, Control means for controlling the operation of the tactile presentation device and the image information output to the display unit, A virtual object haptic presentation system comprising: The tactile presentation device is The aforementioned operating unit, A power output unit that outputs power to move the aforementioned operating unit, A power adjustment unit that adjusts the power output of the power output unit and supplies it to the operation unit, It has, The aforementioned power adjustment unit is A first part is rotatably held in a predetermined position relative to the operator's hand, A second part that is rotatable relative to the first part, A power transmission adjustment means for adjusting the power transmitted between the first part and the second part, It has, The power output from the power output unit is input to the second unit, the power adjusted by the power transmission adjustment means is transmitted from the second unit to the first unit, and the adjusted power is output from the first unit to the operation unit side. The control means is The power output unit and the power adjustment unit are controlled based on the operating position of the operating unit detected by the operating position detection unit. When the operating position of the operating unit is within a predetermined range, the power output unit and the power transmission adjustment means of the power adjustment unit are activated simultaneously to vibrate the operating unit; when the operating position of the operating unit is outside the predetermined range, the power output unit and / or the power adjustment unit are controlled to prevent the operating unit from vibrating. A virtual object haptic presentation system characterized by the following:

[0142] [9th Technical Thought] A virtual object haptic presentation system as described in [Technical Concept #8], The aforementioned power transmission adjustment means is A magnetorheological fluid interposed between the first part and the second part, A magnetic field generating unit that generates a magnetic field to be applied to the magnetoviscous fluid, It has, The power transmitted between the first and second parts is adjusted by adjusting the strength of the magnetic field that the magnetic field generating unit applies to the magnetoviscous fluid. A virtual object haptic presentation system characterized by the following:

[0143] [10th Technical Thought] A virtual object haptic presentation system described in [Technical Concept 8] or [Technical Concept 9], The aforementioned virtual objects are virtual objects that mimic laser guns, machine guns, and other firearms. A virtual object haptic presentation system characterized by the following:

[0144] [11th Technical Thought] A virtual object haptic presentation system described in any one of [Technical Concept 8] to [Technical Concept 10], The aforementioned virtual object is a virtual object modeled after a handgun, The control means is When the operating position of the operating unit is within a predetermined range, the operating unit is vibrated by simultaneously operating the power output unit and the power transmission adjustment means of the power adjustment unit. As the operating position moves within the predetermined range in the direction of finger bending, the operating unit is vibrated more strongly. When the operating position moves beyond the predetermined range in the direction of finger bending, the power output unit and / or the power adjustment unit are controlled to prevent the operating unit from vibrating. A virtual object haptic presentation system characterized by the following:

[0145] [12th Technical Thought] A virtual object haptic presentation system described in any one of [Technical Concept 8] to [Technical Concept 11], The aforementioned virtual object is a virtual object that resembles an electric ball, The control means is When the operating position of the operating unit is within a predetermined range, the power output unit and the power transmission adjustment means of the power adjustment unit are operated simultaneously, and the operating unit is vibrated by periodically reversing the output direction of the power output unit. When the operating position of the operating unit is outside the predetermined range, the power output unit and / or the power adjustment unit are controlled to prevent the operating unit from vibrating. A virtual object haptic presentation system characterized by the following:

[0146] [13th Technical Thought] A virtual object haptic presentation system described in any one of [Technical Concept 8] to [Technical Concept 12], The control means vibrates the operating unit by periodically changing the control parameters of the power output unit and / or the control parameters of the power adjustment unit when the operating position of the operating unit is within a predetermined range. A virtual object haptic presentation system characterized by the following: [Industrial applicability]

[0147] The present invention can be applied, for example, to a virtual object haptic presentation system that presents the operator with the tactile sensation of a virtual object displayed on a screen. [Explanation of Symbols]

[0148] 1. Virtual Object Haptic Presentation System 2. Head-mounted display device 22 First Control Unit 23 Display section 231 VR space 232 A virtual object resembling a hand (a virtual hand object) 234 The Hand of Reality 235 AR space 237 Virtual object of a gun 237a Virtual object of a laser gun 237b Virtual object of a machine gun 238 Virtual object of an electric ball 27 Memory section 271 Tactile Information Database 271A Tactile information for vibration processing 28. First Communications Department 3. Tactile presentation device 31 Power output section 32 Power adjustment section 32A Part 1 32B Part 2 325 coil 326 Magnetoviscous fluid 327a Arm 3292 Angle Sensor 33 Operation position detection unit 34 Second Control Unit 341 Power Output Control Unit 342 MRFD Control Unit 343 Angle acquisition part 35 Transmission Mechanism 37. First mounting section 38 Second mounting section 40 Second Communications Department 5. Control means

Claims

1. Control panel and A power output unit that outputs power to move the aforementioned operating unit, A power adjustment unit that adjusts the power output of the aforementioned power output unit, An operating position detection unit for detecting the operating position of the aforementioned operating unit, Control means for controlling the power output unit and the power adjustment unit based on information detected by the operation position detection unit, Equipped with, The aforementioned power adjustment unit is A first part is rotatably held in a predetermined position relative to the operator's hand, A second part that is rotatable relative to the first part, A power transmission adjustment means for adjusting the power transmitted between the first part and the second part, Includes, The power output from the power output unit is input to the second unit, the power adjusted by the power transmission adjustment means is transmitted from the second unit to the first unit, and the adjusted power is output from the first unit to the operation unit side. The control means vibrates the operating unit by simultaneously operating the power output unit and the power transmission adjustment means of the power adjustment unit when the operating position of the operating unit is within a predetermined range, and controls the power output unit and / or the power adjustment unit to prevent the operating unit from vibrating when the operating position of the operating unit is outside the predetermined range. A tactile presentation device characterized by the following features.

2. A tactile presentation device according to claim 1, The control means vibrates the operating unit by periodically changing the control parameters of the power output unit and / or the control parameters of the power adjustment unit when the operating position of the operating unit is within a predetermined range. A tactile presentation device characterized by the following features.

3. A tactile presentation device according to claim 1 or 2, The aforementioned power transmission adjustment means is A magnetorheological fluid interposed between the first part and the second part, A magnetic field generating unit that generates a magnetic field to be applied to the magnetoviscous fluid, It has, The power transmitted between the first and second parts is adjusted by adjusting the strength of the magnetic field that the magnetic field generating unit applies to the magnetoviscous fluid. A tactile presentation device characterized by the following features.

4. A tactile presentation device according to claim 1 or 2, The aforementioned operating unit includes an attachment part that is attached to the operator's fingers, A tactile presentation device characterized by the following features.

5. A tactile presentation device according to claim 1 or 2, The control means, when the operating position of the operating unit remains within a predetermined range for a predetermined time or longer, controls the power output unit and / or the power adjustment unit to change the amplitude or type of vibration applied to the operating unit. A tactile presentation device characterized by the following features.

6. A tactile presentation device that provides tactile feedback to the operator operating the control unit, A display unit that displays video including virtual objects, An operating position detection unit for detecting the operating position of the aforementioned operating unit, Control means for controlling the operation of the tactile presentation device and the image information output to the display unit, A virtual object haptic presentation system comprising: The tactile presentation device is The aforementioned operating unit, A power output unit that outputs power to move the aforementioned operating unit, A power adjustment unit that adjusts the power output of the power output unit and supplies it to the operation unit, It has, The aforementioned power adjustment unit is A first part is rotatably held in a predetermined position relative to the operator's hand, A second part that is rotatable relative to the first part, A power transmission adjustment means for adjusting the power transmitted between the first part and the second part, It has, The power output from the power output unit is input to the second unit, the power adjusted by the power transmission adjustment means is transmitted from the second unit to the first unit, and the adjusted power is output from the first unit to the operation unit side. The control means is The power output unit and the power adjustment unit are controlled based on the operating position of the operating unit detected by the operating position detection unit. When the operating position of the operating unit is within a predetermined range, the power output unit and the power transmission adjustment means of the power adjustment unit are activated simultaneously to vibrate the operating unit; when the operating position of the operating unit is outside the predetermined range, the power output unit and / or the power adjustment unit are controlled to prevent the operating unit from vibrating. A virtual object haptic presentation system characterized by the following:

7. A virtual object haptic presentation system according to claim 6, The control means vibrates the operating unit by periodically changing the control parameters of the power output unit and / or the control parameters of the power adjustment unit when the operating position of the operating unit is within a predetermined range. A virtual object haptic presentation system characterized by the following:

8. A virtual object haptic presentation system according to claim 6 or 7, The aforementioned power transmission adjustment means is A magnetorheological fluid interposed between the first part and the second part, A magnetic field generating unit that generates a magnetic field to be applied to the magnetoviscous fluid, It has, The power transmitted between the first and second parts is adjusted by adjusting the strength of the magnetic field that the magnetic field generating unit applies to the magnetoviscous fluid. A virtual object haptic presentation system characterized by the following:

9. A virtual object haptic presentation system according to claim 6 or 7, The aforementioned virtual objects are virtual objects that mimic laser guns, machine guns, and other firearms. A virtual object haptic presentation system characterized by the following:

10. A virtual object haptic presentation system according to claim 6 or 7, The aforementioned virtual object is a virtual object modeled after a handgun, The control means is When the operating position of the operating unit is within a predetermined range, the operating unit is vibrated by simultaneously operating the power output unit and the power transmission adjustment means of the power adjustment unit. As the operating position moves within the predetermined range in the direction of finger bending, the operating unit is vibrated more strongly. When the operating position moves beyond the predetermined range in the direction of finger bending, the power output unit and / or the power adjustment unit are controlled to prevent the operating unit from vibrating. A virtual object haptic presentation system characterized by the following:

11. A virtual object haptic presentation system according to claim 6 or 7, The aforementioned virtual object is a virtual object that resembles an electric ball, The control means is When the operating position of the operating unit is within a predetermined range, the power output unit and the power transmission adjustment means of the power adjustment unit are operated simultaneously, and the operating unit is vibrated by periodically reversing the output direction of the power output unit. When the operating position of the operating unit is outside the predetermined range, the power output unit and / or the power adjustment unit are controlled to prevent the operating unit from vibrating. A virtual object haptic presentation system characterized by the following:

Images