Information processing device, information processing method, and drive data generating method
The information processing system with a hand-held operating device provides synchronized haptic feedback, addressing the lack of effective haptic feedback in VR systems to enhance user interaction and immersion in virtual environments.
Patent Information
- Application Number
- JP2023534811
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-07-14
- Filing Date
- 2022-07-12
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-07-12
AI Technical Summary
Existing virtual reality (VR) systems, such as head-mounted displays (HMDs) connected to game devices, lack effective haptic feedback mechanisms to enhance user interaction and immersion during gameplay.
An information processing system that includes a hand-held operating device with a force sense presentation unit and a control unit that generates drive data to provide haptic feedback based on the user's hand movements, synchronizing with virtual space interactions.
Enhances user immersion and entertainment value by providing realistic haptic feedback, allowing users to experience the effects of their actions in virtual environments through tactile and force sensations.
Smart Images

Figure 0007577218000001 
Figure 0007577218000002 
Figure 0007577218000003
Abstract
Description
[Technical field]
[0001] The present invention relates to techniques for providing force and / or haptic feedback to a user. [Background technology]
[0002] Patent Literature 1 discloses a portable computer interface that includes a housing, a mass coupled to the housing, and an actuator coupled to the mass for changing the position of the mass relative to the housing. The portable computer interface disclosed in Patent Literature 1 is used as a control device for a game, and provides feedback to a user by moving the mass. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2010-225155 A Summary of the Invention [Problem to be solved by the invention]
[0004] A head mounted display (HMD) is worn on the user's head and provides the user with a virtual reality (VR) world. Recently, it has become common for the user to connect the HMD to a game device and play the game by operating the operation device while watching the game image displayed on the HMD.
[0005] During gameplay, the user can experience the results of their game operations in a realistic way by receiving feedback from the game through the operation device. Therefore, it is believed that the entertainment value of applications such as games can be improved by effectively providing users with force feedback and / or haptic feedback.
[0006] The present invention has been made in view of the above circumstances, and has an object to realize a technique for providing a user with force feedback and / or haptic feedback. [Means for solving the problem]
[0007] In order to solve the above problems, an information processing device according to one embodiment of the present invention is an information processing device including an operation information acquisition unit that acquires first operation information indicating a movement of an operation device that is held in a user's hand and has a force sense presentation unit that presents a force sense to the user's hand, and a control unit that controls the movement of a virtual device corresponding to the operation device in a virtual space in accordance with the first operation information, wherein the control unit generates drive data for driving the force sense presentation unit based on the movement of the virtual device, the operation information acquisition unit acquires second operation information indicating that the force sense presentation unit has been moved by the user's hand, and the control unit executes a function corresponding to the second operation information in the virtual space.
[0008] Another aspect of an information processing method of the present invention includes acquiring first operation information indicating a movement of an operating device that is held in a user's hand and has a force sense presentation unit that presents a force sense to the user's hand, controlling the movement of a virtual device corresponding to the operating device in a virtual space in accordance with the first operation information, generating drive data for driving the force sense presentation unit based on the movement of the virtual device, acquiring second operation information indicating that the force sense presentation unit has been moved by the user's hand, and executing a function corresponding to the second operation information in the virtual space.
[0009] Yet another embodiment of an information processing device of the present invention is an information processing device including an operation information acquisition unit that acquires operation information indicating the movement of an operation device that is held in a user's hand and has a force sense presentation unit that presents a force sense to the user's first finger and a tactile presentation unit that presents a tactile sense to the user's second finger, and a control unit that controls the movement of a virtual device corresponding to the operation device in a space in which a virtual object exists in accordance with the operation information, and the control unit generates first drive data for driving the force sense presentation unit and second drive data for driving the tactile presentation unit based on the relationship between the virtual device and the virtual object.
[0010] Another aspect of the drive data generation method of the present invention acquires operation information indicating the movement of an operating device that is held in a user's hand and has a force sense presentation unit that presents a force sense to the user's first finger and a tactile presentation unit that presents a tactile sense to the user's second finger, controls the movement of a virtual device corresponding to the operating device in a space in which a virtual object exists in accordance with the operation information, and generates first drive data for driving the force sense presentation unit and second drive data for driving the tactile presentation unit based on the relationship between the virtual device and the virtual object.
[0011] Any combination of the above components and any conversion of the present invention into a method, device, system, etc. are also valid aspects of the present invention. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 illustrates an example of a configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram showing an example of the external shape of an HMD. [Diagram 3] FIG. 2 is a diagram showing functional blocks of an HMD. [Figure 4] FIG. 2 is a diagram showing a state in which the operating device is held by a user. [Diagram 5] FIG. [Figure 6] FIG. 13 is a diagram showing a state in which a finger is placed on the movable part. [Figure 7] 13A and 13B are diagrams showing a movement mechanism for moving a movable part relative to a base body. [Figure 8] FIG. 2 is a diagram showing functional blocks of the operation device. [Figure 9] FIG. 2 is a diagram illustrating functional blocks of the information processing device. [Figure 10] FIG. 4 is a diagram showing an example of a screen displayed on a display panel. [Figure 11] FIG. 13 illustrates the virtual device entering ink. [Figure 12] 13 is a diagram showing an example of a map that defines the relationship between an entry length and a movement amount of a movable part. FIG. [Figure 13] FIG. 4 is a diagram showing an example of a vibration waveform. [Figure 14] 13 is a diagram showing an example of a map that defines the relationship between the moving speed and the vibration amount of a vibration section. FIG. [Figure 15] FIG. 4 is a diagram showing an example of a screen displayed on a display panel. [Figure 16] FIG. 4 is a diagram showing an example of a screen displayed on a display panel. [Figure 17] FIG. 11 is a diagram showing an example of a map that defines the relationship between the elapsed time from a collision and the movement amount of a movable part. [Figure 18] FIG. 4 is a diagram showing an example of a vibration waveform. [Figure 19] FIG. 2 is a diagram showing a state in which the operating device is held by a user. [Figure 20] FIG. [Figure 21] 13 is a diagram showing a movement mechanism that causes a movable part to rotate relatively to a base body. FIG. [Figure 22] 13 is a diagram showing the inclination of a support base frame when the operating device is viewed from the front. FIG. [Figure 23] FIG. 2 is a diagram showing functional blocks of the operation device. [Figure 24] FIG. 2 is a diagram illustrating functional blocks of the information processing device. [Diagram 25] FIG. 13 is a diagram showing an example of an application screen. [Figure 26] FIG. 13 is a diagram showing an example of an application screen. [Figure 27] FIG. 13 is a diagram showing an example of an application screen. [Figure 28] FIG. 13 is a diagram showing an example of an application screen. [Figure 29] FIG. 13 is a diagram showing an example of an application screen. [Diagram 30] FIG. 13 is a diagram showing an example of an application screen. [Diagram 31] FIG. 13 is a diagram showing an example of an application screen. [Diagram 32] FIG. 13 is a diagram showing an example of an application screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] 1 shows a configuration example of an information processing system 1 in an embodiment. The information processing system 1 includes an information processing device 10, a recording device 11, a head mounted display (HMD) 100 worn on the user's head, an operation device 20 held in the user's hand, an imaging device 14 that captures images of the HMD 100 and the operation device 20, and an output device 15 that outputs images and sounds. The output device 15 may be a television. The information processing device 10 is connected to an external network 2 such as the Internet via an access point (AP) 17. The AP 17 has the functions of a wireless access point and a router, and the information processing device 10 may be connected to the AP 17 by a cable or by a known wireless communication protocol.
[0014] The recording device 11 records applications such as system software and game software. The information processing device 10 may download the application from a content server to the recording device 11 via the network 2. The information processing device 10 executes the application and supplies image data and sound data of the application to the HMD 100 and the output device 15. The information processing device 10 and the HMD 100 may be connected by a known wireless communication protocol or by a cable.
[0015] The HMD 100 is a display device that is worn on the user's head and displays images on a display panel located in front of the user's eyes. The HMD 100 separately displays an image for the left eye on the left-eye display panel and an image for the right eye on the right-eye display panel. These images form parallax images seen from left and right viewpoints, achieving stereoscopic vision. Since the user views the display panel through optical lenses, the information processing device 10 supplies the HMD 100 with parallax image data in which optical distortion caused by the lenses has been corrected.
[0016] The HMD 100 provides a virtual reality (VR) image world to the user. The information processing system 1 is provided with a head tracking function, and the display image is updated in conjunction with the movement of the user's head, enhancing the sense of immersion in the image world.
[0017] The output device 15 is not necessary for a user wearing the HMD 100, but by providing the output device 15, another user can view the image displayed on the output device 15. The information processing device 10 may display on the output device 15 the same image as the image viewed by the user wearing the HMD 100, or may display a different image. For example, when a user wearing an HMD and another user play a game together, the output device 15 may display a game image from the viewpoint of the character of the other user.
[0018] The operation device 20 is a controller that is driven by a battery and has one or more operation buttons for inputting operations for an application. When a user operates an operation button of the operation device 20, the operation input is transmitted to the information processing device 10 by wireless communication. A wireless connection using, for example, Bluetooth (registered trademark) protocol may be established between the operation device 20 and the information processing device 10. The information processing device 10 receives the operation input from the operation device 20, controls the progress of the application in response to the operation input, generates image data and sound data of the application, and supplies them to the HMD 100 and the output device 15. The operation device 20 is not limited to a wireless controller, and may be a wired controller connected to the information processing device 10 via a cable.
[0019] The operation device 20 includes an attitude sensor including a three-axis acceleration sensor and a three-axis gyro sensor, and transmits sensor data to the information processing device 10 at a predetermined cycle. The drawing application of the embodiment moves a virtual device 22 corresponding to the operation device 20 in a virtual three-dimensional space in accordance with the movement of the operation device 20 held by the user's hand. For this reason, the information processing device 10 derives the position, attitude, and / or movement of the operation device 20 based on the sensor data transmitted from the operation device 20, and uses the derived position, attitude, and / or movement of the operation device 20 as operation information for moving the virtual device 22.
[0020] The imaging device 14 is a video camera composed of a CCD imaging element or a CMOS imaging element, and captures images of the real space at a predetermined cycle to generate frame images for each cycle. It is preferable that the imaging device 14 is a stereo camera, and the information processing device 10 can measure the distance to the object from the captured images. The imaging speed of the imaging device 14 may be 60 frames / second, which is the same as the frame rate of the HMD 100. The imaging device 14 is connected to the information processing device 10 via a USB (Universal Serial Bus) or other interface.
[0021] The HMD 100 is provided with a marker (tracking LED) for tracking the user's head, and the information processing device 10 detects the movement of the HMD 100 based on the position of the marker included in the captured image. The HMD 100 is equipped with an attitude sensor (a three-axis acceleration sensor and a three-axis gyro sensor), and the information processing device 10 may perform a highly accurate tracking process by acquiring sensor data detected by the attitude sensor from the HMD 100 in conjunction with the use of the captured image of the marker, to detect the position and attitude of the user's head (actually the HMD 100) in real space.
[0022] Here, the position of the HMD 100 refers to position coordinates in a three-dimensional space with a reference position as the origin, and the reference position may be the position coordinates (latitude, longitude) when the power of the HMD 100 is turned on. The attitude of the HMD 100 refers to the inclination in three-axis directions with respect to a reference attitude in a three-dimensional space, and the reference attitude is an attitude in which the line of sight of the user is horizontal, and the reference attitude may be set when the power of the HMD 100 is turned on. Note that various methods have been proposed for tracking processing, and the information processing device 10 may adopt any tracking method as long as it can detect the movement of the HMD 100.
[0023] 2 shows an example of the external shape of the HMD 100. The HMD 100 is composed of an output mechanism 102 and a wearing mechanism 104. The wearing mechanism 104 includes a wearing band 106 that goes around the head when worn by the user to secure the HMD 100 to the head. The wearing band 106 is made of a material or has a structure that allows the length to be adjusted to fit the user's head circumference.
[0024] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the HMD 100, and includes a display panel inside that faces the eyes when worn. The display panel may be a liquid crystal panel, an organic EL panel, or the like. The housing 108 also includes a pair of left and right optical lenses located between the display panel and the user's eyes to expand the user's viewing angle. The HMD 100 may further include speakers or earphones at positions corresponding to the user's ears, and may be configured to allow external headphones to be connected.
[0025] A plurality of light-emitting markers 110a, 110b, 110c, and 110d are provided on the outer surface of the housing 108. In this example, the light-emitting markers 110 are constituted by tracking LEDs, but other types of markers may be used as long as they are photographed by the imaging device 14 and the information processing device 10 can perform image analysis of the marker positions. The number and arrangement of the light-emitting markers 110 are not particularly limited, but they need to be numbered and arranged so as to be capable of detecting the attitude of the HMD 100, and in the example shown, the light-emitting markers 110 are provided at the four corners of the front of the housing 108. Furthermore, the light-emitting markers 110 may be provided on the side or rear of the wearing band 106 so that they can be photographed even when the user has their back turned to the imaging device 14.
[0026] 3 shows functional blocks of the HMD 100. The control unit 120 is a main processor that processes and outputs various data such as image data, sound data, and sensor data, as well as commands. The storage unit 122 temporarily stores the data and commands processed by the control unit 120. The orientation sensor 124 detects orientation information of the HMD 100. The orientation sensor 124 includes at least a three-axis acceleration sensor and a three-axis gyro sensor.
[0027] The communication control unit 128 transmits data output from the control unit 120 to the external information processing device 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 128 also receives data from the information processing device 10 by wired or wireless communication via a network adapter or an antenna, and outputs the data to the control unit 120.
[0028] When the control unit 120 receives image data or sound data from the information processing device 10, it supplies the data to the display panel 130 for display, and also supplies the data to the sound output unit 132 for output. The display panel 130 is composed of a left-eye display panel 130a and a right-eye display panel 130b, and a pair of parallax images are displayed on each display panel. The control unit 120 also causes the communication control unit 128 to transmit sensor data from the attitude sensor 124 and sound data from the microphone 126 to the information processing device 10.
[0029] Example 1 Fig. 4 shows a state in which the operating device 20 of the first embodiment is held by a user. Fig. 4(a) shows a state in which the operating device 20 is viewed from the front, and Fig. 4(b) shows a state in which the operating device 20 is viewed from the side. The operating device 20 includes a base 30 that is held by the user's hand, and a movable part 32 that is a part that is movable relative to the base 30. The movable part 32 functions as a force sense presentation part that presents a force sense to the user's hand, and is driven based on drive data transmitted from the information processing device 10 to provide force sense feedback to the user.
[0030] The operation device 20 is a pen-shaped haptic device, and the base body 30 has two side surfaces arranged approximately parallel to each other. A user uses the operation device 20 by holding both sides of the base body 30 between the thumb and middle finger and placing the index finger on the movable part 32.
[0031] Fig. 5 is a side view of the operating device 20. In the operating device 20, the side on which the movable part 32 is provided is defined as the tip side, and the left side in the side view shown in Fig. 5 is called the front side, and the right side is called the rear side. In the state shown in Fig. 5, the movable part 32 is at a position that protrudes most forward from the base body 30, and this position is the "reference position" of the movable part 32.
[0032] A finger rest 36 is provided on the upper surface of the movable part 32 for the user to place the tip of the index finger while holding the base body 30. A finger engagement part 34 with which the tip of the finger engages is provided on the front end of the finger rest 36. The finger engagement part 34 is erected in a direction different from the moving direction (i.e., the front-rear direction) of the movable part 32, and in the first embodiment, the finger engagement part 34 is erected in a direction perpendicular to the upper surface of the movable part 32. The finger engagement part 34 has a curved surface 34a that is inclined to follow the pad of the finger and has a curvature. The curved surface 34a is inclined with respect to the upper surface of the movable part 32 and is formed concave in the direction in which the fingertip contacts, so that the user can stably contact the tip of the index finger with the curved surface 34a.
[0033] The user holds both sides of the base body 30 between the thumb and middle finger as if holding a pen, and places the pad of the index finger on the finger placement section 36. This allows the user to place the index finger on the finger placement section 36 regardless of the user's dominant hand. A vibration section 40 including a vibrator is provided on the tip side of the base body 30, specifically, on the part where the thumb or middle finger is placed. The vibration section 40 may be provided on both sides of the tip of the base body 30. The vibration section 40 functions as a tactile presentation section that presents a tactile sensation to the user's hand by vibrating, and is driven based on drive data transmitted from the information processing device 10 to provide tactile feedback to the user.
[0034] An operation button 42 that can be operated with the tip of an index finger is provided on the finger rest 36. The operation button 42 is a push button, and a user presses the operation button 42 to turn on a switch (not shown) equipped with a contact structure. The operation button 42 is provided on the finger rest 36 on the rear side of the curved surface 34a. By providing the operation button 42 on the finger rest 36, the user can freely operate the operation button 42 with the index finger while holding the base 30 between the thumb and middle finger.
[0035] 6(a) shows a state in which a finger is placed on the movable part 32 in the reference position. When the user places the pad of the finger on the finger placement section 36, the tip of the finger fits and comes into contact with the curved surface 34a. By the tip of the finger abutting the finger engagement section 34 in this manner, the finger reliably moves together with the finger engagement section 34 when the movable part 32 moves in a direction in which it is drawn into the base body 30.
[0036] 6(b) shows a state in which the movable part 32 moves in a direction in which it is drawn into the base body 30. When the movable part 32 moves, the fingertip moves in conjunction with the movable part 32 due to the frictional force acting between the fingertip and the finger placement part 36. At this time, the fingertip is in contact with the finger engagement part 34 that is erected perpendicular to the direction of movement, so that the fingertip is reliably linked to the movement of the movable part 32.
[0037] Fig. 7 shows a moving mechanism 300 that moves the movable part 32 relative to the base body 30. Fig. 7(a) shows a state of the moving mechanism 300 when the movable part 32 is in a reference position, and Fig. 7(b) shows a state of the moving mechanism 300 when the movable part 32 is retracted from the reference position.
[0038] The moving mechanism 300 includes a first rotary actuator 310, a feed screw 312, a nut 314, a guide mechanism 316, a rod 318, a fixed part 320, and a first rotation angle sensor 322, and slides the movable part 32 in the longitudinal direction of the base body 30. In this moving mechanism 300, the first rotary actuator 310 rotates the feed screw 312 forward or backward, thereby moving the nut 314 in the axial direction of the feed screw 312 (the longitudinal direction of the base body 30). A plurality of rods 318 are fixed to the nut 314, and fixed parts 320 fixed to fixed parts of the movable part 32 are attached to the tips of the rods 318. The movement of the rods 318 in the advancing direction is guided by a guide mechanism 316 fixed to the inner wall of the base body 30. The first rotation angle sensor 322 detects the rotation angle of the first rotation actuator 310 , and the movement of the movable part 32 is controlled based on the detection value of the first rotation angle sensor 322 .
[0039] 8 shows functional blocks of the operation device 20. The control unit 50 is a main processor that processes and outputs various data such as sensor data and drive data, and commands. The storage unit 52 temporarily stores the data and commands processed by the control unit 50. The attitude sensor 54 detects attitude information of the operation device 20. The attitude sensor 54 includes at least a three-axis acceleration sensor and a three-axis gyro sensor.
[0040] The communication control unit 56 transmits data output from the control unit 50 to the external information processing device 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 56 also receives data from the information processing device 10 by wired or wireless communication via a network adapter or an antenna, and outputs the data to the control unit 50.
[0041] When the control unit 50 receives the drive data from the information processing device 10, it drives the movable unit 32 and / or the vibration unit 40. In the operation device 20, the movable unit 32 is a force sense presentation unit that presents a force sense to the user's hand, and the vibration unit 40 is a tactile sense presentation unit that presents a tactile sense to the user's hand. The movable unit 32 provides force sense feedback, and the vibration unit 40 provides tactile feedback, allowing the user to experience an event occurring in an application in a realistic manner. The control unit 50 causes the communication control unit 56 to transmit sensor data from the attitude sensor 54 to the information processing device 10.
[0042] The operation device 20 of the first embodiment is used as a virtual device 22 that emits ink in a drawing application for drawing pictures and characters in a virtual three-dimensional space. The virtual device 22 is displayed on the display panel 130 and the output device 15 as a virtual pen including a pen tip. In this drawing application, the tip of the operation device 20 corresponds to the pen tip of the virtual device 22, and when the user moves the operation device 20 in the real space, the virtual device 22 exhibits a behavior linked to the operation device 20 in the virtual space.
[0043] 9 shows functional blocks of the information processing device 10. The information processing device 10 includes a receiving unit 200 and a transmitting unit 260 as input / output interfaces with the outside. The information processing device 10 further includes an HMD information acquiring unit 210, an operation information acquiring unit 212, and a processing unit 220.
[0044] The information processing device 10 includes a computer, which executes a program to realize various functions shown in FIG. 9. The computer includes hardware such as a memory for loading the program, one or more processors for executing the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of multiple electronic circuits including a semiconductor integrated circuit and an LSI, and the multiple electronic circuits may be mounted on one chip or multiple chips. The functional blocks shown in FIG. 9 are realized by cooperation between hardware and software, and therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof.
[0045] The HMD sensor data receiving unit 202 receives sensor data of the attitude sensor 124 of the HMD 100 worn by the user at a predetermined cycle and supplies the data to the HMD information acquiring unit 210. For example, the cycle in which the HMD 100 transmits sensor data may be set to 11.25 ms. The captured image receiving unit 204 receives images of the HMD 100 from the imaging device 14 at a predetermined cycle and supplies the images to the HMD information acquiring unit 210. For example, the imaging device 14 may capture an image of the space ahead every (1 / 60) seconds, and the captured image receiving unit 204 may receive the captured image every (1 / 60) seconds. The operation data receiving unit 206 receives sensor data of the attitude sensor 54 of the operating device 20 held by the user at a predetermined cycle. For example, the cycle in which the operating device 20 transmits sensor data may be set to 11.25 ms. The operation data receiving unit 206 also receives operation data of the operation button 42 of the operating device 20 held by the user. The operation data receiving unit 206 supplies the sensor data of the attitude sensor 54 and the operation data of the operation buttons 42 to the operation information acquiring unit 212 .
[0046] The HMD information acquisition unit 210 acquires posture information indicating the posture of the HMD 100 in real space and position information indicating the position of the HMD 100 from the sensor data of the HMD 100 and the shooting state of the multiple light-emitting markers 110 included in the captured image. The HMD information acquisition unit 210 may calculate the inclination of the multiple light-emitting markers 110 for tracking included in the captured image, and acquire posture information of the HMD 100 using the calculated inclination and sensor data of a three-axis gyro sensor. The HMD information acquisition unit 210 may also calculate the amount of movement from a reference position using sensor data of a three-axis acceleration sensor, and acquire position information of the HMD 100. The HMD information acquisition unit 210 supplies the posture information and position information of the HMD 100 to the processing unit 220.
[0047] The operation information acquisition unit 212 acquires, from the sensor data of the operation device 20, attitude information indicating the attitude of the operation device 20 in real space and position information indicating the position of the operation device 20. The operation information acquisition unit 212 calculates the attitude information of the operation device 20 mainly using the sensor data of the three-axis gyro sensor, and calculates the position information of the operation device 20 using the sensor data of the three-axis acceleration sensor. The attitude information and position information of the operation device 20 constitute operation information indicating the movement of the operation device 20. The operation information acquisition unit 212 supplies the attitude information and position information of the operation device 20 to the processing unit 220 as operation information indicating the movement of the operation device 20. The operation information acquisition unit 212 also supplies operation data of the operation button 42 to the processing unit 220.
[0048] The processing unit 220 includes a control unit 230, a line of sight direction determination unit 250, a map storage unit 252, a vibration waveform storage unit 254, an image generation unit 256, and a sound generation unit 258. The control unit 230 has a virtual device control unit 232, a relationship identification unit 234, an action determination unit 236, and a drive data generation unit 238.
[0049] The line of sight direction determination unit 250 determines the line of sight of the user according to the attitude information of the HMD 100. The line of sight direction determination unit 250 converts the attitude information of the HMD 100 into the line of sight of the user, and supplies the converted line of sight direction to the image generation unit 256. The image generation unit 256 uses the line of sight provided by the line of sight direction determination unit 250 as information for determining the line of sight direction of the player character in the virtual space. The image generation unit 256 may determine the position and direction of a virtual camera based on the position information of the HMD 100 acquired by the HMD information acquisition unit 210 and the line of sight direction determined by the line of sight direction determination unit 250, and generate an application image.
[0050] The virtual device control unit 232 controls the movement of the virtual device 22 in the virtual space according to the operation information indicating the movement of the operation device 20. Specifically, the virtual device control unit 232 periodically acquires the attitude information and position information of the operation device 20 from the operation information acquisition unit 212, determines the attitude and position of the virtual device 22 in the virtual space according to the attitude information and position information of the operation device 20, and links the virtual device 22 in the virtual space to the movement of the operation device 20 in the real space. Note that the virtual device control unit 232 may determine the position or attitude of the virtual device 22 in the virtual space according to the position information or attitude information of the operation device 20. In other words, the virtual device control unit 232 may link the position of the operation device 20 in the real space with the position of the virtual device 22 in the virtual space, or may link the attitude of the operation device 20 in the real space with the attitude of the virtual device 22 in the virtual space.
[0051] In the drawing application of the first embodiment, in the virtual space, the player character dips the pen tip of the virtual device 22 into an inkwell to determine the color of the lines to be drawn. After that, the pen tip of the virtual device 22 is moved to the drawing start position, and the user moves the operation device 20 while pressing the operation button 42, whereby the trajectory of the tip of the operation device 20 is drawn in the virtual space. The line drawn in the virtual three-dimensional space becomes a shield that protects the player character, and can be used to protect the player character from virtual objects (bullets) flying toward the player character. In this series of progression, the operation device 20 provides the user with force feedback and / or haptic feedback, so that the user can experience the effect on the action in the virtual space. The effect of each scene in the application will be described below.
[0052] <Scene of dipping the pen tip into the inkwell> FIG. 10 shows an example of a screen displayed on the display panel 130 of the HMD 100. A plurality of ink bottles 70a, 70b, and 70c filled with ink are displayed on the display panel 130. The ink bottles 70a, 70b, and 70c are arranged at predetermined positions in the virtual space, and are filled with ink of different colors. Here, the ink bottle 70a is filled with red ink 72a, the ink bottle 70b is filled with blue ink 72b, and the ink bottle 70c is filled with yellow ink 72c. The user moves the virtual device 22 over the ink bottle 70 of the color he or she wants to use, and dips the pen tip into the ink. Note that the player character holding the virtual device 22 is not shown in the display screens shown in FIG. 10, FIG. 15, and FIG. 16.
[0053] Fig. 11 shows the virtual device 22 entering blue ink 72b in an inkpot 70b. Fig. 11(a) shows the pen tip entering slightly below the ink surface, and Fig. 11(b) shows the virtual device 22 entering about half the blue ink 72b. The virtual device control unit 232 controls the movement of the virtual device 22 in accordance with the attitude information and position information of the operation device 20.
[0054] The relationship identifying unit 234 identifies a relative relationship between the virtual device 22 and the blue ink 72b, which is a virtual object, and the effect determining unit 236 determines an effect on the operation device 20 based on the relative relationship identified by the relationship identifying unit 234. In the first embodiment, when the virtual device 22 and the virtual object come into contact with each other, the effect determining unit 236 may determine an effect caused by the contact between the two.
[0055] The relationship identifying unit 234 identifies the positional relationship between the virtual device 22 and the blue ink 72b from the three-dimensional coordinates of the ink bottle 70b arranged at a predetermined position and the three-dimensional coordinates of the virtual device 22. In the first embodiment, the relationship identifying unit 234 derives the length by which the virtual device 22 has entered the blue ink 72b (entry length L) from the identified positional relationship. When the virtual device 22 enters the blue ink 72b, the action determining unit 236 determines to move the movable unit 32, which is a force sense presentation unit, in a direction to pull it into the base 30. At this time, the action determining unit 236 derives the amount of movement of the movable unit 32, which is a force sense presentation unit, based on the entry length L.
[0056] FIG. 12(a) shows an example of a map that defines the relationship between the entry length L and the movement amount of the movable part 32. This relationship is stored as a map in the map storage unit 252, and the movement amount indicates the retraction amount of the movable part 32 from the reference position. Since the relationship shown in FIG. 12(a) is proportional to the entry length and the movement amount, the movable part 32 is retracted into the base body 30 in proportion to the length by which the virtual device 22 has entered the blue ink 72b. When the action determination unit 236 receives the entry length L from the relationship specification unit 234, it refers to the map stored in the map storage unit 252 to obtain the movement amount according to the entry length L and passes it to the drive data generation unit 238. The drive data generation unit 238 generates first drive data for driving the movable part 32 based on the movement amount.
[0057] The map storage unit 252 may store, for each virtual object, a map that defines the relationship between the penetration length L and the movement amount of the movable part 32. For example, the map storage unit 252 may store different maps for the red ink 72a, the blue ink 72b, and the yellow ink 72c.
[0058] 12(b) shows another example of a map that defines the relationship between the entry length L and the movement amount of the movable part 32. This relationship is held in the map holding unit 252. The relationship shown in FIG. 12(b) has a movement profile in which the movable part 32 moves significantly when the virtual device 22 starts to enter the ink, and the movable part 32 moves gradually after the virtual device 22 has entered to a certain extent. The map holding unit 252 holds a map that defines the relationship between the entry length and the movement amount for each virtual object (ink in the first embodiment), so that the user can feel the resistance specific to the virtual object with which the virtual device 22 is in contact, by the force sense presented by the movable part 32.
[0059] Furthermore, in the first embodiment, the relationship identification unit 234 identifies the relative speed between the virtual device 22 and the blue ink 72b, which is a virtual object, and the action determination unit 236 determines an action on the operating device 20 based on the relative speed identified by the relationship identification unit 234.
[0060] The relationship identifying unit 234 identifies the speed (moving speed) at which the virtual device 22 enters the blue ink 72b. Because the blue ink 72b, which is a virtual object, is stationary, the moving speed of the virtual device 22 is equal to the relative speed between the virtual device 22 and the blue ink 72b. The effect determining unit 236 derives a vibration profile of the vibrating unit 40, which is a tactile sense providing unit, based on the moving speed of the virtual device 22. Here, the vibration profile may be defined by a vibration waveform and a vibration amount.
[0061] 13(a) to (c) show examples of vibration waveforms. The horizontal axis indicates time. The vibration waveform storage unit 254 stores a vibration waveform, which is a pattern for vibrating the vibration unit 40, for each virtual object. Here, the vibration waveforms shown in FIGS. 13(a) and (b) are periodic, and the vibration waveform shown in FIG. 13(c) is non-periodic. In this manner, the vibration waveform may be freely set for each virtual object. The vibration waveform is a normalized vibration pattern, and the vibration amount (amplitude) determined according to the moving speed is multiplied by the vibration waveform to derive the vibration profile of the vibration unit 40.
[0062] FIG. 14(a) shows an example of a map that defines the relationship between the moving speed and the vibration amount of the vibration unit 40. This relationship is stored as a map in the map storage unit 252, and the vibration amount indicates the amplitude of the vibration waveform. In the relationship shown in FIG. 14(a), the moving speed and the vibration amount (amplitude) are in a proportional relationship, so that the vibration amplitude of the vibration unit 40 increases in proportion to the moving speed of the virtual device 22. When the action determination unit 236 receives the moving speed from the relationship specification unit 234, it refers to the map stored in the map storage unit 252 to obtain the vibration amount according to the moving speed, and derives a vibration profile by multiplying the vibration waveform corresponding to the virtual object stored in the vibration waveform storage unit 254 by the vibration amount. The action determination unit 236 passes the derived vibration profile to the drive data generation unit 238, and the drive data generation unit 238 generates second drive data for driving the vibration unit 40 based on the vibration profile.
[0063] The map storage unit 252 may store, for each virtual object, a map that defines the relationship between the moving speed and the vibration amount of the vibration unit 40. For example, the map storage unit 252 may store different maps for the red ink 72a, the blue ink 72b, and the yellow ink 72c.
[0064] 14(b) shows another example of a map that defines the relationship between the moving speed and the vibration amount of the vibration unit 40. The map storage unit 252 stores a map that defines the relationship between the moving speed and the vibration amount for each virtual object (ink in the first embodiment), and thereby the user can feel the characteristic of the virtual object that the virtual device 22 is in contact with through the haptic sensation presented by the vibration unit 40.
[0065] The transmission unit 260 transmits the first drive data and the second drive data generated by the drive data generation unit 238 to the operation device 20. In the operation device 20, the control unit 50 drives the movable part 32 based on the first drive data and drives the vibration unit 40 based on the second drive data. Specifically, the control unit 50 controls the first rotation actuator 310 while monitoring the detection value of the first rotation angle sensor 322, and moves the movable part 32 at the reference position in a direction to draw it into the base body 30. Therefore, the movable part 32 is drawn into the base body 30 by a movement amount according to the penetration length of the virtual device 22, and at the same time, the vibration unit 40 vibrates with a vibration profile according to the moving speed of the virtual device 22. According to the information processing system 1 of the embodiment, the entertainment value of the application can be improved by simultaneously providing force feedback and haptic feedback in response to the operation of the operation device 20.
[0066] 15 shows an example of a screen displayed on the display panel 130 of the HMD 100. After dipping the pen tip of the virtual device 22 in the blue ink 72b, the pen tip of the virtual device 22 is moved to a drawing start position, and the user moves the operation device 20 in the real space while pressing the operation button 42, whereby a trajectory of the pen tip of the virtual device 22 corresponding to the tip of the operation device 20 is drawn in the virtual space. In this scene, the relationship specifying unit 234 specifies the moving speed of the virtual device 22, and the action determining unit 236 determines an action on the operation device 20 based on the moving speed specified by the relationship specifying unit 234.
[0067] The relationship identifying unit 234 identifies the moving speed of the virtual device 22 from the moving speed of the operation device 20 with the operation button 42 pressed. The action determining unit 236 derives a vibration profile of the vibration unit 40, which is a tactile sense providing unit, based on the moving speed of the virtual device 22. As described above, the vibration profile may be defined by a vibration waveform and a vibration amount. The driving data generating unit 238 generates second driving data for driving the vibration unit 40 based on the vibration profile.
[0068] The vibration waveform storage unit 254 may store a vibration waveform, which is a pattern for vibrating the vibration unit 40, for each ink color. Furthermore, the map storage unit 252 may store a map that defines the relationship between the movement speed and the vibration amount of the vibration unit 40 for each ink color. By preparing a vibration waveform and vibration amount according to the ink color, it is possible to provide the user with tactile feedback according to the color of the line being drawn, and the user can enjoy the difference in tactile feedback depending on the color of the line.
[0069] 16 shows an example of a screen displayed on the display panel 130 of the HMD 100. In a drawing application, a line drawn by a user maintains its shape and is used as a shield to block an incoming virtual object. In this scene, a shield 24 is attached to the tip of the virtual device 22, and the user points the shield 24 in the direction in which a virtual object (in this example, a cannonball) is coming, to protect the player character from the virtual object.
[0070] The relationship identifying unit 234 identifies a relative relationship between the virtual device 22 and the shield 24, and the cannonball which is a virtual object, and the action determining unit 236 determines an action on the operation device 20 based on the relative relationship identified by the relationship identifying unit 234. In the first embodiment, when the shield 24 and the virtual object come into contact with each other, the action determining unit 236 may determine an action caused by the contact between them.
[0071] The relationship identifying unit 234 determines a collision between the virtual object, which is a bullet, and the shield 24, based on the three-dimensional coordinates of the virtual object and the three-dimensional coordinates of the shield 24. When a collision between the virtual object and the shield 24 is determined, the action determining unit 236 determines to drive the movable unit 32, which is a force sense presentation unit.
[0072] FIG. 17 shows an example of a map that defines the relationship between the time elapsed since the collision and the amount of movement of the movable part 32. This relationship is stored as a map in the map storage unit 252, and the amount of movement indicates the amount of retraction of the movable part 32 from the reference position. The map storage unit 252 may store a map that defines the relationship between the time elapsed since the collision and the amount of movement of the movable part 32 for each virtual object. For example, the map may be set according to the type and size of the virtual object. When the action determination unit 236 is notified of the collision by the relationship identification unit 234, it reads out the map stored in the map storage unit 252 and passes it to the drive data generation unit 238. The drive data generation unit 238 generates first drive data for driving the movable part 32 based on the movement profile defined in the map.
[0073] Furthermore, in the first embodiment, the relationship identifying unit 234 identifies the relative speed between the virtual device 22 and the shell, which is a virtual object, and the action determining unit 236 determines the action on the operation device 20 based on the relative speed identified by the relationship identifying unit 234. Specifically, the action determining unit 236 derives a vibration profile of the vibration unit 40, which is a tactile presentation unit, based on the relative speed between the virtual device 22 and the virtual object. Here, the vibration profile may be defined by a vibration waveform and a vibration amount.
[0074] FIG. 18 shows an example of a vibration waveform. The horizontal axis indicates the elapsed time from the collision. The vibration waveform storage unit 254 stores a vibration waveform, which is a pattern for vibrating the vibration unit 40, for each colliding virtual object. When the action determination unit 236 receives the moving speed from the relationship specification unit 234, it refers to the map stored in the map storage unit 252 to obtain a vibration amount corresponding to the moving speed, and derives a vibration profile by multiplying the vibration waveform corresponding to the virtual object stored in the vibration waveform storage unit 254 by the vibration amount. The action determination unit 236 passes the derived vibration profile to the drive data generation unit 238, and the drive data generation unit 238 generates second drive data for driving the vibration unit 40 based on the vibration profile.
[0075] The transmission unit 260 transmits the first drive data and the second drive data generated by the drive data generation unit 238 to the operation device 20. In the operation device 20, the control unit 50 drives the movable unit 32 based on the first drive data, and drives the vibration unit 40 based on the second drive data. According to the information processing system 1 of the embodiment, the entertainment value of the application can be improved by simultaneously providing force feedback and haptic feedback in response to the operation of the operation device 20.
[0076] Example 2 In the operating device 20 of the first embodiment, the moving mechanism 300 moves the movable part 32 in the longitudinal direction of the base body 30, thereby presenting the user with a force sensation of pushing the fingers in the longitudinal direction of the base body 30. In the operating device 400 of the second embodiment, the movable part 403 is rotated around an axis parallel to the longitudinal direction of the base body 402, thereby presenting the user with a force sensation of moving the fingers around the longitudinal direction of the base body 402.
[0077] 19 shows a state in which a user holds an operating device 400 according to the second embodiment. The operating device 400 includes a base 402 held by the user's hand, and a movable part 403 which is a part movable relative to the base 402. The operating device 400 is a pen-shaped haptic device, and the base 402 has both side surfaces arranged substantially parallel to each other. The user uses the operating device 400 by holding both side surfaces of the base 402 between the thumb and middle finger and placing the index finger on the movable part 403.
[0078] The movable unit 403 functions as a force sense presentation unit that presents a force sense to the hand of the user. The movable unit 403 is driven based on drive data transmitted from the information processing device 10 to provide force sense feedback to the user. The movable unit 403 in the second embodiment is movable relative to the base body 402 in the longitudinal direction of the base body 402 and is further rotatable relative to the base body 402 around an axis parallel to the longitudinal direction of the base body 402.
[0079] A switch installation member 406 having an operation button 407 operable with the tip of a middle finger is provided on a side surface of the base 402. The operation button 407 is a push button, and a user turns on a switch (not shown) having a contact structure by pressing the operation button 407. The user can turn on the switch by pressing the operation button 407 with his / her middle finger.
[0080] The operation device 400 of the second embodiment, like the operation device 20 of the first embodiment, is used as a virtual device 22 that emits ink in a drawing application for drawing pictures and characters in a virtual three-dimensional space. The virtual device 22 is displayed on the display panel 130 and the output device 15 as a virtual pen including a pen tip. In this drawing application, the tip of the operation device 400 corresponds to the pen tip of the virtual device 22, and when the user moves the operation device 400 in the real space, the virtual device 22 behaves in conjunction with the operation device 400 in the virtual space.
[0081] In the drawing application, a force sense in the direction in which the pen tip is pressed is presented to the index finger as the movable part 403 slides relative to the base 402, and a force sense (corresponding to a frictional force acting on the pen tip) in the opposite direction to the direction in which the pen tip moves in the virtual space is presented to the index finger as the movable part 403 rotates relative to the base 402. The force sense presented allows the user to experience drawing pictures and letters by moving the virtual pen.
[0082] 20 is a perspective view of the operating device 400. In the operating device 400, the side where the movable part 403 is provided is defined as the tip side. A vibration part 408 including a vibrator is provided inside the switch installation member 406. The vibration part 408 functions as a tactile sensation providing part that provides a tactile sensation to the user's hand (middle finger) by vibrating, and is driven based on drive data transmitted from the information processing device 10 to provide tactile feedback to the user.
[0083] 20, the movable part 403 is at a position that protrudes most forward from the base body 402, and this position is the "reference position" of the movable part 403. In addition, in the state shown in FIG. 20, the rotation angle of the movable part 403 is zero, and this posture is the "reference posture" of the movable part 403.
[0084] A finger rest 405 is provided on the upper surface of the movable part 403 for the user to place the tip of the index finger while holding the base body 402. Finger engagement parts 404 with which the tip of the finger engages are provided on both side parts and the front end part of the finger rest 405. The finger engagement parts 404 are erected on the upper surface of a beam member of a support base frame 412 described later. The finger engagement parts 404 have a curved surface 404a that is inclined to fit the pad of the finger and has a curvature. The curved surface 404a is inclined with respect to the upper surface of the beam member and is formed concave in the direction in which the fingertip contacts, so that the user can stably contact the tip of the index finger with the curved surface 404a.
[0085] The user holds both sides of base 402 between the thumb and middle finger as if holding a pen, places the pad of the index finger on finger rest section 405, and places the pad of the middle finger on the surface of switch installation member 406. Note that the user may hold base 402 between three or more fingers and place the pad of another finger that is not primarily involved in holding base 402 on finger rest section 405. In this way, base 402 of operating device 400 is held by two or more fingers, so that base 402 can be fixed in the hand without the need for a special gripping mechanism for fixing base 402 to the hand.
[0086] 21 shows a moving mechanism 420 that rotates the movable part 403 relative to the base body 402. The moving mechanism 420 includes a second rotation actuator 410, a support frame 412, a base 414, a stopper 416, and a second rotation angle sensor 418, and rotates the movable part 403 about an axis. The fixed part 320 shown in FIG. 7 is fixed to a fixed part 422 of the movable part 403, so that the movable part 403 can be slid by the moving mechanism 300.
[0087] The base 414 is fixed to the housing of the movable part 403, and rotatably supports the support frame 412 on which the finger rest 405 is installed. The support frame 412 has a gate-shaped structure including a pair of pillar members and a beam member connecting the pair of pillar members, and the pair of pillar members are rotatably connected to the base 414, and the beam member mounts the finger rest 405 on the upper surface. The second rotation actuator 410 is fixed to the base 414, and the motor shaft is connected to one of the pillar members of the support frame 412. In the movement mechanism 420, the second rotation actuator 410 rotates forward or backward, and thus the support frame 412 rotates right or left when the operating device 400 is viewed from the front. A pair of stoppers 416 is formed on the base 414, and the stoppers 416 regulate the rotation of the support frame 412. The second rotation angle sensor 418 detects the rotation angle of the second rotation actuator 410 , and the rotation of the movable part 403 is controlled based on the detection value of the second rotation angle sensor 418 .
[0088] FIG. 22 shows the inclination of the support base frame 412 when the operating device 400 is viewed from the front. 22(a) shows the base 414 and the support frame 412 in the home position, in which the rotation angle is 0 degrees. 22(b) shows a state in which the support base frame 412 has rotated leftward relative to the base 414. The rotation of the support base frame 412 is restricted by a stopper 416, and the maximum rotation angle during left rotation may be −45 degrees. 22(c) shows a state in which the support base frame 412 has rotated rightward relative to the base 414. The rotation of the support base frame 412 is limited by a stopper 416, and the maximum rotation angle during rightward rotation may be +45 degrees.
[0089] 23 shows functional blocks of the operation device 400. The control unit 450 is a main processor that processes and outputs various data such as sensor data and drive data, and commands. The storage unit 52 temporarily stores the data and commands processed by the control unit 450. The attitude sensor 54 detects attitude information of the operation device 400. The attitude sensor 54 includes at least a three-axis acceleration sensor and a three-axis gyro sensor.
[0090] The communication control unit 56 transmits data output from the control unit 450 to the external information processing device 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 56 also receives data from the information processing device 10 by wired or wireless communication via a network adapter or an antenna, and outputs the data to the control unit 450.
[0091] When the control unit 450 receives the drive data from the information processing device 10, it drives the first rotary actuator 310, the second rotary actuator 410 and / or the vibration unit 408. Here, the first rotary actuator 310 is a drive unit for moving the movable unit 403 in the longitudinal direction of the base body 402, and the second rotary actuator 410 is a drive unit for rotating the movable unit 403 around an axis parallel to the longitudinal direction of the base body 402. In the operation device 400, the movable unit 403 is a part (force sense presentation unit) that presents a force sense to the user's hand, and the vibration unit 408 is a vibrator (tactile sense presentation unit) that presents a tactile sense to the user's hand. The movable unit 403 provides force feedback, and the vibration unit 408 provides tactile feedback, so that the user can realistically experience an event occurring in an application. The control unit 450 causes the communication control unit 56 to transmit the sensor data from the attitude sensor 54 to the information processing device 10.
[0092] The operation device 400 of the second embodiment is used as a virtual device 22 that emits ink in a drawing application for drawing pictures and characters in a virtual three-dimensional space. The virtual device 22 is displayed on the display panel 130 and the output device 15 as a virtual pen including a pen tip. In this drawing application, the tip of the operation device 400 corresponds to the pen tip of the virtual device 22, and when the user moves the operation device 400 in the real space, the virtual device 22 behaves in conjunction with the operation device 400 in the virtual space.
[0093] 24 shows another example of the functional blocks of the information processing device 10. The information processing device 10 includes a receiving unit 200 and a transmitting unit 260 as an input / output interface with the outside. The receiving unit 200 includes an HMD sensor data receiving unit 202, a captured image receiving unit 204, and an operation data receiving unit 206. The information processing device 10 further includes an HMD information acquiring unit 210, an operation information acquiring unit 212, and a processing unit 220. The processing unit 220 includes a control unit 230, a line of sight direction determining unit 250, a map holding unit 252, a vibration waveform holding unit 254, an image generating unit 256, and a sound generating unit 258, and the control unit 230 includes a virtual device control unit 232, a relationship identifying unit 234, an action determining unit 236, a drive data generating unit 238, and a function executing unit 240.
[0094] The information processing device 10 includes a computer, which executes a program to realize various functions shown in FIG. 24. The computer includes hardware such as a memory for loading the program, one or more processors for executing the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of multiple electronic circuits including a semiconductor integrated circuit and an LSI, and the multiple electronic circuits may be mounted on one chip or multiple chips. The functional blocks shown in FIG. 24 are realized by cooperation between hardware and software, and therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof.
[0095] In Fig. 24, functional blocks having the same reference numerals as those in Fig. 9 have the same or similar functions as those in Fig. 9. Therefore, the description of the functional blocks having the same reference numerals as those in Fig. 9 will be omitted as appropriate.
[0096] The operation data receiving unit 206 receives sensor data of the attitude sensor 54 of the operation device 400 held by the user at a predetermined cycle. The operation data receiving unit 206 also receives operation data of the operation buttons 407 of the operation device 400 held by the user. The operation data receiving unit 206 supplies the sensor data of the attitude sensor 54 and the operation data of the operation buttons 407 to the operation information acquisition unit 212.
[0097] In the second embodiment, the movable unit 403 in the operation device 400 has a function of providing a force sense to the user's hand, and also has a function of allowing the user to input operation information for realizing a predetermined function in an application by being moved by the user. That is, the movable unit 403 has a function of an input device for inputting operation information. When the operation data receiving unit 206 receives operation information indicating that the movable unit 403 has been moved by the user's hand, the operation data receiving unit 206 supplies the operation information to the operation information acquiring unit 212.
[0098] The operation information acquisition unit 212 acquires, from the sensor data of the operation device 400, attitude information indicating the attitude of the operation device 400 in the real space and position information indicating the position of the operation device 400. The operation information acquisition unit 212 mainly calculates the attitude information of the operation device 400 using the sensor data of the three-axis gyro sensor, and calculates the position information of the operation device 400 using the sensor data of the three-axis acceleration sensor. The attitude information and position information of the operation device 400 constitute operation information indicating the movement of the operation device 400. The operation information acquisition unit 212 supplies the attitude information and position information of the operation device 400 to the processing unit 220 as operation information indicating the movement of the operation device 400. The operation information acquisition unit 212 also supplies the operation data of the operation button 407 and the operation information of the movable part 403 to the processing unit 220.
[0099] The virtual device control unit 232 controls the movement of the virtual device 22 in the virtual space according to the operation information indicating the movement of the operation device 400. Specifically, the virtual device control unit 232 periodically acquires the attitude information and position information of the operation device 400 from the operation information acquisition unit 212, determines the attitude and position of the virtual device 22 in the virtual space according to the attitude information and position information of the operation device 400, and links the virtual device 22 in the virtual space to the movement of the operation device 400 in the real space.
[0100] The virtual device control unit 232 may determine the position or orientation of the virtual device 22 in the virtual space in accordance with the position information or orientation information of the operation device 400. In other words, the virtual device control unit 232 may link the position of the operation device 400 in the real space with the position of the virtual device 22 in the virtual space, or may link the orientation of the operation device 400 in the real space with the orientation of the virtual device 22 in the virtual space.
[0101] In the drawing application of the second embodiment, when the user moves the operation device 400 while pressing the operation button 407, the trajectory of the pen tip of the virtual device 22 corresponding to the tip of the operation device 400 is drawn in the virtual space, and when the user releases the press of the operation button 407, the drawing of the trajectory of the pen tip ends.
[0102] <Scene of drawing lines in space> 25 shows an example of a drawing application screen displayed on the display panel 130 of the HMD 100. In this application screen, a virtual object (eraser) for erasing a drawing made by the user is placed in the lower left. First, the user places the pen tip of the virtual device 22 at the drawing start position.
[0103] 26 shows an example of a screen of a drawing application. When a user moves the operation device 400 in the real space while pressing the operation button 407, a trajectory of the pen tip of the virtual device 22 corresponding to the tip of the operation device 400 is drawn in the virtual space. In this scene, the relationship identifying unit 234 identifies the moving speed of the virtual device 22, and the action determining unit 236 determines an action on the operation device 400 based on the moving speed identified by the relationship identifying unit 234.
[0104] The relationship specification unit 234 specifies the moving speed of the virtual device 22 from the moving speed of the operation device 400 on which the operation button 407 is pressed. The action determination unit 236 derives a vibration profile of the vibration unit 408, which is a tactile sense presentation unit, based on the moving speed of the virtual device 22. As described above, the vibration profile may be defined by a vibration waveform and a vibration amount. The drive data generation unit 238 generates second drive data for driving the vibration unit 408 based on the vibration profile. As described in the first embodiment, the drive data generation unit 238 may generate the second drive data based on a vibration waveform and a vibration amount according to the color of the line. When the user stops pressing the operation button 407, the drawing application stops drawing the trajectory of the pen tip.
[0105] Fig. 27 shows an example of a screen of a drawing application. When pressing of operation button 407 is finished, the drawing application finishes drawing the trajectory of the pen tip in the virtual space. The example shown in Fig. 27 shows a state where the user has finished operating operation button 407 and drawing of line 430 has been finished.
[0106] The drawing application of the second embodiment implements an application in which a drawn line 430 is designated with a pen tip, and the line shape is extended in the direction of movement of the pen tip by moving the pen tip. 28 shows a state in which a drawn line is designated with the pen tip. The designation of the line 430 is performed by placing the pen tip of the virtual device 22 on the line 430. When the line 430 is designated, the drawing application may display the designated line 430 thicker or in a different color to notify the user that the designation of the line 430 has been completed.
[0107] 29 shows a state in which the line shape is stretched in the moving direction of the pen tip. When the user moves the operation device 400 while pressing the operation button 407, the shape of the line 430 is stretched in the moving direction of the pen tip in the virtual space. In this example, the line 430 is a straight line, but if the line 430 is a curved line, a surface in which the curved line shape is stretched in the moving direction of the pen tip is drawn.
[0108] The action determination unit 236 derives the movement direction and movement amount of the movable unit 403, which is a force sense presentation unit, based on the movement of the virtual device 22 that stretches the line shape. Specifically, when the shape of the line 430 is stretched by the virtual device 22, the action determination unit 236 derives the movement direction (rotation direction) of the movable unit 403 based on the stretched direction, and derives the movement amount (rotation amount) of the movable unit 403 in the rotation direction based on the stretched length. When the line shape is stretched to the right with respect to the initial position of the line 430, the action determination unit 236 determines to rotate the movable unit 403 in the opposite direction to the right direction in which the pen tip moves (i.e., the left rotation direction from the user's viewpoint, and the right rotation direction when the operating device 400 is viewed from the front (the rotation direction shown in FIG. 22(c))). On the other hand, when the line shape is extended to the left relative to the initial position of the line 430, the action determination unit 236 determines to rotate the movable part 403 in the opposite direction to the left direction in which the pen tip moves (i.e., a right rotation direction from the user's viewpoint, and a left rotation direction when the operating device 400 is viewed from the front (the rotation direction shown in Figure 22 (b))).
[0109] In the example shown in FIG. 29, since the virtual device 22 is moving in the right direction, the action determination unit 236 determines the rotation direction of the movable part 403 to rotate right when the operation device 400 is viewed from the front. The action determination unit 236 also determines the rotation amount of the movable part 403 according to the movement amount of the virtual device 22. As shown in FIG. 12, the map holding unit 252 holds a map that defines the relationship between the movement amount of the virtual device 22 and the rotation amount of the movable part 403, and the action determination unit 236 may obtain the rotation amount of the movable part 403 according to the movement amount of the virtual device 22 by referring to the map. The action determination unit 236 passes the determined rotation direction and rotation amount to the drive data generation unit 238. The drive data generation unit 238 generates first drive data for driving the second rotation actuator 410 based on the rotation direction and rotation amount.
[0110] The transmission unit 260 transmits the first drive data generated by the drive data generation unit 238 to the operation device 400. In the operation device 400, the control unit 450 drives the second rotation actuator 410 based on the first drive data to rotate the movable part 403. Therefore, the movable part 403 rotates in a rotation direction and by an amount of rotation according to the movement direction and amount of movement of the virtual device 22. At this time, the second rotation angle sensor 418 detects the rotation angle of the second rotation actuator 410, and the rotation of the movable part 403 is controlled based on the detection value of the second rotation angle sensor 418.
[0111] 29, the user's index finger is presented with a force sense in a counterclockwise direction as viewed from the user's viewpoint by the movable part 403. This force sense in the counterclockwise direction corresponds to the frictional force acting on the pen tip when the operation device 400 is moved to the right, so the user can experience the action of extending a line shape to form a surface.
[0112] In the second embodiment, the movable part 403 not only presents a force sense to the user, but also functions as an input device for the user to input an operation to an application. The control unit 450 monitors the angle detected by the second rotation angle sensor 418, and when it detects that the index finger of the user has rotated the movable part 403, causes the communication control unit 56 to transmit operation information indicating that the movable part 403 has been moved by the hand of the user to the information processing device 10.
[0113] For example, it is assumed that the movable part 403 has rotated α degrees in the right direction as viewed from the front from a state in which it is in a reference posture with a rotation angle of 0 degrees as shown in Fig. 22(a) by the first drive data generated by the information processing device 10. Therefore, the second rotation angle sensor 418 detects a rotation angle of +α degrees, and the control part 450 controls the second rotation actuator 410 to maintain the rotation angle of +α degrees.
[0114] In this state, the user's index finger rotates the movable part 403 so that the rotation angle of the movable part 403 deviates from +α degrees. At this time, the control unit 450 accepts this movement of the movable part 403 as operation information for the application based on the detection value of the second rotation angle sensor 418. Note that the control unit 450 may accept the movement of the movable part 403 as operation information for the application when a period during which the rotation angle of the movable part 403 becomes an angle different from +α degrees continues for a predetermined time (for example, 1 second). When the control unit 450 accepts operation information indicating that the movable part 403 has been moved by the user's hand, it causes the communication control unit 56 to transmit the operation information to the information processing device 10. This operation information may include the direction in which the movable part 403 has been moved by the user's hand.
[0115] When the angle detected by the second rotation angle sensor 418 is (α-β), the control unit 450 determines that the user rotated the movable unit 403 in a counterclockwise direction by β degrees when viewed from the front. That is, it determines that the user rotated the movable unit 403 in a direction opposite to the direction in which the movable unit 403 was moved by the first drive data (the direction in which the force sense was applied). On the other hand, when the angle detected by the second rotation angle sensor 418 is (α+β), the control unit 450 determines that the user rotated the movable unit 403 in a clockwise direction by β degrees when viewed from the front. That is, it determines that the user rotated the movable unit 403 in the same direction as the direction in which the movable unit 403 was moved by the first drive data (the direction in which the force sense was applied). The control unit 450 causes the communication control unit 56 to transmit operation information indicating the direction in which the movable unit 403 was moved by the user's hand to the information processing device 10.
[0116] In the information processing device 10, when the operation information acquisition unit 212 acquires operation information indicating that the movable unit 403 has been moved by the user's hand, the operation information is provided to the control unit 230. In the control unit 230, the function execution unit 240 executes an application function corresponding to the operation information in the virtual space.
[0117] FIG. 30 shows a state in which the linear shape is further extended from the pen tip. When the operation information indicates that the movable part 403 has been moved by the user's hand in a direction different from the direction in which the movable part 403 was moved by the drive data, the function execution unit 240 executes a function to further extend the linear shape from the pen tip. As a result, the surface formed by extending the linear shape is further expanded in the moving direction of the pen tip. Since there is an upper limit to the stroke amount in the rotation direction of the movable part 403, the user is likely to move the movable part 403 in the opposite direction to the direction in which the movable part 403 was moved by the drive data. Therefore, it is effective to assign a function of an application to the user moving the movable part 403 in the opposite direction to the direction in which the movable part 403 was moved by the drive data, because a sufficient stroke amount can be secured due to the structure of the movable part 403.
[0118] When the operation information indicates that the movable part 403 has been moved by the user's hand in the same direction as the direction in which the movable part 403 has been moved by the drive data, the function execution unit 240 may execute a function of making the stretching amount of the linear shape shorter than the position of the pen tip. In the drawing application of the second embodiment, the user can adjust the stretching amount of the linear shape by moving the pen tip of the virtual device 22, but the stretching amount can also be adjusted by rotating the movable part 403, so that the stretching amount can be easily fine-tuned.
[0119] The operation information acquisition unit 212 acquires operation information indicating that the movable unit 403 has been moved by the user's hand while the movable unit 403 is presenting a force sense to the user's hand, and the function execution unit 240 executes a function corresponding to the operation information in the virtual space while the movable unit 403 is presenting a force sense to the user's hand. In this way, the control unit 230 can use the deviation from the amount of rotation (+α degrees) of the second rotation actuator 410 based on the first drive data as operation information of the movable unit 403 by the user.
[0120] In the drawing application of the second embodiment, the user can erase a drawn object from the screen by operating the movable part 403 of the eraser located at the bottom left of the screen. <Scene of erasing a drawing> 31 shows a state in which the pen tip of the virtual device 22 is in contact with an object (eraser). The relationship specifying unit 234 specifies the relative relationship between the virtual device 22 and the eraser 434, which is a virtual object, and the action determining unit 236 determines the action on the operation device 400 based on the relative relationship specified by the relationship specifying unit 234.
[0121] The relationship specifying unit 234 specifies the positional relationship between the virtual device 22 and the eraser 434 from the three-dimensional coordinates of the eraser 434 arranged at a predetermined position and the three-dimensional coordinates of the virtual device 22. In the second embodiment, the relationship specifying unit 234 derives the length by which the virtual device 22 is pressed into the eraser 434 from the specified positional relationship. When the virtual device 22 is pressed into the eraser 434, the action determining unit 236 determines to move the movable unit 403, which is a force sense presentation unit, in a direction to pull it into the base body 402. At this time, the action determining unit 236 may derive the movement amount of the movable unit 403, which is a force sense presentation unit, based on the pressing length. The driving data generating unit 238 generates first driving data for driving the first rotation actuator 310 based on the movement amount.
[0122] The transmission unit 260 transmits the first drive data generated by the drive data generation unit 238 to the operation device 400. In the operation device 400, the control unit 450 drives the first rotation actuator 310 based on the first drive data to move the movable part 403. At this time, the first rotation angle sensor 322 detects the rotation angle of the first rotation actuator 310, and the movement of the movable part 403 is controlled based on the detection value of the first rotation angle sensor 322. The user can feel that the virtual device 22 is in contact with the eraser 434 by the force sense presented by the movable part 403.
[0123] 31, the movable part 403 presents a force sense in the pushing direction of the pen tip to the index finger of the user. In this state, when the index finger of the user pushes back the movable part 403, the control unit 450 obtains the amount of movement of the index finger of the user pushing back the movable part 403 from the angle detected by the first rotation angle sensor 322, and causes the communication control unit 56 to transmit operation information indicating that the movable part 403 has been moved by the hand of the user to the information processing device 10.
[0124] For example, it is assumed that the movable part 403 has moved from the reference position by a retraction amount A due to the first drive data generated by the information processing device 10. Therefore, the first rotation angle sensor 322 detects the rotation angle a degrees of the first rotation actuator 310 which corresponds to the retraction amount A, and the control unit 450 controls the first rotation actuator 310 so as to maintain the rotation angle of the first rotation actuator 310 at a degrees.
[0125] In this state, the user's index finger moves the movable part 403 in the sliding direction so that the retraction amount deviates from the retraction amount A. At this time, the control unit 450 accepts this movement of the movable part 403 as operation information for the application based on the detection value of the first rotation angle sensor 322. Specifically, the control unit 450 accepts operation information indicating that the movable part 403 has been moved by the user's hand, and causes the communication control unit 56 to transmit the operation information to the information processing device 10. This operation information may include the amount of movement of the movable part 403 by the user's hand.
[0126] When the angle detected by the second rotation angle sensor 418 is (ab), the control unit 450 determines that the user has moved the movable unit 403 in the direction in which the movable unit 403 is pushed back by an amount of movement equivalent to the detected angle b. In other words, it determines that the user has moved the movable unit 403 in the direction opposite to the direction in which the movable unit 403 was moved by the first drive data (the direction in which the force sense was applied). Note that the control unit 450 may accept the movement of the movable unit 403 as operation information for an application when a period during which the rotation angle of the first rotation actuator 310 becomes an angle different from a degree continues for a predetermined time (for example, 1 second). The control unit 450 causes the communication control unit 56 to transmit operation information indicating that the movable unit 403 has been moved by the user's hand to the information processing device 10.
[0127] In the information processing device 10, the operation information acquisition unit 212 acquires operation information indicating that the movable unit 403 has been moved by the user's hand, and provides the operation information to the control unit 230. In the control unit 230, the function execution unit 240 executes a function corresponding to the operation information in the virtual space. In this example, the function execution unit 240 executes a function to erase the drawn object 432, thereby erasing the drawn object 432.
[0128] 32 shows a state in which the drawing object 432 has been erased. In this way, the function executing unit 240 executes an application function assigned to the operation information based on the operation information of the user operating the movable unit 403, thereby making it possible to appropriately reflect the user's intention to execute a function.
[0129] The present invention has been described above based on a number of embodiments. These embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present invention. In the embodiments, the control unit 230 executes a VR application, but an application other than a VR application may be executed.
[0130] In the embodiment, the effect determination unit 236 derives a vibration profile, and the drive data generation unit 238 generates second drive data according to the vibration profile. In a modified example, the effect determination unit 236 may apply the method of deriving a vibration profile to a sound effect, and the sound generation unit 258 may generate a sound synchronized with the haptic feedback. [Industrial Applicability]
[0131] The present invention can be used in techniques that provide force and / or haptic feedback to a user. [Explanation of symbols]
[0132] 1 Information processing system, 10 Information processing device, 20 Operation device, 22 Virtual device, 30 Base body, 32 Movable part, 34 Finger engagement part, 34a Curved surface, 36 Finger rest, 40 Vibration part, 42 Operation button, 50 Control part, 52 Memory part, 54 Posture sensor, 56 Communication control part, 100 HMD, 102 Output mechanism, 104 Wearing mechanism, 106 Wearing band, 108 Housing, 110 Light emitting ma a sensor, 120, a control unit, 122, a storage unit, 124, a posture sensor, 126, a microphone, 128, a communication control unit, 130, a display panel, 130a, a display panel for the left eye, 130b, a display panel for the right eye, 132, a sound output unit, 200, a receiving unit, 202, an HMD sensor data receiving unit, 204, a captured image receiving unit, 206, an operation data receiving unit, 210, an HMD information acquiring unit, 212, an operation information acquiring unit, 220, a processing unit, 230, a control unit, 232....Virtual device control unit, 234....Relationship identification unit, 236....Action determination unit, 238....Drive data generation unit, 240....Function execution unit, 250....Gaze direction determination unit, 252....Map storage unit, 254....Vibration waveform storage unit, 256....Image generation unit, 258....Sound generation unit, 260....Transmission unit, 300....Movement mechanism, 310....First rotary actuator, 312....Feed screw, 314....Nut, 316....Guide mechanism, 318....Rod, 320.... Fixed portion, 322...first rotation angle sensor, 400...operation device, 402...base, 403...movable portion, 404...finger engagement portion, 404a...curved surface, 405...finger rest portion, 406...switch installation member, 407...operation button, 408...vibration portion, 410...second rotation actuator, 412...support base frame, 414...base, 416...stopper, 418...second rotation angle sensor, 420...moving mechanism, 422...fixed portion, 450...control portion.
Claims
1. An information processing device, comprising one or more processors having hardware, The one or more processors: acquiring first operation information indicating a movement of an operation device that is held in a user's hand and includes a force sense presentation unit that presents a force sense to the user's hand; Controlling a movement of a virtual device corresponding to the operation device in a virtual space in response to first operation information; generating drive data for driving the haptic unit based on the movement of the virtual device; acquiring second operation information indicating that the force sense supply unit is moved by a user's hand; executing a function corresponding to the second operation information in the virtual space; Information processing device.
2. the second operation information includes a direction in which the force sense supply unit is moved by the user's hand; The information processing device according to claim 1 .
3. The second operation information indicates that the force sense supply unit has been moved by the user's hand in a direction different from a direction in which the force sense supply unit has been moved by the drive data. The information processing device according to claim 2 .
4. the second operation information indicating that the force sense supply unit has been moved by the user's hand in a direction opposite to a direction in which the force sense supply unit has been moved by the drive data; The information processing device according to claim 3 .
5. the second operation information indicating that the haptic unit has been moved by the user's hand in the same direction as the direction in which the haptic unit has been moved by the drive data; The information processing device according to claim 2 .
6. The one or more processors: executing a function corresponding to second operation information in a virtual space while the force sense presentation unit is presenting a force sense to the user's hand; The information processing device according to claim 1 .
7. A method for processing information in an information processing device, comprising: The information processing device, acquiring first operation information indicating a movement of an operation device that is held in a user's hand and includes a force sense presentation unit that presents a force sense to the user's hand; Controlling a movement of a virtual device corresponding to the operation device in a virtual space in response to first operation information; generating drive data for driving the haptic unit based on the movement of the virtual device; acquiring second operation information indicating that the force sense supply unit is moved by a user's hand; executing, in the virtual space, a function corresponding to the second operation information; 23. An information processing method comprising:
8. On the computer, a function of acquiring first operation information indicating a movement of an operation device that is held by a user's hand and includes a force sense presentation unit that presents a force sense to the user's hand; a function of controlling a movement of a virtual device corresponding to the operation device in a virtual space in response to first operation information; a function of generating drive data for driving the haptic unit based on a movement of a virtual device; a function of acquiring second operation information indicating that the force sense presentation unit is moved by a user's hand; a function of executing, in a virtual space, a function corresponding to the second operation information; A program to achieve this.
9. An information processing device, comprising one or more processors having hardware, The one or more processors: acquiring operation information indicating a movement of an operating device including a force sense presentation unit that is held by a user's hand and presents a force sense to a first finger of the user, and a tactile sense presentation unit that presents a tactile sense to a second finger different from the first finger of the user and is a separate component from the force sense presentation unit; Controlling the movement of a virtual device corresponding to the operation device in a space in which a virtual object exists in accordance with operation information; generating first drive data for driving the force sense provision unit and second drive data for driving the haptic sense provision unit based on a relationship between a virtual device and a virtual object; Information processing device.
10. The force sense providing unit is a movable unit that moves relative to a base body of the operation device, and the tactile sense providing unit is a vibration unit that generates vibrations.
10. The information processing apparatus according to claim 9,
11. The one or more processors: generating first drive data for driving the force sense presentation unit based on a positional relationship between a virtual device and a virtual object; The information processing device according to claim 9.
12. The one or more processors: generating second drive data for driving the haptic sense providing unit based on a relative speed between the virtual device and the virtual object; The information processing device according to claim 9.
13. The one or more processors: generating second drive data for driving the haptic sense providing unit based on a vibration profile derived from a vibration waveform determined according to a virtual object and a vibration amount determined according to a relative speed between the virtual device and the virtual object; The information processing device according to claim 12.
14. acquiring operation information indicating a movement of an operating device including a force sense presentation unit that is held by a user's hand and presents a force sense to a first finger of the user, and a tactile sense presentation unit that presents a tactile sense to a second finger different from the first finger of the user and is a separate component from the force sense presentation unit; Controlling the movement of a virtual device corresponding to the operation device in a space in which a virtual object exists in accordance with operation information; generating first drive data for driving the force sense provision unit and second drive data for driving the haptic sense provision unit based on a relationship between a virtual device and a virtual object; A driving data generating method comprising:
15. On the computer, a function of acquiring operation information indicating a movement of an operating device including a force sense presentation unit that is held by a user's hand and presents a force sense to a first finger of the user, and a tactile sense presentation unit that presents a tactile sense to a second finger different from the first finger of the user and is a separate component from the force sense presentation unit; a function of controlling a movement of a virtual device corresponding to the operation device in a space in which a virtual object exists, in accordance with operation information; a function of generating first drive data for driving the force sense provision unit and second drive data for driving the haptic sense provision unit based on a relationship between a virtual device and a virtual object; A program to achieve this.
Citation Information
Patent Citations
Tactile force presentation method and tactile force pen tablet
JP2000200140A
Handheld computer interface with haptic feedback
JP2010225155A
Haptic device
JP2010287221A
Tactile sense reproduction device
JP2017123039A
Hardness sense presentation device
JP2019106102A
