Information processing apparatus, information processing system, information processing program, and information processing method

Abstract

To provide force sense feedback control with excellent operability when presenting tactile force sense technology to a user.SOLUTION: The information processing apparatus includes an acquisition unit configured to acquire information related to a physical feature of a virtual object, a setting unit configured to set a first region and a second region different from the first region in a mixed reality space generated such that the virtual object is arranged in a real space or a virtual space in which the virtual object is arranged in conformity with the real space, and a control unit configured to control not to generate a tactile effect based on the information in a case where a position of the virtual object held by a user is in the first region in the mixed reality space or the virtual space. And a control unit configured to perform control to generate the haptic effect when the position is in the second region.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device that presents a haptic sensation to a user in a virtual space or a mixed reality space. [Background technology]

[0002] Cross reality (XR) systems have been proposed that allow users to experience virtual reality using a head-mounted display (HMD). Haptic technology is one of the means for sensing the texture and weight of an object when touching it in these virtual realities. For example, Patent Document 1 discloses a technology that applies a special vibration stimulus to the user, tricking the brain into thinking that the user is feeling a force as if they are touching a virtual object (also called a virtual object). In other words, Patent Document 1 discloses a technology for presenting such haptic sensations. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-142374 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when presenting a haptic sensation such as weight in response to a user's actions such as grasping (grabbing) or releasing a virtual object, the haptic sensation may be presented when the user's posture is unstable. In this case, depending on the type and size (strength) of the presented haptic sensation, there is a concern that the user may be surprised, resulting in poor operability.

[0005] Therefore, an object of the present invention is to provide haptic feedback control that is easy for the user to operate when the user performs an action on a virtual object. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the information processing device of the present invention is characterized by having an acquisition means for acquiring information regarding the physical characteristics of a virtual object; a setting means for setting a first area and a second area different from the first area in a mixed reality space generated so that the virtual object is placed in a real space, or in a virtual space conforming to the real space in which the virtual object is placed; and a control means for controlling not to generate a haptic effect based on the information when the position of the virtual object being held by a user in the mixed reality space or the virtual space is in the first area, and for controlling to generate the haptic effect when the position is in the second area. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technique for performing haptic feedback control that is easy for the user to operate when the user performs an action on a virtual object. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating the internal configuration of the information processing device according to the first embodiment. [Figure 3] 10 is a flowchart illustrating a procedure for determining a force-sense effective area and a force-sense ineffective area in the first embodiment. [Figure 4] 10 is a flowchart illustrating a procedure of a process for presenting force feedback in the first embodiment. [Figure 5] 10A and 10B are diagrams illustrating a first area and a second area based on the position of a virtual object in the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating a first area and a second area based on the position of a user in the first embodiment. [Figure 7] 10A and 10B are diagrams illustrating stepwise changes in force sense parameters in the first embodiment. [Figure 8] 10 is a flowchart illustrating a procedure of a process for presenting force feedback in the second embodiment. [Figure 9] FIG. 10 is a diagram illustrating a first area and a second area based on the position of a virtual object in the second embodiment. [Figure 10] 10A to 10C are diagrams illustrating examples of positions where a virtual object is held in the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Each embodiment will be described below with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing will be assigned the same reference numerals, and duplicate descriptions will be omitted where appropriate. In addition, some of the components, members, and processes will be omitted in each drawing.

[0010] Example 1 <System configuration> The configuration of an information processing system according to the first embodiment will be described with reference to Fig. 1. The information processing system includes an HMD 100 and a user assistive device 110.

[0011] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the user's head. The HMD 100 is equipped with a camera for capturing images of the area in front of the user and a display for displaying images to the user. The HMD 100 displays a composite image on the display that combines the captured image with virtual objects, which are CG (computer graphics) content. This allows the user to experience virtual reality with their own eyes. The HMD 100 also has the function of detecting the user's hands from the captured image and acquiring information related to the position and posture of the hands, thereby allowing the hand movements to affect virtual objects. This allows the user to intuitively operate virtual objects using their hands.

[0012] The user assistance device 110 is a device for presenting the force sensation instructed from the HMD 100 to the user. In the first embodiment, the shape of the user assistance device 110 is described as a glove type, but it may be a bracelet type, a ring type, or the like. Note that the shape of the user assistance device 110 may be a controller that the user holds.

[0013] <Internal Configuration of HMD> Referring to FIG. 2(a), the internal configuration of the HMD 100 will be described. The HMD 100 includes a control unit 200, a communication unit 201, a RAM 202, a ROM 203, an imaging unit 204, a display unit 205, and an attitude sensor unit 206.

[0014] The control unit 200 controls each component of the HMD 100. The control unit 200 executes programs stored in the ROM 203, utilizes the RAM 202 as a work area, and controls the overall processing unit of the HMD 100. The control unit 200 is configured with, for example, one or more processors such as a CPU or a GPU. Note that instead of the control unit 200 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing. The control unit 200 receives an image (captured image) acquired by the imaging unit 204 and orientation information acquired by the orientation sensor unit 206. The control unit 200 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 204 and the optical system of the display unit 205. The control unit 200 then combines the captured image with any CG to generate a composite image and controls the display unit 205 to display it. The control unit 200 also acquires the coordinates of the HMD 100 in real space based on the captured image acquired from the imaging unit 204. This type of composite image, in which CG is superimposed on an image of real space, is presented to the user as a composite image known as MR (mixed reality). Mixed reality spaces are generated by acquiring information such as the position, orientation, and depth of real objects in real space from captured images of the real space and other sensors, and then placing real and virtual objects in the space. By displaying a mixed reality space to the user, the user can perceive virtual objects as if they were actually present in real space. In other words, a space in which real and virtual spaces are combined (fused) can be presented, allowing the user to experience the space as if they were actually present. Mixed reality spaces simultaneously provide users with information from both the virtual and real worlds, enabling them to have virtual experiences in real space and to communicate realistically with people in real space through virtual space, even from remote locations.

[0015] The control unit 200 may acquire the coordinates of the HMD 100 in real space based on the captured image acquired from the imaging unit 204 and the information (position information and orientation information) acquired by the orientation sensor unit 206.

[0016] The control unit 200 estimates the position or posture of each joint point of the user's hand and fingers from the two camera images acquired by the imaging unit 204. The joint points include characteristic points of parts such as the finger joints and fingertips, the back of the hand (palm), and the arm. Each joint point indicates a coordinate position, and the posture can be estimated from information on multiple joint points. Methods for estimating the position or posture of the hand and each joint point can include known object recognition and pose estimation techniques based on machine learning using, for example, a convolutional neural network. Depth position information of each joint point of the hand can be obtained by calculating the distance from the imaging unit 204 to each joint point, for example, by triangulation using stereo matching using the two camera images acquired by the imaging unit 204.

[0017] The control unit 200 controls the position, orientation, and size of the CG in the composite image based on the information (position information and orientation information) acquired by the orientation sensor unit 206. For example, when placing a virtual object represented by CG near a specific object existing in real space in the space represented by the composite image, the control unit 200 makes the virtual object (CG) larger the closer the distance between the specific object and the imaging unit 204. By controlling the position, orientation, and size of the CG in this manner, the control unit 200 can generate a composite image in which a CG object that is not located in real space appears to be located in real space.

[0018] The communication unit 201 performs wireless communication with the user assistive device 110 (communication unit 211) using Bluetooth (registered trademark). Note that wireless communication may also be performed using Wi-Fi (Wireless Fidelity) (registered trademark).

[0019] The RAM 202 is a temporary storage area such as the main memory and work area of ​​the HMD 100. The RAM 202 stores the haptic effective area and haptic ineffective area determined based on the flowchart of Fig. 3, and the haptic parameter amounts determined based on the flowchart of Fig. 4. The RAM 202 is also used as a buffer memory that temporarily stores image data captured by the imaging unit 204, an image display memory for the image display unit 203, a work area for the control unit 201, etc.

[0020] The ROM 203 is an electrically erasable and recordable nonvolatile memory that stores control programs executed by the control unit 201 and mass information corresponding to virtual objects.

[0021] The imaging unit 204 includes two cameras (imaging devices). The two cameras are arranged near the positions of the left and right eyes of the user when wearing the HMD 100 in order to capture an image of a space similar to the space the user normally sees. Images (captured images) captured by the two cameras of a subject (the range in front of the user) are output to the RAM 203. Furthermore, the two cameras in the imaging unit 204 can acquire information on the distance from the two cameras to the subject as distance information by measuring distances with a stereo camera. Note that the number of cameras is not limited to two, and more cameras may be included.

[0022] The display unit 205 is a display that displays, as a three-dimensional image, a composite image of a captured image and a virtual object, an operation menu for various control operations of the HMD 100, and the like. The display unit 205 includes, for example, a liquid crystal panel or an organic EL panel. When the user wears the HMD 100, an organic EL panel is disposed in front of each of the user's eyes. Note that the display unit 205 may also be a device using a semi-transparent half mirror. In this case, for example, the display unit 205 may display an image so that a virtual object appears to be directly superimposed on the real space visible through the half mirror, using a technology generally known as AR (Augmented Reality). The display unit 205 may also display an image of a complete virtual space without using captured images, using a technology generally known as VR (Virtual Reality). A virtual space is a space set based on real space, and the user does not view the real space on the display unit. That is, an image of the virtual space is usually displayed on the display unit, and captured images are not displayed on the display unit while the user is experiencing the virtual space. This allows the user to perceive the virtual space as spreading out before their eyes instead of the real space, giving the user the perception that they are actually in the virtual space.

[0023] The attitude sensor unit 206 acquires attitude (and position) information of the HMD 100. Then, the attitude sensor unit 206 acquires attitude information of the user (the user wearing the HMD 100) that corresponds to the attitude (and position) of the HMD 100. The attitude sensor unit 206 has an inertial measurement unit (IMU) that is configured from an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 206 is used when acquiring information on the user's attitude (attitude information), and the control unit 200 controls the acquisition of information on the user's attitude (attitude information).

[0024] Although the HMD 100 has been described here as an example of a head-mounted display device with a built-in information processing device, the information processing device is not limited to this. The information processing device connected to the head-mounted display device by wire or wirelessly may also be a smartphone, a tablet terminal, or a PC. In this case, part of the processing of the HMD 100 described above may be performed by the information processing device connected to the head-mounted display device by wire or wirelessly.

[0025] <Internal structure of user assistive device> The internal configuration of the user assistive device 110 will be described with reference to FIG. 2(b).

[0026] The user assistive device 110 includes a control unit 210 , a communication unit 211 , a force feedback unit 212 , and a posture sensor unit 213 .

[0027] The control unit 210 controls each component of the user assistive device 110. The control unit 210 controls the force feedback unit 212 in accordance with force feedback parameters acquired from the HMD 100 via the communication unit 211.

[0028] The communication unit 211 performs wireless communication with the HMD 100 (communication unit 201) using Bluetooth. Note that wireless communication may also be performed using Wi-Fi.

[0029] The force feedback unit 212 has an actuator that provides force feedback information, and provides the user with a desired sensation of force, object, weight, movement, or shape change according to the presented force feedback parameters.

[0030] The haptic feedback is not limited to the force sense of mass, and may present a haptic effect based on information about the physical characteristics of a virtual object, which may include at least one of the size, shape, rigidity, weight, thermal characteristics, or texture of the virtual object.

[0031] The attitude sensor unit 213 acquires attitude (and position) information of the user assistance device 110. The attitude sensor unit 213 has an inertial measurement unit (IMU) consisting of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The control unit 210 acquires the attitude (and position) information of the user assistance device 110 and controls transmission of the information to the control unit 200 via the communication unit 211. Note that the user assistance device 110 does not need to have the attitude sensor unit 213. In that case, the control unit 200 of the HMD 100 acquires (estimates) the attitude (and position) information of the user assistance device 110 based on the captured image captured by the imaging unit 204.

[0032] <Process for determining the force-invalid area and force-valid area> A process for determining a haptic invalid area and a haptic valid area in the first embodiment of the present invention will be described with reference to the flowchart of FIG. 3 . The haptic invalid area is an area in which, when the user assist device 110 is in the haptic invalid area, control is not performed to provide a haptic sense to the user as if a mass were generated, and exemplifies a first area in the present invention. The haptic valid area is an area in which, when the user assist device 110 is in the haptic valid area, control is performed to provide a haptic sense to the user as if a mass were generated, and exemplifies a second area in the present invention. This flowchart starts when a virtual object is displayed in the mixed reality space and is executed at least the first time. The timing of execution of this flowchart is not limited to when a virtual object is displayed in the mixed reality space. For example, the timing may be when the user starts up the HMD or when the user starts up an HMD application. The following flowchart may be executed when a mode for providing haptic feedback is selected. The following flowchart may be executed in response to a reset operation performed by the user at a predetermined timing. Note that the following flowchart may be executed not only before a virtual object is placed, but also after the virtual object has been placed. By executing the following flowchart before a virtual object is placed, the virtual object can be placed according to the set force sense effective area and force sense invalid area. Furthermore, by executing the following flowchart after a virtual object has been placed, the force sense effective area and force sense invalid area can be set in accordance with the placement of the virtual object.

[0033] In step S301, the control unit 200 activates the imaging unit 204 to start capturing images of the surrounding real space, acquires the captured images in real time, and proceeds to step S302. Note that if the HMD has already been activated and the imaging unit 204 has already started capturing images, step S301 may be omitted.

[0034] In step S302, the control unit 200 generates an image of a mixed reality space in which a virtual object is superimposed on a real space (real image), and proceeds to step S303. Note that, when force feedback is to be presented for a virtual object displayed in a virtual space, an image in which a virtual object is arranged in a virtual space may be generated based on an image of the real space acquired by the imaging unit 204.

[0035] In step S303, the control unit 200 causes the display unit 205 to display an image of the mixed reality space, and the process proceeds to step S304. Note that, in the case of displaying a virtual object in a virtual space, the control unit 200 may cause the display unit 205 to display an image of the virtual space.

[0036] In step S304, the control unit 200 determines a haptic effective area and a haptic invalid area. An example of determining the haptic effective area and the haptic invalid area is shown in FIGS. 5(a) and 6(a). FIG. 5(a) is a diagram illustrating a scene in which a user wearing an HMD 100 and a virtual object 501 exist in a mixed reality space 500. Based on the coordinates of the virtual object 501, a region 502 surrounding the virtual object 501 is defined as the haptic invalid area, and a region 503 remaining in the mixed reality space 500 excluding the haptic invalid area 502 is defined as the haptic effective area. In the case of a virtual space, the user illustrated in FIGS. 5(a) and 6(a) may be an avatar in the virtual space. FIG. 6(a) is a diagram illustrating a scene in which a user wearing an HMD 100 and a virtual object 601 exist in a mixed reality space 600. Here, a certain area 603 in front of the user based on the position of the HMD 100 is set as the haptic effective area, assuming that it is the area where the user checks the weight of the virtual object with their hand, and the area 602 excluding the haptic effective area 603 from the mixed reality space 600 is set as the haptic ineffective area.

[0037] The haptic effective area may be an area up to a predetermined distance from the user's torso, or an area up to a predetermined distance from a part of the user's body. The haptic effective area may also be moved to follow the movement of the user (HMD). When the haptic effective area is set to a predetermined area from the user's torso, the weight is checked with the hands close to the torso, allowing the weight to be checked in a more stable state than when the hands are far from the torso.

[0038] When a virtual object is placed in a mixed reality space, the control unit 200 may set a haptic-valid area and a haptic-invalid area according to the position of the real object.

[0039] Furthermore, the control unit 200 may set a haptic effective area and a haptic ineffective area according to the position of a virtual object placed in the virtual space. Here, in order to set the haptic effective area and the haptic ineffective area, the target of the virtual object according to the position may be a virtual object with which the user can interact, such as by grasping it, or may be a virtual object with which interaction is not possible.

[0040] Furthermore, the haptic-enabled area and the haptic-disabled area may be set in accordance with the user's operation. For example, the user may draw a line by hand to define the boundary of the area, or the boundary of the area may be defined in accordance with the position or orientation of the controller.

[0041] Furthermore, for example, after a virtual object is placed, a force sense effective area and a force sense invalid area may be set in an area desired by the user. Alternatively, a force sense effective area and a force sense invalid area may be set in an area desired by the user depending on the location of the user, and the virtual object may be placed in accordance with the set force sense effective area and force sense invalid area.

[0042] In step S305, the control unit 200 performs display control so that the haptic valid area and haptic invalid area determined in step S304 can be visually distinguished. That is, the control unit 200 visually notifies the user of the boundary between the haptic valid area and the haptic invalid area via the display unit 205. Note that the processing of step S305 may not be performed. Note that only one of the haptic valid area and the haptic invalid area may be visually displayed.

[0043] Note that the force sense effective area and the force sense invalid area are not limited to being set in the mixed reality space (MR), but may also be set in the virtual space (VR) or the augmented reality space (AR). Furthermore, when switching the screen from the mixed reality space (MR) to the virtual space (VR), the force sense effective area and the force sense invalid area set in the mixed reality space (MR) may be carried over.

[0044] In this way, by setting the haptic feedback effective area and the haptic feedback ineffective area in accordance with this flowchart, the user can prepare to receive haptic feedback.

[0045] <Processing for determining force parameters according to the position where the virtual object is being held> Referring to the flowchart in Fig. 4, a process for determining haptic parameters according to the position where a virtual object is grasped in Example 1 of the present invention will be described. This flowchart illustrates a control means in the present invention. This flowchart starts when a virtual object is displayed in mixed reality space, and is repeatedly executed at regular periodic intervals of, for example, several tens of milliseconds while the virtual object is displayed. Execution of this flowchart allows the user to receive haptic feedback.

[0046] In step S401, the control unit 200 acquires a captured image in real time, and the process proceeds to step S402.

[0047] In step S402, the control unit 200 generates an image of a mixed reality space in which a virtual object is superimposed on a real space (real image), and proceeds to step S403. Note that, when providing force feedback for a virtual object displayed in a virtual space, an image in which a virtual object is arranged in a virtual space may be generated based on an image of the real space acquired by the imaging unit 204.

[0048] In step S403, the control unit 200 causes the display unit 205 to display an image of the mixed reality space, and the process proceeds to step S404. Note that, in the case of displaying a virtual object in a virtual space, the control unit 200 may cause the display unit 205 to display an image of the virtual space.

[0049] In step S404, the control unit 200 detects the position of the user's hand from the captured image acquired by the imaging unit 204 and determines whether the user's hand is gripping the virtual object displayed on the display unit 205. Specifically, the control unit 200 compares the position of the user's wrist joint point with the position where the virtual object is located in the coordinate system of the captured image to determine whether the user is gripping the object. If the control unit 200 determines that the user is gripping the object, the process proceeds to step S405. If the control unit 200 does not determine that the user is gripping the object, the process proceeds to step S413. Whether the user's hand is gripping the virtual object displayed on the display unit 205 may be determined by recognizing a hand gesture made by the user. For example, if the user makes a pinching gesture, such as bringing their thumb and index finger together, near the virtual object, the control unit 200 may determine that the user's hand is gripping the virtual object displayed on the display unit 205. If the user assistive device 110 is a controller, the control unit 200 may determine that the virtual object is being gripped when a predetermined button on the controller is pressed. Note that, in a case where an operation for gripping a virtual object is determined in advance, it may be determined whether or not the virtual object is being gripped based on whether or not an operation for gripping the virtual object is being performed.

[0050] In step S405, the control unit 200 acquires mass information of the virtual object being held from the ROM 203 and determines a maximum haptic parameter according to the mass information. The maximum haptic parameter is a parameter that reproduces the mass of the virtual object if it were in real space, and the heavier the mass information of the virtual object, the larger the maximum haptic parameter, and the lighter the mass information of the virtual object, the smaller the maximum haptic parameter.

[0051] In step S406, the control unit 200 determines whether the position where the virtual object acquired by the imaging unit 204 is held is within the force-sense effective area. If the control unit 200 determines that the position where the virtual object is held is within the force-sense effective area, the process proceeds to step S407, and if the control unit 200 determines that the position where the virtual object is held is within the force-sense invalid area, the process proceeds to step S408.

[0052] Here, the position where a virtual object is grasped will be described with reference to Figures 10(a), 10(b), 10(c), and 10(d). Figure 10(a) illustrates a scene in which a virtual object 1002 is placed in a mixed reality space 1001. Here, the index finger and thumb of a user's hand 1003 are separated, and no grasping hand gesture is being performed.

[0053] 10(b) illustrates a scene in which a user's hand 1003 directly grasps the virtual object 1002 at position 1014 in the mixed reality space 1001, which is closer to the virtual object 1002 than in FIG. 10(a). Here, a scene is also illustrated in which the user directly grasps the virtual object 1002 by making a gesture of bringing the index finger and thumb together. That is, in the mixed reality space 1001, the user's hand 1013 and the virtual object 1002 are in contact at position 1014. At this time, the position at which the virtual object is grasped is position 1014.

[0054] 10(c) illustrates a scene in which a user's hand 1023 in the mixed reality space 1001 makes a gesture of bringing their index finger and thumb together in the same position in the mixed reality space 1001 as in FIG. 10(a), thereby indirectly grasping a virtual object 1002. Here, it is assumed that a position 1024 of the virtual object 1002 has been selected in correspondence with the position where the index finger and thumb are brought together at position 1025 in the mixed reality space 1001. In this case, the position where the virtual object is grasped may be position 1025 instead of position 1024.

[0055] 10(d) illustrates a scene in which a user's hand 1023 is holding a virtual object 1002 by making a gesture of bringing the index finger and thumb together at a position closer to the virtual object 1002 in the mixed reality space 1001 than in FIG. 10(a). Here, the user's hand 1013 and the virtual object 1002 may or may not be in contact with each other in the mixed reality space 1001. In this case, a scene is assumed in which the position at which the virtual object 1002 is held is position 1034, which is the position of the center of gravity (center) of the virtual object 1002.

[0056] As described above, the position where a virtual object is grasped is not limited to the position where the hand and the virtual object come into contact, even when the virtual object is directly grasped by a hand gesture. When a virtual object is indirectly grasped by a hand gesture, the position may be a predetermined position of the hand or a predetermined position of the virtual object. The position where the virtual object is grasped may be a predetermined coordinate position of the virtual object. For example, it may be the coordinate position of the center of gravity of the virtual object or the coordinate position of the center of the virtual object. When an operation to grasp a virtual object is performed using a controller, the position where the virtual object is grasped may be a selected position of the virtual object or a predetermined position of the virtual object. When an operation to grasp a virtual object is performed using a controller, the position where the virtual object is grasped may be a predetermined position of the controller, or, if a virtual ray is emitted from the controller, it may be the point of contact between the ray and the controller.

[0057] 5(b), 5(c), 5(d), and 5(e) and 6(b), 6(c), 6(d), and 6(e) show examples of determining the haptic sense effective area and the haptic sense invalid area. FIG. 5(b) illustrates a scene in which the user is holding a virtual object 501 with one hand, and the holding position is within the haptic sense invalid area 502, so the haptic sense is invalid. FIG. 5(c) illustrates a scene in which the user is holding a virtual object 501 with one hand, and the holding position is within the haptic sense effective area 503, so the haptic sense is valid. FIG. 5(d) illustrates a scene in which the user is holding a virtual object 501 with both hands, and both holding positions are within the haptic sense effective area 503, so the haptic sense is valid. FIG. 5(e) is a diagram illustrating a scene in which the user is holding a virtual object 501 with both hands, and one of the holding positions is within the force-sense invalid area 502, resulting in the force sense being invalid.

[0058] FIG. 6(b) is a diagram illustrating a scene in which a user is holding a virtual object 601 with one hand, and the holding position is within the force-sense invalid area 602, so the force sense is invalid. FIG. 6(c) is a diagram illustrating a scene in which a user is holding a virtual object 601 with one hand, and the holding position is within the force-sense valid area 603, so the force sense is valid. FIG. 6(d) is a diagram illustrating a scene in which a user is holding a virtual object 601 with both hands, and both holding positions are within the force-sense valid area 603, so the force sense is valid. FIG. 6(e) is a diagram illustrating a scene in which a user is holding a virtual object 601 with both hands, and one of the holding positions is within the force-sense invalid area 602, so the force sense is invalid.

[0059] In step S407, the control unit 200 acquires the current haptic parameter being presented to the user from the RAM 202 and determines whether the acquired haptic parameter is less than the maximum haptic parameter determined in step S405. If the haptic parameter is less than the maximum haptic parameter, the process proceeds to step S409, where the haptic parameter is increased in stages. Otherwise, the process proceeds to step S410. If there is no current haptic parameter in the first control, the current haptic parameter is set to zero. Now, with reference to FIG. 7(a), an example of increasing the haptic parameter in stages will be described. In FIG. 7(a), a haptic effective area 502 and a haptic invalid area 503 are set. Also, a scene is assumed in which a virtual object 701 is moved from the haptic invalid area 503 to the position of virtual object 702, virtual object 703, and virtual object 704, in that order. Here, the positions of virtual objects 701 and 702 are within the force-sense invalid area 503, and the positions of virtual objects 703 and 704 are within the force-sense valid area 502. Because virtual object 701 is within the force-sense invalid area 502, no force-sense parameters are presented when the user grasps it. That is, the force-sense parameters are zero. When virtual object 701, which was located in the force-sense invalid area 503, moves to the position of virtual object 702 in the force-sense invalid area 503, no force-sense parameters are presented. Next, when virtual object 702, which was located in the force-sense invalid area 503, moves to the position of virtual object 703 in the force-sense valid area 502, a first-stage force-sense parameter 705 is presented. Subsequently, when virtual object 704 is located in the force-sense valid area 502, a second-stage force-sense parameter 706 is presented. Here, it is assumed that force parameter 706 is larger than force parameter 705. In this manner, the force-sense parameters may be adjusted in stages according to the distance. The force parameter may be adjusted in stages over time.

[0060] In step S409, the control unit 200 increases the haptic parameter by one level, stores the increased haptic parameter in the RAM 202, and then proceeds to step S414.

[0061] In step S410, the control unit 200 notifies the user that the haptic parameters are the maximum, and proceeds to step S414. That is, the notification is made when the generation of the haptic effect up to the maximum haptic parameter is completed. The method of notifying the user that the haptic parameters are the maximum may be, for example, by vibration via a vibration unit (not shown) of the user assist device 110. Note that the notification is not limited to vibration; the control unit 200 may also notify the user by controlling the display unit 205 to display that the haptic parameters are the maximum.

[0062] In step S408, the control unit 200 obtains the current haptic parameter being presented to the user from the RAM 202 and determines whether this haptic parameter is zero. If the haptic parameter is not zero, the control unit 200 proceeds to step S411, and if so, proceeds to step S412.

[0063] In step S411, the control unit 200 lowers the haptic parameter by one level, stores the lowered haptic parameter in the RAM 202, and then proceeds to step S414.

[0064] An example of gradually lowering the haptic parameter will now be described with reference to FIG. 7(b). In FIG. 7(b), a haptic effective area 502 and a haptic invalid area 503 are set. A scene is also assumed in which a virtual object 711 is moved from the haptic effective area 502 to the position of a virtual object 712, a position of a virtual object 713, and a position of a virtual object 714, in this order. Here, the positions of the virtual objects 711 and 712 are within the haptic effective area 502, and the positions of the virtual objects 713 and 714 are within the haptic invalid area 503. Since the virtual object 711 is within the haptic effective area 502, when the user is holding the virtual object 711, a haptic parameter 715 is presented to the user. When the virtual object 711, which was located in the haptic effective area 502, moves to the position of the virtual object 712 in the haptic effective area 502, a haptic parameter 716 having the same magnitude as the haptic parameter 715 is presented. Next, when virtual object 712 located in the haptic invalid area 502 moves to the position of virtual object 713 in the haptic invalid area 503, haptic parameter 717, which is a haptic parameter one step lower than haptic parameter 715 (haptic parameter 716), is presented. Subsequently, when virtual object 712 moves to the position of virtual object 714 in the haptic invalid area 502, no haptic parameter is presented. In other words, the haptic parameter is set to zero. In this way, the haptic parameter may be adjusted in stages in response to entry into the haptic invalid area 502. In other words, by entering the haptic invalid area 502, the haptic parameter presented in the haptic invalid area 502 is gradually reduced, and finally the haptic parameter is set to zero. Note that after entry into the haptic invalid area, the haptic parameter may be adjusted in stages according to the distance, or according to the time.

[0065] In step S412, the control unit 200 sets the haptic parameter to zero and stores the haptic parameter in the RAM 202. Alternatively, the control unit 200 proceeds to step S414 with the haptic parameter set to zero and stored in the RAM 202.

[0066] In step S413, the control unit 200 sets the haptic parameter to zero, stores the haptic parameter in the RAM 202, and proceeds to step S414.

[0067] In step S414, the control unit 200 notifies the user assist device 110 of the determined haptic parameter via the communication unit 201, and ends this process. If the haptic parameter is zero, no haptic sensation is presented through the user assist device 110. If the haptic parameter is not zero, a haptic sensation is presented through the user assist device 110 in accordance with the haptic parameter.

[0068] According to the flowchart of FIG. 4 described above, it is possible to gradually change the strength of the haptic parameter depending on the position in the mixed reality space where the user's hand is gripping the virtual object.

[0069] 4, if the control unit 200 determines that the position where the virtual object is being gripped is within the haptic effective area, the control unit 200 may determine the haptic parameter to be the maximum haptic parameter. In this case, the control unit 200 stores the maximum haptic parameter in the RAM 202 and proceeds to step S414.

[0070] If the control unit 200 determines in step S408 that the haptic parameter is not zero, the control unit 200 may set the haptic parameter to zero without going through any stages.

[0071] When the force parameter is equal to or greater than the threshold value or equal to or less than the threshold value, the parameter may be changed by two or more steps at once, rather than by one step.

[0072] Note that haptic feedback is not limited to being provided based on the haptic active area and haptic inactive area set in a mixed reality (MR) space, but may also be provided based on the haptic active area and haptic inactive area set in a virtual reality (VR) space or an augmented reality (AR) space.

[0073] Example 2 In the first embodiment, a process for determining a haptic parameter according to a position where a virtual object is held has been described with reference to the flowchart of FIG. 4. In the second embodiment, a process for determining a haptic parameter according to the position of the held virtual object, instead of according to the position where the virtual object is held, will be described. With reference to the flowchart of FIG. 8, a process for determining a haptic parameter according to the position of the held virtual object in the second embodiment of the present invention will be described. This flowchart illustrates a control means of the present invention. This flowchart starts when a virtual object is displayed in the mixed reality space, and is repeatedly executed at regular periodic intervals of, for example, several tens of milliseconds while the virtual object is displayed. Note that in the flowchart of FIG. 8, a description of steps S401 to S405 and steps S407 to S414, which overlap with the contents described in FIG. 4, will be omitted.

[0074] 8, in step S806, control unit 200 determines whether the position of the held virtual object is within the haptic effective area. If control unit 200 determines that the position of the held virtual object is within the haptic effective area, the process proceeds to step S407, and if control unit 200 determines that the position of the held virtual object is within the haptic invalid area, the process proceeds to step S408.

[0075] Here, with reference to FIGS. 9(a), 9(b), 9(c), 9(d), and 9(e), a scene in which it is determined whether to enable the force sense will be described. FIG. 9(a) is a diagram illustrating a scene in which a user wearing an HMD 100 and a virtual object 901 exist in a mixed reality space 900. Here, a scene is assumed in which the virtual object 901 is placed on the surface of a virtual object 902. Based on the coordinates of the virtual object 901, an area 903 that coincides with the virtual object 901 is defined as a force sense invalid area, and an area 904 obtained by excluding the force sense invalid area 903 from the mixed reality space 900 is defined as a force sense valid area. In other words, the area of ​​the virtual object 901 when the virtual object 901 is placed by default is defined as the force sense invalid area. FIG. 9(b) illustrates a scene in which a user wearing an HMD 100 existing in the mixed reality space 900 is holding the virtual object 901. Here, a scene is assumed in which the user is merely holding the virtual object but not moving it. That is, the virtual object 901 is within the force-sense invalid area 903 and is in contact with the surface of the virtual object 902 .

[0076] 9(c) is a diagram illustrating a scene in which the haptic sense is enabled when a part of a held virtual object enters the haptic sense effective area. In FIG. 9(c), a scene is assumed in which a user wearing an HMD 100 present in a mixed reality space 900 holds a virtual object 901 and lifts it up. By lifting the virtual object 901, the virtual object 901 moves away from the surface of a virtual object 902, and a part of the held virtual object 901 enters a haptic sense effective area 904. That is, in step S806, the control unit 200 determines that the position of the held virtual object 901 is within the haptic sense effective area. Note that in this case, even if the virtual object 901 is being held, the position of the held virtual object is determined to be within the haptic sense invalid area only when the virtual object is completely contained within the haptic sense invalid area.

[0077] Next, FIGS. 9(d) and 9(e) are diagrams illustrating a scene in which the haptic sense is disabled when a part of a held virtual object enters a haptic invalid area, and the haptic sense is enabled when the entire virtual object is included in the haptic valid area. FIG. 9(d) illustrates a scene in which a user wearing an HMD 100 present in a mixed reality space 900 is holding a virtual object 901 and lifting it up. By lifting the virtual object 901, the virtual object 901 is separated from the surface of a virtual object 902, and a part of the held virtual object 901 enters a haptic valid area 904. That is, in step S806, the control unit 200 determines that the position of the held virtual object 901 is within the haptic invalid area. FIG. 9(e) illustrates a scene in which a user wearing an HMD 100 present in a mixed reality space 900 is holding a virtual object 901 and lifting it up. 9(d), by lifting the virtual object 901, it separates from the surface of the virtual object 902, and the entire virtual object 901 being held is included in the force-sense effective area 904. That is, in step S806, the control unit 200 determines that the position of the held virtual object 901 is within the force-sense effective area. In this case, even if the virtual object 901 is being held, the position of the held virtual object is determined to be within the force-sense effective area only when the virtual object is completely included within the force-sense effective area.

[0078] 9(a), 9(b), 9(c), 9(d), and 9(e), the area of ​​virtual object 901 when virtual object 901 is placed may be set as the force-sense invalid area, and the surface and interior areas of virtual object 902 may also be set as force-sense invalid areas. In this case, when the virtual object is moved so as to invade the interior of virtual object 902, force-sense feedback is not performed, which has the effect of preventing the sense of immersion from being impaired.

[0079] 9(a), the force feedback may be controlled depending on whether the virtual object 901 is in contact with the surface of the virtual object 902. In other words, the force feedback may be performed when the virtual object 901 is grasped and then released from the surface of the virtual object 902.

[0080] Note that instead of placing virtual object 901 on virtual object 902, virtual object 901 may be placed on a real object. In this case, force feedback may be performed depending on whether or not the virtual object is in contact with the surface of the real object. Note that not only the surface of the real object but also the interior area of ​​the real object may be set as the force-sense invalid area.

[0081] In this way, in the second embodiment, a situation is assumed in which the area or volume of a virtual object is taken into consideration when determining whether the virtual object is within the haptic effective area or the haptic invalid area. Note that the area or volume of a real object such as a controller or the user's hand may also be taken into consideration when determining whether the virtual object is within the haptic effective area or the haptic invalid area.

[0082] According to the embodiment described above, it is possible to provide force feedback control with good operability in operations for gripping and releasing a virtual object having mass.

[0083] When multiple virtual objects are placed, a force sense invalid area and a force sense effective area may be set for each virtual object. That is, the force sense invalid area of ​​each virtual object may be set to match the area of ​​each virtual object when placed. When multiple virtual objects are placed, a common force sense invalid area and force sense effective area may be set. When multiple virtual objects are placed, a force sense invalid area for each virtual object, a force sense effective area for each virtual object, a force sense invalid area common to the virtual objects, and a force sense effective area common to the virtual objects may be set arbitrarily.

[0084] Note that haptic feedback is not limited to being provided based on the haptic active area and haptic inactive area set in a mixed reality (MR) space, but may also be provided based on the haptic active area and haptic inactive area set in a virtual reality (VR) space or an augmented reality (AR) space.

[0085] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0086] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0087] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0088] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0089] In addition, in each of the examples described above, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0090] The disclosure of this embodiment includes the following configuration, method, and program.

[0091] [Configuration 1] obtaining means for obtaining information about physical characteristics of a virtual object; a setting means for setting a first area and a second area different from the first area in a mixed reality space generated by placing the virtual object in a real space, or in a virtual space conforming to the real space and in which the virtual object is placed; a control means for controlling not to generate a haptic effect based on the information when a position of the virtual object being held by a user is in the first area in the mixed reality space or the virtual space, and for controlling to generate the haptic effect when the position is in the second area; An information processing device comprising:

[0092] [Configuration 2] The control means controls to generate a haptic effect based on the information when at least a portion of the virtual object enters the second area. 2. The information processing device according to configuration 1,

[0093] [Configuration 3] The control means controls the haptic effect based on the information to be gradually generated over a predetermined time when at least a portion of the virtual object enters the second area. 2. The information processing device according to configuration 1,

[0094] [Configuration 4] The control means controls not to generate a haptic effect based on the information when the position where the virtual object is held is in a first area, and controls to generate the haptic effect when the position where the virtual object is held is in a second area different from the first area. 2. The information processing device according to configuration 1,

[0095] [Configuration 5] The position where the virtual object is held is a contact point between the virtual object and the user's hand, a predetermined position on the virtual object, or a predetermined position on the user's hand. 5. The information processing device according to configuration 4.

[0096] [Configuration 6] When generating a first haptic effect related to weight based on the information, the control means generates the haptic effect so that it gradually becomes heavier over a predetermined period of time. 6. The information processing device according to any one of configurations 3 to 5.

[0097] [Configuration 7] The device further includes a notification unit that notifies the user that the generation of the haptic effect based on the information has been completed when the generation of the haptic effect based on the information has been completed. 6. The information processing device according to any one of configurations 3 to 5.

[0098] [Configuration 8] When a first virtual object is placed on a second virtual object or a real object, the control means generates a haptic effect based on the information when the user grasps the first virtual object and releases it from the surface of the second virtual object or the real object. 2. The information processing device according to configuration 1,

[0099] [Configuration 9] The first area is an area in which the virtual object is placed by default. 9. The information processing device according to any one of configurations 1 to 8.

[0100] [Configuration 10] The first area is provided for each virtual object. 10. The information processing device according to any one of configurations 1 to 9.

[0101] [Configuration 11] The second region is a region extending from the torso of the user to a predetermined distance. 11. The information processing device according to any one of configurations 1 to 10.

[0102] [Configuration 12] The second area is an area up to a predetermined distance from the display device. 12. The information processing device according to any one of configurations 1 to 11.

[0103] [Configuration 13] The physical characteristics include at least one of the size, shape, rigidity, weight, thermal characteristics, or texture of the virtual object. 13. The information processing device according to any one of configurations 1 to 12.

[0104] [Configuration 14] The method further includes transmitting means for transmitting the haptic effect generated by the control means to an actuator. 14. The information processing device according to any one of configurations 1 to 13.

[0105] [Configuration 15] The actuator is attached to the user's hand. 15. The information processing device according to configuration 14.

[0106] [Configuration 16] The actuator is provided in a controller held or worn by the user. 15. The information processing device according to configuration 14.

[0107] [Configuration 17] The controller is a ring-shaped controller. 17. The information processing device according to configuration 16.

[0108] [Configuration 18] The display control means further includes a display control means for controlling the display of the first area and the second area so that they can be visually distinguished from each other. 18. The information processing device according to any one of configurations 1 to 17.

[0109] [Configuration 19] The display control means controls the display means to display the boundary between the first area and the second area. 19. The information processing device according to configuration 18.

[0110] [Configuration 20] The display control means controls the display means to display the first area and the second area in different colors. 19. The information processing device according to configuration 18.

[0111] [Control method] an acquisition step of acquiring information about physical characteristics of the virtual object; a control step of controlling not to generate a haptic effect based on the information when the virtual object held by the user is in a first area, and controlling to generate the haptic effect when the virtual object held by the user is in a second area different from the first area; 1. A method for controlling an information processing device, comprising:

[0112] [program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 20.

[0113] [system] an acquisition device for acquiring information about physical characteristics of a virtual object; a control device that controls not to generate a haptic effect based on the information when the virtual object held by the user is in a first area, and controls to generate the haptic effect when the virtual object held by the user is in a second area different from the first area; An information processing system comprising:

Claims

1. obtaining means for obtaining information about physical characteristics of a virtual object; a setting means for setting a first area and a second area different from the first area in a mixed reality space generated by placing the virtual object in a real space, or in a virtual space conforming to the real space and in which the virtual object is placed; a control means for controlling not to generate a haptic effect based on the information when a position of the virtual object being held by a user is in the first area in the mixed reality space or the virtual space, and for controlling to generate the haptic effect when the position is in the second area; An information processing device comprising:

2. The control means controls to generate a haptic effect based on the information when at least a portion of the virtual object enters the second area.

2. The information processing apparatus according to claim 1, wherein:

3. The control means controls the haptic effect based on the information to be gradually generated over a predetermined time when at least a portion of the virtual object enters the second area.

2. The information processing apparatus according to claim 1, wherein:

4. The control means controls not to generate a haptic effect based on the information when the position where the virtual object is held is in a first area, and controls to generate the haptic effect when the position where the virtual object is held is in a second area different from the first area.

2. The information processing apparatus according to claim 1, wherein:

5. The position where the virtual object is held is a contact point between the virtual object and the user's hand, a predetermined position on the virtual object, or a predetermined position on the user's hand.

5. The information processing apparatus according to claim 4,

6. When generating a first haptic effect related to weight based on the information, the control means generates the haptic effect so that it gradually becomes heavier over a predetermined period of time.

5. The information processing apparatus according to claim 3, wherein the information processing apparatus is a computer.

7. The device further includes a notification unit that notifies the user that the generation of the haptic effect based on the information has been completed when the generation of the haptic effect based on the information has been completed.

5. The information processing apparatus according to claim 3, wherein the information processing apparatus is a computer.

8. When a first virtual object is placed on a second virtual object or a real object, the control means generates a haptic effect based on the information when the user grasps the first virtual object and releases it from the surface of the second virtual object or the real object.

2. The information processing apparatus according to claim 1, wherein:

9. The first area is an area in which the virtual object is placed by default.

2. The information processing apparatus according to claim 1, wherein:

10. The first area is provided for each virtual object.

2. The information processing apparatus according to claim 1, wherein:

11. The second region is a region extending from the torso of the user to a predetermined distance.

2. The information processing apparatus according to claim 1, wherein:

12. The second area is an area up to a predetermined distance from the display device.

2. The information processing apparatus according to claim 1, wherein:

13. The physical characteristics include at least one of the size, shape, rigidity, weight, thermal characteristics, or texture of the virtual object.

2. The information processing apparatus according to claim 1, wherein:

14. The method further includes transmitting means for transmitting the haptic effect generated by the control means to an actuator.

2. The information processing apparatus according to claim 1, wherein:

15. The actuator is attached to the user's hand.

15. The information processing apparatus according to claim 14,

16. The actuator is provided in a controller held or worn by the user.

15. The information processing apparatus according to claim 14,

17. The controller is a ring-shaped controller.

17. The information processing apparatus according to claim 16,

18. The display control means further includes a display control means for controlling the display of the first area and the second area so that they can be visually distinguished from each other.

2. The information processing apparatus according to claim 1, wherein:

19. The display control means controls the display means to display the boundary between the first area and the second area.

19. The information processing apparatus according to claim 18,

20. The display control means controls the display means to display the first area and the second area in different colors.

19. The information processing apparatus according to claim 18,

21. an acquisition step of acquiring information about physical characteristics of the virtual object; a setting step of setting a first area and a second area different from the first area in a mixed reality space generated by placing the virtual object in a real space, or in a virtual space conforming to the real space and in which the virtual object is placed; a control step of controlling not to generate a haptic effect based on the information when a position of the virtual object being held by the user is in the first area in the mixed reality space or the virtual space, and controlling to generate the haptic effect when the position is in the second area; 1. A method for controlling an information processing device, comprising:

22. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

23. an acquisition device for acquiring information about physical characteristics of a virtual object; a setting device that sets a first area and a second area different from the first area in a mixed reality space that is generated so that the virtual object is placed in a real space, or in a virtual space that conforms to the real space and in which the virtual object is placed; a control device that controls not to generate a haptic effect based on the information when a position of the virtual object being held by a user is in the first area in the mixed reality space or the virtual space, and controls to generate the haptic effect when the position is in the second area; An information processing system comprising:

Images