Information processing device

Abstract

This technology provides haptic feedback in aerial haptics that is natural and comfortable for the user. [Solution] The information processing device of the present invention includes an acquisition means for acquiring information on the position and orientation of an operating part, which is a part of the user performing the operation, and a determination means for determining the position and orientation of a virtual operating part corresponding to the operating part in a virtual space presented to the user, based on the position and orientation of the operating part. In the virtual space, a tactile sensation that reproduces the virtual tactile sensation that the virtual operating part receives from a virtual object is provided to the operating part by ultrasound from a tactile device, and the determination means determines the position and orientation of the virtual operating part so that the virtual operating part moves by a distance greater than the movement distance of the operating part.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus that is used together with a tactile device (tactile generation device) that presents a tactile sensation by ultrasonic waves and controls a virtual space presented (displayed) to a user.

Background Art

[0002] Patent Document 1 discloses aerial tactile feedback (aerial haptics) using a continuous distribution of sound energy called a "sound field". By using aerial haptics, a user can obtain a tactile sensation related to a virtual space such as an augmented reality (AR) space or a mixed reality (MR) space without wearing a tactile glove or the like.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in aerial haptics, since the ultrasonic waves emitted from the tactile device that forms the sound field attenuate, the tactile sensation attenuates as the distance from the tactile device increases and eventually disappears. Such a change in the tactile sensation gives a user a sense of discomfort. For example, when a user pinches up a virtual object with a finger, even though the user continues to hold the same virtual object, the tactile sensation changes (attenuates or disappears) as the finger moves away from the tactile device.

[0005] An object of the present invention is to provide a technique that enables tactile presentation without giving a user a sense of discomfort in aerial haptics.

Means for Solving the Problems

[0006] A first aspect of the present invention is an information processing device comprising: an acquisition means for acquiring information on the position and orientation of an operating part, which is a part of a user performing an operation; and a determination means for determining the position and orientation of a virtual operating part corresponding to the operating part in a virtual space presented to the user, based on the position and orientation of the operating part; wherein in the virtual space, a tactile sensation that reproduces the virtual tactile sensation that the virtual operating part receives from a virtual object is provided to the operating part by ultrasound from a tactile device; and the determination means determines the position and orientation of the virtual operating part so that the virtual operating part moves by a distance greater than the movement distance of the operating part.

[0007] A second aspect of the present invention is an information processing method comprising: an acquisition step of acquiring information on the position and orientation of an operating part, which is a part of a user performing an operation; and a determination step of determining the position and orientation of a virtual operating part corresponding to the operating part in a virtual space presented to the user, based on the position and orientation of the operating part; wherein in the virtual space, a tactile sensation that reproduces the virtual tactile sensation that the virtual operating part receives from a virtual object is given to the operating part by ultrasound from a tactile device; and in the determination step, the position and orientation of the virtual operating part are determined such that the virtual operating part moves by a distance greater than the movement distance of the operating part.

[0008] A third aspect of the present invention is a program for causing a computer to function as each of the means of the information processing apparatus. A fourth aspect of the present invention is a computer-readable storage that stores the program for causing a computer to function as each of the means of the information processing apparatus. It is a medium. [Effects of the Invention]

[0009] According to the present invention, it becomes possible to provide tactile feedback in aerial haptics without causing discomfort to the user. [Brief explanation of the drawing]

[0010] [Figure 1]This is a block diagram of the information processing system according to Embodiment 1. [Figure 2] This is an external view of the HMD and haptic device. [Figure 3] This is a schematic diagram of user operations. [Figure 4] This is a flowchart of the information processing system according to Embodiment 1. [Figure 5] This is a diagram illustrating the process of S406. [Figure 6] This is a diagram illustrating the process of S407. [Figure 7] This is an explanatory diagram showing whether or not the S407 process is performed. [Figure 8] This is a block diagram of the information processing system according to Embodiment 2. [Figure 9] This is a flowchart of the information processing system according to Embodiment 2. [Figure 10] This is a diagram illustrating part of the processing in S901 and S902. [Modes for carrying out the invention]

[0011] <Embodiment 1> Embodiment 1 of the present invention will now be described. Figure 1 is a block diagram showing an example of the functional configuration of an information processing system 1 according to Embodiment 1. Figure 2(A) is an external view showing an example of the appearance of the HMD 100, and Figure 2(B) is an external view showing an example of the appearance of the haptic device 150. The information processing system 1 includes a head-mounted display (HMD) 100 as an example of an information processing device to which the present invention is applied, and a haptic device 150. The HMD 100 presents a virtual space such as an augmented reality (AR) space or a mixed reality (MR) space to the user. The haptic device 150 provides tactile feedback using ultrasound (aerial haptics). Note that the present invention is not limited to HMDs and can be applied to various information processing devices. For example, the present invention can be applied to a personal computer connected to an HMD. Instead of an HMD, a holographic display or the like may be used as a device to present a virtual space to the user.

[0012] The HMD100 will be described. Each component of the HMD100 is connected to the internal bus 116, and data transmission and reception within the HMD100 are performed via the internal bus 116.

[0013] The CPU 101 performs various controls by expanding and executing the programs stored in the ROM 102 in the system memory 103. The ROM 102 stores programs executable by the CPU 101 and various setting values of the HMD100. The system memory 103 consists of, for example, a RAM (a volatile memory using semiconductor elements, etc.).

[0014] The imaging optical unit 104 is composed of lenses, an aperture, etc., and performs focus adjustment and exposure adjustment. The sensor 105 is a sensor such as a CMOS that converts the optical image of the subject into an electrical signal to obtain an image, and converts the analog image signal of the subject into digital image data by an A / D conversion circuit and outputs it. Although one sensor is shown in FIG. 1, the HMD100 may have a plurality of sensors. Also, optical information may be converted into an electrical signal using a depth sensor such as LiDAR. The imaging processing unit 106 performs image processing such as noise removal on the digital image data output from the sensor 105.

[0015] The data transfer control unit 107 controls the writing / readout of data to / from the DRAM 108. The DRAM 108 is a memory for temporarily storing data.

[0016] The user operation part acquisition unit 109 detects a user operation part (for example, a hand), which is a part of the user who performs an operation in the virtual space, from the image (image of the real space) after the processing of the imaging processing unit 106, and acquires information on the position and posture (for example, the direction of the hand) of the user operation part. The information on the position and posture of the user operation part is, for example, 3D data representing the position and posture of the user operation part. The 3D data may further represent the shape of the user operation part.

[0017] Based on the information (the position and orientation of the user operation part) acquired by the user operation part acquisition unit 109, the virtual operation part generation unit 110 determines the position and orientation of a virtual operation part, which is a virtual object corresponding to the user operation part in the virtual space, and generates the virtual operation part. The generation of the virtual operation part is, for example, the generation of 3D data representing the position and orientation of the virtual operation part. The 3D data may further represent the shape of the virtual operation part. The virtual operation part generation unit 110 repeatedly updates the position and orientation of the virtual operation part so that the movement of the virtual operation part is linked to the movement of the user operation part.

[0018] The virtual object acquisition unit 111 includes a communication unit (not shown) and acquires virtual object information from the outside of the HMD 100. The virtual object information is information on virtual objects arranged in the virtual space and includes, for example, 3D data representing the position, shape, and orientation of the virtual object and tactile information representing the texture and weight of the virtual object.

[0019] The communication unit 112 communicates with the outside of the HMD 100 by wire or wirelessly. In Embodiment 1, the communication unit 112 of the HMD 100 communicates (transmits and receives electrical signals) with the communication unit 156 of the tactile device 150.

[0020] The tactile limit distance acquisition unit 113 acquires information on the maximum distance from the tactile device 150 at which the tactile device 150 can apply a certain tactile sensation to the user operation part. The distance from the tactile device 150 is, for example, the distance from the ultrasonic output surface of the tactile device 150. In Embodiment 1, it is assumed that the tactile limit distance acquisition unit 113 acquires the information on the maximum distance by calculation, but the tactile limit distance acquisition unit 113 may acquire the information on the maximum distance from the outside of the HMD 100. For example, the tactile limit distance acquisition unit 113 may acquire the information on the maximum distance from the tactile device 150 via the communication unit 112. When acquiring the information on the maximum distance from the tactile device 150, the maximum distance may be a fixed distance determined according to the ability of the tactile device 150 (e.g., the maximum output intensity of ultrasonic waves), or a distance that changes according to the setting change of the tactile device 150. The maximum output intensity of ultrasonic waves may be a predetermined fixed intensity, or an intensity that changes according to the setting change of the tactile device 150.

[0021] The virtual movement ratio adjustment unit 114 adjusts the degree of movement of the virtual operation part generated by the virtual operation part generation unit 110 based on the maximum distance acquired by the tactile limit distance acquisition unit 113.

[0022] The display unit 115 presents (displays) to the user an image of the virtual space, including virtual operation parts generated by the virtual operation part generation unit 110, virtual objects, and virtual objects acquired by the virtual object acquisition unit 111.

[0023] The haptic device 150 will now be described. Each component of the haptic device 150 is connected to the internal bus 159, and data transmission and reception within the haptic device 150 are performed via the internal bus 159.

[0024] The CPU 151 performs various controls by loading the programs stored in the ROM 152 into the system memory 153 and executing them. The ROM 152 is used by the CPU 151 to execute Possible programs and various settings for the haptic device 150 are stored. The system memory 153 consists of, for example, RAM (volatile memory using semiconductor elements).

[0025] The data transfer control unit 154 controls the writing and reading of data to and from the DRAM 155. The DRAM 155 is a memory for temporarily storing data.

[0026] The communication unit 156 communicates with the outside of the haptic device 150 via wired or wireless connection. In Embodiment 1, the communication unit 156 of the haptic device 150 communicates (sends and receives electrical signals) with the communication unit 112 of the HMD 100.

[0027] The ultrasonic control unit 157 controls the waveform of the output ultrasonic waves. By controlling the ultrasonic wave waveform, it is possible to control the tactile sensation provided to the user's operating area.

[0028] The tactile output unit 158 ​​emits ultrasound. For example, the tactile output unit 158 ​​includes multiple ultrasonic speakers arranged in an array, and each of the multiple ultrasonic speakers emits ultrasound. Touch is felt at the point where the sound energy of the ultrasound emitted from the multiple ultrasonic speakers is concentrated.

[0029] In Figure 1, at least some of the components included in the HMD 100 may be provided on an external device of the HMD 100 (e.g., a tactile device 150). Similarly, in Figure 1, at least some of the components included in the tactile device 150 may be provided on an external device of the tactile device 150 (e.g., the HMD 100).

[0030] Figures 3(A) to 3(D) are schematic diagrams illustrating an example of user operation. Figure 3(A) shows the real world. As shown in Figure 3(A), the user wears the HMD 100 and operates with their hand (user hand) 301. Figure 3(B) shows the image of the virtual space 302 displayed by the display unit 115 and visible to the user. In Figure 3(B), a virtual hand 303 corresponding to the user hand 301 and a virtual object 304 are displayed within the virtual space 302 (field of view). When the user moves the user hand 301, the virtual hand 303 moves in the same way as the user hand 301. Figure 3(C) shows the image of the virtual space 302, indicating the state after the user has moved their hand from the state in Figure 3(B). In Figure 3(C), the virtual hand 303 (index finger and thumb of the virtual hand 303) is in contact with the virtual object 304. In the state shown in Figure 3(C), the user's hand 301 is given a tactile sensation that reproduces the virtual tactile sensation that the virtual hand 303 receives from the virtual object 304 in the virtual space 302, via ultrasound from the tactile device 150. Figure 3(D) shows the user's hand 301 in the state shown in Figure 3(C). The index finger 305a of the user's hand 301 is given a tactile sensation that reproduces the virtual tactile sensation that the index finger of the virtual hand 303 receives from the virtual object 304, and the thumb 305b of the user's hand 301 is given a tactile sensation that reproduces the virtual tactile sensation that the thumb of the virtual hand 303 receives from the virtual object 304.

[0031] Because the ultrasound emitted from the tactile device 150 attenuates, the tactile sensation weakens and eventually disappears as the user moves away from the tactile device 150. Such changes in tactile sensation can cause discomfort to the user. By increasing the output intensity (amplitude) of the ultrasound emitted by the tactile device 150 in accordance with the increasing distance from the tactile device 150 to the user's operating area, unwanted changes in tactile sensation (attenuation or disappearance) can be suppressed. However, since there is an upper limit (maximum output intensity) to the output intensity of the ultrasound emitted by the tactile device 150, unwanted changes in tactile sensation (attenuation or disappearance) will occur if the user's operating area moves too far away from the tactile device 150.

[0032] Therefore, in Embodiment 1, the position and orientation of the virtual operating part are determined so that the virtual operating part moves a greater distance than the distance moved by the user operating part. By doing this, even when the user wants to move the virtual operating part a large distance, the user operating part will move the tactile device 1 By preventing the point from deviating too far from 50 and suppressing unnecessary tactile changes, it is possible to provide tactile feedback that feels natural to the user.

[0033] Figure 4 is a flowchart illustrating an example of the operation of the information processing system 1 according to Embodiment 1. The operation of the HMD 100 (S401-S410) is achieved by the CPU 101 of the HMD 100 loading the program stored in the ROM 102 into the system memory 103 and executing it. The operation of the tactile device 150 (S451-S454) is achieved by the CPU 151 of the tactile device 150 loading the program stored in the ROM 152 into the system memory 153 and executing it.

[0034] The operation of HMD100 will now be explained. In S401, the CPU101 determines whether or not the virtual control part has touched the virtual object. For example, the CPU101 determines whether or not the virtual hand 303 has touched the virtual object 304. If the virtual control part has touched the virtual object, the process proceeds to S402; otherwise, it proceeds to S403.

[0035] Various known methods can be used to determine S401. For example, a method can be used to compare the 3D coordinates of a virtual manipulation part in a 3D virtual space with the 3D coordinates of a virtual object, or a method can be used to compare the 2D coordinates of a virtual manipulation part in a 2D image of a virtual space with the 2D coordinates of a virtual object.

[0036] When the CPU 101 makes the determination in S401, it causes the virtual operation part generation unit 110 to generate a virtual operation part and the virtual object acquisition unit 111 to acquire virtual object information. The CPU 101 then obtains information on the position and orientation (and shape) of the virtual operation part from the virtual operation part generation unit 110, and information on the position, orientation, and shape of the virtual object from the virtual object acquisition unit 111, as information necessary for making the determination in S401.

[0037] In S402, the CPU 101 controls the communication unit 112 to output a command to the haptic device 150 to provide haptic feedback (a command to output ultrasonic waves). This command includes, for example, information on haptic intensity and information on the position and orientation of the user's operating part. The haptic intensity is calculated based on, for example, the type of virtual object 304 and how the virtual hand 303 holds and touches the virtual object 304.

[0038] In S403, the CPU 101 controls the communication unit 112 to output a command to the haptic device 150 that does not provide tactile feedback (a command that does not output ultrasonic waves).

[0039] In S404, the CPU 101 determines whether the virtual operation part possesses a virtual object. For example, the CPU 101 determines whether the virtual hand 303 possesses a virtual object 304. If the virtual operation part possesses a virtual object, the process proceeds to S405; otherwise, it proceeds to S410.

[0040] Various known methods can be used to determine S404. For example, methods that use the contact time between the virtual operating part and the virtual object, or methods that perform mechanical calculations based on the position and orientation (and shape) of the virtual operating part and the position, orientation, and shape of the virtual object can be used.

[0041] The CPU 101 may or may not use the information acquired in S401 to make the determination in S404. When making the determination in S404, the CPU 101 may also have the virtual operation part generation unit 110 generate a virtual operation part and the virtual object acquisition unit 111 acquire virtual object information. The CPU 101 may then acquire information on the position and orientation (and shape) of the virtual operation part from the virtual operation part generation unit 110 and information on the position, orientation, and shape of the virtual object from the virtual object acquisition unit 111 as information necessary for making the determination in S404.

[0042] In S405, the CPU 101 determines whether the degree of movement of the virtual manipulation part has been adjusted (i.e., whether the process in S407 has been performed). If the degree of movement has been adjusted, the process proceeds to S408; otherwise, it proceeds to S406.

[0043] In S406, the CPU 101 determines whether the user's operating area has moved away from the tactile device 150. For example, the CPU 101 determines whether the user's hand 301 has moved away from the tactile device 150. If the user's operating area has moved away from the tactile device 150, the process proceeds to S407; otherwise, it proceeds to S410. The direction of movement is not particularly limited, but in Embodiment 1, it is assumed that the CPU 101 determines whether the user's operating area has moved away from the tactile device 150 in a direction perpendicular to the ultrasonic output surface of the tactile device 150.

[0044] An example of the processing in S406 will be explained using Figure 5. The CPU 101 causes the user operation part acquisition unit 109 to acquire the 3D coordinates p(x,y,z) of the user's hand 301 at a predetermined period. Based on the time change of the 3D coordinates p(x,y,z), the CPU 101 acquires the movement vector v(x,y,z) of the user's hand 301. The CPU 101 has in advance stored information on the normal vector n(x,y,z) of the ultrasonic output surface of the tactile device 150. The CPU 101 may also detect the tactile device 150 from the image processed by the imaging processing unit 106 (image in real space) and acquire the normal vector n(x,y,z) information. The CPU 101 calculates the value of the dot product of the normal vector n(x,y,z) and the movement vector v(x,y,z) as the evaluation value N of the movement of the user's hand 301 away from the tactile device 150 in the direction of the normal vector n(x,y,z). The CPU 101 determines whether the evaluation value N is greater than a predetermined threshold. If the evaluation value N is greater than the predetermined threshold, the CPU 101 determines that the user's hand 301 has moved away from the tactile device 150 in the direction of the normal vector n(x,y,z). If the evaluation value N is less than or equal to the predetermined threshold, the CPU 101 determines that the user's hand 301 has not moved away from the tactile device 150 in the direction of the normal vector n(x,y,z).

[0045] In S407, the CPU 101 causes the virtual movement ratio adjustment unit 114 to adjust the degree of movement of the virtual operation part, thereby moving the virtual operation part. The degree of movement of the virtual operation part is, for example, the ratio (linking coefficient) of the movement distance of the virtual operation part to the movement distance of the user operation part.

[0046] An example of the S407 process will be explained using Figures 6(A) to 6(D). Figures 6(A) and 6(C) show the image of the virtual space 302 displayed by the display unit 115 and viewed by the user. Figures 6(B) and 6(D) show the haptic device 150 as seen from the side.

[0047] The default value for the ratio (linking coefficient) of the movement distance of the virtual hand 303 to the movement distance of the user hand 301 is assumed to be a one-dimensional coefficient k (a predetermined ratio). This means that when the user hand 301 moves with movement vector v(x,y,z) in real space, the virtual hand 303 moves with movement vector k·v(x,y,z) in virtual space 302. Figure 6(A) shows k·v(x,y,z).

[0048] Assume that the output intensity of the ultrasonic waves from the tactile device 150 is Pw0, and that the user's hand 301 is at a distance d from the tactile device 150 in the direction of the normal vector n(x,y,z) of the ultrasonic output surface of the tactile device 150. In this case, the tactile intensity Pw(d) received by the user's hand 301 is expressed by the following equation 1. In equation 1, D(d) is the ultrasonic attenuation coefficient corresponding to the distance d. Figure 6(B) shows Pw0, n(x,y,z), d, and Pw(d).

number

[0049] The virtual object acquisition unit 111 acquires the maximum distance (maximum towing distance) h that the virtual object 304 can move in the virtual space 302. The maximum towing distance h is predetermined. Figure 6(C) shows h.

[0050] The virtual object acquisition unit 111 acquires the tactile intensity Pw1 when the virtual hand 303 moves (lifts) the virtual object 304 at the maximum traction distance h. The tactile intensity Pw1 may be interpreted as the virtual touch that the virtual hand 303 receives from the virtual object 304, or as the intensity of the touch that is given to the user hand 301 that reproduces the said virtual touch. The virtual object acquisition unit 111 calculates the tactile intensity Pw1 based on, for example, the type of virtual object 304 and how the virtual hand 303 holds the virtual object 304. Figures 6(C) and 6(D) show Pw1.

[0051] The tactile limit distance acquisition unit 113 acquires (determines) the maximum distance H from the tactile device 150 in the direction of the normal vector n(x,y,z) at which the tactile device 150 can provide the user's hand 301 with a tactile sensation of tactile intensity Pw1, based on the tactile intensity Pw1. For example, the tactile limit distance acquisition unit 113 acquires information on the tactile intensity Pw(d) in equation 1 (information showing the correspondence between tactile intensity and the distance from the tactile device 150) from the tactile device 150 via the communication unit 112. Then, the tactile limit distance acquisition unit 113 acquires the distance d at which tactile intensity Pw(d) = Pw1 as the maximum distance H. The information on tactile intensity Pw(d) is, for example, information on the tactile intensity Pw(d) when the ultrasonic output intensity Pw0 is at its maximum output intensity. Figure 6(D) shows H.

[0052] The tactile limit distance acquisition unit 113 may calculate the maximum distance H based on the tactile intensity Pw1, the ultrasonic output intensity Pw0, and ultrasonic attenuation information (information on the attenuation coefficient D(d); information on the correspondence between the distance from the tactile device 150 and the ultrasonic attenuation coefficient). The output intensity Pw0 is, for example, the maximum output intensity. In that case, the tactile limit distance acquisition unit 113 acquires the ultrasonic output intensity Pw0 information from the tactile device 150 via the communication unit 112. The attenuation information is, for example, predetermined. By substituting the tactile intensity Pw1 into the tactile intensity Pw(d) in Equation 1, the attenuation coefficient D1 corresponding to the tactile intensity Pw1 can be calculated, and the distance d where the attenuation coefficient D(d) = D1 can be acquired as the maximum distance H.

[0053] The virtual movement ratio adjustment unit 114 calculates (determines) the adjusted linkage coefficient r·k based on the maximum distance H and the maximum traction distance h, using the adjustment degree r. The adjustment degree r is a coefficient of the same dimension as the linkage coefficient k, and if the linkage coefficient k is a one-dimensional coefficient, then the adjustment degree r is also a one-dimensional coefficient. The adjusted linkage coefficient r·k is, for example, the ratio of the maximum traction distance h to the distance traveled by the user's hand 301 when the user's hand 301 moves a maximum distance H away from the tactile device 150 in order to move the virtual object 304, and is calculated from the following equation 2. In equation 2, H0 is the distance from the tactile device 150 to the user's hand 301 in the direction of the normal vector n(x,y,z) at time S406. Figure 6(D) shows H0.

number

[0054] The virtual movement ratio adjustment unit 114 uses the following equation 3 to convert the movement vector v(x,y,z) of the user hand 301 to the movement vector v'(x,y,z) of the virtual hand 303 according to the linkage coefficient k before adjustment.

number

[0055] The virtual movement ratio adjustment unit 114 adjusts the direction of the normal vector n(x,y,z) of the ultrasonic output surface. Adjustment is performed only in relation to the adjustment degree r. Therefore, the virtual movement ratio adjustment unit 114 orthogonally projects the movement vector v'(x,y,z) of the virtual hand 303 in the direction of the vector n'(x,y,z) in virtual space, which corresponds to the normal vector n(x,y,z) of the ultrasonic output surface. This makes it possible to obtain components in the movement vector v'(x,y,z) of the virtual hand 303 that correspond to the direction of the normal vector n(x,y,z) and the vector n'(x,y,z). The vector n'(x,y,z) in virtual space is a vector obtained by multiplying the normal vector n(x,y,z) of the ultrasonic output surface by the linkage coefficient k. The vector v'' obtained by orthogonally projecting the movement vector v'(x,y,z) of the virtual hand 303 in the direction of the vector n'(x,y,z) in virtual space can be calculated using the following equation 4.

number

[0056] The virtual movement ratio adjustment unit 114 then calculates the movement vector u using the following equation 5 and moves the virtual hand 303 using the calculated movement vector u. In the following equation 5, 1 is a unit vector.

number

[0057] The difference between performing the S407 process and not performing it will be explained using Figures 7(A) to 7(C). Figure 7(A) shows the tactile device 150 as seen from the side. Figures 7(B) and 7(C) show the image of the virtual space 302 displayed by the display unit 115 and visible to the user. When the S407 process is performed, as shown in Figures 7(A) and 7(B), when the user's hand 301 moves away from the tactile device 150 with a movement vector v(x,y,z), the virtual hand 303 moves with a movement vector u. At this time, in the direction of the normal vector n(x,y,z) of the ultrasonic output surface, the user's hand 301 moves away from the tactile device 150 by a distance H, and in the direction of vector n'(x,y,z), the virtual object 304 is lifted by a distance h. Since the virtual object 304 cannot be lifted any further in the direction of vector n'(x,y,z), the user hand 301 will not move any further away from the haptic device 150. If the process in S407 is not performed, as shown in Figures 7(A) and 7(C), when the user hand 301 moves away from the haptic device 150 in the direction of movement vector v(x,y,z), the virtual hand 303 only moves in the direction of movement vector k·v(x,y,z). Therefore, there is room to lift the virtual object 304, and the user hand 301 moves any further away from the haptic device 150.

[0058] In S408, the CPU 101 determines whether the virtual operation part has released the virtual object. For example, the CPU 101 determines whether the virtual hand 303 has released the virtual object 304. If the virtual operation part has released the virtual object, the process proceeds to S409; otherwise, it proceeds to S410. The determination in S408 can be made in the same way as the determination in S404.

[0059] In S409, the CPU 101 instructs the virtual movement ratio adjustment unit 114 to release the adjustment in S407. As a result, for example, when the user hand 301 moves with movement vector v(x,y,z), the virtual hand 303 moves with movement vector k·v(x,y,z). The virtual movement ratio adjustment unit 114 also adjusts the coordinates of the virtual hand 303 so that when the user hand 301 is at coordinate p(x,y,z), the virtual hand 303 is positioned at coordinate k·p(x,y,z). This adjustment may be instantaneous or stepwise.

[0060] In S410, the CPU 101 determines whether the HMD 100 has received a command to terminate its operation. Commands to terminate the HMD 100 include, for example, a command corresponding to the operation of turning off the power of the HMD 100, or a command corresponding to the operation of stopping the application that provides the virtual space to the user. If the HMD 100 has received a command to terminate its operation, it terminates the operation shown in Figure 4; otherwise, it proceeds to S401.

[0061] According to the above operation, at the first timing when the virtual operating part begins to move the virtual object, the linkage coefficient is increased from a predetermined ratio (k). Then, at the second timing when the contact of the virtual operating part with the virtual object is released, the linkage coefficient is returned to the predetermined ratio. At the same time, the position of the virtual operating part is updated to the position of the virtual operating part if the predetermined ratio were used as the linkage coefficient from the first timing to the second timing. Note that the first and second timings are not limited to these. For example, the first timing may be the timing when the virtual operating part makes contact with the virtual object.

[0062] The operation of the haptic device 150 will now be described. In S451, the CPU 151 determines whether the communication unit 156 has received the instruction S402 or S403 from the HMD 100. If an instruction has been received, the process proceeds to S452; otherwise, it proceeds to S454.

[0063] In S452, the CPU 151 instructs the ultrasonic control unit 157 to control (adjust) the ultrasonic waves output from the tactile output unit 158. For example, when the command S402 is received, the ultrasonic waves with the phase, frequency, and amplitude corresponding to the command are determined to be output from the tactile output unit 158. When the command S403 is received, it is determined that the tactile output unit 158 ​​will not output ultrasonic waves.

[0064] Various known methods can be used to adjust the ultrasound. For example, methods that perform acoustic calculations based on tactile intensity or the position and orientation of the user's operating part can be used. Information on tactile intensity and information on the position and orientation of the user's operating part are included, for example, in the S402 instruction.

[0065] In S453, the CPU 151 causes the tactile output unit 158 ​​to output ultrasonic waves (or not output ultrasonic waves) according to the processing result of S452.

[0066] In S454, the CPU 151 determines whether the haptic device 150 has received a command to terminate its operation. A command to terminate the operation of the haptic device 150 is, for example, a command corresponding to the operation of turning off the power of the haptic device 150. If the haptic device 150 has received a command to terminate its operation, it terminates the operation shown in Figure 4; otherwise, it proceeds to S451.

[0067] As described above, according to Embodiment 1, the position and orientation of the virtual operating part are determined such that the virtual operating part moves a greater distance than the distance moved by the user operating part. By doing so, even when the user wants to move the virtual operating part a large distance, it is possible to suppress the user operating part from moving too far away from the haptic device 150, suppress unnecessary changes in tactile sensation, and provide tactile presentation that does not cause discomfort to the user.

[0068] Furthermore, by adjusting the linkage coefficient (the degree of movement of the virtual control part) only in a specific direction, the user's discomfort when operating the virtual control part can be reduced. When the adjustment of the linkage coefficient is released, the virtual control part is returned to its position as it would be without the adjustment, thereby eliminating the positional shift caused by the adjustment and reducing the user's discomfort after the adjustment of the linkage coefficient is released.

[0069] In addition, in S406, the CPU 101 may calculate the normal vector n(x,y,z) from multiple vectors parallel to the ultrasonic output surface. The CPU 101 may pre-store information on multiple vectors parallel to the ultrasonic output surface, or it may detect the ultrasonic output surface from the image processed by the imaging processing unit 106 (image in real space) and obtain information on multiple vectors parallel to the ultrasonic output surface. If the ultrasonic output surface is curved, a plane similar to the actual ultrasonic output surface may be considered instead of the actual ultrasonic output surface (curved surface) so that the normal vector n(x,y,z) is uniquely determined.

[0070] In S406, if the evaluation value N is too large, it is highly likely that the movement of the user-operated part is not intentional. Therefore, in addition to the lower threshold, an upper threshold may also be used as the threshold for the evaluation value N. The CPU 101 may determine whether the evaluation value N is within a predetermined range (greater than or equal to the lower threshold and less than or equal to the upper threshold). If the evaluation value N is within the predetermined range, the CPU 101 may determine that the user-operated part has moved away from the tactile device 150 in the direction of the normal vector n(x,y,z). If the evaluation value N is not within the predetermined range, the CPU 101 may determine that the user-operated part has not moved away from the tactile device 150 in the direction of the normal vector n(x,y,z).

[0071] In S407, the CPU 101 may acquire (determine) attenuation information (information on the attenuation coefficient D(d)) of the ultrasound output from the tactile device 150 based on the ultrasound frequency. For example, the CPU 101 may acquire ultrasound frequency information from the tactile device 150 via the communication unit 112 and acquire attenuation information based on the frequency information.

[0072] In S407, the CPU 101 may acquire (determine) attenuation information (information on the attenuation coefficient D(d)) of the ultrasonic waves output from the tactile device 150 based on the temperature of the real space. A temperature sensor for detecting the temperature of the user's operating area or the surrounding area (nearby) of the tactile device 150 may be provided on the HMD 100 or on the tactile device 150. If the temperature sensor is provided for tactile information, the CPU 101 may acquire temperature information of the real space from the tactile device 150 via the communication unit 112 and acquire attenuation information based on the temperature information. The CPU 101 may acquire attenuation information based on a combination of information including temperature information, frequency information, or both.

[0073] In S407, the degree of adjustment r does not have to depend on the distance d from the tactile device 150 to the user's hand 301, or it may be a degree of adjustment r(d) that changes according to the distance d. In the case of a degree of adjustment r(d), the interlocking coefficient after adjustment also changes according to the distance d. In this case, the CPU 101 may cause the virtual object acquisition unit 111 to acquire a tactile intensity Pw1, which changes according to the distance d, as the tactile intensity Pw1.

[0074] In S407, the linkage coefficient k may be a 3x3 coefficient matrix, and in the initial state, the degree of movement of the virtual operation part (the ratio of the movement distance of the virtual operation part to the movement distance of the user operation part) may differ depending on the direction.

[0075] In S407, calculations using the linkage coefficient may be used to convert or obtain 2D coordinates (2D movement vectors) instead of 3D coordinates (3D movement vectors). For example, if the linkage coefficient is k, calculations using the linkage coefficient k may convert 3D coordinates w(x,y,z) in real space to 2D coordinates k·w(x,y) in a 2D image in virtual space. 2D coordinates w(x,y) in real space on a plane perpendicular to the direction the user's face is facing may be converted to 2D coordinates k·w(x,y) in a 2D image in virtual space. In this case, the linkage coefficient k may be a 2x2 coefficient matrix, and the degree of movement of the virtual manipulation part may differ depending on the direction.

[0076] The adjusted coupling coefficient initially determined in S407 may be used until the adjustment of the coupling coefficient is released in S409. Alternatively, the process in S407 may be repeated until the adjustment of the coupling coefficient is released in S409, and the adjusted coupling coefficient may be updated when there are changes in parameters such as ultrasonic output intensity or tactile intensity.

[0077] Although the linkage coefficient (the degree of movement of the virtual operating part) is adjusted based on the maximum distance H and the maximum traction distance h, the linkage coefficient may be adjusted based on the maximum distance H, not the maximum traction distance h. For example, another value greater than H-H0 may be used instead of the maximum traction distance h. The linkage coefficient may be adjusted without being limited to a specific direction. Alternatively, the linkage coefficient may be used at all times without adjustment, and the position and attitude of the virtual operating part may be determined based only on the position and attitude (and linkage coefficient) of the user operating part.

[0078] At least some of the processes described as being performed by the HMD100 may be performed by an external device (e.g., a haptic device 150). At least some of the processes described as being performed by the haptic device 150 may be performed by an external device (e.g., the HMD100). In these cases, the HMD100 and the haptic device 150 may output (transmit) information necessary for the external device's processing to the external device. The HMD100 and the haptic device 150 may also acquire (receive) the processing results from the external device.

[0079] <Embodiment 2> Embodiment 2 of the present invention will now be described. Note that the same configurations and processes as in Embodiment 1 will not be described below, and only configurations and processes different from those in Embodiment 1 will be described.

[0080] Figure 8 is a block diagram showing an example of the functional configuration of the information processing system 1 according to Embodiment 2. The display unit 801 has the functions of the display unit 115 in Embodiment 1 (Figure 1) and the function of acquiring information regarding the user's field of view (field of view information). The field of view adjustment unit 802 adjusts the field of view displayed on the display unit 801. In Embodiment 2, the field of view adjustment unit 802 is a zoom change unit that changes the zoom magnification of the image of the virtual space presented to the user. The movement time difference adjustment unit 803 adjusts the delay time between the movement of the virtual operation part and the movement of the user operation part.

[0081] Figure 9 is a flowchart illustrating an example of the operation of the information processing system 1 according to Embodiment 1. The operation of the HMD 100 (S401-S410 and S901-S905) is achieved by the CPU 101 of the HMD 100 loading the program stored in the ROM 102 into the system memory 103 and executing it. The operation of the tactile device 150 (S451-S454) is achieved by the CPU 151 of the tactile device 150 loading the program stored in the ROM 152 into the system memory 153 and executing it. In Figure 9, the same processes (steps) as in Embodiment 1 (Figure 4) are denoted by the same reference numerals as in Embodiment 1 (Figure 4).

[0082] In S901, similar to S407 in Figure 4, the CPU 101 causes the virtual movement ratio adjustment unit 114 to adjust the degree of movement of the virtual operating part, thereby moving the virtual operating part. The method for obtaining the maximum traction distance h differs between S901 and S407.

[0083] In S901, the CPU 101 causes the display unit 801 to acquire the user's field of view information. Various known methods can be used to acquire the field of view information. For example, a method can be used to acquire information for a predetermined range centered on the center of the image displayed by the display unit 801. Since the display unit 801 displays an image of a virtual space, the user's field of view may be interpreted as the perceptual range, which is the range that the user perceives in the virtual space. Based on the field of view information (and the vector n' corresponding to the normal vector n), the CPU 101 acquires (determines) the maximum towing distance h. For example, the CPU 101 determines the maximum towing distance h such that when the virtual object is lifted by the maximum towing distance h, the virtual object remains within the field of view (perceptual range).

[0084] In S902, the CPU 101 causes the field of view adjustment unit 802 to adjust the field of view displayed on the display unit 801 based on the distance d from the tactile device 150 to the user's hand 301 in the direction of the normal vector n. In Embodiment 2, the field of view adjustment unit 802 changes the zoom magnification of the virtual space image presented to the user from a predetermined magnification (default value) so that the perceptual range narrows as the distance d increases and widens as the distance d decreases. For example, the field of view adjustment unit 802 adjusts the zoom magnification according to the ratio of distance d to maximum distance H. When distance d (≦H) approaches the maximum distance H, the field of view adjustment unit 802 increases the zoom magnification so that the perceptual range narrows by zooming in, and when distance d moves away from the maximum distance H, it decreases the zoom magnification so that the perceptual range widens by zooming out. However, zooming out is limited so that the perceptual range does not widen beyond the initial state.

[0085] The processing in S902 allows the user to perceive that they are moving the virtual object by a larger amount, thereby preventing the user's operating area from moving too far away from the haptic device 150. The field of view adjustment unit 802 may adjust the zoom magnification so that the virtual operating area is located outside the perceptual range when the distance d coincides with the maximum distance H.

[0086] Part of the processing in S901 and S902 will be explained using Figures 10(A) and 10(B). Figures 10(A) and 10(B) show the image of the virtual space 302 displayed by the display unit 801 and visible to the user. In Figure 10(A), the maximum traction distance h is determined so that when the virtual object 304 is lifted by the maximum traction distance h, the virtual object 304 is contained within the pre-adjustment (default) perception range 1000. In Figure 10(B), when the virtual hand 303 moves with the movement vector u, the perception range is adjusted from the perception range 1000 in Figure 10(A) to a narrower perception range 1001 by zooming in. As a result, the virtual hand 303 that moved with the movement vector u is located outside the perception range 1001 and is no longer visible to the user. At this time, the virtual object 304 may or may not be visible to the user.

[0087] In S903, CPU101 instructs the field of view adjustment unit 802 to cancel the adjustment performed by S902.

[0088] According to the above operation, at the first timing when the virtual control part begins to move the virtual object, the zoom magnification of the image in the virtual space is increased from a predetermined magnification. Then, at the second timing when the contact of the virtual control part with respect to the virtual object is released, the zoom magnification is returned to the predetermined magnification. The change in zoom magnification may be instantaneous or gradual. Furthermore, as described in Embodiment 1, the first and second timings are not limited to these.

[0089] As described above, according to Embodiment 2, even if the maximum traction distance h cannot be obtained from the outside, the same effect as Embodiment 1 can be obtained by determining the maximum traction distance h based on the user's field of view (visual range) information. In addition, the zoom magnification of the virtual space image is adjusted based on the distance from the tactile device 150 to the user's hand 301. This allows the user to get the feeling that they are moving the virtual object by a larger amount, and consequently, it is possible to suppress the user's operating part from moving too far away from the tactile device 150. Furthermore, by releasing the zoom magnification adjustment when the linkage coefficient adjustment is released, the user's discomfort after the linkage coefficient adjustment is released can be reduced.

[0090] In S901, the CPU 101 may instruct the movement time difference adjustment unit 803 to increase the delay time between the movement of the virtual operation part and the movement of the user operation part from a predetermined delay time (default value). By doing so, the user can be made to feel that the virtual object 304 is heavy. In this case, in S409, the CPU 101 may instruct the movement time difference adjustment unit 803 to return the delay time to the predetermined delay time.

[0091] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0092] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0093] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0094] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0095] This embodiment includes the following configurations, methods, programs, and media. (Composition 1) An acquisition means for acquiring information on the position and orientation of the operating part, which is the part of the user performing the operation, A determination means for determining the position and orientation of a virtual operating part corresponding to the operating part in the virtual space presented to the user, based on the position and orientation of the operating part. It has, In the aforementioned virtual space, a tactile sensation that reproduces the virtual tactile sensation received by the virtual operating part from a virtual object is applied to the operating part by ultrasound from a tactile device. The determination means determines the position and orientation of the virtual operating part such that the virtual operating part moves by a distance greater than the movement distance of the actual operating part. An information processing device characterized by the following: (Configuration 2) The determination means determines the ratio of the movement distance of the virtual operating part to the movement distance of the operating part, based on the maximum distance from the tactile device at which the tactile sensation reproducing the virtual tactile sensation can be provided to the operating part. The information processing device according to configuration 1, characterized by the above. (Composition 3) The determination means determines the ratio which changes according to the distance from the tactile device to the operating part. The information processing apparatus according to configuration 2, characterized in that... (Composition 4) The determination means determines the ratio based on a first maximum distance over which the virtual object can move in the virtual space, and a second maximum distance, which is the maximum distance from the tactile device to which the virtual operating part can provide the operating part with the tactile sensation that reproduces the virtual tactile sensation when the virtual operating part moves the virtual object at the first maximum distance. An information processing device according to configuration 2 or 3, characterized by the above. (Composition 5) The determination means determines the ratio of the first maximum distance to the distance the operating part moves when the operating part moves away from the second maximum distance from the tactile device in order to move the virtual object. The information processing apparatus according to configuration 4, characterized by the features described above. (Composition 6) The determination means determines the second maximum distance based on the virtual touch sensation or the intensity of the touch sensation when the virtual operating part moves the virtual object by the first maximum distance. The information processing apparatus according to configuration 4 or 5, characterized by the above. (Composition 7) The determination means determines the second maximum distance based on the virtual touch or the intensity of the touch when the virtual operating part moves the virtual object at the first maximum distance, and the correspondence between the intensity of the touch provided from the tactile device to the operating part and the distance from the tactile device to the operating part. The information processing device according to configuration 6, characterized by the features described therein. (Composition 8) The determination means determines the second maximum distance based on the virtual touch or the intensity of the touch when the virtual operating part moves the virtual object at the first maximum distance, the output intensity of the ultrasound from the touch device, and the attenuation information of the ultrasound. The information processing device according to configuration 6, characterized by the features described therein. (Composition 9) The output intensity of the ultrasound from the tactile device is the maximum output intensity. The information processing apparatus according to configuration 8, characterized by the above. (Composition 10) The determination means determines the attenuation information based on the frequency of the ultrasound. The information processing apparatus according to configuration 8 or 9, characterized by the above. (Composition 11) The determination means determines the decay information based on the temperature in the real space. The information processing apparatus according to configuration 8 or 9, characterized by the above. (Composition 12) The determination means uses an intensity that changes according to the distance from the tactile device to the operating part as the intensity of the virtual touch or touch when the virtual operating part moves the virtual object at the first maximum distance. An information processing device according to any one of configurations 6 to 11, characterized by the above. (Composition 13) The determination means determines the first maximum distance based on the perceptual range, which is the range perceived by the user in the virtual space. An information processing device according to any one of configurations 4 to 12, characterized by the above. (Composition 14) Zoom changing means for changing the zoom magnification of the image of the virtual space presented to the user, such that the perceptual range narrows in response to an increase in the distance from the tactile device to the operating part, and the perceptual range widens in response to a decrease in the distance from the tactile device to the operating part. It further possesses The information processing device according to configuration 13, characterized by the above. (Composition 15) The determination means increases the ratio of the movement distance of the virtual operating part to the movement distance of the operating part from a predetermined ratio at a first timing, and returns the ratio to the predetermined ratio at a second timing after the first timing. The zoom changing means increases the zoom magnification from a predetermined magnification at the first timing, and returns the zoom magnification to the predetermined magnification at the second timing. The information processing apparatus according to configuration 14, characterized by the features described herein. (Composition 16) The determination means increases the ratio of the movement distance of the virtual operating part to the movement distance of the operating part from a predetermined ratio at a first timing, and returns the ratio to the predetermined ratio at a second timing that occurs after the first timing. The aforementioned information processing device is Delay changing means that, at the first timing, increases the delay between the movement of the virtual operating part and the movement of the operating part from a predetermined delay, and at the second timing, returns the delay to the predetermined delay. It further possesses An information processing device according to any one of configurations 1 to 15, characterized by the above. (Composition 17) The aforementioned determination means is At the first timing, the ratio of the movement distance of the virtual operating part to the movement distance of the operating part is increased from a predetermined ratio. At a second timing that occurs after the first timing, the ratio is returned to the predetermined ratio, and the position of the virtual operation part is updated to the position of the virtual operation part when the predetermined ratio is used as the ratio from the first timing to the second timing. An information processing device according to any one of configurations 1 to 16, characterized by the above. (Composition 18) The first timing is the timing at which the virtual operation part begins to move the virtual object. An information processing device according to any one of configurations 15 to 17, characterized by the above. (Composition 19) The second timing is the timing at which contact between the virtual operating part and the virtual object is released. An information processing device according to any one of configurations 15 to 18, characterized by the above. (Composition 20) The aforementioned determination means is With respect to directions not perpendicular to the ultrasonic output surface of the tactile device, the ratio of the movement distance of the virtual operating part to the movement distance of the operating part is not changed. The ratio of the movement distance of the virtual operating part to the movement distance of the operating part is changed in the direction perpendicular to the ultrasonic output surface of the tactile device. An information processing device according to any one of configurations 1 to 19, characterized by the above. (method) An acquisition step to obtain information on the position and orientation of the operating part, which is the part of the user performing the operation, A determination step in which, based on the position and orientation of the aforementioned operating part, the position and orientation of a virtual operating part corresponding to the aforementioned operating part in the virtual space presented to the user is determined. It has, In the aforementioned virtual space, a tactile sensation that reproduces the virtual tactile sensation received by the virtual operating part from a virtual object is applied to the operating part by ultrasound from a tactile device. In the determination step, the position and orientation of the virtual operating part are determined such that the virtual operating part moves by a distance greater than the movement distance of the actual operating part. An information processing method characterized by the following: (program) A program for causing a computer to function as one of the means of an information processing device described in any of configurations 1 to 20. (medium) A computer-readable storage medium that stores a program for causing the computer to function as one of the means of the information processing device described in any of configurations 1 to 20. [Explanation of Symbols]

[0096] 100: Head-mounted display (HMD) 101: CPU 109: User operation part acquisition unit 110: Virtual operation part generation unit 114: Virtual Movement Ratio Adjustment Unit

Claims

1. An acquisition means for acquiring information on the position and orientation of the operating part, which is the part of the user performing the operation, A determination means for determining the position and orientation of a virtual operating part corresponding to the operating part in the virtual space presented to the user, based on the position and orientation of the operating part. It has, In the aforementioned virtual space, a tactile sensation that reproduces the virtual tactile sensation received by the virtual operating part from a virtual object is applied to the operating part by ultrasound from a tactile device. The determination means determines the position and orientation of the virtual operating part such that the virtual operating part moves by a distance greater than the movement distance of the actual operating part. An information processing device characterized by the following:

2. The determination means determines the ratio of the movement distance of the virtual operating part to the movement distance of the operating part, based on the maximum distance from the tactile device at which the tactile sensation reproducing the virtual tactile sensation can be provided to the operating part. The information processing apparatus according to feature 1.

3. The determination means determines the ratio which changes according to the distance from the tactile device to the operating part. The information processing apparatus according to feature 2.

4. The determination means determines the ratio based on a first maximum distance over which the virtual object can move in the virtual space, and a second maximum distance, which is the maximum distance from the tactile device to which the tactile sensation can be provided to the operating part, reproducing the virtual tactile sensation when the virtual operating part moves the virtual object at the first maximum distance. The information processing apparatus according to feature 2.

5. The determination means determines the ratio of the first maximum distance to the distance the operating part moves when the operating part moves away from the second maximum distance from the tactile device in order to move the virtual object. The information processing apparatus according to feature 4.

6. The determination means determines the second maximum distance based on the virtual touch sensation or the intensity of the touch sensation when the virtual operating part moves the virtual object by the first maximum distance. The information processing apparatus according to feature 4.

7. The determination means determines the second maximum distance based on the virtual touch or the intensity of the touch when the virtual operating part moves the virtual object at the first maximum distance, and the correspondence between the intensity of the touch provided from the tactile device to the operating part and the distance from the tactile device to the operating part. The information processing apparatus according to feature 6.

8. The determination means determines the second maximum distance based on the virtual touch or the intensity of the touch when the virtual operating part moves the virtual object at the first maximum distance, the output intensity of the ultrasound from the touch device, and the attenuation information of the ultrasound. The information processing apparatus according to feature 6.

9. The output intensity of the ultrasound from the tactile device is the maximum output intensity. The information processing apparatus according to feature 8.

10. The determination means determines the attenuation information based on the frequency of the ultrasound. The information processing apparatus according to feature 8.

11. The determination means determines the decay information based on the temperature in the real space. The information processing apparatus according to feature 8.

12. The determination means uses an intensity that changes according to the distance from the tactile device to the operating part as the intensity of the virtual touch or touch when the virtual operating part moves the virtual object at the first maximum distance. The information processing apparatus according to feature 6.

13. The determination means determines the first maximum distance based on the perceptual range, which is the range perceived by the user in the virtual space. The information processing apparatus according to feature 4.

14. Zoom changing means for changing the zoom magnification of the image of the virtual space presented to the user, such that the perceptual range narrows in response to an increase in the distance from the tactile device to the operating part, and the perceptual range widens in response to a decrease in the distance from the tactile device to the operating part. It further possesses The information processing apparatus according to feature 13.

15. The determination means increases the ratio of the movement distance of the virtual operating part to the movement distance of the operating part from a predetermined ratio at a first timing, and returns the ratio to the predetermined ratio at a second timing after the first timing. The zoom changing means increases the zoom magnification from a predetermined magnification at the first timing, and returns the zoom magnification to the predetermined magnification at the second timing. The information processing apparatus according to feature 14.

16. The determination means increases the ratio of the movement distance of the virtual operating part to the movement distance of the operating part from a predetermined ratio at a first timing, and returns the ratio to the predetermined ratio at a second timing after the first timing. The aforementioned information processing device is Delay changing means that, at the first timing, increases the delay between the movement of the virtual operation part and the movement of the operation part from a predetermined delay, and at the second timing, returns the delay to the predetermined delay. It further possesses The information processing apparatus according to feature 1.

17. The aforementioned determination means is At the first timing, the ratio of the movement distance of the virtual operating part to the movement distance of the operating part is increased from a predetermined ratio. At a second timing that occurs after the first timing, the ratio is returned to the predetermined ratio, and the position of the virtual operation part is updated to the position of the virtual operation part when the predetermined ratio is used as the ratio from the first timing to the second timing. The information processing apparatus according to feature 1.

18. The first timing is the timing at which the virtual operation part begins to move the virtual object. The information processing apparatus according to feature 15.

19. The second timing is the timing at which contact between the virtual operating part and the virtual object is released. The information processing apparatus according to feature 15.

20. The aforementioned determination means is With respect to directions not perpendicular to the ultrasonic output surface of the tactile device, the ratio of the movement distance of the virtual operating part to the movement distance of the operating part is not changed. The ratio of the movement distance of the virtual operating part to the movement distance of the operating part is changed in the direction perpendicular to the ultrasonic output surface of the tactile device. The information processing apparatus according to feature 1.

21. An acquisition step to obtain information on the position and orientation of the operating part, which is the part of the user performing the operation, A determination step in which, based on the position and orientation of the aforementioned operating part, the position and orientation of a virtual operating part corresponding to the aforementioned operating part in the virtual space presented to the user are determined. It has, In the aforementioned virtual space, a tactile sensation that reproduces the virtual tactile sensation received by the virtual operating part from a virtual object is applied to the operating part by ultrasound from a tactile device. In the determination step, the position and orientation of the virtual operating part are determined such that the virtual operating part moves by a distance greater than the movement distance of the actual operating part. An information processing method characterized by the following:

22. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 20.

23. A computer-readable storage medium storing a program for causing the computer to function as one of the means of the information processing apparatus described in any one of claims 1 to 20.

Images