Information processing device

Abstract

This technology can suppress the presentation of unnatural tactile sensations. [Solution] The information processing device of the present invention is characterized by having control means for controlling the display of the virtual space so that predetermined contacts corresponding to tactile sensations that cannot be presented to the user by the haptic device used as user contact with a virtual object do not occur in the virtual space.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, and particularly to a technique for presenting a tactile sensation in a virtual space to a user.

Background Art

[0002] Patent Document 1 discloses airborne tactile feedback (airborne haptics) using a continuous distribution of sound energy called a "sound field". By using airborne haptics, a user can obtain a tactile sensation related to a virtual space such as an augmented reality (AR) space or a mixed reality (MR) space without wearing a tactile glove or the like.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in airborne haptics, the tactile sensation that can be presented to the user from the tactile device is limited due to, for example, the upper and lower limits of the intensity of ultrasonic waves that can be output from the tactile device that forms the sound field. In other types of tactile feedback as well, the tactile sensation that can be presented to the user from the tactile device is limited. Therefore, it may not be possible to present a suitable tactile sensation to the user from the tactile device, which may give the user a sense of discomfort. For example, when the user suddenly moves a finger, a tactile sensation deviating from the tactile sensation imagined by the user may be presented to the user, and the user may feel a sense of discomfort.

[0005] An object of the present invention is to provide a technique capable of suppressing the presentation of a tactile sensation that causes discomfort.

Means for Solving the Problems

[0006] A first aspect of the present invention is an information processing apparatus characterized by having control means for controlling the display of a virtual space such that, in the virtual space, a predetermined contact corresponding to a touch that cannot be presented to the user by the haptic device used as a user's contact with a virtual object does not occur.

[0007] A second aspect of the present invention is an information processing method characterized by having a step of controlling the display of a virtual space such that, in the virtual space, a predetermined contact corresponding to a touch that cannot be presented to the user by the haptic device used as a user's contact with a virtual object does not occur.

[0008] A third aspect of the present invention is a program for causing a computer to function as each of the means of the information processing apparatus. A fourth aspect of the present invention is a computer-readable storage medium that stores a program for causing a computer to function as each of the means of the information processing apparatus. [Effects of the Invention]

[0009] According to the present invention, it is possible to suppress the presentation of unnatural tactile sensations. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram of the display system. [Figure 2] This is an external view of the display device. [Figure 3] This is a schematic diagram of the tactile sensation generation area. [Figure 4] This is a flowchart for aerial haptics processing. [Figure 5] This is a schematic diagram illustrating the usage of the display system and the composite image. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described.

[0012] (System Configuration) FIG. 1 is a block diagram showing the configuration of the display system according to the present embodiment. The display system of FIG. 1 includes a display device 100 and a tactile device 200 (tactile generation device).

[0013] (Display Device) As shown in FIG. 1, the display device 100 includes a control unit 101, an information processing unit 102, a communication unit 103, a primary storage unit 104, a secondary storage unit 105, an imaging unit 106, a display unit 107, and an operation unit 108. Further, the display device 100 includes a sensor unit 119, a user part detection unit 120, a motion vector detection unit 121, a user behavior prediction unit 122, and a tactile estimation unit 123. These components are connected to a bus 112, and data transmission and reception between the components are performed via the bus 112.

[0014] The control unit 101 is, for example, a CPU, and controls each part of the display device 100.

[0015] The information processing unit 102 is a processing circuit (arithmetic unit) that performs various information processing (arithmetic processing) such as arithmetic operations, matrix operations, and physical operations. For example, the information processing unit 102 performs arithmetic processing on the image data obtained by the imaging unit 106 (such as arithmetic processing for obtaining various evaluation values related to the image data) and arithmetic processing on the data acquired by the communication unit 103.

[0016] The communication unit 103 is a communication interface that communicates with external devices.

[0017] The primary storage unit 104 is, for example, a DRAM, and temporarily stores data used by the control unit 101 and the information processing unit 102.

[0018] The secondary storage unit 105 is, for example, a flash memory, and stores data used by the control unit 101 and the information processing unit 102, and processing results of the information processing unit 102 (for example, encoded recorded images).

[0019] The imaging unit 106 has, for example, an optical lens, an image sensor, an A / D converter, etc., and converts light from the outside (subject) into digital data (image data).

[0020] The display unit 107 is a display for displaying various images. The display unit 107 may have a touch panel that accepts touch operations with a user's finger, a stylus, etc. The display unit 107 provides a virtual space such as an Augmented Reality (AR) space or a Mixed Reality (MR) space to the user. In the virtual space, the user can touch a virtual object. The position and posture of the user's part in the virtual space are linked to the position and posture of the corresponding part in the real space. As the user's part in the virtual space, an image of the member being imaged may be displayed, or a 3D model (CG) corresponding to the part may be displayed. When the display device 100 is an optical see-through type display device, the user's part in the virtual space may be the user's part itself in the real space. space itself.

[0021] The operation unit 108 has operation members that accept user operations, and has, for example, buttons and dials. The above-described touch panel is also an example of an operation member. When a user operation on these operation members is performed, the control unit 101 performs control corresponding to the user operation. A signal corresponding to a user operation on an external device may be acquired by the communication unit 103, and the control unit 101 may perform control corresponding to the signal.

[0022] The sensor unit 119 has various sensors. For example, the sensor unit 119 has a gyro sensor, an acceleration sensor, and a GPS (Global Positioning System) sensor, and acquires information on the angular velocity and the current position of the display device 100. Information on the angular velocity and the current position of the display device 100 may be acquired by a method different from the method using the gyro sensor, the acceleration sensor, and the GPS sensor.

[0023] The user part detection unit 120 acquires information on the position and orientation (orientation) of a user part (a predetermined part of the user) by detecting it from image data (images of real space) obtained by the imaging unit 106. The method for detecting a user part is not particularly limited; for example, a feature extraction process using a convolutional neural network (CNN) may be used to detect the user part. Various parts can be detected by switching the weights in the CNN. In this embodiment, the user part to be detected is assumed to be the hand, but the detection target is not limited to the hand; it may be fingers, arms, face, etc. One part may be detected, or multiple parts may be detected.

[0024] The motion vector detection unit 121 detects feature points from the image data of the current frame obtained by the imaging unit 106 and detects the motion vector of the feature points from past frames (for example, the frame immediately preceding the current frame) to the current frame. The three-dimensional coordinates of the feature points are obtained, and a three-dimensional motion vector is obtained. The method for detecting the motion vector is not particularly limited, and for example, known methods such as correlation methods or block matching methods may be used to detect the motion vector.

[0025] The user behavior prediction unit 122 predicts the user's future behavior based on the user's behavior up to the present. User behavior may be interpreted as a change in the position and orientation of the user's body part over time. The method for predicting user behavior is not particularly limited. In this embodiment, the user behavior prediction unit 122 predicts the position, orientation, and velocity of the user's body part in a future frame based on information acquired by the user body part detection unit 120, the motion vector detection unit 121, and the sensor unit 119. A future frame is, for example, the frame following the current frame. In this embodiment, the imaging unit 106 is fixed relative to the display device 100. Therefore, the motion vector detected by the motion vector detection unit 121 indicates movement relative to the display device 100. Based on the angular velocity and current position of the display device 100 acquired by the sensor unit 119, the user behavior prediction unit 122 removes the translational and rotational components of the display device 100 from the motion vector detected by the motion vector detection unit 121. This obtains the velocity of the user's body part. The user behavior prediction unit 122 predicts the position, orientation, and velocity of the user part in future frames based on the acquired velocity of the user part and the position and orientation of the user part detected by the user part detection unit 120.

[0026] Furthermore, in order for the user behavior prediction unit 122 to consider more detailed user behavior up to the present, the motion vector detection unit 121 may detect the time change of the motion vector up to the present. The user behavior prediction unit 122 will consider a period of multiple frames in the future. User behavior may be predicted. In this case, the prediction result for the first frame, which is a future frame, may be used to predict the second frame, which is after the first frame.

[0027] The haptic estimation unit 123 predicts the user's contact with a virtual object and estimates the corresponding haptic sensation based on the future user behavior predicted by the user behavior prediction unit 122. If the position and orientation of the virtual object change over time, the behavior of the virtual object is further considered when predicting contact and estimating haptic sensation. The prediction of contact and estimation of haptic sensation are not particularly limited. In this embodiment, the haptic estimation unit 123 predicts the user's contact with a virtual object and estimates the corresponding haptic intensity by performing a collision simulation using physical calculations. The following data and information are used in the collision simulation. Information other than that obtained by the user behavior prediction unit 122 may be stored in advance in the primary storage unit 104 or may be obtained externally by the communication unit 103. • Data of a 3D model of the user's body part • Information on the weight of the user's body part (3D model) • Information acquired by the user behavior prediction unit 122 (information on the position, orientation, and velocity of the user's body parts in future frames) • Data of the 3D model of a virtual object • Information on the weight of virtual objects (3D models) • Information on the behavior of virtual objects

[0028] Figure 2 is an external view of the display device 100. The display device 100 is a video see-through type display device and has a frame, two display units 21a and 21b corresponding to the display unit 107, and two imaging units 24a and 24b corresponding to the imaging unit 106. The frame has a rim 25 to which the display units 21a and 21b are attached to the bottom surface, and temples 26a and 26b attached to both sides of the rim 25. The imaging units 24a and 24b capture images of the area in front of the display device 100 (real space). In the display unit 21a, light from a display element (not shown) is guided to the right eye of the user wearing the display device 100 by an image projection unit 22a and a light guide unit 23a. The light guided to the user's right eye represents, for example, an image with an image captured by the imaging unit 24a (an image of real space) as the background. Similarly, in the display unit 21b, light from a display element (not shown) is guided to the left eye of the user wearing the display device 100 by the image projection unit 22b and the light guide unit 23b. The light guided to the user's left eye represents, for example, an image with an image captured by the imaging unit 24b (an image of real space) as the background. Note that an optical see-through type display device may be used as the display device 100. In the case of an optical see-through type display device, the display device 100 does not need to have imaging units 24a and 24b. The user can see in front of the display device 100 (real space itself, not an image) through the display units 21a and 21b. The user can simultaneously see the image displayed on the display units 21a and 21b and in front of the display device 100.

[0029] (Tactile device) As shown in Figure 1, the tactile device 200 includes a control unit 201, an information processing unit 202, a communication unit 203, a primary storage unit 204, a secondary storage unit 205, an operation unit 208, a tactile generation unit 220, and a drive control unit 221. These components are connected to a bus 212, and data is transmitted and received between the components via the bus 212.

[0030] The control unit 201 is, for example, a CPU, and controls the various parts of the haptic device 200.

[0031] The information processing unit 202 is a processing circuit (arithmetic unit) that performs various information processing (arithmetic operations) such as arithmetic operations, matrix operations, and physical calculations. For example, the information processing unit 202 is connected to the communication unit 203. It performs calculations and other processing on the acquired data.

[0032] The communication unit 203 is a communication interface that communicates with external devices.

[0033] The primary storage unit 204 is, for example, a DRAM, and temporarily stores data used by the control unit 201 and the information processing unit 202.

[0034] The secondary storage unit 205 is, for example, a flash memory, and stores data used by the control unit 201 and the information processing unit 202, as well as the processing results of the information processing unit 202.

[0035] The operation unit 208 has operating members that accept user input, such as buttons and dials. When a user input is performed on these operating members, the control unit 201 performs control corresponding to that user input. A signal corresponding to a user input to an external device may be acquired by the communication unit 203, and the control unit 201 may perform control corresponding to that signal.

[0036] The tactile sensation generating unit 220 outputs ultrasonic waves to provide the user with a tactile sensation.

[0037] The drive control unit 221 drives the haptic generation unit 220 (haptic driver 2201, described later) in response to the haptic signal received by the communication unit 203. The haptic signal includes frequency information, intensity information, position information, and synchronization information. The drive control unit 221 drives the haptic generation unit 220 so that ultrasonic waves with the frequency indicated by the frequency information and the intensity indicated by the intensity information reach the position indicated by the position information, and that a tactile sensation is presented in synchronization with the display device 100.

[0038] Figure 3 is a schematic diagram showing the configuration of the tactile sensation generating unit 220. The tactile sensation generating unit 220 has a plurality of haptic drivers 2201 arranged in two dimensions. The haptic drivers 2201 are, for example, ultrasonic transducers. By individually controlling the vibration of each haptic driver 2201, a desired tactile sensation can be expressed.

[0039] (Communication between display devices and haptic devices) Communication between the display device 100 and the haptic device 200 is performed using communication units 103 and 203. The display device 100 sends a haptic reproduction signal request to the haptic device 200. When the haptic device 200 receives a haptic reproduction signal request from the display device 100, it sends a haptic reproduction signal to the display device 100.

[0040] A tactile reproduction signal request is a signal that requests a tactile reproduction signal. The tactile reproduction signal is a signal that indicates information about the tactile device, for example, a signal that indicates the tactile sensation that can be presented to the user from the tactile device 200. In this embodiment, the tactile reproduction signal indicates frequency range information, intensity range information, and region information for each type of tactile sensation, such as "warm," "cold," "hard," "soft," "smooth," and "rough." The region information indicates a three-dimensional region in which the tactile sensation can be presented to the user. The frequency range information indicates the range (upper and lower limits) of the ultrasonic frequency corresponding to the tactile sensation that can be presented to the user for each position within the region indicated by the region information. The intensity range information indicates the range (upper and lower limits) of the ultrasonic intensity corresponding to the tactile sensation that can be presented to the user for each position within the region indicated by the region information.

[0041] When the display device 100 determines which tactile sensation to present to the user, it transmits a tactile signal to the tactile device 200. Upon receiving the tactile signal, the tactile device 200 presents the tactile sensation to the user by driving each haptic driver 2201 according to the tactile signal.

[0042] (Aerial haptics processing) Figure 4 is a flowchart of the aerial haptics processing performed in the display system according to this embodiment. For example, when the display device 100 and the haptic device 200 are activated, and the display device 100 runs an application that provides a virtual space to the user, and a connection is established between the display device 100 and the haptic device 200, the aerial haptics processing shown in Figure 4 begins. When the aerial haptics processing shown in Figure 4 is performed, the user 10 places the haptic device 200 near the user 10 and wears the display device 100 on their head, as shown in Figure 5(A).

[0043] In step S401, the control unit 101 of the display device 100 transmits a tactile reproduction signal request to the tactile device 200 via the communication unit 103.

[0044] In step S403, the control unit 101 receives a tactile reproduction signal from the tactile device 200 via the communication unit 103 and stores the received tactile reproduction signal in the secondary storage unit 105.

[0045] In step S404, the control unit 101 obtains virtual object information, such as the 3D model, material, mass, and behavior patterns of the virtual object, from a database (not shown) on the cloud via the communication unit 103. Here, the behavior patterns may indicate one or more actions, such as the virtual object approaching the user, the virtual object running away from the user, or the virtual object playing by itself, and may also indicate changes in the virtual object's position, posture, facial expression, etc. In this embodiment, the virtual object information includes information on multiple virtual objects. The virtual object information may be stored in the secondary storage unit 105 in advance.

[0046] In step S405, the control unit 101 selects one or more virtual objects from a plurality of virtual objects indicated by the virtual object information acquired in step S404. The method of selecting virtual objects is not particularly limited. For example, the control unit 101 displays a list of a plurality of virtual objects indicated by the virtual object information on the display unit 107 and selects the specified virtual objects in response to a user operation using the operation unit 108 to specify one or more virtual objects. The control unit 101 may also automatically select virtual objects according to the area of ​​the virtual space that the user is viewing.

[0047] In step S406, the control unit 101 sets an initial state (position, orientation, shape, behavior, light source, etc.) of the virtual object selected in step S405, based on the information of the virtual object acquired in step S404.

[0048] In step S407, the control unit 101 controls the information processing unit 102 based on the state of the set virtual object to generate a virtual object image corresponding to the position and orientation of the user (display device 100). The method for generating the virtual object image is not particularly limited. For example, a 3D model of the virtual object can be placed in the virtual space based on the state of the set virtual object, and a 2D virtual object image can be generated by performing coordinate deformation or geometric deformation according to the position and orientation of the user (display device 100).

[0049] In step S408, the control unit 101 controls the information processing unit 102 to combine (superimpose) the virtual object image generated in step S407 onto the image obtained by the imaging unit 106 (a real-space image captured from real space). This generates a composite image in which the virtual object image is combined with the real-space image. The control unit 101 then displays the composite image on the display unit 107. Figure 5(B) is a schematic diagram showing an example of a composite image. In the composite image in Figure 5(B), the user's hand 11 and the virtual object fish 20 are displayed.

[0050] In step S409, the control unit 101 controls the user body part detection unit 120 to detect the position and orientation of the user's hand from the real-world image.

[0051] In step S410, the control unit 101 controls the motion vector detection unit 121 to detect the motion vectors of feature points from the real-space image.

[0052] In step S411, the control unit 101 controls the user behavior prediction unit 122 to predict the position, orientation, and velocity of the user's hand in future frames. The prediction uses the hand position and orientation detected in step S409, the motion vector detected in step S410, and the angular velocity and current position of the display device 100 acquired by the sensor unit 119.

[0053] In step S412, the control unit 101 updates the state of the virtual object based on the virtual object's behavior pattern.

[0054] In step S413, the control unit 101 controls the tactile estimation unit 123 to predict whether or not the user's hand will make contact with the virtual object, and to estimate the tactile sensation corresponding to that contact. If contact occurs, the process proceeds to step S415; otherwise, it proceeds to step S407. For example, the tactile intensity is estimated by a collision simulation based on the hand position, posture, and velocity predicted in step S411 and the state of the virtual object after updating in step S412. If the tactile intensity is greater than 0, it is determined that contact has occurred, and if the tactile intensity is 0, it is determined that no contact has occurred.

[0055] In step S415, the control unit 101 determines whether the tactile sensation estimated in step S413 is a tactile sensation that can be presented to the user from the tactile device 200. The tactile sensation that can be presented to the user from the tactile device 200 can be determined from the tactile reproduction signal received in step S403. If the estimated tactile sensation can be presented, the process proceeds to step S417; otherwise (if the estimated tactile sensation cannot be presented), the process proceeds to step S416. For example, the control unit 101 determines whether the intensity of the ultrasound corresponding to the estimated tactile sensation is within the intensity range indicated by the intensity range information. If the intensity of the ultrasound is within the intensity range, the control unit 101 determines that the estimated tactile sensation can be presented; otherwise, it determines that the estimated tactile sensation cannot be presented.

[0056] In step S416, the control unit 101 controls the display of the virtual space so that no contact (a predetermined contact) corresponding to tactile sensations that cannot be presented to the user by the haptic device 200 occurs as user contact with the virtual object. The specific method of controlling the display of the virtual space will be described later.

[0057] In step S417, the control unit 101 transmits a tactile signal to the tactile device 200 via the communication unit 103, indicating the estimated tactile sensation (if the control in step S416 was performed, the tactile sensation corresponding to the contact between the virtual object and the user after the control).

[0058] In step S418, the control unit 201 of the haptic device 200 controls the drive control unit 221 based on the haptic signal transmitted in step S417 to drive the haptic generation unit 220 (haptic driver 2201) to present a tactile sensation to the user.

[0059] In step S419, the control unit 101 and the control unit 201 determine whether or not to terminate the aerial haptics processing shown in Figure 4. For example, if the power of the display device 100 or the tactile device 200 is turned off, or if the application providing the virtual space to the user is terminated, the system may terminate. If the connection between the display device 100 and the haptic device 200 is disconnected, the aerial haptics processing ends. If the aerial haptics processing does not end, proceed to step S407.

[0060] A specific example of the processing (control) in step S416 will be explained. Figures 5(C) to 5(H) are schematic diagrams showing an example of a composite image.

[0061] Figure 5(C) shows an example of a composite image when step S416 is not performed. User 10's hand 11 is touching the rear of the virtual object fish 20. In this case, the rough texture of the scales should be presented to user 10 (hand 11), but the tactile device 200 is unable to present such a texture. As a result, the user is presented with a texture that deviates from the rough texture, causing the user to feel uncomfortable.

[0062] In this embodiment, the behavior of the object is controlled so that a situation like that shown in Figure 5(C) does not occur. For example, if the situation shown in Figure 5(C) is predicted, the behavior of the fish 20 is controlled so that the hand 11 does not come into contact with the fish 20. In Figure 5(D), the behavior of the fish 20 is controlled so that it swims away from the hand 11. By doing this, the user 10 (hand 11) is not presented with tactile sensations, and the user's discomfort can be suppressed.

[0063] If the situation shown in Figure 5(C) is predicted, the display in the virtual space may be controlled so that the hand 11 makes contact with the fish 20 that is not the contact shown in Figure 5(C). The tactile device 200 can present the user 10 (hand 11) with the smooth texture of the fish 20's head. In Figure 5(E), the fish 20's behavior is controlled so that its head makes contact with the hand 11. In this way, the user 10 (hand 11) can be presented with a smooth texture that feels natural. Note that the part of the fish 20 that comes into contact with the hand 11 can be any part that corresponds to a texture that the tactile device 200 can present, and is not limited to the fish 20's head.

[0064] The reason the haptic device 200 cannot provide the tactile sensation corresponding to the contact in Figure 5(C) is that the scene in Figure 5(C) is a cold scene. The haptic device 200 can provide the user 10 (hand 11) with the tactile sensation corresponding to the contact in Figure 5(C) if the scene is warm. In Figure 5(F), a heater 30, which is a virtual object separate from the fish 20, is added to the virtual space. By doing this, the scene in the virtual space can be transitioned from a cold scene to a warm scene, and the user 10 (hand 11) can be presented with a tactile sensation that does not feel unnatural.

[0065] The reason the haptic device 200 cannot provide a tactile sensation corresponding to the situation in Figure 5(C) is that the scene in Figure 5(C) is a hot scene. The haptic device 200 can provide the user 10 (hand 11) with a tactile sensation corresponding to the contact in Figure 5(C) if the scene is cold. In Figure 5(G), a rain effect has been added to the display of the virtual space. By doing this, the scene in the virtual space can be transitioned from a hot scene to a cold scene, and the user 10 (hand 11) can be presented with a tactile sensation that does not feel unnatural.

[0066] By controlling the material of the fish 20, the tactile sensation corresponding to the situation in Figure 5(C) may be controlled to a tactile sensation that the tactile device 200 can present. The tactile device 200 can present the tactile sensation of bubbles to the user 10 (hand 11). In Figure 5(H), the material of the fish 20 has been changed to bubbles. By doing so, the user 10 (hand 11) can be presented with the tactile sensation of bubbles that feels natural. When the user touches the fish 20, the tactile sensation of bubbles is presented to the user 10 (hand 11), and then the fish 20 disappears.

[0067] As described above, according to this embodiment, the display of the virtual space is controlled so that predetermined contacts corresponding to tactile sensations that cannot be presented to the user by the haptic device used do not occur as user contact with the virtual object. In this way, the presentation of unnatural tactile sensations can be suppressed.

[0068] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0069] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0070] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0071] For example, the controls shown in Figures 5(D) to 5(H) may be combined and executed as appropriate. Furthermore, while an example has been described in which the intensity of the ultrasonic wave corresponding to the tactile sensation intensity is determined to determine whether the estimated tactile sensation can be presented or not, the system is not limited to this. The system may also determine whether the tactile sensation intensity (contact intensity) is within a predetermined range, whether the magnitude of the motion vector is within a predetermined range, or whether the acceleration detected by the sensor unit 119 (output value of the acceleration sensor) is within a predetermined range. If the system determines the motion vector or acceleration, collision simulation becomes unnecessary, reducing the processing load. In addition, the result of determining whether the estimated tactile sensation can be presented or not can be obtained at an earlier stage.

[0072] We have described an example of predicting future contact between a user and a virtual object, but it is also possible to evaluate contact once it occurs. This way, a tactile sense of discomfort may be presented temporarily, but the continuous presentation of tactile discomfort can be suppressed.

[0073] Although an example of applying the present invention to a display device has been described, the information processing device to which the present invention can be applied is not limited to a display device. For example, the present invention can also be applied to a personal computer connected to a display device. Furthermore, the table information processing device to which the present invention is applied may acquire various data and information (for example, information on the position and posture of user body parts) generated within the display device 100 from an external source.

[0074] Although an example of applying the present invention to aerial haptics has been described, the present invention is also applicable to tactile feedback other than aerial haptics (for example, tactile feedback through vibration or temperature modulation of the part that comes into contact with the user).

[0075] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0076] This embodiment includes the following configurations, methods, programs, and media. (Composition 1) Control means for controlling the display of the virtual space, such that, as user contact with a virtual object, predetermined contacts corresponding to tactile sensations that cannot be presented to the user by the haptic device being used do not occur. An information processing device characterized by having the following features. (Configuration 2) An acquisition means for acquiring information from the aforementioned tactile device, Estimation means for predicting the user's contact with the virtual object and estimating the tactile sensation corresponding to such contact. It further possesses, The control means controls the display of the virtual space based on the information acquired by the acquisition means and the tactile sensation estimated by the estimation means. The information processing device according to configuration 1, characterized by the above. (Composition 3) Second acquisition means for acquiring information on the position and posture of a predetermined part of the user. It further possesses, The estimation means performs contact prediction and tactile estimation based on the time changes in the position and posture of the predetermined part. The information processing apparatus according to configuration 2, characterized in that... (Composition 4) The position and orientation of the predetermined part in the virtual space are linked to the position and orientation of the predetermined part in the real space. The second acquisition means detects the predetermined part from the image captured of the real space. The information processing apparatus according to configuration 3, characterized by the features described herein. (Composition 5) The estimation means performs contact prediction and tactile estimation based on the time changes in the position and orientation of the predetermined part and the time changes in the position and orientation of the virtual object. An information processing apparatus according to configuration 3 or 4, characterized by the above. (Composition 6) A determination means determines whether the estimated touch is a touch that can be presented to the user from the touch device, based on the information acquired by the acquisition means and the touch estimated by the estimation means. It further possesses, The information processing apparatus according to any one of configurations 3 to 5, characterized in that the control means controls the display of the virtual space based on the determination result of the determination means. (Composition 7) The control means controls the behavior of the virtual object so that the user does not make contact with the virtual object when the predetermined contact is predicted. An information processing device according to any one of configurations 2 to 6, characterized by the above. (Composition 8) The control means controls the display of the virtual space so that when the predetermined contact is predicted, the user makes contact with the virtual object that is not the predetermined contact. An information processing device according to any one of configurations 2 to 6, characterized by the above. (Composition 9) Controlling the display of the virtual space includes controlling the behavior of the virtual objects. An information processing apparatus according to any one of configurations 1 to 6 and 8, characterized by the above. (Composition 10) The control of the virtual object's behavior is a control that causes a predetermined part of the virtual object to come into contact with the user. The information processing apparatus according to configuration 9, characterized by the features described therein. (Composition 11) Controlling the display of the virtual space includes controlling the addition of another virtual object to the virtual space. An information processing apparatus according to any one of configurations 1 to 6 and 8 to 10, characterized by the above. (Composition 12) The information processing apparatus according to any one of configurations 1 to 6 and 8 to 11, characterized in that the control of the display of the virtual space includes control of adding effects to the display of the virtual space. (Composition 13) Controlling the display of the virtual space includes controlling the material of the virtual objects. An information processing device according to any one of configurations 1 to 6 and 8 to 12, characterized by the above. (method) A step to control the display of the virtual space so that, as user contact with a virtual object in the virtual space, predetermined contacts corresponding to tactile sensations that cannot be presented to the user by the haptic device being used do not occur. An information processing method characterized by having the following features. (program) A program for causing a computer to function as one of the means of the information processing device described in any of configurations 1 to 13. (medium) A computer-readable storage medium that stores a program for causing the computer to function as one of the means of the information processing device described in any of configurations 1 to 13. [Explanation of symbols]

[0077] 100: Display device 101: Control unit

Claims

1. Control means for controlling the display of the virtual space, such that, as user contact with a virtual object, predetermined contacts corresponding to tactile sensations that cannot be presented to the user by the haptic device being used do not occur. An information processing device characterized by having the following features.

2. An acquisition means for acquiring information from the aforementioned tactile device, Estimation means for predicting the user's contact with the virtual object and estimating the tactile sensation corresponding to such contact. It further possesses, The control means controls the display of the virtual space based on the information acquired by the acquisition means and the tactile sensation estimated by the estimation means. The information processing apparatus according to feature 1.

3. Second acquisition means for acquiring information on the position and posture of a predetermined part of the user. It further possesses, The estimation means performs contact prediction and tactile estimation based on the time changes in the position and posture of the predetermined part. The information processing apparatus according to feature 2.

4. The position and orientation of the predetermined part in the virtual space are linked to the position and orientation of the predetermined part in the real space. The second acquisition means detects the predetermined part from the image captured of the real space. The information processing apparatus according to claim 3.

5. The estimation means performs contact prediction and tactile estimation based on the time changes in the position and orientation of the predetermined part and the time changes in the position and orientation of the virtual object. The information processing apparatus according to claim 3.

6. A determination means determines whether the estimated touch is a touch that can be presented to the user from the touch device, based on the information acquired by the acquisition means and the touch estimated by the estimation means. It further possesses, The information processing apparatus according to claim 3, characterized in that the control means controls the display of the virtual space based on the determination result of the determination means.

7. The control means controls the behavior of the virtual object so that the user does not make contact with the virtual object when the predetermined contact is predicted. The information processing apparatus according to feature 2.

8. The control means controls the display of the virtual space so that when the predetermined contact is predicted, the user makes contact with the virtual object that is not the predetermined contact. The information processing apparatus according to feature 2.

9. Controlling the display of the virtual space includes controlling the behavior of the virtual objects. The information processing apparatus according to feature 1.

10. Controlling the behavior of the virtual object involves a predetermined portion of the virtual object. This is a control that makes contact with the siphon. The information processing apparatus according to feature 9.

11. Controlling the display of the virtual space includes controlling the addition of another virtual object to the virtual space. The information processing apparatus according to feature 1.

12. The information processing apparatus according to claim 1, characterized in that the control of the display of the virtual space includes control of adding effects to the display of the virtual space.

13. Controlling the display of the virtual space includes controlling the material of the virtual objects. The information processing apparatus according to feature 1.

14. A step to control the display of the virtual space so that, as user contact with a virtual object in the virtual space, predetermined contacts corresponding to tactile sensations that cannot be presented to the user by the haptic device being used do not occur. An information processing method characterized by having the following features.

15. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 13.

16. A computer-readable storage medium storing a program for causing the computer to function as one of the means of the information processing apparatus described in any one of claims 1 to 13.

Images