Information processing device

Abstract

This technology can suppress the presentation of unnatural tactile sensations. [Solution] The information processing apparatus of the present invention is characterized by comprising: an acquisition means for acquiring information on the position and posture of each of a plurality of users; and a control means for controlling the display of the virtual space based on a plurality of pieces of information corresponding to each of the plurality of users acquired by the acquisition means, so as to prevent contact with a virtual object in the virtual space from occurring that corresponds to a touch sensation that cannot be simultaneously presented to the two or more users from the haptic device being used.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, and particularly to a technique for presenting a sense of touch in a virtual space to a user.

Background Art

[0002] Patent Document 1 discloses airborne tactile feedback (airborne haptics) using a continuous distribution of sound energy called a "sound field". By using airborne haptics, a user can obtain a sense of touch related to a virtual space such as an Augmented Reality (AR) space or a Mixed Reality (MR) space without wearing a tactile glove or the like.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in airborne haptics, the sound field changes due to interference between the user and the ultrasonic waves from the tactile device. Therefore, when one sound field is shared by User A and User B, the sense of touch obtained by User A changes due to interference between User B and the ultrasonic waves (for example, the sense of touch obtained by User A deviates from the sense of touch imagined by User A), which may give the user a sense of discomfort.

[0005] An object of the present invention is to provide a technique capable of suppressing the presentation of a sense of touch that causes discomfort.

Means for Solving the Problems

[0006] A first aspect of the present invention is an information processing apparatus characterized by comprising: an acquisition means for acquiring information on the position and orientation of each of a plurality of users; and a control means for controlling the display of the virtual space based on a plurality of pieces of information corresponding to each of the plurality of users acquired by the acquisition means, such that contact corresponding to touch that cannot be simultaneously presented to the two or more users from the haptic device being used as contact of two or more users to a virtual object in the virtual space does not occur.

[0007] A second aspect of the present invention is an information processing method characterized by comprising: an acquisition step of acquiring information on the position and orientation of each of a plurality of users; and a control step of controlling the display of the virtual space based on a plurality of pieces of information corresponding to each of the plurality of users acquired in the acquisition step, such that contact corresponding to touch that cannot be simultaneously presented to the two or more users from the haptic device being used as contact of two or more users to a virtual object in the virtual space does not occur.

[0008] A third aspect of the present invention is a program for causing a computer to function as each of the means of the information processing apparatus. A fourth aspect of the present invention is a computer-readable storage medium that stores a program for causing a computer to function as each of the means of the information processing apparatus. [Effects of the Invention]

[0009] According to the present invention, it is possible to suppress the presentation of unnatural tactile sensations. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram of the display system according to Embodiment 1. [Figure 2] This is an external view of the display device. [Figure 3] This is a schematic diagram of the tactile sensation generation area. [Figure 4]This is a flowchart of the aerial haptics processing according to Embodiment 1. [Figure 5] This is a schematic diagram illustrating the usage of the display system and the composite image. [Figure 6] This is a schematic diagram of the process in step S415. [Figure 7] This is a flowchart of the process in step S416. [Figure 8] This is a block diagram of the display system according to Embodiment 2. [Figure 9] This is a flowchart of the aerial haptics processing according to Embodiment 2. [Modes for carrying out the invention]

[0011] <Embodiment 1> Embodiment 1 of the present invention will be described.

[0012] (System Configuration) Figure 1 is a block diagram showing the configuration of a display system according to Embodiment 1. The display system in Figure 1 includes display devices 100, 300 and a tactile device 200 (tactile feedback device). The number of display devices may be more than two. The number of tactile devices may also be more than one.

[0013] (display device) As shown in Figure 1, the display device 100 includes a control unit 101, an information processing unit 102, a communication unit 103, a primary storage unit 104, a secondary storage unit 105, an imaging unit 106, a display unit 107, and an operation unit 108. Furthermore, the display device 100 includes a sensor unit 110, a user body part detection unit 111, a user body part selection unit 112, and a tactile sensation estimation unit 113. These components are connected to a bus 109, and data is transmitted and received between the components via the bus 109. In the description of the display device 100, the user is assumed to be the user of the display device 100. The configuration of the display device 300 is the same as that of the display device 100, so the description of the display device 300 is omitted.

[0014] The control unit 101 is, for example, a CPU, and controls each part of the display device 100.

[0015] The information processing unit 102 is a processing circuit (arithmetic unit) that performs various information processing (arithmetic processing) such as arithmetic operations, matrix operations, and physical operations. For example, the information processing unit 102 performs arithmetic processing on the image data obtained by the imaging unit 106 (such as arithmetic processing for obtaining various evaluation values related to the image data) and arithmetic processing on the data acquired by the communication unit 103.

[0016] The communication unit 103 is a communication interface for communicating with external devices.

[0017] The primary storage unit 104 is, for example, a DRAM, and temporarily stores data used by the control unit 101 and the information processing unit 102.

[0018] The secondary storage unit 105 is, for example, a flash memory, and stores data used by the control unit 101 and the information processing unit 102, the processing results of the information processing unit 102 (for example, encoded recorded images), and the like.

[0019] The imaging unit 106 includes, for example, an optical lens, an imaging device (image sensor), an A / D converter, etc., and converts light from the outside (subject) into digital data (image data).

[0020] The display unit 107 is a display that shows various images. The display unit 107 may have a touch panel that accepts touch operations from the user's finger or stylus. The display unit 107 provides the user with a virtual space such as an Augmented Reality (AR) space or a Mixed Reality (MR) space. In the virtual space, the user can touch virtual objects. The user's position and orientation in the virtual space are linked to the user's position and orientation in the real space. The user in the virtual space may be an image of the user, or a 3D model (CG) corresponding to the user may be displayed. If the display device 100 is an optical see-through type display device, the user in the virtual space may be the user in the real space itself.

[0021] The operation unit 108 has operating members that accept user input, such as buttons and dials. The touch panel mentioned above is also an example of an operating member. When a user input is performed on these operating members, the control unit 101 performs control corresponding to that user input. A signal corresponding to a user input to an external device may be acquired by the communication unit 103, and the control unit 101 may perform control corresponding to that signal.

[0022] The sensor unit 110 has various sensors. For example, the sensor unit 110 has a gyro sensor, an accelerometer, and a GPS (Global Positioning System) sensor to acquire information on the angular velocity and current position of the display device 100. The angular velocity and current position information of the display device 100 may be acquired by a method other than the method using the gyro sensor, accelerometer, and GPS sensor.

[0023] The user part detection unit 111 detects a user from real space and acquires information on the user's position and orientation. The method for detecting a user and the method for acquiring information on the user's position and orientation are not particularly limited, but in Embodiment 1, the user part detection unit 111 detects a user from image data (images of real space) obtained by the imaging unit 106. More specifically, the user part detection unit 111 detects one or more user parts (parts of the user) from the image data obtained by the imaging unit 106 and acquires information on the position and orientation of the detected user parts. The method for detecting user parts (users) is not particularly limited, for example, a convolutional neural network. User body parts may be detected by feature extraction using a Neural Network (CNN). By switching the weights in the CNN, various user body parts can be detected. For example, hands, fingers, arms, and faces can be detected. In Embodiment 1, it is assumed that hands are detected, but the detection target is not limited to hands; it may also be fingers, arms, faces, etc.

[0024] The user body part selection unit 112 selects a user body part detected by the user body part detection unit 111 as the user body part to be presented with tactile sensation. In Embodiment 1, if one user body part is detected by the user body part detection unit 111, that user body part is selected. If multiple user body parts are detected by the user body part detection unit 111, one of those multiple user body parts is selected. In Embodiment 1, one hand is selected, but the selection target is not limited to the hand; it may be fingers, arms, face, etc. Multiple user body parts may be selected. The method of selecting user body parts is not particularly limited. For example, the user body part detection unit 111 stores the information of the detected user body part in the primary storage unit 104, and the display unit 107 displays the detection result of the user body part based on that information. The user performs a user operation using the operation unit 108 to specify the detected user body part, and the user body part selection unit 112 selects the specified user body part in response to the user operation.

[0025] The tactile estimation unit 113 estimates the tactile sensation corresponding to the contact of a user part with a virtual object. Here, the user part is the user part selected by the user part selection unit 112. In Embodiment 1, the tactile estimation unit 123 estimates the tactile intensity corresponding to the contact of a user part with a virtual object by performing interference simulation using physical calculations. The following data and information are used in the interference simulation. Information other than the information acquired by the user part detection unit 111 may be stored in advance in the primary storage unit 104 or may be acquired externally by the communication unit 103. • Data of a 3D model of the user's body part • Information acquired by the user body part detection unit 111 (information on the position and orientation of the user body part) • Data of the 3D model of a virtual object • Information on the position and orientation of virtual objects

[0026] Figure 2 is an external view of the display device 100. The display device 100 is a video see-through type display device and has a frame, two display units 21a and 21b corresponding to the display unit 107, and two imaging units 24a and 24b corresponding to the imaging unit 106. The frame has a rim 25 to which the display units 21a and 21b are attached to the bottom surface, and temples 26a and 26b attached to both sides of the rim 25. The imaging units 24a and 24b capture images of the area in front of the display device 100 (real space). In the display unit 21a, light from a display element (not shown) is guided to the right eye of the user wearing the display device 100 by an image projection unit 22a and a light guide unit 23a. The light guided to the user's right eye represents, for example, an image with an image captured by the imaging unit 24a (an image of real space) as the background. Similarly, in the display unit 21b, light from a display element (not shown) is guided to the left eye of the user wearing the display device 100 by the image projection unit 22b and the light guide unit 23b. The light guided to the user's left eye represents, for example, an image with an image captured by the imaging unit 24b (an image of real space) as the background. Note that an optical see-through type display device may be used as the display device 100. In the case of an optical see-through type display device, the display device 100 does not need to have imaging units 24a and 24b. The user can see in front of the display device 100 (real space itself, not an image) through the display units 21a and 21b. The user can simultaneously see the image displayed on the display units 21a and 21b and in front of the display device 100.

[0027] (Tactile device) As shown in Figure 1, the haptic device 200 includes a control unit 201, an information processing unit 202, a communication unit 203, a primary storage unit 204, a secondary storage unit 205, an operation unit 208, a sensor unit 210, a haptic generation unit 211, and a drive control unit 212. These components are connected to a bus 209, and data is transmitted and received between the components via the bus 209. In describing the haptic device 200, the user is assumed to be the user of the display device 100 and the user of the display device 300.

[0028] The control unit 201 is, for example, a CPU, and controls the various parts of the haptic device 200.

[0029] The information processing unit 202 is a processing circuit (arithmetic unit) that performs various information processing (arithmetic operations) such as arithmetic operations, matrix operations, and physics calculations. For example, the information processing unit 202 performs calculations on data acquired by the communication unit 203.

[0030] The communication unit 203 is a communication interface that communicates with external devices.

[0031] The primary storage unit 204 is, for example, a DRAM, and temporarily stores data used by the control unit 201 and the information processing unit 202.

[0032] The secondary storage unit 205 is, for example, a flash memory, and stores data used by the control unit 201 and the information processing unit 202, as well as the processing results of the information processing unit 202.

[0033] The operation unit 208 has operating members that accept user input, such as buttons and dials. When a user input is performed on these operating members, the control unit 201 performs control corresponding to that user input. A signal corresponding to a user input to an external device may be acquired by the communication unit 203, and the control unit 201 may perform control corresponding to that signal.

[0034] The sensor unit 210 has various sensors. For example, the sensor unit 210 has a gyro sensor, an accelerometer, and a GPS sensor to acquire information on the angular velocity and current position of the tactile device 200. Information on the angular velocity and current position of the tactile device 200 may be acquired in a manner different from that using the gyro sensor, accelerometer, and GPS sensor.

[0035] The tactile sensation generating unit 211 outputs ultrasonic waves to provide the user with a tactile sensation.

[0036] The drive control unit 212 drives the haptic generation unit 211 (haptic driver 213, described later) in response to the haptic signal received by the communication unit 203. The haptic signal includes frequency information, intensity information, position information, and synchronization information. The drive control unit 212 drives the haptic generation unit 211 so that the following haptic presentations are made. In response to the tactile signal received from the display device 100, ultrasonic waves with the frequency indicated by the frequency information and the intensity indicated by the intensity information are delivered to the position indicated by the position information, and a tactile sensation is presented in synchronization with the display on the display device 100. In response to the tactile signal received from the display device 300, ultrasonic waves with the frequency indicated by the frequency information and the intensity indicated by the intensity information are delivered to the position indicated by the position information, and tactile sensations are presented in synchronization with the display on the display device 100.

[0037] Figure 3 is a schematic diagram showing the configuration of the tactile sensation generating unit 211. The tactile sensation generating unit 211 has a plurality of haptic drivers 213 arranged in two dimensions. The haptic drivers 213 are, for example, ultrasonic transducers. By individually controlling the vibration of each haptic driver 213, a desired tactile sensation can be expressed.

[0038] (Communication between display devices and haptic devices) Communication between the display device 100 and the tactile device 200 is performed using communication units 103 and 203. The display device 100 transmits a tactile reproduction signal request to the tactile device 200. Upon receiving the tactile reproduction signal request from the display device 100, the tactile device 200 transmits a tactile reproduction signal to the display device 100. Furthermore, the display device 100 and the tactile device 200 transmit and receive information acquired by the sensor units 110 and 210 to each other.

[0039] A tactile reproduction signal request is a signal that requests a tactile reproduction signal. A tactile reproduction signal is a signal that indicates information about a tactile device, for example, a signal that indicates the tactile sensation that can be presented to the user from the tactile device 200. In Embodiment 1, the tactile reproduction signal indicates frequency range information, intensity range information, and region information for each type of tactile sensation, such as "warm," "cold," "hard," "soft," "smooth," and "rough." The region information indicates a three-dimensional region in which the tactile sensation can be presented to the user. The frequency range information indicates the range (upper and lower limits) of the ultrasonic frequency corresponding to the tactile sensation that can be presented to the user for each position within the region indicated by the region information. The intensity range information indicates the range (upper and lower limits) of the ultrasonic intensity corresponding to the tactile sensation that can be presented to the user for each position within the region indicated by the region information.

[0040] When the display device 100 determines the tactile sensation to present to the user of the display device 100, it transmits a tactile signal to the tactile device 200. When the tactile device 200 receives the tactile signal from the display device 100, it presents the tactile sensation to the user of the display device 100 by driving each haptic driver 213 according to the tactile signal. The display device 100 may communicate with multiple tactile devices, and the tactile sensation determined by the display device 100 may be presented to the user of the display device 100 by ultrasound from the multiple tactile devices.

[0041] Communication between the display device 300 and the haptic device 200 is the same as communication between the display device 100 and the haptic device 200, and the haptic device 200 presents tactile sensations to the user of the display device 300 according to the tactile signals from the display device 300.

[0042] (Aerial haptics processing) Figure 4 is a flowchart of the aerial haptics processing performed in the display system according to Embodiment 1. For example, when the display devices 100 and 300 and the haptic device 200 are activated, and the display devices 100 and 300 execute an application that provides a virtual space to the user, and the connection between the display devices 100 and 300 and the haptic device 200 is established, the aerial haptics processing shown in Figure 4 begins. When the aerial haptics processing shown in Figure 4 is executed, the haptic device 200 is positioned near the user 10 of the display device 100 and the user 30 of the display device 300, as shown in Figure 5(A). The user 10 wears the display device 100 on their head, and the user 30 wears the display device 300 on their head. The processing of the display device 300 is omitted below, but the same processing as that of the display device 100 is performed in the display device 300.

[0043] In step S401, the control unit 101 of the display device 100 transmits a tactile reproduction signal request to the tactile device 200 via the communication unit 103.

[0044] In step S402, the control unit 201 of the tactile device 200 receives a tactile reproduction signal request from the display device 100 via the communication unit 203 and transmits a tactile reproduction signal to the display device 100 via the communication unit 203.

[0045] In step S403, the control unit 101 receives a tactile reproduction signal from the tactile device 200 via the communication unit 103 and stores the received tactile reproduction signal in the secondary storage unit 105.

[0046] In step S404, the control unit 101 obtains virtual object information, such as the 3D model, material, mass, and behavioral patterns of the virtual object, from a database (not shown) on the cloud via the communication unit 103. The virtual object information may be stored in the secondary storage unit 105 in advance.

[0047] In step S405, the control unit 101 selects one or more virtual objects from a plurality of virtual objects indicated by the virtual object information acquired in step S404. The method of selecting virtual objects is not particularly limited. For example, the control unit 101 displays a list of a plurality of virtual objects indicated by the virtual object information on the display unit 107 and selects the specified virtual objects in response to a user operation using the operation unit 108 to specify one or more virtual objects. The control unit 101 may also automatically select virtual objects according to the area of ​​the virtual space that the user 10 is viewing. In Embodiment 1, it is assumed that the same virtual object is selected in both the display device 100 and the display device 300.

[0048] In step S406, the control unit 101 sets an initial state (position, orientation, shape, behavior, light source, etc.) of the virtual object selected in step S405, based on the information of the virtual object acquired in step S404.

[0049] In step S407, the control unit 101 controls the information processing unit 102 based on the state of the set virtual object to generate a virtual object image corresponding to the position and orientation of the user 10 (display device 100). The method for generating the virtual object image is not particularly limited. For example, a 3D model of the virtual object can be placed in the virtual space based on the state of the set virtual object, and a 2D virtual object image can be generated by performing coordinate deformation or geometric deformation according to the position and orientation of the user 10 (display device 100).

[0050] In step S408, the control unit 101 controls the information processing unit 102 to combine (superimpose) the virtual object image generated in step S407 onto the image obtained by the imaging unit 106 (a real-space image captured from real space). This generates a composite image in which the virtual object image is combined with the real-space image. The control unit 101 then displays the composite image on the display unit 107. Figure 5(B) is a schematic diagram showing an example of a composite image. In the composite image of Figure 5(B), the hand 11 of user 10, user 30, and the virtual object dog 50 are displayed.

[0051] In step S409, the control unit 101 controls the user body part detection unit 111 to detect the position and orientation of the user's hand 11 from the real-world image.

[0052] In step S410, the control unit 101 acquires information on the position and orientation of the user's hand 31 from the display device 300 via the communication unit 103. The control unit 101 also acquires information on the position and orientation of the tactile device 200 from the tactile device 200 via the communication unit 103. The method of acquiring this information is not particularly limited. This information does not have to be acquired from outside the display device 100. This information may be generated inside the display device 100, for example, by detecting the hand 31 and the tactile device 200 from a real-space image.

[0053] In step S411, the control unit 101 determines whether or not to perform user body part selection. If user body part selection is to be performed, the process proceeds to step S412; otherwise, it proceeds to step S413. For example, if user 10 instructs the user to perform user body part selection on the menu screen using the operation unit 108, user body part selection is performed. In step S409, an unnecessary hand, such as user 30's hand 31 or user 10's other hand (a hand different from hand 11), may be detected. In such cases, user 10 instructs the user to perform user body part selection in order to select the necessary hand 11.

[0054] In step S412, the control unit 101 controls the user body part selection unit 112 to select the hand 11.

[0055] In step S413, the control unit 101 updates the state of the virtual object based on the virtual object's behavior pattern.

[0056] In step S414, the control unit 101 controls the tactile estimation unit 113 to estimate the tactile sensation corresponding to the user 10 (hand 11)'s contact with the virtual object. For example, the tactile sensation is estimated by interference simulation based on the position and orientation of the hand 11 detected in step S409 and the state of the virtual object after it has been updated in step S413.

[0057] In step S415, the control unit 101 determines whether a contact occurs that corresponds to a tactile sensation that cannot be simultaneously presented to the users 10 and 30 (hands 11 and 31) from the haptic device 200 as contact with the virtual object. This determination considers the position and orientation of hand 11, the position and orientation of hand 31, the position and orientation of the haptic device 200, etc. The following are taken into consideration. Estimated tactile sensations for hand 11 (tactile intensity, ultrasonic intensity, ultrasonic frequency, etc.) and estimated tactile sensations for hand 31 may also be taken into consideration. If a contact corresponding to an unpresentable tactile sensation occurs, proceed to step S416; otherwise, proceed to step S419. Details of this determination will be described later. In Embodiment 1, the display of the virtual space is controlled so that a contact corresponding to an unpresentable tactile sensation does not occur.

[0058] In step S416, the control unit 101 determines whether it can control the display of the virtual space so that contact corresponding to tactile sensations that cannot be presented does not occur. This determination is made using the tactile reproduction signal acquired in step S403 and the virtual object information acquired in step S404. If control is possible, the process proceeds to step S418; otherwise (if contact corresponding to tactile sensations that cannot be presented occurs no matter how the display of the virtual space is controlled), the process proceeds to step S417. Details of this determination will be described later.

[0059] In step S417, the control unit 101 displays a warning on the display unit 107 as a predetermined notification, indicating that a suitable tactile sensation corresponding to contact with the virtual object cannot be obtained. The notification in step S417 is not particularly limited, and for example, the notification may be given by sound instead of display.

[0060] In step S418, the control unit 101 controls the display of the virtual space. In conjunction with the control of the display of the virtual space, the control unit 101 also updates the estimated tactile sensation. In Embodiment 1, the display of the virtual object being touched by the user 10 (hand 11) is controlled. Details of this will be described later. Note that the control of the display of the virtual space is not limited to the display of the virtual object being touched by the hand 11. For example, other virtual objects that affect the tactile sensation (such as a heater) or display effects (such as snow) may be added.

[0061] In step S419, the control unit 101 transmits a tactile signal to the tactile device 200 via the communication unit 103, indicating the estimated tactile sensation for the user 10 (hand 11). If the process proceeds to step S418, the control unit 101 transmits a tactile signal indicating the updated tactile sensation in step S418.

[0062] In step S420, the control unit 201 of the haptic device 200 controls the drive control unit 212 based on the haptic signal transmitted in step S419 to drive the haptic generation unit 211 (haptic driver 213) to present a tactile sensation to the user 10 (hand 11).

[0063] In step S421, the control unit 101 and the control unit 201 determine whether or not to terminate the aerial haptics processing shown in Figure 4. For example, if the power to the display device 100 or the haptic device 200 is turned off, or if the application that provides the virtual space to the user is turned off, and the connection between the display device 100 and the haptic device 200 is disconnected, the aerial haptics processing is terminated. If the aerial haptics processing is not terminated, the process proceeds to step S407.

[0064] A specific example of the process (decision) in step S415 will be explained.

[0065] Figures 6(A) and 6(B) are schematic diagrams showing an example of the processing in step S415. Figures 6(A) and 6(B) show an example of determining whether a predetermined number or more user body parts are included in the space based on the information from the tactile device 200. If a predetermined number or more user body parts are included in the space based on the information from the tactile device 200, it is determined that contact corresponding to an unpresentable touch occurs; otherwise, it is determined that no contact corresponding to an unpresentable touch occurs. The space based on the information from the tactile device 200 may be the space indicated by the information, or it may be the space set (determined) by the control unit 101 based on the information. The space based on the information from the tactile device 200 may be, for example, the space based on the tactile reproduction signal, and the tactile device 2 00 represents a space where tactile sensations can be presented to the user. The predetermined number is not particularly limited, but here we will assume that the predetermined number = 2.

[0066] In Figure 6(A), only the user's hand 11 is included in the space 600 based on the information from the tactile device 200, and the number of user body parts (1) included in the space 600 is less than the predetermined number (2). Therefore, it is determined that no contact corresponding to tactile sensations that cannot be presented will occur.

[0067] In Figure 6(B), the space 600 contains the hand 11 of user 10 and the hand 31 of user 30, and the number of user parts (2) included in the space 600 is equal to the predetermined number (2). Therefore, it is determined that contact corresponding to an unpresented tactile sensation occurs.

[0068] Figures 6(C) and 6(D) are schematic diagrams showing another example of the process in step S415. In Figures 6(C) and 6(D), the control unit 101 sets up a space including the hand 11 and the tactile device 200 based on the position of the user 10's hand 11 and the position of the tactile device 200. The control unit 101 then determines whether or not a predetermined number of user parts are included in the set space. If a predetermined number of user parts are included in the set space, it is determined that contact corresponding to an unpresentable tactile sensation will occur; otherwise, it is determined that no contact corresponding to an unpresentable tactile sensation will occur. The method for setting up the space including the hand 11 and the tactile device 200 is not particularly limited. Here, it is assumed that a cone-shaped space 601 is set up with the output surface of the tactile device 200 (the surface that emits ultrasonic waves) as the base and the position of the hand 11 as the apex. The predetermined number is not particularly limited, but here it is assumed that the predetermined number = 2.

[0069] In Figure 6(C), only the user's hand 11 is contained within space 601, and the number of user body parts (1) contained within space 601 is less than the predetermined number (2). Therefore, it is determined that no contact corresponding to tactile sensations that cannot be presented occurs.

[0070] In Figure 6(B), the space 601 contains the hand 11 of user 10 and the hand 31 of user 30, and the number of user parts (2) included in the space 601 is equal to the predetermined number (2). Therefore, it is determined that contact corresponding to an unpresented tactile sensation occurs.

[0071] The method for determining step S415 is not limited to the method described above. For example, it may be determined whether the estimated tactile intensity for users 10,30 (hands 11,31) is within a predetermined range. The determination of whether the tactile intensity is within a predetermined range may include determining whether the tactile intensity is greater than or equal to threshold T1, determining whether the tactile intensity is less than or equal to threshold T2 (>T1), or both. The predetermined range is, for example, the range corresponding to the intensity range indicated by the tactile reproduction signal (intensity range information).

[0072] An electromagnetic field analysis may be performed based on the position and orientation of hands 11,31 and the position and orientation of the tactile device 200. The results may be compared with the estimated tactile sensations for hands 11,31 to determine whether or not contact corresponding to unpresented tactile sensations occurs. The information (data) necessary for the determination may be output from the communication unit 103 to an external device (e.g., a cloud server), the external device may perform the determination, and the determination result may be obtained from the external device. Information (e.g., a table) showing the correspondence between the information necessary for the determination and the determination result may be prepared in advance, and the determination result corresponding to the current information necessary for the determination may be obtained from the information showing the correspondence.

[0073] Figure 7 is a flowchart of the process (decision) in step S416.

[0074] In step S416-1, the control unit 101, based on the virtual object information acquired in step S404, resolves the contact corresponding to the tactile sensation that cannot be presented. The system determines whether or not a behavioral pattern (such as position or posture) exists. If a behavioral pattern exists, the system proceeds to step S418 in Figure 4; otherwise, it proceeds to step S416-2. A behavioral pattern that resolves contact corresponding to an unpresentable tactile sensation may be interpreted as a behavioral pattern that allows users 10,30 (hands 11,31) to obtain a suitable tactile sensation. In step S416-1, for example, without changing the contact position of hands 11,31 with respect to the virtual object, the system searches for a behavioral pattern that resolves contact corresponding to an unpresentable tactile sensation by making the same determination as in step S415 for each of the multiple behavioral patterns.

[0075] In step S416-2, the control unit 101 determines whether the number of virtual objects can be increased. For example, the control unit 101 determines whether there is space to add virtual objects to the space based on the tactile reproduction signal acquired in step S403 (the space in which the tactile device 200 can present tactile sensations to the user). If the number of virtual objects can be increased (there is space), the process proceeds to step S416-3; otherwise, the process proceeds to step S416-4.

[0076] In step S416-3, the control unit 101 determines whether increasing the number of virtual objects can eliminate contact corresponding to tactile sensations that cannot be presented. If it can, the process proceeds to step S416-5; otherwise, it proceeds to step S416-4. In step S416-3, for example, the same determination as in step S415 is repeatedly made while changing the number of virtual objects without changing the contact positions of the hands 11, 31 with respect to the virtual objects. This searches for a state (number, position, orientation, etc.) in which contact corresponding to tactile sensations that cannot be presented is eliminated.

[0077] In step S416-4, the control unit 101 determines whether changing the size of the virtual object (enlarging or shrinking) will eliminate contact corresponding to tactile sensations that cannot be presented. If it can be eliminated, the process proceeds to step S416-5; otherwise, it proceeds to step S417 in Figure 4. Here, the case of changing the size of the virtual object without increasing the number of virtual objects may be considered, or the case of changing the size of the virtual object (shrinking) and increasing the number of virtual objects may be considered. In step S416-4, for example, the same determination as in step S415 is repeatedly made while changing the size of the virtual object without changing the contact position of the hands 11, 31 with respect to the virtual object. This searches for a state (size, position, orientation, number, etc. of the virtual object) that eliminates contact corresponding to tactile sensations that cannot be presented.

[0078] If increasing the number of virtual objects can eliminate touches corresponding to tactile sensations that cannot be presented, in step S416-5, the control unit 101 assigns each of the increased virtual objects to each user. Then, the process proceeds to step S418 in Figure 4. For example, if one virtual object is increased to two virtual objects, the two virtual objects are assigned to two users 10 and 30, respectively. Through the control in step S418 in Figure 4, each virtual object is displayed so that each user can touch the virtual object assigned to them. At this time, each user's display device may show only the virtual object assigned to them, or it may show all of the increased virtual objects.

[0079] A specific example of the process in step S418 is explained below. In step S418, the state of the virtual object is controlled according to the state discovered in the search in step S416 (size, position, orientation, number, etc. of the virtual object). In addition, the estimated tactile sensation is updated accordingly.

[0080] In Figure 5(C), user 10 is touching the dog 50's face with hand 11, and user 30 is touching the dog 50's tail with hand 31. In this case, hand 11 and hand 31 are separated, and hand 11 Since this provides a suitable tactile sensation for hand 31, the process proceeds from step S415 to step S419 in Figure 4. Then, the composite image shown in Figure 5(D) (an image showing user 10 touching dog 50's face with hand 11 and user 30 touching dog 50's tail with hand 31) is displayed on the display unit 107. At this time, user 10 can obtain the tactile sensation of touching dog 50's face, and user 30 can obtain the tactile sensation of touching dog 50's tail.

[0081] In Figure 5(E), user 10 is touching dog 50's face with hand 11, and user 30 is touching dog 50's forelegs with hand 31. In this case, hands 11, 31, and the tactile device 200 are close together, and it is not possible to present suitable tactile sensations to hands 11 and 31, so the process proceeds from step S415 to step S416 in Figure 4. Now, suppose that in step S416, exploration discovers that if dog 50 lies down, suitable tactile sensations can be presented to hands 11 and 31. In this case, in step S418, the state of dog 50 is controlled so that dog 50 lies down. When dog 50 lies down, user 10 moves hand 11 to continue touching dog 50's face, and user 30 moves hand 31 to continue touching dog 50's forelegs. Then, the composite image shown in Figure 5(F) (an image in which user 10 is touching dog 50's face with hand 11, and user 30 is touching dog 50's foreleg with hand 31) is displayed on the display unit 107. At this time, user 10 can obtain the sensation of touching dog 50's face, and user 30 can obtain the sensation of touching dog 50's foreleg.

[0082] In the state shown in Figure 5(E), if a behavioral pattern of dog 50 that can provide suitable tactile sensations to hands 11 and 31 cannot be found through searching, and if it is discovered through searching that dogs 51 and 52 can be displayed instead of dog 50, then suitable tactile sensations can be provided to hands 11 and 31, then in step S418, the state of dog 50 is controlled so that dogs 51 and 52 (both of which are the same dogs as dog 50) are displayed instead of dog 50. With dogs 51 and 52 displayed, user 10 moves hand 11 to continue touching dog 50's face, and user 30 moves hand 31 to continue touching dog 50's forelegs. Then, the composite image shown in Figure 5(G) (an image of user 10 touching dog 51's face with hand 11, and user 30 touching dog 52's forelegs with hand 31) is displayed on the display unit 107. At this time, user 10 can obtain the tactile sensation of touching dog 51's (dog 50's) face, and user 30 can obtain the tactile sensation of touching dog 52's (dog 50's) forelegs.

[0083] According to Embodiment 1, the display of the virtual space is controlled so that, as contact between two or more users of a virtual object in the virtual space, contact corresponding to tactile sensations that cannot be simultaneously presented to two or more users from the haptic device being used does not occur. In this way, the presentation of unnatural tactile sensations can be suppressed.

[0084] <Embodiment 2> Embodiment 2 of the present invention will now be described. In Embodiment 2, information on the user's future position and orientation is obtained as information on the user's position and orientation. In the following, the same configurations and processes as in Embodiment 1 will be omitted from the description, and configurations and processes that differ from those in Embodiment 1 will be described.

[0085] Figure 8 is a block diagram showing the configuration of the display system according to Embodiment 2. In the display system of Figure 8, the display device 100 has the plurality of components shown in Figure 1, a motion vector detection unit 114, and a user behavior prediction unit 115.

[0086] The motion vector detection unit 114 detects feature points from the image data of the current frame obtained by the imaging unit 106, and detects the motion vector of the feature points from past frames (for example, the frame immediately preceding the current frame) to the current frame. The 3D coordinates of the feature points are obtained, and the 3D motion vector is obtained. The method for detecting the motion vector is not particularly limited. Alternatively, motion vectors may be detected using known methods such as correlation methods or block matching methods.

[0087] The user behavior prediction unit 115 predicts the user's future behavior based on the user's behavior up to the present. User behavior may be interpreted as a change in the position and orientation of the user's body part over time. The method for predicting user behavior is not particularly limited. In Embodiment 2, the user behavior prediction unit 115 predicts the position, orientation, and velocity of the user's body part in a future frame based on information acquired by the user body part detection unit 111, the motion vector detection unit 114, and the sensor unit 110. The future frame is, for example, the frame following the current frame. In Embodiment 2, the imaging unit 106 is fixed relative to the display device 100. Therefore, the motion vector detected by the motion vector detection unit 114 indicates movement relative to the display device 100. Based on the angular velocity and current position of the display device 100 acquired by the sensor unit 110, the user behavior prediction unit 115 removes the translational and rotational components of the display device 100 from the motion vector detected by the motion vector detection unit 114. This allows the velocity of the user's body part to be obtained. The user behavior prediction unit 115 predicts the position, orientation, and velocity of the user part in future frames based on the acquired velocity of the user part and the position and orientation of the user part detected by the user part detection unit 111.

[0088] Furthermore, in order for the user behavior prediction unit 115 to consider more detailed user behavior up to the present, the motion vector detection unit 114 may detect the time change of the motion vector up to the present. The user behavior prediction unit 115 may also predict user behavior over a period of multiple future frames. In this case, the prediction result for the first frame, which is a future frame, may be used to predict the second frame, which is after the first frame.

[0089] Figure 9 is a flowchart of the aerial haptics processing performed in the display system according to Embodiment 2.

[0090] The process in steps S401 to S409 is performed in the same manner as in Embodiment 1 (Figure 4).

[0091] In step S921, the control unit 101 controls the motion vector detection unit 114 to detect the motion vectors of feature points from the image obtained by the imaging unit 106 (a real-space image captured of real space).

[0092] In step S922, the control unit 101 controls the user behavior prediction unit 115 to predict the position, orientation, and velocity of the user's hand 11 in future frames. The prediction uses the position and orientation of the hand 11 detected in step S409, the motion vector detected in step S921, and the angular velocity and current position of the display device 100 acquired by the sensor unit 110.

[0093] The processing in steps S410 to S420 is performed in the same manner as in Embodiment 1 (Figure 4). However, the prediction result from step S922 is used as the position and orientation of the user 10's hand 11.

[0094] According to Embodiment 2, the same processing as in Embodiment 1 is performed using information on the user's future position and orientation. This makes it possible to prevent contact that corresponds to tactile sensations that cannot be presented.

[0095] Note that the various controls described above are performed by a single piece of hardware (e.g., a processor or circuit). This is acceptable, or it is not. Multiple pieces of hardware (for example, multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0096] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0097] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0098] Although an example of applying the present invention to a display device has been described, the information processing device to which the present invention can be applied is not limited to a display device. For example, the present invention can also be applied to a personal computer connected to a display device. Furthermore, the table information processing device to which the present invention is applied may acquire various data and information (for example, information on the position and posture of user body parts) generated within the display device 100 from an external source.

[0099] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0100] This embodiment includes the following configurations, methods, programs, and media. (Composition 1) For each of the multiple users, an acquisition means for acquiring the location and orientation information of the user, Based on the multiple pieces of information corresponding to each of the multiple users acquired by the acquisition means, a control means controls the display of the virtual space so that contact corresponding to tactile sensations that cannot be simultaneously presented to the two or more users from the haptic device being used does not occur as contact of two or more users to a virtual object in the virtual space. An information processing device characterized by having the following features. (Configuration 2) The user's position and orientation in the virtual space are linked to the user's position and orientation in the real space. The acquisition means detects the first user among the plurality of users from the real space and acquires the position and orientation information of the first user. The information processing device according to configuration 1, characterized by the above. (Composition 3) The acquisition means detects the first user from the image captured of the real space. The information processing apparatus according to configuration 2, characterized in that... (Composition 4) The acquisition means acquires location and orientation information of a second user, who is different from the first user, from outside the information processing device. An information processing device according to configuration 2 or 3, characterized by the above. (Composition 5) The aforementioned information regarding the user's position and posture is information regarding the position and posture of the user's body parts. The acquisition means detects multiple body parts of the first user, The aforementioned information processing device is Selection means for selecting at least one of the plurality of parts of the first user. It further possesses, The control means uses the position and posture information of the part selected by the selection means as the position and posture information of the first user. An information processing device according to any one of configurations 2 to 4, characterized by the above. (Composition 6) The aforementioned information regarding the user's position and posture is information regarding the position and posture of the user's body parts. The control means sets a space that includes the body parts of the first user among the plurality of users and the haptic device, and controls the display of the virtual space so that no more than a predetermined number of user body parts are included in the set space. An information processing device according to any one of configurations 1 to 5, characterized by the above. (Composition 7) The aforementioned information regarding the user's position and posture is information regarding the position and posture of the user's body parts. The acquisition means further acquires information from the tactile device, The control means controls the display of the virtual space so that no more than a predetermined number of user body parts are included in the space based on the information of the tactile device. An information processing device according to any one of configurations 1 to 5, characterized by the above. (Composition 8) The space based on the information from the haptic device is a space in which the haptic device can present tactile sensations to the user. The information processing apparatus according to configuration 7, characterized by the features described above. (Composition 9) The acquisition means acquires information about the user's future location and orientation as information about the user's location and orientation. An information processing device according to any one of configurations 1 to 8, characterized by the above. (Composition 10) The control means controls at least one of the position and orientation of the virtual object. An information processing device according to any one of configurations 1 to 9, characterized by the above. (Composition 11) The control means controls the number of virtual objects. An information processing device according to any one of configurations 1 to 10, characterized by the above. (Composition 12) The control means controls the size of the virtual object. An information processing device according to any one of configurations 1 to 11, characterized by the above. (Composition 13) The control means controls the display of the virtual space to issue a predetermined notification if it is not possible to control the display of the virtual space so that contact corresponding to the tactile sensation that cannot be presented does not occur. An information processing device according to any one of configurations 1 to 12, characterized by the above. (method) For each of the multiple users, an acquisition step is made to acquire information on the location and orientation of the user, Information processing method characterized by having a control step of controlling the display of the virtual space based on a plurality of pieces of information corresponding to each of the plurality of users acquired in the acquisition step, so as to prevent contact with a virtual object in the virtual space from occurring that corresponds to a touch that cannot be simultaneously presented to the two or more users from the haptic device being used. (program) A program for causing a computer to function as one of the means of the information processing device described in any of configurations 1 to 13. (medium) A computer-readable storage medium that stores a program for causing the computer to function as one of the means of the information processing device described in any of configurations 1 to 13. [Explanation of symbols]

[0101] 100: Display device 101: Control unit

Claims

1. For each of the multiple users, an acquisition means for acquiring the location and orientation information of the user, Based on the multiple pieces of information corresponding to each of the multiple users acquired by the acquisition means, a control means controls the display of the virtual space so that contact corresponding to tactile sensations that cannot be simultaneously presented to the two or more users from the haptic device being used does not occur as contact of two or more users to a virtual object in the virtual space. An information processing device characterized by having the following features.

2. The user's position and orientation in the virtual space are linked to the user's position and orientation in the real space. The acquisition means detects a first user among the plurality of users from the real space and acquires information on the position and orientation of the first user. The information processing apparatus according to feature 1.

3. The acquisition means detects the first user from the image captured of the real space. The information processing apparatus according to feature 2.

4. The acquisition means acquires location and orientation information of a second user, who is different from the first user, from outside the information processing device. The information processing apparatus according to feature 2.

5. The aforementioned information regarding the user's position and posture is information regarding the position and posture of the user's body parts. The acquisition means detects multiple body parts of the first user, The aforementioned information processing device is Selection means for selecting at least one of the plurality of parts of the first user. It further possesses, The control means uses the position and posture information of the part selected by the selection means as the position and posture information of the first user. The information processing apparatus according to feature 2.

6. The aforementioned information regarding the user's position and posture is information regarding the position and posture of the user's body parts. The control means sets a space that includes the body parts of a first user among the plurality of users and the haptic device, and controls the display of the virtual space so that no more than a predetermined number of user body parts are included in the set space. The information processing apparatus according to feature 1.

7. The aforementioned information regarding the user's position and posture is information regarding the position and posture of the user's body parts. The acquisition means further acquires information from the tactile device, The control means controls the display of the virtual space so that no more than a predetermined number of user body parts are included in the space based on the information of the tactile device. The information processing apparatus according to feature 1.

8. The space based on the information from the haptic device is a space in which the haptic device can present tactile sensations to the user. The information processing apparatus according to feature 7.

9. The acquisition means acquires information about the user's future location and orientation as information about the user's location and orientation. The information processing apparatus according to feature 1.

10. The control means controls at least one of the position and orientation of the virtual object. The information processing apparatus according to feature 1.

11. The control means controls the number of virtual objects. The information processing apparatus according to feature 1.

12. The control means controls the size of the virtual object. The information processing apparatus according to feature 1.

13. The control means controls the display of the virtual space to issue a predetermined notification if it is not possible to control the display of the virtual space so that contact corresponding to the tactile sensation that cannot be presented does not occur. The information processing apparatus according to feature 1.

14. For each of the multiple users, an acquisition step is made to acquire information on the location and orientation of the user, Information processing method characterized by having a control step of controlling the display of the virtual space based on a plurality of pieces of information corresponding to each of the plurality of users acquired in the acquisition step, so as to prevent contact with a virtual object in the virtual space from occurring that corresponds to a touch that cannot be simultaneously presented to the two or more users from the haptic device used.

15. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 13.

16. A computer-readable storage medium storing a program for causing the computer to function as one of the means of the information processing apparatus described in any one of claims 1 to 13.

Images