Information processing apparatus, method, and program

Abstract

The present technology relates to an information processing apparatus, method, and program that make it possible to perform tactile presentation closer to the intention of a creator. This information processing apparatus comprises a processing unit that, on the basis of behavior information indicating the behavior of a user, generates probability data related to the probability of performing tactile presentation at a future time point including the time when the behavior information is acquired. The present technology can be applied to a tactile presentation system.

Description

Information processing device, method, and program

[0001] The present technology relates to an information processing device, method, and program, and more particularly to an information processing device, method, and program that enable tactile presentation closer to the creator's intention.

[0002] For example, content that utilizes tactile presentation is known, such as presenting a tactile sensation when a tactile interaction occurs, such as when a user touches an object in a virtual space.

[0003] When a haptic device requires a startup time to present haptic sensations, it is not possible to present haptic sensations at the appropriate timing if the device is activated after a haptic interaction has occurred. Even if a predicted activation time is used, there is a large error, making it difficult to present haptic sensations at the appropriate timing.

[0004] For example, as a technology for presenting tactile sensations at the appropriate time, a method has been proposed in which the tactile sensation is presented at the appropriate time by utilizing the distance from an object in the virtual space to the user's hand, etc., and the delay time of the tactile presentation device, which is known in advance.

[0005] A method has also been proposed for generating rendering data of a subsequent scene that is expected to occur in the receiver's virtual space at a time later than the current time, based on sensor information of the current time obtained from the sender (see, for example, Patent Document 1).

[0006] International Publication No. 2024 / 034336

[0007] However, with the above-mentioned techniques, it has been difficult to provide tactile feedback in accordance with the intention of the creator of the content or the like.

[0008] For example, in a method that uses the distance from an object in a virtual space to a user's hand or the like and the known delay time of a haptic device, the predicted user behavior may not match the actual behavior. In such cases, the device may be driven in a way that the creator did not intend, such as providing a haptic sensation even though the user is not touching the object.

[0009] Specifically, consider a haptic presentation using a haptic presentation device that requires 0.5 seconds to start up. If the haptic presentation (stimulation) starts when the user is predicted to have 0.5 seconds left until the user's hand or other object comes into contact, and the predicted time until the object is touched changes due to a change in the user's behavior, etc., the haptic presentation will continue even though the user has not touched the object.

[0010] The present technology has been developed in light of these circumstances, and makes it possible to present tactile sensations that are closer to the creator's intentions.

[0011] An information processing device according to a first aspect of the present technology includes a processing unit that generates, based on behavior information indicating a user's behavior, probability data regarding the probability of causing a tactile presentation at a future time including the time when the behavior information is acquired.

[0012] The information processing method or program of the first aspect of the present technology includes a step of generating, based on behavior information indicating a user's behavior, probability data regarding the probability of providing a tactile presentation at a future time including the time when the behavior information is acquired.

[0013] In a first aspect of the present technology, probability data regarding the probability of causing a tactile presentation at a future time including the time when the behavior information indicating a user's behavior is acquired is generated based on the behavior information.

[0014] An information processing device according to a second aspect of the present technology includes a signal generation unit that generates a drive signal for providing tactile presentation based on probability data regarding the probability of providing tactile presentation at a future time including the time when behavior information indicating the user's behavior is acquired.

[0015] The information processing method or program of the second aspect of the present technology includes a step of generating a drive signal for providing tactile presentation based on probability data regarding the probability of providing the tactile presentation at a future time including the time when behavior information indicating the user's behavior is acquired.

[0016] In a second aspect of the present technology, a drive signal for providing a tactile sensation is generated based on probability data regarding the probability of providing a tactile sensation at a future time including the time when behavior information indicating a user's behavior is acquired.

[0017] 1 is a diagram illustrating a haptic presentation operation for a haptic interaction. A diagram illustrating the use of stimulation probability data. A diagram illustrating an example of the configuration of an information processing device. A flowchart illustrating haptic presentation processing. A diagram illustrating an example of stimulation probability data. A diagram illustrating an example of stimulation probability data. A diagram illustrating an example of stimulation probability data. A diagram illustrating an example of stimulation probability data. A diagram illustrating an example of stimulation probability data. A diagram illustrating an example of stimulation probability data. A diagram illustrating an example of stimulation probability data. A diagram illustrating an example of the format of stimulation probability data. A diagram illustrating an example of the format of stimulation probability data. A diagram illustrating an example of calculation of stimulation occurrence probability. A diagram illustrating an example of generation of a drive signal. A diagram illustrating an example of generation of a drive signal. A diagram illustrating adjustment of a stimulation occurrence threshold. A diagram illustrating adjustment of a stimulation occurrence threshold. A diagram illustrating an example of a screen for setting. A diagram illustrating processing allocation. A diagram illustrating an example of the configuration of a server. A diagram illustrating an example of the configuration of a computer.

[0018] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0019] <First embodiment> <About the present technology> For example, when a haptic sensation is presented at the timing when a haptic interaction such as when a user touches an object occurs, it has been difficult to present a haptic sensation at an appropriate timing even if the haptic sensation presentation device is driven based on a predicted result of the user's behavior and a delay time such as the startup time of the haptic sensation presentation device. In other words, it has been difficult to present a haptic sensation as intended by a creator of content, etc.

[0020] Therefore, this technology provides flexibility in the approach to tactile presentation, making it possible to present tactile sensations that are closer to the creator's intentions.

[0021] Flexibility in the tactile presentation policy is achieved by taking into account the characteristics of the tactile presentation device and adjusting the probability that tactile presentation will not occur even when an object (virtual object) is being touched, i.e., a tactile interaction is occurring, and the probability that tactile presentation will occur even when a tactile interaction is not occurring.

[0022] Specifically, in this technology, the probability of a haptic interaction occurring (or occurring) for each time period from the current time to a future time until the haptic presentation and a stimulus occurrence threshold are defined. For example, the stimulus occurrence threshold is defined by the content creator.

[0023] Furthermore, for example, the probability that a haptic interaction will occur at a future time (hereinafter also referred to as a stimulation occurrence probability) is calculated using information such as the positions, velocities, and accelerations of the user and objects in the virtual space of the content. This time-series stimulation probability data indicating the stimulation occurrence probability at each future time is used for tactile presentation by the haptic presentation device.

[0024] By doing so, even when using a tactile presentation device that takes a long time to start up, it is possible to select a tactile presentation method (policy) that is suitable for the content and provide a tactile presentation that is closer to the creator's intention. In other words, the quality of the content experience can be improved. Furthermore, since malfunctions of the tactile presentation can be reduced, power consumption in the tactile presentation device, etc. can be reduced.

[0025] The effect of improving the quality of experience by using stimulation probability data will be described with reference to FIGS.

[0026] As a premise, let us assume that in a period of 1000 frames of content, the number of frames in which haptic presentation should be performed, i.e., the number of frames in which haptic interaction actually occurred, is 200, and the number of frames in which haptic interaction did not occur is 800. In other words, let us assume that there are 200 frames in which there is contact with a virtual object, and 800 frames in which there is no contact.

[0027] In particular, the time-series transition between contact and non-contact states is not taken into account, so for example, the first 200 frames may be in contact and the next 800 frames may be in non-contact, or vice versa.

[0028] For 1000 frames of such content, if haptic presentation is performed in the frames where haptic interaction occurs, the ideal presentation result would be as shown in Figure 1.

[0029] In this example, tactile presentation (stimulus presentation) was performed in all frames (200 frames) where actual tactile interaction occurred, i.e., where there was contact with the virtual object, and no tactile presentation was performed in all frames (800 frames) where there was no contact with the virtual object and no tactile interaction occurred.

[0030] Therefore, there is no malfunction in which a tactile sensation is not presented in a frame where a virtual object is in contact (hereinafter also referred to as a contact / non-presentation operation), or a malfunction in which a tactile sensation is presented in a frame where a virtual object is not in contact (hereinafter also referred to as a non-contact / presentation operation). In other words, the contact / non-contact of a virtual object corresponds to the presence or absence of a tactile sensation, and no errors (malfunctions) occur in the tactile sensation presentation.

[0031] However, in reality, malfunctions (errors) occur, such as a contact / non-presentation operation in which no tactile sensation is presented despite a contact state, or a non-contact / presentation operation in which a tactile sensation is presented despite a non-contact state, as shown on the left side of Figure 2.

[0032] In this example, contact / non-presentation actions occur for 100 frames and non-contact / presentation actions occur for 100 frames, resulting in malfunctions occurring in a total of 200 frames.

[0033] Ideally, haptic interaction would always be performed correctly, but in reality, device limitations, such as the startup time of the haptic device, can cause malfunctions (errors). Such malfunctions can also occur when a user's behavior is predicted and haptic feedback is used.

[0034] Therefore, this technology introduces stimulation probability data and a stimulation occurrence threshold, enabling tactile presentation that is closer to the creator's intention.

[0035] For example, in the present technology, if the stimulation occurrence probability at a predetermined time indicated by the stimulation probability data is equal to or greater than a stimulation occurrence threshold, a tactile sensation is presented at the predetermined time.

[0036] In this case, the creator can provide flexibility to the tactile presentation by appropriately specifying the stimulation generation threshold in accordance with the creator's intention, i.e., the tactile presentation policy (hereinafter also referred to as the presentation policy), which is determined by the content, etc. In other words, the tactile presentation can be performed in accordance with the presentation policy, and the occurrence of malfunctions (errors) unintended by the creator can be reduced.

[0037] Specifically, for example, it is assumed that the creator's presentation policy is to reduce contact / non-presentation actions in which no tactile sensation is presented despite a contact state.

[0038] In such a case, the creator can reduce the contact / non-presentation action as shown in the upper right of the figure by lowering the stimulus generation threshold, that is, by setting the stimulus generation threshold low.

[0039] In this example, correct tactile presentation for the contact condition increased from 100 to 140 frames, and errors for the contact / no-presentation behavior decreased from 100 to 60 frames.

[0040] This shows that adjusting the stimulation generation threshold allows for tactile presentation that is closer to the creator's presentation policy (intention), which is to reduce contact / non-presentation actions, that is, to ensure that tactile presentation is as accurate as possible in response to the contact state.

[0041] On the other hand, for example, suppose that the creator's presentation policy is to reduce non-contact / presentation actions in which tactile presentation is performed even in a non-contact state.

[0042] In such a case, the creator can reduce the non-contact / presentation action as shown in the lower right of the figure by increasing the stimulus generation threshold, that is, by setting the stimulus generation threshold high.

[0043] In this example, the correct behavior of no tactile presentation for the no-touch condition increases from 700 frames to 740 frames, and the error of the no-touch / presentation behavior decreases from 100 frames to 60 frames.

[0044] This shows that by adjusting the stimulation generation threshold, tactile presentation can be achieved that is closer to the creator's presentation policy (intention), which is to reduce non-contact / presentation operations, that is, to avoid tactile presentation as much as possible when in a non-contact state.

[0045] As described above, by introducing probability into tactile presentation, the creator can specify the presentation policy, in other words, the stimulus generation threshold, thereby enabling tactile presentation that is closer to the creator's intention.

[0046] <Configuration Example of Information Processing Apparatus> FIG. 3 is a diagram showing a configuration example of an embodiment of an information processing apparatus to which the present technology is applied.

[0047] The information processing device 11 may be configured as a single device such as a VR (Virtual Reality) device, or may be configured as a plurality of devices such as a motion capture system, a head-mounted display, a tactile presentation device, etc. For example, when the information processing device 11 is configured as a plurality of devices, it can be said that the information processing device 11 functions as a tactile presentation system.

[0048] For example, the information processing device 11 plays content created by a creator. The content may consist of only video, or may consist of video and accompanying sound, or may consist of only sound (audio). For example, the video of the content may be a video of a virtual space (virtual space) including objects and users as subjects, and the sound of the content may include at least the sound of the objects. Alternatively, the content may be AR (Augmented Reality) content obtained by superimposing virtual objects on real space.

[0049] Furthermore, when content is being played back, a tactile sensation is presented to the user in accordance with the user's movements, etc. For example, even if the content consists of audio only, a tactile sensation can be presented in accordance with the user's movements when the user makes a specific movement (gesture) during audio playback, or when a specific sound is played back.

[0050] In the following description, it is assumed that the content includes at least a video. In particular, it is assumed that the content is a video that includes one or more objects (virtual objects) in a virtual space and at least one user in the virtual space as a video object.

[0051] In the following, a haptic interaction will be referred to as an occurrence of a predetermined event, such as a user touching (coming into contact with) a predetermined object in a virtual space, such as an object, a wall, another user, a flame, or a beam, a user entering a predetermined area, or a predetermined playback time. Furthermore, a haptic presentation will be performed at the timing when a haptic interaction occurs. That is, the occurrence of a haptic interaction will be referred to as the occurrence of an event that requires a haptic presentation.

[0052] The information processing device 11 includes an input unit 21 , a control unit 22 , a tactile sense providing unit 23 , a display unit 24 , a communication unit 25 , and an audio output unit 26 .

[0053] The input unit 21 is composed of a motion capture device attached to a predetermined part of the user and having any sensor such as an acceleration sensor, an angular velocity sensor, or an image sensor, and receives input from the user.

[0054] For example, when the user moves the part (wearing part) to which the input unit 21 is attached or moves their line of sight, the input unit 21 detects (senses) the user's movement and supplies the detection result as behavior information indicating the user's behavior (movement) to the control unit 22. The behavior information indicates the movement of the user's body parts and the movement of the user's line of sight in the real space, i.e., the user's behavior, and can also be said to indicate the user's behavior in the virtual space.

[0055] For example, behavior information can be used to estimate or identify the position and orientation (posture) of the user and each part of the user's body, such as the user's hands, the speed and acceleration of each part, the direction of movement of each part, the user's line of sight (gaze position), and the angle (flexion angle) of each joint part of the user, such as the user's elbows and knees, in the virtual space.

[0056] The input unit 21 may be composed of a mouse, keyboard, buttons, a touch panel superimposed on the display unit 24, etc., which can input the user's movements (behavior) in the virtual space, or it may be composed of a camera (image sensor), etc., which is placed at a position away from the user and can detect the movements of each part of the user's body and their line of sight.

[0057] The control unit 22 is composed of one or more processors and controls the overall operation of the information processing device 11. For example, the control unit 22 drives the tactile presentation unit 23, which is a tactile presentation device, based on behavior information from the input unit 21, and generates and outputs a drive signal for causing the tactile presentation to be performed. That is, the control unit 22 controls the tactile presentation by the tactile presentation unit 23.

[0058] The control unit 22 includes a processing unit 31 and a signal generating unit 32. For example, the processing unit 31 and the signal generating unit 32 are realized by the control unit 22 executing a program.

[0059] The processing unit 31 generates content data for playing back content based on data for generating content prepared in advance and behavior information supplied from the input unit 21 .

[0060] For example, the data for generating content includes scenario data indicating a content scenario, such as image data of each object in the virtual space, audio data of each object, and the movement of the object at each time. The data for generating content may be acquired by the control unit 22 from an external device such as a server via the communication unit 25, or may be recorded in advance in a memory or the like provided in the control unit 22.

[0061] Scenario data is data that indicates information such as the movement of objects at each time, the number of objects, the placement of objects, the type of content scene at each time, whether or not various events occur, etc. For example, scenario data is generated in advance by a creator for content.

[0062] Furthermore, as content data, video data for displaying a video (image) of the virtual space including objects, users, etc. is generated, and this video data is supplied from the control unit 22 to the display unit 24 .

[0063] In addition to the video data, audio data for reproducing the sounds of objects and sound effects in the virtual space may be generated as content data. In this case, the audio data of the content is supplied from the control unit 22 to the sound output unit 26.

[0064] Based on the behavior information supplied from the input unit 21, the processing unit 31 calculates the probability of tactile interaction occurring (stimulation occurrence probability), in other words, the probability of tactile presentation, at each future time including the time the behavior information was acquired, to generate stimulation probability data and supply it to the signal generation unit 32.

[0065] The stimulation probability data is periodically updated and supplied as stream data (streaming data) to the signal generation unit 32. Content data, pre-prepared scenario data, and other content generation data may also be used to generate the stimulation probability data as appropriate. Note that the stimulation probability data is not limited to data indicating the stimulation occurrence probability itself at each time, but may be any data related to the stimulation occurrence probability, such as the mean value and variance of a probability distribution, i.e., any data that can identify the stimulation occurrence probability at each time.

[0066] The processing unit 31 also supplies the signal generating unit 32 with tactile data, which is data related to tactile presentation, such as the stimulation pattern and stimulation intensity at the time of tactile presentation.

[0067] The tactile data may be recorded in advance in a memory or the like provided in the control unit 22, or may be acquired by the control unit 22 as appropriate from an external device such as a server via the communication unit 25.

[0068] For example, the tactile data is data consisting of metadata and stimulation information indicating the intensity and stimulation pattern of the tactile stimulation.

[0069] The metadata includes data indicating, for example, the sampling frequency of the waveform information of the tactile stimulus, the type of tactile presentation (tactile stimulation) to be performed on each channel, i.e., the modality of the tactile stimulation, and the modalities that can be used when presenting the tactile sensation, such as whether hot, cold, or electrical stimulation is possible.

[0070] The stimulation information may be, for example, information indicating the temperature of the tactile presentation in the hot / cold device serving as the tactile presentation unit 23, such as 25°C, 35°C, or relatively +5°C, or information indicating the stimulation intensity of the electrical stimulation device serving as the tactile presentation unit 23, such as intensity 0, 3, or 5.

[0071] Furthermore, for example, the stimulation information may be information indicating the vibration waveform when a vibration stimulus is provided as a tactile presentation using a vibration device, information indicating the temperature change pattern when a temperature stimulus is provided as a tactile presentation using a hot / cold device, or information indicating the compression pattern when compression is provided as a tactile presentation using a compression device.

[0072] The haptic data may be supplied from the processing unit 31 to the signal generating unit 32 before or when the content is played back, and the haptic data may be used thereafter, or may be supplied from the processing unit 31 to the signal generating unit 32 each time it is needed as stream data. In particular, when the haptic data includes time-series information such as a stimulus waveform or stimulus pattern as stimulation information, it is preferable to supply the haptic data to the signal generating unit 32 in advance.

[0073] The signal generating unit 32 is supplied with tactile sense providing device information, which is information relating to the tactile sense providing unit 23 , from the tactile sense providing unit 23 , and is also supplied with a stimulation generation threshold value from the control unit 22 .

[0074] For example, the tactile presentation device information includes at least information indicating the rise speed of the tactile presentation unit 23 which is a tactile presentation device, that is, information indicating the rise time (delay time) of the tactile presentation unit 23 .

[0075] The rise time here refers to, for example, the time from when stimulation probability data is generated by the processing unit 31, when the tactile presentation unit 23 presents a tactile sensation based on the stimulation probability data, until the user actually perceives the stimulation (tactile stimulation) due to the tactile presentation. In other words, the rise time may include all of the system delays due to communication and processing (computation) in the control unit 22, the tactile presentation unit 23, etc., the time required for physical changes in the devices related to the tactile presentation, and the time related to the perceptual process.

[0076] Furthermore, for example, the tactile presentation device information may include data corresponding to the metadata of the tactile data, such as data indicating tactile sensations that can be presented by the tactile presentation unit 23, such as hot / cold or vibration, information regarding drive signals that can be handled by the tactile presentation unit 23, parameters related to the content, and information about the body part to which the tactile presentation unit 23 is attached.

[0077] For example, a stimulation timing tolerance window may be considered as an example of a content-related parameter. The stimulation timing tolerance window is information indicating the time range (tolerance time) that is allowed for a delay in the timing of tactile presentation, i.e., the time range within which a human being does not perceptually sense a delay in a tactile stimulus. For example, the stimulation timing tolerance window may be defined (prepared) for each type of tactile presentation device serving as the tactile presentation unit 23. Furthermore, since the time range (tolerance time) that is allowed for a delay in the timing of tactile presentation is expected to differ for each user, if multiple stimulation timing tolerance windows are set, each user may be able to select a stimulation timing tolerance window that is appropriate for them via a UI (User Interface) or the like.

[0078] Furthermore, for example, the stimulation generation threshold supplied from the control unit 22 to the signal generation unit 32 may be included in the data for content generation described above, or may be obtained by the control unit 22 from an external device such as a server via the communication unit 25 separately from the data for content generation.

[0079] The signal generating unit 32 generates a drive signal based on the stimulation probability data and tactile data from the processing unit 31, the stimulation occurrence threshold, and the tactile presentation device information from the tactile presentation unit 23, and supplies the drive signal to the tactile presentation unit 23 to cause tactile presentation. The drive signal may be output as stream data (streaming data) including zero data indicating that tactile presentation is not to be performed, or may be output only at the timing when tactile presentation is to be performed.

[0080] The tactile presentation unit 23 comprises one or more tactile presentation devices attached (worn) to a user who receives the tactile presentation, i.e., a user who views the content, and is driven based on a drive signal supplied from the signal generation unit 32 to present a tactile sensation to the user. For example, the tactile presentation device may be comprised of an actuator or the like, and presents a tactile sensation by giving a tactile stimulus to the user according to the type of the tactile presentation device.

[0081] The type of tactile presentation (tactile stimulation) may be any type, such as temperature stimulation (hot or cold), stimulation by wind, pressure, vibration, force stimulation, stimulation related to friction or texture (touch), electrical stimulation, distributed stimulation, etc. As described above, tactile presentation is not limited to when a user touches an object (material, etc.) in a virtual space, but can also be performed (tactile interaction occurs) when any event occurs, such as a predetermined user behavior or a content scenario.

[0082] The display unit 24 is made up of a display, and displays an image based on image data (video data) supplied from the control unit 22. For example, the display unit 24 displays a video of the content.

[0083] The communication unit 25 communicates with an external device such as a server under the control of the control unit 22. For example, the communication unit 25 transmits information supplied from the control unit 22 to the server or the like, or receives information transmitted from the server or the like and supplies the information to the control unit 22.

[0084] The audio output unit 26 is made up of one or more speakers, and reproduces sound based on the audio data supplied from the control unit 22. For example, the audio output unit 26 reproduces the sound of the content.

[0085] Note that some or all of the input unit 21 , the tactile sense providing unit 23 , the display unit 24 , and the sound output unit 26 may be configured as separate devices different from the information processing device 11 .

[0086] Furthermore, at least one of the processing unit 31 and the signal generation unit 32 may be realized by a device other than the information processing device 11, i.e., an external server or the like. For example, the processing unit 31 may be realized by a server constituting an external cloud, or the functions of the processing unit 31 and the signal generation unit 32 may be provided in a device having the tactile presentation unit 23 other than the information processing device 11.

[0087] <Explanation of Tactile Presentation Processing> The operation of the information processing device 11 will be described. That is, the tactile presentation processing by the information processing device 11 will be described below with reference to the flowchart in Fig. 4. For example, the tactile presentation processing is performed when playback of content starts.

[0088] In step S11 , the processing unit 31 acquires the behavior information output from the input unit 21 .

[0089] For example, the input unit 21 constantly detects the user's movements (behavior) and outputs behavior information indicating the detection results, so the processing unit 31 acquires the behavior information output from the input unit 21.

[0090] In step S12 , the processing unit 31 generates stimulation probability data based on the behavior information acquired in step S11 and data for generating content prepared in advance, and supplies the generated stimulation probability data to the signal generating unit 32 .

[0091] Specifically, for example, the processing unit 31 generates content data based on the behavior information and data for content generation.

[0092] Once the content data is generated, the content data is then supplied at an appropriate timing to the display unit 24 and the sound output unit 26. That is, the control unit 22 supplies video data as content data to the display unit 24 to display the video of the content, and supplies audio data as content data to the sound output unit 26 to play the sound of the content.

[0093] In addition, the processing unit 31 generates stimulation probability data based on the content data, in other words, data for content generation, and behavior information at a timing before playback of a frame of content (hereinafter also referred to as the current frame) based on the generated content data.

[0094] At this time, not only the content data in the current frame of the content to be played back, but also the content data of frames (past frames) before the current frame may be used to generate the stimulation probability data.Similarly, not only the latest (current) behavior information but also past behavior information may be used to generate the stimulation probability data.

[0095] Specifically, for example, the processing unit 31 calculates the probability of a stimulus occurring in a frame after the current frame (future), i.e., in a future time, based on at least one of information about the user's body at the current time (current frame), such as the position, orientation (posture), speed, acceleration, movement direction, and bending angle of each part of the user's joints in the virtual space. Note that the probability of a stimulus occurring in the current frame (current time) may also be calculated.

[0096] Furthermore, for example, the calculation of the probability of a stimulus occurring may use at least one of the following information: the position, orientation (posture), speed, acceleration, direction of movement, and information about the body (e.g., information such as the flexion angle of a joint) of each part of one or more objects (video objects) in the virtual space in the current frame.

[0097] Similarly, at least one of the position, orientation (posture), speed, acceleration, direction of movement, and information about the body (such as the bending angle of joints) of each part of the user in the virtual space in past frames, or at least one of the position, orientation (posture), speed, acceleration, direction of movement, and information about the body (such as the bending angle of joints) of each part of one or more objects (image objects) in the virtual space in past frames may be used to calculate the probability of stimulation occurrence.

[0098] In addition, the calculation of the stimulus occurrence probability may use the user's line of sight in the virtual space in the current frame, i.e., the line of sight position (the position where the user is looking), scenario data of the content, etc.

[0099] As an example, by using content data (video data) of the current frame and past frames, it is possible to identify the relative distance, orientation, and other positional relationships between the user and an object in the virtual space at the present (current frame) and past (past frame), as well as the relative speed and acceleration. Using this identification result, it is possible to determine the probability of a haptic interaction, such as contact between the user and an object, occurring at each time point (each frame) in the future, including the present, as the stimulation occurrence probability.

[0100] Furthermore, for example, by using the bending angle of each part of the user in the virtual space, it is possible to identify the shape of each part of the user, such as the shape of the user's hand, etc. From this identification result, it is possible to estimate (identify) whether the user's behavior is grasping an object or colliding with an object, and based on this estimation result, it is possible to calculate a more accurate probability of stimulus occurrence.

[0101] In addition, tactile data is supplied at appropriate timing from the processing unit 31 to the signal generating unit 32. Furthermore, tactile presentation device information is supplied from the tactile presentation unit 23 to the signal generating unit 32 at appropriate timing.

[0102] In step S13 , the signal generating unit 32 generates a drive signal based on the stimulation probability data and tactile data supplied from the processing unit 31 , the stimulation occurrence threshold, and the tactile presentation device information supplied from the tactile presentation unit 23 .

[0103] For example, the signal generation unit 32 determines whether or not to provide a tactile sensation (whether or not a tactile interaction will occur) at each future time, i.e., at each future frame including the current frame, based on the stimulation probability data, the stimulation occurrence threshold, and the tactile sensation provision device information. For example, to determine whether or not to provide a tactile sensation, information on the rise speed, such as the rise time, and the stimulation timing allowance window, which are included in the tactile sensation provision device information, are appropriately used.

[0104] Then, based on the result of the determination of whether or not to provide tactile presentation and the tactile data, the signal generation unit 32 generates a drive signal for providing tactile presentation with appropriate intensity and pattern at the time (frame) at which tactile presentation is to be provided.

[0105] The signal generating unit 32 supplies (outputs) the generated drive signal to the tactile presentation unit 23, causing the tactile presentation unit 23 to present a tactile sensation of appropriate intensity and pattern at appropriate timing. The tactile presentation unit 23 is driven in response to the drive signal from the signal generating unit 32, thereby presenting a tactile sensation by providing any tactile stimulus such as vibration to the user wearing the tactile presentation unit 23.

[0106] As a result, while viewing the content, the user receives a tactile sensation at the timing when a tactile interaction occurs in response to the user's own movement, etc. In other words, the user perceives the tactile stimulus provided by the tactile sensation providing unit 23 as a tactile sensation.

[0107] In step S14, the control unit 22 determines whether to terminate the process of providing haptic presentation in accordance with the playback of the content. For example, the control unit 22 determines to terminate the process of providing haptic presentation when the playback of the content has ended or when the user instructs to stop the playback of the content.

[0108] If it is determined in step S14 that the process is not yet finished, the process then returns to step S11, and the above-described process is repeated.

[0109] On the other hand, if it is determined in step S14 that the processing is to be ended, the control unit 22 stops the processing of each unit of the information processing device 11, and the tactile presentation processing ends.

[0110] In this way, the information processing device 11 generates stimulation probability data according to behavior information, etc., and generates a drive signal for tactile presentation based on the stimulation probability data and information related to the rise speed of the tactile presentation unit 23, etc.

[0111] In this way, by controlling the tactile presentation using the stimulation probability data, it is possible to provide a tactile presentation in accordance with the presentation policy intended by the content creator, such as a presentation policy that reduces contact / non-presentation actions or a presentation policy that reduces non-contact / presentation actions. In other words, it is possible to provide a tactile presentation that is closer to the creator's intention. Furthermore, it is possible to reduce the occurrence of malfunctions (errors) unintended by the creator, thereby reducing power consumption for driving, etc.

[0112] <Specific Examples of Stimulation Probability Data> Specific examples of stimulation probability data will be described with reference to FIGS.

[0113] FIG. 5 shows an example of the data structure of stimulation probability data supplied from the processing unit 31 to the signal generating unit 32 at a certain time.

[0114] Stimulation probability data, i.e., the probability of stimulation occurring at each time, is generated by using the position, posture, and speed of the user's finger or other body part or object at each time to predict the occurrence (state) of tactile interaction at each future time, including the time when behavioral information is acquired (acquisition time).

[0115] The calculation (computation) of the stimulus occurrence probability may be performed using a learning model generated by data analysis or the like, or may be performed using a heuristic design model.

[0116] Generally, probability information is not used to determine whether to provide a tactile sensation. In other words, whether to provide a tactile sensation is determined based on sparse information such as whether the stimulus occurrence probability is 0 or 1.

[0117] In contrast, in the information processing device 11, the probability of a stimulus occurring at each time is expressed as a number between 0 and 1 (a number greater than or equal to 0 and less than or equal to 1), and this probability of a stimulus occurring allows flexibility in the tactile presentation.

[0118] FIG. 5 shows an example of stimulation probability data generated when a user is performing a behavior (action) of moving their finger closer to an object (object) in a virtual space.

[0119] Here, the stimulation probability data is data for one tactile presentation device that provides tactile presentation in the tactile presentation unit 23, that is, data for one channel.

[0120] In particular, the stimulation probability data is composed of time information indicating each time in the future including the time when the behavior information is acquired, and probability information indicating the probability of stimulation occurring at each time in the future.

[0121] Here, the column marked with the words "x seconds later," i.e., the upper part of the stimulation probability data, indicates time information, and the column marked with the words "probability of a certain stimulation occurring," i.e., the lower part of the stimulation probability data, indicates probability information.

[0122] In other words, the stimulation probability data can be said to be a stimulation occurrence probability array in which future times indicated by time information and stimulation occurrence probabilities at those times indicated by probability information are arranged for each time.

[0123] The time information is, for example, information indicating a time x seconds after the current time when the time when the stimulation probability data is output to the signal generating unit 32 is taken as the current time. In particular, in this example, the interval (width) between each time is a constant width of 0.1 seconds, and the time information is information indicating each time at equal intervals from 0 seconds later (x=0) to 1.0 seconds later. Note that the time information may be any information as long as it can identify each of a plurality of times.

[0124] The probability information indicates the probability that a tactile interaction will occur at each time (stimulation occurrence probability), i.e., the probability that a predetermined stimulus will be generated for the user through the tactile interaction. In other words, the probability information indicates the probability that a predetermined type of tactile stimulus (tactile presentation) will be provided at each time.

[0125] In particular, the tactile level (level), such as the intensity of the stimulus, is not taken into consideration here, and the probability of whether or not a stimulus is occurring, that is, the probability of presenting a tactile sensation, is included as probability information.

[0126] For example, in this example, the probability of a stimulus occurring at the time "0.6 seconds later" is "0.9," which indicates that the probability of a tactile sensation being presented 0.6 seconds after the current time is 0.9.

[0127] As an example of generating a drive signal when using such stimulation probability data, the following case can be considered.

[0128] It is assumed here that the startup time of the tactile presentation unit 23 is 0.5 seconds. In other words, it is assumed that the time required for the user to perceive the tactile presentation is 0.5 seconds, based on the time when the stimulation probability data is generated (output).

[0129] In such a case, for example, suppose the creator specifies (selects) presentation policy α, which aims to reduce contact / non-presentation actions in which no tactile sensation is presented despite a contact state, and specifies a stimulation generation threshold of "0.6" for that presentation policy α.

[0130] The creator may specify (set) both the presentation policy and the stimulus generation threshold, or when the creator specifies the presentation policy, a predetermined stimulus generation threshold may be automatically specified for the presentation policy. Also, when the creator specifies the stimulus generation threshold, a predetermined presentation policy may be automatically specified for the stimulus generation threshold.

[0131] When the presentation policy α and the stimulation occurrence threshold value of "0.6" are specified and the stimulation probability data shown in Figure 5 are supplied, the signal generation unit 32 targets the period from 0 seconds to 0.5 seconds, for example, in step S13 of Figure 4, since the start-up time of the tactile presentation unit 23 is 0.5 seconds.

[0132] If the maximum value of the stimulation occurrence probability at each time from 0 seconds to 0.5 seconds is equal to or greater than the stimulation occurrence threshold "0.6", the signal generation unit 32 determines to provide a tactile sensation and generates a drive signal. In this example, the stimulation occurrence probability "0.8" at the time "0.5 seconds later" is equal to or greater than the stimulation occurrence threshold "0.6", so a drive signal is generated and a tactile sensation is provided.

[0133] Also, for example, suppose that a creator specifies a presentation policy β that aims to reduce non-contact / presentation actions in which tactile sensations are presented even when the device is not in contact, and specifies a stimulation generation threshold of "0.9" for that presentation policy β.

[0134] In this case, for example, in step S13 of Fig. 4, if the maximum value of the stimulation occurrence probability at each time from 0 seconds to 0.5 seconds is equal to or greater than the stimulation occurrence threshold "0.9", the signal generation unit 32 determines to provide a tactile sensation and generates a drive signal. In this example, the maximum value, the stimulation occurrence probability "0.8" at the time "0.5 seconds later", is less than the stimulation occurrence threshold "0.9", so no drive signal is generated and no tactile sensation is provided.

[0135] Furthermore, for example, suppose that the creator specifies a presentation policy γ that indicates that the timing of tactile presentation is particularly important, and specifies a stimulus generation threshold of "0.6" in the presentation policy γ.

[0136] In this case, for example, in step S13 of Figure 4, if the maximum value of the stimulation occurrence probability at each time is equal to or greater than the stimulation occurrence threshold value of "0.6", the signal generation unit 32 determines that tactile presentation will be performed at the time when that maximum value is reached, and generates a drive signal.

[0137] In this example, since the stimulation occurrence probability of "0.9" at the time "0.6 seconds later," when the stimulation occurrence probability reaches its maximum value, is equal to or greater than the stimulation occurrence threshold value of "0.6," a drive signal is generated in advance to prepare for the tactile presentation so that the tactile presentation will be performed at that time "0.6 seconds later." Then, a drive signal is output at the timing "0.1 seconds later." As a result, the tactile presentation unit 23 will perform the tactile presentation at "0.6 seconds later," when the stimulation occurrence probability is at its maximum.

[0138] There are also cases where the signal generating unit 32 can generate a drive signal based on the stimulation probability data within a time shorter than the time interval at which the stimulation probability data is supplied to the signal generating unit 32. In other words, there are also cases where the signal generating unit 32 can generate a drive signal faster than the update rate of the stimulation probability data.

[0139] In such a case, the signal generating unit 32 may update the stimulation probability data by subtracting the real time (elapsed time) indicated by the system clock from the time indicated by the time information of the stimulation probability data, and generate the next drive signal based on the updated stimulation probability data.

[0140] A typical example of stimulus probability data for the structure shown in FIG. 5 is shown in FIG.

[0141] In the example shown in FIG. 6, the stimulation probability data is also made up of time information indicating each time in the future and probability information indicating the stimulation occurrence probability at each time in the future.

[0142] The portion indicated by arrow Q11 shows an example of stimulation probability data generated when the user's finger is a predetermined distance away from an object in the virtual space and the finger is not moving. In this example, because the finger is stationary (not moving), the stimulation occurrence probability is set to 0.0 in the near future, and the stimulation occurrence probability is not very high even after that.

[0143] The portion indicated by arrow Q12 shows an example of stimulation probability data generated when the user is bringing their finger closer to an object in virtual space. That is, the portion indicated by arrow Q12 shows the same stimulation probability data as the example in Figure 5. In this example, as the finger approaches the object, the stimulation occurrence probability increases as time progresses, and then decreases at subsequent times.

[0144] The part indicated by arrow Q13 shows an example of stimulation probability data generated when the user's finger is in contact with an object in virtual space. In this example, the finger is in contact with the object and haptic interaction has already occurred, so the stimulation occurrence probability at time "0 seconds later" is set to 1.0, and the stimulation occurrence probability decreases over time.

[0145] The portion indicated by arrow Q14 shows an example of stimulation probability data generated when the user's finger is moving away from an object in virtual space. In this example, because the finger is moving away from the object, the stimulation occurrence probability is particularly low in the near future, and the stimulation occurrence probability is also low in the future.

[0146] FIG. 7 shows an example of stimulation probability data indicating the state of tactile sensation (stimulation) that can be achieved in tactile presentation based on a drive signal, that is, the stimulation occurrence probability for each tactile level.

[0147] In this example, there are three states of stimulation (stimulation states) that the tactile sense providing unit 23 can provide to the user, from a first level to a third level.

[0148] As an example, the first level is an "off" stimulation state in which no tactile sensation is presented, i.e., no stimulation is given, the second level is a "weak" stimulation state in which a weak stimulation is given, and the third level is a "strong" stimulation state in which a stronger stimulation than that given at "weak" is given. Therefore, even when tactile sensations are presented by the same tactile presentation device, the tactile sensations presented, more specifically, the intensity of the tactile sensation, differ for each stimulation state.

[0149] FIG. 7 shows an example of one channel of stimulation probability data generated when a user moves his / her finger closer to an object in a virtual space.

[0150] In Figure 7, the stimulation probability data is composed of time information similar to the example in Figure 5, probability information for the stimulation state "first level", probability information for the stimulation state "second level", and probability information for the stimulation state "third level".

[0151] In particular, the column marked with the words "Probability of the stimulus being at the first level," i.e., the second row from the top of the stimulus probability data, indicates probability information for the "first level" stimulus state. The third row from the top marked with the words "Probability of the stimulus being at the second level" indicates probability information for the "second level" stimulus state, and the fourth row from the top marked with the words "Probability of the stimulus being at the third level" indicates probability information for the "third level" stimulus state. Therefore, the time information is common to the probability information for each stimulus state.

[0152] For example, the probability information for the "first level" stimulation state indicates the probability that a tactile interaction that provides a "first level" stimulation state will occur at each time indicated by the time information (stimulus occurrence probability), i.e., the probability of providing a "first level" stimulation state tactile presentation.

[0153] In this case, for example, when the first level is an "off" stimulation state in which no tactile presentation is provided, the probability information of the "first level" becomes information indicating the probability that no tactile interaction will occur, that is, the probability that no tactile stimulation will be provided.

[0154] Furthermore, for example, the probability information of the "second level" stimulation state indicates the probability (stimulus occurrence probability) of a tactile interaction that provides a stimulus of the "second level" stimulation state occurring at each time indicated by the time information, i.e., the probability of providing a tactile presentation of the "second level" stimulation state. Similarly, the probability information of the "third level" stimulation state indicates the probability (stimulus occurrence probability) of a tactile interaction that provides a stimulus of the "third level" stimulation state occurring at each time indicated by the time information.

[0155] Since the state is one of the first to third levels at each time, the sum of the stimulation occurrence probability of the "first level" stimulation state, the stimulation occurrence probability of the "second level" stimulation state, and the stimulation occurrence probability of the "third level" stimulation state at one time is always 1.0. Therefore, any one of the probability information of the "first level" stimulation state, the probability information of the "second level" stimulation state, and the probability information of the "third level" stimulation state does not have to be included in the stimulation probability data.

[0156] As an example of generating a drive signal when using such stimulation probability data, the following case can be considered.

[0157] It is assumed here that the rise time of the tactile sensation providing unit 23 is 0.5 seconds. In addition, since the stimulation occurrence probability of the stimulation state "first level" at the time "0 seconds later" is "1.0", the tactile sensation is currently being provided at the first level.

[0158] In such a case, for example, suppose the creator specifies (selects) presentation policy α, which aims to reduce malfunctions (errors) in which the stimulus state is not changed correctly, and specifies a stimulus generation threshold of "0.6" for that presentation policy α.

[0159] 4, for example, since the rise time of the tactile sense providing unit 23 is 0.5 seconds, the signal generating unit 32 targets the period from 0 seconds to 0.5 seconds. Furthermore, since the stimulation state at the current time is the "first level," the signal generating unit 32 only refers to the probability information of the "second level" and the "third level" other than the "first level."

[0160] If the maximum value of the stimulation occurrence probability of each stimulation state indicated by the probability information of the "second level" and the "third level" at each time from 0 seconds to 0.5 seconds is equal to or greater than the stimulation occurrence threshold value of "0.6", the signal generating unit 32 changes the stimulation state of the tactile presentation to the stimulation state with the maximum value.

[0161] Here, the maximum value of the stimulation occurrence probabilities of the "second level" and the "third level" at each time from 0 seconds to 0.5 seconds is the stimulation occurrence probability of "0.8" for the "second level" at the time "0.5 seconds later." Since this stimulation occurrence probability of "0.8" is equal to or greater than the stimulation occurrence threshold value of "0.6," a drive signal for providing tactile presentation in a stimulation state of the "second level" is generated and output. As a result, the stimulation state of the tactile presentation is switched from the "first level" to the "second level."

[0162] Also, for example, suppose that the creator specifies (selects) a presentation policy β that aims to reduce malfunctions (errors) in which the stimulus state is erroneously changed, and specifies (selects) a stimulus generation threshold of "0.9" for that presentation policy β.

[0163] In this case, the signal generation unit 32, for example, in step S13 of Figure 4, targets the period from 0 seconds to 0.5 seconds, as in the case of presentation policy α, and refers to the probability information of the ``second level'' and ``third level'' during that period.

[0164] If the maximum value of the stimulation occurrence probability of each stimulation state indicated by the probability information of the "second level" and the "third level" at each time from 0 seconds to 0.5 seconds is equal to or greater than the stimulation occurrence threshold value of "0.9", the signal generating unit 32 changes the stimulation state of the tactile presentation to the stimulation state with the maximum value.

[0165] Here, the maximum value of the stimulation occurrence probabilities for the "second level" and the "third level" at each time from 0 seconds to 0.5 seconds is the stimulation occurrence probability of "0.8" for the "second level" at the time "0.5 seconds later." However, because this stimulation occurrence probability of "0.8" is less than the stimulation occurrence threshold value of "0.9," no change is made to the stimulation state, and it remains at "first level."

[0166] Furthermore, for example, suppose that the creator specifies a presentation policy γ that indicates that the timing of tactile presentation is particularly important, and specifies a stimulus generation threshold of "0.6" in the presentation policy γ.

[0167] In this case, for example, in step S13 of Figure 4, if the maximum value of the stimulation occurrence probability at each time indicated by the probability information of the "second level" and "third level" other than the "first level" is equal to or greater than the stimulation occurrence threshold value of "0.6", the signal generation unit 32 changes the stimulation state of the tactile presentation to the stimulation state that corresponds to that maximum value.

[0168] In this example, the stimulation occurrence probability of "0.9" for the "second level" at the time "0.6 seconds later," which is the maximum value, is greater than or equal to the stimulation occurrence threshold "0.6." Therefore, a drive signal is generated in advance to prepare for tactile presentation so that the stimulation state will be changed from "first level" to "second level" at the time "0.6 seconds later." Then, a drive signal is output at the timing "0.1 seconds later." As a result, the tactile presentation unit 23 will present a tactile sensation in a stimulation state of "second level" at the time "0.6 seconds later." In other words, the stimulation state of tactile presentation will be switched from "first level" to "second level."

[0169] FIG. 8 shows an example of stimulation probability data including probability information for each of a plurality of channels.

[0170] In this example, the stimulation probability data is composed of time information similar to that in the example of FIG. 5 and probability information for each of channels 1ch to 3ch.

[0171] The probability information for each channel is probability information for each of a plurality of different tactile presentation devices that make up the tactile presentation unit 23 .

[0172] As an example, a tactile presentation device worn on the right hand of a user receiving tactile presentation may be designated as a tactile presentation device of 1ch, i.e., channel 1, and a tactile presentation device worn on the user's left hand may be designated as a tactile presentation device of 2ch, i.e., channel 2, and a channel may be assigned to each location where the tactile presentation device is worn.

[0173] It is also conceivable that a channel is assigned to each type of tactile stimulus, i.e., each type (modality) of tactile presentation device, such as, for example, a tactile presentation device of 1ch (channel 1) for vibration stimulation and a tactile presentation device of 2ch (channel 2) for temperature stimulation.In addition, it is also conceivable that a channel is assigned to each user who is the target of the tactile presentation, such as, for example, a tactile presentation device of 1ch (channel 1) is attached to user A and a tactile presentation device of 2ch (channel 2) is attached to user B.

[0174] In this example, the time information is common to the probability information of each of the three channels, and the interval (width) between each time indicated by the time information is a fixed width of 0.1 seconds. The probability information for each channel is similar to that shown in Figure 5. Hereinafter, the interval (width) between each time indicated by the time information, i.e., the length between adjacent times, will also be referred to as the time interval.

[0175] Stimulation probability data consisting of such common time information and probability information for each of a plurality of channels associated with the time information is useful in cases where a plurality of tactile presentation devices are provided in the tactile presentation unit 23. Note that, although an example in which the number of channels is three has been described in Fig. 8, the number of channels, i.e., the number of channels for which probability information is stored in the stimulation probability data, can be any number equal to or greater than two.

[0176] FIG. 9 shows an example of stimulation probability data including time information and probability information for each of a plurality of channels.

[0177] In this example, the stimulation probability data is composed of time information and probability information for each channel from ch 1 to ch 3. The time information and probability information for each channel is the same as in the example of FIG.

[0178] In the example of Fig. 8, the time information is common to all channels, but in the example of Fig. 9, not only the probability information but also the time information is prepared for each channel, so that the intervals between times indicated by the time information (time intervals) can be made different for each channel.

[0179] For example, in Figure 9, the interval between adjacent times (time interval) indicated by the time information for 1ch (channel 1) is 0.1 seconds, while the interval between adjacent times in the time information for 2ch (channel 2) is 0.01 seconds, and the interval between adjacent times in the time information for 3ch (channel 3) is 0.5 seconds.

[0180] When multiple tactile presentation devices are provided in the tactile presentation unit 23, the rise speed may differ depending on the type of tactile presentation device, or the movement speed may differ depending on the location where the tactile presentation device is attached. In this case, the time interval requiring the stimulation occurrence probability, i.e., the appropriate time interval, may differ for each channel (tactile presentation device), but by configuring the stimulation probability data as shown in Figure 9, it is possible to use information on the appropriate time interval for each channel.

[0181] For example, the time interval in the time information for each channel may be determined based on at least one of the following: characteristics of the tactile presentation device such as rise speed; type of tactile presentation device (modal type); attachment location of the tactile presentation device; positional relationship such as relative distance and orientation between the attachment location of the tactile presentation device and the object; relative speed and acceleration between the attachment location of the tactile presentation device and the object; processing speed of the processing unit 31 and the signal generation unit 32; stimulation timing tolerance window; etc.

[0182] The intervals between adjacent times (time intervals) in the time information for each channel may be constant intervals (equal intervals) or may be unequal intervals.

[0183] 9, the processing unit 31 may generate time information and probability information at fixed time intervals for each channel determined by the characteristics of the tactile presentation device, etc. Alternatively, the processing unit 31 may generate time information and probability information while dynamically changing the intervals between times indicated by the time information for each channel according to the relative distance between the attachment portion and the object, etc.

[0184] For example, in a virtual space, if the distance between the part of the user where the haptic presentation device is worn (wearing part) and an object is equal to or greater than a predetermined threshold, the time interval may be lengthened, and if the distance between the wearing part and the object is less than the threshold, the time interval may be shortened. Also, if the rise speed of the haptic presentation device is fast, the time interval may be lengthened, and if the rise speed is slow, the time interval may be shortened.

[0185] FIG. 10 shows another example of stimulation probability data including time information and probability information for each of a plurality of channels.

[0186] 10, like the example shown in Fig. 9, the stimulation probability data includes time information and probability information for each of multiple channels, but the length (array length) of the time information and probability information differs for each channel. That is, the number of times (specified by the time information) included in the time information and the number of stimulation occurrence probabilities included in the probability information differ for each channel.

[0187] For example, on 1ch (channel 1), time information and probability information are generated for each of 11 times from "0 seconds later" to "1.0 seconds later." In other words, the sequence length of the time information and probability information is "11."

[0188] In contrast, the sequence length for 2ch (channel 2) is "14", and the sequence length for 3ch (channel 3) is "12".

[0189] These array lengths may be determined by the processing unit 31 based on at least one of the following: characteristics of the tactile presentation device such as rise speed, the type of tactile presentation device (modal type), the attachment position of the tactile presentation device, the positional relationship such as the relative distance and orientation between the attachment position of the tactile presentation device and the object, the relative speed and acceleration between the attachment position of the tactile presentation device and the object, and the processing speed of the processing unit 31 and the signal generation unit 32. Furthermore, the array length for each channel may be a predetermined fixed length, or may be dynamically changed.

[0190] For example, when the user's hand, which is the part where the tactile presentation device is worn, is moving vigorously, i.e., when the speed of the hand is equal to or greater than a threshold, the array length of the channel corresponding to the hand may be increased.Other examples include changing the array length according to the variance σ in the distribution of the stimulus occurrence probability, which is obtained when calculating the stimulus occurrence probability based on the relative speed between the part where the device is worn and the object, as described below.

[0191] Fig. 11 shows an example of stimulation probability data consisting of one channel's worth of time information and probability information. In particular, Fig. 11 shows an example in which the width (time interval) between adjacent times within one piece of time information is different. In other words, the intervals between multiple times indicated (specified) by the time information are unequal.

[0192] For example, the example shown at the top of the figure is an example in which the time interval between each time point changes depending on the rise speed (rise time) of the tactile presentation device and the stimulus timing allowable window.

[0193] In this example, the time interval between "0.2 seconds later" and "0.4 seconds later," which are close to the current time, is set to the shorter time interval of "0.1 seconds," while the time intervals for other time periods are set to "0.2 seconds," "0.4 seconds," "1.0 seconds," etc.

[0194] In particular, the time interval is shorter for time portions closer to the current time, i.e., the time resolution is higher, and the time interval is longer for time portions further away from the current time, i.e., the time resolution is lower. In this case, the position and length of the time portion with the short time interval of "0.1 seconds" are determined by the processing unit 31 based on the startup time of the tactile presentation device and the stimulation timing allowable window.

[0195] The example shown at the bottom of the figure is one in which the time interval between each time point changes depending on the probability of stimulation occurring, which is calculated based on factors such as the relative speed between the part of the user's body in the virtual space where the tactile presentation device is worn and the object.

[0196] In this example, the time interval is shorter, 0.05 seconds, between 0.4 seconds later and 0.7 seconds later, when the probability information indicates a high probability of a stimulus occurring, and the time intervals for other times are 0.2 seconds, 0.3 seconds, etc.

[0197] At times when the probability of a stimulus occurring is high, there is a high possibility that tactile presentation will be performed, so at such times, the time interval is shortened, i.e., the temporal resolution is made high, and at times when the probability of a stimulus occurring is low, the time interval is lengthened, i.e., the temporal resolution is made low.

[0198] In this case, the time intervals at each time point may not be constant but may be varied depending on the probability of the stimulus occurring. For example, in this example, the time intervals at the time points where the probability of the stimulus occurring is high are set to "0.05 seconds", but this time interval may also be varied depending on the probability of the stimulus occurring.

[0199] The time intervals between these times may be determined by the processing unit 31 based on at least one of the following: characteristics of the tactile presentation device such as rise speed; type of tactile presentation device (modal type); attachment location of the tactile presentation device; positional relationship such as relative distance and orientation between the attachment location of the tactile presentation device and the object; relative speed and acceleration between the attachment location of the tactile presentation device and the object; processing speed of the processing unit 31 and the signal generation unit 32; stimulus timing tolerance window; stimulus occurrence probability; etc.

[0200] Making the time intervals variable for each time within the stimulus probability data, i.e., within the time information, i.e., making the time intervals different, is particularly useful when the array length of the time information or probability information is fixed.

[0201] The stimulation probability data is output from the processing unit 31 to the signal generating unit 32 in the form of streaming data, for example, at predetermined time intervals.

[0202] An example of the format of stream data (stimulus probability data) in such a case is shown in FIG.

[0203] FIG. 12 shows an example of a format when the stimulation probability data has the data structure shown in FIG. 9, for example.

[0204] 12, for example, frame data Fch1 indicated by arrow Q51 is data for one channel of the stimulation probability data generated at a certain timing by the processing unit 31. In other words, data for one channel is data for one frame.

[0205] The frame data Fch1 is made up of a frame header placed at the beginning and probability information following the frame header.

[0206] The frame header contains channel information indicating the channel of the probability information, the time interval dt between adjacent times in the time information, and the data length Len of the frame data itself. In particular, in frame data Fch1, the channel information is "Ch1" indicating channel 1 (1ch).

[0207] 9, the time intervals between the times indicated by the time information for each channel are constant, so the time information for each channel can be restored using the time interval dt. In other words, the time interval dt functions as time information.

[0208] The frame data also stores probability information in a predetermined data format. For example, the data format of the probability information may be any format, such as PCM (Pulse Code Modulation). In this example, the probability information for channel 1 (1ch) is obtained by PCM-modulating data obtained by arranging the stimulus occurrence probabilities at 11 time points from "0 seconds later" to "1.0 seconds later" shown in FIG. 9 , and the resulting data is stored in frame data Fch1.

[0209] 9, there are three channels of data, and frame data is generated for each of these channels: frame data Fch1 for channel 1 (1ch), frame data Fch2 for channel 2 (2ch), and frame data Fch3 for channel 3 (3ch).

[0210] Then, as indicated by arrow Q52, stream data "Stream(i)" at a predetermined timing i is generated, which consists of frame data Fch1 to Fch3 and a stream header.

[0211] In this example, the stream header includes frame number information Nframe indicating the number of frame data included in the stream data, resolution information Bit indicating the resolution of the probability information in each frame data, i.e., the data size, and a data size Size indicating the total data length of the frame data included in the stream data. The stream header also includes the update frequency of the stimulation probability data, i.e., the refresh rate Fs of the stream data, and a timestamp "timestamp" indicating the time of transmission of the stream data by the processing unit 31. For example, the timestamp indicating the transmission time may be the value of a master clock within the processing unit 31, indicating the time at which the stream data (stimulation probability data) is output.

[0212] The processing unit 31 outputs one piece of stream data at a predetermined clock timing, i.e., at a predetermined time, and outputs the next piece of stream data one clock or several clocks later. Information indicating the output frequency of the stream data at this time is the refresh rate Fs.

[0213] Therefore, as indicated by arrow Q53, stream data is sequentially supplied from the processing unit 31 to the signal generating unit 32 at a predetermined output frequency.

[0214] Although an example where the time intervals between each time in the time information are equal has been described here, if the time intervals are unequal, the format of the stimulation probability data will be, for example, as shown in Fig. 13. That is, Fig. 13 shows an example format when the stimulation probability data has the data structure shown in Fig. 11, for example.

[0215] In this example, frame data for one channel is made up of a frame header, time information, and probability information.

[0216] The frame header includes channel information "Ch1" and the data length Len of the frame data. The time information and probability information are in a predetermined data format, such as PCM format.

[0217] For example, if the stimulation probability data is as shown in the upper part of Figure 11, the data obtained by arranging information indicating each of the 11 times from "0 seconds later" to "16.0 seconds later" shown in the upper part of Figure 11 is modulated using the PCM method, and the resulting data is stored in the frame data as time information.

[0218] The above frame data itself, or frame data to which a stream header shown in Fig. 12 is added as appropriate, is generated as stream data at a predetermined timing. In particular, when the stimulus probability data includes data for multiple channels, data consisting of the frame data for each channel and the stream header is generated as stream data.

[0219] <Calculation Example of Stimulus Occurrence Probability> A specific example of the calculation of the stimulus occurrence probability performed by the processing unit 31 in step S12 of FIG. 4 will be described.

[0220] The stimulus occurrence probability can be obtained by calculating, for example, the probability distribution shown in Fig. 14. In Fig. 14, the horizontal axis represents time (point in time) t, and the vertical axis represents the stimulus occurrence probability.

[0221] In this example, the probability distribution of the stimulus occurrence probability at each time t is a normal distribution. Note that the probability distribution is not limited to a normal distribution and can be freely set by the content creator or the like.

[0222] In this example, the processing unit 31 calculates the stimulus occurrence probability f(t) at time t by calculating the following formula (1): μ in formula (1) represents the mean value of the probability distribution (normal distribution) calculated by the following formula (2), and σ in formula (1) represents the variance of the probability distribution calculated by the following formula (3).

[0223]

[0224]

[0225]

[0226] In equations (2) and (3), v represents the relative velocity (hereinafter also referred to as relative velocity) between the object and the user in the virtual space, more specifically, the part of the user where the haptic presentation device is worn, and the object in the current frame (current time) of the content. Also, d in equation (2) represents the relative distance (hereinafter also referred to as relative distance) between the object and the user in the virtual space, more specifically, the part of the user where the haptic presentation device is worn, and the object in the current frame (current time) of the content. The current time is time t=0.

[0227] In equation (3), a, b, and c represent predetermined coefficients (constants), and v ｉ indicates the relative speed of the content at a time before the current time (past time), and v ａｖｅ is the relative velocity v at multiple times in the past ｉ The average value of

[0228] Furthermore, in equation (3), g(x eye ) is the user's gaze direction or gaze position in the virtual space at the current time of the content. eye is a function that takes x as an argument (variable), and h(t) is a predetermined function based on scenario data prepared in advance for the content. eye is the gaze direction.

[0229] For example, it can be seen that the mean value μ of the probability distribution shown by equation (2) is the time until the attachment part comes into contact with the object, which is calculated from the relative velocity v and relative distance d at the current time.

[0230] Furthermore, when we look at the variance σ shown in equation (3), the variance σ is the variance of the relative velocity v at the current time, the variance of the past relative velocity, and the line of sight direction x at the current time. eye , and varies depending on the pre-set scenario data.

[0231] In other words, as the relative velocity v at the current time increases, the variance σ also increases. ｉ -v ａｖｅ ) increases, the variance σ also increases.eye The variance σ changes depending on the scenario data.

[0232] For example, the line of sight x eye can be obtained from the user's eye position based on behavior information. ｉ -v ａｖｅ ), the relative distance d can be obtained from the content data of the current frame and past frames, in other words, from the behavior information and data for content generation.

[0233] Furthermore, the value of the function h(t) based on scenario data, for example, changes depending on the number and arrangement of objects in the virtual space at time t indicated by the scenario data, the content scene, etc. As an example, the scenario data includes information set by the creator indicating the type of scene at time t, such as whether the scene is a quiet scene or a lively scene, and the value of the function h(t) is assumed to be large in lively scenes.

[0234] As described above, the probability of stimulus occurrence changes at each time point depending on the user's movement (behavior) and the object's movement.

[0235] For example, the processing unit 31 calculates equation (1) based on behavior information, content data, scenario data, etc., and generates information consisting of stimulus occurrence probabilities f(t) at multiple discrete times t as probability information.

[0236] <Generation Example of Drive Signal> An example of generation of a drive signal by the signal generating section 32 will be described with reference to FIGS. 15 and 16. FIG.

[0237] For example, the signal generation unit 32 generates a drive signal to be output to the tactile sense providing unit 23 based on the stimulation probability data shown in Fig. 15. In Fig. 15, the horizontal axis represents time, and the vertical axis represents the stimulation occurrence probability. In this example, the stimulation occurrence threshold is set to th1.

[0238] As a method of generating a drive signal, for example, the signal generation unit 32 may compare the stimulus occurrence threshold th1 with the stimulus occurrence probability at each time based on information indicating the rise time of the tactile presentation device that constitutes the tactile presentation unit 23, and generate a drive signal according to the comparison result.

[0239] In this case, the signal generator 32 determines the time at which the tactile sensation will be presented by the tactile sensation presentation device if a drive signal is generated at the current time based on the information indicating the rise time of the tactile sensation presentation device, and designates this time as the rise time t1. In other words, the rise time t1 is determined as the time (area) at which the comparison with the stimulation generation threshold th1 is performed.

[0240] The signal generation unit 32 compares the stimulation occurrence probability at the rising time t1 with the stimulation occurrence threshold th1, and starts tactile stimulation if the stimulation occurrence probability at the rising time t1 is equal to or greater than the stimulation occurrence threshold th1. That is, the signal generation unit 32 generates a drive signal for providing a tactile sensation (applying a tactile stimulation) and outputs the drive signal to the tactile sensation presentation unit 23.

[0241] A stimulation timing tolerance window may be used to generate the drive signal. The time width indicated by the stimulation timing tolerance window is set to a time width in which a human being does not perceive a delay in response to a tactile stimulation.

[0242] For example, the signal generating unit 32 sets a period W1 having a time width indicated by the stimulation timing allowable window centered on the rising time t1 as the period (area) for comparing with the stimulation occurrence threshold th1. For example, the period W1 may be a period of ±0.2 seconds centered on the rising time t1.

[0243] If the maximum value of the stimulation occurrence probability at each time within the period W1 is equal to or greater than the stimulation occurrence threshold th1, the signal generating unit 32 determines that tactile stimulation is to be started, and generates a drive signal.

[0244] The signal generator 32 may also dynamically change the stimulus generation threshold th1 in accordance with input operations (designation operations) by the user, scenario data related to content prepared in advance by a creator or the like, the frequency of tactile presentation (tactile stimulation) in a period immediately before the current time (past period), etc. The stimulus generation threshold th1 may also be dynamically changed in accordance with the variance σ of the stimulus generation probability, the relative velocity v in the virtual space related to the content, the relative acceleration between the user and the object, the relative positional relationship between the user and the object, etc.

[0245] As an example, if the user inputs a value to change the stimulation threshold th1 or a change amount (correction amount) in the stimulation threshold th1, the stimulation threshold th1 may be changed in accordance with the change.

[0246] Furthermore, the stimulus generation threshold th1 may be changed depending on the type of scene of the content indicated by the scenario data, for example.

[0247] In this case, for example, in a quiet scene, the stimulation generation threshold th1 may be changed to a higher (larger) value to reduce non-contact / presentation actions that result in tactile presentation despite a non-contact state, whereas in a noisy scene, the stimulation generation threshold th1 may be changed to a lower (smaller) value to reduce contact / non-presentation actions that result in no tactile presentation despite a contact state.

[0248] Furthermore, for example, if the frequency of tactile presentation (tactile stimulation) in the immediately preceding period is high, the tactile stimulation becomes noise, so it is conceivable to change the stimulation generation threshold th1 to a higher value in order to reduce the frequency of tactile presentation. Conversely, if the frequency of tactile presentation in the immediately preceding period is low, it is conceivable to change the stimulation generation threshold th1 to a lower value in order to increase the frequency of tactile presentation.

[0249] Alternatively, a determination may be made as to whether or not to start tactile stimulation based on the difference between each time (timestamp) indicated by the time information included in the stimulation probability data and the current time indicated by the value of the master clock of the signal generation unit 32, and a drive signal may be generated based on the determination result.

[0250] For example, as shown in FIG. 16, assume that stimulation probability data indicated by arrow Q81 is input to the signal generating unit 32 at time t indicated by the value of the master clock of the signal generating unit 32.

[0251] The upper row of the stimulation probability data shown in FIG. 16 indicates time information, and the lower row of the stimulation probability data indicates probability information.

[0252] In this example, the processing unit 31 supplies stimulation probability data to the signal generating unit 32 every 0.2 seconds, and when the maximum value of the stimulation occurrence probability at each time within 0.5 seconds from the current time is equal to or greater than the stimulation occurrence threshold value of "0.6", it is determined that tactile presentation is to be performed, and a drive signal is generated.

[0253] In this case, at time t, the maximum value of the stimulation occurrence probability at each of the times "0.3 seconds later," "0.4 seconds later," and "0.5 seconds later" indicated by the time information of the stimulation probability data, "0.4," is less than the stimulation occurrence threshold value of "0.6." Therefore, at time t, it is not determined that tactile presentation is to be performed, and no drive signal is generated.

[0254] Furthermore, at time t+0.1 s, which is 0.1 seconds after time t, the stimulation probability data has not yet been updated. In other words, new stimulation probability data has not yet been generated, and therefore no stimulation probability data is transmitted from the processing unit 31 to the signal generating unit 32.

[0255] However, since the signal generating unit 32 can perform processing at a speed faster than the processing speed of the processing unit 31, more specifically, the speed at which the stimulation probability data is updated, it is possible to determine whether to present tactile sensation and generate a drive signal even at time "t+0.1s".

[0256] Therefore, for example, the stimulation probability data at time "t+0.1 s" may be estimated (predicted) based on the latest stimulation probability data held by the signal generation unit 32, and the resulting stimulation probability data may be used to determine whether to present a tactile sensation and to generate a drive signal.

[0257] Specifically, for example, the time information and probability information in the stimulation probability data at time t are shifted by 0.1 seconds, which is the actual time since the stimulation probability data at time t was supplied (generated), thereby generating stimulation probability data at time "t+0.1 seconds."

[0258] As a result, the stimulation probability data for time "t+0.1 s" is obtained, for example, as indicated by arrow Q82. Here, the stimulation probability at each time is predicted (estimated) under the assumption that the stimulation occurrence probability at each time indicated by the probability information of the stimulation probability data shifts as time passes.

[0259] Therefore, for example, the stimulation occurrence probability of "0.9" at time "0.6 seconds later" in the stimulation probability data at time t is shifted by 0.1 seconds, which is the actual elapsed time, and that stimulation occurrence probability of "0.9" becomes the stimulation occurrence probability at time "0.5 seconds later" in the stimulation probability data at time "t+0.1s."

[0260] At time "t+0.1 s", the signal generating unit 32 determines whether to provide a tactile sensation based on the stimulation probability data that it has updated through prediction (estimation) in this manner.

[0261] In this example, the maximum value of the stimulation occurrence probability "0.9" at each of the times "0.3 seconds later," "0.4 seconds later," and "0.5 seconds later" indicated by the time information of the updated stimulation probability data is equal to or greater than the stimulation occurrence threshold value "0.6." Therefore, the signal generating unit 32 determines that a tactile sensation should be presented, generates a drive signal for the tactile sensation presentation, and outputs the drive signal to the tactile sensation presentation unit 23.

[0262] In this way, when stimulation probability data is not supplied, the signal generator 32 estimates stimulation probability data for the current timing based on the last supplied stimulation probability data and the time elapsed since the stimulation probability data was supplied (generated), and performs processing using the estimated result. In this way, it is possible to prevent delays in the timing of tactile presentation.

[0263] Furthermore, at time "t+0.2 s", which is 0.1 seconds after time "t+0.1 s", new stimulation probability data, for example, as indicated by arrow Q83, is supplied from the processing unit 31 to the signal generating unit 32.

[0264] Therefore, the signal generating unit 32 performs a process of determining whether to provide a tactile sensation and generating a drive signal using this new stimulation probability data.

[0265] <Regarding Adjustment of Stimulus Generation Threshold> While it has been described that the stimulus generation threshold may be changed based on input operations by the user, scenario data, etc., the stimulus generation threshold may also be changed depending on the part of the user that receives the tactile sensation (presentation part) or the user profile. In other words, the stimulus generation threshold may be adjusted depending on the presentation part, etc.

[0266] In this case, for example, the signal generating unit 32 corrects the stimulation generation threshold using a correction value corresponding to at least one of the presentation area of ​​the tactile presentation based on the drive signal, the user profile, the type of tactile presentation (modal type), etc.

[0267] For example, since humans have different tactile spatial resolution and perception thresholds depending on the part of the body they contact, the quality of the user's experience can be improved by setting the stimulation threshold taking advantage of the characteristics of each part of the body they contact.

[0268] Therefore, for example, as shown in Fig. 17, the stimulation generation threshold may be different for each presentation part of the user. In other words, the stimulation generation threshold may be adjusted (corrected) for each presentation part.

[0269] In the example of FIG. 17, examples of correction values ​​of stimulation generation thresholds for the user's arms, hands, and feet as presentation sites are shown.

[0270] Specifically, for the user's arm, the preset stimulation threshold is corrected by a correction value of "-0.2." For example, if the preset stimulation threshold is "0.6," the corrected stimulation threshold of "0.4" is used when generating a drive signal to be supplied to the tactile presentation device worn on the user's arm.

[0271] Similarly, the stimulation threshold for the user's hands is corrected by a correction value of "+0.3", and the stimulation threshold for the user's feet is corrected by a correction value of "-0.4".

[0272] It is generally known that the human hand is highly sensitive to stimuli. Therefore, for the presentation site "hand," a correction is made to raise (increase) the stimulation threshold in order to reduce non-contact / presentation actions that result in tactile presentation even in a non-contact state.

[0273] In contrast, for the arms and legs, which are less sensitive to stimuli than the hands, a correction is made to lower (reduce) the stimulation threshold in order to reduce contact / non-presentation actions in which no tactile sensation is presented despite a contact state.

[0274] Furthermore, as shown in FIG. 18, for example, the stimulation generation threshold may be adjusted (corrected) according to a user profile, which is information about the user receiving the tactile presentation, such as the age and sex of the user.

[0275] Since humans have different tactile spatial resolution and perception thresholds depending on their age and gender, if the stimulation generation threshold is set taking into account these characteristics such as age, the quality of the user's experience can be improved, as in the example of Figure 17.

[0276] In this example, the correction value of the stimulation threshold for each combination of user age and gender is displayed as information about the user. For example, the stimulation threshold for a user who is a "teenager" and a "male" is corrected by a correction value of "+0.3."

[0277] It is generally known that women and young people have high tactile sensitivity. Therefore, for female and younger users, a correction is made to raise the stimulation threshold to reduce non-contact / presentation actions, which result in tactile presentation even when there is no contact.

[0278] In contrast, since men and the elderly have low tactile sensitivity, for male users and older users, a correction is made to lower the stimulation generation threshold in order to reduce contact / non-presentation actions in which no tactile sensation is presented despite a contact state.

[0279] Priority may be set for presentation parts, user profiles, and modalities (types of tactile presentation).

[0280] It is known that sensitivity to tactile presentation varies depending on the presentation area, user profile, modality, etc. Therefore, for example, content creators define perceptual priorities taking into account such differences in sensitivity (perception). Then, for example, using a correction value according to the defined priority, the stimulus generation threshold is appropriately corrected for at least one combination of presentation area, user profile, and modality. In this case, the priority is ultimately (effectively) registered as the amount of change in the stimulus generation threshold, i.e., the correction value, but the creator can confirm (recognize) it as a change in the priority label.

[0281] Alternatively, the stimulus generation threshold may be corrected (adjusted) depending on the type (modal) of tactile presentation, such as hot / cold, vibration, etc. In other words, a correction value for the stimulus generation threshold may be determined for each modal.

[0282] For example, cold stimuli have characteristics such as a lower perception threshold compared to warm stimuli, so the quality of the tactile presentation experience can be improved by determining a correction value according to this characteristic and correcting (adjusting) the stimulus generation threshold.

[0283] The correction (adjustment) of the stimulation threshold may be performed based on a combination of two or more of the presentation region, the user profile, and the modality.

[0284] Furthermore, adjustment of the stimulation generation threshold for each presentation part, each user profile, and each modality, i.e., determination of correction values ​​and priorities, may all be set manually by the content creator, or may be performed by the signal generation unit 32 or the like based on tactile presentation device information, user profile information, content data, scenario data, etc. In such cases, appropriate correction values ​​and priorities are determined depending on, for example, the content and the arrangement (wearing position) of the tactile presentation device.

[0285] When creating content, the creator may be able to set various settings regarding the stimulus occurrence probability and the stimulus occurrence threshold.

[0286] In such a case, the creator's device may display, for example, a screen (UI (User Interface) screen) shown in FIG.

[0287] In the example shown in Figure 19, the creator can select a probability distribution, such as a normal distribution, to use when calculating the stimulus occurrence probability by operating the drop-down list provided in the area indicated by arrow Q91 on the screen.

[0288] Additionally, for example, the creator can specify (adjust) the stimulus generation threshold by operating the slider indicated by arrow Q92. Note that it may be possible to use intuitive labels such as "stricter presentation" or "loose presentation" to allow the creator to more easily adjust the stimulus generation threshold.

[0289] Furthermore, for example, by operating the slider indicated by arrow Q93, the creator can specify the positive and negative widths of the stimulation timing tolerance window, and by operating the slider indicated by arrow Q94, the creator can specify (adjust) the correction value of the stimulation threshold for each tactile presentation part, such as the hand or arm. By operating the slider indicated by arrow Q95, the creator can also specify (adjust) the correction value of the stimulation threshold for each user profile.

[0290] These various settings may be configured to be performed by the user rather than the creator. In such a case, for example, the control unit 22 displays the screen shown in Fig. 19 on the display unit 24, and the control unit 22 selects a probability distribution and changes the correction value of the stimulus generation threshold based on a signal supplied from the input unit 21 in response to a user operation.

[0291] <Regarding Allocation of Each Process> Incidentally, each process for playing back content and providing tactile sensations may be performed by the information processing device 11 alone, or may be performed by the information processing device 11 and another device.

[0292] For example, each process may be shared between the information processing device 11, which is a terminal that functions as a client that plays back the content, and a server such as a cloud server connected to the information processing device 11.

[0293] Regarding whether each process is performed on the client or server side, there are four possible patterns shown in FIG.

[0294] In FIG. 20, pattern 1 is an example in which the input unit 21, the processing unit 31, the signal generating unit 32, and the tactile sense providing unit 23 are all provided on the client side.

[0295] Pattern 2 is an example in which the input unit 21, the signal generation unit 32, and the tactile sense providing unit 23 are provided on the client side, and only the processing unit 31 is provided on the server side.

[0296] Pattern 3 is an example in which the input unit 21, the processing unit 31, and the tactile sense providing unit 23 are provided on the client side, and only the signal generating unit 32 is provided on the server side.

[0297] Pattern 4 is an example in which the input unit 21 and the tactile sense providing unit 23 are provided on the client side, and the processing unit 31 and the signal generating unit 32 are provided on the server side.

[0298] In the case of any of Patterns 2 to 4, the information processing device 11 is connected to a server 71 via a network, as shown in Fig. 21, for example, and a tactile presentation system is realized by the information processing device 11 and the server 71. The server 71 may be a single device, or may be realized by multiple devices constituting a cloud.

[0299] In the example of FIG. 21, the server 71 includes a control unit 81 and a communication unit 82 .

[0300] The control unit 81 controls the overall operation of the server 71. The communication unit 82 communicates with the information processing device 11 via the network.

[0301] For example, in pattern 2, the control unit 22 on the client (information processing device 11 ) side does not implement the function of the processing unit 31 , and the control unit 81 of the server 71 functions as the processing unit 31 .

[0302] In this case, the behavior information obtained by the information processing device 11 is transmitted to the server 71 by the communication unit 25. Then, the communication unit 82 of the server 71 receives the behavior information transmitted from the information processing device 11, and the control unit 81 generates stimulation probability data based on the received behavior information, etc.

[0303] This stimulation probability data is transmitted by the communication unit 82 to the information processing device 11, received by the communication unit 25, and supplied from the communication unit 25 to the control unit 22 (signal generation unit 32). Note that tactile data may also be transmitted together with the stimulation probability data.

[0304] In pattern 3, the control unit 22 on the client (information processing device 11 ) side does not realize the function of the signal generation unit 32 , and the control unit 81 of the server 71 functions as the signal generation unit 32 .

[0305] In this case, the stimulation probability data and tactile data obtained by the information processing device 11 are transmitted to the server 71 by the communication unit 25 and received by the communication unit 82 of the server 71. The control unit 81 then generates a drive signal based on the received stimulation probability data and tactile data and, as appropriate, tactile presentation device information obtained from the information processing device 11, etc. This drive signal is transmitted by the communication unit 82 to the information processing device 11, received by the communication unit 25, and supplied from the communication unit 25 to the tactile presentation unit 23 via the control unit 22.

[0306] In pattern 4, the control unit 22 on the client (information processing device 11) side does not realize the functions of the processing unit 31 and the signal generation unit 32, and the control unit 81 of the server 71 functions as the processing unit 31 and the signal generation unit 32.

[0307] In this case, behavior information obtained by the information processing device 11 is transmitted to the server 71 by the communication unit 25 and received by the communication unit 82 of the server 71. The control unit 81 then generates stimulation probability data using the received behavior information and, as appropriate, tactile presentation device information obtained from the information processing device 11, etc., and generates a drive signal from the stimulation probability data. This drive signal is transmitted by the communication unit 82 to the information processing device 11, received by the communication unit 25, and supplied from the communication unit 25 to the tactile presentation unit 23 via the control unit 22.

[0308] In addition, when some of the processing such as patterns 2 to 4 is performed on the server 71 side, the input unit 21 and the tactile sense providing unit 23 do not need to perform processing for the same user.

[0309] That is, behavior information of a certain user may be transmitted to the information processing device 11 of another user, or a drive signal may be transmitted to the information processing device 11 of each of a plurality of users.

[0310] A specific example will be described below. Note that, hereinafter, a certain user will be referred to as user A, and a user different from user A will be referred to as user B. Note that the number of users who become user A and user B is not limited to one, and may be multiple.

[0311] (Input Unit on User A's Side, Tactile Sense Presentation Unit on User A's Side) First, a case can be considered in which both the input unit 21 and the tactile sense presentation unit 23 are provided on the user A's side.

[0312] In this example, it is assumed that a single user A receives haptic feedback in a first space, which is a VR environment or a virtual space. Specifically, it is conceivable that the user A uses a device including a sensor as the input unit 21 and an actuator as the haptic feedback unit 23 to experience standalone content.

[0313] (Input Unit on User A's Side, Tactile Sense Providing Unit on User B's Side) A case can be considered in which the input unit 21 is provided on the user A's side, and the tactile sense providing unit 23 is provided on the user B's side.

[0314] In this example, for example, multiple users exist, and the movement of user A in a first space is sensed using a sensor or the like as the input unit 21, and a drive signal is generated by performing appropriate processing based on the behavior information obtained as a result. Then, based on the drive signal, a tactile sensation is presented to user B in a second space different from the first space.

[0315] Specifically, for example, at a predetermined timing, such as the moment when user A wearing a motion capture device as input unit 21 presses a button in the virtual space, a tactile sensation is presented to user B wearing a tactile vest as tactile presentation unit 23 through vibration stimulation.

[0316] (Input Unit on User A Side, Tactile Sense Providing Unit on User A and User B Sides) A ​​case can be considered in which the input unit 21 is provided on the user A side, and the tactile sense providing unit 23 is provided on both the user A side and the user B side.

[0317] In this example, for example, multiple users exist, and the movement of user A in a first space is sensed using a sensor or the like as the input unit 21, and a drive signal is generated by performing appropriate processing based on the behavior information obtained as a result. Then, based on the drive signal, a tactile sensation is presented to user A in the first space and user B in the second space.

[0318] Specifically, for example, at a predetermined timing, such as the moment when user A wearing a motion capture device as input unit 21 presses a button in the virtual space, tactile sensations are presented to user A and user B wearing a tactile vest as tactile presentation unit 23 through vibration stimulation.

[0319] (Input Units on User A Side and User B Side, Tactile Presentation Unit on User A Side) A case can be considered in which the input unit 21 is provided on both the user A side and the user B side, and the tactile presentation unit 23 is provided on the user A side.

[0320] In this example, for example, multiple users are present, and the movements of user A in a first space are sensed using a sensor or the like as the input unit 21, and the movements of user B in a second space are sensed by the input unit 21. A drive signal is generated by performing appropriate processing based on the behavior information obtained as a result of these sensing operations. In this case, stimulation probability data is calculated from the behavior information of user A and the behavior information of user B, and a drive signal is generated. Then, a tactile sensation is presented to user A in the first space based on the drive signal.

[0321] Specifically, for example, when user A wearing a device that has a sensor as the input unit 21 and an actuator as the tactile presentation unit 23 and user B wearing a device that only has a sensor as the input unit 21 high-five, a tactile presentation through vibration stimulation may be provided only to user A.

[0322] (Input unit on user A side and user B side, tactile presentation unit on user A side and user B side) It is conceivable that the input unit 21 is provided on both user A side and user B side, and the tactile presentation unit 23 is also provided on both user A side and user B side.

[0323] In this example, for example, multiple users are present, and the movement of user A in the first space is sensed using a sensor or the like as the input unit 21, and the movement of user B in the second space is sensed by the input unit 21. A drive signal is generated by performing appropriate processing based on the behavior information obtained as a result of these sensing operations. In this case, stimulation probability data is calculated from the behavior information of user A and the behavior information of user B, and drive signals for user A and user B, or a drive signal common to user A and user B, are generated. Then, based on the drive signals, a tactile sensation is presented to user A in the first space and user B in the second space.

[0324] Specifically, for example, when user A wearing a device that has a sensor as the input unit 21 and an actuator as the tactile presentation unit 23 and user B wearing a device that has a sensor as the input unit 21 and an actuator as the tactile presentation unit 23 high-five each other, tactile sensations are presented to both user A and user B through vibration stimulation.

[0325] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0326] FIG. 22 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0327] In the computer, a CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0328] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0329] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0330] In a computer configured as described above, the CPU 501 loads, for example, a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the above-described series of processes.

[0331] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0332] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0333] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0334] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0335] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0336] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0337] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0338] Furthermore, the present technology can also be configured as follows.

[0339] (1) An information processing device including a processing unit that generates, based on behavior information indicating a user's behavior, probability data regarding the probability of causing a haptic presentation at a future time including the time when the behavior information was acquired. (2) The information processing device described in (1), in which the processing unit generates the probability data based on at least one of a position, posture, direction of movement, information about the body, speed, and acceleration of an object in space. (3) The information processing device described in (2), in which the processing unit generates the probability data based on at least one of a position, posture, direction of movement, information about the body, speed, and acceleration of the object at a predetermined time and at least one of a position, posture, direction of movement, information about the body, speed, and acceleration of the object prior to the predetermined time. (4) The information processing device described in (2) or (3), in which the behavior information is information that can estimate or identify at least one of a position, posture, direction of movement, speed, acceleration, line of sight, and information about the body of the user in the space. (5) The information processing device according to any one of (2) to (4), wherein the processing unit generates the probability data based on scenario data generated for content including at least one of an image of the space including the object as a subject and a sound of the object. (6) The information processing device according to any one of (2) to (5), wherein the probability data includes time information capable of identifying each of a plurality of future times including the time of acquisition of the behavior information, and probability information indicating the probability at each of the times. (7) The information processing device according to (6), wherein the interval between the plurality of times identified by the time information is determined based on at least one of a characteristic of a tactile presentation device that provides the tactile presentation, a wearing position of the tactile presentation device, a type of tactile presentation, a relative positional relationship between the user and the object, a relative velocity between the user and the object, a relative acceleration between the user and the object, a processing speed of the processing unit, a processing speed of a signal processing unit that generates a drive signal for the tactile presentation device based on the probability data, and an allowable time delay for the timing of the tactile presentation.(8) The information processing device according to (6) or (7), wherein the number of times specified by the time information is determined based on at least one of a characteristic of a tactile presentation device that provides the tactile presentation, a wearing position of the tactile presentation device, a type of tactile presentation, a relative positional relationship between the user and the object, a relative velocity between the user and the object, a relative acceleration between the user and the object, a processing speed of the processing unit, and a processing speed of a signal processing unit that generates a drive signal for the tactile presentation device based on the probability data. (9) The information processing device according to any one of (6) to (8), wherein the probability data includes, as the probability information, first probability information indicating a probability of providing a first tactile presentation and second probability information indicating a probability of providing a second tactile presentation. (10) The information processing device according to (9), wherein the first tactile presentation and the second tactile presentation are tactile presentations of different intensities provided by the same tactile presentation device, or tactile presentations provided by different tactile presentation devices. (11) The information processing device according to (9) or (10), wherein the probability data includes, as the time information, first time information of the first probability information and second time information of the second probability information. (12) The information processing device according to (11), wherein the number of times specified by the first time information is different from the number of times specified by the second time information. (13) The information processing device according to (11) or (12), wherein the intervals between the multiple times specified by the first time information are different from the intervals between the multiple times specified by the second time information. (14) The information processing device according to any one of (6) to (13), wherein the intervals between the multiple times specified by the time information are unequal.(15) The information processing device according to (14), wherein the interval between adjacent times specified by the time information is determined based on at least one of the probability at the time, a characteristic of the tactile presentation device that provides the tactile presentation, a wearing position of the tactile presentation device, the type of tactile presentation, the relative positional relationship between the user and the object, the relative velocity between the user and the object, the relative acceleration between the user and the object, the processing speed of the processing unit, the processing speed of a signal processing unit that generates a drive signal for the tactile presentation device based on the probability data, and an allowable time for a timing delay of the tactile presentation. (16) An information processing method, wherein an information processing device generates, based on behavior information indicating user behavior, probability data regarding the probability of providing tactile presentation at a future time including the time when the behavior information was acquired. (17) A program that causes a computer to execute processing including the step of generating, based on behavior information indicating user behavior, probability data regarding the probability of providing tactile presentation at a future time including the time when the behavior information was acquired. (18) An information processing device comprising: a signal generation unit that generates a drive signal for performing a haptic presentation based on probability data regarding the probability of performing the haptic presentation at a future time including the time when behavior information indicating user behavior is acquired. (19) The information processing device described in (18), wherein the signal generation unit generates the drive signal based on the probability data and a predetermined threshold. (20) The information processing device described in (19), wherein the signal generation unit changes the threshold based on at least one of a user input, scenario data of content related to the haptic presentation, the frequency of occurrence of the haptic presentation, the variance of the probability, a relative positional relationship between the user and an object in a space related to the content, a relative velocity between the user and the object, and a relative acceleration between the user and the object. (21) The information processing device described in (19) or (20), wherein the signal generation unit corrects the threshold with a correction value according to at least one of a presentation portion of the haptic presentation, a user profile, and a type of the haptic presentation.(22) The information processing device according to any one of (18) to (21), wherein the signal generation unit generates the drive signal based on the probability data and an allowable time for a delay in the timing of the tactile presentation. (23) The information processing device according to any one of (18) to (22), wherein the signal generation unit generates the drive signal based on the probability data and information related to a rise speed of a tactile presentation device that performs the tactile presentation. (24) The information processing device according to any one of (18) to (23), wherein the signal generation unit generates the drive signal based on the probability data and an elapsed time since the probability data was supplied. (25) An information processing method, wherein an information processing device generates the drive signal for performing the tactile presentation based on probability data related to a probability of performing tactile presentation at a future time including a time when behavior information indicating a user's behavior is acquired. (26) A program that causes a computer to execute a process including a step of generating a drive signal for providing a tactile sensation based on probability data regarding the probability of providing the tactile sensation at a future time including the time when behavior information indicating a user's behavior is acquired.

[0340] REFERENCE SIGNS LIST 11 information processing device, 21 input unit, 22 control unit, 23 tactile presentation unit, 24 display unit, 25 communication unit, 26 acoustic output unit, 31 processing unit, 32 signal generation unit

Claims

1. An information processing device comprising a processing unit that generates, based on behavior information indicating a user's behavior, probability data regarding the probability of providing a tactile sensation at a future time including the time when the behavior information is acquired.

2. The information processing device according to claim 1, wherein the processing unit generates the probability data based on at least one of the object's position, posture, movement direction, information about its body, speed, and acceleration in space.

3. The information processing device of claim 2, wherein the processing unit generates the probability data based on at least one of the object's position, posture, direction of movement, information about the body, speed, and acceleration at a predetermined time, and at least one of the object's position, posture, direction of movement, information about the body, speed, and acceleration prior to the predetermined time.

4. The information processing device according to claim 2, wherein the behavior information is information that can estimate or identify at least one of the user's position, posture, direction of movement, speed, acceleration, gaze direction, and information about the body in the space.

5. The information processing device according to claim 2, wherein the processing unit generates the probability data based on scenario data generated for content consisting of at least one of an image of the space including the object as a subject and a sound of the object.

6. The information processing device according to claim 2, wherein the probability data includes time information that can identify each of a plurality of future times including the time when the behavior information was acquired, and probability information that indicates the probability at each of the times.

7. The information processing device of claim 6, wherein the intervals between the multiple times specified by the time information are determined based on at least one of the characteristics of the tactile presentation device that provides the tactile presentation, the location where the tactile presentation device is worn, the type of tactile presentation, the relative positional relationship between the user and the object, the relative speed between the user and the object, the relative acceleration between the user and the object, the processing speed of the processing unit, the processing speed of a signal processing unit that generates a drive signal for the tactile presentation device based on the probability data, and the allowable time for delay in the timing of the tactile presentation.

8. The information processing device of claim 6, wherein the number of times specified by the time information is determined based on at least one of the characteristics of the tactile presentation device that provides the tactile presentation, the location where the tactile presentation device is worn, the type of tactile presentation, the relative positional relationship between the user and the object, the relative speed between the user and the object, the relative acceleration between the user and the object, the processing speed of the processing unit, and the processing speed of a signal processing unit that generates a drive signal for the tactile presentation device based on the probability data.

9. An information processing device as described in claim 6, wherein the probability data includes, as the probability information, first probability information indicating the probability of causing a first tactile presentation and second probability information indicating the probability of causing a second tactile presentation.

10. The information processing device according to claim 9, wherein the first tactile presentation and the second tactile presentation are tactile presentations of different intensities by the same tactile presentation device, or tactile presentations by different tactile presentation devices.

11. The information processing device according to claim 9, wherein the probability data includes, as the time information, first time information of the first probability information and second time information of the second probability information.

12. The information processing device according to claim 11, wherein the number of times specified by the first time information is different from the number of times specified by the second time information.

13. The information processing device according to claim 11, wherein the intervals between the multiple times specified by the first time information are different from the intervals between the multiple times specified by the second time information.

14. The information processing device according to claim 6, wherein the time intervals between the multiple times specified by the time information are unequal.

15. The information processing device of claim 14, wherein the interval between adjacent times specified by the time information is determined based on at least one of the probability at the time, the characteristics of the tactile presentation device that provides the tactile presentation, the location where the tactile presentation device is worn, the type of tactile presentation, the relative positional relationship between the user and the object, the relative velocity between the user and the object, the relative acceleration between the user and the object, the processing speed of the processing unit, the processing speed of a signal processing unit that generates a drive signal for the tactile presentation device based on the probability data, and the allowable time for delay in the timing of the tactile presentation.

16. An information processing method in which an information processing device generates, based on behavior information indicating a user's behavior, probability data regarding the probability of causing a tactile presentation at a future time including the time when the behavior information is acquired.

17. A program that causes a computer to execute a process including a step of generating probability data regarding the probability of providing a tactile sensation at a future time including the time when the behavior information indicating the user's behavior is acquired, based on the behavior information.

18. An information processing device comprising a signal generation unit that generates a drive signal for providing a tactile sensation based on probability data regarding the probability of providing a tactile sensation at a future time including the time when behavior information indicating a user's behavior is acquired.

19. The information processing device according to claim 18, wherein the signal generating section generates the drive signal based on the probability data and a predetermined threshold value.

20. The information processing device of claim 19, wherein the signal generation unit varies the threshold based on at least one of user input, scenario data of the content related to the tactile presentation, the frequency of occurrence of the tactile presentation, the variance of the probability, the relative positional relationship between the user and the object in the space related to the content, the relative speed between the user and the object, and the relative acceleration between the user and the object.

21. The information processing device according to claim 19, wherein the signal generating unit corrects the threshold value with a correction value according to at least one of the presentation portion of the tactile presentation, the user profile, and the type of the tactile presentation.

22. The information processing device according to claim 18, wherein the signal generating unit generates the drive signal based on the probability data and an allowable time for a lag in the timing of the tactile presentation.

23. The information processing device according to claim 18, wherein the signal generating unit generates the drive signal based on the probability data and information relating to the rise speed of the tactile presentation device that provides the tactile presentation.

24. The information processing device according to claim 18, wherein the signal generating section generates the drive signal based on the probability data and the time elapsed since the probability data was supplied.

25. An information processing method in which an information processing device generates a drive signal for providing a tactile presentation based on probability data regarding the probability of providing the tactile presentation at a future time including the time when behavior information indicating a user's behavior is acquired.

26. A program that causes a computer to execute processing including a step of generating a drive signal for providing a tactile sensation based on probability data regarding the probability of providing the tactile sensation at a future time including the time when behavior information indicating the user's behavior is acquired.

Images