Sensory presentation device

Abstract

To eliminate the deviation of perceptible timing, when presenting a specific content such that a visual sense, an auditory sense and a tactile sense are linked together.SOLUTION: When providing contents in which voice and / or video, and tactile sense effect are linked together, a sensibility presentation device delays timing in which each signal is outputted to each device, by predetermined delay time, respectively, and outputs the signal from each control unit, depending on perception time until a human perceives each of an auditory sense and / or a visual sense, a passive tactile sense and an active tactile sense.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to control for appropriately adjusting a sensation providing device depending on the situation. [Background technology]

[0002] Various types of tactile presentation devices have been proposed that allow users to experience tactile effects in conjunction with content. Active tactile effects, such as vibration, pressure, wind, humidity, and heat, that can be felt even when the user is stationary, have already been provided in conjunction with video content such as movies. Furthermore, for interface devices that respond to human movements, consideration has been given to providing passive tactile effects that can be felt in response to the user's movements, such as the resistance felt when gripping a device in the hand, such as hardness or softness. Patent Document 1 proposes a system that uses feedback from sensors to adjust the intensity of the tactile sensations provided by a tactile presentation device.

[0003] Furthermore, when tactile presentation devices provide content, they rarely provide only tactile information; rather, it is generally considered that a better experience can be provided to the user by providing tactile information that is suited to the visual and auditory information in parallel with the information provided visually and aurally. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-29563 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when attempting to provide at least one of the auditory and visual senses in addition to the tactile sense, the timing of the user's experience may not be synchronized, resulting in a feeling of discomfort with the content.

[0006] Therefore, the object of this invention is to improve the quality of experience of content by synchronizing the timing of the user's experience when providing sensations of specific content using an audio output device, a video output device, or both together with a haptic output device. [Means for solving the problem]

[0007] This invention is an auditory control unit that outputs sound to an audio output device (speaker: earphone); A visual control unit that outputs images to an optical output device (liquid crystal / organic EL monitor), and and, A sensory presentation device has a haptic control unit capable of outputting both an active haptic effect that acts regardless of a user's motion and a passive haptic effect that acts in response to a user's motion, and selectively provides an active haptic sensation that acts regardless of a user's motion and a passive haptic sensation that acts in response to a user's motion, When providing content that links the audio and / or video with the haptic effect, the timing of outputting each signal to each device is determined as follows: The auditory and / or visual sense, the active tactile sense, and the passive tactile sense are each output from each control unit with a predetermined delay time depending on the perception time from transmission by the control unit to perception by a human. The above problem was solved by using a sensory presentation device.

[0008] The audio output device, optical output device, and haptic output device included in or connected to the sensory presentation device have different time lags depending on the connection configuration and characteristics of each device, from receiving a signal sent from the control unit to expressing that signal as audio, video, or haptic effects. More precisely, it was found that the time it takes for a user to perceive an active haptic effect that occurs regardless of the user's actions differs from the time it takes for a passive haptic effect that occurs in response to the user's actions.

[0009] For example, displays on an OLED monitor directly attached to a smartphone are processed relatively quickly due to the device's short circuit and fast response time. In contrast, audio playback from earphones connected to a smartphone via near-field wireless communication requires encryption, wireless transmission, wireless reception, and decryption, which naturally slows the audio playback time. Haptic devices vary in speed, depending on the actuators and other media and mechanisms used to implement them. The time lag until the actual effect is realized also varies between active and passive devices. Furthermore, the time it takes for the user's brain to perceive the stimuli perceived by the eardrum, retina, nerves, etc., produced by the audio, video, and haptic effects varies depending on the conditions. By delaying the output from each control unit by a predetermined delay time so that the final perception timing is synchronized based on the time lag between the output from the control unit and the time lag due to human perception, it is possible to reduce or eliminate the lag in the timing of these sensations actually perceived by the user.

[0010] The sensation presentation device according to the present invention includes a first tactile output device that actively operates to provide the active tactile sensation; a second haptic output device that operates passively in response to an external force to provide the passive haptic sensation; An embodiment having the following may be adopted.

[0011] Furthermore, the sensation presentation device according to the present invention has a haptic attribute that defines whether the haptic effect is the active haptic sensation or the passive haptic sensation; An embodiment may be adopted in which the predetermined delay time is invoked in response to the haptic attribute.

[0012] a control device connected to the optical output device and the audio output device; a sensation presentation device including a haptic output device connected to the control device and having the haptic output device and the haptic control unit, The control device A delay time table is provided in which a delay time for each sensation is set, and a transmitting means is implemented to transmit each signal from the control unit in accordance with the delay time value of the delay time table. The embodiment can be adopted. Here, for each of the senses, an embodiment can be adopted in which delay times are set for active haptic effects and passive haptic effects.

[0013] The sensation presentation method according to the present invention comprises: an auditory control unit that outputs sound to an audio output device; a visual control unit that outputs an image to an optical output device; and and, A sensory presentation method using a sensory presentation device, which has a haptic control unit capable of outputting both an active haptic effect that acts regardless of a user's motion and a passive haptic effect that acts in response to a user's motion, and selectively provides an active haptic effect that acts regardless of a user's motion and a passive haptic effect that acts in response to a user's motion, comprising: When providing content that links the audio and / or video with the haptic effect, the timing of outputting each signal to each device is determined as follows: The above problem can be solved by a sensory presentation method characterized in that the auditory and / or visual senses, the active tactile sense, and the passive tactile sense are each output from each control unit with a predetermined delay time depending on the perception time from transmission from the control unit to perception by a human. [Effects of the Invention]

[0014] This invention makes it possible to provide both tactile effects that are actively provided by the device and tactile effects that are passively provided by the device, and when a user uses a sensory presentation device that outputs at least one of audio and video, the time lag that occurs when experiencing the provided content with multiple senses can be reduced or eliminated, thereby reducing the sense of discomfort when using the content, improving the sense of immersion, and improving the accuracy of the presented sensations. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a functional block diagram of an embodiment of a sensory presentation device according to the present invention. [Figure 2] (a) Example of a delay time table, (b) Example of a delay time table based on vision, (c) Example of a delay time table based on a combination of senses [Figure 3] FIG. 1 is a diagram illustrating an example of data processing between a control device and a haptic output device in an example of a sensory presentation device according to the present invention. [Figure 4] Conceptual diagram of an MRF device, an example of a haptic output device used in this invention. [Figure 5] 1 is a conceptual diagram of a tap unit, which is an example of a sensory presentation device according to the present invention. [Figure 6] FIG. 1 is a process flow example when a user uses the sensory presentation device according to the present invention. [Figure 7] A continuation of Figure 6: Example of processing flow [Figure 8] A continuation of Figure 6: Example of processing flow [Figure 9] Example of object selection on the monitor output by the app DETAILED DESCRIPTION OF THE INVENTION

[0016] The present invention will be described in detail below. The present invention is directed to an audio control unit that outputs audio to an audio output device, a visual control unit that outputs video to an optical output device, and and, The present invention relates to a sensory presentation device having a haptic control unit capable of outputting both an active haptic effect that acts regardless of the user's actions and a passive haptic effect that acts in response to the user's actions to a haptic output device. It also relates to a sensory presentation method using the sensory presentation device. Here, "outputting both" does not necessarily mean outputting them simultaneously, but also includes cases where either of the two can be selected and output depending on the situation. Of course, it is more preferable to be able to selectively output both simultaneously.

[0017] In this invention, hearing refers to one of the five senses, and refers to a sensation experienced through the eardrum. In this invention, sound refers to an element that can be perceived through hearing. Specific examples include, but are not limited to, human voices, animal cries, sound effects, and music. Essentially, sound is experienced through the eardrum by vibrating the air using an audio output device such as a speaker, headphones, or earphones, which reproduces sound from an electrical signal. Note that this also includes sounds that are not vibrated by air but are instead vibrated by another medium, such as bone conduction.

[0018] In this invention, "vision" refers to one of the five senses, a sensation experienced through the retina of the eye. In this invention, "image" refers to an element perceived through this vision. Specifically, an image is reproduced as light by an image output device, such as an LCD monitor, an OLED monitor, a CRT monitor, a projector projecting onto a screen, VR goggles, or a retinal scanning display, which reproduces an image from an electrical signal, and is experienced by the eyeball and the retina. Examples of such images include text displays, photographs, live-action images, animated images, and 3D rendered images. They may be displayed on the entire display screen or only on a portion of the screen. Furthermore, images in a broad sense can also include light projections, blackouts, flashing rays of light, and light spots, even if they are not text or pictures. For example, spotlight illumination and LED illumination are also included in images in a broad sense.

[0019] In this invention, tactile sensation refers to one of the five senses, a sensation that can be felt directly or indirectly through the skin, muscles, nerves, etc., and is used in a broad sense unless otherwise specified. In the narrow sense, tactile sensation refers to one of the sensations known as those felt through the skin: touch, pressure, pain, cold, and warmth. However, in this invention, tactile sensation includes not only tactile sensation in the narrow sense but also other sensations such as vibration and contact sensation. The tactile effect provided by this invention is an effect that realizes these sensations so that they are actually experienced by the human body. A device that realizes this tactile effect by receiving an electrical signal is called a tactile output device. The tactile output device used in this invention is one that realizes a tactile sensation of varying intensity depending on the current and voltage of the electrical signal, rather than simply turning the electrical signal on and off. Examples of haptic output devices include actuators such as motors that receive an electrical signal to drive a device, balloons that receive an electrical signal to heat a device and inflate a gas, electric fans that receive an electrical signal to rotate their rotors to generate airflow, and magnetorheological fluid devices that receive an electrical signal to increase the rotational resistance of a device. These haptic output devices include both active devices that operate independently of the user's actions to provide a haptic effect to the user, and passive devices that operate passively in response to the user's actions to provide a haptic effect. The definition of whether a haptic effect is active or passive is called a haptic attribute. Examples of the former, actively operating devices, include devices that deform to provide a sense of pressure felt on the skin of the hand or other device, built-in vibrators in game controllers and smartphones that provide vibrations in the hand, built-in vibrators in chairs that provide vibrations throughout the body, and motors that rotate or accelerate seats. Examples of the latter passively acting devices include devices that change the resistance felt in the hands or feet when the user applies force to a controller, such as a glove or other device worn on the body, or a fishing rod reel or bicycle pedal.

[0020] The sensory presentation device according to the present invention is a device or system that allows a user to experience both the active and passive tactile sensations, as well as either or both of vision and hearing. The device may be mounted in a single housing, or may be a device that is an integrated system in which the above-mentioned devices are distributed across multiple housings and connected by wire or wirelessly.

[0021] Fig. 1 shows a functional block diagram of an embodiment of a sensory presentation device 10 according to the present invention. The sensory presentation device 10 comprises a haptic output device 11 that houses a haptic output device 14 and generates a haptic sensation for the user, and a control device 51 that houses an audio output device 75 and an optical output device 76. The haptic output device 11 and the control device 51 are connected by wire or short-distance wirelessly and are capable of communicating with each other, constituting a system that can operate as an integrated device.

[0022] The sensory presentation device 10 includes a haptic output device 14 that provides haptic effects in response to electrical signals. The device may be either an active haptic output device 14a, which actively operates to provide a haptic sensation, such as an actuator (e.g., a motor or vibrator) or a heated balloon, which directly contacts the user to provide a haptic sensation, or a non-contact, indirect haptic sensation, such as a fan that generates wind pressure. Alternatively, the device may be a passive haptic output device 14b, which passively operates a device (e.g., an MRF device) that increases or decreases resistance to the user's movement, or an actuator (e.g., a motor) that increases or decreases resistance to the user's movement. While the figure shows a single device containing both the active haptic output device 14a and the passive haptic output device 14b, the device may instead house multiple devices capable of providing each of these haptic effects, or a device housing both devices and a device housing one of the devices. A single device capable of outputting both passive and active haptics, such as a motor, may also be housed. 1 illustrates an example in which an active haptic output device 14a and a passive haptic output device 14b are stored in a haptic output device 11 connected to a control device 51, and the active haptic output device 14a and the passive haptic output device 14b are worn by a user separately from a smartphone. In the following description, the active haptic output device 14a and the passive haptic output device 14b may be collectively referred to as the haptic output device 14.

[0023] The haptic output device 11 has a haptic control unit 21 that controls operations and commands, such as transmitting or causing to be transmitted electrical signals to the active haptic output device 14a and the passive haptic output device 14ab. The haptic control unit 21 receives instructions from a control unit 61 of the control device 51 and causes the haptic output device 14 to output a haptic effect at a predetermined timing. Alternatively, the master-slave relationship may be reversed, such that the control units 61 (auditory control unit 71, visual control unit 72) that receive instructions from the haptic control unit 21 output the effect at a predetermined timing. Note that, in the illustrated embodiment, a single haptic control unit 21 controls both the active haptic output device 14a and the passive haptic output device 14b, but the haptic control unit 21 may be divided into multiple units that work together and control each device individually.

[0024] The haptic control unit 21 also has a haptic signal storage unit (not shown), which is memory used for calculations and the like. The haptic signal storage unit temporarily records signals and stores information, results, commands, and the like required for calculations by the haptic control unit 21. The memory of the haptic signal storage unit may be volatile memory, but it is more preferable if it also has nonvolatile memory. If it has nonvolatile memory, a high-performance haptic output device 11 can store records and histories of individual output changes and refer to them for further optimization of the output. Although not shown, the program that operates the haptic control unit 21 may be stored in the haptic signal storage unit if it is nonvolatile memory, or it may be stored separately.

[0025] In addition, it is preferable that the haptic output device 11 has a sensor 16 that measures the displacement of the device itself for the tactile sensation that the haptic output device 14 provides, as well as the position, displacement, load, etc. when the user operates the haptic output device 14.

[0026] Furthermore, the haptic output device 11 has a power supply 25 necessary for operating the device itself. It may be a battery, or may be connected to an external power source. In the case of a battery, if the amount of current required by the haptic output device 11 is small, a replaceable primary battery will suffice, but if the amount of current required is large, a secondary battery that is charged from an external power source will be easier to operate. Also, if communication with the control device 51 is carried out via a wired cable, the control device 51 may be powered as an external power source.

[0027] Communication between the individual devices constituting the sensory presentation device 10 may be wired or wireless. In the case of wired communication, power may be supplied via a wired cable. The standard is not particularly limited; at the time of filing the present application, USB cables, Lightning (registered trademark) cables, Thunderbolt (registered trademark) cables, etc. are available, but any standard that enables similar or upwardly compatible communication may be used. In the case of wireless communication, any short-range wireless communication standard may be used, including various wireless LAN standards, Bluetooth (registered trademark), Bluetooth LE, and wireless USB. However, because the amount of data required to implement at least the adjustments in this invention is small, standards that are relatively slow and consume little power, such as Bluetooth and Bluetooth LE, are preferably used. Of course, high-speed communication standards that require large amounts of data for operations other than those required by this invention may also be used.

[0028] The control device 51 communicates with the haptic output device 11, controls the haptic output device 11 to provide the user with a tactile sensation as part of the content, and is also responsible for outputting audio and video. Examples of such a control device 51 include a smartphone, portable game console, tablet device, etc., which has earphones wirelessly connected to a built-in speaker as the audio output device 75 and a touch panel stored in the device as the optical output device 76. Another example is a home theater set connected to a surround speaker set via a wired connection as the audio output device 75 and a projector for screen projection as the optical output device 76. Still another example is a personal computer, stationary game console, or VR goggles, which has an LCD monitor or organic EL monitor as the optical output device 76, wired headphones as the audio output device 75, and a controller with a vibration function as the haptic output device 14 connected to the haptic output device 11. The combination of devices may be interchanged between these examples. Here, a smartphone will be primarily used as an example of the control device 51, but the present invention is not limited thereto.

[0029] In addition, the control device 51 has an input device 78 that is responsible for selection and input for experiencing sensations. Specifically, examples include a touch panel integrated with the optical output device 76, a mouse, a trackball, a controller, a keyboard, etc. Also, the haptic output device 11 may be a part of the input device 78.

[0030] Furthermore, a server 81 provided at the end of a network 82 to which the control device 51 is connected via a router or a terminal may assume some of the functions of the control device 51 described below. For this purpose, it is preferable that the control device 51 has a network interface (NWIF) 69 such as a wired LAN function, a wireless LAN function, or a function for connecting to a mobile communication network. The network interface 69 may also be used for communication with the haptic output device 11, or may be independent. Since the communication volume differs greatly, it is often preferable for the network interface 69 to be independent. The figure shows an example in which the network interface 69 is independent.

[0031] The control device 51 has a control unit 61 that performs calculations and the like. Specifically, it is a calculation device such as a CPU or GPU, and controls the behavior of the device, such as reading content, calculations, receiving inputs and outputs to individual devices, and communicating with the haptic output device 11. Of the control unit 61, an element responsible for outputting audio to the audio output device 75 is referred to as an auditory control unit 71. Furthermore, an element responsible for outputting video to the optical output device 76 is referred to as a visual control unit 72. These may be separate hardware components, or a single or multiple control units 61 may each perform their respective roles according to a program. The figure shows a configuration in which the control unit 61, which is the CPU of the control device 51, taken as an example of a smartphone, performs the roles of both the auditory control unit 71 and the visual control unit 72.

[0032] In addition, the control device 51 has a memory unit (not shown) that stores data and programs. This memory unit preferably has both a non-volatile memory or a magnetic disk used as storage and a volatile memory used for calculations. In the figure, it is not distinguished whether data or programs are loaded.

[0033] Application software (hereinafter abbreviated as "apps") such as games, movies, virtual spaces, simulators, etc. that allow the user to experience haptic output as part of the content are stored in the storage unit of control device 51. The software may be pre-installed in control device 51, or may be downloaded and installed from server 81 via network 82.

[0034] The app includes audio and video content and causes the haptic output device 11 to output haptics linked to the audio and video, thereby providing the user with a haptic sensation. Examples of such content include, but are not limited to, a reproduction of the haptic sensation experienced by a character in a movie, a reproduction of the tactile sensation of an object that appears in a game, a simulator that reproduces the tactile sensation of an object touched in a virtual space, the sensation of a cat or dog, and a reproduction of the resistance of a ball when batting in a virtual batting game. This invention can be used regardless of whether the content includes active or passive haptics.

[0035] The app 65 stores a media signal database containing media signals such as audio and video signals, including programs for reproducing the content, via the audio output device 75, optical output device 76, and haptic output device 14 (active haptic output device 14a and passive haptic output device 14b). This media signal database preferably includes a video signal database containing still images and video, an audio signal database containing sound effects and audio, and a haptic signal database containing haptic effect signals. These databases are preferably in the form of audio files, image files, signal intensity files, and other files that can be handled independently for each sensation to be provided, making it easier to handle delay times. However, it is desirable for these files to contain information on the timing of linking by default, and for the database to be constructed so that the files can be linked and presented to the user. Video files that combine audio and video can also be used, but the synchronization must be deactivated or the timing of the synchronization must be shifted when output to the device. It should be noted that the media signal database does not need to store all necessary media signals in the storage unit 62, and may be added temporarily or permanently by downloading them as needed via the network.

[0036] The sensory presentation device 10 according to the present invention further includes a delay time database, which is associated with the media signal database and delays the timing of output to each control unit and device by a predetermined delay time. This delay time is preferably determined mainly from two factors. One is a hardware-dependent adjustment time for adjusting the time lag of signal processing that depends on the hardware configuration. The other is a perception-dependent adjustment time that corresponds to the perception time required for humans to perceive the audio, video, and haptic effects output from the device.

[0037] Among these, hardware-dependent adjustment times include, for example, the processing time for encryption and decryption when communicating wirelessly with wireless earphones or a wirelessly connected haptic output device 11, the processing time for 3D graphics processing, the time for processing via a digital audio interface, the time lag until mechanisms such as the actuators of the haptic output device 11 start operating, and the time lag due to the response speed of materials that exhibit characteristics.

[0038] On the other hand, the perception-dependent adjustment time compensates for the fact that the time it takes for a human to perceive a sensory stimulus differs between hearing, vision, and touch. To compensate for this difference in perception time, the time is adjusted so that stimuli that take longer to be perceived are earlier and stimuli that take shorter to be perceived are delayed. In particular, visual images tend to be perceived slightly later, so it is preferable to delay auditory and tactile stimuli more than visual stimuli and output them from the control unit.

[0039] The delay times recorded in this delay time database may be recorded separately for hardware-dependent adjustment times and perception-dependent adjustment times, but it is preferable to record them as the total time for each of the auditory, visual, and tactile senses, since this total time can be applied as the delay time directly and the delay processing itself can be sped up. Furthermore, the perception time may change depending on the output content, such as a blurry image or a clear image. The delay time may be set for each content.

[0040] The delay times recorded in this delay time database are the delay times for each sense when an active haptic effect is provided by the active haptic output device 14a and the delay times for each sense when a passive haptic effect is provided by the passive haptic output device 14b, and if these are recorded as separate values, it is possible to select and delay the appropriate delay time for each haptic effect. Note that separate tables may be used, the delay times may be recorded in different columns in the same table, or the delay times may be specified as a combination of one value and its correction value.

[0041] The delay times recorded in the delay time database can be set in milliseconds for each of the auditory, visual, and tactile senses. The perception-dependent adjustment time, among the delay times, can be set to a value based on the average human perception speed. However, since there are some individual differences, it is preferable to be able to change the perception-dependent adjustment time so as to reduce the sense of discomfort felt by the user. However, since adjustment is done in milliseconds and is difficult for users who are not accustomed to it, multiple patterns of arrangements of the perception-dependent adjustment time can be prepared, allowing the user to test use and select the arrangement that causes the least discomfort.

[0042] On the other hand, for the hardware-dependent adjustment time, it is advisable to have the app record a pre-measured value of the hardware-dependent adjustment time in a typical operating environment as a default value. For example, since commonly sold smartphones and game consoles often have common hardware specifications, it is advisable to measure these common values ​​in advance. On the other hand, for uncommon hardware specifications, it is preferable to use the default value while being able to adjust it as appropriate. In particular, it is desirable for the app to perform tests that actually perform the operations of each of the audio output device 75, optical output device 76, active haptic output device 14a, and passive haptic output device 14b, and to measure and set the actual processing time. This is because the value changes depending on the hardware configuration and processing capacity of the control device 51.

[0043] Each delay time is basically 0 or a positive value; the larger the positive value, the greater the delay, resulting in a slower transmission of the output from the control unit. However, negative values ​​may also be set depending on the situation. Figure 2(a) shows an example of this delay time table, with separate tables for haptic attributes for cases where active haptic effects are provided by the active haptic output device 14a and passive haptic effects are provided by the passive haptic output device 14b. For example, if the haptic output device 11 is the haptic output device 14b that operates passively in response to user operation, it is possible to adjust the delay time with other senses by transmitting a signal in advance before the user's operation reaches a predetermined position. In this case, it is recommended to calculate the user's operation speed, etc., using the sensor 16 and estimate the time it will take to reach a predetermined position. On the other hand, if the active haptic output device 14a acts regardless of the user's operation, the delay time will be different. The delay time for active haptics and hearing is shorter than for passive haptics, while the delay time for vision is longer. This is because the perception-dependent adjustment period of the human operator varies depending on whether the operator is passive or active, and the hardware-dependent adjustment period also varies due to the passive haptic output device 14b's characteristic of receiving and experiencing the user's movements. The perception speed of each sense, which is the basis for the perception-dependent adjustment time, is thought to be approximately 120-190 ms for tactile sensation, 180-230 ms for visual sensation, and 120-180 ms for auditory sensation, from the time of receiving a stimulus to the time of perception. It is advisable to set the delay time according to this perception speed.

[0044] Another method for setting the delay time is to use the timing of transmission by the control unit corresponding to the auditory, visual, or tactile signal provided by the content, which has the longest perception time, as a reference, and set the delay time for the timing of transmission by the corresponding control unit for the signals related to the other senses. Specifically, the delay time for the reference sense signal is set to 0 ms, and the signals for the other senses are set to a delay time from that reference. Although this depends on the hardware-dependent adjustment time, the delay time to the optical output device 76 for vision, which often has the longest perception time, is set to 0 ms, and the delay times for the other senses are set to positive times. An example of this delay time table is shown in Figure 2(b). In this case, the delay times for senses other than the reference sense use different values ​​depending on the tactile attribute, whether active or passive tactile.

[0045] Another method for setting the delay time is to register different settings depending on the combination of senses presented by the content. This is to address the fact that when multiple senses are perceived simultaneously and closely, the user's impression changes, resulting in a change in the actual perception time. For example, content that combines visual, tactile, and auditory perception is more likely to be perceived faster than content that is perceived solely through visual and tactile sensations. Furthermore, the perception time for tactile sensations differs between a combination of tactile and visual sensations and a combination of tactile and auditory sensations. Therefore, it is recommended to set different delay times for each combination and select the appropriate combination from a table depending on the combination of senses presented by the content. An example of this delay time table is shown in Figure 2(c). For content that combines tactile, visual, and auditory sensations, the delay time for each sense is shorter than for content that combines tactile, visual, and auditory sensations alone or for content that combines tactile and auditory sensations alone. In this example, content that combines tactile, visual, and auditory sensations reduces the visual sensation by 2 ms and the tactile sensation by 5 ms compared to content that is solely tactile and visual sensations. The reason there are differences in the shortened time is that the delay time for vision is originally short, so shortening it by the same amount would create a sense of discomfort. Also, when comparing content that combines haptic, visual, and auditory senses with content that combines only haptic and auditory senses, the delay time is made shorter than that for haptic and auditory senses only. In this example, haptic sense is shortened by 5 ms, and auditory sense is also shortened by 5 ms. Because the difference in perception time between haptic and auditory senses is small, the amount of shortening by combination can be made the same. Furthermore, in this case too, different values ​​are used depending on the haptic attribute, whether it is active haptic or passive haptic.

[0046] The operation of the sensory presentation device 10 according to the present invention when triggered by the operation of the haptic output device 11 will be described with reference to Fig. 3. A tapping unit using a magnetorheological fluid and a motor will be used as the haptic output device 11. An example of the structure of a passive haptic output device 14b incorporated in this tapping unit is shown in Fig. 4, and an embodiment of the structure of the tapping unit is shown in Figs. 5(a) and 5(b).

[0047] A magnetorheological fluid (MRF) device (hereinafter abbreviated as "MRF device") is used as the passive tactile output device 14b shown in FIG. 4. This MRF device is composed of a rotating shaft 41, a disk 32, yokes 34 and 35, a coil 37, a magnetorheological fluid 38, and casings 31 and 36. As shown in FIG. 4, a space is provided around the disk 32 attached to the rotating shaft 41, sandwiched between the yokes 34 and 35. The space contains a magnetorheological fluid 38, whose viscosity changes depending on the strength of the applied magnetic field. The coil 37, supported by the yoke 35 and generating a magnetic field (indicated by the arrow in the figure), is also housed in the space. The rotating shaft 41 is supported by a bearing 39 and is integrated with the disk 32, which is surrounded by the magnetorheological fluid 38. The resistance of the magnetorheological fluid 38 when rotating the rotating shaft 41 can be adjusted by adjusting the amount of current supplied to the coil 37, which generates the magnetic field. When a user applies force to rotate this rotating shaft 41, the viscosity of the magnetorheological fluid 38 is increased or decreased, thereby increasing or decreasing the resistance to the disk 32 that is integrated with the rotating shaft 41, allowing the user to experience a passive tactile effect on their movement as "difficulty in rotating" the rotating shaft 41.

[0048] In this invention, a motor (not shown) that serves as the active tactile output device 14a is connected to the rotating shaft 41 of this passive tactile output device 14b, and the rotating shaft 41 is set to rotate whether or not it is due to the user's movement.

[0049] Figures 5(a) and 5(b) show a tap unit, a haptic output device 11 using such an MRF device and a motor. It includes a base 40, which is fixed by the thumb or palm, and a finger rest 42 that rotates in conjunction with a rotation axis 41 of the MRF device attached to the base 40. The finger rest 42 corresponds to one or all of the index finger, middle finger, ring finger, and little finger. When the user bends and grasps the finger, it rotates around a fulcrum 43. When the finger rest 42 is pressed, a first link member 44 and a second link member 45 linked via a pin 46 rotate as shown in Figure 5(c). This movement rotates the rotation axis 41 relative to the base 40. The resistance to this rotation is increased or decreased by the magnetorheological fluid 38 of the MRF device. Meanwhile, the motor drives the rotation axis 41 to reverse this movement, causing the finger rest 42 to open, providing an active tactile sensation that works without the user's movement. When the motor is driven and the rotary shaft 41 is rotated relative to the base 40, the second link member 45 rotates as shown by the thick arrow in Figure 4(c), and the first link member 44 is pressed via the pin 46, causing the finger rest part 42 to also rotate as shown by the thick arrow. This allows the user to experience tactile sensations as pressure felt by the fingers and the displacement of the finger position pressed by the part.

[0050] The sensor 16 attached to the tap unit, which includes the MRF device and motor, can be an angle sensor that acquires the rotational position of the second link member 45 relative to the rotation axis 41 as an angle. The initial position, where the user is not applying any force, is 0°, and it can be displaced up to 90° from there. Furthermore, by operating the motor, this rotational position can be freely displaced from minimum to maximum, and forces in the expanding and contracting directions can be experienced as haptic effects. The haptic effect can be realized by changing the resistance value depending on the degree to which the rotational position is "grasped," "expanded," or "contracted." At 0°, the current value is 0.0 A, and no resistance is felt when gripping. However, as the rotational position advances and the grip advances, the user begins to feel the resistance, which gives the sensation of "grasping" the target content. The resistance value increases further as the grip advances. To achieve this, the current applied to the motor is maintained at 0.0 A, while the current applied to the magnetorheological fluid 38 is increased, increasing the resistance to rotate the disk 32 relative to the rotation axis 41. This increases the resistance when applying force to the tap unit to further advance the rotational position. On the other hand, to operate the rotational position using a motor, the current applied to the magnetorheological fluid 38 is reduced to reduce the resistance, and then the current applied to the motor is increased to rotate the device. At 90°, the grip feels empty. However, as current flows through the motor and the rotational position begins to return toward 0°, the fingers experience a sensation of movement or expansion from within the grip, and the fingers begin to spread apart. This spreading and pushing force are manifested as a pushing force on the finger rest 42, depending on the current flowing through the motor, which is set differently depending on the rotational position. Finally, when the rotational position returns to 0°, the current flowing through the motor becomes 0.0 A, and the finger rest 42 stops spreading (returning).

[0051] The image displayed on the optical output device 76 changes in response to the sensations of "grasping," "expanding," and "shrinking." Initially, an image (〇A.jpg) showing the state where nothing is being touched is displayed. However, as the user grasps the content, the image changes to an image (〇B.jpg) showing the content being deformed by the touch. As the user grasps the content further, the displayed image changes to show the result of irreversible deformation of the content (〇C.jpg → 〇D.jpg). On the other hand, if the motor is driven to allow the user to experience the content returning to its original position before the grasp advances, the displayed image returns to its original state (〇C.jpg → 〇B.jpg). Note that while this example shows the image and audio files being specified individually, it is also possible to depict a 3D object, and then render and display the image as force is applied. In addition, in accordance with the contact and the reversible or irreversible deformation, a contact sound (OP.mp3), a popping sound (OQ.mp3), and a return sound (not shown) that are suited to the content are reproduced from the audio output device 75.

[0052] The app 65 of the control device 51 has, in its storage unit, multiple content tables that combine tactile, visual, and auditory information as content that can be experienced by the control device 51. In FIG. 3, a table showing the tactile sensation of the ball, its deformed shape, and the sounds of contact and popping is shown at the forefront. A content database 67 is provided as a collection of information for reproducing the sensation of touching such content. The content database 67 is preferably stored in the storage unit 62 of the control device 51. Specifically, the image files and audio files specified in the table, along with the table values, are preferably stored so that they can be called up as a set. If they are not stored at the time of installation of the app 65, it is desirable that they be called up from the server 81 and stored in a state where they can be immediately sent to each device. This is because calling them from the server would not be fast enough to accommodate delays in the millisecond range.

[0053] Furthermore, the storage unit 62 of the control device 51 has a table of delay times used in the present invention for each of the tactile, visual, and auditory contents. Based on the values ​​in this delay time table, transmission from the control unit 61 (the "auditory control unit 71" and the "visual control unit 72") and the tactile control unit 21 is delayed.

[0054] In order to allow the user to experience content that combines haptics, vision, and hearing according to this content table, the haptic portion of the content table for the selected content (here, a balloon as an example), i.e., the table of rotational positions and current values ​​(base haptic data), is transmitted from control device 51 to haptic output device 11 (S1). At this time, an embodiment may be such that either an active or passive haptic effect is selected depending on the selected content, or an embodiment may be such that the same content exerts either or both an active and passive haptic effect depending on the situation. Here, an example is shown in which a table for experiencing an active haptic effect is selected, assuming that the inflation of a balloon is experienced.

[0055] In the haptic output device 11, the base haptic data is stored in the haptic signal storage unit 22. The delay time (35 ms in this case) specified when outputting this base haptic data from the haptic control unit 21 is also transmitted. This delay time is also stored in the haptic signal storage unit 22. The optical output device 76 is pre-displayed with an image file (〇A.jpg) representing the initial state of the content. In addition, to prepare the user for the experience, the haptic output device 14 is displayed as if it is being squeezed as far as possible, and when the user squeezes it, the rotation position is set to 90°. The user is now ready to experience the content.

[0056] When the haptic output device 14 starts operating while the user is waiting, the haptic control unit 21 acquires the angle θ, which is the rotational position, using the sensor 16, references the table of base haptic data in the haptic signal storage unit 22 to read it, and applies a voltage to the motor of the haptic output device 14 to apply a current value corresponding to that rotational position (S2). This causes the rotational position to shift back from 90°. The shift is detected and the changed rotational position is transmitted to the control device 51 (S3). Subsequently, the haptic signal storage unit 22 references the base haptic data to read a current value for outputting the resistance at that rotational position as a haptic effect, and the amount of current is transmitted to the haptic output device 14 with a delay corresponding to the delay time (S4). In parallel with S4, the control unit receives the changed value of the rotational position transmitted from the haptic output device 11 (S5). The audio file and image file corresponding to this changed rotation position are read from the content table of the selected content, and the image file is sent to the optical output device 76 to switch the display, and if it is time to output sound, the audio file is output from the audio output device 75 (S6). The timing of this display switching and audio output is also delayed according to the delay time table. When counting this delay time, it is preferable that the reference time of the delay timing between the control unit 61 and the haptic control unit 21 is synchronized. That is, the delay time is counted from the reference time that is matched with the visual and auditory delay times, taking into account the time required for the data to be sent from the haptic control unit 21 to the control unit 61 and for S5 and S6 to be processed.

[0057] Furthermore, if the selected content is a table that provides a passive haptic effect, for example, by experiencing the sensation of gripping a ball, base haptic data for the passive case is transmitted to the haptic output device 11 (corresponding to S1). The haptic output device 11 stores the haptic table in the haptic signal storage unit 22. The haptic output device 11 also transmits a delay time (45 ms in this case) to be specified when outputting the haptic table for the passive case from the haptic control unit 21. This delay time is also stored in the haptic signal storage unit 22. The optical output device 76 displays an image file representing the initial state of the content in advance. This completes the preparations for the user to experience the content.

[0058] When the user starts the operation, the haptic control unit 21 acquires the angle θ, which is the rotational position, using the sensor 16, and transmits it to the control device 51 when it changes by a predetermined amount (for example, in units of 1°) (corresponding to S2). At the same time, the haptic signal storage unit 22 references the haptic table to read a current value for outputting the resistance at that rotational position as a haptic effect, and transmits that current value to the haptic output device 14 with a timing delayed by the delay time (corresponding to S3). Concurrently with this transmission, the control unit receives the changed value of the rotational position transmitted from the haptic output device 11 (corresponding to S5). The control unit reads an audio file and an image file corresponding to this changed rotational position from the content table of the selected content, transmits the image file to the optical output device 76 to change the display, and, if it is time to output sound, outputs the audio file from the audio output device 75 (corresponding to S6). The timing of this display change and audio output is also delayed according to the delay time table. When counting this delay time, it is preferable that the reference times for the delay timing between the control unit 61 and the haptic control unit 21 are synchronized. In other words, the output of the tactile control unit 21 is sent to the control unit 61, and the delay time of the output of the tactile control unit 21 is counted from a reference point that is combined with the visual and auditory delay times, anticipating the time required for the processing corresponding to S5 and S6 to be processed.

[0059] An example of the process when a user uses the sensory presentation device 10 according to the present invention will be described with reference to the flowcharts in Figures 6 to 8. First (S101), the app 65 is launched on the smartphone, which is the control device 51 (S102). Next, the tap unit, which is the haptic output device 11, is turned on (S103), and the smartphone and tap unit are paired via Bluetooth to establish a wireless connection (S104). Note that the launch of the app (S102) may be performed after S104 in some embodiments. Additionally, the app 65 initially holds a table of delay times for haptic, visual, and auditory sensations, which is read out (S105).

[0060] A user operating the app 65 via the input device 78 selects an object displayed on the monitor for which they wish to experience a haptic sensation (S110). An example of the display on the monitor is shown in FIG. 9. First, the route for selecting the ball object on the right will be explained (S111). This object is selected as the object for experiencing a passive haptic sensation. Upon receiving the selection instruction, the control unit 61 retrieves the content table associated with the selected object from the content database 67 (S112). At the same time, if there are any image files and audio files specified in the table, they are also retrieved and stored in the storage unit 62 (S113). Next, a table of rotational positions and current amounts is extracted from the content table and transmitted to the tap unit (S114). At the same time, a delay time for the passive haptic sensation is also transmitted. The tap unit stores the transmitted haptic table and delay time in the haptic signal storage unit 22 (S115). Subsequent processing then differs depending on whether the object is active or passive (S116). If it is active (S116→Yes), then preparation is complete and the control device 51 displays an image file of the initial state of the ball object, which is the selected content, and a selection cursor on the screen of the optical output device 76 (S117).

[0061] The haptic control unit 21 acquires the angle of the rotational position from the angle sensor 16 (S121). It monitors whether the angle reaches the next rotational position where the current value is to be changed, as defined in the haptic table stored in the haptic signal storage unit 22 (S122). If the angle has not been reached (S122 → No), it continues to acquire angles from the angle sensor (S122). If the next rotational position has been reached (S122 → Yes), this is set as the reference time (S123), and the system waits for the haptic delay time from the reference time (S124). After waiting, the system applies a current value to the haptic output device 14 to realize the haptic effect (S125). After application, the user experiences the haptic effect after a time lag equivalent to human perception.

[0062] Meanwhile, in parallel with setting the reference time (S123), the haptic control unit 21 notifies the control device 51 that the rotational position has shifted (S130). The control unit 61 of the control device 51 starts counting the visual delay time and the auditory delay time from the reference time (S131, S141). The control unit 61 (visual control unit 72) waits the visual delay time (S131) ​​and then outputs an image file to the optical output device 76 (S132). Furthermore, the control unit 61 (auditory control unit 71) waits the auditory delay time (S141) and then outputs an audio file to the audio output device 75 (S142). As a result, the user perceives the resistance that represents the feeling of touching the ball through their tactile sense, in synchronization with the user's perception that the image file of the ball has been displayed, and also hears and perceives the sound of the ball being touched (S150).

[0063] If the user continues to experience the same object by further rotating it (S151 → Yes), the process returns to monitoring the rotational position using the sensor (S122). As the rotational position advances, a different image file is displayed. For example, as the image changes from a contact state to an image of a ball exploding, the feeling of grasping the ball disappears, the resistance applied to the haptic output device 14 disappears, and the sound of the ball exploding is displayed. This is also output in accordance with the delay time for each sense, so that the timing at which the ball's explosion is perceived by touch, hearing, and vision coincides (S150).

[0064] If the user selects another object (for example, the balloon-shaped object on the left) and experiences a different sensation (S161 → Yes), the process returns to step S111 (S162). In this case, the content table (tactile table) associated with the selected object is called up.

[0065] The flow when the selected object (content) provides an active haptic effect will be described following the branching from S116 → No onwards. The control device 51 displays instructions on the screen of the optical output device 76 to grip the tap unit (S211). When the tap unit is pressed in and reaches a rotation position of 90° (S212), an image file of the initial state of the balloon object, which is the selected content (for example, a deflated balloon before inflation), and a selection cursor are displayed (S213). At the same time, a start button is displayed, and when the start button is tapped, output by the haptic output device begins (S214).

[0066] The haptic control unit 21 acquires the angle of the rotation position from the sensor 16, which is an angle sensor (S221). Here, it is initially 90°. From that angle, the current value defined in the base haptic data stored in the haptic signal storage unit 22 is referenced (S222). A reference time is then specified (S223). A haptic delay time is then waited from that reference time (S224), and after waiting, a current value for realizing the haptic effect is applied to the haptic output device 14 (S225). When a current is applied, the motor is driven, and after a time lag equivalent to the human perception time, the user experiences the haptic effect.

[0067] Meanwhile, in parallel with setting the reference time (S223), the haptic control unit 21 notifies the control device 51 that the rotational position has been displaced (S230). The control unit 61 of the control device 51 starts counting the visual delay time and the auditory delay time from the reference time (S231, S241). The control unit 61 (visual control unit 72) waits the visual delay time (S231) and then outputs an image file to the optical output device 76 (S232). Furthermore, the control unit 61 (auditory control unit 71) waits the auditory delay time (S241) and then outputs an audio file to the audio output device 75 (S242). As a result, the user haptically perceives the displacement of the finger rest part 42, which gives the feeling that the balloon is starting to inflate, in synchronization with the user's perception that the image file showing the balloon starting to inflate has been displayed, and simultaneously perceives a sound effect indicating that the balloon has started to inflate (S250).

[0068] If the user continues to move the finger position further on the same object to continue experiencing it (S251 → Yes), the sensor continues to monitor the rotation position (S252). The motor drives the rotation position to return, and when it reaches the next rotation position boundary (S252 → Yes), that rotation position is acquired (S221), a reference time for experiencing the tactile, visual, and auditory sensations at that rotation position is specified (S223), and similar processing is performed (S224~, S230~).

[0069] The experience of this object ends when the rotation position returns to 0° and no longer moves forward, or when the user consciously inputs an intention to end the experience of this object through the input device 78 (S251 → No). If the user selects another object and experiences a different sensation (S261 → Yes), the process returns to step S111 (S262). At this time, the content table (base haptic data) associated with the selected object is called up. If the user ends the experience (S261 → No), the application 65 is terminated (S271). [Explanation of symbols]

[0070] 10 Sensory presentation device 11 Tactile output device 14 Haptic Output Devices 14a Active tactile output device 14b Passive tactile output device 16 sensors 21 Tactile control unit 22 Tactile signal memory unit 25 Power supply 31 Casing 32 disk 34 York 35 York 37 Coil 38 Magnetorheological fluid 39 Bearings 40 base 41 Rotation axis 42 parts 43 Fulcrum 44 First link material 45 Second link material 46 pins 51 Control device 61 Control Unit 62 Storage section 65 apps 67 Content Database 69 Network Interfaces 71 Hearing control section 72 Visual control section 75 Audio Output Devices 76 Optical Output Devices 78 Input Devices 81 servers 82 Network

Claims

1. an auditory control unit that outputs sound to an audio output device; a visual control unit that outputs an image to an optical output device; and and, A sensory presentation system has a haptic control unit capable of outputting both an active haptic effect that acts regardless of a user's motion and a passive haptic effect that acts in response to a user's motion, and selectively provides an active haptic sensation that acts regardless of a user's motion and a passive haptic sensation that acts in response to a user's motion, When providing content that links the audio and / or video with the haptic effect, the timing of outputting each signal to each device is set as follows: The auditory and / or visual senses, the active tactile sense, and the passive tactile sense are each output from each control unit with a delay of a predetermined delay time in accordance with a perception time from transmission by the control unit to perception by a human; the haptic effect has a haptic attribute that defines whether the haptic effect is an active haptic or a passive haptic; the predetermined delay time is invoked in response to the haptic attribute; Sensory presentation system.

2. an auditory control unit that outputs sound to an audio output device; a visual control unit that outputs an image to an optical output device; and and, a haptic control unit capable of outputting both an active haptic effect that acts regardless of a user's action and a passive haptic effect that acts in response to a user's action, a control device connected to the optical output device and the audio output device; a haptic output device connected to the control device, the haptic output device having the haptic output device and the haptic control unit; A sensation presentation system that selectively provides an active tactile sensation that acts regardless of a user's movement and a passive tactile sensation that acts in response to a user's movement, When providing content that links the audio and / or video with the haptic effect, the timing of outputting each signal to each device is set as follows: The auditory and / or visual senses, the active tactile sense, and the passive tactile sense are each output from each control unit with a delay of a predetermined delay time in accordance with a perception time from transmission by the control unit to perception by a human; The control device A sensory presentation system that has a delay time table that sets a delay time for each sense, and that executes a transmission means that transmits each signal from the control unit according to the delay time value of the delay time table.

3. As the tactile output device, a first haptic output device that actively operates to provide the active haptic sensation; a second haptic output device that operates passively in response to an external force to provide the passive haptic sensation; The sensory presentation system according to claim 1 or 2, comprising:

4. an auditory control unit that outputs sound to an audio output device; a visual control unit that outputs an image to an optical output device; and and, A sensory presentation method using a sensory presentation system that has a haptic control unit capable of outputting both an active haptic effect that acts regardless of a user's motion and a passive haptic effect that acts in response to a user's motion, and that selectively provides an active haptic effect that acts regardless of a user's motion and a passive haptic effect that acts in response to a user's motion, comprising: When providing content that links the audio and / or video with the haptic effect, the timing of outputting each signal to each device is set as follows: The auditory and / or visual senses, the active tactile sense, and the passive tactile sense are each output from each control unit with a delay of a predetermined delay time in accordance with a perception time from transmission by the control unit to perception by a human; the haptic effect has a haptic attribute that defines whether the haptic effect is an active haptic or a passive haptic; The predetermined delay time is called in response to the tactile attribute. Sensory presentation methods.

5. an auditory control unit that outputs sound to an audio output device; a visual control unit that outputs an image to an optical output device; and and, a haptic control unit capable of outputting both an active haptic effect that acts regardless of a user's action and a passive haptic effect that acts in response to a user's action, a control device connected to the optical output device and the audio output device; a haptic output device connected to the control device, the haptic output device having the haptic output device and the haptic control unit; A sensation presentation method using a sensation presentation system that selectively provides an active tactile sensation that acts regardless of a user's movement and a passive tactile sensation that acts in response to a user's movement, comprising: When providing content that links the audio and / or video with the haptic effect, the timing of outputting each signal to each device is set as follows: The auditory and / or visual senses, the active tactile sense, and the passive tactile sense are each output from each control unit with a delay of a predetermined delay time in accordance with a perception time from transmission by the control unit to perception by a human; The control device A delay time table is provided in which a delay time for each of the senses is set, and a transmitting means is executed to transmit each of the signals from the control unit according to the delay time value of the delay time table. Sensory presentation methods.

Images