Information processing device and method, and program
The information processing device addresses perceptual deterioration in tactile stimulation by determining modulation frequencies based on tactile characteristics and user states, enhancing the efficiency and quality of tactile experiences by minimizing noise and interference.
Patent Information
- Application Number
- PCT/JP2024/017718
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-02
- Filing Date
- 2024-05-14
- Publication Date
- 2024-12-05
AI Technical Summary
Existing tactile stimulation technologies face challenges in maintaining perceptual quality due to modulation, as determining the optimal modulation frequency based on resonance characteristics and human sensitivity is difficult, leading to perceptual deterioration and auditory noise.
An information processing device that determines the modulation frequency of a content signal based on tactile perceptual characteristics for each presentation site, information about the unmodulated signal, and user state, using a determining section and modulating section to perform modulation, thereby encoding and generating an encoded signal with presentation site, modulation intensity, and non-modulated signal information.
This approach effectively suppresses perceptual deterioration while enhancing drive efficiency and reducing auditory noise, ensuring a consistent and improved user experience by aligning modulation frequencies with sensitive tactile receptors and avoiding interference between modulated and unmodulated signal bands.
Smart Images

Figure JP2024017718_05122024_PF_FP_ABST
Abstract
Description
Information processing device, method, and program
[0001] The present technology relates to an information processing device, method, and program, and more particularly to an information processing device, method, and program that are capable of suppressing perceptual degradation due to modulation.
[0002] In recent years, technologies have been developed that provide tactile stimulation to users by vibrating a device worn by the user. Here, tactile stimulation refers to a physical phenomenon that causes the user to feel a sense of touch through a vibration phenomenon or the like. The generation of tactile stimulation is also called tactile presentation.
[0003] Tactile presentation technology is used in devices in various fields. For example, in terminal devices equipped with a touch panel such as a smartphone, the touch panel vibrates in response to a user's touch operation, providing a tactile stimulus to the user's finger, thereby enabling the user to experience the sensation of touching a button or other item displayed on the touch panel.
[0004] For example, in a music listening device such as headphones, it is possible to emphasize the deep bass of the music being played by providing tactile stimuli in sync with the music being played.
[0005] For example, in devices that provide computer games or VR (Virtual Reality), the controller can be vibrated to provide tactile stimulation depending on the operation using the controller or the content scene, thereby improving the user's sense of immersion in the content.
[0006] Furthermore, for example, technology has been developed that provides a tactile stimulus to a user based on a tactile signal received from an external device.
[0007] However, there are a wide variety of devices used to present tactile stimuli, each with significantly different characteristics. Therefore, it is possible that a tactile sensation that can be presented with one device may not be presented with another. Furthermore, vibrations during tactile presentation can generate unintended auditory noise, which can impair the user experience more than the benefits of the tactile presentation.
[0008] As a solution to this problem, the use of amplitude modulation has been proposed. Amplitude modulation is a process that converts signal components into signals of a certain frequency. In particular, amplitude modulation, which modulates high frequency bands to the resonant frequency of the tactile presentation device or the peak sensitivity of humans, has the advantages of absorbing the characteristic differences of the tactile presentation device, increasing drive efficiency, and reducing auditory noise.
[0009] For example, Patent Document 1 discloses a technique for receiving device characteristics from a tactile presentation device and performing amplitude modulation in accordance with the device characteristics.
[0010] JP 2019-67396 A
[0011] However, determining the modulation frequency by considering only the resonance characteristics of the tactile presentation device or the human sensitivity peak does not take into account the degree of perceptual quality degradation (hereinafter referred to as perceptual degradation), such as changes in the tactile sensation felt due to amplitude modulation, or differences in tactile perception characteristics depending on the presentation location and frequency of the tactile stimulus.
[0012] The present technology has been made in view of such circumstances, and is intended to make it possible to suppress perceptual degradation due to modulation.
[0013] An information processing device according to one aspect of the present technology includes a frequency determination unit that determines a modulation frequency of a content signal based on at least one of tactile perception characteristics for each presentation site, information regarding an unmodulated signal that does not undergo modulation, and a user state, and a modulation unit that modulates the content signal using the modulation frequency.
[0014] An information processing device according to another aspect of the present technology includes an encoding unit that encodes a content signal and generates an encoded signal including at least information indicating a presentation portion, modulation intensity information, and an unmodulated signal.
[0015] In one aspect of the present technology, the modulation frequency of a content signal is determined based on at least one of the tactile perception characteristics of each presentation site, information regarding an unmodulated signal that does not undergo modulation, and the user state, and modulation is performed on the content signal using the modulation frequency.
[0016] In another aspect of the present technology, a content signal is encoded to generate an encoded signal including at least information indicating a presentation portion, modulation intensity information, and an unmodulated signal that is not modulated.
[0017] 9 is a diagram showing haptic perception characteristics. FIG. 10 is a block diagram showing a configuration example of a haptic presentation system according to an embodiment of the present technology. FIG. 11 is a block diagram showing a configuration example of the encoding device of FIG. 2. FIG. 12 is a block diagram showing a configuration example of the haptic reproduction device of FIG. 2. FIG. 13 is a diagram showing a processing flow of the haptic presentation system of FIG. 2. FIG. 14 is a block diagram showing a configuration example of the modulation unit of FIG. 4. FIG. 15 is a diagram showing a method of calculating a modulation tolerance range. FIG. 16 is a diagram showing a method of calculating a modulation frequency f. FIG. 17 is a diagram showing a first processing example. FIG. 18 is a diagram showing an example of a transmission data format. FIG. 19 is a diagram showing a modified example of the first processing of FIG. 9. FIG. 19 is a diagram showing a second processing example. FIG. 10 is a flowchart illustrating decoding and modulation processing of the haptic reproduction device. FIG. 11 is a block diagram showing a configuration example of a computer.
[0018] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Overview of the present technology 2. Configuration 3. First processing example 4. Second processing example 5. Other
[0019] 1. Overview of the Technology Overview FIG. 1 is a diagram showing tactile perception characteristics.
[0020] Figure 1 shows the tactile perception characteristics for each frequency for Pacinian corpuscles, Meissner's corpuscles, Ruffini endings, and Merkel's discs.
[0021] Pacinian corpuscles, Meissner corpuscles, Ruffini endings, and Merkel's disks are receptors.
[0022] As shown in FIG. 1, the tactile perception characteristics vary depending on the frequency, i.e., the receptors that are most sensitive, i.e., that predominantly respond, differ.
[0023] Furthermore, the tactile perception properties of each receptor are different. Among the Pacinian corpuscles, Meissner's corpuscles, Ruffini endings, and Merkel's discs, the only receptors that are least sensitive to frequency changes are the Pacinian corpuscles, which respond to relatively high frequencies.
[0024] Therefore, in order to suppress perceptual degradation (perceptual degradation) due to modulation, such as changes in the sense of touch, it is necessary to keep the modulation frequency within the range in which the Pacinian corpuscles, which are insensitive to frequency changes, respond predominantly. Furthermore, this perceptual mechanism differs depending on the part of the body that senses (hands, feet, back, etc.).
[0025] Furthermore, when only a portion of the band of a haptic signal is modulated, the haptic signal includes a signal in the band to be modulated (hereinafter referred to as the modulated signal) and a signal in the band that is not modulated (hereinafter referred to as the unmodulated signal). Therefore, when only a portion of the band of a haptic signal is modulated, if the signal in the band that is not modulated (hereinafter referred to as the unmodulated signal) and the modulated signal overlap on the frequency axis, they interfere with each other, resulting in an unintended waveform, which also causes perceptual degradation. Here, as an example of a modulated signal and an unmodulated signal, the high-frequency side, which is obtained by applying a known frequency division process (such as a low-pass filter), can be considered the modulated signal, and the low-frequency side can be considered the unmodulated signal. Alternatively, the entire band of the haptic signal may be modulated. In this case, the unmodulated signal can be omitted from the encoded data, thereby reducing the amount of data.
[0026] Therefore, in the present technology, the modulation frequency is determined based on at least one of information on tactile perception characteristics and unmodulated signals, and also on the user state and the like.
[0027] This allows modulation to be performed with minimal perceptual degradation while absorbing differences in perceptual characteristics due to modulation, reducing noise, and improving drive efficiency.
[0028] 2. Configuration System Configuration FIG. 2 is a block diagram showing an example configuration of a tactile presentation system according to an embodiment of the present technology.
[0029] In FIG. 2, the tactile presentation system 1 is configured with a production environment 11 and a presentation environment 12 as environments for realizing tactile presentation.
[0030] In the production environment 11, the haptic signals of the content obtained by producing the target haptic information (tactile stimulation) are encoded, and haptic encoded data Dc obtained by the encoding is acquired.
[0031] In the presentation environment 12, a tactile emotion stimulus is presented based on a tactile signal obtained by decoding the tactile encoded data Dc.
[0032] The tactile presentation system 1 in Figure 2 includes an encoding device 21 in a production environment 11 to which one or more tactile signals 20-1 to 20-m are input, and also includes a tactile playback device 22 in a presentation environment 12 that is configured to be able to acquire tactile encoded data Dc, and one or more tactile presentation devices 23-1 to 23-n connected to the tactile playback device 22.
[0033] When it is not necessary to distinguish between the tactile signals 20-1 to 20-m, they will simply be referred to as the tactile signal 20. When it is not necessary to distinguish between the tactile presentation devices 23-1 to 23-n, they will be referred to as the tactile presentation devices 23.
[0034] The waveform of the haptic signal 20 is typically artificially created by a creator. However, the haptic signal 20 may also be an actual vibration signal obtained from a vibration sensor such as a piezo pickup or an acceleration sensor.
[0035] The encoding device 21 is configured with, for example, a CPU (Central Processing Unit) and a DSP (Digital Signal Processor), and encodes each haptic signal 20 according to a predetermined transmission data format. The encoding device 21 then stores the haptic encoded data Dc obtained by encoding in, for example, an internal storage device.
[0036] The haptic reproduction device 22 is configured with a CPU, a DSP, etc., and acquires the haptic encoded data Dc. For example, the haptic encoded data Dc encoded in the production environment 11 is acquired by the haptic reproduction device 22 via a required network such as the Internet. Alternatively, the haptic encoded data Dc may be acquired by the haptic reproduction device 22 via a portable storage medium on which the haptic encoded data Dc is recorded by the encoding device 21.
[0037] The haptic reproduction device 22 decodes the acquired haptic encoded data Dc and drives each haptic presentation device 23 based on the haptic signal obtained by the decoding.
[0038] The tactile presentation device 23 is a device that generates a tactile stimulus. In Fig. 2, the tactile presentation device 23 is configured by, for example, a vibration device such as a vibrator or an actuator. Furthermore, each tactile presentation device 23 in Fig. 2 may be worn on a different part of the human body of the touch receiver.
[0039] The tactile presentation device 23 may be a device that is not attached to a part of the human body. For example, the tactile presentation device 23 may be composed of focused ultrasound waves or an air cannon using a speaker array, a pressure presentation device that controls the amount of fluid inside the device, an electrical stimulation device that directly stimulates tactile receptors, etc. Also, a part of the tactile presentation device 23 may be attached to a part of the human body, and another part of the tactile presentation device 23 may not be attached to a part of the human body.
[0040] <Configuration of Encoding Device> FIG. 3 is a block diagram showing an example of the configuration of the encoding device 21 in FIG.
[0041] 3 is configured to include a plurality of haptic signal input units 31-1 to 31-m, an encoding unit 32, a control unit 33, a storage unit 34, a communication unit 35, and a bus 36. When there is no need to distinguish between the haptic signal input units 31-1 to 31-m, they will be referred to as haptic signal input unit 31.
[0042] The encoding unit 32, the control unit 33, the storage unit 34, and the communication unit 35 are connected via a bus 36 and are configured to be able to communicate data with one another.
[0043] The haptic signal input from the haptic signal input unit 31 is input to the encoding unit 32 .
[0044] The encoder 32 is configured by, for example, a DSP. The encoder 32 encodes the input haptic signal in accordance with a predetermined transmission data format to obtain encoded haptic data Dc. The transmission data format will be described in detail later.
[0045] The control unit 33 is configured to include a microcomputer having, for example, a CPU, a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and performs overall control of the encoding device 21 by executing processing in accordance with a program stored in the ROM.
[0046] The control unit 33 is capable of performing data communication with an external device connected to the network via the communication unit 35. In particular, the control unit 33 is capable of transmitting the haptic encoded data Dc obtained by the encoding unit 32 to the external device via the communication unit 35.
[0047] The storage unit 34 is configured to include a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 34 is used to store various types of data in the encoding device 21. For example, the storage unit 34 stores data necessary for control by the control unit 33. The haptic encoded data Dc obtained by the encoding unit 32 may also be stored in the storage unit 34 under the control of the control unit 33.
[0048] The communication unit 35 is configured to be able to perform data communication with an external device via a network such as the Internet.
[0049] <Configuration of Haptic Reproduction Device> FIG. 4 is a block diagram showing an example of the configuration of the haptic reproduction device 22 of FIG.
[0050] In FIG. 4, the tactile reproduction device 22 is shown together with the tactile presentation device 23 .
[0051] The haptic reproduction device 22 is configured to include a plurality of amplifiers 41-1 to 41-n, a plurality of D / A (Digital to Analog) converters 42-1 to 42-n, a modulation unit 43, a decoding unit 44, a control unit 45, a storage unit 46, a communication unit 47, and a bus 48. When there is no need to distinguish between the amplifiers 41-1 to 41-n and the D / A converters 42-1 to 42-n, they will be referred to as amplifiers 41 and D / A converters 42, respectively.
[0052] The modulation unit 43, the decoding unit 44, the control unit 45, the storage unit 46, and the communication unit 47 are connected via a bus 48 and are configured to be able to communicate data with one another.
[0053] The control unit 45 is configured to include, for example, a microcomputer having a CPU, ROM, RAM, etc., and controls the entire tactile reproduction device 22 and controls the tactile presentation device 23 to present tactile stimulation.
[0054] The storage unit 46 is, for example, a storage device similar to the storage unit 34 in FIG. 3, and is used to store various data used by the control unit 45 and the like.
[0055] The communication unit 47 is configured to be able to perform data communication with an external device via a network such as the Internet. The haptic encoded data Dc transmitted from the encoding device 21 is received by the communication unit 47.
[0056] The decoding unit 44 decodes the haptic encoded data Dc input via the communication unit 47 using a method described below to obtain a haptic signal for each tactile presentation device 23. The haptic signal obtained by the decoding unit 44 is input to the modulation unit 43.
[0057] The modulation unit 43 determines a modulation frequency for the haptic signal obtained by the decoding unit 44 based on the information encoded by the encoding unit 32 so that the haptic reproduction device 22 can be driven efficiently and perceptual degradation is minimized, and modulates the decoded signal.
[0058] Each haptic signal that has passed through the modulation unit 43 is input to a corresponding D / A converter 42 and D / A converted, then adjusted to an appropriate dynamic range by a corresponding amplifier 41, and output to a corresponding tactile presentation device 23. As described above, a corresponding D / A converter 42 may be provided for each haptic signal that has passed through the modulation unit 43, or a configuration may be adopted in which each haptic signal is collectively processed by a single D / A converter 42. Alternatively, a configuration may be adopted in which a corresponding D / A converter 42 is provided for some of the haptic signals, and the remaining haptic signals are collectively processed by a single D / A converter 42.
[0059] This allows each tactile presentation device 23 to be driven based on the tactile signal, and the tactile stimulus created in the production environment can be given to the haptic receiver.
[0060] Although only tactile signals have been mentioned in the above explanation, it is also possible to record audio signals and video signals together with the tactile signals, thereby providing the tactile receiver with sound and video together with tactile information.
[0061] <Processing Flow of Tactile Presentation System> FIG. 5 is a diagram showing the processing flow of the tactile presentation system 1. As shown in FIG.
[0062] FIG. 5 shows an example in which a haptic signal created by a creator 50 is coded by a coding device 21, and is decoded by a haptic reproduction device 22 owned by a user 51, and a predetermined haptic sensation is presented.
[0063] A creator 50 creates a haptic signal 53 using a design tool 52. The created haptic signal 53 is associated with a presentation site to a user 51 in the design tool 52. The haptic signal 53 associated with the presentation site is input to the encoding device 21.
[0064] The encoding device 21 obtains the haptic encoded data Dc multiplexed in the encoding unit 32. The haptic encoded data Dc is then transmitted from the communication unit 35 to the haptic reproduction device 22. At that time, the encoding unit 32 also encodes haptic part information indicating the presentation part associated with the haptic signal.
[0065] In the haptic reproduction device 22, the haptic encoded data Dc is received by the communication unit 47 and decoded by the decoding unit 44. Based on the decoded information, the haptic signals obtained by the decoding are transmitted to the haptic presentation device 23 worn on the presentation site associated with each haptic signal. In FIG. 5 , the haptic reproduction device 22 is worn by the user 51 and connected to a sensor 54 that measures the state of the user 51 (e.g., the state of use of the haptic presentation device 23). The sensor 54 may be any sensor capable of measuring the state of the user 51, such as a body temperature sensor, a blood flow sensor, a blood pressure sensor, a pulse sensor, or an electroencephalogram sensor. Alternatively, a sensor other than those described above (e.g., an imaging sensor such as a camera, an acceleration sensor, an angular velocity sensor, etc.) may also be used.
[0066] At this time, the modulation unit 43 determines a modulation frequency based on the tactile site information encoded by the encoding unit 32, information about the unmodulated signal (described later), external user input, or input from the sensor 54, and performs modulation processing on the tactile signal. This modulation processing modulates the tactile signal to a signal of a certain frequency while retaining the physical or perceptual intensity information of the signal before modulation. For example, amplitude modulation is used for the modulation processing.
[0067] The modulated signal is presented to the user 51 as a tactile stimulus through the tactile presentation device 23 .
[0068] <Configuration of Modulation Unit> FIG. 6 is a block diagram showing an example configuration of the modulation unit 43 in FIG.
[0069] The modulation section 43 is configured to include a presentation part acquisition section 60 , a presentation part table 61 , a non-modulation band acquisition section 62 , a modulation tolerance range calculation section 63 , a frequency determination section 64 , and an amplitude modulation section 65 .
[0070] The decoded signal obtained by the decoding unit 44 is supplied to the presentation part acquisition unit 60 and the non-modulation band acquisition unit 62. The decoded signal obtained by the decoding unit 44 includes haptic encoded data Dc.
[0071] The presentation part acquisition unit 60 acquires presentation part information using an ID (referred to as a part ID) indicating the presentation part included in the decoded signal (i.e., the haptic encoded data Dc) supplied from the decoding unit 44 and a presentation part table 61 stored in the storage unit 46. The presentation part acquisition unit 60 outputs the acquired presentation part information to a modulation allowable range calculation unit 63.
[0072] The presentation part table 61 is a table made up of information indicating, for example, that ID=0 is the thumb, ID=1 is the index finger, ..., ID=10 is the right upper arm, ..., ID=44 is the sole of the left foot, .... Here, which presentation part each ID corresponds to may be determined in advance by the creator 50 or by the user 51.
[0073] The part ID may be assigned when the haptic signal is created. Instead of acquiring the part ID and presentation part information from the presentation part table 61, the haptic reproduction device 22 may receive device information of the haptic presentation device 23 and acquire, from the received device information, attachment part information indicating the part on which the presentation device 23 is attached, i.e., presentation part information. The user 51 may also add, delete, or otherwise modify the IDs assigned when the haptic signal is created, as appropriate.
[0074] The non-modulated band acquisition unit 62 acquires the band of the non-modulated signal (component) that is not modulated by the amplitude modulation unit 65 (hereinafter referred to as the non-modulated band). The decoded signal (i.e., the haptic encoded data Dc) supplied from the decoding unit 44 also includes data of the non-modulated signal. Therefore, the non-modulated band acquisition unit 62 performs a known frequency analysis on the data of the non-modulated signal to acquire non-modulated band information indicating the value of the frequency range (bandwidth) of the non-modulated band. The non-modulated band acquisition unit 62 outputs the acquired non-modulated band information to the modulation tolerance calculation unit 63.
[0075] The modulation tolerance calculation unit 63 calculates the modulation tolerance range based on the presentation part information supplied from the presentation part acquisition unit 60 and the non-modulation band information supplied from the non-modulation band acquisition unit 62 .
[0076] When amplitude-modulating a haptic signal and presenting it to a predetermined part of the user, the modulation tolerance calculation unit 63 calculates a range of modulation frequencies (hereinafter referred to as the modulation tolerance range) that will not cause perceptual degradation before and after amplitude modulation. In other words, if the modulation frequency at which the haptic signal is amplitude-modulated is within the modulation tolerance range obtained by the modulation tolerance calculation unit 63, the user's perception will not be easily affected by the amplitude modulation. The method for calculating the modulation tolerance range will be described later with reference to FIG. 7.
[0077] The modulation tolerance calculation unit 63 outputs information indicating the calculated modulation tolerance range to the frequency determination unit 64 .
[0078] The frequency determination unit 64 determines the modulation frequency. First, the frequency determination unit 64 acquires the original modulation frequency (hereinafter, desired modulation frequency) f0. This desired modulation frequency f0 is set so as to obtain the benefits of amplitude modulation using known methods, such as reduction of vibration noise, the resonance frequency of the tactile presentation device 23, and human tactile sensitivity, and is stored in advance in the storage unit 46.
[0079] Next, the frequency determination unit 64 calculates the modulation frequency f from the desired modulation frequency f0 and the permissible modulation range indicated by the permissible modulation range information supplied from the permissible modulation range calculation unit 63. The frequency determination unit 64 outputs the calculated modulation frequency f to the amplitude modulation unit 65.
[0080] The parameter α, which indicates the degree of avoidance of perceptual degradation and is used to calculate the modulation frequency f, may be stored in advance in the storage unit 46, or may be set according to the user's preference through input 56 from the user 51. As described above with reference to FIG. 5, the user 51 wears a sensor 54 that is connected to the haptic reproduction device 22 and measures the user's own condition. The parameter α may be determined based on the input value of the sensor 54. The method for calculating the modulation frequency f and the parameter α will be described later with reference to FIG. 8.
[0081] The amplitude modulation unit 65 modulates the decoded haptic signal based on the modulation frequency f determined by the frequency determination unit 64, and outputs the modulated haptic signal.
[0082] <Method of Calculating Permissible Modulation Range> FIG. 7 is a diagram showing a method of calculating the permissible modulation range.
[0083] The modulation tolerance is calculated from two perspectives.
[0084] The first aspect is the tactile perception characteristics of the tactile receptors shown in Figure 7A. Amplitude modulation is a process that changes the frequency of a signal. Therefore, the stimulation frequencies before and after amplitude modulation are both set within the frequency range (hereinafter referred to as the Pacinian band) 70 to which Pacinian corpuscles, which are insensitive to frequency changes, respond predominantly.
[0085] The second point of view is interference with the non-modulation band shown in B of Fig. 7. As described above, since modulation needs to be performed only within the Pacinian band 70, a modulation band 72 and a non-modulation band 73 are generated within the frequency spectrum 71 of the original signal. Therefore, the modulation frequency is set outside the non-modulation band 73 so that the non-modulated signal in the non-modulation band 73 does not interfere with the modulated signal.
[0086] For example, modulating a modulation band of 100 to 600 Hz to 300 Hz means changing 600 Hz to 300 Hz and also changing 100 Hz to 30 Hz. If the Pacinian band is 150 Hz or higher, the 100 to 150 Hz band will be modulated from outside the Pacinian band to within the band. In other words, to meet the condition of "modulating within the Pacinian band" to suppress degradation, a band that cannot be modulated (in this case, below 150 Hz) will emerge, and this band will have to become at least a non-modulated band.
[0087] However, as will be described later, depending on the degree of degradation to be tolerated, the entire band may be modulated, and then the modulation frequency may be determined to minimize the degradation as much as possible (in this case, the condition for the non-modulated band is 0 Hz≦f, so modulation is essentially performed with only the Pacinian band in mind).
[0088] Note that "outside the Pacinian band" does not equal "non-modulated band" because, in the example above, it is assumed that the non-modulated band may extend up to 300 Hz. Pacinian corpuscles are merely "insensitive" to frequency changes. Therefore, in the case of an ideal vibration device with excellent frequency characteristics and low noise, the smaller the modulated band is, the higher the perceived quality will be, albeit slightly. Therefore, it is assumed that the non-modulated band may overlap with the Pacinian band.
[0089] The modulation tolerance calculation unit 63 calculates a frequency range that satisfies the above two conditions.
[0090] A more specific calculation method will now be described.
[0091] First, the tactile reception characteristics of a human are estimated to obtain the Pacinian band 70. The Pacinian band 70 can be obtained by the modulation tolerance calculation unit 63 reading out the frequency range for each body part pre-stored in the storage unit 46 in accordance with the presentation body part information acquired by the presentation body part acquisition unit 60. Note that if it is considered that the difference in the Pacinian band 70 depending on the body part of a human is small, only one frequency range may be recorded in the storage unit 46 and read out regardless of the presentation body part.
[0092] Here, if the Pacinian band 70 is [fp1, fp2], the first condition is imposed as shown in the following equation (1) so that the modulation frequency f falls within this Pacinian band [fp1, fp2]: fp1 ≦ f ≦ fp2 (1)
[0093] Furthermore, if the non-modulation band 73 is [fnm1, fnm2], the second condition is imposed as shown in the following equation (2) so that the modulation frequency f is not included in the non-modulation band [fnm1, fnm2] acquired by the non-modulation band acquisition unit 62: f<fnm1, fnm2<f (2)
[0094] The modulation tolerance calculation unit 63 calculates the range of modulation frequency f that satisfies these two conditions simultaneously as the modulation tolerance range.
[0095] <Method of Calculating Modulation Frequency> FIG. 8 is a diagram showing a method of calculating the modulation frequency f.
[0096] As shown in FIG. 8A, if the desired modulation frequency f0 satisfies the modulation tolerance range 80, the frequency determination unit 64 outputs the desired modulation frequency f0 as is as the modulation frequency f.
[0097] On the other hand, as shown in B of FIG. 8, if the desired modulation frequency f0 does not satisfy the modulation tolerance 80, the modulation tolerance calculation unit 63 determines a new modulation frequency f based on the modulation tolerance 80.
[0098] When the frequency that satisfies the modulation tolerance range 80 and is closest to the desired modulation frequency f0 is defined as f_near, the modulation frequency f is determined by the following equation (3): f = αf0 + (1 - α)f_near, 0≦α≦1 (3)
[0099] Here, α is a parameter that indicates the degree of avoidance of perceptual degradation in order to balance the avoidance of perceptual degradation and the enjoyment of the benefits of amplitude modulation. When the parameter α=1, the modulation tolerance range is ignored and the benefits of amplitude modulation are prioritized. When the parameter α=0, avoidance of perceptual degradation is given top priority.
[0100] As described above, this parameter α may be one that is pre-stored in the storage unit 46 depending on the use case, or may be set according to the user's preference through input 56 by the user 51. Furthermore, the parameter α may be determined based on the input value of the above-mentioned sensor 54 that is connected to the haptic reproduction device 22.
[0101] For example, the sensor 54 may be an acceleration sensor that measures the degree of contact between the tactile presentation device 23 and the user 51, and the parameter α may be set in the frequency determination unit 64 in accordance with the contact information measured by the sensor 54. This will be described in detail later.
[0102] This allows users to maximize the benefits of amplitude modulation when the modulation frequency provides the benefits without causing perceptual degradation. On the other hand, when the modulation frequency provides the benefits of amplitude modulation but may cause perceptual degradation, modulation can be performed by balancing perceptual degradation with the benefits of amplitude modulation.
[0103] As described above, by setting the parameter α that indicates the degree of avoidance of perceptual degradation and determining the modulation frequency f, amplitude modulation can be performed while balancing the benefits of amplitude modulation and avoidance of perceptual degradation depending on the presentation area, the frequency band of the unmodulated signal, and the state of the user 51 (the state of use of the tactile presentation device 23), and the user can perceive the tactile sensation that the creator originally intended to present while enjoying the benefits of amplitude modulation.
[0104] 3. First Processing Example Reproduction Process of Haptic Signal Created by Creator FIG. 9 shows a first processing example in which a haptic signal created by a creator is reproduced by a haptic presentation device unknown to the user.
[0105] The creator 90 creates a haptic signal to be presented using a design tool 91. A human body diagram 92 is displayed in the design tool 91. When the creator 90 selects a part on the human body diagram 92, the design tool 91 associates the haptic signal with the selected presentation part.
[0106] The design tool 91 generates four haptic signals 93-1 to 93-4 in response to the selection of the creator 90. The generated four haptic signals 93-1 to 93-4 are associated by the design tool 91 with, for example, the left hand, the right hand, the back, and the soles of both feet, respectively.
[0107] As a modified example, the creator 90 may associate a part ID corresponding to a desired part by manually inputting the part ID corresponding to the desired part into a UI of a text box provided in the design tool 91. Furthermore, in cases where the creator 90 using the design tool 91 is a beginner or unfamiliar with creating content using haptic signals, it may be preferable for the creator 90 to refer to content data created by well-known creators. Therefore, content data created by well-known creators may be stored as reference data within the design tool 91 or on a server, so that the creator 90 can select it as appropriate.
[0108] The encoder 21 encodes the haptic signals 93-1 to 93-4 created by the design tool 91 to generate encoded haptic data Dc. The encoder 21 assigns at least a region ID and unmodulated signal data to each of the haptic signals 93-1 to 93-4 for each region. For signals in the modulation band, only the intensity data or the signal itself is encoded.
[0109] The generated haptic encoded data Dc is acquired by the haptic reproduction device 22 of the user 94 and decoded by the decoding unit 44. The decoding unit 44 outputs each decoded signal to the modulation unit 43 connected to the haptic presentation device 23.
[0110] The modulation unit 43 identifies the tactile sensation presentation region from the decoded region ID. In Fig. 9, IDs corresponding to the left hand, right hand, back, and soles of the feet have been assigned by the design tool 91, so the modulation unit 43 acquires one of these pieces of tactile sensation presentation region information as the presentation region.
[0111] Based on the tactile site information, the modulation unit 43 reads out the Pacinian band information for each site pre-stored in the storage unit 46 of the tactile reproduction device 22, and determines the value as the Pacinian band. In parallel, the modulation unit 43 estimates the unmodulated band from the data of the decoded unmodulated signal.
[0112] The modulation unit 43 calculates the modulation allowable range based on the obtained Pacinian band information and non-modulated band information. For example, if the Pacinian band is 150 Hz or higher for the left and right hands and soles of the feet, 200 Hz or higher for the back, and the non-modulated bands are all 0 to 200 Hz, then the modulation allowable range for the left and right hands and soles of the feet is 200 Hz or lower, which is the range of f that simultaneously satisfies the Pacinian band condition of 150 Hz or lower and the non-modulated band condition of 200 Hz or lower. Similarly, a range of 200 Hz or higher is calculated for the back as the modulation allowable range that is included in the Pacinian band but not included in the non-modulated band.
[0113] The modulator 43 determines the modulation frequency based on the modulation tolerance and the desired modulation frequency. For example, suppose that tactile presentation devices 23-1 and 23-2 with a resonant frequency of 300 Hz are used for the left and right hands, tactile presentation device 23-3 with a resonant frequency of 200 Hz is used for the soles of the feet, and tactile presentation device 23-4 with a resonant frequency of 50 Hz is used for the back. The desired modulation frequency is set to the same value as the resonant frequency to improve drive efficiency.
[0114] In the example of Figure 9, the parameter α is set to 0 to avoid perceptual degradation. In this case, the desired modulation frequency of 300 Hz for the left and right hands satisfies the modulation tolerance range of 200 Hz or more. Therefore, the modulation frequency is set to 300 Hz, as is the desired modulation frequency. The desired modulation frequency of 200 Hz for the soles also satisfies the modulation tolerance range of 200 Hz or more, so the modulation frequency is set to 200 Hz.
[0115] On the other hand, the desired modulation frequency of 50 Hz for the back does not satisfy the modulation tolerance range of 200 Hz or more. Therefore, the modulator 43 adopts f = 0 × f0 + 1 × f_near = 200 Hz as the modulation frequency based on the frequency f_near = 200 Hz that satisfies the modulation tolerance range and is closest to the desired modulation frequency.
[0116] The modulation unit 43 modulates these signals using the respective modulation frequencies, adds them to the unmodulated signals, and outputs the results from the tactile presentation devices 23-1 to 23-4, respectively, to present tactile stimulation to the user 94.
[0117] As a result, since the creator does not know what type of tactile presentation device the user will use, it is not clear at the production stage whether modulation that takes perceptual degradation into consideration is necessary.However, the tactile reproduction device 22 can determine the modulation frequency based on tactile perception characteristics and unmodulated signals only when it is necessary to avoid perceptual degradation, and when it is not necessary to avoid perceptual degradation, it can maximize the benefits of amplitude modulation (for example, by setting the parameter α = 1 and determining the desired modulation frequency f0 as the modulation frequency).
[0118] <Modification of Encoding> When it is determined that at least a portion of the band of the haptic signal is to be modulated in the haptic reproduction device 22, only the intensity data of the signal in the modulation band may be encoded. In this case, the amount of information can be reduced compared to encoding the waveform data itself.
[0119] The "intensity data" here may be waveform amplitude or perceived intensity information converted into a human perceptual quantity. The perceived intensity information may be a Sensation Level, which indicates how many times the amplitude is above the vibration detection threshold, which is the threshold for detecting vibrations in humans.
[0120] When it is unknown whether modulation will be performed in the haptic reproduction device 22, the waveform data is encoded as is. This allows the haptic reproduction device 22 to select whether or not to perform modulation.
[0121] Furthermore, the amount of information may be reduced by encoding non-modulated signal data and modulation band signal data when waveform data is encoded as is using a known encoding technique.
[0122] Furthermore, the boundary frequency between the modulated band and the non-modulated band, which was set when the signal was created, may be included in the haptic encoded data Dc. In this case, the process of acquiring the non-modulated band in the modulator 43 changes from signal analysis such as spectrum analysis to simple data reading. This increases the amount of information in the haptic encoded data Dc, but in return, it reduces the load of the non-modulated band acquisition process on the haptic reproduction device 22.
[0123] <Transmission Data Format> Fig. 10 is a diagram showing an example of a transmission data format of data transmitted from the encoding device 21 to the haptic reproduction device 22. Fig. 10 is merely one example of a transmission data format, and the configuration of the transmission format may be a format configuration different from that shown in Fig. 10 (for example, the content or order of the data).
[0124] The data is configured to include areas 100 to 103 and data area 104-1, data area 104-2, etc. When there is no need to distinguish between the data area 104-1, data area 104-2, etc., they will be referred to as the data area 104.
[0125] Area 100 stores an identifier. Area 101 stores the signal sampling frequency. Area 102 stores the quantization precision. Area 103 stores the number of samples of signal data for each multiplexed signal. Data area 104 stores the signal data for each multiplexed signal.
[0126] The data area 104 is configured to include areas 111 to 116.
[0127] A region ID flag indicating whether or not a region ID is associated with the signal presentation region is stored in region 111. A region ID is stored in region 112 when the region ID flag is 1. When the region ID flag is 0, the data in region 112 is omitted.
[0128] Intensity data (intensity information) is stored in area 113. The intensity data is modulation intensity information used for modulation. As described above, when the waveform data is encoded as is, the waveform data is stored in area 113, for example.
[0129] Area 114 stores a boundary frequency flag indicating whether or not there is a boundary frequency between the modulated band and the non-modulated band.
[0130] Area 115 stores the boundary frequency if the boundary frequency flag is 1. If the boundary frequency flag is 0, the data in area 115 is omitted.
[0131] Area 116 stores data of a non-modulated signal, which is a component that is not modulated.
[0132] While it is desirable to transmit the entire contents of the data format shown in Fig. 10, there may be cases where the communication bandwidth is insufficient, or the haptic reproduction device's computational resources or remaining battery power is insufficient. Therefore, in the above-mentioned cases, instead of transmitting the entire contents of the data format shown in Fig. 10, it is also possible to transmit only a necessary portion (for example, transmitting only the data area 104). Furthermore, when packet loss occurs, the missing portion of the signal may be retransmitted as appropriate.
[0133] <Presented Part Determination Process in Haptic Movie> FIG. 11 is a diagram showing presented part determination process in a haptic movie, which is a modification of the first process in FIG.
[0134] FIG. 11 shows an example in which a creator adds a haptic signal to existing video content or creates new video content that includes a haptic signal.
[0135] When the creator 90 creates a tactile signal using the design tool 91, the creator 90 does not specify the presentation location of the tactile signal; the design tool 91 determines and assigns the appropriate presentation location based on the properties of the tactile signal and sensor information (modal signals) obtained from other sensors.
[0136] In creating haptic content, it is assumed that a video signal 120, an audio signal 121, and haptic signals 122 to 124 are handled in a time-synchronized manner. Here, as a first method, the design tool 91 determines the presentation portions of the haptic signals 122 to 124 from the properties obtained by analyzing the haptic signals.
[0137] For example, suppose that it is found by known techniques such as calculating the peak position of a cross-correlation function or spectral analysis that haptic signals 122 and 123 are synchronized in time and contain many low-frequency components. Therefore, it is determined that the synchronized low-frequency components can be presented to both feet to create a powerful impact, and the left foot and right foot are designated as the presentation sites for haptic signals 122 and 123, respectively.
[0138] As another example, the presentation portion may be determined based on the properties of the video signal 120 and the audio signal 121 .
[0139] For example, based on known acoustic scene analysis techniques or the results of scene analysis of video signal 120, it may be determined that the sound contained in section 131 of audio signal 121 is a gunshot, and that this sound is well synchronized with haptic signal waveform 132 in the same section 131.
[0140] In this case, the design tool 91 first determines that the haptic signal waveform 132 is associated with a gunshot, and then designates the ear as the presentation location for the haptic signal 124 in order to create the sense of tension caused by, for example, a bullet grazed by the ear.
[0141] For example, this process may use a table showing the correspondence between signal properties and presentation areas. For example, this table may be stored in advance in the storage unit 34 of the encoding device 21, or may be set in advance by the creator 90 through the design tool 91. This table may also be obtained from the Internet, or may be obtained by the design tool learning the creator's presentation area settings. The creator may also select these tables as appropriate.
[0142] In this way, by determining the presentation site of a haptic signal, the cost of creating haptic content, which requires specifying the site for a large number of signals, can be reduced. Then, the modulation processing of the present technology can be used based on the site information specified in this way.
[0143] 4. Second Processing Example Dynamic Setting Process for Avoiding Degree of Perceptual Degradation in VR (Virtual Reality) FIG. 12 is a diagram showing a second processing example in which a haptic sensation is presented to a user in VR, AR (Augmented Reality), or the like.
[0144] Consider, for example, a VR game as an application to which the present technology can be applied.
[0145] A user 141, who is a player of the game, wears a haptic reproduction device 22 and a plurality of haptic presentation devices 23 connected thereto on his or her body. In this state, when a virtual user 143 in a VR space 142 touches a virtual substance 144, haptic encoded data Dc is transmitted to the haptic reproduction device 22 according to the properties and contact method set for the virtual substance 144, and a haptic stimulus is presented to the user 141 via the haptic presentation device 23. In this way, the user 141 can have an experience as if he or she were touching the virtual substance 144 in the VR space 142.
[0146] In the second processing example, for example, the hands and soles of the feet play an important role in ascertaining the feel of an object or the feel of the ground, and therefore, deterioration of perception in these areas should be avoided as much as possible.
[0147] On the other hand, the back may not require precise discrimination of stimuli, but the presentation area is large and the sensitivity is low, so power consumption may be high to provide sufficient stimulation. Therefore, in areas with large presentation areas and low sensitivity such as the back, it is highly valuable to make maximum use of amplitude modulation and improve the drive efficiency of the tactile presentation device 23.
[0148] Therefore, for example, if the presentation site is a hand or a sole, the haptic reproduction device 22 sets the parameter α=0 to avoid perceptual degradation and determines the modulation frequency with emphasis on the modulation tolerance range. On the other hand, for other sites, such as the abdomen or upper arm, the haptic reproduction device 22 sets the parameter α=0.5 to avoid perceptual degradation and improve perceptual sensitivity to some extent.
[0149] For example, in a region where tactile sensation is particularly weak, such as the back, the tactile reproduction device 22 sets the parameter α to 1, that is, ignores the modulation tolerance range and prioritizes improving the drive efficiency.
[0150] As described above, the parameter α, which indicates the degree to which perceptual degradation is avoided, is adjusted depending on the region where the tactile stimulus is presented. This allows perceptual degradation to be avoided in regions with high sensitivity, the benefits of improved efficiency due to modulation to be enjoyed in regions with low sensitivity, and modulation to be performed after adjusting the parameter so that the benefits of both can be achieved to some extent in regions with low sensitivity.
[0151] The balance between avoiding perceptual degradation and enjoying the benefits of modulation may also be determined by connecting another sensor 145 provided in the system including the user 141 to the haptic reproduction device 22 and using the information from that sensor.
[0152] For example, the sensor 145 is an acceleration sensor that measures the degree of contact between the user 141 and each tactile presentation device 23 .
[0153] If the degree of contact is low, the tactile presentation to the user 141 becomes unstable, and there is a possibility that perception may be interrupted by a low-energy stimulus. Since interruption of the tactile presentation causes a greater loss to the user experience than perceptual degradation of the signal itself, it is desirable to avoid this as a priority.
[0154] In this case, the tactile reproduction device 22 may ignore the modulation tolerance range and prioritize modulation to the resonant frequency of the tactile presentation device 23 or a frequency higher than the human tactile sensitivity so that the user 141 can perceive the tactile sensation with high sensitivity. In other words, the parameter α may be set to a large value.
[0155] On the other hand, when the degree of contact is high, there is no need to worry, and it is thought that the user 141 will carefully perceive the presented tactile sensation. In this case, the user 141 will be more likely to notice perceptual degradation, so the tactile reproduction device 22 sets the parameter α to a small value to prevent perceptual degradation.
[0156] As another example, the sensor 145 may be a motion capture system that measures motion state information indicating the motion state of the user 141, such as the speed of the swing of the arms.
[0157] It is generally believed that humans are less sensitive to subtle differences in tactile sensations when they are moving than when they are stationary. Therefore, when the user 141 is moving significantly, the haptic reproduction device 22 may set the parameter α to a large value to prioritize driving efficiency and improved perceptual quality. On the other hand, when the user 141 is moving slowly, the haptic reproduction device 22 may set the parameter α to a small value to prioritize avoiding perceptual degradation.
[0158] This allows modulation to be performed to reflect the degree of contact and the state of movement, preventing interruptions in stimulation and degradation of perception appropriately according to the current state, even in use cases such as VR and AR, where the wearing state of the tactile presentation device 23 constantly changes as the user 141 moves around and the tactile presentation device 23 itself may become separated from the user, or where the user's state of movement changes and their perception sensitivity may also change.
[0159] <Decoding and Modulation Processing> FIG. 13 is a flowchart illustrating the decoding and modulation processing of the haptic reproduction device 22.
[0160] FIG. 13 shows a process in which the haptic encoded data Dc is decoded by the decoder 44 and a modulated signal is output by the modulator 43 .
[0161] In step S11, the decoding unit 44 receives the haptic encoded data Dc.
[0162] In step S12, the decoding unit 44 decodes the received signal to obtain the decoded signal (i.e., the haptic encoded data Dc).
[0163] In step S13, the presentation part acquisition unit 60 determines whether or not a part ID is included in the decoded signal. If it is determined in step S13 that a part ID is included in the decoded signal, the process proceeds to step S14.
[0164] In step S14, the presentation part acquisition section 60 acquires presentation part information based on the part ID and the presentation part table 61. The presentation part acquisition section 60 outputs the acquired presentation part information to the modulation allowable range calculation section 63.
[0165] If it is determined in step S13 that the decoded signal does not contain a region ID, the process proceeds to step S15.
[0166] In step S15, the presentation part acquisition unit 60 receives device information of the tactile presentation device 23 and acquires wearing part information, i.e., presentation part information, from the received device information. The presentation part acquisition unit 60 outputs the acquired presentation part information to the modulation allowable range calculation unit 63.
[0167] After steps S14 and S15, the process proceeds to step S16.
[0168] In step S16, the modulation tolerance calculation unit 63 acquires the Pacinian band based on the presentation part information.
[0169] In step S17, the non-modulated band acquisition unit 62 determines whether the decoded signal includes a boundary frequency. If it is determined in step S17 that the decoded signal does not include a boundary frequency, the process proceeds to step S18.
[0170] In step S18, the non-modulation band acquisition unit 62 performs signal analysis.
[0171] If it is determined in step S17 that the decoded signal contains a boundary frequency, the process skips step S18 and proceeds to step S19.
[0172] In step S19, the non-modulation band acquisition unit 62 acquires the non-modulation band (width) based on the boundary frequency or the result of the signal analysis. The non-modulation band acquisition unit 62 outputs the acquired non-modulation band information to the modulation allowable range calculation unit 63.
[0173] In step S20, the modulation tolerance calculation unit 63 calculates the modulation tolerance based on the Pacinian band acquired in step S17 and the non-modulation band information supplied from the non-modulation band acquisition unit 62. The modulation tolerance calculation unit 63 outputs modulation tolerance information indicating the calculated modulation tolerance to the frequency determination unit 64.
[0174] In step S21, the frequency determination unit 64 acquires the desired modulation frequency f stored in advance in the storage unit 46.
[0175] In step S22, the frequency determination unit 64 acquires a parameter α indicating the degree of avoidance of perceptual degradation from the storage unit 46, an external input, etc. Note that the parameter α may be calculated based on sensor information acquired from another sensor 54 connected to the haptic reproduction device 22, as described above.
[0176] In step S23, the frequency determination unit 64 determines the modulation frequency f based on the modulation tolerance range information, the desired modulation frequency f0, and the degree of avoidance of perceptual degradation. The frequency determination unit 64 outputs the calculated modulation frequency f to the amplitude modulation unit 65.
[0177] In step S24, the amplitude modulation unit 65 performs amplitude modulation on the decoded haptic signal based on the modulation frequency f determined by the frequency determination unit 64, and outputs the result as a modulated signal.
[0178] In step S25, the decoding unit 44 determines whether all signals have been decoded. If it is determined in step S25 that all signals have not been decoded, the process returns to step S11, and the subsequent processes are repeated.
[0179] If it is determined in step S25 that all signals have been decoded, the decoding and modulation process of FIG. 13 ends.
[0180] Note that the part acquisition in steps S13 to S15 in FIG. 13, the non-modulation band acquisition in steps S17 to S19, the acquisition of the desired modulation frequency f0 in step S21, and the acquisition of the degree of avoidance of perceptual degradation in step S22 are processes performed in parallel, and the order of these processes may be reversed.
[0181] <5. Other> <Effects of the Present Technology> As described above, in the present technology, the modulation frequency of a content signal is determined based on at least one of the tactile perception characteristics of each presentation part, information on a non-modulated signal that does not undergo modulation, and a user state, and the content signal is modulated at the modulation frequency.
[0182] This makes it possible to suppress perceptual degradation due to modulation while enjoying the benefits of modulation, such as improved drive efficiency and perceptual sensitivity, and reduced noise.
[0183] In the above description, a haptic signal of content has been used as an example, but the present technology is not limited to a haptic signal of content, and can also be applied to an audio signal or a video signal of content.
[0184] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.
[0185] FIG. 14 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.
[0186] The CPU 301 , ROM 302 , and RAM 303 are connected to one another via a bus 304 .
[0187] An input / output interface 305 is also connected to the bus 304. An input unit 306 including a keyboard, a mouse, etc., and an output unit 307 including a display, a speaker, etc. are connected to the input / output interface 305. In addition, a storage unit 308 including a hard disk, a nonvolatile memory, etc., a communication unit 309 including a network interface, etc., and a drive 310 that drives removable media 311 are also connected to the input / output interface 305.
[0188] In a computer configured as described above, the CPU 301 performs the above-described series of processes by, for example, loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program.
[0189] The program executed by the CPU 301 is provided, for example, by being recorded on a removable medium 311 or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit 308 .
[0190] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.
[0191] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.
[0192] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.
[0193] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.
[0194] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.
[0195] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.
[0196] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.
[0197] <Examples of Combinations of Configurations> The present technology may also be configured as follows. (1) An information processing device comprising: a frequency determination unit that determines a modulation frequency of a content signal based on at least one of tactile perception characteristics for each presentation site, information related to an unmodulated signal, and a user state; and a modulation unit that modulates the content signal at the modulation frequency. (2) The information processing device described in (1), wherein the frequency determination unit performs modulation at the modulation frequency determined by the frequency determination unit when avoidance of perceptual degradation is necessary, and performs modulation at the modulation frequency using a predetermined desired frequency as the modulation frequency when avoidance of perceptual degradation is not necessary. (3) The information processing device described in (1) or (2), wherein the content signal is at least one of a haptic signal, an audio signal, and a video signal. (4) The user state is a usage state of a device worn by a user. (5) The information processing device described in any of (1) to (3), wherein the frequency determination unit determines the modulation frequency according to a parameter indicating a degree of avoidance of perceptual degradation. (6) The information processing device according to (5), wherein the parameters are set according to sensor information acquired from a sensor worn by the user. (7) The information processing device according to (6), wherein the sensor information is at least one of motion state information and contact degree information. (8) The information processing device according to (5), wherein the parameters are set by a user's instruction or set in advance for each presentation site. (9) The information processing device according to any of (1) to (8), wherein the frequency determination unit determines the modulation frequency based on a frequency range to which Pacinian corpuscles at the presentation site predominantly respond, as the tactile perception characteristic for each presentation site. (10) The information processing device according to any of (1) to (9), wherein the frequency determination unit determines the modulation frequency based on a bandwidth of the unmodulated signal, as information related to the unmodulated signal.(11) The information processing device according to any of (1) to (10), further comprising: a part information acquisition unit that acquires presentation part information indicating the presentation part, wherein the frequency determination unit determines the modulation frequency based on tactile perception characteristics of the presentation part indicated by the presentation part information. (12) The information processing device according to any of (1) to (11), further comprising: a non-modulated signal information acquisition unit that acquires information about the non-modulated signal. (13) The information processing device according to any of (1) to (12), further comprising: a modulation tolerance range calculation unit that calculates a modulation tolerance range, which is a range within which modulation is permissible, based on the presentation part information indicating the presentation part and information about the non-modulated signal, wherein the frequency determination unit determines the modulation frequency based on the modulation tolerance range. (14) The information processing device according to (11), further comprising: a decoding unit that decodes an encoded signal of the content signal, the encoded signal including at least a presentation part identifier that identifies the presentation part, modulation intensity information, and the non-modulated signal, wherein the part information acquisition unit acquires the presentation part information using the decoded presentation part identifier. (15) The information processing device according to (14), wherein the presentation part identifier is set by an instruction from a creator. (16) The information processing device according to (14), wherein the presentation part identifier is set based on at least one of an analysis result of the content signal and other sensor information. (17) The information processing device according to (14), wherein the encoded signal includes information related to the unmodulated signal. (18) The information processing device according to (11), wherein the part information acquisition unit acquires the presentation part information from a presentation device that presents information corresponding to the content signal. (19) An information processing method, wherein an information processing device determines a modulation frequency of a content signal based on at least one of tactile perception characteristics for each presentation part, information related to an unmodulated signal, and a user state, and modulates the content signal using the modulation frequency.(20) A program that causes a computer to function as a frequency determination unit that determines a modulation frequency of a content signal based on at least one of tactile perception characteristics for each presentation part, information related to a non-modulated signal that does not undergo modulation, and a user state, and a modulation unit that modulates the content signal at the modulation frequency. (21) An information processing device comprising: an encoding unit that encodes the content signal and generates an encoded signal that includes at least information indicating the presentation part, modulation intensity information, and a non-modulated signal that does not undergo modulation.
[0198] 1 Tactile presentation system, 11 Production environment, 12 Presentation environment, 20, 20-1 to 20-m Tactile signal, 21 Encoding device, 22 Tactile reproduction device, 23-1 to 23-n Tactile presentation device, 31, 31-1 to 31-m Tactile signal input unit, 32 Encoding unit, 33 Control unit, 34 Memory unit, 35 Communication unit, 36 Bus, 41-1 to 41-n Amplifier, 42-1 to 42-n D / A converter, 43 Modulation unit, 44 Decoding unit, 45 Control unit, 46 Memory unit, 47 Communication unit, 48 Bus, 50 Creator, 51 User, 52 Design tool, 53 Tactile signal, 54 Sensor, 60 Presentation part acquisition unit, 61 Presentation part table, 62 Non-modulation band acquisition unit, 63 Modulation tolerance range calculation unit, 64 Frequency determination unit, 65 Amplitude modulation unit, 70 Pacinian band, 71 Frequency spectrum, 72 Band, 73 Non-modulation band, 80 Modulation tolerance range, 90 Creator, 91 Design tool, 92 Human body diagram, 93-1 to 93-4 Haptic signal, 94 User, 120 Video signal, 121 Audio signal, 122 to 124 Haptic signal, 141 User, 142 VR game, 144 Virtual material, 145 Sensor
Claims
1. An information processing device comprising: a frequency determination unit that determines a modulation frequency of a content signal based on at least one of the tactile perception characteristics of each presentation part, information regarding a non-modulated signal that does not undergo modulation, and a user state; and a modulation unit that performs modulation on the content signal at the modulation frequency.
2. The information processing device of claim 1, wherein the frequency determination unit performs modulation using the modulation frequency determined by the frequency determination unit when it is necessary to avoid perceptual degradation, and performs modulation using a predetermined desired frequency as the modulation frequency when it is not necessary to avoid perceptual degradation.
3. The information processing device according to claim 1, wherein the content signal is at least one of a haptic signal, an audio signal, and a video signal.
4. The information processing device according to claim 1, wherein the user state is a usage state of a device attached to the user.
5. The information processing device according to claim 1, wherein the frequency determination unit determines the modulation frequency in accordance with a parameter indicating a degree of avoidance of perceptual degradation.
6. The information processing device according to claim 5, wherein the parameters are set in accordance with sensor information acquired from a sensor attached to the user.
7. The information processing device according to claim 6, wherein the sensor information is at least one of exercise state information and contact degree information.
8. The information processing device according to claim 5, wherein the parameters are set by a user's instruction or are set in advance for each of the presentation parts.
9. An information processing device as described in claim 1, wherein the frequency determination unit determines the modulation frequency based on a frequency range to which the Pacinian corpuscles in the presentation site predominantly respond as a tactile perception characteristic for each presentation site.
10. The information processing device according to claim 1, wherein the frequency determination unit determines the modulation frequency based on a bandwidth of the unmodulated signal as information about the unmodulated signal.
11. An information processing device as described in claim 1, further comprising a part information acquisition unit that acquires presentation part information indicating the presentation part, wherein the frequency determination unit determines the modulation frequency based on the tactile perception characteristics of the presentation part indicated by the presentation part information.
12. The information processing device according to claim 1, further comprising a non-modulated signal information acquisition unit that acquires information related to the non-modulated signal.
13. An information processing device as described in claim 1, further comprising a modulation tolerance range calculation unit that calculates a modulation tolerance range, which is a range within which modulation is acceptable, based on presentation part information indicating the presentation part and information regarding the unmodulated signal, and the frequency determination unit determines the modulation frequency based on the modulation tolerance range.
14. The information processing device of claim 11, further comprising a decoding unit that decodes an encoded signal of the content signal that includes at least a presentation part identifier that identifies the presentation part, modulation intensity information, and the unmodulated signal, and the part information acquisition unit acquires the presentation part information using the decoded presentation part identifier.
15. The information processing device according to claim 14, wherein the presentation part identifier is set based on at least one of the analysis result of the content signal and other sensor information.
16. The information processing apparatus according to claim 14, wherein the encoded signal includes information about the unmodulated signal.
17. The information processing device according to claim 11, wherein the part information acquisition unit acquires the presented part information from a presentation device that presents information corresponding to the content signal.
18. An information processing method in which an information processing device determines a modulation frequency of a content signal based on at least one of tactile perception characteristics for each presentation area, information regarding a non-modulated signal that does not undergo modulation, and a user state, and modulates the content signal at the modulation frequency.
19. A program that causes a computer to function as a frequency determination unit that determines the modulation frequency of a content signal based on at least one of the tactile perception characteristics of each presentation part, information on a non-modulated signal that does not undergo modulation, and a user state, and a modulation unit that performs modulation on the content signal at the modulation frequency.
20. An information processing device comprising an encoding unit that encodes a content signal and generates an encoded signal including at least information indicating a presentation portion, modulation intensity information, and a non-modulated signal that is not modulated.
Citation Information
Patent Citations
Haptic pitch control
JP2019067396A
Method and apparatus for efficiently providing tactile information
JP2006351012A
Electronic device
JP2011183374A
Methods and systems for communicating emotions
JP2011523364A
Perception in the tactile system
JP2018507485A
