Tactile transmission system, tactile transmission device, tactile transmission program, and tactile transmission method

Abstract

To prevent howling and loop back of a contact signal in a bi-directional tactile sense transmission system.SOLUTION: A tactile sense transmission system 100 has a tactile sense transmission device 3 and an other tactile sense transmission device 3, and comprises: a vibration measuring unit 31 that measures vibration occurring in the tactile sense transmission device 3; a calculation unit that calculates perceptual information specified from the vibration measured by the vibration measuring unit 31; a conversion unit that converts a signal related to the vibration into a predetermined frequency while maintaining the perceptual information calculated by the calculation unit; and a signal output unit that causes an oscillator 32 of the other tactile sense transmission device 3 to output the signal after the conversion by the conversion unit as output vibration.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The technology described in this specification relates to a tactile transmission system, a tactile transmission device, a tactile transmission program, and a tactile transmission method.

Background Art

[0002] Technologies for communication by voice or video even at a remote location, such as telephones, video conferences, and social Virtual Reality (VR), have been developed. By transmitting tactile vibrations in addition to voice and video, more advanced communication than before becomes possible.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In tactile communication that transmits vibrations generated when a user touches a housing bidirectionally, it is desirable to mount a vibration sensor for measuring vibrations and an actuator for generating vibrations on the same housing. At this time, the contact vibration between the user and the housing added as an input appears centered on the amplitude modulation wave of the natural vibration of the housing. On the other hand, as an output, when the housing is vibrated by an actuator for tactile presentation, vibrations near the natural vibration of the housing also occur.

[0005] When a plurality of such tactile input / output devices are connected bidirectionally, since the sensor signal and the drive signal of the actuator looped back are driven near the same natural vibration, there is a possibility of a howling phenomenon occurring.

[0006] To suppress this feedback, acoustic signals are typically suppressed using methods such as bandpass filters to control signals in the natural vibration frequency band. However, in the case of tactile transmission, the tactile information to be transmitted also exists in the same frequency band, so lowering the gain may lead to a decrease in the quality of the contact information.

[0007] In one aspect, the technology described herein aims to suppress howling and loopback of contact signals in bidirectional haptic transmission systems. [Means for solving the problem]

[0008] In one aspect, the bidirectional haptic transmission system is a haptic transmission system having a haptic transmission device and another haptic transmission device, comprising: a vibration measuring unit that measures vibrations generated in the haptic transmission device; a calculation unit that calculates perceptual information identified from the vibrations measured by the vibration measuring unit; a conversion unit that converts a signal related to the vibrations to a predetermined frequency while maintaining the perceptual information calculated by the calculation unit; and a signal output unit that causes the converted signal converted by the conversion unit to be output as an output vibration to the vibrator of the other haptic transmission device, wherein the conversion unit attenuates a specific frequency band by filtering so as to suppress loopback of vibrations output by the vibrator. In another aspect, the bidirectional haptic transmission system is a haptic transmission system having a haptic transmission device and another haptic transmission device, comprising: a vibration measuring unit for measuring vibrations generated in the haptic transmission device; a calculation unit for calculating perceptual information identified from the vibrations measured by the vibration measuring unit; a conversion unit for converting a signal related to the vibration to a predetermined frequency while maintaining the perceptual information calculated by the calculation unit; and a signal output unit for causing the converted signal converted by the conversion unit to be output as an output vibration to the vibrator of the other haptic transmission device, wherein the conversion unit converts the signal to a frequency other than the resonant frequency of the housing of the haptic transmission device as the predetermined frequency. [Effects of the Invention]

[0009] One aspect of this is that it can suppress howling and loopback of contact signals in bidirectional tactile transmission systems. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic block diagram showing an example of the hardware configuration of a bidirectional haptic transmission system in an embodiment. [Figure 2] (a) and (b) are block diagrams schematically showing an example of the software configuration of the bidirectional haptic transmission system shown in Figure 1. [Figure 3] (a) is a block diagram showing a first example of the signal input processing unit shown in Figure 2, and (b) is a block diagram showing a first example of the signal output processing unit shown in Figure 2. [Figure 4] (a) is a block diagram showing a second example of the signal input processing unit shown in Figure 2, and (b) is a block diagram showing a second example of the signal output processing unit shown in Figure 2. [Figure 5] (a) is a block diagram showing a third example of the signal input processing unit shown in Figure 2, and (b) is a block diagram showing a third example of the signal output processing unit shown in Figure 2. [Figure 6] (a) to (c) are graphs illustrating the frequency conversion process performed by the frequency conversion unit shown in Figures 3 and 4. [Figure 7] This graph illustrates the amplitude measurement results for through output (when the signal is output without processing). [Figure 8] Figure 1 shows a table illustrating the results of suppressing howling when Intensity Segment Modulation (ISM) is used in the bidirectional haptic transmission system. [Figure 9] (a) is a graph illustrating the measurement results of output vibration in the case of through output, and (b) is a graph illustrating the measurement results of output vibration when using ISM. [Figure 10] This figure shows a first example of an environmentally installed vibration input / output device. [Figure 11]It is a diagram showing a second example of an environment-installed vibration input / output device. [Figure 12] It is a diagram showing a third example of an environment-installed vibration input / output device. [Figure 13] It is a diagram showing a list band-type vibration input / output device. [Figure 14] It is a diagram showing a body pillow-type vibration input / output device. [Figure 15] It is a diagram showing a tool attachment-type vibration input / output device. [Figure 16] It is a block diagram schematically showing a configuration example of a terminal as an embodiment. [Figure 17] (a) to (c) are graphs for briefly explaining the ISM process. [Figure 18] It is a graph showing the discriminability of vibrations by humans. [Figure 19] It is a sample waveform of vibrations used in a forced three-choice discrimination experiment conducted to determine the discriminability shown in the graph shown in FIG. 18. [Figure 20] It is a graph showing the waveforms of signals before and after conversion for each segment by the terminal shown in FIG. 16. [Figure 21] It is a graph representing the amplitude threshold value Tf used in the calculation of correction energy. [Figure 22] It is a graph representing the exponential threshold value b used in the calculation of correction energy. [Figure 23] It is a diagram for explaining the use of a window function in the terminal shown in FIG. 16. [Figure 24] It is a graph for explaining an example of synthesis of a low frequency and a high frequency in the terminal shown in FIG. 16. [Figure 25] It is a graph showing a specific example of the waveforms of signals before and after conversion by the terminal shown in FIG. 16. [Figure 26] It is a block diagram for explaining a functional configuration example of the ISM unit in the terminal shown in FIG. 16. [Figure 27] It is a block diagram for explaining a first embodiment of the vibration waveform generation process in the terminal shown in FIG. 16. [Figure 28]Figure 26 is a block diagram illustrating the details of the energy control process shown. [Figure 29] As a second embodiment of the vibration waveform generation process at the terminal shown in Figure 16, Figure 26 is a block diagram illustrating the separation process of low-frequency components in the energy control process. [Figure 30] (a) to (c) are graphs illustrating examples of generating oscillations according to ISM without exaggerating the waveform. [Figure 31] (a) to (c) are graphs illustrating the first example of emphasizing and separating high-frequency components above 3000Hz from a sound source. [Figure 32] (a) to (c) are graphs illustrating a second example of emphasizing and separating high-frequency components above 3000Hz from a sound source. [Figure 33] (a) to (c) are graphs illustrating an example of emphasizing and separating low-frequency components below 1000Hz from a sound source. [Figure 34] This is a block diagram illustrating the first modified example of the energy control process shown in Figure 26. [Figure 35] This is a block diagram illustrating a second modified example of the energy control process shown in Figure 26. [Figure 36] Figure 26 is a block diagram illustrating the details of the energy synthesis process shown. [Figure 37] Figure 26 is a block diagram illustrating the details of the process for generating the corrected vibration waveform shown. [Figure 38] This block diagram shows an example of a DAC configuration when using multiple vibration devices in the bidirectional tactile transmission system shown in Figure 1. [Figure 39] This block diagram shows an example of a DAC configuration when using a single vibration device in the bidirectional tactile transmission system shown in Figure 1. [Modes for carrying out the invention]

[0011] The embodiments will now be described with reference to the drawings. However, the embodiments shown below are merely illustrative, and there is no intention to exclude various modifications or applications of techniques not explicitly shown in the embodiments. In other words, these embodiments can be implemented with various modifications without departing from their spirit.

[0012] Furthermore, each figure is not intended to represent only the components shown in the figure, but may include other components. In the following figures, parts denoted by the same reference numerals indicate the same or similar parts unless otherwise specified.

[0013] [A] Embodiment [A-1] Bidirectional transmission processing of tactile sensations

[0014] Figure 1 is a schematic block diagram showing an example of the hardware configuration of the bidirectional haptic transmission system 100 in an embodiment.

[0015] The bidirectional haptic transmission system 100 comprises, as an example, two terminals 1 (in other words, haptic transmission devices), two Universal Serial Bus (USB) audio interfaces 2, two vibration input / output devices 3 (vibration input / output devices #1 and #2), and two amplifiers 4. Each of the two vibration input / output devices 3 comprises a vibration sensor 31 (in other words, a vibration measuring unit) and a vibrator 32 (in other words, an oscillator).

[0016] In Figure 1, the solid arrows indicate the signal flow when vibrations generated in vibration input / output device #2 are transmitted to vibration input / output device #1, and the dashed arrows indicate the signal flow when vibrations generated in vibration input / output device #1 are transmitted to vibration input / output device #2.

[0017] The vibration sensor 31 measures vibration and transmits the signal to terminal 1 via the USB audio interface 2. Terminal 1 performs ISM processing or through-output processing and transmits the signal to another terminal 1 at a remote location. The remote terminal 1 outputs vibrations generated from the transmitted signal to the vibrator 32 via the USB audio interface 2 and amplifier 4. Note that tactile sensations may be transmitted between three or more parties by providing three or more sets of terminals 1, USB audio interfaces 2, vibration input / output devices 3 and amplifiers 4. Furthermore, a mute function may be provided to turn off the vibration output from the vibrator 32 for users who do not wish to experience tactile sensation transmission.

[0018] Further details about ISM will be discussed later using Figures 17 to 39, etc.

[0019] Figures 2(a) and 2(b) are block diagrams schematically showing an example of the software configuration of the bidirectional haptic transmission system 100 shown in Figure 1.

[0020] As shown in Figures 2(a) and 2(b), the bidirectional tactile transmission system 100 functions as a vibration measurement unit 211, a signal amplification unit 212, a signal input processing unit 213, a signal transmission unit 214, a signal reception unit 215, a signal output processing unit 216, a signal amplification unit 217, and a vibration presentation unit 218 at each of the bases #1 and #2.

[0021] The functions of the vibration measurement unit 211 and the signal amplification unit 212 are realized by the vibration sensor 31 shown in Figure 1, the function of the signal input processing unit 213 is realized by the USB audio I / F 2 shown in Figure 1 and the CPU 11 of terminal 1 which will be described later using Figure 16, and the function of the signal transmission unit 214 may be realized by the communication I / F (not shown) of terminal 1 which will be described later using Figure 16. Furthermore, the function of the signal reception unit 215 is realized by the communication I / F (not shown) of terminal 1 which will be described later using Figure 16, the function of the signal output processing unit 216 is realized by the USB audio I / F 2 shown in Figure 1 and the CPU 11 of terminal 1 which will be described later using Figure 16, the function of the signal amplification unit 217 is realized by the amplifier 4 shown in Figure 1, and the function of the vibration presentation unit 218 is realized by the vibrator 32 shown in Figure 1.

[0022] The vibration measurement unit 211 measures vibrations in the housing that the user comes into contact with. The signal amplification unit 217 amplifies the signal measured by the vibration measurement unit 211. The signal input processing unit 213 performs signal processing described later using Figures 3 to 5, etc. The signal transmission unit 214 transmits the signal processed by the signal input processing unit 213 to another location. The tactile signal may be transmitted in synchronization with the video and audio. Alternatively, the tactile signal may be transmitted included in the audio channel of the video, etc.

[0023] The signal receiving unit 215 receives signals transmitted from other locations. The signal output processing unit 216 performs signal processing on the signals received by the signal receiving unit 215, as described later using Figures 3 to 5. The signal amplification unit 217 amplifies the signals processed by the signal output processing unit 216. The vibration presentation unit 218 vibrates the housing that the user touches in response to the signals amplified by the signal amplification unit 217.

[0024] Figure 3(a) is a block diagram showing a first example of the signal input processing unit 213 shown in Figure 2, and (b) is a block diagram showing a first example of the signal output processing unit 216 shown in Figure 2.

[0025] As shown in Figure 3(a), the signal input processing unit 213 includes the functions of an AD converter 2131, a bandstop filter 2132, an equalizer 2133, and a frequency converter 2134. The signal output processing unit 216 includes the functions of a buffer 2161 and a DA converter 2162, as shown in Figure 3(b).

[0026] The AD conversion unit 2131 function is implemented in the USB audio I / F 2 shown in Figure 1, and the bandstop filter 2132, equalizer 2133, and frequency conversion unit 2134 functions may be implemented in the CPU 11 of terminal 1, which will be described later using Figure 16. The buffering function 2161 function is implemented in the memory 12 or communication I / F (not shown) of terminal 1, which will be described later using Figure 16, and the DA conversion unit 2162 function may be implemented in the USB audio I / F 2 shown in Figure 1.

[0027] On the transmitting side, the analog signal is converted to a digital signal by the AD converter 2131, the frequency band used for vibration presentation is removed by the bandstop filter 2132, the signal in a specific band is emphasized by the equalizer 2133, and the frequency is converted by the frequency converter 2134 before the presentation vibration is transmitted to the receiving side. Note that the equalizer 2133 may be omitted.

[0028] On the receiving side, the presentation vibration received is buffered by the buffering unit 2161, and the digital signal of the presentation vibration is converted into an analog signal by the DA conversion unit 2162.

[0029] Figure 4(a) is a block diagram showing a second example of the signal input processing unit 213 shown in Figure 2 (see reference numeral 213a), and Figure 4(b) is a block diagram showing a second example of the signal output processing unit 216 shown in Figure 2 (see reference numeral 216a).

[0030] As shown in Figure 4(a), the signal input processing unit 213a includes the functions of an AD converter 2131, a bandstop filter 2132, and an equalizer 2133. Furthermore, as shown in Figure 4(b), the signal output processing unit 216a includes the functions of a buffer 2161, a DA converter 2162, and a frequency converter 2163.

[0031] The AD conversion unit 2131 function is implemented in the USB audio I / F 2 shown in Figure 1, and the bandstop filter 2132 and equalizer 2133 functions may be implemented in the CPU 11 of terminal 1, which will be described later using Figure 16. The frequency conversion unit 2163 function is implemented in the CPU 11 of terminal 1, which will be described later using Figure 16, and the buffering unit 2161 function is implemented in the memory 12 of terminal 1, which will be described later using Figure 16, and the DA conversion unit 2162 function may be implemented in the USB audio I / F 2 shown in Figure 1.

[0032] On the transmitting side, the analog signal is converted to a digital signal by the AD converter 2131, the frequency band used for vibration presentation is removed by the bandstop filter 2132, and the signal in a specific band is emphasized by the equalizer 2133 before the signal is transmitted to the receiving side. Note that the equalizer 2133 may be omitted.

[0033] On the receiving side, the frequency conversion unit 2163 performs frequency conversion on the received signal, the frequency-converted presentation vibration is buffered by the buffering unit 2161, and the digital signal of the presentation vibration is converted to an analog signal by the DA conversion unit 2162.

[0034] Figure 5(a) is a block diagram showing a third example of the signal input processing unit 213 shown in Figure 2 (see reference numeral 213b), and Figure 5(b) is a block diagram showing a third example of the signal output processing unit 216 shown in Figure 2 (see reference numeral 216b).

[0035] As shown in Figure 5(a), the signal input processing unit 213b includes the functions of an AD converter 2131, a bandstop filter 2132, an equalizer 2133, and an intensity calculation unit 2135. The signal output processing unit 216b, as shown in Figure 5(b), includes the functions of a buffer 2161, a DA converter 2162, and an oscillation waveform generation unit 2164.

[0036] The AD conversion unit 2131 function is implemented in the USB audio I / F 2 shown in Figure 1, and the bandstop filter 2132, equalizer 2133, and intensity calculation unit 2135 functions may be implemented in the CPU 11 of terminal 1, which will be described later using Figure 16. The vibration waveform generation unit 2164 function is implemented in the CPU 11 of terminal 1, which will be described later using Figure 16, and the buffering function 2161 function is implemented in the memory 12 of terminal 1, which will be described later using Figure 16, and the DA conversion unit 2162 function may be implemented in the USB audio I / F 2 shown in Figure 1.

[0037] On the transmitting side, the analog signal is converted to a digital signal by the AD conversion unit 2131, the frequency band used for vibration presentation is removed by the band stop filter 2132, the signal in a specific band is emphasized by the equalizer 2133, and the vibration intensity is calculated by the intensity calculation unit 2135 before the signal is transmitted to the receiving side. Since the vibration intensity only needs to be updated at a rate of about 100 Hz compared to the waveform signal, it has the effect of reducing the amount of data transmitted. Note that the equalizer 2133 may be omitted.

[0038] On the receiving side, vibrations are generated based on the vibration intensity received by the vibration waveform generation unit 2164, the frequency-converted presented vibrations are buffered by the buffering unit 2161, and the digital signal of the presented vibrations is converted to an analog signal by the DA conversion unit 2162.

[0039] Next, the frequency conversion process performed by the frequency conversion units 2134 and 2163 will be described in (1) to (3) below.

[0040] (1) Frequency conversion processing example I Figures 6(a) to 6(c) are graphs illustrating the frequency conversion process performed by the frequency conversion units 2134 and 2163 shown in Figures 3 and 4.

[0041] The frequency conversion units 2134 and 2163 maintain the envelope (in other words, perceptual information) and present it as an amplitude-modulated wave. Figure 6(a) shows the original signal v0(t), Figure 6(b) shows the upper and lower envelope signals eup(t) and elow(t) of the original signal as dotted lines, and Figure 6(c) shows the amplitude-modulated wave vam(t) generated so as to be contained within the envelope in Figure 6(b). The amplitude-modulated wave vam(t) is expressed by the following equation.

number

[0042] Here, A(t) is the amplitude of the amplitude-modulated wave, f is the carrier frequency of the presented amplitude-modulated wave, and voff(t) is the offset of the amplitude-modulated wave.

[0043] The carrier frequency f is selected to be different from the frequency components of the original signal. It is best to choose a frequency of around 150-400 Hz, which is easily perceptible to humans. This allows us to perceive fluctuations in the envelope of the original signal.

[0044] (2) Frequency conversion processing example II The frequency conversion units 2134 and 2163 may perform frequency conversion processing by mapping using the equivalent subjective intensity of the stimulus. Assume that the subjective intensity of an oscillation with frequency f and amplitude A can be obtained by the function S(A, f). When the representative frequency f0 of the original signal's oscillation can be identified, the amplitude A is determined such that the subjective intensity S(A0, f0) and the subjective intensity S(A, f) of the oscillation after the frequency change are equivalent. The equivalent subjective intensity function may be, for example, an equi-sensory curve of oscillation obtained by the magnitude balance method.

[0045] (3) Frequency conversion processing example III The frequency conversion units 2134 and 2163 may perform frequency conversion processing using only the perceptual intensity represented by the following equation 2. The perceptual intensity of the representative frequency f0 of the original signal is determined using the following equation, and the amplitude A is determined so that the perceptual intensity of the vibration after frequency conversion is equivalent.

number

[0046] Here, A is the amplitude, Tf is the amplitude threshold at frequency f, and bf is an exponential value that depends on frequency f.

[0047] ISM is a method that utilizes time-divided perceptual intensity, and may be an additional time-division process added to the processing in the frequency conversion units 2134 and 2163 described above.

[0048] Figure 7 is a graph illustrating the measurement results of the amplitude at the through output.

[0049] In the bidirectional tactile transmission system 100 shown in Figure 1, howling was verified under the following conditions. Sensor: Piezoelectric vibration sensor VS-BV203 Vibrator: Audio Exciter TEAX09C005-8 USB Audio: OCTA-CAPTURE Amplifier: SA-36A PRO Device: Matte polystyrene board measuring 100 mm (height) x 150 mm (width)

[0050] In the case of through output, howling was observed as shown in Figure 7. On the other hand, when ISM processing was performed, howling did not occur even when the sensor sensitivity was increased compared to the case of through output. In other words, it was confirmed that frequency modulation is effective against vibration howling.

[0051] Figure 8 is a table illustrating the results of suppressing howling when using ISM in the bidirectional tactile transmission system 100 shown in Figure 1.

[0052] We investigated whether feedback would occur by changing the modulation frequency (fm) after conversion using ISM. The natural frequency of the board is approximately 500Hz, and the fm range is 200-1000Hz.

[0053] In Figure 8, ○ indicates that the device does not oscillate even when vibration is applied, △ indicates that it oscillates with slight vibration, and × indicates that it self-excites even without vibration.

[0054] When the fm frequency was equal to the natural frequency of the board, it was confirmed that even slight vibrations caused howling without touching the board. Furthermore, it was confirmed that howling occurred when vibrations were applied to the board at twice the natural frequency. On the other hand, it was confirmed that howling did not occur at other frequencies even when vibrations were applied to the board. In other words, it was confirmed that howling can be prevented by converting the vibration to a frequency far from the natural frequency using ISM.

[0055] Figure 9(a) is a graph illustrating the measurement results of output vibration in the case of through output, and (b) is a graph illustrating the measurement results of output vibration when using ISM.

[0056] ISM can convert vibrations into any desired carrier frequency. By applying a filter to the sensor's measured values ​​to remove the ISM carrier frequency component, the vibrations caused by the oscillator are eliminated.

[0057] The frequency after conversion using ISM was set to 200 Hz, and the 200 Hz frequency was removed using the bandstop filter 2132, presenting an impulse waveform as the initial oscillation.

[0058] As shown in Figure 9(a), in the case of through output, the initial oscillation indicated by code G1 was dampened while the loopback indicated by codes G2 and G3 was repeated multiple times. On the other hand, as shown in Figure 9(b), in the case of ISM, no loopback occurred after the initial oscillation indicated by code G4. This confirmed that the combination of ISM processing and filtering processing is effective in preventing loopback.

[0059] From here, we will explain specific examples of vibration input / output devices 3, 3a to 3e using Figures 10 to 15.

[0060] Figure 10 shows a first example of an environmentally installed vibration input / output device 3.

[0061] In the vibration input / output device 3 shown in Figure 10, a vibration sensor 31 and a vibrator 32 (in other words, an oscillator) are arranged on the back surface of the contact plate 301.

[0062] When a user touches the contact plate 301, the vibration input / output device 3 detects and presents tactile sensations. Two rectangular fixing parts 302 are attached along two sides of the back of the contact plate 301, and the contact plate 301 may be fixed to a desk, a laptop-type terminal 1, a keyboard (not shown), etc. The contact plate 301 may be fixed at both ends and raised in the middle to create a structure that allows the front to vibrate easily. An image may be projected onto the contact plate 301 to display the image of the recipient.

[0063] Figure 11 shows a second example of an environmentally installed vibration input / output device 3a.

[0064] The vibration input / output device 3a shown in Figure 11, like the vibration input / output device 3 shown in Figure 10, has a contact plate 301 fixed by two fixing parts 302. The vibration input / output device 3a is equipped with one vibration sensor 31 and multiple (four in the illustrated example) vibrators 32 on the back of the contact plate 301. In addition, a contact position detection sensor 33 (in other words, a contact position detection sensor) may be separately provided, for example, by detecting the contact position using a depth camera that tracks the hand position with a stereo camera. That is, in the vibration input / output device 3 shown in Figure 1, a contact position detection sensor 33 may be added.

[0065] In this way, based on the detection results from the contact position detection sensor 33, the sensation of movement of vibration may be output to the vibrator 32. For example, based on the contact position information (e.g., coordinate information) on the contact plate 301 detected by the contact position detection sensor 33, vibration is distributed and output to multiple vibrators 32 of the receiving vibration input / output device 3a. By arranging multiple vibrators 32, the vibration intensity and stimulation time difference of each vibrator 32 can be adjusted according to the contact position of the communication partner, thereby virtually representing the sensation of movement of the contact object.

[0066] Figure 12 shows a third example of an environmentally installed vibration input / output device 3b.

[0067] The vibration input / output device 3c shown in Figure 12, like the vibration input / output devices 3 and 3a shown in Figures 10 and 11, has a contact plate 301 fixed by two fixing parts 302. The vibration input / output device 3c is equipped with multiple (four in the illustrated example) vibration sensors 31 and multiple (four in the illustrated example) vibrators 32 on the back of the contact plate 301.

[0068] By providing multiple vibration sensors 31, the contact position can be estimated based on the difference in input intensity, and the sensation of vibration movement can be output to the vibrator 32 based on the input from each of the multiple vibration sensors 31. Furthermore, by providing multiple vibrators 32, it is also possible to vibrate the entire surface of the contact plate. In addition, by adjusting the vibration intensity and time difference of each vibration according to the contact position of the communication partner, it is possible to virtually represent the sensation of movement of the contact object.

[0069] Figure 13 shows a wristband-type vibration input / output device 3c.

[0070] The vibration input / output device 3c shown in Figure 13 is equipped with multiple vibration sensors 31 and multiple vibrators 32 on the inner surface of a wristband worn by the user on their wrist. The vibration input / output device 3c may be various wearable devices that the user can wear. In Figure 13, two vibration sensors 31 and two vibrators 32 are explicitly shown, but multiple vibration sensors 31 and vibrators 32 may be provided around the user's wrist, for example, four each in the up, down, left, and right directions.

[0071] A vibration sensor 31 is attached near the wrist to measure vibrations transmitted from the user's fingertips upon contact with an object. By using multiple vibration sensors 31, differences in vibration depending on the part of the arm can be obtained. A vibrator 32 is also attached near the wrist to display vibrations transmitted from the communication partner. By using multiple vibrators 32, differences in vibration depending on the part of the body can be represented.

[0072] Figure 14 shows a 3D vibration input / output device in the shape of a body pillow.

[0073] The vibration input / output device 3d shown in Figure 14 has a vibration sensor 31 and a vibrator 32 positioned on a body pillow, cushion, or the like that can come into contact with or be grasped by the user's body. The number of vibration sensors 31 and vibrators 32 can be varied. The vibration input / output device 3d can come into contact with a large area of ​​the user's body, such as the abdomen or legs, and measure the vibration of the user's body using the vibration sensor 31, while transmitting vibrations to the user's body using the vibrator 32. Alternatively, the vibration sensor 31 and vibrator 32 may be positioned on a handheld controller, which can be grasped by the user.

[0074] Figure 15 shows a tool-mounted vibration input / output device 3e.

[0075] The vibration input / output device 3e shown in Figure 15 has a vibration sensor 31 and a vibrator 32 attached to a user-holdable tool such as a wrench or screwdriver. The number of vibration sensors 31 and vibrators 32 can be varied. The vibrations transmitted to the tool are measured by the vibration sensor 31, and when the vibrator 32 is attached to the tool, the hand is stimulated, and the stimulation is transmitted bidirectionally to a remote user, conveying the feeling of operation and other information, enabling remote technical guidance and other applications.

[0076] In addition, the vibration input / output devices 3, 3a to 3e shown in Figures 10 to 15 may output sound, video, or light simultaneously with the output of the converted signal from the vibrator 32.

[0077] [A-2] Terminal Figure 16 is a block diagram schematically showing an example configuration of terminal 1 as an embodiment.

[0078] Terminal 1 comprises a Central Processing Unit (CPU) 11, memory 12, and storage device 13.

[0079] Memory 12 is a storage device that includes Read Only Memory (ROM) and Random Access Memory (RAM).

[0080] The storage device 13 is a device that stores data in a read-write manner, and may be, for example, a Hard Disk Drive (HDD), Solid State Drive (SSD), or Storage Class Memory (SCM). The storage device 13 stores the generated training data, learning models, etc.

[0081] The CPU 11 is a processing unit that performs various control and calculations, and realizes various functions by executing the Operating System (OS) and programs stored in the memory 12. That is, as shown in Figure 16, the CPU 11 may function as a calculation unit 113A, a conversion unit 114A, and a signal output unit 115A.

[0082] The CPU 11 is an example of a computer and, exemplarily, controls the operation of the entire terminal 1. The device for controlling the operation of the entire terminal 1 is not limited to the CPU 11, and may be, for example, one of the following: MPU, DSP, ASIC, PLD, FPGA, or dedicated processor. Alternatively, the device for controlling the operation of the entire terminal 1 may be a combination of two or more of the following: CPU, MPU, DSP, ASIC, PLD, FPGA, and dedicated processor. Note that MPU is an abbreviation for Micro Processing Unit, DSP is an abbreviation for Digital Signal Processor, and ASIC is an abbreviation for Application Specific Integrated Circuit. Also, PLD is an abbreviation for Programmable Logic Device, and FPGA is an abbreviation for Field Programmable Gate Array.

[0083] The calculation unit 113A calculates perceptual information identified from the vibrations measured by the vibration sensor 31. The calculation unit 113A may calculate the envelope identified from the vibrations measured by the vibration sensor 31 as perceptual information. The calculation unit 113A may calculate the subjective intensity of the stimulus as perceptual information from the waveform of the vibrations measured by the vibration sensor 31. The calculation unit 113A may calculate the perceptual intensity (in other words, the energy of the signal) identified from the vibrations measured by the vibration sensor 31 as perceptual information. The calculation unit 113A may divide the signal related to the vibrations measured by the vibration sensor 31 into predetermined time intervals and calculate the perceptual intensity for each of the divided predetermined time intervals.

[0084] The conversion unit 114 converts the vibration signal to a predetermined frequency while maintaining the perceptual information calculated by the calculation unit 113A. The conversion unit 114A may convert the signal to a frequency other than the resonant frequency of the housing of the vibration input / output device 3. The conversion unit 114A may output the vibration signal as an amplitude-modulated signal converted so that the predetermined frequency becomes the carrier frequency, while maintaining the envelope calculated by the calculation unit 113A. The conversion unit 114A may convert the vibration signal to a predetermined frequency using a predetermined equivalent subjective intensity map so as to maintain the subjective intensity calculated by the calculation unit 113A. The conversion unit 114A may convert the vibration signal to a waveform with a different frequency while maintaining the perceptual intensity calculated by the calculation unit 113A. The conversion unit 114A may adjust the perceptual intensity of signals in a specific frequency band among the frequency components of the signal and convert the waveform, while converting signals outside the specific frequency band to a waveform with a different frequency while maintaining the perceptual intensity. The conversion unit 114A may adjust the perceptual intensity and convert the waveform of signals extracted based on specific signal features, while converting signals not extracted by specific features into waveforms with different frequencies while maintaining the perceptual intensity. The conversion unit 114A may attenuate a specific frequency band by filtering so that loopback of vibrations output from the vibrator 32 is suppressed. This filtering may be implemented by the bandstop filter 2132 shown in Figures 3 to 5.

[0085] The signal output unit 115A causes the converted signal, converted by the conversion unit 114A, to be output as an output vibration to the vibrator 32 of the other vibration input / output device 3. The signal output unit 115A may also output the amplitude-modulated signal, converted by the conversion unit 114A, to the vibrator 32 of the other vibration input / output device 3.

[0086] Furthermore, of the functions of the calculation unit 113A, conversion unit 114A, and signal output unit 115A, the transmitting terminal 1 may be equipped with any of the following combinations: calculation unit 113A only, calculation unit 113A and conversion unit 114A, or calculation unit 113A, conversion unit 114A, and signal output unit 115A. Also, of the functions of the calculation unit 113A, conversion unit 114A, and signal output unit 115A, the receiving terminal 1 may be equipped with any of the following combinations: signal output unit 115A only, conversion unit 114A and signal output unit 115A, or calculation unit 113A, conversion unit 114A, and signal output unit 115A.

[0087] [A-3] ISM Figures 17(a) to (c) are graphs that briefly explain the ISM processing used in the distribution of perceptual intensity shown in Figure 1.

[0088] ISM is a technique that modulates high-frequency vibrations to low frequencies while maintaining their tactile feel. The original signal shown in Figure 17(a) is transformed, and the vibration intensity for each segment shown in Figure 17(b) is calculated. Then, while maintaining the vibration intensity, the transformed waveform shown in Figure 17(c) is generated.

[0089] In Figure 17, the pre-conversion waveform is 400-600 Hz, while the post-conversion waveform is 200 Hz. However, any frequency can be selected for the post-conversion waveform.

[0090] In generating high-frequency component signals, vibration waveforms equivalent to the distributed perceptual intensity are generated. Simply put, since the waveforms of each vibrator 32 have the same frequency, they can be driven by multiplying the original waveform by a gain value obtained from the distribution coefficient (the same method as for low-frequency components, described later). However, generally, tactile vibrators 32 have a narrow response frequency band, making it difficult to directly generate arbitrary vibration waveforms. Furthermore, when using acoustic signals as the vibration source waveform, there is a problem of noise generation when driven by vibrators 32 because they include frequencies in the audible range.

[0091] Therefore, the signal is converted into an amplitude-modulated wave with an appropriate carrier frequency to generate the distributed perceptual intensity Ik. This results in a single carrier frequency for the generated signal. The carrier frequency can be selected to match the frequency response characteristics of the oscillator. Considering the human perception characteristics of high-frequency vibrations, a carrier frequency in the range of 150 to 400 Hz is appropriate.

[0092] Considering the human perceptual characteristics of high-frequency vibrations, in the high-frequency band, instead of focusing on the waveform itself, we focus on the vibrational energy that correlates with human perceptual characteristics. By replacing the waveform with another waveform that has equivalent vibrational energy, the frequency band can be changed.

[0093] By dividing a continuous, arbitrary vibration signal into appropriate time segments that take into account human perceptual characteristics, and converting each segment into vibrational energy, it becomes possible to convert the signal into any waveform while maintaining the same level of tactile sensation as humans, or even enabling the perception of high-frequency bands that are difficult to perceive.

[0094] By appropriately selecting the frequency of the converted vibration, it becomes possible to efficiently drive the oscillator according to its response range, reduce auditory noise, and convert it to any desired sound source.

[0095] It is said that human perception of vibrations extends to about 1 kHz. Therefore, vibrations above 1 kHz are often ignored. On the other hand, even for vibrations above 1 kHz, if the amplitude of the amplitude fluctuates within a bandwidth that humans can perceive, it is known that the envelope component can be perceived.

[0096] On the other hand, the vibration energy model is known to describe the perceptual characteristics of human vibrations for high-frequency vibrations of approximately 100 Hz or higher. From this, it has been found that even if the carrier frequency of the amplitude-modulated wave is replaced while maintaining the high-frequency vibration energy, the vibrations cannot be distinguished. However, even if the vibration energy is maintained, as mentioned above, the envelope component of the vibration may be perceived as a difference in tactile information, and the range of this perception has not been investigated. Furthermore, although methods have been devised to convert signals based on vibration energy using time division, methods for maintaining low-frequency components have not been considered.

[0097] Figure 18 is a graph showing the discriminability of vibrations by humans. Figure 19 shows the sample vibration waveforms used in a forced three-choice discrimination experiment conducted to determine the discriminability shown in the graph in Figure 18.

[0098] Assuming a conventionally known vibrational energy model, investigating human perceptual discrimination characteristics while maintaining vibrational energy yields the graph shown in Figure 18. In Figure 19, symbols B1 and B2 show the same waveform, while symbol B3 shows a different waveform. Participants were asked to compare the constant-amplitude vibrations shown in symbols B1 and B2 in Figure 19 with the amplitude-modulated stimulus shown in symbol B3, and to identify which was the amplitude-modulated wave. In Figure 18, the correct answer rate obtained in the forced three-choice discrimination experiment is expressed as Sensitivity (d': d-prime), a discrimination performance index based on signal detection theory, where d' being 1 or less means that the correct answer rate is below approximately 60%.

[0099] According to the graph in Figure 18, the upper limit of the frequency range at which the envelope component can be discriminated is approximately 80-125 Hz. Furthermore, it is not necessary to maintain the envelope component above this frequency limit, and it is shown that if the carrier frequency of the amplitude-modulated wave is replaced while maintaining the vibrational energy, the stimulus cannot be distinguished.

[0100] As mentioned above, even if vibrational energy is maintained, if the energy fluctuates in the low-frequency range, this fluctuation may be perceived as a difference in tactile information, and the range of this perception had not been investigated. Therefore, based on the discovery that the upper limit of perceptible low-frequency fluctuations is around 80-125 Hz, we will perform vibrational energy conversion while maintaining the low-frequency components using two measures (see measures [1] and [2] described later).

[0101] Figure 20 is a graph showing the waveforms of the signals before and after conversion for each segment by terminal 1, as shown in Figure 16.

[0102] Human high-frequency perception is based more on vibrational energy than on the waveform itself; therefore, maintaining the vibrational energy will result in the same sensation. However, if the fluctuations in vibrational energy occur at frequencies below approximately 80-125 Hz, it is necessary to reproduce those fluctuations in vibrational energy.

[0103] Therefore, in one example of this embodiment, as a means of maintaining fluctuations in vibration energy below a predetermined frequency (for example, around 80 to 125 Hz), the vibration is time-divided in an interval of approximately 80 to 200 Hz, the vibration energy is determined for each segment, and it is replaced with vibrations having different carrier frequencies.

[0104] In the example shown in Figure 20, the original vibration signal, indicated by symbol C1, and the converted signal, indicated by symbol C2, are transformed so that within the same time segment, the energy of the converted signal becomes the same as the energy of the original vibration signal.

[0105] The time division width (in other words, the division width) should be set to a degree that can represent energy fluctuations of 80-125Hz or less (in other words, to a degree that the peaks of the fluctuations match) (Countermeasure [1]). The frequency of the division width may be greater than 80-125Hz, but if the division width is made too short, the accuracy of estimating the vibration energy of vibrations with periods longer than the division width will deteriorate. Therefore, as per the countermeasure [2] below, vibrations whose energy cannot be estimated are output as waveforms as they are.

[0106] Alternatively, components below a predetermined frequency may be extracted and presented directly as the stimulus vibration (Countermeasure [2]). The predetermined frequency may be 80 to 125 Hz or higher, but components above the predetermined frequency may be represented by the energy control unit 113 of the second signal component. This allows for arbitrary frequency selection. However, setting the predetermined frequency too high may cause noise problems or necessitate the use of a wideband vibration device.

[0107] According to the above measures [1] and [2], the specified frequency may be around 80 to 400 Hz. 400 Hz is the upper limit from the standpoint of noise problems and the performance of the vibration device.

[0108] Setting a predetermined frequency involves selecting the carrier frequency used when converting vibrations. Since the peak vibration frequency at which human perception sensitivity is high is around 200-250 Hz, a practical carrier frequency range is around 150-400 Hz, which enhances sensitivity without creating noise. The carrier frequency may be a constant multiple of the division width. Furthermore, multiple different frequencies may be used as carrier frequencies, and high-frequency ranges above 400 Hz may also be included.

[0109] Furthermore, the predetermined frequency used to separate low and high frequencies does not necessarily have to be the same as the frequency of the division width used to calculate energy.

[0110] Corrected energy, which is vibrational energy corrected to enhance human perceptibility, can be expressed by the following equation.

[0111]

number

[0112] Figure 21 is a graph showing the amplitude threshold Tf used in the calculation of the correction energy.

[0113] As shown in Figure 21, the amplitude threshold varies with frequency. Humans can perceive relatively small amplitudes in the range of approximately 102-103 Hz, but in other ranges, humans cannot perceive anything other than relatively large amplitudes.

[0114] Figure 22 is a graph showing the exponential value bf used in the calculation of the corrected energy.

[0115] The exponential value bf in Figure 22 is an example where the value obtained by linearly interpolating the exponential value bf below 400 Hz, which has been reported in the past, is used.

[0116] Figure 23 illustrates the use of the window function at terminal 1, as shown in Figure 16.

[0117] As shown in symbol D1, a high-frequency signal H(t) is input. As shown in symbol D2, the high-frequency signal H(t) is divided into frames i, i+1, i+2, ... and each frame is divided into signals hi, hi+1, hi+2, ... As shown in symbol D3, the signal h of each divided frame is separated into multiple basis signals g1, g2, g3, ... As shown in symbol D4, based on the frequencies f1, f2, f3, ... of the basis signals g1, g2, g3, ... scalar values ​​Ei, Ei+1, Ei+2, ... are output by combining the correction energies of all basis signals g1, g2, g3, ... As shown in symbol D5, the scalar values ​​Ei, Ei+1, Ei+2, ... of the vibration energy calculated in each frame i are converted into vibration waveforms with equivalent vibration energy but different carrier frequencies, and a windowing process using a window function is performed on the amplitudes ai(t), ai+1(t), ai+2(t), ... of these waveforms. As shown in symbol D6, frame synthesis is performed for the 1st to Nth frames, and the amplitude A(t) of the vibration waveform is output. As shown in symbol D7, a second vibration waveform S2(t) with a carrier frequency such that its amplitude is A(t) is output.

[0118] Figure 24 is a graph illustrating an example of the combination of low-frequency and high-frequency signals at terminal 1 shown in Figure 16.

[0119] The second vibration waveform S2(t), shown as E1, generated from the high-frequency signal H(t) using the window function in Figure 23, is combined with the first vibration waveform S1(t), shown as E2, which is the output of the low-frequency signal L(t) as is. As a result, the combined waveform S1(t)+S2(t), shown as E3, is output.

[0120] Figure 25 is a graph showing specific examples of the signal waveforms before and after conversion by terminal 1, as shown in Figure 16.

[0121] In Figure 25, the waveform of the violin sound before conversion (see symbol F1) and after conversion (see symbol F2) are represented by their amplitude over time.

[0122] High-frequency vibrations, such as those of a violin, generate significant auditory noise with conventional tactile vibrations, and applying a low-pass filter eliminates vibrations that humans can perceive. Therefore, correction energy is calculated so that the waveform becomes a single wavelength with a low-frequency carrier frequency over time.

[0123] Figure 26 is a block diagram illustrating an example of the functional configuration of the ISM unit 1000 in terminal 1 shown in Figure 16.

[0124] The ISM unit 1000 functions as a time-division control unit 112, an energy control unit 113, an energy-vibration conversion unit 114a, and a vibration generation unit 114b. In this embodiment, the ISM unit 1000 controls vibrations containing high-frequency components of approximately 100 Hz or higher generated by the vibrator 32 using signals. The method for controlling vibrations containing high-frequency components of 100 Hz or higher according to the present invention is collectively referred to as ISM.

[0125] The time-division control unit 112 divides the vibration signal X(t), which contains high-frequency components of approximately 100 Hz or higher, into N frames, and inputs the signal hi of the i-th frame to the energy control unit 113. The number of frames N may be determined by a predetermined period and the overlap rate of the windowing process.

[0126] The energy control unit 113 calculates a corrected energy ei for the signal hi of the i-th frame and inputs the calculated corrected energy to the energy vibration conversion unit 114a.

[0127] The energy vibration conversion unit 114a generates a signal A(t) by combining the correction energies e1 to eN of the 1st to Nth frames and inputs it to the second vibration generation unit 114b.

[0128] The vibration generation unit 114b outputs a signal waveform S(t) based on the synthesized signal A(t).

[0129] A first embodiment of the vibration waveform generation process at terminal 1 shown in Figure 16 will be described according to the block diagram (steps S1 to S7) shown in Figure 27.

[0130] The signal removal unit 111a removes components below a predetermined frequency from the acquired unconverted signal X(t) to generate a high-frequency signal H(t), which is then input to the time-division control unit 112 (step S1).

[0131] The time-division control unit 112 divides the high-frequency signal H(t) into N frames and inputs the signal hi of the i-th frame to the energy control unit 113 (step S2). The number of frames N may be determined by a predetermined period and the overlap rate of the windowing process.

[0132] The energy control unit 113 calculates a corrected energy ei for the signal hi of the i-th frame and inputs the calculated corrected energy to the energy oscillation conversion unit 114a (step S3).

[0133] The energy vibration conversion unit 114a generates a signal A(t) by combining the correction energies e1 to eN of the 1st to Nth frames and inputs it to the second vibration generation unit 114b (step S4).

[0134] The second vibration generation unit 114b outputs a second vibration waveform S2(t) based on the synthesized signal A(t) (step S5).

[0135] Meanwhile, the low-pass filter 111b inputs the low-pass signal L(t), obtained by filtering out components below a predetermined frequency from the acquired signal X(t) before conversion, to the first vibration generation unit 114c (step S6).

[0136] The first vibration generation unit 114c outputs a first vibration waveform S1(t) based on the low-frequency signal L(t) (step S7).

[0137] Next, the details of the energy control process shown in step S3 of Figure 27 will be explained according to the block diagram (steps S11 to S14) shown in Figure 28.

[0138] As shown in Figure 28, the energy control unit 113 functions as a base signal separation control unit 113a, a frequency calculation unit 113b, an energy correction parameter calculation unit 113c, and a corrected energy calculation unit 113d.

[0139] The base signal separation control unit 113a separates the input signal, which is the time-divided i-th frame signal hi, into multiple base signals g, and inputs the separated k-th base signal gk to the frequency calculation unit 113b (step S11). For example, the signals may be separated by methods such as short-time Fourier analysis, wavelet analysis, or Empirical Mode Decomposition (EMD).

[0140] The frequency calculation unit 113b calculates the frequency fk of the k-th basis signal gk, for example, by discrete Fourier analysis or Hilbert Spectrum analysis, and inputs it to the energy correction parameter calculation unit 113c (step S12).

[0141] The energy correction parameter calculation unit 113c calculates the exponential value bk and amplitude threshold Tk, as explained using Figures 21 and 22, based on the frequency fk, and inputs them to the corrected energy calculation unit 113d (step S13).

[0142] The correction energy calculation unit 113d calculates the correction energy Ipc for each basis signal gk according to the formula shown in Equation 3, based on the exponential value bk and the amplitude threshold Tk, and outputs a scalar value ei obtained by summing the correction energies of all basis signals gk (step S14).

[0143] Next, as a second embodiment of the vibration waveform generation process at terminal 1 shown in Figure 16, the low-frequency component separation process in the energy control process shown in Figure 26 will be explained according to the block diagram (steps S101 to S105) shown in Figure 29.

[0144] As shown in Figure 29, the energy control unit 113 may function as a base signal separation control unit 113a, a frequency calculation unit 113b, an energy correction parameter calculation unit 113c, and a corrected energy calculation unit 113d, and may also have a function to separate low-frequency components to a low-frequency component synthesis unit 113g.

[0145] The base signal separation control unit 113a separates the input signal, which is the time-divided i-th frame signal hi, into multiple base signals g, and inputs the separated k-th base signal gk to the frequency calculation unit 113b (step S101). For example, the signals may be separated by methods such as short-time Fourier analysis, wavelet analysis, or EMD.

[0146] The frequency calculation unit 113b calculates the frequency fk of the k-th basis signal gk by, for example, discrete Fourier analysis or Hilbert Spectrum analysis, and inputs it to the energy correction parameter calculation unit 113c (step S102).

[0147] The energy correction parameter calculation unit 113c calculates the exponential value bk and amplitude threshold Tk, as explained using Figures 21 and 22, based on the frequency fk, and inputs them to the corrected energy calculation unit 113d (step S103).

[0148] The correction energy calculation unit 113d calculates the correction energy Ipc for each basis signal gk according to the formula shown in Equation 3, based on the exponential value bk and the amplitude threshold Tk, and outputs a scalar value ei obtained by summing the correction energies of all basis signals gk (step S104).

[0149] The low-frequency component synthesis unit 113g synthesizes a base signal gk whose frequency fk is smaller than a predetermined frequency to generate a low-frequency component L(t) (step S105).

[0150] In some cases, when dealing with a sound source containing signals across multiple frequency bands, it is desirable to emphasize the vibrational energy of a specific frequency band and present it as vibration. In such cases, the energy control units 1131 and 1132, which are modified versions applied when adjusting the energy of a fundamental signal present in a predetermined frequency band to transform the waveform, will be explained using Figures 30 to 35.

[0151] Figures 30(a) to (c) are graphs illustrating an example of generating vibrations according to ISM without exaggerating the waveform. In Figure 30, the frequency bands corresponding to the high-frequency components of the cymbal (drum) waveform and the frequency bands corresponding to the piano and bass waveforms are shown from a piano trio piece. In Figures 30(a) to (c), the horizontal axis represents time [s] and the vertical axis represents frequency [Hz]. A darker spectrum indicates high power, and a lighter spectrum indicates low power.

[0152] Figure 30(a) shows the distribution of the sound source spectrum, with the high-frequency component cymbal waveform shown by the dashed line and the low-frequency component piano and bass waveforms shown by the dashed line.

[0153] Figure 30(b) shows the spectral distribution (centered at 200 Hz) after conversion with ISM. In Figure 30(b), the ISM effect extracts the intensity of all frequencies, including cymbals, piano, and bass.

[0154] Figure 30(c) shows an example where the signal is converted using the representative frequency of the base signal, rather than converting it to a signal with a frequency of 200 Hz based on intensity. This visualizes which frequency bands are emphasized.

[0155] Figures 31(a) to (c) are graphs illustrating the first example of emphasizing and separating high-frequency components from a sound source. Figure 31 shows an example of emphasizing and separating the high-frequency components of cymbals (drums) from a piano trio piece. In Figures 31(a) to (c), the horizontal axis represents time [s], and the vertical axis represents frequency [Hz]. A darker spectrum indicates high power, and a lighter spectrum indicates low power.

[0156] Figure 31(a) shows the distribution of the sound source spectrum, with the high-frequency component cymbal waveform shown by the dashed line and the low-frequency component piano and bass waveforms shown by the dashed line.

[0157] Figure 31(b) shows the spectral distribution (centered at 200 Hz) after conversion using ISM. In Figure 31(b), only the intensity above 3000 Hz is boosted by +20 dB (100 times).

[0158] Figure 31(c) shows an example where the signal is converted using the representative frequency of the base signal, rather than converting it to a signal with a frequency of 200 Hz based on intensity. This visualizes which frequency bands are emphasized. In Figure 31(c), the power of the cymbal spectrum is increased.

[0159] Figures 32(a) to (c) are graphs illustrating a second example of emphasizing and separating high-frequency components from a sound source. Figure 32 shows an example of emphasizing and separating the high-frequency components of cymbals (drums) from a piano trio piece. In Figures 32(a) to (c), the horizontal axis represents time [s], and the vertical axis represents frequency [Hz]. A darker spectrum indicates high power, and a lighter spectrum indicates low power.

[0160] Figure 32(a) shows the distribution of the sound source spectrum, with the high-frequency component cymbal waveform shown by the dashed line and the low-frequency component piano and bass waveforms shown by the dashed line.

[0161] Figure 32(b) shows the spectral distribution (centered at 200 Hz) after conversion using ISM. In Figure 32(b), the intensity above 3000 Hz is increased by +20 dB (100 times), while the intensity below 1000 Hz is reduced by -10 dB (1 / 10 times).

[0162] Figure 32(c) shows an example where the signal is converted using the representative frequency of the base signal, rather than converting it to a signal with a frequency of 200 Hz based on intensity. This visualizes which frequency bands are emphasized. In Figure 32(c), the power of the cymbal spectrum is increased.

[0163] Figures 33(a) to (c) are graphs illustrating an example of emphasizing and separating low-frequency components from a sound source. Figure 33 shows an example of emphasizing and separating the low-frequency components of piano and bass from a piano trio piece. In Figures 33(a) to (c), the horizontal axis represents time [s], and the vertical axis represents frequency [Hz]. A darker spectrum indicates high power, and a lighter spectrum indicates low power.

[0164] Figure 33(a) shows the distribution of the sound source spectrum, with the high-frequency component cymbal waveform shown by the dashed line and the low-frequency component piano and bass waveforms shown by the dashed line.

[0165] Figure 33(b) shows the spectral distribution (centered at 200 Hz) after conversion using ISM. In Figure 33(b), the intensity below 1000 Hz is increased by +10 dB (10 times).

[0166] Figure 33(c) shows an example where the signal is converted using the representative frequency of the base signal, rather than converting it to a signal with a frequency of 200 Hz based on intensity. This visualizes which frequency bands are emphasized. In Figure 33(c), the spectral power of the piano and bass is increased.

[0167] The process of emphasizing and separating arbitrary frequency components shown in Figures 30 to 33 is also applicable to the bidirectional haptic transmission system 100 shown in Figure 1. That is, for signals in a specific frequency band among the frequency components of the signal, the energy is adjusted and the waveform is transformed, while for signals outside the specific frequency band, the energy is maintained and the waveform is transformed to have a different frequency. Furthermore, for signals extracted based on specific signal features, the energy is adjusted and the waveform is transformed, while for signals not extracted by specific features, the energy is maintained and the waveform is transformed to have a different frequency.

[0168] A first modified example of the energy control process shown in Figure 26 will be explained according to the block diagram (steps S41 to S45) shown in Figure 34.

[0169] As shown in Figure 34, the energy control unit 1131 functions as a gain calculation unit 113e, in addition to the base signal separation control unit 113a, frequency calculation unit 113b, energy correction parameter calculation unit 113c, and correction energy calculation unit 113d shown in Figure 28.

[0170] The base signal separation control unit 113a separates the input signal, which is the time-divided i-th frame signal hi, into multiple base signals g, and inputs the separated k-th base signal gk to the frequency calculation unit 113b (step S41). For example, the signals may be separated by methods such as short-time Fourier analysis, wavelet analysis, or EMD.

[0171] The frequency calculation unit 113b calculates the frequency fk of the k-th basis signal gk by, for example, discrete Fourier analysis or Hilbert Spectrum analysis, and inputs it to the energy correction parameter calculation unit 113c (step S42).

[0172] The energy correction parameter calculation unit 113c calculates the exponential value bk and amplitude threshold Tk, as explained using Figures 21 and 22, based on the frequency fk, and inputs them to the corrected energy calculation unit 113d (step S43).

[0173] The gain calculation unit 113e outputs predetermined gain values ​​Gk for each frequency band according to the frequency fk of the calculated basis signal gk (step S44). Gk is set to > 1 when energy is to be emphasized, and to 0 ≤ Gk < 1 when energy is to be suppressed. Energy adjustment by emphasis or suppression may be performed for one frequency band or for multiple frequency bands. Furthermore, energy adjustment may be performed for the entire frequency band input to the energy control unit 1131.

[0174] The correction energy calculation unit 113d calculates a gain-adjusted correction energy Ipc for each base signal gk according to the formula shown in Equation 2 below, based on the amplitude A of the separated base signals gk, and outputs a scalar value ei obtained by summing the correction energies of all base signals gk (step S45).

[0175]

number

[0176] As shown in Figure 35, the energy control unit 1132 functions as a gain calculation unit 113e and a signal source identification unit 113f, in addition to the base signal separation control unit 113a, frequency calculation unit 113b, energy correction parameter calculation unit 113c, and correction energy calculation unit 113d shown in Figure 28.

[0177] The base signal separation control unit 113a separates the input signal, which is the time-divided i-th frame signal hi, into multiple base signals g, and inputs the separated k-th base signal gk to the frequency calculation unit 113b (step S51). For example, the signals may be separated by methods such as short-time Fourier analysis, wavelet analysis, or EMD.

[0178] The frequency calculation unit 113b calculates the frequency fk of the k-th basis signal gk by, for example, discrete Fourier analysis or Hilbert Spectrum analysis, and inputs it to the energy correction parameter calculation unit 113c (step S52).

[0179] The energy correction parameter calculation unit 113c calculates the exponential value bk and amplitude threshold Tk, as explained using Figures 21 and 22, based on the frequency fk, and inputs them to the corrected energy calculation unit 113d (step S53).

[0180] The signal source identification unit 113f estimates identification candidates from the input signal hi and the history of hi based on the set signal characteristics, identifies which signal source the basis signal gk belongs to, and outputs the identification result as an ID (identifier) ​​(step S54). The signal source identification unit 113f may have a classifier prepared in advance by machine learning or the like. For example, the characteristics of many instruments may be learned by deep learning, and a group of candidate instruments (e.g., piano, bass, drums) may be estimated to determine which instrument is included in the current input signal hi (or the history of each of the multiple input signals hi if the input signal hi is too short), and it may be identified which instrument the basis signal gk belongs to.

[0181] The gain calculation unit 113e outputs a predetermined gain value Gk for each frequency band according to the ID identified by the signal source identification unit 113f (step S55). Gk is set to > 1 when energy is to be emphasized, and to 0 ≤ Gk < 1 when energy is to be suppressed. Energy adjustment by emphasis or suppression may be performed for one frequency band or for multiple frequency bands. Energy adjustment may also be performed for the entire frequency band input to the energy control unit 1132.

[0182] The correction energy calculation unit 113d calculates a gain-adjusted correction energy Ipc for each base signal gk according to the formula shown in Equation 4, based on the amplitude A of the separated base signals gk, and outputs a scalar value ei obtained by summing the correction energies of all base signals gk (step S56).

[0183] Next, the details of the energy synthesis process shown in step S4 of Figure 26 will be explained according to the block diagram (steps S21 to S23) shown in Figure 36.

[0184] The energy vibration conversion unit 114a functions as an energy equivalent conversion unit 1141a, a windowing processing unit 1142a, and a frame synthesis unit 1143a.

[0185] As shown in Figure 36, the energy equivalent conversion unit 1141a converts the scalar value ei of the vibration energy calculated in each frame i into a vibration waveform having equivalent vibration energy but a different carrier frequency, and outputs the amplitude ai(t) of that waveform to the windowing processing unit 1142a (step S21).

[0186] The windowing processing unit 1142a performs windowing on the amplitude ai(t) of each input frame i using the window function shown in Figure 23, and inputs the processing result to the frame merging unit 1143a (step S22).

[0187] The frame synthesis unit 1143a performs frame synthesis on the input from the windowing processing unit 1142a for the 1st to Nth frames and outputs the amplitude A(t) of the vibration waveform (step S23).

[0188] Next, the details of the process for generating the corrected vibration waveform shown in step S5 of Figure 26 will be explained according to the block diagram (steps S31 and S32) shown in Figure 37.

[0189] As shown in Figure 37, the second vibration generation unit 114b functions as an amplitude vibration conversion unit 1141b and a waveform output unit 1142b. The second vibration generation unit 114b takes the input signal A(t) and outputs a sine wave with a carrier frequency. The phase of the generated waveform may be controlled so that the vibrations are smoothly connected.

[0190] The amplitude-vibration conversion unit 1141b converts the input amplitude A(t) into vibration (step S31).

[0191] The waveform output unit 1142b outputs a sine wave S2(t) with a carrier frequency such that its amplitude is A(t) (step S32).

[0192] [B] effect According to the bidirectional tactile communication system 100, tactile communication program, and tactile communication method in one embodiment, the following effects can be achieved, for example.

[0193] The vibration sensor 31 measures the vibration generated in the vibration input / output device 3. The calculation unit 113A calculates perceptual information identified from the vibration measured by the vibration sensor 31. The conversion unit 114A converts the vibration signal to a predetermined frequency while maintaining the perceptual information calculated by the calculation unit 113A. The signal output unit 115A outputs the converted signal converted by the conversion unit 114A as output vibration to the vibrator 32 of the other vibration input / output device 3.

[0194] This makes it possible to suppress howling and loopback of the contact signal in the bidirectional tactile transmission system 100. Specifically, by modulating the actuator signal to a signal with a frequency different from the natural vibration of the housing while maintaining perceptual information, howling caused by bidirectional communication can be suppressed. In addition, by keeping the actuator drive signal constant at a carrier frequency different from the natural vibration of the housing and removing a signal with this carrier frequency from the sensor signal, interference with the sensor signal can be prevented, and loopback can be suppressed.

[0195] [C] Others The disclosed technology is not limited to the embodiments described above, and can be implemented in various ways without departing from the spirit of each embodiment. Each configuration and each process of each embodiment can be selected or combined as needed.

[0196] Figure 38 is a block diagram showing an example of the DAC configuration when multiple vibration devices 310, 320 are used in the bidirectional tactile transmission system 100 shown in Figure 1.

[0197] In the example shown in Figure 38, the DA conversion unit 2162 shown in Figures 3 to 5 functions as a high-frequency gain adjuster 21a, a low-frequency gain adjuster 21b, a high-frequency vibration device drive circuit 22a, and a low-frequency vibration device drive circuit 22b. In addition, the vibrator 32 shown in Figure 1 functions as a high-frequency vibration device 310 and a low-frequency vibration device 320.

[0198] The high-frequency gain adjuster 21a outputs the second vibration waveform S2(t) input from terminal 1 to the high-frequency vibration device 310 via the high-frequency vibration device drive circuit 22a. The low-frequency gain adjuster 21b outputs the first vibration waveform S1(t) input from terminal 1 to the low-frequency vibration device 320 via the low-frequency vibration device drive circuit 22b.

[0199] Figure 39 is a block diagram showing an example of the DAC configuration when a single vibration device is used in the bidirectional tactile transmission system 100 shown in Figure 1.

[0200] In the example shown in Figure 39, the DA conversion unit 2162 shown in Figures 3 to 5 functions as a high-frequency gain adjuster 21a, a low-frequency gain adjuster 21b, and a vibration device drive circuit 22. The vibrator 32 shown in Figure 1 functions as a vibration device 30.

[0201] The high-frequency gain adjuster 21a and the low-frequency gain adjuster 21b output the second vibration waveform S2(t) and the first vibration waveform S1(t) input from terminal 1 to a common vibration device 30 via a common vibration device drive circuit 22, respectively. [Explanation of symbols]

[0202] 100: Bidirectional haptic transmission system 1: Terminal 11: CPU 1000:ISM Department 111: Frequency rejection control unit 111a: Signal removal section 111b:Low pass filter 111d: Correction energy calculation unit 112: Time Division Control Unit 113A: Calculation section 113,1131,1132: Energy Control Unit 113a: Base signal separation control unit 113b: Frequency calculation unit 113c: Energy correction parameter calculation unit 113d: Correction energy calculation unit 113e: Gain calculation section 113f: Signal source identification section 113g: Low-frequency component synthesis section 114A: Conversion section 114a: Energy vibration conversion unit 114b: Second vibration generation section 114c: 1st vibration generation section 1141a: Energy equivalent conversion unit 1142a: Window hanging processing unit 1143a: Frame composition section 1141b: Amplitude vibration conversion unit 1142b: Waveform output section 115A: Signal output section 12: Memory 13:Storage device 2: USB Audio Interface 21a: High-frequency gain adjuster 21b: Low-frequency gain adjuster 211: Vibration measurement unit 212,217: Signal amplification section 213: Signal input processing unit 2131: AD conversion unit 2132: Bandstop filter 2133: Equalizer 2134, 2163: Frequency conversion section 2135: Intensity Calculation Unit 214: Signal transmission unit 215: Signal receiving unit 216: Signal output section 2161: Buffering 2162: DA conversion unit 2164: Vibration waveform generation section 22: Vibration device drive circuit 22a: High-frequency vibration device drive circuit 22b: Low-frequency vibration device drive circuit 3,3a~3e: Vibration input / output devices 301: Contact plate 302:Fixed part 31: Vibration sensor 310: High-frequency vibration device 32: Vibrator 320: Low-frequency vibration device 33: Sensor for detecting contact position 4: Amplifier

Claims

1. A bidirectional tactile transmission system having a tactile transmission device and another tactile transmission device, The tactile transmission device includes a measuring unit that measures the vibrations generated and the location where the vibrations originate, A calculation unit that calculates perceptual information identified from the vibration measured by the measurement unit, A conversion unit that converts the vibration signal to a predetermined frequency while maintaining the perceptual information calculated by the calculation unit, A signal output unit distributes the converted signal, which has been converted by the conversion unit, to a plurality of vibrators of the other tactile transmission device based on the generation position measured by the measurement unit, and outputs it as output vibration. Equipped with, The conversion unit attenuates a specific frequency band by filtering so that loopback of vibrations output from the plurality of oscillators is suppressed. A bidirectional haptic transmission system.

2. A bidirectional tactile transmission system having a tactile transmission device and another tactile transmission device, The tactile transmission device includes a measuring unit that measures the vibrations generated and the location where the vibrations originate, A calculation unit that calculates perceptual information identified from the vibration measured by the measurement unit, A conversion unit that converts the vibration signal to a predetermined frequency while maintaining the perceptual information calculated by the calculation unit, A signal output unit distributes the converted signal, which has been converted by the conversion unit, to a plurality of vibrators of the other tactile transmission device based on the generation position measured by the measurement unit, and outputs it as output vibration. Equipped with, The conversion unit converts the signal to a frequency other than the resonant frequency of the housing of the tactile transmission device, which is the predetermined frequency. A bidirectional haptic transmission system.

3. The measurement unit includes a contact position detection sensor that detects the vibration generation location as the contact position of the user with the housing of the tactile transmission device. The signal output unit distributes and outputs the vibration movement sensation to the plurality of vibrators based on the detection result by the contact position detection sensor. The bidirectional tactile transmission system according to claim 1 or 2.

4. The measurement unit is equipped with multiple vibration sensors, The signal output unit detects the generation position as the user's contact position with the housing of the tactile transmission device based on the input intensity difference to the plurality of vibration sensors, and distributes the vibration movement sensation to the plurality of vibrators for output. The bidirectional tactile transmission system according to claim 1 or 2.

5. In a contact plate that is installed on the environmental side and transmits vibrations when the user's body comes into contact with it, the measurement unit and the vibrator are arranged. A bidirectional tactile transmission system according to any one of claims 1 to 4.

6. The measurement unit and the vibrator are arranged in a wearable device that can be attached to the user's body. The bidirectional tactile transmission system according to claim 1 or 2.

7. The measuring unit and the vibrator are arranged in a device that the user holds. The bidirectional tactile transmission system according to claim 1 or 2.

8. The output processing of the output vibration in the other tactile transmission device based on the measurement processing of the vibration and the generation location in the aforementioned tactile transmission device is also applied to the output processing of the output vibration in the other tactile transmission device based on the measurement processing of the vibration and the generation location in the aforementioned tactile transmission device. A bidirectional tactile transmission system according to any one of claims 1 to 7.

9. In a computer in a bidirectional haptic transmission system having a haptic transmission device and another haptic transmission device, The vibrations generated in the tactile transmission device and the location where the vibrations originate are measured. The perceptual information identified from the measured vibrations is calculated, While maintaining the calculated perceptual information, the vibration signal is converted to a predetermined frequency. The converted signal is distributed to multiple vibrators of the other tactile transmission device based on the measured generation location and output as output vibration. The signal is converted to a frequency other than the resonant frequency of the housing of the tactile transmission device, as the predetermined frequency. A bidirectional haptic feedback program that initiates processing.

10. In the process of measuring the generation location, the generation location is detected as the contact position of the user with the housing of the haptic transmission device. In the process of outputting the output vibration, based on the detection result, the vibration movement sensation is distributed to the plurality of vibrators and output. The bidirectional haptic transmission program according to claim 9, which causes the computer to perform the processing.

11. The process of measuring the vibration and the location of its occurrence is performed by multiple vibration sensors. In the process of outputting the output vibration, the generation position is detected as the user's contact position with the housing of the tactile transmission device based on the input intensity difference to the plurality of vibration sensors, and the sensation of vibration movement is distributed to the plurality of vibrators and output. The bidirectional haptic transmission program according to claim 9, which causes the computer to perform the processing.

12. A bidirectional tactile transmission method in a bidirectional tactile transmission system having a tactile transmission device and another tactile transmission device, The vibrations generated in the tactile transmission device and the location where the vibrations originate are measured. The perceptual information identified from the measured vibrations is calculated, While maintaining the calculated perceptual information, the vibration signal is converted to a predetermined frequency. The converted signal is distributed to multiple vibrators of the other tactile transmission device based on the measured generation location and output as output vibration. In the aforementioned plurality of oscillators, a specific frequency band is attenuated by filtering so that the loopback of vibrations output is suppressed. A bidirectional haptic communication method for executing a process.

13. A bidirectional tactile transmission method in a bidirectional tactile transmission system having a tactile transmission device and another tactile transmission device, The vibrations generated in the tactile transmission device and the location where the vibrations originate are measured. The perceptual information identified from the measured vibrations is calculated, While maintaining the calculated perceptual information, the vibration signal is converted to a predetermined frequency. The converted signal is distributed to multiple vibrators of the other tactile transmission device based on the measured generation location and output as output vibration. The signal is converted to a frequency other than the resonant frequency of the housing of the tactile transmission device, as the predetermined frequency. A bidirectional haptic communication method for executing a process.

14. In the process of measuring the generation location, the generation location is detected as the contact position of the user with the housing of the haptic transmission device. In the process of outputting the output vibration, based on the detection result, the vibration movement sensation is distributed to the plurality of vibrators and output. The method for bidirectional tactile transmission according to claim 12 or 13.

15. The process of measuring the vibration and the location of its occurrence is performed by multiple vibration sensors. In the process of outputting the output vibration, the generation position is detected as the user's contact position with the housing of the tactile transmission device based on the input intensity difference to the plurality of vibration sensors, and the sensation of vibration movement is distributed to the plurality of vibrators and output. The method for bidirectional tactile transmission according to claim 12 or 13.

Images