Signal processing device, signal processing method, and program

Abstract

To provide a signal interpolation method for making a user less noticeable of losing a tactile sense signal for enhancing immersion of the user to a tactile content.SOLUTION: A signal processing device according to the present technology is provided with an interpolation signal generation unit that generates, as an interpolation signal for interpolating a tactile sense signal lost during transmission, an interpolation signal on the basis of the tactile sense signal in a section adjacent to a lost section and a physical model related to tactile sense or a perception model of human beings.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present technology relates to a signal processing device, a method, and a program for processing haptic signals, and in particular to a technology for interpolating haptic signals that are lost during transmission. [Background technology]

[0002] In recent years, technology has been developed that provides tactile stimulation to users by vibrating a device worn by the user. Here, tactile stimulation refers to a physical phenomenon that causes the user to feel a sense of touch through a vibration phenomenon or the like. Hereinafter, the generation of tactile stimulation will be referred to as "tactile presentation."

[0003] Haptic presentation technologies are used in devices in various fields. For example, in terminal devices equipped with a touch panel, such as smartphones, the touch panel vibrates in response to a user's touch operation, providing a tactile stimulus to the user's finger, thereby enabling the user to experience the sensation of touching a button or other item displayed on the touch panel. Furthermore, in music listening devices, such as headphones, providing a tactile stimulus in sync with the music playback can enhance the bass of the music being played. Furthermore, in devices that provide computer games or virtual reality (VR), for example, providing a tactile stimulus by vibrating the controller in response to the controller's operation or the content scene can enhance the user's sense of immersion in the content.

[0004] Furthermore, technologies have been developed that provide tactile stimulation to a user based on tactile signals received from an external device. For example, Patent Document 1 below discloses a technology that provides tactile stimulation to a user while changing the frequency and amplitude of vibration based on a received signal.

[0005] However, because haptics generally requires that the device used to present the haptic sensation be in contact with the user's body, wired connections between various devices tend to hinder the user experience, and multi-channel connections make handling even more cumbersome. Therefore, it is desirable to use wireless connections between the device that acquires or generates the haptic signal and the device that ultimately presents it to the user.

[0006] On the other hand, when transmitting signals via wireless connection, a good communication environment is not always available, and loss of transmitted signals such as packet loss cannot be avoided. Many countermeasures have been considered for the transmission of audio signals. For example, one countermeasure is to provide a large send / receive buffer to allow for retransmission in the event of packet loss. This creates a margin of time until the signal corresponding to the lost signal is regenerated, and by successfully retransmitting before that time, it becomes possible to recover from the packet loss.

[0007] Packet loss concealment is also known, which conceals the loss of the original signal so that it is not noticeable to the user. Typical methods include filling the lost section with noise or a sine wave, or copying or extending the nearest frame to replace the lost signal.

[0008] However, it is not appropriate to apply the above-described measures for audio signals directly to haptic signals. The method of using large send and receive buffers aims to extend the time from reception to playback, making it difficult to achieve low latency, which is important for the quality of the haptic experience. Furthermore, packet loss concealment for audio signals has been developed as a method that can conceal packet loss well for the sense of hearing, taking into account the characteristics of human hearing, so there is no guarantee that packet loss can also be concealed well for the sense of touch.

[0009] As related prior art, the following Patent Documents 2 and 3 can be cited. Patent Document 2 proposes adjusting the signal length by inserting or removing a 0 section in the signal, and by using this, it is possible to implement a conventional packet loss concealment method for audio. Furthermore, Patent Document 3 proposes that if a signal transmitted to a vibration output device is lost, the signal is retransmitted to ensure the original signal is reproduced. However, this poses a problem of delay in the haptic experience. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-202486 [Patent Document 2] Japanese Patent Publication No. 2020-85033 [Patent Document 3] Japanese Patent Publication No. 2020-116256 Summary of the Invention [Problem to be solved by the invention]

[0011] This technology was developed in consideration of the above circumstances, and aims to provide a signal interpolation method that makes it difficult for users to notice the loss of haptic signals, thereby improving the user's sense of immersion in haptic content. [Means for solving the problem]

[0012] A first signal processing device according to the present technology includes an interpolation signal generation unit that generates an interpolation signal that interpolates a haptic signal that has been lost during transmission, the interpolation signal being based on a haptic signal in a section adjacent to the lost section and a physical model or a human perception model related to haptics. As a result, an interpolated signal based on haptic characteristics is generated as an interpolated signal for a missing section of the haptic signal.

[0013] Furthermore, a second signal processing device according to the present technology includes an interpolation method selection unit that selects an interpolation method for interpolating a haptic signal that is lost during transmission based on the results of signal analysis of the haptic signal that is based on a physical model of haptics or a human perception model, and an interpolation method setting unit that performs processing to generate transmission data for the haptic signal in which additional information indicating the interpolation method selected by the interpolation method selection unit is added to the haptic signal. According to the above configuration, the interpolation process for interpolating haptic signals lost during transmission can be performed using an appropriate method selected based on a physical model related to haptics and a human perception model. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of the configuration of a tactile presentation system according to a first embodiment of the present technology. [Figure 2] 2 is a block diagram illustrating an example of the internal configuration of a transmission device included in the tactile presentation system according to the first embodiment. FIG. [Figure 3] 4 is a diagram showing an example of the data structure of transmission data of a haptic signal in the first embodiment. FIG. [Figure 4] 2 is a block diagram showing an example of the internal configuration of a receiving device and a tactile presentation device included in the tactile presentation system according to the first embodiment. FIG. [Figure 5] FIG. 2 is an explanatory diagram of an interpolation signal generation method according to a first embodiment. [Figure 6] 10 is a flowchart illustrating an example of a processing procedure for realizing an interpolation signal generation method according to a first embodiment. [Figure 7] FIG. 10 is a block diagram illustrating an example of the internal configuration of a receiving device according to a second embodiment. [Figure 8] FIG. 10 is an explanatory diagram of an interpolation signal generation method according to a second embodiment. [Figure 9] 10 is a flowchart illustrating an example of a processing procedure for realizing an interpolation signal generating method according to a second embodiment. [Figure 10]FIG. 10 is a block diagram showing an example of the configuration of a tactile presentation system according to a third embodiment. [Figure 11] FIG. 10 is a block diagram showing an example of the internal configuration of a tactile presentation device and a presentation control device according to a third embodiment. [Figure 12] FIG. 10 is an explanatory diagram of an interpolation signal generation method according to a third embodiment. [Figure 13] 10 is a flowchart illustrating an example of a processing procedure for realizing an interpolation signal generating method according to a third embodiment. [Figure 14] FIG. 10 is a block diagram illustrating an example of the internal configuration of a transmission device according to a fourth embodiment. [Figure 15] FIG. 13 is a diagram illustrating an example of a data structure of packet data in the fourth embodiment. [Figure 16] FIG. 10 is a block diagram illustrating an example of the internal configuration of a receiving device according to a fourth embodiment. [Figure 17] 10 is a flowchart illustrating an example of a processing procedure for realizing the functions of a transmitting device according to a fourth embodiment. [Figure 18] 10 is a flowchart illustrating an example of a processing procedure for realizing the functions of a receiving device according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present technology will be described in the following order with reference to the accompanying drawings. <1. First embodiment> [1-1. Overview of the tactile presentation system] [1-2.Transmitting device] [1-3. Receiving device] [1-4. Interpolation signal generation method as the first embodiment] [1-5. Processing Procedure] 2. Second Embodiment 3. Third Embodiment 4. Fourth Embodiment <5. Variations> <6. Summary of embodiments> <7. This Technology>

[0016] In this specification, the terms are defined as follows. Tactile stimulation: A physical phenomenon that allows a person to perceive touch, such as a vibration phenomenon. Tactile presentation: Producing tactile stimuli. Tactile information: Information perceived through the sense of touch, such as vibration information. Tactile signal: a signal representing a pattern of tactile stimulation, for example a signal representing a vibration waveform. Haptic recipient: The person receiving the haptic presentation. Encoded data: Data obtained by encoding a signal. Sub-concepts include data obtained by compressing and encoding signals or by non-compressing (lossless) encoding signals.

[0017] <1. First embodiment> [1-1. Overview of the tactile presentation system] FIG. 1 is a block diagram showing an example of the configuration of a tactile presentation system 100 according to a first embodiment of the present technology. The tactile presentation system 100 of this embodiment is configured to sense the target tactile information (tactile stimulation) to obtain a tactile signal, transmit this tactile signal to a destination device via a predetermined communication path, and present the tactile sensation to a user as a tactile receiver via a tactile presentation device installed at the destination.

[0018] As shown in the figure, the tactile presentation system 100 includes a transmitting device 1, a receiving device 2, a tactile sensor 3, and a tactile presentation device 4. The tactile sensor 3 is a sensor that senses tactile information, and in this example, an acceleration sensor or a vibration sensor such as a piezo pickup is used. The tactile sensor 3 outputs vibrations and movements as voltage changes when it is brought into contact with the sensing target, that is, the human body in this example. In this example, the tactile sensor 3 is connected to the transmitting device 1 by wire, and is attached to a predetermined part of the human body as the target to sense tactile information corresponding to a tactile stimulus occurring at that part.

[0019] The transmitting device 1 is configured with a computer device such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), and encodes the detection signal (tactile signal) from the tactile sensor 3 in accordance with a predetermined data format, and transmits the encoded data to the receiving device 2 via a network NT, which is a communications network such as the Internet or a LAN (Local Area Network). As will be described later, the encoded data for the tactile signal is packetized and sent to the network NT.

[0020] The receiving device 2 is configured with a computer device such as a CPU or a DSP, decodes the coded data received from the transmitting device 1 via the network NT, and outputs the haptic signal obtained by the decoding to the haptic presentation device 4.

[0021] The tactile presentation device 4 includes a tactile presentation unit 5, and presents a tactile stimulus to a user as a tactile receiver based on the tactile signal input from the receiving device 2. The tactile presentation unit 5 is a device that generates a tactile stimulus, and in this example, a vibration device such as a vibrator or actuator is used. Other examples of the tactile sensation presentation unit 5 include a device that outputs focused ultrasonic waves or an air cannon using a speaker array, a pressure presentation device that controls the amount of fluid inside the device, and an electrical stimulation device that directly stimulates tactile receptors with electricity. In any of these configurations, the human body can sense vibrations.

[0022] Note that, although an example is given here in which a tactile signal is transmitted from the transmitting device 1 to the receiving device 2 via the network NT, it is also conceivable that the transmission of the tactile signal from the transmitting device 1 to the receiving device 2 may be performed using short-range wireless communication such as Bluetooth (registered trademark).

[0023] Although only tactile signals are mentioned here, it is also possible to record audio signals and video signals synchronized with the tactile signals, thereby providing the tactile receiver with sound and video along with tactile information.

[0024] [1-2.Transmitting device] FIG. 2 is a block diagram for explaining an example of the internal configuration of the transmission device 1, and shows the tactile sensor 3 shown in FIG. 1 together with the example of the internal configuration of the transmission device 1. The transmitting device 1 includes a sensor I / F (interface) 11, an encoding unit 12, a memory unit 13, a communication unit 14, and a control unit 15. As shown in the figure, these units are connected to a bus 16, and data communication can be performed via this bus 16.

[0025] The sensor I / F 11 is a communication interface unit for the tactile sensor 3, and receives the tactile signal obtained by the tactile sensor 3 as input.

[0026] The encoding unit 12 is configured, for example, by a DSP, and encodes the haptic signal input from the haptic sensor 3 via the sensor I / F 11. The encoding method can be any of a variety of methods commonly used for encoding audio signals, which are one-dimensional signals like haptic signals. For example, MP3 (MPEG-1 Audio Layer-III) or AAC (Advanced Audio Coding) can be used, or a lossless encoding method such as FLAC (Free Lossless Audio Codec) can be used. Furthermore, considering computational resources, it is also possible to adopt ADPCM (Adaptive Differential Pulse Code Modulation) or the like. In the encoding unit 12, the haptic signal is divided into frames of a fixed length, and encoding is performed for each frame. Hereinafter, a frame of a haptic signal will be referred to as "frame Fr."

[0027] The memory unit 13 is a comprehensive representation of storage devices such as a hard disk drive (HDD) or a solid state drive (SSD), and is used to store various types of data in the transmission device 1. For example, the memory unit 13 stores data necessary for control by the control unit 15. Furthermore, under the control of the control unit 15, the coded data obtained by the coding unit 12 can also be stored in the memory unit 13.

[0028] The communication unit 14 performs data communication with an external device. Specifically, the communication unit 14 in this example has a network communication function and is capable of performing data communication with the receiving device 2 via the network NT shown in FIG.

[0029] The control unit 15 is configured with a microcomputer having, for example, a CPU, a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and performs overall control of the transmitting device 1 by the CPU executing processing in accordance with a program stored in the ROM. For example, the control unit 15 performs data communication with an external device, particularly the receiving device 2, via the communication unit 14. Specifically, the control unit 15 performs processing to transmit coded data of the haptic signal obtained by the coding unit 12 to the receiving device 2. In this example, the control unit 15 packetizes the coded data of the haptic signal and transmits it to the receiving device 2.

[0030] FIG. 3 is a diagram showing an example of the data structure of packet data in this example. In this example, the control unit 15 generates packet data (transmission data) as shown in FIG. 3 based on the encoded data of the haptic signal. In this example, the haptic signal is packetized for each frame Fr, that is, one packet data contains one frame Fr of the haptic signal.

[0031] As shown in the figure, in this case, the packet data includes information such as a sync pattern, a packet ID, a sampling frequency, the number of quantization bits, and the number of signal samples added to the actual data of the haptic signal. The sync pattern is an identifier that indicates the beginning of packet data, and stores data with a predetermined bit pattern. For example, 16 bits are assigned to the sync pattern, and it is considered that the sync pattern is a predetermined bit pattern such as 0xFFFE.

[0032] The packet ID indicates the ID of the packet data. The control unit 15 assigns, for example, ascending serial numbers as packet IDs. Specifically, the control unit 15 assigns ascending serial numbers in the time series order of the haptic signals.

[0033] The sampling frequency and the number of quantization bits respectively indicate the sampling frequency and the number of quantization bits when digitally sampling the haptic signal. The number of signal samples indicates the number of samples of the haptic signal as actual data stored in the packet data, specifically the number of samples in units of frames Fr in this example.

[0034] In FIG. 2, the control unit 15 generates the packet data as described above based on the encoded data of the haptic signal, and performs processing so that the packet data is transmitted to the receiving device 2 via the communication unit 14 .

[0035] [1-3. Receiving device] FIG. 4 is a block diagram showing an example of the internal configuration of the receiving device 2 and the tactile presentation device 4. As shown in FIG. As shown in the figure, the receiving device 2 comprises a first communication unit 21, a memory unit 22, a decoding unit 23, a second communication unit 24, and a control unit 25, and each of these units is connected to a bus 26, enabling data communication via the bus 26. The tactile sense presentation device 4 also includes a communication unit 41 and a drive unit 42 in addition to the tactile sense presentation unit 5 described above.

[0036] In the receiving device 2, the first communication unit 21 performs data communication with an external device, and specifically in this example, has a network communication function and is capable of performing data communication with the transmitting device 1 (communication unit 14) via the network NT shown in Figure 1.

[0037] The memory unit 22 is a comprehensive representation of storage devices such as HDDs and SSDs, and is used to store various types of data in the receiving device 2. For example, the memory unit 22 stores data necessary for control by the control unit 25. Furthermore, based on the control of the control unit 25, the memory unit 22 can also store packet data received from the transmitting device 1 via the first communication unit 21 and decoded data (decoded data of encoded data) obtained by the decoding unit 23.

[0038] The decoding unit 23 decodes the coded data (coded data of the haptic signal) included in the packet data received from the transmitting device 1 via the first communication unit 21, and obtains the haptic signal.

[0039] The second communication unit 24 performs communication processing to output the decoded haptic signal to the haptic presentation device 4. Note that the connection between the receiving device 2 and the haptic presentation device 4 for this communication may be a wired connection or a wireless connection.

[0040] In the tactile presentation device 4, the tactile signal output from the receiving device 2 via the second communication unit 24 is input via the communication unit 41, and the drive unit 42 drives the tactile presentation unit 5 based on the input tactile signal. As a result, a tactile sensation can be presented to a user as a tactile receiver wearing the tactile presentation device 4 according to the tactile information sensed by the tactile sensor 3 shown in FIG.

[0041] In the receiving device 2, the control unit 25 is configured with a microcomputer having, for example, a CPU, a ROM, a RAM, etc., and performs overall control of the receiving device 2 by the CPU performing processing in accordance with a program stored in the ROM. For example, the control unit 25 causes the decoding unit 23 to perform a decoding process on the encoded data included in the packet data received from the transmitting device 1 via the first communication unit 21. Furthermore, the control unit 25 performs a process of outputting the haptic signal decoded by the decoding unit 23 to the haptic presentation device 4 via the second communication unit 24 as a haptic signal output process.

[0042] Furthermore, the control unit 25 has a function as an interpolation signal generation unit F1 as a function according to the embodiment. The interpolation signal generation unit F1 generates an interpolation signal that interpolates for a haptic signal lost during transmission, based on the haptic signal in a section adjacent to the lost section and a physical model or human perception model related to haptics.

[0043] [1-4. Interpolation signal generation method as the first embodiment] The function of the interpolation signal generating unit F1 will be described below. In the first embodiment, an interpolated signal is generated based on a physical model relating to the tactile sense, specifically, an interpolated signal is generated based on a damped sine wave model.

[0044] Damped sine waves are frequently observed in haptic signals, for example as vibration waveforms caused by collisions or sliding between objects. Therefore, by generating an interpolated signal based on a damped sine wave model, the loss of a haptic signal can be made less noticeable to the user, thereby improving the user's sense of immersion in the haptic content.

[0045] FIG. 5 is an explanatory diagram of an interpolation signal generation method according to the first embodiment. First, as shown in the upper part of the figure, on the transmitting device 1 side, the haptic signal is packetized in units of frame Fr, and the packet data is transmitted to the receiving device 2 side. In the figure, three consecutive frames Fr in the haptic signal are designated as frames Fr1, Fr2, and Fr3, and the packet data obtained by packetizing these frames Fr1, Fr2, and Fr3 are designated as packets P1, P2, and P3. As described above, packet data is assigned a packet ID for identifying each packet, and in the figure, the packet IDs assigned to packets P1, P2, and P3 are shown as PID1, PID2, and PID3.

[0046] Here, let us assume that packet P2 among the above-mentioned packets P1, P2, and P3 is lost during transmission to the receiving device 2. On the receiving device 2 side, the control unit 25 performs a packet data loss determination process based on the packet ID information of the received packet data. Specifically, it determines whether or not the packet ID is missing. In the case of FIG. 5, it is determined that PID2 is missing, and therefore the determination result is that packet P2 has been lost.

[0047] In the first embodiment, in response to the detection of loss of a haptic signal in this way, an interpolation signal for interpolating (concealing) the lost section is generated based on the haptic signal in the section adjacent to the lost section (lost packet) and a damped sine wave model shown in the following [Equation 1].

number

[0048] Specifically, in generating an interpolated signal in this case, first, based on the haptic signals of the packet immediately preceding the lost packet (packet P1 in the illustrated example) and the packet immediately following the lost packet (packet P3), a known waveform analysis method is used to calculate the initial amplitude A, waveform envelope (attenuation coefficient) λ for each time t, and frequency f of the haptic signal for three frames including the frame Fr (frame Fr2) of the lost packet and its preceding and succeeding frames (frames Fr1 and Fr3). For the initial amplitude A, the initial amplitude of the frame Fr (frame Fr1) immediately preceding the lost frame Fr is used. Here, the waveform envelope λ for each time t can be calculated using a known envelope calculation method, such as the Hilbert transform. Furthermore, the frequency f for each time t can be calculated using a known envelope calculation method, such as using the peak position of an autocorrelation function.

[0049] After determining the initial amplitude A, the waveform envelope (attenuation coefficient) λ for each time t, and the frequency f for the haptic signal for three frames as described above, these values ​​are used to generate a damped sine wave according to Equation 1. Of the signals generated by the damped sine wave in this way, the signal in the section corresponding to the missing frame Fr (frame Fr2) is obtained as an interpolated signal.

[0050] When it is determined that a packet has been lost, the control unit 25 performs processing so that the interpolated signal generated as described above is output to the tactile presentation device 4 as the haptic signal for the lost section (the signal section of frame Fr2). This reduces the delay in the haptic experience compared to when lost packets are resent, while making the loss of the haptic signal less noticeable to the user, thereby improving the user's sense of immersion in the haptic content.

[0051] [1-5. Processing Procedure] FIG. 6 is a flowchart showing an example of a processing procedure for realizing the interpolation signal generating method according to the first embodiment described above. In this example, the process shown in Fig. 6 is executed by the control unit 25 shown in Fig. 4. Specifically, the CPU in the control unit 25 executes the process based on a program stored in a predetermined storage device such as a ROM of the control unit 25.

[0052] 6, first, in step S101, the control unit 25 performs a process of acquiring decoded data of a received packet. That is, the control unit 25 causes the decoding unit 23 to execute a decoding process on the coded data included in the packet data received from the transmitting device 1 via the first communication unit 21, and acquires a haptic signal (a haptic signal for one frame in this example) obtained by the decoding process.

[0053] In step S102 following step S101, the control unit 25 performs processing to buffer the acquired decoded data. This buffering is performed so that three consecutive frames Fr required for generating an interpolated signal are held. Specifically, if the most recently received frame Fr is frame Fr3, three frames Fr1, Fr2, and Fr3 are held.

[0054] In step S103 following step S102, the control unit 25 determines whether or not a packet loss has been detected. As described above, the determination of packet loss is made by determining whether or not the packet ID attached to the packet data is missing. In this case, whether or not a packet has been lost cannot be determined unless the frame Fr next to the frame Fr that is the target of the loss determination is received.

[0055] If it is determined in step S103 that a packet loss has been detected, the control unit 25 proceeds to step S104 to execute an interpolation signal generation process. Note that the interpolation signal generation method according to the first embodiment, i.e., the method for generating an interpolation signal based on a damped sine wave model, has already been described, and therefore a duplicate description will be avoided. Then, in response to having executed the interpolation signal generation process in step S104, the control unit 25 advances the process to step S105.

[0056] On the other hand, if it is determined in step S103 that no packet loss has been detected, the control unit 25 skips the interpolation signal generation process in step S104 and proceeds to step S105.

[0057] In step S105, the control unit 25 performs a signal output process. In this signal output process, a haptic signal corresponding to the section of the frame Fr immediately before the most recently received frame Fr is output to the haptic presentation device 4. Specifically, if it is determined in step S103 that no packet loss has been detected, the haptic signal of the frame Fr immediately before the most recently received frame Fr (i.e., the haptic signal received without loss) is output, and if it is determined in step S103 that packet loss has been detected, the interpolated signal generated for the frame Fr immediately before the most recently received frame Fr is output.

[0058] In step S106 following step S105, the control unit 25 determines whether the process has ended, that is, whether a predetermined process end condition (for example, an instruction to stop haptic presentation has been given to the user, the haptic presentation device 4 has been disconnected, the power supply of the receiving device 2 has been turned off, etc.) has been met. Determine whether or not.

[0059] If it is determined in step S106 that the process is not finished, the control unit 25 returns to step S101, whereby the next packet is targeted for packet loss determination and interpolation signal generation when packet loss is detected.

[0060] On the other hand, if it is determined in step S106 that the process has ended, the control unit 25 ends the series of processes shown in FIG.

[0061] 2. Second Embodiment FIG. 7 is a block diagram showing an example of the internal configuration of a receiving device 2A according to the second embodiment. In the second embodiment, the configuration of the tactile presentation system 100 is the same as that of the first embodiment except that a receiving device 2A is provided instead of the receiving device 2, and therefore a description with illustrations will be omitted. In the following description, the same parts as those already described will be assigned the same reference numerals and step numbers, and the description thereof will be omitted.

[0062] 7, a receiving device 2A differs from the receiving device 2 of the first embodiment in that a control unit 25A is provided instead of the control unit 25. The control section 25A differs from the control section 25 in that it has an interpolation signal generation section F1A instead of the interpolation signal generation section F1.

[0063] The interpolation signal generation unit F1A generates an interpolation signal for a lost section based on a human perception model related to the tactile sense, rather than a physical model related to the tactile sense. Specifically, the interpolation signal generation unit F1A in this example generates an interpolation signal based on such a perceptual model, that is, based on the tactile subjective intensity and envelope shape of the haptic signal in the adjacent section.

[0064] Regarding the ability of the human tactile sense to distinguish between two vibration stimuli, it is said that for vibrations with frequencies above 200 Hz, the main factors are the tactile subjective intensity, which is determined by the amplitude and frequency, and the envelope shape of the vibration waveform. In other words, by making the tactile subjective intensity and envelope shape the same, it is possible to generate two signals that are difficult for the human tactile sense to distinguish. The second embodiment proposes a method for generating an interpolated signal that utilizes such a human perception mechanism.

[0065] An interpolation signal generation method according to the second embodiment will be described with reference to FIG. First, as a premise, for the sake of explanation, the second embodiment will be described as an example in which the half-overlap method is used as the encoding method for haptic signals. In the half-overlap method, frames Fr are extracted while overlapping half of the signal in the transmitting device 1, and encoding is performed for each frame Fr (see the left side of the figure). In this case, packetization is also performed in units of frames Fr. Here, three consecutive packets in chronological order are also shown as packets P1, P2, and P3. In this case, the receiving device 2A outputs the decoded haptic signals for each packet while overlapping half of the signals.

[0066] Let us now assume that the middle packet (P2) of three consecutive packets (P1 to P3) is lost. As a reminder, the half-overlap method is based on the premise that signals are combined half-and-half with adjacent packets. Therefore, when one packet is lost, it becomes unclear which signal should be combined with the signals of the preceding and following packets, making it impossible to provide an appropriate tactile presentation.

[0067] When the interpolation signal generation unit F1A detects a packet loss, it obtains the haptic subjective intensity and envelope shape of each of the haptic signals of the adjacent packets before and after the lost packet. Regarding the tactile subjective intensity, the sensory gain is a measure of the amplitude D of a certain vibration, which indicates how many times the minimum detection threshold T at the frequency of the vibration is.

number

number

[0068] The envelope shape can be determined by known methods such as applying a low-pass filter to the squared value of the signal, calculating the analytic signal using a Hilbert transform and finding its absolute value, or finding the average amplitude of the signal for each fixed interval.

[0069] After determining the haptic subjective intensity and envelope shape of the haptic signals of the adjacent packets before and after the lost packet, the haptic subjective intensity and envelope shape of the interpolated signal are determined based on the haptic subjective intensity and envelope shape. For example, the haptic subjective intensity value itself is used, and the envelope shape is a value quantified by the number of peaks. In this case, the haptic subjective intensity and envelope shape of the interpolated signal are determined by linearly interpolating the haptic subjective intensity values ​​and the number of peaks of the envelopes of the adjacent packets before and after the lost packet. In the illustrated example, the haptic subjective intensity of the adjacent packet before the lost packet is 50, the number of peaks is 3.5, and the haptic subjective intensity of the adjacent packet after the lost packet is 30, the number of peaks is 2.5. By linearly interpolating these values, the haptic subjective intensity of the interpolated signal is determined to be 40, and the number of peaks is 3.0.

[0070] The interpolation signal generation unit F1A generates a signal having the haptic subjective intensity and the number of peaks of the envelope calculated by such linear interpolation as an interpolation signal for the lost packet. In this case, the frequency of the interpolated signal needs to be 200 Hz or higher, and can be determined based on a value that is easily perceived by humans or the resonance characteristics of the tactile presentation device. For example, an amplitude-modulated wave with a frequency of 200 Hz or higher, a subjective intensity of 40, and three peaks in the envelope is generated. This interpolated signal does not need to have the same frequency as the lost original waveform, according to the human perception mechanism. Therefore, other signal components, such as noise, may be added to the interpolated signal in addition to the amplitude-modulated wave.

[0071] By using the interpolated signal generation method described above, it is possible to interpolate the tactile signal in the missing section based on the tactile subjective intensity and envelope shape, which are the main factors that determine how humans discriminate between tactile stimuli. Therefore, the loss of the haptic signal can be made less noticeable to the user, and the user's sense of immersion in the haptic content can be improved.

[0072] FIG. 9 is a flowchart showing an example of a processing procedure for realizing the interpolation signal generating method according to the second embodiment described above. In this example, the processing shown in Figure 9 is executed by the control unit 25A shown in Figure 7, and specifically, the CPU in the control unit 25A executes it based on a program stored in a predetermined storage device such as a ROM of the control unit 25A.

[0073] In this case, the processing from steps S101 to S103 is executed in the same manner as in the case of FIG. If the control unit 25A determines in step S103 that a packet loss has been detected, the control unit 25A proceeds to step S104A to perform an interpolation signal generation process. Specifically, the interpolation signal generation process in step S104A performs a process of generating an interpolation signal based on the haptic subjective intensities and envelope shapes of adjacent packets before and after the lost packet, as described above.

[0074] Then, in step S105A following step S104A, the control unit 25A performs signal output processing. In this example, the signal output processing in step S105A is signal output processing corresponding to the half overlap method. Specifically, if it is determined in step S103 that packet loss has been detected, processing is performed to cause the tactile presentation device 4 to output, in order, a synthesized signal of the haptic signal in the latter half of packet P1 and the signal in the former half of the interpolated signal, and a synthesized signal of the signal in the latter half of the interpolated signal and the haptic signal in the former half of packet P3, for the haptic signals of the most recently received packet (assumed to be packet P3) and the packet two packets before that (assumed to be packet P1), and the interpolated signal generated in step S104A. Furthermore, if it is determined in step S103 that no packet loss has been detected, a process is performed in which the tactile presentation device 4 outputs, in order, a synthesized signal of the haptic signal for the latter half of packet P1 and the haptic signal for the former half of packet P2 (the packet received immediately before packet P3), and a synthesized signal of the haptic signal for the latter half of packet P2 and the haptic signal for the former half of packet P3.

[0075] The control unit 25A advances the process to step S108 in response to the execution of the signal output process in step S105A, but the process from step S108 onwards is the same as in the case of FIG. 6, so a duplicated description will be avoided.

[0076] 3. Third Embodiment The third embodiment is another example of generating an interpolated signal based on a physical model related to haptics, in which an interpolated signal is generated based on the interlocking characteristics of movement and vibration.

[0077] FIG. 10 is a block diagram showing an example of the configuration of a tactile presentation system 100B according to the third embodiment. As shown in the figure, the tactile presentation system 100B includes a tactile presentation device 4B worn by a user U as a tactile receiver, and a presentation control device 6. In this example, the tactile presentation device 4B is shown as being held in each hand of the user U, but since the presentation control device 6 performs the same processing for each tactile presentation device 4B, the following will explain the processing for only one of the tactile presentation devices 4B as a representative.

[0078] The tactile presentation system 100B is configured as a system that presents tactile sensations in a VR (Virtual Reality) game. A user U as a game player holds a controller-type tactile presentation device 4B that includes a motion sensor (a motion sensor 43 described below) and a tactile presentation unit 5. A motion signal obtained by the motion sensor is input from the tactile presentation device 4B to a presentation control device 6. The presentation control device 6 is configured as a computer device such as a personal computer, and generates a tactile signal linked to the input motion signal, packetizes the tactile signal, and outputs it to the tactile presentation device 4B. Here, the presentation control device 6 identifies the contact pattern of the virtual object that the virtual user U's hand has come into contact with in the VR space from the input movement signal, generates an appropriate tactile signal according to the identified contact pattern, and outputs it to the tactile presentation device 4B. This allows the user U to experience the sensation of touching a virtual object in the VR space.

[0079] FIG. 11 is a block diagram showing an example of the internal configuration of the tactile presentation device 4B and the presentation control device 6. As shown in FIG. As shown in the figure, the tactile presentation device 4B includes a communication unit 41, a drive unit 42, and a tactile presentation unit 5, as well as a motion sensor 43, a decoding unit 23, and a control unit 25B.

[0080] The motion sensor 43 is a sensor that detects a motion signal indicating the motion of the user U, and is configured by, for example, an acceleration sensor, an angular velocity sensor, etc. The motion signal detected by the motion sensor 43 is output to the presentation control device 6 via the communication unit 41.

[0081] As described above, the decoding unit 23 decodes the coded data of the haptic signal to obtain the haptic signal. The control unit 25B is configured with a microcomputer having, for example, a CPU, a ROM, a RAM, etc., and the CPU performs processing according to a program stored in the ROM, thereby performing overall control of the tactile presentation device 4B. For example, the control unit 25B outputs the haptic signal decoded by the decoding unit 23 to the driving unit 42, and causes the haptic sense providing unit 5 to perform a haptic sense providing operation based on the haptic signal. In particular, the control section 25B has a function as an interpolation signal generation section F1B, which will be described later.

[0082] The presentation control device 6 includes a communication unit 61, an encoding unit 62, and a control unit 63. The communication unit 61 performs data communication with the communication unit 41 of the tactile presentation device 4B. Note that the connection between the tactile presentation device 4B and the presentation control device 6 for the data communication may be a wired connection or a wireless connection.

[0083] The control unit 63 is configured with a microcomputer having, for example, a CPU, a ROM, a RAM, etc., and performs overall control of the presentation control device 6 by the CPU performing processing in accordance with a program stored in the ROM. The control unit 63 functions as a haptic signal generation unit 63a. Based on a movement signal input from the haptic presentation device 4B via the communication unit 61, the haptic signal generation unit 63a generates a haptic signal linked to the movement signal. Specifically, from the input movement signal, the haptic signal generation unit 63a analyzes and acquires object information (at least information indicating the type of object) of a virtual object contacted by the virtual hand of the user U in the VR space and information on the manner of contact (tracing, grasping, and speed), and generates an appropriate haptic signal according to the object information and the manner of contact. The control unit 63 also causes the encoding unit 62 to encode the generated haptic signal, packetizes the encoded data, and outputs the packetized data to the haptic presentation device 4B via the communication unit 61.

[0084] An interpolation signal generation method by the interpolation signal generation unit F1B (an interpolation signal generation method according to the third embodiment) will be described with reference to FIG. First, let us assume that in this case, the motion signal from the motion sensor 43 is also separated by frames Fr, just like the haptic signal. Furthermore, let us assume that the control unit 25B in the haptic presentation device 4B is capable of identifying the frame Fr of the haptic signal synchronized with the frame Fr of the motion signal. For example, the control unit 63 in the presentation control device 6 assigns a packet ID to the packet data of the haptic signal with the same number as the frame number of the frame Fr of the motion signal synchronized with the haptic signal, thereby enabling the control unit 25B to identify the frame Fr of the haptic signal synchronized with the frame Fr of the motion signal.

[0085] When the control unit 25B (interpolated signal generating unit F1B) detects the loss of a packet of the haptic signal, it generates an interpolated signal based on the interlocking characteristics of movement and vibration. Specifically, to generate the interpolated signal in this case, a physical model is used that assumes that as the contact speed with an object increases, the amplitude and frequency of the vibrations generated by the contact also increase. This physical model can be rephrased as a model that "when an object is touched quickly, stronger and finer vibrations are transmitted to the hand." In this example, the movement speed of the tactile presentation device 4B obtained by the movement sensor 43 (the movement speed of the hand of the user U) represents the contact speed with the virtual object, and the vibrations to be presented are vibrations generated by contact with the virtual object, so this physical model applies.

[0086] A specific method for generating an interpolated signal will be described below. For the sake of explanation, the frame number of the movement signal and the frame number of the haptic signal synchronized with that frame Fr are treated as the same. Also, it is assumed here that the haptic signal of frame Fr2 has disappeared from the haptic signal in the continuous section from frame Fr1 to frame Fr3.

[0087] First, for frame Fr1, which is the frame preceding the lost frame Fr, the speed of movement identified from the movement signal, and the amplitude (vibration amplitude) and frequency of the haptic signal are obtained. Similarly, for frame Fr3, which is the frame following the lost frame Fr, the speed of movement identified from the movement signal, and the amplitude and frequency of the haptic signal are obtained. Then, based on this acquired information, the relationship between the speed of movement and the amplitude and frequency of the vibration is identified. The speed of movement in frame Fr2, where the haptic signal was lost, is then applied to the identified relationship to generate an interpolated signal for frame Fr2.

[0088] Specifically, a physical model in which the velocity and vibration amplitude / frequency are proportional to each other is created from the velocity, vibration amplitude, and frequency of the motion of frame Fr1 and the velocity, vibration amplitude, and frequency of the motion of frame Fr3.

number

number

[0089] To improve the accuracy of deriving the relationship between movement speed and vibration amplitude and frequency, it is desirable to use multiple frames, rather than just the single frame immediately before the loss, as in this example. In this case, by using only information from the frames preceding the lost frame, it is not necessary to wait for the reception and decoding of several packets after the lost packet to derive the relationship, thereby reducing the delay until the interpolated signal is generated. It is believed that the perception of the surface shape and texture of an object, as presented in this example, is processed in the human brain by strongly linking the vibrations perceived by the skin with the person's own movement. Therefore, a delay in the vibrations applied to the hand alone can significantly affect perception. Therefore, minimizing delay is important.

[0090] By preparing the above-mentioned physical model in accordance with various touching methods and the physical properties of virtual objects, it can be applied to all types of contact with objects. This enables packet loss concealment using interpolated signals linked based on motion information and a physical model, and allows the user to be presented with interpolated signals that provide natural feedback of their own motion.

[0091] This method assumes that the way a person touches an object does not change suddenly compared to before and after the lost frame Fr, in other words, that the person has not suddenly touched a different substance than in the previous frame or grabbed something they were tracing. This assumption is the basis for being able to estimate the signal information of the lost packet from the information of the packets before and after. In reality, it is expected that the way someone touches an object may change suddenly in one frame, but if packet loss is to be concealed, it is important to eliminate unnaturalness, and this is thought to provide sufficient packet loss concealment.

[0092] FIG. 13 is a flowchart showing an example of a processing procedure for realizing the interpolation signal generating method according to the third embodiment described above. In this example, the processing shown in Figure 13 is executed by the control unit 25B shown in Figure 11, and specifically, the CPU in the control unit 25B executes it based on a program stored in a predetermined storage device such as a ROM of the control unit 25B.

[0093] In this case, the processing from steps S101 to S103 is executed in the same manner as in the case of FIG. If the control unit 25B determines in step S103 that a packet loss has been detected, the control unit 25B proceeds to step S104B to perform an interpolation signal generation process. Specifically, the interpolation signal generation process in step S104B performs a process of generating an interpolation signal based on the interlocking characteristics of movement and vibration using a motion signal detected by the motion sensor 43, as described above. Specifically, the interpolation signal generation process is performed based on, for example, the above-mentioned [Equation 5], but since the details have already been described, a repeated description will be avoided.

[0094] In step S105B following step S104B, the control unit 25B performs a signal output process, which includes outputting the interpolated signal generated in step S104B and the haptic signal of the frame Fr to be output (when no packet loss is detected in step S103) to the drive unit 42.

[0095] The control unit 25B advances the process to step S108 in response to the execution of the signal output process in step S105B, but the process from step S108 onwards is the same as in the case of FIG. 6, so a duplicated description will be avoided.

[0096] In the above, as an example of generating an interpolated signal based on a motion signal, a case where the tactile presentation device 4B generates an interpolated signal using a motion signal detected by its own motion sensor 43 has been exemplified, but the interpolated signal generation method of the third embodiment can also be applied to a case where content in which motion signals and tactile signals are recorded is read from a recording medium and transmitted to the tactile presentation device 4B. In this case, it is assumed that the motion corresponding to the motion signal is also reproduced by the tactile presentation device 4B. In this case, if at least the movement signal can be received in the frame Fr where the haptic signal is lost, an interpolated signal can be generated. Generally, movement information can be transmitted with a smaller amount of information than vibration information, so even if transmission and reception of vibration information fails, it is possible that at least the movement information can be transmitted and received. Therefore, it is preferable to generate an interpolated signal based on the movement signal, as in the third embodiment.

[0097] 4. Fourth Embodiment Next, a fourth embodiment will be described, which relates to the selection and setting of an interpolation method.

[0098] FIG. 14 is a block diagram for explaining an example of the internal configuration of a transmission device 1C according to the fourth embodiment, and shows the tactile sensor 3 together with the example of the internal configuration of the transmission device 1C. The transmitting device 1C differs from the transmitting device 1 shown in Fig. 2 in that a control unit 15C is provided instead of the control unit 15. The control unit 15C differs from the control unit 15 in that it has the functions of an interpolation method selection unit F5 and an interpolation method setting unit F6.

[0099] The interpolation method selection unit F5 selects an interpolation method for interpolating haptic signals that were lost during transmission based on the results of signal analysis of the haptic signals, which is based on a physical model of haptics or a human perception model. The interpolation method setting unit F6 performs processing to generate transmission data for the haptic signal in which additional information indicating the interpolation method selected by the interpolation method selection unit F5 is added to the haptic signal. In this embodiment, an interpolation method ID is added as additional information indicating the interpolation method.

[0100] Here, in this example, the interpolation methods to be selected are an interpolation method based on the damped sine wave model described in the first embodiment, and an interpolation method based on the human perception model described in the second embodiment. For this reason, in this example, the interpolation method selection unit F5 performs signal analysis based on the damped sine wave model and the human perception model as signal analysis for selecting an interpolation method. In this case, the selection and setting of the interpolation method is performed in units of frames Fr of the haptic signal.

[0101] As can be understood from the previous description of the second embodiment, the interpolation method based on the human perception model can be suitably applied under conditions where the frequency of the haptic signal is 200 Hz or higher. Therefore, the interpolation method selection unit F5 selects an interpolation method based on the results of signal analysis based on a human perception model, and selects an interpolation method based on the results of determining whether the energy in a band above a predetermined frequency of the tactile signal, specifically in a band above 200 Hz, is above a predetermined threshold. Specifically, if the energy in the band above 200 Hz for the tactile signal of at least the frame Fr for which the interpolation method is selected and set is above a predetermined threshold, an interpolation method based on a human perception model is selected as the interpolation method for the target frame Fr.

[0102] Furthermore, in this example, the interpolation method selection unit F5 performs processing to determine whether or not to select an interpolation method based on an attenuation-limited wave model if the energy in the band above 200 Hz in the haptic signal of the target frame Fr is not above a predetermined threshold. This determination is made by calculating the decay sine wave (Ae) from the haptic signal for three consecutive frames (hereafter referred to as "signal y[t]") consisting of the haptic signal for the target frame Fr and the haptic signals for the frames before and after it, using the following [Equation 6]: -λtThe fit evaluation value for (sin(2πft)) is calculated and the calculation is performed based on the evaluation value.

number

[0103] If the fit evaluation value is equal to or greater than a threshold, the interpolation method selection unit F5 selects an interpolation method based on a damped sine wave model as the interpolation method for the target frame Fr. In this example, if the fit evaluation value is less than the threshold, the interpolation method selection unit F5 selects no interpolation method. In this case, ID information indicating no interpolation method is generated as the interpolation method ID.

[0104] FIG. 15 is a diagram showing an example of the data structure of packet data in the fourth embodiment. As shown in the figure, the selected interpolation method ID can be stored as one of the additional information for the actual data of the haptic signal in the packet data.

[0105] FIG. 16 is a block diagram for explaining an example of the internal configuration of a receiving device 2C according to the fourth embodiment, and shows the tactile presentation device 4 together with the example of the internal configuration of the receiving device 2C. 4, the receiving device 2C is different in that a control unit 25C is provided instead of the control unit 25. The control unit 25C is different from the control unit 25 in that an interpolation signal generation unit F1C is provided instead of the interpolation signal generation unit F1.

[0106] The interpolation signal generation unit F1C selects an interpolation method for the haptic signal based on additional information added to the haptic signal, specifically, in this example, information on the interpolation method ID added to the haptic signal. As can be seen by referring to Figure 15, in this example, an interpolation method ID, which is additional information indicating the interpolation method, is added to each frame Fr (each unit section) of the haptic signal, and therefore the interpolation signal generation unit F1C selects an interpolation method for each frame Fr based on the interpolation method ID added to each frame Fr in this manner.

[0107] FIG. 17 is a flowchart showing an example of a processing procedure for realizing the functions of the transmission device 1C according to the fourth embodiment described above. In this example, the processing shown in Figure 17 is executed by the control unit 15C shown in Figure 14, and specifically, the CPU in the control unit 15C executes it based on a program stored in a predetermined storage device such as a ROM of the control unit 15C.

[0108] First, in step S201, the control unit 15C determines whether a predetermined amount of signal (tactile signal) has been buffered, and if it determines that the predetermined amount of signal has not been buffered, it waits for a predetermined time in step S202 and executes the processing of step S201 again. In this embodiment, the signal analysis process for selecting an interpolation method for the target frame Fr may require haptic signals (signal y[t]) for three consecutive frames, including the target frame Fr and the frames before and after it. Therefore, the process waits until the haptic signals for the three frames are buffered by the processing of steps S201 and S202 described above.

[0109] If it is determined in step S201 that a predetermined amount of signal has been buffered, the control unit 15C proceeds to step S203, where it determines whether the signal energy above 200 Hz is above a threshold. In this example, this determination is performed as a process of determining whether the signal energy above 200 Hz is above a threshold for the above-mentioned signal y[t], i.e., the haptic signal for three consecutive frames, including the target frame Fr and the frames before and after it. Here, the signal energy is calculated, for example, using the average amplitude.

[0110] If it is determined in step S203 that the signal energy above 200 Hz is equal to or greater than the threshold, control unit 15C proceeds to step S204, generates an ID of an interpolation method based on a human perception model (interpolation method ID), and proceeds to step S209.

[0111] On the other hand, if it is determined in step S203 that the signal energy above 200 Hz is not above the threshold, the control unit 15C proceeds to step S205, and performs a process of calculating an evaluation value using [Equation 6] using the above-mentioned signal y[t] as a damped sine wave fitting process. Then, in the following step S206, the control unit 15C determines whether the fitting is good, specifically, whether the evaluation value calculated in step S205 is equal to or greater than a threshold value. If the evaluation value is equal to or greater than the threshold value and the fitting is determined to be good, the control unit 15C proceeds to step S207 to generate an ID of an interpolation method based on the damped sine wave model (interpolation method ID), and proceeds to step S209.

[0112] On the other hand, if it is determined in step S206 that the fitting is not good, the control unit 15C proceeds to step S208, generates an ID indicating that no interpolation method is used (interpolation method ID), and proceeds to step S209.

[0113] In step S209, the control unit 15C executes a process of generating packet data including the generated interpolation method ID. As described in the first and second embodiments, generation of an interpolated signal based on a damped sine wave model and generation of an interpolated signal based on a human perception model are performed using haptic signals from frames before and after the lost frame Fr. In consideration of this, in this example, an interpolation method ID is assigned to the frame Fr immediately following the target frame Fr. That is, the process of step S209 generates packet data to which the generated interpolation method ID is assigned as packet data containing the haptic signal of the frame Fr immediately following the target frame Fr. As a result, when a packet is lost, the receiving device 2C can reference the interpolation method ID assigned to the packet immediately following the lost packet and generate an interpolated signal for the lost packet using an appropriate interpolation method.

[0114] In step S210 following step S209, the control unit 15C performs a process of transmitting the packet data generated in step S209 to the receiving device 2C via the communication unit 14 as a packet data transmission process.

[0115] In step S211 following step S210, the control unit 15C determines whether the processing has ended, that is, whether a predetermined processing end condition has been met, and if it determines that the processing has not ended, the process returns to step S201. As a result, the above-mentioned processing for selecting an interpolation method and processing for setting the selected interpolation method are performed for the next target frame.

[0116] In response to determining in step S211 that the process has ended, the control unit 15C ends the series of processes shown in FIG.

[0117] FIG. 18 is a flowchart showing an example of a processing procedure for realizing the functions of a receiving device 2C according to the fourth embodiment. In this example, the processing shown in Figure 18 is executed by the control unit 25C shown in Figure 16, and specifically, the CPU in the control unit 25C executes it based on a program stored in a predetermined storage device such as a ROM of the control unit 25C.

[0118] In this case, the processing from steps S101 to S103 is executed in the same manner as in the case of FIG. If it is determined in step S103 that a packet loss has been detected, the control unit 25C proceeds to step S110 and refers to the interpolation method ID. As described above, in this example, the interpolation method ID indicating the interpolation method for the lost packet is added to the packet immediately following the lost packet. Therefore, in step S110, the interpolation method ID added to the packet immediately following the lost packet is referred to.

[0119] In step S111 following step S110, the control unit 15C performs processing to generate an interpolated signal using a method identified by the ID. That is, if the interpolation method ID referenced in step S110 indicates an interpolation method based on a damped sine wave model, the control unit 15C generates an interpolated signal using an interpolation method based on a damped sine wave model as described in the first embodiment. If the referenced interpolation method ID indicates an interpolation method based on a human perception model, the control unit 15C generates an interpolated signal using an interpolation method based on a human perception model as described in the second embodiment. Note that if the referenced interpolation method ID indicates no interpolation method, the control unit 15C does not generate an interpolated signal. In other words, in this case, signal interpolation of the lost section is not performed.

[0120] In response to having executed the process of step S111, the control unit 15C advances the process to step S105C to perform signal output processing. The signal output processing of step S105C includes outputting the interpolated signal generated in step S111 and the haptic signal of the frame Fr to be output (when no packet loss is detected in step S103) to the tactile presentation device 4 via the second communication unit 24. Note that, when the half overlap method is adopted as exemplified in the second embodiment, signal output processing corresponding to the half overlap method is performed.

[0121] After executing the signal output process in step S105C, the control unit 25C advances the process to step S108. However, the process from step S108 onwards is the same as in the case of FIG. 6, so a duplicated description will be avoided.

[0122] In the above example, the interpolation method candidates are an interpolation method based on a damped sine wave model and an interpolation method based on a human perception model. However, when these two interpolation methods and an interpolation method based on the interlocking characteristics of motion and vibration are candidates, as in the case of application to a VR system in the third embodiment, the interpolation method can be selected based on information about the application in which the haptic signal is used. Specifically, in this case, the interpolation method selection unit F5 determines whether the application in which the haptic signal is used is a VR application such as the one illustrated in FIG. 10. If a positive result is obtained, the interpolation method based on the interlocking characteristics of motion and vibration is selected as the interpolation method. If a negative result is obtained, the interpolation method can be selected between the interpolation method based on a damped sine wave model and the interpolation method based on a human perception model by performing processing similar to that illustrated in FIG. 17.

[0123] The selection of the interpolation method can also be made based on information about the amount of delay or the amount of calculation when the method is adopted. For example, when both the selection condition (S203) for the interpolation method based on the human perception model and the selection condition (S205 and S206) for the interpolation method based on the damped sine wave model shown in FIG. 17 are satisfied, one of the interpolation methods can be selected based on information on the amount of delay or amount of calculation in each case where these interpolation methods are adopted. Specifically, the amount of calculation required is greater when an interpolation method based on a damped sine wave model is adopted, so if an interpolation method selection is made based on the amount of calculation, the interpolation method based on the human perception model will be selected. Also, the amount of delay required is greater when an interpolation method based on a damped sine wave model is adopted, so even if an interpolation method selection is made based on the amount of delay, the interpolation method based on the human perception model will be selected.

[0124] It is also possible to add an interpolation method ID to each frequency band of a haptic signal, in which case the receiving side may generate an interpolated signal using a different interpolation method for each frequency band. For example, when selecting an interpolation method, the target haptic signal is divided into low-frequency components below 200 Hz and high-frequency components above 200 Hz. The high-frequency components above 200 Hz are determined to have a certain level of energy. If a positive result is obtained, an interpolation method ID indicating an interpolation method based on a perceptual model is generated and added as the interpolation method ID for the high-frequency components. On the other hand, if a negative result is obtained, the signal is deemed not to have a significant impact on perception, and an interpolation method ID indicating no interpolation is added as the interpolation method ID for the high-frequency components in this case. In addition, for low-frequency components below 200 Hz, a determination is made as to whether an interpolation method based on a damped sine wave model is applicable, and if an interpolation method based on a damped sine wave model is applicable, an ID indicating the interpolation method based on the damped sine wave model is generated as the interpolation method ID for the low-frequency components, and this ID is added as the interpolation method ID for the low-frequency components. This makes it possible to set an appropriate interpolation method for each frequency band of a haptic signal according to the characteristics of its waveform.

[0125] Note that when the interpolation method is determined as information about the application or system, such as when the actuator used as the tactile presentation unit 5 is required to handle vibrations of 200 Hz or higher due to its characteristics, when the tactile presentation unit 5 is required to be used in a VR game involving motion information, or when the content is required to handle collision vibrations that can be accurately represented by damped oscillatory waves, signal analysis may be omitted and an interpolation method ID corresponding to the entire interval (entire band) may be added. Also, when the computational resources available for interpolation processing in the receiving device are limited, signal analysis may be omitted and an interpolation method ID available within those resources may be added to the entire interval (entire band). For example, the interpolation method based on a perceptual model has the lowest processing load of the methods exemplified in the above embodiments, and therefore this interpolation method may be selected depending on the computational resources.

[0126] Furthermore, the interpolation method ID is not limited to being added to the packet immediately following the lost packet as exemplified above. For example, it may also be added to the packet immediately preceding the lost packet. Hereinafter, the interpolation method ID added to the packet immediately following the lost packet will be referred to as the "forward packet interpolation method ID," and the interpolation method ID added to the packet immediately preceding the lost packet will be referred to as the "backward packet interpolation method ID." The backward packet interpolation method ID is based on the assumption that when a packet is lost, interpolation is performed based on the haptic signal of the packet received before the lost packet. For example, as in the method exemplified in the third embodiment, the backward packet interpolation method ID can be added when the interpolation signal of the lost packet can be generated only from the haptic signal of the packet immediately before the lost packet.

[0127] It is also possible to add both the "forward packet interpolation method ID" and the "backward packet interpolation method ID" as the interpolation method ID. In this case, it is possible to select which interpolation method ID to use to generate an interpolated signal for a lost packet according to any rule. When a "backward packet interpolation method ID" is used, it is not necessary to wait for the arrival of a packet after a loss in order to generate an interpolated signal, thereby reducing delay. On the other hand, when a "forward packet interpolation method ID" is used, the selection of the interpolation method indicated by the interpolation method ID is made taking into account information from the future beyond the lost packet, so a highly reliable interpolation method can be adopted.

[0128] <5. Variations> Although the embodiments according to the present technology have been described above, the present technology is not limited to the specific examples described above, and various modified configurations can be adopted. For example, in the above first, second, and fourth embodiments, a configuration was exemplified in which a tactile signal indicating tactile information sensed by a tactile sensor 3 is input to a transmitting device (1, 1C), and a tactile presentation is performed on the receiving side based on the tactile signal transmitted by the transmitting device. However, instead, a configuration can be adopted in which a tactile signal recorded on a predetermined recording medium is input to a transmitting device, and a tactile presentation is performed on the receiving side based on the tactile signal transmitted by the transmitting device.

[0129] Furthermore, in the above example, the tactile sensor 3 is separate from the transmitting device (1, 1C), but a configuration in which the tactile sensor 3 is integrated with the transmitting device is also conceivable. It is also possible to adopt a configuration in which the tactile presentation device 4 is integrated with the receiving device (2, 2A, 2C).

[0130] <6. Summary of embodiments> As described above, the first signal processing device (receiving devices 2, 2A, 2C, tactile presentation device 4B) as an embodiment is equipped with an interpolation signal generation unit (F1, F1A, F1B, F1C) that generates an interpolation signal based on the tactile signal in the section adjacent to the lost section and a physical model or human perception model related to tactile sensation as an interpolation signal to interpolate the tactile signal lost during transmission. As a result, an interpolated signal based on haptic characteristics is generated as an interpolated signal for a missing section of the haptic signal. Therefore, the loss of the haptic signal can be made less noticeable to the user, and the user's sense of immersion in the haptic content can be improved.

[0131] In the first signal processing device (receiving device 2, 2C) according to the embodiment, the interpolation signal generating unit (receiving device F1, F1C) generates an interpolation signal based on a damped sine wave model as the generation of an interpolation signal based on a physical model. Damped sine waves are frequently observed in haptic signals, for example as vibration waveforms caused by collisions or sliding between objects. Therefore, by generating an interpolated signal based on a damped sine wave model, the loss of a haptic signal can be made less noticeable to the user, thereby improving the user's sense of immersion in the haptic content.

[0132] Furthermore, in the first signal processing device (receiving device 2A, 2C) as an embodiment, the interpolation signal generation unit (same F1A, F1C) generates an interpolation signal based on the tactile subjective intensity of the tactile signal in an adjacent section as the generation of an interpolation signal based on a perceptual model. This makes it possible to interpolate the tactile signal in the disappearance section based on the tactile subjective intensity, which is the main factor when humans discriminate between tactile stimuli. Therefore, the loss of the haptic signal can be made less noticeable to the user, and the user's sense of immersion in the haptic content can be improved.

[0133] Furthermore, in the first signal processing device (receiving device 2A, 2C) as an embodiment, the interpolation signal generation unit (same F1A, F1C) generates an interpolation signal based on the envelope shape of the haptic signal in adjacent sections as the generation of an interpolation signal based on a perceptual model. This makes it possible to interpolate the missing section of the tactile signal based on the envelope shape of the tactile signal, which is a key factor in humans discriminating between differences in tactile stimuli. Therefore, the loss of the haptic signal can be made less noticeable to the user, and the user's sense of immersion in the haptic content can be improved.

[0134] Furthermore, in the first signal processing device (receiving device 2A, 2C) as an embodiment, the interpolation signal generation unit (receiving device F1A, F1C) generates an interpolation signal by linearly interpolating at least one of the tactile subjective intensity and envelope shape of the tactile signal in adjacent sections. This makes it possible to generate an interpolated signal so that the tactile subjective intensity and envelope shape change continuously for adjacent sections, making it less noticeable to the user when the tactile signal is lost, thereby improving the user's sense of immersion in the tactile content.

[0135] Furthermore, in the first signal processing device (tactile presentation device 4B) as an embodiment, the interpolation signal generation unit (F1B) generates an interpolation signal based on the linkage characteristics between movement and vibration as the generation of an interpolation signal based on a physical model. This makes it possible to appropriately generate an interpolation signal from the movement signal corresponding to the lost section in response to a case where a haptic sensation is presented to a user by a haptic signal linked to a movement signal indicating the user's movement. Therefore, the loss of the haptic signal can be made less noticeable to the user, and the user's sense of immersion in the haptic content can be improved.

[0136] Furthermore, in the first signal processing device (receiving device 2C) as an embodiment, the interpolation signal generating unit (F1C) selects an interpolation method for the haptic signal based on additional information added to the haptic signal. This makes it possible to generate an interpolated signal for the lost section using an appropriate interpolation method that is set in advance before transmitting the haptic signal. Therefore, the loss of the haptic signal can be made less noticeable to the user, and the user's sense of immersion in the haptic content can be improved.

[0137] In addition, in the first signal processing device as an embodiment, additional information is added to each unit section of the haptic signal, and the interpolation signal generation unit (F1C) selects an interpolation method for the haptic signal based on the additional information in the unit section adjacent to the lost section. This makes it possible to select an appropriate interpolation method corresponding to the lost section when the appropriate interpolation method differs depending on which section of the haptic signal has lost signal. Therefore, the loss of the haptic signal can be made less noticeable to the user, and the user's sense of immersion in the haptic content can be improved.

[0138] Furthermore, a first signal processing method as an embodiment is a signal processing method in which a signal processing device generates an interpolated signal that interpolates a haptic signal that was lost during transmission, based on a haptic signal in a section adjacent to the lost section and a physical model or a human perception model related to haptics. This signal processing method can also provide the same functions and effects as the first signal processing device as the above-described embodiment.

[0139] The second signal processing device (transmitting device 1C) according to the embodiment includes an interpolation method selection unit (F5) that selects an interpolation method for interpolating a haptic signal that is lost during transmission based on the results of signal analysis of the haptic signal that is based on a physical model of haptics or a human perception model, and an interpolation method setting unit (F6) that performs processing to generate transmission data for the haptic signal in which additional information indicating the interpolation method selected by the interpolation method selection unit is added to the haptic signal. According to the above configuration, the interpolation process for interpolating haptic signals lost during transmission can be performed using an appropriate method selected based on a physical model related to haptics and a human perception model. Therefore, the loss of the haptic signal can be made less noticeable to the user, and the user's sense of immersion in the haptic content can be improved.

[0140] In the first signal processing device as an embodiment, the interpolation method selection unit performs signal analysis based on a damped sine wave model as signal analysis based on a physical model (see FIG. 17). This makes it possible to appropriately determine whether or not to select an interpolation method based on a damped sine wave model when one of the interpolation method candidates includes an interpolation method based on the damped sine wave model. Therefore, it is possible to ensure that the interpolation processing of the haptic signal is performed using an appropriate interpolation method, making it difficult for the user to notice the loss of the haptic signal, thereby improving the user's sense of immersion in the haptic content.

[0141] Furthermore, in the first signal processing device as an embodiment, the interpolation method selection unit selects an interpolation method based on the results of signal analysis based on a perceptual model, by determining whether the energy in a band above a predetermined frequency of the haptic signal is above a predetermined threshold (see Figure 17). This makes it possible to appropriately determine whether or not to select an interpolation method based on a human perceptual model related to touch when one of the interpolation method options is based on the perceptual model. Therefore, it is possible to ensure that the interpolation processing of the haptic signal is performed using an appropriate interpolation method, making it difficult for the user to notice the loss of the haptic signal, thereby improving the user's sense of immersion in the haptic content.

[0142] Furthermore, in the first signal processing device as an embodiment, the interpolation method selection unit selects an interpolation method based on information about the amount of delay or the amount of calculation when the method is adopted. This makes it possible to give priority to selecting an interpolation method that has a small amount of delay and calculation. Therefore, by reducing the amount of delay in haptic reproduction when interpolation processing is performed in response to signal loss, it is possible to improve the user's sense of immersion and reduce the processing load required for interpolation processing.

[0143] In the first signal processing device as an embodiment, the interpolation method selection unit selects an interpolation method based on information about the application in which the haptic signal is used. This makes it possible to select and set an appropriate interpolation method in cases where one of the options includes an interpolation method that can only be applied to a specific application, such as an application that presents tactile sensations in response to the user's movements, where interpolation processing can be performed using an interpolation method based on the linkage characteristics between movement and vibration. Therefore, the loss of the haptic signal can be made less noticeable to the user, and the user's sense of immersion in the haptic content can be improved.

[0144] Furthermore, in the first signal processing device as an embodiment, the interpolation method selection unit performs signal analysis for each unit interval of the haptic signal using the haptic signals of adjacent unit intervals to select an interpolation method for each unit interval, and the interpolation method setting unit performs processing to generate transmission data to which additional information indicating the interpolation method selected by the interpolation method selection unit is added for each unit interval of the haptic signal (see Figure 17). This makes it possible to select an appropriate interpolation method corresponding to the lost section when the appropriate interpolation method differs depending on which section of the haptic signal has lost signal. Therefore, the loss of the haptic signal can be made less noticeable to the user, and the user's sense of immersion in the haptic content can be improved.

[0145] In addition, a second signal processing method as an embodiment is a signal processing method in which a signal processing device selects an interpolation method for interpolating a haptic signal that is lost during transmission based on the results of signal analysis of the haptic signal that is based on a physical model of tactile sensation or a human perception model, and performs processing so that transmission data for the haptic signal is generated to which additional information indicating the selected interpolation method is added. This signal processing method can also provide the same functions and effects as the second signal processing device as the above-described embodiment.

[0146] Here, as an embodiment, a program that causes a computer device such as a CPU to realize the functions of the interpolation signal generation units (F1, F1A, F1B, F1C) described in Figures 6, 9, 13, 18, etc., and the functions of the interpolation method selection unit (F5) and the interpolation method setting unit (F6) described in Figure 17, etc., can be considered.

[0147] That is, the first program as an embodiment is a program that can be read by a computer device and that causes the computer device to realize the function of generating an interpolation signal that interpolates a tactile signal that has been lost during transmission, based on a tactile signal in a section adjacent to the lost section and a physical model or a human perception model related to tactile sensation. The first signal processing device as the above-described embodiment can be realized by such a first program.

[0148] Furthermore, a second program as an embodiment is a program readable by a computer device that causes the computer device to realize an interpolation method selection function that selects an interpolation method for interpolating a haptic signal that is lost during transmission based on the results of signal analysis of the haptic signal that is based on a physical model of tactile sensation or a human perception model, and an interpolation method setting function that performs processing to generate transmission data for the haptic signal in which additional information indicating the interpolation method selected by the interpolation method selection function is added to the haptic signal. Such a second program can realize the second signal processing device as the above-described embodiment.

[0149] The first and second programs described above can be pre-recorded on a recording medium built into a device such as a computer device, or on a ROM or the like in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magnet Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. The first and second programs can be installed onto a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as the Internet or a LAN (Local Area Network).

[0150] Furthermore, the first and second programs can contribute to the widespread provision of the first and second signal processing devices of the embodiments. For example, by downloading the programs to a personal computer, a portable information processing device, a mobile phone, a game device, an AV (Audio Visual) device, or the like, the personal computer or the like can function as the first and second signal processing devices of the present technology.

[0151] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0152] <7. This Technology> The present technology can also be configured as follows. (1) The present invention includes an interpolation signal generation unit that generates an interpolation signal based on a haptic signal in a section adjacent to the lost section and a physical model or a human perception model related to haptics as an interpolation signal for interpolating a haptic signal lost during transmission. Signal processing device. (2) The interpolation signal generation unit generates an interpolation signal based on a damped sine wave model as the generation of the interpolation signal based on the physical model. The signal processing device according to (1) above. (3) The interpolation signal generation unit generates an interpolation signal based on the tactile subjective intensity of the haptic signal in the adjacent section as the generation of the interpolation signal based on the perceptual model. The signal processing device according to (1) above. (4) The interpolation signal generation unit generates an interpolation signal based on an envelope shape of the haptic signal in the adjacent section as the generation of the interpolation signal based on the perceptual model. The signal processing device according to (1) or (3). (5) The interpolated signal generating unit generates the interpolated signal by linearly interpolating at least one of the tactile subjective intensity and the envelope shape of the tactile signal in the adjacent section. The signal processing device according to (3) above. (6) The interpolation signal generation unit generates an interpolation signal based on a linkage characteristic between motion and vibration as the generation of the interpolation signal based on the physical model. The signal processing device according to (1) above. (7) The interpolation signal generation unit selects an interpolation method for the haptic signal based on additional information added to the haptic signal. The signal processing device according to any one of (1) to (6). (8) the additional information is added to each unit section of the haptic signal; The interpolation signal generation unit selects an interpolation method for a haptic signal based on the additional information in the unit section adjacent to the lost section. The signal processing device according to (7) above. (9) The signal processing device An interpolated signal is generated based on the haptic signal in the section adjacent to the lost section and a physical model or a human perception model related to haptics, as an interpolated signal for interpolating the haptic signal lost during transmission. Signal processing methods. (10) A computer readable program, The computer device is made to realize a function of generating an interpolated signal based on a haptic signal in a section adjacent to the lost section and a physical model or a human perception model related to haptics as an interpolated signal for interpolating a haptic signal lost during transmission. program. (11) an interpolation method selection unit that selects an interpolation method for interpolating a haptic signal that has been lost during transmission based on the results of signal analysis of the haptic signal that is based on a physical model related to haptics or a human perception model; an interpolation method setting unit that performs processing to generate transmission data for a haptic signal in which additional information indicating the interpolation method selected by the interpolation method selection unit is added to the haptic signal, as transmission data for the haptic signal; Signal processing device. (12) The interpolation method selection unit performs signal analysis based on a damped sine wave model as the signal analysis based on the physical model. The signal processing device according to (11) above. (13) The interpolation method selection unit selects the interpolation method based on a result of the signal analysis based on the perceptual model, based on a determination result of whether energy in a band equal to or higher than a predetermined frequency of the haptic signal is equal to or higher than a predetermined threshold. The signal processing device according to (11) or (12). (14) The interpolation method selection unit selects the interpolation method based on information about the amount of delay or the amount of calculation when the method is adopted. The signal processing device according to any one of (11) to (13). (15) The interpolation method selection unit selects the interpolation method based on information about an application in which a haptic signal is used. The signal processing device according to any one of (11) to (14). (16) the interpolation method selection unit performs the signal analysis for each unit section of the haptic signal using the haptic signal of an adjacent unit section, and selects the interpolation method for each unit section; The interpolation method setting unit performs processing to generate the transmission data to which the additional information indicating the interpolation method selected by the interpolation method selection unit is added for each unit section of the haptic signal. The signal processing device according to any one of (11) to (15). (17) The signal processing device An interpolation method for interpolating a haptic signal that has been lost during transmission is selected based on the results of signal analysis of the haptic signal that is based on a physical model related to haptics or a human perception model, and processing is performed to generate transmission data for the haptic signal in which additional information indicating the selected interpolation method is added to the haptic signal. Signal processing methods. (18) A computer readable program, an interpolation method selection function that selects an interpolation method for interpolating a haptic signal that has been lost during transmission based on the results of signal analysis of the haptic signal that is based on a physical model related to haptics or a human perception model; an interpolation method setting function that performs processing so that transmission data in which additional information indicating the interpolation method selected by the interpolation method selection function is added to the haptic signal is generated as transmission data of the haptic signal. program. [Explanation of symbols]

[0153] 100,100B tactile presentation system 1,1C transmitter 2, 2A, 2C receiving device 3 Tactile sensors 4,4B Tactile presentation device 5 Tactile display unit NT Network 11 Sensor I / F (Interface) 12 Encoding section 13 Memory section 14 Communications Department 15,15C Control unit 21 First Communications Department 22 Memory section 23 Decoding unit 24 Second Communications Department 25, 25A, 25B, 25C control section F1, F1A, F1B, F1C Interpolation signal generator 41 Communications Department 42 Drive unit P1, P2, P3 packets Fr1, Fr2, Fr3 frames U User 43 Motion Sensor 6 Presentation control device 61 Communications Department 62 Encoding section 63 Control Unit 63a Tactile signal generator F5 Interpolation method selection F6 Interpolation method setting section

Claims

1. The present invention includes an interpolation signal generation unit that generates an interpolation signal based on a haptic signal in a section adjacent to the lost section and a physical model or a human perception model related to haptics as an interpolation signal for interpolating a haptic signal lost during transmission. Signal processing device.

2. The interpolation signal generation unit generates an interpolation signal based on a damped sine wave model as the generation of the interpolation signal based on the physical model. The signal processing device according to claim 1 .

3. The interpolation signal generation unit generates an interpolation signal based on the perceptual model, the interpolation signal being generated based on the tactile subjective intensities of the haptic signals in the adjacent sections. The signal processing device according to claim 1 .

4. The interpolation signal generation unit generates an interpolation signal based on an envelope shape of the haptic signal in the adjacent section as the generation of the interpolation signal based on the perceptual model. The signal processing device according to claim 1 .

5. The interpolated signal generating unit generates the interpolated signal by linearly interpolating at least one of the tactile subjective intensity and the envelope shape of the tactile signal in the adjacent section. The signal processing device according to claim 3 .

6. The interpolation signal generation unit generates an interpolation signal based on a linkage characteristic between motion and vibration as the generation of the interpolation signal based on the physical model. The signal processing device according to claim 1 .

7. The interpolation signal generation unit selects an interpolation method for the haptic signal based on additional information added to the haptic signal. The signal processing device according to claim 1 .

8. the additional information is added to each unit section of the haptic signal; The interpolation signal generation unit selects an interpolation method for a haptic signal based on the additional information in the unit section adjacent to the lost section. The signal processing device according to claim 7 .

9. The signal processing device An interpolated signal is generated based on the haptic signal in the section adjacent to the lost section and a physical model or a human perception model related to haptics, as an interpolated signal for interpolating the haptic signal lost during transmission. Signal processing methods.

10. A computer readable program, The computer device is made to realize a function of generating an interpolated signal based on a haptic signal in a section adjacent to the lost section and a physical model or a human perception model related to haptics as an interpolated signal for interpolating a haptic signal lost during transmission. program.

11. an interpolation method selection unit that selects an interpolation method for interpolating a haptic signal that has been lost during transmission based on the results of signal analysis of the haptic signal that is based on a physical model related to haptics or a human perception model; an interpolation method setting unit that performs processing to generate transmission data for a haptic signal in which additional information indicating the interpolation method selected by the interpolation method selection unit is added to the haptic signal, as transmission data for the haptic signal; Signal processing device.

12. The interpolation method selection unit performs signal analysis based on a damped sine wave model as the signal analysis based on the physical model. The signal processing device according to claim 11 .

13. The interpolation method selection unit selects the interpolation method based on a result of the signal analysis based on the perceptual model, based on a determination result of whether energy in a band equal to or higher than a predetermined frequency of the haptic signal is equal to or higher than a predetermined threshold. The signal processing device according to claim 11 .

14. The interpolation method selection unit selects the interpolation method based on information about the amount of delay or the amount of calculation when the method is adopted. The signal processing device according to claim 11 .

15. The interpolation method selection unit selects the interpolation method based on information about an application in which a haptic signal is used. The signal processing device according to claim 11 .

16. the interpolation method selection unit performs the signal analysis for each unit section of the haptic signal using the haptic signal of an adjacent unit section, and selects the interpolation method for each unit section; The interpolation method setting unit performs processing to generate the transmission data to which the additional information indicating the interpolation method selected by the interpolation method selection unit is added for each unit section of the haptic signal. The signal processing device according to claim 11 .

17. The signal processing device An interpolation method for interpolating a haptic signal that has been lost during transmission is selected based on the results of signal analysis of the haptic signal that is based on a physical model related to haptics or a human perception model, and processing is performed to generate transmission data for the haptic signal in which additional information indicating the selected interpolation method is added to the haptic signal. Signal processing methods.

18. A computer readable program, an interpolation method selection function that selects an interpolation method for interpolating a haptic signal that has been lost during transmission based on the results of signal analysis of the haptic signal that is based on a physical model related to haptics or a human perception model; an interpolation method setting function that performs processing so that transmission data in which additional information indicating the interpolation method selected by the interpolation method selection function is added to the haptic signal is generated as transmission data of the haptic signal. program.

Images