Decoding device, decoding method, program, encoding device, encoding method
By dividing tactile signals into low and high frequency bands and encoding tactile intensity for the high frequency band, the system effectively reduces data volume while preserving tactile sensation quality, addressing the challenge of increased data load and delays in tactile reproduction systems.
Patent Information
- Application Number
- JP2022505005
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-03-04
- Filing Date
- 2021-01-08
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-01-08
AI Technical Summary
The increase in the number of areas receiving tactile stimulation leads to an increase in the number of tactile channels, resulting in higher data amounts that can cause processing load and transmission delays in tactile reproduction systems.
The system divides tactile signals into two frequency bands, encoding and decoding the low frequency band to maintain tactile reproducibility while reducing data volume by encoding tactile intensity information for the high frequency band, allowing for efficient data compression.
This approach reduces data volume while maintaining tactile reproducibility, optimizing processing load and transmission efficiency, and minimizing delays in tactile sensation presentation.
Smart Images

Figure 0007582291000008 
Figure 0007582291000009 
Figure 0007582291000010
Abstract
Description
[Technical field]
[0001] The present technology relates to a decoding device that decodes an encoded haptic signal, a method thereof, and a program for the decoding device, and a coding device that encodes a haptic signal, a method thereof, and a program for the coding device. [Background technology]
[0002] In recent years, technology has been developed that provides tactile stimulation to a user by vibrating a device worn by the user. Here, the tactile stimulation refers to a physical phenomenon that causes a user to feel a sense of touch through a vibration phenomenon or the like. In addition, the generation of the tactile stimulation is called tactile presentation.
[0003] Tactile presentation technology is used in devices in various fields. For example, in a terminal device equipped with a touch panel such as a smartphone, the touch panel vibrates in response to a touch operation by a user, and a tactile stimulus is given to the user's finger, making it possible to express the feeling of touching a button or the like displayed on the touch panel. In addition, in a music listening device such as a headphone, it is possible to emphasize the deep bass of the music being played by giving a tactile stimulus in accordance with the music being played. In addition, in a device that provides a computer game or VR (Virtual Reality), it is possible to improve the user's immersion in the content by vibrating the controller or the like and giving a tactile stimulus in response to the operation using the controller or the scene of the content.
[0004] In addition, a technology has been developed that provides a tactile stimulus to a user based on a tactile signal received from an external device. For example, Patent Document 1 below discloses a technology that provides a tactile stimulus to a user while changing the frequency and amplitude of vibration based on a received signal. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2016-202486 A Summary of the Invention [Problem to be solved by the invention]
[0006] Here, in a tactile reproduction system that reproduces tactile information, it is considered to provide a plurality of tactile presentation devices and to provide tactile stimulation to a plurality of parts of the human body. However, as the number of parts to which tactile stimulation is applied increases, the number of channels for tactile signals also increases, leading to an increase in the amount of data. An increase in the amount of data for tactile signals is undesirable because it may increase the processing load for tactile reproduction and cause transmission delays.
[0007] The present technology has been developed in consideration of the above circumstances, and aims to reduce the data volume of haptic signals while ensuring the reproducibility of the haptic sensation. [Means for solving the problem]
[0008] The decoding device according to the present technology includes a first decoding unit that decodes coded data obtained by encoding a haptic signal, the coded data including coded data obtained by encoding a first band signal, which is a signal of a first frequency band in the haptic signal, and haptic intensity information, which is information indicating haptic intensity for a second frequency band different from the first frequency band in the haptic signal, the first band signal; a second decoding unit that decodes a second band signal, which is a signal of the second frequency band in the haptic signal, based on the haptic intensity information in the coded data; and a synthesis unit that synthesizes the first band signal decoded by the first decoding unit and the second band signal decoded by the second decoding unit. This makes it possible to decode a haptic signal related to the second frequency band while suppressing degradation of haptic reproducibility for the encoded data generated by the encoding device according to the present technology.
[0009] In the above-described decoding device according to the present technology, it is possible to configure the first frequency band to be a band on the lower frequency side than the second frequency band. For tactile signals in a frequency band above a certain frequency, the human body easily perceives changes in tactile intensity but does not easily perceive changes in frequency.
[0010] In the decoding device relating to the above-mentioned present technology, the haptic intensity information is calculated for each frequency within the second frequency band, and the second decoding unit may be configured to decode the second band signal based on a total value of haptic intensity indicated by the haptic intensity information calculated for each frequency. This makes it possible to reproduce an appropriate tactile intensity for the second frequency band based on the tactile intensity of each frequency in the second band signal before encoding.
[0011] In the decoding device related to the present technology described above, the haptic intensity information is calculated for one or more main frequencies within the second frequency band, and the second decoding unit may be configured to decode the second band signal based on the haptic intensity indicated by the haptic intensity information calculated for the main frequencies. This makes it possible to reproduce an appropriate tactile intensity for the second frequency band based on the tactile intensity of the main frequency in the second band signal before encoding.
[0012] In the decoding device related to the present technology described above, the second decoding unit may be configured to convert the haptic intensity indicated by the haptic intensity information into a signal amplitude, and generate, as a decoded signal of the second band signal, a periodic signal having the converted signal amplitude and whose signal frequency is a frequency within the second frequency band. This makes it possible to decode the second band signal of the encoded data generated by the encoding device according to the present technology in a manner that suppresses degradation of tactile reproducibility.
[0013] In the decoding device according to the present technology described above, it is possible to configure the frequency of the periodic signal to be a frequency that approximately matches a resonance frequency of the tactile presentation device. This can improve the efficiency of driving the tactile presentation device. Note that the tactile presentation device here refers to a device that presents a tactile sensation based on a tactile signal obtained by a decoding device according to the present technology.
[0014] In the decoding device according to the present technology described above, it is considered that the frequency of the periodic signal is set based on the tactile sensitivity characteristics of the human body. This makes it possible to set the frequency of the periodic signal to a frequency at which the human body has high tactile sensitivity, thereby improving the efficiency of driving the tactile presentation device.
[0015] In the above-described decoding device according to the present technology, it is considered that the frequency of the periodic signal is set based on the hearing sensitivity characteristics of the human body. This makes it possible to set the frequency of the periodic signal to a frequency at which the human body has low hearing sensitivity, thereby reducing noise generated by driving the tactile presentation device.
[0016] In the above-described decoding device according to the present technology, the second decoding unit may be configured to set the frequency of the periodic signal based on an operation. This allows the user to arbitrarily select the frequency of the periodic signal.
[0017] In addition, a decoding method according to the present technology is a decoding method for coded data obtained by coding a haptic signal, the coded data including data obtained by coding a first band signal, which is a signal in a first frequency band in the haptic signal, and haptic intensity information, which is information indicating haptic intensity for a second frequency band different from the first frequency band in the haptic signal, the first band signal being decoded, a second decoding procedure for decoding a second band signal, which is a signal in the second frequency band in the haptic signal, based on the haptic intensity information in the coded data, and a synthesis procedure for synthesizing the first band signal decoded by the first decoding procedure and the second band signal decoded by the second decoding procedure. With such a decoding method, the same effect as that of the decoding device according to the present technology described above can be obtained.
[0018] Furthermore, a first program related to the present technology is a program that causes an information processing device to realize a first decoding function for decoding a first band signal, the encoded data being encoded data of a haptic signal, the encoded data including data obtained by encoding a first band signal, which is a signal of a first frequency band in the haptic signal, and haptic intensity information, which is information indicating haptic intensity for a second frequency band different from the first frequency band in the haptic signal, based on the haptic intensity information in the encoded data, a second decoding function for decoding a second band signal, which is a signal of the second frequency band in the haptic signal, based on the haptic intensity information in the encoded data, and a synthesis function for synthesizing the first band signal decoded by the first decoding function and the second band signal decoded by the second decoding function. The above-described decoding device according to the present technology is realized by the first program according to the present technology.
[0019] The encoding device according to the present technology includes a band division unit that divides a haptic signal into a first band signal that is a signal in a first frequency band and a second band signal that is a signal in a second frequency band different from the first frequency band, a first encoding unit that encodes the first band signal, a haptic intensity calculation unit that calculates haptic intensity information indicating haptic intensity for the second frequency band based on the second band signal, and an encoded data generation unit that generates encoded data including data encoded with the first band signal and the haptic intensity information. This makes it possible to efficiently compress information in tactile signals by utilizing the tactile sensitivity characteristics of the human body, whereby in a certain frequency band, both changes in tactile intensity and changes in frequency are easily perceived, but in another frequency band, changes in tactile intensity are easily perceived but changes in frequency are not.
[0020] In the encoding device according to the present technology described above, it is possible to configure the first frequency band to be a band on the lower frequency side than the second frequency band. For tactile signals in a frequency band above a certain frequency, the human body easily perceives changes in tactile intensity but does not easily perceive changes in frequency.
[0021] In the encoding device according to the present technology described above, it is possible to adopt a configuration in which the division frequency between the first frequency band and the second frequency band is variable. By making the division frequency variable, it is possible to adjust the balance between the effect of data reduction by encoding and the effect of suppressing the deterioration of tactile reproducibility.
[0022] In the encoding device according to the present technology described above, the band dividing section may be configured to change the division frequency depending on the stability of communication with an external device. This makes it possible to change the division frequency in a direction that increases the data reduction effect when communication with an external device to which the encoded data is transmitted is unstable, i.e., by lowering the bit rate of the communication data, thereby preventing communication interruptions; conversely, when communication is stable, the division frequency can be changed in a direction that increases tactile reproducibility.
[0023] In the encoding device according to the present technology described above, the band division section may be configured to set the division frequency lower when the communication stability is low than when the communication stability is high. As a result, when communication is unstable, the division frequency is changed in the direction of increasing the data reduction effect, i.e., lowering the bit rate of the communication data, and conversely, when communication is stable, the division frequency is changed in the direction of increasing tactile reproducibility.
[0024] In the encoding device according to the present technology described above, the haptic intensity calculation unit may be configured to calculate the haptic intensity information for each frequency in the second frequency band. This makes it possible to reproduce an appropriate tactile intensity for the second frequency band based on the tactile intensity of each frequency in the second band signal before encoding.
[0025] In the encoding device according to the present technology described above, the haptic intensity calculation unit may be configured to calculate the haptic intensity information only for one or more main frequencies in the second frequency band. This makes it possible to reproduce an appropriate tactile intensity for the second frequency band based on the tactile intensity of the main frequency in the second band signal before encoding.
[0026] In addition, an encoding method according to the present technology is an encoding method including: a band division step of dividing a haptic signal into a first band signal that is a signal in a first frequency band and a second band signal that is a signal in a second frequency band different from the first frequency band; a first encoding step of encoding the first band signal; a haptic intensity calculation step of calculating haptic intensity information indicating haptic intensity for the second frequency band based on the second band signal; and an encoded data generation step of generating encoded data including data obtained by encoding the first band signal and the haptic intensity information. With such an encoding method, the same effect as that of the encoding device according to the present technology described above can be obtained.
[0027] Furthermore, a second program related to the present technology is a program that causes an information processing device to realize a band splitting function that splits a haptic signal into a first band signal that is a signal of a first frequency band and a second band signal that is a signal of a second frequency band different from the first frequency band, a first encoding function that encodes the first band signal, a haptic intensity calculation function that calculates haptic intensity information indicating haptic intensity for the second frequency band based on the second band signal, and an encoded data generation function that generates encoded data including data encoded with the first band signal and the haptic intensity information. The encoding device according to the present technology described above is realized by such a second program according to the present technology. [Brief description of the drawings]
[0028] [Figure 1] 1 is a diagram illustrating an example of the configuration of a haptic reproduction system including an encoding device and a decoding device according to an embodiment of the present technology. [Diagram 2] FIG. 2 is a diagram for explaining an example of the internal configuration of an encoding device according to an embodiment; [Diagram 3] 1 is a diagram showing an example of the internal configuration of a playback device according to a first embodiment; [Figure 4] FIG. 2 is a diagram for explaining an example of the internal configuration of a decoding device according to an embodiment; [Diagram 5] FIG. 1 is an explanatory diagram of an example of use of the tactile reproduction system. [Figure 6] FIG. 4 is an explanatory diagram of a vibration detection threshold curve. [Figure 7] FIG. 2 is a functional block diagram showing functions of an encoding unit in the embodiment. [Figure 8] 1 is a diagram illustrating tactile sensitivity characteristics for each sensory receptor in the human body. [Figure 9] FIG. 1 is an explanatory diagram of tactile intensity. [Figure 10] FIG. 2 is an explanatory diagram of main frequencies. [Figure 11] FIG. 2 is a diagram for explaining an example of an encoded data format. [Figure 12] FIG. 11 is an explanatory diagram of an example of a data structure of tactile intensity information. [Figure 13] 1 is a flowchart illustrating an example of a processing procedure for implementing an encoding method according to an embodiment. [Figure 14] FIG. 13 is a diagram for explaining an example of a machine learning technique in which a DNN model is used to identify a main frequency. [Figure 15] FIG. 2 is a functional block diagram showing functions of a decoding unit in the embodiment. [Figure 16] 11 is a flowchart illustrating an example of a processing procedure for implementing a decoding method according to an embodiment. [Figure 17] 11 is an explanatory diagram of an example in which the frequency of a periodic signal is set based on an operation. FIG. [Figure 18] 11 is an explanatory diagram of an example of control of a division frequency according to communication stability. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0029] Hereinafter, embodiments of the present technology will be described in the following order with reference to the accompanying drawings. <1. Overview of the tactile reproduction system> <2. Encoding device configuration> <3. Configuration of playback device> <4. Configuration of the Decryption Device> <5. Example of use of tactile reproduction system> <6. Tactile reproduction method as an embodiment> [6-1. Issues related to the transmission of tactile signals] [6-2. Encoding method] [6-3. Decryption method] [6-4. Control of division frequency according to communication stability] <7. Summary of the embodiment> <8. This Technology>
[0030] In this specification, each term is defined as follows. Tactile stimulation: A physical phenomenon that allows a person to perceive touch, such as a vibration phenomenon. Tactile presentation: Producing tactile stimuli. Tactile information: Information perceived by the sense of touch, such as vibration information. Tactile Signal: A signal representing a pattern of tactile stimulation, e.g. a signal representing a vibration waveform. Haptic recipient: The person who receives the haptic presentation. Tactile sensitivity: The subjective sensitivity of a tactile stimulus to its intensity. It varies depending on the receptors and parts of the human body. Tactile sensitivity characteristics: Characteristics related to human tactile sensitivity. Varies depending on the part of the body (hands, face, feet, etc.). Encoded data: Data obtained by encoding a signal. Sub-concepts include stream and frame. Tactile encoded data: Data that encodes tactile signals.
[0031] <1. Overview of the tactile reproduction system> FIG. 1 shows an example of the configuration of a haptic reproduction system 1 including an encoding device (encoding device 2) and a decoding device (decoding device 3) as an embodiment of the present technology. First, in this embodiment, the environment for realizing tactile reproduction is divided into a recording environment in which the tactile signal obtained by sensing the target tactile information (tactile stimulation) is encoded and the tactile encoded data Dc obtained by the encoding is recorded, and a reproduction environment in which tactile information is reproduced based on the tactile signal obtained by decoding the tactile encoded data Dc.
[0032] As shown in the figure, the tactile reproduction system 1 comprises, in a recording environment, a plurality of tactile sensors 5 and an encoding device 2 to which the tactile sensors 5 are connected, and, in a reproduction environment, a playback device 4 configured to be capable of acquiring tactile encoded data Dc, a decoding device 3 configured to be capable of wireless communication with the playback device 4, and a plurality of tactile presentation devices 6 connected to the decoding device 3.
[0033] The tactile sensor 5 is a sensor that senses tactile stimuli, and in this example, a vibration sensor such as a piezo pickup or an acceleration sensor is used. The tactile sensor outputs vibrations and movements as voltage changes when it is brought into contact with an object to be sensed, that is, a human body in this example. In this example, each tactile sensor 5 is connected to the encoding device 2 by wire, and each tactile sensor 5 is attached to a different part of the human body as the target to sense the tactile stimulus occurring at each part.
[0034] The encoding device 2 is configured with a computer device such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), and encodes the detection signals (tactile signals) from each tactile sensor 5 in accordance with a predetermined data format, and stores the tactile encoded data Dc obtained by the encoding in, for example, a storage device provided internally.
[0035] The playback device 4 is configured with a computer device such as a CPU and a DSP, and transmits the acquired haptic encoded data Dc to the decoding device 3. For example, the haptic encoded data Dc recorded in the recording environment is acquired by the playback device 4 via a required network such as the Internet, a home network, a LAN (Local Area Network), or a satellite communication network. Alternatively, the haptic encoded data Dc can be recorded on a portable recording medium, and the playback device 4 can acquire the haptic encoded data Dc via the recording medium.
[0036] The decoding device 3 decodes the haptic encoded data Dc received from the playback device 4, and drives each haptic presentation device 6 based on the haptic signal obtained by the decoding.
[0037] The tactile presentation device 6 is a device that generates a tactile stimulus, and in this example, a vibration device such as a vibrator or actuator is used. In this example, each tactile presentation device 6 is attached to a different part of the human body of the touch receiver, and is adapted to reproduce the tactile stimulus sensed by the corresponding tactile sensor 5.
[0038] Here, in this example, each tactile presentation device 6 is connected to the decoding device 3 by wire, and the part surrounded by the dashed line in the figure, i.e., the decoding device 3 and each tactile presentation device 6, is the part that is worn by the haptic receiver. The tactile reproduction system 1 can be configured such that the playback device 4 is provided with the functionality of the decoding device 3 and the playback device 4 is connected to each tactile presentation device 6 by wire, but in that case, there is a risk of causing annoyance to the haptic receiver wearing the tactile presentation device 6. This annoyance is expected to increase as the number of parts to which tactile stimulation is applied increases. The configuration of the tactile reproduction system 1 shown in FIG. 1 makes it possible to prevent such annoyance from being caused to the tactile receiver.
[0039] The tactile reproduction system 1 shown in Figure 1 is configured as a system that reproduces the tactile sensation of each part of the body perceived by a person wearing a tactile sensor 5 in a recipient, and is also capable of handling cases where the two are located remotely.
[0040] In this embodiment, the number of tactile sensors 5 and tactile presentation devices 6, that is, the number of parts of the human body that sense and reproduce tactile stimulation, is at least three.
[0041] <2. Encoding device configuration> Fig. 2 is a diagram for explaining an example of the internal configuration of the encoding device 2. Note that Fig. 2 shows the tactile sensors 5 shown in Fig. 1 together with the example of the internal configuration of the encoding device 2. As shown in the figure, the encoding device 2 includes a plurality of amplifiers 21, a plurality of A / D converters 22, a pre-processing unit 23, an encoding unit 24, a control unit 25, a storage unit 26, a communication unit 27, and a bus 28. The pre-processing unit 23, the encoding unit 24, the control unit 25, the storage unit 26, and the communication unit 27 are connected via the bus 28 and are capable of data communication with one another.
[0042] The detection signal of each tactile sensor 5 is input to a corresponding amplifier 21 and adjusted to an appropriate dynamic range, and then input to a corresponding A / D converter 22 and A / D converted (analog / digital converted). Each A / D converted detection signal (that is, the tactile signal for each part) is input to a pre-processing unit 23. In the pre-processing unit 23, various types of digital signal processing such as noise removal and calibration of the sensor characteristics of the tactile sensor 5 are performed. Each haptic signal that has been subjected to signal processing by the pre-processing unit 23 is input to the encoding unit 24.
[0043] The encoding unit 24 is composed of, for example, a DSP, and encodes each input haptic signal according to a predetermined data format to obtain the above-mentioned encoded haptic data Dc. The encoding of the haptic signal according to this embodiment will be described later.
[0044] The control unit 25 is configured with a microcomputer having, for example, a CPU, a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and performs overall control of the encoding device 2 by executing processing in accordance with a program stored in the ROM. For example, the control unit 25 performs data communication with an external device via the communication unit 27. The communication unit 27 is configured to be capable of performing data communication with an external device via a network such as the Internet, and the control unit 25 is capable of performing data communication with an external device connected to the network via the communication unit 27. In particular, the haptic encoded data Dc obtained by the encoding unit 24 can be transmitted to the external device via the communication unit 27.
[0045] The storage unit 26 is a general representation of storage devices such as a hard disk drive (HDD) or a solid state drive (SSD), and is used to store various types of data in the encoding device 2. For example, the storage unit 26 stores data necessary for control by the control unit 25. Also, under the control of the control unit 25, the haptic encoded data Dc obtained by the encoding unit 24 can be stored in the storage unit 26.
[0046] <3. Configuration of playback device> FIG. 3 is a diagram showing an example of the internal configuration of the playback device 4. As shown in the figure, the playback device 4 includes a control unit 41, a communication unit 42, a media drive 43, a storage unit 44, and a wireless communication unit 45, as well as a bus 46 that connects these units together so that data can be communicated between them.
[0047] The control unit 41 is configured with, for example, a microcomputer having a CPU, ROM, RAM, etc., and performs overall control of the playback device 4.
[0048] The communication unit 42 is configured to be capable of performing data communication with an external device via a network such as the Internet. The control unit 41 is capable of performing data communication with an external device connected to the network via the communication unit 42. In particular, it is possible to cause the communication unit 42 to receive haptic encoded data Dc from an external device such as a server device on the network.
[0049] The media drive 43 is configured to allow a portable recording medium to be detachably attached, and is configured as a reader / writer unit capable of writing and reading data to and from the attached recording medium. Examples of recording media compatible with the media drive 43 include memory cards (e.g., portable flash memory) and optical disk recording media. This media drive 43 makes it possible to read out the haptic encoded data Dc recorded on a portable recording medium.
[0050] The storage unit 44 is a comprehensive representation of storage devices such as an HDD or SSD, and is used to store various types of data in the playback device 4. For example, the storage unit 44 stores data necessary for control by the control unit 41. Furthermore, based on the control of the control unit 41, the storage unit 44 can also store haptic encoded data Dc read by the media drive 43 and haptic encoded data Dc received from an external device by the communication unit 42.
[0051] The wireless communication unit 45 performs short-distance wireless communication using a predetermined communication method such as Bluetooth (registered trademark).
[0052] Here, as part of the overall control described above, the control unit 41 controls the communication unit 42 to receive the haptic encoded data Dc and the media drive 43 to read it. The control unit 41 also controls the wireless communication unit 45 to transmit the haptic encoded data Dc obtained via the communication unit 42 and the media drive 43 to the decoding device 3.
[0053] <4. Configuration of the Decryption Device> FIG. 4 is a diagram for explaining an example of the internal configuration of the decoding device 3, and shows each of the tactile presentation devices 6 together with the example of the internal configuration of the decoding device 3. As shown in FIG. As shown in the figure, the decoding device 3 includes a plurality of amplifiers 31, a plurality of D / A converters 32, a post-processing unit 33, a decoding unit 34, a control unit 35, a wireless communication unit 36, a storage unit 37, an operation unit 38, a display unit 39, and a bus 30. The post-processing unit 33, the decoding unit 34, the control unit 35, the wireless communication unit 36, and the storage unit 37 are connected via the bus 30 and are capable of data communication with one another.
[0054] The control unit 35 is configured with, for example, a microcomputer having a CPU, ROM, RAM, etc., and performs overall control of the decoding device 3.
[0055] The wireless communication unit 36 performs short-range wireless communication using a method such as Bluetooth that enables communication with the wireless communication unit 45 in the playback device 4. The haptic encoded data Dc transmitted from the playback device 4 is received by the wireless communication unit 36.
[0056] The storage unit 37 is a storage device similar to the storage units 26 and 44, and is used to store various data used by the control unit 35 and the like.
[0057] The operation unit 38 collectively represents various operators such as buttons, keys, a touch panel (touch sensor) and the like provided on the decoding device 3, and outputs operation input information to the control unit 35 in response to an operation input.
[0058] The display unit 39 is configured to have a display device such as a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display, and displays various information such as image information based on instructions from the control unit 35.
[0059] The decoding unit 34 decodes the haptic encoded data Dc input via the wireless communication unit 36 using a method described below to obtain a haptic signal for each body part. The haptic signals for each body part obtained by the decoding unit 34 are input to the post-processing unit 33.
[0060] The post-processing unit 33 performs signal processing such as calibration of the tactile presentation device 6 and predetermined filtering, as necessary, on the input tactile signals for each part.
[0061] Each tactile signal that has passed through the post-processing unit 33 is input to a corresponding D / A converter 32 for D / A conversion (digital / analog conversion), then adjusted to an appropriate dynamic range by a corresponding amplifier 31, and output to a corresponding tactile presentation device 6. As a result, each tactile presentation device 6 is driven based on the tactile signal, and the tactile stimulus that is the subject of sensing in the detection environment can be given to the touch recipient (that is, tactile information can be reproduced).
[0062] Although only haptic signals have been mentioned above, audio and video signals may be recorded together with the haptic signals, so that sounds and videos may be provided to the touch recipient along with the haptic information.
[0063] <5. Example of use of tactile reproduction system> Consider playing content that provides tactile feedback in addition to video. FIG. 5 is an explanatory diagram of an example of use of the tactile reproduction system 1. As shown in FIG. 5, when creating content, in addition to video, tactile signals recorded by tactile sensors 5 (in the figure, tactile sensors 5b for the torso, 5h for the fingers, and 5f for the feet) worn by a person Hm1 as a tactile recorder are encoded by an encoding device 2, and the encoded data is recorded in synchronization with the video, and recorded as content Cnt. During playback, the recorded content Cnt is transmitted to a decoding device 3, for example, by wireless communication, and the decoding device 3 decodes the received content Cnt in a decoding section 34. As a result, tactile presentation is performed based on the corresponding tactile signals through tactile presentation devices 6 (in the figure, tactile presentation device 6b for the torso, tactile presentation device 6h for the fingers, and tactile presentation device 6f for the feet) worn by a person Hm2 as a touch receiver.
[0064] Examples of scenes in which haptics are actually presented within the video include scenes in which characters punch (or are punched), shoot (or are shot), are hit by an explosion, or feel the ground shaking. The figure shows example waveforms of haptic signals for the torso, hands, and feet as examples of haptic signals in actual content. Explaining in chronological order, first, in a scene where person Hm1 shoots someone with a gun, vibrations occur in the fingers due to the recoil of the bullet, then in a scene where the other person also shoots a gun at person Hm1's torso, vibrations occur in the torso due to the impact of the gunshot, and then in a scene where an earthquake occurs, the vibrations of the ground are gradually transmitted to the feet, torso, and fingers. For example, by playing such content, the person Hm2 acting as the haptic receiver can experience high-quality reality by reproducing the sense of touch through vibration in addition to video and audio.
[0065] <6. Tactile reproduction method as an embodiment> [6-1. Issues related to the transmission of tactile signals] A tactile reproduction method according to an embodiment will be described below. First, the tactile reproduction method according to the embodiment is a method that focuses on the tactile characteristics of humans. The vibration detection threshold curve shown in Figure 6 has been reported as a guide to human tactile sensitivity. In Figure 6, the horizontal axis represents frequency, and the vertical axis represents the magnitude of the tactile stimulus (vibration: here, displacement). The vibration detection threshold curve in Figure 6 is based on the experimental results of "Four cahnnels mediate the mechanical aspects of touch SJ, Bolanowski 1988". The vibration detection threshold curve shown in Figure 6 is an example of an experiment to determine whether or not humans can sense a vibration as touch, that is, tactile sensitivity. Humans cannot sense vibrations smaller than this curve as touch.
[0066] As can be seen from Figure 6, the frequency at which humans have the highest tactile sensitivity is generally considered to be around 200 Hz. Therefore, devices and applications that generate vibrations are often designed to generate vibrations up to around 200 Hz.
[0067] On the other hand, although it is not shown in the results of Figure 6, it is commonly known that humans can sense vibrations with frequencies up to about 1 kHz as touch. Humans can sense vibrations that contain frequency components of about 1 kHz as different tactile sensations from vibrations that do not. For example, the vibrations generated when removing a cork from a bottle include vibrations with frequencies up to several kHz. If the vibrations are transmitted from a haptic device at frequencies up to several hundred Hz, the user cannot feel the vibrations with a sufficient sense of reality as the haptic sensation of removing the cork from the bottle. Therefore, in order to provide a user with a more realistic tactile sensation, it is necessary to present the tactile sensation using vibrations including frequencies up to about 1 kHz.
[0068] However, as the frequency range of a signal becomes wider, the amount of data in the signal increases, and this increases the possibility of delays in signal transmission and reception. In other words, if the quality of the tactile sensation is improved, a situation may arise in which the tactile sensation cannot be presented at the appropriate time.
[0069] The delay in signal transmission and reception and in tactile presentation will be explained using specific examples. First, the data volume of a haptic signal will be explained. When a haptic signal is transmitted between devices, it is first converted into digital data. The capacity of digital data is expressed as the number of bits required per unit time, i.e., bit rate B. Incidentally, haptic sensitivity depends not only on the frequency but also on the amplitude of vibration. For example, the experimental results in FIG. 6 above show that the amplitude of vibration felt by humans is about 50 dB (-20 dB to 30 dB) or more, and the frequency is about 1000 Hz. In the following, taking into account the distribution of haptic information actually felt by humans, the amplitude of vibration is considered to be about 70 dB.
[0070] When the haptic signal TS is digitized using Linear Pulse Code Modulation (LPCM), the vibration amplitude that can be expressed by 1 bit is 6 dB. In other words, a vibration amplitude of 70 dB requires 12 bits. On the other hand, if the vibration frequency is 1000 Hz, the sampling frequency must be doubled to 2000 Hz, and the bit rate B0 can be calculated using the following [Equation 1]. B0=12bit / sample×2000sample / sec=24kbit / sec...[Formula 1]
[0071] This value itself is extremely small compared to, for example, the bit rate of CD (Compact Disc), a typical format for audio signals, which is 700 kbps / ch. Therefore, it seems unlikely that incorporating this haptic signal as an additional component in some system would pose a major problem.
[0072] However, as mentioned above, it is known that the bandwidth of haptic signals that humans can sense extends up to several kHz. For example, if haptic signals are reproduced up to 2000 Hz, the bit rate will be 48 kbit / sec, twice that of [Equation 1].
[0073] Furthermore, unlike vision (two eyes) and hearing (two ears), touch is present everywhere on the surface of the human body. Just considering the fingertips on both hands, there are ten locations, and if we were to handle all of these tactile signals, the bit rate would be ten times higher, at 480 kbit / sec. If we were to add each finger joint, palm, and other locations, the bit rate would increase dramatically.
[0074] Furthermore, while tactile signals are essentially one-dimensional, the physical phenomenon of vibration can be captured in three axes (x, y, z). To handle all of this, a bit rate of 1440 kbit / sec, three times faster, would be required, exceeding the 1411 kbit / sec of an audio CD.
[0075] In this way, the total amount of data required for haptic signals increases with improvements in the reproducibility of haptic stimulation and with an increase in the number of parts of the user to which haptic stimulation is applied, and this increase in the total amount of data places a heavy load on the network system that transmits the haptic signals.
[0076] Further, factors other than the total data volume as described above may also be considered as causes of delays. For example, when a haptic signal is transmitted by wireless communication, it is assumed that loss of encoded data of the haptic signal occurs due to interference on the transmission path, etc. When data loss occurs, the transmitting device retransmits the data, and a delay in the time until the receiving device completes receiving the data may occur. In other words, as the amount of data to be retransmitted increases, the time required to retransmit the data corresponding to the data loss increases, and as a result, the time until the transmission and reception of the haptic signal is normally completed may be further delayed.
[0077] In this way, if there is a delay in the time until the transmission of the haptic signal is completed, the reproducibility of the haptic sensation may be reduced. Specifically, a situation may occur in which the haptic stimulation is not provided to the user at the appropriate time, and the haptic stimulation is not synchronized with content related to other senses, such as video and sound.
[0078] Next, a specific application of a wireless communication scheme will be described taking into consideration the above-mentioned circumstances. Since a device that presents haptic sensation is installed so as to come into contact with a user, it is generally desirable to communicate with other devices wirelessly in terms of device weight. However, when using a broadband wireless method such as Wi-Fi (registered trademark), the battery of the device becomes large in terms of power consumption, and this is expected to reduce user convenience. In addition, when using Wi-Fi (registered trademark), a processing time is generally required for the procedure from a signal transmission request of a transmitting device to reception processing of a receiving device, so a larger delay may occur compared to other wireless methods.
[0079] On the other hand, short-range wireless systems such as Bluetooth (registered trademark) are capable of communication with lower power consumption and lower latency than other wireless systems, and are therefore considered suitable for transmitting haptic signals. However, short-range wireless systems have a smaller capacity for transmitting data at one time than other wireless systems. For example, in transmitting content that provides a user with a haptic stimulus while synchronizing with video, sound, etc., it is conceivable that the communication capacity allocated to transmitting haptic signals may be insufficient.
[0080] Furthermore, in a service that streams video and audio over the Internet, when haptic sensations are also conveyed to the user, it is conceivable that the communication capacity allocated to the transmission of haptic signals may become insufficient due to, for example, Quality of Service (QoS) functions that respond to network line conditions.
[0081] In this embodiment, in consideration of the above circumstances, the objective is to achieve both low latency of haptic signals and suppression of degradation of haptic reproducibility by reducing the amount of transmitted data without impairing haptic reproducibility as much as possible.
[0082] [6-2. Encoding method] FIG. 7 is a functional block diagram showing the functions of the encoding unit 24. As shown in FIG. As shown in the figure, the encoding unit 24 includes a signal input unit 24a, a band dividing unit 24b, a low-frequency signal encoding unit 24c, a tactile intensity calculation unit 24d, a multiplexing unit 24e, and a signal output unit 24f.
[0083] A predetermined number of channels of haptic signals are input to the signal input unit 24a, each with a fixed number of samples. In the following explanation, the same processing is performed for each channel. The input haptic signals are assumed to have a bandwidth of 2 kHz and a sampling frequency of 4 kHz.
[0084] The signal input unit 24a extracts the input haptic signal into appropriate processing block units (for example, units of the number of samples corresponding to a time period of 5 ms). Hereinafter, this processing block unit will be referred to as a “frame.” In each process after the signal input unit 24a, processing is performed in frame units.
[0085] The band splitting unit 24b performs band splitting processing on the input signal from the signal input unit 24a. Specifically, the input signal is split into two signals, a low frequency signal and a high frequency signal. The split frequencies in this band splitting processing will be described later.
[0086] The low-frequency signal encoding unit 24c receives the low-frequency signal obtained by the band division process by the band division unit 24b, and performs encoding process on the low-frequency signal by a predetermined encoding method. Here, as the coding method for the low-frequency signal, various methods generally used for coding audio signals, which are one-dimensional signals like haptic signals, can be adopted. For example, MP3 (MPEG-1 Audio Layer-III), AAC (Advanced Audio Coding), or FLAC (Free Lossless Audio Codec), which is a lossless coding method, can be adopted. In addition, ADPCM (Adaptive Differential Pulse Code Modulation) can also be adopted in consideration of computational resources. As a coding method for such low frequency signals, a coding method that maintains the signal waveform as much as possible is adopted.
[0087] The haptic intensity calculation unit 24d receives the high-frequency signal obtained by the band division unit 24b and calculates the haptic intensity based on the high-frequency signal. The tactile intensity will be described later.
[0088] The multiplexing unit 24e receives the coded data of the low-frequency signal obtained by the low-frequency signal coding unit 24c (hereinafter referred to as "low-frequency coded data") and the haptic intensity calculated by the haptic intensity calculation unit 24d, and codes the low-frequency coded data, the haptic intensity, and other information as a bit string in accordance with the coded data format shown in Fig. 11, which will be described later. Through this coding, haptic coded data Dc is obtained.
[0089] The signal output unit 24f outputs the haptic encoded data Dc obtained by the multiplexing unit 24e.
[0090] The band dividing section 24b will now be described in detail. FIG. 8 is a diagram illustrating the tactile sensitivity characteristics of each sensory receptor of the human body. It has been reported that there are four types of sensory receptors that sense touch in the human body, as shown in Figure 8. Among them, Meissner's corpuscles are said to be receptors that detect speed, and are also sensitive to changes in frequency. For this reason, the frequency components from a few Hz to about 100 Hz, where Meissner's corpuscles are most easily stimulated, are coded so as to maintain the waveform as much as possible. Therefore, the band division unit 24b divides the band of the tactile signal into a low-frequency band below 100 Hz and a high-frequency band above 100 Hz. The low frequency band and the high frequency band may be divided into arbitrary frequencies depending on the circumstances, as will be described later in detail.
[0091] Next, the tactile intensity calculator 24d will be described. Regarding human tactile sensation, the intensity felt varies depending on the frequency, as can be seen from the vibration detection threshold shown in Figure 6. A tactile intensity model that quantitatively models this has been reported. Here, tactile intensity means the strength of the tactile sensation felt by the human body when a tactile stimulus of a certain magnitude (amplitude) is applied.
[0092] Fig. 9 is an explanatory diagram of tactile intensity, Fig. 9A is a grab of T(f) and Fig. 9B is a graph of a(f), where T(f) means the vibration detection threshold T (the vibration detection threshold explained in Fig. 6) for frequency f, and a(f) means the coefficient a for frequency f.
[0093] The tactile intensity I(f) at frequency f is calculated based on T(f) and a(f) and the amplitude A(f) of the tactile signal at frequency f using the following Equation 2.
number
[0094] If the tactile intensity I is the same signal, the vibration intensity is felt equally even if the frequency is different. Furthermore, among the tactile receptors shown in Figure 8, the Pacinian corpuscles are said to be less sensitive to changes in vibration frequency compared to their reactions to changes in vibration intensity. Therefore, in the perception of high frequencies above 100 Hz, at which the Pacinian corpuscles are most easily stimulated, the same perception can be obtained by tracking the changes in tactile intensity I. Therefore, for high-frequency signals of 100 Hz or higher, the waveform itself is not recorded, but the value of the tactile intensity I(f) for each frequency is recorded. The lower limit frequency for calculating the haptic intensity may be determined in accordance with the above-mentioned band division.
[0095] When calculating and recording the tactile intensity I for one frame, the tactile intensity I of the high-frequency signal in the tactile signal for one frame is calculated as a definite integral of the frequency f as shown in the following [Equation 3].
number
[0096] If the above is considered as digital signal processing, first, the input signal is subjected to a discrete Fourier transform to obtain a spectrum, and then the tactile intensity I(Fbin) for each frequency bin Fbin is calculated using [Equation 4]. Here, AS(Fbin) is the amplitude spectrum value for Fbin, and fFbin is the center frequency of Fbin.
number
[0097] Then, I(Fbin) is added from the minimum value FbinMin to the maximum value FbinMax of the frequency bins within the tactile intensity calculation range according to the following [Equation 5] to obtain the tactile intensity I of the high frequency signal.
number
[0098] In the above example, a representative tactile intensity is calculated for all frequencies from 100 Hz to 2 kHz. However, if the main frequencies in the input signal that are important for reproducing the tactile sensation are known, then the tactile intensity I can be calculated by targeting only the components of those main frequencies. The term "principal frequency" used herein refers to a frequency at which a principal spectrum is obtained. The principal spectrum refers to a spectrum excluding at least noise spectrum.
[0099] An example of determining the main frequency by frequency analysis will be described. First, the input tactile signal is subjected to a discrete Fourier transform to obtain a spectrum, and then the main frequency is determined by examining the peak of the amplitude spectrum above 100 Hz. For example, if the Fourier transform shows that there are peaks at frequency components of 500 Hz, 800 Hz, and 1000 Hz as shown in FIG. 10, then these frequencies, 500 Hz, 800 Hz, and 1000 Hz, are identified as the main frequencies. Here, the peak detection can be performed, for example, by determining whether the amplitude spectrum is equal to or greater than a predetermined threshold value. Note that the method of peak detection is not limited to this, and various known methods can be used, for example.
[0100] When the dominant frequency is identified, the tactile intensity I of the high-frequency signal is calculated based on the amplitude A of the dominant frequency. For example, when the dominant frequency is identified as shown in Fig. 10, the tactile intensity I is calculated by the following [Equation 6].
number
[0101] If the dominant frequency is not identified, the tactile intensity I of the high-frequency signal is calculated using [Equation 5].
[0102] Fig. 11 is a diagram showing an example of the data format of the haptic encoded data Dc. Specifically, Fig. 11 shows an example of the data format of one frame of the haptic encoded data Dc. First, the sync word is an identification code for indicating the beginning of a frame, and stores a pattern that is preferably not contained in other data. As the sampling frequency ID, a pattern ID of the sampling frequency of the haptic signal is recorded. The number of channels is recorded as the total number of channels of the tactile signal. The frame size indicates the size of one frame of haptic encoded data Dc in bytes.
[0103] The first channel data stores encoded data related to a haptic signal of the first channel. The first channel data stores a division frequency, low-frequency encoded data, and haptic intensity information. The divided frequency is a divided frequency by the band dividing unit 24b, and for example, a numerical value indicating the frequency (Hz) is stored. The low-frequency encoded data is encoded data of a low-frequency signal encoded by the low-frequency signal encoding unit 24c.
[0104] The tactile intensity information is shown in detail in FIG. As shown in the figure, a flag storage area is provided for the haptic intensity information. For example, when a main frequency is identified based on the peak detection described above, i.e., when the frequencies for which the tactile intensity is to be calculated are limited, a "1" is stored in the flag, and the area following the flag stores a pair of information including the identifiers of the frequencies for which the tactile intensity is to be calculated (shown in the figure as the identifiers of frequency A and frequency B) and the tactile intensity I for that frequency. On the other hand, when the main frequency is not specified and the specific frequency to be used for calculating the haptic intensity is not limited, the flag is stored as "0", and the subsequent area stores the haptic intensity I for each frequency (each frequency bin) in the high frequency area (from the division frequency to the Nyquist frequency: in this example, 100 Hz to 2 kHz). Note that for the haptic intensity I when the flag is 0, a value for each frequency may be stored, or the total value for each frequency calculated as in [Equation 5] above may be stored.
[0105] The second channel data is stored as encoded data relating to a second channel haptic signal in the same format as the first channel data. If there are other channels, they are added and information is stored in the same manner.
[0106] An example of a processing procedure for implementing the encoding method according to the embodiment described above will be described with reference to the flowchart of FIG. Note that, although an example is given here in which the encoding unit 24 executes the processing for realizing the encoding method of the embodiment as software processing, some or all of the processing described below can also be realized by hardware. The process shown in FIG. 13 is a process for generating one frame of encoded haptic data Dc, and is repeatedly executed for each frame.
[0107] First, in step S101, the encoding unit 24 inputs a signal cut out of one channel of haptic signals to be encoded, with the number of samples for one frame. In step S102 following step S101, the encoding unit 24 performs band division processing on the input signal using division frequencies to obtain a low frequency signal and a high frequency signal.
[0108] In step S103 following step S102, the encoding unit 24 encodes the low-frequency signal obtained by the band splitting process in step S102 using a predetermined encoding method, such as the aforementioned MP3, AAC, FLAC, ADPCM, etc., to generate low-frequency encoded data.
[0109] In step 104 following step S103, the encoding unit 24 determines whether the haptic intensity range is the entire high frequency band. That is, based on the result of analyzing the high frequency signal obtained by the band division process in step 102, it determines whether the haptic intensity calculation target is the entire band of the high frequency signal, specifically, whether the above-mentioned main frequency has not been identified.
[0110] In step 104, if, for example, the dominant frequency is not identified and a positive result is obtained that the haptic intensity range is the entire high frequency band, the encoding unit 24 proceeds to step 105 to calculate the haptic intensity of the entire high frequency signal band (see [Equation 5] above), and then proceeds to step S107.
[0111] On the other hand, if a negative result is obtained in step 104, for example, that the dominant frequency is identified and the haptic intensity range is not the entire high frequency band, the encoding unit 24 proceeds to step 106 to calculate the haptic intensity of the dominant frequency (see [Equation 6] above), and proceeds to step S107. Here, if the dominant frequency is identified, the encoding unit 24 stores an identifier of the identified dominant frequency.
[0112] In step S107, the encoding unit 24 stores the flag of the current channel (the channel currently being processed). Specifically, if the encoding unit 24 obtains a positive result in step S104 that the haptic intensity range is the entire high frequency band, the encoding unit 24 stores "0" as the flag of the current channel, and otherwise stores "1" as the flag of the current channel.
[0113] In step S108 following step S107, the encoding unit 24 determines whether or not the processing for all channels has been completed, that is, whether or not the processing from step S101 to S107 has been completed for all channels of the haptic signal. If the processing for all channels has not been completed, the encoding unit 24 returns to step S101, whereby the same processing for one frame is executed for the next channel.
[0114] On the other hand, if the processing for all channels has been completed, the encoding unit 24 executes a bit stream generation process in step S109, and ends the series of processes shown in FIG. In the bit stream generation process in step S109, multiplexing is performed in accordance with the encoded data format described with reference to FIGS. 11 and 12 to generate a bit stream for the haptic encoded data Dc.
[0115] Although the above describes an example of identifying the main frequency by peak detection, the main frequency can also be identified using a DNN (Deep Neural Network).
[0116] A specific example will be described with reference to FIG. Figure 14 shows a DNN model, in which supervised learning is performed in advance using a training data set. As the training data set, a training input data set is used as an input signal for learning. The learning input dataset is the amplitude spectrum of the haptic signal, and the input layers I1 to In of the DNN model are the center frequencies of each frequency bin, and the input is the amplitude spectrum value of each frequency bin. The output layers A1 to An of the DNN model are combinations of frequencies that are key to tactile reproduction, and the training teacher dataset is a teacher dataset corresponding to each data of the training input dataset, and is a combination of frequencies that are key to tactile reproduction that has been labeled in advance. When training a DNN model, the weight coefficients of each edge of the DNN model are updated based on the probability (likelihood) of the output layers A1 to An when training input data is input and the error backpropagation of the training teacher data.
[0117] When using the DNN model trained in the above manner to identify the main frequencies from an actual tactile signal, the input tactile signal (high-frequency signal) is converted into a spectrum by discrete Fourier transform, and the amplitude spectrum is then input to the trained DNN model, and the output (frequency combination pattern) with the highest likelihood from among the output layers A1 to An is adopted.
[0118] [6-3. Decryption method] FIG. 15 is a functional block diagram showing the functions of the decoding unit 34. As shown in FIG. As shown in the figure, the decoding unit 34 has an encoded data input unit 34a, a demultiplexing unit 34b, a low-frequency signal decoding unit 34c, a haptic intensity acquiring unit 34d, a high-frequency signal generating unit 34e, a band synthesizing unit 34f, and a signal output unit 34g.
[0119] The encoded data input unit 34a receives one frame of encoded haptic data Dc. The beginning of one frame can be detected from the sync word described above.
[0120] The demultiplexer 34b performs demultiplexing processing in accordance with the encoded data format shown in FIGS. 11 and 12, and acquires various pieces of information stored in the haptic encoded data Dc.
[0121] The low-frequency signal decoding unit 34c decodes the low-frequency signal code data acquired by the demultiplexing unit 34b to obtain a low-frequency signal.
[0122] The haptic intensity acquisition unit 34d performs a process of determining the value of a flag based on the haptic intensity information acquired by the demultiplexing unit 34b, and a process of acquiring a haptic intensity according to the determined flag value.
[0123] The high-frequency signal generating unit 34e generates a high-frequency signal based on the haptic intensity acquired by the haptic intensity acquiring unit 34d. The method of generating the high-frequency signal will be described later.
[0124] The band synthesis unit 34f synthesizes the low-frequency signal obtained by the low-frequency signal decoding unit 34c and the high-frequency signal obtained by the high-frequency signal generation unit 34e. The signal output unit 34g outputs the haptic signal obtained by the synthesis in the band synthesis unit 34f.
[0125] The process of the high frequency signal generating unit 34e will be described. The processing of the high frequency signal generating unit 34e differs depending on the value of the flag. When the flag=0, the haptic intensity for the entire band of the high frequency signal is obtained, and a periodic signal with an arbitrary frequency fx within the high frequency band is generated as the high frequency signal based on the haptic intensity for the entire band. Here, a periodic signal refers to a signal (time signal) whose amplitude changes at a predetermined period, and may be, for example, a sine wave signal. When the flag=0, the high-frequency signal generating unit 34e generates a high-frequency signal based on the haptic intensity I of the high-frequency signal, that is, the sum of the haptic intensities calculated for each frequency in the high-frequency band. Specifically, in this case, the amplitude A(fx) of the high-frequency signal S1(t) can be calculated from the obtained tactile intensity I and an arbitrary frequency fx based on [Equation 2], and can be found by the following [Equation 7].
number
[0126] In this way, when the flag = 0, the high-frequency signal generating unit 34e converts the tactile intensity into a signal amplitude and generates a periodic signal having the converted signal amplitude and a signal frequency fx within the high-frequency band as a high-frequency signal.
[0127] Here, when the flag=0, the arbitrary frequency fx may be uniquely determined in advance within the decoding unit . For example, it is considered that the frequency fx is set to a frequency that approximately matches the resonance frequency of the tactile presentation device 6. Approximately matching here is a concept that includes being within a range in which the numerical values are considered to be the same, and means, for example, that in the frequency characteristic curve of the tactile presentation device 6, belonging to a frequency range from the front end frequency to the rear end frequency of the peak portion centered on the resonance frequency. By setting the frequency fx to a frequency that approximately coincides with the resonance frequency of the tactile presentation device 6, the efficiency of driving the tactile presentation device 6 can be improved, and power consumption of the tactile presentation device 6 can be reduced.
[0128] The frequency fx may be set based on the tactile sensitivity characteristics of the human body. Specifically, the frequency fx may be set to a frequency that provides a tactile sensitivity equal to or greater than a predetermined sensitivity. This makes it possible to set the frequency fx to a frequency at which the tactile sensitivity of the human body is high, thereby improving the efficiency of driving the tactile presentation device 6. Therefore, the power consumption of the tactile presentation device 6 can be reduced.
[0129] The frequency fx may be set based on the hearing sensitivity characteristics of the human body. Specifically, the frequency fx may be set at a frequency at which the hearing sensitivity is equal to or lower than a predetermined level. This makes it possible to set the frequency fx to a frequency at which the human body has low hearing sensitivity, thereby reducing noise generated by driving the tactile presentation device 6. This makes it possible to enhance the user's sense of immersion in the tactile content.
[0130] The process when the flag is 1 will be described. When the flag=1, the haptic intensity acquisition unit 34d acquires the main frequency and the haptic intensity for the main frequency. In this case, the high-frequency signal generator 34e generates a high-frequency signal based on the tactile intensity for the main frequency. Specifically, for example, if the main frequencies are frequency A and frequency B, the high-frequency signal S2 is calculated based on frequency A (fa) and the tactile intensity (I(fa)) for frequency A, and frequency B (fb) and the tactile intensity (I(fb)) for frequency B, using the following Equation 8.
number
[0131] An example of a processing procedure for implementing the decoding method according to the embodiment described above will be described with reference to the flowchart of FIG. As in the case of encoding, an example is given here in which the decoding unit 34 executes the processing for realizing the decoding method of the embodiment as software processing, but some or all of the processing described below can also be realized by hardware. The process shown in FIG. 16 is a process for obtaining a haptic signal by decoding one frame's worth of bit stream (haptic encoded data Dc), and is repeatedly executed for each frame.
[0132] First, in step S201, the decoding unit 34 analyzes one frame of haptic encoded data Dc according to the encoded data format. Through this analysis process, the sampling frequency is identified from the sampling frequency ID, and the number of channels, frame size, and each channel data are obtained.
[0133] In step S202 following step S201, the decoding unit 34 decodes the low-frequency encoded data contained in the target channel data to obtain a low-frequency signal.
[0134] In step S203 following step S202, the decoding unit 34 determines whether the value of the flag included in the target channel data is 0 or not. If the flag=0, the decoding unit 34 proceeds to step S204 to obtain the haptic intensity of the entire high frequency band contained in the target channel data, and proceeds to step S206.
[0135] On the other hand, if the flag is not 0, the decoding unit 34 proceeds to step S205 to obtain the frequency identifier for the dominant frequency contained in the target channel data and the haptic intensity for the dominant frequency, and then proceeds to step S206.
[0136] In step S206, the decoding unit 34 performs high-frequency signal generation processing. That is, if the flag is 0, a high-frequency signal is generated according to the above [Equation 7] based on the haptic intensity of the entire high-frequency band acquired in step S204. On the other hand, if the flag is 1, a high-frequency signal is generated according to the above [Equation 8] based on the frequency identifier of the main frequency and the haptic intensity acquired in step S205.
[0137] In step S207 following step S206, the decoding unit 34 performs band synthesis processing, that is, processing for synthesizing the low-frequency signal obtained in step S202 and the high-frequency signal obtained in step S206.
[0138] Next, in step S208, the decoding unit 34 determines whether the processing for all channels has been completed, and if the processing for all channels has not been completed, the process returns to step S202. As a result, the processing from step S202 to S207 is executed for the next channel. On the other hand, if the processing for all channels has been completed, the decoding unit 34 ends the series of processing shown in FIG.
[0139] Here, in the above, an example has been given in which a predetermined frequency is set as the frequency fx when the flag is 0, but the frequency fx can also be set based on an operation. For example, the decoding unit 34 can set the frequency fx based on an operation input via the operation unit 38 shown in FIG.
[0140] FIG. 17 shows an example of a GUI (Graphical User Interface) related to setting the frequency fx. FIG. 17 shows an example of a GUI in which a plurality of frequencies can be set as the frequency fx, rather than a single frequency. In this case, to accept the setting operation of frequency fx, a setting screen as shown in the figure is displayed on screen 39a of display unit 39. This setting screen provides a frequency setting area for each channel of the haptic signal. In the frequency setting area of each channel, a frequency display window 50 (in the figure, display windows 50a, 50b, and 50c are shown for frequency A, frequency B, and frequency C, respectively) and a slider 51 for specifying the frequency value (in the figure, sliders 51a, 51b, and 51c are shown for frequency A, frequency B, and frequency C, respectively) are provided for multiple frequencies to be set and accepted. When the user changes the position of any of the sliders 51, the value in the frequency display window 50 corresponding to that slider 51 changes. This allows the user to specify an arbitrary frequency by manipulating the slider 50 while checking the value in the frequency display window 50.
[0141] Here, when a plurality of frequencies fx are set, the decoding unit 34 generates, for example, a periodic signal for each of the frequencies and generates a high-frequency signal by synthesizing the generated periodic signals. In this case, it is assumed that the haptic intensity of each frequency in the high frequency band is stored as haptic intensity information when the flag in the haptic encoded data Dc is 0. Then, in the process of generating the high frequency signal, the decoding unit 34 performs a calculation similar to the above [Equation 8] based on each set frequency fx and each haptic intensity at that frequency fx to generate the high frequency signal.
[0142] Although the above describes an example in which the frequency fx of the periodic signal is set based on an operation input via the operation unit 38 provided in the decoding device 3, the frequency fx can also be set based on an operation input via an operation unit provided in a device other than the decoding device 3. For example, in a case in which an operation unit is provided in the playback device 4, it is also possible to set the frequency of the periodic signal in response to an operation input to the playback device 4.
[0143] [6-4. Control of division frequency according to communication stability] In the above description, an example has been given in which the division frequency between the low frequency band and the high frequency band is fixed in the band division process during encoding, but the division frequency can also be variable. It is conceivable to increase the division frequency when it is desired to reproduce the low-frequency signal with higher quality, and to decrease the amount of haptic encoded data Dc when it is desired to further reduce the data volume. For example, when the radio wave conditions deteriorate during communication and communication stability is reduced, it is conceivable to dynamically change the division frequency as QoS in order to reduce the signal quality and increase communication stability.
[0144] Here, communication stability refers to an index of the stability of communication, such as communication speed (communication bit rate), latency value such as Ping value (round-trip time from the communication partner until a response packet returns), radio wave strength in wireless communication, and packet loss rate in packet communication.
[0145] An example of changing the division frequency depending on the communication stability will be described with reference to FIG. In Figure 18, the transmission buffer refers to the transmission buffer of the communication unit 27 in the encoding device 2, and when the haptic encoded data Dc obtained by the encoding unit 24 is transmitted to an external device, the haptic encoded data Dc is buffered sequentially on a packet basis. In this example, the communication stability is determined based on the number of free slots in the transmission buffer. Specifically, when a large amount of packet data is stored in the transmission buffer, it can be determined that the communication state has deteriorated and data retransmission requests have been received many times from the receiving device (the receiving side of the haptic encoded data Dc: for example, the playback device 4), and that transmission is not proceeding smoothly. In other words, it can be determined that the communication stability is low (deteriorating).
[0146] Fig. 18 shows division frequencies corresponding to each of the cases where communication stability is normal (Fig. 18A), good (Fig. 18B), and deteriorated (Fig. 18C). The low-frequency band determined by the division frequencies is a band in which the waveform is coded almost as is, and can therefore be called a quality-oriented coding range. Conversely, the high-frequency band is a band in which only the haptic intensity is used as a substitute for the waveform itself, and can therefore be called an efficiency-oriented coding range.
[0147] In other words, when communication stability is good, even if the bit rate is high, there is little possibility of data interruption, so the division frequency is set high (for example, 500 Hz) to improve quality by widening the quality-oriented coding range (low frequency band) as much as possible. Conversely, when communication stability deteriorates, the division frequency is set low (for example, 100 Hz) to widen the efficiency-oriented coding range (high frequency band) and lower the bit rate, with the emphasis on preventing data interruption.
[0148] When performing variable setting of the division frequency according to the communication stability as described above, in the band division process of step S102 shown in FIG. 13, the encoding unit 24 performs a process of determining the communication stability based on the number of free slots in the transmission buffer in the communication unit 27 (for example, a determination of one of three states: good / normal / worse), and a process of setting the division frequency according to the determination result.
[0149] <7. Summary of the embodiment> As described above, the decoding device (3) as an embodiment includes a first decoding unit (low-frequency signal decoding unit 34c) that decodes a first band signal, which is encoded data (haptic encoded data Dc) obtained by encoding a haptic signal, and which includes encoded data obtained by encoding a first band signal, which is a signal of a first frequency band in the haptic signal, and haptic intensity information, which is information indicating haptic intensity (haptic intensity) for a second frequency band different from the first frequency band in the haptic signal. The first band signal is a signal of a first frequency band in the haptic signal, and the encoded data includes haptic intensity information, which is information indicating haptic intensity for a second frequency band different from the first frequency band in the haptic signal. The first band signal is a signal of a second frequency band in the haptic signal, and the second band signal is a signal of a second frequency band in the haptic signal. The second band signal is a signal of a second frequency band in the haptic signal, based on the haptic intensity information in the encoded data. The second band signal is a signal of a second frequency band in the haptic signal ... This makes it possible to decode the haptic signal related to the second frequency band while suppressing deterioration of haptic reproducibility for the encoded data generated by the encoding device according to the embodiment. Therefore, it is possible to reduce the data amount of the haptic signal while suppressing the deterioration of the reproducibility of the haptic sensation.
[0150] In the decoding device according to the embodiment, the first frequency band is a band on the lower frequency side than the second frequency band. For tactile signals in a frequency band above a certain frequency, the human body easily perceives changes in tactile intensity but does not easily perceive changes in frequency. Therefore, with the above configuration, it is possible to realize information compression of tactile signals by utilizing the tactile sensitivity characteristics of the human body, thereby making it possible to reduce the data volume of tactile signals while suppressing a decrease in tactile reproducibility.
[0151] Furthermore, in the decoding device as an embodiment, haptic intensity information is calculated for each frequency in the second frequency band, and the second decoding unit decodes the second band signal based on the total value of haptic intensity indicated by the haptic intensity information calculated for each frequency (see steps S204 → S206 in Figure 16). This makes it possible to reproduce an appropriate tactile intensity for the second frequency band based on the tactile intensity of each frequency in the second band signal before encoding. Therefore, it is possible to suppress the deterioration of the reproducibility of the tactile sensation.
[0152] Furthermore, in the decoding device as an embodiment, haptic intensity information is calculated for one or more main frequencies within the second frequency band, and the second decoding unit decodes the second band signal based on the haptic intensity indicated by the haptic intensity information calculated for the main frequencies (see steps S205 → S206 in Figure 16). This makes it possible to reproduce an appropriate tactile intensity for the second frequency band based on the tactile intensity of the main frequency in the second band signal before encoding. Therefore, it is possible to suppress the deterioration of the reproducibility of the tactile sensation.
[0153] Furthermore, in the decoding device as an embodiment, the second decoding unit converts the haptic intensity indicated by the haptic intensity information into a signal amplitude, and generates a periodic signal having the converted signal amplitude and a signal frequency within the second frequency band as a decoded signal of the second band signal (see [Equation 7]). This makes it possible to decode the second band signal of the encoded data generated by the encoding device according to the present technology in a manner that suppresses degradation of tactile reproducibility. Therefore, it is possible to reduce the data amount of the haptic signal while suppressing the deterioration of the reproducibility of the haptic sensation.
[0154] Furthermore, in the decoding device according to the embodiment, the frequency of the periodic signal is set to a frequency that approximately matches the resonant frequency of the tactile presentation device. This improves the efficiency of driving the tactile presentation device. Therefore, the power consumption of the tactile presentation device can be reduced.
[0155] Furthermore, in the decoding device according to the embodiment, the frequency of the periodic signal is set based on the tactile sensitivity characteristics of the human body. This makes it possible to set the frequency of the periodic signal to a frequency at which the human body has high tactile sensitivity, thereby improving the efficiency of driving the tactile presentation device. Therefore, the power consumption of the tactile presentation device can be reduced.
[0156] In the decoding device according to the embodiment, the frequency of the periodic signal is set based on the hearing sensitivity characteristics of the human body. This makes it possible to set the frequency of the periodic signal to a frequency at which the human body has low hearing sensitivity, thereby reducing noise generated by driving the tactile presentation device. This can enhance the user's sense of immersion in the haptic content.
[0157] Furthermore, in the decoding device according to the embodiment, the second decoding unit sets the frequency of the periodic signal based on the operation (see FIG. 17). This allows the user to arbitrarily select the frequency of the periodic signal.
[0158] In addition, a decoding method as an embodiment is a decoding method for coded data obtained by coding a haptic signal, the coded data including coded data obtained by coding a first band signal, which is a signal in a first frequency band in the haptic signal, and haptic intensity information, which is information indicating haptic intensity for a second frequency band different from the first frequency band in the haptic signal, the coded data having the following steps: a first decoding procedure for decoding the first band signal; a second decoding procedure for decoding a second band signal, which is a signal in a second frequency band in the haptic signal, based on the haptic intensity information in the coded data; and a synthesis procedure for synthesizing the first band signal decoded by the first decoding procedure and the second band signal decoded by the second decoding procedure. With such a decoding method, it is possible to obtain the same functions and effects as those of the decoding device according to the above embodiment.
[0159] Moreover, the encoding device as an embodiment includes a band division unit (24b) that divides a haptic signal into a first band signal, which is a signal in a first frequency band, and a second band signal, which is a signal in a second frequency band different from the first frequency band, a first encoding unit (low-frequency signal encoding unit 24c) that encodes the first band signal, a haptic intensity calculation unit (24d) that calculates haptic intensity information indicating the haptic intensity for the second frequency band based on the second band signal, and an encoded data generation unit (multiplexing unit 24e) that generates encoded data (haptic encoded data Dc) including data obtained by encoding the first band signal and the haptic intensity information. This makes it possible to efficiently compress information in tactile signals by utilizing the tactile sensitivity characteristics of the human body, whereby in a certain frequency band, both changes in tactile intensity and changes in frequency are easily perceived, but in another frequency band, changes in tactile intensity are easily perceived but changes in frequency are not. Therefore, it is possible to reduce the data amount of the haptic signal while suppressing the deterioration of the reproducibility of the haptic sensation.
[0160] In the decoding device according to the embodiment, the first frequency band is a band on the lower frequency side than the second frequency band. For tactile signals in a frequency band above a certain frequency, the human body easily perceives changes in tactile intensity but does not easily perceive changes in frequency. Therefore, with the above configuration, it is possible to realize information compression of tactile signals by utilizing the tactile sensitivity characteristics of the human body, thereby making it possible to reduce the data volume of tactile signals while suppressing a decrease in tactile reproducibility.
[0161] Furthermore, in the decoding device according to the embodiment, the division frequency between the first frequency band and the second frequency band is variable. By making the division frequency variable, it is possible to adjust the balance between the effect of data reduction by encoding and the effect of suppressing the deterioration of tactile reproducibility. Therefore, it is possible to optimize the balance between the data reduction effect and the effect of suppressing the decline in tactile reproducibility according to various circumstances, such as when, for some reason, one wants to prioritize the data reduction effect over tactile reproducibility, or conversely, when one wants to prioritize tactile reproducibility over the data reduction effect.
[0162] Furthermore, in the decoding device according to the embodiment, the band dividing section changes the divided frequency depending on the stability of communication with an external device (see FIG. 18). This makes it possible to change the division frequency in a direction that increases the data reduction effect when communication with an external device to which the encoded data is transmitted is unstable, i.e., by lowering the bit rate of the communication data, thereby preventing communication interruptions; conversely, when communication is stable, the division frequency can be changed in a direction that increases tactile reproducibility. Therefore, the balance between the data reduction effect and the effect of suppressing the deterioration of haptic reproducibility can be appropriately adjusted according to the communication stability.
[0163] Furthermore, in the decoding device according to the embodiment, the band dividing section sets the division frequency lower when communication stability is low than when communication stability is high. As a result, when communication is unstable, the division frequency is changed in the direction of increasing the data reduction effect, i.e., lowering the bit rate of the communication data, and conversely, when communication is stable, the division frequency is changed in the direction of increasing tactile reproducibility. Therefore, it is possible to appropriately adjust the balance between preventing communication interruptions and preventing deterioration of tactile reproducibility.
[0164] Furthermore, in the decoding device as the embodiment, the haptic intensity calculation unit calculates the haptic intensity information for each frequency in the second frequency band (see step S105 in FIG. 13). This makes it possible to reproduce an appropriate tactile intensity for the second frequency band based on the tactile intensity of each frequency in the second band signal before encoding. Therefore, it is possible to suppress the deterioration of the reproducibility of the tactile sensation.
[0165] Furthermore, in the decoding device as the embodiment, the haptic intensity calculation unit calculates the haptic intensity information only for one or more main frequencies in the second frequency band (see step S106 in FIG. 13). This makes it possible to reproduce an appropriate tactile intensity for the second frequency band based on the tactile intensity of the main frequency in the second band signal before encoding. Therefore, it is possible to suppress the deterioration of the reproducibility of the tactile sensation.
[0166] In addition, an encoding method as an embodiment is an encoding method having a band splitting step of splitting a haptic signal into a first band signal, which is a signal in a first frequency band, and a second band signal, which is a signal in a second frequency band different from the first frequency band, a first encoding step of encoding the first band signal, a haptic intensity calculation step of calculating haptic intensity information indicating haptic intensity for the second frequency band based on the second band signal, and an encoded data generation step of generating encoded data including data encoded from the first band signal and the haptic intensity information.
[0167] The encoding method according to this embodiment can also provide the same functions and effects as the encoding device according to the above embodiment.
[0168] The functions of the encoding unit (24) and decoding unit (34) described above can be realized as software processing by a CPU, etc. The software processing is executed based on a program.
[0169] A first program as an embodiment is a program that causes an information processing device to realize a first decoding function for decoding a first band signal, the encoded data being a haptic signal, the encoded data including encoded data of a first band signal, which is a signal of a first frequency band in the haptic signal, and haptic intensity information, which is information indicating haptic intensity for a second frequency band different from the first frequency band in the haptic signal, based on the haptic intensity information in the encoded data, a second decoding function for decoding a second band signal, which is a signal of a second frequency band in the haptic signal, based on the haptic intensity information in the encoded data, and a synthesis function for synthesizing the first band signal decoded by the first decoding function and the second band signal decoded by the second decoding function. The first program described above can realize the decoding device according to the embodiment.
[0170] In addition, a second program as an embodiment is a program that causes an information processing device to realize a band splitting function that splits a haptic signal into a first band signal that is a signal in a first frequency band and a second band signal that is a signal in a second frequency band different from the first frequency band, a first encoding function that encodes the first band signal, a haptic intensity calculation function that calculates haptic intensity information indicating haptic intensity for the second frequency band based on the second band signal, and an encoded data generation function that generates encoded data including data encoded with the first band signal and the haptic intensity information. Such a second program can realize the encoding device according to the above-mentioned embodiment.
[0171] The above-mentioned first and second programs can be recorded in advance on a recording medium built into a device such as a computer device, or in a ROM or the like in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magnet Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Furthermore, the first and second programs can be installed into a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as the Internet or a LAN (Local Area Network).
[0172] In addition, the first and second programs are suitable for providing a wide range of the decoding device and encoding device of the embodiment. For example, by downloading the programs to a personal computer, a portable information processing device, a mobile phone, a game device, an AV (Audio Visual) device, etc., the personal computer can function as the decoding device or encoding device of the present technology.
[0173] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.
[0174] <8. This Technology> The present technology can also be configured as follows. (1) a first decoding unit that decodes a first band signal, the coded data being a haptic signal, the coded data including data obtained by coding a first band signal that is a signal of a first frequency band in the haptic signal and haptic intensity information that is information indicating a haptic intensity for a second frequency band in the haptic signal that is different from the first frequency band; a second decoding unit that decodes a second band signal, which is a signal in the second frequency band of the haptic signal, based on the haptic intensity information in the encoded data; a synthesis unit that synthesizes the first band signal decoded by the first decoding unit and the second band signal decoded by the second decoding unit. Decryption device. (2) The first frequency band is a band lower than the second frequency band. The decoding device according to (1) above. (3) the haptic intensity information is calculated for each frequency within the second frequency band; The second decoding unit is The second band signal is decoded based on a total value of the haptic intensity indicated by the haptic intensity information calculated for each of the frequencies. The decoding device according to (1) or (2). (4) the haptic intensity information is calculated for one or more dominant frequencies within the second frequency band; The second decoding unit is Decoding the second band signal based on a haptic intensity indicated by the haptic intensity information calculated for the principal frequency. A decoding device according to any one of (1) and (2). (5) The second decoding unit is converting the haptic intensity indicated by the haptic intensity information into a signal amplitude; A periodic signal having the converted signal amplitude and a signal frequency within the second frequency band is generated as a decoded signal of the second band signal. A decoding device according to any one of (1) to (3). (6) The frequency of the periodic signal is set to be substantially equal to the resonance frequency of the tactile presentation device. The decoding device according to (5) above. (7) The frequency of the periodic signal is set based on the tactile sensitivity characteristics of the human body. The decoding device according to (5) or (6). (8) The frequency of the periodic signal is set based on the hearing sensitivity characteristics of the human body. A decoding device according to any one of (5) to (7). (9) The second decoding unit is Setting the frequency of the periodic signal based on the operation A decoding device according to any one of (5) to (8). (10) a first decoding step of decoding a first band signal, the encoded data being a haptic signal, the encoded data including data obtained by encoding a first band signal that is a signal of a first frequency band in the haptic signal, and haptic intensity information that is information indicating a haptic intensity for a second frequency band in the haptic signal that is different from the first frequency band; a second decoding step of decoding a second band signal, which is a signal in the second frequency band of the haptic signal, based on the haptic intensity information in the encoded data; a synthesis procedure for synthesizing the first band signal decoded by the first decoding procedure and the second band signal decoded by the second decoding procedure. Decryption method. (11) a first decoding function for decoding a first band signal, the encoded data being a haptic signal, the encoded data including data obtained by encoding a first band signal, which is a signal of a first frequency band in the haptic signal, and haptic intensity information, which is information indicating a haptic intensity for a second frequency band in the haptic signal that is different from the first frequency band; a second decoding function that decodes a second band signal, which is a signal in the second frequency band of the haptic signal, based on the haptic intensity information in the encoded data; a synthesis function of synthesizing the first band signal decoded by the first decoding function and the second band signal decoded by the second decoding function in the information processing device. program. (12) a band splitting unit that splits a haptic signal into a first band signal that is a signal in a first frequency band and a second band signal that is a signal in a second frequency band different from the first frequency band; a first encoding unit that encodes the first band signal; a haptic intensity calculation unit that calculates haptic intensity information indicating a haptic intensity for the second frequency band based on the second band signal; an encoded data generation unit that generates encoded data including data obtained by encoding the first band signal and the haptic intensity information; Encoding device. (13) The first frequency band is a band lower than the second frequency band. The encoding device according to (12) above. (14) A division frequency between the first frequency band and the second frequency band is variable. The encoding device according to (12) or (13) above. (15) The band splitting unit includes: The division frequency is changed according to the stability of communication with the external device. The encoding device according to (14) above. (16) The band splitting unit includes: When the communication stability is low, the division frequency is set lower than when the communication stability is high. The encoding device according to (15) above. (17) The tactile intensity calculation unit Calculating the haptic intensity information for each frequency within the second frequency band. An encoding device according to any one of (12) to (16). (18) The tactile intensity calculation unit Calculating the tactile intensity information only for one or more dominant frequencies within the second frequency band. An encoding device according to any one of (12) to (16). (19) a band division step of dividing a haptic signal into a first band signal that is a signal in a first frequency band and a second band signal that is a signal in a second frequency band different from the first frequency band; a first encoding step for encoding the first band signal; a haptic intensity calculation step of calculating haptic intensity information indicating a haptic intensity for the second frequency band based on the second band signal; and generating encoded data including data obtained by encoding the first band signal and the haptic intensity information. Encoding method. (20) a band division function that divides a haptic signal into a first band signal that is a signal in a first frequency band and a second band signal that is a signal in a second frequency band different from the first frequency band; a first encoding function for encoding the first band signal; a haptic intensity calculation function that calculates haptic intensity information indicating a haptic intensity for the second frequency band based on the second frequency band signal; an encoded data generating function for generating encoded data including data obtained by encoding the first band signal and the haptic intensity information, program. [Explanation of symbols]
[0175] 1. Tactile reproduction system 2 Encoding device 3. Decryption Device 5. Tactile Sensor 6 Tactile presentation device Dc Tactile coding data 24 Encoding section 25 Control Unit 26 Memory section 27 Communications Department 34 Decoding section 35 Control section 38 Control section 39 Display section 41 Control section 42 Communications Department 24a Signal input section 24b Band division section 24c Low frequency signal encoder 24d Tactile intensity calculation unit 24e Multiplexer 34a Encoded data input section 34b Demultiplexer 34c Low-frequency signal decoding section 34d Tactile Intensity Acquisition Unit 34e High frequency signal generator 34f Band synthesis section
Claims
1. a first decoding unit that decodes a first band signal, the coded data being a haptic signal, the coded data including data obtained by coding a first band signal that is a signal of a first frequency band in the haptic signal and haptic intensity information that is information indicating a haptic intensity for a second frequency band in the haptic signal that is different from the first frequency band; a second decoding unit that decodes a second band signal, which is a signal in the second frequency band of the haptic signal, based on the haptic intensity information in the encoded data; a synthesis unit that synthesizes the first band signal decoded by the first decoding unit and the second band signal decoded by the second decoding unit. Decryption device.
2. The first frequency band is a band lower than the second frequency band. The decoding device according to claim 1 .
3. the haptic intensity information is calculated for each frequency within the second frequency band; The second decoding unit is The second band signal is decoded based on a total value of the haptic intensity indicated by the haptic intensity information calculated for each of the frequencies. The decoding device according to claim 1 .
4. the haptic intensity information is calculated for one or more dominant frequencies within the second frequency band; The second decoding unit is Decoding the second band signal based on a haptic intensity indicated by the haptic intensity information calculated for the principal frequency. The decoding device according to claim 1 .
5. The second decoding unit is converting the haptic intensity indicated by the haptic intensity information into a signal amplitude; A periodic signal having the converted signal amplitude and a signal frequency within the second frequency band is generated as a decoded signal of the second band signal. The decoding device according to claim 1 .
6. The frequency of the periodic signal is set to be substantially equal to the resonance frequency of the tactile presentation device. The decoding device according to claim 5.
7. The frequency of the periodic signal is set based on the tactile sensitivity characteristics of the human body. The decoding device according to claim 5.
8. The frequency of the periodic signal is set based on the hearing sensitivity characteristics of the human body. The decoding device according to claim 5.
9. The second decoding unit is Setting the frequency of the periodic signal based on the operation The decoding device according to claim 5.
10. a first decoding step of decoding a first band signal, the encoded data being a haptic signal, the encoded data including data obtained by encoding a first band signal that is a signal of a first frequency band in the haptic signal, and haptic intensity information that is information indicating a haptic intensity for a second frequency band in the haptic signal that is different from the first frequency band; a second decoding step of decoding a second band signal, which is a signal in the second frequency band of the haptic signal, based on the haptic intensity information in the encoded data; a synthesis procedure for synthesizing the first band signal decoded by the first decoding procedure and the second band signal decoded by the second decoding procedure. Decryption method.
11. a first decoding function for decoding a first band signal, the encoded data being a haptic signal, the encoded data including data obtained by encoding a first band signal, which is a signal of a first frequency band in the haptic signal, and haptic intensity information, which is information indicating a haptic intensity for a second frequency band in the haptic signal that is different from the first frequency band; a second decoding function that decodes a second band signal, which is a signal in the second frequency band of the haptic signal, based on the haptic intensity information in the encoded data; a synthesis function of synthesizing the first band signal decoded by the first decoding function and the second band signal decoded by the second decoding function in the information processing device. program.
12. a band splitting unit that splits a haptic signal into a first band signal that is a signal in a first frequency band and a second band signal that is a signal in a second frequency band different from the first frequency band; a first encoding unit that encodes the first band signal so that the first band signal can be decoded; a haptic intensity calculation unit that calculates haptic intensity information indicating a haptic intensity for the second frequency band based on the second band signal; an encoded data generation unit that generates encoded data including data obtained by encoding the first band signal and the haptic intensity information; Encoding device.
13. The first frequency band is a band lower than the second frequency band. Encoding device according to claim 12.
14. A division frequency between the first frequency band and the second frequency band is variable. Encoding device according to claim 12.
15. The band splitting unit includes: The division frequency is changed according to the stability of communication with the external device. Encoding device according to claim 14.
16. The band splitting unit includes: When the communication stability is low, the division frequency is set lower than when the communication stability is high. Encoding device according to claim 15.
17. The tactile intensity calculation unit Calculating the haptic intensity information for each frequency within the second frequency band. Encoding device according to claim 12.
18. The tactile intensity calculation unit Calculating the tactile intensity information only for one or more dominant frequencies within the second frequency band. Encoding device according to claim 12.
19. a band division step of dividing a haptic signal into a first band signal that is a signal in a first frequency band and a second band signal that is a signal in a second frequency band different from the first frequency band; a first encoding step for encoding the first band signal in a decodable manner; a haptic intensity calculation step of calculating haptic intensity information indicating a haptic intensity for the second frequency band based on the second band signal; and generating encoded data including data obtained by encoding the first band signal and the haptic intensity information. Encoding method.
20. a band division function that divides a haptic signal into a first band signal that is a signal in a first frequency band and a second band signal that is a signal in a second frequency band different from the first frequency band; a first encoding function for encoding the first band signal in a decodable manner; a haptic intensity calculation function that calculates haptic intensity information indicating a haptic intensity for the second frequency band based on the second frequency band signal; an encoded data generating function for generating encoded data including data obtained by encoding the first band signal and the haptic intensity information, program.
Citation Information
Patent Citations
Vibration signal creation program, vibration signal creation system, vibration signal creation device, vibration signal creation method and data output program
JP2016202486A
Decoding device, decoding method, and program
WO2020003727A1
Decoding device, decoding method, and program
WO2020031483A1
