Decoding device, decoding method, and decoding program
The decoding device obtains the intermediate representation signal and device characteristic information, and generates tactile signals adapted to human perception, solving the user experience and data efficiency problems caused by differences in device characteristics in the prior art, and achieving efficient tactile signal generation and transmission.
Patent Information
- Application Number
- CN202380029303.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-31
- Filing Date
- 2023-03-03
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to generate high-quality tactile signals independently of the output device, and the creator's workload increases due to differences in device characteristics, affecting user experience and data transmission efficiency.
A decoding device and method are designed to generate tactile signals adapted to human perception by obtaining intermediate representation signals and device characteristics information, and realize efficient data storage and transmission independently of output device characteristics.
It enables the generation of high-quality haptic signals without relying on output device characteristics, reducing the workload of the creator and improving data transmission and storage efficiency.
Smart Images

Figure CN118922803A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a decoding device, a decoding method, and a decoding program provided to implement a signal decoding process in haptic technology. Background Art
[0002] There is known a tactile presentation (haptics) technology that implements tactile presentation through vibration stimulation or the like so that a user can obtain a desired sensory effect.
[0003] For example, a technique for controlling the timing of a plurality of haptic signals to allow a user to simultaneously perceive two or more sensory effects is known (eg, Patent Document 1). Also known is a technique for appropriately encoding a plurality of haptic signals (eg, Patent Document 2).
[0004] Reference List Patent Literature Patent Document 1: WO 2019 / 138867 A Patent Document 2: JP 2019-219785 A Summary of the invention Technical issues According to known techniques, multiple haptic signals may be presented to a user, thereby making it possible to improve the user's haptic experience.
[0005] At the same time, tactile presentation is easily affected by the characteristics of the device that outputs the tactile signal or the vibrator (actuator) included in the device, etc. Therefore, depending on the output device, it may be difficult to reflect the intention of the creator of the tactile signal. In addition, in order to accurately reflect the creator's own intention, the creator of the tactile signal must create the signal in consideration of the characteristics of the device to which the signal will be output, thereby increasing the creator's workload.
[0006] In view of this, the present disclosure proposes a decoding device, a decoding method, and a decoding program that can generate a tactile signal independently of an output environment.
[0007] Solutions to the Problem According to an embodiment of the present disclosure, a decoding device includes: an acquisition unit that acquires an intermediate representation signal that records information related to a representation of tactile presentation, and characteristic information related to an output unit that implements tactile presentation based on the intermediate representation signal; and a generation unit that generates a tactile signal as a signal for controlling an output unit's output based on the intermediate representation signal acquired by the acquisition unit. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 is a diagram showing an overview of information processing according to an embodiment of the present disclosure.
[0009] Figure 2 is a diagram showing a configuration example of a conversion device according to the embodiment.
[0010] Figure 3 is a conceptual diagram illustrating encoding into an intermediate representation signal according to the embodiment.
[0011] Figure 4 is a diagram showing an example of an intermediate representation signal according to the described embodiment.
[0012] Figure 5 is a diagram (1) showing an example of audio source separation according to the embodiment.
[0013] Figure 6 is a diagram (2) showing an example of audio source separation according to the embodiment.
[0014] Figure 7 is a diagram (3) showing an example of audio source separation according to the embodiment.
[0015] Figure 8 is a diagram illustrating an attack in an intermediate representation signal according to the described embodiment.
[0016] Fig. 9 is a flowchart showing the procedure of the conversion process according to the embodiment.
[0017] Fig.10 is a diagram showing a configuration example of a decoding device according to the embodiment.
[0018] Fig.11 is a conceptual diagram illustrating decoding of an intermediate representation signal according to the embodiment.
[0019] Fig.12 is a diagram showing an example of an intermediate representation signal as a target of a decoding process according to the embodiment.
[0020] Fig.13 is a diagram (1) showing an example of a decoding process according to the embodiment.
[0021] Fig.14 is a diagram (2) showing an example of the decoding process according to the embodiment.
[0022] Fig.15 is a diagram (3) showing an example of the decoding process according to the embodiment.
[0023] Fig.16 is a diagram showing an example of the generation process according to the embodiment.
[0024] Fig.17is a diagram (1) showing an example of an adjustment process based on the characteristics of a tactile presentation device.
[0025] Fig.18 is a diagram (2) showing an example of adjustment processing based on the characteristics of the tactile presentation device.
[0026] Fig.19 is a diagram (3) showing an example of adjustment processing based on the characteristics of the tactile presentation device.
[0027] Fig. 20 is a diagram showing an extended example of the conversion process according to the embodiment.
[0028] Fig.21 is a diagram showing an example of an adjustment process according to a time change.
[0029] Fig. 22 is a diagram (1) showing an example of an adjustment process adapted to human perception.
[0030] Fig.23 is a diagram (2) showing an example of an adjustment process adapted to human perception.
[0031] Fig.24 is a diagram (3) showing an example of an adjustment process adapted to human perception.
[0032] Fig.25 is a diagram showing an example of adjustment processing related to signal superposition.
[0033] Fig.26 is a flowchart showing the procedure of a decoding process according to the embodiment.
[0034] Fig. 27 is a diagram showing the flow of a tactile presentation process according to a modification.
[0035] Fig.28 is a hardware configuration diagram showing an example of a computer that implements the function of the conversion device. DETAILED DESCRIPTION
[0036] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In each of the following embodiments, the same parts are denoted by the same reference numerals, and repeated description will be omitted.
[0037] The present disclosure will be described in the following order.
[0038] 1. Embodiment 1-1. Overview of information processing according to the embodiment 1-2. Configuration of conversion device according to the embodiment 1-3. Procedure of conversion processing according to the embodiment 1-4. Configuration of a decoding device according to an embodiment 1-5. Procedure of decoding process according to the embodiment 2. Modification of the Embodiment 2-1. Equipment Configuration 3. Other embodiments 4. Effects of the conversion device according to the present disclosure 5. Effects of the decoding device according to the present disclosure 6. Hardware Configuration (1. Example) (1-1. Overview of information processing according to the embodiment) First, refer to Figure 1 An overview of information processing according to the embodiment is described. Figure 1 is a diagram showing an overview of information processing according to the embodiment.
[0039] Figure 1 The information processing system 1 according to the embodiment is shown. The information processing system 1 includes a conversion device 100, a decoding device 200, and a tactile presentation device 10. The information processing system 1 is a system that controls a series of processes for realizing tactile presentation intended by a creator 20 on the tactile presentation device 10.
[0040] The information processing according to the embodiment is performed by Figure 1 . Specifically, the conversion device 100 converts any signal created by the creator 20 into an intermediate representation signal 24 for a characteristic representation signal independent of the tactile presentation device 10, etc. In addition, the decoding device 200 decodes the intermediate representation signal 24 into a signal to be output on various tactile presentation devices 10. That is, the conversion device 100 and the decoding device 200 play the roles of encoding function and decoding function in a series of tactile presentation processes, respectively. In the following description, the signal before undergoing the processing of the conversion device 100 may be referred to as a "conversion source signal". In addition, in the following description, the tactile signal represents a signal having a waveform expressing the vibration generated by the vibration of the output unit (vibrator (actuator)) of the tactile presentation device 10. The tactile signal may also be referred to as a command or parameter provided to the tactile presentation device 10 so as to implement vibration in the tactile presentation device 10.
[0041] Figure 1The conversion device 100 shown in the figure is an information processing device that converts conversion source signals of various modes into an intermediate representation signal 24. Examples of the conversion device 100 include, for example, a personal computer (PC), a server device, and a tablet terminal. The conversion source signal is any signal that is to be output as a tactile signal through the conversion process according to the embodiment. Examples of the conversion source signal include: sound data in which songs, environmental sounds, etc. are recorded; moving image data; and a tactile signal generated by a device other than the decoding device 200 and rendered for a specific type of tactile presentation device 10.
[0042] The decoding device 200 is an information processing device that generates a haptic signal based on the intermediate representation signal 24. Examples of the decoding device 200 include a PC, a server device, and a tablet terminal.
[0043] The tactile presentation device 10 is an information processing device having a function of vibrating an output unit based on a tactile signal. Examples of the tactile presentation device 10 include a game controller 10A, a headset 10B, a wristband type device 10C, and a vest (suit) type device 10D. The tactile presentation device 10 includes one or more output units, and vibrates the output unit based on the tactile signal, thereby implementing tactile presentation (giving stimulation) to the corresponding body part of the user. The output unit is an element that converts an electrical signal into vibration, and corresponds to devices such as a motor, a linear vibrator, and a piezoelectric element actuator, for example.
[0044] A user touching the tactile presentation device 10 can receive a tactile presentation corresponding to the flow of content such as video and audio displayed on the display while enjoying the content, thereby being able to enjoy the content with a higher sense of reality. Specifically, the user can enjoy a tactile presentation synchronized with the time lapse of the presentation of displayed content such as moving images or audio.
[0045] The creator 20 is a person who creates content or signals for tactile presentation. For example, the creator 20 creates content such as moving images or audio. Alternatively, the creator 20 creates a signal (tactile signal) dedicated to tactile presentation in order to enhance the user's sense of reality. For example, in the case of implementing tactile presentation by using the game controller 10A, the creator 20 sets a scene as a target of tactile presentation in the game content, sets the degree of output (intensity or frequency of vibration) to be used to express the scene, and designs a tactile signal to be actually output.
[0046] As described above, the tactile presentation technology is a useful technology that can present information to the user through the tactile presentation device 10 or give additional vibration to video and audio media, thereby allowing the user to have a higher sense of reality. The tactile presentation technology is widely used in various applications, including entertainment applications, in which the game controller 10A vibrates in combination with game sounds and a dedicated vibration device vibrates in combination with music as a method for making people with hearing impairments enjoy music, and an application that sends useful information to the user by using the vibration of a smart phone. Generally speaking, the tactile signal output from the tactile presentation device 10 is expressed in combination with an acoustic signal such as music or ambient sound, or is expressed using a tone so that the creator 20 conveys the desired information to the user.
[0047] However, the tactile signal is a time signal represented by the frequency or intensity used when the tactile presentation device 10 vibrates, so the output of the tactile signal on the tactile presentation device 10 with different frequency responses or output methods makes the output different each time, thereby there is a risk of generating an expression that is not intended by the creator 20. In order to avoid this situation, it is conceivable to prepare a plurality of tactile signals corresponding to various tactile presentation devices 10 in advance. However, this situation will increase the amount of data, resulting in reduced data storage and transmission efficiency. In addition, in the case where a tactile signal is newly generated from a time signal such as an acoustic signal instead of a specially prepared tactile signal, the source time signal is directly converted into a tactile signal by frequency shift or the like. At this time, in the case where the time signal of the conversion source is a time signal obtained by superimposing various individual signals, the generated signal may be a tactile signal in which multiple signals are superimposed and expression cannot be implemented according to the source signal. For example, in the case where it is desired to output a tactile signal having a dramatic sense of emphasizing the drum sound included in the music signal as a tactile signal, when a tactile signal including a high ratio of vibration corresponding to a human voice or a guitar is generated, it may not be possible to give a user an appropriate sense of reality.
[0048] In view of this, the conversion device 100 according to the embodiment uses the following conversion processing to allow high-quality tactile sensation to be presented without depending on the tactile presentation device 10 to which the output will be implemented, and to allow efficient data storage and transmission. Specifically, in order to express the conversion source signal as the source of the tactile signal through information with a higher abstraction level, the conversion device 100 converts the conversion source signal into an intermediate representation signal 24 expressed by multiple parameters corresponding to human perception. In addition, the decoding device 200 according to the embodiment decodes the intermediate representation signal 24 expressed at a high abstraction level, and generates a tactile signal to be actually output on the tactile presentation device 10. With this configuration, the conversion device 100 and the decoding device 200 can achieve tactile presentation adapted to human perception (excellent realism) without depending on the characteristics of the tactile presentation device 10, while improving the efficiency of data transmission and data retention.
[0049] In the following text, reference will be made to Figure 1 The following describes an overview of the information processing flow according to the embodiment. Figure 1 As shown in , the creator 20 generates a conversion source signal 22 as an arbitrary signal. For example, the creator 20 creates the conversion source signal 22 as audio content to be provided to a user through a network.
[0050] The conversion device 100 acquires the conversion source signal 22 created by the creator 20 (step S11). The conversion device 100 performs the conversion process according to the embodiment to convert the conversion source signal 22 into the intermediate representation signal 24. The details of the conversion process will be described later.
[0051] Subsequently, the decoding device 200 acquires the intermediate representation signal 24 through the network (step S12). At this time, the decoding device 200 acquires characteristic information about the tactile presentation device 10 to which the output is to be implemented. For example, if the tactile signal is to be output on the game controller 10A, the decoding device 200 acquires the characteristic information of the game controller 10A. Subsequently, the decoding device 200 performs the decoding and generation processing according to the embodiment, thereby decoding the intermediate representation signal 24 based on the intermediate representation signal 24 and the characteristic information of the tactile presentation device 10 to generate the tactile signal 26. The details of the decoding and generation processing will be described later.
[0052] The decoding device 200 implements control to send the generated haptic signal 26 to various haptic presentation devices 10 so as to output the signal on the haptic presentation device 10 (step S13). For example, the decoding device 200 sends the haptic signal 26A generated based on the characteristics of the game controller 10A to the game controller 10A. In addition, the decoding device 200 sends the haptic signal 26B generated based on the characteristics of the headphone 10B to the headphone 10B. In this way, the decoding device 200 can send a haptic signal optimized for each haptic presentation device 10, thereby making it possible to achieve an optimal output according to the characteristics of the haptic presentation device 10.
[0053] Next, we will refer to Figure 2 The subsequent drawings describe the configurations of the conversion device 100 and the decoding device 200 according to the embodiment and the details of the conversion process and the decoding process.
[0054] (1-2. Configuration of Conversion Device According to Embodiment) Next, we will refer to Figure 2 The configuration of the conversion device 100 according to the embodiment is described. Figure 2 : is a diagram showing a configuration example of the conversion device 100 according to the embodiment.
[0055] like Figure 2 As shown in , the conversion device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. It should be noted that the conversion device 100 may include an input device (e.g., a touch panel, a keyboard, or a pointing device such as a mouse, a voice input microphone, or an input camera (line of sight input or gesture input)) for implementing various operation inputs from an administrator or the like who operates the conversion device 100.
[0056] The communication unit 110 is implemented by, for example, a network interface card (NIC). The communication unit 110 is connected to a network N (Internet, near field communication (NFC), Bluetooth (registered trademark), etc.) in a wired or wireless channel, and transmits and receives information to / from the creator 20, the decoding device 200, other information devices, etc. through the network N.
[0057] The storage unit 120 is implemented by a semiconductor memory element such as a random access memory (RAM) and a flash memory, or other storage devices such as a hard disk or an optical disk. The storage unit 120 stores the acquired conversion source signal, the converted intermediate representation signal, and the like.
[0058] The control unit 130 is implemented by executing a program (e.g., a conversion program according to the embodiment, etc.) stored in the conversion device 100 by a central processing unit (CPU), a micro processing unit (MPU), etc. using a random access memory (RAM), etc. as a work area. In addition, the control unit 130 is a controller and can be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).
[0059] like Figure 2 As shown in FIG. 1 , the control unit 130 includes an acquisition unit 131, a conversion unit 132, and a transmission unit 133, and implements or executes the functions and actions of the information processing described below. The internal configuration of the control unit 130 is not limited to Figure 2 The configuration shown in , and may be another configuration as long as it is a configuration that implements the information processing described below.
[0060] The acquisition unit 131 acquires various data used by the processing unit to implement processing in the later stage. For example, the acquisition unit 131 acquires a conversion source signal as a signal to be converted by the conversion unit 132 as a source of a tactile signal.
[0061] The conversion unit 132 converts the conversion source signal acquired by the acquisition unit 131 into an intermediate representation signal represented by at least one parameter. For example, the conversion unit 132 converts the conversion source signal into an intermediate representation signal represented by one or more parameters corresponding to human perception.
[0062] Specifically, the conversion unit 132 converts the conversion source signal into an intermediate representation signal, wherein the intermediate representation signal includes, as parameters: an attack as a signal representing a steep rise in an output value, a harmonic component as a signal having a fundamental frequency, a noise component as a signal not having a fundamental frequency, and information indicating a ratio between the harmonic component and the noise component.
[0063] Figure 3 is a conceptual diagram showing encoding into an intermediate representation signal according to the embodiment. Figure 3 In the example shown, the acquisition unit 131 acquires the conversion source signal 30. Figure 3 As shown in , the conversion source signal 30 includes: music data such as songs, or an acoustic signal 32 obtained by, for example, recording natural sounds, environmental sounds, etc. In addition, the conversion source signal 30 may include a tactile signal 34, such as a tactile signal created for a specific actuator, or a tactile signal obtained after decoding an acoustic signal, etc., by a device other than the decoding device 200.
[0064] The conversion unit 132 replaces the information included in the conversion source signal 30 with information having a high abstraction level, thereby converting the conversion source signal 30 into the intermediate representation signal 40. Specifically, the conversion unit 132 converts parameters such as output value and frequency included in the conversion source signal 30 into a signal defined by parameters adapted to human perception. In the embodiment, the conversion unit 132 converts the signal into the intermediate representation signal 40, and the intermediate representation signal 40 includes, as parameters: an attack as a signal representing a steep rise in the output value, an attenuation occurring as time changes, and information indicating a ratio between a harmonic component and a noise component.
[0065] In this way, the conversion unit 132 replaces parameters such as output value and frequency included in the conversion source signal 30 with parameters adapted to human perception, thereby making it possible to retain information related to tactile presentation in a format independent of a device or an actuator. In addition, the conversion unit 132 converts a signal such as a tactile signal 34 including characteristic information to be applied to a specific tactile presentation device 10 into an intermediate representation signal 40 expressed by parameters not including the characteristic information, thereby making it possible to retain the tactile signal retained in the existing format in a newly replaced format independent of an output destination.
[0066] Figure 4 An example of an intermediate representation signal according to the described embodiment is shown. Figure 4 40 is a diagram showing an example of an intermediate representation signal according to the embodiment. Figure 4 The example uses a waveform to express each parameter included in the intermediate representation signal 40. It should be noted that Figure 4 The vertical axis of the conversion source signal 30 shown in φ represents the output value, and the horizontal axis represents the time.
[0067] exist Figure 4 In the intermediate representation signal 40 shown in FIG. 1 , the waveform 50 indicates the attack information. Specifically, the waveform 50 indicates the position of the attack on the time axis, the time length of each attack, and the frequency corresponding to the attack. The intermediate representation signal 40 according to the embodiment defines two types of attack lengths, the attack lengths are classified into the first attack ( Figure 4 ) and a second attack with a relatively shorter output time ( Figure 4 The waveform 50 also indicates the classification of these attacks. In addition, the frequency display 52 indicates the frequencies assigned to the attacks by the density or darkness of the hatching.
[0068] For example, Figure 4The attack 54 shown in is the first attack for a long time output. In addition, the frequency assigned to the attack 54 is indicated by the density or darkness of the hatching in the area 56, and for example, a relatively low frequency (about 80 Hz) is assigned to the attack 54. On the other hand, the attack 58 is the second attack for a short time output. The conversion unit 132 assigns the length and frequency of the attack 54 and the attack 58 based on the information included in the conversion source signal 30. In addition, the conversion unit 132 assigns the overall volume (output value) of the intermediate representation signal 40 based on the information included in the conversion source signal 30. Figure 4 In the example shown, the volume is represented as the amplitude (vertical axis) of waveform 50 .
[0069] Waveform 60 shows the ratio between the harmonic component and the noise component. Figure 4 In the example of , it is assumed that as the vertical axis of the waveform 60 is closer to 1, the ratio of the harmonic component is larger, and as the vertical axis of the waveform 60 is closer to 0, the ratio of the noise component is larger.
[0070] Waveform 62 shows the fundamental frequency of the harmonic components. Figure 4 In the example of FIG. 6 , the vertical axis of the waveform 62 is a numerical value indicating the fundamental frequency of the harmonic component.
[0071] Waveform 64 indicates the frequencies included in the noise component after passing through the low-pass filter. As described above, the noise component is a component that does not include the fundamental frequency, that is, a component composed of noise. For example, the frequency indicated by waveform 64 is not the fundamental frequency, but indicates the frequency mainly included in the noise component. With this configuration, even for the same noise component, it is possible to express whether the noise component is a relatively high noise component (wind noise in the case of natural sound, etc.) or a relatively low noise component.
[0072] When converting the conversion source signal 30 into the intermediate representation signal 40 , the conversion unit 132 separates the conversion source signal into individual elements constituting the conversion source signal 30 , and converts the separated signals into the intermediate representation signal 40 .
[0073] For example, in the case of an acoustic signal 32 which is music data such as a song and in which a plurality of musical instrument sounds are superimposed, the conversion unit 132 separates the acoustic signal 32 into individual musical instrument sounds and converts the separated signals into the intermediate representation signal 40. In addition, in the case where the conversion source signal is a natural sound, an environmental sound, etc., the conversion unit 132 separates the conversion source signal into a harmonic component which is a signal having a fundamental frequency and a noise component which is a signal not having a fundamental frequency. In this way, by separating the conversion source signal 30 into the elements constituting the conversion source signal 30, the conversion unit 132 can implement the conversion into the intermediate representation signal 40 which appropriately reflects the representation in the conversion source signal 30.
[0074] Will refer to Figures 5 to 7 An example describing audio source separation. Figure 5 is a diagram (1) showing an example of audio source separation according to the embodiment.
[0075] Figure 5 The example of shows an example in which the conversion unit 132 separates the song 68 as an example of the conversion source signal. In this example, the song 68 is a popular song in which a plurality of instrument sounds and vocals are mixed. In this case, the conversion unit 132 separates the audio source for each instrument sound constituting the song 68 using a known audio source separation technique. For example, the conversion unit 132 separates the song 68 by using an audio source separation process for each instrument sound, which is a non-harmonic sound separation method that separates steep sounds in the time direction (such as drum sounds) by using a neural network or by applying a median filter in the time-frequency domain.
[0076] As an example, the conversion unit 132 separates the song 68 into drum sounds, bass sounds, guitar sounds, and vocals. In this case, the separated instrument sounds can each be an element of attack, low-frequency vibration, high-frequency vibration, and emphasized medium-frequency vibration in the intermediate representation signal. When the intermediate representation signal generated from such elements is finally decoded into a tactile signal output by the tactile presentation device 10, the signal can be a tactile signal with a sense of drama taking into account the arrangement of the song.
[0077] Next, the separation of natural sound and environmental sound will be described. Figure 6 2 is a diagram showing an example of audio source separation according to the embodiment. It should be noted that natural sound and ambient sound refer to sounds recorded in nature or in a town, etc.
[0078] Figure 6 The example of shows an example in which the conversion unit 132 separates the natural sound 72 as an example of the conversion source signal. In this case, the conversion unit 132 separates the sound constituting the natural sound 72 into a harmonic component and a noise component. For example, if the natural sound 72 is the sound of wind, it is assumed that the sound has fewer harmonic components and more noise components. The noise component can be regarded as a signal component that generally has similar power in the frequency domain. In addition, the harmonic component can be regarded as a signal component in which a specific frequency has high power in the frequency domain.
[0079] When the intermediate representation signal generated from such an element is finally decoded into a haptic signal to be output by the haptic presentation device 10 , the signal may be a rough haptic signal including band-limited noise as a main component.
[0080] Incidentally, when separating natural sounds or environmental sounds, the conversion unit 132 can also, for example, extract only the sound included in the sound to be particularly emphasized. As an example, the conversion unit 132 can separate the bird song mixed in the wind sound as an element. For example, the conversion unit 132 can separate the bird song from other natural sounds by using a machine learning model dedicated to extracting the bird song.
[0081] Next, we will describe the Figure 6 Example of separation of natural sounds and ambient sounds. Figure 7 is a diagram (3) showing an example of audio source separation according to the embodiment.
[0082] Figure 7 The example of φ shows an example in which the conversion unit 132 separates the ambient sound 76 as an example of the conversion source signal. The ambient sound 76 is assumed to be sound data obtained by recording a situation including many car engine sounds or the like. Figure 7 The ambient sound 76 shown in the example of can be a sound effect produced for a game or video content (the sound of a car driving in a game, the sound of an object hitting a wall or floor, etc.). In this case, signal separation will be implemented to separate into harmonic components and noise components.
[0083] That is, similar to Figure 6 , the conversion unit 132 separates the sound constituting the environmental sound 76 into a harmonic component and a noise component. For example, in the case of a car driving sound, the road noise includes a high ratio of noise components, and the engine sound includes a high ratio of harmonic components. For this reason, it is assumed that the environmental sound 76 mainly including the car engine sound has a high ratio of harmonic components and a relatively low ratio of noise components after separation. In the separation of the noise component, the conversion unit 132 can also use a known separation technique, such as a method of separating a sound that does not change for a long time into noise (spectral subtraction method, etc.), or a process of separating steep frequency components for each time frame of the spectrogram.
[0084] When from Figure 7 When the intermediate representation signal generated by the separated elements in the example shown in is finally decoded into a haptic signal to be output by the haptic presentation device 10 , the intermediate representation signal may be a highly vibrating haptic signal including a sine wave as a main component.
[0085] Next, the process of extracting information from each separated audio source and actually converting the information into an intermediate representation signal will be described. It should be noted that all the intermediate representation signal extraction methods described below can be applied to each separated signal obtained by signal separation, or can be selectively applied to each separated signal with individual consideration of the characteristics of the separated signal. In addition, the methods can be applied directly to the converted source signal before separation.
[0086] The generation of attacks in the intermediate representation signal will first be described. Figure 8 is a diagram showing an attack in an intermediate representation signal according to the embodiment. Hereinafter, the result of transforming a time signal (conversion source signal) to be converted into an intermediate representation signal into a spectrogram (time-frequency representation by short-time Fourier transform) having a specific frame width and a specific offset width is represented as “X tf ” (t represents time and f represents frequency).
[0087] As described above, the attack is to parameterize a portion of the conversion source signal having a steep power change in the time direction. For example, regarding the output value change for each time unit in the conversion source signal, the conversion unit 132 refers to the difference with the value obtained by leveling the output value within a predetermined time width, and extracts the interval where the reference value exceeds the reference output value as an attack.
[0088] In the embodiment, the attack parameters are calculated using, for example, a process that removes steep temporal changes from the amplitude profile for each time frame, such as a median filter. In the following description, the amplitude profile is denoted as “V t ”.
[0089] When the amplitude profile of the steep time change is removed from the "V t ^sm”, the “V” indicating only the attack outline is expressed by the following formula (1): t ^a". Figure 8 As shown in the “V t The visual representation of "^sm" is demonstrated as a waveform from which the steep time changes are removed.
[0090] Next, in the "V t In the example, the conversion unit 132 combines the groups connected in the time direction into one attack. For example, the conversion unit 132 extracts Figure 8 The attacks 80 and 82 are shown as groups in a predetermined time interval.
[0091] In addition, each frequency power of the conversion source signal in the attack part is usually highly correlated with the tactile signal as the corresponding signal. That is, the lower the frequency of the conversion source signal, the lower the frequency of the corresponding tactile signal will actually be. Therefore, the conversion unit 132 can allocate a frequency corresponding to each attack based on the conversion source signal.
[0092] For example, the conversion unit 132 allocates the frequency corresponding to each attack based on the weighted average frequency of the signal in the interval corresponding to the attack in the conversion source signal. That is, in order to provide frequency information for each attack, the conversion unit 132 calculates the weighted average frequency by the frequency power in the frame where the target attack is located, for example, as shown in the following formula (2).
[0093]
[0094] In the above formula (2), "i" is the index indicating each attack, and "Ti" is the set of time frames where each attack is located.
[0095] Here, the tactile characteristics of the attack change depending on the time length, so a mechanism for separately extracting attacks with different lengths may be included to express the change in characteristics. That is, the conversion unit 132 may convert the conversion source signal into an intermediate representation signal, the intermediate representation signal including a first attack with a long output duration and a second attack with a shorter output duration than the first attack. For example, attack length classification may be implemented by changing the filter length in the median filter described above. That is, when the filter length is reduced, only short attacks are extracted, and when the filter length is increased, longer attacks are extracted. Figure 8 In the example shown, attack 80 is an example of a short attack and attack 82 is an example of a long attack.
[0096] Using the previous processing, the conversion unit 132 can set the position information, length information "T i "、Frequency information "f i ^ave" and power information "V (tϵTi) ^a" as the attack parameter for the i-th attack. Figure 8 Attack 82 is shown as an example of the i-th attack, the power of which is indicated as "V (tϵTi) ^a", the information about the attack also includes frequency information "f i ^ave". It should be noted that Figure 8 The "V t ^sm" corresponds to Figure 4 In the waveform 59 shown, the frequency information "f i^ave" corresponds to area 56. That is, the conversion unit 132 extracts the Figure 4 The information of the attack parameters is visualized in the waveform 50 shown. Figure 4 and 8 The waveform in the example shown in shows that the attack parameters in the intermediate representation signal and the parameters to be described later will be recorded as encoded digital information in practice.
[0097] Next, the noise component ratio in the intermediate representation signal will be described. As described above, the conversion unit 132 separates the conversion source signal into a harmonic component and a noise component, and the parameter indicating the noise component ratio is the ratio of the noise component included in each time frame in the conversion source signal. The noise ratio is high in noisy sounds such as wind sounds, and is low in sounds with many harmonic components such as wind instrument sounds.
[0098] The calculation of the noise ratio parameter can be implemented using the separation of noise components and harmonic components exemplified in the signal separation for sound effects, etc. Here, when the noise ratio parameter is calculated by "X tf ^n” (n indicates the noise component) and “X tf When the spectrogram of each separated component is expressed by ^h" (h indicates harmonic component), the noise ratio parameter "N" in time frame t is expressed by the following formula (3): t ”.
[0099]
[0100] The noise ratio function represented by the above formula (3) is useful for purposes such as adjusting the roughness of a tactile signal.
[0101] Next, the frequency of the noise component in the intermediate representation signal will be described. As described above, the conversion unit 132 can use the frequency corresponding to the noise component as one of the parameters of the intermediate representation signal. The noise component frequency parameter is, for example, a parameter used to determine the noise component when the haptic signal is output.
[0102] For example, the conversion unit 132 may calculate the frequency range of the bandpass filter calculated from the noise component included in the conversion source signal as the frequency parameter of the noise component. The noise component frequency parameter is useful for purposes such as expressing a rough tactile sensation (eg, irregular vibration) corresponding to the conversion source signal.
[0103] Next, the frequency of the harmonic component in the intermediate representation signal will be described. As described above, the conversion unit 132 can use the frequency corresponding to the harmonic component as one of the parameters of the intermediate representation signal. The harmonic component frequency parameter is, for example, a parameter used to determine the harmonic component output as a tactile signal.
[0104] For example, the conversion unit 132 may calculate the frequency of the sine wave extracted as the harmonic component included in the conversion source signal as the frequency parameter of the harmonic component. The harmonic frequency component parameter is useful for the purpose of expressing a vibrotactile sensation (eg, regular vibration) corresponding to the conversion source signal, for example.
[0105] As described above, based on the conversion source signal, the conversion unit 132 assigns frequencies corresponding to each of the harmonic component and the noise component. With this configuration, the conversion unit 132 can generate an intermediate representation signal capable of accurately reproducing the tactile presentation intended by the creator 20 or the like.
[0106] Will return Figure 2 The description is continued. The sending unit 133 sends the intermediate representation signal converted by the conversion unit 132 to a processing unit in a subsequent stage. For example, the sending unit 133 sends the intermediate representation signal to the decoding device 200 that decodes the intermediate representation signal.
[0107] (1-3. Procedure of conversion processing according to the embodiment) Fig. 9 The flow of the conversion process according to the embodiment is shown. Fig. 9 is a flowchart showing the procedure of the conversion process according to the embodiment.
[0108] like Fig. 9 As shown in FIG. 1 , first, the conversion device 100 acquires a conversion source signal (step S101 ). Subsequently, the conversion device 100 performs a signal separation process on the acquired conversion source signal (step S102 ).
[0109] Next, the conversion device 100 extracts a tactile representation from each separated signal (step S103). The tactile representation is an element that can be a source of tactile presentation to the user, such as the attack, noise component and harmonic component described above. The conversion device 100 integrates the extracted tactile representation (step S104).
[0110] The conversion device 100 converts the conversion source signal into an intermediate representation signal based on the integration information (step S105 ) and then transmits the intermediate representation signal to a device (decoding device 200 , etc.) capable of decoding the intermediate representation signal via a network or the like (step S106 ).
[0111] (1-4. Configuration of Decoding Device According to Embodiment) Next, the decoding process of the intermediate representation signal will be described. Fig.10 The configuration of the decoding device 200 according to the embodiment is described. Fig.10: is a diagram showing a configuration example of the decoding device 200 according to the embodiment.
[0112] like Figure 2 As shown in , the decoding device 200 includes a communication unit 210, a storage unit 220, and a control unit 230. It should be noted that the decoding device 200 may include an input device (e.g., a touch panel, a keyboard, or a pointing device such as a mouse, a voice input microphone, or an input camera (line of sight input or gesture input)) for implementing various operation inputs from a user operating the decoding device 200 or the like.
[0113] The communication unit 210 is implemented by, for example, a NIC. The communication unit 210 is connected to the network N in a wired or wireless channel, and transmits / receives information to / from devices such as the tactile presentation device 10 and the conversion device 100 through the network N.
[0114] The storage unit 220 is implemented by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 220 stores the acquired intermediate representation signal, the decoded haptic signal, and the like.
[0115] The control unit 230 is implemented by executing a program (e.g., a decoding program according to the embodiment) stored in the decoding device 200 using a RAM or the like as a work area by a CPU, an MPU, etc. In addition, the control unit 230 is a controller and can be implemented by using an integrated circuit such as an ASIC or an FPGA, for example.
[0116] like Fig.10 As shown in FIG. 1 , the control unit 230 includes an acquisition unit 231, a generation unit 232, and an output control unit 233, and implements or executes the functions and actions of the information processing described below. The internal configuration of the control unit 230 is not limited to Fig.10 The configuration shown in , and may be another configuration as long as it is a configuration that implements the information processing described below.
[0117] The acquisition unit 231 acquires various data for processing by the processing unit in the following stage. For example, the acquisition unit 231 acquires an arbitrary signal recording information about the expression of tactile presentation. Specifically, the acquisition unit 231 acquires an intermediate representation signal obtained by converting the conversion source signal by the conversion device 100.
[0118] That is, the acquisition unit 231 acquires the intermediate representation signal, wherein as parameters, the intermediate representation signal includes: an attack as information representing a steep rise in an output value, a harmonic component as information having a fundamental frequency, a noise component as information not having a fundamental frequency, and information indicating a ratio between the harmonic component and the noise component.
[0119] In addition, the acquisition unit 231 acquires characteristic information related to the output unit that implements the tactile presentation based on the intermediate representation signal, etc. The output unit may also be referred to as the tactile presentation device 10. That is, the acquisition unit 231 acquires information such as the characteristics of the element that actually vibrates based on the tactile signal and the characteristics of the tactile presentation device 10 that controls the element. The characteristic information may include information such as the part of the human body on which the output unit of the tactile presentation device 10 is worn, the number of output units included in the tactile presentation device 10, and the like.
[0120] The generating unit 232 generates a tactile signal as a signal for controlling the output of the output unit based on the intermediate representation signal acquired by the acquiring unit 231. For example, the generating unit 232 decodes the intermediate representation signal acquired by the acquiring unit 231, and adjusts the decoded signal based on the characteristic information, thereby generating a tactile signal. Alternatively, the generating unit 232 adjusts the signal obtained by decoding the intermediate representation signal based on the characteristic information, and thereby generates a tactile signal. That is, the generating unit 232 has a function of decoding the intermediate representation signal. As described above, when decoding the intermediate representation signal into the tactile signal as a decoding process, the generating unit 232 may adjust the intermediate representation signal itself based on the characteristic information, etc., and then synthesize (generate) the intermediate representation signal into the tactile signal, or may first decode the intermediate representation signal into the tactile signal, and then adjust the tactile signal based on other information such as the characteristic information. It should be noted that the generation unit 232 does not necessarily need to use all of the acquired characteristic information, but may use, for example, only minimum information necessary for output (such as information for identifying an output unit serving as an output destination) as the characteristic information.
[0121] Fig.11 is a conceptual diagram showing decoding of an intermediate representation signal according to the embodiment. Fig.11 In the example shown, the acquisition unit 231 acquires the intermediate representation signal 40 and the device information (characteristic information) 42. Fig.11 As shown in , the intermediate representation signal 40 includes parameters such as attack, time change and noise components. In addition, the device information 42 includes information such as frequency characteristics of the output unit and the position and number of the output units provided in the tactile presentation device 10.
[0122] The generation unit 232 decodes the intermediate representation signal 40 based on the acquired information, and thereby generates a tactile signal 300 for actually driving the output unit. Fig.11 As shown in , the generating unit 232 further decodes the intermediate representation signal 40 using the device information, so that a plurality of haptic signals corresponding to the respective haptic presentation devices 10 can be generated from one intermediate representation signal 40 .
[0123] With this configuration, the generation unit 232 can present an appropriate sensation to the user regardless of the characteristics of the device to be output. Specifically, the generation unit 232 handles the intermediate expression in a state where the device-dependent information is removed, making it possible to generate an appropriate tactile signal even when the actuator intended for output by the creator 20 is changed. In addition, the generation unit 232 appropriately decodes the common data distributed in a device-independent state for each device, making it possible to reduce the amount of data related to data distribution, etc.
[0124] Although the embodiment describes an example in which the generation unit 232 generates a haptic signal based on the intermediate representation signal 40 generated by the conversion device 100, the information to be decoded by the generation unit 232 is not limited to the intermediate representation signal 40. That is, the generation unit 232 can generate a haptic signal based on the method described below as long as the signal is a signal including an axis having a high level of abstraction based on some haptic representation (for example, a signal obtained by encoding an expression based on human perception, such as roughness, hardness, and intensity).
[0125] Fig.12 An example of an intermediate representation signal 40 handled by the generation unit 232 is shown. Fig.12 is a diagram showing an example of an intermediate representation signal 40 that is a target of a decoding process according to the embodiment. Fig.12 The intermediate representation signal 40 shown in FIG. 1 is a signal generated by the conversion device 100 and is related to Figure 4 The intermediate representation signal 40 is the same as the intermediate representation signal 40 shown in . That is, the intermediate representation signal 40 does not include device-dependent information, and includes the attack, noise component and harmonic component ratio and the frequency of these components as parameters adapted to human perception. Based on the parameters included in the intermediate representation signal 40, the generation unit 232 generates a tactile signal to be actually output by the tactile presentation device 10.
[0126] The following will refer to Figures 13 to 25 The details of the decoding process and the generation process according to the embodiment are described. Fig.13 is a diagram (1) showing an example of a decoding process according to the embodiment.
[0127] Fig.13 The example of exemplifies a decoding process based on attack parameters in the intermediate representation signal 40 . Fig.13 A waveform 50 is shown that visually represents the attack parameter.
[0128] As an example, a waveform 310 including two arbitrary attacks among the waveform 50 will be described. The waveform 310 is a simplified representation of only two triangle waves each indicating an attack. The vertical axis of the waveform 310 schematically indicates the intensity of the attack. The waveform 312 indicates the level of the main frequency of the harmonic component in the intermediate representation signal 40.
[0129] As an example, when decoding waveform 50, generation unit 232 processes the signal so that the higher the intensity of the attack, the larger the amplitude in the decoded signal, and the higher the frequency of the original signal, the higher the frequency in the decoded signal. For example, generation unit 232 decodes waveform 310 and waveform 312 into Fig.13 The waveform 314 shown in FIG. 314 is a waveform indicating the signal strength through the height of the amplitude, and the frequency level through the number of repetitions of the amplitude.
[0130] Next, we will refer to Fig.14 Another example is described. Fig.14 is a diagram (2) showing an example of the decoding process according to the embodiment. Fig.14 The example of exemplifies a decoding process based on a noise component in the intermediate representation signal 40 . Fig.14 A waveform 60 indicating the ratio between the noise component and the harmonic component and a waveform 64 indicating the frequency of the noise component are shown.
[0131] Waveform 320 is a portion of waveform 60 for a specific time period that only schematically shows a change in noise ratio. The vertical axis of waveform 320 indicates the ratio of noise components, for example, the larger the value of the vertical axis, the more noise components are present. In addition, waveform 322 indicates a change in the frequency of the noise component in the amount of time corresponding to waveform 320. The vertical axis of waveform 322 indicates the level of the frequency of the noise component.
[0132] As an example, the generation unit 232 sets the amplitude of the noise component when the waveform 60 and the waveform 64 are decoded so that the larger the overall volume and the noise ratio, the larger the amplitude will be. In addition, the generation unit 232 is set so that the higher the frequency of the noise component, the higher the frequency of the noise component of the decoded signal. As described above, the overall volume is a parameter indicating the magnitude of the output signal according to time, and corresponds to the vertical axis of the waveform 50 in the case of the signal before decoding (intermediate representation signal 40).
[0133] For example, the generation unit 232 decodes the waveform 320 and the waveform 322 into Fig.14The waveform 324 shown in FIG. 324 shows the magnitude and frequency of the noise component in the tactile signal. It should be noted that the waveform 324 shows the strength (magnitude) of the signal through the height of the amplitude, and shows the level of the frequency through the number of repetitions of the amplitude.
[0134] Next, we will refer to Fig.15 Another example is described. Fig.15 is a diagram (3) showing an example of the decoding process according to the embodiment. Fig.15 The example of shows a decoding process based on the harmonic components of the intermediate representation signal 40. Fig.15 A waveform 60 indicating the ratio between the noise component and the harmonic component and a waveform 62 indicating the frequency of the harmonic component are shown.
[0135] Waveform 330 is a portion of waveform 60 for a specific time period that only schematically shows a change in noise ratio. The vertical axis of waveform 330 indicates the ratio of noise components, for example, the larger the value of the vertical axis, the more noise components are present. In addition, waveform 332 indicates a change in the frequency of the harmonic component in the amount of time corresponding to waveform 330. The vertical axis of waveform 332 indicates the level of the frequency of the harmonic component.
[0136] As an example, the generation unit 232 sets the amplitude of the harmonic component when the waveform 60 and the waveform 62 are decoded so that the smaller the overall volume and noise ratio, the larger the amplitude will be. In addition, the setting of the generation unit 232 is such that the higher the frequency of the harmonic component, the higher the frequency of the harmonic component of the decoded signal.
[0137] For example, the generation unit 232 decodes the signal having the waveform 330 and the waveform 332 into a signal having a waveform 330 and a waveform 332. Fig.15 The waveform 334 shown in FIG. 334 indicates the magnitude and frequency of the harmonic components in the tactile signal. It should be noted that the waveform 334 indicates the strength (magnitude) of the signal through the height of the amplitude, and indicates the level of the frequency through the number of repetitions of the amplitude.
[0138] The above describes an example in which the generation unit 232 extracts information indicated by the waveform 314, the waveform 324, and the waveform 334 from the parameters included in the intermediate representation signal 40. The generation unit 232 integrates this information to generate a tactile signal. Fig.16 to describe the process. Fig.16 is a diagram showing an example of the generation process according to the embodiment.
[0139] exist Fig.16In the example shown, the generation unit 232 generates information indicated by the waveform 314, the waveform 324, and the waveform 334. Specifically, the generation unit 232 combines the amplitudes corresponding to the three waveforms along the time axis.
[0140] Furthermore, the generation unit 232 integrates the overall volume in the intermediate representation signal 40 . Fig.16 The waveform 336 shown in FIG. 4 shows the magnitude of the volume in the intermediate representation signal 40 as a waveform along the time axis.
[0141] The generation unit 232 combines a waveform obtained by integrating the information indicated by the waveform 314 , the waveform 324 , and the waveform 334 with the overall volume, thereby generating a haptic signal indicated by the waveform 340 . Fig.16 The example of illustrative embodiments schematically illustrates the amplitude (output value) and frequency included in the haptic signal using a waveform 340 .
[0142] Using the previous processing, the generation unit 232 can generate a tactile signal from the intermediate representation signal 40. Here, the generation unit 232 can also use the device information and various types of information to generate a tactile signal with higher reproducibility. Figures 17 to 25 Describe an extended example similar to this.
[0143] Fig.17 is an illustration ( 1 ) of an example of an adjustment process based on the characteristics of the tactile presentation device 10 . Fig.17 The graph 350 shown in FIG. 3 shows the frequency characteristics in a specific tactile presentation device 10. The example in the graph 350 shows that the tactile presentation device 10 has a significant peak near 70 Hz. In the case where data owned by a manufacturer of the actuator or the like can be acquired in advance as characteristic information, the acquisition unit 231 acquires such data. When there is no data indicating the characteristic information, the acquisition unit 231 can acquire the characteristic information of the tactile presentation device 10 by issuing a predetermined test signal or the like and observing the result (reaction) of the test signal or the like.
[0144] The generating unit 232 may implement a predetermined adjustment process with reference to the characteristic information, such as the graph 350 acquired by the acquiring unit 231 . Fig.17 The waveform 352 shown in FIG. 3 schematically shows the waveform of the tactile signal before adjustment. The vertical axis of the waveform 352 and the waveform 354 indicates the amplitude, and the horizontal axis indicates the frequency. As shown in the waveform 352, the signal before adjustment is uniform regardless of the frequency.
[0145] Referring to the characteristic information shown in the graph 350 , the generation unit 232 adjusts the waveform 352 to a waveform 354 . Fig.17The waveform 354 shown in FIG. 3 schematically shows the tactile signal after adjustment by the generation unit 232. As shown in the waveform 354, in the signal after adjustment, the amplitude near 70 Hz having a peak in the graph 350 is smaller than the amplitude of the waveform 352, and the amplitudes of other frequency bands are larger than the amplitude of the waveform 352.
[0146] In this way, the generation unit 232 adjusts the output value for each frequency in the decoded signal based on the frequency characteristics of the output unit obtained as the characteristic information, and thereby generates a tactile signal. That is, the generation unit 232 adjusts the output of the tactile signal based on information indicating the specific frequency characteristics of the tactile presentation device 10, for example, so that the actual output value becomes substantially constant. In other words, as post-processing of the decoding, the generation unit 232 corrects the tactile signal to reduce the output at the frequency at which vibration is likely to occur as a characteristic of the device, and to increase the output at the frequency at which vibration is unlikely to occur. With this operation, the generation unit 232 can achieve the output intended by the original intermediate representation signal, regardless of the characteristics of the device or actuator.
[0147] It should be noted that the characteristic information also includes information indicating not only the difference in frequency but also the difference in time response (eg, the time interval from application of voltage to occurrence of vibration). The generation unit 232 can also respond to such characteristic information by adjusting the haptic signal.
[0148] Fig.18 2 is a diagram showing an example of an adjustment process based on the characteristics of the tactile presentation device 10. A graph 360 shows the time response characteristics for a particular tactile presentation device 10. Specifically, the graph 360 shows the time required for the amplitude to reach the intended output value after the voltage is applied, and the time required for the amplitude to reach 0 after the voltage is turned off. Fig.18 As shown in FIG. 360 , the time response characteristics vary depending on the frequency. In this example, it is assumed that the tactile presentation device 10 has a fast response at 200 Hz and a slow response at 50 Hz corresponding to the curve 360. As a typical trend, the closer the frequency is to the resonant frequency of the vibrator, the slower the time response of the vibrator.
[0149] Fig.18 The waveform 362 shown in the diagram schematically illustrates the signal input to the tactile presentation device 10 shown in the graph 360. The waveform 362 includes an attack 364 and an attack 368 (e.g., referred to as an attack) representing the amplitude of vibrating the tactile presentation device 10. In this example, the attack 364 has a frequency of 200 Hz (indicating the number of vibrations) and the attack 368 has a frequency of 50 Hz.
[0150] Without implementing specific adjustments, as shown in the lower portion of waveform 362 , the amplitude corresponding to attack 364 and the amplitude corresponding to attack 368 result in the haptic signal as shown.
[0151] Here, the generation unit 232 performs a predetermined adjustment process. The waveform 372 represents the signal after the waveform 362 is adjusted by the generation unit 232. For example, the generation unit 232 adjusts the rise and decay timing of the decoded signal.
[0152] Specifically, the generation unit 232 shifts the amplitude of the attack 364 corresponding to the frequency with the fast response to a slightly earlier time. Fig.18 As shown in FIG. 3 , the amplitude 374 in the waveform 372 will be a waveform in which the amplitude becomes 0 earlier than the end time point 366 of the attack 364. In addition, the generation unit 232 shifts the amplitude of the attack 368 corresponding to the frequency of the slow response to a slightly later time. Accordingly, as Fig.18 As shown in , the amplitude 376 in the waveform 372 is a waveform whose amplitude becomes 0 later than the end time point 370 of the attack 368. These operations make it possible for the generation unit 232 to implement output adapted to perception according to the characteristics of the tactile presentation device 10.
[0153] In addition to the adjustment regarding the time, the generation unit 232 can also adjust the magnitude of the amplitude (that is, the input voltage of the tactile presentation device 10). Fig.19 Describe an example of this. Fig.19 is a diagram ( 3 ) showing an example of adjustment processing based on the characteristics of the tactile presentation device 10 .
[0154] Fig.19 Graph 360 and waveform 362 are shown again. Here, the generation unit 232 adjusts the amplitude of the waveform 362 to generate a tactile signal for implementing an output adapted to human perception. For example, by slightly suppressing the magnitude of the amplitude for frequencies with a high response speed, the generation unit 232 can reproduce an output adapted to human perception. In addition, by slightly increasing the magnitude of the amplitude for frequencies with a low response speed, the generation unit 232 can reproduce an output adapted to human perception.
[0155] Waveform 384 indicates the waveform corresponding to the tactile signal adjusted by the generation unit 232. That is, the generation unit 232 slightly attenuates the input voltage of the attack 364 corresponding to the frequency with a fast response. Fig.19 In the example of , the generating unit 232 attenuates the output value 380 corresponding to the attack 364. Accordingly, the amplitude 386 corresponding to the attack 364 has a slightly lower output value compared to the waveform 362. In addition, the generating unit 232 slightly amplifies the input voltage of the attack 368 corresponding to the frequency of the full response. Fig.19 In the example of FIG. 3 , the generation unit 232 amplifies the output value 382 corresponding to the attack 368 . Accordingly, the amplitude 388 corresponding to the attack 368 has a slightly increased output value compared to the waveform 362 .
[0156] In this way, the generation unit 232 adjusts the output timing or output value of the decoded signal based on the time response characteristic of the output unit acquired as the characteristic information, thereby generating a tactile signal. This makes it possible for the generation unit 232 to achieve the ideal output that the original intermediate representation signal corresponding to the time response characteristic of the tactile presentation device 10 has.
[0157] Incidentally, for a signal corresponding to a frequency with a slow response time, the generation unit 232 may implement a conditioning process of inputting a signal with an opposite phase so as to converge the vibration in advance. With this configuration, the generation unit 232 may suppress the vibration, thereby making it possible to control the output in the tactile presentation device 10 with a slow response time to an ideal time.
[0158] Meanwhile, in the case where the conversion source signal is accompanied by a steep frequency change, it is conceivable to use a method of retaining the information itself in the intermediate representation signal in the encoding process rather than the decoding process, so that the change can be reproduced through tactile representation.
[0159] Will refer to Fig. 20 Describe an example of this. Fig. 20 is a diagram showing an extended example of the conversion process according to the embodiment. Fig. 20 The processing shown in is performed by, for example, the conversion unit 132 of the conversion device 100 .
[0160] Fig. 20 The waveform 390 shown in shows a converted source signal having a record of the magnitude of the amplitude. In this case, by expressing the magnitude of the amplitude using the conversion process described above, information about the intermediate representation signal can be retained as an attack parameter. On the other hand, the waveform 392 shows a converted source signal having no record of the magnitude of the amplitude but having a record of a steep frequency change at time 394 or time 396. In this case, it is possible that the attack parameter will not be recorded in the conversion process described above. In addition, since steep frequency changes have a great influence on human perception, it is desirable to reproduce the frequency changes as a tactile representation.
[0161] Accordingly, for steep frequency changes in the converted source signal, such as when the fundamental frequency changes greatly in a very short time frame, the conversion device 100 can record the change as an attack parameter. Waveform 398 schematically shows information obtained by converting waveform 390 or waveform 392 into an intermediate representation signal.
[0162] In this way, as an attack, the conversion unit 132 of the conversion device 100 can extract, among the frequency changes in each time unit in the conversion source signal, an interval in which a frequency change exceeding a predetermined reference occurs in a predetermined time width. This enables the conversion device 100 to incorporate a steep frequency change as an attack parameter into the intermediate representation signal, thereby making it possible to generate an intermediate representation signal including a more diverse tactile representation.
[0163] Next, the adjustment process for the case where the attack continues in the intermediate representation signal will be described. Fig.21 is a diagram showing an example of an adjustment process according to a time change.
[0164] In human perception, the time interval within which two sounds can be recognized as different sounds is known by experience. Fig.21 The waveform 400 in illustratively shows a signal including two attacks. In this case, the tactile signal is indicated by a waveform having two amplitude peaks as indicated by the waveform 402. However, when the time interval 404 between the two attacks is less than a predetermined time (about 50 ms), there is a possibility that humans recognize the two sounds as one sound. The waveform 406 schematically shows a human perception detecting the tactile signal indicated by the waveform 400 and the waveform 402 as an output.
[0165] In this case, the tactile representation originally intended to present two attacks may be weakened. In order to cope with this situation, the generation unit 232 implements a predetermined adjustment process.
[0166] For example, when decoding generates a tactile signal as indicated by waveform 402, the generation unit 232 adjusts the tactile signal so that the attack 410 of the two attacks is shifted forward to a slightly earlier time as indicated by waveform 408. In this way, by separating the two attacks by a time interval (e.g., 50 ms or more) that allows humans to perceive different sounds, the generation unit 232 performs the adjustment so as not to combine the two attacks into one attack. The generation unit 232 can not only perform the adjustment of the attack 410 by shifting the attack to an earlier time, but also perform the adjustment of the attack 410 by slightly increasing the amplitude of the attack. This also enables the generation unit 232 to increase the sensitivity of the attack to human perception.
[0167] Furthermore, as another adjustment example, the generation unit 232 may adjust the haptic signal so as to shift the attack 414 , which is the later attack of the two attacks, to a slightly later time as shown in the waveform 412 .
[0168] As described above, the generation unit 232 adjusts the signal obtained by decoding the intermediate representation signal based on the parameter, thereby generating a tactile signal, wherein the parameter is given based on the perceptual sensitivity of a human being to whom the tactile presentation is to be output through the output unit. Specifically, the generation unit 232 adjusts the decoded signal based on the parameter, wherein the parameter is pre-set based on the perceptual sensitivity of a human being to whom the tactile presentation is to be output through the output unit.
[0169] As an example, the generating unit 232 has decoded a signal including a plurality of output intervals intended to be isolated and output in the intermediate representation signal (eg Fig.21 In the case of the waveform 400 shown in FIG. 1 ), and in the case where the time intervals between the plurality of output intervals are within the predetermined time (e.g., 50 ms) set as the parameter, the generation unit implements adjustment to widen the time intervals between the plurality of output intervals and generates a tactile signal. Alternatively, the generation unit 232 may implement adjustment to increase one of the output values corresponding to the plurality of output intervals, or implement adjustment to extend one of the output time lengths corresponding to the plurality of output intervals.
[0170] Another example of a process for adjusting a haptic signal according to human perception will be described next. Fig. 22 is a diagram (1) showing an example of an adjustment process adapted to human perception.
[0171] The curve graph 420 schematically shows the human perception intensity when various tactile signals having the same frequency are output from the tactile presentation device 10. As shown in the curve graph 420, when a human feels a specific tactile signal, the human has a strong perception of the signal immediately after feeling the signal, but when the signal lasts for more than a specific presentation time (e.g., 1 second), the sensitivity to the signal decreases. Therefore, the generation unit 232 adjusts the vibration intensity according to the perception characteristics to achieve the intended human perception intensity. For example, as shown in the curve graph 422, the generation unit 232 can adjust the amplitude of the tactile signal to gradually decay.
[0172] In this way, when a specific frequency and output value output in the decoded signal exceeds a predetermined time (eg, 1 second) set by a parameter, the generation unit 232 can implement adjustments to change the frequency or output value according to time.
[0173] In addition to the previous examples, the generation unit 232 can generate a tactile signal by adjusting the frequency or output value of the decoded signal based on information related to the human body part to which the output unit will output the tactile presentation (as a type of characteristic information of the device). For example, the sensitivity varies depending on the output destination of the tactile signal. It is shown that human fingertips have high sensitivity to sounds of about 200 Hz, and the generation unit 232 can appropriately adjust the tactile signal according to the sensitivity. In this case, by pre-reserving data related to human frequency characteristics corresponding to each body part and applying the retained information as a parameter, the generation unit 232 can implement adjustments adapted to each body part.
[0174] Another example of a process for adjusting a haptic signal according to human perception will be described next. Fig.23 is a diagram (2) showing an example of an adjustment process adapted to human perception.
[0175] The frequencies include frequencies that are highly perceptible to humans and frequencies that are not highly perceptible to humans. Therefore, the generation unit 232 can shorten the output time length of the signal corresponding to the highly perceptible frequency or reduce the amplitude of the signal. In other words, the generation unit 232 can adjust the signal decoded based on the preset parameters, and the preset parameters are based on the human perception sensitivity related to the frequency among the human perception characteristics.
[0176] Graph 430 shows an example of the relationship between frequency and vibration intensity. For example, assume that frequency band 432 in graph 430 includes frequencies that are highly perceptible to humans. In this case, as shown in adjustment process 436, generation unit 232 reduces the vibration intensity of the signal corresponding to frequency band 432. On the other hand, assume that frequency band 434 in graph 430 includes frequencies that are not highly perceptible to humans. In this case, as shown in adjustment process 436, generation unit 232 amplifies the vibration intensity corresponding to frequency band 434. This makes it possible for generation unit 232 to more appropriately implement the tactile representation intended by creator 20 or the like.
[0177] Another example of a process for adjusting a haptic signal according to human perception will be described next. Fig.24 is a diagram (3) showing an example of an adjustment process adapted to human perception.
[0178] It is known through experience that when the presentation of signals with similar frequencies lasts for a certain length of time, humans tend to have a negative feeling as a perception. Therefore, in the case where the tactile signal includes the presentation of signals with similar frequencies, the generation unit 232 can implement adjustments to mix the noise component into the signal. Fig.24The waveform 440 in FIG. 4 shows an example of a continuous presentation including signals having similar frequencies. The signal of the waveform 440 is a signal having a fundamental frequency as shown in the graph 442 (the horizontal axis of the graph 442 represents frequency and the vertical axis represents amplitude).
[0179] When such a signal is observed, the generation unit 232 mixes a noise component at the time of decoding to make adjustments so as not to allow a signal having a similar frequency to continue. Fig.24 The waveform 444 in the figure schematically shows the signal after adjustment by the generation unit 232. In addition, the graph 446 shows the frequency of the signal after adjustment. In this case, by superimposing the noise component while avoiding the change of the fundamental frequency of the original signal, the generation unit 232 can adjust the signal to a signal that does not cause negative feelings in humans without changing the basic characteristics of the signal.
[0180] Next, we will refer to Fig.25 Another example of adjustment is described. Fig.25 is a diagram showing an example of adjustment processing related to signal superposition.
[0181] Fig.25 The waveform 450 shown in FIG. 4 shows an example of superimposing a signal decoded from an attack parameter and a signal decoded from a noise component or a harmonic component. In the interval 452 and the interval 454 of the waveform 450, signals having similar amplitudes are superimposed.
[0182] Here, when the element of the attack is clearly output, the tactile signal is regarded as having a sense of drama and having an enhanced effect of tactile presentation. Accordingly, the generation unit 232 can refer to the tactile signal, especially the interval including the attack, and in the case where the signal other than the attack overlaps with the range before and after the interval including the attack (for example, within 50ms), and can implement adjustments such as reducing the signal amplitude before and after the attack signal in order to emphasize the attack signal. In addition, in the case where the attack interval continues for multiple times as in the waveform 450, the generation unit 232 can mute the signal superimposed with the attack at an earlier position. This makes it possible for the generation unit 232 to more effectively implement the tactile presentation corresponding to the attack.
[0183] Fig.25 The waveform 460 in FIG. 4 shows the signal after being conditioned by the generating unit 232. Fig.25In the example in FIG. 4 , the generation unit 232 removes the noise and harmonic components other than the attack in the interval 462, that is, performs adjustment so that only the signal corresponding to the attack is highlighted. In addition, the generation unit 232 also reduces the noise and harmonic components in the interval 464, that is, performs adjustment so that only the signal corresponding to the attack is highlighted. In this way, in the case where the information decoded from the attack and the information decoded from the parameters other than the attack interfere with each other, the generation unit 232 can perform adjustment so as to attenuate the output value decoded from the parameters other than the attack.
[0184] The foregoing describes an example of the adjustment process according to the embodiment. It should be noted that the adjustment process described above can be fully or selectively applied at the time of decoding.
[0185] Will return by Fig.10 The output control unit 233 outputs the tactile signal generated by the generation unit 232 to the tactile presentation device 10 . Specifically, the output control unit 233 sends the tactile signal to the tactile presentation device 10 through the network, and controls the tactile presentation to be output on the tactile presentation device 10 .
[0186] (1-5. Procedure of Decoding Processing According to Embodiment) Fig.26 The flow of the conversion process according to the embodiment is shown. Fig.26 is a flowchart showing the procedure of a decoding process according to the embodiment.
[0187] First, the decoding device 200 acquires an intermediate representation signal represented by parameters having a high abstraction level adapted to human perception (step S201 ). Subsequently, the decoding device 200 acquires device information including frequency characteristics of the tactile presentation device 10 and the like (step S202 ).
[0188] Subsequently, the decoding device 200 starts to generate a haptic signal corresponding to the device to which the haptic signal is to be output (step S203 ). At this time, the decoding device 200 determines whether there is a difference with a reference characteristic in the device to which the haptic signal is to be output (step S204 ).
[0189] The decoding device 200 refers to the device information, and when it is determined that there is a specific difference in the characteristics ("Yes" in step S204), the decoding device 200 determines the parameters to be used for decoding according to the characteristics (step S205). The parameters in this case are not limited to amplitude, frequency, etc., and also include the adjustment parameters in the adjustment process described above (values indicating the degree of increase / decrease of amplitude, the degree of shift of time, etc.).
[0190] After determining the parameters used for decoding, or when there is no difference with the reference characteristics (No in step S204), the decoding device 200 generates a haptic signal (step S206). Subsequently, the decoding device 200 may output the haptic signal to the haptic presentation device 10, or may retain the haptic signal in the storage unit 220.
[0191] (2. Modification of the embodiment) The information processing according to the above-described embodiment may include various modifications. Modifications of the embodiment will be described below.
[0192] (2-1, Equipment Configuration) The conversion device 100 and the decoding device 200 described above are not necessarily independent devices, and may be provided as processing units in an existing tactile presentation process. In this case, the processing units corresponding to the conversion device 100 and the decoding device 200 are incorporated into the existing tactile presentation process. Fig. 27 Describe this point. Fig. 27 is a diagram showing the flow of a tactile presentation process according to a modification.
[0193] Fig. 27 The process shown in includes a flow of outputting a tactile signal in an existing format to the tactile presentation device 10 through a conversion process and a decoding process according to the embodiment in a tactile presentation process. In this case, a tactile signal in an existing format is acquired by a tactile signal encoding device 500, and the tactile signal encoding device 500 includes an encoding unit that performs a conversion process according to the embodiment (step S301). Subsequently, the encoding unit generates an intermediate representation signal through the conversion process according to the embodiment (step S302).
[0194] Subsequently, the encoding unit sends the intermediate representation signal to the tactile signal decoding device 510, which includes a decoding unit that performs the decoding process according to the embodiment (step S303). The decoding unit generates a tactile signal from the intermediate representation signal and outputs the generated tactile signal to the tactile presentation device 10 (step S304).
[0195] exist Fig. 27 In the example of , the encoding unit and the decoding unit may be incorporated in the same device, or the encoding unit and the decoding unit may be incorporated in the tactile presentation device 10. That is, the conversion process and the decoding process according to the embodiment may be incorporated as the encoding process and the decoding process in a series of tactile presentation processes regardless of the device configuration. For example, Fig. 27 The encoding unit and the decoding unit shown in may be provided as a plug-in operating in the tactile presentation device 10 as software.
[0196] (3. Other embodiments) In addition to each embodiment described above, the processing according to each embodiment described above can be implemented in various different forms (modifications).
[0197] In addition, among the processes described in the previous embodiments, all or part of the processes described as being automatically implemented may be implemented manually, or the processes described as being manually implemented may be automatically implemented by known methods. In addition, unless otherwise indicated, the processing procedures, specific names, and information including various data and parameters described in the previous description or drawings may be changed in any manner. For example, the various information shown in each figure is not limited to the information shown.
[0198] In addition, each component of each device is provided as a functional and conceptual illustration and therefore does not necessarily need to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the form depicted in the drawings, and all or part of the device can be functionally or physically distributed or integrated into any unit according to various loads and usage conditions.
[0199] Furthermore, the embodiments and modifications described above may be appropriately combined within the scope that can be implemented without dealing with contradictions.
[0200] The effects described in this specification are merely examples, and thus there may be other effects not limited to the exemplified effects.
[0201] (4. Effects of the conversion device according to the present disclosure) As described above, the conversion device according to the present disclosure (the conversion device 100 in the embodiment) includes an acquisition unit (the acquisition unit 131 in the embodiment) and a conversion unit (the conversion unit 132 in the embodiment). The acquisition unit acquires a conversion source signal as a source of a tactile signal. The conversion unit converts the conversion source signal acquired by the acquisition unit into an intermediate representation signal represented by at least one parameter. For example, the conversion unit converts the conversion source signal into an intermediate representation signal represented by a single or multiple parameters corresponding to human perception.
[0202] In this way, the conversion device converts the signal used to present tactile information into an intermediate representation signal expressed by parameters corresponding to human perception without depending on the output environment. With this configuration, the conversion device can generate a tactile signal independently of the output environment.
[0203] In addition, the conversion unit separates the conversion source signal into elements constituting the conversion source signal, and converts the separated signals into intermediate representation signals. For example, in the case where the conversion source signal is an acoustic signal in which a plurality of musical instrument sounds are superimposed, the conversion unit separates the acoustic signal into individual musical instrument sounds, and converts the separated signals into intermediate representation signals. In addition, the conversion unit separates the conversion source signal into a harmonic component as a signal having a fundamental frequency and a noise component as a signal not having a fundamental frequency.
[0204] In this way, the conversion device first separates a plurality of individual signals included in the conversion source signal and then generates an intermediate representation signal, thereby making it possible to generate a feature that appropriately reflects each of the separated signals.
[0205] Furthermore, the acquisition unit acquires, as a conversion source signal, a haptic signal including characteristic information to be applied to a specific haptic presentation device. The conversion unit converts the haptic signal including characteristic information to be applied to the specific haptic presentation device into an intermediate representation signal represented by parameters not including the characteristic information.
[0206] In this way, the conversion device can replace the existing haptic signal with information that does not include device-dependent information, so there is no need to send or process information corresponding to a large number of devices to which output will be implemented. This makes it possible for the conversion device to effectively utilize resources related to data transmission and information processing.
[0207] In addition, the conversion unit converts the conversion source signal into an intermediate representation signal, wherein the intermediate representation signal includes, as parameters, an attack as information representing a steep rise in the output value, a harmonic component as information having a fundamental frequency, a noise component as information not having a fundamental frequency, and information indicating a ratio between the harmonic component and the noise component.
[0208] In this way, by expressing the signal with information such as the ratio of attack, noise and harmonic components, a tactile presentation adapted to human perception can be implemented, and the conversion device can achieve a tactile presentation that further resonates with human sensitivity.
[0209] Furthermore, the conversion unit refers to a difference between an output value change for each time unit in the conversion source signal and an average value of the output value in a predetermined time width, and extracts a section in which the reference value exceeds the reference output value as an attack.
[0210] In this way, the conversion device defines the attack based on the output value of the conversion source signal, thereby making it possible to achieve a tactile presentation with a sense of drama as intended by the conversion source signal.
[0211] In addition, the conversion unit converts the conversion source signal into an intermediate representation signal including a first attack and a second attack, wherein the first attack has a long output duration and the second attack has an output duration shorter than the first attack. In addition, the conversion unit assigns a frequency corresponding to each attack based on the conversion source signal. For example, the conversion unit assigns a frequency corresponding to each attack based on a weighted average frequency of a signal in an interval corresponding to the attack in the conversion source signal.
[0212] In this way, by attaching length and frequency information to the attack, the conversion device can appropriately reproduce the tactile presentation intended by the converted source signal.
[0213] Furthermore, the conversion unit assigns a frequency corresponding to each of the harmonic component and the noise component based on the conversion source signal.
[0214] In this way, by allowing the noise component and the harmonic component to have frequency information, the conversion device can reproduce tactile presentation that is difficult to reproduce using a simple time signal, such as the roughness intended by the conversion source signal.
[0215] Furthermore, as an attack, the conversion unit extracts a section in which a frequency change exceeding a predetermined reference occurs in a predetermined time width among frequency changes in each time unit in the conversion source signal.
[0216] In this way, by capturing the steep frequency changes as attacks, the conversion device can appropriately replace the event expressed by the conversion source signal with a haptic presentation.
[0217] (5. Effects of the decoding device according to the present disclosure) The decoding device according to the present disclosure (the decoding device 200 in the embodiment) includes an acquisition unit (the acquisition unit 231 in the embodiment) and a generation unit (the generation unit 232 in the embodiment). The acquisition unit acquires: an intermediate representation signal including a record of information related to a representation of tactile presentation; and characteristic information related to an output unit that implements tactile presentation based on the intermediate representation signal. The generation unit generates a tactile signal as a signal that controls the output of the output unit based on the intermediate representation signal acquired by the acquisition unit. For example, the generation unit adjusts a signal obtained by decoding the intermediate representation signal based on the characteristic information, or decodes the intermediate representation signal after adjusting the intermediate representation signal based on the characteristic information, and thereby generates a tactile signal.
[0218] In this way, the decoding device obtains an intermediate representation signal that does not include a record of device-dependent information but only includes a record of information related to the representation of the tactile presentation, and then decodes the signal based on the characteristic information of the output destination. This makes it possible to implement appropriate tactile presentation for various output destinations.
[0219] The acquisition unit acquires the frequency characteristics of the output unit as characteristic information. The generation unit adjusts the output value of each frequency based on the frequency characteristics, and thereby generates a tactile signal. The acquisition unit may acquire the time response characteristics of the output unit as characteristic information. The generation unit adjusts the output timing or output value of the decoded signal based on the time response characteristics, and thereby generates a tactile signal. In addition, the acquisition unit may acquire information related to the human body part to which the output unit outputs the tactile presentation as characteristic information. The generation unit adjusts the frequency or output value of the decoded signal according to the information about the human body part to which the output unit outputs the tactile presentation, and thereby generates a tactile signal.
[0220] In this way, the decoding device adjusts the output value and the like based on the characteristic information of the output destination, thereby making it possible to implement tactile presentation that reflects the creator's intention of the original signal regardless of the mode of the output unit or the type of tactile presentation device.
[0221] Furthermore, the generation unit generates a haptic signal by conditioning the signal obtained by decoding the intermediate representation signal based on parameters given based on the perceptual sensitivity of a human being to whom the haptic presentation is to be outputted through the output unit.
[0222] In this way, by implementing adjustments adapted to human perception, the decoding device can implement a more effective tactile presentation.
[0223] As an example, in the case where the generation unit has decoded a signal including a plurality of output intervals intended to be isolated and output in the intermediate representation signal, and the time intervals between the plurality of output intervals are within a predetermined time set as the parameter, the generation unit implements an adjustment to widen the time intervals between the plurality of output intervals and generates a tactile signal. The generation unit may implement an adjustment to amplify any output value corresponding to the plurality of output intervals and generate a tactile signal. In addition, the generation unit may implement an adjustment to extend any output time length corresponding to the plurality of output intervals and generate a tactile signal.
[0224] In this way, by adjusting the output value and timing of a signal or the like that is difficult to feel through human perception, the decoding device can implement a tactile presentation that does not obscure the intention of the creator of the original signal.
[0225] In addition, when a specific frequency and output value outputted in the decoded signal exceed a predetermined time set by a parameter, the generation unit performs adjustment to change the frequency or output value according to time. In addition, the generation unit adjusts the decoded signal based on the parameter, wherein the parameter is preset based on human perception sensitivity related to frequency.
[0226] In this way, when a signal is observed that is difficult for humans to feel or that is likely to cause negative feelings in humans, the decoding device can generate a signal adjusted to address these problems, thereby making it possible to implement a tactile presentation that makes humans feel comfortable.
[0227] Furthermore, the acquisition unit acquires an intermediate representation signal, wherein the intermediate representation signal includes, as parameters, an attack as information representing a steep rise in the output value, a harmonic component as information having a fundamental frequency, a noise component as information not having a fundamental frequency, and information indicating a ratio between the harmonic component and the noise component. The generation unit decodes information related to the output value and the frequency from each of the parameters, and thereby generates a tactile signal.
[0228] In this way, since the decoding device generates a haptic signal from an intermediate representation signal comprising parameters adapted to human perception, it is possible to implement a haptic rendering in a more intuitive way.
[0229] Furthermore, in the case where information decoded from the attack and information decoded from parameters other than the attack interfere with each other, the generation unit may implement adjustment so as to attenuate the output value decoded from the parameters other than the attack.
[0230] In this way, by adjusting the information that interferes with each other in the decoding, the decoding device can implement a very dramatic tactile presentation of the output.
[0231] (6. Hardware configuration) Information devices such as the conversion device 100 and the decoding device 200 according to the various embodiments described above are composed of, for example, Fig.28 The configuration shown in FIG. 1 is implemented by a computer 1000. Hereinafter, the conversion device 100 according to the embodiment will be described as an example. Fig.28 10 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the conversion device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The respective components of the computer 1000 are interconnected by a bus 1050.
[0232] The CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400 to control each component. For example, the CPU 1100 expands the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.
[0233] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000 , and the like.
[0234] The HDD 1400 is a non-transitory computer-readable recording medium that records a program executed by the CPU 1100 , data used by the program, and the like. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure as an example of the program data 1450 .
[0235] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from other devices through the communication interface 1500 or transmits data generated by the CPU 1100 to other devices.
[0236] The input / output interface 1600 is an interface for connecting the input / output device 1650 to the computer 1000. For example, the CPU 1100 receives data from an input device such as a touch panel, a keyboard, a mouse, a microphone, or a camera through the input / output interface 1600. In addition, the CPU 1100 sends data to an output device such as a display, a speaker, or a printer through the input / output interface 1600. In addition, the input / output interface 1600 can serve as a medium interface for reading a program recorded on a predetermined recording medium. Examples of the medium include optical recording media such as a digital versatile disk (DVD) or a phase-change rewritable disk (PD), magneto-optical recording media such as a magneto-optical disk (MO), tape media, magnetic recording media, and semiconductor memories.
[0237] For example, when the computer 1000 functions as the conversion device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes the information processing program loaded on the RAM 1200 so as to implement the functions of the control unit 130, etc. In addition, the HDD 1400 stores the conversion program and data according to the present disclosure in the storage unit 120. As another example, when the CPU 1100 executes the program data 1450 read from the HDD 1400, the CPU 1100 can acquire these programs from another device through the external network 1550.
[0238] It should be noted that the present technology may also have the following configurations.
[0239] (1) A decoding device comprising: an acquisition unit, which acquires an intermediate representation signal recording information related to the representation of the tactile presentation, and characteristic information related to the output unit implementing the tactile presentation based on the intermediate representation signal; and The generating unit generates a haptic signal as a signal outputted by the controlling output unit based on the intermediate representation signal acquired by the acquiring unit.
[0240] (2) The decoding device according to (1), wherein the generating unit generates the tactile signal in the following manner: A signal obtained by decoding the intermediate representation signal is conditioned based on the characteristic information, or the intermediate representation signal is first conditioned based on the characteristic information and then the intermediate representation signal is decoded.
[0241] (3) A decoding device according to (2), wherein The acquiring unit acquires the frequency characteristic of the output unit as characteristic information, and The generating unit generates a haptic signal by adjusting an output value of each frequency in the decoded signal based on the frequency characteristic.
[0242] (4) A decoding device according to (2) or (3), wherein The acquisition unit acquires the time response characteristic of the output unit as characteristic information, and The generating unit generates a haptic signal by adjusting an output timing or an output value of the decoded signal based on the time response characteristic.
[0243] (5) A decoding device according to any one of (2) to (4), wherein The acquisition unit acquires information related to the human body part to which the output unit will output the tactile presentation as characteristic information, and The generation unit generates a tactile signal by adjusting a frequency or an output value of the decoded signal according to information about a human body part to which the output unit is to output a tactile presentation.
[0244] (6) A decoding device according to any one of (2) to (5), wherein The generation unit generates a haptic signal by adjusting the decoded signal based on parameters that are preset based on the perceptual sensitivity of a human being to whom the output unit is to output a haptic presentation.
[0245] (7) A decoding device according to (6), wherein When the generation unit has decoded a signal including multiple output intervals in the intermediate representation signal that are intended to be isolated and output, and when the time intervals between the multiple output intervals are within a predetermined time set as the parameter, the generation unit implements adjustment to widen the time intervals between the multiple output intervals and generates a tactile signal.
[0246] (8) A decoding device according to (7), wherein The generating unit performs adjustment to increase any output value corresponding to the plurality of output intervals, and generates a haptic signal.
[0247] (9) A decoding device according to (7) or (8), wherein The generating unit implements adjustment to extend any output time length corresponding to the plurality of output intervals, and generates a haptic signal.
[0248] (10) A decoding device according to any one of (6) to (9), wherein The generating unit performs adjustment to change the frequency or the output value according to time when the constant frequency and the output value outputted in the decoded signal exceed a predetermined time set by a parameter.
[0249] (11) A decoding device according to any one of (6) to (10), wherein The generation unit adjusts the decoded signal based on parameters, wherein the parameters are preset based on frequency-dependent human perception sensitivity.
[0250] (12) A decoding device according to any one of (2) to (11), wherein The acquisition unit acquires an intermediate representation signal, wherein the intermediate representation signal includes, as parameters: an attack as information representing a steep rise in the output value, a harmonic component as information having a fundamental frequency, a noise component as information not having a fundamental frequency, and information indicating a ratio between the harmonic component and the noise component, and The generating unit generates a haptic signal by decoding information related to an output value and a frequency from each of the parameters.
[0251] (13) A decoding device according to (12), wherein The generation unit performs adjustment to attenuate an output value decoded from the parameter other than the attack in a case where the information decoded from the attack and the information decoded from the parameter other than the attack interfere with each other.
[0252] (14) A decoding method, comprising: performing the following operations by a computer: Acquire an intermediate representation signal recording information related to a representation of the tactile presentation, and characteristic information related to an output unit implementing the tactile presentation based on the intermediate representation signal; and A haptic signal is generated as a signal for controlling an output unit based on the acquired intermediate representation signal.
[0253] (15) A decoding program for causing a computer to function as: an acquisition unit, which acquires an intermediate representation signal recording information related to the representation of the tactile presentation, and characteristic information related to the output unit implementing the tactile presentation based on the intermediate representation signal; and The generating unit generates a haptic signal as a signal outputted by the controlling output unit based on the intermediate representation signal acquired by the acquiring unit.
[0254] Reference numerals list 10——Tactile presentation device 100 - Conversion equipment 110 - Communication unit 120 - Storage unit 130——Control unit 131 - Get Unit 132 - Conversion unit 133——Sending unit 200——Decoding device 210 - Communication unit 220——Storage unit 230——Control unit 231 - Get Unit 232 - Generate Unit 233——Output control unit
Claims
1. A decoding device, comprising: an acquisition unit, which acquires an intermediate representation signal recording information related to the representation of the tactile presentation, and characteristic information related to the output unit implementing the tactile presentation based on the intermediate representation signal; as well as The generating unit generates a haptic signal as a signal outputted by the controlling output unit based on the intermediate representation signal acquired by the acquiring unit.
2. The decoding device according to claim 1, wherein the generating unit generates the tactile signal by: A signal obtained by decoding the intermediate representation signal is conditioned based on the characteristic information, or the intermediate representation signal is first conditioned based on the characteristic information and then the intermediate representation signal is decoded.
3. The decoding device according to claim 2, wherein The acquiring unit acquires the frequency characteristic of the output unit as characteristic information, and The generating unit generates a haptic signal by adjusting an output value of each frequency in the decoded signal based on the frequency characteristic.
4. The decoding device according to claim 2, wherein The acquisition unit acquires the time response characteristic of the output unit as characteristic information, and The generating unit generates a haptic signal by adjusting an output timing or an output value of the decoded signal based on the time response characteristic.
5. The decoding device according to claim 2, wherein The acquisition unit acquires information related to the human body part to which the output unit will output the tactile presentation as characteristic information, and The generation unit generates a tactile signal by adjusting a frequency or an output value of the decoded signal according to information about a human body part to which the output unit is to output a tactile presentation.
6. The decoding device according to claim 2, wherein The generation unit generates a haptic signal by adjusting the decoded signal based on parameters that are preset based on the perceptual sensitivity of a human being to whom the output unit is to output a haptic presentation.
7. The decoding device according to claim 6, wherein When the generation unit has decoded a signal including multiple output intervals in the intermediate representation signal that are intended to be isolated and output, and when the time intervals between the multiple output intervals are within a predetermined time set as the parameter, the generation unit implements adjustment to widen the time intervals between the multiple output intervals and generates a tactile signal.
8. The decoding device according to claim 7, wherein The generating unit performs adjustment to increase any output value corresponding to the plurality of output intervals, and generates a haptic signal.
9. The decoding device according to claim 7, wherein The generating unit implements adjustment to extend any output time length corresponding to the plurality of output intervals, and generates a haptic signal.
10. The decoding device according to claim 6, wherein The generating unit performs adjustment to change the frequency or the output value according to time when the constant frequency and the output value outputted in the decoded signal exceed a predetermined time set by a parameter.
11. The decoding device according to claim 6, wherein The generation unit adjusts the decoded signal based on parameters, wherein the parameters are preset based on frequency-dependent human perception sensitivity.
12. The decoding device according to claim 2, wherein The acquisition unit acquires an intermediate representation signal, wherein the intermediate representation signal includes, as parameters: an attack as information indicating a steep rise in the output value, a harmonic component as information having a fundamental frequency, a noise component as information not having a fundamental frequency, and information indicating a ratio between the harmonic component and the noise component, and The generating unit generates a haptic signal by decoding information related to an output value and a frequency from each of the parameters.
13. The decoding device according to claim 12, wherein The generation unit performs adjustment to attenuate an output value decoded from the parameter other than the attack in a case where the information decoded from the attack and the information decoded from the parameter other than the attack interfere with each other.
14. A decoding method, comprising: performing the following operations by a computer: Acquire an intermediate representation signal recording information related to a representation of the tactile presentation, and characteristic information related to an output unit implementing the tactile presentation based on the intermediate representation signal; and A haptic signal is generated as a signal for controlling an output unit based on the acquired intermediate representation signal.
15. A decoding program for causing a computer to act as: an acquisition unit, which acquires an intermediate representation signal recording information related to the representation of the tactile presentation, and characteristic information related to the output unit implementing the tactile presentation based on the intermediate representation signal; and The generating unit generates a haptic signal as a signal outputted by the controlling output unit based on the intermediate representation signal acquired by the acquiring unit.
Citation Information
Patent Citations
Information processing apparatus, information processing method, and program
JP2019219785A
Information processing device, method, and program
WO2019138867A1
