Conversion device, conversion method, and conversion program
The tactile signal is converted into an intermediate representation signal by the conversion device, and the decoding device generates a suitable tactile signal based on the device characteristics, which solves the problem of the influence of equipment characteristics in the prior art and realizes efficient tactile signal transmission and storage.
Patent Information
- Application Number
- CN202380029301.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-31
- Filing Date
- 2023-03-22
- Publication Date
- 2024-11-08
AI Technical Summary
Existing haptic presentation technologies are difficult to generate high-quality haptic signals independently of the characteristics of the output device, and the complexity of the device characteristics considerations increases the workload of the creator and affects the user experience.
A conversion device and a decoding device are designed to obtain the conversion source signal and convert it into an intermediate representation signal, and independently of the output device characteristics, the decoding device generates a suitable tactile signal based on the device characteristics, thereby realizing efficient data storage and transmission.
It realizes the generation of high-quality haptic signals without relying on the characteristics of the output device, improves the user experience and reduces the complexity of data storage and transmission.
Smart Images

Figure CN118922802A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a conversion device, a conversion method, and a conversion program that perform signal conversion processing in haptic technology. Background Art
[0002] There is known a tactile presentation (haptic) technology that enables a user to obtain a desired sensory effect by performing tactile presentation using vibration stimulation or the like.
[0003] For example, there is known a technique for controlling the timing of a plurality of haptic signals to allow a user to simultaneously perceive two or more perception effects (eg, Patent Document 1). There is also known a technique for appropriately encoding a plurality of haptic signals (eg, Patent Document 2).
[0004] Citation List
[0005] Patent Literature
[0006] Patent Document 1: WO 2019 / 138867A
[0007] Patent Document 2: JP 2019-219785A Summary of the invention
[0008] Technical issues
[0009] According to known techniques, multiple tactile signals can be appropriately presented to a user, so that the user's tactile experience can be improved.
[0010] At the same time, tactile presentation is easily affected by the characteristics of the device that outputs the tactile signal or the vibrator (actuator) included in the device, etc. Therefore, depending on the output device, it may be difficult to reflect the intention of the creator of the tactile signal. In addition, in order to accurately reflect the creator's own intention, the creator of the tactile signal must create the signal in consideration of the characteristics of the device to which the signal is to be output, which increases the creator's workload.
[0011] In view of this, the present disclosure proposes a conversion device, a conversion method, and a conversion program that can generate a tactile signal independently of an output environment.
[0012] Solution to the problem
[0013] A conversion device according to an embodiment of the present disclosure includes: an acquisition unit that acquires a conversion source signal as a source of a tactile signal; and a conversion unit that converts the conversion source signal acquired by the acquisition unit into an intermediate representation signal represented by at least one parameter. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 is a diagram showing an overview of information processing according to an embodiment of the present disclosure.
[0015] Figure 2 is a diagram showing a configuration example of a conversion device according to this embodiment.
[0016] Figure 3 is a conceptual diagram illustrating encoding into an intermediate representation signal according to an embodiment.
[0017] Figure 4 is a diagram showing an example of an intermediate representation signal according to the embodiment.
[0018] Figure 5 is a diagram (1) showing an example of audio source separation according to an embodiment.
[0019] Figure 6 2 is a diagram showing an example of audio source separation according to an embodiment.
[0020] Figure 7 is a diagram (3) showing an example of audio source separation according to an embodiment.
[0021] Figure 8 is a diagram showing a burst in an intermediate representation signal according to the embodiment.
[0022] Fig. 9 is a flowchart showing the procedure of the conversion process according to this embodiment.
[0023] Fig.10 is a diagram showing a configuration example of a decoding device according to this embodiment.
[0024] Fig.11 is a conceptual diagram illustrating decoding of an intermediate representation signal according to this embodiment.
[0025] Fig.12 is a diagram showing an example of an intermediate representation signal which is a target of decoding processing according to this embodiment.
[0026] Fig.13 is a diagram (1) showing an example of a decoding process according to an embodiment.
[0027] Fig.14 2 is a diagram showing an example of a decoding process according to an embodiment.
[0028] Fig.15 3 is a diagram showing an example of decoding processing according to this embodiment.
[0029] Fig.16 : is a diagram showing an example of generation processing according to this embodiment.
[0030] Fig.17 is a diagram (1) showing an example of adjustment processing based on the characteristics of a tactile presentation device.
[0031] Fig.18 2 is a diagram showing an example of adjustment processing based on the characteristics of a tactile presentation device.
[0032] Fig.19 3 is a diagram showing an example of adjustment processing based on the characteristics of the tactile presentation device.
[0033] Fig. 20 : is a diagram showing an extended example of the conversion process according to this embodiment.
[0034] Fig.21 is a diagram showing an example of adjustment processing according to time changes.
[0035] Fig. 22 is a diagram (1) showing an example of adjustment processing adapted to human perception.
[0036] Fig.23 FIG. 2 is a diagram showing an example of adjustment processing suitable for human perception.
[0037] Fig.24 FIG. 3 is a diagram showing an example of adjustment processing suitable for human perception.
[0038] Fig.25 2 is a diagram showing an example of adjustment processing related to signal superposition.
[0039] Fig.26 is a flowchart showing the procedure of the decoding process according to this embodiment.
[0040] Fig. 27 is a diagram showing the flow of a tactile sense presentation process according to a modification.
[0041] Fig.28 is a hardware configuration diagram showing an example of a computer that realizes the function of the conversion device. DETAILED DESCRIPTION
[0042] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In each of the following embodiments, the same components are denoted by the same reference numerals, and repeated descriptions thereof will be omitted.
[0043] The present disclosure will be described in the following order.
[0044] 1. Example
[0045] 1-1. Overview of information processing according to the embodiment
[0046] 1-2. Configuration of conversion device according to embodiment
[0047] 1-3. Procedure of conversion processing according to the embodiment
[0048] 1-4. Configuration of decoding device according to embodiment
[0049] 1-5. Process of decoding processing according to the embodiment
[0050] 2. Modification of the Embodiment
[0051] 2-1. Equipment configuration
[0052] 3. Other embodiments
[0053] 4. Effects of the conversion device according to the present disclosure
[0054] 5. Effects of the decoding device according to the present disclosure
[0055] 6. Hardware Configuration
[0056] (1. Example)
[0057] (1-1. Overview of Information Processing According to Embodiment)
[0058] First, refer to Figure 1 An outline of information processing according to the embodiment is described. Figure 1 is a diagram showing an overview of information processing according to the embodiment.
[0059] Figure 1 The information processing system 1 according to this embodiment is shown. The information processing system 1 includes a conversion device 100, a decoding device 200, and a tactile presentation device 10. The information processing system 1 is a system that controls a series of processes for realizing tactile presentation intended by a creator 20 on the tactile presentation device 10.
[0060] The information processing according to this embodiment is performed by Figure 1 The conversion device 100 and the decoding device 200 shown in the figure are executed. Specifically, the conversion device 100 converts any signal created by the creator 20 into an intermediate representation signal 24 for representing the signal independently of the characteristics of the tactile presentation device 10, etc. In addition, the decoding device 200 decodes the intermediate representation signal 24 into a signal to be output on various tactile presentation devices 10. That is, the conversion device 100 and the decoding device 200 respectively play the role of encoding function and decoding function in a series of tactile presentation processes. In the following description, the signal before undergoing the processing of the conversion device 100 can be referred to as a "conversion source signal". In addition, in the following description, the tactile signal represents a signal having a waveform representing the vibration generated by the vibration of the output unit (vibrator (actuator)) of the tactile presentation device 10. The tactile signal can be interpreted as a command or parameter provided to the tactile presentation device 10 so as to realize vibration in the tactile presentation device 10.
[0061] Figure 1The conversion device 100 shown is an information processing device that converts conversion source signals of various modes into intermediate representation signals 24. For example, examples of the conversion device 100 include a personal computer (PC), a server device, and a tablet terminal. The conversion source signal is an arbitrary signal to be output as a tactile signal through the conversion process according to the present embodiment. Examples of the conversion source signal include: sound data in which songs, environmental sounds, etc. are recorded; moving image data; and a tactile signal generated by a device other than the decoding device 200 and rendered for a specific type of tactile presentation device 10.
[0062] The decoding device 200 is an information processing device that generates a haptic signal based on the intermediate representation signal 24. Examples of the decoding device 200 include a PC, a server device, and a tablet terminal.
[0063] The tactile presentation device 10 is an information processing device having a function of vibrating an output unit based on a tactile signal. Examples of the tactile presentation device 10 include a game controller 10A, a headset 10B, a wristband type device 10C, and a vest (suit) type device 10D. The tactile presentation device 10 includes one or more output units, and vibrates the output unit based on the tactile signal, thereby performing tactile presentation (giving stimulation) to the corresponding body part of the user. The output unit is an element that converts an electrical signal into vibration, and corresponds to, for example, a device such as an eccentric motor, a linear vibrator, and a piezoelectric element actuator.
[0064] A user touching the tactile presentation device 10 can receive a tactile presentation corresponding to a content stream such as a video and audio displayed on a display, etc. while enjoying the content, thereby being able to enjoy the content with a higher sense of reality. Specifically, the user can enjoy a tactile presentation synchronized with the time lapse of the presentation of displayed content such as a moving image or audio.
[0065] The creator 20 is a person who creates content or signals for tactile presentation. For example, the creator 20 creates content such as moving images or audio. Alternatively, the creator 20 creates a signal (tactile signal) dedicated to tactile presentation in order to enhance the user's sense of reality. For example, in the case of performing tactile presentation by using the game controller 10A, the creator 20 sets a scene in the game content as an object of tactile presentation, sets the degree of output to be used to represent the scene (intensity or frequency of vibration), and designs the tactile signal to be actually output.
[0066] As described above, the tactile presentation technology is a useful technology that can present information to the user through the tactile presentation device 10, or provide additional vibration to video and audio media, thereby allowing the user to have a higher sense of reality. The tactile presentation technology is widely used in applications including entertainment applications, where the game controller 10A vibrates in conjunction with game sounds, and a dedicated vibration device vibrates in conjunction with music as a method for enabling people with hearing difficulties to enjoy music, and applications that transmit useful information to the user by using the vibration of a smart phone. Typically, the tactile signal output from the tactile presentation device 10 is expressed in conjunction with an acoustic signal such as music or ambient sound, or is expressed in a tone so that the creator 20 conveys the desired information to the user.
[0067] However, the tactile signal is a time signal represented by a frequency or intensity used when the tactile presentation device 10 vibrates, and therefore, outputting the tactile signal on the tactile presentation device 10 with different frequency responses or output methods will make each output different, with the risk of generating an expression that the creator 20 does not expect. In order to avoid this, it is conceivable to prepare a plurality of tactile signals corresponding to various tactile presentation devices 10 in advance. However, this case will increase the amount of data, resulting in deterioration of data storage and transmission efficiency. In addition, in the case where a tactile signal is newly generated from a time signal such as an acoustic signal instead of a specially prepared tactile signal, the source time signal is directly converted into a tactile signal by frequency shifting or the like. At this time, in the case where the signal of the conversion source is a time signal obtained by superimposing various individual signals, the generated signal may be a tactile signal superimposed with multiple signals, and the tactile signal cannot perform expression according to the source signal. For example, when a tactile signal including a high vibration ratio corresponding to singing or guitar is generated in the case where a tactile signal having a vivid feeling of emphasizing the drum sound included in the music signal is expected to be output as a tactile signal, there is a possibility that an appropriate sense of reality cannot be given to the user.
[0068] In view of this, the conversion device 100 according to the present embodiment uses the following conversion processing to enable high-quality tactile presentation without relying on the tactile presentation device 10 on which the output is to be performed, and to enable efficient data storage and transmission. Specifically, in order to express the conversion source signal as the source of the tactile signal through information with a higher degree of abstraction, the conversion device 100 converts the conversion source signal into an intermediate representation signal 24 expressed by multiple parameters corresponding to human perception. In addition, the decoding device 200 according to the present embodiment decodes the intermediate representation signal 24 expressed at a high degree of abstraction, and generates a tactile signal to be actually output on the tactile presentation device 10. With this configuration, the conversion device 100 and the decoding device 200 can achieve tactile presentation (excellent realism) suitable for human perception without relying on the characteristics of the tactile presentation device 10 while improving the efficiency of data transmission and data retention.
[0069] Below, we will refer to Figure 1 The following describes an overview of the flow of information processing according to this embodiment. Figure 1 As shown, the creator 20 generates a conversion source signal 22 as an arbitrary signal. For example, the creator 20 creates the conversion source signal 22 as audio content to be provided to a user via a network.
[0070] The conversion device 100 acquires the conversion source signal 22 created by the creator 20 (step S11). The conversion device 100 performs the conversion process according to the present embodiment to convert the conversion source signal 22 into the intermediate representation signal 24. The details of the conversion process will be described below.
[0071] Thereafter, the decoding device 200 acquires the intermediate representation signal 24 via the network (step S12). At this time, the decoding device 200 acquires characteristic information about the tactile presentation device 10 on which the output is to be performed. For example, in the case where a tactile signal is to be output on the game controller 10A, the decoding device 200 acquires characteristic information of the game controller 10A. Thereafter, the decoding device 200 performs the decoding and generation processing according to the present embodiment, thereby decoding the intermediate representation signal 24 based on the intermediate representation signal 24 and the characteristic information of the tactile presentation device 10 to generate a tactile signal 26. The details of the decoding and generation processing will be described below.
[0072] The decoding device 200 controls to send the generated tactile signal 26 to various tactile presentation devices 10 to output the signal on the tactile presentation device 10 (step S13). For example, the decoding device 200 sends the tactile signal 26A generated based on the characteristics of the game controller 10A to the game controller 10A. In addition, the decoding device 200 sends the tactile signal 26B generated based on the characteristics of the earphone 10B to the earphone 10B. In this way, the decoding device 200 can send a tactile signal optimized for each tactile presentation device 10, so that the optimal output can be achieved according to the characteristics of the tactile presentation device 10.
[0073] Next, we will refer to Figure 2 and subsequent drawings describe the configurations of the conversion device 100 and the decoding device 200 according to this embodiment and the details of the conversion process and the decoding process.
[0074] (1-2. Configuration of Conversion Device According to Embodiment)
[0075] Next, we will refer to Figure 2 The configuration of the conversion device 100 according to this embodiment is described. Figure 2 is a diagram showing a configuration example of the conversion device 100 according to this embodiment.
[0076] like Figure 2 As shown, the conversion device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. Note that the conversion device 100 may include an input component (e.g., a touch panel, a keyboard, or a pointing device such as a mouse, a voice input microphone, or an input camera (line of sight input or gesture input)) that performs various operation inputs from an administrator or the like who operates the conversion device 100.
[0077] The communication unit 110 is implemented by, for example, a network interface card (NIC). The communication unit 110 is connected to a network N (Internet, near field communication (NFC), Bluetooth (registered trademark), etc.) via a wired or wireless channel, and sends and receives information to and from the creator 20, the decoding device 200, other information devices, etc. via the network N.
[0078] The storage unit 120 is implemented by a semiconductor memory element such as a random access memory (RAM) and a flash memory or other storage devices such as a hard disk or an optical disk. The storage unit 120 stores the acquired conversion source signal, the converted intermediate representation signal, and the like.
[0079] The control unit 130 is implemented by executing a program (e.g., a conversion program according to an embodiment, etc.) stored in the conversion device 100 through a central processing unit (CPU), a microprocessing unit (MPU), etc., with a random access memory (RAM) or the like as a working area. In addition, the control unit 130 is a controller and can be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).
[0080] like Figure 2 As shown, the control unit 130 includes an acquisition unit 131, a conversion unit 132 and a sending unit 133, and implements or executes the functions and actions of the information processing described below. The internal configuration of the control unit 130 is not limited to Figure 2 The configuration shown in , and may be another configuration as long as it is a configuration that executes the information processing described below.
[0081] The acquisition unit 131 acquires various data used by the processing unit to perform processing in a later stage. For example, the acquisition unit 131 acquires a conversion source signal as a signal to be subjected to conversion processing by the conversion unit 132 as a source of a tactile signal.
[0082] The conversion unit 132 converts the conversion source signal acquired by the acquisition unit 131 into an intermediate representation signal represented by at least one parameter. For example, the conversion unit 132 converts the conversion source signal into an intermediate representation signal represented by a single or multiple parameters corresponding to human perception.
[0083] Specifically, the conversion unit 132 converts the conversion source signal into an intermediate representation signal, which includes a burst, a harmonic component, a noise component, and information indicating the ratio between the harmonic component and the noise component as parameters, wherein the burst is a signal indicating a sharp rise in the output value, the harmonic component is a signal with a fundamental frequency, and the noise component is a signal without a fundamental frequency.
[0084] Figure 3 is a conceptual diagram showing encoding into an intermediate representation signal according to an embodiment. Figure 3 In the example shown, the acquisition unit 131 acquires the conversion source signal 30. Figure 3 As shown, the conversion source signal 30 includes: music data such as a song; or an acoustic signal 32 obtained by recording, for example, natural sounds, environmental sounds, etc. In addition, the conversion source signal 30 may include a tactile signal 34, for example, a tactile signal created for a specific actuator or a tactile signal obtained after the acoustic signal, etc. has been decoded by a device other than the decoding device 200.
[0085] The conversion unit 132 replaces the information contained in the conversion source signal 30 with information with a high degree of abstraction, thereby converting the conversion source signal 30 into the intermediate representation signal 40. Specifically, the conversion unit 132 converts parameters such as output value and frequency contained in the conversion source signal 30 into a signal defined by parameters suitable for human perception. In an embodiment, the conversion unit 132 converts the signal into the intermediate representation signal 40, which includes a burst as a parameter, attenuation that occurs over time, and information indicating the ratio between a harmonic component and a noise component, wherein the burst is a signal indicating a sharp rise in the output value.
[0086] In this way, the conversion unit 132 replaces parameters such as the output value and frequency included in the conversion source signal 30 with parameters suitable for human perception, so that information related to tactile presentation can be retained in a format independent of a device or an actuator. In addition, the conversion unit 132 converts a signal such as a tactile signal 34 including characteristic information to be applied to a specific tactile presentation device 10 into an intermediate representation signal 40 expressed by parameters that do not include the characteristic information, so that the tactile signal maintained in the existing format can be maintained in the newly replaced format independently of the output destination.
[0087] Figure 4 An example of an intermediate representation signal according to this embodiment is shown. Figure 4 is a diagram showing an example of an intermediate representation signal according to the embodiment. Figure 4 The example uses a waveform to express each parameter included in the intermediate representation signal 40 when the conversion source signal 30 is converted into the intermediate representation signal 40. Note that Figure 4The vertical axis of the conversion source signal 30 shown in represents the output value, and the horizontal axis represents the time.
[0088] exist Figure 4 In the intermediate representation signal 40 shown, the waveform 50 indicates the burst information. Specifically, the waveform 50 indicates the position of the burst on the time axis, the time length of each burst, and the frequency corresponding to the burst. The intermediate representation signal 40 according to this embodiment defines two types of burst lengths, which are classified into outputting a relatively long first burst ( Figure 4 ) and outputs a relatively short second burst ( Figure 4 ). Waveform 50 indicates the classification of those bursts. In addition, frequency display 52 indicates the frequencies assigned to the bursts by the density or darkness of the shading.
[0089] For example, Figure 4 The burst 54 shown in is a first burst output for a long time. In addition, the frequency assigned to the burst 54 is indicated by the density or darkness of the shading in the area 56, and for example, a relatively low frequency (about 80 Hz) is assigned to the burst 54. On the other hand, the burst 58 is a second burst output for a short time. The conversion unit 132 allocates the length and frequency of the burst 54 and the burst 58 based on the information included in the conversion source signal 30. In addition, the conversion unit 132 allocates the overall volume (output value) of the intermediate representation signal 40 based on the information included in the conversion source signal 30. Figure 4 In the example shown, the quantity is represented as the amplitude of the waveform 50 (vertical axis).
[0090] Waveform 60 represents the ratio between the harmonic component and the noise component. Figure 4 In the example of FIG. 6 , it is assumed that the ratio of the harmonic component is larger when the vertical axis of the waveform 60 is closer to 1, and the ratio of the noise component is larger when the vertical axis of the waveform 60 is closer to 0.
[0091] Waveform 62 represents the fundamental frequency of the harmonic components. Figure 4 In the example of FIG. 6A , the vertical axis of the waveform 62 is a numerical value indicating the fundamental frequency of the harmonic component.
[0092] Waveform 64 represents the frequency included in the noise component after passing through the low-pass filter. As described above, the noise component is a component that does not include the fundamental frequency, that is, a component composed of noise. For example, the frequency indicated by waveform 64 is not the fundamental frequency, but indicates the frequency most included in the noise component. With this configuration, even for the same noise component, it is possible to indicate whether the noise component is a relatively high noise component (wind noise, etc. in the case of natural sound) or a relatively low noise component.
[0093] When converting the conversion source signal 30 into the intermediate representation signal 40 , the conversion unit 132 separates the conversion source signal into elements constituting the conversion source signal 30 , and converts the separated signals into the intermediate representation signal 40 .
[0094] For example, in the case of an acoustic signal 32 which is music data such as a song and in which a plurality of musical instrument sounds are superimposed, the conversion unit 132 separates the acoustic signal 32 into individual musical instrument sounds and converts the separated signals into the intermediate representation signal 40. In addition, in the case where the conversion source signal is a natural sound, an environmental sound, or the like, the conversion unit 132 separates the conversion source signal into a harmonic component which is a signal having a fundamental frequency and a noise component which is a signal not having a fundamental frequency. In this way, by separating the conversion source signal 30 into the elements constituting the conversion source signal 30, the conversion unit 132 can perform conversion into the intermediate representation signal 40 which appropriately reflects the representation in the conversion source signal 30.
[0095] Reference Figures 5 to 7 An example of audio source separation according to the embodiment is described. Figure 5 is a diagram (1) showing an example of audio source separation according to an embodiment.
[0096] Figure 5 The example of FIG. 1 shows an example in which the conversion unit 132 separates song 68 as an example of a conversion source signal. In this example, song 68 is a pop song in which a plurality of instrument sounds and vocals are mixed. In this case, the conversion unit 132 uses a known audio source separation technique to separate the audio source of each instrument sound constituting song 68. For example, the conversion unit 132 separates song 68 by performing an audio source separation process on each instrument sound using a neural network, or by a non-harmonic sound separation method that separates steep sounds (such as drum sounds) in the time direction by applying a median filter in the time-frequency domain.
[0097] As an example, the conversion unit 132 separates the song 68 into drum sounds, bass, guitar sounds, and voices. In this case, the separated instrument sounds may each be an element of an attack, a low-frequency vibration, a high-frequency vibration, and an aggravated medium-frequency vibration in the intermediate representation signal. When the intermediate representation signal generated from such elements is finally decoded into a tactile signal output by the tactile presentation device 10, the signal may be a tactile signal having an exciting feeling, considering the composition of the song.
[0098] Next, separation of natural sound and environmental sound will be described. Figure 6 2 is a diagram showing an example of audio source separation according to an embodiment. Note that natural sound and ambient sound represent sounds recorded in nature or in a town, etc.
[0099] Figure 6The example of shows an example in which the conversion unit 132 separates a natural sound 72 as an example of a conversion source signal. In this case, the conversion unit 132 separates the sound constituting the natural sound 72 into a harmonic component and a noise component. For example, if the natural sound 72 is the sound of wind, it is assumed that the sound has fewer harmonic components and more noise components. The noise component can be considered as a signal component having similar power widely in the frequency domain. In addition, the harmonic component can be considered as a signal component in which a specific frequency has a large power in the frequency domain.
[0100] When an intermediate representation signal generated from such an element is finally decoded into a haptic signal to be output by the haptic presentation device 10 , the signal may be a rough haptic signal including band-limited noise as a main component.
[0101] Incidentally, for example, when separating natural sounds or environmental sounds, the conversion unit 132 may also extract only the sound to be particularly emphasized included in the sound. For example, the conversion unit 132 may separate the sound of birdsong mixed in the sound of wind as an element. For example, the conversion unit 132 may separate the sound of birdsong from other natural sounds by using a machine learning model specifically for extracting the sound of birdsong.
[0102] Next, we will describe Figure 6 Examples of different separations of natural and ambient sounds. Figure 7 is a diagram (3) showing an example of audio source separation according to an embodiment.
[0103] Figure 7 The example of exemplifies an example in which the conversion unit 132 separates an environmental sound 76 as an example of a conversion source signal. The environmental sound 76 is assumed to be sound data obtained by recording a situation including many car engine sounds and the like. Figure 7 The ambient sound 76 shown in the example of can be a sound effect produced for a game or a video content (the sound of a car driving in a game, the sound of an object hitting a wall or floor, etc.). In this case, the signal separation is also performed as a separation of harmonic components and noise components.
[0104] That is to say, Figure 6Similarly, the conversion unit 132 separates the sounds constituting the ambient sound 76 into harmonic components and noise components. For example, in the case of a car driving sound, the road noise includes a high ratio of noise components, and the engine sound includes a high ratio of harmonic components. Therefore, it is assumed that the ambient sound 76 mainly including the car engine sound has a high harmonic component ratio and a relatively low noise component ratio after separation. In the separation of the noise component, the conversion unit 132 can also use a known separation technique, such as a method of separating a sound that does not change for a long time as noise (spectral subtraction, etc.) or a process of separating steep frequency components for each time frame of the spectrogram.
[0105] When from Figure 7 When the intermediate representation signal generated by the separated elements in the illustrated example is finally decoded into a haptic signal to be output by the haptic presentation device 10 , the intermediate representation signal may be a highly vibrating haptic signal including a sine wave as a main component.
[0106] Next, the process of extracting information from each separated audio source and actually converting the information into an intermediate representation signal will be described. Note that all the intermediate representation signal extraction methods described below can be applied to each of the separated signals obtained by signal separation, or can be selectively applied to each of the separated signals individually in consideration of the characteristics of the separated signals. In addition, the method can be directly applied to the converted source signal before separation.
[0107] First, the generation of bursts in the intermediate representation signal will be described. Figure 8 1 is a diagram showing a burst in the intermediate representation signal according to the embodiment. In the following, the result of transforming the time signal (transformation source signal) to be transformed into the intermediate representation signal into a spectrogram (time frequency representation using short-time Fourier transform) having a certain frame width and a certain shift width is represented as “X tf ” (t represents time and f represents frequency).
[0108] As described above, a burst is a portion of the conversion source signal that has a sharp power change in the time direction in order to parameterize. For example, regarding the output value change per time unit in the conversion source signal, the conversion unit 132 refers to the difference from the value obtained by averaging the output value within a predetermined time width, and extracts the interval where the referenced value exceeds the reference output value as a burst.
[0109] In this embodiment, the burst parameter is calculated by removing the sharp time changes of the amplitude profile of each time frame, such as using a median filter. In the following description, the amplitude profile is represented as "V t ”.
[0110] When the amplitude profile from which the steep time variations are removed is "V t^sm”, indicating only the “V t ^a" is represented by the following formula (1). t The visual representation of ^sm" is demonstrated as a waveform from which the steep time variations have been removed, such as Figure 8 shown.
[0111]
[0112] Next, in the "V t In the example, the conversion unit 132 combines the groups connected in the time direction into one burst. For example, the conversion unit 132 extracts Figure 8 Burst 80 and burst 82 are shown as a group.
[0113] In addition, each frequency power of the conversion source signal in the burst portion is generally highly correlated with the tactile signal as the corresponding signal. That is, the lower the frequency of the conversion source signal, the lower the frequency of the corresponding tactile signal will be. Therefore, the conversion unit 132 can allocate a frequency corresponding to each burst based on the conversion source signal.
[0114] For example, the conversion unit 132 allocates the frequency corresponding to each burst based on the weighted average frequency of the signal in the interval corresponding to the burst in the conversion source signal. That is, in order to provide frequency information for each burst, the conversion unit 132 calculates the weighted average frequency by the frequency power in the frame where the object burst is located, for example, as shown in the following formula (2).
[0115]
[0116] In the above formula (2), i is an index representing each burst. i ” is the set of time frames in which each burst is located.
[0117] Here, the tactile characteristics of the burst change depending on the time length, and therefore, a mechanism for separately extracting bursts with different lengths may be included in order to express such changes in characteristics. That is, the conversion unit 132 may convert the conversion source signal into an intermediate representation signal, which includes a first burst with a long output duration and a second burst with a shorter output duration compared to the first burst. For example, burst length classification can be achieved by changing the filter length in the above-mentioned median filter. That is, when the filter length is reduced, only short bursts are extracted, and when the filter length is increased, longer bursts are extracted. Figure 8 In the example shown, burst 80 is an example of a short burst, while burst 82 is an example of a long burst.
[0118] Through the above processing, the conversion unit 132 can convert the position information, length information “T i"、Frequency information "f i ^ave" and power information "V( t∈Ti )^a" is set as the burst parameter of the i-th burst. Figure 8 Burst 82 is shown as an example of the i-th burst, the power of the burst is indicated as “V(t∈T i )^a", which is the rise amount, and the information about the burst also includes frequency information "f i ^ave". Note, Figure 8 The “V t ^sm" corresponds to Figure 4 The waveform 59 shown in FIG. 5 is a waveform 59, and the frequency information “f i ^ave" corresponds to area 56. That is, the conversion unit 132 extracts the converted source signal which is visualized as Figure 4 The information of the burst parameters in the waveform 50 shown in FIG. Figure 4 and Figure 8 In the example shown, the burst parameters are represented by waveforms, but in practice, the burst parameters in the intermediate representation signal and the parameters to be described below will be recorded as encoded numerical information.
[0119] Next, the noise component ratio in the intermediate representation signal will be described. As described above, the conversion unit 132 separates the conversion source signal into a harmonic component and a noise component, and the parameter indicating the noise component ratio is the ratio of the noise component included in each time frame in the conversion source signal. In a noisy sound such as the sound of wind, the noise ratio is high, and in a sound with many harmonic components such as the sound of a wind instrument, the noise ratio is low.
[0120] The calculation of the noise ratio parameter can be performed using the separation of the noise component and the harmonic component exemplified in the signal separation for sound effects, etc. Here, when the spectrum diagram of each separated component is represented by "X tf ^n" (n represents the noise component) and "X tf ^h" (h represents the harmonic component), the noise ratio parameter "N t ” is expressed by the following formula (3).
[0121]
[0122] The noise ratio parameter represented by the above formula (3) is useful for purposes such as adjusting the roughness of a tactile signal.
[0123] Next, the frequency of the noise component in the intermediate representation signal will be described. As described above, the conversion unit 132 may use the frequency corresponding to the noise component as one of the parameters of the intermediate representation signal. The noise component frequency parameter is, for example, a parameter for determining the noise component when outputting a haptic signal.
[0124] For example, the conversion unit 132 may calculate the frequency range of the bandpass filter calculated from the noise component included in the conversion source signal as the frequency parameter of the noise component. The noise component frequency parameter is useful for expressing, for example, a rough tactile sensation (e.g., irregular vibration) corresponding to the conversion source signal.
[0125] Next, the frequency of the harmonic component in the intermediate representation signal will be described. As described above, the conversion unit 132 may use the frequency corresponding to the harmonic component as one of the parameters of the intermediate representation signal. The harmonic component frequency parameter is, for example, a parameter for determining the harmonic component output as a haptic signal.
[0126] For example, the conversion unit 132 may calculate the frequency of the sine wave extracted as the harmonic component included in the conversion source signal as the frequency parameter of the harmonic component. The harmonic component frequency parameter is useful for expressing a vibrotactile sensation (eg, regular vibration) corresponding to the conversion source signal, for example.
[0127] As described above, based on the conversion source signal, the conversion unit 132 assigns a frequency corresponding to each of the harmonic component and the noise component. With this configuration, the conversion unit 132 can generate an intermediate representation signal that can accurately reproduce the tactile presentation desired by the creator 20 or the like.
[0128] By returning to Figure 2 The description is continued. The transmission unit 133 transmits the intermediate representation signal converted by the conversion unit 132 to a processing unit in a subsequent stage. For example, the transmission unit 133 transmits the intermediate representation signal to the decoding device 200 which decodes the intermediate representation signal.
[0129] (1-3. Conversion Processing Procedure According to Embodiment)
[0130] Fig. 9 The flow of the conversion process according to this embodiment is shown. Fig. 9 is a flowchart showing the procedure of the conversion process according to this embodiment.
[0131] like Fig. 9 As shown, first, the conversion device 100 acquires a conversion source signal (step S101). Subsequently, the conversion device 100 performs a signal separation process on the acquired conversion source signal (step S102).
[0132] Next, the conversion device 100 extracts a tactile representation from each separated signal (step S103). The tactile representation is an element that can be a source of tactile presentation to the user, such as the above-mentioned burst, noise component and harmonic component. The conversion device 100 integrates the extracted tactile representation (step S104).
[0133] The conversion device 100 converts the conversion source signal into an intermediate representation signal based on the integration information (step S105). Thereafter, the conversion device 100 transmits the intermediate representation signal to a device (decoding device 200, etc.) capable of performing decoding processing on the intermediate representation signal via a network or the like (step S106).
[0134] (1-4. Configuration of Decoding Device According to Embodiment)
[0135] Next, the decoding process of the intermediate representation signal will be described. Fig.10 The configuration of the decoding device 200 according to this embodiment is described. Fig.10 : is a diagram showing a configuration example of the decoding device 200 according to this embodiment.
[0136] like Figure 2 As shown, the decoding device 200 includes a communication unit 210, a storage unit 220, and a control unit 230. Note that the decoding device 200 may include an input device (e.g., a touch panel, a keyboard, or a pointing device such as a mouse, a voice input microphone, or an input camera (line of sight input or gesture input)) that performs various operation inputs from a user who operates the decoding device 200, etc.
[0137] The communication unit 210 is implemented by, for example, a NIC. The communication unit 210 is connected to the network N in a wired or wireless channel, and transmits / receives information to / from devices such as the tactile presentation device 10 and the conversion device 100 via the network N.
[0138] The storage unit 220 is implemented by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 220 stores the acquired intermediate representation signal, the decoded haptic signal, and the like.
[0139] The control unit 230 is implemented by executing a program (e.g., a decoding program according to the present embodiment) stored in the decoding device 200 using a RAM or the like as a work area by a CPU or an MPU or the like. In addition, the control unit 230 may be a controller and may be implemented by using an integrated circuit such as an ASIC or an FPGA.
[0140] like Fig.10 As shown, the control unit 230 includes an acquisition unit 231, a generation unit 232, and an output control unit 233, and implements or executes the functions and operations of the information processing described below. The internal configuration of the control unit 130 is not limited to Fig.10 The configuration shown in , and may be another configuration as long as it is a configuration that executes the information processing described below.
[0141] The acquisition unit 231 acquires various data for processing by the processing unit in the next stage. For example, the acquisition unit 231 acquires any signal recording information about the expression of tactile presentation. Specifically, the acquisition unit 231 acquires the intermediate representation signal obtained by converting the conversion source signal by the conversion device 100.
[0142] That is, the acquisition unit 231 acquires an intermediate representation signal, which includes a burst, a harmonic component, a noise component, and information indicating the ratio between the harmonic component and the noise component as parameters, wherein the burst is information indicating a sharp rise in the output value, the harmonic component is information having a fundamental frequency, and the noise component is information not having a fundamental frequency.
[0143] In addition, the acquisition unit 231 acquires characteristic information about the output unit that performs tactile presentation based on the intermediate representation signal, etc. The output unit can be interpreted by the tactile presentation device 10. That is, the acquisition unit 231 acquires information such as the characteristics of the element that actually vibrates based on the tactile signal and the characteristics of the tactile presentation device 10 that controls the element. The characteristic information may include information such as the part of the human body on which the output unit of the tactile presentation device 10 is worn, the number of output units included in the tactile presentation device 10, and the like.
[0144] The generating unit 232 generates a tactile signal as a signal for controlling the output of the output unit based on the intermediate representation signal acquired by the acquiring unit 231. For example, the generating unit 232 decodes the intermediate representation signal acquired by the acquiring unit 231, and adjusts the decoded signal based on the characteristic information, thereby generating the tactile signal. Alternatively, the generating unit 232 adjusts the signal obtained by decoding the intermediate representation signal based on the characteristic information, thereby generating the tactile signal. That is, the generating unit 232 has a function of decoding the intermediate representation signal. As described above, when the intermediate representation signal is decoded into the tactile signal as a decoding process, the generating unit 232 may adjust the intermediate representation signal itself based on the characteristic information, etc., and then synthesize (generate) the intermediate representation signal into the tactile signal, or may first decode the intermediate representation signal into the tactile signal, and then adjust the tactile signal based on other information such as the characteristic information. Note that the generating unit 232 does not necessarily use all the acquired characteristic information, but may, for example, use only the minimum information required for output (such as information for identifying the output unit used as the output destination) as the characteristic information.
[0145] Fig.11 is a conceptual diagram showing decoding of an intermediate representation signal according to this embodiment. Fig.11 In the example shown, the acquisition unit 231 acquires the intermediate representation signal 40 and the device information (characteristic information) 42. Fig.11As shown, the intermediate representation signal 40 includes parameters such as burst, time variation and noise components. In addition, the device information 42 includes information such as frequency characteristics of the output unit and the position and number of the output units provided in the tactile presentation device 10.
[0146] The generation unit 232 decodes the intermediate representation signal 40 based on the acquired information, thereby generating a tactile signal 300 for actually driving the output unit. Fig.11 As shown, the generating unit 232 further uses the device information to decode the intermediate representation signal 40 , so a plurality of haptic signals corresponding to the respective haptic presentation devices 10 can be generated from one intermediate representation signal 40 .
[0147] With this configuration, the generation unit 232 can present an appropriate sensation to the user regardless of the characteristics of the device to be output. Specifically, the generation unit 232 processes the intermediate representation in a state where the device-dependent information is removed, so that even when the actuator to be used for output by the creator 20 is changed, an appropriate tactile signal can be generated. In addition, the generation unit 232 appropriately decodes the common data published in a device-independent state for each device, so that the amount of data related to data publishing, etc. can be reduced.
[0148] Although the embodiment describes an example in which the generation unit 232 generates a haptic signal based on the intermediate representation signal 40 generated by the conversion device 100, information to be decoded by the generation unit 232 is not limited to the intermediate representation signal 40. That is, the generation unit 232 may generate a haptic signal based on the method described below as long as the signal is a signal including an axis with a high level of abstraction based on some kind of haptic representation (for example, a signal obtained by encoding an expression based on human perception such as roughness, hardness, and intensity).
[0149] Fig.12 An example of an intermediate representation signal 40 processed by the generation unit 232 is shown. Fig.12 is a diagram showing an example of an intermediate representation signal 40 which is a target of decoding processing according to this embodiment. Fig.12 The intermediate representation signal 40 shown in FIG. 1 is a signal generated by the conversion device 100 and is related to Figure 4 The intermediate representation signal 40 is the same as the intermediate representation signal 40 shown in . That is, the intermediate representation signal 40 does not include device-related information, and includes the ratio of the burst, noise component and the harmonic component, the frequency of these components, which are parameters suitable for human perception. Based on the parameters included in the intermediate representation signal 40, the generation unit 232 generates a tactile signal to be actually output by the tactile presentation device 10.
[0150] The following will refer to Figures 13 to 25Details of the decoding process and the generation process according to this embodiment are described. Fig.13 is a diagram (1) showing an example of a decoding process according to an embodiment.
[0151] Fig.13 The example of illustrative embodiments shows a decoding process from the intermediate representation signal 40 based on burst parameters. Fig.13 A waveform 50 is shown that visually conveys the burst parameters.
[0152] In the waveform 50, a waveform 310 including two arbitrary bursts will be described as an example. The waveform 310 is a simplified representation of only two triangular waves, each indicating a burst. The vertical axis of the waveform 310 schematically indicates the intensity of the burst. The waveform 312 indicates the level of the main frequency of the harmonic component in the intermediate representation signal 40.
[0153] As an example, when decoding waveform 50, generation unit 232 performs signal processing so that the higher the strength of the burst, the larger the amplitude in the decoded signal, and the higher the frequency of the original signal, the higher the frequency in the decoded signal. For example, generation unit 232 decodes waveform 310 and waveform 312 into waveforms such as Fig.13 A waveform such as waveform 314 shown in FIG. Notice that waveform 314 indicates the strength of the signal by the height of the amplitude, and indicates the level of the frequency by the number of repetitions of the amplitude.
[0154] Next, we will refer to Fig.14 Another example is described. Fig.14 2 is a diagram showing an example of a decoding process according to an embodiment. Fig.14 The example of shows a decoding process based on a noise component in the intermediate representation signal 40. Fig.14 A waveform 60 representing the ratio between the noise component and the harmonic component and a waveform 64 representing the frequency of the noise component are shown.
[0155] Waveform 320 is a portion of waveform 60 schematically showing a specific time period of change in only the noise ratio. The vertical axis of waveform 320 indicates the ratio of the noise component, indicating, for example, that the larger the value of the vertical axis, the more noise components there are. In addition, waveform 322 indicates the change in the frequency of the noise component in the amount of time corresponding to waveform 320. The vertical axis of waveform 322 indicates the level of the frequency of the noise component.
[0156] As an example, the generating unit 232 sets the amplitude of the noise component when decoding the waveform 60 and the waveform 64 so that the larger the overall volume and noise ratio, the larger the amplitude will be. In addition, the generating unit 232 sets it so that the higher the frequency of the noise component, the higher the frequency of the noise component of the decoded signal. As described above, the overall volume is a parameter indicating the amplitude of the output signal changing over time, and in the case of the signal before decoding (intermediate representation signal 40), the overall volume corresponds to the vertical axis of the waveform 50.
[0157] For example, the generation unit 232 decodes the waveform 320 and the waveform 322 into a Fig.14 The waveform 324 shown in FIG. 324 indicates the magnitude and frequency of the noise component in the tactile signal. Note that the waveform 324 indicates the strength (magnitude) of the signal by the height of the amplitude, and indicates the level of the frequency by the number of repetitions of the amplitude.
[0158] Next, we will refer to Fig.15 Another example is described. Fig.15 3 is a diagram showing an example of decoding processing according to this embodiment. Fig.15 The example of shows a decoding process based on the harmonic components of the intermediate representation signal 40. Fig.15 A waveform 60 indicating the ratio between the noise component and the harmonic component and a waveform 62 indicating the frequency of the harmonic component are shown.
[0159] The waveform 330 is a portion of the waveform 60 schematically showing only a change in the noise ratio for a specific time period. The vertical axis of the waveform 330 represents the ratio of the noise component, for example, the larger the value of the vertical axis, the more noise components there are. In addition, the waveform 332 represents the change in the frequency of the harmonic component in the amount of time corresponding to the waveform 330. The vertical axis of the waveform 332 represents the level of the frequency of the harmonic component.
[0160] As an example, the generation unit 232 sets the amplitude of the harmonic component when the waveform 60 and the waveform 62 are decoded so that the smaller the overall volume and noise ratio, the larger the amplitude will be. In addition, the generation unit 232 is set so that the higher the frequency of the harmonic component, the higher the frequency of the harmonic component of the decoded signal.
[0161] For example, the generation unit 232 decodes the signal having the waveform 330 and the waveform 332 into a signal such as Fig.15 A waveform such as waveform 334 shown in FIG. 334 indicates the magnitude and frequency of the harmonic components in the haptic signal. Note that waveform 334 indicates the strength (magnitude) of the signal by the height of the amplitude, and indicates the level of the frequency by the number of repetitions of the amplitude.
[0162] The above describes an example in which the generation unit 232 extracts information indicated by the waveform 314, the waveform 324, and the waveform 334 from the parameters included in the intermediate representation signal 40. The generation unit 232 integrates this information to generate a tactile signal. Fig.16 This process is described. Fig.16 : is a diagram showing an example of generation processing according to this embodiment.
[0163] exist Fig.16 In the example shown, the generation unit 232 integrates the information indicated by the waveform 314, the waveform 324, and the waveform 334. Specifically, the generation unit 232 combines the amplitudes along the time axis corresponding to the three waveforms.
[0164] Furthermore, the generation unit 232 integrates the overall quantity in the intermediate representation signal 40 . Fig.16 The waveform 336 shown in FIG. 4 illustrates the magnitude of a volume in the intermediate representation signal 40 as a waveform along the time axis.
[0165] The generation unit 232 combines a waveform obtained by integrating information represented by the waveform 314 , the waveform 324 , and the waveform 334 with the overall quantity, thereby generating a haptic signal represented by the waveform 340 . Fig.16 The example of illustrative embodiments schematically illustrates the amplitude (output value) and frequency included in a haptic signal using waveform 340 .
[0166] Using the above-mentioned processing, the generation unit 232 can generate a haptic signal from the intermediate representation signal 40. Here, the generation unit 232 can also use the device information and various types of information to generate a haptic signal with higher reproducibility. Figures 17 to 25 Describe an example of an extension like this.
[0167] Fig.17 is a diagram ( 1 ) illustrating an example of adjustment processing based on the characteristics of the tactile presentation device 10 . Fig.17 The graph 350 illustrated in FIG. 3 illustrates the frequency characteristics in a specific tactile presentation device 10. The example illustrated in the graph 350 indicates that the tactile presentation device 10 has a unique peak around 70 Hz. In the case where data owned by a manufacturer of an actuator or the like can be acquired in advance as characteristic information, the acquisition unit 231 acquires such data. When there is no data indicating characteristic information, the acquisition unit 231 can acquire the characteristic information of the tactile presentation device 10 by transmitting a predetermined test signal or the like and observing the result (reaction) of the test signal or the like.
[0168] The generation unit 232 may perform a predetermined adjustment process with reference to the characteristic information such as the graph 350 acquired by the acquisition unit 231 . Fig.17The waveform 352 shown in FIG. 3 schematically shows the waveform of the tactile signal before adjustment. The vertical axis of the waveform 352 and the waveform 354 represents the amplitude, and the horizontal axis represents the frequency. As shown in the waveform 352, the signal before adjustment is uniform and has nothing to do with the frequency.
[0169] Referring to the characteristic information shown in the graph 350 , the generation unit 232 adjusts the waveform 352 to a waveform 354 . Fig.17 The waveform 354 shown in FIG3 schematically shows the waveform of the tactile signal after adjustment by the generation unit 232. As shown in the waveform 354, in the adjusted signal, the amplitude near 70 Hz having a peak in the graph 350 is smaller than the amplitude of the waveform 352, and the amplitudes of other frequency bands are larger than the amplitude of the waveform 352.
[0170] In this way, the generation unit 232 adjusts the output value for each frequency in the decoded signal based on the frequency characteristics of the output unit obtained as characteristic information, thereby generating a tactile signal. That is, the generation unit 232 adjusts the output of the tactile signal, for example, based on information indicating a specific frequency characteristic of the tactile presentation device 10, so that the actual output value becomes substantially constant. In other words, as post-processing of decoding, the generation unit 232 corrects the tactile signal so as to reduce the output at a frequency where vibration is likely to occur and increase the output at a frequency where vibration is less likely to occur, as a characteristic of the device. Through this operation, the generation unit 232 can achieve the desired output of the original intermediate representation signal regardless of the characteristics of the device or actuator.
[0171] Note that the characteristic information also includes information indicating not only a difference in frequency but also a difference in response time (eg, a time interval from application of voltage to occurrence of vibration). The generation unit 232 may also process such characteristic information by adjusting the haptic signal.
[0172] Fig.18 2 is a diagram showing an example of an adjustment process based on the characteristics of the tactile presentation device 10. A graph 360 indicates the time response characteristics of a specific tactile presentation device 10. Specifically, the graph 360 indicates the time required for the amplitude to reach the expected output value after the voltage is applied, and the time required for the amplitude to reach 0 after the voltage is turned off. Fig.18 360 , but the time response characteristics vary according to the frequency. In this example, it is assumed that the tactile presentation device 10 corresponding to the graph 360 has a characteristic of fast response at 200 Hz and slow response at 50 Hz. As a typical trend, the closer the frequency is to the resonant frequency of the vibrator, the slower the time response of the vibrator.
[0173] Fig.18The waveform 362 shown in the diagram schematically illustrates a signal input to the tactile presentation device 10 shown in the graph 360. The waveform 362 includes a burst 364 and a burst 368 representing the amplitude (referred to as a burst for convenience) of vibrating the tactile presentation device 10. In this example, the burst 364 has a frequency of 200 Hz (indicating the amount of vibration), and the burst 368 has a frequency of 50 Hz.
[0174] Without performing specific adjustments, as shown in the lower portion of waveform 362 , the amplitude corresponding to burst 364 and the amplitude corresponding to burst 368 produce a haptic signal as shown.
[0175] Here, the generation unit 232 performs a predetermined adjustment process. The waveform 372 represents a signal after the waveform 362 has been adjusted by the generation unit 232. For example, the generation unit 232 adjusts the rise and decay timing of the decoded signal.
[0176] Specifically, the generation unit 232 shifts the amplitude of the burst 364 corresponding to the frequency with the fast response to a slightly earlier time. Fig.18 As shown, the amplitude 374 in the waveform 372 will be a waveform in which the amplitude becomes 0 earlier than the end time point 366 of the burst 364. In addition, the generation unit 232 shifts the amplitude of the burst 368 corresponding to the frequency with slow response to a later time. Fig.18 As shown, the amplitude 376 in the waveform 372 is a waveform whose amplitude becomes 0 later than the end time point 370 of the burst 368. These operations enable the generation unit 232 to perform output suitable for perception according to the characteristics of the tactile presentation device 10.
[0177] In addition to the adjustment regarding time, the generation unit 232 may also adjust the magnitude of the amplitude (ie, the input voltage to the tactile presentation device 10). Fig.19 Describe an example of this situation. Fig.19 3 is a diagram showing an example of adjustment processing based on the characteristics of the tactile presentation device 10 .
[0178] Fig.19 Graph 360 and waveform 362 are shown again. Here, the generation unit 232 adjusts the amplitude of the waveform 362 to generate a tactile signal for performing an output suitable for human perception. For example, for a frequency with a high response speed, the generation unit 232 can reproduce an output suitable for human perception by slightly suppressing the magnitude of the amplitude. In addition, for a frequency with a low response speed, the generation unit 232 can reproduce an output suitable for human perception by slightly increasing the magnitude of the amplitude.
[0179] Waveform 384 indicates a waveform corresponding to the haptic signal adjusted by generation unit 232. That is, generation unit 232 slightly attenuates the input voltage for burst 364 corresponding to the frequency with fast response. Fig.19 In the example of , the generation unit 232 attenuates the output value 380 corresponding to the burst 364. Therefore, the amplitude 386 corresponding to the burst 364 has a slightly lower output value compared to the waveform 362. In addition, the generation unit 232 slightly amplifies the input voltage for the burst 368 corresponding to the frequency with slow response. Fig.19 In the example of FIG. 3 , generation unit 232 amplifies output value 382 corresponding to burst 368 . Therefore, amplitude 388 corresponding to burst 368 has a slightly increased output value compared to waveform 362 .
[0180] In this way, the generation unit 232 adjusts the output timing or output value of the decoded signal based on the time response characteristic of the output unit acquired as the characteristic information, thereby generating a tactile signal. This allows the generation unit 232 to achieve the ideal output possessed by the original intermediate representation signal corresponding to the time response characteristic of the tactile presentation device 10.
[0181] Incidentally, for a signal corresponding to a frequency with a slow time response, the generation unit 232 can perform adjustment processing of inputting a signal with an opposite phase so as to converge the vibration earlier. With this configuration, the generation unit 232 can suppress the vibration so that the output in the tactile presentation device 10 with a slow time response to an ideal time can be controlled.
[0182] Meanwhile, in the case where the conversion source signal is accompanied by a sharp frequency change, it is conceivable to use a method of retaining information in the intermediate representation signal itself in the encoding process rather than in the decoding process so that the change can be reproduced by tactile representation.
[0183] Reference Fig. 20 Describe an example of this situation. Fig. 20 : is a diagram showing an extended example of the conversion process according to this embodiment. Fig. 20 The illustrated processing is performed by, for example, the conversion unit 132 of the conversion device 100 .
[0184] Fig. 20The waveform 390 shown in represents a conversion source signal having a magnitude record of the amplitude. In this case, by using the above conversion process to represent the magnitude of the amplitude, information about the intermediate representation signal can be maintained as a burst parameter. On the other hand, the waveform 392 represents a conversion source signal having no magnitude record of the amplitude at time 394 or time 396 but a record of a sharp frequency change. In this case, there is a possibility that the burst parameter is not recorded in the above conversion process. However, since the sharp frequency change has a great influence on human perception, it is desirable to reproduce the frequency change as a tactile representation.
[0185] Therefore, in the case where the frequency of the conversion source signal changes dramatically, for example, in the case where the fundamental frequency changes greatly within a very short time frame, the conversion device 100 can record the change as a burst parameter. Waveform 398 schematically shows information obtained by converting waveform 390 or waveform 392 into an intermediate representation signal.
[0186] In this way, the conversion unit 132 of the conversion device 100 can extract, as a burst, a section in which a frequency change exceeding a predetermined reference occurs within a predetermined time width in the frequency change per time unit in the conversion source signal. This enables the conversion device 100 to incorporate a steep frequency change as a burst parameter into the intermediate representation signal, so that an intermediate representation signal including a wider variety of tactile representations can be generated.
[0187] Next, the adjustment process in the case where a burst continues in the intermediate representation signal will be described. Fig.21 is a diagram showing an example of adjustment processing according to time changes.
[0188] In human perception, the time interval within which two sounds can be recognized as different sounds is empirically known. Fig.21 The waveform 400 in illustratively shows a signal including two bursts. In this case, the tactile signal is indicated by a waveform having two amplitude peaks, as indicated by waveform 402. However, when the time interval 404 between the two bursts is less than a predetermined time (approximately 50 ms), there is a possibility that a person recognizes the two sounds as one sound. Waveform 406 schematically shows a human perception detecting the tactile signal indicated by waveforms 400 and 402 as an output.
[0189] In this case, the tactile representation originally intended to present two bursts may be weakened. In order to deal with this situation, the generation unit 232 performs a predetermined adjustment process.
[0190] For example, when the decoding has generated a tactile signal as indicated by waveform 402, the generation unit 232 adjusts the tactile signal so as to shift the burst 410 of the two bursts forward to a slightly earlier time as indicated by waveform 408. In this way, by separating the two bursts with a time interval (e.g., 50 ms or longer) at which a person can perceive different sounds, the generation unit 232 performs the adjustment so as not to combine the two bursts into one burst. The generation unit 232 can perform the adjustment of the burst 410 not only by shifting the burst to an earlier time, but also by slightly increasing the amplitude of the burst. This also enables the generation unit 232 to increase the sensitivity of the burst to human perception.
[0191] Furthermore, as another adjustment example, the generation unit 232 may adjust the haptic signal so as to slightly shift the burst 414 , which is the latter of the two bursts, to a later time, as shown in the waveform 412 .
[0192] As described above, the generation unit 232 adjusts the signal obtained by decoding the intermediate representation signal based on the parameters presented according to the perceptual sensitivity of the person to whom the output unit is to output the tactile presentation, thereby generating the tactile signal. Specifically, the generation unit 232 adjusts the decoded signal based on the parameters pre-set according to the perceptual sensitivity of the person to whom the output unit is to output the tactile presentation.
[0193] As an example, after the generation unit 232 has decoded a signal including a plurality of output intervals (such as Fig.21 In the case of the waveform 400 shown in ), wherein the plurality of output intervals are intervals to be isolated and output in the intermediate representation signal, and when the time intervals between the plurality of output intervals are within a predetermined time (e.g., 50 ms) set as a parameter, the generation unit makes an adjustment to widen the time intervals between the plurality of output intervals, and generates a tactile signal. Alternatively, the generation unit 232 may perform an adjustment to increase one of the output values corresponding to the plurality of output intervals, or perform an adjustment to extend one of the output time lengths corresponding to the plurality of output intervals.
[0194] Next, another example of a process of adjusting a haptic signal according to human perception will be described. Fig. 22 is a diagram (1) showing an example of adjustment processing adapted to human perception.
[0195] The graph 420 schematically shows the intensity perceived by a person when a tactile signal of the same frequency is output from the tactile presentation device 10. As shown in the graph 420, when a human perceives a tactile signal, the person has a strong perception of the signal immediately after feeling the signal, but when the signal lasts for more than a certain presentation time (e.g., 1 second), the sensitivity of the signal decreases. Therefore, the generation unit 232 adjusts the vibration intensity according to the perception characteristics so that the intensity perceived by humans is achieved as expected. For example, as shown in the graph 422, the generation unit 232 can adjust the amplitude of the tactile signal to gradually decay.
[0196] In this way, in the case where a specific frequency and output value are output in the decoded signal for more than a predetermined time (eg, 1 second) set by a parameter, the generation unit 232 may perform adjustment to change the frequency or output value according to time.
[0197] In addition to the above examples, as a type of characteristic information of the device, the generation unit 232 can generate a tactile signal by adjusting the frequency or output value of the decoded signal based on information related to the human body part to which the tactile presentation will be output by the output unit. For example, the sensitivity varies depending on the output destination of the tactile signal, which is indicated by the fact that the fingertips of a person have high sensitivity to sounds of about 200 Hz, and the generation unit 232 can appropriately adjust the tactile signal according to the sensitivity. In this case, by pre-maintaining data related to the frequency characteristics of a person corresponding to each body part, etc., and using the maintained information as a parameter, the generation unit 232 can make adjustments suitable for each body part.
[0198] Next, another example of a process of adjusting a haptic signal according to human perception will be described. Fig.23 FIG. 2 is a diagram showing an example of adjustment processing suitable for human perception.
[0199] The frequency includes a frequency that is highly perceptible to humans and a frequency that is not highly perceptible to humans. Therefore, the generation unit 232 can shorten the output time length of the signal or reduce the amplitude of the signal corresponding to the highly perceptible frequency. That is, the generation unit 232 can adjust the decoded signal based on a preset parameter based on the human perception sensitivity related to the frequency in the human perception characteristics.
[0200] Graph 430 shows an example of the relationship between frequency and vibration intensity. For example, assume that frequency band 432 in graph 430 includes frequencies that are highly perceptible to humans. In this case, as shown in adjustment process 436, generation unit 232 reduces the vibration intensity of the signal corresponding to frequency band 432. On the other hand, assume that frequency band 434 in graph 430 includes frequencies that are not highly perceptible to humans. In this case, as shown in adjustment process 436, generation unit 232 amplifies the vibration intensity corresponding to frequency band 434. This allows generation unit 232 to more appropriately implement the tactile representation desired by creator 20 or the like.
[0201] Next, another example of a process of adjusting a haptic signal according to human perception will be described. Fig.24 FIG. 3 is a diagram showing an example of adjustment processing suitable for human perception.
[0202] It is known empirically that when the presentation of signals of similar frequencies continues for a certain length of time, a person tends to have a negative feeling as a perception. Therefore, in the case where the tactile signal includes the presentation of signals having similar frequencies, the generation unit 232 may perform adjustment to mix a noise component into the signal. Fig.24 The waveform 440 in FIG. 4 shows an example including signals having similar frequencies being presented continuously. The signal of the waveform 440 is a signal having a fundamental frequency as shown in the graph 442 (the horizontal axis of the graph 442 represents frequency, and the vertical axis represents amplitude).
[0203] When such a signal is observed, the generation unit 232 mixes a noise component at the time of decoding to make adjustments so as not to allow a signal having a similar frequency to continue. Fig.24 The waveform 444 in illustratively shows the signal adjusted by the generation unit 232. In addition, the frequency of the signal after adjustment is shown in the graph 446. In this case, by superimposing the noise component while avoiding the change of the fundamental frequency of the original signal, the generation unit 232 can adjust the signal to a signal that does not cause negative feelings in humans without changing the basic characteristics of the signal.
[0204] Next, we will refer to Fig.25 Another example of adjustment is described. Fig.25 2 is a diagram showing an example of adjustment processing related to signal superposition.
[0205] Fig.25 The waveform 450 shown in FIG. 4 shows an example in which a signal decoded from a burst parameter and a signal decoded from a noise component or a harmonic component are superimposed. In the interval 452 and the interval 454 of the waveform 450, the signals are superimposed with similar amplitudes.
[0206] Here, when the burst element is clearly output, the tactile signal is considered to have a significant feeling and an enhanced tactile presentation effect. Therefore, the generation unit 232 can refer to the tactile signal, especially the interval including the burst, and in the case where the signal other than the burst overlaps with the range before and after the burst interval (for example, within 50ms), and can perform adjustments such as reducing the signal amplitude before and after the burst signal to emphasize the burst signal. In addition, in the case where the burst interval lasts for multiple times as in waveform 450, the generation unit 232 can mute the signal superimposed with the earlier burst. This allows the generation unit 232 to more effectively perform the tactile presentation corresponding to the burst.
[0207] Fig.25 The waveform 460 in FIG. 4 represents the signal after being adjusted by the generation unit 232. Fig.25 In the example of , the generation unit 232 removes the noise and harmonic components in the interval 462 other than the burst, that is, performs adjustment so as to highlight only the signal corresponding to the burst. In addition, the generation unit 232 also reduces the noise and harmonic components in the interval 464, that is, performs adjustment so as to highlight only the signal corresponding to the burst. In this way, in the case where the information decoded from the burst and the information decoded from the parameter other than the burst interfere with each other, the generation unit 232 can perform adjustment to attenuate the output value decoded from the parameter other than the burst.
[0208] An example of the adjustment process according to the present embodiment has been described above. Note that the above-mentioned adjustment process can be applied entirely or selectively at the time of decoding.
[0209] Will return to Fig.10 The output control unit 233 outputs the tactile signal generated by the generation unit 232 to the tactile presentation device 10. Specifically, the output control unit 233 sends the tactile signal to the tactile presentation device 10 via the network, and controls to output the tactile presentation on the tactile presentation device 10.
[0210] (1-5. Procedure of Decoding Process According to Embodiment)
[0211] Fig.26 The flow of the conversion process according to this embodiment is shown. Fig.26 is a flowchart showing the procedure of the decoding process according to this embodiment.
[0212] First, the decoding device 200 acquires an intermediate representation signal represented by parameters having a high level of abstraction suitable for human perception (step S201 ) Then, the decoding device 200 acquires device information including frequency characteristics and the like of the tactile presentation device 10 (step S202 ).
[0213] Subsequently, the decoding device 200 starts generating a haptic signal corresponding to the device to which the haptic signal is to be output (step S203 ). At this time, the decoding device 200 determines whether there is a difference from a reference characteristic in the device to which the haptic signal is to be output (step S204 ).
[0214] The decoding device 200 refers to the device information, and when it is determined that there is a specific difference in the characteristics (step S204; yes), the decoding device 200 determines the parameters to be used for decoding according to the characteristics (step S205). The parameters in this case are not limited to amplitude, frequency, etc., and also include adjustment parameters in the above-mentioned adjustment processing (values indicating the degree of increase / decrease of amplitude, degree of time shift, etc.).
[0215] After determining the parameters for decoding, or when there is no difference with the reference characteristics (step S204; No), the decoding device 200 generates a haptic signal (step S206). Thereafter, the decoding device 200 may output the haptic signal to the haptic presentation device 10, or may store the haptic signal in the storage unit 220.
[0216] (2. Modification of Embodiment)
[0217] The information processing according to the above-described embodiment may include various modifications. Modifications of this embodiment will be described below.
[0218] (2-1. Equipment configuration)
[0219] The above-mentioned conversion device 100 and decoding device 200 do not have to be independent devices, and can be provided as a processing unit in an existing tactile presentation process. In this case, the processing unit corresponding to the conversion device 100 and the decoding device 200 is incorporated into the existing tactile presentation process. This point will be referred to as Fig. 27 Give a description. Fig. 27 is a diagram showing the flow of a tactile sense presentation process according to a modification.
[0220] Fig. 27 The example shown in includes a process of outputting a tactile signal in an existing format in a tactile presentation process to the tactile presentation device 10 through a conversion process and a decoding process according to an embodiment. In this case, a tactile signal in an existing format is acquired by a tactile signal encoding device 500 including an encoding unit that performs a conversion process according to an embodiment (step S301). Thereafter, the encoding unit generates an intermediate representation signal through a conversion process according to the embodiment (step S302).
[0221] Subsequently, the encoding unit sends the intermediate representation signal to the tactile signal decoding device 510 including a decoding unit that performs the decoding process according to the present embodiment (step S303). The decoding unit generates a tactile signal from the intermediate representation signal and outputs the generated tactile signal to the tactile presentation device 10 (step S304).
[0222] exist Fig. 27 In the example of , the encoding unit and the decoding unit may be incorporated into the same device, or the encoding unit and the decoding unit may be incorporated into the tactile presentation device 10. That is, the conversion process and the decoding process according to the embodiment may be incorporated into a series of tactile presentation processes as the encoding process and the decoding process regardless of the device configuration. For example, Fig. 27 The encoding unit and the decoding unit shown in FIG. 1 may be provided as a plug-in that operates as software in the tactile presentation device 10 .
[0223] (3. Other embodiments)
[0224] The processing according to each of the above-described embodiments may be performed in various different forms (modifications) in addition to each of the above-described embodiments.
[0225] In addition, in each process described in the above embodiments, all or part of the process described as being automatically performed may be performed manually, or the process described as being manually performed may be automatically performed by a known method. In addition, unless otherwise specified, the process, specific name, and information including various data and parameters described in the above description or drawings may be changed in any way. For example, the various information shown in each drawing is not limited to the information shown.
[0226] In addition, each component of each device is provided as a functional and conceptual illustration, and therefore does not necessarily need to be physically configured as shown in the illustration. That is, the specific form of dispersion / integration of each device is not limited to those shown in the drawings, and all or part of it can be functionally or physically dispersed or integrated into any unit according to various loads and usage conditions.
[0227] Furthermore, the above-described embodiments and modifications can be appropriately combined within an achievable range without dealing with conflicts.
[0228] The effects described in this specification are merely examples, and thus, there may be other effects without being limited to the exemplified effects.
[0229] (4. Effects of the conversion device according to the present disclosure)
[0230] As described above, the conversion device (conversion device 100 in the embodiment) according to the present disclosure includes an acquisition unit (acquisition unit 131 in the embodiment) and a conversion unit (conversion unit 132 in the embodiment). The acquisition unit acquires a conversion source signal as a tactile signal source. The conversion unit converts the conversion source signal acquired by the acquisition unit into an intermediate representation signal represented by at least one parameter. For example, the conversion unit converts the conversion source signal into an intermediate representation signal represented by a single or multiple parameters corresponding to human perception.
[0231] In this way, the conversion device converts the signal for presenting tactile information into an intermediate representation signal expressed by parameters corresponding to human perception without depending on the output environment. With this configuration, the conversion device can generate a tactile signal independently of the output environment.
[0232] In addition, the conversion unit separates the conversion source signal into elements constituting the conversion source signal, and converts the separated signals into intermediate representation signals. For example, in the case where the conversion source signal is an acoustic signal in which a plurality of musical instrument sounds are superimposed, the conversion unit separates the acoustic signal into individual musical instrument sounds, and converts the separated signals into intermediate representation signals. In addition, the conversion unit separates the conversion source signal into a harmonic component which is a signal having a fundamental frequency and a noise component which is a signal not having a fundamental frequency.
[0233] In this manner, the conversion device first separates a plurality of separate signals included in a conversion source signal and then generates an intermediate representation signal, so that the intermediate representation signal appropriately reflecting the characteristics of each separated signal can be generated.
[0234] Furthermore, the acquisition unit acquires a haptic signal including characteristic information to be applied to a specific haptic presentation device as a conversion source signal. The conversion unit converts the haptic signal including characteristic information to be applied to the specific haptic presentation device into an intermediate representation signal represented by parameters not including the characteristic information.
[0235] In this way, the conversion device can replace the existing tactile signal with information that does not include device-dependent information, and therefore does not need to send or process information corresponding to a large number of devices to which output is to be performed. This allows the conversion device to effectively utilize resources related to data transmission and information processing.
[0236] In addition, the conversion unit converts the conversion source signal into an intermediate representation signal, which includes a burst, a harmonic component, a noise component, and information indicating the ratio between the harmonic component and the noise component as parameters, wherein the burst is information indicating a sharp rise in the output value, the harmonic component is information having a fundamental frequency, and the noise component is information not having a fundamental frequency.
[0237] In this way, by representing the signal with information such as the ratio of burst, noise and harmonic components, tactile presentation suitable for human perception can be performed, and the conversion device can achieve tactile presentation that can further resonate with human sensibility.
[0238] Furthermore, the conversion unit refers to a difference from a flattened value of the output value within a predetermined time width in an output value variation per time unit in the conversion source signal, and extracts a section where the reference value exceeds the reference output value as a burst.
[0239] In this way, the conversion device defines the burst based on the output value of the conversion source signal, so that a tactile presentation with an exciting feeling expected by the conversion source signal can be achieved.
[0240] In addition, the conversion unit converts the conversion source signal into an intermediate representation signal, the intermediate representation signal including a first burst having a long output duration and a second burst having a shorter output duration than the first burst. In addition, the conversion unit allocates a frequency corresponding to each burst based on the conversion source signal. For example, the conversion unit allocates a frequency corresponding to each burst based on a weighted average frequency of a signal in an interval corresponding to the burst in the conversion source signal.
[0241] In this way, by appending length and frequency information to the bursts, the conversion device can appropriately reproduce the tactile presentation desired by the converted source signal.
[0242] In addition, the conversion unit assigns a frequency corresponding to each of the harmonic component and the noise component based on the conversion source signal.
[0243] In this way, by allowing the noise component and the harmonic component to have frequency information, the conversion device can reproduce a tactile presentation that is difficult to reproduce using a simple time signal, such as a rough feeling desired by converting the source signal.
[0244] Furthermore, the conversion unit extracts, as the burst, a section in which a frequency change exceeding a predetermined reference appears within a predetermined time width among frequency changes per time unit in the conversion source signal.
[0245] In this way, by capturing sharp frequency changes as bursts, the conversion device can appropriately replace the event expressed by the converted source signal with a haptic presentation.
[0246] (5. Effects of the decoding device according to the present disclosure)
[0247] The decoding device (decoding device 200 in the embodiment) according to the present disclosure includes an acquisition unit (acquisition unit 231 in the embodiment) and a generation unit (generation unit 232 in the embodiment). The acquisition unit acquires: an intermediate representation signal, including a record of information related to a representation of tactile presentation; and characteristic information related to an output unit that performs tactile presentation based on the intermediate representation signal. The generation unit generates a tactile signal based on the intermediate representation signal acquired by the acquisition unit, the tactile signal being a signal that controls the output of the output unit. For example, the generation unit adjusts a signal obtained by decoding the intermediate representation signal based on the characteristic information, or decodes the intermediate representation signal after adjusting the intermediate representation signal based on the characteristic information, thereby generating a tactile signal.
[0248] In this way, the decoding device obtains an intermediate representation signal that does not include a record of device-related information, but only includes a record of information related to the representation of the tactile presentation, and thereafter decodes the signal based on the characteristic information of the output destination. This makes it possible to perform appropriate tactile presentation to various output destinations.
[0249] The acquisition unit acquires the frequency characteristic of the output unit as characteristic information. The generation unit adjusts the output value of each frequency in the decoded signal based on the frequency characteristic, thereby generating a tactile signal. The acquisition unit may acquire the time response characteristic of the output unit as characteristic information. The generation unit adjusts the output timing or output value of the decoded signal based on the time response characteristic, thereby generating a tactile signal. In addition, the acquisition unit may acquire information about the human body part to which the tactile presentation is output by the output unit as characteristic information. The generation unit adjusts the frequency or output value of the decoded signal based on the information about the human body part to which the tactile presentation is output by the output unit, thereby generating a tactile signal.
[0250] In this way, the decoding device adjusts the output value and the like based on the characteristic information of the output destination, so that tactile presentation reflecting the creator's intention of the original signal can be performed regardless of the mode of the output unit or the type of the tactile presentation device.
[0251] Furthermore, the generation unit adjusts the signal obtained by decoding the intermediate representation signal based on a parameter given by the perceptual sensitivity of a person to whom the output unit is to output the tactile presentation, thereby generating the tactile signal.
[0252] In this way, the decoding device can perform more effective tactile rendering by performing adjustments suitable for human perception.
[0253] As an example, in a case where the generation unit has decoded a signal including a plurality of output intervals, and in a case where the time intervals between the plurality of output intervals are within a predetermined time set as a parameter, the generation unit makes an adjustment to widen the time intervals between the plurality of output intervals, and generates a tactile signal, wherein the plurality of output intervals are intervals to be isolated and output in the intermediate representation signal. The generation unit may adjust any one of the output values corresponding to the plurality of output intervals to be amplified, and generate a tactile signal. In addition, the generation unit may perform an adjustment so as to extend any one of the output time lengths corresponding to the plurality of output intervals, and generate the tactile signal.
[0254] In this way, by adjusting the output value and timing of a signal or the like that is difficult to be felt by human perception, the decoding device can perform tactile presentation that does not drown out the creator's intention of the original signal.
[0255] In addition, when a specific frequency and output value are output in the decoded signal for more than a predetermined time set by a parameter, the generation unit performs adjustment to change the frequency or output value according to time. In addition, the generation unit adjusts the decoded signal based on a parameter that is preset based on human perception sensitivity related to frequency.
[0256] In this way, in the event that a signal is observed that is difficult for humans to feel or is more likely to produce a negative feeling in humans, the decoding device can generate a signal that is adjusted to address these problems so that a tactile presentation that is comfortable for humans can be performed.
[0257] In addition, the acquisition unit acquires an intermediate representation signal including a burst as a parameter, a harmonic component, a noise component, and information indicating a ratio between the harmonic component and the noise component, wherein the burst is information indicating a sharp rise in the output value, the harmonic component is information having a fundamental frequency, and the noise component is information not having a fundamental frequency. The generation unit decodes the information related to the output value and the frequency according to each parameter, thereby generating a tactile signal.
[0258] In this way, since the decoding device generates a haptic signal from an intermediate representation signal including parameters suitable for human perception, haptic presentation can be performed more intuitively.
[0259] Furthermore, in a case where information decoded from the burst and information decoded from the parameter other than the burst interfere with each other, the generation unit may perform adjustment to attenuate an output value decoded from the parameter other than the burst.
[0260] In this way, by adjusting information that interferes with each other in decoding, the decoding device can perform a noticeable tactile presentation of the output.
[0261] (6. Hardware Configuration)
[0262] For example, by having Fig.28 The computer 1000 of the illustrated configuration realizes information devices such as the conversion device 100 and the decoding device 200 according to the above-described respective embodiments. Hereinafter, the conversion device 100 according to the embodiment will be described as an example. Fig.28 10 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the conversion device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The respective components of the computer 1000 are interconnected by a bus 1050.
[0263] The CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400 to control each component. For example, the CPU 1100 expands the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.
[0264] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000 , and the like.
[0265] The HDD 1400 is a non-transitory computer-readable recording medium that records a program executed by the CPU 1100 , data used by the program, and the like. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of the program data 1450 .
[0266] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from other devices via the communication interface 1500 or transmits data generated by the CPU 1100 to other devices.
[0267] The input / output interface 1600 is an interface for connecting the input / output device 1650 to the computer 1000. For example, the CPU 1100 receives data from an input device such as a touch pad, a keyboard, a mouse, a microphone, or a camera via the input / output interface 1600. In addition, the CPU 1100 sends data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface for reading a program recorded on a predetermined recording medium, etc. Examples of media include optical recording media such as digital versatile disks (DVDs) or phase-change rewritable disks (PDs), magneto-optical recording media such as magneto-optical disks (MOs), magnetic tape media, magnetic recording media, and semiconductor memories.
[0268] For example, when the computer 1000 is used as the conversion device 100 according to the present embodiment, the CPU 1100 of the computer 1000 executes the information processing program loaded on the RAM 1200 so as to realize the functions of the control unit 130, etc. In addition, the HDD 1400 stores the conversion program and data according to the present disclosure in the storage unit 120. When the CPU 1100 executes the program data 1450 read from the HDD 1400, as another example, the CPU 1100 can acquire these programs from another device via the external network 1550.
[0269] Note that the present technology may also have the following configurations.
[0270] (1) A conversion device comprising:
[0271] an acquisition unit that acquires a conversion source signal to be used as a source of a tactile signal; and
[0272] The conversion unit converts the conversion source signal acquired by the acquisition unit into an intermediate representation signal represented by at least one parameter.
[0273] (2) The conversion device according to (1), wherein
[0274] Conversion Unit
[0275] The source signal is converted into an intermediate representation signal represented by parameters corresponding to human perception.
[0276] (3) The conversion device according to (1) or (2), wherein
[0277] Conversion Unit
[0278] The conversion source signal is separated into elements constituting the conversion source signal, and the separated signals are converted into intermediate representation signals.
[0279] (4) The conversion device according to (3), wherein
[0280] Conversion unit,
[0281] When the conversion source signal is an acoustic signal on which a plurality of musical instrument sounds are superimposed, the acoustic signal is separated into individual musical instrument sounds, and the separated signals are converted into intermediate representation signals.
[0282] (5) The conversion device according to (3) or (4), wherein
[0283] Conversion Unit
[0284] The conversion source signal is separated into a harmonic component which is a signal having a fundamental frequency and a noise component which is a signal not having a fundamental frequency.
[0285] (6) The conversion device according to any one of (1) to (5), wherein
[0286] Get Unit
[0287] acquiring a tactile signal including characteristic information to be applied to a specific tactile presentation device as a conversion source signal, and
[0288] Conversion Unit
[0289] A haptic signal including characteristic information to be applied to a specific haptic rendering device is converted into an intermediate representation signal represented by parameters not including the characteristic information.
[0290] (7) The conversion device according to any one of (2) to (6), wherein
[0291] Conversion Unit
[0292] The conversion source signal is converted into an intermediate representation signal, which includes a burst, a harmonic component, a noise component, and information indicating a ratio between the harmonic component and the noise component as parameters, wherein the burst is information indicating a sharp rise in an output value, the harmonic component is information having a fundamental frequency, and the noise component is information not having a fundamental frequency.
[0293] (8) The conversion device according to (7), wherein
[0294] Conversion Unit
[0295] In the output value variation per time unit in the conversion source signal, the difference from the flattened value of the output value within a predetermined time width is referenced, and a section where the reference value exceeds the reference output value is extracted as a burst.
[0296] (9) The conversion device according to (7) or (8), wherein
[0297] Conversion Unit
[0298] The converted source signal is converted into an intermediate representation signal including a first burst having a long output duration and a second burst having a shorter output duration than the first burst.
[0299] (10) The conversion device according to any one of (7) to (9), wherein
[0300] Conversion Unit
[0301] A frequency corresponding to each burst is allocated based on the conversion source signal.
[0302] (11) The conversion device according to (10), wherein
[0303] Conversion Unit
[0304] The frequency corresponding to each burst is allocated based on the weighted average frequency of the signal in the section corresponding to the burst in the conversion source signal.
[0305] (12) The conversion device according to any one of (7) to (11), wherein
[0306] Conversion Unit
[0307] A frequency corresponding to each of the harmonic component and the noise component is allocated based on the conversion source signal.
[0308] (13) The conversion device according to any one of (7) to (12), wherein
[0309] Conversion Unit
[0310] A section in which a frequency change exceeding a predetermined reference appears within a predetermined time width among frequency changes per time unit in the conversion source signal is extracted as the burst.
[0311] (14) A conversion method comprising:
[0312] Through the computer,
[0313] obtaining a converted source signal that becomes a source of a haptic signal; and
[0314] The acquired conversion source signal is converted into an intermediate representation signal represented by at least one parameter.
[0315] (15) A conversion program for causing a computer to function as:
[0316] an acquisition unit that acquires a conversion source signal as a source of a tactile signal; and
[0317] The conversion unit converts the conversion source signal acquired by the acquisition unit into an intermediate representation signal represented by at least one parameter.
[0318] Reference numerals list
[0319] 10Haptic Presentation Devices
[0320] 100 conversion devices
[0321] 110 Communication Unit
[0322] 120 storage units
[0323] 130 control unit
[0324] 131 Acquisition Unit
[0325] 132 conversion units
[0326] 133 Sending unit
[0327] 200 decoding equipment
[0328] 210 Communication Unit
[0329] 220 storage units
[0330] 230 control unit
[0331] 231 Acquisition Unit
[0332] 232 Generation Unit
[0333] 233 Output Control Unit
Claims
1. A conversion device, comprising: an acquisition unit, which acquires a conversion source signal as a source of a tactile signal; as well as The conversion unit converts the conversion source signal acquired by the acquisition unit into an intermediate representation signal represented by at least one parameter.
2. The conversion device according to claim 1, wherein Conversion Unit The source signal is converted into an intermediate representation signal represented by parameters corresponding to human perception.
3. The conversion device 1 according to claim 1, wherein: Conversion Unit The conversion source signal is separated into elements constituting the conversion source signal, and the separated signals are converted into intermediate representation signals.
4. The conversion device according to claim 3, wherein Conversion unit, When the conversion source signal is an acoustic signal on which a plurality of musical instrument sounds are superimposed, the acoustic signal is separated into individual musical instrument sounds, and the separated signals are converted into intermediate representation signals.
5. The conversion device according to claim 3, wherein Conversion Unit The conversion source signal is separated into a harmonic component which is a signal having a fundamental frequency and a noise component which is a signal not having a fundamental frequency.
6. The conversion device according to claim 1, wherein Get Unit acquiring a tactile signal including characteristic information to be applied to a specific tactile presentation device as a conversion source signal, and Conversion Unit A haptic signal including characteristic information to be applied to a specific haptic rendering device is converted into an intermediate representation signal represented by parameters not including the characteristic information.
7. The conversion device according to claim 2, wherein: Conversion Unit The conversion source signal is converted into an intermediate representation signal, which includes a burst, a harmonic component, a noise component, and information indicating a ratio between the harmonic component and the noise component as parameters, wherein the burst is information indicating a sharp rise in an output value, the harmonic component is information having a fundamental frequency, and the noise component is information not having a fundamental frequency.
8. The conversion device according to claim 7, wherein: Conversion Unit In the output value variation per time unit in the conversion source signal, the difference from the flattened value of the output value within a predetermined time width is referenced, and a section where the reference value exceeds the reference output value is extracted as a burst.
9. The conversion device according to claim 7, wherein Conversion Unit The converted source signal is converted into an intermediate representation signal including a first burst having a long output duration and a second burst having a shorter output duration than the first burst.
10. The conversion device according to claim 7, wherein: Conversion Unit A frequency corresponding to each burst is allocated based on the conversion source signal.
11. The conversion device according to claim 10, wherein: Conversion Unit The frequency corresponding to each burst is allocated based on the weighted average frequency of the signal in the section corresponding to the burst in the conversion source signal.
12. The conversion device according to claim 7, wherein Conversion Unit A frequency corresponding to each of the harmonic component and the noise component is allocated based on the conversion source signal.
13. The conversion device according to claim 7, wherein: Conversion Unit A section in which a frequency change exceeding a predetermined reference appears within a predetermined time width among frequency changes per time unit in the conversion source signal is extracted as the burst.
14. A conversion method, comprising: Through the computer, obtaining a converted source signal that becomes a source of a tactile signal; as well as The acquired conversion source signal is converted into an intermediate representation signal represented by at least one parameter.
15. A conversion program for causing a computer to function as: an acquisition unit that acquires a conversion source signal as a source of a tactile signal; and The conversion unit converts the conversion source signal acquired by the acquisition unit into an intermediate representation signal represented by at least one parameter.
Citation Information
Patent Citations
Information processing apparatus, information processing method, and program
JP2019219785A
Information processing device, method, and program
WO2019138867A1
