Generating vibrotactile signals from audio content for playback by haptic acoustic transducers

Abstract

A method of generating one or more haptic drive signals from an input signal representative of real-time audio is provided. A transient extraction process is performed on the input signal to determine a transient component of the real-time audio, and complementary transient haptic signals and steady-state haptic signals are generated in accordance with the transient extraction process. The transient haptic signal and the steady-state haptic signal are then used to generate one or more haptic drive signals, which may be used to drive one or more haptic actuators. There is also provided an audio system comprising one or more digital signal processors adapted to perform the method. The system may include one or more haptic actuators driven by the generated one or more haptic drive signals.

Description

Technical Field

[0001] The present invention relates to the derivation and processing of drive signals for vibrotactile acoustic transducers, particularly when mounted in a seat, to enhance and / or augment the reception of a conventional audio speaker system, resulting in a multi-sensory experience. Background Art

[0002] Haptics (from the Greek haptikos, meaning “related to the sense of touch”), in T. Jaiswal, R. Yadav and P. Kedia, International Journal of Advanced Science and Engineering Research, April 2018 International Journal of Advance Research in Science and Engineering "Haptic Technology - A Comprehensive Research Review and Its Applications" Haptic Technology – Comprehensive Review Study with its Applications ”) is a form of technology that simulates the sensation of touch by applying force. The technology is commonly used in the gaming, virtual reality (VR) and film industries to enhance the sensory experience rendered by rumble effects rendered on acoustic transducers installed in objects that are in direct tactile contact with the user (such as game controllers, wearable devices or seats). Because these rumble effects can increase the immersion of movie and VR experiences, tactile feedback can also be used to enhance the experience of musical performances.

[0003] When experiencing music at high sound pressure levels (SPL), listeners will be able to feel tactile sensations rendered in the form of vibrations from the low frequencies of the audio. This sensation is expected to be reproduced in home and car audio solutions using seat-mounted or wearable acoustic transducers to enhance the listening experience. It is generally believed that humans can hear frequencies in the range of 20 Hz to 20,000 Hz. As reported by B. Remache-Vinueza et al. in the September 2021 issue of Sensors ( "Sensors” )'s "Audio tactile rendering: A review of techniques and methods for conveying music information through touch" (" Audio-Tactile Rendering: A Review on Technology and Methods to Convey Musical Information through the Sense of Touch ”), humans can perceive vibration frequencies ranging from 0.3Hz to 1,000Hz.

[0004] like Figure 1 As shown, there is an overlapping region 13 between the audible spectrum 12 and the tactile sensing range 11, which means that an audio signal containing content in this overlapping frequency range can be rendered by an acoustic transducer mounted on a seat or body to provide tactile enhancement to the input audio. However, this approach may sound unnatural to the listener because different areas of the body have different sensitivities to tactile stimulation.

[0005] First, a method for creating more natural haptic performance involves performing time alignment between a conventional acoustic transducer that executes the audible part of a signal and a haptic actuator that renders haptic feedback. A delay line can be introduced to correct for a time mismatch that may be caused by the positions of various transducers in space and their relative proximity to the listener, as well as different delays in different parts of the audio system. Improving the time alignment of haptic and sound responses is beneficial for the coherence of the overall system. However, the effect can be further enhanced through signal manipulation to create a more realistic haptic experience.

[0006] Although humans can experience vibrotactile perception up to 1000 Hz, it is not practical for haptic actuators to render up to such high frequencies. Mechanoreception refers to the ability of humans to use mechanoreceptors, a type of nerve ending, to detect stimuli such as pressure changes and touch sensations. One of the main mechanoreceptive channels responsible for perceiving vibrations in the somatosensory system (also known as somesthesia, such as haptic perception) is the Pacinian channel. Pacinian corpuscles are nerve endings responsible for the skin's sensitivity to vibration. According to "A Systematic Approach to Musical Vibrotactile Feedback" by Birnbaum D.M. and Wanderley M.M. published in "Proceedings of the International Computer Music Conference" ( Proceedings of the International Computer Music Conference " (Copenhagen, Denmark, August 27 - 31, 2007, pp. 397 - 404)), the Pacinian corpuscles have a sensitive range of 40 Hz to 500 Hz, which means that the frequency response of haptic actuators can be limited to this range in terms of its band. A Systematic Approach to Musical Vibrotactile Feedback

[0007] Since the human ear is most sensitive to frequencies above 250 Hz (see, for example, "Loudness, Its Definition, Measurement, and Calculation" by H. Fletcher and W. A. Munson published in "Journal of the Acoustical Society of America" ( "Journal of the Acoustical Society of America” " (Vol. 5, 1933, pp. 82 - 108)), it is useful for acoustic transducers to only render haptic signals below this value to avoid interfering with the sound performance of other parts of the audio system. Therefore, the frequency range that haptic enhancement may focus on is 40 Hz to 250 Hz. "Loudness, its definition, measurement and calculation”

[0008] As Y. Cho et al. published in "International Conference on New Actuators" in 2014 ( "International Conference on New Actuators” "Haptic Pad: Automatically Generating Vibrotactile Feedback Based on Audio Signals for Immersive Interaction in Multimedia" ( "Haptic Cushion: Automatic Generation of Vibro-tactile Feedback Based on Audio Signal for Immersive Interaction with Multimedia” ​​) and also found in the published patent US11,340,704B2, synthesizing a new signal based on the original audio can be used to generate tactile data in the required frequency range. However, the generation of new content can be seen as an excessive modification of the original audio, which may not be appropriate.

[0009] A. Sonza et al., published in Medical Engineering and Physics in 2015 ("Medical Engineering and Physics”) Whole-body vibration perception maps and associated acceleration loads on the lower leg, hip, and head "A whole body vibration perception map and associated acceleration loads at the lower leg, hip and head” ), different areas of the body have different perceived sensitivities to vibration. Therefore, it is helpful for a haptic rendering system to be able to distribute and weight different levels of haptic signals to different areas of the system in contact with the body to accommodate different sensitivity ranges.

[0010] Bone conduction is another consideration for tactile systems, as described in Sakuragi R., Ikeno S., Okazaki R., Kajimoto H. (2015) in Proceedings of the International Conference on Artificial Reality and Telepresence and the European Graphics Association Symposium on Virtual Environments. "Proceedings of the International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments” )'s "CollarBeat: Whole-body vibrotactile presentation via the clavicle to enrich the music listening experience" ( "CollarBeat: Whole Body Vibrotactile Presentation via the Collarbone to Enrich Music "Listening Experience” Bone conduction can be a factor influencing tactile perception, so it is helpful to be able to weight the signal differently for drivers near parts of the body where the effects of bone conduction will be more or less significant.

[0011] Signal decomposition and processing of individual elements is important for advanced control of the tactile experience. As seen in US11,340,704B2, artificial intelligence (AI) can be used to extract harmonic components and percussive sounds and process them separately. However, the use of AI can be detrimental as it can take up a lot of processing resources, which can be a problem when the capabilities of the digital signal processor (DSP) are limited, and the AI ​​must be trained on a large list of program material covering the various audio types to be rendered by the system, which can be time consuming and expensive. Furthermore, it is likely that such AI training will be performed on a limited range of program material, which raises the issue that certain types may be omitted, which is detrimental to overall performance, for example when rendering more obscure audio.

[0012] The temporal aspect of audio signals can be divided into transient signals and steady-state signals. According to JO Smith's Introduction to Digital Filtering and Audio Applications ("Introduction to Digital Filters with Audio Applications” )(W3K Publishing, October 2007, ISBN 978-0-9745607-1-7), a transient event can be defined as a broadband event that bursts in an otherwise steady-state signal. A transient event can also be classified as an event where the broadband energy of the signal or the energy in a specific frequency range changes rapidly. This definition is sufficient for creating an algorithm to track transient events. Examples of such transient events can be found in Figure 2 , the Figure 2 presented against an otherwise steady-state background.

[0013] A transient event can be perceived by an individual as a sense of impact, while a steady-state signal can be interpreted as a vibration effect in the tactile system. Therefore, it is useful to separate transient signals from steady-state signals in an audio stream to be rendered haptically.

[0014] Since the in-vehicle processing capabilities of a DSP may be limited, for example when the technology is used for enhancing infotainment systems in the automotive industry, it is desirable to create a lightweight solution to derive haptic signals in real time from an incoming audio stream. Importantly, this technology should have a low computational cost so that the process can be executed on an in-vehicle DSP.

[0015] Therefore, there is a need for an improved method for deriving haptic signals from an incoming audio stream that is computationally efficient while allowing for more refined haptic feedback that can be used to enhance and / or reinforce the audible reception from a traditional audio speaker system to provide a multi-sensory experience. SUMMARY OF THE INVENTION

[0016] According to one aspect of the present invention, there is provided a method for generating one or more haptic drive signals from an input signal representative of real-time audio, the method comprising the steps of: Receiving the input signal; Performing transient extraction processing on the input signal to determine the transient components of the real-time audio; Generating a transient haptic signal from the input signal based on the transient extraction processing and a steady-state haptic signal , wherein the transient haptic signal and the steady-state haptic signal are complementary such that , wherein is the input signal on which the transient extraction processing has been performed; and Generating one or more haptic drive signals based on one or both of the transient haptic signal and the steady-state haptic signal.

[0017] Accordingly, the present invention provides a method of using transient analysis to estimate the time of occurrence of a transient event and thereby generate complementary transient and steady-state haptic signals from a real-time audio stream. Then, one or more haptic drive signals are generated using the transient and steady-state haptic signals, which can be used to drive a haptic actuator.

[0018] The transient extraction analysis can be performed in several ways, but generally speaking, the transient estimate can be expressed as the relationship between a short-term (microscopic) dynamic envelope (typical time frame of 0 ms - 100 ms) and a long-term (macroscopic) dynamic envelope (typical time frame of 200 ms - 1000 ms).

[0019] The received input audio stream can be processed algorithmically to derive a transient estimate, which is preferably limited between 0 and 1. A metric derived from the so-called "crest factor" meets this requirement, which provides the relationship between the peak value of the signal and the effective value or root mean square (RMS) value. The transient estimate analyzes the relationship between the amplitude of the transient peak of the real-time audio stream and the average amplitude in the previous audio time frame, thus outputting a higher value when a transient is detected.

[0020] Thus, in a preferred embodiment, performing the transient extraction process includes deriving a real-time transient estimate from the input signal C(t) , where C(t) has a value in the range of 0 ≤ C(t) ≤ 1 and represents the transient component of the real-time audio; and the transient haptic signal T (t) is generated according to , and the steady-state haptic signal S(t) is generated according to .

[0021] Then, the transient estimate can be used to create a transient haptic signal, for example, by multiplying it with the received input audio signal. Then, the complementary signal of the derived transient estimate signal can be multiplied with the same band-limited audio signal to create a steady-state haptic signal. If these signals are added together, they should recreate the audio signal from which they were derived. This allows the system to control the enhancement or attenuation of the transient or steady-state elements of the audio stream, thus avoiding over-modification or coloration and maintaining purity.

[0022] Preferably, the input audio signal is band-limited before the transient estimation step. This band-limitation can be performed before or after receiving the input signal. The selected band-limitation will typically be determined by the frequency response of the haptic actuator drive unit as well as the psychoacoustic and somatosensory system's perceptual sensitivity. For example, for optimal haptic response, the band-limitation can be in the frequency range of 40 to 250 Hz.

[0023] The generated haptic drive signals can be distributed to a large number of haptic actuators. This distribution is affected by factors such as transducer response, drive unit location, and other user-controlled parameters. The actuators can receive only transient signals, only steady-state signals, or a weighted sum of both signals.

[0024] These generated signals can be derived from a mono audio stream, or alternatively, the processing can be applied in real-time to any number of discrete audio streams, thereby creating transient haptic signals and steady-state haptic signals for each audio channel.

[0025] The method can also include generating one or more acoustic drive signals in the audible frequency range based on the input signal. In this way, audible drive signals and haptic drive signals are generated, both of which can be used to drive appropriate transducers, thereby providing the user with a combination of rich audio sensory feedback and haptic sensory feedback.

[0026] According to a second aspect of the invention, a computer-readable medium includes computer-executable instructions that, when executed on one or more processors of an audio system, cause the system to perform the method of the first aspect. In this way, the method of the first aspect of the invention can be implemented by one or more processors of an audio system to generate haptic drive signals for driving haptic actuators.

[0027] The computer-readable medium of the second aspect of the invention can provide an update or enhancement to an existing digital signal processor sound source system. In this way, the existing system can be updated by providing the update.

[0028] According to a third aspect of the invention, an audio system includes one or more digital signal processors adapted to perform the method of the first aspect.

[0029] In certain embodiments, the audio system includes a user interface for receiving user input parameters. In this way, the user can control certain characteristics of the haptic drive signals.

[0030] Preferably, the audio system includes one or more haptic transducers for providing haptic feedback, each of the one or more haptic transducers being driven by one of the one or more haptic drive signals. Each haptic drive signal can be generated to optimally drive its corresponding haptic transducer.

[0031] In some embodiments, one or more haptic transducers are adapted for use in a seat occupied by a user. The haptic transducers may be located in the backrest of the seat, under the seat, and in the leg area. Further haptic transducers may be provided for the floor. In other embodiments, the haptic transducers are adapted for use in a wearable device worn by the user. Depending on the particular application, a particular pattern of actuators may be provided to achieve optimal sensory feedback while appropriately adjusting the drive signals.

[0032] The audio system may also include one or more acoustic transducers for providing an audible signal, each acoustic transducer being driven by an acoustic drive signal of the one or more acoustic drive signals. In this way, both acoustic transducers and haptic transducers are provided, allowing the user to experience a combination of rich audio sensory stimuli and haptic sensory stimuli optimized for a particular arrangement and audio type.

[0033] Those skilled in the art will understand that the present invention is capable of various implementations depending on the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Examples of the present invention will be described in detail with reference to the accompanying drawings, in which: Figure 1 showing a frequency domain representation of an audio segment, in which an overlapping region between a haptic sensory area and an audible acoustic area is defined; Figure 2 showing a time domain representation of a transient event occurring in an audio stream that was originally in a steady state; Figure 3 showing an exemplary setup of a car seat, in which one acoustic transducer is mounted on the backrest of the seat; Figure 4 is a flowchart showing advanced transient processing of an audio signal; Figure 5 showing a time domain analysis of a mostly steady state signal containing a transient, where (a) is the original signal, (b) is the real-time transient estimate, (c) is the calculated transient signal, and (d) is the calculated steady state signal; Figure 6 is in Figure 5 a schematic representation of the process applied in the time domain analysis; Figure 7 is a flowchart showing more details of the required and optional processing steps for combining a transient stream and a steady state stream before output; Figure 8 is a flowchart showing more details of the required and optional processing steps for not combining a transient stream and a steady state stream before output; Figure 9 showing an exemplary configuration of a seat having two internally mounted haptic transducers, one in the backrest and the other under the seat; and Figure 10 An exemplary configuration of a seat is shown, which is provided with a plurality of smaller haptic transducers mounted in the backrest, a large actuator under the seat, a leg actuator, and a floor vibrator. Detailed Description

[0035] The present invention can be applied in many different ways depending on the audio system used. Some exemplary implementations will be described below with reference to the accompanying drawings.

[0036] The present invention derives transient haptic signals and steady-state haptic signals in real time from audio source content and distributes these signals to one or more acoustic transducers. Embodiments of the present invention are used to enhance the experience of an audio system to create a realistic haptic perception consistent with the auditory reception of sound.

[0037] The motivation for separating the transient haptic stream and the steady-state haptic stream from the original source audio stems from considerations in user experience and transducer mechanical design. Figure 3 A basic arrangement is shown including a single transducer 31 mounted to the backrest of a seat 30, although the transducer can be positioned and mounted arbitrarily depending on the use case. In this case, the single transducer should be able to effectively reproduce both steady-state and transient haptic content.

[0038] Generally, actuators employ a heavy moving mass to facilitate high energy transfer at low frequencies, but this in turn results in a slower and weaker transient response. Conversely, a lighter moving mass actuator may have a fast enough transient response, but may not provide sufficient energy at low frequencies when driven with an unprocessed input audio signal. Therefore, in order to accommodate actuator responses that may vary depending on the design, it is beneficial to weight the transient or steady-state elements present in the drive signal. Additionally, from a user experience perspective, it is also advantageous to be able to control the balance between the steady-state (vibration) and transient (impact) components so that the two different resulting sensations can be blended together to suit individual tastes and also calibrated for different audio program materials.

[0039] For the purposes of the present invention, an audio signal containing the full range of audio frequencies can be processed to determine the transient component and the steady-state component. However, generally prior to processing, it is preferred to band-limit the audio signal to the frequency band of interest. Sometimes, the audio signal may be naturally band-limited. Otherwise, it may be necessary to band-limit the audio signal to the operable frequency band of interest through frequency filtering.

[0040] One basic method for preparing a band-limited signal is to apply low-pass filtering with a cut-off frequency in the 1 kHz range to the input audio signal, thereby obtaining a band-limited signal whose frequency components are consistent with the limits of human tactile response. See "Audio-Tactile Rendering: A Review of Techniques and Methods for Communicating Musical Information through the Sense of Touch" by B. Remache-Vinueza et al., published in Sensors in September 2021 ( "Sensors” ). Audio-Tactile Rendering: A Review on Technology and Methods to Convey Musical Information through the Sense of Touch ).

[0041] When deriving the band-limited signal, a more comprehensive method takes into account the responses of both human auditory and tactile perceptions under the cross-frequency influence of acoustic phenomena. As mentioned earlier, according to the discussion in "Audio-Tactile Rendering: A Review of Techniques and Methods for Communicating Musical Information through the Sense of Touch" by B. Remache-Vinueza et al., published in Sensors in September 2021 ( "Sensors” ), Audio-Tactile Rendering: A Review on Technology and Methods to Convey Musical Information through the Sense of Touch ), Figure 1 Figure 13 shows the frequency-related overlapping region in human responses, which exists between the tactile response area 11 and the auditory response area 12 of the audio segment. Thus, the so-called infrasound phenomenon (i.e., f < 20 Hz) can be perceived by tactile means, frequencies in the range of 20 to 1 kHz can be perceived by both tactile and auditory means, and for frequencies greater than 1 kHz, the human tactile response becomes ineffective.

[0042] In addition, the main mechanism for generating tactile vibration perception through skin contact is by means of the Pacinian corpuscle channels that are effectively responsive to tactile phenomena between 40 and 500 Hz. See "A Systematic Approach to Musical Vibration Tactile Feedback" by Birnbaum D.M. and Wanderley M.M., published in the Proceedings of the International Computer Music Conference ( Proceedings of the International Computer Music Conference ), (Copenhagen, Denmark, August 27 - 31, 2007, pp. 397 - 404) ( A Systematic Approach to Musical Vibrotactile Feedback ). In addition, for frequencies greater than 250 Hz, the human ear becomes more sensitive to acoustic phenomena (see, for example, "Loudness, Its Definition, Measurement, and Calculation" by H. Fletcher and W. A. Munson, published in the Journal of the Acoustical Society of America ( "Journal of the Acoustical Society of America” ), (Vol. 5, 1933, pp. 82 - 108) ( "Loudness, its definition, measurement and calculation” ).

[0043] Based on these considerations, a preferred embodiment of the present invention includes deriving a band-limited signal present in the frequency range of 40 - 250 Hz. This band-limiting enables effective tactile response excitation while reducing the likelihood of the tactile actuator output having a negative impact on the sound auditory reception quality within the overall system.

[0044] In some cases, such as in multi-channel audio content, the low-frequency effects (LFE) channel may contain an appropriate frequency range to drive the tactile actuator and provide tactile feedback. In these cases, the LFE channel or similar audio content containing frequencies within the range of human tactile perception can be used as the input audio-tactile signal for this processing.

[0045] The resulting band-limited input signal is analyzed to calculate the above-mentioned transient tactile stream and steady-state tactile stream. This analysis can be performed in several ways, but generally speaking, the transient estimate can be expressed as the relationship between a short-term (microscopic) dynamic envelope (typical time frame of 0 ms - 100 ms) and a long-term (macroscopic) dynamic envelope (typical time frame of 200 ms - 1000 ms). This processing represents the core of the present invention, and a flowchart 40 of the overall method is shown in Figure 4 In it. The received input audio signal 41 is subjected to a transient extraction process 42, and the resulting tactile drive signal 43 is output.

[0046] To this end, the relationship between the peak value calculated from the band-limited input signal and the effective value or root mean square (RMS) value can be determined. A useful definition of this relationship is the peak factor, which is equivalent to the peak-to-average power ratio of the signal ( ), as defined by T. J. Rouphael in "Wireless Knowledge 101: Peak-to-Average Power Ratio (PAPR)" published in 2009 ( "Wireless 101: Peak to average power ratio (PAPR)” )(available online: https: / / www.eetimes.com / wireless-101-peak-to-average-power-ratio-papr / , accessed on July 27, 2022), when expressed in decibels, this peak-to-average power ratio ( ) is defined as follows:

[0047] where, is the maximum value of the square of the audio signal within a certain time frame of the signal, and is the average value of the square of the signal calculated over the same time frame interval.

[0048] Although the above provides an exemplary method for deriving a transient estimate, other methods can be used to calculate this value, provided that it represents the relationship between the peak and average signal values and can be reduced to a mapping range from 0 to 1. For example, the transient estimate can be derived through the following relationship :

[0049] In all instances of this calculation, and will be bounded between 0 and 1 because the original signal will be an audio signal with an amplitude range of -1 to 1, representing full-scale audio. Therefore, the transient estimate will also be bounded between 0 and 1, which is particularly advantageous as a scaling factor.

[0050] Using the transient estimate as described above , two complementary audio-tactile signals can be generated as follows:

[0051] wherein, is the transient tactile signal, is the steady-state tactile signal, and is the input audio signal on which the transient extraction process is performed to allow any filtering before processing. In this way, the two complementary tactile signals contain all the relevant information from the input signal.

[0052] The time range used for calculating the peak and the effective value can be adjustable. For a smaller dynamic envelope time range for peak and RMS tracking, C(t) the value will be more responsive to smaller changes in the signal. For a longer time range, the resulting transient estimate will be smoother and thus represent more significant transient events in the signal.

[0053] Figure 5 shows the process in the time domain, starting from the Figure 5 input signal shown in (a), which is a seemingly steady-state input signal 50 with a transient 51 inserted therein. Applying the transient extraction process to this signal results in the Figure 5 transient estimate shown in (b) , where the central "sawtooth" feature 53 represents the transient on a roughly constant zero background level 52. Multiplying the input signal by results in the Figure 5 transient tactile signal shown in (c) , while multiplying the input signal by results inFigure 5 The steady-state tactile signal shown in (d) .

[0054] Figure 6 Schematically shows the steps involved in the process 60, starting from the input signal 61, performing a transient tracking analysis 62 on the input signal 61 to obtain a transient estimate and its complement , and then, these two are respectively multiplied by the input signal at 63 and 64 to generate a transient tactile signal and a steady-state tactile signal .

[0055] Additional algorithms can be used to process the signal, such as squaring the signal or adding gain to change its response intensity, but it should be noted that the range of the signal should be limited between 0 and 1. Any method functionally equivalent to the above process can also generate representative transient tactile signals and steady-state tactile signals.

[0056] Once the transient tactile signal and the steady-state tactile signal have been derived, they can be used to generate one or more tactile drive signals for driving one or more tactile actuators. Each transient tactile signal and steady-state tactile signal can be used alone as a drive signal with or without further processing, but they can also be combined to generate a tactile drive signal.

[0057] For example, the signal to be sent to the i-th actuator

[0058] wherein is the steady-state gain (or weight) of the tactile drive signal of the i-th actuator, is the transient gain (or weight) of the i-th actuator, is the time-domain representation of the transient signal extracted from the original audio, and is the time-domain representation of the steady-state signal extracted from the original audio. In some embodiments, the steady-state gain value and the transient gain value can be fixed and optimized according to the tactile actuator to be driven, while in other embodiments, the gain values can vary, for example, by user input.

[0059] In one implementation, the steady-state gain value and the transient gain value can be represented by the following:

[0060] wherein, for the i-th actuator, is a user input parameter whose value is between -1 and 1, is a static scale factor used to control the strength of the user parameter, and and are static weight values for the steady-state signal and the transient signal, respectively.

[0061] When = 0, the steady-state gain and the transient gain are directly equal to the static weight values. When is positive, the level of the transient signal will increase and the level of the steady-state signal will decrease, each by an equal number of decibel values equal to because represents the logarithmic decibel gain in this implementation. In other implementations, when represents the linear gain, the components and can be replaced with and .

[0062] Depending on the application, the transient tactile signal and the steady-state tactile signal can be filtered and / or time-corrected either before or after the weighting is applied. For a given scenario, this processing can be sequenced and applied in the most efficient manner.

[0063] Figure 7 is a flowchart showing the processing steps 70 according to an embodiment, where the transient tactile signal and the steady-state tactile signal are combined to generate one or more tactile drive signals. The main steps are shown in solid boxes, while the optional steps are shown in dashed boxes. Optionally, the received input signal 71 is filtered 72 to extract the tactile frequency components before performing the transient extraction analysis 73. The resulting transient tactile signal and steady-state tactile signal are weighted 74t, 74s and can optionally be further filtered 75t, 75s before summing 76. Then, a time correction 77 can be applied to the combined signal before outputting one or more tactile drive signals 78.

[0064] Figure 8 is a flowchart showing the processing steps 80 according to an embodiment, where the transient tactile signal and the steady-state tactile signal are not combined to generate one or more tactile drive signals. The main steps are again shown in solid boxes, while the optional steps are shown in dashed boxes. Optionally, the received input signal 81 is filtered 82 to extract the tactile frequency components before performing the transient extraction analysis 83. Then, appropriate weighting 84t, 84s is applied to the resulting transient tactile signal and steady-state tactile signal, after which further filtering 85t, 85s and / or time correction 86t, 86s can be optionally performed without summing before outputting one or more tactile drive signals 87.

[0065] After reviewing the input signal analysis and haptic drive signal generation, we now continue to consider the configuration and properties of haptic actuators that can be driven in embodiments of the present invention. As previously mentioned, Figure 3 A single acoustic transducer 31 mounted in the backrest of seat 30 is shown, but this is just one of the many actuator configurations that the technology can cover. Any number of actuators can be provided at various locations within a seat, wearable device, or other surface, and a variety of mounting strategies can be deployed. Additionally, the orientation of the actuators can be configured in multiple ways.

[0066] In an exemplary scenario, Figure 3 the large backrest actuator 31 shown may have poor transient performance, in which case the static weight of the transient signal is typically set to be greater than the static weight of the steady-state signal value. In the case of this single actuator, both the transient signal and the steady-state signal will be sent to the same drive unit. To calibrate this drive unit, is set to 2 (+6 dB), and is set to 1 (+0 dB). This exemplary weight will double the intensity of the transient component of the received signal. Additionally, if the scale factor is set to 6, this will provide the user with a 12 dB (i.e., 2 range to control the signal. If the user is listening to dance music, for example, they may choose to increase the parameter to boost the transient, or if they are listening to orchestral music, they may decrease the value.

[0067] In another example, such as as Figure 9 shown, where seat 90 is equipped with more than one actuator, value and value can be used to weight the backrest actuator 91, for example, to significantly enhance the transient component of the signal, while the actuator 92 under the seat is weighted to enhance or specifically present the steady-state component of the signal. Additionally, actuators mounted on the floor or wearable legs can be used to simulate the sensations experienced when standing in a venue performing high SPL music. A steady-state or vibration effect can be rendered under the user's feet, and the whole-body vibration sensation will be similar to the vibration transmitted through the ground from a high-volume bass driver and conducted through the feet.

[0068] Such as Figure 3 in a very simple setup does not require weighting. The transient signal can be sent without weighting to drive the backrest actuator 31, which is equivalent to setting and 。The transient signal can be subjected to some calibration equalization filtering to calibrate the performance of the actuator before the content rendering stage. Similarly, the steady-state signal can be sent without weighting to drive the actuator under the seat, which is equivalent to the setting in the combined signal model and 。Similarly, the steady-state signal can undergo some calibration equalization before the content rendering stage.

[0069] Although large mass movers have been shown as possible haptic actuators for some solutions, several smaller acoustic transducers can be additionally installed to render haptic enhancements. Figure 10 Shows the use of different actuators within the seat 100 according to their position and use. The seat back contains multiple actuators, where a medium-sized actuator 101 is located in the occupant's spinal region and is supplemented by smaller actuators 102 near the periphery. Larger actuators are used for the lower seat area 103 and the leg area 104 of the seat, while additional actuators 105 are embedded in the floor to render a vibration effect under the user's feet. In this case, it is useful to heavily weight the transient part of the signal for transducers closer to the occupant, e.g., the transducer 101 closest to the user's spine, to create a tightly focused transient performance. The steady-state component can be weighted to spread the vibration effect over a larger surface area, effectively vibrating the entire seat.

[0070] When haptic processing is implemented as part of a larger multi-speaker system, there may be a temporal mismatch between the perceived auditory signal and the haptic signal, as some of the drive units will render signals within the human audible range, which can be caused by various factors including sound source proximity, actuator speed, and signal transmission medium. To correct this, a delay line can be implemented before any signal to delay either the haptic signal or the auditory signal. Here, the application of a delay for this purpose is referred to as time correction.

[0071] In summary, an audio signal containing frequencies within the human haptic perception range can be decomposed into complementary transient and steady-state components. The initial signal can optionally be band-limited or directly input without prior processing. The decomposed haptic signals can be weighted and distributed to multiple acoustic transducers designed to render haptic content. The individual haptic signals can be directly routed to the output channels or combined before output. Time correction can optionally be applied to facilitate integration into a multimedia system. Weighting enables the generated haptic signals to be system-agnostic as they can be rendered on any number of actuators and can adapt to their transient response. Alternatively, the user can be allowed to control the relative weighting of the transient and steady-state signals.

Claims

1. A method for generating one or more haptic drive signals from an input signal representative of real-time audio, the method comprising the steps of: Receiving the input signal; Performing transient extraction processing on the input signal to determine transient components of the real-time audio; Generate transient tactile signals from the input signal according to the transient extraction process and steady-state tactile signals , where the transient tactile signals and the steady-state tactile signals are complementary, such that , where is the input signal on which the transient extraction process has been performed; And Generating the one or more haptic drive signals based on one or both of the transient haptic signal and the steady-state haptic signal.

2. The method according to claim 1, wherein: Performing transient extraction processing includes deriving a real-time transient estimate from the input signal C(t) , where C(t) has a value in the range of 0 ≤ C(t) ≤ 1 and represents the transient component of the real-time audio; And The transient tactile signal T(t) is generated according to , and the steady-state tactile signal S(t) is generated according to .

3. The method according to claim 2, wherein, C(t) defined according to the following formula: Wherein, represents the maximum value of the square of the audio signal within the time frame of the signal, and represents the average value of the square of the signal calculated over the same time frame, such that and .

4. The method according to any one of claims 1 to 3, wherein Naturally or derived from a previous frequency filtering, the received input signal is band-limited.

5. The method according to any one of claims 1 to 3 further comprises the following steps: The input signal is frequency filtered prior to the transient extraction process to bandlimit the input signal, where is the input signal after filtering.

6. The method according to claim 4 or 5, wherein The band-limited input signal is in a frequency range of 40 to 250 Hz.

7. The method according to any one of claims 1 to 3, wherein The input signal is a multi-channel signal, and the method further comprises downmixing the input signal to a mono signal before the transient extraction processing.

8. The method according to any one of claims 1 to 3, wherein The input signal includes a low-frequency effects (LFE) channel.

9. The method according to any one of the preceding claims, wherein Generating the one or more haptic drive signals includes: applying time correction to one or both of the transient haptic signal and the steady-state haptic signal.

10. The method according to any one of the preceding claims, wherein, Generating the one or more haptic drive signals includes: applying frequency filtering to one or both of the transient haptic signal and the steady-state haptic signal.

11. The method according to any one of the preceding claims, wherein, Generating the one or more haptic drive signals includes: combining the transient haptic signal and the steady-state haptic signal.

12. The method according to claim 11, wherein Generating the one or more haptic drive signals further includes: weighting the transient haptic signal and the steady-state haptic signal before combining them such that the i-th haptic drive signal is generated according to the following formula: wherein, is is the weight applied to the steady-state tactile signal is the weight applied to the transient tactile signal to generate the i-th tactile drive signal , where i ≥ 1 13. The method according to claim 12, wherein, The weights applied to each of the transient haptic signal and the steady-state haptic signal include a static weight component and a variable weight component determined from user input parameters.

14. The method according to claim 13, wherein The weights applied to the steady-state tactile signals and the weights applied to the transient tactile signals are calculated as follows: wherein, is a user parameter of an input within a range of -1 to 1, is a static scale factor in decibels for controlling the intensity of the user parameter, and and are static weight values of the steady-state haptic signal and the transient haptic signal respectively for generating the i-th haptic drive signal.

15. The method according to any one of claims 11 to 14, wherein Before combining the transient haptic signal and the steady-state haptic signal to generate the i-th haptic drive signal, a plurality of different haptic drive signals are generated by applying different weights to the transient haptic signal and the steady-state haptic signal.

16. The method according to any one of the preceding claims further comprises: Generating one or more acoustic drive signals in an audible frequency range based on the input signal.

17. A computer-readable medium, comprising computer-executable instructions that, when executed on one or more processors of an audio system, cause the system to perform the method according to any one of claims 1 to 16.

18. An audio system, comprising one or more digital signal processors adapted to perform the method according to any one of claims 1 to 16.

19. The audio system according to claim 18, comprising a user interface for receiving user input parameters.

20. The audio system according to claim 18 or claim 19, comprising one or more haptic transducers for providing haptic feedback, each of the haptic transducers being driven by a haptic drive signal among the one or more haptic drive signals.

21. The audio system according to claim 20, wherein, Each haptic drive signal is generated to optimally drive its corresponding haptic transducer.

22. The audio system according to claim 20 or claim 21, wherein, One or more of the haptic transducers are adapted to be used in a seat, a wearable device, or a floor.

23. The audio system according to any one of claims 18 to 22, as recited in claim 16, includes one or more acoustic transducers for providing audible feedback, each of the acoustic transducers being driven by an acoustic drive signal of the one or more acoustic drive signals.

Images