Audio and haptic signal processing

Abstract

Audio and haptic signal processing systems and methods are provided. A method for haptic drive signal generation is provided. The method includes receiving an input audio signal for driving an audio speaker. The method also includes deriving, from the audio signal, data components indicative of behavior of the audio speaker when driven by the input audio signal, each data component associated with a respective frequency range. The method also includes determining a haptic drive component from each data component. A gain provided to each of the haptic drive components is selected, the gain being selected for each respective frequency range. Each selected gain is applied to the respective haptic drive component to produce a gain-adjusted haptic drive component. A haptic drive signal is determined from the gain-adjusted haptic drive component, the haptic drive signal being designed to generate a mechanical behavior of the haptic actuator when used to drive the haptic actuator.

Description

Technical Field

[0001] The present disclosure relates to haptic feedback generation. In particular, the present disclosure relates to methods and systems for generating a haptic drive signal for driving a haptic actuator, the haptic drive signal being generated based on an audio signal for driving an audio speaker. Background Art

[0002] Haptic feedback or haptic technology, kinesthetic communication, 3D touch, etc. refers to the use of vibration (or in some cases, force or motion) to simulate the user's touch experience. Haptic feedback is commonly used in devices such as mobile phones, game controllers, joysticks and steering wheels, where various components of the device are configured to vibrate in order to improve the perceived user experience. Haptic technology is also used in virtual, augmented and extended reality applications, such as for creating or controlling virtual objects and enhancing remote control of devices. Haptic feedback is also used in wearable technology and the Internet of Things (IOT), also to improve the user experience or control of devices.

[0003] In cases such as mobile phones and gaming applications, the user experience is often improved by linking tactile feedback with the audio output. To obtain a realistic, immersive user experience, the acoustic output of the speaker should generally be consistent with the output vibration (i.e., tactile response). Furthermore, certain tactile responses may improve the user experience to a greater extent than other tactile responses, and thus customized tactile responses may be desirable.

[0004] However, maintaining consistency between audio output and haptic responses can be challenging, particularly when trying to provide certain haptic responses on multiple devices that may have different speakers / haptic actuators with different properties and / or different geometric locations of the speakers and haptic actuators, and therefore may respond differently to different audio and haptic signals. Summary of the invention

[0005] The following provides aspects of improving the prior art and systems for tactile drive signal generation, disclosing at least improved consistency between audio output and tactile response. Such consistency may include providing a tactile drive signal that closely corresponds to the audio output signal, both in terms of time synchronization between the signals (so that a particular audio output is synchronized in time with the tactile output) and in terms of the profile of the signal (so that the actual tactile signal perceived is consistent with the audio output). The consistency provided by these disclosed aspects may be independent of the specific audio speakers and / or tactile actuators used, and independent of the specific tactile signals generated. In other words, depending on the use case, a customized tactile signal may be generated for a given audio signal (e.g., where certain frequency components are emphasized for tactile response, as further discussed herein), all of which provides a user experience that is consistent with the audio signal, regardless of the specific type of speaker or actuator used.

[0006] According to a first aspect, an embodiment provides a method for generating a tactile drive signal. The method includes receiving an input audio signal for driving an audio speaker. The method also includes deriving data components from the audio signal indicating the behavior of the audio speaker when driven by the input audio signal, each data component being associated with a corresponding frequency range. The method also includes determining a tactile drive component from each data component. A gain is selected to be provided to each of the tactile drive components, the gain being selected for each corresponding frequency range. Each selected gain is applied to the corresponding tactile drive component to produce a gain-adjusted tactile drive component. A tactile drive signal is determined from the gain-adjusted tactile drive component, the tactile drive signal being designed to produce mechanical behavior of the tactile actuator when used to drive the tactile actuator.

[0007] Optionally, each data component is indicative of the mechanical behavior of an audio speaker when driven by an audio signal across a corresponding frequency range.

[0008] When the audio speaker is driven by the input audio signal, the data component may indicate speaker displacement, speaker acceleration, or speaker output sound pressure level.

[0009] The derivation of the data components may include predicting a mechanical behavior of an audio speaker when driven by an audio signal; and determining the data components from the prediction of the mechanical behavior.

[0010] The prediction may include inputting the audio signal into a model of the mechanical behavior of the audio speaker, the model being configured to generate a prediction of the mechanical behavior of the audio speaker when driven by the audio signal input into the model. The prediction may also include adjusting one or more parameters of the model based on operational feedback from the audio speaker. The operational feedback may include voltage and / or current feedback from the audio speaker. Determining the data component from the prediction of the mechanical behavior may include filtering the prediction of the mechanical behavior according to the audio frequency, filtering the prediction of the mechanical behavior into the data component.

[0011] Determining a haptic drive component from each data component may include applying a moving average or other digital signal processing method, such as a finite impulse response (FIR) or infinite impulse response (IIR) filter, to each data component to generate a corresponding haptic drive component.

[0012] The gain provided to each of the haptic drive components may be selectable by a user.The gain provided to each of the haptic drive components may be selectable by a user via a graphical user interface GUI.

[0013] Each tactile drive component may indicate a mechanical behavior of the tactile actuator when driven by a tactile drive signal determined from the corresponding tactile drive component. Determining the tactile drive signal from the gain-adjusted tactile drive component may include determining the tactile drive signal from the gain-adjusted tactile drive component to produce a mechanical behavior of the tactile actuator indicated by the gain-adjusted tactile drive component. Determining the tactile drive signal from the gain-adjusted tactile drive component may include inputting the gain-adjusted tactile drive component into a model of the mechanical behavior of the tactile actuator, the model being configured to generate a tactile drive signal that will recreate the mechanical behavior indicated by the gain-adjusted tactile drive component when used to drive the tactile actuator. Determining the tactile drive signal from the gain-adjusted tactile drive component may also include adjusting one or more parameters of the model based on operational feedback from the tactile actuator. The operational feedback may include voltage and / or current feedback from the tactile actuator. In other words, the resulting mechanical behavior may be a desired mechanical behavior corresponding to the gain-adjusted tactile drive component. The gain-adjusted haptic drive component may represent a desired mechanical behavior produced by the haptic actuator. In this case, the haptic drive signal is determined based on the gain-adjusted drive component as a drive signal that will produce the desired mechanical behavior in the haptic actuator as indicated by the gain-adjusted drive component.

[0014] Providing a gain to a tactile drive component may be based on a frequency range associated with the tactile drive component. A first tactile drive component may be associated with a first frequency range component, and a second tactile drive component may be associated with a second frequency range. The first frequency range may be a lower frequency range than the second frequency range. Providing a gain to a tactile drive component based on a frequency range associated with the tactile drive component may include providing a higher gain to a first tactile drive component associated with the first frequency range component than to a second tactile drive component associated with the second frequency range. The gain provided to the second tactile drive component can be a zero gain.

[0015] The method may also include: synchronizing an audio output of an audio speaker with a tactile output of a tactile actuator. The synchronization may include: generating an audio output from an audio speaker using an input audio signal including a pilot input signal (the pilot tone may be the input signal, or may be included as part of the audio signal—e.g., embedded in the signal with other audio data), and receiving a speaker response signal from the audio speaker in response to the generated audio output; generating a tactile output from the tactile actuator using the pilot input signal, and receiving a tactile response signal from the tactile actuator in response to the generated tactile output; determining a time delay from the speaker response signal and the tactile response signal; and adjusting the input audio signal to the audio speaker and the input tactile signal to the tactile actuator to adjust the time delay.

[0016] According to this aspect, the embodiment also provides a system for tactile drive signal generation, the system comprising one or more processors configured to receive an input audio signal for driving an audio speaker. The one or more processors are further configured to derive data components from the audio signal indicating the behavior of the audio speaker when driven by the input audio signal, each data component being associated with a corresponding frequency range; determine a tactile drive component from each data component; select a gain provided to each of the tactile drive components, the gain being selected for each corresponding frequency range; apply each selected gain to the corresponding tactile drive component to generate a gain-adjusted tactile drive component; and determine a tactile drive signal from the gain-adjusted tactile drive component, the tactile drive signal being designed to produce a mechanical behavior of the tactile actuator when used to drive the tactile actuator.

[0017] Optionally, each data component is indicative of the mechanical behavior of an audio speaker when driven by an audio signal across a corresponding frequency range.

[0018] When the audio speaker is driven by the input audio signal, the data component may indicate speaker displacement, speaker acceleration, or speaker output sound pressure level.

[0019] The derivation of the data components may include predicting a mechanical behavior of an audio speaker when driven by an audio signal; and determining the data components from the prediction of the mechanical behavior.

[0020] The prediction may include inputting the audio signal into a model of the mechanical behavior of the audio speaker, the model being configured to generate a prediction of the mechanical behavior of the audio speaker when driven by the audio signal input into the model. The prediction may also include adjusting one or more parameters of the model based on operational feedback from the audio speaker. The operational feedback may include voltage and / or current feedback from the audio speaker. Determining the data component from the prediction of the mechanical behavior may include filtering the prediction of the mechanical behavior according to the audio frequency, filtering the prediction of the mechanical behavior into the data component.

[0021] Determining a haptic drive component from each data component may include applying a moving average or other digital signal processing method, such as a finite impulse response (FIR) or infinite impulse response (IIR) filter, to each data component to generate a corresponding haptic drive component.

[0022] The gain provided to each of the haptic drive components may be selectable by a user.The gain provided to each of the haptic drive components may be selectable by a user via a graphical user interface GUI.

[0023] Each tactile drive component may indicate a mechanical behavior of the tactile actuator when driven by a tactile drive signal determined from the corresponding tactile drive component. Determining the tactile drive signal from the gain-adjusted tactile drive component may include determining the tactile drive signal from the gain-adjusted tactile drive component to produce a mechanical behavior of the tactile actuator indicated by the gain-adjusted tactile drive component. Determining the tactile drive signal from the gain-adjusted tactile drive component may include inputting the gain-adjusted tactile drive component into a model of the mechanical behavior of the tactile actuator, the model being configured to generate a tactile drive signal that will recreate the mechanical behavior indicated by the gain-adjusted tactile drive component when used to drive the tactile actuator. Determining the tactile drive signal from the gain-adjusted tactile drive component may also include adjusting one or more parameters of the model based on operational feedback from the tactile actuator. The operational feedback may include voltage and / or current feedback from the tactile actuator. In other words, the resulting mechanical behavior may be a desired mechanical behavior corresponding to the gain-adjusted tactile drive component. The gain-adjusted haptic drive component may represent a desired mechanical behavior produced by the haptic actuator. In this case, the haptic drive signal is determined based on the gain-adjusted drive component as a drive signal that will produce the desired mechanical behavior in the haptic actuator as indicated by the gain-adjusted drive component.

[0024] Providing a gain to a tactile drive component may be based on a frequency range associated with the tactile drive component. A first tactile drive component may be associated with a first frequency range component, and a second tactile drive component may be associated with a second frequency range. The first frequency range may be a lower frequency range than the second frequency range. Providing a gain to a tactile drive component based on a frequency range associated with the tactile drive component may include providing a higher gain to a first tactile drive component associated with the first frequency range component than to a second tactile drive component associated with the second frequency range. The gain provided to the second tactile drive component can be a zero gain.

[0025] The one or more processors of the system may be further configured to synchronize an audio output of an audio speaker with a tactile output of a tactile actuator. The synchronization may include generating an audio output from an audio speaker using an input audio signal including a pilot input signal (the pilot tone may be the input signal, or may be included as part of the audio signal—e.g., embedded in the signal with other audio data), and receiving a speaker response signal from the audio speaker in response to the generated audio output; generating a tactile output from the tactile actuator using the pilot input signal, and receiving a tactile response signal from the tactile actuator in response to the generated tactile output; determining a time delay from the speaker response signal and the tactile response signal; and adjusting the input audio signal to the audio speaker and the input tactile signal to the tactile actuator to adjust the time delay.

[0026] The system may also include a memory connected to the one or more processors.The system may also include an audio speaker and / or a haptic actuator connected to the one or more processors.

[0027] According to this aspect, the embodiment also provides a computer-readable medium comprising instructions, which, when executed, cause one or more circuits of an apparatus for processing data to perform any of the steps of the method described in this aspect.

[0028] According to a second aspect, an embodiment provides a method for generating a tactile drive signal, the method comprising: receiving an input audio signal for driving an audio speaker; determining a signal indicating the mechanical behavior of the audio speaker when driven by the audio signal based on the audio signal; and determining a tactile drive signal for a tactile actuator from the signal indicating the mechanical behavior of the audio speaker.

[0029] According to a second aspect, an embodiment provides a system for generating a tactile drive signal, the system comprising one or more processors configured to perform the following operations: receiving an input audio signal for driving an audio speaker; determining a signal indicating the mechanical behavior of the audio speaker when driven by the audio signal based on the audio signal; and determining a tactile drive signal for a tactile actuator from the signal indicating the mechanical behavior of the audio speaker.

[0030] According to a third aspect, an embodiment provides a method for synchronizing an audio output of an audio speaker with a tactile output of a tactile actuator, the method comprising: generating an audio output from the audio speaker using an input audio signal including a pilot input signal, and receiving a speaker response signal from the audio speaker in response to the generated audio output; generating a tactile output from the tactile actuator using the pilot input signal, and receiving a tactile response signal from the tactile actuator in response to the generated tactile output; determining a time delay from the speaker response signal and the tactile response signal; and adjusting the input audio signal to the audio speaker and the input tactile signal to the tactile actuator so as to adjust the time delay.

[0031] According to a third aspect, an embodiment provides a system for synchronizing an audio output of an audio speaker with a tactile output of a tactile actuator, the system comprising one or more processors configured to perform the following operations: generate an audio output from the audio speaker using a pilot input signal, and receive a speaker response signal from the audio speaker in response to the generated audio output; generate a tactile output from the tactile actuator using a pilot input signal, and receive a tactile response signal from the tactile actuator in response to the generated tactile output; determine a time delay from the speaker response signal and the tactile response signal; and adjust the input audio signal to the audio speaker and the input tactile signal to the tactile actuator to adjust the time delay.

[0032] According to a further aspect, an embodiment provides a computer readable medium having instructions thereon, the instructions being configured to cause one or more processors to perform the method according to any one of the first aspect, the second aspect or the third aspect.

[0033] In the case of referring to functional modules or units in the device embodiments for performing various functions or steps of the described methods, it should be understood that these modules or units can be implemented in hardware, software, or a combination of the two. When implemented in hardware, these modules can be implemented as one or more hardware modules, such as one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). When implemented in software, these modules can be implemented as one or more computer programs executed on one or more processors. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figures are provided to help describe various aspects thereof.

[0035] Figure 1 is a schematic representation of an exemplary system for implementing aspects of the present disclosure;

[0036] Figure 2 is a schematic representation of an exemplary system illustrating functional modules for implementing aspects of the present disclosure;

[0037] Figure 3A is a schematic representation of an exemplary graphic equalizer according to aspects of the present disclosure;

[0038] Figure 3B is from Figure 3A Graphic equalizer produces a graphical representation of the tactile and audio output;

[0039] Figure 4 is a graphical representation illustrating a speaker and haptic acceleration according to aspects of the present disclosure;

[0040] Figure 5 is a flow chart illustrating a process for generating a haptic drive signal according to aspects of the present disclosure;

[0041] Figure 6 is a schematic representation of an exemplary system illustrating functional modules for implementing aspects of the present disclosure;

[0042] Figure 7 is a flow chart illustrating a process for generating a haptic drive signal according to aspects of the present disclosure;

[0043] Figure 8 is a schematic representation of an exemplary system for implementing aspects of the present disclosure, according to an embodiment;

[0044] Fig. 9 is a schematic representation of an exemplary system illustrating functional components for implementing aspects of the present disclosure according to an embodiment;

[0045] Fig.10 is a graph illustrating an exemplary application of various aspects of the present disclosure;

[0046] Fig.11 is a flow chart illustrating a process for generating a haptic drive signal according to aspects of the present disclosure;

[0047] Fig.12 is a flow chart illustrating a process for time synchronization of audio and haptic output according to aspects of the present disclosure.

[0048] Fig.13 yes Figure 8 A schematic diagram of an exemplary system showing additional details of the system.

[0049] Detailed description of the drawings

[0050] The present disclosure will first provide a description of the main components of an exemplary system for implementing various aspects of the present disclosure. Another aspect of the present disclosure will then be discussed, and then the specific exemplary embodiments for implementing various aspects of the present disclosure will be turned to, and the details of implementing the embodiments described herein will be further discussed. Alternative aspects may be designed without departing from the scope of the present disclosure. In addition, well-known elements of the present disclosure will not be described in detail or will be omitted to avoid making the relevant details of the present disclosure difficult to understand.

[0051] Tactile signal generation

[0052] Reference Figure 1 , a schematic representation of the main components of an exemplary system 10 for implementing various aspects of the present disclosure is shown. System 10 is a system for haptic drive signal generation, which includes an audio-haptic drive signal generation module 11, which can be a general processing unit. This processing module 11 can be implemented in hardware or software, and can include, for example, one or more processors and a memory connected thereto, which is configured to perform various method steps for generating haptic drive signals, as will be discussed in detail below.

[0053] The system 10 also includes an audio speaker 12 and a tactile actuator 13, such as a linear resonant actuator (LRA), a piezoelectric resonator, or any other suitable tactile actuator. Although for ease of discussion, the system 10 has been shown as including the audio speaker 12 and the tactile actuator 13, it should be understood that aspects of the present disclosure are not limited to systems including the speaker 12 and the actuator 13. For example, the system may include only the audio-tactile drive signal generating module 11, which may then be connected to a separate audio speaker (or multiple audio speakers, as the case may be) and a separate tactile actuator (or multiple tactile actuators, as the case may be). The audio speaker is connected to the audio-tactile drive signal generating module so that the audio-tactile drive signal generating module can transmit an audio signal to the audio speaker to drive the audio speaker, and the audio speaker can transmit a signal (e.g., a feedback signal) back to the audio-tactile drive signal generating module. The audio signal here is an electronic signal that can be converted by the audio speaker into a corresponding audio output. As will be described with respect to Fig.13 As discussed in more detail, in some embodiments, the audio signal output from the audio-tactile drive signal generation module is a digital signal that can be passed to an audio amplifier ( Figure 1) to convert the digital signal into an analog output. The analog audio signal is then passed to the audio speaker. In other words, the audio signal can be converted from a digital format to an analog format before being passed to the audio speaker. Similarly, the feedback signal can be passed to the audio amplifier to be converted into a digital feedback signal, wherein the digital feedback signal is passed to the audio-tactile drive signal generation module. The feedback signal can be, for example, a current or voltage signal indicating the performance characteristics of the audio speaker, as will be further described below. Similarly, the tactile actuator is connected to the audio-tactile drive signal generation module so that the audio-tactile drive signal generation module can pass the tactile drive signal to the tactile actuator to drive the tactile actuator, and the tactile actuator can pass the signal (e.g., feedback signal) back to the audio-tactile drive signal generation module. The tactile drive signal is an electronic signal here, such as a voltage signal, which can be converted into a corresponding tactile output by the tactile actuator. The feedback signal can be, for example, a current or voltage signal indicating the performance characteristics of the tactile actuator, as will be further described below. Like the audio signal, in some embodiments, the haptic drive signal output from the audio-haptic drive signal generation module is a digital signal, which can be transmitted to the haptic driver circuit ( Figure 1 The haptic drive signal may be converted from a digital format to an analog format before being passed to the haptic actuator. Similarly, the feedback signal may be passed to the haptic driver circuit to be converted to a digital feedback signal, which is passed to the audio-haptic drive signal generation module.

[0054] Now refer to Figure 2 , shows the functional components of an exemplary audio-haptic drive signal generation module 11. Each component may be implemented in software or hardware as appropriate—for example, the components may be implemented by the one or more processors of module 11.

[0055] The audio-tactile driving signal generating module 11 includes a speaker module 21, a filter module 22, an envelope tracking module 23 (in Figure 2 ), an amplitude modulator 24, and a tactile waveform generator 25. It should be noted that in some embodiments, the envelope tracking module 23 can be an audio-to-tactile conversion module, that is, the module performs the function of audio-to-tactile conversion. In other cases, the combination of the envelope tracking module 23 and the amplitude modulator can act as an audio-to-tactile conversion module. It should be understood that other alternatives are also possible. It should be understood that, as Figure 2 The modules shown are for ease of discussion and may not represent separate components—modules may be combined or omitted depending on implementation and use cases.

[0056] The speaker module 21 is configured to receive an input audio signal u(t) for driving an audio speaker (in Figure 2 26 in the figure). This is an electronic signal that is converted by the audio speaker 12 into an audio output when it is passed to the audio speaker 12. In other words, the audio signal u(t) is an electronic signal representation of the audio output, or in other words, it is an electronic signal that can be used to drive the audio speaker to produce an audio output corresponding to the signal. The input audio signal u(t) can be a digital representation of the speaker drive signal, such as pulse code modulation (PCM). The input audio signal u(t) can be converted from a digital format to an analog format, for example, by a speaker audio amplifier. The analog signal can be an analog voltage drive signal that is passed to the speaker for conversion into an audio output. The audio signal u(t) can be a digital file representing a music track, a soundtrack for a video game or movie, a ringtone or application alert, or any suitable audio output. The audio signal can be stored in a memory, such as a memory of the audio-tactile drive signal generation module 11, or in another suitable memory of the system.

[0057] exist Figure 2 In the arrangement shown, a speaker module 21 and a filter module 22 (here a bandpass filter bank (BPF)) are configured to derive data components from the audio signal indicating the behavior of the audio speaker when driven by the input audio signal, each data component being associated with a corresponding frequency range.

[0058] In more detail, the speaker module 21 is configured to predict the mechanical behavior of the audio speaker 12 when driven by an audio signal. The prediction of the mechanical behavior includes inputting the audio signal u(t) into a model of the mechanical behavior of the audio speaker 12. The model is configured to generate a prediction of the mechanical behavior of the audio speaker when driven by the audio signal input into the model, i.e., the model outputs the expected behavior of the audio speaker when driven by a given audio signal. The output is a signal (or function) a s (t), which indicates the expected mechanical behavior of the loudspeaker 12 when driven by the input audio signal u(t).

[0059] The loudspeaker module 21 includes or has access to a model stored, for example, in a memory of the drive signal generation module 11, which may be a memory of the loudspeaker module 21 or a separate memory. In the present embodiment, the mechanical behavior is the acceleration of the loudspeaker diaphragm, which is the second derivative of the loudspeaker displacement. It should be understood that other mechanical behaviors are also suitable, such as the loudspeaker displacement or the pressure output of the loudspeaker diaphragm. For a given audio signal input to the loudspeaker model 21, the model provides the expected loudspeaker acceleration when the audio speaker is driven by the input audio signal.

[0060] The model may be defined by various predetermined electromechanical parameters of the specific audio speaker being used (the model may vary from speaker to speaker, i.e., the model may be tailored to a specific audio speaker in order to accurately predict the behavior of that specific speaker). The electromechanical parameters include those that allow the acceleration of the speaker for any given audio input to be determined. The speaker may be represented using a parametric lumped element model, but it should be understood that other models are possible. The model may be discretized into filters for determining the output displacement for any input voltage. The parameters of the speaker are lumped element parameters. These parameters describe the behavior of the speaker across various input signal amplitudes and across frequency bands of interest. Here is a reference for speaker modeling: https: / / www.aes.org / e-lib / browse.cfm?elib=13881.

[0061] The speaker module 21 may also be configured to adjust one or more parameters of the model based on operational feedback from the audio speaker. The speaker module 21 may be configured to receive operational feedback including current and voltage feedback and measured speaker electromechanical parameters (in Figure 2 , speaker current (1), voltage feedback (2), and speaker electromechanical parameters (3). The feedback is live feedback from the audio speaker 12 when the audio speaker is driven by an audio signal (e.g., an audio signal different from u(t)). This live feedback allows the speaker module 21 to determine the actual performance of the speaker in real time, and then adjust the model accordingly. This allows the speaker module 21 to take into account specific (e.g., current) operating conditions experienced by the audio speaker, such as temperature, age of the speaker, etc., in order to provide the most accurate and up-to-date model for the speaker as possible. In one example, the current and voltage feedback can be used to derive the current electromechanical parameters, which can then be used to update the parameters of the model. For example, the electromechanical parameters of the model can be adjusted to match those electromechanical parameters calculated based on the feedback, which will cause the behavior predicted by the model to match the actual behavior of the speaker. The speaker module can be configured to update the model using feedback at a specific frequency (e.g., every millisecond, ten milliseconds, 100 milliseconds, etc.). Although Figure 2 The arrangement shown in involves speaker current and voltage feedback to optimize the model, but it will be appreciated that embodiments not involving such feedback are possible. In such embodiments, the predicted behavior is simply determined by a set of predetermined models that are set up in advance, for example during initial setup or manufacturing. If a lumped element model is used, for example for some voltage inputs, and a specific lumped element model of the speaker and some recorded sensed currents, the speaker can be characterized. The lumped element parameters, if updated, represent a time-varying speaker. With knowledge of the specific model, the sensed current / voltage waveforms can be used to track the speaker parameters over time.

[0062] The speaker module may also be configured to adjust the input audio based on the predicted speaker behavior, for example to protect the speaker. This may be done using only the model, i.e. without live feedback, or including live feedback. In this way, the speaker module acts as a speaker protection module, such as a limiter, which prevents audio signals that would cause an audio response that could damage the speaker. For example, an audio input that the model predicts would cause mechanical behavior that could damage the speaker is modified so that the modified audio signal causes mechanical behavior that is safe for the speaker to perform (i.e., the risk of speaker damage is low).

[0063] Once the expected speaker acceleration a is output from the speaker module 21 s (t), it is passed to the BPF 22. It should be noted that in some embodiments, the mechanical behavior of the speaker may not be predictable at all, in which case the speaker module 21 (if present) simply passes the audio signal directly to the BPF module 22.

[0064] BPF module 22 is based on the mechanical behavior a s The determination of the data components by the BPF module 22 includes filtering the prediction of the mechanical behavior according to the audio frequency. The prediction of the mechanical behavior is filtered into the data components. In other words, the expected speaker acceleration a s (t) passes through a filter bank that extracts the acceleration a from the loudspeaker according to a specific frequency band. s (t) extracts the data component - the loudspeaker acceleration a associated with an audio range s The data components of (t) will be organized in a filter bank, and the loudspeaker acceleration a associated with the second audio range s The data components of (t) will be organized in the second filter bank, and so on. Therefore, the term "data component" is used here to refer to the loudspeaker acceleration a s The audio signal u(t) will be distributed over the frequency spectrum depending on the audio signal in question - it could be an audio signal with a lot of bass, tenor, alto, soprano, etc., or a profile distributed over different frequencies, etc. This audio frequency distribution will correspond to the expected loudspeaker acceleration a s (t) has a corresponding relationship. For example, the audio frequency profile can be the Fourier transform of the expected loudspeaker acceleration as(t) in the time domain. The expected loudspeaker acceleration a sSome components of (t) will be caused by audio components across a certain frequency range—for example, a bass audio signal component will generate the expected loudspeaker acceleration a s (t). Therefore, the speaker acceleration a s The data components of (t) are associated with audio frequencies and can therefore be filtered into filter banks according to the frequency range to which they are associated. In some embodiments, specific frequency bands up to a certain cutoff frequency are used to isolate the speaker acceleration. These frequency bands can be selected by the manufacturer. The frequency bands can depend on the choice of speaker and the specific oscillation frequency of the haptic oscillator. The output of the BPF unit 22 is the filtered speaker acceleration a f (t), which represents derived data components that indicate the behavior of the audio speaker 12 when driven by the input audio signal, each data component being associated with a corresponding frequency range. The behavior here is the mechanical acceleration of the speaker diaphragm, but it can also be the speaker displacement of the diaphragm, or other suitable measure, as discussed above. It should also be understood that filters other than bandpass filters can be used.

[0065] Filtered loudspeaker acceleration a f (t) (representing all data components) is then passed to the envelope tracking module 23. The processing up to this point can be considered as audio signal processing and is performed by the module group 28. The filtered speaker acceleration a f The transfer of (t) to the envelope tracking module 23 may be considered the beginning of haptic signal processing performed by the module group 29. The conversion from audio processing to haptic processing may be performed during the haptic signal processing. Specifically, a haptic drive component is determined from each data component, a gain is selected to be provided to each of the haptic drive components, the gain being selected for each corresponding frequency range, and each selected gain is applied to the corresponding haptic drive component to produce a gain-adjusted haptic drive component.

[0066] Determining the haptic drive component from each data component may include taking a moving average integration window over the data component. This is to establish a "profile" or "envelope" for the data component - the goal is to smooth out oscillations of subcomponents of the data component that occur across the frequency range of the data component. In this case, as the filtered speaker acceleration a for a particular frequency range f The tactile driving component a of the envelope of (t) env (t)(i.e., a fThe signal data component (t) indicates the desired mechanical property of the tactile actuator across the frequency range, here the acceleration of the tactile actuator. By using the envelope of the filtered speaker acceleration to determine the desired mechanical property of the tactile actuator (e.g., the desired acceleration), consistency between the audio output and the tactile response can be maintained. The specific envelope of the current audio signal will result in a tactile response that reflects the audio signal (i.e., is consistent with the audio signal), and more specifically results in the mechanical behavior of the speaker that is expected to be produced by the audio signal driving the speaker. This means that, since the properties of the speaker in question are taken into account (via the speaker model), consistency is maintained regardless of the specific speaker used. This means that the speaker acceleration a f (t) represents the mechanical properties of the specific speaker being driven, and thus the haptic driving component is adapted to the mechanical behavior of the specific speaker when the speaker is driven by the audio signal.

[0067] Then, for each haptic drive component a, the frequency associated with the haptic drive component is determined based on env (t) Select a gain and apply it to the corresponding tactile drive component. Thus, a gain is applied to each tactile drive component a based on the frequency range associated with the tactile drive component. env (t) providing separate gains. This allows tactile drive components associated with certain frequencies to be emphasized based on the use case, while tactile drive components associated with other frequencies are de-emphasized. For example, a tactile response to a low-frequency component may be emphasized on the basis of improving the user experience by providing a strong tactile response to bass, for example because the speaker in question cannot reproduce the audio output of the low frequency, so the tactile response is used instead. In one example, a first tactile drive component is associated with a first frequency range component, and a second gain is associated with a second tactile drive component associated with a second frequency range, the first frequency range being a lower frequency range than the second frequency range, and wherein providing a gain to the tactile drive component based on the frequency range associated with the tactile drive component includes providing a higher gain to the first tactile drive component associated with the first frequency range component than to the second tactile drive component associated with the second frequency range. For example, the second gain can be a zero gain.

[0068] The gain provided to each of these tactile drive components may be selectable by a user. For example, the gain may be selected via a graphical user interface GUI, such as that shown in FIG. 3 , which is divided into two parts: Figure 3A The Graphic Equalizer on the left is shown, which offers a variety of preset modes to cover various use cases. Figure 3BThe haptic and audio response for a selected preset mode is shown—in the case of FIG3 , “Game Mode.” A preset represents a predetermined selection of gains to be applied to the haptic drive component in order to emphasize the haptic response of certain audio frequencies, depending on the use case. For example, a game preset may emphasize low-frequency audio associated with explosions, gunshots, etc., so that these noises produce a strong haptic response, which may improve the user experience. The graphic equalizer may also provide a custom mode in which the haptic response may be increased for a given frequency range (a given haptic component a). env (t)) manually set each gain so that the user has maximum flexibility to adjust the tactile response according to preference. On the right side of Figure 3, the game preset tactile response is shown for the input audio. Here it is shown how to emphasize the tactile drive components associated with certain frequency ranges and de-emphasize the tactile drive components associated with other frequency ranges by applying gains. GUI user interface elements such as sliders can represent the adjustment of linear, logarithmic or other scales of gains based on human perception of tactile vibration intensity.

[0069] Once the gain is applied, the gain-adjusted haptic drive component a GA (t) determines a tactile drive signal, which is designed to generate a mechanical behavior of the tactile actuator when used to drive the tactile actuator. Figure 2 In the arrangement of , this occurs in two stages. First, the gain-adjusted haptic drive component a GA (t) is passed to the amplitude modulator 24, which uses the gain-adjusted tactile drive component a GA (t) modulates a tone at the haptic actuator—such as a tone at the linear resonant actuator (LRA) resonant frequency F0 tone 27. Here, a GA (t) represents all the individual gain-adjusted haptic drive components. These components provide the amplitude envelope to which the F0 tone is modulated. For example, these components can be recombined into a single signal before modulation. Modulation can be achieved using the following equation:

[0070] a h (t) = a GA (t)*sin(2ΠF0t)

[0071] The output of this module 24 is a h (t), which is indicative of the expected mechanical behavior of the tactile actuator across the entire frequency range (in Figure 2 Where is the signal or function of LRA acceleration), that is, the output signal LRA acceleration a h (t) Gain is taken into account. Figure 4A graphical representation of the LRA acceleration and how it aligns with the speaker acceleration is provided; the speaker acceleration provides the envelope of the tactile response, which provides the amplitude of the tone (or signal), here the LRA resonant frequency tone, which is modulated to produce the LRA acceleration a h (t). In other words, the signal / tone is modulated using an envelope—the envelope sets the amplitude of the signal, where the frequency of the signal is set by the resonant frequency of the LRA.

[0072] Secondly, the tactile actuator acceleration a h (t) is fed into the tactile waveform generator 25. The tactile waveform generator 25 is configured to determine the tactile drive signal (or waveform), here the LRA electrical drive voltage u h (t), that is, when the tactile driving signal is used to drive the tactile actuator, the tactile actuator is driven by the tactile actuator acceleration a h In other words, the waveform generator 25 analyzes the desired tactile actuator acceleration a h (t) and determines the drive voltage that produces the mechanical behavior. The details of the operation of the waveform generator 25 are discussed in detail under a separate heading below. However, in brief, the electrical drive voltage u is generated h (t) includes the actuator acceleration a h (t) is input into a model that maps the desired mechanical behavior of the haptic actuator to the drive signal required to produce that behavior. This is a reverse mapping compared to the model used for the haptic actuator module, which receives the input haptic drive signal and determines the expected haptic actuator behavior.

[0073] The waveform generator 25 includes or has access to a model stored, for example, in a memory of the drive signal generation module 11, which may be a memory of the waveform generator 25 or a separate memory. In this embodiment, the mechanical behavior is the acceleration of the LRA actuator. It should be understood that other mechanical behaviors may also be suitable, such as actuator displacement or pressure output of the actuator.

[0074] The model may be defined by various predetermined electromechanical parameters of the particular haptic actuator being used (the model may be actuator specific, i.e., the model may be tailored for a particular actuator in order to accurately predict the behavior of that particular actuator). The electromechanical parameters include those that allow the determination of the drive voltage for the desired mechanical behavior (in this case, for the desired actuator acceleration). For example, the electromechanical parameters may include one or more of the resonant frequency of the haptic actuator, the spring stiffness (k) of the haptic actuator, the mass (m), the mechanical damping factor (b), the electromechanical coupling factor (BL), the resistance, the inductance, or any combination thereof.

[0075] The tactile waveform generator 25 may also be configured to adjust one or more parameters of the model based on operational feedback from the tactile actuator 13. Specifically, when the tactile actuator is driven by the drive signal, the audio-tactile drive signal generation module 11 may perform one or more feedback measurements on the tactile actuator. In one aspect, an intelligent tactile driver may be an example of a component for performing feedback measurements. Specifically, a feedback sensor and / or a zero-crossing detector may be configured to perform feedback measurements. These components may be functional components of the tactile waveform generator 25, or may be separate components. For example, the tactile waveform generator 25 may be configured to receive operational feedback including current and voltage feedback and LRA electromechanical parameters measured directly from the tactile actuator 13 (in Figure 2 In one embodiment, the feedback sensor (e.g., waveform generator module 25) measures the LRA current (1), voltage feedback (2), and LRA electromechanical parameters (3)), i.e., module 25 performs the feedback measurement—or a separate component may perform the measurement and transmit the measurement result to module 25. As described above, the feedback may be converted from an analog signal received from the tactile actuator to a digital signal for use by the tactile waveform generator 25. In either case, the feedback sensor (e.g., waveform generator module 25) may measure the voltage across the tactile actuator that occurs due to the tactile actuator being driven. The voltage may be measured while the tactile actuator is driven with a drive signal. That is, the voltage may be measured simultaneously. Alternatively or in addition, the feedback sensor may measure the current flowing through the tactile actuator that occurs due to the tactile actuator being driven. The current may also be measured simultaneously while the tactile actuator is driven with a drive signal. On the other hand, the zero crossing detector may measure the back electromotive force (BEMF) induced in the tactile actuator due to the tactile actuator being driven with a drive signal and detect the zero crossing. The zero-crossing detector may also measure other characteristics of the BEMF, such as waveform ring-down, envelope decay, etc. Likewise, the zero-crossing detector may be part of module 25 or may be a separate component that communicates the measurement results to module 25 .

[0076] The feedback is from the haptic actuator when the haptic actuator is driven by a haptic drive signal (e.g., different from u h The haptic waveform generator 25 may provide real-time feedback of the haptic actuator being driven by a haptic drive signal (t). The real-time feedback allows the haptic waveform generator 25 to determine the actual performance of the haptic actuator in real time, and the model may then be adjusted accordingly. This may allow the waveform generator 25 to take into account specific (e.g., current) operating conditions experienced by the haptic actuator, such as temperature, age of the actuator, etc., in order to provide the most accurate and up-to-date model for the actuator as possible.

[0077] In one example, the current and voltage feedback can be used to derive current electromechanical parameters, which can then be used to update the parameters of the model. For example, the electromechanical parameters of the model can be adjusted to match those calculated from the feedback, which will produce the haptic drive signal u h(t), the haptic drive signal causes the haptic actuator to move in a desired manner (i.e., as indicated by a h (t) indicated) mechanically. Figure 2 The arrangement shown involves speaker current and voltage feedback to optimize the model, but it will be appreciated that embodiments not involving such feedback are also possible. In such embodiments, the drive voltage is simply determined by a predetermined set of models that are set in advance, such as during initial setup or manufacture.

[0078] Once the haptic drive signal u has been generated h (t), audio signal u(t) and tactile driving signal u h (t) can then be passed to the speaker 12 and actuator 13 to produce audio and tactile output. Importantly, the tactile drive signal has been generated from the expected mechanical behavior of the speaker 12 when driven by the audio signal in question. This maintains consistency between the audio and tactile output - the tactile output reflects the audio output, not only at the electronic level (i.e., the tactile drive signal does not simply reflect the input audio signal), but also takes into account the mechanical behavior of the speaker. In addition, live feedback can be provided to ensure that the mechanical behavior of the speaker and actuator is accurately modeled, and to ensure that the expected behavior of the tactile actuator is achieved and is appropriate for the specific speaker and actuator in question. Finally, gain is applied to the tactile drive component based on the frequency range of the audio, allowing certain components of the audio signal to be converted into a larger or smaller tactile response depending on the use case.

[0079] refer to Figure 5 Flowchart of the process, in use, Figure 1 The system for tactile drive signal generation operates as follows. Figure 5 At step 51, an audio signal is received by the audio-haptic drive signal generation module 11, for example, at the speaker module 21. The input audio may be received from a memory, from a server or a network, etc., and may represent a music track, audio for a video game, audio for a movie, etc. At step 52, the audio-haptic drive signal generation module 11 derives data components from the audio signal that indicate the behavior of the audio speaker when driven by the input audio signal, each data component being associated with a corresponding frequency range. As discussed above, this step may be performed by the speaker module 21 and the filter module 22. Each data component may be as described with respect to Figure 2discussed and may be determined in the same manner. For example, each data component may indicate the mechanical behavior of an audio speaker when driven by an audio signal across a corresponding frequency range, and specifically, the data component may indicate a speaker displacement, a speaker acceleration, or a speaker pressure when the audio speaker is driven by an input audio signal. As discussed above, deriving the data component may include predicting the mechanical behavior of an audio speaker when driven by an audio signal, and determining the data component from the prediction of the mechanical behavior. The prediction may include inputting the audio signal into a model of the mechanical behavior of the audio speaker (e.g., the model discussed above), the model being configured to generate a prediction of the mechanical behavior of the audio speaker when driven by an audio signal input into the model, such as with respect to Figure 2 The prediction can be the expected loudspeaker acceleration a s (t). The prediction may also include adjusting one or more parameters of the model based on operational feedback from the audio speaker, wherein the operational feedback is voltage and / or current feedback from the audio speaker, as previously discussed. Determining the data component from the prediction of the mechanical behavior may also include filtering the prediction of the mechanical behavior according to the audio frequency, filtering the prediction of the mechanical behavior into the data component to produce, for example, a filtered speaker acceleration a f (t).

[0080] In step 53, module 11 determines the tactile drive component a from each data component. env (t), and at step 54, module 11 selects a gain to provide to each of the tactile drive components, the gain being selected for each corresponding frequency range, and applies each selected gain to the corresponding tactile drive component to produce a gain-adjusted tactile drive component. These steps may be performed by envelope tracking module 23. The tactile drive components and gains may be as described above with respect to Figure 2discussed above and may be determined in the same manner. For example, determining the tactile drive component from each data component may include applying a moving average (or a digital FIR or IIR filter, as appropriate) to each data component to produce a corresponding tactile drive component. The gain provided to each of these tactile drive components may be selectable by a user, for example, via the GUI of FIG. 3 . As discussed above, a gain may be provided to the tactile drive component based on a frequency range associated with the tactile drive component. For example, a first tactile drive component may be associated with a first frequency range component, and a second tactile drive component may be associated with a second frequency range, the first frequency range being a lower frequency range than the second frequency range, and wherein providing a gain to the tactile drive component based on the frequency range associated with the tactile drive component includes providing a higher gain to the first tactile drive component associated with the first frequency range component than to the second tactile drive component associated with the second frequency range. The gain provided to the second tactile drive component can be a zero gain. Each tactile drive component a env (t) may indicate a mechanical behavior of the haptic actuator when driven by a haptic drive signal determined from the corresponding haptic drive component, for example, as described with respect to Figure 2 The acceleration of the haptic actuator in question.

[0081] Finally, at step 55, module 11 determines a haptic drive signal from the gain-adjusted haptic drive component, the haptic drive signal being designed to produce a mechanical behavior of the haptic actuator when used to drive the haptic actuator. Figure 2 modulator 24 and waveform generator 25, as discussed above. For example, determining the tactile drive signal from the gain-adjusted tactile drive component may include determining the tactile drive signal from the gain-adjusted tactile drive component to produce appropriate mechanical behavior of the tactile actuator indicated by the gain-adjusted tactile drive component. This may be achieved by amplitude modulating the tone / signal at the tactile actuator (such as a resonant frequency F0 tone of a linear resonant actuator (LRA)) using the gain-adjusted tactile drive component aGA(t), i.e., the signal / tone is modulated using an envelope defined by the tactile drive component - the envelope sets the amplitude of the signal, and the frequency of the signal is set by the resonant frequency, such as the LRA acceleration a h Here, the modulated acceleration a is obtained based on the acceleration envelope represented by the gain-adjusted tactile drive component and the resonant frequency of the tactile actuator. h (t)

[0082] The modulation signal is then input into a model of the mechanical behavior of the haptic actuator, which is configured to generate a haptic drive signal that will recreate the mechanical behavior indicated by the modulation signal when used to drive the haptic actuator. This may include adjusting one or more parameters of the model based on operational feedback from the haptic actuator, such as voltage and / or current feedback from the haptic actuator. This may be as described with respect to Figure 2 The waveform generator 25 is performed as described.

[0083] It should be noted that although the description includes Figure 2 The present disclosure of all modules and functions of the present invention is not limited to the aspects of the present invention, but other aspects are also contemplated. For example, according to some aspects, frequency-based filtering and the provision of gain may not be necessary. In such an arrangement, the audio-tactile drive signal generation module 11 can receive an input audio signal for driving an audio speaker (this can be as described with respect to Figure 2 Module 11 may then determine, based on the audio signal, a signal indicative of the mechanical behavior of the audio speaker when driven by the audio signal. This may be performed by a speaker module such as module 21 and may be performed in the same manner as described with respect to that module. For example, an expected speaker acceleration a may be determined s (t). Module 11 may then determine a tactile drive signal for the tactile actuator based on the signal indicative of the mechanical behavior of the audio speaker. Here, the tactile envelope may be directly derived from the expected speaker acceleration a s (t), i.e., not through BPF module 22. In this case, envelope tracking module 23 may not apply a gain to the haptic drive component. Instead, the envelope of the entire signal is determined. A tone at the haptic actuator (such as a linear resonant actuator (LRA) resonant frequency F0 tone) can then be amplitude modulated using this envelope to determine the haptic acceleration a by a modulator such as modulator 24. h (t), and thereby the drive voltage can be determined, for example, by waveform generator 25 as discussed above. It will therefore be appreciated that aspects are not limited to applying gain to a particular haptic drive component—in certain aspects, embodiments can be provided that derive the haptic drive signal directly from the expected mechanical properties of the speaker. The steps of this process are Fig.11 As shown in the flowchart, that is, in step 1101, an input audio signal for driving an audio speaker is received, in step 1102, a signal indicating the mechanical behavior of the audio speaker when driven by the audio signal is determined based on the audio signal, and in step 1103, a tactile driving signal for a tactile actuator is determined according to the signal indicating the mechanical behavior of the audio speaker.

[0084] As a further alternative, aspects involving filtering and gain application but not deriving the haptic components from the expected mechanical properties of the speaker are also possible. In such embodiments, the input audio may be fed directly into a filtering module (such as BPF module 22) and the haptic envelope (haptic drive component) determined directly from the input audio. This may be similar to the aspects regarding Figure 2 The haptic output can be done in the same manner as described above, for example, each haptic drive component is associated with a frequency range. Gains can then be applied to the derived haptic drive components, as discussed above. A modulator (such as modulator 24) can modulate these components to provide a modulated signal, and the signal can then be passed to a waveform generator (such as generator 25) to generate a drive voltage to drive the actuator. In such an arrangement, the haptic output will be linked to the audio by being derived directly from the audio input signal.

[0085] Before describing specific embodiments of various aspects of the present disclosure, we will now turn to another aspect of the present disclosure.

[0086] Time synchronization of audio and haptic output

[0087] The audio and tactile output may be processed as discussed above and delivered to, e.g. Figure 1 audio and tactile speakers. However, if the two outputs are not well synchronized in time, the tactile response may not appear to the user to be consistent with the audio output, which may reduce the user experience. Therefore, aspects of the present disclosure provide methods and systems for time synchronization of audio output and tactile output. It should be understood that time synchronization can be implemented in conjunction with the aspects discussed above for the generation of tactile drive signals, or separately (i.e., it can be used to synchronize audio and tactile outputs that are not generated as discussed above).

[0088] Reference Figure 6 , system 61 is used to synchronize the audio output of the audio speaker with the tactile output of the tactile actuator. Figure 6 Functional modules for performing synchronization are shown, which may be implemented in hardware or software as appropriate. System 61 may be implemented as Figure 1 A part of the system 10 , for example within the audio-haptic drive signal generation module 11 .

[0089] First, the system i) generates an audio output from an audio speaker using an input audio signal including a pilot input signal (the input audio signal may be a pilot tone, or may include a pilot tone in addition to other audio data), and receives a speaker response signal from the audio speaker in response to the generated audio output, and ii) generates a tactile output from a tactile actuator using an input signal including the pilot input signal, and receives a tactile response signal from the tactile actuator in response to the generated tactile output. Figure 6 In the example shown, a single input audio signal including a pilot audio tone (e.g., a predetermined audio signal) is generated, and thereby audio output and tactile output are generated from an audio speaker (e.g., speaker 12) and a tactile actuator (e.g., actuator 13), respectively. Generating audio output and tactile output from the pilot input signal may follow, for example, the method described above with respect to Figure 2 and Figure 5 The process described above, wherein the signal u(t) represents a pilot tone. It will be appreciated that alternatives are possible, such as simply providing separate pilot audio and haptic signals to generate audio and haptic outputs.

[0090] Once the audio output and tactile output have been generated by the pilot tone (pilot input signal), feedback voltage and / or current signals are received. Here, a feedback voltage signal v(t) is received from the audio speaker and a feedback voltage signal v(t) is received from the tactile actuator. h (t). Both signals pass through the filter module, here is Figure 6 The BPF modules 62 and 63 are used to extract the loudspeaker response signal v from these filter modules. pTa (t) and tactile response signal v PTh (t). These signals are voltage response signals specific to the pilot tone. For example, the pilot tone may have a specific frequency, or may be designed to generate a specific output frequency from the speaker and actuator. Filter modules 62 and 63 may filter this frequency to eliminate feedback signals not due to the pilot tone, thereby extracting feedback specific to the pilot tone (the specific feedback is the response signal v pTa (t) and v PTh (t)).

[0091] Response signal v pTa (t) and v PTh (t) is then passed to a delay estimation module 64 where the time delay between the response signals is calculated. More specifically, the speaker acceleration and tactile actuator acceleration generated by the pilot tone can be derived from the response signals (e.g., using a model similar to that described above that maps voltage outputs to corresponding accelerations based on the electromechanical properties of the speaker or actuator preconfigured into the model), and the time difference between the accelerations can be determined. In some embodiments, the time difference between the corresponding speaker and tactile acceleration can be determined by applying a specific delay to the tactile acceleration. The cross-correlation between the speaker and the tactile acceleration is then calculated based on the candidate delay value. Similarly, the cross-correlation between the speaker and the tactile acceleration is calculated for a set of delays that take values ​​between a certain delay range (e.g., -0.5 seconds to +0.5 seconds). The delay value that maximizes the cross-correlation between the speaker and the tactile acceleration is the estimated delay of the system, Figure 6 It is denoted as t0 in the figure. It should be understood that other fractional delay estimation processes in the frequency domain are also possible.

[0092] Once the time delay is determined based on the speaker response signal and the haptic response signal, the next haptic drive signal u to be sent to the haptic actuator is h (t) is input into the delay estimation module 64. The next haptic drive signal is then offset by this time delay so as to eliminate the delay between the audio output and the haptic output. This is represented by the output drive signal u h (t-t0), where t0 is the calculated time delay. This is equivalent to closed-loop feedback on the tactile drive voltage to eliminate the delay. Of course, it should be understood that such closed-loop feedback can be applied equally to the audio drive signal or to both, as long as the signal adjustment is such as to eliminate the delay between the audio and tactile output. The steps of this synchronization process are as follows: Fig.12 . In step 1201, an audio output is generated from an audio speaker using a pilot input signal, and a speaker response signal is received from the audio speaker in response to the generated audio output. In step 1202, a tactile output is generated from a tactile actuator using the pilot input signal, and a tactile response signal is received from the tactile actuator in response to the generated tactile output. In step 1203, a time delay is determined from the speaker response signal and the tactile response signal. In step 1204, an input audio signal to the audio speaker and an input tactile signal to the tactile actuator are adjusted to adjust the time delay.

[0093] Speaker Module

[0094] The operation of the speaker module 21 of the audio-haptic drive signal generation module 11 according to aspects of the present disclosure will now be described in more detail.

[0095] The speaker module 21 uses a model of the loudspeaker 12 (specifically, the displacement transfer function h(t) of the loudspeaker 12) and the input audio signal u(t) to predict (e.g., estimate) the displacement of the loudspeaker (audio speaker) 12 for the input audio signal u(t) (e.g., which may be a speech signal).

[0096] More specifically, in order to estimate the displacement, the original audio / speech signal (input signal) u(t) is filtered using the impulse response of the displacement transfer function h(t) of the loudspeaker to estimate the displacement e(t) of the input audio / speech signal. If the impulse response of the displacement transfer function h(t) of the loudspeaker 12 is known, the displacement e(t) can be estimated by e(t)=h(t)*u(t), where * represents the convolution of two sequences. For example, this can be performed by an IIR filter, but other specific implementations are also possible.

[0097] As about Figure 2 As discussed, the model of the speaker 12 (i.e., the transfer function h(t)) is a known function stored in memory and available to the speaker module 21. The transfer function h(t) may be initially predetermined based on known or expected electromechanical properties of the speaker, such as during manufacture or initial setup of the system 10. These properties may be known properties common to speakers of the type of speaker 12, or may be determined by testing the speaker 12 during manufacture. These properties will determine the initial parameters that define the model / transfer function. The parameters of the model may then be updated in real time as the current electromechanical properties of the speaker (i.e., its properties in operation) are determined by live feedback, as discussed above.

[0098] Once e(t) has been determined, the second derivative of e(t) may be determined by the speaker module 21 to determine the expected speaker acceleration a s (t). It will be appreciated that other suitable metrics, such as expected loudspeaker pressure, may be used as appropriate. This may be achieved by using an appropriate model / transfer function.

[0099] The speaker module may also be configured to adjust the input audio based on the predicted speaker behavior, for example to protect the speaker. This may be done using only the model, i.e. without live feedback, or including live feedback. In this way, the speaker module acts as a speaker protection module, such as a limiter, which prevents audio signals that would cause an audio response that could damage the speaker. For example, an audio input that the model predicts would cause mechanical behavior that could damage the speaker is modified so that the modified audio speaker causes mechanical behavior that is safe for the speaker to perform (i.e., the risk of speaker damage is low).

[0100] For example, a maximum speaker displacement may be calculated based on the speaker's transfer function h(t), and the audio input signal may be modified to constrain the audio speaker to keep the displacement below or equal to the maximum displacement.

[0101] In addition to limiting the displacement, the speaker module may determine in parallel adjustments to the audio signal for optimal audio output given known operating conditions of the speaker. For example, the speaker module may simultaneously limit the displacement and also adapt the signal to maximize the perceived volume of the limited displacement. Obviously, other limiting and optimization functions are possible. For example, this function may be performed together with the determination of the speaker acceleration and may be performed by appropriate software running on the hardware of the system.

[0102] Waveform Generator

[0103] The operation of waveform generator 25 of audio-haptic drive signal generation module 11 according to aspects of the present disclosure will now be described in more detail.

[0104] The generator 25 can calculate the velocity vel(t) and displacement e of the tactile actuator based on the modulated acceleration (t). H (t). In one aspect, the following equations can be applied to calculate vel(t) and e H (t):

[0105]

[0106] The generator 25 can be based on the modulated acceleration a h (t), velocity vel(t), displacement e H (t) and one or more parameters of the haptic actuator to calculate the electromagnetic force F(t) to move the moving mass of the haptic actuator. In one aspect, the following equation can be applied to calculate F(t):

[0107] F(t)=m h *a h (t)+b h *vel(t)+k h *e H (t)

[0108] In the above equation for F(t), m h It can represent the mass of the moving part of the tactile actuator, b h can represent the damping factor of the tactile actuator, and k h may represent the spring stiffness of the haptic actuator. Note that these are identified as haptic actuator parameters and can be tracked by continuously monitoring feedback measurements, such as Figure 2 In other words, the parameters of the tactile actuator can be continuously monitored and updated through closed-loop feedback.

[0109] The waveform generator 25 may calculate the voltage u of the driving signal based on the electromagnetic force F(t) and the one or more parameters of the haptic actuator. h (t) and the current I(t) through the haptic actuator. In one aspect, the voltage u can be calculated using the following equation: h (t) and current I(t):

[0110] I(t) = F(t) / BL h (6)

[0111]

[0112] In the above, for I(t) and u h In the two equations of (t), the resistance R of the tactile actuator may represent the ohmic voltage drop, L may represent the inductance, and BL h The drive signal u may represent the electromechanical coupling factor of the tactile actuator. Again, these may be tracked by the tracker 219 by continuously monitoring the feedback measurements.h (t) can be an analog drive voltage applied to a haptic actuator by a device (e.g., an audio-tactile drive signal generation module, which can be, for example, part of a mobile device). In one aspect, a digital coder / decoder CODEC and / or a smart haptic driver can be examples of components for performing the process steps described in this section. In some embodiments, an amplifier can be configured to drive a haptic actuator using the generated drive signal.

[0113] Figure 7 The waveform generator 25 is shown in FIG. h (t) is a flowchart of the steps that can be taken when generating a drive voltage. These steps are as follows. In step 71, based on the modulated acceleration a h (t) to calculate the velocity vel(t) and displacement e of the tactile actuator H (t). In step 72, the electromagnetic force F(t) is calculated based on the velocity vel(t), the displacement e(t) and the one or more parameters of the tactile actuator. In step 73, the voltage u of the drive signal is calculated based on the electromagnetic force F(t) and the one or more parameters of the tactile actuator. h (t) and the current I(t) through the tactile actuator. These steps can be implemented as described above in this section.

[0114] Additional embodiments

[0115] Now about Figure 8 , Fig. 9 and Fig.13 Specific implementations for implementing various aspects of the present disclosure are discussed. Figure 8 An audio-tactile drive signal generation module 80 is shown, which may correspond to Figure 1 Module 11, but it should be understood that module 11 is not limited to Figure 8 The modules shown in FIG. Figure 8 The main functional modules of this embodiment are shown. Fig.13 Shows Figure 8 Module 80 includes an audio-tactile signal generator 81, which can perform Figure 2 All functions described herein are described and may be implemented by appropriate hardware and / or software (e.g., by a processor). Module 80 also includes an audio driver 82 such as an audio amplifier and a haptic driver 83, which may perform analog-to-digital (ADC) or digital-to-analog (DAC) conversion of signals transmitted between the audio-haptic signal generator and the speaker and actuator.

[0116] The audio-tactile signal generator 81 may include a processing unit 1301 that may implement the various processing steps discussed herein. For example, the processing unit may be implemented in one or more processors and appropriate software. Figure 2 and Figure 6 Functional modules. Fig.13 As shown, the processing unit includes an audio synchronization waveform generation module, which illustrates the various modules described here for performing audio to tactile conversion. The audio-tactile signal generator 81 has an output terminal 1302 for digital audio signal output and digital tactile signal output. These outputs are connected to an interface and register unit 1303, which receives signals and directs the signals to their appropriate destinations. The processing unit 1301 also has an input terminal for receiving feedback from speakers and actuators (such as temperature, battery voltage, and sensed current and voltage). These feedbacks are fed into an algorithm unit, which updates the speaker and actuator models as discussed above.

[0117] The audio driver 82 may receive an audio input signal (eg, Figure 2 The audio driver is configured to drive the speaker with the input signal u(t) shown in FIG. 1 . In some embodiments, the audio driver is directly connected to the speaker to drive the speaker. In addition, the audio driver may perform Figure 2 The feedback measurement discussed - the audio driver can measure the voltage and current values ​​of the speaker as well as other relevant operating parameters of the speaker (e.g., temperature) while the speaker is operating. The audio driver can feed this back to the audio-haptic drive signal generation module 80 so that the module 80 updates the speaker module (or transfer function) for generating the speaker acceleration a s (t). The audio driver may include a feedback sensor for performing these feedback measurements on the speaker. The audio driver may also convert the digital input signal u(t) into a corresponding analog drive voltage for driving the speaker. In more detail, the audio driver 82 may receive an audio signal output from the audio-tactile signal generator 81. The audio signal may be received at an interface module 1304 that performs various functions (such as speaker protection and signal routing). The received audio signal (which is a digital signal) is passed to the DAC1305 for digital-to-analog conversion before being passed to the speaker. In addition, the interface module 1304 receives a feedback signal, which may then be passed to the signal generator 81. For example, temperature, current, and voltage readings from the speaker are obtained, converted to digital values ​​by a separate ACD converter 1306, and the digital values ​​are passed to the interface module 1304. The temperature, current, and voltage readings may be obtained by appropriate sensors.

[0118] The haptic driver 83 may receive a haptic driving signal from the audio-haptic signal generator 81, such as Figure 2 The tactile driving signal u h (t). This signal can be Figure 2 The haptic driver 83 may be configured to drive the haptic actuator using the received drive signal. In some embodiments, the haptic driver 83 is directly connected to the haptic actuator for driving the actuator. In addition, the haptic driver may perform Figure 2 The feedback measurement discussed - the haptic driver can measure the voltage and current values ​​of the speaker and other relevant operating parameters of the speaker (e.g., temperature) when the speaker is operating. The audio driver can feed this back to the audio-haptic drive signal generation module 80 so that the module 80 updates the haptic module (or transfer function) for generating the haptic acceleration a h (t). The tactile driver may include a feedback sensor for performing these feedback measurements on the actuator. The tactile driver may also convert the digital tactile drive signal into a corresponding analog drive voltage for driving the actuator. In more detail, the tactile driver 83 may receive a tactile signal output from the audio-tactile signal generator 81. The tactile signal may be received at a tactile driver interface module 1307 that performs various functions (such as actuator protection and signal routing). The received tactile signal (which is a digital signal) is passed to the DAC 1308 for digital-to-analog conversion before being passed to the actuator (here, the LRA). In addition, the interface module 1307 receives a feedback signal, which may then be passed to the signal generator 81. For example, temperature, current, and voltage readings from the actuator are obtained, converted to digital values ​​by a separate ACD converter 1309, and the digital values ​​are passed to the interface module 1307. The temperature, current, and voltage readings may be obtained by appropriate sensors.

[0119] Steering Fig. 9 , showing example implementations for implementing various aspects of the present disclosure. Fig. 9 A system 90 for generating tactile signals is shown, the system comprising: Figure 8 The audio-tactile signal generator 81 is a functional module implemented in hardware and / or software. Fig. 9 The functional modules show the details of various aspects of the present disclosure and can be Figure 2 It should be noted that Figure 2 The system is not limited to Fig. 9 Specific implementation plan - Fig. 9 The implementation scheme shows Figure 2 Possible specific implementations of the system.

[0120] like Fig. 9 As shown, the input audio data (for example, the audio signal u(t), such as Fig. 9The audio signal is input to the speaker displacement prediction and acoustic output filter 91 (shown as unit 99 in FIG. 21 ). This filter module 91 derives the speaker displacement e(t) from the input audio and outputs the displacement to a power estimator 92 which determines the signal power from the displacement e(t). In this way, modules 91 and 92 act as the speaker module 21 of FIG. 21 , i.e., they are functionally equivalent. It should be noted that in Fig. 9 In some embodiments, the signal power is derived from the speaker displacement rather than from the speaker acceleration, however it will be appreciated that either may be used as appropriate for a particular use case. In any case, the signal power remains representative of the relevant mechanical properties of the speaker - the power quantity is used merely for ease of operation. It will be appreciated that conversion to signal power may not be necessary, and that simply using a direct displacement or acceleration signal may also be possible. The derived power signal is then separated into data components (indicative of the behavior of the audio speaker when driven by an input audio signal - the displacement of the speaker membrane) according to frequency. This is done by Fig. 9 Filter bank 93 is executed. Components of the power signal associated with sub-bass (20Hz to 60Hz) audio are separated into the sub-bass band library. Components of the power signal associated with bass (60Hz to 250Hz) audio are separated into the bass band library. Components of the power signal associated with low-mid frequency (250Hz to 500Hz) audio are separated into the low-mid frequency band library. Components of the power signal associated with mid frequency (500Hz to 2kHz) audio are separated into the mid frequency band library. Components of the power signal associated with high-mid frequency (2kHz to 4kHz) audio are separated into the high-mid frequency band library. Components of the power signal associated with presence (4kHz to 6kHz) audio are separated into the presence band library. Components of the power signal associated with brightness (6kHz to 20kHz) audio are separated into the brightness band library. Each data component is associated with a corresponding frequency range. It should be noted that although in Fig. 9 The full frequency range has been shown in , but a subset of the full range may be used depending on the use case. For example, for narrower range applications, the brightness and sub-bass band libraries may be discarded. For other applications, the lower range (e.g., sub-bass and bass) band libraries may be discarded, or the higher range band libraries (brightness and presence) may be discarded, or a wider band library may be used (e.g., sub-bass and bass combination, mid-range band library combination, presence and brightness combination, etc.), or a narrower band library may be used. It should be understood that many permutations are possible depending on the use case.

[0121] Filter bank 93 is Figure 2 An exemplary implementation of the filter module 22 is shown.

[0122] Once the frequency band library has been determined, each data component is passed to the integration module 94, which performs a separate integration operation on each data component. The operation is a moving average integration within the integration window, and the envelope of the tactile drive signal in each frequency range is determined. The envelope of a given frequency range is referred to herein as a tactile drive component. The moving average integration window can be one cycle or half a cycle of the actuator resonant frequency, that is, T = (1 / F0) or T = (0.5 / F0). The integration operation of the data component can be expressed as follows:

[0123]

[0124] or

[0125]

[0126] Depends on whether signal power p, direct displacement e, or acceleration a is used. The period can be fixed for all frequency bands, or can be set individually for each frequency band - for example, each frequency band can have its own period. The period T can also be set to capture specific types of sounds. For example, low-frequency sounds may require a longer integration period to ensure that the signal power (i.e., displacement, acceleration, or pressure) from the sound is captured in the envelope (e.g., a smaller period may only capture a portion of the signal). For higher frequency sounds, a smaller period can be used.

[0127] Each haptic drive component p env (t) is then passed to a gating module 95 which applies a nonlinear noise gating function, which may be controlled by the graphic equalizer of FIG3. Here, the threshold Th may be n and gain G n Applied to each haptic drive component p env (t) n , where n is a specific frequency range associated with the tactile drive component. In other words, each tactile drive component is associated with its own threshold and gain. The threshold is used to eliminate abnormal sub-components within the envelope, that is, it acts as a smoothing function to eliminate artifacts in the signal. The gains are used to emphasize or de-emphasize their corresponding tactile drive components and can be controlled by the user. In some use cases, the user can also control the setting of the threshold, such as via the graphic equalizer of Figure 3. For example, a standard mode may be provided in which only the gain is selectable, and a separate advanced mode is provided for the technician in which both the threshold and the gain are selectable by the equalizer. It should be understood that in some embodiments, a threshold is not used.

[0128] The integration module 94 and the gating module 95 are exemplary implementations of the envelope tracking module 23 .

[0129] The gain-adjusted haptic drive components are then recombined into the full signal p by appropriate operations. GA (t) (which represents all gain adjusted components) and is passed to the modulator 96 which modulates the signal at the resonant frequency - this can be expressed here as p h (t) = p GA (t)*sin(2ΠF0t).

[0130] Modulator 96 is Figure 2 An exemplary implementation of the amplitude modulator 24 is shown.

[0131] Finally, the modulation signal p h (t) is passed to the waveform designer 97. If the acceleration is used directly, the waveform designer obtains (a h (t)) modulation signal p h (t), and generates the actuator drive voltage u h (t), such as Fig. 9 As shown in unit 98 in FIG. 97 , the waveform designer 97 is Figure 2 An exemplary implementation of the waveform generator 25 and as described with respect to Figure 2 and operate as described under the Waveform Generator heading.

[0132] It should be noted that in the above, aspects of the present disclosure have been discussed that involve the use of frequency bands. However, embodiments that do not use frequency bands are also possible. For example, a point conversion can be performed where certain sounds with known audio waveforms are converted into tactile responses in the time domain. Here, the audio-to-tactile conversion will follow the same Figure 2 The same principle as shown in Figure 1 is that the audio waveform will be used to generate a consistent tactile response. However, here, no segmentation operation based on frequency is performed because the input audio signal will have a discrete waveform that can be converted into a consistent tactile response using aspects of the present disclosure. For example, in a gaming application, a gunshot may have a specific waveform. The expected mechanical behavior of the speaker can be determined from this waveform, and a consistent tactile response can be generated from this, for example, by using the models and transformations discussed here (but without the filter bank). In addition, the tactile response can be variable - the generated tactile waveform can vary based on different metrics, such as who pulled the trigger (for example, if the user pulled the trigger or another player pulled the trigger, a different tactile response will be generated). This can be detected based on user input or based on data retrieved from the game engine.

[0133] Sample Application

[0134] One application of various aspects of the present disclosure is to target tactile responses for a library of lower audio frequency bands (e.g., bass and sub-bass). The reason is that audio speakers often perform less than ideally in these lower frequency bands. For example, the audio speakers of a mobile phone may not achieve optimal performance in the bass and sub-bass ranges. Therefore, by emphasizing the tactile response for these lower frequency ranges, the tactile response can supplement the audio in these lower ranges, making the reduced performance of the speakers in these ranges less obvious to the user. Here, there is both audio and tactile response at the low frequencies; the tactile response supplements the audio response. An alternative is to combine the audio-to-tactile conversion discussed here with psychoacoustic bass. In this approach, the audio speaker actually has no audio response at the fundamental frequency of a given lower bass frequency. In this case, the speaker can play certain harmonic frequencies of the lower bass frequency, while the tactile actuator converts the missing fundamental frequency in the audio waveform into a tactile response. For example, when the missing fundamental frequency is 85 Hz, its second harmonic 170 Hz and third harmonic 255 Hz can be played through the audio speaker, and the fundamental frequency 85 Hz can be converted from the audio to tactile and played through the tactile actuator. Here, the conversion may first derive (e.g., mathematically) the expected fundamental frequency waveform from the harmonics, and then use the derived fundamental frequency audio waveform to generate a tactile response, for example, according to the aspects described herein.

[0135] Fig.10 This principle is illustrated in FIG. 1 , where a graph illustrating increasing actuator activation across a frequency range of decreasing audio activity is provided. Specifically, curve 101 illustrates actuator activity, illustrating that activity increases over a frequency range of decreasing audio activity. 2 Hz and 10 3 Hz, where the audio response 102 is low. The tactile actuator activity then decreases to 10 3 with 10 4 The lower levels between 10 and 20 Hz where speaker activity reaches its highest point.

[0136] This tactile response is achieved by selecting an appropriate gain, for example, via the graphic equalizer of FIG. Fig. 9 In an embodiment of the present invention, the haptic drive components associated with the sub-bass and bass frequency ranges may be provided with high gains, i.e., high gain values ​​G1 and G2 may be selected, and zero or near-zero gains are selected for the remaining haptic drive components. In this way, in the gain-adjusted signal, the low-frequency haptic drive components are emphasized while the remaining components are suppressed. When converted to the final voltage drive signal, the emphasized low-frequency components will be converted to a larger drive signal and will therefore produce a larger activity in the haptic actuator for this audio range. In other words, the low-frequency audio components of the audio signal will result in larger accelerations of the haptic actuator - they will be converted to the emphasized haptic drive components.

[0137] It will be appreciated that for other applications, the gain may be tailored differently to achieve a different tactile response.

[0138] Aspects of the present disclosure have applications in a variety of use cases, such as in mobile phones, wearable devices, virtual reality, gaming, and IOT devices. Applications include audio-synchronized haptics, game vibrations (such as gunshots, bomb explosions, and car engines), application alerts, ringtones, and notifications involving both audio and haptics, and audio and / or video playback with haptic responses, for example, users watching videos and / or listening to audio content on mobile devices for an immersive experience, tracking wireless headset latency (e.g., Bluetooth latency), and considering playback using, for example, speakers attached to your glasses or frames or other headphones (using micro speakers), etc.

[0139] Advantages include improved haptic waveform generation, i.e., a waveform better suited to the audio signal and the properties of both the speaker and the actuator; improved timing synchronization between audio and haptic output, and improved bass perception, for example on a handheld device.

Claims

1. A method for generating a tactile drive signal, the method comprising: receiving an input audio signal for driving an audio speaker; deriving from the audio signal data components indicative of behavior of the audio speaker when driven by the input audio signal, each data component being associated with a respective frequency range; determining a haptic drive component from each data component; selecting a gain provided to each of the haptic drive components, the gain being selected for each respective frequency range; applying each selected gain to the corresponding haptic drive component to produce a gain-adjusted haptic drive component; as well as A haptic drive signal is determined from the gain-adjusted haptic drive component, the haptic drive signal being designed to produce a mechanical behavior of the haptic actuator when used to drive the haptic actuator. 2 . The method of claim 1 , wherein each data component is indicative of a mechanical behavior of the audio speaker when driven by the audio signal across the corresponding frequency range. 3 . The method of claim 2 , wherein the data component indicates speaker acceleration when the audio speaker is driven by the input audio signal.

4. The method of claim 2, wherein deriving the data component comprises: predicting the mechanical behavior of the audio speaker when driven by the audio signal; as well as The data component is determined from the prediction of the machine behavior.

5. The method of claim 4 , wherein the predicting comprises inputting the audio signal into a model of the mechanical behavior of the audio speaker, the model being configured to generate a prediction of the mechanical behavior of the audio speaker when driven by the audio signal input into the model.

6. The method of claim 5, wherein the predicting further comprises adjusting one or more parameters of the model based on operational feedback from the audio speaker.

7. The method of claim 6, wherein the operational feedback comprises voltage and / or current feedback from the audio speaker.

8. The method of claim 4, wherein determining the data component from the prediction of the mechanical behavior comprises filtering the prediction of the mechanical behavior into the data component based on audio frequency. 9 . The method of claim 1 , wherein determining the haptic drive component from each data component comprises applying a moving average to each data component to generate a corresponding haptic drive component.

10. The method of claim 1, wherein the gain provided to each of the haptic drive components is selectable by a user. 11 . The method of claim 10 , wherein the gain provided to each of the haptic drive components is selectable by a user via a graphical user interface (GUI). 12 . The method of claim 1 , wherein each haptic drive component indicates a mechanical behavior of the haptic actuator when driven by a haptic drive signal determined from the corresponding haptic drive component.

13. The method of claim 12, wherein determining a haptic drive signal from the gain-adjusted haptic drive component comprises: The haptic drive signal is determined from the gain-adjusted haptic drive component to produce the mechanical behavior of the haptic actuator indicated by the gain-adjusted haptic drive component.

14. The method of claim 13, wherein determining the haptic drive signal from the gain-adjusted haptic drive component comprises: amplitude modulating a tone at a resonant frequency of the tactile actuator using the gain adjusted tactile drive component to generate a modulation signal; as well as The modulation signal is input into a model of the mechanical behavior of the haptic actuator, the model being configured to generate a haptic drive signal that, when used to drive the haptic actuator, will recreate the mechanical behavior indicated by the modulation signal. 15 . The method of claim 14 , wherein determining the haptic drive signal from the gain-adjusted haptic drive component further comprises adjusting one or more parameters of the model based on operational feedback from the haptic actuator. 16 . The method of claim 15 , wherein the operational feedback comprises voltage and / or current feedback from the haptic actuator. 17 . The method of claim 1 , further comprising providing a gain to the haptic drive component based on the frequency range associated with the haptic drive component.

18. The method of claim 17, wherein a first tactile drive component is associated with a first frequency range component and a second tactile drive component is associated with a second frequency range, the first frequency range being a lower frequency range than the second frequency range, and wherein providing a gain to the tactile drive component based on the frequency range associated with the tactile drive component comprises providing a higher gain to the first tactile drive component associated with the first frequency range component than a gain provided to the second tactile drive component associated with the second frequency range. The method of claim 18 , wherein the gain provided to the second haptic drive component can be a zero gain.

20. The method of claim 1, further comprising: synchronizing the audio output of the audio speaker with the tactile output of the tactile actuator, the synchronizing comprising: generating an audio output from the audio speaker using a pilot input signal and receiving a speaker response signal from the audio speaker in response to the generated audio output; generating a tactile output from the tactile actuator using a pilot input signal, and receiving a tactile response signal from the tactile actuator in response to the generated tactile output; determining a time delay from the speaker response signal and the tactile response signal; An input audio signal to the audio speaker and an input haptic signal to the haptic actuator are adjusted to adjust the time delay.

21. A system for haptic drive signal generation, the system comprising one or more processors configured to: receiving an input audio signal for driving an audio speaker; deriving from the audio signal data components indicative of behavior of the audio speaker when driven by the input audio signal, each data component being associated with a respective frequency range; determining a haptic drive component from each data component; selecting a gain provided to each of the haptic drive components, the gain being selected for each respective frequency range; applying each selected gain to the corresponding haptic drive component to produce a gain-adjusted haptic drive component; as well as A haptic drive signal is determined from the gain-adjusted haptic drive component, the haptic drive signal being designed to produce a mechanical behavior of the haptic actuator when used to drive the haptic actuator.

22. The system of claim 21, wherein each data component is indicative of a mechanical behavior of the audio speaker when driven by the audio signal across the corresponding frequency range.

23. The system of claim 22, wherein the data component indicates speaker acceleration when the audio speaker is driven by the input audio signal.

24. The system of claim 23, wherein deriving the data component comprises: predicting the mechanical behavior of the audio speaker when driven by the audio signal; as well as The data component is determined from the prediction of the machine behavior.

25. The system of claim 24, wherein the prediction comprises inputting the audio signal into a model of the mechanical behavior of the audio speaker, the model being configured to generate a prediction of the mechanical behavior of the audio speaker when driven by the audio signal input into the model.

26. The system of claim 25, wherein the predicting further comprises adjusting one or more parameters of the model based on operational feedback from the audio speaker.

27. The system of claim 26, wherein the operational feedback comprises voltage and / or current feedback from the audio speaker.

28. The system of claim 24, wherein determining the data component from the prediction of the mechanical behavior comprises filtering the prediction of the mechanical behavior into the data component based on audio frequency.

29. The system of claim 21, wherein determining the haptic drive component from each data component comprises applying a moving average to each data component to generate a corresponding haptic drive component.

30. The system of claim 21, wherein the gain provided to each of the haptic drive components is selectable by a user.

31. The system of claim 30, wherein the gain provided to each of the haptic drive components is selectable by a user via a graphical user interface (GUI).

32. The system of claim 21, wherein each haptic drive component indicates a mechanical behavior of the haptic actuator when driven by a haptic drive signal determined from the corresponding haptic drive component.

33. The system of claim 32, wherein determining a haptic drive signal from the gain-adjusted haptic drive component comprises: The haptic drive signal is determined from the gain-adjusted haptic drive component to produce the mechanical behavior of the haptic actuator indicated by the gain-adjusted haptic drive component.

34. The system of claim 33, wherein determining the haptic drive signal from the gain-adjusted haptic drive component comprises: amplitude modulating a tone at a resonant frequency of the tactile actuator using the gain adjusted tactile drive component to generate a modulation signal; as well as The modulation signal is input into a model of the mechanical behavior of the haptic actuator, the model being configured to generate a haptic drive signal that, when used to drive the haptic actuator, will recreate the mechanical behavior indicated by the modulation signal. 35 . The system of claim 34 , wherein determining the haptic drive signal from the gain-adjusted haptic drive component further comprises adjusting one or more parameters of the model based on operational feedback from the haptic actuator.

36. The system of claim 35, wherein the operational feedback comprises voltage and / or current feedback from the tactile actuator.

37. The system of claim 21, wherein the one or more processors are further configured to provide a gain to the haptic drive component based on the frequency range associated with the haptic drive component.

38. A system according to claim 37, wherein a first tactile drive component is associated with a first frequency range component and a second tactile drive component is associated with a second frequency range, the first frequency range is a lower frequency range than the second frequency range, and wherein providing a gain to the tactile drive component based on the frequency range associated with the tactile drive component includes providing a higher gain to the first tactile drive component associated with the first frequency range component than a gain provided to the second tactile drive component associated with the second frequency range.

39. The system of claim 38, wherein the gain provided to the second haptic drive component can be a zero gain.

40. The system of claim 21, wherein the one or more processors are further configured to: synchronizing the audio output of the audio speaker with the tactile output of the tactile actuator, the synchronizing comprising: generating an audio output from the audio speaker using a pilot input signal and receiving a speaker response signal from the audio speaker in response to the generated audio output; generating a tactile output from the tactile actuator using a pilot input signal, and receiving a tactile response signal from the tactile actuator in response to the generated tactile output; determining a time delay from the speaker response signal and the tactile response signal; An input audio signal to the audio speaker and an input haptic signal to the haptic actuator are adjusted to adjust the time delay.

41. The system of claim 21, further comprising: A memory coupled to the one or more processors.

42. The system of claim 21, further comprising: The audio speaker and / or the haptic actuator, the audio speaker and / or the haptic actuator are connected to the one or more processors.

43. A computer-readable medium comprising instructions which, when executed, cause one or more circuits of an apparatus for processing data to perform the method steps of any one of claims 1 to 20.

Images