Method and apparatus for outputting a haptic signal to a haptic transducer

Abstract

The embodiments described herein relate to methods and apparatus for outputting haptic signals to haptic transducers. A method for triggering the output of a haptic signal to a haptic transducer includes: receiving an audio signal for output via an audio output transducer; determining whether the audio signal includes a haptic trigger based on an indication of a rate of change of the amplitude of the audio signal; and, in response to determining that the audio signal includes a haptic trigger, triggering the output of the haptic signal to the haptic transducer.

Description

[0001] This application is a divisional application of the invention patent application with an application date of April 4, 2019, an application number of 201980023336.0, and a title of "Method and Apparatus for Outputting Tactile Signals to a Tactile Transducer". Technical Field

[0002] The embodiments described herein relate to methods and apparatus for outputting tactile signals to a tactile transducer based on received audio signals, the received audio signals being output via a loudspeaker. Background Art

[0003] For some applications, it may be desirable to generate tactile effects on mobile devices and other devices (such as the user interfaces of smart home devices or vehicles). In some examples, it may be desirable for these tactile effects to be synchronized with specific audio events to enhance the user experience. In some cases (e.g., where the audio source material (e.g., a specific song of a known type) is pre-defined in the system), it is conceivable that a series of tactile events designed to be output along with the song can be stored and replayed independently of the known audio.

[0004] However, there may be examples of audio playback that do not have a predictable or pre-defined nature. For example, in a gaming application, the rendered audio output may include a combination of stored elements (e.g., background music) and other sounds (which may depend on the gameplay). The precise timing and output of the latter category of sounds may depend on the user's actions in the game. Due to such user control, the output of a stored series of tactile events may be inappropriate. Summary of the Invention

[0005] According to some embodiments, there is provided a method for triggering a tactile signal that is output to a tactile transducer. The method includes: receiving an audio signal for output via an audio output transducer; determining whether the audio signal includes a tactile trigger based on an indication of a rate of change of the amplitude of the audio signal; and in response to determining that the audio signal includes a tactile trigger, triggering the tactile signal to be output to the tactile transducer.

[0006] In some embodiments, the determining step includes: generating a first weighted sum representation of the audio signal using a first number of samples of a modified form of the audio signal; generating a second weighted sum representation of the audio signal using a second number of samples of the modified form of the audio signal, wherein the second number of samples is greater than the first number of samples; and comparing the first weighted sum representation with the second weighted sum representation. This comparison can provide an indication of the rate of change of the amplitude of the audio signal.

[0007] In some embodiments, the comparing step includes: determining that the audio signal includes a haptic trigger when the first weighted sum representation exceeds the second weighted sum representation. In some embodiments, the comparing step includes: determining that the audio signal includes a haptic trigger when the first weighted sum representation exceeds the second weighted sum representation plus a predetermined offset.

[0008] In some embodiments, the method further includes adjusting a first set of weighting coefficients used to generate the first weighted sum representation and / or adjusting a second set of weighting coefficients used to generate the second weighted sum representation based on desired haptic trigger characteristics.

[0009] In some embodiments, the method further includes adjusting the first number of samples and / or the second number of samples based on desired haptic trigger characteristics.

[0010] In some embodiments, the method further includes determining whether the audio signal includes a haptic trigger based on a rate of change of an amplitude of the audio signal.

[0011] According to some embodiments, there is provided a method that includes: receiving an audio signal for output via an audio output transducer; determining a haptic signal for output to a haptic transducer based on the audio signal and at least one criterion; and dynamically adjusting the at least one criterion based on a parameter, where the parameter includes one or more of the following: information about the content of media playback being played back along with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and a battery charge state of the device.

[0012] In some embodiments, the method further includes: determining that the audio signal includes a haptic trigger in response to the audio signal meeting a first criterion of the at least one criterion; and outputting the haptic signal to the transducer in response to determining that the audio signal includes the haptic trigger.

[0013] The first criterion may include a threshold rate of change of the audio signal.

[0014] In some embodiments, the method further includes determining the haptic signal based on the haptic trigger and a second criterion of the at least one criterion.

[0015] In some embodiments, the method further includes generating the haptic signal from the haptic trigger based on a third criterion of the at least one criterion.

[0016] The third criterion may include one or more of the following: maximum amplitude, allowable dynamic range, and allowable frequency range.

[0017] In some embodiments, the adjusting step includes: in response to the parameter indicating that the future repetition rate of the tactile signal event should be reduced, adjusting the at least one criterion to reduce the repetition rate of the tactile signal event derived from the audio signal.

[0018] In some embodiments, the parameter indicates whether the audio signal contains speech, and the adjusting step includes: in response to the parameter indicating that the audio signal contains speech, adjusting the at least one criterion to reduce the repetition rate of the tactile signal event.

[0019] In some embodiments, the parameter includes the battery charge state of the device, and the adjusting step includes in response to the parameter indicating that the battery charge state is below a threshold level, adjusting the at least one criterion to reduce the repetition rate of the tactile signal event.

[0020] In some embodiments, the parameter includes information about the content of the media playback played back together with the audio signal; and the adjusting step includes adjusting the at least one criterion based on the content of the media playback to select a tactile signal.

[0021] The media playback may include video data.

[0022] In some embodiments, the parameter includes information about historical tactile events associated with the audio signal; and the adjusting step includes adjusting the at least one criterion to optimize the expected future output of repeated tactile events.

[0023] According to some embodiments, a method for generating a tactile signal for output to a tactile transducer is provided. The method includes: receiving an audio signal for output through an audio output transducer; determining whether the audio signal contains a tactile trigger; in response to determining that the audio signal contains the tactile trigger, by undersampling data in the audio signal containing the tactile trigger, mapping energy at frequencies outside an expected frequency range in the tactile trigger to frequencies within the expected frequency range in the tactile signal, such that energy at frequencies above the expected frequency range in the tactile trigger is converted to frequencies within the expected frequency range, to generate the tactile signal.

[0024] In some embodiments, the method further includes adjusting the dynamic range of the tactile trigger to generate the tactile signal.

[0025] In some embodiments, the mapping step further includes distorting the audio signal energy at frequencies below the expected frequency range in the haptic trigger such that corresponding harmonic energy is generated within the expected frequency range in the haptic signal.

[0026] According to some embodiments, a processor for triggering a haptic signal that is output to a haptic transducer is provided. The processor includes: an input configured to receive an audio signal for output via an audio output transducer; a haptic trigger detection block configured to determine whether the audio signal includes a haptic trigger based on an indication of a rate of change of the amplitude of the audio signal; and a haptic signal generator configured to, in response to determining that the audio signal includes a haptic trigger, trigger the haptic signal to be output to the haptic sensor.

[0027] According to some embodiments, a processor is provided that outputs a first haptic signal to a haptic transducer in a device. The processor includes: an input configured to receive an audio signal for output via an audio output transducer; a haptic signal determination block configured to determine a haptic signal for output to the haptic transducer based on the audio signal and at least one criterion, wherein the processor is configured to dynamically adjust the at least one criterion based on a parameter, wherein the parameter includes one or more of the following: information about the content of media playback played back together with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and the battery charge state of the device.

[0028] According to some embodiments, a processor for generating a haptic signal for output to a haptic transducer is provided. The processor includes: an input configured to receive an audio signal for output via an audio output transducer; a haptic trigger detection block configured to determine whether the audio signal includes a haptic trigger; and a haptic signal generator configured to, in response to determining that the audio signal includes the haptic trigger, map the energy at frequencies outside the expected frequency range in the haptic trigger to frequencies within the expected frequency range in the haptic signal by undersampling the data in the audio signal that includes the haptic trigger, such that the energy at frequencies above the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range to generate the haptic signal.

[0029] According to some embodiments, an electronic device including a processor is provided, the processor being configured to trigger a haptic signal to be output to a haptic transducer. The processor includes: an input configured to receive an audio signal for output through an audio output transducer; a haptic trigger detection block configured to determine whether the audio signal includes a haptic trigger based on an indication of a rate of change of the amplitude of the audio signal; and a haptic signal generator configured to trigger the haptic signal to be output to the haptic transducer in response to determining that the audio signal includes a haptic trigger.

[0030] According to some embodiments, an electronic device including a processor is provided, the processor being configured to output a first haptic signal to a haptic transducer in a device. The processor includes: an input configured to receive an audio signal for output through an audio output transducer; a haptic signal determination block configured to determine a haptic signal for output to the haptic transducer based on the audio signal and at least one criterion; wherein the processor is configured to dynamically adjust the at least one criterion based on a parameter, the parameter including one or more of the following: information about the content of media playback played back together with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and the battery charge state of the device.

[0031] According to some embodiments, an electronic device including a processor is provided, the processor being configured to generate a haptic signal for output to a haptic transducer. The processor includes: an input configured to receive an audio signal for output through an audio output transducer; a haptic trigger detection block configured to determine whether the audio signal includes a haptic trigger; and a haptic signal generator configured to map energy at frequencies outside an expected frequency range in the haptic trigger to frequencies within the expected frequency range in the haptic signal by undersampling the haptic trigger data in response to determining that the audio signal includes the haptic trigger, such that energy at frequencies higher than the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range to generate the haptic signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] To better understand embodiments of the present disclosure and to illustrate how embodiments of the present disclosure may be implemented, reference is now made, by way of example only, to the accompanying drawings in which:

[0033] Figure 1 An example schematic diagram is illustrated showing a portion of a device 100 having an audio output transducer and a haptic output transducer;

[0034] Figure 2 Illustrates an embodiment of a processor for triggering a haptic signal H OUT where the haptic signal H OUT is output to a haptic transducer;

[0035] Figure 3 is an example flowchart illustrating a method for determining whether an audio signal includes a haptic trigger according to some embodiments;

[0036] Figure 4a Illustrates an example audio signal, a first weighted sum representation of the audio signal, and a second weighted sum representation of the audio signal;

[0037] Figure 4b Illustrates an example audio signal and a rate of change of the audio signal;

[0038] Figure 5 Illustrates an embodiment of a processor for triggering a haptic signal that is output to a haptic transducer in a device according to some embodiments;

[0039] Figure 6 Illustrates an embodiment of a processor including an audio output transducer and a haptic transducer according to some embodiments, wherein the processor outputs a haptic signal to the haptic transducer;

[0040] Figure 7 is an example flowchart illustrating a method for triggering a haptic signal that is output to a haptic transducer according to some embodiments;

[0041] Figure 8 is an example flowchart illustrating a method for outputting a first haptic signal to a haptic transducer in a device according to some embodiments;

[0042] Figure 9 is an example flowchart illustrating a method for outputting a haptic signal to a haptic transducer according to some embodiments. DETAILED DESCRIPTION

[0043] The following description sets forth example embodiments in accordance with the present disclosure. Other example embodiments and implementations will be apparent to those of ordinary skill in the art. Additionally, those of ordinary skill in the art will recognize that various equivalent techniques may be applied in lieu of or in combination with the embodiments discussed below, and all such equivalents should be considered to be covered by the present disclosure.

[0044] Figure 1is an example schematic diagram showing a portion of a device 100 having a corresponding audio output transducer 102 and a haptic output transducer 104. The device 100 may include any electronic device, such as: a portable device; a battery-powered device; a computing device; a communication device; a gaming device; a mobile phone; a personal media player; a laptop computing device, a tablet computing device or a notebook computing device, a smart watch, a virtual reality (VR) device or an augmented reality (AR) device or a smart home device. For clarity, Figure 1 elements of the device 100 that are not relevant to the present disclosure are not shown, but those skilled in the art should understand that in addition to the Figure 1 elements and components shown therein, the device 100 may include other elements and components.

[0045] The device 100 includes a processor 106, which may be, for example, an application processor. The processor 106 is interfaced with a signal processor 108, which may be, for example, a digital signal processor (DSP). The signal processor 108 is interfaced with an audio output amplifier 110, which is configured to output an audio signal to drive the audio output transducer 102. The signal processor 108 is also interfaced with a haptic output amplifier 112, which is configured to output a haptic signal to drive the haptic transducer 104.

[0046] The processor 106 executes and provides the operating environment of the device 100 to allow software applications (such as games) to be executed by the device 100. Such applications typically receive user input. The user input may include one or more of the following: touch user input and / or gesture user input that may be detected by a touch-sensitive surface (such as a touch screen) of the device (not shown); kinematic user input (such as rotating or tilting the device) that may be detected by sensors (such as an accelerometer or gyroscope of the device) (also not shown); and audio user input (such as spoken commands) that may be detected by sensors (such as a microphone of the device) (also not shown). In response to detecting user input, the application is operable to generate an appropriate output at the device. For example, the application is operable to cause the image displayed on a display (not shown) of the device to be updated and to cause an appropriate audio effect to be output by the audio output transducer 102. The application is also operable to cause an appropriate haptic output to be provided by the haptic output transducer 104 in response to detecting user input.

[0047] For example, in a driving game application, a user can interact with the touch screen of device 100, e.g., by touching a graphical representation of an accelerator pedal or a brake pedal presented on the touch screen of the device, so as to accelerate or brake a vehicle in the game. Such a touch is detected by the touch screen and causes the image displayed on the display of the device to be updated by the application to give a visual indication that the vehicle is accelerating or braking. The update of the image displayed on the display may be accompanied by appropriate audio effects output by the output audio transducer 102, such as an engine revving effect when touching the accelerator pedal, or a tire screeching effect when touching the brake pedal.

[0048] To turn the vehicle, the user can rotate the device counterclockwise to turn left or clockwise to turn right. In response to such kinetic input detected by the accelerometer and / or gyroscope of the device, when appropriate, the image displayed on the display is updated by the application to give a visual indication of the vehicle's movement. Additionally, appropriate audio effects, such as a tire screeching effect, may be output by the output audio transducer 102.

[0049] Audio user inputs (such as voice commands) can additionally be used, for example, to perform special functions such as firing a weapon. When a valid voice command or other audio user input is detected, when appropriate, the image displayed on the display can be updated to give a visual indication of the effect of the command, and appropriate audio effects, such as a gunfire sound effect, can be output by the output audio transducer 102.

[0050] The application can also be configured to use the tactile output transducer 104 to provide additional sensory information to the user. In the driving game application embodiment discussed above, in addition to updating the image displayed on the display and outputting appropriate audio effects, device 100 can also provide tactile output in response to user input. For example, the tactile output transducer 104 can be actuated to cause device 100 to vibrate while updating the image displayed on the display and outputting audio effects through the output audio transducer 102, thereby providing additional sensory information to the user.

[0051] In such an application, haptic output can be designed into the application from the start. In other words, the designer of the application can identify the user input, visual output, and audio output to be associated with the haptic output and manually program the application to provide the haptic output in response to the identified user input or in combination with the identified visual output or identified audio output. As should be understood, at the programming stage, this manual programming is a time-consuming process. Alternatively, some simple processing (e.g., filtering or correction) can be applied to the audio signal, and this processed signal can be used to drive the haptic transducer. However, the disadvantage of this method is that the haptic output reflects the amplitude of the audio output, losing the desired effect of making the haptic response only highlight or emphasize certain events. In addition, keeping the haptic circuit continuously active is detrimental to the battery life in a mobile device.

[0052] Accordingly, it may be desirable to be able to selectively scan the audio signal for relevant events and derive a suitable haptic output. In some applications (such as, game applications), the user experience for events (such as, a shot on the screen, or for example, other "loud" virtual events, or events with specific characteristics) can be enhanced through haptic response. Thus, these events can be considered "haptic triggers". Accordingly, in some embodiments, haptic triggers are characterized by portions in the audio signal where there are large energy changes or high-speed changes in energy.

[0053] Figure 2 Illustrated is a processor 200 for triggering a haptic signal H OUT which is output to a haptic transducer 201. In some embodiments, the haptic signal H OUT can be driven to the haptic transducer 201 after being amplified by a haptic amplifier 208. OUT

[0054] The processor 200 includes an input 202 for receiving an audio signal A IN which is for output through an audio output transducer 203. The audio signal A IN can be driven through the audio output transducer 203 after being amplified by an audio amplifier 207. The output transducer 203 and the processor 200 can be part of the same device or can be an external transducer on a separate device, such as a speaker or a headset. Thus, the device including the processor 200 can include an output suitable for connection to such an external transducer. IN

[0055] The processor 200 further includes a haptic trigger detection block 204. The haptic trigger detection block 204 is configured to receive the audio signal A IN and, based on the audio signal AIN determine the audio signal A based on an indication of the rate of change of the amplitude IN whether it includes a haptic trigger. A haptic trigger can be considered as a part of the audio signal that has a certain characteristic indicating its suitability for output together with the haptic signal.

[0056] For example, if the rate of change of the amplitude of the audio signal is higher than a first threshold, then this rate of change can indicate that an audio event occurs in the audio signal A IN which can serve as a basis for a haptic event, thereby highlighting or emphasizing the audio event to the user.

[0057] Then, the haptic trigger detection block 204 can indicate to the haptic signal generator 205 whether the audio signal A IN includes a haptic trigger. In response to receiving an indication that the audio signal includes a haptic trigger, the haptic signal generator 205 can trigger the haptic signal H OUT wherein the haptic signal H OUT is output to the haptic transducer 201.

[0058] In some embodiments, the indication includes an indication of the type of the occurring audio event. For example, the indication can include information indicating that the audio signal includes a gunshot sound. Thus, the haptic signal generator 205 can generate an appropriate haptic signal H based on the indicated type of audio event in the audio signal that causes the haptic trigger, for example, by selecting a pre - defined haptic signal from the memory 209 OUT .

[0059] In some embodiments, the indication can indicate the rate of change of the amplitude of the audio signal A IN . Thus, the haptic signal generator 205 can generate an appropriate haptic signal H based on the rate of change of the amplitude of the audio signal A IN . For example, if the rate of change of the amplitude of the audio signal A OUT is low, for example, lower than a second threshold (the second threshold is higher than the first threshold), then the haptic signal generator 205 can generate a first type of haptic signal H IN . For example, the first type of haptic signal can include beeping or piece - wise linear envelope (PWLE) type waveforms, which are embodiments of simple and effective ways to store haptic waveforms. OUT However, if the rate of change of the amplitude of the audio signal A

[0060] is high (e.g., higher than the second threshold), then the haptic signal generator 205 can generate a second type of haptic signal H IN . The second type of haptic signal H OUT . OUTIt may include a waveform that is shorter and more impulsive than the first type of tactile signal. For example, the second type of tactile signal may include a "more aggressive" or "snapping" waveform and is stored as a waveform file composed of sampled discrete data in a device (or memory).

[0061] In some embodiments, the indication may include an audio signal A IN itself. In other words, the tactile trigger detection block 204 may output the audio signal A to the tactile signal generator in response to detecting that the audio signal contains a tactile trigger. IN In this embodiment, the tactile signal generator may generate a tactile signal by processing the audio signal A that contains a tactile trigger. The processing applied may be as described later with reference to IN Figure 6

[0062] In other embodiments, the tactile signal generator 205 may receive the audio signal A IN as a separate input from the indication. In this embodiment, the indication may mark for the tactile signal generator 205 when the audio signal A IN contains a tactile trigger, so that when to generate a tactile signal H IN from the audio signal A OUT

[0063] In some embodiments, particularly in embodiments where a tactile signal H IN is generated from the audio signal A OUT , the audio signal A IN may be output to the audio amplifier 207 after being delayed by the delay block 206. In particular, the delay block 206 may be configured to introduce a delay into the path between the input 202 and the audio amplifier 207, which is the same as the delay introduced by the processing in the tactile signal generator 205 between the input 202 and the tactile amplifier 208.

[0064] Figure 3 Illustrates a method for determining whether an audio signal contains a tactile trigger according to some embodiments. For example, Figure 3 the method illustrated in Figure 2 may be performed by the tactile trigger detection block 204 illustrated in Figure 4a . To illustrate an embodiment of this method, refer to the example audio signal 400 in

[0065] ​​​In step 301, the haptic trigger detection block 204 generates a first weighted sum representation 402 of the modified form 401 of the audio signal 400 using a first number of samples of the modified form 401 of the audio signal 400. The first weighted sum representation 402 provides a representation of the fluctuations in energy in the modified form 401 of the audio signal 400.

[0066] In step 302, the haptic trigger detection block 204 generates a second weighted sum representation 404 of the modified form 401 of the audio signal 400 using a second number of samples of the modified form 401 of the audio signal 400, where the second number of samples is greater than the first number of samples used in step 301.

[0067] When more samples are used to generate the weighted sum representation of the signal, the representation is less responsive to the occurrence of peaks and valleys in the audio signal. Thus, the second weighted sum representation 404 is less responsive to peaks and valleys in the modified form 401 of the audio signal 400 compared to the first weighted sum representation 402.

[0068] In step 303, the haptic trigger detection block 204 can compare the first weighted sum representation 402 with the second weighted sum representation 404. This comparison can indicate the rate of change of the amplitude of the audio signal 400.

[0069] In step 304, the haptic trigger detection block 204 can determine whether the audio signal 400 contains a haptic trigger based on the comparison made in step 303.

[0070] For example, when the first weighted sum representation 402 exceeds the second weighted sum representation 404 (as illustrated by the marker 406 in Figure 4a ), the haptic trigger detection block 204 can determine that the audio signal 400 contains a haptic trigger.

[0071] In some embodiments, when the first weighted sum representation 402 exceeds the second weighted sum representation 404 plus a predetermined offset, the haptic trigger detection block 204 can determine that the audio signal contains a haptic trigger.

[0072] It should be understood that during some applications, the processor can adjust the sensitivity of the haptic output. For example, during some applications, the processor 200 can adjust a first set of weighting coefficients (which can include one or more coefficients) used to generate the first weighted sum representation and / or adjust a second set of weighting coefficients (which can include one or more coefficients) used to generate the second weighted sum representation based on desired haptic trigger characteristics. In other words, the weighted sums can be changed such that a higher or lower rate of change of the amplitude of the audio signal is required to trigger a haptic event.

[0073] Adjustment of the first or second weighting factor may also adjust the frequency response, which is included in the respective weighted sum representation. For example, the second weighted sum may be configured to filter out more high frequencies than the first weighted sum; or, create a high-pass filtering effect so that the second weighted sum ignores any bass content. In this way, the comparison of the weighted sums may search for a haptic trigger within a desired frequency range.

[0074] In some embodiments, the haptic trigger detection block may determine whether the audio signal no longer contains a haptic trigger based on an indication of the rate of change of the amplitude of the audio signal. In some embodiments, the comparison between the first weighted sum representation 402 and the second weighted sum representation 404 may determine the length of the haptic signal or the amplitude of the haptic signal. For example, when the difference between the first weighted sum representation 402 and the second weighted sum representation 404 is greater, the haptic signal is longer or the amplitude of the haptic signal is higher.

[0075] In response to determining that the audio signal no longer contains a haptic trigger, the haptic trigger detection block may indicate this condition to the haptic signal generator 205.

[0076] In some embodiments, when the first weighted sum representation 402 drops back below the second weighted sum representation 404, the haptic trigger detection block may determine that the audio signal no longer contains a haptic trigger.

[0077] In some embodiments, when the first weighted sum representation 402 drops to a predetermined offset below the second weighted sum representation 404, the haptic trigger detection block 204 may determine that the audio signal no longer contains a haptic trigger. In this embodiment, a certain hysteresis may be used.

[0078] It should be understood that in some embodiments, multiple different weighted sum representations and associated offsets may be used to detect different types of haptic triggers corresponding to different types of audio events. Then, the haptic signal may be generated according to which type of haptic trigger is present.

[0079] To trigger the haptic signal output to the haptic transducer such that the haptic signal is synchronized with the audio signal containing the haptic trigger, the processing for generating the haptic signal may have time limitations. This factor may effectively limit the range and functionality of the decision points available for registering haptic triggers based on changes in the energy level in the audio signal.

[0080] Figure 4b An embodiment according to some embodiments is illustrated, in which the rate of change of the audio signal is used to determine whether the audio signal contains a haptic trigger.

[0081] In this embodiment, an audio signal 408 and a rate of change 410 of this audio signal are illustrated. When the rate of change 410 of the audio signal is higher than a predetermined threshold 412, a haptic trigger can be detected. This detection is illustrated by a flag 414.

[0082] Figure 5 A processor 500 for triggering a haptic signal H according to some embodiments is illustrated. OUT The haptic signal H OUT is output to a haptic transducer 201 in the device. Features similar to those illustrated in Figure 2 have been given the same reference numerals. In some embodiments, the haptic signal H OUT can be driven through the haptic transducer 201 after being amplified by a haptic amplifier 208.

[0083] The processor 500 includes an input 501 for receiving an audio signal A IN for output through an audio output transducer 203. The audio signal A IN can be driven through the audio output transducer 203 after being amplified by an audio amplifier 207.

[0084] The processor 500 includes a haptic signal determination block 502. The haptic signal determination block can be configured to determine a haptic signal based on an audio signal and at least one criterion for output to a transducer. In some embodiments, the haptic signal determination block 502 includes a haptic trigger detection block 204 and a haptic signal generator 205 as illustrated in Figure 2 . In these embodiments, in response to the audio signal satisfying a first criterion of at least one criterion, it is determined that the audio signal includes a haptic trigger, and in response to determining that the audio signal includes a haptic trigger, a haptic signal is output to the transducer. As described with reference to Figure 2 , the first criterion can thus include a threshold rate of change of the amplitude of the audio signal, which can be represented by a comparison between a first weighted sum representation of the audio signal and a second weighted sum representation of the audio signal.

[0085] In this embodiment, at least one criterion can include a second criterion. For example, the second criterion can include a second threshold as described above. In other words, if the rate of change of the amplitude of the audio signal A IN is high (e.g., higher than the second threshold), the haptic signal generator 205 can generate a second type of haptic signal H OUT . The second type of haptic signal H OUTIt may include waveforms that are shorter and more pulsating than the first type of haptic signal. For example, the second type of haptic signal may include a click waveform. Alternatively, the second criterion may include the relationship between the value of the rate of change of the amplitude of the audio signal and the amplitude of the determined haptic signal. In other words, in some embodiments, when the rate of change is higher, the amplitude of the generated haptic signal is higher.

[0086] In some embodiments, the haptic signal determination block 502 may generate a haptic signal from a haptic trigger based on a third criterion among at least one criterion. For example, the third criterion may include one or more of the following: the maximum amplitude of the haptic signal, the allowable dynamic range of the haptic signal, and the allowable frequency range of the haptic signal. Thus, a haptic signal may be generated from the audio signal such that the haptic signal satisfies the third criterion.

[0087] It should be understood that other methods may be utilized to generate a haptic signal from an audio signal, and the other methods may use different criteria to determine whether a haptic signal will be output and what the haptic signal will be. However, Figure 2 the illustrated embodiment in [document] may be considered a fast response decision stage, which can thus quickly provide a haptic signal to maintain synchronization with the audio signal.

[0088] Then, the processor 500 may be configured to dynamically adjust at least one criterion based on a parameter, where the parameter includes one or more of the following: information about the content of media playback being played back together with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and the battery charge state of the device. Thus, compared to determining the haptic signal, this process of dynamically adjusting the criterion based on the parameter can be performed over a longer period of time, thereby providing greater flexibility in the types of information that can be used to determine the haptic signal.

[0089] In other words, the haptic signal determination block 502 may include a process with lower latency than the process required to dynamically adjust at least one criterion. Thus, this higher-latency process can be used for: a) applying more processing power to determining the haptic signal, and b) adopting algorithms and techniques more suitable for larger audio data sets. By adjusting at least one criterion, the higher-latency process can effectively train or adapt the lower-latency process without affecting the latency of the output haptic signal.

[0090] The criteria described above may be adjusted by the processor to reduce the repetition rate of haptic signal events derived from the audio signal. For example, in some cases, the parameter may indicate that the repetition rate of the occurring haptic signal events is too high (e.g., the user may have indicated through user input that the haptic events are too frequent) and that the future repetition rate should be reduced.

[0091] For example, in cases where the parameter includes the threshold change rate of the amplitude of an audio signal, the adjustment may include adjusting the number of samples for the first weighted sum representation and / or the second weighted sum representation for the audio signal. Alternatively, the adjustment may include increasing or decreasing an offset to decrease or increase the frequency of detecting a tactile trigger.

[0092] In some embodiments, the processor includes a voice detection block 503 configured to determine whether the audio signal contains speech. Thus, the parameter may include information about the content of the audio signal that indicates whether the audio signal contains speech. It should be understood that in some embodiments, the processor 500 may receive an indication of whether the audio signal contains speech from an external processor.

[0093] In these embodiments, the processor may be configured to adjust at least one criterion to reduce the repetition rate of tactile signal events in response to a parameter indicating that the audio signal contains speech. This adjustment may be configured to prevent plosive consonants (such as "p") in the speech of the audio signal from causing an unwanted tactile response. In some embodiments, at least one criterion may be adjusted to turn off the generation of any tactile signals in response to the speech indicated by the parameter.

[0094] In some embodiments, the parameter includes the battery charging state of the device. In these embodiments, the processor may adjust at least one criterion to reduce the repetition rate of tactile signal events in response to the parameter indicating that the battery state is below a threshold level. In other words, if the device's battery is low, the tactile response may be reduced to save battery power for applications that are considered more important for the operation of the device. For example, in response to the parameter indicating that the device's battery is below a threshold level, at least one criterion may be adjusted to reduce one or more of the following: the intensity, length, or temporal density of the tactile signal output to the tactile transducer. Thus, such an adjustment to at least one criterion may provide a finer management of the trade-off between low battery situations and the quality of the user experience.

[0095] In some embodiments, the parameter includes information about the content of media playback that is played back along with the audio signal. For example, the audio signal may be played with some video data, such as for a gaming application. In these embodiments, the processor may be configured to adjust at least one criterion for selecting a haptic signal based on the content of the media playback (e.g., the program content of the media playback). In other words, if the game content is a military-based game, the criterion can be adjusted to distinguish between a shot represented by an "on-screen" image (e.g., muzzle flash) in the video data and an "off-screen" shot not represented by an image in the video data. Thus, the criterion for generating the haptic signal can be adjusted to produce different haptic signals for the two scenarios, even though the haptic triggers in the two scenarios may be very similar.

[0096] In some embodiments, the parameter includes information about historical haptic events associated with the audio signal. For example, the haptic history block 504 may store a data history of the haptic signals output during a particular audio stream. The haptic history block 504 may include processing configured to, for example, look for repeated haptic signals output during the audio signal. The haptic history block 504 may then output a parameter to adjust at least one criterion to optimize any expected future output of the repeated haptic signals.

[0097] In some embodiments, it may be desirable that audio signal A IN be time-synchronized with some other media content (e.g., video data). In the case where haptic signals derived from the audio signal are added, it may be necessary to control a third synchronization variable. The haptic signals may be delayed relative to the input audio of the system, even slightly. However, if the audio signal can be arranged to arrive at the processor before the other media content, the output audio and haptic signals from the processor can be aligned with the other media content.

[0098] Figure 6 Illustrated is a processor 600 that includes an audio output transducer 203 and a haptic transducer 201, where the processor 600 outputs a haptic signal H to the haptic transducer 201 OUT . Features similar to those illustrated in Figure 2 have been given the same reference numerals. In some embodiments, the haptic signal H OUT can be driven through the haptic transducer 201 after being amplified by the haptic amplifier 208.

[0099] The processor 600 includes an input 601 that is configured to receive an audio signal for output through the audio output transducer 203 (e.g., a speaker).

[0100] The processor 600 includes a haptic trigger detection block 602 configured to determine whether an audio signal contains a haptic trigger. The haptic trigger detection block 602 can be configured to be similar to the haptic trigger detection block 204 in Figure 2 . Alternatively, other methods of detecting a haptic trigger in an audio signal can be performed by the haptic trigger detection block 602.

[0101] The haptic trigger detection block 204 can then indicate to the haptic signal generator 205 whether the audio signal A IN contains a haptic trigger. In some embodiments, the indication includes the haptic trigger of the audio signal A IN . In other words, the haptic trigger detection block 204 can output to the haptic signal generator 205 an audio signal that contains a haptic trigger. In some embodiments, the haptic signal generator includes an input for receiving the audio signal A IN , and the indication includes information indicating the position of the haptic trigger within the audio signal.

[0102] In response to receiving an indication that the audio signal A IN contains a haptic trigger, the haptic signal generator 205 maps the energy at frequencies outside the expected frequency range in the received haptic trigger (received from the haptic trigger detection block or received as part of the audio signal) to frequencies within the expected frequency range to generate a haptic signal. Specifically, the haptic signal generator maps the frequencies by undersampling the data in the audio signal that contains the haptic trigger, such that the energy at frequencies above the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range, thereby generating a haptic signal.

[0103] The expected frequency range can include frequencies that, when input into the haptic transducer 201, result in a haptic event that can be felt by the user. Mapping the higher frequencies by undersampling the haptic trigger means that in the haptic event caused by the haptic signal, the user will feel the energy in the haptic trigger. By undersampling the data in the audio signal that contains the haptic trigger, the haptic signal generator can thus create a haptic signal representing the audio signal that contains the haptic trigger, but the haptic signal is within a frequency range suitable for reproduction by the haptic transducer 201.

[0104] The haptic signal generator 205 can be configured to output the haptic signal to the haptic transducer.

[0105] In some embodiments, the haptic signal generator may be configured to adjust the dynamic range of a haptic trigger to generate a haptic signal. In other words, while a typical 16-bit audio signal may have a dynamic range of up to and exceeding 90 dB, the user's resolution sensitivity to haptic events caused by haptic signals is much smaller, e.g., 20 dB to 30 dB. Subsequently, if low-level audio material is found to form part of the audio signal during a haptic trigger, the dynamic range of the haptic trigger can be adjusted to suit the user's sensitivity to the resulting haptic event.

[0106] In some embodiments, the haptic signal generator may further be configured to distort the audio signal energy at frequencies below the expected frequency range in a haptic trigger such that corresponding energy is generated within the expected frequency range in the haptic signal. Similarly, the user does not sense low-frequency content below the expected frequency range. Thus, the haptic signal generator can modify or otherwise distort the low frequencies of the haptic trigger to deliberately generate high-level harmonics within the expected frequency band so that the user can sense the haptic event.

[0107] It should be understood that processors 200, 500, and 600 may be implemented as at least a part of signal processor 108 in a device (such as device 100 illustrated in Figure 1 ).

[0108] Figure 7 Illustrated is a method for triggering a haptic signal that is output to a haptic transducer. The method may be executed by a processor, such as Figure 2 processor 200 or any suitable circuitry in

[0109] In step 701, the method includes receiving an audio signal for output via an audio output transducer.

[0110] In step 702, the method includes determining whether the audio signal includes a haptic trigger based on an indication of the rate of change of the amplitude of the audio signal.

[0111] In step 703, the method includes triggering a haptic signal in response to determining that the audio signal includes a haptic trigger, the haptic signal being output to a haptic transducer.

[0112] Figure 8 Illustrated is a method for outputting a first haptic signal to a haptic transducer in a device. The method may be executed by a processor, such as Figure 5 processor 500 or any suitable circuitry in

[0113] In step 801, the method includes receiving an audio signal for output via an audio output transducer.

[0114] In step 802, the method includes determining, based on an audio signal and at least one criterion, a haptic signal for output to a haptic transducer.

[0115] In step 803, the method includes dynamically adjusting at least one criterion based on a parameter, where the parameter includes one or more of the following: information about the content of media playback played back together with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and the battery charge state of the device.

[0116] Figure 9 A method for generating a haptic signal for output to a haptic transducer is illustrated. The method may be executed by a processor, such as Figure 6 processor 600 in or any suitable circuitry.

[0117] In step 901, the method includes receiving an audio signal for output through an audio output transducer.

[0118] In step 902, the method includes determining whether the audio signal includes a haptic trigger.

[0119] In step 903, the method includes, in response to determining that the audio signal includes a haptic trigger, by undersampling data in the audio signal including the haptic trigger, mapping energy at frequencies outside an expected frequency range in the haptic trigger to frequencies within the expected frequency range in the haptic signal, such that energy at frequencies above the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range in the haptic signal. Accordingly, a method and apparatus for outputting a haptic signal to a haptic transducer based on a received audio signal, the received audio signal being output through a speaker, are provided.

[0120] It should be noted that the above-described embodiments illustrate rather than limit the present invention, and those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. The word "comprising" does not exclude the presence of elements or steps other than those listed in the claims, "a" or "an" does not exclude a plurality, and a single feature or other unit may implement the functions of several units recited in the claims. Any reference numerals or reference signs in the claims should not be construed as limiting the scope of the claims. Terms such as amplification or gain include applying a scaling factor less than 1 to a signal.

[0121] Of course, it should be understood that multiple embodiments of the analog conditioning circuit or multiple blocks or portions thereof as described above may be co-integrated with other blocks or portions thereof or with other functions of the host device on an integrated circuit such as a smart codec.

[0122] Accordingly, those skilled in the art will recognize that some aspects of the apparatus and methods described above may be embodied as processor control code, for example, located on a non-volatile carrier medium such as a magnetic disk, a CD-ROM or a DVD-ROM, a programmed memory such as a read-only memory (firmware), or on a data carrier such as an optical signal carrier or an electrical signal carrier. For many applications, embodiments of the present invention will be implemented on a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). Thus, the code may include conventional program code or microcode or, for example, code for setting up or controlling an ASIC or an FPGA. The code may also include code for dynamically configuring a reconfigurable device such as a reprogrammable logic gate array. Similarly, the code may include code for a hardware description language such as Verilog TM or VHDL (Very High Speed Integrated Circuit Hardware Description Language). As those skilled in the art will understand, the code may be distributed among multiple coupled components that communicate with each other. In appropriate cases, code that runs on a field programmable (re)programmable analog array or similar device to configure analog hardware may also be used to implement the embodiments.

[0123] It should be understood that, in particular, those of ordinary skill in the art who benefit from this disclosure should understand that the various operations described herein, particularly those associated with the figures, may be implemented by other circuits or other hardware components. The order of each operation that performs a given method may be changed, and various elements of the systems illustrated herein may be added, reordered, combined, omitted, modified, etc. It is intended that this disclosure include all such modifications and changes, and thus, the above description should be considered illustrative rather than restrictive.

[0124] Similarly, although this disclosure refers to specific embodiments, certain modifications and changes may be made to those embodiments without departing from the scope and coverage of this disclosure. Additionally, any benefits, advantages, or solutions to problems described herein with respect to specific embodiments are not to be construed as critical, essential, or necessary features of the elements.

[0125] Likewise, other embodiments that are obvious to those of ordinary skill in the art who benefit from this disclosure should be considered to be included herein.

Claims

1. A method for generating a haptic signal for output to a haptic transducer, comprising: receiving, by a processor, an audio signal for output by an audio output transducer; determining, by the processor, whether the audio signal contains a haptic trigger based on an indication that a rate of change of an amplitude of the audio signal is higher than a threshold, the threshold indicating that the audio signal is suitable for output together with a haptic signal; responsive to determining that the audio signal contains the haptic trigger, generating, by the processor, the haptic signal by mapping energy at frequencies outside an expected frequency range in the haptic trigger to frequencies within the expected frequency range in the haptic signal, wherein the mapping is performed by: undersampling data in the audio signal containing the haptic trigger such that energy at frequencies above the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range, wherein the expected frequency range includes frequencies that, when input to the haptic transducer, cause a haptic event that can be felt by a user.

2. The method according to claim 1, further comprising: adjusting, by the processor, a dynamic range of the haptic trigger to generate the haptic signal.

3. The method according to claim 1, wherein the mapping step further comprises: distorting audio signal energy at frequencies below the expected frequency range in the haptic trigger such that corresponding harmonic energy is generated within the expected frequency range in the haptic signal.

4. A processor for generating a haptic signal for output to a haptic transducer, comprising: an input configured to receive an audio signal for output by an audio output transducer; a haptic trigger detection block configured to determine whether the audio signal contains a haptic trigger based on an indication that a rate of change of an amplitude of the audio signal is higher than a threshold, the threshold indicating that the audio signal is suitable for output together with a haptic signal; and a haptic signal generator configured to: responsive to determining that the audio signal contains the haptic trigger, map energy at frequencies outside an expected frequency range in the haptic trigger to frequencies within the expected frequency range in the haptic signal to generate the haptic signal, wherein the mapping is performed by: undersampling data in the audio signal containing the haptic trigger such that energy at frequencies above the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range, wherein the expected frequency range includes frequencies that, when input to the haptic transducer, cause a haptic event that can be felt by a user.

Images