Method and apparatus for outputting a haptic signal to a haptic transducer

Abstract

The embodiments described herein relate to methods and apparatus for outputting haptic signals to haptic transducers. A method for triggering a haptic signal to be output to a haptic transducer includes: receiving an audio signal for output through an audio output transducer; determining whether the audio signal includes a haptic trigger based on an indication of a rate of change of the amplitude of the audio signal; and, in response to determining that the audio signal includes a haptic trigger, triggering the haptic signal to be output to the haptic transducer.

Description

[0001] This application is a divisional application of the invention patent application with the application date of April 4, 2019, the application number of 201980023336.0, and the title of "Method and apparatus for outputting a haptic signal to a haptic transducer". Technical Field

[0002] The embodiments described herein relate to a method and apparatus for outputting a haptic signal to a haptic transducer based on a received audio signal, the received audio signal being output through a speaker. Background Art

[0003] For some applications, it may be desirable to generate haptic effects on mobile devices and other devices such as user interfaces of smart home devices or vehicles. In some examples, it may be desirable for these haptic effects to be synchronized with specific audio events to enhance the user experience. In some cases (e.g., where the audio source material is pre-defined in the system, such as a specific song of a known type), it is conceivable that a series of haptic events designed to be output along with the song can be stored and replayed independently of the known audio.

[0004] However, there may be examples of audio playback that do not have a predictable or pre-defined nature. For example, in a gaming application, the rendered audio output can include a combination of stored elements (e.g., background music) and other sounds (which may depend on the gameplay). The precise timing and output of the latter category of sounds may depend on the user's actions in the game. Due to such user control, the output of a stored series of haptic events may be inappropriate. Summary of the Invention

[0005] According to some embodiments, there is provided a method for triggering a haptic signal that is output to a haptic transducer. The method includes: receiving an audio signal for output through an audio output transducer; determining whether the audio signal includes a haptic trigger based on an indication of a rate of change of an amplitude of the audio signal; and in response to determining that the audio signal includes a haptic trigger, triggering the haptic signal to be output to the haptic transducer.

[0006] In some embodiments, the determining step includes: generating a first weighted sum representation of the audio signal using a first number of samples of a modified form of the audio signal; generating a second weighted sum representation of the audio signal using a second number of samples of the modified form of the audio signal, wherein the second number of samples is greater than the first number of samples; and comparing the first weighted sum representation with the second weighted sum representation. This comparison can provide an indication of the rate of change of the amplitude of the audio signal.

[0007] In some embodiments, the comparing step includes: determining that the audio signal includes a haptic trigger when the first weighted sum representation exceeds the second weighted sum representation. In some embodiments, the comparing step includes: determining that the audio signal includes a haptic trigger when the first weighted sum representation exceeds the second weighted sum representation plus a predetermined offset.

[0008] In some embodiments, the method further includes adjusting a first set of weighting coefficients used to generate the first weighted sum representation, and / or adjusting a second set of weighting coefficients used to generate the second weighted sum representation, based on desired haptic trigger characteristics.

[0009] In some embodiments, the method further includes adjusting the first number of samples and / or the second number of samples based on desired haptic trigger characteristics.

[0010] In some embodiments, the method further includes determining whether the audio signal includes a haptic trigger based on a rate of change of the amplitude of the audio signal.

[0011] According to some embodiments, there is provided a method that includes: receiving an audio signal for output via an audio output transducer; determining a haptic signal for output to a haptic transducer based on the audio signal and at least one criterion; and dynamically adjusting the at least one criterion based on a parameter, where the parameter includes one or more of the following: information about the content of media playback being played back along with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and the battery charge state of the device.

[0012] In some embodiments, the method further includes: determining that the audio signal includes a haptic trigger in response to the audio signal meeting a first criterion of the at least one criterion; and outputting the haptic signal to the transducer in response to determining that the audio signal includes the haptic trigger.

[0013] The first criterion may include a threshold rate of change of the audio signal.

[0014] In some embodiments, the method further includes determining the haptic signal based on the haptic trigger and a second criterion of the at least one criterion.

[0015] In some embodiments, the method further includes generating the haptic signal from the haptic trigger based on a third criterion of the at least one criterion.

[0016] The third criterion may include one or more of the following: maximum amplitude, allowable dynamic range, and allowable frequency range.

[0017] In some embodiments, the adjusting step includes: in response to the parameter indicating that the future repetition rate of the haptic signal event should be reduced, adjusting the at least one criterion to reduce the repetition rate of the haptic signal event derived from the audio signal.

[0018] In some embodiments, the parameter indicates whether the audio signal contains speech, and the adjusting step includes: in response to the parameter indicating that the audio signal contains speech, adjusting the at least one criterion to reduce the repetition rate of the haptic signal event.

[0019] In some embodiments, the parameter includes the battery charging state of the device, and the adjusting step includes in response to the parameter indicating that the battery charging state is below a threshold level, adjusting the at least one criterion to reduce the repetition rate of the haptic signal event.

[0020] In some embodiments, the parameter includes information about the content of the media playback played back together with the audio signal; and the adjusting step includes adjusting the at least one criterion based on the content of the media playback to select the haptic signal.

[0021] The media playback may include video data.

[0022] In some embodiments, the parameter includes information about historical haptic events associated with the audio signal; and the adjusting step includes adjusting the at least one criterion to optimize the expected future output of repeated haptic events.

[0023] According to some embodiments, a method for generating a haptic signal for output to a haptic transducer is provided. The method includes: receiving an audio signal for output through an audio output transducer; determining whether the audio signal contains a haptic trigger; in response to determining that the audio signal contains the haptic trigger, by undersampling the data in the audio signal containing the haptic trigger, mapping the energy at frequencies outside the expected frequency range in the haptic trigger to frequencies within the expected frequency range in the haptic signal, such that the energy at frequencies above the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range to generate the haptic signal.

[0024] In some embodiments, the method further includes adjusting the dynamic range of the haptic trigger to generate the haptic signal.

[0025] In some embodiments, the mapping step further includes distorting the audio signal energy at frequencies below the expected frequency range in the haptic trigger such that corresponding harmonic energy is generated within the expected frequency range in the haptic signal.

[0026] According to some embodiments, a processor for triggering a haptic signal that is output to a haptic transducer is provided. The processor includes: an input configured to receive an audio signal for output via an audio output transducer; a haptic trigger detection block configured to determine whether the audio signal includes a haptic trigger based on an indication of a rate of change of the amplitude of the audio signal; and a haptic signal generator configured to, in response to determining that the audio signal includes a haptic trigger, trigger the haptic signal to be output to the haptic sensor.

[0027] According to some embodiments, a processor is provided that outputs a first haptic signal to a haptic transducer in a device. The processor includes: an input configured to receive an audio signal for output via an audio output transducer; a haptic signal determination block configured to determine a haptic signal for output to the haptic transducer based on the audio signal and at least one criterion, wherein the processor is configured to dynamically adjust the at least one criterion based on a parameter, and the parameter includes one or more of the following: information about the content of media playback played back together with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and the battery charge state of the device.

[0028] According to some embodiments, a processor for generating a haptic signal for output to a haptic transducer is provided. The processor includes: an input configured to receive an audio signal for output via an audio output transducer; a haptic trigger detection block configured to determine whether the audio signal includes a haptic trigger; and a haptic signal generator configured to: in response to determining that the audio signal includes the haptic trigger, map the energy at frequencies outside the expected frequency range in the haptic trigger to frequencies within the expected frequency range in the haptic signal by undersampling the data in the audio signal that includes the haptic trigger, such that the energy at frequencies above the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range, to generate the haptic signal.

[0029] According to some embodiments, an electronic device including a processor is provided, the processor being configured to trigger a haptic signal to be output to a haptic transducer. The processor includes: an input configured to receive an audio signal for output through an audio output transducer; a haptic trigger detection block configured to determine whether the audio signal includes a haptic trigger based on an indication of a rate of change of an amplitude of the audio signal; and a haptic signal generator configured to trigger the haptic signal to be output to the haptic transducer in response to determining that the audio signal includes a haptic trigger.

[0030] According to some embodiments, an electronic device including a processor is provided, the processor being configured to output a first haptic signal to a haptic transducer in a device. The processor includes: an input configured to receive an audio signal for output through an audio output transducer; a haptic signal determination block configured to determine a haptic signal for output to the haptic transducer based on the audio signal and at least one criterion; wherein the processor is configured to dynamically adjust the at least one criterion based on a parameter, the parameter including one or more of the following: information about content of media playback played back together with the audio signal, information about content of the audio signal, information about historical haptic events associated with the audio signal, and a battery charge state of the device.

[0031] According to some embodiments, an electronic device including a processor is provided, the processor being configured to generate a haptic signal for output to a haptic transducer. The processor includes: an input configured to receive an audio signal for output through an audio output transducer; a haptic trigger detection block configured to determine whether the audio signal includes a haptic trigger; and a haptic signal generator configured to map energy at frequencies outside an expected frequency range in the haptic trigger to frequencies within the expected frequency range in the haptic signal by undersampling the haptic trigger data in response to determining that the audio signal includes the haptic trigger, such that energy at frequencies higher than the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range to generate the haptic signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] To better understand embodiments of the present disclosure and to illustrate how to implement embodiments of the present disclosure, reference is now made, by way of example only, to the accompanying drawings, in which:

[0033] Figure 1 An example schematic diagram is illustrated showing a portion of a device 100 having an audio output transducer and a haptic output transducer;

[0034] Figure 2 Illustrates an embodiment of a processor for triggering a haptic signal H OUT where the haptic signal H OUT is output to a haptic transducer;

[0035] Figure 3 is an example flowchart illustrating a method for determining whether an audio signal contains a haptic trigger according to some embodiments;

[0036] Figure 4a Illustrates an example audio signal, a first weighted sum representation of the audio signal, and a second weighted sum representation of the audio signal;

[0037] Figure 4b Illustrates an example audio signal and the rate of change of the audio signal;

[0038] Figure 5 Illustrates an embodiment of a processor for triggering a haptic signal that is output to a haptic transducer in a device according to some embodiments;

[0039] Figure 6 Illustrates an embodiment of a processor including an audio output transducer and a haptic transducer according to some embodiments, wherein the processor outputs a haptic signal to the haptic transducer;

[0040] Figure 7 is an example flowchart illustrating a method for triggering a haptic signal that is output to a haptic transducer according to some embodiments;

[0041] Figure 8 is an example flowchart illustrating a method for outputting a first haptic signal to a haptic transducer in a device according to some embodiments;

[0042] Figure 9 is an example flowchart illustrating a method for outputting a haptic signal to a haptic transducer according to some embodiments. DETAILED DESCRIPTION

[0043] The following description sets forth example embodiments in accordance with the present disclosure. Other example embodiments and implementations will be apparent to those of ordinary skill in the art. Additionally, those of ordinary skill in the art will recognize that various equivalent techniques may be applied in place of or in combination with the embodiments discussed below, and all such equivalents should be considered to be encompassed by the present disclosure.

[0044] Figure 1is an example schematic diagram showing a portion of a device 100 having a corresponding audio output transducer 102 and a haptic output transducer 104. The device 100 can include any electronic device, such as: a portable device; a battery-powered device; a computing device; a communication device; a gaming device; a mobile phone; a personal media player; a laptop computing device, a tablet computing device or a notebook computing device, a smart watch, a virtual reality (VR) device or an augmented reality (AR) device or a smart home device. For clarity, Figure 1 elements in the device 100 that are not relevant to the present disclosure are not shown, but those skilled in the art should understand that in addition to the Figure 1 elements and components shown therein, the device 100 can include other elements and components.

[0045] The device 100 includes a processor 106, which can be, for example, an application processor. The processor 106 is interfaced with a signal processor 108, which can be, for example, a digital signal processor (DSP). The signal processor 108 is interfaced with an audio output amplifier 110, which is configured to output an audio signal to drive the audio output transducer 102. The signal processor 108 is also interfaced with a haptic output amplifier 112, which is configured to output a haptic signal to drive the haptic transducer 104.

[0046] The processor 106 executes and provides an operating environment for the device 100 to allow software applications (such as games) to be executed by the device 100. Such applications typically receive user input. The user input can include one or more of the following: touch user input and / or gesture user input that can be detected by a touch-sensitive surface (such as a touch screen) of the device (not shown); kinematic user input (such as rotating or tilting the device) that can be detected by sensors (such as an accelerometer or gyroscope of the device) (also not shown); and audio user input (such as spoken commands) that can be detected by sensors (such as a microphone of the device) (also not shown). In response to detecting the user input, the application is operable to generate an appropriate output at the device. For example, the application is operable to cause the image displayed on a display (not shown) of the device to be updated and to cause an appropriate audio effect to be output by the audio output transducer 102. The application is also operable to cause an appropriate haptic output to be provided by the haptic output transducer 104 in response to detecting the user input.

[0047] For example, in a driving game application, a user can interact with the touch screen of device 100, e.g., by touching a graphical representation of an accelerator pedal or a brake pedal presented on the touch screen of the device, so as to accelerate or brake a vehicle in the game. Such a touch is detected by the touch screen and causes the image displayed on the display of the device to be updated by the application to give a visual indication that the vehicle is accelerating or braking. The update of the image displayed on the display may be accompanied by appropriate audio effects output by the output audio transducer 102, such as an engine revving up effect when touching the accelerator pedal, or a tire screeching effect when touching the brake pedal.

[0048] To turn the vehicle, the user can rotate the device in a counterclockwise direction to turn left or in a clockwise direction to turn right. In response to such kinetic input detected by the accelerometer and / or gyroscope of the device, the image displayed on the display is updated by the application, when appropriate, to give a visual indication of the vehicle's movement. Additionally, appropriate audio effects, such as a tire screeching effect, may be output by the output audio transducer 102.

[0049] Audio user input (such as a voice command) can additionally be used, for example, to perform special functions such as firing a weapon. When a valid voice command or other audio user input is detected, the image displayed on the display can be updated, when appropriate, to give a visual indication of the effect of the command, and appropriate audio effects, such as a gunfire sound effect, may be output by the output audio transducer 102.

[0050] The application can also be configured to use the tactile output transducer 104 to provide additional sensory information to the user. In the driving game application embodiment discussed above, in addition to updating the image displayed on the display and outputting appropriate audio effects, device 100 can also provide tactile output in response to user input. For example, the tactile output transducer 104 can be actuated to cause device 100 to vibrate while updating the image displayed on the display and outputting audio effects through the output audio transducer 102, so as to provide additional sensory information to the user.

[0051] In such an application, tactile output can be designed into the application at the beginning. In other words, the designer of the application can identify the user input, visual output, and audio output to be associated with the tactile output, and manually program the application to provide tactile output in response to the identified user input or in combination with the identified visual output or the identified audio output. As should be understood, this manual programming is a time-consuming process during the programming stage. Alternatively, some simple processing (e.g., filtering or correction) can be applied to the audio signal, and this processed signal can be used to drive the tactile transducer. However, the disadvantage of this method is that the tactile output reflects the amplitude of the audio output, and loses the desired effect of making the tactile response only highlight or emphasize certain events. In addition, making the tactile circuit continuously active is unfavorable to the battery life in the mobile device.

[0052] Therefore, it may be desirable to be able to selectively scan the audio signal for relevant events and derive appropriate tactile outputs. In some applications (such as gaming applications), the user's experience of events (such as shooting on the screen, or, for example, other "loud" virtual events, or events with specific characteristics) can be enhanced by tactile responses. Therefore, these events can be considered "tactile triggers". Therefore, in some embodiments, tactile triggers are characterized by portions of the audio signal where large energy changes or high-speed changes in energy occur.

[0053] Figure 2 An example of a method for triggering a haptic signal H according to some embodiments is shown. OUT The processor 200, the tactile signal H OUT is output to the tactile transducer 201. In some embodiments, the tactile signal H OUT The signal may be driven to the tactile transducer 201 after being amplified by the tactile amplifier 208 .

[0054] The processor 200 comprises an input 202 for receiving an audio signal A IN , the audio signal A IN Used to output through the audio output transducer 203. Audio signal A IN The audio signal may be driven through an audio output transducer 203 after being amplified by an audio amplifier 207. The output transducer 203 may be part of the same device as the processor 200, or may be an external transducer on a separate device, such as a speaker or a headset. Thus, a device including the processor 200 may include an output suitable for connection to such an external transducer.

[0055] The processor 200 further includes a tactile trigger detection block 204. The tactile trigger detection block 204 is configured to receive an audio signal A IN , and based on the audio signal AIN determine the audio signal A based on an indication of the rate of change of the amplitude IN whether it includes a haptic trigger. A haptic trigger can be considered as a part of the audio signal that has a certain characteristic indicating its suitability for output together with a haptic signal.

[0056] For example, if the rate of change of the amplitude of the audio signal is higher than a first threshold, then this rate of change can indicate that an audio event occurs in the audio signal A IN which can serve as a basis for a haptic event, thereby highlighting or emphasizing the audio event to the user.

[0057] Then, the haptic trigger detection block 204 can indicate to the haptic signal generator 205 whether the audio signal A IN includes a haptic trigger. In response to receiving an indication that the audio signal includes a haptic trigger, the haptic signal generator 205 can trigger the haptic signal H OUT wherein the haptic signal H OUT is output to the haptic transducer 201.

[0058] In some embodiments, the indication includes an indication of the type of the audio event that occurs. For example, the indication can include information indicating that the audio signal includes a gunshot sound. Thus, the haptic signal generator 205 can generate an appropriate haptic signal H based on the indicated type of audio event in the audio signal that causes the haptic trigger, for example, by selecting a pre - defined haptic signal from the memory 209 OUT .

[0059] In some embodiments, the indication can indicate the rate of change of the amplitude of the audio signal A IN . Thus, the haptic signal generator 205 can generate an appropriate haptic signal H based on the rate of change of the amplitude of the audio signal A IN . For example, if the rate of change of the amplitude of the audio signal A OUT is low, for example, lower than a second threshold (the second threshold is higher than the first threshold), then the haptic signal generator 205 can generate a first type of haptic signal H IN . For example, the first type of haptic signal can include beeping or a piece - wise linear envelope (PWLE) type of waveform, which are embodiments of simple and effective ways of storing haptic waveforms. OUT However, if the rate of change of the amplitude of the audio signal A

[0060] is high (e.g., higher than the second threshold), then the haptic signal generator 205 can generate a second type of haptic signal H IN . The second type of haptic signal H OUT . OUTIt may include a waveform that is shorter and more impulsive than the first type of haptic signal. For example, the second type of haptic signal may include a "more aggressive" or "snapping" waveform and is stored in the device (or memory) as a waveform file composed of sampled discrete data.

[0061] In some embodiments, the indication may include the audio signal A IN itself. In other words, the haptic trigger detection block 204 may output the audio signal A to the haptic signal generator in response to detecting that the audio signal contains a haptic trigger. IN In this embodiment, the haptic signal generator may generate a haptic signal by processing the audio signal A that contains a haptic trigger. The processing applied may be as described later with reference to IN Figure 6

[0062] In other embodiments, the haptic signal generator 205 may receive the audio signal A IN as an input separate from the indication. In this embodiment, the indication may mark for the haptic signal generator 205 when the audio signal A IN contains a haptic trigger, so that a haptic signal H IN is generated from the audio signal A OUT

[0063] In some embodiments, particularly in embodiments where a haptic signal H IN is generated from the audio signal A OUT , the audio signal A IN may be output to the audio amplifier 207 after being delayed by the delay block 206. In particular, the delay block 206 may be configured to introduce a delay into the path between the input 202 and the audio amplifier 207 that is the same as the delay introduced by the processing in the haptic signal generator 205 between the input 202 and the haptic amplifier 208.

[0064] Figure 3 Illustrates a method for determining whether an audio signal contains a haptic trigger according to some embodiments. For example, Figure 3 the method illustrated in Figure 2 may be performed by the haptic trigger detection block 204 illustrated in Figure 4a . To illustrate an embodiment of this method, reference is made to the example audio signal 400 of

[0065] ​​​In step 301, the haptic trigger detection block 204 generates a first weighted sum representation 402 of the modified form 401 of the audio signal 400 using a first number of samples of the modified form 401 of the audio signal 400. The first weighted sum representation 402 provides a representation of the fluctuations in the energy in the modified form 401 of the audio signal 400.

[0066] In step 302, the haptic trigger detection block 204 generates a second weighted sum representation 404 of the modified form 401 of the audio signal 400 using a second number of samples of the modified form 401 of the audio signal 400, where the second number of samples is greater than the first number of samples used in step 301.

[0067] When more samples are used to generate the weighted sum representation of the signal, the representation is less responsive to the occurrence of peaks and valleys in the audio signal. Thus, the second weighted sum representation 404 is less responsive to the peaks and valleys in the modified form 401 of the audio signal 400 compared to the first weighted sum representation 402.

[0068] In step 303, the haptic trigger detection block 204 may compare the first weighted sum representation 402 with the second weighted sum representation 404. This comparison may indicate the rate of change of the amplitude of the audio signal 400.

[0069] In step 304, the haptic trigger detection block 204 may determine whether the audio signal 400 contains a haptic trigger based on the comparison made in step 303.

[0070] For example, when the first weighted sum representation 402 exceeds the second weighted sum representation 404 (as illustrated by the marker 406 in Figure 4a ), the haptic trigger detection block 204 may determine that the audio signal 400 contains a haptic trigger.

[0071] In some embodiments, when the first weighted sum representation 402 exceeds the second weighted sum representation 404 plus a predetermined offset, the haptic trigger detection block 204 may determine that the audio signal contains a haptic trigger.

[0072] It should be understood that during some applications, the processor may adjust the sensitivity of the haptic output. For example, during some applications, the processor 200 may adjust a first set of weighting coefficients (which may include one or more coefficients) used to generate the first weighted sum representation and / or adjust a second set of weighting coefficients (which may include one or more coefficients) used to generate the second weighted sum representation based on the desired haptic trigger characteristics. In other words, the weighted sums may be changed such that a higher or lower rate of change of the amplitude of the audio signal is required to trigger a haptic event.

[0073] Adjustment of the first or second weighting factor may also adjust the frequency response, which is included in the respective weighted sum representation. For example, the second weighted sum may be configured to filter out more high frequencies than the first weighted sum; or, a high-pass filtering effect may be created so that the second weighted sum ignores any bass content. In this way, the comparison of the weighted sums may search for a haptic trigger within a desired frequency range.

[0074] In some embodiments, the haptic trigger detection block may determine whether the audio signal no longer contains a haptic trigger based on an indication of the rate of change of the amplitude of the audio signal. In some embodiments, the comparison between the first weighted sum representation 402 and the second weighted sum representation 404 may determine the length of the haptic signal or the amplitude of the haptic signal. For example, when the difference between the first weighted sum representation 402 and the second weighted sum representation 404 is greater, the haptic signal is longer or the amplitude of the haptic signal is higher.

[0075] In response to determining that the audio signal no longer contains a haptic trigger, the haptic trigger detection block may indicate this condition to the haptic signal generator 205.

[0076] In some embodiments, when the first weighted sum representation 402 drops back below the second weighted sum representation 404, the haptic trigger detection block may determine that the audio signal no longer contains a haptic trigger.

[0077] In some embodiments, when the first weighted sum representation 402 drops to a predetermined offset below the second weighted sum representation 404, the haptic trigger detection block 204 may determine that the audio signal no longer contains a haptic trigger. In this embodiment, a certain hysteresis may be used.

[0078] It should be understood that in some embodiments, multiple different weighted sum representations and associated offsets may be used to detect different types of haptic triggers corresponding to different types of audio events. Then, a haptic signal may be generated based on which type of haptic trigger is present.

[0079] To trigger the haptic signal output to the haptic transducer such that the haptic signal is synchronized with the audio signal containing the haptic trigger, the processing for generating the haptic signal may have time limitations. This factor may effectively limit the range and functionality of the decision points available for registering haptic triggers based on changes in the energy level in the audio signal.

[0080] Figure 4b An embodiment according to some embodiments is illustrated, in which the rate of change of the audio signal is used to determine whether the audio signal contains a haptic trigger.

[0081] In this embodiment, an audio signal 408 and a rate of change 410 of this audio signal are illustrated. When the rate of change 410 of the audio signal is higher than a predetermined threshold 412, a haptic trigger can be detected. This detection is illustrated by a flag 414.

[0082] Figure 5 A processor 500 for triggering a haptic signal H according to some embodiments is illustrated. OUT The haptic signal H OUT is output to a haptic transducer 201 in the device. Features similar to those illustrated in Figure 2 have been given the same reference numerals. In some embodiments, the haptic signal H OUT can be driven through the haptic transducer 201 after being amplified by a haptic amplifier 208.

[0083] The processor 500 includes an input 501 for receiving an audio signal A IN for output through an audio output transducer 203. The audio signal A IN can be driven through the audio output transducer 203 after being amplified by an audio amplifier 207.

[0084] The processor 500 includes a haptic signal determination block 502. The haptic signal determination block can be configured to determine a haptic signal based on the audio signal and at least one criterion for output to the transducer. In some embodiments, the haptic signal determination block 502 includes a haptic trigger detection block 204 and a haptic signal generator 205 as illustrated in Figure 2 . In these embodiments, in response to the audio signal satisfying a first criterion of at least one criterion, it is determined that the audio signal includes a haptic trigger, and in response to determining that the audio signal includes a haptic trigger, a haptic signal is output to the transducer. As described with reference to Figure 2 , the first criterion can thus include a threshold rate of change of the amplitude of the audio signal, which can be represented by a comparison between a first weighted sum representation of the audio signal and a second weighted sum representation of the audio signal.

[0085] In this embodiment, the at least one criterion can include a second criterion. For example, the second criterion can include a second threshold as described above. In other words, if the rate of change of the amplitude of the audio signal A IN is high (e.g., higher than the second threshold), the haptic signal generator 205 can generate a second type of haptic signal H OUT . The second type of haptic signal H OUTIt may include a waveform that is shorter and more pulsatile than the first type of haptic signal. For example, the second type of haptic signal may include a click waveform. Alternatively, the second criterion may include the relationship between the value of the rate of change of the amplitude of the audio signal and the amplitude of the determined haptic signal. In other words, in some embodiments, when the rate of change is higher, the amplitude of the generated haptic signal is higher.

[0086] In some embodiments, the haptic signal determination block 502 may generate a haptic signal from a haptic trigger based on a third criterion among the at least one criterion. For example, the third criterion may include one or more of the following: the maximum amplitude of the haptic signal, the allowable dynamic range of the haptic signal, and the allowable frequency range of the haptic signal. Thus, a haptic signal may be generated from the audio signal such that the haptic signal meets the third criterion.

[0087] It should be understood that other methods may be utilized to generate a haptic signal from the audio signal, and the other methods may use different criteria to determine whether the haptic signal will be output and what the haptic signal will be. However, Figure 2 the illustrated embodiments in may be considered a fast response decision-making phase, and this fast response decision-making phase may thus quickly provide a haptic signal so as to maintain synchronization with the audio signal.

[0088] Then, the processor 500 may be configured to dynamically adjust the at least one criterion based on a parameter, where the parameter includes one or more of the following: information about the content of the media playback played back together with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and the battery charge state of the device. Thus, compared to determining the haptic signal, this processing of dynamically adjusting the criterion based on the parameter may be performed over a longer time period, thereby providing greater flexibility in the types of information that can be used to determine the haptic signal.

[0089] In other words, the haptic signal determination block 502 may include processing with lower latency than the processing required to dynamically adjust the at least one criterion. Thus, this higher-latency processing may be used for: a) applying more processing power to determining the haptic signal, and b) adopting algorithms and techniques more suitable for larger audio data sets. By adjusting the at least one criterion, the higher-latency processing may effectively train or adapt the lower-latency processing without affecting the latency of the output haptic signal.

[0090] The criteria described above may be adjusted by the processor so as to reduce the repetition rate of haptic signal events derived from the audio signal. For example, in some cases, the parameter may indicate that the repetition rate of the occurring haptic signal events is too high (e.g., the user may have indicated through user input that the haptic events are too frequent) and that the future repetition rate should be reduced.

[0091] For example, in the case where the parameter includes a threshold change rate of the amplitude of an audio signal, the adjustment may include adjusting the number of samples for the first weighted sum representation and / or the second weighted sum representation of the audio signal. Alternatively, the adjustment may include increasing or decreasing an offset to decrease or increase the frequency of detecting a tactile trigger.

[0092] In some embodiments, the processor includes a voice detection block 503 configured to determine whether an audio signal contains speech. Thus, the parameter may include information about the content of the audio signal that indicates whether the audio signal contains speech. It should be understood that in some embodiments, the processor 500 may receive an indication of whether the audio signal contains speech from an external processor.

[0093] In these embodiments, the processor may be configured to adjust at least one criterion to reduce the repetition rate of tactile signal events in response to a parameter indicating that the audio signal contains speech. This adjustment may be configured to prevent plosive consonants (such as "p") in the speech of the audio signal from causing an unwanted tactile response. In some embodiments, at least one criterion may be adjusted to turn off the generation of any tactile signals in response to speech indicated by the parameter.

[0094] In some embodiments, the parameter includes the battery charging state of the device. In these embodiments, the processor may adjust at least one criterion to reduce the repetition rate of tactile signal events in response to the parameter indicating that the battery state is below a threshold level. In other words, if the device's battery is low, the tactile response may be reduced, thus saving battery power for applications that are considered more important for the operation of the device. For example, in response to the parameter indicating that the device's battery is below a threshold level, at least one criterion may be adjusted to reduce one or more of the following: the intensity, length, or temporal density of the tactile signal output to the tactile transducer. Thus, such an adjustment of at least one criterion may provide a more refined management of the trade-off between low battery situations and the quality of the user experience.

[0095] In some embodiments, the parameter includes information about the content of media playback that is played back along with the audio signal. For example, the audio signal may be played with some video data, such as for a gaming application. In these embodiments, the processor may be configured to adjust at least one criterion for selecting a haptic signal based on the content of the media playback (e.g., the program content of the media playback). In other words, if the game content is a military-based game, the criterion can be adjusted to distinguish between a shot represented by an "on-screen" image (e.g., muzzle flash) in the video data and an "off-screen" shot not represented by an image in the video data. Thus, the criterion for generating the haptic signal can be adjusted to produce different haptic signals for the two scenarios, even though the haptic triggers in the two scenarios may be very similar.

[0096] In some embodiments, the parameter includes information about historical haptic events associated with the audio signal. For example, the haptic history block 504 may store a data history of the haptic signals output during a specific audio stream. The haptic history block 504 may include processing configured to, for example, look for repeated haptic signals output during the audio signal. The haptic history block 504 may then output a parameter to adjust at least one criterion to optimize any expected future output of the repeated haptic signals.

[0097] In some embodiments, it may be desirable for the audio signal A IN to be time-synchronized with some other media content (e.g., video data). In the case where a haptic signal derived from the audio signal is added, it may be necessary to control a third synchronization variable. The haptic signal may be delayed relative to the input audio of the system, even slightly. However, if the audio signal can be arranged to arrive at the processor before the other media content, the output audio and haptic signals from the processor can be aligned with the other media content.

[0098] Figure 6 Illustrated is a processor 600 that includes an audio output transducer 203 and a haptic transducer 201, where the processor 600 outputs a haptic signal H to the haptic transducer 201 OUT . Features similar to those illustrated in Figure 2 have been given the same reference numerals. In some embodiments, the haptic signal H OUT can be driven through the haptic transducer 201 after being amplified by a haptic amplifier 208.

[0099] The processor 600 includes an input 601 that is configured to receive an audio signal for output via an audio output transducer 203 (e.g., a speaker).

[0100] Processor 600 includes a haptic trigger detection block 602 configured to determine whether an audio signal contains a haptic trigger. The haptic trigger detection block 602 may be configured to be similar to the haptic trigger detection block 204 in Figure 2 . Alternatively, other methods of detecting a haptic trigger in an audio signal may be performed by the haptic trigger detection block 602.

[0101] The haptic trigger detection block 204 may then indicate to the haptic signal generator 205 whether the audio signal A IN contains a haptic trigger. In some embodiments, the indication includes the haptic trigger of the audio signal A IN . In other words, the haptic trigger detection block 204 may output to the haptic signal generator 205 an audio signal that contains a haptic trigger. In some embodiments, the haptic signal generator includes an input for receiving the audio signal A IN , and the indication includes information indicating the position of the haptic trigger within the audio signal.

[0102] In response to receiving an indication that the audio signal A IN contains a haptic trigger, the haptic signal generator 205 maps the energy at frequencies outside the expected frequency range in the received haptic trigger (received from the haptic trigger detection block or received as part of the audio signal) to frequencies within the expected frequency range to generate a haptic signal. Specifically, the haptic signal generator maps the frequencies by undersampling the data in the audio signal that contains the haptic trigger such that the energy at frequencies above the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range, thereby generating a haptic signal.

[0103] The expected frequency range may include frequencies that, when input into the haptic transducer 201, result in a haptic event that can be felt by the user. Mapping the higher frequencies by undersampling the haptic trigger means that in the haptic event caused by the haptic signal, the user will feel the energy in the haptic trigger. By undersampling the data in the audio signal that contains the haptic trigger, the haptic signal generator can thus create a haptic signal representing the audio signal that contains the haptic trigger, but the haptic signal is within a frequency range suitable for reproduction by the haptic transducer 201.

[0104] The haptic signal generator 205 may be configured to output the haptic signal to the haptic transducer.

[0105] In some embodiments, the tactile signal generator may be configured to adjust the dynamic range of a tactile trigger to generate a tactile signal. In other words, while a typical 16-bit audio signal may have a dynamic range of up to and exceeding 90 dB, a user's resolution sensitivity to tactile events caused by tactile signals is much smaller, e.g., 20 dB to 30 dB. Subsequently, if low-level audio material is found to form part of the audio signal during a tactile trigger, the dynamic range of the tactile trigger may be adjusted to suit the user's sensitivity to the resulting tactile event.

[0106] In some embodiments, the tactile signal generator may further be configured to distort the audio signal energy at frequencies below the expected frequency range in a tactile trigger such that corresponding energy is generated within the expected frequency range in the tactile signal. Similarly, the user does not sense low-frequency content below the expected frequency range. Thus, the tactile signal generator may correct or otherwise distort the low frequencies of the tactile trigger to intentionally generate high levels of harmonics within the expected frequency band so that the user may sense the tactile event.

[0107] It should be understood that processors 200, 500, and 600 may be implemented as at least a portion of signal processor 108 in a device (such as device 100 illustrated in Figure 1 ).

[0108] Figure 7 Illustrated is a method for triggering a tactile signal that is output to a tactile transducer. The method may be performed by a processor, such as Figure 2 processor 200 or any suitable circuitry in

[0109] In step 701, the method includes receiving an audio signal for output via an audio output transducer.

[0110] In step 702, the method includes determining whether the audio signal includes a tactile trigger based on an indication of the rate of change of the amplitude of the audio signal.

[0111] In step 703, the method includes triggering a tactile signal in response to determining that the audio signal includes a tactile trigger, the tactile signal being output to a tactile transducer.

[0112] Figure 8 Illustrated is a method for outputting a first tactile signal to a tactile transducer in a device. The method may be performed by a processor, such as Figure 5 processor 500 or any suitable circuitry in

[0113] In step 801, the method includes receiving an audio signal for output via an audio output transducer.

[0114] In step 802, the method includes determining, based on an audio signal and at least one criterion, a haptic signal for output to a haptic transducer.

[0115] In step 803, the method includes dynamically adjusting at least one criterion based on a parameter, where the parameter includes one or more of the following: information about the content of media playback being played back along with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and the battery charge state of the device.

[0116] Figure 9 A method for generating a haptic signal for output to a haptic transducer is illustrated. The method may be executed by a processor, such as Figure 6 processor 600 in or any suitable circuitry.

[0117] In step 901, the method includes receiving an audio signal for output via an audio output transducer.

[0118] In step 902, the method includes determining whether the audio signal includes a haptic trigger.

[0119] In step 903, the method includes, in response to determining that the audio signal includes a haptic trigger, by undersampling data in the audio signal including the haptic trigger, mapping energy at frequencies outside an expected frequency range in the haptic trigger to frequencies within the expected frequency range in the haptic signal, such that energy at frequencies above the expected frequency range in the haptic trigger is converted to frequencies within the expected frequency range in the haptic signal. Accordingly, a method and apparatus for outputting a haptic signal to a haptic transducer based on a received audio signal, the received audio signal being output via a speaker, are provided.

[0120] It should be noted that the embodiments mentioned above illustrate rather than limit the present invention, and those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. The word "comprising" does not exclude the presence of elements or steps other than those listed in the claims, "a" or "an" does not exclude a plurality, and a single feature or other unit may implement the functions of several units recited in the claims. Any reference numerals or reference signs in the claims should not be construed as limiting the scope of the claimed claims. Terms such as amplification or gain include that a scaling factor less than 1 may be applied to a signal.

[0121] Of course, it should be understood that multiple embodiments of the analog conditioning circuit or multiple blocks or portions thereof as described above may be co-integrated with other blocks or portions thereof or with other functions of the host device on an integrated circuit such as a smart codec.

[0122] Accordingly, those skilled in the art will recognize that some aspects of the apparatus and methods described above may be embodied as processor control code, for example, located on a non-volatile carrier medium such as a magnetic disk, a CD-ROM or a DVD-ROM, a programmed memory such as a read-only memory (firmware), or on a data carrier such as an optical signal carrier or an electrical signal carrier. For many applications, embodiments of the present invention will be implemented on a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). Thus, the code may include conventional program code or microcode or, for example, code for setting up or controlling an ASIC or an FPGA. The code may also include code for dynamically configuring a reconfigurable device such as a reprogrammable logic gate array. Similarly, the code may include code for a hardware description language such as Verilog TM or VHDL (Very High Speed Integrated Circuit Hardware Description Language). As those skilled in the art will appreciate, the code may be distributed among multiple coupled components that communicate with each other. In appropriate cases, code for operating on a field programmable (re)programmable analog array or similar device to configure analog hardware may also be used to implement the embodiments.

[0123] It should be understood that, in particular, those of ordinary skill in the art who benefit from the present disclosure should understand that the various operations described herein, particularly those associated with the figures, may be implemented by other circuits or other hardware components. The order of each operation of a given method may be changed, and various elements of the systems illustrated herein may be added, reordered, combined, omitted, modified, etc. It is intended that the present disclosure include all such modifications and changes, and thus the above description should be considered illustrative rather than restrictive.

[0124] Similarly, although the present disclosure refers to specific embodiments, certain modifications and changes may be made to those embodiments without departing from the scope and coverage of the present disclosure. In addition, any benefits, advantages, or solutions to problems described herein with respect to specific embodiments are not to be construed as critical, essential, or necessary features of the elements.

[0125] Likewise, other embodiments that are obvious to those of ordinary skill in the art who benefit from the present disclosure should be considered to be included herein.

Claims

1. A method for outputting a first haptic signal to a haptic transducer in a device, comprising: Receiving an audio signal for output via an audio output transducer; Performing a first process based on the audio signal and at least one criterion to determine a haptic signal for output to the haptic transducer and synchronized with the audio signal for output via the audio output transducer; And While performing the first process, performing a second process to dynamically adjust the at least one criterion used in the first process based on a parameter used in the first process, wherein the second process is separate from the first process, the second process is slower than the first process, and the second process uses more processing power than the first process, wherein the parameter includes one or more of the following: information about the content of media playback played back together with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and the battery charge state of the device.

2. The method according to claim 1, further comprising: Responsive to the audio signal satisfying a first criterion of the at least one criterion, determining that the audio signal includes a haptic trigger, and Responsive to determining that the audio signal includes the haptic trigger, outputting the haptic signal to the haptic transducer.

3. The method according to claim 2, wherein the first criterion includes a threshold rate of change of the audio signal.

4. The method according to claim 2, further comprising: Determining the haptic signal based on the haptic trigger and a second criterion of the at least one criterion.

5. The method according to claim 2, further comprising: Generating the haptic signal from the haptic trigger based on a third criterion of the at least one criterion.

6. The method according to claim 5, wherein the third criterion includes one or more of the following: maximum amplitude; allowable dynamic range; and allowable frequency range.

7. The method according to claim 1, wherein the adjusting step includes adjusting the at least one criterion to reduce the repetition rate of haptic signal events derived from the audio signal in response to the parameter indicating that the future repetition rate of haptic signal events should be reduced.

8. The method according to claim 7, wherein the parameter indicates whether the audio signal includes speech, and wherein the adjusting step includes: Adjusting the at least one criterion to reduce the repetition rate of haptic signal events in response to the parameter indicating that the audio signal includes speech.

9. The method according to claim 7, wherein the parameter includes the battery charge state of the device, and wherein the adjusting step includes: Adjusting the at least one criterion to reduce the repetition rate of haptic signal events in response to the parameter indicating that the battery charge state is below a threshold level.

10. The method according to claim 1, wherein the parameter includes information about the content of the media playback played back together with the audio signal; and wherein the adjusting step includes adjusting the at least one criterion to select a haptic signal based on the content of the media playback.

11. The method according to claim 10, wherein the media playback includes video data.

12. The method according to claim 1, wherein the parameter includes information about historical haptic events associated with the audio signal; and wherein the adjusting step includes adjusting the at least one criterion to optimize the expected future output of repeated haptic events.

13. The method according to claim 1, wherein the step of performing the second processing has a higher time delay than the step of performing the first processing.

14. The method according to claim 1, wherein the step of performing the second processing trains or adapts the first processing without affecting the time delay of the step of performing the first processing.

15. A processor for outputting a first haptic signal to a haptic transducer in a device, comprising: an input configured to receive an audio signal for output through an audio output transducer; a haptic signal determination block configured to perform a first processing to determine a haptic signal based on the audio signal and at least one criterion provided by a second processing, the haptic signal being for output to the haptic transducer and synchronized with the audio signal for output through the audio output transducer, wherein the processor is configured to perform the second processing while performing the first processing to dynamically adjust the at least one criterion used by the first processing based on a parameter used in the first processing, wherein the second processing is separate from the first processing, the second processing is slower than the first processing, and the second processing uses more processing power, wherein the parameter includes one or more of the following: information about the content of the media playback played back together with the audio signal, information about the content of the audio signal, information about historical haptic events associated with the audio signal, and the battery charge state of the device, and wherein the step of performing the second processing does not affect the time delay of the step of performing the first processing.

16. The processor according to claim 15, wherein the processor is configured to adjust the at least one criterion to reduce the repetition rate of haptic signal events during the audio signal in response to the parameter indicating that the future repetition rate of haptic signal events should be reduced.

17. The processor according to claim 16, wherein the parameter indicates whether the audio signal contains speech, and wherein the processor is configured to adjust the at least one criterion to reduce the repetition rate of haptic signal events in response to the parameter indicating that the audio signal contains speech.

18. The processor according to claim 16, wherein the parameter includes the battery charging state of the device, and the processor is configured to adjust the at least one criterion to reduce the repetition rate of tactile signal events in response to the parameter indicating that the battery charging state is below a threshold level.

19. The processor according to claim 15, wherein the parameter includes information about the content of the media playback played back together with the audio signal; and wherein the processor is configured to adjust the at least one criterion based on the content of the media playback to select tactile signals.

Images