Audio streaming device

Abstract

An audio streaming device for streaming an audio signal to at least one hearing assistance device (32). The audio streaming device comprises at least one input transducer (21), an electronic sensor device (28) adapted to sense a gravity acting on the audio streaming device (20), and a transmitter (30) adapted to stream audio from the at least one input transducer (21) to the at least one hearing assistance device (32). The audio streaming device (20) further comprises a shock protection unit (31) adapted to compare an output signal from the electronic sensor device (28) with a first threshold value and to stop streaming of an audio stream from the at least one input transducer (21) by the transmitter (30) when the output signal from the electronic sensor device (28) exceeds the first threshold value.

Description

TECHNICAL FIELD

[0001] The present invention relates to an audio streaming device for streaming audio signals to at least one hearing assistance device. More specifically, the present invention relates to an audio streaming device for streaming audio signals to at least one hearing assistance device. The audio streaming device comprises at least one input transducer, and a transmitter adapted to stream audio from the at least one input transducer. The present invention also relates to a method of managing an audio streaming device. BACKGROUND

[0002] EP2769557A2 is a prior art example of a movable microphone assembly and a method for capturing an audio signal from a sound by using such a microphone assembly. EP3566468A1 is a prior art example of a microphone assembly to be worn in front of a user's chest. SUMMARY

[0003] It is an object of the present invention to provide an audio streaming device for streaming audio signals to a hearing assistance device and to protect a user from annoying audio caused by handling the audio streaming device.

[0004] This object is achieved with an audio streaming device for streaming audio signals to at least one hearing assistance device and comprising at least one input transducer, an electronic sensor device adapted to sense a gravitational force acting on the audio streaming device, and a transmitter adapted to stream audio from the at least one input transducer to the at least one hearing assistance device, wherein the audio streaming device further comprises a microprocessor adapted to compare an output signal from the electronic sensor device with a first threshold value and to stop streaming of an audio stream from the at least one input transducer by the transmitter when the output signal from the electronic sensor device exceeds the first threshold value. According to a second aspect of the present invention, a method of managing an audio streaming device comprising at least one input transducer, an electronic sensor device, and a transmitter is provided, wherein the method comprises streaming audio from the at least one input transducer to at least one hearing assistance device, sensing a gravitational force acting on the audio streaming device, comparing the sensed gravitational force with a first threshold value, and stopping the streaming of the audio stream when the sensed gravitational force exceeds the first threshold value. Optional features define various embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0005] The invention will be described in further detail with reference to the preferred aspects and to the accompanying drawings, in which:

[0006] Figure 1 A perspective view of an audio streaming device according to one embodiment of the present invention is shown;

[0007] Figure 2 A side view of an audio streaming device is illustrated Figure 1 A side view of an audio streaming device is illustrated

[0008] Figure 3 An audio streaming device and an embodiment of a hearing aid according to an embodiment of the application is schematically illustrated;

[0009] Figure 4 A relationship between acceleration and output voltage for an accelerometer used in an audio streaming device according to an embodiment of the application is illustrated.

[0010] Figure 5 A flow chart illustrating a decision process for a microprocessor used in an audio streaming device according to the application is illustrated; and

[0011] Figure 6 A state diagram for an audio streaming device according to an embodiment of the application is illustrated. DETAILED DESCRIPTION

[0012] Figure 1 A perspective view of an audio streaming device 20 according to an embodiment of the application is shown. The audio streaming device 20 is in the illustrated embodiment disc-shaped, but in other embodiments can assume other shapes serving the purpose. The audio streaming device 20 has a plurality of microphones 23, and in the illustrated example the number is three. In other examples the number can be higher or lower. When the audio streaming device 20 performs a beamforming, the processing relies on a predetermined geometry, which can be as Figure 1 The center of the audio streaming device 20 is here marked as the origin of the vector 22, and the center point is equidistant to the three microphones 23, since the distance between the center point and each of the three microphones 23 is equal. Furthermore, the angle a defined by the lines through the center point and the respective microphone 23 is the same, 120°. The lines are equiangular, since all lines intersect at the center point and form an angle of 120°.

[0013] However, the audio streaming device 20 can be formed as a bar with aligned several microphones, as a hemi-sphere or as a ball structure with integrated multiple (+10) microphones.

[0014] A vector marker is used for the direction of the beam formed by the audio streaming device 20. In some embodiments, the audio streaming device 20 aims to pick up audio originating from this direction and to remove interference introduced by noise and reverberation originating from other directions. Beamforming can be considered as a multi-dimensional filtering in space and time, a signal processing method involving spatially distributed sensors. By means of beamforming, the audio streaming device 20 aims to place virtual microphones at various locations without the need for physical movement. These virtual microphones can be used for applications like conference calls, as well as for picking up speech during a meeting and immediately streaming the audio signal to a set of hearing aids worn by one or more hearing impaired persons in the conference room. There are several beamforming algorithms for combining audio data, and these algorithms usually rely on passing the audio signal through digital filters.

[0015] Figure 2 It is illustrated that the audio streaming device 20 is tilted with respect to the ground Figure 1 A side view of the audio streaming device 20 is shown. For this purpose, the audio streaming device 20 has a stand 24 that can be attached or pivotally fastened to the audio streaming device 20 when needed. The stand 24 can also be used for the purpose of a clip for mounting the audio streaming device 20 to clothing, whereby the audio streaming device 20 can be used as an audio following microphone for a hearing aid user. The stand tilts the audio streaming device 20 with respect to the base surface 10, which influences the angle between the beam direction and the plane defined by the three microphones 21.

[0016] Figure 3 An audio streaming device 20 and one embodiment of a hearing assistance device according to one embodiment of the application are schematically illustrated. The audio streaming device 20 has a plurality of, for example three, microphones 21. The picked up sound is converted by the microphones 21 into electrical audio signals, which are converted into digital signals by respective A / D converters 25. The A / D converters 25 are in some embodiments provided by Delta-Sigma converters. The three digital audio signals originating from the respective microphones are then fed to a beamforming digital signal processor 26, which processes the captured audio signals in such a way that several sound beams are created, the directions of which are uniformly spread over the plane defined by the three microphones 21. The microphones 21 are in one embodiment omnidirectional microphones.

[0017] In the beamforming digital signal processor 26, the uniformly spread beams are created by applying appropriate phase differences by delaying and summing the audio signals of pairs of microphones 21.

[0018] The audio streaming device 20 further comprises an acceleration sensor 28, for example a 3-axis accelerometer, and a microprocessor 27. The acceleration sensor 28 outputs a measure of the acceleration of the audio streaming device 20 along three orthogonal axes. Figure 4 An example of such a measurement is illustrated, and it can be seen that there is a linear relationship between the acceleration and the output voltage when the acceleration in the measurement direction is within the dynamic range of the accelerometer. This is important for the accelerometer output for all three axes. The microprocessor 27 receives the accelerometer output for all three axes, and based on this, the microprocessor 27 calculates the overall acceleration acting on the audio streaming device 20 and the direction of the overall acceleration. When the audio streaming device 20 rests, for example on a table, the gravitational force will be the only contribution to the acceleration, and the orientation of the audio streaming device 20 can be determined by means of the overall acceleration measurement with respect to the gravitational force, which can be determined by combining the amount of acceleration measured along each axis. These data are forwarded to the beamforming digital signal processor 26 for beam selection.

[0019] The beamforming digital signal processor 26 further comprises a functionality for estimating the quality of speech present in the available beams. The beamforming digital signal processor 26 further comprises a beam selection functionality for selecting one of the uniformly spread beams resulting from the delay and sum beamforming, which beam best meets criteria set based on the speech quality estimate and the audio source direction of the desired beam. Once the desired beam has been selected, the beamforming digital signal processor 26 is adapted to adaptively adjust the phase differences applied for the beamforming in order to maximize the quality of speech present in the selected beam.

[0020] The beamforming digital signal processor 26 outputs the processed audio signal to an encoder and packetizing unit 29, in which the processed audio signal is compressed and encoded according to a predefined streaming media audio encoding format in order to represent the audio signal with a minimum number of bits while maintaining the quality. This effectively reduces the bandwidth required to transmit the audio stream. The encoded audio signal is then placed as a payload in data packets transmitted to a radio 30, which modulates and amplifies the data packets for transmission. In one embodiment, the audio stream is transmitted according to the Bluetooth TM The core specification version 5.2 transmits the audio stream, in which the audio streaming device 20 acts as an audio source for one or more persons having a corresponding audio receiving device.

[0021] The microprocessor 27 receives the accelerometer output of all three axes and calculates the total acceleration acting on the audio streaming device 20. From these values, the microprocessor 27 determines an accelerometer measurement that defines a state in which the audio streaming device 20 has been resting stably on a surface for a period of time, e.g. more than 5 seconds. Then, the microprocessor 27 compares at least one of the detected accelerometer measurements with an accelerometer measurement that defines a state in which the audio streaming device 20 is at rest. In case the comparison exceeds a first predetermined threshold, the microprocessor 27 assumes that the audio streaming device 20 is moving on a surface or falling from a surface. This can result in noise that will be streamed to the hearing aid.

[0022] To avoid emitting annoying audio to one or more hearing aids, the microprocessor 27 is connected to a shock protection unit 31 for interfering with the audio stream transmission at reasonable times. In the illustrated embodiment, the encoder and packetizing unit 29 also provides a copy of the data packets transmitted to the radio 30 to the shock protection unit 31. When the microprocessor 27 recognizes, based on the output of the acceleration sensor 28, an increased risk that the audio stream can contain acoustic shocks that should be prevented from reaching the hearing aid user, the shock protection unit 31 can instruct the radio 30 to discard the next data packet from the encoder and packetizing unit 29 and instead use a replacement data packet from the shock protection unit 31. Figure 3

[0023] In one embodiment, the replacement data packet from the shock protection unit 31 can be a copy of a previous data packet transmitted to the radio 30 that has been buffered in the shock protection unit 31. In one embodiment, the shock protection unit 31 has an audio classifier that classifies the audio samples and transmits a pre-stored audio sample that matches the audio classification of the previous data packet transmitted from the encoder and packetizing unit 29. In both embodiments, the shock protection unit 31 compensates for the lost audio packet by means of packet loss concealment (PLC) on the transmitter side (in the audio streaming device 20).

[0024] In yet another embodiment, the shock protection unit 31 simply disables the radio 30 until the risk of sending acoustic shocks is over. The lost audio packets are then compensated by means of packet loss concealment (PLC) on the receiver side (in the hearing aid 32) by the controller 39.

[0025] ​The hearing aid 32 has at least one input transducer or microphone 33 that picks up audio signals. The audio signals are digitized in an A / D converter 34, e.g. a Delta-Sigma converter, and fed to a digital signal processor 35 that is adapted to amplify and condition the audio signals intended for presentation to the hearing aid user. The amplification and conditioning is performed in accordance with predetermined settings stored in the hearing aid 32 to mitigate the hearing loss by amplifying sound at frequencies in those parts of the audible frequency range where the user suffers from a hearing deficit. The amplified and conditioned audio signals are reproduced for the user via a receiver or speaker 36. The at least one microphone 33, the A / D converter 34, the digital signal processor 35 and the speaker 36 provide an audio signal path with hearing loss mitigation.

[0026] Furthermore, the hearing aid 32 comprises a radio 37 device adapted to receive and demodulate an audio stream received as data packets. The radio device 37 can be used for inter-aural communication, or for communication with another remote device such as a smart phone. The audio stream from the radio device 37 is passed through a decoder and depacketizer unit 38, where the compressed data stream is unpacked and decoded again. The received audio is thereafter loaded into the digital signal processor 35.

[0027] A controller 39 controls the reception of data packets and is among other things responsible for packet loss concealment (PLC), which is a technique to mask the effects of packet loss in IP-communicated audio. Due to multipath propagation, a single data packet can be affected by a poor signal-to-noise ratio (SNR) and thus be corrupted by the receiver. Packet loss concealment comprises a method to account for and compensate for speech packet loss by replacing the lost packet with audio content corresponding to the most recently received audio packet, playing the most recently received packet again or synthesizing a segment of the audio stream based on the audio packets. The controller 39 also controls the mixing of the received audio stream and the audio present in the audio signal path of the hearing aid 32.

[0028] Figure 4 The output from one channel in an inertial sensor system or inertial measurement unit (IMU) is illustrated, such as the acceleration sensor 28 (e.g. a 3-axis accelerometer). An inertial measurement unit is an electronic sensor device that provides an output from which the orientation, velocity and gravity of the electronic sensor device can be calculated. By measuring the amount of acceleration caused by gravity, an accelerometer can calculate the angle of tilt of itself relative to the horizontal. Some accelerometers use the piezoelectric effect in crystal structures that are stressed by acceleration forces, thereby creating a voltage across the structure. Alternatively, accelerometers can operate by sensing changes in capacitance. A dynamic range of ±1.5g is sufficient for use as a tilt sensor and for detecting a drop of the audio streaming device 20.

[0029] When the audio streaming device 20 is placed on a stable surface, the audio streaming device 20 and thus the acceleration sensor 28 will not observe any acceleration and the acceleration sensor 28 will output a voltage V2 corresponding to an acceleration equal to 0. It can be seen that if the observed acceleration is below a value of e.g. 2g (twice the gravity), there is a linear relationship between the observed acceleration and the output voltage of the acceleration sensor 28. The output voltage will vary between VI and V3.

[0030] From Figure 4 It can be seen that a small deviation of the resulting acceleration equal to 0 will result in a corresponding small deviation of the voltage V2. According to the present application, small accelerations, e.g. caused by small vibrations and people walking in the room, will not affect the audio beam. However, when a larger acceleration is observed, e.g. due to the audio streaming device 20 sliding along the surface it is placed on, the microprocessor 27 will analyze the output voltage from the acceleration sensor 28. When the output voltage differs more than ΔV TH from the voltage V2, corresponding to an acceleration equal to 0, the acceleration is considered likely to cause annoying sound and the streaming of the emitter will be stopped by the microprocessor 27.

[0031] In one embodiment, the first threshold corresponds to an acceleration of 0.05*g, where g is the gravity. In some embodiments, the first threshold corresponds to an acceleration of 0.1*g, and in other embodiments, the first threshold corresponds to an acceleration of 0.2*g or 0.4*g. In order to prevent emission of audio streams containing noise due to scratching etc., it is important to apply a first threshold lower than the value corresponding to 1*g, as this would represent free fall.

[0032] Figure 5 A flow chart illustrating the decision process of the microprocessor in the audio streaming device 20 according to the present application is shown. When the audio streaming device 20 is placed stably to a surface 10, it starts by picking up audio signals from the environment by means of the microphone 21, as shown in step 51. In parallel with this, the acceleration sensor 28, e.g. a 3-axis accelerometer, picks up inertial sensor data, and in step 53, the microprocessor 27 compares the sensor data to a first threshold. The sensor data can consist of sensor data from each of the three axes and a total acceleration calculated from the sensor data of the three axes, as discussed above. Any combination of available sensor data can be compared to suitable thresholds for detecting scratching, movement and dropping of the audio streaming device 20. In step 53, in case the microprocessor 27 detects that the sensor data exceeds the comparison of the first threshold, in step 54, the microprocessor 27 disables the emission of the picked up audio. The same is the case if only one of the four parameters exceeds the relevant predetermined threshold.

[0033] At step 55, the microprocessor 27 compares the sensor data to a second threshold. This comparison is to ensure that the audio streaming device 20 is not moving. In case the audio streaming device 20 is moving in an unpredictable manner, it makes no sense to adapt the beamforming provided in the audio streaming device 20 during the movement, which is why the microprocessor 27 disables the adaptive beamforming in step 56. Then at step 58, the audio picked up by the microphone 21 and processed by the beamforming digital signal processor 26 will be encoded and transmitted based on the parameters determined before the movement was detected. Once the audio streaming device 20 stops moving and the microprocessor 27 recognizes that the sensor data is below the second threshold, the microprocessor 27 instructs the beamforming digital signal processor 26 to apply the adaptive beamforming again in step 57. Thereafter at step 58, the audio signal will be encoded and transmitted.

[0034] Once the microprocessor 27 detects that the sensor data is below the first threshold at step 53, the audio streaming device 20 continues to pick up audio and monitor the inertial sensor data and starts to stream the audio data.

[0035] Figure 6 A state diagram of the audio streaming device 20 according to one embodiment of the present application is illustrated. The audio streaming device 20 has a stable state 60, in which it acts as a remote microphone, e.g. placed on a table (horizontal and no movement for a period of time). When the acceleration sensor 28 detects a movement corresponding to touching the audio streaming device 20, and the microprocessor 27 observes that the sensor data exceeds the first threshold, the audio streaming device 20 enters an unknown state 62, in which the audio streaming of the signal picked up by means of the microphone 21 is interrupted. The microprocessor 27 mutes and / or filters out low frequencies from the microphone 21 for a short time. In some embodiments, the audio streaming device 20 applies a packet loss concealment technique on the transmitter side to mask the interrupted audio signal. The packet loss concealment technique includes zero insertion, waveform substitution (reconstructing the lost gap by repeating a portion of the audio signal that has been transmitted) or model-based methods (algorithms apply speech models to interpolate and extrapolate speech gaps).

[0036] In some embodiments, the transmission of the audio stream is interrupted when the audio streaming device 20 is in the unknown state 62. Then until the packet loss concealment technique is applied on the receiver side by the radio 37 in the hearing aid 32.

[0037] When using a wireless technology standard, such as Bluetooth TMWhen exchanging data, the transmitted microphone audio will have a latency greater than 10 ms due to the codec applied. By entering the unknown state 62, the transmission of the audio stream is interrupted for e.g. 10-20 ms, leaving the receiver or transmitter with packet loss concealment to clear the gap.

[0038] In one embodiment, a timer in the microprocessor 27 is used to set a predefined time period. If the event detected by the accelerometer 28 ends when this predefined time period expires, the audio streaming device 20 reverts to the stable state 60.

[0039] Once an event is detected and the unknown state 62 is entered, a high pass filtering of the audio signal picked up by the microphone 21 is initiated, and once the high pass filtered audio reaches the processing stage, e.g. after 0-20 ms, the transmission of the audio stream resumes. The filtering of the microphone audio continues until the total duration of the accelerometer movement or event reaches a timeout. The timer for the tabletop detection is reset to eliminate a repeated event.

[0040] Figure 6 It is also explained that the audio streaming device 20 applies adaptive beamforming when it is in the stable state 60 where it acts as a remote microphone, since the beamforming digital signal processor 26 is adapted to adaptively adjust the applied phase differences for beamforming in order to maximize the quality of speech present in the selected beam.

[0041] When the acceleration sensor 28 detects a movement corresponding to touching or turning the audio streaming device 20, and the microprocessor 27 observes that the sensor data exceeds a second threshold, the audio streaming device 20 enters a semi-stable state 61 where the adaptive beamforming is interrupted. This semi-stable state 61 is maintained until the microprocessor 27, based on the sensor data, considers that the audio streaming device 20 is again stably resting on a table, whereupon the audio streaming device 20 reverts to the stable state 60. However, if further movements are detected and also exceed the first threshold, the audio streaming device 20 enters the unknown state 62 where the audio streaming of the signal picked up by means of the microphone 21 is interrupted. In some embodiments, the first threshold and the second threshold are the same.

Claims

1. An audio streaming device for streaming audio signals to at least one hearing assistance device, and comprising - at least one input transducer, - an electronic sensor device (28) adapted to sense a gravitational force acting on the audio streaming device (20), and - a transmitter adapted to stream audio from the at least one input transducer to the at least one hearing assistance device, wherein the audio streaming device (20) further comprises a microprocessor (27) adapted to: - compare an output signal from the electronic sensor device (28) to a first threshold, and - stop streaming the audio stream from the at least one input transducer through the transmitter when the output signal from the electronic sensor device (28) exceeds the first threshold; and wherein the transmitter is adapted to stream audio as data packets, and the audio streaming device is characterized by a shock protection unit (31) adapted to instruct the transmitter to discard the next data packet and instead use a replacement data packet from the shock protection unit (31) when the output signal from the electronic sensor device exceeds the first threshold.

2. The audio streaming device according to claim 1, wherein the audio streaming device (20) further comprises a digital signal processor (26) adapted to provide beamforming based on the at least one input transducer, wherein the microprocessor (27) is further adapted to: - compare the output signal from the electronic sensor device (28) to a second threshold, and - stop the beamforming digital signal processor (26) from changing currently set beamforming parameters when the output signal from the electronic sensor device (28) exceeds the second threshold.

3. The audio streaming device according to claim 2, wherein the digital signal processor (26) is adapted to generate a plurality of uniformly spread beams by imposing appropriate phase differences on audio signals from the at least one input transducer in a delay-and-sum beamforming process.

4. The audio streaming device according to claim 3, wherein the digital signal processor (26) is adapted to select one of the plurality of uniformly spread beams, the selected beam being the beam that best fulfills criteria set based on speech quality estimates and direction of sound sources.

5. The audio streaming device according to claim 4, wherein the digital signal processor (26) is adapted to adaptively adjust the phase differences imposed for the selected beam in order to maximize speech quality present in the selected beam.

6. The audio streaming device according to claim 1, wherein the electronic sensor device (28) is adapted to provide motion measurements as accelerations along three orthogonal axes.

7. Audio streaming device according to claim 6, wherein the electronic sensor device (28) comprises a 3-axis accelerometer, wherein the microprocessor (27) is adapted to calculate a measure of the overall acceleration, and wherein the shock protection unit (31) is adapted to compare at least one motion measure with the first threshold, the at least one motion measure comprising the measure of the overall acceleration or a measure of the acceleration along each of the three orthogonal axes.

8. Audio streaming device according to claim 1, wherein the replacement data packet from the shock protection unit (31) is a copy of a previously transmitted data packet.

9. Audio streaming device according to claim 1, wherein the replacement data packet from the shock protection unit (31) is a pre-stored data packet selected by the shock protection unit (31) from a plurality of pre-stored data packets, when the pre-stored data packet is the most matching in audio classification to a previous data packet transmitted by the radio device (30).

10. Audio streaming device according to claim 1, wherein the first threshold represents an acceleration less than free fall.

11. Audio streaming device according to claim 10, wherein the first threshold represents a resultant acceleration of 0.4*g, wherein g is the gravitational force.

12. Method of managing an audio streaming device, the audio streaming device comprising at least one input transducer, an electronic sensor device and a transmitter, wherein the method comprises: - transmitting an audio stream from the at least one input transducer to at least one hearing assistance device, - sensing a gravitational force acting on the audio streaming device, - comparing the sensed gravitational force with a first threshold, - stopping the transmission of the audio stream when the sensed gravitational force exceeds the first threshold, and wherein the transmitter is adapted to stream audio as data packets, and the method is characterized in that instructing the transmitter to drop the next data packet and to use instead a replacement data packet from a shock protection unit when the sensed gravitational force exceeds the first threshold.

13. Method according to claim 12, comprising providing motion measures for accelerations along three orthogonal axes.

Images