Audio stream management

Abstract

The invention provides an audio system for managing audio streams for sleep assistance. The system detects a sleep state or a transition between sleep states for a user, and adapts an audio stream from the system accordingly. The audio system is also configured to control audio applications installed on an electronic device.

Description

[0001] AUDIO STREAM MANAGEMENT

[0002] TECHNICAL FIELD

[0003] [1] The present invention relates to control of audio streams from audio applications installed on an electronic device. Embodiments relate to management of audio for sleep assistance.

[0004] BACKGROUND

[0005] [2] A person may use an audio device, for example headphones or earbuds, in bed to play audio tracks that provide sleep assistance. For example, a user may choose to play audio books, guided relaxation tracks, or music to reduce the time it takes them to fall asleep. However, if the user falls asleep the audio may continue, which disturbs the user's sleep later in the night.

[0006] [3] To provide sleep assistance, it may be desirable to provide a more complex auditory experience, for example, to provide coloured noise from a first application concurrently with a sleep story from a second audio application, or to provide assistive sounds, such as natural sounds or even white noise, during sleep to protect the sleep state of the user. Protecting a sleeping user against disruptive background noise, such as snoring, loud traffic events and even morning birdsong, still poses a significant problem. Additionally, it is difficult to change the audio to provide such experiences without user intervention.

[0007] [4] Changing the audio playing from a headphone or earbud, for example changing an audio book to coloured noise, typically involves a user interacting with a screen on an electronic device such as a mobile phone. It is known that interacting with a screen before bed reduces sleep quality, and it is therefore recommended to avoid interacting with screens when preparing to sleep.

[0008] [5] It is therefore an aim of the present invention to provide an audio system for providing sleep assistance that addresses at least some of the problems outlined above.

[0009] SUMMARY OF THE INVENTION

[0010] [6] In an aspect, the invention provides a method of adapting an audio stream from a first audio application based on user state. The method comprises defining a first state and a second state for a user, defining an audio setting for the first state and an audio setting for the second state, detecting a transition from the first state to the second state, and adapting the audio stream, wherein the adapted audio stream is the audio setting for the second state.

[0011] [7] The method described herein advantageously allows an audio stream to adapt automatically, without requiring user input. Thus, the user is able to avoid interactions with screens, which may be desirable in some instances (such as when the user is preparing to sleep).

[0012] [8] In a further aspect, the invention provides a system for adapting an audio stream based on user state, wherein a first state is associated with a first audio setting, and a second state is associated with a second audio setting. The system comprises an electronic device configured to transmit an audio stream and a control system. The control system is configured to detect a transition from the first state to the second state and instruct the electronic device to adapt the audio stream, wherein the adapted audio stream is the audio setting for the second state.

[0013] [9] In a further aspect, the invention provides a system for controlling a first application of a plurality of audio applications. The system comprises an electronic device having installed thereon the plurality of audio applications, each application configured to provide an audio stream, and a control system. The control system is configured to receive an indication to play or pause a first audio stream, and send instructions to play or pause the first audio stream to the electronic device, wherein the instructions are readable only by the first application, the first application being configured to provide the first audio stream.

[0014]

[0010] Typically, control of multiple audio applications on the same device are linked: only one audio application can play an audio stream at once, and devices require the audio stream from the first audio application to stop before the audio stream from the second audio application can start. This is found to be disadvantageous for sleep assistance applications, where it is desirable for the user to be able to fully personalise the audio experience, combining audio streams from multiple applications. The approach provided by the present invention allows sleep assistance applications to provide such a fully personalised audio experience. The system described herein advantageously allows control of the first audio stream to be separated from control of other audio streams provided by other audio applications, allowing audio streams from multiple applications to play concurrently.

[0015]

[0011] In a further aspect, the invention provides a method for controlling a first application of a plurality of audio applications installed on an electronic device. The audio application may be a dedicated audio playing application, or an application that may also carry out other functions. The method comprises receiving an indication to play or pause a first audio stream, sending instructions to the electronic device to play or pause the first audio stream, wherein the instructions are readable only by the first application, the first application being configured to provide the first audio stream, and playing or pausing the first audio stream.

[0016]

[0036] In a further aspect, the invention provides a control system for applications implemented on an electronic device, wherein the electronic device has installed thereon an audio application for use in a plurality of defined user states; wherein there is provided a visual user interface adapted for use by the user in at least one defined user state; and wherein there is provided a haptic user interface adapted for use by the user in at least one other defined user state.

[0017]

[0037] The system described herein is able to provide a screenless user interface in particular user states, which is advantageous in user states where interacting with a screen may be undesirable, for example when preparing to sleep.

[0018]

[0038] In a further aspect, the invention provides a method of controlling an audio application installed on an electronic device, wherein the audio application is adapted to operate in a plurality of defined user states, the method comprising: providing a visual user interface for the user for use in at least one defined user state; and providing a haptic user interface for the user for use in at least one other defined user state.

[0019]

[0039] In a further aspect, the invention provides a method of operating a sleep assistance device comprising an audio player and an audio player controller to reduce impact of a background noise event, the method comprising for a background noise type defined by a plurality of aural characteristics: triggering the audio player controller to provide protection against the background noise type; the audio player controller providing protective audio content, wherein the protective audio content has one or more aural characteristics of the background noise; and the audio player playing the protective audio to reduce impact of a background noise event.

[0020] BRIEF DESCRIPTION OF THE DRAWINGS

[0021]

[0040] Specific embodiments of the invention will now be described, by way of example, with reference to the accompanying figures, of which:

[0022]

[0041] Figure 1 is a schematic block diagram showing the overview of an exemplary audio system for providing sleep assistance in accordance with embodiments of the present invention;

[0042] Figure 2 is an illustration showing an example user interface for an audio application in accordance with embodiments of aspects of the present invention;

[0023]

[0043] Figure 3 is a table outlining example audio settings that may be chosen by a user for three sleep states, asleep, trying to sleep, and not trying to sleep in accordance with an embodiment of an aspect of the present invention;

[0024]

[0044] Figure 4 is a flow diagram showing an overview of the method implemented by the system of Figure 1 to adapt audio using the detected state of a user;

[0025]

[0045] Figure 5 is a state model showing three sleep states and the transitions between them, in accordance with an embodiment of an aspect of the present invention;

[0026]

[0046] Figure 6 is a table outlining possible audio stream combinations for the three sleep states shown in Figure 5;

[0027]

[0047] Figure 7 is a similar state model to that shown in Figure 5, additionally showing example audio settings for each of the sleep states;

[0028]

[0048] Figure 8 is a flow diagram showing a prior art method of controlling audio streams;

[0029]

[0049] Figure 9 is a flow diagram showing a method of controlling audio streams according to embodiments of an aspect of the present invention.

[0030]

[0050] Figure 10 is a schematic block diagram showing the overview of an exemplary audio system for providing sleep assistance in which embodiments of the invention may be employed;

[0031]

[0051] Figure 11 sets out steps of a method according to an aspect of the invention;

[0032]

[0052] Figure 12 sets out a method of classifying and acclimatising for background noise according to an embodiment of the invention;

[0033]

[0053] Figure 13 illustrates a protective audio generation module according to an embodiment of the disclosure; and

[0054] Figure 14 illustrates a background noise classifier according to an embodiment of the disclosure;

[0034] DETAILED DESCRIPTION

[0035]

[0055] An audio system according to an embodiment of aspects of the present invention is used to provide sleep assistance for a user. The system detects a sleep state or a transition between sleep states for the user, and adapts an audio stream from the system accordingly. Here, “sleep state” is used to describe a bodily state of a user defined with respect to sleep, and while this may be defined physically, it may be further defined mentally (for example, in relation to determined user intent). Certain embodiments below utilise three sleep states, or user states, or states - the terms are used interchangeably in this specification - being asleep, awake and trying to sleep, and awake and not trying to sleep, but in principle other types of state may be used (for example, “asleep” could be subdivided into “deep sleep” and “shallow sleep”).

[0036]

[0056] Figure 1 shows the overview of the adaptive audio system 10 for providing sleep assistance, in accordance with an embodiment according to aspects of the present invention. The system 10 provides sleep assistance by adapting the audio stream depending on the detected state of a user. As indicated above the term ‘state’ refers to a body state of the user. The system 10 includes an electronic device 12, such as a phone, a computer, or a tablet, an audio device 14, for example a headphone, earphone or earbud, and a control system 16. As illustrated, the electronic device 12 includes a transmitter 18 and an audio application 20 configured to detect user state and stream audio. The audio application comprises a user interface (III) 22 for interacting with a user. In some embodiments, the electronic device 12 may also have one or more third-party audio applications 24 installed, as illustrated in Figure 1.

[0037]

[0057] The audio device 14 comprises a user interface 26, for example a button or switch, a sensor 28, a speaker 30, and a receiver 32. The transmitter 18 of the electronic device 12 is configured to transmit an audio stream 34 to the receiver 32 of the audio device 14, the transmission facilitated by a communications network such as Bluetooth. Both the electronic device 12 and the audio device 14 are also coupled to and controlled by the control system 16. Via the control system 16, communication between the electronic device 12 and the audio device 14 also occurs in the opposite direction, i.e. , from the audio device 14 to the electronic device 12. The control system 16 comprises a processor 36 and a datastore 38 which stores an algorithm 40 and audio settings 42, each of which will be described later. The control system 16 may be integrated with the user’s electronic device 12, or it may be provided by a cloud service, as a separate device, or with distributed functionality (for example, between the electronic device 12 and a remote server (not shown).

[0038]

[0058] The audio application 20 on the electronic device 12 includes a plurality of audio tracks including spoken and non-spoken audio tracks. The electronic device 12 is configured to transmit an audio stream that comprises one or more audio tracks (for example a combination of spoken and non-spoken audio) from the audio application 20 to the audio device 14. The audio track is then played through the speaker 30. The combination of spoken and non-spoken audio and thus the resultant audio stream is chosen by the userand accordingly is optimised to provide sleep assistance for the user. A different combination is chosen for each of a set of sleep states.

[0039]

[0059] In the audio application 20, the audio tracks may be categorised into a spoken library and non-spoken library. The spoken library may include songs sleep stories, or guided relaxations, while the non-spoken library may include music (without words), coloured noise (CN) or soundscapes (SS).

[0040]

[0060] To play an audio stream from the audio application 20, the audio application 20 receives an instruction (either from a user via the III 22 of the audio application 20, or via a command from the control system 16, discussed later) to play an audio stream. The audio application 20 provides the audio stream to the transmitter 18, which then transmits the audio stream to the receiver 32 in the audio device 14 via the communications network. The receiver 32 receives the audio stream and sends the audio stream to the speaker 30 to be played in real-time.

[0041]

[0061] The user can control the audio stream playing, for example pause the audio stream or change the audio track, by providing an indication to the III 22 of the audio application 20. An example III 22 of the audio application 20 is shown in Figure 2, where only the primary user interaction means are shown. As illustrated, the III 22 of the audio application 20 comprises a means for playing / pausing the audio stream, for example a play / pause button 44, and a means for changing the audio track, the skip backward and skip forward buttons 46, 48, The user may provide an indication to the III 22 to pause or play the audio by touching the pause / play button 44, to skip to the next audio track by touching the skip forward button 48. Although not shown in Figure 2, in some embodiments the user may also navigate to the spoken or non-spoken library to select a new audio track (for example a different combination of spoken and non-spoken audio).

[0042]

[0062] The audio application 20 receives the indication via the III 22 and takes the appropriate action to control the audio stream. In the above examples, if the user touches the play / pause button 44 on the III 22, the audio application 20 stops transmitting the audio stream from the audio application 20, and if the user selects a new audio track, the audio application 20 stops transmission of the previous stream, and begins to transmit the new audio stream to the audio device 14 via the transmitter 18. The play / pause 44 on the III 22 of audio application operates in a different manner to typical play / pause buttons, and this is discussed in detail later with reference to Figure 9.

[0043]

[0063] The user can also play or pause the audio stream from the audio application 20 by providing an indication to the III 26 of the audio device 14, The III 26 on the audio device 14 has the same function as the play / pause button 44 on the III 22, i.e., operates to play or pause the audio stream from the audio application 20.

[0044]

[0064] The audio stream from the audio application 20 may be streamed concurrently with audio streams from third party applications 24. For example, a meditation audio track from a third-party application can play concurrently with a CN audio stream from the audio application 20. The play / pause button 44 on the III 22 is configured such that interaction affects only the audio stream originating from the audio application 20, and does not affect (play or pause) any audio streams from third-party audio applications 24.

[0045]

[0065] The III 26 on the audio device 14 operates to control the audio stream originating from the audio application 20 as follows. The III 26 on the audio device 14 receives an indication from a user, and the audio device 14 sends the indication to the control system 16. The processor 36 generates a control signal comprising instructions to play or pause the audio stream from the audio application 20. The processor 36 then sends the control signal to the electronic device 12. The electronic device 12 receives the control signal, and the control signal is read by the audio application 20. The electronic device 20 then begins or stops transmitting the audio stream from the audio application 20.

[0046] Aspect 1 - Adapt based on user state

[0047]

[0066] The method described above relates to adapting the audio stream following receipt of an input from the user, that is, interaction of the user with the Ills 22, 26. However, in a first aspect of the present invention, the audio stream from the audio application 20 adapts automatically, without requiring user input. This is advantageous as the system is able to automatically provide sleep assistance to the user as they move between a plurality of states. In such aspects, the audio stream adapts depending on the detected state of the user, and thus prior to use, a plurality of states are defined. In some embodiments, the states relate to the different body states of a user during different sleep stages. The state of the user is predicted using sensor data and user interaction data (user interactions with the electronic device 12 or audio device 14). Once the state is predicted, an audio setting corresponding to the predicted state is selected and transmitted from the electronic device 12 to the audio device 14, as is discussed in greater detail below.

[0048]

[0067] With reference again to Figure 1 , in embodiments of the first aspect of the present invention, where the system 10 is used to automatically adapt an audio stream, the sensor 28 in the audio device 14 receives data from the surrounding environment. It should be noted that the sensor may include multiple sensors, for example for detecting the user’s sleep state, the sensor 28 may comprise a photoplethysmogram (PPG) sensor, an accelerometer, and a positional sensor. In some embodiments of the first aspect of the present invention, the audio device 14 may also comprise sensors for monitoring data, for example audio breathing data may be received via a microphone on the audio device 14. Each sensor 28 is configured to monitor a different type of physiological data for the user (e.g. heart rate data, heart rate variability data, movement data, head position data). The sensor data is provided to the control system 16, where the processor 36 monitors the received data. When a variation in sensor data is detected, the processor 36 retrieves the algorithm 40 from the datastore 38, and uses the algorithm 40 and the received sensor data to predict the user’s sleep state. Any interactions with the III 26 or III 22 are also provided to the control system 16 as interaction data. This interaction data also contributes to the algorithm’s prediction of user state.

[0049]

[0068] Alongside the algorithm 40, the datastore 38 also stores audio settings 42, which comprise data on the chosen audio stream for each of the user states. The audio settings 42 are defined by the user prior to using the system 10, and thus are personalised for each user. Example audio settings 42 that may be chosen by a user for three states, asleep, trying to sleep, and not trying to sleep, are shown in Figure 3. For the asleep state the user may select to stream coloured noise (CN) from the audio application 20, and to not stream any audio from third-party applications 24. For the audio setting 42 corresponding to the trying to sleep state, the user may choose to define an audio stream comprising coloured noise and a guided relaxation audio track from the audio application 20, and to not stream any audio from third-party audio applications 24. An example audio setting 42 corresponding to the not trying to sleep state for a particular user may comprise streaming no audio track from the audio application 20, and streaming audio from third-party applications 24. Additionally, the user may select to delay the start of the not trying to sleep audio stream by a set amount of time, for example 5 minutes, although any time may be used. If a delay of 5 minutes is defined, the audio stream for the trying to sleep state will begin 5 minutes after the system 10 detects the user has entered this state (e.g. via sensor detection of the user sitting up, discussed later). This offers advantages as if a user sits up in the middle of the night, the asleep / trying to sleep audio continues to play for a set amount of time, allowing the user to more easily fall back asleep. It should be appreciated that additional states may also be included, for example a ‘preparing for sleep’ state. Thus, the system of the present invention is multi-modal, wherein three or more states may be defined for a user, and the system detects transitions between each of the multiple states and adjusts the audio accordingly.

[0050]

[0069] The processor 36 predicts the user state using the algorithm 40, and then the processor retrieves the audio setting 42 corresponding to the predicted state from the datastore 38. The processor 36 then generates a control signal comprising instructions for the electronic device 12 to perform the audio setting 42 corresponding to the predicted state. The control signal is sent to the electronic device 12, and the electronic device adjusts the audio stream being transmitted to correspond to the received audio setting 42. The adjustment may affect the audio stream from the audio application 20, and / or audio streams from third party audio applications 24 (as described above with reference to Figure 3). The adapted audio stream is received by the audio device 14 and plays through the speaker 30. It should be noted that the requested audio setting 42 may include settings to stop audio from the audio application 20, and thus in this case the control signal comprises instructions to stop the transmission of the audio stream between the electronic device 12 and the audio device 14.

[0051]

[0070] An overview of the method 50 implemented by the system of Figure 1 for automatically adapting the audio stream using the body state of the user, according to an embodiment of the first aspect of the present invention is illustrated by the flowchart in Figure 4. The method firstly comprises defining, at Step 52, a plurality of user states, and in present methods at least two user states are required. Preferably, three user states are defined. An audio setting corresponding to each user state is then defined at Step 54, and stored. Defining the audio settings is carried out prior to using the system to automatically adapt audio streams. During use, sensor data is received and monitored at Step 56, and the system determines, at Step 58, if a transition between user states has occurred. A transition between user states is detected, as outlined above, using the algorithm 40, sensor data, and user interaction data. If a transition between user states is not detected, the system returns to Step 56, and continues to receive and monitor sensor data. If, at Step 58, a transition between states is detected, the audio stream is adapted, at Step 60, so that the adapted audio stream is the audio setting corresponding to the new state of the user.

[0052]

[0071] In an embodiment implementing this first aspect of the present invention the system is used to automatically adapt audio as a user moves between states relating to different stages of sleep. Adapting the audio to correspond to different body states in this way allows the user to create a personalised system for ensuring quality sleep. For example, the system may be configured to play coloured noise (CN) when the user is trying to sleep (if CN helps the specific user to fall asleep quickly) and then fade out the CN when the user falls asleep. Additionally, since the system automatically adapts the audio using sensor data, the user is not required to interact with their phone before bed (or during the night if they wake up and want to restart an audio stream). This is advantageous as it is known that interaction with screens can affect sleep quality. This feature is discussed in more detail below.

[0053]

[0072] A state model for such an embodiment of this first aspect of the present invention is illustrated by the schematic diagram in Figure 5. Three sleep states are shown in the described embodiment: asleep, trying to sleep (the user is awake but wants to sleep), and not trying to sleep (the user is awake and not trying to sleep). These are referred to as State 72, 74, and 76 respectively in Figure 5. Each of the sleep states are defined using the characteristics listed below:

[0054] • Asleep 72 = asleep and lying down;

[0055] • Trying to sleep 74 = awake and lying down;

[0056] • Not trying to sleep 76 = awake and sitting up.

[0057] It should be appreciated that in other embodiments of the first aspect of the present invention, additional sleep states may also be included - previously the example of subdividing the “asleep” state was given, but this may also be done for waking states (for example, a ‘preparing for sleep state’ could be defined).

[0058]

[0073] A user state is in this embodiment predicted using two separate characteristics: awake or asleep, and sitting up or lying down. The characteristics are determined using sensor data: physiological data received from the sensors is used to determine if the user is asleep or awake, and lying down or sitting up. User interaction data may also contribute to the user state prediction.

[0059]

[0074] Physiological data is monitored by the sensor 28 and provided to the control system 16. In relation to definition of sleep states, physiological data may comprise heart rate data, heart rate variability data, and movement data, which may be monitored using a PPG sensor and an accelerometer. Physiological data further comprises head position data, monitored using a position sensor.

[0060]

[0075] The processor 36 monitors the physiological data, and if a variation in data is detected, the processor 36 retrieves the algorithm 40 from the datastore 38. The algorithm predicts if the user is asleep or awake, and user position, i.e., if the user us lying down or sitting up. Thus, the algorithm 40 determines the state of the user.

[0076] The algorithm 40 comprises a machine learning (ML) model, trained using labelled physiological data of the same type received by the sensor. Physiological data comprises PPG sensor data (which indicates heart rate and heart rate variability), accelerometer data (indicating movement), and position data (indicating the user’s head position). PPG data and accelerometer data is used to determine if the user is asleep or awake. Heart rate decreases and heart rate variability increases when a user is asleep compared to awake, as does a user’s movement. Movement is also known to follow certain patterns when a user is asleep. Thus, during training the algorithm 40 receives heart rate data, heart rate variability data, and movement data, where the state of the user, asleep vs awake, is known. The algorithm 40 thus learns data patterns that indicate a user being asleep or awake: multiple statistical characteristics of the way the heart rate, heart rate variability and movement data change over the defined states are used to make the state prediction. In some embodiments of the first aspect of the present invention, thresholds may be learnt, where data below or above the thresholds may indicate if the user is asleep or awake. Thus, a ML model is generated that is able to predict if the user is asleep or awake using data from a PPG sensor and an accelerometer.

[0061]

[0077] Position data is used by the algorithm 40 to determine if the user is sitting up or lying down. Position data, specifically orientation data of the audio device 14 (for example, a headphone or an earbud), is monitored by the sensor 28 and received by the processor 36. The algorithm 40 determines the angle of the audio device 14 using the orientation data, and if the angle is within a threshold angle (for example, 20 degrees) of the angle when a user is sitting up straight, then the algorithm determines that the user is sitting up. If the angle of the audio device 14 is within a threshold angle (for example, 20 degrees) of the angle when a user is lying down, the algorithm 40 determines that the user is lying down. It should be noted that a calibration process may be required for each user, as the earbud may sit naturally at different angles for each user.

[0062]

[0078] The algorithm 40 therefore determines if the user is awake or asleep, and sitting up or lying down. Using these two characteristics, alongside any detected user interactions with the electronic device 12 or audio device 14, the sleep state of the user is predicted.

[0063]

[0079] While a ML model is described, any other suitable prediction algorithm may be used.

[0064]

[0080] Using the sensors 28 (including the PPG sensor, the accelerometer, and the position sensor), the algorithm 40 may determine if the audio device 14, that is the headphone / earbud, is being worn. This may be indicated by, for example, there being no heart rate data available, or position data being out with the thresholds for lying down or sitting up. If the algorithm 40 determines that the headphone / earbud is not being worn, the headphone / earbud automatically powers off after some time delay, the length of which can be selected by the user.

[0065]

[0081] Returning to the state model in Figure 5, the state model illustrates transitions between sleep states. As shown, transitions may comprise events such as the user falling asleep, waking up, sitting up, or lying down, which are detected using sensor data as described above. Transitions may also be detected by input actions, such as interacting with the III 26 on the audio device 14, the III 22 of the audio application 20, or with any third-party application 24.

[0066]

[0082] Considering firstly states 72 and 74 (the asleep state and the trying to sleep state), a transition 73a indicates a transition from state 72 to state 74. Transition 73a is detected by a wake-up event, that is, sensor data and the algorithm 40 determines that the user is awake but still lying down (the user interacting with the III 26 on the audio device 14 or with the III 22 on the audio application 20 also contributes to this determination). Accordingly, to detect transition 73a the system uses sensor data and interaction data. A transition 73b indicates a transition in the opposite direction, from state 74 to state 72. Transition 73b is indicated by a fall asleep event (detected using sensor data, user interaction data (no interactions) and the algorithm 40).

[0067]

[0083] Transitions between states 74 and 76 are indicated by transitions 75a and 75b. Transition 75a (from state 74 to state 76) comprises a sit up event (indicated by sensor data, user interaction data, and the algorithm 40, i.e. , the user is awake and sitting up). For example, the user may press the play / pause button 44 on the III 22 to stop the trying to sleep audio stream. Transition 75b in the opposite direction (state 76 to 74) comprises a lie down event, detected using sensor data (the user is awake and lying down) as described previously.

[0068]

[0084] Finally, transitions 77a and 77b are transitions between states 76 and 72 (the not trying to sleep state to the asleep state). Transition 77a is the transition from state 76 to 72, and comprises detection of a fall asleep event (detected using sensor data, user interaction data (no interactions) and the algorithm 40). Transition 77b (from state 72 to 76) comprises detection of a sit up event (i.e., the user is awake and sitting up, detected using sensor data, interaction data, and the algorithm 40).

[0069]

[0085] As described, in some instances an interaction between the user and one of the user interfaces 22, 26 indicates a transition between user state, and thus causes a change in the audio stream. For example, an interaction with the play / pause button 44 or the III 26 (which both carry out the same function) causes an indication to be provided to the control system 16. The processor 36 instructs the electronic device 12 to play or pause the audio stream (dependent on if the audio is currently streaming or not) from the audio application 20, and the electronic device 12 adapts the audio stream accordingly. Interacting with the III 26 or the play / pause button 44 to pause or play the audio allows the user to indicate to the system 10 that they have woken, or are no longer trying to sleep, and would like to adjust the audio stream from the audio application 20 accordingly. Interacting with the III 26 or the play / pause button 44 does not affect audio streams from any third- party applications 24 installed on the electronic device 12.

[0070]

[0086] As mentioned briefly, each user state 72, 74, 76 is associated with an audio setting 42, which is chosen by the user prior to using the system 10. The audio settings comprise an audio stream from the audio application 20, any time delays, and may also comprise an audio stream from a third- party application 24. Examples of audio streams from the audio application 20 for the sleep states 72, 74, 76 included in Figure 5 are listed in Figure 6.

[0071]

[0087] The audio stream from the audio application 20 is configured to combine spoken and nonspoken audio, with different combinations chosen for each user state. For example, as shown in Figure 6, a user may define, for the audio stream corresponding to State 72, to play non-spoken audio, such as SS or CN, or for no audio to be streamed from the audio application 20. For the audio setting 42 corresponding to State 74, the user may select to play spoken audio such as a music playlist with spoken words, non-spoken audio such as a soundscape (SS) or coloured noise (CN), or both. In some embodiments of the first aspect of the present invention, the user may select to have no audio playing when in State 74. Finally, for the audio stream corresponding to State 76, the user my select any music playlist, audio book, cognitive behavioural therapy for insomnia (CBTi), sleep hygiene, or even no audio. These audio streams may originate from the third-party applications 24 or the audio application 20. Additionally, as described previously, alongside the chosen audio stream, the user can define any time delays in the audio settings 42. For example, the user may define that the audio stream for State 76 begins 5 minutes after the user sits up (that is, the system detects transitions 75a or 77b).

[0072]

[0088] Figure 7 shows a similar state model to that illustrated in Figure 5, however Figure 7 additionally shows example audio settings 42 corresponding to each of the sleep states 72, 74, 76. In the illustrated example, third-party audio applications 24 are installed on the electronic device 12, and are also able to transmit audio streams to the audio device 14. The audio streams from the third- party audio applications 24 are referred to as external audio. The example audio settings 42 for the sleep states 72, 74, 76 described in Figure 7 are the exemplary audio settings described with reference to Figure 3.

[0073]

[0089] The audio setting 42 for each of the states 72, 74, 76 is chosen by the user prior to use and stored in the datastore 38. Considering firstly State 72, the corresponding audio setting, as described in Figure 3, comprises a CN audio stream with no external audio playing. The corresponding audio settings 42 for State 74 comprise an audio stream including CN and a guided relaxation audio track from the audio application 20, again with no external audio. Accordingly, when transition 73a is detected, the control system 16 sends instructions to the electronic device 12 to introduce the State 74 audio setting. Introducing the State 74 sleep audio comprises pausing the State 72 audio stream (coloured noise) and playing the pre-selected State 74 audio stream (coloured noise and guided relaxation). Similarly, when transition 73b is detected, the control system 16 instructs the electronic device 12 to pause the State 74 audio stream from the audio application (CN and guided relaxation), and to play the State 72 audio stream (coloured noise), i.e. , pause the guided relaxation component of the audio stream. Although not illustrated, in some embodiments of the first aspect of the present invention, external audio is played concurrently with audio from the audio application 20. The external audio may be, for example, a sleep story. Therefore, the corresponding audio setting 42 for State 74 may also include external audio.

[0074]

[0090] The audio setting corresponding to State 76 in the present example comprises streaming external audio, which begins 5 minutes after the transition to State 76 is detected (it should be noted that in some examples, the audio setting for State 76 may be to do nothing, that is, to not stream any audio from the electronic device 12). Thus, on detection of transition 75a, the control system 16 instructs the electronic device 12 to pause the State 74 audio stream from the audio application 20. External audio begins playing 5 minutes after transition 75a is detected. Conversely, when transition 75b is detected, the control system 16 is configured to instruct the electronic device 12 to pause the external audio, and play the pre-selected audio stream corresponding to State 74 from the audio application 20 (CN and guided relaxation). In some embodiments of the first aspect, the audio settings State 74 may also comprise instructions to keep playing external audio from third-party applications 24 concurrently with the audio stream from the audio application 20.

[0075]

[0091] The final transitions considered in the present example are those occurring between States 76 and 72. If transition 77a is detected, the control system 16 instructs the electronic device 12 to pause external audio from third-party applications 24, and to play the audio stream corresponding to State 72, i.e., a CN audio stream from the audio application 20. When transition 77b is detected, the control system 16 instructs the electronic device 12 to pause the CN (State 72 audio stream) and to resume streaming external audio 5 minutes after the transition 77b is detected, in order to match the audio settings corresponding to State 76.

[0076]

[0092] The user may also choose to fade in or fade our audio streams during state transitions to create a seamless experience. To fade in or fade out audio streams, the user incorporates this when defining the audio setting for each state. Aspect 2 - Separation

[0077]

[0093] As outlined, the audio system 10 is configured to stream audio from the audio application 20, and adapt the audio based on detected user states. External audio from third-party audio applications 24 installed on the electronic device 12 may also be streamed between the electronic device 12 and the audio device 14. In a second aspect of the present invention, control of the audio stream from the audio application 20 is separated from control of external audio originating from third party audio apps 24. Accordingly, audio streams from the audio application 20 can be played concurrently with external audio from third-party apps 24. This offers many advantages, for example sleep stories from third-party apps 24 can be played concurrently with an audio stream, e.g CN, from the audio application 20 when the user is trying to sleep (State 74). Additionally, smooth transitions between audio streams from the audio application 20 and third-party audio applications 24 are possible. For example, the user may play external audio (e.g. an audio book) from a third-party audio application 24 when they are not trying to sleep, and then a non-spoken audio stream (soundscapes) from the audio application 20 can be smoothly introduced to play concurrently with the external audio when the user is detected as lying down and trying to sleep. Additionally, the ability to control external audio and audio from the audio application independently allows users to create a more flexible, personalised audio experience.

[0078]

[0094] Typically, audio streams from multiple audio applications 20, 24 cannot play at the same time since the operating system of the electronic device 12 controls all installed audio apps 20, 24 in the manner illustrated by the flowchart in Figure 8. Figure 8 shows the flow of actions 200 that occur in prior art systems, once a user interacts with a play / pause III on a typical audio application while external audio originating from a third party app is playing, and the typical audio application is open but not transmitting an audio stream. The user interacts, at Step 210, with the play / pause III on the typical audio application. The operating system then sends, at Step 220, a request to the presently playing third party app 24. The third-party apps 24 stops transmitting audio and the audio stream from the typical audio application starts at Step 230. At this stage, a user may interact at Step 240, with the play / pause button on the typical audio application 20 for a second time. As a result, the audio stream from the audio application 20 stops at Step 250, and any audio from third-party apps 24 does not restart. Returning to Step 230, following Step 230 the user may interact, at Step 260, with a play / pause III on a third-party audio app 24. The operating system sends, at Step 270, a request to the presently playing typical audio application. The audio stream from the typical audio application stops and then external audio from the interacted with third party app 24 starts at Step 280.

[0095] Therefore, in typical prior system behaviours, interacting with a pause or play button on an audio application sends the request to pause or play to the app that most recently streamed audio. Similarly, in typical audio behaviour a III on a Bluetooth connected peripheral (such as a headphone) will send the request to play or pause audio to the app that most recently streamed audio. This means that audio streams from different audio applications 20, 24 cannot run concurrently. In prior art systems, if external audio from a third-party app 24 is playing, interacting with the III 26 on the audio device 14 or the play / pause button 44 on the audio application 20 will cause the operating system to first stop the external audio before playing the audio from the audio application 20.

[0079]

[0096] However, the method of the second aspect of the present invention allows the audio stream from the audio application 20 to play concurrently with external audio from third party apps 24. This is because the control of the audio stream from the audio application 20 is separated from control of external audio, which is carried out by the operating system of the electronic device 12. The method of controlling the audio from the audio application 300 according to a second aspect of the present invention is discussed below with reference to Figure 9.

[0080]

[0097] The method is implemented by III 26 on the audio device 14 or by the play / pause button 44 on the III 22 of the audio application 20 of the electronic device 12 (which carry out the same function) and by the control system 16. Rather than sending a notification to the operating system of the electronic device 12, interacting with the III 26 on the audio device or play / pause button 44 in the audio application 20 sends a notification to the control system 16.

[0081]

[0098] As illustrated in the flowchart in Figure 9, consider again the case when the electronic device 12 is playing external audio from third-party applications 24, and the audio application 20 is open on the electronic device 12 but not transmitting any audio. The control system 16 receives, at Step 310, an indication of a user interaction with the pause / play button 44 on the III 22 of the electronic device 12, or the III 26 on the audio device 14. The control system 16 then sends, at Step 320, a command to the electronic device 12, the command comprising instructions for the audio application 20 to begin transmitting audio. The audio application 20 reads the command and transmits, at Step 330, the audio stream from the electronic device 12 to the audio device 14. There is no effect on the external audio, and thus the external audio continues to play, simultaneously with the audio stream from the audio application 20. Although not illustrated, if the control system 16 was to receive an indication of a user interacting with the III 26 or the play / pause button 44 for a second time, i.e. while both the external audio and the audio from the audio application 20 is streaming, a control signal is sent to the electronic device 12 comprising instructions to pause the audio from the audio application 20. Audio from the audio application 20 stops playing. The external audio originating from third-party apps 24 is not affected and continues to play.

[0099] Thus, the control method 300 of the second aspect of the present invention means that interaction with the III 26, 22 always affects the audio stream from the audio application 20, with no change in the external audio. This is contrary to typical operation, where, if a play / pause button is pressed, the Operating System of the electronic device 12 sends the instruction to the last played audio application, which may be the external audio, as described with reference to Figure 8.

[0082]

[0100] This ability to separate control of the audio stream from external audio is enabled using a bespoke, non-standard Bluetooth command. Upon receipt of an indication of an interaction with the Ul 26, 44, the control system 16 generates a bespoke command readable only by the audio application 20. Thus, when the command is sent to the electronic device 12, the command is only understood by the audio application 20, which then takes the appropriate action. The command is not readable by any third-party apps 24, and thus there is no change to external audio.

[0083]

[0101] Accordingly, audio from the audio application 20 and external audio from third party applications 24 is separated: interacting with the Ul 26, 44 does not affect audio from third-party apps 24 and external audio from third-party apps 24 does not affect audio from the audio application 20. External audio may be paused, as discussed with reference to Figure 5, through detection of a transition between user states, such as a falling asleep event or a lie down event, or by interacting with the Ul of the third-party application 24.

[0084] Aspect 3 - Screenless adaptation

[0085]

[0102] As noted above, in embodiments a transition from one defined user state to another - such as from normal use to “trying to sleep” - may itself trigger specific actions or a change in audio application operation. As has been suggested above, one specific area for change may relate to user interfaces.

[0086]

[0103] Users are often cautioned against screen use while preparing to sleep. When users are actively trying to sleep, screen use may move from disadvantageous to an active distraction. When a user is trying to sleep, it may be desirable to provide a user interface that does not require screen use.

[0087]

[0104] The presence of the U I 26 on the audio device 14 means that the user does not need to interact with any applications on the electronic device 12 to start / stop the audio stream from the audio application 20. A haptic user interface, such as a pressable button on the audio device 14, allows for such control of the audio application 20 as is necessary in user states such as the “trying to sleep” state, as the audio application 20 is configured so that any detail in user choices to be made has already been configured. This allows the user to avoid interaction with the screen of the electronic device 12, which is beneficial in the sleep application described above, as it is known that interaction with electronic devices 12 before bed affects sleep quality. If this is adapted to be the default mode of user control of the audio application 20 in the “trying to sleep” state, then the user experience in the “trying to sleep” state can be screenless.

[0088]

[0105] The presence of the sensors that are used to detect state changes consequently can support a screenless experience, and an indication that the user is interacting with the haptic interface may contribute to determination of user state. As described with reference to Figure 7, transitions between user states are detected using sensor data, and the new user state directs which audio stream is transmitted from the audio application 20.

[0089]

[0106] In some embodiments of the third aspect of the present invention, a III on the audio device 14 (e.g., Ill 26) is provided as the primary user interface for the “trying to sleep” state. Interactions with this III may be used as an indicator (though not in itself determinative) that the user may be in the “trying to sleep” state. Typically, a single press of the III 26 causes the audio stream from the audio application to play or pause. The III 26 may be additionally configured so that two, three, or four presses of the primary play III in quick succession (for example, a time threshold less than 1s) may have other meanings - a suitable set of interactions could be configured for the detected user state. For example, interacting with the III 26 using two, three, or four presses causes the audio application 20 to switch to other audio on a user playlist for the “trying to sleep” state if the user is determined to be in that state. Other approaches are possible - in some embodiments of the third aspect of the present invention, the user may be able to switch the audio settings manually, rather than relying on sensor data and predicted state changes.

[0090]

[0107] Consequently, screenless control of the audio application 20 may extend beyond turning an audio stream on and off. A button press or combination of button presses may correspond to selection between different playlists, or between different tracks of one playlist and thus the user is able to switch between a plurality of audio streams again without interacting with the screen of the electronic device 12. This may be done through multiple presses of a single button (e.g. two presses cause the system to skip forward to the next audio track), or by provision of multiple buttons or other controls either on the audio device 14, the electronic device 12 (without requiring activation of the screen), or even on another device, for example linked by Bluetooth to the audio device.

[0091]

[0108] This approach may not be limited to the “trying to sleep” state. For example, other user states may be defined, such as a state between “not trying to sleep” and “trying to sleep” - this may be, for example, a state in which the user is sitting up but it is determined that they are in their “pre-sleep” routine. Use of the haptic interface may be used in a determination that the user is in such a state - while use of the visual user interface may be used in a determination that they are in an “active” state. As it would be recommended to minimise screen interaction when preparing for sleep, the audio application 20 is therefore adapted for providing a screenless user interface in such a “presleep” state as well.

[0092]

[0109] While a screenless user interface may be implemented effectively with haptic controls such as buttons, there are other screenless options available - for example, the audio application 20 may be adapted also to respond to audio commands. It is envisaged that a haptic approach may be more suitable for the user’s transition to sleep, but the inclusion of an audio control option may allow the opportunity for the user to provide more complex commands if required.

[0093] Aspect 4 - Sleep Protection

[0094]

[0110] In a fourth aspect of the present invention, a sleep assistance device is used to provide sleep assistance for a user. The sleep assistance device provides an audio stream adapted to provide protective audio to the user to protect against background noise. Particular embodiments of the fourth aspect of the present invention, as will be described below, are adapted to operate differently according to the sleep state of the user.

[0095]

[0111] The term “sleep state” has been defined above, and may include “deep sleep” and “shallow sleep”, for example. In embodiments of the fourth aspect of the invention, protection in “shallow sleep” may be particularly significant, as this is in practice when a user is most likely to be wakened by a background noise event.

[0096]

[0112] Figure 10 shows the adaptive audio system 10 of Figure 1 , adapted to include some additional components for providing sleep assistance, as explained below.

[0097]

[0113] As illustrated in Figure 10, the audio application 20 in embodiments according to a fourth aspect of the invention further contains a protective audio module 27. The protective audio module 27 operates to provide protective audio for inclusion in the audio stream to be provided to the user to prevent a background noise event from waking them. In the environment of Figure 10, a variety of exemplary background noise types are shown - here snoring 2, traffic noise 4 and birdsong 6 - any of which can be protected against by the protective audio module 27. This operates in connection with a protective audio generation module 275 shown as a functional module of the control system 16, implemented by a suitably programmed processor 36. The operation of the protective audio module 27 and of the protective audio generation module 275 will be described in detail below. In some of the embodiments of the fourth aspect of the invention, described below, it is necessary for the sleep assistance system to detect background sounds. For this purpose, a microphone 29 is provided - this may be an existing microphone, as will be the case if electronic device 12 is a mobile phone or a computer. It should be appreciated that the microphone 29 may be provided in other system components, or as an entirely separate component, if appropriate.

[0098]

[0114] As described previously with reference to Figure 1 , and the first aspect of the present invention, while the audio stream can be adapted following receipt of an input from the user, that is, interaction of the user with the Ills 22, 26, in embodiments of the present invention certain audio content is provided that is system-selected, rather than user-selected. This can be used to enable the system to automatically provide sleep assistance to the user as they move between a plurality of states, but in particular in embodiments of the fourth aspect of the invention to provide protective audio. In some cases, the audio stream adapts depending on the detected state of the user, and thus prior to use, a plurality of states are defined. In some embodiments of the fourth aspect of the present invention, the states relate to the different body states of a user during different sleep stages. The state of the user may then be predicted using sensor data and user interaction data (user interactions with the electronic device 12 or audio device 14). Once the state is predicted, an audio setting corresponding to the predicted state can be selected and transmitted from the electronic device 12 to the audio device 14.

[0099]

[0115] Such a system may be used to automatically adapt audio as a user moves between states relating to different stages of sleep. Adapting the audio to correspond to different body states in this way allows the user to create a personalised system for ensuring quality sleep. For example, the system may be configured to play coloured noise (CN) when the user is trying to sleep (if CN helps the specific user to fall asleep quickly) and then fade out the CN when the user falls asleep. Additionally, since the system automatically adapts the audio using sensor data, the user is not required to interact with their phone before bed (or during the night if they wake up and want to restart an audio stream). This is advantageous as it is known that interaction with screens can affect sleep quality. This feature is discussed in more detail below.

[0100]

[0116] A state model for such an embodiment of the fourth of the present invention is illustrated by the schematic diagram in Figure 5. The description above in relation to the first aspect of the invention applies equally to the fourth aspect of the invention It should be noted that other user states in addition to those shown in Figure 5 can be defined - for example, it may be able to separate asleep states into “deep sleep” and “light sleep”. This could be relevant in implementations as “light sleep” is where there is likely to be the greatest risk of a user waking up. There may be additional criteria used to define user states - for example, additional factors may be considered such as “travel”. Potentially here a user could indicate that they were travelling, and this may act to enable appropriate forms of protective audio content when required.

[0101]

[0117] Also as described previously (with reference to Figure 8), typically, audio streams from multiple audio applications 20, 24 cannot play at the same time since the operating system of the electronic device 12 controls all installed audio apps 20, 24. However, as described previously in relation to the second aspect of the present invention, the approach taken here allows the audio stream from the audio application 20 to play concurrently with external audio from third party apps 24.

[0102]

[0118] Against this context, embodiments of the fourth aspect of the invention will now be described in detail. It should be noted that while the context described is particularly appropriate for implementation of the invention, embodiments of the fourth aspect of the invention can be applied in different contexts - that is, they can be applied to different types of sleep assistance devices that also provide audio content, with different device and system architectures.

[0103]

[0119] The method of the invention will now be described in general terms with reference to Figure 11. First of all - potentially some time before sleep protection is required, and potentially as early as initial configuration of the sleep assistance device - there is a characterisation 400 of one or more types of background noise in terms of its aural characteristics. This may be such as to allow the sleep assistance device to determine whether or not a background noise event has occurred, or it may simply be such as to allow sound with the aural characteristics of the background noise type to be generated. This characterisation allows protective content for that background noise type to be provided. In practical use, the provision of protective content is triggered 410 by a trigger event - this could be for example by user choice, on the user falling asleep, or on the detection of background noise of that type. The result of this triggering process is the provision 420 of protective content for the background noise type - specific types of protective content and approaches to producing protective content will be described in more detail below. This protective content is then played 430 to the user by the audio player - again, strategies for playing protective content, and in particular for providing protective content with other audio content, will be described in more detail below.

[0104]

[0120] As described above, the first step in the method of the present invention is to characterise 400 background noise events. The characterisation 400 is described in more detail by the method 500 shown in the flowchart in Figure 12. The method 500 may be performed by the protective audio generation module 275 shown in Figure 10. The method comprises firstly classifying 502 background noise. In other words, characterising a background noise event based on various aural characteristics. The classification allows the system 10 to learn patterns in background noise events in a particular environment, and thus determine whether or not a background noise event has occurred, and predict likely background noise events.

[0105]

[0121] The method further comprises determining 504 an acclimatisation strategy. The acclimatisation strategy is the way in which the resultant audio will reduce the impact of a background noise: the way in which the novelty of a disturbance (e.g. snoring) is reduced. In this way, the resultant audio prevents a user’s sleep from being disturbed. In general, the acclimatisation strategy comprises preventing the background noise from being recognised by the user as a trigger event. Trigger events trigger a change in sleep states for a user (normally, causing a user to wake up from light sleep), and thus it is beneficial to increase the threshold for any trigger event. Trigger events will be discussed in greater detail later.

[0106]

[0122] The acclimatisation strategy is determined based on the classification of the background noise event. This is important since different background noise events require different acclimatisation strategies to reduce the impact. For example, a first background noise event may have the noise type ‘snoring’, and a first profile (typically a mix of temporal and frequency information). A second background noise may have a noise type ‘traffic’, and a second aural profile. As the first background noise and the second background noise have different aural characteristics, different acclimatisation strategies are required to reduce the impact of each of these noise events for a user. Example acclimatisation strategies include conditioning a user to the background noise (for example, playing the background noise event at a low volume consistently, though it is noted that this may not be pleasant for some users), overlaying volume profiles of trigger noises with a soundscape, and mixing the trigger noises into a soundscape.

[0107]

[0123] Finally, the method for characterising background noise events comprises implementing 506, the acclimatisation strategy. That is, generating and playing protective audio content in line with the determined acclimatisation strategy, that acts to reduce the likelihood that the background noise event will trigger waking of the sleeping user. The protective audio content comprises one or more aural characteristics of the background noise event. For example, the protective audio content may contain the same noise type (e.g. snoring). The protective audio content however may comprise the background noise event (e.g. snoring) with changes in one or more aural characteristics (e.g. frequency), such that the user is not triggered by the protective audio content, in the same way they would be by the noise event itself. This technique conditions the user to the background noise. Additionally, or alternatively, the protective audio content may comprise masking of the background noise event (reducing the difference between the ambient audio and the background noise event). In some embodiments of the fourth aspect of the present invention, the implementation may comprise adapting the protective audio content in real-time to protect against any unpredictable background noise events, as is discussed later.

[0108]

[0124] Considering the classification 502 step in more detail, as discussed briefly with reference to Figure 10, there exists multiple different background noise events that may disturb a user during sleep, causing them to transition between sleep states (e.g., from light sleep to awake). Background noise events that a user may experience include traffic sounds (noisy traffic, aeroplanes), human sounds (snoring, a baby crying, neighbours), and natural sounds (birdsong). While some of these noises are more generic (for example motorway traffic noise may not vary greatly from user to user every user), some background noises are highly user-specific, for example, a partner’s snoring.

[0109]

[0125] It should be noted that embodiments of the fourth aspect of the present invention may be directed to protection against only one type of background noise - for example, snoring - whereas other embodiments of the fourth aspect of the present invention may be directed to protection against a number of types of background noise, or any background noise. Where only specific types of background noise are being protected against, it is not necessary for a classification to cover “all” background noise. In such cases, the classification step only needs to establish the characteristics of the type or types of background noise from which protection is required, so that these can be used in the preparation of protective audio, and in some cases, used in a triggering step where this is based on determination that a background noise event has taken place.

[0110]

[0126] In embodiments of the fourth aspect of the present invention, the protective audio generation module 275 classifies background noise events and determine an acclimatisation strategy for reducing the impact of a background noise event. The protective audio generation module 275 is shown in greater detail in the block diagram in Figure 13.

[0111]

[0127] The module 275 is a model comprising two model components: a classifier 602 and a strategy generator 604. A background noise event is provided to the classifier 602 (this may be input to the module 275 by a user in a configuration step, or measured by the control system 16). The classification of the background noise event is provided to the strategy generator 604. The strategy generator 604 then determines an acclimatisation strategy for the background noise event based on the classification.

[0128] The classifier 602 classifies the background noise event using various aural characteristics. An example classifier is shown in Figure 14. In this example, at the top level, the background noise event is classified by noise type. Examples of noise types include snoring, traffic, baby, and nature, among many others. The background noise event may be further classified using aural characteristics such as amplitude analysis, peak frequency, spectral content, and dynamic range. Accordingly, the classifier characterises the background noise event using multiple different classification categories and can perform the classification process for any background noise event. The classifier may also use machine learning methods to isolate different sounds within a detected background noise event, and classify each sound separately. This is useful if a background noise is detected that comprises sounds from multiple sources (e.g., birdsong and snoring). It is not beneficial to classify these sounds using their combined aural characteristics, as the play back of such a combined sound in protective audio is not likely to assist in reducing the impact of either.

[0112]

[0129] As a model, the audio generation module 275 is trained prior to the system 10 being used to provide predictive audio. The protective audio generation module 275 does not operate to classify background noise events and generate a strategy in real time. Rather, the trained module 275 is stored (for example in a cloud data storage system), and accessed by the control system 16 to determine the acclimatisation strategy corresponding to a detected / predicted background noise event.

[0113]

[0130] Background noise events may be generic or user specific. The module 275 is trained differently depending on if the system 10 is being used to protect against generic or user-specific noise events.

[0114]

[0131] When the system 10 is being used to reduce the impact of generic background noise, the classifier 602 may be pre-trained at an initial configuration stage. In such instances, the model is trained using a set of generic background noises (e.g. traffic), and suitable acclimatisation strategies for reducing the impact of the generic background noise.

[0115]

[0132] In embodiments of the fourth aspect of the present invention where the system is being used to reduce the impact of user-specific background noises, the model is trained during use in the user’s environment (again, before being used to provide protective audio). User-specific background noise events may be measured locally from a user’s phone (for example via a microphone). A classifier trained for generic background noise may then be further trained for detected background noise events based on the experience of that user. In this way, the classifier is trained to classify both generic and user-specific background noise events, and thus the system can be used to protect against a combination of both types of background noise,

[0116]

[0133] Alternatively, a classifier originally not adapted for any background noise may be trained specifically to the user’s local environment. In this way, patterns of background noise events for specific users can be learned and predicted.

[0117]

[0134] Once trained, the model is configured to classify a background noise event and determine a suitable acclimatisation strategy, that reduces the impact of the background noise event. Acclimatisation strategies for learned background noise events are stored, and a suitable strategy is accessed by the control system 16 when triggered to play protective audio content.

[0118]

[0135] As described briefly with respect to Figure 11 , the method of the present invention comprises triggering 410 of protective noise protection. In other words, the control system 16 begins to play protective audio content in response to a trigger. Multiple different trigger types may be used, as described below.

[0119]

[0136] The trigger may be a user trigger, where a user action initiates playing of the protective audio content. This may be via a user interaction with a III on the electronic device 12, or via a change in user state, for example, a user falling asleep. A user trigger is useful in instances where the user is aware of background noise events that are likely to occur in their environment - for example, sleeping next to someone who snores, or being in a known flight path.

[0120]

[0137] Another trigger for the control system to play the protective audio may be the detection of a background noise event. For example, the control system may detect snoring, and then begin to play the protective audio content that protects against snoring. This requires the control system to use the background noise event characteristics for detection of background noise events, as well as for preparation of protective audio content. Particular strategies may be employed here to reduce the risk of triggering, in particular synchronization of background noise events and protective audio noise events. For example, on detection of a single ‘snore’, a snore sound could be played at a similar time, so acclimatisation sounds can be played in direct response to similar background noises to normalise them. Such a dynamic approach may be used instead of, or as well as, a more static approach in which protective audio content may be played on at a regular frequency (say, every 4 seconds) without background noises necessarily taking place.

[0138] Another trigger for the control system may be prediction of a background noise event. As described above, the audio generation module 275 may be trained using background noise events from the user’s local environment. Thus, the audio generation module 275 learns patterns in background noise for the local environment, and can predict background noise events. For example, the audio generation module 275 may learn that a background noise event corresponding to birdsong begins at a certain time each night, and thus can predict this background noise event before it occurs. Prediction of the noise event triggers the control system 16 to introduce protective audio content against birdsong, before the event occurs. A further possibility is for a noise signature of the current environment to be taken and used to give an indication of the upcoming noise challenge based on historic records (i.e. the environment may be recognised as that of the user’s normal bedroom - so triggers would be based on what usually happens in that environment, but if noise is detected characteristic of a different environment, such as for example aeroplane noise, a different approach might be adopted). Such an approach could assist the system in prediction of the noise challenges ahead, and so of what protective audio content may be required.

[0121]

[0139] Additionally, triggering of the control system 16 may comprise triggering against specific sounds. The control system 16 may be configured to only activate and play protective audio content on detection / prediction of specific background noise events. For example, the control system may be configured to activate and play protective audio content when snoring is detected, to prevent a partner’s snoring from disturbing a user during the night. However, the control system may be configured to not trigger protective noise on detection / prediction of bird song, as the user may want to hear this in order to help them wake up in the morning. Such configurations may be set by the user prior to the system being used to provide protective audio.

[0122]

[0140] Alternatively, the control system 16 may be configured to trigger for background noise in general. In such instances, the controller is triggered to play protective audio content when any background noise event is detected or predicted. Thus, the chance of a user being disturbed by background noises during sleep is greatly reduced.

[0123]

[0141] The protective audio content generated for a particular background noise type will be dependent on the aural characteristics of that background noise type. The basic problem to be addressed is that a standard physiological response to a significant noise disturbance during sleep - particularly during light sleep - is to wake the sleeper if the noise is assessed as being characteristic of a significant change in the user’s environment, and so potentially of a threat. This determination will be made on the basis of both intensity and familiarity - if the background noise exceeds a threshold related to both of these two factors, then it is likely that the sleeper will be woken.

[0124]

[0142] The functional effect of the protective audio content should be for it to be more difficult for a background noise event to exceed the threshold for waking the user. Manipulating either or both of the (relative) intensity and the familiarity of the background noise event can be achieved by establishing an appropriate aural background through the protective audio content. Familiarity of the background noise event type may be increased - and the threshold consequently increased - by playing protective audio content that has one or more aural characteristics of the background noise type, but which is determined to be unlikely in itself to cross the threshold. For example, this may be noise with the same frequency and temporal profile as the background noise type, but at a volume significantly below that likely to trigger the sleeper (in some embodiments of the fourth aspect of the present invention, this may be based on direct evidence - if sleep states of a user are tracked and background noise is also tracked by the sleep assistance device, then it may be possible to determine a threshold for background noise events of a particular background noise type to waken the user.

[0125]

[0143] Other strategies for providing protective audio content may be used instead, or as well, as the provision of "acclimatising” audio content of the type described above - for example, masking audio content may be played at particular frequencies to reduce the relative intensity of a background noise event at key frequencies, again reducing the likelihood that it will break the threshold. The same general considerations will apply, however, - the purpose of the protective audio content is to prevent a background noise event of a relevant background noise type from exceeding the threshold that will lead to waking the user, particularly when the user is in a light sleep state.

[0126]

[0144] While the nature of the protective audio content for a particular background noise type is primarily determined by its expected effectiveness in preventing the background noise event from exceeding the threshold, how protective audio content is provided to the user depends on particular factors. One is whether multiple background noise types need, or may need, to be protected against. This relates to the provision of multiple audio streams and its relationship to sleep protection more generally. Another is the current sleep state of the user. A third is user preference. These factors will be considered in turn below.

[0127]

[0145] Multiple Background Noises: It may be necessary to protect against multiple background noises at the same time - for example, a sleeper may require both protection against snoring and traffic noise, or snoring and birdsong. A variety of strategies may be employed to do this effectively. A general consideration will be to ensure that the protective audio content does not, either in itself or in combination with any other audio content provided to the user, itself trigger a waking threshold or otherwise disrupt the user. The following approaches may be taken:

[0128] • Determine whether all protective audio content needs to be provided at the current time. One possibility is to prioritise particular protective audio at particular times of day, or in particular sleep states, so that there is not an overload of protective audio content. For example, there could be a focus on snore protection earlier in the night and protection from birdsong from dawn onwards. Statistical mapping can be used to determine when these background noises typically occur in the user’s environment, and so to provide the best times for providing protective audio. It may therefore be possible to provide effective protection against multiple background noise types by effective scheduling of the protective audio content to be used.

[0129] • If it is desirable to provide more than one type of protective audio content at one time, these could be combined in such a manner that both will be effective but also so that no threshold will be breached by the combination. Different protective audio events could be spaced in time, for example. While such protective audio content could in principle be provided as multiple audio streams by the audio generation module 270 (and then reconciled at the sleep protection module 27 of the audio application), it may in practice be desirable for it to be provided as one audio track from the audio generation module 270 to the sleep protection module 27 in the audio application 20, which provides the sleep protection audio stream for playing to the user.

[0130]

[0146] Multiple Audio Streams: As has already been established, embodiments of the present invention may be implemented in a sleep assistance system adapted to combine multiple audio streams. While this has been described above in connection with providing customised sleep assistance content and third-party audio content, it is also relevant to the combination of protective audio content and any other content. As will be discussed below, protective audio content may be provided in more than one sleep state, and in some of these sleep states a user may have preferred audio content that the sleep assistance system is configured to play: for example, the user may have a meditation soundtrack for a “preparing to sleep” state, and a white noise soundtrack when sleep is established - alternatively the sleep assistance software may have established defaults for particular situations. The protective audio content can be combined with such content in the manner described earlier. The combination of the user determined - or sleep assistance system determined - audio may also be assessed to ensure that a threshold will not be breached. If this is determined to be a risk, the protective audio content may again be modified.

[0147] Sleep State of a User: As indicated previously, protective audio content may be provided in different sleep states depending on what is used as a triggering event. If so, the protective audio content may be presented differently in different sleep states. In a “preparing for sleep” state, it may be desirable to start playing protective audio to condition the user, for example against expected snoring, but the protective audio content should not be at a level that will disturb the user’s enjoyment of other audio content played at that time (such as a meditation track or a story). If there is a differentiation between light and deep sleep, there may be different approaches taken - it may for example be possible to play more intense protective audio content in a deep sleep state, as it will be perceived by the sleeping user but will be unlikely to wake them, than in a shallow sleep state, where a waking event may be more easily triggered.

[0131]

[0148] User Preference: User preference may also drive the use of protective audio content. The user may for example bar the use of protective audio content at certain times (such as in the “preparing for sleep” state while listening to a story) even when triggered.

[0132]

[0149] Schedule and Adaptation: It is possible for provision of protective audio content to be highly scheduled - determined in advance by the audio generation module 270 or by the sleep protection module 27 in the audio application - or highly adaptive (or anywhere in between). In an adaptive approach, the sleep protection module 27, or possibly the audio generation module, determines dynamically what protective audio is required, for example by a combination of scheduling and background noise detection. In this case, the audio generation module 270 may generate protective audio content dynamically based on present needs - and possibly also based on any other audio content currently being played. Again, this protective audio content may be adjusted and optimised for current requirements. Figure 14 illustrates such functionality as being present within a classifier module 602 - such functionality could however be provided at other points within the overall sleep assistance system.

Claims

CLAIMS1. A method of adapting an audio stream from a first audio application based on user state, the method comprising: defining a first state and a second state for a user; defining an audio setting for the first state and an audio setting for the second state; detecting a transition from the first state to the second state; and adapting the audio stream, wherein the adapted audio stream is the audio setting for the second state.

2. The method of claim 1 , wherein the first state is a first sleep state, the second state is a second sleep state, and detecting a transition comprises detecting a transition between sleep states.

3. The method of claim 2, wherein the first sleep state and the second sleep state are comprised within a plurality of sleep states, wherein each sleep state is a bodily state of a user.

4. The method of claim 3, wherein said plurality of sleep states include a state in which the user is asleep, a state in which the user is awake and attempting to sleep, and a state in which the user is awake and is not attempting to sleep.

5. The method of any of claims 2 to 4, the method further comprising receiving data from a sensor on an audio device, wherein detecting the transition from the first state to the second state is based on sensor data.

6. The method of claim 5, wherein the step of receiving senor data comprises receiving physiological data for the user.

7. The method of claim 6, wherein the step of receiving physiological data comprises receiving heart rate and / or heart variability data, movement data, and position data.

8. The method of claim 6 or claim 7, wherein detecting a transition from the first state to the second state comprises: detecting a variation in physiological data; and predicting the second state using an algorithm and the physiological data.

9. The method of claim 8 wherein the algorithm is a machine learning algorithm.

10. The method of any preceding claim further comprising transmitting the adapted audio stream from the audio application to an associated audio device.

11. The method of any preceding claim, wherein the audio setting for the first state or the audio setting for the second state or both comprise a spoken audio component and a non-spoken audio component.

12. The method of any preceding claim, further comprising providing a time delay between detecting the transition from the first state to the second state and adapting the audio stream to the audio setting for the second state.

13. The method of any preceding claim, wherein the audio setting for the first state or the audio setting for the second state or both comprise an audio stream from a first audio application and an audio stream from a second audio application, wherein the stream from the first audio application and the audio stream from the second audio application play concurrently.

14. The method of any preceding claim, wherein the audio setting for the first state is different to the audio setting for the second state.

15. A system for adapting an audio stream based on user state, wherein a first state is associated with a first audio setting, and a second state is associated with a second audio setting, the adaptive audio system comprising: an electronic device configured to transmit an audio stream; and a control system, wherein the control system is configured to: detect a transition from the first state to the second state and instruct the electronic device to adapt the audio stream, wherein the adapted audio stream is the audio setting for the second state.

16. The system of claim 15, wherein the control system is further configured to receive sensor data and detect a transition from the first state to the second state using sensor data.

17. The system of claim 15 or 16 further comprising an audio device configured to play the adapted audio stream, wherein the control system is further configured to detect a transition fromthe first state to the second state on receiving an indication of an interaction on a user interface of the audio device.

18. The system of any of claims 15 to 17, wherein the control system is further configured to detect a transition from the first state to the second state on receiving an indication of an interaction on a user interface of the electronic device.19 A system for controlling a first application of a plurality of audio applications, the system comprising: an electronic device having installed thereon the plurality of audio applications, each application configured to provide an audio stream; and a control system configured to: receive an indication to play or pause a first audio stream; and send instructions to play or pause the first audio stream to the electronic device, wherein the instructions are readable only by the first application, the first application being configured to provide the first audio stream.

20. The system of claim 19, wherein the first application includes a first user interface means, and the control system is configured to receive the indication from the first user interface means.

21. The system of claim 19 or 20 further comprising an audio device for playing the plurality of audio streams, wherein the control system is further configured to instruct the electronic device to transmit the plurality of audio streams to the audio device.

22. The system of claim 21, wherein the audio device is an earphone.

23. The system of claim 21 or 22, wherein the audio device comprises a second user interface means, and the control system is configured to receive the indication from the second user interface means.

24. The system of claim 23, wherein the second user interface means is a haptic user interface.

25. The system of any of claims 19 to 24, wherein the first audio stream and a second audio stream provided by a second application are configured to play concurrently, and the control system is configured to play or pause only the first audio stream in response to receiving an indication.

26. The system of any of claims 19 to 25, wherein the control system is further configured to receive sensor data and to play or pause the first audio stream, the second audio stream, or both based on sensor data.

27. The system of any of claims 19 to 26, wherein the first audio stream comprises a spoken audio component and a non-spoken audio component.

28. The system of any of claims 19 to 27, wherein the first audio stream is configured to provide sleep assistance.

29. A method for controlling a first application of a plurality of audio applications installed on an electronic device, the method comprising: receiving an indication to play or pause a first audio stream; sending instructions to the electronic device to play or pause the first audio stream, wherein the instructions are readable only by the first application, the first application being configured to provide the first audio stream; and playing or pausing the first audio stream.

30. The method of claim 29 further comprising playing the first audio stream and a second audio stream provided by a second application concurrently, wherein receiving an indication does not play or pause the second audio stream.

31. A control system for applications implemented on an electronic device, wherein the electronic device has installed thereon an audio application for use in a plurality of defined user states; wherein there is provided a visual user interface adapted for use by the user in at least one defined user state; and wherein there is provided a haptic user interface adapted for use by the user in at least one other defined user state.

32. The control system of claim 31 , wherein the defined user states are sleep or waking states of the user.33 The control system of claim 32, wherein the at least one other defined user state is a waking state comprising preparation for sleep.

34. The control system of any of claims 31 to 33, wherein the control system is adapted to determine whether the user is in one of the defined user states by detected signals.

35. The control system of claim 34, wherein the detected signals comprise sensor readings from the user.

36. The control system of claim 34 or claim 35, wherein the detected signals comprise command signals from the user.

37. The control system of claim 36, wherein the command signals are provided from the haptic user interface.

38. The control system of any of claims 31 to 37, wherein in the at least one other defined user state, the haptic user interface provides for starting and stopping an audio stream provided by the audio application.

39. The control system of claim 37 or claim 38, wherein the haptic user interface provides for switching content of an audio stream provided by the audio application according to a predetermined selection.

40. An electronic device comprising a control system as claimed in any of claims 31 to 39.

41. The electronic device of claim 40, wherein the electronic device is a mobile telephone.

42. The electronic device of claim 40 or claim 41 , wherein the electronic device comprises the haptic user interface.

43. An audio system comprising the electronic device of any of claims 40 to 42 and an audio device in wireless communication with the electronic device.

44. The audio system of claim 43, wherein the audio device is an earphone.

45. The audio system of claim 44, wherein the earphone is an in-ear device.

46. The audio system of any of claims 43 to 45, wherein the audio device comprises the haptic user interface.

47. The audio system of claim 46, wherein the haptic user interface comprises one or more buttons.

48. The audio system of any of claims 43 to 45, wherein the audio system comprises a further device in wireless communication with at least the electronic device, and wherein the further device comprises the haptic user interface.

49. A method of controlling an audio application installed on an electronic device, wherein the audio application is adapted to operate in a plurality of defined user states, the method comprising: providing a visual user interface for the user for use in at least one defined user state; and providing a haptic user interface for the user for use in at least one other defined user state.

50. The method of claim 59, wherein the defined user states are sleep or waking states of the user.

51. The method of claim 49 or claim 50, wherein the at least one other defined user state is a waking state comprising preparation for sleep.

52. The method of any of claims 49 to 51 , further comprising detecting signals from the user and determining whether the user is in one of the defined user states.

53. The method of claim 52, wherein the detected signals comprise sensor readings from the user.

54. The method of claim 52 or claim 53, wherein the detected signals comprise user interactions with either the visual user interface or the haptic user interface.

55. The method of any of claims 49 to 54, comprising starting and stopping an audio stream provided by the audio application in response to a command received through the haptic user interface.

56. The method of any of claims 49 to 55, comprising switching content of an audio stream provided by the audio application according to a predetermined selection in response to a command received through the haptic user interface.

57. A method of operating a sleep assistance device comprising an audio player and an audio player controller to reduce impact of a background noise event, the method comprising for a background noise type defined by a plurality of aural characteristics: triggering the audio player controller to provide protection against the background noise type; the audio player controller providing protective audio content, wherein the protective audio content has one or more aural characteristics of the background noise; and the audio player playing the protective audio to reduce impact of a background noise event.

58. The method of claim 57, wherein the background noise type is snoring, a transportation noise or a natural sound .

59. The method of claim 57 or claim 58, wherein the method of operating the sleep assistance device comprises providing protection against a plurality of background noise types.

60. The method of any of claims 57 to 59, wherein triggering of the audio player controller comprises triggering for the background noise type, or for any background noise.

61. The method of any of claims 57 to 60 further comprising characterising the background noise type based on the plurality of aural characteristics62. The method of claim 61, further comprising training a model to characterise the background noise type.

63. The method of claim 62, wherein the model is trained to detect any background noise, or to detect a specific a background noise type, wherein training the model to detect a specific background noise type comprising training the model based on user data.

64. The method of any of claims 57 to 63, further comprising blending, by the audio player controller, the protective audio content with other audio content65. The method of any of claims 57 to 64, wherein the audio player playing the protective audio to reduce the impact of a background noise event comprises preventing the background noise event from being recognised as a threshold event, wherein triggering the audio player controller to provide protection against the background noise event is based on detection of the threshold event.

66. The method of claim 65, wherein the threshold event comprises a volume, a volume at a frequency, a volume associated with a frequency profile, or a temporal profile of a background sound.67 The method of claim 65, wherein the threshold event comprises a recognition of a background sound type and meeting of a threshold once the sound has been recognised.

68. The method of any of claims 65 to 67, further comprising playing, by the audio player, a sub-threshold version of the background noise type, or preventing, by the audio player controller, individual events from reaching the threshold.

69. The method of claim 68, wherein playing the sub-threshold version of the background noise type comprises providing a sub-threshold signal comprising one or more aural characteristics of the background noise type.

70. The method of claim 68, wherein preventing individual events from reaching the threshold comprises providing, by the audio player controller, masking noise.

71. The method of any of claims 57 to 70, wherein the audio player playing the protective audio comprises playing a consistent protective audio or playing an adaptive protective audio.

Images