Audio Stream Adaptation

Abstract

The invention provides a method of adapting an audio stream from a first audio application based on user state. The method comprises defining a first state and a second state for a user, defining an audio setting for the first state and an audio setting for the second state, detecting a transition from the first state to the second state, and adapting the audio stream, wherein the adapted audio stream is the audio setting for the second state.

Description

TECHNICAL FIELD [1] The present invention relates to adapting an audio stream from an audio application BACKGROUND [2] A person may use an audio device, for example headphones or earbuds, in bed to play audio tracks that provide sleep assistance. For example, a user may choose to play audio books, guided relaxation tracks, or music to reduce the time it takes them to fall asleep. However, if the user falls asleep the audio may continue, which disturbs the user's sleep later in the night. Similarly, the user may want to play coloured noise when sleeping to improve sleep quality and block out sounds from the environment, but coloured noise may not be optimised to help the user fall asleep. [3] However, changing the audio playing from a headphone or earbud, for example changing an audio book to coloured noise, typically involves a user interacting with a screen on an electronic device such as a mobile phone. It is known that interacting with a screen before bed reduces sleep quality, and it is therefore recommended to avoid interacting with screens when preparing to sleep. [4] It is therefore an aim of the present invention to provide an audio system for providing sleep assistance that addresses at least some of the problems outlined above. SUMMARY OF THE INVENTION [5] In an aspect, the invention provides a method of adapting an audio stream from a first audio application based on user state. The method comprises defining a first state and a second state for a user, defining an audio setting for the first state and an audio setting for the second state, detecting a transition from the first state to the second state, and adapting the audio stream, wherein the adapted audio stream is the audio setting for the second state. [6] The method described herein advantageously allows an audio stream to adapt automatically, without requiring user input. Thus, the user is able to avoid interactions with screens, which may be desirable in some instances (such as when the user is preparing to sleep). [7] The first state may be a first sleep state, the second state may be a second sleep state, and detecting a transition may comprise detecting a transition between sleep states. [8] Accordingly, the method is able to automatically provide sleep assistance to the user as they move between a plurality of different sleep states, ensuring quality sleep for the user. [9] The first sleep state may be comprised within a plurality of sleep states, wherein each sleep state is a bodily state of a user.

[10] The plurality of sleep states may include a state in which the user is asleep, a state in which the user is awake and attempting to sleep, and a state in which the user is awake and is not attempting to sleep.

[11] The method may further comprise receiving data from a sensor on an audio device, wherein detecting the transition from the first state to the second state is based on sensor data.

[12] The sensor data may comprise physiological data for the user.

[13] The physiological data may comprise heart rate and / or heart variability data, movement data, and position data.

[14] Detecting a transition from the first state to the second state may comprise detecting a variation in physiological data, and predicting the second state using an algorithm and the physiological data.

[15] The algorithm may be a machine learning algorithm.

[16] The method may further comprise transmitting the adapted audio stream from the audio application to an associated audio device.

[17] The audio setting for the first state or the audio setting for the second state or both may comprise a spoken audio component and a non-spoken audio component.

[18] Thus, the method allows for a combination of different audio components, that can be personalised for a user for each of the defined user states.

[19] The method may further comprise providing a time delay between detecting the transition from the first state to the second state and adapting the audio stream to the audio setting for the second state.

[20] The audio setting for the first state or the audio setting for the second state or both may comprise an audio stream from a first audio application and an audio stream from a second audio application, wherein the stream from the first audio application and the audio stream from the second audio application play concurrently.

[21] The audio setting for the first state may be different to the audio setting for the second state.

[22] In another aspect, the invention provides a system for adapting an audio stream based on user state, wherein a first state is associated with a first audio setting, and a second state is associated with a second audio setting. The system comprises an electronic device configured to transmit an audio stream and a control system. The control system is configured to detect a transition from the first state to the second state and instruct the electronic device to adapt the audio stream, wherein the adapted audio stream is the audio setting for the second state.

[23] The control system may be further configured to receive sensor data and detect a transition from the first state to the second state using sensor data.

[24] The system may further comprise an audio device configured to play the adapted audio stream, wherein the control system is further configured to detect a transition from the first state to the second state on receiving an indication of an interaction on a user interface of the audio device.

[25] The control system may be further configured to detect a transition from the first state to the second state on receiving an indication of an interaction on a user interface of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[26] Specific embodiments of the invention will now be described, by way of example, with reference to the accompanying figures, of which:

[27] Figure 1 is a schematic block diagram showing the overview of an exemplary audio system for providing sleep assistance in accordance with embodiments of the present invention;

[28] Figure 2 is an illustration showing an example user interface for an audio application in accordance with embodiments of aspects of the present invention;

[29] Figure 3 is a table outlining example audio settings that may be chosen by a user for three sleep states, asleep, trying to sleep, and not trying to sleep in accordance with an embodiment of an aspect of the present invention;

[30] Figure 4 is a flow diagram showing an overview of the method implemented by the system of Figure 1 to adapt audio using the detected state of a user;

[31] Figure 5 is a state model showing three sleep states and the transitions between them, in accordance with an embodiment of an aspect of the present invention;

[32] Figure 6 is a table outlining possible audio stream combinations for the three sleep states shown in Figure 5;

[33] Figure 7 is a similar state model to that shown in Figure 5, additionally showing example audio settings for each of the sleep states;

[34] Figure 8 is a flow diagram showing a prior art method of controlling audio streams;

[35] Figure 9 is a flow diagram showing a method of controlling audio streams according to embodiments of an aspect of the present invention. DETAILED DESCRIPTION

[36] An audio system according to an embodiment of aspects of the present invention is used to provide sleep assistance for a user. The system detects a sleep state or a transition between sleep states for the user, and adapts an audio stream from the system accordingly. Here, “sleep state” is used to describe a bodily state of a user defined with respect to sleep, and while this may be defined physically, it may be further defined mentally (for example, in relation to determined user intent). Certain embodiments below utilise three sleep states, or user states, or states - the terms are used interchangeably in this specification - being asleep, awake and trying to sleep, and awake and not trying to sleep, but in principle other types of state may be used (for example, “asleep” could be subdivided into “deep sleep” and “shallow sleep”).

[37] Figure 1 shows the overview of the adaptive audio system 10 for providing sleep assistance, in accordance with an embodiment according to aspects of the present invention. The system 10 provides sleep assistance by adapting the audio stream depending on the detected state of a user. As indicated above the term ‘state’ refers to a body state of the user. The system 10 includes an electronic device 12, such as a phone, a computer, or a tablet, an audio device 14, for example a headphone, earphone or earbud, and a control system 16. As illustrated, the electronic device 12 includes a transmitter 18 and an audio application 20 configured to detect user state and stream audio. The audio application comprises a user interface (III) 22 for interacting with a user. In some embodiments, the electronic device 12 may also have one or more third-party audio applications 24 installed, as illustrated in Figure 1.

[38] The audio device 14 comprises a user interface 26, for example a button or switch, a sensor 28, a speaker 30, and a receiver 32. The transmitter 18 of the electronic device 12 is configured to transmit an audio stream 34 to the receiver 32 of the audio device 14, the transmission facilitated by a communications network such as Bluetooth. Both the electronic device 12 and the audio device 14 are also coupled to and controlled by the control system 16. Via the control system 16, communication between the electronic device 12 and the audio device 14 also occurs in the opposite direction, i.e., from the audio device 14 to the electronic device 12. The control system 16 comprises a processor 36 and a datastore 38 which stores an algorithm 40 and audio settings 42, each of which will be described later.

[39] The audio application 20 on the electronic device 12 includes a plurality of audio tracks including spoken and non-spoken audio tracks. The electronic device 12 is configured to transmit an audio stream that comprises one or more audio tracks (for example a combination of spoken and non-spoken audio) from the audio application 20 to the audio device 14. The audio track is then played through the speaker 30. The combination of spoken and non-spoken audio and thus the resultant audio stream is chosen by the userand accordingly is optimised to provide sleep assistance for the user. A different combination is chosen for each of a set of sleep states.

[40] In the audio application 20, the audio tracks may be categorised into a spoken library and non-spoken library. The spoken library may include songs sleep stories, or guided relaxations, while the non-spoken library may include music (without words), coloured noise (CN) or soundscapes (SS).

[41] To play an audio stream from the audio application 20, the audio application 20 receives an instruction (either from a user via the UI 22 of the audio application 20, or via a command from the control system 16, discussed later) to play an audio stream. The audio application 20 provides the audio stream to the transmitter 18, which then transmits the audio stream to the receiver 32 in the audio device 14 via the communications network. The receiver 32 receives the audio stream and sends the audio stream to the speaker 30 to be played in real-time.

[42] The user can control the audio stream playing, for example pause the audio stream or change the audio track, by providing an indication to the UI 22 of the audio application 20. An example UI 22 of the audio application 20 is shown in Figure 2, where only the primary user interaction means are shown. As illustrated, the III 22 of the audio application 20 comprises a means for playing / pausing the audio stream, for example a play / pause button 44, and a means for changing the audio track, the skip backward and skip forward buttons 46, 48, The user may provide an indication to the UI 22 to pause or play the audio by touching the pause / play button 44, to skip to the next audio track by touching the skip forward button 48. Although not shown in Figure 2, in some embodiments the user may also navigate to the spoken or non-spoken library to select a new audio track (for example a different combination of spoken and non-spoken audio).

[43] The audio application 20 receives the indication via the U122 and takes the appropriate action to control the audio stream. In the above examples, if the user touches the play / pause button 44 on the UI 22, the audio application 20 stops transmitting the audio stream from the audio application 20, and if the user selects a new audio track, the audio application 20 stops transmission of the previous stream, and begins to transmit the new audio stream to the audio device 14 via the transmitter 18. The play / pause 44 on the UI 22 of audio application operates in a different manner to typical play / pause buttons, and this is discussed in detail later with reference to Figure 9.

[44] The user can also play or pause the audio stream from the audio application 20 by providing an indication to the UI 26 of the audio device 14, The UI 26 on the audio device 14 has the same function as the play / pause button 44 on the UI 22, i.e., operates to play or pause the audio stream from the audio application 20.

[45] The audio stream from the audio application 20 may be streamed concurrently with audio streams from third party applications 24. For example, a meditation audio track from a third-party application can play concurrently with a CN audio stream from the audio application 20. The play / pause button 44 on the UI 22 is configured such that interaction affects only the audio stream originating from the audio application 20, and does not affect (play or pause) any audio streams from third-party audio applications 24.

[46] The UI 26 on the audio device 14 operates to control the audio stream originating from the audio application 20 as follows. The UI 26 on the audio device 14 receives an indication from a user, and the audio device 14 sends the indication to the control system 16. The processor 36 generates a control signal comprising instructions to play or pause the audio stream from the audio application 20. The processor 36 then sends the control signal to the electronic device 12. The electronic device 12 receives the control signal, and the control signal is read by the audio application 20. The electronic device 20 then begins or stops transmitting the audio stream from the audio application 20. Aspect 1 - Adapt based on user state

[47] The method described above relates to adapting the audio stream following receipt of an input from the user, that is, interaction of the user with the Uis 22, 26. However, in an aspect of the present invention, the audio stream from the audio application 20 adapts automatically, without requiring user input. This is advantageous as the system is able to automatically provide sleep assistance to the user as they move between a plurality of states. In such aspects, the audio stream adapts depending on the detected state of the user, and thus prior to use, a plurality of states are defined. In some embodiments, the states relate to the different body states of a user during different sleep stages. The state of the user is predicted using sensor data and user interaction data (user interactions with the electronic device 12 or audio device 14). Once the state is predicted, an audio setting corresponding to the predicted state is selected and transmitted from the electronic device 12 to the audio device 14, as is discussed in greater detail below.

[48] With reference again to Figure 1, in embodiments where the system 10 is used to automatically adapt an audio stream, the sensor 28 in the audio device 14 receives data from the surrounding environment. It should be noted that the sensor may include multiple sensors, for example for detecting the user’s sleep state, the sensor 28 may comprise a photoplethysmogram (PPG) sensor, an accelerometer, and a positional sensor. In some embodiments the audio device 14 may also comprise sensors for monitoring data, for example audio breathing data may be received via a microphone on the audio device 14. Each sensor 28 is configured to monitor a different type of physiological data for the user (e.g. heart rate data, heart rate variability data, movement data, head position data). The sensor data is provided to the control system 16, where the processor 36 monitors the received data. When a variation in sensor data is detected, the processor 36 retrieves the algorithm 40 from the datastore 38, and uses the algorithm 40 and the received sensor data to predict the user’s sleep state. Any interactions with the UI 26 or UI 22 are also provided to the control system 16 as interaction data. This interaction data also contributes to the algorithm’s prediction of user state.

[49] Alongside the algorithm 40, the datastore 38 also stores audio settings 42, which comprise data on the chosen audio stream for each of the user states. The audio settings 42 are defined by the user prior to using the system 10, and thus are personalised for each user. Example audio settings 42 that may be chosen by a user for three states, asleep, trying to sleep, and not trying to sleep, are shown in Figure 3. For the asleep state the user may select to stream coloured noise (CN) from the audio application 20, and to not stream any audio from third-party applications 24. For the audio setting 42 corresponding to the trying to sleep state, the user may choose to define an audio stream comprising coloured noise and a guided relaxation audio track from the audio application 20, and to not stream any audio from third-party audio applications 24. An example audio setting 42 corresponding to the not trying to sleep state for a particular user may comprise streaming no audio track from the audio application 20, and streaming audio from third-party applications 24. Additionally, the user may select to delay the start of the not trying to sleep audio stream by a set amount of time, for example 5 minutes, although any time may be used. If a delay of 5 minutes is defined, the audio stream for the trying to sleep state will begin 5 minutes after the system 10 detects the user has entered this state (e.g. via sensor detection of the user sitting up, discussed later). This offers advantages as if a user sits up in the middle of the night, the asleep / trying to sleep audio continues to play for a set amount of time, allowing the user to more easily fall back asleep. It should be appreciated that additional states may also be included, for example a ‘preparing for sleep’ state. Thus, the system of the present invention is multi-modal, wherein three or more states may be defined for a user, and the system detects transitions between each of the multiple states and adjusts the audio accordingly.

[50] The processor 36 predicts the user state using the algorithm 40, and then the processor retrieves the audio setting 42 corresponding to the predicted state from the datastore 38. The processor 36 then generates a control signal comprising instructions for the electronic device 12 to perform the audio setting 42 corresponding to the predicted state. The control signal is sent to the electronic device 12, and the electronic device adjusts the audio stream being transmitted to correspond to the received audio setting 42. The adjustment may affect the audio stream from the audio application 20, and / or audio streams from third party audio applications 24 (as described above with reference to Figure 3). The adapted audio stream is received by the audio device 14 and plays through the speaker 30. It should be noted that the requested audio setting 42 may include settings to stop audio from the audio application 20, and thus in this case the control signal comprises instructions to stop the transmission of the audio stream between the electronic device 12 and the audio device 14.

[51] An overview of the method 50 implemented by the system of Figure 1 for automatically adapting the audio stream using the body state of the user, according to an embodiment of an aspect of the present invention is illustrated by the flowchart in Figure 4. The method firstly comprises defining, at Step 52, a plurality of user states, and in present methods at least two user states are required. Preferably, three user states are defined. An audio setting corresponding to each user state is then defined at Step 54, and stored. Defining the audio settings is carried out prior to using the system to automatically adapt audio streams. During use, sensor data is received and monitored at Step 56, and the system determines, at Step 58, if a transition between user states has occurred. A transition between user states is detected, as outlined above, using the algorithm 40, sensor data, and user interaction data. If a transition between user states is not detected, the system returns to Step 56, and continues to receive and monitor sensor data. If, at Step 58, a transition between states is detected, the audio stream is adapted, at Step 60, so that the adapted audio stream is the audio setting corresponding to the new state of the user.

[52] In an embodiment implementing this aspect of the present invention the system is used to automatically adapt audio as a user moves between states relating to different stages of sleep. Adapting the audio to correspond to different body states in this way allows the user to create a personalised system for ensuring quality sleep. For example, the system may be configured to play coloured noise (CN) when the user is trying to sleep (if CN helps the specific user to fall asleep quickly) and then fade out the CN when the user falls asleep. Additionally, since the system automatically adapts the audio using sensor data, the user is not required to interact with their phone before bed (or during the night if they wake up and want to restart an audio stream). This is advantageous as it is known that interaction with screens can affect sleep quality. This feature is discussed in more detail below.

[53] A state model for such an embodiment of this aspect of the present invention is illustrated by the schematic diagram in Figure 5. Three sleep states are shown in the described embodiment: asleep, trying to sleep (the user is awake but wants to sleep), and not trying to sleep (the user is awake and not trying to sleep). These are referred to as State 72, 74, and 76 respectively in Figure 5. Each of the sleep states are defined using the characteristics listed below: • Asleep 72 = asleep and lying down; • Trying to sleep 74 = awake and lying down; • Not trying to sleep 76 = awake and sitting up. It should be appreciated that in other embodiments, additional sleep states may also be included -previously the example of subdividing the “asleep” state was given, but this may also be done for waking states (for example, a ‘preparing for sleep state’ could be defined).

[54] A user state is in this embodiment predicted using two separate characteristics: awake or asleep, and sitting up or lying down. The characteristics are determined using sensor data: physiological data received from the sensors is used to determine if the user is asleep or awake, and lying down or sitting up. User interaction data may also contribute to the user state prediction.

[55] Physiological data is monitored by the sensor 28 and provided to the control system 16. In relation to definition of sleep states, physiological data may comprise heart rate data, heart rate variability data, and movement data, which may be monitored using a PPG sensor and an accelerometer. Physiological data further comprises head position data, monitored using a position sensor.

[56] The processor 36 monitors the physiological data, and if a variation in data is detected, the processor 36 retrieves the algorithm 40 from the datastore 38. The algorithm predicts if the user is asleep or awake, and user position, i.e., if the user us lying down or sitting up. Thus, the algorithm 40 determines the state of the user.

[57] The algorithm 40 comprises a machine learning (ML) model, trained using labelled physiological data of the same type received by the sensor. Physiological data comprises PPG sensor data (which indicates heart rate and heart rate variability), accelerometer data (indicating movement), and position data (indicating the user’s head position). PPG data and accelerometer data is used to determine if the user is asleep or awake. Heart rate decreases and heart rate variability increases when a user is asleep compared to awake, as does a user’s movement. Movement is also known to follow certain patterns when a user is asleep. Thus, during training the algorithm 40 receives heart rate data, heart rate variability data, and movement data, where the state of the user, asleep vs awake, is known. The algorithm 40 thus learns data patterns that indicate a user being asleep or awake: multiple statistical characteristics of the way the heart rate, heart rate variability and movement data change over the defined states are used to make the state prediction. In some embodiments, thresholds may be learnt, where data below or above the thresholds may indicate if the user is asleep or awake. Thus, a ML model is generated that is able to predict if the user is asleep or awake using data from a PPG sensor and an accelerometer.

[58] Position data is used by the algorithm 40 to determine if the user is sitting up or lying down. Position data, specifically orientation data of the audio device 14 (for example, a headphone or an earbud), is monitored by the sensor 28 and received by the processor 36. The algorithm 40 determines the angle of the audio device 14 using the orientation data, and if the angle is within a threshold angle (for example, 20 degrees) of the angle when a user is sitting up straight, then the algorithm determines that the user is sitting up. If the angle of the audio device 14 is within a threshold angle (for example, 20 degrees) of the angle when a user is lying down, the algorithm 40 determines that the user is lying down. It should be noted that a calibration process may be required for each user, as the earbud may sit naturally at different angles for each user.

[59] The algorithm 40 therefore determines if the user is awake or asleep, and sitting up or lying down. Using these two characteristics, alongside any detected user interactions with the electronic device 12 or audio device 14, the sleep state of the user is predicted.

[60] While a ML model is described, any other suitable prediction algorithm may be used.

[61] Using the sensors 28 (including the PPG sensor, the accelerometer, and the position sensor), the algorithm 40 may determine if the audio device 14, that is the headphone / earbud, is being worn. This may be indicated by, for example, there being no heart rate data available, or position data being out with the thresholds for lying down or sitting up. If the algorithm 40 determines that the headphone / earbud is not being worn, the headphone / earbud automatically powers off after some time delay, the length of which can be selected by the user.

[62] Returning to the state model in Figure 5, the state model illustrates transitions between sleep states. As shown, transitions may comprise events such as the user falling asleep, waking up, sitting up, or lying down, which are detected using sensor data as described above. Transitions may also be detected by input actions, such as interacting with the UI 26 on the audio device 14, the UI 22 of the audio application 20, or with any third-party application 24.

[63] Considering firstly states 72 and 74 (the asleep state and the trying to sleep state), a transition 73a indicates a transition from state 72 to state 74. Transition 73a is detected by a wake-up event, that is, sensor data and the algorithm 40 determines that the user is awake but still lying down (the user interacting with the UI 26 on the audio device 14 or with the UI 22 on the audio application 20 also contributes to this determination). Accordingly, to detect transition 73a the system uses sensor data and interaction data. A transition 73b indicates a transition in the opposite direction, from state 74 to state 72. Transition 73b is indicated by a fall asleep event (detected using sensor data, user interaction data (no interactions) and the algorithm 40).

[64] Transitions between states 74 and 76 are indicated by transitions 75a and 75b. Transition 75a (from state 74 to state 76) comprises a sit up event (indicated by sensor data, user interaction data, and the algorithm 40, i.e., the user is awake and sitting up). For example, the user may press the play / pause button 44 on the UI 22 to stop the trying to sleep audio stream. Transition 75b in the opposite direction (state 76 to 74) comprises a lie down event, detected using sensor data (the user is awake and lying down) as described previously.

[65] Finally, transitions 77a and 77b are transitions between states 76 and 72 (the not trying to sleep state to the asleep state). Transition 77a is the transition from state 76 to 72, and comprises detection of a fall asleep event (detected using sensor data, user interaction data (no interactions) and the algorithm 40). Transition 77b (from state 72 to 76) comprises detection of a sit up event (i.e., the user is awake and sitting up, detected using sensor data, interaction data, and the algorithm 40).

[66] As described, in some instances an interaction between the user and one of the user interfaces 22, 26 indicates a transition between user state, and thus causes a change in the audio stream. For example, an interaction with the play / pause button 44 or the UI 26 (which both carry out the same function) causes an indication to be provided to the control system 16. The processor 36 instructs the electronic device 12 to play or pause the audio stream (dependent on if the audio is currently streaming or not) from the audio application 20, and the electronic device 12 adapts the audio stream accordingly. Interacting with the UI 26 or the play / pause button 44 to pause or play the audio allows the user to indicate to the system 10 that they have woken, or are no longer trying to sleep, and would like to adjust the audio stream from the audio application 20 accordingly. Interacting with the UI 26 or the play / pause button 44 does not affect audio streams from any third-party applications 24 installed on the electronic device 12.

[67] As mentioned briefly, each user state 72, 74, 76 is associated with an audio setting 42, which is chosen by the user prior to using the system 10. The audio settings comprise an audio stream from the audio application 20, any time delays, and may also comprise an audio stream from a third-party application 24. Examples of audio streams from the audio application 20 for the sleep states 72, 74, 76 included in Figure 5 are listed in Figure 6.

[68] The audio stream from the audio application 20 is configured to combine spoken and nonspoken audio, with different combinations chosen for each user state. For example, as shown in Figure 6, a user may define, for the audio stream corresponding to State 72, to play non-spoken audio, such as SS or CN, or for no audio to be streamed from the audio application 20. For the audio setting 42 corresponding to State 74, the user may select to play spoken audio such as a music playlist with spoken words, non-spoken audio such as a soundscape (SS) or coloured noise (CN), or both. In some embodiments, the user may select to have no audio playing when in State 74. Finally, for the audio stream corresponding to State 76, the user my select any music playlist, audio book, cognitive behavioural therapy for insomnia (CBTi), sleep hygiene, or even no audio. These audio streams may originate from the third-party applications 24 or the audio application 20. Additionally, as described previously, alongside the chosen audio stream, the user can define any time delays in the audio settings 42. For example, the user may define that the audio stream for State 76 begins 5 minutes after the user sits up (that is, the system detects transitions 75a or 77b).

[69] Figure 7 shows a similar state model to that illustrated in Figure 5, however Figure 7 additionally shows example audio settings 42 corresponding to each of the sleep states 72, 74, 76. In the illustrated example, third-party audio applications 24 are installed on the electronic device 12, and are also able to transmit audio streams to the audio device 14. The audio streams from the third-party audio applications 24 are referred to as external audio. The example audio settings 42 for the sleep states 72, 74, 76 described in Figure 7 are the exemplary audio settings described with reference to Figure 3.

[70] The audio setting 42 for each of the states 72, 74, 76 is chosen by the user prior to use and stored in the datastore 38. Considering firstly State 72, the corresponding audio setting, as described in Figure 3, comprises a CN audio stream with no external audio playing. The corresponding audio settings 42 for State 74 comprise an audio stream including CN and a guided relaxation audio track from the audio application 20, again with no external audio. Accordingly, when transition 73a is detected, the control system 16 sends instructions to the electronic device 12 to introduce the State 74 audio setting. Introducing the State 74 sleep audio comprises pausing the State 72 audio stream (coloured noise) and playing the pre-selected State 74 audio stream (coloured noise and guided relaxation). Similarly, when transition 73b is detected, the control system 16 instructs the electronic device 12 to pause the State 74 audio stream from the audio application (CN and guided relaxation), and to play the State 72 audio stream (coloured noise), i.e., pause the guided relaxation component of the audio stream. Although not illustrated, in some embodiments external audio is played concurrently with audio from the audio application 20. The external audio may be, for example, a sleep story. Therefore, the corresponding audio setting 42 for State 74 may also include external audio.

[71] The audio setting corresponding to State 76 in the present example comprises streaming external audio, which begins 5 minutes after the transition to State 76 is detected (it should be noted that in some examples, the audio setting for State 76 may be to do nothing, that is, to not stream any audio from the electronic device 12). Thus, on detection of transition 75a, the control system 16 instructs the electronic device 12 to pause the State 74 audio stream from the audio application 20. External audio begins playing 5 minutes after transition 75a is detected. Conversely, when transition 75b is detected, the control system 16 is configured to instruct the electronic device 12 to pause the external audio, and play the pre-selected audio stream corresponding to State 74 from the audio application 20 (CN and guided relaxation). In some embodiments, the audio settings State 74 may also comprise instructions to keep playing external audio from third-party applications 24 concurrently with the audio stream from the audio application 20.

[72] The final transitions considered in the present example are those occurring between States 76 and 72. If transition 77a is detected, the control system 16 instructs the electronic device 12 to pause external audio from third-party applications 24, and to play the audio stream corresponding to State 72, i.e., a CN audio stream from the audio application 20. When transition 77b is detected, the control system 16 instructs the electronic device 12 to pause the CN (State 72 audio stream) and to resume streaming external audio 5 minutes after the transition 77b is detected, in order to match the audio settings corresponding to State 76.

[73] The user may also choose to fade in or fade our audio streams during state transitions to create a seamless experience. To fade in or fade out audio streams, the user incorporates this when defining the audio setting for each state. Aspect 2 - Separation

[74] As outlined, the audio system 10 is configured to stream audio from the audio application 20, and adapt the audio based on detected user states. External audio from third-party audio applications 24 installed on the electronic device 12 may also be streamed between the electronic device 12 and the audio device 14. In an aspect of the present invention, control of the audio stream from the audio application 20 is separated from control of external audio originating from third party audio apps 24. Accordingly, audio streams from the audio application 20 can be played concurrently with external audio from third-party apps 24. This offers many advantages, for example sleep stories from third-party apps 24 can be played concurrently with an audio stream, e.g CN, from the audio application 20 when the user is trying to sleep (State 74). Additionally, smooth transitions between audio streams from the audio application 20 and third-party audio applications 24 are possible. For example, the user may play external audio (e.g. an audio book) from a third-party audio application 24 when they are not trying to sleep, and then a non-spoken audio stream (soundscapes) from the audio application 20 can be smoothly introduced to play concurrently with the external audio when the user is detected as lying down and trying to sleep. Additionally, the ability to control external audio and audio from the audio application independently allows users to create a more flexible, personalised audio experience.

[75] Typically, audio streams from multiple audio applications 20, 24 cannot play at the same time since the operating system of the electronic device 12 controls all installed audio apps 20, 24 in the manner illustrated by the flowchart in Figure 8. Figure 8 shows the flow of actions 200 that occur in prior art systems, once a user interacts with a play / pause UI on a typical audio application while external audio originating from a third party app is playing, and the typical audio application is open but not transmitting an audio stream. The user interacts, at Step 210, with the play / pause UI on the typical audio application. The operating system then sends, at Step 220, a request to the presently playing third party app 24. The third-party apps 24 stops transmitting audio and the audio stream from the typical audio application starts at Step 230. At this stage, a user may interact at Step 240, with the play / pause button on the typical audio application 20 for a second time. As a result, the audio stream from the audio application 20 stops at Step 250, and any audio from third-party apps 24 does not restart. Returning to Step 230, following Step 230 the user may interact, at Step 260, with a play / pause UI on a third-party audio app 24. The operating system sends, at Step 270, a request to the presently playing typical audio application. The audio stream from the typical audio application stops and then external audio from the interacted with third party app 24 starts at Step 280.

[76] Therefore, in typical prior system behaviours, interacting with a pause or play button on an audio application sends the request to pause or play to the app that most recently streamed audio. Similarly, in typical audio behaviour a UI on a Bluetooth connected peripheral (such as a headphone) will send the request to play or pause audio to the app that most recently streamed audio. This means that audio streams from different audio applications 20, 24 cannot run concurrently. In prior art systems, if external audio from a third-party app 24 is playing, interacting with the UI 26 on the audio device 14 or the play / pause button 44 on the audio application 20 will cause the operating system to first stop the external audio before playing the audio from the audio application 20.

[77] However, the method of the present invention allows the audio stream from the audio application 20 to play concurrently with external audio from third party apps 24. This is because the control of the audio stream from the audio application 20 is separated from control of external audio, which is carried out by the operating system of the electronic device 12. The method of controlling the audio from the audio application 300 according to an aspect of the present invention is discussed below with reference to Figure 9.

[78] The method is implemented by UI 26 on the audio device 14 or by the play / pause button 44 on the UI 22 of the audio application 20 of the electronic device 12 (which carry out the same function) and by the control system 16. Rather than sending a notification to the operating system of the electronic device 12, interacting with the UI 26 on the audio device or play / pause button 44 in the audio application 20 sends a notification to the control system 16.

[79] As illustrated in the flowchart in Figure 9, consider again the case when the electronic device 12 is playing external audio from third-party applications 24, and the audio application 20 is open on the electronic device 12 but not transmitting any audio. The control system 16 receives, at Step 310, an indication of a user interaction with the pause / play button 44 on the UI 22 of the electronic device 12, or the UI 26 on the audio device 14. The control system 16 then sends, at Step 320, a command to the electronic device 12, the command comprising instructions for the audio application 20 to begin transmitting audio. The audio application 20 reads the command and transmits, at Step 330, the audio stream from the electronic device 12 to the audio device 14. There is no effect on the external audio, and thus the external audio continues to play, simultaneously with the audio stream from the audio application 20. Although not illustrated, if the control system 16 was to receive an indication of a user interacting with the UI 26 or the play / pause button 44 for a second time, i.e. while both the external audio and the audio from the audio application 20 is streaming, a control signal is sent to the electronic device 12 comprising instructions to pause the audio from the audio application 20. Audio from the audio application 20 stops playing. The external audio originating from third-party apps 24 is not affected and continues to play.

[80] Thus, the control method 300 of the present invention means that interaction with the UI 26, 22 always affects the audio stream from the audio application 20, with no change in the external audio. This is contrary to typical operation, where, if a play / pause button is pressed, the Operating System of the electronic device 12 sends the instruction to the last played audio application, which may be the external audio, as described with reference to Figure 8.

[81] This ability to separate control of the audio stream from external audio is enabled using a bespoke, non-standard Bluetooth command. Upon receipt of an indication of an interaction with the UI 26, 44, the control system 16 generates a bespoke command readable only by the audio application 20. Thus, when the command is sent to the electronic device 12, the command is only understood by the audio application 20, which then takes the appropriate action. The command is not readable by any third-party apps 24, and thus there is no change to external audio.

[82] Accordingly, audio from the audio application 20 and external audio from third party applications 24 is separated: interacting with the UI 26, 44 does not affect audio from third-party apps 24 and external audio from third-party apps 24 does not affect audio from the audio application 20. External audio may be paused, as discussed with reference to Figure 5, through detection of a transition between user states, such as a falling asleep event or a lie down event, or by interacting with the UI of the third-party application 24. Aspect 3 - Screenless adaptation

[83] As noted above, in embodiments a transition from one defined user state to another - such as from normal use to “trying to sleep” - may itself trigger specific actions or a change in audio application operation. As has been suggested above, one specific area for change may relate to user interfaces.

[84] Users are often cautioned against screen use while preparing to sleep. When users are actively trying to sleep, screen use may move from disadvantageous to an active distraction. When a user is trying to sleep, it may be desirable to provide a user interface that does not require screen use.

[85] The presence of the UI 26 on the audio device 14 means that the user does not need to interact with any applications on the electronic device 12 to start / stop the audio stream from the audio application 20. A haptic user interface, such as a pressable button on the audio device 14, allows for such control of the audio application 20 as is necessary in user states such as the “trying to sleep” state, as the audio application 20 is configured so that any detail in user choices to be made has already been configured. This allows the user to avoid interaction with the screen of the electronic device 12, which is beneficial in the sleep application described above, as it is known that interaction with electronic devices 12 before bed affects sleep quality. If this is adapted to be the default mode of user control of the audio application 20 in the “trying to sleep” state, then the user experience in the “trying to sleep” state can be screenless.

[86] The presence of the sensors that are used to detect state changes consequently can support a screenless experience, and an indication that the user is interacting with the haptic interface may contribute to determination of user state. As described with reference to Figure 7, transitions between user states are detected using sensor data, and the new user state directs which audio stream is transmitted from the audio application 20.

[87] In some embodiments, a UI on the audio device 14 (e.g., UI 26) is provided as the primary user interface for the “trying to sleep” state. Interactions with this UI may be used as an indicator (though not in itself determinative) that the user may be in the “trying to sleep” state. Typically, a single press of the UI 26 causes the audio stream from the audio application to play or pause. The UI 26 may be additionally configured so that two, three, or four presses of the primary play UI in quick succession (for example, a time threshold less than 1s) may have other meanings - a suitable set of interactions could be configured for the detected user state. For example, interacting with the UI 26 using two, three, or four presses causes the audio application 20 to switch to other audio on a user playlist for the “trying to sleep” state if the user is determined to be in that state. Other approaches are possible - in some embodiments, the user may be able to switch the audio settings manually, rather than relying on sensor data and predicted state changes.

[88] Consequently, screenless control of the audio application 20 may extend beyond turning an audio stream on and off. A button press or combination of button presses may correspond to selection between different playlists, or between different tracks of one playlist and thus the user is able to switch between a plurality of audio streams again without interacting with the screen of the electronic device 12. This may be done through multiple presses of a single button (e.g. two presses cause the system to skip forward to the next audio track), or by provision of multiple buttons or other controls either on the audio device 14, the electronic device 12 (without requiring activation of the screen), or even on another device, for example linked by Bluetooth to the audio device.

[89] This approach may not be limited to the “trying to sleep” state. For example, other user states may be defined, such as a state between “not trying to sleep” and “trying to sleep” - this may be, for example, a state in which the user is sitting up but it is determined that they are in their “pre-sleep” routine. Use of the haptic interface may be used in a determination that the user is in such a state -while use of the visual user interface may be used in a determination that they are in an “active” state. As it would be recommended to minimise screen interaction when preparing for sleep, the audio application 20 is therefore adapted for providing a screenless user interface in such a “presleep” state as well.

[90] While a screenless user interface may be implemented effectively with haptic controls such as buttons, there are other screenless options available - for example, the audio application 20 may be adapted also to respond to audio commands. It is envisaged that a haptic approach may be more suitable for the user’s transition to sleep, but the inclusion of an audio control option may allow the opportunity for the user to provide more complex commands if required.

Claims

1. A method of adapting an audio stream from a first audio application based on user state, the method comprising:defining a first state and a second state for a user;defining an audio setting for the first state and an audio setting for the second state; detecting a transition from the first state to the second state; andadapting the audio stream, wherein the adapted audio stream is the audio setting for the second state.

2. The method of claim 1, wherein the first state is a first sleep state, the second state is a second sleep state, and detecting a transition comprises detecting a transition between sleep states.

3. The method of claim 2, wherein the first sleep state and the second sleep state are comprised within a plurality of sleep states, wherein each sleep state is a bodily state of a user.

4. The method of claim 3, wherein said plurality of sleep states include a state in which the user is asleep, a state in which the user is awake and attempting to sleep, and a state in which the user is awake and is not attempting to sleep.

5. The method of any of claims 2 to 4, the method further comprising receiving data from a sensor on an audio device, wherein detecting the transition from the first state to the second state is based on sensor data.

6. The method of claim 5, wherein the step of receiving senor data comprises receiving physiological data for the user.

7. The method of claim 6, wherein the step of receiving physiological data comprises receiving heart rate and / or heart variability data, movement data, and position data.

8. The method of claim 6 or claim 7, wherein detecting a transition from the first state to the second state comprises:detecting a variation in physiological data; andpredicting the second state using an algorithm and the physiological data.

9. The method of claim 8 wherein the algorithm is a machine learning algorithm.

10. The method of any preceding claim further comprising transmitting the adapted audiostream from the audio application to an associated audio device.

11. The method of any preceding claim, wherein the audio setting for the first state or the audio setting for the second state or both comprise a spoken audio component and a non-spoken audio component.

12. The method of any preceding claim, further comprising providing a time delay between detecting the transition from the first state to the second state and adapting the audio stream to the audio setting for the second state.

13. The method of any preceding claim, wherein the audio setting for the first state or the audio setting for the second state or both comprise an audio stream from a first audio application and an audio stream from a second audio application, wherein the stream from the first audio application and the audio stream from the second audio application play concurrently.

14. The method of any preceding claim, wherein the audio setting for the first state is different to the audio setting for the second state.

15. A system for adapting an audio stream based on user state, wherein a first state is associated with a first audio setting, and a second state is associated with a second audio setting, the adaptive audio system comprising:an electronic device configured to transmit an audio stream; anda control system, wherein the control system is configured to:detect a transition from the first state to the second state andinstruct the electronic device to adapt the audio stream, wherein the adapted audio stream is the audio setting for the second state.

16. The system of claim 15, wherein the control system is further configured to receive sensor data and detect a transition from the first state to the second state using sensor data.

17. The system of claim 15 or 16 further comprising an audio device configured to play the adapted audio stream, wherein the control system is further configured to detect a transition fromthe first state to the second state on receiving an indication of an interaction on a user interface of the audio device.

18. The system of any of claims 15 to 17, wherein the control system is further configured to detect a transition from the first state to the second state on receiving an indication of an interaction on a user interface of the electronic device.Application No: GB2405002.3Examiner:Contract Unit ExaminerClaims searched: 1-18Date of search: 21 November 2024Patents Act 1977: Search Report under Section 17Documents considered to be relevant:Category Relevant to claims Identity of document and passage or figure of particular relevance X 1-18 US2018 / 078735 Al (DALGLEISH MARTIN ET AL) paragraphs [0002] - [0012], [0024] -[0065]; figures 1, 2, 7 X 1,15 US2020 / 320890 Al (KUMAR AGRAWAL AMIT ET AL) abstract; figure 3, paragraphs [0001] - [0004], [0014] - [0052] X 1,15 US2016 / 330311 Al (DU JIANGHONG JULIE ET AL) abstract; figure 4, paragraphs [0001], [0002], [0011] - [0041]; figures 1-3 X 1, 15 EP4192037 Al (HUAWEI TECH CO LTD) abstract; figure 1, paragraphs [0004] -[0017], [0171], [0180], [0291] X 1,15 US2017 / 149945 Al (LEE DANIEL ET AL) paragraphs [0004] - [0038]; figures 1, 2 X 1, 15 US2022 / 249017 Al (ZHOU HUIHUI ET AL) paragraphs [0002] - [0008], [0030] - [0041]; figure 1 X 1,15 US2018 / 232197 Al (ISHIHARA ATSUSHI ET AL) abstract; figure 1, paragraphs [0001] -[0009], [0021] - [0045]; figures 2, 3Categories:X Document indicating lack of novelty or inventive step A Document indicating technological background and / or state of the art. Y Document indicating lack of inventive step if P Document published on or after the declared priority date but combined with one or more other documents of same category. before the filing date of this invention. & Member of the same patent family E Patent document published on or after, but with priority date earlier than, the filing date of this application.Field of Search:Search of GB, EP. WO &US patent documents classified in the following areas of the UKCX :Worldwide search of patent documents classified in the following areas of the IPC____________G06F_______________________________________________________The following online and other databases have been used in the preparation of this search reportInternational Classification:Subclass Subgroup Valid From G06F 0003 / 16 01 / 01 / 2006

Images