Method and system for context-dependent automatic volume compensation
Patent Information
- Application Number
- CN202211165579.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-08-09
- Filing Date
- 2022-09-23
- Publication Date
- 2026-11-03
- Estimated Expiration
- 2042-09-23
AI Technical Summary
Existing headsets and headphones cannot automatically adjust volume in different environments, resulting in a poor user experience, especially when switching between quiet and noisy environments, requiring manual volume adjustment.
By integrating sensors and processors into electronic devices, the system utilizes microphone signals and sensor data to determine the context, selects an appropriate volume compensation model, and automatically adjusts the volume to adapt to ambient noise and user activity.
It enables automatic volume adjustment in different environments, improving user experience, reducing manual user intervention, and enhancing the listenability of audio content.
Smart Images

Figure CN115866489B_ABST
Abstract
Description
[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 248,342, filed September 24, 2021, which is incorporated herein by reference. Technical Field
[0002] One aspect of this disclosure relates to methods and systems for context-dependent automatic volume compensation. Other aspects are also described. Background Technology
[0003] A headset is an audio device that includes a pair of speakers, each placed in the user's ear when worn on or around the user's head. Similar to a headset, headphones (or in-ear headsets) are two separate audio devices, each with a speaker that inserts into the user's ear. Headsets and headphones are typically wired to a separate playback device, such as a digital audio player, which drives each speaker of the device with an audio signal to generate sound (e.g., music). Headsets and headphones provide a convenient way for users to listen to audio content privately without having to broadcast it to others nearby. Summary of the Invention
[0004] One aspect of this disclosure is a method performed by an electronic device (e.g., a programmed processor integrated therein), such as a wearable device (e.g., a pair of smart glasses, a smartwatch, a pair of wireless headsets, etc.), for performing context-dependent automatic volume compensation. The electronic device receives an audio signal, which may contain audio content desired by the user, such as musical works, podcasts, movie soundtracks, etc., and receives a microphone signal using one or more microphones, which includes audio (or ambient noise) of the environment in which the electronic device is located. The electronic device determines the context of the electronic device and selects a volume compensation model from several volume compensation models based on the determined context. The electronic device processes the audio signal according to the selected volume compensation model and the microphone signal, and uses the processed audio signal to drive one or more speakers of the electronic device.
[0005] In one aspect, the context of an electronic device can be determined based on the audio content of an audio signal. For example, when the audio content does not include speech, the selected volume compensation model may include a broadband compressor for compressing the entire frequency range of the audio signal, while when the audio content includes speech, the selected volume compensation model may include a multi-band compressor for compressing a subset of one or more frequency bands of the entire frequency range of the audio signal. In some aspects, the context of an electronic device includes indications that one or more software applications are being executed by a programmed processor of the electronic device, wherein the audio signal may be associated with a software application that a user of the electronic device is interacting with. In another aspect, the context of an electronic device is based on sensor data from one or more sensors of the electronic device, such as a Global Positioning System (GPS) sensor, camera, microphone, thermistor, inertial measurement unit (IMU), and accelerometer. In some aspects, the context of an electronic device includes user activities, such as at least one of the following: interactions between the user and the electronic device (e.g., a device that receives user input via one or more input devices), and physical activities performed by the user when the electronic device is part of or attached to the user (e.g., when worn or held by the user). In some aspects, the context of an electronic device is the location of the device.
[0006] In one aspect, the electronic device determines a change in the context of the electronic device, selects a different volume compensation model from several volume compensation models based on the change in context, and processes an audio signal according to the selected different volume compensation model and a microphone signal. In one aspect, each volume compensation model includes at least one of the following: one or more scalar gain values to be applied to the audio signal, a wideband compressor or a multiband compressor, a compression ratio, a start-up time for applying the compression ratio to the wideband compressor or multiband compressor, and a release time for removing the compression ratio from the wideband compressor or multiband compressor.
[0007] In one aspect, processing the audio signal according to the selected volume compensation model and microphone signal includes using the selected volume compensation model to compensate the audio signal for the environment. In some aspects, the electronic device is a portable device. In another aspect, the electronic device is a wearable device, such as a pair of smart glasses or a smartwatch. In yet another aspect, one or more speakers are integrated within the electronic device, wherein the electronic device does not include hardware volume controls arranged to adjust the sound output level of one or more speakers of the electronic device.
[0008] According to another aspect of this disclosure, a method executed by an audio playback software application, which is executed by a programmed processor of an electronic device excluding volume controls, to perform context-dependent automatic volume compensation. The electronic device receives an audio signal including audio content and receives sensor data from one or more sensors arranged to sense the condition of the environment in which the electronic device is located. The electronic device determines a device snapshot including the current state of each of one or more software applications being executed by the electronic device, wherein the one or more software applications include the audio playback software application. The electronic device determines at least one audio adjustment parameter of a volume compensator based on the sensor data, the snapshot of the one or more software applications, and the audio content of the audio signal. The device processes the audio signal using the volume compensator according to the determined audio adjustment parameter and uses the processed audio signal to drive one or more speakers.
[0009] In one aspect, the current state of each of the one or more software applications indicates at least one software application currently being executed by the electronic device, the user of the electronic device interacting with the software application, and whether the audio content of the audio signal is associated with the software application. In another aspect, the device snapshot is a first device snapshot including a first state of the software application being executed by the electronic device, and the method further includes determining a second device snapshot including a second state different from the first state of the software application; determining different audio adjustment parameters based at least on the second state of the software application; and processing the audio signal according to the determined different audio adjustment parameters. In some aspects, determining at least one audio adjustment parameter includes determining a scalar gain value that a volume compensator will apply to the audio signal, and a compression ratio, start-up time, and release time that the volume compensator will compress the audio signal.
[0010] The above overview does not constitute an exhaustive list of all aspects of this disclosure. It is contemplated that this disclosure encompasses all systems and methods that can be practiced by all suitable combinations of the aspects outlined above and those disclosed in the detailed embodiments below and specifically pointed out in the claims. Such combinations may have specific advantages not specifically set forth in the foregoing summary. Attached Figure Description
[0011] Multiple aspects are illustrated in the accompanying drawings by way of example rather than limitation, and similar reference numerals in the drawings indicate similar elements. It should be noted that references to "a" or "an" aspect in this disclosure do not necessarily refer to the same aspect, and each refers to at least one. Furthermore, for the sake of brevity and to reduce the total number of drawings, a single drawing may be used to illustrate features of more than one aspect, and for a particular aspect, not all elements in that drawing may be necessary.
[0012] Figure 1 A block diagram of a system based on one aspect is shown.
[0013] Figure 2 A block diagram of an output device with automatic volume compensation dependent on the execution context of one aspect is shown.
[0014] Figure 3 An example of a data structure including a volume compensation model is shown, based on several aspects.
[0015] Figure 4 It is a flowchart of a process for performing context-dependent automatic volume compensation based on one aspect.
[0016] Figure 5 It is a flowchart of the process for determining the context of an output device based on one aspect. Detailed Implementation
[0017] Various aspects of this disclosure will now be explained with reference to the accompanying drawings. Unless the shape, relative position, and other aspects of the components described in any aspect are explicitly defined, the scope of this disclosure is not limited to the components shown, which are for illustrative purposes only. Furthermore, while numerous details have been set forth, it should be understood that some embodiments may be implemented without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of the description. Moreover, unless the meaning is explicitly contrary, all scopes shown herein are to be considered to include the endpoints of each scope.
[0018] Figure 1 A block diagram of a system (or audio system) 1 according to one aspect is shown. Specifically, system 1 includes a playback device 2, an output device 3, a network (e.g., the Internet) 4, and a content server 5. In one aspect, the system may include more or fewer elements, such as having an additional content server, or excluding a content server and / or playback device. In this case, the output device may perform all (or most) audio signal processing operations as described herein.
[0019] In one aspect, content server 5 may be a standalone electronic server, computer (e.g., desktop computer), or a cluster of server computers configured to store, stream, and / or receive digital content such as audio content (e.g., as one or more audio signals of any audio format). In another aspect, content server may store video and / or audio content such as movies for streaming (transmission) to one or more electronic devices. As shown, the server (e.g., via network 4) is communicatively coupled to playback device 2 to stream (e.g., audio) content for playback (e.g., via output device). In another aspect, content server is communicatively coupled (e.g., directly) to output device.
[0020] In one aspect, the playback device can be any electronic device (e.g., having electronic components such as one or more processors, memory, etc.) capable of streaming audio content in any format, such as stereo audio signals, for playback (e.g., via one or more speakers integrated within the playback device and / or via one or more output devices, as described herein). For example, the playback device can be a desktop computer, a laptop computer, a digital media player, etc. In another aspect, the device can be a portable electronic device (e.g., handheld), such as a tablet computer, a smartphone, etc. In yet another aspect, the playback device can be a wearable device (e.g., a device designed to be worn (e.g., attached to) a user's clothing and / or body), such as a smartwatch.
[0021] In one aspect, the output device 3 can be any (e.g., portable) electronic device including at least one speaker and configured to output (or play back) sound by driving the speaker with an audio signal. For example, as shown, the device is a wireless headset (e.g., an in-ear headset or earphone) designed to be positioned on (or in) a user's ear and designed to output sound into the user's ear canal. In some aspects, the earphone can be a sealed type with a flexible earpiece end designed to acoustically seal the entrance to the user's ear canal relative to the surrounding environment by blocking or occluding it within the ear canal. As shown, the output device includes a left earphone for the user's left ear and a right earphone for the user's right ear. In this case, each earphone can be configured to output at least one audio channel of video content (e.g., the right earphone outputs the right audio channel in a dual-channel input of, for example, a stereo recording of a musical work, and the left earphone outputs the left channel). In another aspect, the output device can be any electronic device including at least one speaker and arranged for wear by a user and arranged to output sound by driving the speaker with an audio signal. For example, the output device can be any type of headset, such as over-ear (or on-ear) headphones that at least partially cover the user's ears and are arranged to direct sound into the user's ears. On the other hand, the output device can be a wearable electronic device, such as smart glasses or a smartwatch.
[0022] In some respects, the output device may be a head-mounted device, as illustrated herein. In other respects, the output device may be any electronic device arranged to output sound to the surrounding environment. Examples may include standalone speakers, smart speakers, home theater systems, or infotainment systems integrated into vehicles. In another respect, the output device as a head-mounted device may be arranged to output sound to the surrounding environment. For example, when the output device is a pair of smart glasses, the output device may include “external” speakers arranged to project sound into the surrounding environment (e.g., in a direction away from at least a portion of the wearer, such as the ear or ear canal), in contrast to “internal” speakers of a pair of head-mounted receivers arranged to project sound into (or toward) the user’s ear canal when worn.
[0023] As described herein, an output device can be a wireless device that can be communicatively coupled to a playback device to exchange (e.g., audio) data. For example, a playback device can be configured to establish a wireless connection with an output device via a wireless communication protocol (e.g., Bluetooth or any other wireless communication protocol). During the established wireless connection, the playback device can exchange (e.g., transmit and receive) data packets (e.g., Internet Protocol (IP) packets) with the output device, which may include audio digital data of any audio format.
[0024] In one aspect, an output device may include electronic components for performing audio signal processing operations, such as one or more processors, memory, etc. In another aspect, an output device may not include one or more user controls for adjusting audio playback. For example, an output device may not include (e.g., physical) volume controls, such as adjustable knobs or (e.g., physical) buttons. In some aspects, an output device may not include any physical controls for configuring (or instructing) the device to perform one or more operations, such as adjusting the volume. In another aspect, an output device may include one or more controls (e.g., a power button), but may still not include (e.g., dedicated) controls for adjusting the volume level of the sound output at the output device. Therefore, an output device may be configured to perform context-dependent automatic volume compensation (AVC) to automatically adjust the volume level based on one or more criteria. For example, when a user moves from a quiet environment (e.g., a house) to a noisy environment (e.g., a busy intersection), the output device may adjust the volume level to compensate for the noise without requiring the user to manually adjust the volume controls (e.g., by turning up the volume). Further details regarding context-dependent AVC are described herein.
[0025] On the other hand, either or both of playback device 2 and output device 3 can be designed to receive user input. For example, when the playback device is a smartphone, the device may include a touch-sensitive display (not shown) arranged to receive user input when a user of the device touches the display (e.g., by tapping the screen with one or more fingers). On the other hand, the device may be designed to sense a user's voice command as user input. For example, the playback (and / or output) device may include one or more microphones arranged to sense speech (and ambient sound). The device may be configured to detect the presence of speech within the signals from one or more microphones. Once detected, the device can analyze the speech to determine whether the speech contains a voice command to perform one or more actions. Further details regarding output (and / or playback) devices that receive user input are described herein.
[0026] On the other hand, playback device 2 can be communicatively coupled to output device 3 via other methods. For example, both devices can be coupled via a wired connection. In this case, one end of the wired connection can be (e.g., fixedly) connected to the output device, while the other end can have a connector, such as a media jack or a Universal Serial Bus (USB) connector, that inserts into a jack on the playback device. Once connected, the playback device can be configured to drive one or more speakers on the output device via the wired connection using one or more audio signals. For example, the playback device can transmit the audio signals as digital audio (e.g., PCM digital audio). On the other hand, the audio can be transmitted in an analog format.
[0027] In some respects, playback device 2 and output device 3 may be different (separate) electronic devices, as illustrated herein. In other respects, playback device may be part of (or integrated with) output device. For example, at least some components of playback device (such as one or more processors, memory, etc.) may be part of output device, and / or at least some components of output device may be part of playback device. In this case, at least some of the operations performed by playback device (e.g., streaming audio content from audio content server 5) may be performed by output device.
[0028] Figure 2 A block diagram of an output device 3 performing context-dependent automatic AVC is shown. Specifically, the output device can perform context-dependent AVC to adapt the sound output of the output device based on the context of the output device (e.g., when using audio signal 21 to drive speaker 26). The output device is configured to automatically compensate for the volume level of the sound output (and / or perform one or more audio signal processing operations) based on context analysis of the output device (and / or the user of the output device). As described herein, such analysis may involve analyzing 1) the environment in which the output device is located, 2) a device snapshot of the output device (e.g., the device snapshot may indicate what software application is being executed, the activity of the user of the output device, etc.), and / or 3) the audio content being played back by the output device. Based on at least some of this analysis, the output device can determine the context of the output device. For example, the output device can determine that the device (and the user of the device) is in a quiet environment (e.g., based on analysis of one or more microphone signals captured by microphone 22), and therefore the output device can reduce the overall volume level. The volume level can be reduced because less sound output may be needed to mask ambient noise in a quiet environment. Therefore, context analysis allows output devices to optimize the listener experience by adapting the volume level and / or performing one or more audio signal processing operations (e.g., dynamic range compression) on one or more audio signals played back by the output device. This article describes more about output devices that perform context-dependent AVC.
[0029] In one aspect, the output device includes one or more sensors 31, including a microphone 22, a camera 23, an accelerometer 24, an inertial measurement unit (IMU) 25, a speaker 26, a controller 20, and a memory 36. In another aspect, the output device may include more or fewer elements, such as having multiple (e.g., two or more) microphones and / or speakers, or may not include one or more sensors, such as an IMU and / or an accelerometer.
[0030] Memory 36 can be any type of storage medium (e.g., non-transitory machine-readable), such as random access memory, CD-ROM, DVD, magnetic tape, optical data storage devices, flash memory devices, and phase-change memory. In one aspect, the memory can be a component of an output device (e.g., integrated into the output device). For example, the memory can be part of controller 20. In some aspects, the memory can be a separate device, such as a data storage device. In this case, the memory can be communicatively coupled to controller 20 (e.g., via a network interface) so that the controller performs one or more of the operations described herein.
[0031] As shown, the memory stores an operating system (OS) 38 and one or more software applications 37, which, when executed by the controller, cause the output device to perform one or more operations, as described herein. In one aspect, the memory may include more or fewer applications. The OS 38 is a software component responsible for the management and coordination of activities and the sharing of resources of the output device (e.g., controller resources, memory, etc.). In one aspect, the OS acts as a host for applications (e.g., application 37) running on the device. In some aspects, these applications may run on top of the OS. In one aspect, the OS provides an interface to the hardware layer (not shown) of the output device and may include one or more software drivers that communicate with the hardware layer. For example, these drivers may receive and process data packets received from one or more other devices (e.g., one or more sensors in sensor 31, etc.) communicatively coupled to the device via the hardware layer.
[0032] As described herein, the memory includes one or more software applications 37 that include instructions that, when executed by the controller 20 (e.g., one or more processors), cause the output device to perform one or more operations. For example, the output device may include a navigation application that retrieves route selection (navigation) instructions (e.g., from a remote server via network 4) and presents the route selection instructions (e.g., audible instructions via speaker 26) to a user of the output device. Other types of software applications may include alarm applications, navigation applications, map applications (for presenting maps and / or location information to a user), media (e.g., audio and / or video) playback applications, social media applications (e.g., applications providing a user interface for an online social media platform), exercise applications (e.g., applications that track a user's physical activity), healthcare applications (e.g., applications that set and track a user's health-oriented goals), and telephone applications (allowing users to make telephone calls via cellular networks such as 4G Long Term Evolution (LTE) networks of network 4).
[0033] Controller 20 may be a dedicated processor such as an application-specific integrated circuit (ASIC), a general-purpose microprocessor, a field-programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and dedicated state machines). The controller is configured to perform audio signal processing operations and / or networking operations. For example, controller 20 may perform context-dependent AVC operations to adjust the volume (or sound) level of the sound output from one or more speakers 26 of the output device. Further details regarding the operations performed by controller 20 are described herein.
[0034] In one aspect, one or more sensors 31 are configured to detect an environment (e.g., where the output device is located) and generate sensor data based on the environment. Microphone 22 can be any type of microphone (e.g., a differential pressure gradient microelectromechanical system (MEMS) microphone) configured to convert acoustic energy caused by sound waves propagating in the acoustic environment into a microphone signal. In one aspect, camera 23 is configured to capture image data (e.g., still digital images and / or video represented by a series of digital images). In some aspects, the camera is a complementary metal-oxide-semiconductor (CMOS) image sensor capable of capturing digital images including image data representing the camera's field of view, where the field of view includes a scene of the environment in which the output device is located. In some aspects, the camera can be a charge-coupled device (CCD) camera type. In one aspect, the camera can be positioned anywhere around the output device to capture one or more fields of view. In some aspects, the device can include multiple cameras (e.g., where each camera may have a different field of view).
[0035] Accelerometer 24 is arranged and configured to receive (detect or sense) speech vibrations generated when a user (e.g., a user who may be wearing an output device) speaks, and to generate an accelerometer signal representing (or including) the speech vibrations. Specifically, the accelerometer is configured to sense bone conduction vibrations transmitted from the user's vocal cords to the user's ear (ear canal) during speaking and / or humming. For example, when the output device is a wireless headset, the accelerometer may be located on or within the headset at any location that can contact a part of the user's body to sense vibrations caused by the user speaking. The IMU is designed to measure the position and / or orientation of the output device. For example, the IMU may generate sensor (or motion) data indicating changes in the orientation of the output device (e.g., with respect to any X, Y, Z axis) and / or changes in the position of the device. Thus, the IMU may generate motion data indicating the direction and velocity of the output device moving from one location (e.g., to another).
[0036] In one aspect, the output device may include an additional sensor 31. For example, the output device may include a thermistor (or temperature sensor) configured to detect (e.g., ambient) temperature as sensor data. In another aspect, the thermistor may be arranged to measure the internal temperature of the output device (e.g., the temperature of electronic components, such as a processor). As another example, the sensor may include a Global Positioning System (GPS) sensor that can generate location data indicating the position of the output device. In one aspect, based on the location data, the controller 20 may determine motion data indicating the direction and / or speed of movement of the output device.
[0037] The speaker 26 may be, for example, an electrically driven driver specifically designed for sound output in a particular frequency band, such as a woofer, tweeter, or midrange driver. In one aspect, the speaker may be a “full-range” (or “full-frequency”) electrically driven driver that reproduces as much of the audible frequency range as possible. In another aspect, when the output device includes two or more speakers, each speaker may be of the same type (e.g., all full-range), or one or more speakers may differ from the others, such as one being a woofer and another a tweeter. In some aspects, the speaker 26 may be an internal speaker or an external speaker, as described herein.
[0038] In one aspect, any element described herein may be part of (or integrated into) an output device (e.g., integrated into the housing of the output device). In another aspect, at least some of the elements may be one or more separate electronic devices (e.g., a portion thereof) communicatively coupled to the output device (e.g., via Bluetooth connection) via a network interface of the output device and a controller. For example, a speaker may be integrated into the output device, while one or more sensors in sensor 31 may be integrated into another device such as playback device 2. In this case, the playback device may transmit sensor data to the output device, as described herein. In another aspect, the controller and one or more sensors may be integrated into another device. In this case, the other device may perform one or more audio signal processing operations (e.g., context-dependent AVC operations, as described herein) to generate one or more audio signals. Once generated, the signals may be transmitted via speaker 26 to the output device for playback.
[0039] As described herein, controller 20 is configured to perform audio signal processing operations, such as context-dependent AVC. In one aspect, these operations can be performed while the controller is playing back sound. For example, controller 20 is configured to receive audio signal 21 (which may include audio content desired by the user, such as musical works, podcasts, etc.) and use this signal to drive speaker 26. To perform context-dependent AVC operations, the controller includes several operation blocks. As shown, the controller includes a device snapshot detector 28, a context engine and decision logic (or context engine) 29, a volume compensation model database 27, and a volume compensator 30.
[0040] Device snapshot detector 28 is configured to determine a device snapshot of the output device. Specifically, the snapshot may include the current state of one or more software applications that the electronic device is executing (and / or is not currently executing). In one aspect, the current state may include an indication of whether one or more software applications (which of the one or more software applications) with instructions stored in the memory of the output device are currently being executed by the electronic device (e.g., one or more programmed processors of the electronic device) (e.g., where the software applications perform one or more digital signal operations). For example, when the software application is a navigation application, the current state of the application may indicate that the application is active (e.g., running in the foreground) and that the application (e.g., retrieving route selection instructions from a remote server via network 4) and presenting the route selection instructions (e.g., presenting audible instructions to the user via speaker 26). In another aspect, the current state may indicate whether the application is executing in the background (e.g., unlike when the application is running in the foreground, when the application is running in the background, any activity / operation of the application is currently invisible to or does not attract the attention of the user of the output device) or running in the foreground, as described with respect to a navigation application.
[0041] In one aspect, a snapshot may include data associated with a software application stored and / or executed by the output device. Specifically, a snapshot may indicate the type of software application being executed (e.g., whether the software application is an alarm application or a navigation application). A snapshot may also indicate whether any application is playing back sound. As described herein, when the controller drives the speaker 26 with audio signal 21, the controller may perform context-dependent AVC. In one aspect, a snapshot may indicate whether the audio signal is associated with one or more software applications. For example, a snapshot may indicate that audio signal 21 is associated with (or played back by) an audio playback software application executing on the output device (e.g., where the user of the device opens the application and requests playback of audio content).
[0042] On the other hand, snapshots can include data about the amount of resources (on the output device) used by each application during execution. For example, resources could indicate the amount of memory and (e.g., processing resources of one or more processors) on the output device. Data could indicate how long the software application has been running since it was activated (e.g., opened by a user of the output device).
[0043] In some aspects, a snapshot may include historical data from one or more software applications on the output device. For example, historical data may indicate how frequently a software application is opened and closed by a user of the device (e.g., over a period of time), or how long a software application runs once opened (or activated) by a user (e.g., an average over a period of time). Historical data may indicate the average amount of resources used by a software application during that period. In other aspects, a snapshot may include historical data determined by one or more software applications. For example, a snapshot may include healthcare-related data (e.g., a user's sleep schedule, meal times, etc.). In some aspects, historical data may include any information about one or more software applications on the output device. In some aspects, a device snapshot may include data such as which software applications (e.g., relative to other software applications) are regularly executed by the output device. In other aspects, a device snapshot may indicate which software applications require more (e.g., above a threshold) device resources than other applications. In other aspects, a device snapshot may include any type of historical data regarding one or more software applications.
[0044] On the other hand, snapshots can indicate whether (and how) a user is interacting with a software application. For example, detector 28 can make this determination based on receiving user input 32 (e.g., while the software application is executing). In one aspect, user input can be received at an output device. For example, user input can be a voice command captured by microphone 22, which includes instructions for the software application (e.g., a request for navigation instructions from a navigation application being executed by the output device). On the other hand, user input can be received via one or more input devices communicatively coupled to the output device (e.g., a portion of the output device), such as physical control buttons or a touch-sensitive display (not shown) showing a graphical user interface (GUI) of the software application. For example, user input can indicate (e.g., based on a tap on the screen) a selection of one or more UI items displayed on the screen. On the other hand, detectors can receive user input via other methods.
[0045] On the other hand, a snapshot can indicate whether a software application is (e.g., currently) presenting data to a user. As described herein, a snapshot can indicate whether a particular software application is running in the background or in the foreground. Therefore, a snapshot can indicate what information (data) of the software application is being presented (or output) to the user when the application is in the foreground. For example, when an output device is communicatively coupled to a display (not shown), a snapshot can indicate whether the display is showing the GUI of the software application. On the other hand, a snapshot can indicate whether the software application is playing back one or more audio signals associated with the application via one or more speakers 26. Specifically, a snapshot can indicate whether the audio content of the audio signal 21 that the output device is (or will) play back is associated with the software application. For example, a snapshot can indicate that the audio signal includes audio content of the software application (e.g., when the software application is an alarm application, a snapshot can indicate that the audio content is a ringing tone to be played back).
[0046] On the other hand, a snapshot may include data (or information) related to the media content (such as audio content and / or video content) being played back by the output device. For example, when an audio playback software application being executed by the output device drives the speaker 26 with the audio signal 21, the snapshot may include metadata related to the audio content contained in the audio signal (e.g., when the audio content is a song, the metadata may include the song title, the singer of the song, the genre of the song, the duration of the song, etc.).
[0047] As described above, a snapshot can indicate whether user input 32 is received at the output device and / or whether a software application (e.g., via the output device) presents data to the user. On the other hand, a snapshot can include information related to one or more software applications from an electronic device (e.g., playback device 2), which is communicatively coupled to the output device and (at least partially) executes one or more software applications. For example, the playback device may include memory arranged to store one or more software applications (e.g., such as application 37) and may include one or more processors arranged to execute the applications. In some aspects, applications being executed by two devices can be configured to interact with each other (e.g., exchange data) via wired and / or wireless networks. For example, the playback device may be executing a software application (which may be executed by the output device) such as a navigation application and may receive user input (e.g., a user tap on the touch-sensitive display of the playback device displaying the graphical user interface (GUI) of the navigation application) to perform navigation operations. In response, the playback device can transmit user input (e.g., as one or more instructions) to the output device (e.g., its device snapshot detector) to indicate user interaction (e.g., a user request for direction). Alternatively, the snapshot detector can receive data from the playback device indicating whether the device is presenting data from a software application, such as whether a navigation application running on the playback device is displaying navigation instructions via the playback device's screen.
[0048] The volume compensation model database 27 includes one or more volume compensation models, each with one or more audio adjustment parameters. The volume compensator 30 can use these one or more audio adjustment parameters to process one or more audio signals (e.g., audio signal 21) for playback by the speaker 26. In some aspects, the database 37 may (e.g., at least partially) be stored in memory 36 and / or controller 30, as shown. In one aspect, the database may store a table (e.g., as a data structure) including one or more volume compensation models, each associated with (or having) one or more audio adjustment parameters. Figure 3An example of such a data structure 35 stored within database 27 is shown. Specifically, the data structure is a table of one or more volume compensation models and one or more associated audio adjustment parameters. As shown, the data structure includes two models (a first model and a second model), but as described herein, it may include more (or fewer) models. Each model within the data structure includes one or more audio adjustment parameters. For example, both the first and second models include scalar gain values (V1, V2), which the volume compensator 30 can apply to one or more audio signals to attenuate (or increase) the signal level of the applied signal. Each model is also associated with a compressor type of the volume compensator to reduce the dynamic range of the applied audio signal. In one aspect, the database may have one or more different compressor types. For example, the first model includes a wideband compressor that compresses the entire (e.g., audible) frequency range of the audio signal when applied by the volume compressor (e.g., the audio signal may have a frequency range between 20 Hz and 20 kHz). The second model includes a multiband compressor that, when applied, compresses a subset of one or more frequency bands of the entire frequency range of the audio signal. For example, a multi-band compressor can compress only low-frequency content. On the other hand, a multi-band compressor can compress different frequency bands differently. For example, a multi-band compressor can compress low-frequency content (e.g., frequency content below a first threshold), intermediate-range frequency content (e.g., frequency content between the first threshold and a second threshold above the first threshold), and high-frequency content (e.g., frequency content above the second threshold) differently from each other.
[0049] The model also includes compression ratios (R1, R2), each specifying the amount of attenuation the compressor will apply to one or more signals. Additionally, the model includes a startup time (T). A1 T A2 ) and release time (T) R1 T R2 The start-up time indicates the amount of time it takes for one or more audio signals to become fully compressed, and the release time indicates the amount of time it takes to release (or remove) the compression from the signal. Therefore, when the volume compensator 30 applies the first model to the audio signal 21, the compensator will apply a wideband compressor with a compression ratio of R1, having a start-up time T for applying the compressor 9. A1 and the release time T for removing the broadband compressor R1 .
[0050] In one aspect, the model may include one or more additional audio conditioning parameters. For example, the parameters may include one or more thresholds (e.g., in dB) that the volume compensator uses to determine whether to engage a particular compressor. In another aspect, the model may include one or more audio filters, such as low-pass, band-pass, and high-pass filters. In yet another aspect, one or more models may include a limiter configured to restrict levels below a threshold (e.g., a maximum) level. In some aspects, the model may include a spatial filter that allows the volume compensator to spatially render the audio signal. For example, the spatial filter may include one or more head-related transfer functions (HRTFs), or equivalently, one or more head-related impulse responses (HRIRs), which, when applied to one or more audio signals, can produce spatial audio (e.g., binaural rendered audio signals).
[0051] On the other hand, one or more models may include multiple audio conditioning parameters. For example, a second model may include one or more compression ratios, each applied to a different set of one or more frequency bands when the multi-band compressor compresses the audio signal. On the other hand, one or more models (e.g., compared to other models) may include fewer audio conditioning parameters. For example, a model may not include scalar gain values, but only compressor parameters (e.g., compressor type, ratio, and start / release time).
[0052] In one aspect, the volume compensation model can be a predefined model, which may have already been defined in a controlled environment (e.g., in a laboratory). In another aspect, at least some of the models in the model can be user-defined (e.g., based on user input received by the output device). In some aspects, the volume compensation model can be derived based on user preferences and / or based on the model selection of the context engine 29 (e.g., over time). This paper describes further details regarding deriving models based on the selection of the context engine.
[0053] In one aspect, volume compensation models (e.g., stored in database 27) can be associated with one or more contexts of the output device. Specifically, each model can be configured to compensate (or adapt) the sound output of the output device according to a specific context (or scenario). In one aspect, the model can be configured to optimize the audio content of the audio signal to be compensated by volume compensator 30. For example, a multi-band compressor may be a preferred type of compressor when the audio content of the audio signal to be compressed contains speech in order to improve intelligibility. Thus, a second model can be configured to optimally adapt the sound output of an audio signal that includes speech (e.g., a podcast). A wideband compensator may be optimal for audio content that does not include speech (or not only speech, such as musical works). Thus, a first model can be configured to optimally adapt the sound output of an audio signal that includes musical works. In another aspect, the model can be associated with specific environmental conditions. For example, a first model can be associated with an output device in a noisy environment (e.g., in an environment where the noise level is above a threshold), and therefore the scalar gain value can be high (e.g., above the gain threshold). Conversely, the second model can be associated with an output device in a quiet environment (e.g., a noise level below a threshold), and therefore the scalar gain value can be low (e.g., below the gain threshold).
[0054] On the other hand, for example, a model may be configured to compensate for sound output based on a determined context of the output device, such as the activity being performed by the user of the output device. For example, data structure 35 may include a model configured to optimize sound output while the user of the output device is cycling and listening to music (e.g., where the model includes gain values to increase the sound level of the sound output to compensate for wind noise). Further details of models configured to compensate for sound output based on a determined context of the output device are described herein.
[0055] Context engine 29 is configured to determine the context of the output device, using which the engine determines (or selects) one or more volume compensation models to adapt the sound output (e.g., volume) of the output device. Further details regarding the use of volume compensation models to adapt sound output are described herein. In one aspect, the “context” of the output device can be the state of the device (e.g., operational state, physical state, etc.) and / or the activities or tendencies of the user of the device. For example, the context engine can perform internal analysis of the output device and / or external analysis of the environment and / or state (or activities) of the user of the device (e.g., based on sensor data, device snapshots of the output device, etc.), and use at least some of this information to determine the overall context of the device.
[0056] In one aspect, the context engine can analyze the environment in which the output device is located to determine details (or information) about the environment (details that may indicate whether the volume level of the sound output should be adjusted). In another aspect, the context engine 29 can use sensor data obtained from one or more sensors 31 to analyze the environment in which the output device is located. For example, the context engine can determine the location of the output device (e.g., within the environment). To do this, the context engine can receive GPS sensor data indicating the (e.g., precise) location of the output device. In another aspect, the context engine can determine the location of the output device based on one or more sensors. For example, the context engine can use image data captured by camera 23 to perform object recognition algorithms to identify the location of the output device (e.g., identifying a pedestrian crossing and moving cars indicating that the user and the output device are at a busy (and noisy) intersection). As another example, when identifying trees and benches, the context engine can determine that the output device is in a park, which may typically be quiet. In yet another aspect, the context engine can determine the location based on a snapshot of the device determined by detector 28 (and received from that detector). For example, the snapshot may indicate that a navigation application is being executed and the location of the output device along the navigation route currently being presented to the user. On the other hand, the context engine can determine location based on historical data (e.g., from sensor 31). For example, the context engine can determine that the output device was in a specific location at a specific time based on historical data (e.g., trends or patterns) that indicate (e.g., for a threshold number of days, etc.) in the past (approximately) a specific location. For example, historical location data could indicate that the user and the output device were dining at a restaurant around 6 p.m. On the other hand, along with (or instead of) location identification, the context engine can identify objects within the location. As described herein, using image data captured by a camera, the context engine can determine which objects are in the environment.
[0057] As described herein, details about the environment can indicate whether the volume level of the sound output should be adjusted. Specifically, the presence of ambient noise can be determined based on at least some sensor data. For example, a context engine can determine the ambient noise level within the environment based on detected activities and / or objects. Returning to the previous example at a busy intersection, the context engine can determine that the output device is in a noisy environment based on an estimate of the noise generated by identified objects within the environment. Therefore, the context engine can determine (or estimate) the ambient noise level based on an estimate of the noise caused by identified moving cars, identified fire trucks with their lights on, etc.
[0058] In some aspects, the context engine 29 can determine whether the environment in which the output device is located includes ambient noise, and can determine the noise level of the noise. For example, the context engine can obtain one or more microphone signals from microphone 22, and can process the microphone signals to determine the noise level of the ambient noise contained therein. In one aspect, the noise level can indicate how much spectral content the ambient noise has across one or more frequency bands. For example, the level can indicate that the ambient noise includes more low-frequency spectral content than high-frequency spectral content (e.g., above a threshold). In another aspect, the context engine can determine the type of ambient noise contained in the environment. For example, the context engine can analyze the ambient noise to identify the type of noise, such as whether the noise includes musical works, and / or whether the noise includes speech (e.g., by performing a speech activity detection (VAD) algorithm on the microphone signals).
[0059] On the other hand, the context engine 29 can use sensor data to determine whether the output device is stationary or moving within the environment. For example, the context engine can determine movement based on motion data received from the IMU 25. Alternatively, the context engine can determine that the output device is moving based on GPS sensor data and / or on changes in the environment (e.g., changes in objects within image data captured by the camera 23).
[0060] In some respects, the context engine can analyze the audio signal 21 to determine the audio content it contains. For example, the context engine 29 can receive the audio signal 21 that the controller 20 can use to drive the speaker 26, and determine the type of audio content to be played back by the output device (e.g., currently or to be played back) based on the analysis of the audio content. Specifically, the engine can perform a VAD operation to determine whether the audio content contains speech. In another respect, the engine can perform spectral analysis on the audio signal to determine the audio content contained therein, such as whether the audio content is a musical work, and the spectral content of that work (e.g., having spectral content lower than high spectral content, etc.). In yet another respect, the context engine can use a device snapshot to determine information related to the audio signal, as described herein.
[0061] In one aspect, the context engine can be configured to determine whether a user of an output device is performing a physical activity (e.g., when the output device is part of or connected to the user). Specifically, the context engine can determine the activity based on user input. For example, using a device snapshot received from device snapshot detector 28, the context engine can determine whether one or more software applications associated with the physical activity are being used. For instance, if it is determined that a workout software application has been activated (or opened) by the user and the user has (e.g., via user input 32) requested the application to track a workout (e.g., running), the context engine can determine that the user is jogging outdoors. In another aspect, the context engine can use entries within a calendar software application to determine that the user is performing a specific activity at a specific location (e.g., exercising in a noisy gym) (this indicates that the user is exercising at a specific time during a specific day of the week).
[0062] As described herein, a context engine can determine whether a user is performing a physical activity based on user input. On the other hand, a context engine can determine whether a user is active based on the analysis of sensor data and / or device snapshots. For example, a context engine can determine that a user is driving a car based on navigation information within a device snapshot and / or based on location / motion data. As another example, a context engine can determine that a user is dining based on location data (e.g., obtained from GPS sensors, map / navigation software applications, etc.) indicating that the user is at a specific restaurant. Along with location data, a context engine can determine that a user is dining based on image data captured by a camera (e.g., this image data may include objects such as plates, forks, glasses, etc.).
[0063] On the other hand, the context engine can determine whether a user is performing other activities, such as talking to another person. For example, the context engine can determine whether a user is making a phone call based on data obtained from a phone application. On the other hand, the context engine can determine whether a user is having a conversation based on sensor data. For example, the context engine can determine whether a user is speaking based on whether the accelerometer signal generated by accelerometer 24 is above a threshold. Furthermore, the context engine can determine whether another person is within the field of view of camera 23 and whether this person has facial features indicating that they are speaking (e.g., whether their lips are moving).
[0064] On the other hand, the context engine can determine whether a user is performing an activity based on historical data (e.g., obtained from device snapshots). Specifically, the context engine can determine that a user is performing a specific activity based on one or more patterns within the historical data (e.g., recurring patterns). For example, the context engine can determine that a user is at home during those times based on location data received in the past by the output device indicating that the user is typically at home between 6 p.m. and 9 p.m.
[0065] In one aspect, the context engine 29 may determine the (e.g., overall) context of the output device based on one or more determinations described herein. For example, the context engine may determine the context as the user (and output device) walks toward a busy intersection on a sidewalk, based on location data, user activity (e.g., walking direction received via a navigation application), and noise levels. Thus, one or more determinations by the context engine may indicate the context of the user and / or the output device. In one aspect, when determining the context, the context engine may be configured to select one or more volume compensation models associated with the context from the database 27. Further details regarding the selection of one or more models are described herein.
[0066] In one respect, the context of the determined output device can indicate how the sound output should be adjusted based on the estimated (determined or assumed) ambient noise within the environment. Returning to the previous example, when determining that a user is exercising in a noisy gym or at a busy intersection, the context can indicate the presence of significant ambient noise (e.g., above a threshold). Conversely, when determining that a user is having dinner in a park or at home, the context can indicate the presence of very little (below a threshold) ambient noise. In another respect, the context can indicate what spectral content is also present in the environment in which the device is located. For example, when determining that a user is near a fire truck with its lights and sirens on, the context can indicate that the environment has an increased amount (e.g., above a threshold value) of intermediate-range frequency content (e.g., between 500 Hz and 1500 Hz).
[0067] Context engine 29 is configured to determine (or select) one or more volume compensation models from volume compensation model database 27 based on the context of the determined output device. As described herein, a volume compensation model can be associated with one or more contexts of the output device. In this case, the context engine can use the determined context to perform a table lookup of data structure 35 to select one or more volume compensation models associated with the determined context. When a model with a matching context is found, the context engine can select that model. In one aspect, one or more models can be specific to the particular environment in which the output device is located. For example, when the context indicates that a fire truck is nearby, the model can minimize the spectral impact of the siren's sound. As another example, the model can be optimized for user activities, such as having audio conditioning parameters that minimize the user's perception of wind noise while cycling.
[0068] On the other hand, the context engine can select one or more audio adjustment parameters from one or more volume compensation models based on the determined context. Therefore, the context engine can mix and match audio adjustment parameters from various compensation models to create (or build) an optimized volume compensation model for the determined context.
[0069] The volume compensator 30 is configured to receive the audio signal 21 and one or more selected volume compensation models from the context engine 29, and is configured to process the audio signal according to the selected volume compensation model (e.g., adapt the audio signal to the sound output). For example, the model may indicate a specific gain value to be applied to the audio signal (e.g., to increase the signal level of the audio signal due to the context of the output device in a noisy environment). Thus, the compensator may apply a scalar gain to increase the level of the audio signal, and the processed audio signal may be used to drive the speaker 26.
[0070] In one aspect, the volume compensator can process an audio signal based on a selected volume compensation model and a microphone signal. Specifically, the volume compensator can (optionally) acquire the microphone signal and can use the microphone signal to apply the volume compensation model to the audio signal. For example, when the ambient noise level exceeds a threshold, the volume compensator can process the audio signal according to the model. Conversely, when the ambient noise level drops below the threshold, the compensator may not process (or may partially process) the audio signal. For example, when the ambient noise level drops below the threshold, the volume compensator can adjust the compression ratio and / or scalar gain value (e.g., because the environment is quiet), but can maintain the start / release time. As another example, the volume compensator can measure the background noise level and then dynamically adjust the input gain on the limiter (or compressor) of the volume compensation model. Alternatively, the volume compensator can (e.g., based on the measured background noise level) adjust the threshold and gain on a multi-band compressor.
[0071] As described herein, volume compensation models can be predefined (or created) in a controlled environment. In some aspects, volume compensation models can be determined (or defined) over a period of time based on the user's listening patterns at the output device. Specifically, controller 20 can create a volume compensation model based on the user's adjustment of the volume level of the sound output determined by the output device within a specific context. For example, the context engine can determine (e.g., based on sensor data) that the user is performing a physical activity, such as running outdoors. The context engine can also determine that the output device has received user input to increase the volume level (e.g., via a voice command captured by microphone 22). Therefore, the context engine can create a volume compensation model with a scalar gain value to increase the sound output. Furthermore, the context engine can derive audio conditioning parameters based on sensor data. For example, when a user is running, the microphone can capture a significant amount (e.g., above a threshold) of wind noise. Therefore, the context engine can select one or more audio conditioning parameters of the compressor of the optimized model to reduce the impact of wind noise on the sound output.
[0072] In one aspect, controller 20 may be configured to perform (additional) audio signal processing operations based on elements coupled to the controller. For example, when the output device includes two or more “outside-the-ear” speakers arranged to output sound to an acoustic environment rather than speakers arranged to output sound to a user’s ears (e.g., as speakers in an in-ear headset), the controller may include an audio output beamformer configured to generate speaker driver signals that produce spatially selective sound output when driving two or more speakers. Thus, when used to drive speakers, the output device may generate a directional beam pattern that can be directed to a location within the environment.
[0073] In some aspects, controller 20 may include a sound pickup beamformer configured to process audio (or microphone) signals generated by two or more external microphones of the output device to form a directional beam pattern (as one or more audio signals) for spatially selective sound pickup in certain directions, thereby increasing sensitivity to the location of one or more sound sources. In some aspects, the controller may perform audio processing operations (e.g., perform spectral shaping) on the audio signals containing the directional beam pattern.
[0074] In one aspect, context-dependent AVC operations can be performed by (or in combination with) an audio playback software application executed by the output device. For example, the playback application can be configured to drive speaker 26 with audio signal 21. In another aspect, the playback application can play back the audio signal in response to user input (e.g., the application uses a microphone signal to detect a voice command to play back a musical piece). Therefore, when playing back the audio signal, the playback application can execute AVC operations of the operation block of controller 20, as described herein, to adapt the sound output according to the context of the output device (e.g., environment, user activity, audio content, etc.).
[0075] Figure 4 and Figure 5 These are flowcharts that respectively include processes 40 and 50, which can be executed by output device 3 (e.g., its controller 20). On the other hand, at least some of the operations in the operation can be executed by one or more software applications (e.g., audio playback software applications) being executed by the device (e.g., the device's controller).
[0076] Figure 4 This is a flowchart of process 40 for performing context-dependent AVC according to one aspect. The process begins when the controller receives (or receives) audio signals including audio content such as music, podcasts, etc. (e.g., signal 21, such as...). Figure 2 (as shown in box 41). The controller uses one or more microphones to acquire microphone signals including audio (e.g., ambient noise) of the environment in which the electronic device is located (in box 42). The controller determines the context of the output device (in box 43). For example, when the user of the device is running, the context engine 29 may determine that the output device is at a noisy intersection. Or, when the user of the device is reading a book, the context engine may determine that the output device is in a quiet room. This determination may be based on sensor data from one or more sensors 31 and / or on a determined snapshot of the device. Figure 5 More details are described regarding determining the context.
[0077] Controller 20 selects a volume compensation model from a plurality of volume compensation models (e.g., stored in data structure 35 within database 27) based on a determined context (at box 44). Specifically, as described herein, the controller determines one or more audio adjustment parameters of the volume compensator based on sensor data from one or more sensors 31, device snapshots, and / or audio content of the audio signal. As described herein, each (or at least some) of the models may be associated with one or more contexts. Thus, context engine 29 may perform a lookup of tables in data structure 35 to select the model associated with the determined context. The controller processes the audio signal according to the selected volume compensation model and the microphone signal (at box 45). Specifically, the controller uses volume compensator 30 to process the audio signal according to one or more audio adjustment parameters of the volume compensation model. In one aspect, the volume compensator may use the microphone signal to determine how to apply the volume compensation model to the audio signal. For example, the volume compensator may adjust (or apply) one or more audio adjustment parameters based on the audio noise level of the noise contained in the microphone signal. In particular, when the noise level changes (e.g., along with the spectral content contained therein), the compensator may adjust the compression ratio of the model's associated compressor. Therefore, the compensator can adjust the dynamic range of the audio signal according to the ambient noise level. The controller uses the processed audio signal to drive one or more speakers of the output device (at box 46).
[0078] Variations of process 40 can be performed in some aspects. For example, a particular operation may not be performed in the exact order shown and described. The particular operation may not be performed in a consecutive series of operations, and different particular operations may be performed in different aspects. For example, an output device may use the acquired audio signal to drive one or more speakers while at least some of the operations are performed by a controller. Specifically, once the audio signal is acquired, the controller may perform operations in blocks 42 to 46 (at least some of the blocks), while the output device uses the audio signal to drive the signal. Once the signal is processed, at block 45, the controller may use the processed signal to drive the speaker, as described herein.
[0079] As described herein, the controller receives audio signal 21 and processes the audio signal according to a selected model. Alternatively, the controller may receive multiple (one or more) audio signals. For example, the controller may receive one audio signal associated with an audio playback application (e.g., containing musical pieces) and another audio signal associated with a navigation application (e.g., containing verbal navigation instructions). In this case, the controller may process the audio signals differently based on the determined context. For example, the controller may determine that the user of the output device is interacting with the audio playback application (e.g., searching for new musical pieces for playback). Therefore, the controller may determine that the user is more interested in the audio content of the audio playback application than the audio content of the navigation application. In response, the controller may select a different volume compensation model for each audio signal, whereby the volume compensator processes each audio signal according to its associated model. Once processed, the volume compensator can (e.g., by performing a matrix mixing operation) mix the audio signals for playback. Thus, in this example, the audio content of the audio playback application may have a higher volume level than the audio content of the navigation application. On the other hand, instead of choosing a different model for each signal, the volume compensator can process the signal differently based on a model (e.g., by performing some audio signal processing operations on one signal instead of another).
[0080] On the other hand, the controller can perform at least some of these operations while using audio signals to drive the speaker (e.g., continuously). Therefore, the controller can continuously determine whether the context of the output device has changed. For example, the controller can execute process 40 to determine that the context of the output device is that the user is running outdoors. In response, the controller can select a volume compensation model (or one or more adjustment parameters) and process the audio signal according to the model. The controller can continuously monitor data (e.g., sensor data, device snapshot data, etc.) to determine whether the context has changed. Continuing with the previous example, the controller can determine that the user is no longer running outdoors based on sensor data (e.g., a decrease in IMU data) and on device snapshots (e.g., a workout software application indicating that the user has completed an outdoor running workout, etc.). Furthermore, the controller can (e.g., based on the data described herein) determine that the user is sitting in a quiet room. As a result of determining a change in context (or determining a new context), the controller can perform at least some of the operations of process 40 according to the changed context. For example, the controller can select a different volume compensation model (e.g., different audio adjustment parameters) based on the changed context. For example, since the user is sitting in a quiet room, the applied scalar gain value can be reduced. The controller can then process the audio signal according to different models (and microphone signals).
[0081] Figure 5 This is a flowchart of a process 50 for determining the context of an output device, based on one aspect. Specifically, the operations described in this process can be performed by a controller 20 of the output device. The process begins with the controller 20 detecting signals from one or more sensors (e.g., Figure 2 Sensor 31 receives sensor data, the one or more sensors being arranged to sense the condition of the environment in which the output device is located (at box 51). The controller determines a device snapshot (at box 52) including the current state of each of one or more software applications being executed by the output device. For example, the device snapshot may include the current state of one or more software applications (e.g., one or more operations being performed), which may include a snapshot of a playback software application (which may be performing one or more context-dependent AVC operations, as described herein). The controller determines the context of the output device based on the device snapshot, the audio content of the acquired audio signal, and / or sensor data (at box 53). For example, the context of the device may be that the user of the device is jogging outdoors, based on a workout application running on the device and based on location (e.g., GPS) data.
[0082] Variations of process 50 can be performed in some aspects. For example, a particular operation may not be performed in the exact order shown and described. The particular operation may not be performed in a consecutive series of operations, and different particular operations may be performed in different aspects. In one aspect, the context may be determined based on less data (e.g., only based on a device snapshot). For example, the context engine may determine (e.g., within a defined range) that the user is eating dinner based on previously determined user eating patterns.
[0083] As described above, the output device can be configured to perform context-dependent AVC operations to adjust the volume level of the sound output. In one aspect, the output device can perform such operations when the user of the device cannot manually adjust the volume level. Specifically, the output device may not include (e.g., hardware) a volume control arranged to adjust the sound output level of one or more speakers of the output device. Therefore, the output device can dynamically and automatically compensate for the volume level based on the context of the output device, maintaining an optimal user experience for the listener regardless of the context in which the user and device are located.
[0084] According to one aspect of this disclosure, an electronic device includes a processor and a memory having instructions that, when executed by the processor, cause the electronic device to acquire an audio signal including audio content; acquire sensor data from one or more sensors arranged to sense the condition of the environment in which the electronic device is located; determine a device snapshot including the current state of each of one or more software applications being executed by the electronic device, wherein the one or more software applications being executed include an audio playback software application; determine at least one audio adjustment parameter of a volume compensator based on the sensor data, the snapshot of the one or more software applications, and the audio content of the audio signal; process the audio signal using the volume compensator according to the determined audio adjustment parameter; and drive one or more speakers using the processed audio signal.
[0085] As is widely recognized, the use of personally identifiable information should comply with privacy policies and practices that are generally accepted to meet or exceed industry or governmental requirements for protecting user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly explained to users.
[0086] As previously described, one aspect of this disclosure may be a non-transitory machine-readable medium (such as a microelectronic memory) storing instructions thereon that program one or more data processing units (generally referred to herein as a "processor") to perform network operations, context-dependent AVC operations, and (other) audio signal processing operations, as described herein. In other aspects, some of these operations may be performed by specific hardware components containing hard-wired logic. Alternatively, those operations may be performed by any combination of programmed data processing units and fixed hard-wired circuit components.
[0087] In one aspect, the context of the electronic device is based on sensor data from one or more sensors of the electronic device, including a Global Positioning System (GPS) sensor, a camera, a microphone, a thermistor, an inertial measurement unit (IMU), and an accelerometer. In some aspects, the context of the electronic device is the location of the electronic device. In another aspect, the device determines changes to the context of the electronic device; selects a different volume compensation model from a plurality of volume compensation models based on the changes to the context; and processes an audio signal according to the selected different volume compensation model and a microphone signal. In some aspects, each volume compensation model includes at least one of the following: 1) one or more scalar gain values to be applied to the audio signal, 2) a wideband compressor or a multiband compressor, 3) a compression ratio, 4) a start-up time for applying the compression ratio to the wideband compressor or multiband compressor, and 5) a release time for removing the compression ratio from the wideband compressor or multiband compressor. In another aspect, processing the audio signal according to the selected volume compensation model and a microphone signal includes using the selected volume compensation model to audio-compensate the audio signal for the environment. In one aspect, the electronic device is a portable device. In another aspect, the electronic device is a wearable device. In some respects, wearable devices are a pair of smart glasses or a smartwatch.
[0088] While certain aspects have been described and illustrated in the accompanying drawings, it should be understood that such aspects are merely illustrative of the broad disclosure and not limiting, and that this disclosure is not limited to the specific structures and arrangements shown and described, as various other modifications will be apparent to those skilled in the art. Therefore, the description is to be regarded as exemplary and not restrictive.
[0089] In some aspects, this disclosure may include languages such as "[element A] and [element B]". This language may refer to one or more of these elements. For example, "at least one of A and B" may refer to "A", "B", or "A and B". Specifically, "at least one of A and B" may refer to "at least one of A and at least one of B" or "at least either A or B". In some aspects, this disclosure may include languages such as "[element A], [element B] and / or [element C]". This language may refer to any of these elements or any combination thereof. For example, "A, B and / or C" may refer to "A", "B", "C", "A and B", "A and C", "B and C", or "A, B and C".
Claims
1. A method executed by one or more programmed processors of an electronic device, the method comprising: Obtain audio signals; A microphone signal, including audio of the environment in which the electronic device is located, is obtained using one or more microphones; The context of the electronic device is determined based on the audio content of the audio signal; Based on the determined context, a volume compensation model is selected from multiple volume compensation models, where: When the audio content does not include speech, the selected volume compensation model includes a wideband compressor for compressing the entire frequency range of the audio signal, and When the audio content includes speech, the selected volume compensation model includes a multi-band compressor for compressing a subset of one or more frequency bands across the entire frequency range of the audio signal. The audio signal is processed according to the selected volume compensation model and the microphone signal; as well as The processed audio signal is used to drive one or more speakers of the electronic device.
2. The method of claim 1, wherein the context of the electronic device includes an indication that the programmed processor of the electronic device is executing one or more software applications.
3. The method of claim 2, wherein the audio signal is associated with a software application in one or more software applications that is being used by a user of the electronic device.
4. The method of claim 1, wherein the context of the electronic device includes the activities of the user of the electronic device.
5. The method of claim 4, wherein the user's activities include at least one of: interactions between the user and the electronic device, and physical activities performed by the user when the electronic device is part of or connected to the user.
6. The method of claim 1, wherein the one or more speakers are integrated within the electronic device, wherein the electronic device does not include hardware volume controls arranged to adjust the sound output level of the one or more speakers of the electronic device.
7. An electronic device, comprising: One or more microphones; One or more speakers; One or more processors; as well as A memory, wherein the memory stores instructions that, when executed by the one or more processors, cause the electronic device to: Obtain audio signal, The microphone signal, including audio of the environment in which the electronic device is located, is obtained using the one or more microphones. The context of the electronic device is determined based on the audio content of the audio signal. Based on the determined context, a volume compensation model is selected from multiple volume compensation models, where: When the audio content does not include speech, the selected volume compensation model includes a wideband compressor for compressing the entire frequency range of the audio signal, and When the audio content includes speech, the selected volume compensation model includes a multi-band compressor for compressing a subset of one or more frequency bands across the entire frequency range of the audio signal. The audio signal is processed according to the selected volume compensation model and the microphone signal, and The processed audio signal is used to drive the one or more speakers.
8. The electronic device of claim 7, wherein the context of the electronic device includes an indication that the electronic device is executing one or more software applications.
9. The electronic device of claim 8, wherein the audio signal is associated with a software application in one or more software applications that is being interacted with by a user of the electronic device.
10. The electronic device of claim 7, wherein the context of the electronic device includes the activities of the user of the electronic device.
11. The electronic device of claim 10, wherein the user's activity includes at least one of: interaction between the user and the electronic device, and physical activities performed by the user when the electronic device is part of or connected to the user.
12. The electronic device of claim 7, wherein the one or more speakers are integrated within the electronic device, and wherein the electronic device does not include hardware volume controls arranged to adjust the sound output level of the one or more speakers of the electronic device.
13. A method executed by an audio playback software application, said audio playback software application being executed by one or more programmed processors of an electronic device, said method comprising: Obtain an audio signal including audio content; Sensor data is obtained from one or more sensors arranged to sense the conditions of the environment in which the electronic device is located; Determine a device snapshot including the current state of each of one or more software applications being executed by the electronic device, wherein the one or more software applications being executed include the audio playback software application; At least one audio adjustment parameter of the volume compensator is determined based on the sensor data, the snapshot of the one or more software applications, and the audio content of the audio signal; The audio signal is processed using the volume compensator according to the determined audio adjustment parameters; as well as Use processed audio signals to drive one or more speakers. The device snapshot is a first device snapshot including a first state of a software application being executed by the electronic device, and the method further includes: Determine a second device snapshot that includes the software application in a second state different from the first state; Different audio adjustment parameters are determined at least based on the second state of the software application; and The audio signal is processed according to the different audio adjustment parameters determined.
14. The method of claim 13, wherein the current state of each of the one or more software applications indicates at least one of the software applications currently being executed by the electronic device, whether the user of the electronic device is interacting with the software application, and whether the audio content of the audio signal is associated with the software application.
15. The method of claim 13, wherein determining the at least one audio adjustment parameter comprises determining The scalar gain value of the volume compensator to be applied to the audio signal, and The volume compensator is required to compress the audio signal by its compression ratio, start time, and release time.
16. The method of claim 13, wherein the one or more sensors comprise at least one of a Global Positioning System (GPS) sensor, a camera, an accelerometer, a thermistor, an inertial measurement unit (IMU), and a microphone.
17. The method of claim 13, wherein the electronic device is a wearable device.
Citation Information
Patent Citations
Dynamic modification of audio signals
US10884696B1
