Hierarchical priority alarm evader matrix

Abstract

A system and method for organizing incoming data from an audio source and a vehicle system input into one or more matrices to hierarchically prioritize the incoming data into one or more of a low priority mixer, a medium priority mixer, and a high priority mixer. The hierarchically organized incoming data is prioritized and assigned as an input or trigger to one or more evaders to generate a mixed audio signal combining attenuated and evaded incoming data. The mixed audio signal is transmitted over the communication link for playback by the personal audio device.

Description

Technical Field

[0001] The present invention relates to an integrated in-vehicle personal audio system and, more particularly, to a ducker for an alarm in an in-vehicle personal audio system. Background Art

[0002] Vehicles typically include a vehicle entertainment system (such as an in-car entertainment (ICE) or in-vehicle infotainment (IVI) system), which includes various hardware and software components that enhance the driver and passenger experience by providing audio or video entertainment, video gaming capabilities, Internet connectivity, and associated features to the passengers and driver. A given vehicle may include multiple devices, such as mounted units, rear-seat entertainment (RSE) units mounted on vehicle seats, and / or portable devices connected to the vehicle (wirelessly or otherwise). Typically, each device is connected to a central hub, such as a head unit mounted near the driver's seat, typically within the vehicle's dashboard.

[0003] This creates a personalized zone that connects each vehicle occupant to the vehicle. Indicators may include, but are not limited to, emergency vehicle alarms, navigation prompts, chimes indicating a door is ajar or the trunk is open, and the like. Within the personalized zone, indicators that are typically received visually are also presented audibly to each occupant within their personalized zone. Indicators can be located both outside and inside the vehicle cabin. Furthermore, indicators and signals are blended and presented to each occupant based on their level of engagement with the vehicle. For example, the driver is located in a personalized zone where signals and alerts are presented to the driver with high priority. Alternatively, passengers in the back seat do not necessarily need to be exposed to navigation prompts and / or safety alerts.

[0004] One disadvantage is that occupants will be exposed to multiple types of inputs, alerts, and communication signals, but some are more critical to the occupants than others, and presenting them with all of the signals simultaneously may be counterproductive to the purpose of the alerts. It is desirable to allow critical safety alerts to be presented in parallel with less critical alerts and indicators in a manner that appropriately ducks or attenuates the less critical alerts compared to alerts deemed more critical. Summary of the Invention

[0005] One or more embodiments of the present subject matter are systems and methods for organizing incoming data from audio sources and vehicle system inputs into one or more matrices to hierarchically prioritize the incoming data into one or more of a low priority mixer, a medium priority mixer, and a high priority mixer. The hierarchically organized incoming data is prioritized and assigned as inputs or triggers to one or more duckers to generate a mixed audio signal that combines the attenuated and ducked incoming data. The mixed audio signal is transmitted over a communication link for playback by a personal audio device.

[0006] One or more embodiments organize incoming data from audio sources and vehicle systems into one or more matrices to organize incoming data from sources inside the vehicle into a seat media matrix, a communication zone matrix, and a voice sensing matrix.

[0007] One or more embodiments distribute the incoming data as an input or trigger to one or more of a media ducker, a communication ducker, a low priority ducker, and a medium priority ducker. Generating a mixed audio signal further comprises: a media mixer mixing signals from sources inside the vehicle and an ambiance mixer mixing signals from sources outside the vehicle.

[0008] One or more embodiments generate a mixed audio signal by attenuating and ducking incoming data having a lower priority in response to a trigger having a higher priority.

[0009] One or more embodiments generate the mixed audio signal by mixing signals from the media mixer and the ambiance mixer using a seat mixer based on priorities assigned to inputs to the one or more duckers and the presence of triggers to the one or more duckers.

[0010] One or more embodiments transmit the mixed audio signal to stimulate an audio playback device or a haptic actuator. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the present disclosure are particularly pointed out in the appended claims. However, other features of one or more embodiments will become more apparent and will be best understood by referring to the following detailed description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 Block diagram of the audio system;

[0013] Figure 2 is a block diagram of an example vehicle system;

[0014] Figure 3 is a block diagram of an example system in an example vehicle;

[0015] Figure 4 is a block diagram of sound zones within an example vehicle interior;

[0016] Figure 5 It is a block diagram that details the signal flow for each sound zone;

[0017] Figure 6 is a flow chart of method steps for synchronizing audio streams between an audio unit and a personal audio device;

[0018] Figure 7 is a flow chart of method steps for generating a mixed audio signal based on data from a vehicle environment; and

[0019] Figure 8 is a flow chart of method steps for processing incoming audio data according to hierarchical priority ordering. DETAILED DESCRIPTION

[0020] Detailed embodiments of the present invention are described herein. However, it should be understood that the disclosed embodiments are merely exemplary and that the present invention may be embodied in various alternative forms. The drawings are not necessarily drawn to scale. Some features may be exaggerated or minimized to illustrate details of particular components. Therefore, the specific structural and functional details disclosed herein are not to be construed as limiting, but merely as a representative basis for teaching those skilled in the art to employ the present invention.

[0021] In the following description, multiple instances of similar objects are described, where a reference number identifies the object and additional numbers identify the instance when necessary (eg, 110(1), 402(1)).

[0022] Figure 1 1 is a block diagram of an audio system 100, such as one in a vehicle, configured to implement one or more elements of the present subject matter. Audio system 100 has a computing device 102, an audio source 104, a sensor 106, an input / output (I / O) device 108, personal audio devices 110(1) to 110(n), a network 112, and a remote data source 114. Computing device 102 has a processing unit 116 and a memory 118. Memory 118 has an audio unit 120 and a data storage area 122. Data storage area 122 has a plurality of configuration files 124.

[0023] The computing device 102 includes a processing unit 116 and a memory 118. In one or more embodiments, the computing device 102 is a device including one or more processing units 116, such as a system on a chip (SoC). In one or more embodiments, the computing device 102 is a mobile computing device that is connected (wired and / or wirelessly) to other devices in the vehicle, such as a tablet computer, a mobile phone, a media player, etc. In some embodiments, the computing device 102 is a head unit included in the vehicle system. Additionally or alternatively, the computing device 102 can be a detachable device that is installed in a portion of the vehicle as part of a stand-alone console. Typically, the computing device 102 is configured to coordinate the overall operation of the audio system 100. The embodiments disclosed herein contemplate any technically feasible system that is configured to implement the functionality of the audio system 100 via the computing device 102. The functionality and technology of the audio system 100 are also applicable to other types of vehicles, including consumer vehicles, commercial trucks, airplanes, helicopters, etc.

[0024] The processing unit 116 may include one or more central processing units (CPUs), digital signal processing units (DSPs), microprocessors, application-specific integrated circuits (ASICs), neural processing units (NPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and the like. The processing unit 116 typically includes a programmable processor that executes program instructions to manipulate input data and generate output. In some embodiments, the processing unit 116 may include any number of processing cores, memories, and other modules for facilitating program execution. For example, the processing unit 116 may receive input from a user via the I / O device 108 and generate pixels for display on the I / O device 108 (e.g., a display device). In some embodiments, the processing unit 116 is configured to execute an audio unit 120 to control audio playback on one or more devices (including one or more personal audio devices 110). In such instances, the audio unit 120 organizes additional audio components based on data from multiple devices, prioritizes and mixes the additional audio components to provide a mixed audio signal that incorporates information about the vehicle environment from both inside and outside the vehicle.

[0025] The memory 118 may include a memory module or a collection of memory modules. The memory 118 typically includes a memory chip, such as a random access memory (RAM) chip, that stores applications and data for processing by the processing unit 116. In one or more embodiments, the memory 118 includes non-volatile memory, such as an optical drive, a magnetic drive, a flash drive, or other storage device. In some embodiments, a separate data storage area, such as a remote data source 114 connected to the network 112 (e.g., a "cloud storage device"), is connected to the audio unit 120. The audio unit 120 within the memory 118 is executed by the processing unit 116 to implement the overall functionality of the computing device 102 and, therefore, coordinate the operation of the audio system 100 as a whole.

[0026] The audio unit 120 processes incoming data from one or more devices and controls the personal audio device 110 and / or other vehicle speakers to reproduce audio signals. In one or more embodiments, the audio unit 120 establishes a communication channel in a configuration to provide high-fidelity audio content in the audio signal reproduced by the personal audio device 110. In one or more embodiments, the audio unit 120 extends connectivity to the personal audio devices 110(1)-110(n). For example, the audio unit 120 can be simultaneously connected to six separate personal audio devices inside a vehicle. In some embodiments, the audio unit 120 detects the personal audio device 110 when it enters or approaches the vehicle. For example, the audio unit 120 can use ultra-wideband (UWB) communication and / or near-field communication (NFC) to detect that the personal audio device 110 is near or within the vehicle. In one or more embodiments, the audio unit 120 collects various types of data and provides notifications to the user about the data. For example, the audio unit 120 may provide super hearing functionality to a given personal audio device 110 by acquiring sounds from an external microphone and / or a built-in microphone (e.g., emergency vehicles, pedestrians, bicyclists outside the vehicle, ambient sounds, voices of other occupants, etc.) and mixing the acquired sounds with the audio signal so that the user hears the acquired sounds when using the personal audio device 110.

[0027] The data storage area 122 is a portion of the memory 118 that locally stores various data, including configuration files 124 (e.g., user configuration files, device configuration files, etc.) and other data (not shown), such as content items, data tables (e.g., a table mapping audio tones to events), and / or application data associated with the audio unit 120 (e.g., security application data, metadata, etc.). In one or more embodiments, the data storage area 122 may be included in volatile memory and may correspond to a sector of non-volatile memory. In some embodiments, the computing device 110 may synchronize data between the volatile memory and the non-volatile memory so that copies of the data are stored in both the volatile memory and the non-volatile memory. In some embodiments, the data storage area 122 stores downloaded content items received via the network 112 to enable playback when the network 112 is inaccessible or when the vehicle's network resources are limited.

[0028] The profiles 124 include device profiles associated with a specific personal audio device 110 and / or user profiles associated with a specific user. In some embodiments, a user profile is linked to one or more device profiles, and vice versa. In such instances, the audio unit 120 prompts the user to select which personal audio device 110 is being used. In one or more embodiments, the profiles 124 include configuration information associated with audio reproduction of audio signals using the personal audio device 110 and / or vehicle speakers. Such configurations may include, for example, specific equalizer settings, selected speaker preferences, auditory augmented reality (AR) preferences, etc. The audio unit 120 uses the profiles to determine which audio output devices (e.g., personal audio device 110(1), one or more vehicle speakers, etc.) to use to reproduce audio signals and how to modify incoming audio signals.

[0029] Audio source 104 comprises one or more data sources that provide audio signals for reproduction. In one or more embodiments, audio source 104 is included in a device within the vehicle, such as an entertainment subsystem included in the vehicle's head unit, a rear-seat entertainment console, a device installed in the vehicle, or the like. In some embodiments, audio source 104 is included in a mobile device, wearable device, and / or other portable device connected to audio unit 120. Additionally or alternatively, audio source 104 may be located remotely from the vehicle. For example, audio source 104 may be included in remote data source 114. In such instances, remote data source 114 streams the audio signal to computing device 110, which then transmits the audio signal to personal audio device 110 for reproduction.

[0030] Sensors 106 include one or more devices that perform measurements and / or acquire data related to certain entities in the environment. In one or more embodiments, sensors 106 generate sensor data related to a user and / or objects in the environment that are not the user. In some embodiments, sensors 106 are coupled to and / or included within computing device 110 and transmit the sensor data to processing unit 116.

[0031] In one or more embodiments, the sensors 106 include audio sensors, such as microphones and / or microphone arrays that record external sounds and / or built-in microphones that record sounds within the vehicle's cabin. In some embodiments, the sound data acquired by the microphones is processed by the audio unit 120 performing various natural language (NL) processing techniques, sentiment analysis, and / or speech analysis to determine the semantics of phrases spoken in the environment.

[0032] In some embodiments, the sensor 106 includes an optical sensor, such as an RGB camera, an infrared camera, a depth camera, and / or a camera array, which includes two or more of such cameras. Other optical sensors may include imagers and laser sensors. In some embodiments, the sensor 106 includes physical sensors that register the body position and / or movement of the user and / or personal audio device 110, such as touch sensors, pressure sensors, position sensors (e.g., accelerometers and / or inertial measurement units (IMUs)), motion sensors, etc. In some embodiments, the sensor 106 includes physiological sensors, such as heart rate monitors, electroencephalogram (EEG) systems, radio sensors, thermal sensors, galvanic skin response sensors (e.g., sensors that measure changes in skin resistance caused by emotional stress), contactless sensor systems, magnetoencephalogram (MEG) systems, etc.

[0033] Input / output (I / O) devices 108 include devices capable of receiving input, such as a keyboard, mouse, touch-sensitive screen, microphone, and other input devices for providing input data to computing device 110. In one or more embodiments, I / O devices 108 are associated with a specific console, such as a personalized screen mounted to a portion of a seat, a personal audio device 110, personalized speakers within the vehicle, console-specific input components, and the like. Additionally or alternatively, I / O devices 108 include various devices capable of providing output, such as a display screen, speakers, and the like. In one or more embodiments, one or more of I / O devices 108 are incorporated into computing device 110 or are external to computing device 110. In some embodiments, computing device 110 and / or one or more I / O devices 108 may be components of an advanced driver assistance system (ADAS) or an entertainment subsystem included in the vehicle. In one or more embodiments, audio unit 120 responds to various inputs received by one or more I / O devices 108. For example, a vehicle may include a head unit that includes a user interface. In such instances, the audio unit 120 may respond via one or more touch inputs and / or NL voice inputs received by the head unit.

[0034] The personal audio device 110 (e.g., 110(1) to 110(n)) includes one or more headphones, earbuds, wearable devices, and / or other types of personalized audio devices that reproduce audio signals to provide audio content to an individual. In some embodiments, the personal audio device 110 provides directional sound in a manner that provides personalized audio to the listener (e.g., a collar that provides a directional speaker). In some embodiments, the personal audio device 110 is configured to provide equalization, noise cancellation, audio augmented reality, high-fidelity audio, and / or other features. In some embodiments, the personal audio device 110 uses a dedicated protocol and / or is manufactured using dedicated hardware to communicate with the audio unit 120 with low latency.

[0035] The network 160 enables communication between the computing device 110 and other devices in the network 160 via wired and / or wireless communication protocols, satellite networks, V2X networks (including Bluetooth, Bluetooth Low Energy (BLE), Wireless Local Area Network (Wi-Fi), cellular protocols and / or Near Field Communication (NFC)).

[0036] Remote data source 114 comprises a computing device in communication with computing device 102, such as a laptop computer, tablet computer, smartphone, cellular phone, desktop computer, teleconferencing system, etc. In some embodiments, the remote data source is a software service or a device that executes a media application, such as audio software, video software, audio calling software, messaging software, a media service platform, etc. In such an example, remote data source 114 can use such a media application to communicate with an audio unit 120 operating within the vehicle.

[0037] The computing device 102 executes instructions to the audio unit 120 to control the playback of audio from one or more vehicle components and / or personal audio devices 110(1)-110(n) within the vehicle. Specifically, the processing unit 116 receives data from the sensor 106, the I / O device 108, and / or the personal audio devices 110(a)-110(n). The audio unit 120 processes the data to detect the presence of at least one personal audio device, such as 110(1). Upon detecting the at least one personal audio device 110(1), the audio unit 120 determines whether the personal audio device 110(1) is known, and if so, whether the personal audio device 110(1) is associated with a known user and / or user profile. When the personal audio device 110(1) is known, the audio unit 120 loads a profile 124 for the user and / or the personal audio device 110(1). The profile 124 may include configuration information associated with audio playback using the personal audio device 110(1) and / or vehicle speakers, including one or more specific equalizer settings, selected speaker preferences, auditory augmented reality (AR) preferences, etc. The audio unit 120 uses the profile 124 to determine which audio output devices (e.g., the personal audio device 110(1), one or more vehicle speakers, etc.) to use to reproduce the audio signal and how to modify the incoming audio signal. The audio unit 120 establishes a communication link with the personal audio device 110(1). In one or more embodiments, the audio unit 120 determines the location of the personal audio device 110(1), determines a sound zone for the determined location, and manages the speakers within the sound zone. Sound zones are referred to later in this document. Figure 4 In one or more embodiments, the audio unit 120 mutes the speakers in the sound zone so that only the personal audio device 110(1) can reproduce the audio signal in the sound zone. Alternatively, in one or more embodiments, the audio unit 120, together with the personal audio device 110(1), controls the set of speakers in the sound zone so that the personal audio device 110(1) and the set of speakers jointly reproduce the audio signal.

[0038] When the audio unit 120 streams the audio signal to the sound zone for reproduction, the audio unit 120 continues to process incoming data from various devices such as, but not limited to, sensors 106, I / O devices 10, vehicle components, remote data sources 114, etc. to determine whether to mix the additional audio component with the audio signal. The additional audio component may include an audio clip recorded by a microphone, such as a sound from outside the vehicle or a speech from another vehicle occupant. Additionally or alternatively, the additional audio component may include an audio tone associated with an event type triggered by the incoming data (e.g., an audio notification indicating a navigation tone, an alarm notification based on a controller area network message, etc.). When the audio unit 120 determines to add an additional audio component, the audio unit 120 mixes the audio signal with the additional component, thereby ducking the audio signal as needed to produce an attenuated version of the audio signal that enables the user to hear the additional audio component. The audio unit 120 transmits the mixed audio signal to the speakers in the sound zone for reproduction.

[0039] Figure 2 According to one or more embodiments, Figure 1 1. Block diagram of an example vehicle system 200 of an audio unit 120. As shown, the vehicle system 200 includes, but is not limited to, a sensing module 210, a head unit 220, an output module 230, and a collection of personal audio devices 110 (e.g., 100(1) to 110(N)). The sensing module 210 includes, but is not limited to, vehicle sensors 214, occupant-facing sensors 216, and cabin sensors 218. The head unit 220 includes, but is not limited to, the audio unit 120, an entertainment subsystem 222, a network module 224, and a navigation subsystem 226. The output module 230 includes, but is not limited to, a speaker 232, a display 234, and a human-machine interface (HMI) 236. The audio unit 120 includes, but is not limited to, an input processing module 262, an output generation module 264, and a synchronization module 266.

[0040] In operation, the audio unit 120 establishes a connection with the set of personal audio devices 110. The audio unit 120 transmits audio signals to the personal audio device 110 and processes incoming data from the sensing module 210 and / or components of the head unit 220 to determine whether to add additional audio components to a given audio signal. The audio unit 120 will then add audio clips and / or audio tones to the audio signal to provide the user with information about the vehicle environment.

[0041] The sensing module 210 includes various types of sensors, including vehicle sensors 214 (e.g., outward-facing cameras, external microphones, accelerometers, etc.), occupant-facing sensors 216 (e.g., cameras, microphones, motion sensors, etc.), and / or non-occupant-facing cabin sensors 218 (e.g., pressure sensors, temperature sensors, etc.). In one or more embodiments, the sensing module 210 provides a combination of sensor data describing the context of the vehicle and the occupants present within the vehicle. For example, the sensing module 210 may provide a set of values associated with the occupants (e.g., the occupant's position, the position of the personal audio device 110, noise levels, etc.). In such instances, the audio unit 120 may identify which seats within the vehicle are occupied.

[0042] In one or more embodiments, vehicle sensors 214 also include other external sensors. Such external sensors may include optical sensors, road vibration sensors, temperature sensors, and the like. In some embodiments, sensing module 210 and / or network module 224 acquire other external data, such as geolocation data (e.g., GNSS systems, including Global Positioning System (GPS), Glonass, Galileo, etc.), demographic data, psychographic data, and the like. In such instances, audio unit 120 may process the incoming data. For example, audio unit 120 may add notification tones about points of interest to the user based on the vehicle's geolocation. In another example, audio unit 120 may provide an alert from a driver assistance system upon detecting an alert condition and provide an audio clip of ambient sounds associated with the alert condition. In some embodiments, navigation data and / or geolocation data are combined and associated with a specific media configuration. For example, when navigation subsystem 226 generates a route between locations, audio unit 120 provides audio tones associated with the navigation route (e.g., a voice-generated clip with navigation instructions).

[0043] The head unit 220 is a component of the vehicle that is installed anywhere within the vehicle's passenger compartment in any technically feasible manner. In some embodiments, the head unit 220 includes any number and type of instruments and applications and provides any number of input and output mechanisms. For example, the head unit 220 enables a user (e.g., driver and / or passenger) to control the entertainment subsystem 222 and / or navigation subsystem 226. The head unit 220 supports any number of input and output data types and formats, as known in the art. For example, the head unit 220 may include built-in Bluetooth for hands-free calling and / or audio streaming, a universal serial bus (USB) connection, speech recognition, rearview camera input via the sensing module 210, video output for any number and type of displays 234 via the output module 230, and any number of audio outputs. Generally, any number of sensors (e.g., sensors 106, 214, 216, 218), displays 234, receivers, transmitters, etc. may be integrated into the head unit 220, or may be implemented externally to the head unit 220. Additionally or alternatively, audio unit 120 may be integrated into host unit 220, or may be implemented external to host unit 220. In one or more embodiments, the external device (eg, personal audio unit 110) communicates with host unit 220 in any technically feasible manner.

[0044] The entertainment subsystem 222 provides various information to the user and / or one or more other occupants of the vehicle via the output module 230. The navigation subsystem 226 provides information about the vehicle's location and path-finding information for reaching the route. For example, the navigation subsystem may provide route information associated with the vehicle via the head unit 220 and / or the HMI 236. The entertainment subsystem 222 enables playback of audio and / or video content from various media sources (e.g., internal sources or external media providers via the network module 224). In one or more embodiments, the entertainment subsystem enables playback of content via the audio unit 120 and / or other output devices included in the output module 230 (e.g., the output device 108, the speaker 232, the display 234, and / or the HMI 236).

[0045] The output module 230 performs one or more actions in response to actions performed by the head unit 220 and / or the audio unit 120. For example, the output module 230 may generate one or more output signals in response to received media signals to drive the output of the speaker 232 and / or the display 234. In another example, the output module 230 may generate one or more output signals to modify the human-machine interface (HMI) 236 to display notification messages and / or alerts. In some embodiments, the HMI 236 is a separate component from the display 234, such as when the HMI 236 is included as part of the windshield. In such instances, the head unit 220 may specify whether the notification is displayed via the display 234 or via the HMI 236.

[0046] The audio unit 120 includes various modules for managing the processing of data and the transmission of audio signals for reproduction. In one or more embodiments, the audio unit 120 also implements one or more of the input processing module 262, the output generation module 264, and / or the synchronization module 266 to establish a communication link with one or more of the personal audio devices 110 and mix additional audio components into the audio signal provided by the audio source 104.

[0047] The input processing module 262 processes various types of data from one or more components and / or devices. For example, the input processing module 262 processes incoming data from one or more devices to detect where a given personal audio device 110 is located within the vehicle. For example, the input processing module 262 may process sensor data from a sensor 106 (e.g., a cabin sensor 218) that indicates the location of the personal audio device 110 within the vehicle. In some embodiments, the audio unit 120 receives messages directly from the personal audio device 110(N). Additionally or alternatively, the input processing module 262 processes incoming data from various devices associated with the environment, such as audio data from a microphone that records the vehicle's external environment, speech from other occupants of the vehicle, events associated with the operation of the entertainment subsystem 222, the network module 224, the navigation subsystem 226, and the like.

[0048] The output generation module 264 provides the audio signal to the personal audio device 110 and / or the speakers 232 for reproduction. In one or more embodiments, the output generation module 264 responds to a determination to mix additional audio content with the audio signal by generating the additional audio content and generating a mixed audio signal comprising the audio signal provided by the audio source 104 and the additional audio content. In some embodiments, the output generation module 264 extracts the audio content from the incoming data. For example, the output generation module 264 generates an audio clip by extracting a portion of the ambient sound data from a microphone included in the vehicle sensor 214. In another example, the output generation module 264 adds a notification sound indicating a change in the audio track based on a notification message generated by the entertainment subsystem 222. In one or more embodiments, the output generation module 264 applies various equalization, noise reduction, and / or other sound modification techniques based on one or more configuration files 124. For example, the output generation module 264 retrieves a device configuration file 124 associated with the personal audio device 110 and uses the device configuration file as an output configuration file to modify the audio signal, and then transmits the modified audio signal to the personal audio device 110 for reproduction.

[0049] The synchronization module 266 establishes communication with the personal audio device 110 and performs various techniques to establish a communication link with the personal audio device 110 and reduce latency in transmitting audio signals to the personal audio device 110. For example, the synchronization module 266 exchanges messages with the personal audio device 110 to determine the location of the personal audio device 110 within the vehicle and exchanges a set of messages to determine the characteristics of the communication link to be established (e.g., negotiating a capability set for the communication link, determining whether to establish a backchannel, determining content protection, payload composition, etc.). After determining the characteristics, the audio unit 120 establishes a communication link with the personal audio device 110. Additionally or alternatively, the synchronization module 266 performs various techniques to reduce latency between the audio source 104 providing the audio signal and the personal audio device 110. For example, the audio unit 120 may devote more processing resources to increase throughput on the communication link.

[0050] In one or more embodiments, the synchronization module 266 establishes communication with a given personal audio device 110 upon determining that the personal audio device 110 is an authenticated component for interoperability with the audio unit 120. In such an example, upon determining that the personal audio device 110 is not authenticated for operation with the audio unit 120, the synchronization module 266 establishes a connection with the speaker 232 in place of the personal audio device 110. Additionally or alternatively, the synchronization module 266 establishes a connection with the speaker 232 when the personal audio device 110 is not present or powered on.

[0051] Figure 3According to one or more embodiments, Figure 1 1. Block diagram of an example system 300 in a vehicle 302 with an audio unit 120. As shown, the system 300 includes, but is not limited to, the vehicle 302, a mobile device 320, the personal audio unit 110, a communication link 312 (wired and / or wireless), and an audio link 332.

[0052] In operation, the mobile device 320 is connected to the audio unit 120 to provide an audio signal. The audio unit 120 simultaneously establishes a streaming session with each of the personal audio devices 110(1), 110(2), and controls the speakers 232(1), 232 located near the personal audio devices 110(1), 110(N), respectively, and provides the audio signal via the audio link 332. The audio unit 120 receives data from the vehicle sensors 214 and / or the cabin sensors 218 and generates a mixed audio signal that incorporates the information provided by the sensors 214, 218. The audio unit 120 transmits the audio signal and / or the mixed audio signal to the personal audio device 110 via the communication link 312 and / or transmits the audio signal and / or the mixed audio signal to the speaker 232 via the audio link 332.

[0053] In one or more embodiments, the personal audio device 110 establishes the communication link 312 using a communication protocol such as Bluetooth. In one or more embodiments, the audio unit 120 specifies the characteristics of the communication link 312 and / or the audio link 332 for the corresponding personal audio device 110. For example, the audio unit 120 may mute the speaker 232(1). In such an instance, the audio unit 120 suppresses the provision of the audio signal to the speaker 232(1). Alternatively, in some embodiments, the audio unit 120 may transmit an inaudible audio signal (e.g., a noise cancellation signal) for reproduction by the speaker 232(1). In another example, the audio unit 120 may generate a set of component audio signals based on the audio signal and may transmit the component audio signals separately to the personal audio device 110(N) and the speaker 232(N). In such an instance, the personal audio device 110(N) and the speaker 232(N) reproduce the corresponding component signals to collectively reproduce the audio signal provided by the audio source 104.

[0054] Figure 4 Described according to one or more embodiments include Figure 1 1. An example of an interior 400 of a vehicle 302 with an audio unit 120. As shown, the interior 400 includes, but is not limited to, the personal audio device 110 and sound zones 402 (eg, 402(1), 402(2), etc.).

[0055] In one or more embodiments, the vehicle 302 includes multiple sets of speakers 232 that generate multiple sound zones (e.g., 402(1), 402(2)) located at different locations within the vehicle 302 and provide personalized audio to the occupants. In some embodiments, the vehicle 302 includes multiple sets of speakers 232 that generate sound zones 402(1)-402(6) in areas of the vehicle corresponding to individual seats or larger areas (e.g., the back row). In such instances, each sound zone 402 is located in an area near a seat so that when the sound zone is seated at a specific location, the occupant's ears are located within the sound zone 402. For example, as shown, when the occupant is seated in the front passenger seat, the occupant's ears are located within the sound zone 402(2).

[0056] In one or more embodiments, one or more output devices 140 associated with a corresponding console and / or other speakers 232 positioned proximate to the seats generate the sound zones 402. In such instances, the audio unit 120 controls the speakers 232 of a given sound zone 402 so that the personal audio device 110 operates in conjunction with or in place of the speakers 232 in the sound zone 402. In such instances, occupants of the sound zone hear the reproduced audio signal, while other occupants hear little or no reproduced audio signal.

[0057] In one or more embodiments, the audio unit 120 determines a sound zone based on the location of each personal audio device 110. For example, the audio unit 120 determines that the personal audio device 110(4) is being worn by an occupant in the rear row. In such an example, the audio unit 120 modifies the sound zone 402(5) to include the entire rear row. The audio unit 120 then controls the set of speakers 232 in the rear row based on the configuration associated with the personal audio device 110(4) (e.g., only the subwoofer is selected) and / or the user's preferences (e.g., a stored user profile has a preference to mute all speakers 232 in the sound zone 402(5).

[0058] Figure 5is a block diagram detailing the signal flow 500 for each sound zone 402 as processed by the computing device 102 to generate a mixed audio signal from one or more incoming data sources, such as media input from a personal audio device (audio source 104) and system inputs 106, 108, for playback in each sound zone 402. According to one or more embodiments, the incoming data is processed and prioritized in a hierarchical manner to generate whether and how the incoming data should be played back at each personal sound zone 402(1)-402(N). The audio input may include, but is not limited to, media input from the audio sources 104(1)-104(N) of each sound zone 402(1)-402(N), a built-in microphone 502, an external microphone 504, engine noise 506, navigation prompts 508, and other vehicle chimes 510 (e.g., seat belt, door ajar, trunk open, etc.). Occupants of each sound zone will only hear the reproduced audio signal for their sound zone, and they will not hear the reproduced signal for other occupants in other sound zones.

[0059] Built-in microphone 502 is a sensor located within the vehicle cabin that detects sounds, speech, and noise. The audio input signal from built-in microphone 502 may be pre-processed as needed. For example, the audio signal may undergo virtual venue processing 512, where an algorithm is applied to tune the audio signal to recreate the audio in a manner that transforms the vehicle cabin into a venue such as an auditorium or stadium. As another example, the audio input signal from built-in microphone 502 may undergo in-car communication (ICC) processing 514, which processes the voice audio signal in a manner that ensures clear two-way communication between occupants in the vehicle cabin. Signal processing may include, but is not limited to, automatically setting music volume or road noise in a manner that allows conversation between front-to-back passengers from different rows. In yet another example, incoming data from built-in microphone 502 may undergo processing 516 to reduce echo and background noise to eliminate unwanted cabin noise for occupants who are talking on a phone or interacting with a voice assistant.

[0060] The external microphone 504 is a sensor located outside the vehicle body that detects sounds, speech, and noises outside the vehicle that may need to be communicated to the occupants inside the vehicle. Incoming data from the external microphone 504 source can be processed 518 to include an audio component that provides the occupants inside the vehicle with an enhanced perception of external sounds. This feature enhances communication possibilities, for example, enabling the driver to understand what is happening outside the vehicle. For example, the external microphone 504 provides an audio component that can be mixed into a mixed audio signal that is transmitted to the driver to provide information about emergency vehicles, pedestrians, and cyclists outside the vehicle. The external microphone 504 signal can also be processed 520 to reduce echo and background noise for occupants who may interact with pedestrians or cyclists outside the vehicle.

[0061] Once the incoming data is processed as needed, it is managed according to one or more embodiments for hierarchical prioritization. Engine noise 506, navigation prompts 508, and vehicle system ringtones 510 are other audio input signals that can be prioritized for playback based on the user profiles of each personal zone and the occupants in the personal zone. There are multiple hierarchical levels, and each hierarchical level involves a mixer block that mixes the levels into a composite signal. The composite signal is then presented to the ducker block as a trigger input. The ducker block attenuates all lower priority audio signals to generate a network of simultaneous signals that are mixed and blended in a functional manner so that the occupants can understand the information in a way that they can understand. Hierarchical prioritization is especially important for high priority signals such as safety ringtones and emergency alarms.

[0062] For simplicity, in the examples provided below, the specific embodiments process signal flows with three levels of hierarchical priority. However, it should be noted that the subject matter of the present invention is applicable to more or less than three levels of hierarchical priority without departing from the scope of the present invention.

[0063] Furthermore, the examples provided below describe a plurality of matrices and mixers. It should be noted that the inventive subject matter is applicable to more or fewer matrices and mixers than those described below without departing from the scope of the inventive subject matter.

[0064] Incoming audio data is managed in a matrix. At a first hierarchical level 500A, incoming data is organized in matrix blocks from which a user can select it or route it for further processing, e.g., in a mixer block or a ducker block. Desired content can be selected from each matrix, e.g., by volume adjustment and / or by personal profile settings. Figure 5In the example shown, media inputs 104(1)-104(N) are organized in a seat media matrix block 522 that includes all possible sources available for user selection. Each user selects their own source, and all users can share media. A communication zone matrix block 524 manages communications 514 between occupants in each personalized sound zone within the vehicle. A voice sensing matrix block 526 manages signals associated with occupants in each personalized sound zone within the vehicle who are on a call or interacting with a voice assistant 520.

[0065] Incoming audio data is prioritized hierarchically into one or more priority levels. Each hierarchical priority level has an associated mixer block to combine incoming data with the same hierarchical priority. For example, incoming data considered high priority is assigned to a high priority mixer block 540. Incoming data considered medium priority is assigned to a medium priority mixer block 534. Incoming data considered low priority is assigned to a low priority mixer block 527. Incoming data considered high priority may include, but is not limited to, ringtones 510 from various vehicle systems that alert occupants to important or critical events related to occupant safety in the vehicle. Other incoming data considered high priority may be related to events such as, but not limited to, the detection of an emergency vehicle 518. Incoming audio considered medium priority may include, but is not limited to, navigation prompts 508. Examples of low priority audio data are engine noise 506, communications 520 via an external microphone 504, and incoming data processed at a virtual venue processing block 512.

[0066] The output of each mixer block 527, 534, 540 can trigger a corresponding ducker (described in detail later herein) to duck or attenuate lower priority incoming data below higher priority incoming audio data. The output of each mixer block 527, 534, 540 can also be an input to another mixer at the second hierarchical level 500B.

[0067] The low priority mixer block 528 mixes incoming audio data from audio sources that are considered low priority for playback in one or more personal sound zones. For example, the virtual field 512 signal, the voice signal and / or noise signal 520 from the external microphone 504, and the engine noise 506 may be considered low priority for playback in one or more personal sound zones. The output of the low priority mixer block 528 is the input 530 of the low priority ducker block 532.

[0068] The medium priority mixer block 534 mixes audio signals from audio sources that are considered medium priority for playback at one or more personal sound zones, such as navigation prompts 508. The output of the medium priority mixer block 534 is the input 536 of the medium priority ducker block 538.

[0069] The high-priority mixer block 540 mixes audio signals from audio sources that are considered high priority for playback in one or more personal sound zones. For example, the audio signal from the external microphone 504 detecting an emergency vehicle 518 can be considered high priority, particularly for the occupant in the driver's seat. A signal that can also be considered high priority is the vehicle safety chime 510 that can indicate lane departure warnings, collision warnings, unfastened seatbelts, a partially opened door, a partially opened trunk, low tire pressure, and the like. The output of the high-priority mixer block 540 is not ducked because it is important to always playback the output in the appropriate personal sound zone. The signal output from the high-priority mixer block 540 can serve as a trigger for the low-priority ducker 532 and the medium-priority ducker 538. In addition, the output from the high-priority mixer block 540 can serve as an input to one or more mixer blocks described later in this article.

[0070] The voice sensing mixer block 542 mixes the incoming data from each of the voice sensing matrix block 526, the middle priority mixer block 534 and the high priority mixer block 540. Output from the voice sensing mixer block 542 can serve as a trigger 554 for a media ducker block 550. Output from the seat media matrix 522 is an input 552 of the media ducker block 550. Trigger 554 from the voice sensing mixer block 542 will cause the media ducker block 550 to duck or decay the input 552 from the seat media matrix so that the incoming data with higher priority takes precedence in the mixed signal that is finally presented to the occupant in the sound zone. For example, for the driver's sound zone, the output of the high priority mixer 540 is more critical than the output of the middle priority mixer 534, and the output of the middle priority mixer is more important to the driver than the output of the low priority mixer. In addition, the output from the voice sensing matrix 526 (for example, the incoming data from a phone call) can take precedence over the entertainment media from the seat media matrix block 522. The media ducker block 550 is capable of mixing the incoming audio data appropriately to generate input to the media mixer block 546 .

[0071] The communication zone mixer block 544 also receives incoming data from the high priority mixer block 534 and the medium priority mixer block 540. The output from the communication zone matrix block 524 is an input 560 to the communication ducker block 558. The output of the communication zone mixer block 544 is a trigger 556 to the communication ducker block 558. The trigger 556 causes the communication ducker block 558 to duck or attenuate the input 560 from the communication zone matrix block 524 so that the signal with a higher priority is presented more prominently in the mixed audio signal being generated for any of the sound zones 402. For example, the navigation prompt 534 will trigger the communication ducker block 556 to attenuate or block the input 560 from the communication zone matrix 524 so that the navigation prompt is more prominent in the signal being generated for presentation to the sound zone occupied by the occupants of the driving vehicle.

[0072] The media mixer block 546 mixes the signals from the media ducker block 550 and the communication ducker block 558 with the audio signals input by the signal processed by the virtual venue 512 algorithm. A composite signal from each of the media ducker block 550 and the communication ducker block 558, where the composite signal has been hierarchically prioritized and mixed with the signal processed by the virtual venue 512, generates a composite signal that is input to the seating mixer block 546. The composite signal 562 generated by the media mixer block 546 represents media-related incoming data that is attenuated and ducked in a manner to be presented to the personal sound zone in accordance with the hierarchical priorities set and processed in the audio unit 120, any stored personal profile settings, and personal audio devices detected at the sound zone and linked to the audio unit, and communication-related incoming data that is attenuated and ducked in a manner to be presented to the personal sound zone 402 in accordance with the hierarchical priorities set and processed in the audio unit 120, any stored personal profile settings, and personal audio devices detected at the sound zone and linked to the audio unit.

[0073] The ambiance mixer block 548 mixes the signals from the low priority ducker block 532, the medium priority ducker block 538, and the high priority ducker block 540. The composite signal 564 generated by the ambiance mixer block 548 represents signals outside the vehicle cabin, such as, but not limited to, emergency vehicles 518, voice sensing from pedestrians or cyclists around the vehicle 520, and engine noise 505. Additionally or alternatively, the composite signal generated by the ambiance mixer block 548 represents navigation prompts 508 and / or vehicle chimes 510, which may be safety-related, such as, but not limited to, lane departure warnings, collision warnings, seatbelt chimes, door ajar chimes, and / or trunk ajar chimes.

[0074] The seat mixer 566 receives composite signals 562 and 564 from the media mixer 546 and the ambiance mixer 548, where the composite signals are mixed and managed according to the hierarchical priorities generated as part of the signal flow and the personal zones to which the signals are to be delivered. Since each personal zone has a personal audio device that is linked to and subscribed to an audio source, the composite signal generated at the seat mixer 566 includes signals associated with the specific personal zone at the assigned hierarchical priority. For example, the driver subscribes to their own media content, which may include but is not limited to in-car communication content, navigation prompts, vehicle ringtones, emergency vehicle detection, and selected in-vehicle infotainment media sources. In another example, an occupant in a personal zone located at the rear of the vehicle may subscribe only to their own media content and in-vehicle communication content. In this case, the composite signal for the occupant driving the vehicle may be much more comprehensive than the composite signal for the occupant in the rear passenger seat sound zone.

[0075] The seat mixer 566 accordingly outputs a mixed audio signal 568 to each personal zone 402(1)-402(N). The output of the seat mixer can be presented in an auditory form, such as through headphones, or can be presented as a tactile signal, such as an actuator in a seat in the personal zone activated by the mixed audio signal. The mixed audio signal 568 can be heard by an occupant of the corresponding sound zone, such as through headphones 570 worn by the occupant. Additionally or alternatively, the mixed audio signal can include a tactile alert 572. The mixed audio signal 568 causes the actuator to deliver a physical sensation, such as a vibration, tap, or click, to the occupant.

[0076] Figure 7 is a flow chart of method steps for generating a mixed audio signal according to one or more embodiments. Figures 1 to 4 Although the method steps are described with reference to a system, those skilled in the art will appreciate that any system configured to perform the method steps in any order falls within the scope of one or more embodiments. Various audio sources 104 provide media input to the computing device. System inputs 106, 108 may include, but are not limited to, audio signals from a built-in microphone 502, an external microphone 504, engine noise 506, navigation prompts 508, and other vehicle system tones 508.

[0077] As shown, the method 600 begins at step 602, where the audio unit 120 detects a personal audio device 110 in the vehicle 302. In one or more embodiments, the audio unit 120 processes incoming data from one or more devices to detect the location of a given personal audio device 110 (N) within the vehicle 302. In some embodiments, the input processing module 262 included in the audio unit 120 processes sensor data from a sensor 130 (e.g., a cabin sensor 218) indicating the location of the personal audio device 110 (N) within the vehicle 302. In some embodiments, the audio unit 120 receives messages directly from the personal audio device 110 (N). In such instances, the synchronization module 266 of the audio unit 120 exchanges messages with the personal audio device 110 (N) to determine the location of the personal audio device 110 (N) within the vehicle 302.

[0078] At step 604, the audio unit 120 identifies the user associated with the personal audio device 110(N). In one or more embodiments, the audio unit 120 identifies the device identifier associated with the personal audio device 110(N). In such instances, the audio unit 120 scans the profile 124 in the data storage area 122 to determine whether the personal audio device 110(N) is associated with a specific user and / or user profile. Alternatively, in some embodiments, the user provides input via an input device (e.g., a mobile device 320, a vehicle console, etc.) to provide a check-in. In such instances, the audio unit 120 uses the information provided by the user during the check-in to locate the profile 124 associated with the user.

[0079] At step 606, the audio unit 120 determines whether any profile 124 stored in the data store 122 is associated with the user. When the audio unit 120 identifies a profile 124 for the user, the audio unit 120 proceeds to step 608, where the audio unit 120 loads the profile 124 from the data store 122. Otherwise, when the audio unit 120 does not identify a profile 124 for the user, it proceeds to step 610. In some embodiments, the audio unit 120 generates a profile for the user for streaming to the personal audio device 110(N).

[0080] At step 610, the audio unit 120 establishes a communication link 312 with the personal audio device 110(N). In one or more embodiments, the audio unit 120 creates a direct connection with the personal audio device 110(N) by establishing the communication link 312(N) with the personal audio device 110(N). In one or more embodiments, the synchronization module 266 included in the audio unit 120 exchanges a set of messages with the personal audio device 110(N) to determine the characteristics of the communication link 312(N) (e.g., negotiate the capability set of the communication link 312(N), determine whether to establish a reverse channel, determine content protection, payload composition, etc.). After determining the characteristics, the audio unit 120 establishes the communication link 312(N) with the personal audio device 110(N).

[0081] At step 612, the audio unit 120 determines a sound zone that includes the personal audio device 110(N). In one or more embodiments, the audio device's input processing module 262 processes a set of input data to determine the location of the personal audio device 110(N) and identifies a sound zone to provide personalized audio to a user located proximate to the personal audio device 110(N). In one or more embodiments, a subset of the speakers 232 within the vehicle may be associated with a predetermined sound zone (e.g., the sound zone 402(1)). In such instances, the audio unit 120 identifies the subset of speakers 232 after identifying the sound zone 402(1). Alternatively, the audio unit 120 uses a selection algorithm to identify a set of speakers 232 in the vehicle 302 that are to provide sound to the sound zone 402(1).

[0082] At step 614, the audio unit 120 manages the vehicle speakers 232 for the sound zone 402. In one or more embodiments, the audio unit 120 manages a set of speakers 232(N) associated with the sound zone 402(1) identified in step 612. In some embodiments, the audio unit 120 mutes the speakers 232(N) for the sound zone 402(1) so that the personal audio device 110(N) reproduces the audio signal instead of the speakers 232(N). Alternatively, the audio unit 120 may manage the speakers 232(N) to reproduce the audio signal in conjunction with the personal audio device 110(N). In such instances, the audio unit 120 provides portions to the personal audio device 110(N) and the speakers 232(N), respectively, for reproduction within the sound zone 402(1).

[0083] At step 616, the audio unit 120 synchronizes the audio stream to the sound zone 402. In one or more embodiments, the synchronization module 266 of the audio unit 120 synchronizes the audio stream provided to the personal audio device 110(N) via the communication link 312(N) and / or to the speakers 232(N) in the sound zone 402(1) via the audio link 332(N) to enable the devices to reproduce the audio signals. In one or more embodiments, the synchronization module 266 performs various techniques to reduce the delay between the audio source providing the audio signal and the personal audio device 110(N) reproducing the audio signal. For example, the audio unit 120 may devote more processing resources to increase the throughput through the communication links 312(N), 332(N).

[0084] Figure 7 A flow chart illustrating method steps for generating a mixed audio signal based on data from a vehicle environment according to various embodiments. Figures 1 to 5 The method steps are described with respect to a system, but one skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the various embodiments.

[0085] As shown, method 700 begins at step 702, where the audio unit processes incoming data. In various embodiments, the input processing module 262 of the audio unit 116 processes data from various devices (e.g., sensors 130, remote data sources 170, entertainment subsystem 222, network module 224, navigation subsystem 226, etc.). The incoming data is associated with the environment, such as audio data from a microphone recording the vehicle's external environment, speech from other occupants of the vehicle, events associated with the operation of the entertainment subsystem 222, network module 224, navigation subsystem 226, etc. In various embodiments, the input processing module 262 processes the incoming data, while the output generation module 264 streams the audio signal to the personal audio device 150 and the speakers 232(N) for the sound zone 402(1). In such instances, the input processing module 262(N) can continuously process the incoming data while establishing a communication link 312(N) with the personal audio device 150(N).

[0086] At step 704, the audio unit 116 determines whether to add an audio component to the audio signal based on the incoming data. In various embodiments, the input processing module 262 determines whether the incoming data indicates that an additional audio component is to be mixed with the audio signal. For example, the input processing module 262 receives a CAN message indicating an alarm condition from a driver assistance system. In such an instance, the input processing module 262 determines to provide the additional audio component based on the alarm condition. If the audio unit 116 determines to add an audio component, the audio unit 116 proceeds to step 606; otherwise, the audio unit 116 determines not to add an audio component and returns to step 602 to further process the incoming data.

[0087] At step 706, the audio unit 116 determines whether to add the audio included in the incoming data. In various embodiments, the input processing module 262 determines to add the audio content included in the incoming data. For example, the input processing module 262 responds to a CAN message from a driver assistance system by adding ambient sounds recorded by the vehicle sensors 214 of the external environment. In another example, the input processing module 262 detects a conversation between two or more vehicle occupants, including a user. In such an instance, the input processing module 262 determines to include speech from the vehicle occupants recorded by the occupant-facing sensors 216 and / or the cabin microphone 218. If the audio unit 116 determines to add audio content from the incoming data, the audio unit 116 proceeds to step 708. Otherwise, the audio unit 116 proceeds to step 710.

[0088] At step 708, the audio unit 116 extracts an audio segment from the incoming data. In various embodiments, the input processing module 262 identifies the audio segment to be included as additional audio content. In such instances, the input processing module 262 extracts the audio segment from the collection of incoming data. In some embodiments, the input processing module 262 further processes the audio segment. For example, when the audio segment is ambient sound of the environment, the input processing module 262 filters out portions of the ambient sound, such as by isolating specific sounds in the environment (e.g., emergency sirens, traffic notification sounds, etc.). In some embodiments, the input processing module 262 generates an audio loop from the audio segment so that a longer version of the audio segment is added to the audio signal.

[0089] At step 710, the audio unit 116 determines whether to generate an audio tone. In various embodiments, the input processing module 262 determines whether to generate an audio tone to mix with the audio signal. For example, the input processing module 262 may detect an event associated with an audio tone at step 604 (e.g., an alarm condition, a navigation event, a notification provided by the entertainment subsystem 222, etc.). In addition to audio content, an audio tone may also be generated (e.g., an alarm tone in addition to ambient sounds) or to indicate a non-audio event. When the audio unit 116 determines that an audio tone is to be generated, the audio unit 116 proceeds to step 712; otherwise, the audio unit 116 determines not to add an audio tone and proceeds to step 714.

[0090] At step 712, the audio unit 116 generates an audio tone corresponding to the determined event type. In various embodiments, after determining the event type to be indicated by the audio tone, the audio unit 116 identifies the audio tone corresponding to the event type. In some embodiments, the data store 118 stores a collection of tones corresponding to specific event types. For example, the data store 118 may store an air horn tone mapped to an alarm event type, a ringing alarm mapped to an incoming call event type, a harp string tone mapped to a navigation event type, and the like. In such instances, the output generation module 264 of the audio unit 116 generates the audio tone mapped to the determined event type.

[0091] At step 714, audio unit 116 generates a mixed audio stream with an additional audio component. In various embodiments, the output generation module 264 of audio unit 116 mixes additional audio components (for example, audio fragments and / or audio tones) with audio signals to generate mixed audio signals. In various embodiments, mixed audio signals comprise the attenuated version of the audio signal provided by audio source and the additional component. In this way, the loudspeaker in sound zone 402 reproduces the audio that provides the contextual information about vehicle environment during audio signal playback. In some embodiments, audio unit 116 mixes signals by significantly attenuating or mute the specific scope of audio signals. In this type of instance, the additional audio component is more prominent in the mixed audio signal, and the user is more likely to hear during playback.

[0092] At step 716, the audio unit 116 transmits the mixed audio signal to the sound zone for reproduction. In various embodiments, the output generation module 264 of the audio unit 116 transmits the mixed audio signal to the personal audio device 150(N) and the speaker 232(N) in the sound zone 402(1) for reproduction. In some embodiments, the output generation module 264 generates separate component signals from the mixed audio signal. In such instances, the audio unit 116 transmits the respective component signals to the devices in the sound zone 402(1) to reproduce the components of the mixed audio signal. After transmitting the mixed audio signal for reproduction, the audio unit 116 returns to step 602 to process other incoming data.

[0093] Figure 8 A flow chart illustrating a method 800 for hierarchically prioritizing incoming data to be processed by an audio unit is shown. Figure 7 As described in step 702 in .

[0094] As shown, the method 800 begins at step 802, where the audio unit 120 receives incoming data. In one or more embodiments, the input processing module 262 of the audio unit 120 receives data from various devices (e.g., audio source 104, sensor 106, I / O device 108, remote data source 114, personal audio device 110, entertainment subsystem 222, network module 224, and navigation subsystem 226). In one or more embodiments, the input processing module 262(N) continuously processes the incoming data while the communication link 312(N) is established with the personal audio device 110(N).

[0095] At step 804, the audio unit 120 organizes the incoming data into one or more matrices within which the incoming data is collected and organized. In the example described herein, the one or more matrices receiving the incoming data may be, but are not limited to, the seat media matrix block 522, the communication zone matrix block 524, and the voice sensing matrix block 526. For example, incoming data from the audio sources 104(a)-104(n) is organized in the seat media matrix block 522. In another example, audio data from the in-vehicle communication system 514 is organized in the communication zone matrix block 524.

[0096] At step 806, the audio unit also prioritizes the incoming data into multiple hierarchical levels, such as high priority, medium priority, and low priority. The incoming data is prioritized by assigning the incoming audio data into audio components, which can be added to the mixed audio signal 568 by one of the low priority mixer block 528, the medium priority mixer block 534, and the high priority mixer block 540. For example, the input processing module 262 receives incoming data in the form of an alert that an emergency vehicle has been detected. The emergency vehicle detection incoming data 518 is assigned to the high priority mixer block 540. In another example, the input processing module receives incoming data in the form of navigation prompts from the navigation subsystem 508. Navigation prompts are audio that can be added to the incoming data and are assigned to the medium priority mixer block 534.

[0097] Incoming data is selected from one or more of the matrices at step 808. The selected data is intended for playback on a personal audio device associated with a particular sound zone.

[0098] At step 810, the audio input unit 120 directs the prioritized incoming data from the high priority mixer block 540, the medium priority mixer block 534, and the low priority mixer block 528 as inputs to be mixed with the already organized signals from the corresponding matrix blocks 522, 524, 526. For example, the voice sensing incoming data from the voice sensing matrix block 526 is mixed at the voice sensing mixer with the medium priority mixing signal and / or the high priority mixing signal from the medium priority mixer block 534 and the high priority mixer block 540 according to its hierarchical priority sorting performed at the first hierarchical level. For example, the voice data from the voice sensing matrix block 526 will be mixed with the incoming audio from the medium priority mixer block 534 and the incoming audio from the low priority mixer block 528, the medium priority mixer block 534, and the high priority mixer block 540. Incoming data from the high priority mixer block 540 will take precedence over incoming audio from the medium priority mixer block 534 , and incoming data from the medium priority mixer block 534 will take precedence over incoming audio from the voice sensing matrix block 526 .

[0099] At step 812, the audio input unit 120 directs the incoming data selected from the matrix as input to the corresponding ducker block. For example, the incoming data from the seating media matrix block 522 is media-related and is therefore input to the media ducker block 546. The incoming data from the communication zone matrix block 524 is communication-related and is therefore input to the communication ducker block 556.

[0100] At step 814, the audio unit 120 directs the prioritized and mixed incoming data from the mixer to the corresponding ducker as a trigger. For example, the incoming data mixed at the voice sensing mixer block 542 triggers the media ducker block 550. The incoming data of voice sensing takes precedence over the media incoming data so that when the incoming data is to be combined and added to the mixed audio signal, the media incoming data is handed over to the voice signal. In another example, the incoming data output from the communication mixer block 544 is a trigger for the communication ducker block 558. The incoming data from the medium priority mixer block 534 and the high priority mixer block 540 is a trigger for the low priority ducker block 530. And the incoming data from the high priority mixer block 540 is a trigger for the medium priority ducker block 538.

[0101] At step 816, the audio unit 120 directs the mixed prioritized and mixed incoming data from the mixers to the corresponding duckers as inputs. The incoming data from the low priority mixer block 528 is the input to the low priority ducker block 532, and the incoming data from the medium priority mixer block 534 is the input to the medium priority ducker block 538.

[0102] In practice, the prioritized and blended incoming data from the high-priority mixer and the medium-priority mixer takes precedence over incoming data related to communications. For example, incoming data related to a detected emergency vehicle or navigation prompts would take precedence over incoming data related to communications between occupants or between occupants and a virtual assistant.

[0103] At step 818, the prioritized and mixed signals are ducked and attenuated when triggered according to the hierarchical prioritization. At this point, the ducked, attenuated and mixed incoming data is processed (e.g., Figure 7 ), to generate a mixed audio signal.

[0104] The audio unit of the present subject matter hierarchically prioritizes audio signals provided to one or more personal audio devices and the location of the personal audio devices within a vehicle. The audio unit detects a given personal audio device and the location of the personal audio device within the vehicle. The audio unit loads a stored profile for the personal audio device and / or a user associated with the personal audio device. The audio unit establishes a communication link with the personal audio device to provide an audio stream for playback by the personal audio device. In various embodiments, the audio unit determines an audio zone for the personal audio device based on the location of the personal audio device within the vehicle and manages one or more speakers in the sound zone, such as by muting the speakers or by providing portions of the audio stream to the speakers to reproduce the audio stream in conjunction with the personal audio device.

[0105] After streaming the audio signal to the personal audio device, the audio unit processes incoming data received from one or more vehicle components, such as audio data, sensor data, and / or other types of data. The audio unit processes the data to determine whether to add an audio portion to the audio stream. As the audio unit hierarchically prioritizes the audio portions and determines whether and when to add an audio portion, it extracts an audio clip from the incoming data or adds an audio tone notification. The audio unit mixes the additional audio portion with the audio stream to generate a mixed audio stream for transmission to the personal audio device and speakers in the sound zone.

[0106] At least one technical advantage of an integrated in-vehicle personal audio system over the prior art is that, utilizing the disclosed technology, the integrated in-vehicle personal audio system can provide high-fidelity audio reproduction of audio signals from within a vehicle while providing contextual information about the driving environment to the vehicle's occupants. This information is presented simultaneously, allowing critical safety alerts to be presented in parallel with less critical alerts and indicators, with the less critical alerts appropriately ducked or attenuated to those deemed more critical. Furthermore, by incorporating compatible personal audio components to generate personalized sound zones for a given occupant, the integrated in-vehicle personal audio system provides a less complex and more cost-effective solution for providing high-fidelity audio to an individual than conventional in-vehicle audio systems. These technical advantages provide one or more technical advances over prior art approaches.

[0107] Any and all combinations of any claim elements recited in any claim and / or any elements described in this application, in any manner, are within the intended scope of the invention and protection.

[0108] The description of the various embodiments has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0109] Aspects of the embodiments of the present invention may be embodied as systems, methods, or computer program products. Thus, aspects of the present disclosure may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or a combination of software and hardware implementations, which may be collectively referred to herein as a "module," "system," or "computer." In addition, any hardware and / or software technology, process, function, component, engine, module, or system described in this disclosure may be implemented as a circuit or a collection of circuits. In addition, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0110] Any combination of computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media would include the following: an electrical connection having one or more conductors, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash drive), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0111] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each frame in the flowchart and / or block diagram and the frame combinations in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine. Instructions enable the function / action specified in one or more frames of the flowchart and / or block diagram to be implemented when the processor of the computer or other programmable data processing device is executed. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, specific application processors or field programmable gate arrays.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, segment, or portion of a code that includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions mentioned in the boxes may not appear in the order mentioned in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially simultaneously, or the boxes may sometimes be executed in reverse order, depending on the functionality involved. It will also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified function or action, or a combination of dedicated hardware and computer instructions.

[0113] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the basic scope is determined by the following claims.

Claims

1. A computer-implemented method comprising: detecting, via an audio unit in a vehicle, one or more personal audio devices within the vehicle; establishing a link with at least a first personal audio device of the one or more personal audio devices; receiving incoming data from sources internal and external to the vehicle; Organizing incoming data from audio sources and vehicle system inputs into one or more matrices; hierarchically prioritizing the incoming data into one or more of a low priority mixer, a medium priority mixer, and a high priority mixer; mixing the organized incoming data with the prioritized incoming data and distributing the incoming data as inputs or triggers to one or more duckers; Attenuate and duck prioritized incoming data; generating a mixed audio signal combining the attenuated and ducked incoming data from each of the one or more duckers; as well as The mixed audio signal is transmitted over the link for playback by the first personal audio set.

2. The method of claim 1 , wherein the step of organizing incoming data from audio sources and vehicle systems into one or more matrices further comprises: Incoming data from sources within the vehicle is organized into a seat media matrix, a communication zone matrix, and a voice sensing matrix.

3. The method of claim 1 , wherein the step of mixing the organized incoming data with the prioritized incoming data further comprises: Mixing is performed in the voice sensing mixer and the communication mixer.

4. The method of claim 1 , wherein the step of assigning the incoming data as an input or trigger to each of the one or more duckers further comprises: The incoming data is distributed to one or more of a media ducker, a communication ducker, a low priority ducker, and a medium priority ducker.

5. The method of claim 1 , wherein the step of generating a mixed audio signal further comprises: A media mixer mixes signals from sources inside the vehicle, and an ambience mixer mixes signals from sources outside the vehicle.

6. The method of claim 5 , wherein the step of generating a mixed audio signal further comprises: Incoming data having a lower priority is attenuated and ducked in response to a trigger having a higher priority.

7. The method of claim 6, wherein the step of generating the mixed audio signal further comprises: A seating mixer mixes the signals from the media mixer and the ambiance mixer based on the priorities assigned to the inputs of the one or more duckers and the presence of triggers for the one or more duckers.

8. The method of claim 1 , wherein the step of transmitting the mixed audio signal further comprises: The mixed audio signal stimulates at least one of an audio playback device or a haptic actuator.

9. A system comprising: an audio unit in a vehicle, the audio unit having a processor and a memory for storing an audio management application, the audio unit detecting one or more personal audio devices in the vehicle; a link between the audio unit and at least a first personal audio device of the one or more personal audio devices; incoming data, the incoming data being received at the audio unit; The processor is coupled to the memory to execute the audio management application by performing the following steps: Organize incoming data into one or more matrices; hierarchically prioritizing the incoming data into one or more of a low priority mixer, a medium priority mixer, and a high priority mixer; blending the organized incoming data with the prioritized incoming data and assigning the blended incoming data as input or trigger to one or more duckers; mixing incoming data assigned to one or more duckers, wherein input to each of the one or more duckers is attenuated and ducked relative to a trigger to each of the one or more duckers; as well as A mixed audio signal generated by the audio management application, wherein the mixed audio signal is played back by the first personal audio set.

10. The system of claim 9, wherein the one or more matrices further comprise a seating media matrix, a communication zone matrix, and a voice sensing matrix.

11. The system of claim 9, further comprising: A voice sensing mixer and a communication mixer for mixing the organized incoming data with the prioritized incoming data.

12. The system of claim 9, further comprising: a media mixer for mixing signals from sources within the vehicle; and an ambience mixer for mixing signals from sources external to the vehicle.

13. The system of claim 12, further comprising: A seat mixer is provided for mixing signals from the media mixer and the ambiance mixer based on priorities assigned to the inputs of the one or more duckers and the presence of triggers for the one or more duckers.

14. The system of claim 9, further comprising: At least one of an audio playback device or a haptic actuator activated by the mixed audio signal being played back by the first personal audio device.

15. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: detecting, via an audio unit of the vehicle, one or more personal audio devices within the vehicle; establishing a link with at least one of the one or more personal audio devices; receiving incoming data from sources internal and external to the vehicle; Assigning hierarchical priorities to incoming data; generating a mixed audio signal combining prioritized incoming data sources, wherein lower hierarchical priority incoming data is ducked and attenuated below higher hierarchical priority incoming data; as well as The mixed audio signal is transmitted over the link for playback by the one or more personal audio devices.

16. The one or more non-transitory computer-readable media of claim 15, wherein the step of assigning a priority to the incoming data further comprises: assigning a first hierarchical priority level to incoming data from a plurality of sources and system inputs; blending incoming data from the plurality of sources and system inputs according to assigned priorities; assigning a second hierarchical priority level to incoming data and the mixed incoming data hierarchically prioritized at the first hierarchical priority level, the second hierarchical priority level taking precedence over the first hierarchical level and mixed with the incoming data from the plurality of sources and system inputs, wherein the incoming data at the first hierarchical priority level is ducked or attenuated below the incoming data prioritized at the second hierarchical level; as well as A third hierarchical priority level is assigned to incoming data and the mixed incoming data hierarchically prioritized at the second hierarchical priority level, wherein the incoming data at the third hierarchical priority level takes precedence over the incoming data at the second hierarchical level, wherein the incoming data at the second hierarchical level is dodged or attenuated to be lower than the incoming data at the third hierarchical level.

17. The one or more non-transitory computer-readable media of claim 15, wherein transmitting the mixed audio signal over the link for playback by the one or more personal audio devices further comprises: The mixed audio signal stimulates at least one of an audio playback device or a haptic actuator.

Images