Hybrid in-car speaker and headphone based acoustical augmented reality system

Abstract

To provide an appropriate hybrid in-car speaker and a headphone based acoustical augmented reality system.SOLUTION: A vehicle audio system 100 for a vehicle 102 may include a plurality of headsets 116 associated with a vehicle occupant and configured to transmit audio signals to an associated occupant; a plurality of interior microphones 118 configured to receive an interior audio signal from within a vehicle cabin; a plurality of exterior microphones 120 configured to receive an exterior audio signal from outside of the vehicle; and a processor 110 configured to receive at least one of the interior audio signals from the interior microphone 118 and the exterior audio signal from the exterior microphones 120, determine whether at least one of the interior audio signal and exterior audio signal includes a trigger command or an alert signal, and transmit in response to the audio signals including a trigger command or an alert signal the relevant audio signal to the headset 116.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 62 / 787,981, filed January 3, 2020, the disclosure of which is incorporated herein by reference in its entirety.

[0002] Disclosed herein is a hybrid in-car speaker and headphone based audio augmented reality system. [Background technology]

[0003] Vehicles are often equipped with a variety of infotainment systems, including in-car radios, screens for watching movies, headphones, spatial audio systems, etc. Often, vehicles have multiple occupants, each of whom wants to use a different infotainment system. However, existing loudspeaker technology in the vehicle can prevent content separation between the various occupants. Summary of the Invention [Means for solving the problem]

[0004] The in-vehicle audio system may include at least one internal microphone configured to receive an internal audio signal from within the vehicle cabin, at least one vehicle speaker disposed within the vehicle cabin and configured to transmit an audio signal to a driver, at least one first headset associated with a vehicle occupant and configured to transmit an audio signal to the associated occupant, at least one external microphone configured to receive an external audio signal obtained from outside the vehicle, and a processor programmed to: receive an internal audio signal from the at least one internal microphone, receive an external audio signal from the at least one external microphone, transmit the internal audio signal to the at least one vehicle speaker and to the at least one internal microphone to facilitate conversation between the driver and the at least one occupant, determine whether the external audio signal includes an alert signal, and in response to an external audio signal including the alert signal, stop transmitting the internal audio signal to the vehicle speaker and transmit the alert signal to the vehicle speaker.

[0005] The in-vehicle audio system can include a first headset associated with a first vehicle occupant and configured to transmit an audio signal to the first vehicle occupant, at least one external microphone configured to receive an external audio signal obtained from outside the vehicle, at least one vehicle speaker configured to transmit the audio signal, and a processor programmed to: transmit an internal audio signal obtained from within the vehicle to the at least one vehicle speaker, receive the external audio signal from the at least one external microphone, determine whether the external audio signal includes an alert signal, and in response to the external audio signal including the alert signal, interrupt the internal audio signal at the vehicle speaker and transmit the external audio signal.

[0006] An in-vehicle audio method may include receiving an internal audio signal from at least one internal microphone; receiving at least one trigger command including an identification of to which of a subset of a plurality of headsets and speakers, each associated with a vehicle occupant, the internal audio signal is to be transmitted; and in response to the trigger command, transmitting the internal audio signal to the subset of headsets and speakers, at least one of the subset of headsets and speakers being an in-vehicle speaker. The present specification also provides, for example, the following items: (Item 1) An in-car audio system, at least one internal microphone configured to receive an internal audio signal from within the vehicle cabin; at least one vehicle speaker disposed within the vehicle cabin and configured to transmit an audio signal to a driver; at least one first headset associated with a first vehicle occupant and configured to transmit an audio signal to the associated occupant; at least one external microphone configured to receive an external audio signal captured from outside the vehicle; 1. A processor comprising: receiving the internal audio signal from the at least one internal microphone and receiving the external audio signal from the at least one external microphone; transmitting the internal audio signal to the at least one vehicle speaker and to the at least one internal microphone to facilitate communication between the driver and at least one passenger; determining whether the external audio signal includes an alert signal; in response to the external audio signal including the alert signal, ceasing transmission of the internal audio signal to the vehicle speaker and transmitting the alert signal to the vehicle speaker; the processor being programmed to The in-vehicle audio system. (Item 2) Item 1 , wherein the alert signal comprises an audible sound related to an emergency situation. (Item 3) Item 1 . The system of item 1 , wherein the alert signal includes an audible siren. (Item 4) Item 1 . The system of item 1 , further comprising a second headset associated with a second vehicle occupant. (Item 5) 5. The system of claim 4, wherein the processor is programmed to transmit the internal audio signal to the first headset and the second headset. (Item 6) 2. The system of claim 1, wherein the processor is further programmed to receive a trigger command, the first headset includes a plurality of headsets, and the trigger command indicates which of the plurality of headsets is to receive the internal audio signal. (Item 7) 7. The system of claim 6, wherein the trigger command is an audible command. (Item 8) An in-car audio system, a first headset associated with a first vehicle occupant and configured to transmit an audio signal to the first vehicle occupant; at least one external microphone configured to receive an external audio signal captured from outside the vehicle; at least one vehicle speaker configured to transmit an audio signal; 1. A processor comprising: transmitting an internal audio signal obtained from within the vehicle to at least one vehicle speaker; receiving the external audio signal from the at least one external microphone; determining whether the external audio signal includes an alert signal; interrupting the internal audio signal and transmitting the external audio signal through the vehicle speaker in response to the external audio signal including the alert signal. the processor being programmed to The in-vehicle audio system. (Item 9) 9. The system of claim 8, wherein the alert signal comprises an audible sound related to an emergency situation. (Item 10) 9. The system of claim 8, wherein the alert signal comprises an audible siren. (Item 11) 9. The system of claim 8, wherein the processor is further programmed to receive a plurality of known alert sounds and compare the external audio signal to the known alert sounds to determine whether the external audio signal includes an alert signal. (Item 12) 9. The system of claim 8, further comprising at least one second headset associated with a second vehicle occupant different from the first vehicle occupant. (Item 13) Item 13. The system of item 12, wherein the first headset is associated with a driver of a vehicle, and the processor is further programmed to transmit the external audio signal only to the first headset associated with the driver and not to the at least one second headset. (Item 14) 1. An in-car audio method, comprising: receiving an internal audio signal from at least one internal microphone; receiving at least one trigger command including an identification of to which of a subset of a plurality of headsets and speakers the internal audio signal is to be transmitted, each of the headsets and speakers being associated with a vehicle occupant; in response to the trigger command, transmitting the internal audio signal to the subset of headsets and speakers, at least one of the subset of headsets and speakers being a vehicle speaker; The in-car audio method comprising: (Item 15) Item 15. The method of item 14, wherein the trigger command comprises a command received via a human machine interface (HMI). (Item 16) Item 15. The method of item 14, wherein the trigger command is an audible command. (Item 17) Item 15. The method of item 14, wherein the internal audio signal includes a voice signal obtained from the at least one internal microphone. (Item 18) Item 15. The method of item 14, further comprising continuously transmitting media signals to other headsets not included in the subset of headsets. (Item 19) 16. The method of claim 15, wherein the trigger command identifies a subset of occupants that will receive the internal audio signal.

[0007] Embodiments of the present disclosure are pointed out with particularity in the appended claims, however other features of the various embodiments will become more apparent and will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] 1 illustrates an exemplary video audio system for a vehicle. [Diagram 2] 2 shows an example block diagram of the audio system of FIG. 1; [Diagram 3] 1 illustrates an example process for an audio system that allows external sounds to interrupt a driver's current content. [Figure 4]1 illustrates an example process for an audio system where an interior sound can interrupt one of the occupants' current contents. [Diagram 5] 2 shows an alternative view of the system of FIG. 1; [Figure 6] 2 shows an alternative view of the system of FIG. 1; [Figure 7] 2 shows an alternative view of the system of FIG. 1; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Where necessary, detailed embodiments of the present invention are disclosed herein, but it will be understood that the disclosed embodiments are merely examples of the invention that may be embodied in various and alternative forms. The figures are not necessarily to scale, and some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to variously use the present invention.

[0010] Disclosed herein is a vehicle audio system that includes a network of inward-facing cabin microphones and outward-facing interior environment microphones to provide an audio augmented reality system for enhanced communication and increased awareness of environmental factors. The system may include a number or array of microphones within the vehicle to capture audio. Approximately one microphone or microphone array per passenger seat may be included. External microphones may capture important acoustic signatures from the external environment. Each passenger may use a set of personal headphones. The driver may enjoy audio through the vehicle speakers. The vehicle's human machine interface (HMI) or audio processing unit may control user inputs and preferences and map microphone signals to various headphone and microphone outputs.

[0011] The system also includes an adaptive system to detect emergency sound signatures (e.g., ambulance sirens) to assist the driver in awareness and cognitive driving tasks. The system can increase the driver's awareness of external events, such as playing real-time environmental sounds to avoid accidents and increase awareness. The system can also enable private conversations between the two occupants, personalized media for each occupant, and parental monitoring of each occupant's conversations. The system can layer these functions to provide a flexible and robust system that handles many types of user interactions. The system can further enable a high-performance audio system that rivals the current industry best, and the system can be fully optimized for driver seat optimization. This can allow the system to provide the best listening experience for the driver, rather than tuning appropriately for every seat.

[0012] While the driver listens to the vehicle speakers, the remaining passengers may wear headphones to create private sound zones with each passenger's selected media and content. A typical vehicle audio system may rely on standard speaker technology to distribute acoustic energy throughout the vehicle cabin, which may prevent content separation between different occupants. Headphones may help solve this problem, but they may also prevent external sounds from being perceived. Additionally, headphones may hinder natural conversation between occupants.

[0013] In the disclosed system, since each passenger relies on passive acoustic separation via their headphones, performance expectations can be much better compared to state of the art "active controller" solutions such as Individual Sound Zones (ISZ). With In-Car Communication (ICC), passengers can participate in a very natural conversation even though they are acoustically separated. The driver uses the existing speaker system, making the most of the shared expertise in car tuning and system architecture design, allowing the driver to achieve a vast and expansive sound experience. A certain amount of "bleed-through" from the driver's audio content can leak through other passengers' headphones. To negate some of this, an integrated noise cancellation algorithm is incorporated to reduce low frequency propagation at all passenger seats. Thus, the headphones provide a high degree of passive contrast (cancellation) for all mid-range and high frequencies, while the system can cancel low frequencies at a more targeted rate. As this system offers more degrees of freedom, it provides better and more robust separation between passenger sound zones, ultimately providing a cleaner and less distracting listening experience for all passengers.

[0014] FIG. 1 illustrates an exemplary vehicle audio system 100 for a vehicle 102. The system 100 may include a vehicle 102 and a remote network 106. The vehicle 102 may include a processor 110 disposed within the vehicle. The processor 110 may be included in a vehicle head unit or a vehicle ECU. The processor 110 may include various vehicle systems, such as navigation, infotainment, and autonomous vehicle systems. The processor 110 may include an audio system processor 122 (shown in FIG. 2) configured to determine when and where to play a particular audio signal within the vehicle 102. The vehicle processor 110 and / or the audio system processor 122 may each include a controller (not shown) configured to control various vehicle systems and devices. The processors 110, 122 may be interchangeable in that each may process any of the processes described herein.

[0015] The vehicle 102 may include multiple interior infotainment devices 112. The infotainment devices 112 may be devices configured to present content to a user within the vehicle cabin, either audibly, visually, tactilely, etc. For example, the infotainment devices 112 may include vehicle speakers configured to present audible sounds within the vehicle 102. The infotainment devices 112 may include displays or projectors configured to present information visually. One or more of these devices may comprise the infotainment device 112. For example, the infotainment devices 112 may include devices that provide both audio and video information (e.g., both speakers and displays). Although the vehicle speakers 114 are shown separately, they may also be included in the infotainment device 112. The speakers 114 may emit audible sounds intended to be heard by the driver.

[0016] Infotainment device 112 may present content to a vehicle occupant, such as media content including music, video, and the like. The content may include various forms of infotainment, alerts, and the like. In one example, infotainment device 112 may display and play audio content requested by a vehicle occupant with a particular radio station, artist, song, content, and the like. In another example, the information may present requested navigation commands defined by a driving route to a desired destination.

[0017] The infotainment devices 112 may be located in various locations throughout the vehicle 102. In the example shown in FIG. 1, the first infotainment device 112a is located in the vehicle's head unit. The second infotainment device 112b and the third infotainment device 112c are located in the rear headrests and configured to be visible to the rear passengers. These locations are merely examples. Additionally, the infotainment devices 112 may be each passenger's portable personal device, such as the passenger's phone or tablet. The vehicle speakers 114 may be included in the infotainment device 112 or may function as additional infotainment devices 112. The infotainment devices 112 and the vehicle speakers 114 may be used interchangeably herein and may be configured to primarily provide high quality audio to the vehicle driver. More or fewer infotainment devices 112 may be included in the vehicle.

[0018] Each passenger may be associated with a headset 116. The headset 116 may include at least one set of headphones configured to be worn by each passenger so that each passenger can enjoy media content without disturbing other passengers. The headset 116 may also be configured to selectively emit sounds, such as the voices of other passengers. The headset 116 may be a wireless headset connected to a user device via wireless communication. The headset 116 may also be connected to the infotainment device 112 or other user devices via a wired connection. The headset 116 may include a microphone configured to capture noises, such as voices. The processor 110 may control the output to each headset 116, including various signal processing, inputs, preferences, etc., to improve the quality of the audio output. FIG. 1 illustrates a headset associated with each seat position, including headsets 116a, 116b, 116c (collectively referred to as headsets 116). Notably, the headsets are not associated with the driver.

[0019] The vehicle 102 may also include multiple interior microphones 118a, 118b, 118c, 118d (collectively referred to as interior microphones 118). The interior microphones 118 may face the interior of the vehicle 102 and may be configured to capture ambient noise, voices, and other acoustic events. Interior audio signals captured by the interior microphones 118 may be transmitted to the passenger headsets 116. Approximately one interior microphone 118 may be included at each passenger seat to capture audio signals from each passenger. The interior microphones 118 may include a microphone array and may include more or fewer microphones. In the example of FIG. 1, a one-to-one ratio of the interior microphones 118 and the headsets 116 is shown, but higher or lower ratios may be appreciated. Additionally, the microphones 114 may include a microphone array. Often, such microphones are located in the headliner or seat back.

[0020] The vehicle 102 may also include multiple external microphones 120a, 120b, 120c, 120d (collectively referred to as external microphones 120) facing the exterior of the vehicle 102 and configured to capture sounds outside the vehicle, such as emergency vehicle sirens, traffic announcements, and other acoustic events. The external microphones 120 may be located at various locations around the vehicle 102. In the example shown in FIG. 1, a left external microphone 120a, a front external microphone 120b, a right external microphone 120c, and a rear external microphone 120d may be included. More or fewer microphones may be included, and the locations of the microphones in FIG. 1 are exemplary configurations.

[0021] The processor 122 can receive external audio signals from at least one external microphone and determine whether those sounds should be passed to one or more of the vehicle occupants. For example, if an occupant, particularly the driver, is listening to music through a headset, the processor 122 can determine that a siren from an emergency vehicle should be passed to the driver and emit the sound through the driver's headset 116a.

[0022] Although not shown, the vehicle 102 may include a connected vehicle system, including one or more systems facilitated through connected car or connected vehicle telematics. These systems may include features available on another mobile device, typically the driver's mobile device. The processor 110 of the vehicle 102 may wirelessly communicate with the mobile device (not shown) to access data in the mobile device, such as the driver's calendar, navigation system, GPS antenna, etc. The data may include data specific and local to the driver's mobile device, such as music, photos, etc. The data may also include dynamic data provided from external sources, such as weather information, traffic information, etc. This data may pertain to cases of general interest to the driver of the vehicle and may indicate to the processor 122 that information should be passed to the driver. In addition to or instead of the connected vehicle system, the processor 110 may receive data from the network 106, other mobile devices, etc.

[0023] FIG. 2 illustrates an example block diagram of the vehicle audio system 100 of FIG. 1. The processor 122 can receive audio signals from the external microphone 120 and the internal microphone 118. The processor 122 can determine whether these signals should be passed to one or more of the vehicle occupants. In one embodiment, the processor 122 can determine whether the external audio signal includes an alert, such as a siren. If so, the processor 122 can pass this sound to the vehicle speaker 114 but not to the passenger headset 116. In another embodiment, the processor 122 can determine that a private conversation is taking place between the driver and the front seat passenger. In this embodiment, the audio signal captured by the internal microphone 118 is passed to the first and second headsets 116a, 116b but not to the remaining headsets 116c, 116d. In this embodiment, the rear seat passengers can enjoy their respective media content without interruption.

[0024] In yet another example, a parent passenger may choose to listen to and monitor conversations between child passengers. The parent may choose to hear each passenger's voice via the infotainment device 112 or HMI. In this example, audio signals captured from the internal microphone 118 associated with that passenger may be passed to the parent headset 116 and / or driver speaker 114a.

[0025] In an example where the occupant selects to listen to internal audio picked up by the internal microphone 118, the occupant may make such selection on the infotainment device 112 or another device, such as their own smartphone, tablet, etc. This may be done via a touch screen on either the infotainment device 112, the vehicle display, or the occupant's personal device. In another example, the selection may be made audibly by giving a voice command. For example, the occupant may say "listen to rear passenger conversation" or "start a private conversation with the driver."

[0026] 3 illustrates an example process 300 for the audio system 100 where an external sound may interrupt a driver's current content. The process 300 may begin at block 305 where the processor 122 receives an external audio signal from the external microphone 120.

[0027] In block 310, the processor 122 may determine whether the external audio signal includes an alert. The processor may determine this by analyzing the audio signal for sounds that resemble a particular alert, such as a siren or announcement. The sound is classified as an alert signal in response to the processor 122 recognizing it as a sound typically heard in an emergency situation. The processor 122 may be in communication with a database of known sounds, whereby certain characteristics of the audio signal are compared to characteristics of known alert sounds. Additionally or alternatively, the processor 122 may include a self-learning and adaptive capability to learn certain sounds.

[0028] If the processor 122 determines that the external audio signal includes an alert sound, the process proceeds to block 315. Otherwise, the process 300 proceeds to block 305.

[0029] In block 315, the processor 122 may send the external audio signal to one of the vehicle speakers. In this example, the external audio signal may be sent only to the driver's vehicle speaker 114 and not to the passenger headset 116. The processor 122 may return the vehicle speaker 114 to its normal content once the alert sound is no longer discernible in the external audio signal. The process 300 may then end.

[0030] 4 illustrates an example process 400 of the audio system 100 in which an interior sound may interrupt one of the occupant's current contents. The process 400 may begin at block 405 where the processor 122 receives an internal audio signal from the internal microphone 118.

[0031] At block 410, the processor 122 may determine whether a trigger command has been received from one of the occupants. The trigger command may include a touch screen initiated command at the infotainment device 112 or an audible command recognized by the internal microphone 118. Other forms of trigger commands may also be included, such as an audible trigger command, such as a trigger word, that when received by the internal microphone 118 may be recognized as a trigger command by the processor 122. For example, the driver may say, "Enter conversation mode with front passenger." If a trigger command has been received, the process 400 proceeds to block 415. If not, the process 400 proceeds to block 405.

[0032] In block 415, the processor 122 may transmit the selected audio signal indicated by the trigger command to the associated headset. That is, if the passenger desires to converse with the driver, the audio signals detected by the internal microphones 118 associated with the passenger and the driver are transmitted to the other respective headsets so that the conversation can be heard in each headset. Process 400 may then end.

[0033] Both process 300 and process 400 may occur simultaneously. Processor 122 may maintain a hierarchy of commands, and in some embodiments, external audio signals, including alerts, may take precedence over occupant made commands regarding internal audio signals (e.g., a siren may be sent to the driver even if the driver chooses to talk to the occupant).

[0034] Figure 5 shows an alternative view of the system 100 of Figure 1. As in Figure 1, multiple microphones and speakers are located inside and outside the vehicle. In this example, the interior speakers 114 are specifically labeled as a first speaker 114a, a second speaker 114b, a third speaker 114c, and a fourth speaker 114d (collectively referred to as speakers 114). Although the vehicle audio system may use only four channels and four speakers, this is merely exemplary and other numbers of channels and speakers may be included.

[0035] The processor 122 may be a digital signal processor configured to mix, tune, and amplify the audio signal and provide the audio signal to the speaker 114 and the headset 116. The left front passenger mixer 144 may be located in the vehicle 102. Although shown near the driver's seat, the mixer 144 may be located anywhere in the vehicle and configured to generate audio signals for the driver's listening experience. For example, the mixer 144 may generate high quality audio signals based on an external audio signal to emit an alert noise to the driver. The mixer 144 may also generate high quality music audio based on active noise cancellation for the driver at the driver's specific location in the vehicle. Since other passengers are likely receiving audio signals via their respective headsets 116, the audio signals generated by the vehicle speaker 114 may be focused on the driver's location and preferences to create a driver-specific listening experience.

[0036] Figure 6 shows an alternative view of the system 100 of Figure 1. In this view, a mixer is located at the left front passenger or driver's position, similar to that of Figure 5. The remaining passenger positions are associated with headphones. In this example, the driver can receive an external audio signal, including a specific alert signal, as described above. The remaining passengers can enjoy the audio signal through their respective headsets 116 without being disturbed by the alert signal.

[0037] FIG. 7 shows an alternative diagram of the system 100 of FIG. 1. In this diagram, the driver and front passenger may have a private conversation. The rear passenger may be acoustically isolated. In the example of FIG. 7, a first microphone 118a may pick up an audio signal (e.g., a voice signal) originating from the driver. A second microphone 118b may pick up an audio signal from the front passenger. The processor 122 may perform digital processing on each signal and transmit an audio signal from the front passenger to the driver speaker 114a and transmit an audio signal from the driver to a first headset 116a associated with the front passenger. Thus, the front passenger may receive audio related to a conversation with the driver through his or her headset, while the driver may receive the audio at the vehicle speaker 114. Thus, the front passenger may carry on a conversation while the rear passenger is acoustically isolated.

[0038] Notably, Figures 6 and 7 are not mutually exclusive examples. If an alert signal is detected by the external microphone 120, the alert signal may be passed to the driver via the speaker 114. That is, the driver's audio or conversation may be interrupted by an incoming alert, such as a siren from an emergency vehicle.

[0039] Computing devices described herein generally include computer-executable instructions, which may be executable by one or more computing devices or hardware devices, such as those listed above. Computer-executable instructions may be compiled or interpreted from computer programs written using a variety of programming languages ​​and / or technologies, either alone or in combination, such as, but not limited to, Java, C, C++, Visual Basic, Java Script, Perl, etc. Generally, a processor (e.g., a microprocessor) receives instructions, for example, from a memory, a computer-readable medium, etc., and executes these instructions to thereby perform one or more processes, including one or more of the processes described herein. Such instructions and other data may be stored and transmitted using a variety of computer-readable media.

[0040] Although exemplary embodiments have been described above, it is not intended that these embodiments describe every possible form of the invention. Rather, the words used herein are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the invention. Moreover, features of various implementing embodiments may be combined to form further embodiments of the invention.

Claims

[Claim 1] The invention described in this specification.

Images