Audio stream mixing system and method

Abstract

A system and method for mixing a second audio stream with a first audio stream in an audio output device is provided. The system is configured to perform the method, which includes: buffering the first audio stream and outputting the first audio stream as an unmodified output via the audio output device; determining at least one insertion point within the first audio stream; modifying the first audio stream at the insertion point to avoid content loss; outputting the second audio stream at the insertion point; and resuming the unmodified output of the first audio stream at or near the completion of the second audio stream. Modifying the first audio stream may include pausing and / or distorting the first audio stream at the insertion point. The audio output device may be a vehicle head unit or a wireless device, such as a mobile phone.

Description

Technical Field

[0001] The inventive concept relates to the field of audio stream processing, and more particularly to combining multiple audio output streams. Background Art

[0002] More and more audio sources are competing for the audio channels of user devices. For example, in a vehicle, multiple different audio sources may compete for the attention of the driver and passengers. In a car, as perhaps the most common example, competing audio devices may include, but are not limited to:

[0003] ·radio

[0004] Stream music, audio books, and other media

[0005] Car Assistant

[0006] Navigation system

[0007] Car Condition and Safety Notices

[0008] In the event that an announcement, such as from a car, assistant, or navigation system, must be played, any continuous stream from the infotainment system may be lost. Outgoing infotainment streams may include music and / or talk, and may originate from any of a number of channels or media, such as radio, mobile device, music, vehicle hard drive, compact disc (CD), and universal serial bus (USB), etc.

[0009] In the event that a second audio stream, such as an announcement from a vehicle or navigation system, is received, the first (e.g., infotainment) stream is typically muted or suppressed, for example by setting the output volume of the infotainment stream to 0 or close to 0. In this case, the muted or suppressed infotainment audio output may be completely lost. As a result, for example, the listener may miss information from a radio news program or from an audio book that she / he is listening to the infotainment stream. This may greatly degrade the infotainment experience for all listeners.

[0010] In addition, announcements are often output without regard to the first (eg, infotainment) stream output. As a result, announcements may appear in the middle of words or sentences, which may further reduce the consumption of infotainment content by all listeners. Summary of the invention

[0011] According to aspects of the inventive concept, a method for mixing a second audio stream with a first audio stream in an audio output device is provided, the method comprising: buffering the first audio stream; determining an insertion point within the first audio stream; outputting the buffered first audio stream, including modifying the output of at least a portion of the first audio stream at the insertion point to accommodate the output of the second audio stream, thereby minimizing or avoiding degradation or loss of content of the first audio stream; outputting the second audio stream at the insertion point; and continuing to output the buffered first audio stream after the second audio stream is completed.

[0012] In various embodiments, the second audio stream is received during output of the first audio stream.

[0013] In various embodiments, buffering of the first audio stream occurs in response to receiving the second audio stream.

[0014] In various embodiments, the first audio stream is a radio stream.

[0015] In various embodiments, the first audio stream is a playback of content from a tangible storage medium local to the audio output device.

[0016] In various embodiments, the tangible storage medium is an optical disk, a UBus medium, a hard drive, or a computer memory.

[0017] In various embodiments, the second audio stream is received by the audio output device together with an emergency identifier or a maximum delay identifier.

[0018] In various embodiments, the method includes identifying a maximum delay for output of the second audio stream based on the urgency identifier or the maximum delay identifier.

[0019] In various embodiments, the method further includes determining the insertion point based at least in part on the urgency identifier or the maximum delay identifier.

[0020] In various embodiments, determining the insertion point comprises finding a gap or pause within the first audio stream.

[0021] In various embodiments, determining the insertion point includes analyzing the first audio stream using one or more speech analysis techniques to find the end of sentences, phrases, words, or other natural breaking points.

[0022] In various embodiments, the one or more speech analysis techniques include at least one of voice activity detection (VAD), automatic speech recognition (ASR), and natural language understanding (NLU).

[0023] In various embodiments, determining at least one insertion point includes looking for a low volume level within the first audio stream.

[0024] In various embodiments, modifying the first audio stream at the insertion point includes pausing the first audio stream at the insertion point.

[0025] In various embodiments, modifying the first audio stream at the insertion point comprises modifying a time-frequency structure of the first audio stream.

[0026] In various embodiments, modifying the first audio stream at the insertion point includes warping the first audio stream at or near the insertion point.

[0027] In various embodiments, outputting the second audio stream at the insertion point includes modifying the second audio stream.

[0028] In various embodiments, modifying the second audio stream includes modifying a time-frequency structure of the second audio stream.

[0029] In various embodiments, modifying the second audio stream includes modifying a voice style of the second audio stream to improve intelligibility relative to the first audio stream.

[0030] In various embodiments, the audio output device is part of or forms part of the vehicle head unit.

[0031] In various embodiments, the second audio stream is an announcement or warning from a vehicle navigation system, a vehicle monitoring system, or a text-to-speech system.

[0032] In various embodiments, the second audio stream is an announcement or alert from an advertising system.

[0033] In various embodiments, the audio output device is a wireless portable device including a mobile phone, a tablet computer, or a phablet.

[0034] According to another aspect of the inventive concept, an audio stream mixing system is provided, comprising: one or more processors coupled to one or more computer storage devices, one or more first audio stream sources, one or more second audio stream sources, and one or more audio output devices. The one or more processors are configured to: buffer a first audio stream from a first audio stream source in the one or more computer storage devices; determine an insertion point within the first audio stream; output the buffered first audio stream via the one or more audio output devices, wherein the one or more processors are configured to modify the output of at least a portion of the first audio stream at the insertion point to accommodate the output of the second audio stream, thereby minimizing or avoiding degradation or loss of content of the first audio stream, the second audio stream being received from the second audio stream source; output the second audio stream via the one or more audio output devices at the insertion point; and continue to output the buffered first audio stream via the one or more audio output devices after the second audio stream is completed.

[0035] In various embodiments, the second audio stream is received during output of the first audio stream.

[0036] In various embodiments, the one or more processors are configured to buffer the first audio stream in response to receiving the second audio stream.

[0037] In various embodiments, the first audio stream is a radio stream.

[0038] In various embodiments, the first audio stream is a playback of content from a tangible storage medium local to the audio output device.

[0039] In various embodiments, the tangible storage medium is an optical disk, a UBus medium, a hard drive, or a computer memory.

[0040] In various embodiments, the second audio stream is received by the audio output device together with an emergency identifier or a maximum delay identifier.

[0041] In various embodiments, the one or more processors are configured to identify a maximum delay for output of the second audio stream based on the urgency identifier or the maximum delay identifier.

[0042] In various embodiments, the one or more processors are configured to determine the insertion point based at least in part on the urgency identifier or the maximum delay identifier.

[0043] In various embodiments, the one or more processors are configured to look for a gap or pause within the first audio stream to determine the insertion point.

[0044] In various embodiments, the one or more processors are configured to analyze the first audio stream using one or more speech analysis techniques to find the end of sentences, phrases, words or other natural breaking points to determine the insertion points.

[0045] In various embodiments, the one or more speech analysis techniques include at least one of voice activity detection (VAD), automatic speech recognition (ASR), and natural language understanding (NLU).

[0046] In various embodiments, the one or more processors are configured to look for a low volume level within the first audio stream to determine the insertion point.

[0047] In various embodiments, the one or more processors are configured to pause the first audio stream at the insertion point as a modification to the first audio stream.

[0048] In various embodiments, the one or more processors are configured to modify a time-frequency structure of the first audio stream as a modification of the first audio stream.

[0049] In various embodiments, the one or more processors are configured to warp the first audio stream at or near the insertion point as a modification to the first audio stream.

[0050] In various embodiments, the one or more processors are configured to modify the second audio stream and output the second audio stream at the insertion point as a modified second audio stream.

[0051] In various embodiments, the modified second audio stream comprises a modified time-frequency structure.

[0052] In various embodiments, the one or more processors are configured to modify the voice style of the second audio stream to improve intelligibility relative to the first audio stream.

[0053] In various embodiments, the audio output device is part of or forms part of the vehicle head unit.

[0054] In various embodiments, the second audio stream is an announcement or warning from a vehicle navigation system, a vehicle monitoring system, or a text-to-speech system.

[0055] In various embodiments, the second audio stream is an announcement or alert from an advertising system.

[0056] In various embodiments, the audio output device is a wireless portable device including a mobile phone, a tablet computer, or a phablet. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention will become more apparent in view of the accompanying drawings and the accompanying detailed description. The embodiments depicted therein are provided by way of example and not limitation, wherein like reference numerals refer to like or similar elements. The drawings are not necessarily drawn to scale, but rather emphasize illustrating aspects of the present invention. In the drawings:

[0058] Figure 1 is a block diagram of an embodiment of an architecture that can perform audio stream mixing according to aspects of the inventive concepts;

[0059] Figure 2 is a block diagram of an embodiment of an audio stream mixing system according to aspects of the inventive concept;

[0060] Figure 3 According to the aspects of the inventive concept, Figure 2 Embodiment of the method for audio stream mixing performed by the system;

[0061] Figure 4A is an embodiment of a first audio stream;

[0062] Figure 4B is modified to accommodate a second audio stream according to aspects of the inventive concept Figure 4A An embodiment of a first audio stream;

[0063] Figure 4C is modified to accommodate a second audio stream according to aspects of the inventive concept Figure 4A Another embodiment of the first audio stream;

[0064] Figure 4D is modified to accommodate a second audio stream according to aspects of the inventive concept Figure 4A A further embodiment of the first audio stream;

[0065] Figure 5 is an embodiment of a first audio stream;

[0066] Figure 6 is modified to accommodate a second audio stream according to aspects of the inventive concept Figure 5 An embodiment of a first audio stream;

[0067] Figure 7 is modified to accommodate a second audio stream according to aspects of the inventive concept Figure 5 Another embodiment of the first audio stream. DETAILED DESCRIPTION

[0068] Various aspects of the inventive concept will be described more fully below with reference to the accompanying drawings, in which some typical embodiments are shown. However, the inventive concept may be embodied in many different forms and should not be construed as limited to the typical embodiments set forth herein.

[0069] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used to distinguish one element from another element, but do not mean a desired sequence of elements. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present invention. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. The term "or" is not used in an exclusive sense, but in an inclusive sense.

[0070] It should be understood that when an element is referred to as being "located on" or "connected to" or "coupled to" another element, the element may be directly located on, connected to, or coupled to the other element, or intervening elements may be present. Conversely, when an element is referred to as being "directly located on" or "directly connected to" or "directly coupled to" another element, there are no intervening elements. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., "located between" versus "directly between," "adjacent to" versus "directly adjacent to," etc.).

[0071] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "comprises", "comprising", "includes", and / or "including" when used herein specify the presence of the features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0072] Spatially relative terms such as "under," "beneath," "lower," "above," "upper," etc. may be used to describe the relationship of an element and / or feature to another element and / or feature, for example, as shown in the figures. It should be understood that the spatially relative terms are intended to encompass different orientations of the device in use and / or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is flipped, elements described as "under other elements or features" and / or "under other elements or features" will be oriented "above other elements or features." The device may be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein are interpreted accordingly.

[0073] Exemplary embodiments are described herein with reference to cross-sectional illustrations that are schematic illustrations of idealized exemplary embodiments (and intermediate structures). As such, variations from the shapes of the figures that result, for example, from manufacturing techniques and / or tolerances, are anticipated. Thus, the exemplary embodiments should not be construed as limited to the particular shapes of the regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing.

[0074] To the extent that functional features, operations and / or steps are described herein or otherwise understood to be included in various embodiments of the inventive concept, such functional features, operations and / or steps may be embodied in functional blocks, units, modules, operations and / or methods. And to the extent that such functional blocks, units, modules, operations and / or methods include computer program code, such computer program code may be stored in a computer-readable medium such as a non-transitory memory and medium that can be executed by at least one computer processor.

[0075] According to aspects of the inventive concept, system and method behaviors are more convenient for a user (e.g., a driver and / or passenger of a vehicle) and reduce or eliminate loss of content in a first (e.g., infotainment) audio stream due to interrupting announcements or warnings as a second audio stream.

[0076] Figure 1 1 is a block diagram of an embodiment of an architecture 100 that may perform audio stream mixing according to aspects of the inventive concepts. Figure 1 In the overall architecture 100, various systems and devices can communicate via one or more wired and / or wireless communication systems, networks and / or channels (collectively represented by cloud 10). The communication system can include the Internet, global positioning system (GPS), cellular network, Wi-Fi, Bluetooth, etc.

[0077] exist Figure 1 In the embodiment, vehicles are generally represented as cars and trucks 40. Although the present invention will generally be described in the context of vehicles, the inventive concept can be applied to any of a variety of vehicles that include audio output devices (e.g., airplanes, bicycles, trains, etc.). In addition, in some embodiments, the inventive concept can be applied to other types of audio output devices, which can include but are not limited to mobile phones, tablet computers, tablet phones, personal computers, laptop computers, and televisions (including smart TVs), etc. (in Figure 1 Where the description herein refers to a vehicle or a vehicle head unit, it should be understood that the same description may also apply to other types of audio output devices (such as the above-mentioned audio output devices, etc.) unless otherwise noted.

[0078] In general, the phrase "audio output device" as used herein means an electronic device configured to output at least a first audio stream and a second audio stream as an interruption warning or announcement. The first audio stream may provide any of various types of content, such as infotainment, etc. The second audio stream includes content to be output during the output of the first audio stream, which may result in the loss or significant degradation of at least a portion of the first audio stream content. The second audio stream may include content selectively output by a user during the output of the first audio stream or time-sensitive content.

[0079] exist Figure 1 In the embodiment, the first audio stream source 20 can provide the first audio stream 1 to at least one audio output device (such as a vehicle 40 and / or one or more other types of audio output devices 50 and 60, etc.). The first audio stream 1 can be or include music, speech and / or other forms of audio (such as infotainment). Figure 1 The first audio stream source 20 is shown external to the vehicle 40, but the first audio stream source 20 may also be local to the vehicle 40, for example, output from a local or onboard storage medium or system.

[0080] The second audio stream source 30 can deliver the second audio stream 2 to at least one audio output device (such as the vehicle 40 and / or one or more other types of audio output devices 50 and 60, etc.). The second audio stream 2 may include information that would normally be output as an interruption to the output of the first audio stream, for example, to cause a loss or significant degradation of a portion of the first audio stream content. Examples of types of second audio streams include, but are not limited to, navigation system output (e.g., time-sensitive directions for the driver), incoming text messages converted to output speech, and vehicle status output, etc. Although Figure 1 The second audio stream source 30 is shown external to the vehicle 40, but the second audio stream source 30 may also be local to the vehicle 40, for example, output from a local or onboard storage medium or system.

[0081] In some embodiments, the second audio stream 2 may include advertising announcements, public alerts and warnings, or other time-sensitive information (e.g., from an external second audio stream source 30). In some embodiments, the second audio stream 2 may be time-sensitive based on the location, condition, or status of the vehicle 40, or the proximity of the vehicle 40 to other vehicles, structures, businesses (e.g., shopping malls, restaurants, stadiums, and / or entertainment venues), points of interest, or traffic, road, or weather conditions. The external second audio stream source 30 may, for example, include a navigation system, an advertising and marketing system, and / or an emergency, warning, and advisory system that transmits the second audio stream 2 to the vehicle 40 via one or more communication networks and / or the system 10.

[0082] The external application system 70 may also provide the vehicle 40 with content that is independent of the first audio stream 1 or the second audio stream 2, but that may be used to deliver and / or output the first audio stream 1 and / or the second audio stream 2. Such external systems may include GPS, cellular service systems, vehicle monitoring systems, and / or other systems that deliver, receive, and / or exchange information and data with the vehicle, its systems, and / or devices within the vehicle (such as mobile phones of vehicle occupants, etc.). For example, such external application systems 70 may be used to locate and / or track the vehicle and / or its occupants and / or to evaluate or report on the status of the vehicle or its subsystems.

[0083] Figure 2 is a block diagram of an embodiment of an audio stream mixing system 200 according to aspects of the inventive concepts. Figure 2 The audio stream mixing system 200 may be part of or hosted by the vehicle 40 (or other audio output device). In some embodiments, the audio stream mixing system 200 may be part of a vehicle head unit responsible for processing and outputting the first audio stream 1 and the second audio stream 2 to, for example, a vehicle occupant. In some embodiments, the vehicle head unit (including the audio stream mixing system 200) and a remote system (e.g., the first audio stream content source 20 or the second audio stream content source 30 and / or the external application system 70) may cooperate to perform the audio stream mixing function.

[0084] In the present embodiment, the vehicle (or vehicle head unit) 40 includes at least one processor 210, which is coupled to at least one computer storage device or memory 220, for example, via at least one bus (BUS). The computer storage 220 can take the form of any now known or later developed computer storage device, such as, but not limited to, a hard drive, various forms of random access memory, various forms of read-only memory, and / or an optical disk. The computer storage 220 can store the audio stream mixing system 200 to perform its tasks (including performing Figure 3 The computer storage unit 220 may also be configured to store and / or buffer the first audio stream 1, the second audio stream 2 and / or a modified form thereof.

[0085] At least one communication interface module 230 may be configured to enable the vehicle 40 and / or the audio stream mixing system 200 to communicate and exchange information and data with systems and subsystems external to the vehicle and local to the vehicle. For example, the vehicle 40 and / or the audio stream mixing system 200 may be configured to receive a first audio stream 1 from at least one first audio stream source 20 (whether local, onboard, or external) and a second audio stream 2 from at least one second audio stream source 30 (whether local, onboard, or external) via the communication interface module 230. The communication interface module 230 may also enable communication with an external application 70.

[0086] In various embodiments, the first audio stream source 20 may be external to the audio stream mixing system 200 of the vehicle 40, such as an AM / FM radio, satellite radio, or other form of telematics or entertainment system, etc. In various embodiments, the first audio source 20 may be local to the audio stream mixing system 200 of the vehicle 40, such as an application 240 located locally to the vehicle 40 for outputting music, audio books, or other audio content from a disc, a local hard drive, a USB device, or from a local device (e.g., a mobile phone or portable audio player) coupled to the vehicle 40 via a wired or tethered connection or a short-range wireless connection (e.g., Bluetooth), etc.

[0087] In various embodiments, the second audio stream source 30 may be external to the audio stream mixing system 200, such as a remote navigation system, a public warning or alarm system, a text system in which text is converted to speech (text-to-speech (TTS)) in the vehicle 40, or an advertising system, etc. In various embodiments, the second audio stream source 30 may be local to the audio stream mixing system 200, such as an application 240 located locally in the vehicle 40 for outputting navigation information from an in-vehicle or onboard navigation system or warnings and announcements from a vehicle monitoring and / or alarm system, etc.

[0088] The first audio stream 1 and the second audio stream 2 may be output via one or more local (eg, in-vehicle or onboard) audio output devices 260 , such as a local or onboard vehicle entertainment system including one or more speakers, or the like.

[0089] In various embodiments, Figure 2 The audio stream mixing system 200 indicated within the dashed line in FIG. 2 is configured to utilize the processor 210 and computer storage 220 of the audio output device (eg, vehicle 40), so that these shared assets are Figure 2 200 is shown within the dashed line of the audio stream mixing system 200. In other embodiments, the audio stream mixing system 200 may use different or dedicated processors and / or computer storage.

[0090] In the present embodiment, the audio stream mixing system 200 includes an audio analysis processor 270 configured to buffer the first audio stream 1 in the computer storage 220 and analyze the first audio stream 1 to determine one or more insertion points for outputting the second audio stream 2. In some embodiments, the audio analysis processor 270 may analyze the first audio stream 1 to locate pauses, audio gaps, and / or low audio levels as insertion points. When the first audio stream 1 is buffered, the first audio stream may have a delayed output (e.g., about a few seconds).

[0091] The input / output control module 250 can manage the buffering and output of the first audio stream 1 via the local audio device 260. The input / output control module 250 can also manage the reception and storage of the second audio stream 2, which can also be buffered in the computer storage 220 until it is output at the insertion point of the first audio stream 1. Therefore, the output of the second audio stream may be delayed and inserted at the insertion point of the first audio stream. The second audio stream 2 may have an associated emergency identifier that affects or indicates the maximum delay before outputting the second audio stream. In some embodiments, the second audio stream source 30 may include a second audio stream 2 and an emergency identifier or include the emergency identifier as part of the second audio stream 2. In some embodiments, the input / output control module 250 and / or the audio analysis processor 270 may associate an emergency level according to a specific second audio stream source 30, wherein different second audio stream sources may have different associated emergency levels. For example, navigation instructions as a second audio stream may have a predetermined emergency level with a maximum delay, for example, must be output within 10 seconds or less.

[0092] The audio stream mixing system 200 may include an audio modification processor 280 configured to modify the first audio stream 1 and the optional second audio stream 2 to complete the mixing of the second audio stream 2 into or with the first audio stream 1 with minimal or no loss or degradation of the content of the first audio stream 1. The audio modification processor may perform the mixing in conjunction with the input / output control module 250 using one or more of a variety of techniques.

[0093] In some embodiments, when a new interrupting second audio stream 2 arrives, the audio stream source 30 may indicate urgency and / or an acceptable maximum delay. Based on the indicated urgency and / or maximum delay, the buffered first audio stream 1 is analyzed to find the least disruptive insertion point of the interruption in the future audio. A part of the analysis performed by the audio stream mixing system 200 may be to classify the first audio stream as music or speech, and then find gaps or pauses in the speech as possible insertion points. One or more of various speech analysis techniques (e.g., using voice activity detection (VAD), automatic speech recognition (ASR) and / or natural language understanding (NLU) processing to find the end of a sentence, phrase, word or other natural interruption point) may be used to find gaps or pauses in speech and / or music lyrics. In some embodiments, additionally or alternatively, speech and / or music may be analyzed for gaps or pauses or points with low volume as possible insertion points.

[0094] In various embodiments, once the insertion point is identified, if the first audio stream 1 is from a medium that can be stopped (e.g., a CD, USB, hard drive, or other computer storage medium), it will be paused at that point and subsequently resumed without losing any of the first audio stream content. However, in some embodiments, as part of the modification of the first audio stream, the pause or gap in the content of the first audio stream can be omitted. In various embodiments, if the first audio stream 1 cannot be stopped (e.g., from a radio), the audio before and after the planned insertion of the second audio stream can be distorted or otherwise modified in the time domain and / or frequency domain to make room for the second audio stream 2.

[0095] In some embodiments, the portion of the first audio stream 1 before and after the insertion point of the second audio stream 2 may be warped to, for example, 110% or 120% of its natural speed. Preferably, the warping of the portion of the first audio stream does not significantly reduce the intelligibility of the warped portion. As an example, if the second audio stream 2 is 6 seconds long and a warp of 120% is used, the first audio stream will be completely caught up after 5 x 6s = 30s seconds. The warp may be determined by the audio modification processor 280 and implemented by the input / output control module 250 to access the buffered first audio stream 1 and second audio stream 2.

[0096] In other embodiments, another technique that can be used to modify the first audio stream 1 is to discard unwanted bits of the first audio stream (such as long pauses, etc.), which can also be implemented by the audio modification processor 280 and the input / output control module 250. In other embodiments, if the first audio stream 1 is music, the audio modification processor 280 can adjust the volume of the first audio stream to low or silent (volume level of 0 or close to 0) as the background and output the second audio stream more prominently in the foreground, then text-to-speech (TTS) or other second audio stream announcements can be customized to be in front of the first audio stream (e.g., music) while still being fully understandable.

[0097] In various embodiments, the audio modification processor 280 and the input / output control module 250 can increase the level of the speech output as the second audio stream 2 based on the first audio stream level. Another more complex option that can be used in some embodiments is to adjust not only the level of the output, but also the time-frequency structure of the output. In various embodiments, this can be implemented similarly to the Lombard effect - when people are in noise, they speak louder and make phonological adjustments: vowel lengthening, formant shifting, etc., to produce improved speech intelligibility in a noisy environment. The resulting audio stream speech can be output with some modifications to improve intelligibility in the case of noise, for example by using parametric coding to improve quality.

[0098] In various embodiments, the audio modification processor 280 and the input / output control module 250 can utilize the TTS multi-style concept to distinguish the second audio stream from the first audio stream. In the case where the mixed audio stream includes playing the second audio stream 2 as the foreground of the background first audio stream 1, this method can be used. The selection of the voice for the second audio stream can be carried out according to the determined voice style used in the first audio stream, wherein the difference between the two voice styles is intended to improve the intelligibility of the second audio stream in the foreground. For example, according to the voice in the first audio stream, a formal / clearer voice or a less formal, more friendly (but not clear enough) voice can be used for the second audio stream. In some embodiments, the voice style of the second audio stream can be selected to match specific needs (higher intelligibility relative to the background, timing relative to other sources, etc.).

[0099] Figure 3 According to the aspects of the inventive concept, Figure 2 An embodiment of a method 300 for mixing audio streams by a system. As described herein, a second audio stream (e.g., an announcement or warning to make an interruption) is mixed with a first audio stream (e.g., infotainment). In some embodiments, the second audio stream is received during the output of the first audio stream and can initiate Figure 3Method 300. For example, in some embodiments, receiving the second audio stream may initiate buffering and / or determination of insertion points in the first audio stream. In some embodiments, buffering and determination of insertion points of the first audio stream may be performed independently of receiving the second audio stream.

[0100] In step 310, a first audio stream 1 is received from a local or external first audio stream source 20 and buffered by an audio output device (e.g., vehicle 40 or devices 50 and 60). The buffered first audio stream 1 may be buffered and output as a modified or unmodified audio stream via, for example, the audio output device of vehicle 40. In step 312, a second audio stream 2 is received by the audio output device from a second audio stream source 30, which may also be buffered. Buffering the second audio stream 2 enables the system 200 to perform an analysis to determine a position within the first audio stream 1 at which the second audio stream 2 should be output. The second audio stream 2 may have an associated urgency level identifier or a maximum delay identifier to assist in determining a position within the first audio stream 1 at which the second audio stream 2 may be inserted and output by the audio output device.

[0101] In step 314, the first audio stream 1 is analyzed to determine one or more insertion points for inserting the second audio stream 2 (e.g., an announcement or warning). As an example, the insertion point can be a portion of the first audio stream having a pause, a gap, and / or a low volume. The determination of the one or more insertion points can be determined based on one or more insertion point criteria, which include, for example, volume level or amplitude, audio signal frequency, audio signal timing, and / or content-based features of the audio (such as natural language processing attributes).

[0102] If multiple potential insertion points are determined, step 314 may include selecting an insertion point from the multiple insertion points. Selecting the insertion point may be based on the earliest insertion point, the insertion point with the longest gap or pause, the insertion point with the lowest volume, the insertion point with the least disruptive effect on the content of the first audio stream based on speech and / or natural language processing of the first audio stream, or a combination of two or more thereof.

[0103] One or more of a variety of speech analysis techniques may be used (e.g., using voice activity detection (VAD), automatic speech recognition (ASR), and / or natural language understanding (NLU) processing to find the end of sentences, phrases, words, or other natural break points) to find gaps or pauses in speech and / or music in the first audio stream 1. In some embodiments, additionally or alternatively, the speech and / or music may be analyzed for gaps and / or pauses based on the volume level of the first audio stream as possible insertion points, where a zero, substantially zero, or relatively low volume level may be determined to indicate a gap and / or pause in speech or music in the first audio stream.

[0104] In some embodiments, the audio stream mixing system 200 and step 314 can implement thresholds for gaps, pauses, and / or low volume levels, so that a portion of the first audio stream 1 is analyzed against one or more thresholds to determine whether the portion has a sufficiently low volume level within a sufficiently long duration of a gap and / or pause to be designated as a suitable insertion point. In some embodiments, duration cannot be used in such a determination, for example, any point where the volume level (or amplitude) of the first audio stream 1 is zero, substantially zero, and / or below a threshold can be determined as an insertion point. Using any of a variety of voice, speech, and / or natural language processing techniques, a zero, substantially zero, or below-threshold volume at the end of a paragraph, sentence, phrase, or word can be determined as an insertion point. With respect to the volume level threshold, the volume level threshold can be, for example, based on a predetermined absolute value of amplitude, a predetermined percentage of the maximum amplitude of all or part of the first audio stream, or a predetermined percentage of the average amplitude of all or part of the first audio stream.

[0105] If urgency or a maximum delay is associated with the second audio stream 2, the position in the first audio stream 1 at which the second audio stream 2 is to be inserted may additionally or alternatively be based at least in part on the urgency or the maximum delay. For example, if the urgency level and / or the maximum delay associated with the second audio stream is 10 seconds, step 314 may include determining at least one insertion point within the next 10 seconds of the first audio stream. In this case, any predetermined threshold may be relaxed or suspended to ensure that the second audio stream 2 is played within the associated maximum delay. For example, within such a maximum delay, the audio stream mixing system 200 and step 314 may include, for example, using the above-mentioned techniques to determine the best insertion point (such as the lowest volume portion or the least disruptive position in a natural language string, etc.) to insert the second audio stream.

[0106] In step 316, the first audio stream 1 is modified at the selected insertion point to accommodate the insertion of the second audio stream 2. Modifying the first audio stream may, for example, include pausing the first audio stream, distorting the first audio stream, adjusting the volume and / or frequency aspects of the first audio stream. The first audio stream may be modified before, during, and / or after the insertion of the second audio stream. The distorted portion of the first audio stream may be used to allow the first audio stream to catch up and / or resume the unmodified output after the insertion of the second audio stream.

[0107] If the first audio stream can be stopped (e.g., output from a CD, USB, or hard drive), the modification of the first audio stream 1 can be to pause the first audio stream at the insertion point, and the second audio stream can be inserted at the pause at step 318. In some cases, even any buffered first audio stream from an external source audio stream (e.g., radio) can be paused when necessary.

[0108] However, if the first audio stream cannot be stopped (e.g., radio), and if the second audio stream has an urgency or maximum delay that specifies that the second audio stream be output before a gap, pause, or low volume portion of the first audio stream, the system 200 may determine to output the second audio stream 2 as a foreground audio output (in step 318), wherein the first audio stream 1 is simultaneously output as background audio. As discussed above, various techniques may be used to modify the first audio stream 1 and / or the second audio stream 2 so that both are simultaneously and understandably output. In some embodiments, the second audio stream may also be modified to increase or enhance its intelligibility. For example, in some embodiments, as discussed above, the second audio stream is played as foreground content over the first audio stream as background content, and the speech or aspects of the speech of the second audio stream may be modified to enhance intelligibility.

[0109] After inserting the second audio stream, the output of the first audio stream is restored in step 320, either as a modified or unmodified output.

[0110] Figure 4A is a diagram 400 used to describe an embodiment of a first audio stream. Figure 4B is used to describe a method for modifying a second audio stream according to aspects of the inventive concept Figure 4A FIG410 is a graph of an embodiment of a first audio stream. Figure 4C is used to describe a method for modifying a second audio stream according to aspects of the inventive concept Figure 4A FIG420 is a graph of another embodiment of a first audio stream. Figure 4D is used to describe a method for modifying a second audio stream according to aspects of the inventive concept Figure 4A Graphs 430 of yet another embodiment of a first audio stream of FIG. In these graphs, the audio stream is plotted with amplitude on the y-axis and time on the x-axis.

[0111] exist Figures 4A to 4D In the example, the first audio stream is referenced as 412, similar to the first audio stream 1 above. The second audio stream is referenced as 414. Figure 4A In FIG. 4 , only the first audio stream 412 is shown as buffered, and its duration is time t1. Figure 4B In FIG. 4 , the second audio stream 414 is also shown, and its duration is d. The second audio stream 414 is inserted into the first audio stream 412 at the insertion point I. Figure 4B In the embodiment, the modification of the first audio stream 412 is to pause the first audio stream at the insertion point for a duration d (i.e., the duration of the second audio stream 418 or approximately the duration). The time t2, which is the end of the first audio stream 412 after the insertion of the second audio stream 414, is equal to or approximately equal to t1+d≈t2.

[0112] exist Figure 4C , the first audio stream 414 is modified at insertion point I to accommodate simultaneous output of the first audio stream 412 and the second audio stream 414. The modified portion of the first audio stream 412 is indicated by reference numeral 416. The first audio stream and the second audio stream may be modified to accommodate intelligible output of the second output stream without significant degradation or loss of content in the first audio stream.

[0113] exist Figure 4D In the example, a portion 416a of the first audio stream before the insertion point I is modified (e.g., warped) and the second audio stream is inserted at the insertion point. At or near the output end of the second audio stream 414, a portion 416b of the first audio stream is modified (e.g., warped) so that the first audio stream maintains its original end time t1. In other embodiments, only a portion of the first audio stream inserted before or after the second audio stream 414 may be modified (e.g., warped). In other embodiments, even in the case where a portion of the first audio stream inserted before and / or after the second audio stream is modified, the completion of the first audio stream does not necessarily occur at or substantially at t1, and it may occur after t1 but before t1+d.

[0114] Figure 5 is an example of a graph 500 of a first audio stream 510, the graph being represented with amplitude on the y-axis and time on the x-axis. Figure 5 , the audio stream mixing system 200 determines two potential insertion points 512, 514. Insertion point 512 occurs at time t2. Each insertion point has zero volume or low volume for a period of time, the period of time being d1 for insertion point 512 and d2 for insertion point 514. Here, d1>d2. In various embodiments, insertion point 512 may be selected as an insertion point, for example, because insertion point 512 is the first insertion point, because insertion point 512 is the longest insertion point when the second audio stream has an associated urgency or maximum delay that stipulates that the second audio stream must be output before the time of the second insertion point 514, or because of some combination thereof. In various embodiments, for example, the second insertion point 514 may be selected as an insertion point when speech or natural language processing applied to the first audio stream 510 indicates that the second insertion point occurs at a more natural and less disruptive portion of the content of the first audio stream (e.g., at the end of a sentence or line or section in music content).

[0115] Figure 6 is a modified to accommodate the second audio stream 610 according to aspects of the inventive concept Figure 5 600 is an embodiment of a first audio stream of a graphic. Figure 6In [description], the first insertion point 512 is selected. The second audio stream 610 is inserted at time t2 of the first audio stream 510. The duration of the second audio stream 610 is d', which is greater than the duration d2 of the insertion point 512 (see Figure 5 ). The first audio stream 512 has been paused to insert the second audio stream 610, and the first audio stream 510 has been resumed after the second audio stream 610 is completed. In various embodiments, as Figure 6 shows, the low volume of the first audio stream 510 (which may be a gap or pause in the content of the first audio stream 510) has been completely omitted, for example, not included when resuming the first audio stream after outputting the second audio stream. In this embodiment, due to the insertion into the first audio stream, the completion of the first audio stream 510 may occur after time t1.

[0116] Figure 7 is another embodiment of the graph 700 of the first audio stream 510 and Figure 6 that is modified according to aspects of the inventive concept to accommodate Figure 5 and Figure 6 the second audio stream 610. In Figure 7 , the first audio stream 510 has been modified to include a first part 510' before inserting the second audio stream 610 and a second part 510" after inserting the second audio stream 610. The first part 510' and the second part 510" have been distorted (e.g., accelerated) so that the first audio stream is completed at time t1, which is the Figure 5 completion time of the unmodified first audio stream 510.

[0117] Distorting the first part 510' of the first audio stream 510 causes the insertion point 510 to move forward in time to time t3, where t3 < t2 (in Figure 5 ). As Figure 6 shows, when resuming the output of the first audio stream using the output of the second part 510" of the first audio stream 510, there is no need to include the low-level part of the first audio stream at the insertion point.

[0118] In other embodiments, only the first portion or the second portion of the first audio stream may be distorted. Additionally or alternatively, the second audio stream or a portion thereof may be distorted. In some embodiments, only the second audio stream may be distorted to minimize or eliminate the need to modify the first audio stream. For example, if the second audio stream is distorted so that the second audio stream fits understandably within the duration of the insertion point, the first audio stream may be modified to background or zero volume while the second audio stream is played in the foreground. As another example, the first audio stream may be paused while the second audio stream is output; the first audio stream may then be resumed with the low volume level of the first audio stream within the insertion omitted. In another embodiment, the first audio stream may be distorted (e.g., slowed down) at the insertion point until the second audio stream is completed.

[0119] In various embodiments, utilizing aspects of the inventive concepts, other techniques for modifying the first audio stream and / or the second audio stream may be implemented to mix the first audio stream and the second audio stream with minimal loss of content of the first audio stream.

[0120] Although the above describes what is considered to be the best mode and / or other preferred embodiments, it should be understood that various modifications may be made therein, and the invention may be implemented in various forms and embodiments, and may be applied in many applications, only some of which are described herein. The following claims are intended to claim protection for the contents described literally and all equivalents thereof, including all modifications and variations that fall within the scope of each claim.

[0121] It should be understood that, for clarity, certain features of the present invention described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the present invention described in the context of a single embodiment for brevity may also be provided separately or in any suitable sub-combination.

[0122] For example, it is to be understood that all features recited in any claims (whether independent or dependent) may be combined in any given way.

Claims

1. A method for mixing a second audio stream with a first audio stream in an audio output device, the method comprising: buffering the first audio stream; determining an insertion point within the first audio stream; Outputting the buffered first audio stream, including modifying output of at least a portion of the first audio stream at the insertion point to accommodate output of the second audio stream, thereby minimizing or avoiding degradation or loss of content of the first audio stream due to output of the second audio stream; outputting the second audio stream at the insertion point; as well as After the second audio stream is completed, the buffered first audio stream continues to be output.

2. The method according to claim 1, wherein: The second audio stream is received during output of the first audio stream.

3. The method according to claim 1, wherein: Buffering of the first audio stream occurs in response to receiving the second audio stream.

4. The method according to claim 1, wherein: The first audio stream is a radio stream.

5. The method according to claim 1, wherein: The first audio stream is a playback of content from a tangible storage medium local to the audio output device.

6. The method according to claim 5, wherein: The tangible storage medium is an optical disk, a UBus medium, a hard drive or a computer memory.

7. The method according to claim 1, wherein: The second audio stream is received by the audio output device together with an emergency identifier or a maximum delay identifier.

8. The method according to claim 7, wherein: The method comprises: A maximum delay for output of the second audio stream is identified based on the urgency identifier or the maximum delay identifier.

9. The method according to claim 7, further comprising: The insertion point is determined based at least in part on the urgency identifier or the maximum delay identifier.

10. The method according to claim 1, wherein: Determining the insertion point includes finding a gap or pause within the first audio stream.

11. The method according to claim 1, wherein: Determining the insertion points includes analyzing the first audio stream using one or more speech analysis techniques to find the end of sentences, phrases, words, or other natural breaking points.

12. The method according to claim 11, wherein: The one or more speech analysis techniques include at least one of voice activity detection (VAD), automatic speech recognition (ASR), and natural language understanding (NLU).

13. The method according to claim 1, wherein: Determining the insertion point includes looking for a low volume level within the first audio stream.

14. The method according to claim 1, wherein: Modifying the first audio stream at the insertion point includes pausing the first audio stream at the insertion point.

15. The method according to claim 1, wherein: Modifying the first audio stream at the insertion point includes modifying a time-frequency structure of the first audio stream.

16. The method according to claim 1, wherein: Modifying the first audio stream at the insertion point includes warping the first audio stream at or near the insertion point.

17. The method according to claim 1, wherein: Outputting the second audio stream at the insertion point includes modifying the second audio stream.

18. The method according to claim 17, wherein: Modifying the second audio stream includes modifying a time-frequency structure of the second audio stream.

19. The method according to claim 17, wherein: Modifying the second audio stream includes modifying a voice style of the second audio stream to improve intelligibility relative to the first audio stream.

20. The method according to claim 1, wherein: The audio output device is or forms part of a vehicle head unit.

21. The method according to claim 20, wherein: The second audio stream is an announcement or warning from a vehicle navigation system, a vehicle monitoring system, or a text-to-speech system.

22. The method according to claim 1, wherein: The second audio stream is an announcement or warning from an advertising system.

23. The method according to claim 1, wherein: The audio output device is a wireless portable device including a mobile phone, a tablet computer or a phablet.

24. An audio stream mixing system, comprising: One or more processors coupled to one or more computer storage devices, one or more first audio stream sources, one or more second audio stream sources, and one or more audio output devices, wherein the one or more processors are configured to: buffering a first audio stream from a first audio stream source in the one or more computer storage devices; determining an insertion point within the first audio stream; outputting the buffered first audio stream via the one or more audio output devices, wherein the one or more processors are configured to modify output of at least a portion of the first audio stream at the insertion point to accommodate output of a second audio stream, thereby minimizing or avoiding degradation or loss of content of the first audio stream due to output of the second audio stream, the second audio stream being received from a second audio stream source; outputting the second audio stream via the one or more audio output devices at the insertion point; and Continue to output the buffered first audio stream via the one or more audio output devices after the second audio stream is completed.

25. The system of claim 24, wherein: The second audio stream is received during output of the first audio stream.

26. The system of claim 24, wherein: The one or more processors are configured to buffer the first audio stream in response to receiving the second audio stream.

27. The system of claim 24, wherein: The first audio stream is a radio stream.

28. The system of claim 24, wherein: The first audio stream is a playback of content from a tangible storage medium local to the audio output device.

29. The system of claim 28, wherein: The tangible storage medium is an optical disk, a UBus medium, a hard drive or a computer memory.

30. The system of claim 24, wherein: The second audio stream is received by the audio output device together with an emergency identifier or a maximum delay identifier.

31. The system of claim 30, wherein: The one or more processors are configured to: A maximum delay for output of the second audio stream is identified based on the urgency identifier or the maximum delay identifier.

32. The system of claim 30, wherein: The one or more processors are configured to: The insertion point is determined based at least in part on the urgency identifier or the maximum delay identifier.

33. The system of claim 24, wherein: The one or more processors are configured to look for a gap or pause within the first audio stream to determine the insertion point.

34. The system of claim 24, wherein: The one or more processors are configured to analyze the first audio stream using one or more speech analysis techniques to find the end of a sentence, phrase, word, or other natural break point to determine the insertion point.

35. The system of claim 34, wherein: The one or more speech analysis techniques include at least one of voice activity detection (VAD), automatic speech recognition (ASR), and natural language understanding (NLU).

36. The system of claim 24, wherein: The one or more processors are configured to look for a low volume level within the first audio stream to determine the insertion point.

37. The system of claim 24, wherein: The one or more processors are configured to pause the first audio stream at the insertion point as a modification to the first audio stream.

38. The system of claim 24, wherein: The one or more processors are configured to modify the time-frequency structure of the first audio stream as a modification of the first audio stream.

39. The system of claim 24, wherein: The one or more processors are configured to warp the first audio stream at or near the insertion point as a modification to the first audio stream.

40. The system of claim 24, wherein: The one or more processors are configured to modify the second audio stream and output the second audio stream at the insertion point as a modified second audio stream.

41. The system of claim 40, wherein: The modified second audio stream includes a modified time-frequency structure.

42. The system of claim 40, wherein: The one or more processors are configured to modify a voice style of the second audio stream to improve intelligibility relative to the first audio stream.

43. The system of claim 24, wherein: The audio output device is or forms part of a vehicle head unit.

44. The system of claim 43, wherein: The second audio stream is an announcement or warning from a vehicle navigation system, a vehicle monitoring system, or a text-to-speech system.

45. The system of claim 24, wherein: The second audio stream is an announcement or warning from an advertising system.

46. ​​The system of claim 24, wherein: The audio output device is a wireless portable device including a mobile phone, a tablet computer or a phablet.

Images