Monitoring of audio signals

Abstract

This application is entitled "Monitoring of Audio Signals". An apparatus and method for monitoring audio output are disclosed. The apparatus can include means for providing one or more primary audio signals based on signals from one or more first microphones associated with an audio capture device, and one or more secondary audio signals based on signals from one or more second microphones associated with an audio monitoring device, the audio monitoring device being separate from the audio capture device and configured for outputting the one or more primary audio signals and the one or more secondary audio signals through one or more loudspeakers. The apparatus can include means for modifying one or both of the primary audio signals and the secondary audio signals such that the output of the one or more primary audio signals is distinguishable from the output of the one or more secondary audio signals.

Description

Technical Field

[0001] The example embodiments relate to an apparatus, method, and computer program relating to the monitoring of audio signals (e.g., monitoring of audio signals representing content captured by an audio capture device). Background Technology

[0002] When capturing audio, for example, using a mobile device with one or more microphones, users may want to monitor the audio being captured in real time. For example, unwanted noise may be present in the captured audio due to environmental conditions (such as wind) and / or due to user processing. Based on the monitored audio, users can adjust the positioning or processing of the capture device to avoid capturing or at least reduce unwanted noise in the captured audio. Summary of the Invention

[0003] The scope of protection sought by the various embodiments of the present invention is defined by the independent claims. Embodiments and features described in this specification that are not within the scope of the independent claims (if any) are to be interpreted as examples useful for understanding the various embodiments of the invention.

[0004] According to a first aspect, this specification describes an apparatus comprising components for: providing one or more primary audio signals based on signals from one or more first microphones associated with an audio capture device; providing one or more secondary audio signals based on signals from one or more second microphones associated with an audio monitoring device, the audio monitoring device being separate from the audio capture device and configured to output one or more primary audio signals and one or more secondary audio signals through one or more speakers; and modifying one or both of the primary audio signals and secondary audio signals such that the output of one or more primary audio signals is distinguished from the output of one or more secondary audio signals.

[0005] The device may further include: a component for monitoring one or more characteristics of one or both of the main audio signal and the auxiliary audio signal, wherein the modification component is triggered based on the monitored one or more characteristics to temporarily modify one or both of the main audio signal and the auxiliary audio signal.

[0006] The monitoring component is configured to monitor the amplitude of one or both of the main audio signal and the auxiliary audio signal, wherein the modification component is triggered based on the monitored amplitude of one of the main audio signal and the auxiliary audio signal exceeding a predetermined threshold.

[0007] The audio monitoring device may include components for performing noise cancellation processing on signals from one or more second microphones, one or more auxiliary audio signals representing the artifacts of the noise cancellation processing, which can be heard by one or more speakers.

[0008] The modified component can be configured to disable noise cancellation processing.

[0009] The modified component can be configured to disable one or more second microphones.

[0010] The modification component can be configured to modify the amplitude of one of the main audio signal and the auxiliary audio signal relative to the other of the main audio signal and the auxiliary audio signal.

[0011] The modification component can be configured to increase the amplitude of one or more main audio signals relative to the amplitude of one or more auxiliary audio signals.

[0012] One or more master audio signals can represent spatial audio, and the modification component is configured to modify the spatial position perceived when one or more master audio signals are output through one or more speakers.

[0013] The device may also include components for determining the receiving direction associated with one or more auxiliary audio signals, wherein the modifying components may be configured to modify the spatial position such that one or more primary audio signals are perceived from a direction different from the receiving direction associated with one or more auxiliary audio signals when output through one or more speakers.

[0014] The device may also include components for determining the orientation or location of the audio capture device relative to the audio monitoring device, wherein the modifying components may be configured to modify the spatial location such that one or more main audio signals are perceived substantially from the orientation or location of the audio capture device when output through one or more speakers.

[0015] The modification component can be configured to modify one or both of the main audio signal and the auxiliary audio signal by means of audio synthesis processing and / or by means of audio filtering, such that at least some audio properties of one of the main audio signal and the auxiliary audio signal are modified in a manner different from that of the other of the main audio signal and the auxiliary audio signal.

[0016] The modification component can be configured to process one or both of the main audio signal and the auxiliary audio signal by means of a selected audio synthesis process and / or audio filter, selecting based on the characteristics of one or both of the main audio signal and the auxiliary audio signal.

[0017] The device can be an audio monitoring device. The device may include a set of headphones or headsets.

[0018] According to a second aspect, this specification describes a method comprising: providing one or more primary audio signals based on signals from one or more first microphones associated with an audio capture device; providing one or more secondary audio signals based on signals from one or more second microphones associated with an audio monitoring device, the audio monitoring device being separate from the audio capture device and configured to output the one or more primary audio signals and the one or more secondary audio signals through one or more speakers; and modifying one or both of the primary audio signals and the secondary audio signals such that the output of the one or more primary audio signals is distinguished from the output of the one or more secondary audio signals.

[0019] The method may further include: monitoring one or more characteristics of one or both of the primary audio signal and the secondary audio signal, wherein modification is triggered based on the monitored one or more characteristics to temporarily modify one or both of the primary audio signal and the secondary audio signal.

[0020] The monitoring can monitor the amplitude of one or both of the main audio signal and the auxiliary audio signal, wherein a modification is triggered based on the monitored amplitude of one of the main audio signal and the auxiliary audio signal exceeding a predetermined threshold.

[0021] The audio monitoring device can be configured to perform noise cancellation processing on signals from one or more second microphones, and one or more auxiliary audio signals represent the pseudo-effects of the noise cancellation processing, which are audible through one or more speakers.

[0022] The modification can disable noise cancellation processing.

[0023] The modification can disable one or more secondary microphones.

[0024] The modification allows you to adjust the amplitude of one of the main audio signal and the auxiliary audio signal relative to the other.

[0025] The modification can increase the amplitude of one or more main audio signals relative to the amplitude of one or more auxiliary audio signals.

[0026] One or more master audio signals can represent spatial audio, and modification can alter the perceived spatial location of one or more master audio signals when output through one or more speakers.

[0027] The method may further include: determining a receiving direction associated with one or more auxiliary audio signals, wherein the modification may alter the spatial location such that one or more primary audio signals, when output through one or more speakers, are perceived from a direction different from the receiving direction associated with the one or more auxiliary audio signals.

[0028] The method may further include: determining the orientation or location of the audio capture device relative to the audio monitoring device, wherein the modification may alter the spatial location such that one or more main audio signals are perceived substantially from the orientation or location of the audio capture device when output through one or more speakers.

[0029] The modification can be made by means of audio synthesis processing and / or by means of audio filtering to modify one or both of the main audio signal and the auxiliary audio signal, such that at least some audio properties of one of the main audio signal and the auxiliary audio signal are modified in a manner different from that of the other of the main audio signal and the auxiliary audio signal.

[0030] Modifications can be made by using a selected audio synthesis process and / or audio filter to process one or both of the main audio signal and the auxiliary audio signal, based on the characteristics of one or both of the main audio signal and the auxiliary audio signal.

[0031] This method can be performed by an audio monitoring device, such as a set of headphones or headsets.

[0032] According to a third aspect, this specification describes a computer program comprising instructions for causing a device to perform at least the following operations: providing one or more primary audio signals based on signals from one or more first microphones associated with an audio capture device; providing one or more secondary audio signals based on signals from one or more second microphones associated with an audio monitoring device, the audio monitoring device being separate from the audio capture device and configured to output the one or more primary audio signals and the one or more secondary audio signals through one or more speakers; and modifying one or both of the primary audio signals and the secondary audio signals such that the output of the one or more primary audio signals is distinguished from the output of the one or more secondary audio signals.

[0033] The example embodiments may also provide any features of the second aspect.

[0034] According to a fourth aspect, this specification describes a computer-readable medium (e.g., a non-transitory computer-readable medium) including program instructions stored thereon for performing at least the following operations: providing one or more primary audio signals based on signals from one or more first microphones associated with an audio capture device; providing one or more secondary audio signals based on signals from one or more second microphones associated with an audio monitoring device, the audio monitoring device being separate from the audio capture device and configured to output the one or more primary audio signals and the one or more secondary audio signals through one or more speakers; and modifying one or both of the primary audio signals and the secondary audio signals such that the output of the one or more primary audio signals is distinguished from the output of the one or more secondary audio signals.

[0035] According to a fifth aspect, this specification describes an apparatus comprising: at least one processor; and at least one memory including computer program code, which, when executed by the at least one processor, causes the apparatus to: provide one or more main audio signals based on signals from one or more first microphones associated with an audio capture device; provide one or more auxiliary audio signals based on signals from one or more second microphones associated with an audio monitoring device, the audio monitoring device being separate from the audio capture device and configured to output the one or more main audio signals and the one or more auxiliary audio signals through one or more speakers; and modify one or both of the main audio signals and the auxiliary audio signals such that the output of the one or more main audio signals is distinguished from the output of the one or more auxiliary audio signals. Attached Figure Description

[0036] Exemplary embodiments will now be described by way of non-limiting example with reference to the accompanying drawings, in which:

[0037] Figure 1 The illustration shows a scenario including a user operating an audio capture device and monitoring the captured audio;

[0038] Figure 2 This is a flowchart of the processing operations according to the example embodiment;

[0039] Figure 3 This is a partial flowchart indicating an example processing operation, which may include... Figure 2 The modification operations indicated in the flowchart;

[0040] Figure 4 This is a partial flowchart of the processing operations according to another example embodiment;

[0041] Figure 5 It shows that in the subsequent time Figure 1 The scene;

[0042] Figure 6 A schematic diagram of a device that can be configured according to some example embodiments; and

[0043] Figure 7 It is a non-transitory medium capable of carrying computer-readable code, according to some example embodiments. Detailed Implementation

[0044] Example embodiments may relate to an apparatus, method, and computer program relating to the monitoring of audio signals (e.g., audio signals representing content captured by an audio capture device).

[0045] It may be necessary to monitor, at least in real-time or near real-time, the audio signals captured by an audio capture device including one or more microphones. For example, ambient noise such as wind might be picked up by one or more microphones and captured without the user's awareness. Due to the user's hand movement on the audio capture device, processing the noise may not be noticeable to the user during capture. For this reason, the user may want to monitor, in real-time or near real-time, what is captured by one or more speakers using another device (i.e., an audio monitoring device) so that the user can react to avoid or mitigate unwanted noise.

[0046] Monitoring can provide a form of real-time or near-real-time feedback that prompts and subsequently guides the user to change the location and / or processing of the audio capture device so that the captured audio content has little or no unwanted noise. However, the user may not necessarily know whether the noise includes noise on the audio signal based on signals received or picked up by one or more microphones associated with the audio capture device (hereinafter referred to as the "primary audio signal"), or noise on the audio signal based on similar noise on signals received or picked up by one or more microphones associated with the audio monitoring device (hereinafter referred to as the "secondary audio signal").

[0047] The term "based on" indicates that some signal processing may be performed at the audio capture device and / or audio monitoring device after one or more microphones have received the audio signal.

[0048] For example, an audio capture device may include one or more signal processing functions performed on signals received by one or more of its microphones, such as noise cancellation, spatialization, automatic granular control, and compression.

[0049] For example, an audio monitoring device may include one or more microphones associated with processing functions such as noise cancellation, which can generate one or more auxiliary audio signals that can be simultaneously heard by a user as the primary audio signal via one or more speakers. The two can be difficult to distinguish.

[0050] The example implementation involves modifying one or both of the primary and secondary audio signals such that the output of one or more primary audio signals is distinguished from the output of one or more secondary audio signals. This thus provides enhanced feedback to the user.

[0051] The example embodiments involve the use of audio capture devices and separate audio monitoring devices.

[0052] The audio capture device may include any device having one or more first microphones for providing one or more master audio signals for transmission to an audio monitoring device. The audio monitoring device may include one or more speakers for outputting one or more master audio signals for monitoring purposes. The audio capture device may also include a storage component for storing a representation of the one or more master audio signals, such as a digital representation. The storage component may include any suitable data storage component, such as one or more memory modules, such as, but not limited to, solid-state memory, hard disk drives, and / or removable memory cards or modules. In some embodiments, the audio capture device may transmit the master audio signals to an external storage system or device, which is another form of memory module.

[0053] Audio capture devices may include, but are not limited to, smartphones, digital assistants, digital music players, personal computers, laptops, tablets, or wearable devices such as smartwatches. Audio capture devices may also include one or more decoders for decoding audio data into a format suitable for output by the speakers of an audio monitoring device.

[0054] Audio capture devices can establish communication sessions with other devices, such as audio monitoring devices, using wired or wireless communication channels. User equipment may include components for using short-range wireless communication such as Bluetooth, Zigbee, or WiFi. Considering the envisioned real-time or near-real-time monitoring nature, the wireless communication channel can use low-latency technologies, such as Bluetooth 5.0.

[0055] The audio capture device may also include a display screen and / or one or more control buttons. The display screen may be touch-sensitive. The audio capture device may include one or more antennas for communicating with external devices, including audio monitoring devices.

[0056] Audio monitoring devices can include any device having one or more speakers for outputting one or more master audio signals received from an audio capture device. For example, an audio monitoring device can include one or more headphones, earphones, earbuds, or speakers from a wearable device such as a virtual reality headset. If a pair of such speakers is provided in the received audio signal, the pair of such speakers can output mono, stereo, and possibly spatial sound if these sounds are provided in the received audio signal. In some embodiments, an audio monitoring device may include only one speaker, which, for example, forms part of a single headphone or earbud, and therefore can only output mono sound.

[0057] The example embodiment focuses on an audio monitoring device as a headphone device, a term that will be used hereinafter. This term can be used as a general term to cover such examples or known equivalents. The example embodiment relates to a headphone device including a first earphone and a second earphone. For the avoidance of doubt, the embodiment can also be implemented in a headphone device including only one earphone.

[0058] The headset device may also include one or more input transducers, such as one or more microphones. One or more microphones can provide a component for the user wearing the headset device to participate in, for example, telephone calls, if the audio capture device has such functionality. The one or more microphones may also be associated with active noise cancellation (ANC) processing capabilities that the headset device may provide.

[0059] The headphone device may also include functionality that enables it to participate in a communication session with the aforementioned audio capture device. For this purpose, the headphone device may include one or more antennas.

[0060] ANC (sometimes called Active Noise Cancellation (ANR)) uses an electrical or electronic system associated with one or more microphones and one or more speakers (such as speakers in headphone devices). The ANC system performs signal processing (e.g., by processing ambient sound received by the one or more microphones) to generate a cancellation signal for output by the one or more speakers. The cancellation signal is used to reduce or eliminate the user's perception of ambient sound at output by means of destructive interference. For example, an ANC system can generate a cancellation signal that is out of phase with the received ambient sound.

[0061] In an earphone device including a first earphone and a second earphone, each earphone may include a microphone, an ANC system, and a speaker. For each earphone, the microphone can receive ambient sound waves, which are then converted into an ambient sound signal and processed by the ANC system to generate a cancellation signal, which is output by the earphone's speaker. Therefore, each earphone can have independent ANC functionality. Alternatively, an ANC system shared by the first and second earphones can receive ambient sound signals from the microphones of both earphones and can generate a corresponding cancellation signal to send back to the first and second earphones.

[0062] An ANC system can operate in several modes. For example, in so-called feedforward mode, the ANC system receives ambient sound signals via one or more microphones located outside each earpiece, typically on the opposite side of the speaker. In this way, a cancellation signal can be generated instantaneously before the user hears the ambient sound. In so-called feedback mode, the ANC system receives ambient sound signals via one or more microphones located inside each earpiece, typically between the speaker and the user's ear. In this way, the cancellation signal can be based on what the user will hear from the speaker. For example, a so-called hybrid mode can utilize signals received from both external and internal microphones in each earpiece to generate a cancellation signal. In this way, the benefits of both feedforward and feedback modes can be combined to generate a cancellation signal. For example, feedforward mode is better at reducing or canceling higher frequency signals compared to feedback mode, but feedback mode is better at reducing or canceling signals over a wider frequency range.

[0063] The ANC system can also provide the aforementioned hear-through mode or transparent mode. Similar to the other modes described above, hear-through mode can be user-selectable, for example, via the user device's user interface or by tapping a controller on the headset. Hear-through mode can be used when a user wants to hear at least some ambient sounds received through one or more microphones on one or more headsets.

[0064] Users can choose which of the above ANC modes to use in a specific situation, such as via the user interface of the user device or by tapping the controller on the headset device. However, for the avoidance of doubt, the example embodiments are not limited to any particular type of ANC system or ANC system that provides the above modes.

[0065] ANC systems may not be perfect at eliminating ambient noise. For example, ambient noise such as wind may not be completely eliminated, and when monitoring one or more main audio signals from the audio capture device, some noise artifacts may still be heard in one or more so-called secondary audio signals, making it impossible for the user to know the source of the noise.

[0066] Figure 1A scenario is illustrated involving a user 10 operating an audio capture device 30 (e.g., a smartphone) to capture audio and possibly video of an event 20, which generates sound waves 22 for capture. The user 10 holds the audio capture device 30 in a specific first orientation for proper capture. The audio capture device 30 may include a display screen 32 and one or more microphones 34. The audio capture device 30 may also include one or more cameras (not shown). Use of the audio capture device 30 may involve the user pointing one or more microphones 34 at the event 20, and the display screen 32 may or may not provide some indication of capture performance, and any video being captured (if appropriate). The one or more microphones 34 receive the sound waves 22 of the event 20, as well as possible other noise, such as wind noise 50 and / or processing noise 60, which are collectively digitally encoded into a main audio signal and may be stored on one or more memory modules of the audio capture device 30.

[0067] User 10 can use a headphone device 40, consisting of a first earbud 40A and a second earbud 40B with corresponding speakers, to monitor the main audio signal in real time or near real time. The main audio signal can be transmitted by the audio capture device 30 via a communication channel 65, which can be Bluetooth 5.0 or other low-latency channels, as described above. User 10 can therefore monitor what they perceive as being captured by the audio capture device 30, and can therefore modify the first direction / orientation, or even the processing of the audio capture device, to mitigate unwanted noise.

[0068] However, if the headphone device 40 is able to output an auxiliary audio signal simultaneously with the main audio signal through the corresponding speakers of the earbuds 40A and 40B, the user may not be able to accurately perceive the content being captured. Therefore, unnecessary adjustments or adjustments detrimental to the capture quality may be made. For example, if the headphone device 40 includes an ANC system as described above, the auxiliary audio signal may include artifacts caused by the pickup of wind noise, etc., by one or more second microphones 34.

[0069] Figure 2 It is a flowchart indicating, for example, processing operations that can be performed by headphone device 40, although the above operations can be performed by an external system.

[0070] Processing operations can be performed through hardware, software, firmware, or a combination thereof.

[0071] The first operation 200 may be optional and may include enabling a monitoring mode. That is, when user 10 wishes to begin audio monitoring of the captured audio, the monitoring mode may be enabled via, for example, a user interface on display 32 and / or via a voice command detectable by audio capture device 30. Alternatively or additionally, the monitoring mode may be enabled based on the received noise level, for example, because it exceeds a predetermined threshold, and the monitoring mode may be disabled if the noise returns in the opposite direction. Alternatively or additionally, the monitoring mode may be enabled by means of headphone device 40.

[0072] The second operation 201 may include providing one or more main audio signals based on signals received from one or more first microphones associated with the audio capture device.

[0073] The third operation 202 may include providing one or more auxiliary audio signals based on signals from one or more second microphones associated with the headphone device 40 (as a given example of an audio monitoring device).

[0074] The fourth operation 203 may include: modifying one or both of the primary audio signal and the secondary audio signal such that the output of one or more primary audio signals is distinguished from the output of one or more secondary audio signals.

[0075] As will be explained below, another operation may include monitoring one or more characteristics of one or both of the primary and secondary audio signals, wherein modification is triggered based on the monitored one or more characteristics to temporarily alter one or both of the primary and secondary audio signals. For example, monitoring may involve monitoring the amplitude of one or both of the primary and secondary audio signals, and triggering modification based on the monitored amplitude of one of the primary and secondary audio signals exceeding a predetermined threshold. For example, modification may be triggered if the secondary audio signal exceeds a predetermined threshold, indicating a specific level of ambient noise that cannot be removed. Modification may be canceled when the monitored amplitude returns from the predetermined threshold, or due to some other detected condition (such as a cancel input or associated voice command via display 32). The predetermined threshold may be a threshold relative to a corresponding characteristic of the primary audio signal. For example, modification may be triggered if, at a given time, the amplitude of the secondary audio signal is greater than the amplitude of the primary audio signal by a predetermined threshold.

[0076] There are multiple options available for modifying the primary and / or secondary audio signals to enhance the user’s ability to distinguish between primary and secondary audio signals monitored, for example, from an ANC system.

[0077] For example, Figure 3 It is another flowchart, whose instructions can serve as Figure 2The modification operation is performed as part of the fourth operation 203 of the process, and can be used alone or in combination.

[0078] For example, the first example modification operation 301 may include: modifying the amplitude (volume) of one of the main signal and the auxiliary signal relative to the other of the main audio signal and the auxiliary audio signal. For example, modification operation 301 may include: increasing the amplitude of one or more main audio signals relative to the amplitude of one or more auxiliary audio signals. Alternatively or additionally, the amplitude of one or more auxiliary audio signals may be decreased relative to the amplitude of one or more main audio signals.

[0079] The first example modification operation 301 can be performed when external noise affects both the audio capture device 30 and the headphone device 40. The first example modification operation 301 can be performed when the amount of noise included in the second audio signal is not too large (below a predetermined threshold, possibly related to the main audio signal) and the user will be able to easily identify the boosted main audio signal.

[0080] In the case of performing mono rendering, such as through only one earbud 40A of the headphone device 40, and / or when external noise only affects one or more second microphones 34 of the headphone device, the user interface of the display 32 can confirm via some visual indication that the audio capture device 30 is capturing audio without detecting additional noise above a predetermined threshold.

[0081] The second example modification operation 303 may include modifying the spatial position of one or more of the primary audio signals (or, one or more secondary audio signals). In this context, where the one or more primary audio signals represent spatial audio, the modification may include changing the perceived spatial position of the one or more primary audio signals when output through one or more speakers of the headphone device 40. Moving one or more primary audio signals may also help the user distinguish such signals from the one or more secondary audio signals.

[0082] For example, as part of the second example modification operation 303, the reception direction associated with one or more auxiliary audio signals can be determined, for example, based on which of the first earbud 40A and the second earbud 40B receives the most noise or noise above a predetermined threshold. The modification may include modifying the spatial location such that one or more primary audio signals, when output through one or more corresponding speakers of the first earbud 40A and the second earbud 40B, are perceived from a direction different from the reception direction associated with the one or more auxiliary audio signals.

[0083] For example, as part of the second example modification operation 303, the orientation or location of the audio capture device 30 relative to the headphone device 40 can be determined or assumed, for example, it is typically located substantially centrally relative to the headphone device. The modification may include altering the spatial location such that one or more main audio signals are perceived substantially from the orientation or location of the audio capture device 30 when output through one or more corresponding speakers of the first earbud 40A and the second earbud 40B.

[0084] Temporary modifications to spatial location can be combined with the magnitude modifications described above for the first example modification operation 301.

[0085] The third example modification operation 305 may include disabling one or more second microphones 34 of the audio headphone device 40. This will remove noise artifacts from the aforementioned second microphones 34 and leave only one or more main audio signals. This effectively disables the ANC processing function of the headphone device 40.

[0086] The fourth example modification operation 307 may include disabling the ANC processing function of the headphone device 40 to effectively achieve the same effect as described above, or alternatively, reducing the amount of ANC processing to perform less noise removal.

[0087] The fifth example modification operation 309 may include synthesizing and / or filtering one or both of the primary audio signal and the secondary audio signal. This may include using one or more synthesizer and / or filter modules to distinguish, for example, one or more secondary audio signals, so that they sound different while preserving the characteristics of the original signal. For example, one or more wind noise reduction filters may be enabled to modify one or more secondary audio signals to make them less noticeable.

[0088] As part of the fifth example modification operation 309, multiple different audio synthesis modules and / or filter modules may be provided, wherein one of these modules is selected based on one or more characteristics of one or both of the main audio signal and the auxiliary audio signal. For example, the one or more characteristics may be based on the type of noise detected in the auxiliary audio signal (e.g., wind noise, processing noise, or other types of noise) and / or which audio channels are most affected.

[0089] For example, if one or more characteristics of the auxiliary audio signal indicate wind noise, a wind noise reduction filter can be applied to the auxiliary audio signal, such as a high-pass filter with a cutoff frequency of approximately 50-150 Hz. If it is determined that the wind noise on the auxiliary audio signal is higher than a predetermined threshold, the modification described above, disabling one or more second microphones 34, can instead be performed.

[0090] For example, if one or more characteristics of the auxiliary audio signal indicate wind noise in only one channel—that is, the left or right channel associated with the left and right microphones of one or more second microphones 34—wind noise reduction can be performed only for that channel, or the associated second microphone can be disabled. In some example embodiments, the audio signal in an unaffected channel can replace the audio signal in an affected channel, thereby making the auxiliary audio signal a mono audio signal.

[0091] For example, if one or more characteristics of the auxiliary audio signal indicate processing noise, different noise reduction filtering and / or disabling operations can be performed in a similar manner to wind noise, but using filters with responses suitable for mitigating processing noise. In practice, processing noise is more likely to occur on only one channel, and therefore filtering and / or disabling only one channel and / or channel replacement operations are more likely to be used for this type of noise.

[0092] For example, if one or more characteristics of the auxiliary audio signal indicate noise due to ANC processing and / or pass-through operation of the ANC system, then ANC processing and / or pass-through operation can be disabled. If only one channel is affected, then only that channel can have its ANC processing and pass-through operation disabled. Similarly, audio signals in unaffected channels can replace audio signals in affected channels.

[0093] For example, if one or more characteristics of the auxiliary audio signal indicate any form of noise and one or more characteristics of the main audio signal indicate little or no noise, the main audio signal can be converted into a mono audio signal, and may be louder than the auxiliary audio signal, so that the user can spatially distinguish between the main audio signal and the auxiliary audio signal.

[0094] For example, artificial wind noise can be mixed into a mono signal, thus separating and distinguishing it from auxiliary noise on the left or right.

[0095] Figure 4 It is an instruction Figure 2 and Figure 3 Another flowchart variant of the flowchart.

[0096] Following the third operation 202, the additional operation 402 may include monitoring one or more characteristics of one or both of the primary audio signal and the secondary audio signal. For example, the characteristics may include the amplitude of one or both of the primary audio signal and the secondary audio signal.

[0097] Another operation 403 may include determining whether a predetermined triggering condition is met.

[0098] If satisfied, the additional operation 404 may include: temporarily modifying one or both of the primary audio signal and the secondary audio signal based on one or more of the monitored characteristics, for example, to make the output of one of the audio signals distinct from the other of the audio signals.

[0099] For example, the amplitude of one or both of the primary and secondary audio signals can be monitored, and modification can be triggered based on the monitored amplitude of one of the primary and secondary audio signals exceeding a predetermined threshold. Temporarily, this means that the modification is completely or gradually canceled after a specific period of time and / or when it is detected that one or more monitored features have returned to exceeding the threshold in the counter direction.

[0100] Figure 5 It shows the results in different subsequent time ranges. Figure 1 The scenario shows that user 10 has adjusted the orientation of audio capture device 30 based on feedback provided by headphone device 40 to avoid or mitigate the capture of previously experienced wind noise and processing noise.

[0101] The example embodiments can therefore help users monitor captured audio (even under noisy conditions) and can utilize the distinguishing aspects described herein to adjust the positioning and / or processing of the audio capture device to avoid or mitigate the capture of unwanted audio (such as wind noise or processing noise).

[0102] Example device

[0103] Figure 6 An apparatus according to some example embodiments is shown, which may include any one of an audio capture device 30 or a headphone device 40. The apparatus may be configured to perform the operations described herein, such as those described with reference to any disclosed process. The apparatus includes at least one processor 600 and at least one memory 601 directly or closely connected to the processor. The memory 601 includes at least one random access memory (RAM) 601a and at least one read-only memory (ROM) 601b. Computer program code (software) 605 is stored in the ROM 601b. The apparatus may be connected to a transmitter (TX) and a receiver (RX). The apparatus may optionally be connected to a user interface (UI) for indicating the apparatus and / or for outputting data. At least one processor 600 is arranged, together with at least one memory 601 and computer program code 605, to cause the apparatus to perform at least one method according to any of the foregoing processes, such as the methods and related features disclosed with reference to the flowcharts herein.

[0104] Figure 7A non-transitory medium 700 according to some embodiments is illustrated. The non-transitory medium 700 is a computer-readable storage medium. It may be, for example, a CD, DVD, USB flash drive, Blu-ray disc, etc. The non-transitory medium 700 stores computer program code to cause a device to perform any of the aforementioned processes, for example, the methods and related features disclosed with respect to the flowcharts herein.

[0105] The names of network elements, protocols, and methods are based on current standards. In other versions or other technologies, the names of these network elements and / or protocols and / or methods may differ, as long as they provide the corresponding functionality. For example, embodiments may be deployed in 2G / 3G / 4G / 5G networks and further generations of 3GPP, but may also be deployed in non-3GPP radio networks such as WiFi.

[0106] The memory module can be volatile or non-volatile. It can be, for example, RAM, SRAM, flash memory, FPGA block RAM, DCD, CD, USB memory stick, and Blu-ray disc.

[0107] Unless otherwise stated or clearly indicated from the context, different statements about two entities mean that they perform different functions. This does not necessarily mean that they are based on different hardware. That is, each entity described in this specification may be based on different hardware, or some or all of the entities may be based on the same hardware. This does not necessarily mean that they are based on different software. That is, the various entities described in this specification may be based on different software, or some or all of the entities may be based on the same software. Each entity described in this specification may be implemented in the cloud.

[0108] As a non-limiting example, implementations of any of the foregoing blocks, devices, systems, techniques, or methods include implementations as hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof. Some embodiments may be implemented in the cloud.

[0109] It should be understood that the above description represents what is currently considered a preferred embodiment. However, it should be noted that the description of the preferred embodiment is given by way of example only, and various modifications can be made without departing from the scope defined by the appended claims.

Claims

1. A device for monitoring audio signals, comprising: A component for providing one or more main audio signals based on signals from one or more first microphones associated with an audio capture device; A component for providing one or more auxiliary audio signals based on signals from one or more second microphones associated with an audio monitoring device, said audio monitoring device being separate from said audio capture device, and configured to output said one or more main audio signals and said one or more auxiliary audio signals through one or more speakers; A component for monitoring one or more characteristics of one or both of the main audio signal and the auxiliary audio signal; as well as A component for modifying one or both of the primary audio signal and the secondary audio signal such that the output of the one or more primary audio signals is distinguished from the output of the one or more secondary audio signals, wherein the component for modification is triggered based on the one or more monitored characteristics to temporarily modify one or both of the primary audio signal and the secondary audio signal.

2. The apparatus of claim 1, wherein the monitoring component is configured to: monitor the amplitude of one or both of the main audio signal and the auxiliary audio signal, wherein the modification component is triggered based on the monitored amplitude of one of the main audio signal and the auxiliary audio signal exceeding a predetermined threshold.

3. The apparatus according to any one of the preceding claims, wherein the audio monitoring device comprises: Components for performing noise cancellation processing on the signals from the one or more second microphones, wherein the one or more auxiliary audio signals represent pseudo-effects of the noise cancellation processing, and the pseudo-effects are audible through the one or more speakers.

4. The apparatus of claim 3, wherein the modification component is configured to disable noise cancellation processing.

5. The apparatus of claim 1 or 2, wherein the modification component is configured to disable the one or more second microphones.

6. The apparatus of claim 1 or 2, wherein the modifying component is configured to modify the amplitude of one of the main audio signal and the auxiliary audio signal relative to the other of the main audio signal and the auxiliary audio signal.

7. The apparatus of claim 6, wherein the modifying component is configured to increase the amplitude of the one or more main audio signals relative to the amplitude of the one or more auxiliary audio signals.

8. The apparatus of claim 1 or 2, wherein the one or more main audio signals represent spatial audio, and the modifying component is configured to modify the spatial position perceived when the one or more main audio signals are output through the one or more speakers.

9. The apparatus according to claim 8, further comprising: A component for determining a receiving direction associated with the one or more secondary audio signals, wherein the modifying component is configured to modify the spatial position such that the one or more primary audio signals, when output through the one or more speakers, are perceived from a direction different from the receiving direction associated with the one or more secondary audio signals.

10. The apparatus according to claim 8, further comprising: A component for determining the direction or location of the audio capture device relative to the audio monitoring device, wherein the modifying component is configured to modify the spatial location such that the one or more main audio signals, when output through the one or more speakers, are perceived substantially from the direction or location of the audio capture device.

11. The apparatus of claim 1 or 2, wherein the modifying component is configured to modify one or both of the primary audio signal and the secondary audio signal by means of audio synthesis processing and / or by means of audio filtering, such that at least some audio properties of one of the primary audio signal and the secondary audio signal are modified in a manner different from that of the other of the primary audio signal and the secondary audio signal.

12. The apparatus of claim 11, wherein the modifying component is configured to process one or both of the primary audio signal and the secondary audio signal by means of a selected audio synthesis process and / or audio filter, the selection being based on the characteristics of one or both of the primary audio signal and the secondary audio signal.

13. The apparatus according to claim 1 or 2, wherein the apparatus is the audio monitoring device.

14. A method for monitoring audio signals, the method comprising: One or more master audio signals are provided based on signals from one or more first microphones associated with the audio capture device; Based on signals from one or more second microphones associated with the audio monitoring device, one or more auxiliary audio signals are provided, the audio monitoring device being separate from the audio capture device and configured to output the one or more main audio signals and the one or more auxiliary audio signals through one or more speakers; Monitoring one or more characteristics of one or both of the primary audio signal and the secondary audio signal; and Modify one or both of the primary audio signal and the secondary audio signal such that the output of the one or more primary audio signals is distinguished from the output of the one or more secondary audio signals, wherein the modification includes: temporarily modifying one or both of the primary audio signal and the secondary audio signal based on the one or more monitored characteristics.

Images