Generation of audio and video signals

Abstract

The device includes receivers 101 and 102 that receive microphone and video signals capturing a scene. An image generator 113 generates an image signal containing an image generated from the video signal, and a zoom determiner 115 determines zoom characteristics such as the zoom coefficient of the image. An audio component generator 107 generates a stereo signal component of a given monaural audio signal, containing a monaural audio signal positioned at a certain location within the stereo image of the stereo signal component. A coupler 109 generates a stereo audio signal containing at least one stereo signal component, and an output unit 111 generates an audio-video signal containing the stereo audio signal and the image signal. In this approach, the position is determined based on zoom characteristics and typically also based on the direction of arrival of the given monaural audio signal. This approach may improve the consistency between visual and auditory spatial cues.

Description

[Technical Field]

[0001] The present invention relates to the generation of audio and video signals, and more particularly to the generation of audio and video signals including stereo audio signals and image signals for video conferencing applications, but is not limited thereto. [Background technology]

[0002] Various applications and services based on capturing audio and video from a scene and transmitting the corresponding data signals to a remote location for local playback are becoming widespread. For example, the recent proliferation of various video conferencing applications that provide both audio and video has been remarkable, and they are now playing an important role not only in the workplace but also in daily life. In recent years, various forms of remote audio and video communication applications using diverse communication media (especially the internet) have become even more widespread, and there is a demand for further expansion and improvement of the services, options, and user experience offered.

[0003] Acoustic, particularly speech, capture has become increasingly important over the past few decades. Speech capture is becoming crucial in a wide range of applications, including communications, video conferencing, gaming, and voice user interfaces. However, in many scenarios and applications, the source of the desired speech is usually not the only sound source in the environment. In a typical acoustic environment, microphones pick up numerous other sound / noise sources, some of which may be the desired sound source.

[0004] Until now, remote voice communication using landlines, mobile networks, or the internet has been virtually limited to the transmission of monaural audio content. However, the constraint of using monaural signals may limit the user experience. For example, video conferencing applications based on monaural signals may not adequately represent the environment in which the sound is captured. In particular, unlike face-to-face communication, the use of monaural signals limits spatial hearing, which is the ability to extract individual sound sources, such as the intended speaker, from a complex acoustic scene.

[0005] The introduction of stereo communication is being proposed for various remote audio and communication applications, particularly internet-based remote conferencing applications. By introducing stereo signals, it becomes possible to transmit spatial dimensions, which is expected to improve the user experience in many applications.

[0006] In general, monaural audio capture in a room tends to be perceived as limited and insufficient in many situations, failing to provide a adequate user experience. In many applications, including spatial components is desirable, for example, to make it easier to distinguish between different speakers.

[0007] On the recording side, a stereo microphone can be used to capture the spatial characteristics of an environment suitable for stereo distribution. Such a stereo microphone typically includes two microphone elements with different directivity, each microphone element capturing one channel of the stereo signal. There are various recording techniques, but basically, two beams are formed, one directed to the left and the other to the right. The two outputs become the left and right channels of the stereo pair. For example, in X / Y recording, two identical cardioid microphones are placed one above the other, facing each other at a 90-degree angle. The two microphones have different radiation / sensitivity characteristics, and their respective outputs become the L and R channels of the stereo pair.

[0008] As another example, in M(id) / S(ide) recording, a Mid microphone with omnidirectional or cardioid characteristics is pointed towards the center of the scene, and a Side microphone with so-called figure-eight or dipole characteristics is mounted perpendicular to the Mid microphone. The sensitivity of a perfect dipole is the same for two sound sources 180 degrees apart, but there is a 180-degree phase difference between the output signals. This characteristic can be used to create two new beams by adding or subtracting the (potentially) scaled signal of the dipole output to the output of the Mid microphone. These beams can have any direction, including X / Y characteristics. Compared to X / Y techniques, the M / S method is more flexible, and the width of the stereo image can be widened or narrowed by changing the coefficients in the mixing matrix.

[0009] However, while this approach delivers desirable performance in many scenarios and applications, it is not optimal in all scenarios. For example, in many scenarios, spatial information may be relatively limited, and spatially separating different sound sources can be difficult, especially for sound sources located far from a stereo microphone. Furthermore, this approach tends to be relatively complex and resource-intensive in many scenarios.

[0010] Furthermore, applications involving both audio and video introduce additional complexity because scene information is delivered through different signals, paths, and processing. However, since these different modalities are perceived as a single, consistent input, it is highly desirable that the audio and video domains and renderings are consistent with each other. Moreover, it is desirable that this consistency be maintained even if variations or adaptations occur on the audio and / or video sides.

[0011] Therefore, improving the approach is beneficial, and in particular, an approach that achieves reduced complexity, increased flexibility, easier implementation, lower costs, improved audio / video acquisition, improved spatial recognition / identification of sound sources, improved sound source isolation, improved support for remote audio applications, improved support for video conferencing, improved audio-video consistency, improved adaptability to changes in video characteristics, improved user experience, and / or improved performance is desirable. [Overview of the Initiative]

[0012] Therefore, the present invention aims to suitably mitigate, reduce, or eliminate one or more of the above-mentioned drawbacks, either individually or in any combination.

[0013] According to one aspect of the present invention, an apparatus for generating audio and video signals is provided. The apparatus includes an audio receiver that receives at least one microphone signal from a microphone configuration that captures a scene having multiple sound sources; an extractor that extracts at least one monaural audio signal representing one of the multiple sound sources from at least one microphone signal; a video receiver that receives a video signal from a camera that captures a scene; an image generator that generates an image signal including an image generated from the video signal; a zoom determiner that determines the zoom characteristics of the image, wherein the zoom characteristics include at least one of zoom level and zoom direction; an audio component generator that generates an intermediate stereo signal component for each of at least one monaural audio signals, wherein the first intermediate stereo signal component of the first monaural audio signal includes a first monaural audio signal positioned at a first position in the stereo image of the first intermediate stereo signal component; a coupler that generates a stereo audio signal including at least one intermediate stereo signal component; and an output unit that generates an audio and video signal including a stereo audio signal and an image signal, wherein the audio component generator determines a first position according to the zoom characteristics.

[0014] This approach can result in an improvement in user experience, operation, and / or performance in many embodiments. In many scenarios, this approach can improve the consistency between the spatial cues provided by the audio and video / images of the audio-visual signal. This approach may be able to dynamically adjust the audio according to the image of the scene, improving the user experience.

[0015] This approach may be able to achieve an improved stereo signal with a stereo image that can provide a better user perception and experience in many embodiments and scenarios. This approach may be able to improve the degree of freedom in generating the spatial arrangement of appropriate sound sources within the stereo image. Typically, this approach can reduce the sensitivity of the provided stereo image to specific characteristics of the captured acoustic scene and specific capture options. In particular, in many scenarios and applications, it is possible to increase the flexibility in providing a stereo image that reflects the characteristics of the acoustic scene but is not completely constrained by them.

[0016] This approach significantly improves the discrimination between sound sources in many embodiments and enables the listener who listens to the stereo image of the stereo signal to separate and identify individual sound sources in many scenarios.

[0017] Also, in many embodiments, each sound source can be represented as more clearly defined, for example, as a point sound source.

[0018] This approach can facilitate or improve the capture of the spatial characteristics of sound sources in many embodiments, and these are represented (e.g., emphasized or modified) within the stereo image of the generated audio signal.

[0019] For example, the inventors have found that the capture of conventional stereo information by stereo signals from stereo microphones (especially two microphones with different directivities) tends to be insufficient in many applications. In particular, it is difficult to separate distant sound sources, and in conventional stereo signals representing sound captured using stereo microphones, it is difficult to separate / distinguish sound sources, and they tend not to be clearly defined spatially. The above approach can provide stereo signals that can more clearly separate / distinguish different sound sources and / or the sound sources are clearly defined spatially in many scenarios.

[0020] This approach may be able to generate stereo signals that can provide a better spatial user experience for the person listening to the stereo signals.

[0021] The scene may be a real-world scene / physical scene such as a room, for example. The sound source may be a real-world sound source such as a speaker in the scene / room.

[0022] The zoom characteristics may be variable zoom characteristics / parameters, for example specifically a variable zoom coefficient / level and / or a variable zoom direction (specifically the direction in which the image is zoomed in or out).

[0023] The monaural audio signal may specifically be a speech signal. The sound source may specifically be a speaker or participant in a teleconference.

[0024] According to an optional feature of the present invention, the zoom characteristics include a zoom level / zoom coefficient.

[0025] Thereby, the operation and / or performance is improved in many embodiments, and typically the user experience is improved.

[0026] The zoom characteristic may be a zoom level / zoom coefficient, and specifically, it may be a field of view relative to a reference field of view such as the maximum field of view. The zoom coefficient / level may increase as the field of view (of the image) decreases.

[0027] According to an optional feature of the present invention, the zoom characteristics include the zoom direction.

[0028] This results in improved operation and / or performance in many embodiments, typically leading to an improved user experience.

[0029] According to an optional feature of the present invention, the apparatus further includes a direction-determining device that determines the direction of arrival of sound from a sound source, and a sound component generator determines a first position according to the direction of arrival.

[0030] This approach can, in many embodiments, lead to improvements in user experience, operation, and / or performance. In many scenarios, this approach can improve the coherence between the spatial cues and information provided by the audio and video / images in the audio-video signal. This approach may allow for dynamic adjustment of audio to match the image of the scene, thereby improving the user experience. For example, this allows the audio-video signal to provide a spatial representation of the scene that more closely corresponds to the physical spatial characteristics of the scene. For example, the audio-video signal can represent sound sources with spatial characteristics that match the spatial characteristics of physical sound sources in the scene, and can provide spatial information that reflects the relative placement of sound sources, for example.

[0031] In many embodiments, the direction of arrival is determined as the direction of the sound beam that captures the sound from the sound source.

[0032] According to an optional feature of the present invention, the audio component generator sets the first position to the position of an endpoint in the stereo image when the direction of arrival is outside the field of view of the image in the zoom characteristics.

[0033] This results in improved operation and / or performance in many embodiments, typically leading to an improved user experience.

[0034] According to an optional feature of the present invention, the output unit includes information indicating that the sound source is located outside the image when it is detected that the direction of arrival is outside the field of view of the image in the zoom characteristics.

[0035] This improves operation and / or performance in many embodiments, typically resulting in an improved user experience. In many embodiments, it can improve the customization of audio and video signal rendering on remote rendering devices.

[0036] According to an optional feature of the present invention, the image generator adjusts the zoom parameters of the image according to the direction of arrival.

[0037] This improves operation and / or performance in many embodiments, typically resulting in an improved user experience. Zoom control parameters are specifically parameters that control the zoom applied to an image, and more specifically, parameters that control the zoom coefficient and / or zoom direction applied to generate the image.

[0038] According to an optional feature of the present invention, at least one microphone signal comprises multiple microphone signals, the extractor comprises multiple audio beamformers that receive the multiple microphone signals, each audio beamformer generates a beam audio signal representing the sound captured by the beam formed by the audio beamformer, the monaural audio signal is the beam audio signal of a first beam formed by a first audio beamformer among the multiple beamformers, and the audio component generator determines a first position according to the direction of the first beam.

[0039] This can result in particularly advantageous operation and performance, typically enabling efficient or robust operation with favorable effects and / or performance.

[0040] A beamformer can receive multiple microphone signals from multiple microphones and perform beamforming on the microphone signals. Multiple microphones can form a microphone array (e.g., multiple microphones arranged in a row). Beamforming may be performed by additive addition / combination of the microphone signals. In some cases, each filter may be filtered by an adaptive filter before addition / combination. The adapter may be configured to perform beamforming by adjusting the coefficients / weights of additive addition or filtering.

[0041] The adapter may also be configured to adjust each of the multiple audio beamformers to capture the sound source, which is typically multiple different sound sources.

[0042] According to an optional feature of the present invention, the audio component generator determines a first position such that the distance from the center of the stereo image increases when the zoom characteristic shows an increase in the zoom level.

[0043] This results in improved operation and / or performance in many embodiments, typically leading to an improved user experience.

[0044] According to an optional feature of the present invention, the audio component generator determines a first position such that the distance from the center of the stereo image decreases when the zoom characteristic shows an increase in the zoom level.

[0045] This results in improved operation and / or performance in many embodiments, typically leading to an improved user experience.

[0046] According to an optional feature of the present invention, the audio component generator adjusts the audio level of the intermediate stereo signal component in the stereo audio signal according to the zoom level indicated by the zoom characteristic.

[0047] This results in improved operation and / or performance in many embodiments, typically leading to an improved user experience.

[0048] According to an optional feature of the present invention, the audio component generator adjusts the audio level of the intermediate stereo signal component in the stereo audio signal according to a first position relative to the field of view of the image at the zoom level.

[0049] This improves operation and / or performance in many embodiments, typically resulting in an improved user experience. The field of view area may be a field of view angle / region (e.g., viewport) that includes the line of sight direction of the scene contained in the image at the current scale factor.

[0050] In some embodiments, if the first position is outside the field of view of the image at the zoom level, the audio component generator reduces the audio level of the intermediate stereo signal component in the stereo audio signal.

[0051] This results in improved operation and / or performance in many embodiments, typically leading to an improved user experience.

[0052] In some embodiments, the image generator adjusts the image zoom control parameters according to the number of monaural audio signals generated.

[0053] This results in improved operation and / or performance in many embodiments, typically leading to an improved user experience.

[0054] According to an optional feature of the present invention, the apparatus is a video conferencing apparatus further comprising a transmitter configured to transmit audio and video signals to a remote video conferencing device.

[0055] In many embodiments, this approach can provide an improved video conferencing system. In many embodiments, this video conferencing system can provide an improved stereo signal that improves speaker representation and typically allows for clearer and easier distinction between different speakers.

[0056] The monaural audio signal may specifically be a speech signal. The sound source may specifically be a speaker or participant in a teleconference.

[0057] According to one aspect of the present invention, a method for generating an audio-video signal is provided. The method includes the steps of: receiving at least one microphone signal from a microphone configuration that captures a scene having multiple sound sources; extracting at least one monaural audio signal from the at least one microphone signal that represents one of the multiple sound sources; receiving a video signal from a camera that captures a scene; generating an image signal including an image generated from the video signal; determining the zoom characteristics of the image, wherein the zoom characteristics include at least one of a zoom level and a zoom direction; generating an intermediate stereo signal component for each of the at least one monaural audio signal, wherein the first intermediate stereo signal component of the first monaural audio signal includes a first monaural audio signal positioned at a first position in the stereo image of the first intermediate stereo signal component; generating a stereo audio signal including at least one intermediate stereo signal component; generating an audio-video signal including a stereo audio signal and an image signal; and determining a first position according to the zoom characteristics.

[0058] The above and other aspects, features, and advantages of the present invention will be described and made apparent with reference to the embodiments described below. [Brief explanation of the drawing]

[0059] Hereinafter, embodiments of the present invention, which are merely examples, will be described with reference to the following drawings. [Figure 1] Figure 1 shows an example of elements of a device for generating a stereo audio signal according to a partial embodiment of the present invention. [Figure 2]Figure 1 shows an example of elements of a device for generating a stereo audio signal according to a partial embodiment of the present invention. [Figure 3] Figure 3 shows an example of an audio beamformer. [Figure 4] Figure 4 shows an example of the camera's field of view. [Figure 5] Figure 5 shows an example of the geometric relationship between different origins for capturing audio or video. [Figure 6] Figure 6 shows some elements of a possible configuration of a processor for implementing elements of an acoustic device according to some embodiments of the present invention. [Modes for carrying out the invention]

[0060] The following description will focus specifically on embodiments of the present invention applicable to systems that acquire and transmit speech and video, such as video conferencing equipment for video conferencing applications. However, it will be understood that the approach of the present invention is applicable to many other systems and scenarios.

[0061] Figure 1 shows an example of an audio-video device configured to generate an audio-video signal that includes audio and video of a real-world scene capture. In this specific example, the audio-video device is a video conferencing device configured to capture speakers in the environment / scene and transmit that information to a remote device.

[0062] The audio / video device includes an audio receiver 101 configured to receive at least one microphone signal from a microphone configuration that captures an environment / scene in which multiple sound sources may be present. Specifically, the audio receiver 101 may receive a single microphone signal from a single microphone placed in the environment, but in many embodiments it may receive multiple microphone signals, for example, specifically, two microphone signals from a stereo microphone, or more microphone signals from a microphone array (e.g., from a linear microphone array consisting of 4, 8, or 16 microphones).

[0063] As a specific example, the audio receiver 101 may receive microphone signals from a microphone array that captures multiple speakers in a room who are using a video conferencing application with remote devices / participants.

[0064] Furthermore, the audio and video device includes a video receiver 102 configured to receive video signals from a camera capturing a scene. The camera may be a camera built into the video conferencing device, or it may be an external camera, such as a webcam, video camera, or any other camera capable of capturing a physical / real-world scene and generating a corresponding video signal.

[0065] Therefore, the audio-visual equipment receives signals that provide both visual and auditory captures of the real world / physical scene. In this specific example, both visual and auditory captures of multiple speakers present in the same room and participating in the same video conference are performed and provided to the audio-visual equipment / video conferencing equipment in the form of video signals and multiple microphone signals. The audio-visual equipment is configured to generate an output audio signal containing stereo signals representing the speakers and an image signal containing images of the scene. Thus, the output audio-visual signals provide a combination of visual and auditory presentations of the real world scene / speakers.

[0066] The device can employ specific approaches to improve the speaker's auditory presentation, such as ensuring an improved relationship between the acoustic representation and the visual presentation, typically increasing the consistency between the two.

[0067] The device includes an extractor 103 connected to a receiver 101 that receives microphone signals. The extractor 103 is configured to extract one or more monaural audio signals, each representing one of several sound sources in a scene. Thus, the extractor 103 generates multiple monaural audio signals, and each monaural audio signal (at least one of them) is an estimate of the sound from a single sound source. The extractor 103 is configured to extract monaural audio signals from microphone signals.

[0068] Therefore, the extractor 103 can extract multiple monaural audio signals, each typically representing / estimated to represent an acoustic source from a single sound source.

[0069] It will be understood that the method for extracting a monaural audio signal from a received microphone signal may vary depending on the embodiment.

[0070] For example, the extractor 103 may be configured to extract a monaural audio signal by selecting a specific time segment that is thought to correspond to one speaker / one sound source. For example, a single microphone signal may be received and divided by the extractor 103 into multiple consecutive time segments, each having a length of, for example, 20 milliseconds. Based on the characteristics obtained by analyzing the microphone signal within each time segment, the extractor 103 can assign each segment to a specific speaker ID among multiple detected speakers. For example, by considering the frequency distribution and comparing it with the average frequency distribution stored for multiple different speaker IDs, each time segment can be assigned to the speaker ID that shows the closest match.

[0071] In other embodiments, multiple different sound sources may be separated by filtering. For example, if multiple different sound sources have substantially different frequency spectra, they can be easily distinguished by using multiple filters with different passbands. For example, a bass guitar and a guitar can be extracted or separated by using filters with different passbands. In some cases, frequency filtering can also be used to distinguish speakers, such as distinguishing between a male voice and a female voice.

[0072] Examples of algorithms that can be used to separate multiple sound sources, particularly multiple speakers, are described in “Audio Source Separation and Speech Enhancement” by Emmanuel Vincent (ed.), Tuomas Virtanen (ed.), and Sharon Gannot (editor) (John Wiley & Sons, October 2018, ISBN: 978-1-119-27989-1).

[0073] The following description focuses on embodiments that extract monaural audio signals by spatial distinction, particularly by using beamforming to form a beam directed towards a specific sound source.

[0074] In particular, as shown in the specific example in Figure 2, the extractor 103 comprises a set of beamformers 201 connected to a microphone array containing M microphones via an audio receiver 101. In this example, the multiple beamformers 201 are illustrated as a single functional unit (i.e., extractor 103). This reflects the fact that the multiple beamformers 201 can be implemented by a single processing unit that performs the operation of all beamformers (e.g., in parallel or sequentially). Thus, the multiple beamformers 201 can be considered equivalent to a single beamformer 201 that forms multiple beams.

[0075] Each beamformer 201 is configured to form a beam and generate a monaural audio signal representing the sound captured by that beam.

[0076] The number of beamformers N, and therefore the number of monaural audio signals N produced, may vary depending on the embodiment. In many embodiments, the number of beamformers N 201 may be 2, 3, 5, or possibly 10 or less. In many embodiments, the number of beamformers N may be 5, 10, or 20 or less. In many embodiments, the system may be designed to include a number of beamformers corresponding to the (maximum) number of speakers / sound sources that are to be individually represented by the monaural audio signals.

[0077] The beamformer 201 is connected to an adapter 203 configured to adjust the beamformer. Specifically, the adapter 203 controls the beamformer 201 to adjust the beam it forms, which includes directing the beam towards the direction of a detected sound source, and typically dynamically adjusting the beam to follow (track) the sound source as it moves.

[0078] In many embodiments, the adapter 203 may include functions such as searching for / detecting new sound sources and forming a new beam toward them, determining when to end sound source tracking, and switching the beam toward new sound sources. For example, the adapter 203 can control the beamformer to search for sound sources and track the detected sound source as long as the average signal level (e.g., a signal level averaged over a time interval long enough to compensate for temporary pauses in speech) exceeds a predetermined threshold, and otherwise stop tracking and search for new sound sources.

[0079] Therefore, the beamformer 201 is an adaptive beamformer in which the directivity can be controlled by the adapter 203 adjusting the parameters of the beamforming operation.

[0080] Each beamformer 201 can be a filter-and-combine (or filter-and-sum in many embodiments) beamformer. A beamforming filter may be applied to each microphone signal, and the filtered outputs may be combined. In some embodiments, this combination may involve applying multiple functions to different signals (e.g., frequency compensation to compensate for differences in frequency sensitivity between microphones, variable delay, nonlinear gain compensation, etc.). However, a filter-and-combine beamformer is typically implemented as a filter-and-sum beamformer by simply summing the filtered outputs.

[0081] Figure 3 shows a simplified example of a filter-and-additive beamformer based on a microphone array with only two microphones 301. In this example, each microphone is connected to beamforming filters 303 and 305, and their respective outputs are added in an adder 307 to generate a beamformed audio output signal. The beamforming filters 303 and 305 have impulse responses f1 and f2, which are tuned to form beams in a given direction. Typically, a microphone array has two or more microphones, and it will be understood that the principle in Figure 3 can be easily extended by adding a beamforming filter for each microphone.

[0082] Each beamformer can include such filter and summation architectures for beamforming (e.g., those described in US7146012 and US7602926). However, it will be understood that in many embodiments, the microphone array 301 may include three or more microphones.

[0083] In most embodiments, the time-domain impulse response of each beamforming filter is not a single Dirac pulse (a simple delay, and therefore corresponding to gain and phase offset in the frequency domain), but rather an impulse response typically spanning a time interval of 2, 5, 10, or 30 milliseconds or more.

[0084] The impulse response may often be implemented by a beamforming filter, which is a finite impulse response (FIR) filter having multiple coefficients. In such embodiments, the adapter 203 is configured to adjust beamforming by adjusting the filter coefficients. In many embodiments, the FIR filter has a coefficient corresponding to a fixed time offset (typically a sample time offset), and the adjustment is achieved by adjusting the value of the coefficient. In other embodiments, the beamforming filter typically has a significantly smaller number of coefficients (e.g., two or three), but the timing of these coefficients is (also) adjustable.

[0085] A notable advantage of beamforming filters having an extended impulse response rather than simple variable delay (or simple frequency-domain gain / phase compensation) is that the beamformer can be adapted not only to the strongest (typically direct) signal component, but also to include other signal paths, typically corresponding to reflected waves. Therefore, this approach improves performance in most real-world environments, specifically in reflective and / or reverberant environments, and / or for sound sources located further away from the microphone array.

[0086] A key element in the performance of adaptive beamformers is the adaptation of directivity (this directivity is generally called the beam, but the extended impulse response results in directivity that has not only a spatial component but also a temporal component (i.e., for example, the beam is formed as a temporal change of reflection)).

[0087] In the system shown in Figure 2, the adapter 203 is configured to adjust the beamforming parameters of the beamformer, specifically by adjusting the coefficients of the beamforming filter to form a specific (spatial and temporal) beam.

[0088] In different embodiments, different adaptive algorithms may be used, and various optimization parameters are known to those skilled in the art. For example, the adapter 203 may be adjusted to adjust the beamforming parameters to maximize the output signal value of the beamformer. As a specific example, consider a beamformer that filters the received microphone signal with a forward matching filter and sums the filter outputs. The output signal is filtered by an inverse adaptive filter having a filter response conjugate to the forward filter (corresponding to a time-reversed impulse response in the time domain in the frequency domain response). An error signal is generated as the difference between the input signal and the output of the inverse adaptive filter. By adjusting the filter coefficients to minimize the error signal, the output power is maximized. This may further result in the generation of a noise reference signal from the error signal. Details of such approaches are described in US7146012 and US7602926.

[0089] Furthermore, the approaches described in US7146012 and US7602926 are based on adjustments using both the sound source signal z(n) and the noise reference signal x(n) obtained from the beamformer, and similar approaches can be applied to the beamformers shown in Figure 2 or Figure 3.

[0090] In fact, the beamformer 201 may have a beamformer configuration that specifically corresponds to the configurations disclosed in US7146012 and US7602926.

[0091] The adapter 203 may be configured to capture a desired sound source and adapt beamforming to represent it as a beamformed audio output signal. Furthermore, the adapter 203 may generate a noise reference signal to estimate the remaining captured acoustics. That is, the noise reference signal indicates the noise that would be captured if the desired sound source were not present.

[0092] If the beamformer 201 is as disclosed in US7146012 and US7602926, the noise reference signal may be generated as an error signal. However, different approaches may be used in other embodiments. For example, in some embodiments, the noise reference signal may be generated by subtracting the generated beamformed speech output signal from the microphone signal from a microphone (e.g., an omnidirectional) microphone. Alternatively, if this noise reference microphone is sufficiently far from other microphones and does not contain the intended speech, it may be generated as the microphone signal itself. In another embodiment, the set of beamformers 201 may be configured to generate a second beam having a null point in the direction of the maximum value of the beam that generates the beamformed speech output signal, and the noise reference signal may be generated as the speech captured by this complementary beam.

[0093] In some embodiments, post-processing, such as noise suppression, may be applied to the output of the voice capture device. This may improve performance, for example, in voice communication. Such post-processing may include nonlinear processing, but in some cases, such as with some speech recognition devices, it may be advantageous to limit it to linear processing only.

[0094] Therefore, the adapter 203 is configured to control the beamformer to search for a sound source, and if a sound source is detected, to dynamically adjust the beam to track the sound source. In some embodiments, the adapter 203 may further adjust the beam shape, for example, by narrowing the beam width after initial detection.

[0095] In this specific example, the device is intended to capture speech signals from speakers within the environment. In this case, the adapter 203 may be configured to adjust based on whether the sound source is actually a speech signal. For example, the adapter 203 may extract characteristics of the captured speech (e.g., frequency distribution, activity pattern, etc.) for a specific detected sound source and determine whether these match characteristics typically found in speech signals. If they match, the adapter 203 continues tracking this sound source; otherwise, it may ignore the sound source and move on to searching for other sound sources that represent a speaker.

[0096] Therefore, a beamformer forms a beam toward the detected sound source (which could specifically be different speakers in the environment). Each beamformer provides a single monaural audio signal representing the sound captured by the beam it forms. Thus, a set of beamformers / a bank of beamformers / multiple beamformers generates multiple monaural signals, and typically each monaural signal represents a different sound source or speaker in the environment.

[0097] The extractor 103 (multiple beamformers in the example in Figure 2) is connected to the audio component generator 107, which is configured to generate an intermediate stereo signal component for one or more, typically all, monaural audio signals. For a given monaural audio signal, the generator 107 is configured to generate an intermediate stereo signal component, which is a stereo signal component in which the monaural audio signal is positioned at a specific location within the stereo image of the intermediate stereo signal component. The intermediate stereo signal component includes two channels, namely a first channel and a second channel corresponding to the left channel and the right channel (or vice versa). Thus, unlike a monaural audio signal which has a single value at a given time (specifically, one sample value per sample time), the intermediate stereo signal component has two values ​​corresponding to the left channel and the right channel. Thus, this two-component stereo representation forms a stereo image from the left position corresponding to the rendering position of the left channel to the right position corresponding to the rendering position of the right channel (and vice versa).

[0098] The generator 107 is configured to generate an intermediate stereo signal component from a monaural audio signal so that the monaural audio signal is placed at a position selected or determined as a desired position within the stereo image.

[0099] Specifically, with respect to a given monaural audio signal from a beamformer, the generator 107 generates an intermediate stereo signal component such that, in at least some situations, the monaural audio signal has a specific position within the stereo image. This position is determined based on the direction of arrival of the corresponding sound, and, for example, as described later, based on the position of the speaker in the image of the received video signal.

[0100] In many examples, the intermediate stereo signal component for a particular monaural audio signal is generated to have a left channel component and a right channel component containing the monaural audio signal. Typically, the phases of the monaural audio signal in the left and right channels are identical, and the position within the stereo image is controlled by adjusting the relative level / amplitude / signal strength between the two channels. If the level of the left channel is not zero and the level of the right channel is zero or close to zero, the monaural audio signal is perceived to be at the rendering position of the left channel within the stereo image. If the level of the right channel is not zero and the level of the left channel is zero or close to zero, the monaural audio signal is perceived to be at the rendering position of the right channel. If the levels of the two channels are identical, the perceived position is in the center of the stereo image; otherwise, for other relative amplitude ratios, it is perceived to be located between the left and right rendering positions. Therefore, in many embodiments, the generator 107 is configured to perform amplitude panning to position a given monaural audio signal at a desired position within the stereo image of the intermediate stereo signal component representing that monaural audio signal.

[0101] The generator 107 may also be configured to position a monaural audio signal within the stereo image of the corresponding intermediate stereo signal component by scaling and adding the first monaural audio signal to two channels of the intermediate stereo signal component, where the relative scaling factor between the two channels (i.e., the left channel and the right channel) is determined based on the desired position within the stereo image for the monaural audio signal / sound source.

[0102] In many embodiments, the generator 107 is configured to determine independent (scalar) scale coefficients for the left and right channels of the intermediate stereo signal components, respectively. The two scale coefficients may be determined as appropriate functions depending on the desired position / direction of arrival, for example, as a function of the angular position φ, which represents the direction of arrival / incident angle indicated by the beam direction / angle.

[0103] As a specific example, the left (L) channel and right (R) channel of the intermediate stereo signal component are determined using amplitude panning, as shown in the following equation. Li = f(φ i )·Si Ri=(1-f(φ i ))· * Si Here, f(φ i ) is a function of the beam direction / incident angle φ of the beamformer i, and the range of this function is limited to 0 to 1. Si is the monaural audio signal of the beam / beamformer with index i (the symbol · indicates multiplication).

[0104] Assuming the beamforming is based on a linear array, φ i If we define f(-π / 2) as an angle in the range of -π / 2 to +π / 2, then by setting f(-π / 2) to 0 and f(+π / 2) to 1, for example, the full range of the stereo image between the left and right speakers can be used during playback.

[0105] In some embodiments, the generator 107 may be configured to determine the scale factor / gain / amplitude of one channel as a function of the beam direction of the beamformer that generates the first monaural audio signal, and then determine the scale factor / gain / amplitude of the other channel such that the sum of the two scale factors / gain / amplitudes satisfies a predetermined criterion. This is equivalent to determining the two scale factors using two functions, but with the functions designed such that the sum of the scale factors satisfies the criterion.

[0106] The criterion may be that the sum of two scale coefficients equals a specific value. This value is usually a predetermined value, such as a fixed value. For example, the sum of the two functions above for the scale coefficients of the left and right channels is always 1, and the function for the left channel is simply given as the value obtained by subtracting the scale coefficient of the right channel from 1.

[0107] Therefore, the generator 107 may be configured to generate intermediate stereo signal components such that the mono audio signal is positioned at a desired angle within the stereo image by determining the relative amplitude of the mono audio signal in the left and right channels in accordance with the determined angle.

[0108] Thus, the generator 107 typically generates multiple intermediate stereo signal components, each component usually corresponding to a single sound source, and especially in video conferencing applications, each intermediate stereo signal component typically represents a specific speaker. Furthermore, the position of each intermediate stereo signal component within the stereo image may be determined, for example, so that multiple speakers are perceived as being located in different positions within the scene.

[0109] As will be described later, this positioning may change depending on the zoom characteristics of the image in the output audio and video signals.

[0110] The generator 107 is connected to the coupler 109, and the generated intermediate stereo signal components are input to the coupler 109. The coupler 109 is configured to generate a stereo audio signal that includes at least some of the intermediate stereo signal components (at least for a specific time interval). In many embodiments, the stereo audio signal is generated to include all of the intermediate stereo signal components input from the generator 107.

[0111] In many embodiments, the coupler 109 may simply add up all left channel signals / values ​​from all intermediate stereo signal components, and then add up all right channel signals / values. Thus, the coupler 109 may add up channels of (at least some) intermediate stereo signal components. In some embodiments, the coupler 109 may not be limited to simple addition of different intermediate stereo signal components, but may perform actions such as adjusting the relative levels between different intermediate stereo signal components, selecting a subset of intermediate stereo signal components, or filtering one or more intermediate stereo signal components.

[0112] Therefore, the coupler generates a stereo audio signal such that the intermediate stereo signal component maintains its position within the stereo image. That is, the position of a particular mono audio signal within the stereo image of the stereo audio signal is the same as the position of that mono audio signal within the corresponding intermediate stereo signal component.

[0113] In particular, in some embodiments, the summation may be weighted summation, where different intermediate stereo signal components may be assigned different weights (however, within the same intermediate stereo signal component, the weights of different channels are the same).

[0114] In this specific example, the coupler is connected to the output unit / communicator 111, which, as will be described later, is configured to generate an audio-video signal that includes a stereo signal and an image signal generated from the received video signal.

[0115] The communication device 111 may transmit voice and video signals via a network such as the Internet, a wireless communication link, a dedicated line, or another communication link capable of transmitting voice data.

[0116] Furthermore, the communication device 111 may be configured to encode and transmit audio and video signals in any suitable manner, using various audio and / or video encoding schemes, speech signal encoding schemes, appropriate modulation and coding techniques, etc., as is well known to those skilled in the art.

[0117] Therefore, in a specific example of a video conferencing application, the device can generate an audio-video signal that is transmitted to one or more remote devices participating in the conference. This audio-video signal typically includes a stereo audio signal containing voices corresponding to multiple different speakers, with the multiple different speakers positioned within a stereo image. Furthermore, this signal includes an image signal (specifically a video signal) containing images of the scene.

[0118] The receiving video conferencing device can provide the user with a spatial perception and representation of the speaker simply by rendering the received stereo signal. For example, the stereo audio signal is simply appropriately decoded and presented via headphones or earphones.

[0119] More generally, a stereo signal may be provided that represents multiple sound sources (e.g., different instruments) positioned at different locations within the stereo image of the stereo signal.

[0120] Therefore, unlike conventional approaches that capture a stereo signal with a stereo image by directly capturing two channels in the environment using, for example, a stereo microphone or other stereo capture configuration, this approach can use a dedicated approach that selects and isolates individual sound sources, such as speakers, using multiple sound beams, and then constructs a stereo image with each sound source positioned at a specific location.

[0121] As described above, the audio / video device is further configured to generate an image signal / video signal from the received video signal and include this generated image signal in the output audio / video signal.

[0122] The video receiver 102 is connected to the image generator 113, which is configured to generate an image signal that includes an image generated from a video signal.

[0123] In some embodiments, the image generator 113 may simply generate an image signal that includes an image or frame of the video signal received from the camera, or in some cases, the received video signal itself may be used directly as an image signal (after possibly removing overhead data or metadata).

[0124] In many embodiments, the image generator 113 may be configured to process images / frames of a video signal to generate an image signal. For example, in some embodiments, the frame rate and / or, for example, the image resolution may be changed. Alternatively, or in addition to this, the image generator 113 may be configured to re-encode video images / frames, or to modify them by performing, for example, noise reduction, color grading, dynamic range conversion, or any other appropriate image processing.

[0125] In some embodiments, the image generator 113 may be configured to generate an image of the image signal by performing a dynamic zoom on part or all of the image / frame received from the camera. The image generator 113 may also be configured to apply a digital zoom to the received image, for example, by selecting a portion of the received image.

[0126] In other embodiments, the camera is controlled to provide dynamically changing zoom, and multiple received images may reflect multiple different zoom levels and viewport sizes of the scene. In other embodiments, variable zoom may be applied by the camera, and the captured image may reflect the current zoom. The video signal received from the camera may include multiple frames and images with different zooms, thereby providing viewports of different sizes for the scene.

[0127] In some embodiments, for example, the zoom may be controlled manually by the user, for example, by the user directly operating the digital zoom of the camera or image generator 113. In some embodiments, a remote user of the video conferencing application controls the zoom via a data signal transmitted from the far-end video conferencing device.

[0128] Zooming can specifically be an operation that changes the field of view (hereinafter also simply referred to as the field of view) represented by one or more images. Typically, zooming in from a particular viewpoint corresponds to a reduction in the field of view, and zooming out corresponds to an expansion of the field of view. That is, zooming in narrows the field of view of the image from a given viewpoint, and zooming out widens it. The zoom level or zoom factor of an image may represent the difference between the (current) field of view from the viewpoint and a reference field of view (e.g., nominal or virtual field of view). The reference field of view may be, for example, the maximum field of view of the image and / or the maximum field of view of the camera capturing the image. In some cases, the maximum field of view may simply be set as a fixed nominal value, such as 180°.

[0129] For example, suppose a camera has a maximum field of view of 40°. In this case, the camera can zoom in to narrow this field of view. The further the field of view is from the maximum field of view, the higher the zoom coefficient or zoom level. The smaller the field of view, the higher the zoom coefficient / level.

[0130] Zooming in may reduce the image's viewport, and zooming out may increase it. Therefore, the higher the zoom level of the image, the smaller the displayed field of view / viewport / scene area. In most cases, zooming (the operation) changes the image's field of view and / or the size of the viewport.

[0131] The zoom direction is the direction in which the field of view changes during zooming, specifically the direction in which zoom-in / field of view reduction occurs. More specifically, it may be the direction of the center or middle of the image's field of view (when the zoom level changes). The image generator 113 is connected to the communication device 111, and the image signal generated by the image generator 113 is sent to the communication device 111. The communication device 111 incorporates the image signal into the audio-video signal. Thus, the audio-video signal is generated to include the generated image signal and the generated stereo audio signal, providing a synthesized audio-video signal that provides visual and auditory perceptual information (including spatial information) about the captured scene. The audio-video signal may be transmitted or delivered to, for example, one or more remote devices / destinations. For example, in a video conferencing application, the audio-video signal may be transmitted to a remote device, which may directly display the image signal and play the stereo signal through speakers or headphones.

[0132] In this system, the generated image that can be transmitted to other parties, such as the far end in the case of a video conferencing application, may change dynamically by dynamically adjusting the zoom level / zoom coefficient and / or zoom direction.

[0133] For example, the audio-visual equipment may be configured in a video conferencing setup where the audio-visual equipment transmits an image signal containing images / video of the room in which active participants are visible in the image.

[0134] For example, in some embodiments, the image generator 113 may substantially implement a function corresponding to an "automatic cameraman" that automatically zooms in / out on a person depending on whether the person is the active speaker or not. For example, if there are two speakers in a room, the image generator 113 may generate an image that includes both if both are active at the same time, or if neither is active. On the other hand, if only one speaker is active, the image generator 113 may zoom in on that speaker and include only that speaker in the generated image. This can be achieved, for example, by selecting an image region of the input image, or by the image generator 113 controlling the camera to increase the zoom factor and point it towards the person speaking.

[0135] In audio-video signals, which include both audio and video (for example, in video conferencing applications), it is desirable to ensure consistency between visual and auditory perception. Visual representations inherently provide spatial information of the scene to remote participants. However, if only a monaural audio representation is provided, essential spatial information is not provided. In a common configuration where speakers are placed near the left and right sides of the screen, all sound tends to be perceived as emanating from the center of the screen. As a result, users may feel that the audio is not in sync with the video.

[0136] In the case of the audio-video device shown in Figure 1 or Figure 2, a stereo signal is generated and transmitted along with the image signal, allowing spatial information to be provided to the user via audio as well. However, it is highly desirable that the spatial information provided by both video and audio modalities in this manner maintains consistency.

[0137] For example, if an image contains two speakers (on the left and on the right), a common approach would typically represent the audio with the left speaker panning to the left (e.g., panning to the left speaker) and the right speaker panning to the right (e.g., panning to the right speaker). However, if the left speaker stops speaking and an automated cameraman function (which may be controlled based on acoustics) zooms in on the right speaker and positions that person in the center of the screen, the acoustic image would still position the speaker to the right of the provided stereo image, which would be perceived as inconsistent by the remote user.

[0138] The devices in Figures 1 and 2 offer a more flexible approach and can typically significantly improve the spatial user experience. This approach can achieve a more coherent spatial perception overall, particularly by generating stereo audio signals that provide spatial perception that is more aligned with and follows the changes in the visual spatial position provided by the image signals.

[0139] The audio-video device includes a zoom processor 115 configured to determine the zoom characteristics of an image. Specifically, the zoom characteristics may be a zoom coefficient / level and / or zoom direction. For example, in the case of digital zoom performed by an image generator 113, the zoom processor 115 may determine the zoom coefficient as the proportion of the received image / frame included in the image produced by the image generator 113. Zoom coefficient 1 may correspond to the entire image, and zoom coefficient 2 may correspond to an output image representing one-quarter of the original image (the zoom coefficient 2 is applied in both the horizontal and vertical directions). The zoom coefficient is usually determined as part of a zoom operation. Most zoom operations include a parameter that directly indicates the currently applied zoom coefficient, which is often used directly by the zoom processor 115 or converted to a different, e.g., more appropriate zoom coefficient representation.

[0140] In some embodiments, the zoom characteristic may indicate the direction of the zoom being performed. For example, if a camera is controlled to zoom in on a particular object or part of a scene, the zoom coefficient may increase and the imaging direction may change (the camera may be physically controlled by changes in orientation, etc.). The zoom instruction may reflect the capture direction. Similarly, if digital zoom is substantially performed by selecting a particular sub-region of the received frame for the output image, the zoom instruction may indicate which region of the input frame has been selected.

[0141] In many embodiments, the audio-visual device includes a direction determiner 117 configured to determine the direction of arrival of sound from one or more sound sources in a scene. In many embodiments, as shown, for example, in Figure 2, the direction determiner 117 may be configured to determine the direction of arrival as the direction of a beam formed toward the sound source. Often, the beam direction is determined based on a coefficient determined, for example, by an adapter 203. In this case, the direction of arrival can be considered identical to the beam direction.

[0142] Therefore, in many embodiments, the audio-video device includes a direction determiner 117 configured to determine the direction of sound captured by beams formed by a plurality of audio beamformers. The direction determiner 117 may determine the direction of sound captured by each beamformer / beam. Specifically, each beamformer / beam may track a single sound source, such as a speaker, and the direction determiner 117 may be configured to determine the direction to that sound source (often by considering the direction of the formed beam as the direction to the sound source).

[0143] In many embodiments, the direction of the captured sound is determined as the direction of the formed beam. Therefore, the direction determiner 117 may be configured to determine the direction of each beam formed by the beamformer and use this direction as an indicator of the direction to the sound source captured by that beam. This allows the adapter 203 to control the beam to detect and then track the sound source, thus estimating the direction of the sound source captured by the sound beam with very high accuracy. Typically, the beam is directed in the direction from which the sound is arriving at the microphone array, so the beam direction reflects the (primary) direction of arrival of the sound captured by that beam (usually the direction of arrival of the direct sound rather than reflected sound), and therefore the direction of arrival from the detected sound source. The determined direction typically reflects the direction from which a person at the microphone array would perceive the sound source as arriving.

[0144] A variety of approaches are known for determining the direction of the formed sound beam, and it will be understood that the direction determiner 117 can use any suitable approach. In some embodiments, the direction determiner 117 may determine the direction of the beam by evaluating, for example, a coefficient applied to each microphone signal or the taps of a beamforming filter.

[0145] For example, if a beamformer forms a beam by weighting and combining signals from multiple substantially omnidirectional microphones (typically using complex weights representing both amplitude and phase / delay), the beam direction can be easily determined from the applied coefficients, as is well known to those skilled in the art.

[0146] As another example, if the beamformer is based on only two microphone inputs, implemented in the frequency domain, and the filter coefficients F1 and F2 belong to microphone 1 and microphone 2 respectively, then the product F1·F2 * ( * By performing the inverse Fourier transform (where indicates conjugate), the peak can be found from this cross-correlation, and the propagation time difference td can be calculated from it. This propagation time difference has the following relationship with the incident angle φ1. cos(φ1) = td·c / d Here, c is the speed of sound and d is the distance between the two microphones. Appropriate exemplary approaches include, for example, US6774934, or “Fast Cross-correlation for TDOA Estimation on Small Aperture Microphone Arrays” by F. Grondin et al. (arXiv:2204.13622).

[0147] In some embodiments, the beam may not directly follow / track the sound source, for example, changing slower than the movement of the sound source or remaining nearly constant. In many cases, such differences are acceptable, and the beam direction can be used directly as an estimate of the direction of arrival of the captured sound. However, in some embodiments, it may be determined that the direction of arrival does not necessarily coincide with the beam direction in which it is formed. For example, in some embodiments, the direction determiner 117 may not directly consider the coefficients used by the beamformer, but may use a different algorithm to detect the direction of arrival of the sound signal captured by the microphone array. Such an algorithm may, for example, directly consider the microphone signal independently of beamforming and be directly based on the microphone signal. The direction of arrival is determined, for example, as the strongest sound signal detected within the angular interval corresponding to the beam angle interval of a given beamformer / beam. This direction of arrival can then be used as the direction of the corresponding monaural signal generated by the beamformer.

[0148] The direction determiner 117 is connected to the generator 107, which receives direction estimates (for example, the direction of each beam in this specific example). Thus, the generator 107 receives the monaural audio signal from the beamformer 201 / extractor 103 and the direction estimates for each beam / beamformer.

[0149] In many embodiments, generator 107 is configured to generate intermediate stereo signal components from these monaural audio signals, which is performed such that each monaural audio signal is placed at a position within the stereo image according to the arrival direction / direction estimation.

[0150] For example, when the zoom characteristic meets certain conditions, generator 107 may be configured to place the monaural audio signals from each beam / beamformer according to the direction of the beam. This is particularly effective, for example, in a scenario where the zoom factor is low enough and all speakers are included in the image. For example, as a nominal setting of the camera, it is conceivable to capture an image with a sufficiently wide viewing angle that includes all possible speaker positions. In this case, generator 107 may generate the positions of each monaural audio signal to coincide with the direction determined by direction determiner 117.

[0151] As shown in FIG. 4, the camera may have a predetermined angular range corresponding to [Φ v,l , Φ v,r from a predetermined camera position at the lowest zoom level. The microphone array is placed at the same position as the camera and can provide beams in an angular direction within a predetermined range [Φ a,l , Φ a,r . The origin of the polar coordinate system may be set at the center of the camera. For applications such as video conferencing, for example, a linear microphone array symmetrically arranged around the camera is often used for audio, and the polar coordinate systems of video and audio share the same origin (i.e., the center of the camera). In this case, the mapping from video space information to audio space information is relatively simple. The arrival direction of the audio may be represented by an angular value and may be mapped to an angular direction within the video range. Then, this audio angle is mapped to an angle within the stereo signal. For example, any angle within the angle range of the camera (i.e., within the angle range of the received video signal) may be directly mapped to the corresponding angle within the stereo image. However, for angles outside the captured video capture angle range, they are mapped to the angle at the nearest end of the capture range.

[0152] For example, if a camera captures wide-angle images within the angular range [-60°, 60°], and a microphone and beamforming configuration can capture and determine angular direction within the angular range [-90°, 90°], then in mapping the arriving angle to the video angular range, the arriving angle will not change within the [-60°, 60°] interval, will be set to -60° for angles less than -60°, and to 60° for angles greater than 60°. Therefore, the arriving angle may be limited to the angular range of the captured frames of the video signal.

[0153] The generator 107 may be configured to map this restricted angle of arrival to the angle range of the intermediate stereo signal components and the stereo image of the stereo signal. For example, the angle range can be mapped to the entire stereo image and scaled so that one endpoint is perfectly mapped to the left channel and the other endpoint to the right channel. For example, if the stereo image is represented by the angle range [-180°, 180°], the stereo image angle of a particular monaural audio signal can be determined by scaling the restricted angle of arrival of that monaural audio signal with a coefficient that reflects the difference in the angle interval. That is, in this example, the restricted angle of arrival may be scaled by a coefficient of 3.

[0154] This approach, in a scenario where audio is played using two speakers positioned on either side of a display showing a corresponding image, makes the voice of a particular speaker appear to be emanating from the direction corresponding to the location where the sound was actually emitted in the scene. Therefore, a high degree of consistency is achieved between the presented visual and auditory information.

[0155] Furthermore, the generator 107 may be configured to adaptively correct the determined position according to the zoom coefficient. For example, if the line of sight does not change due to zooming (i.e., the camera zooms without a change in posture, or the image generator 113 selects the central region of the received video frame), the angular interval displayed in the image of the image signal may become smaller as the zoom coefficient increases. For example, if the zoom coefficient is 2, the original angular interval [-60°, 60°] is reduced to [-30°, 30°]. In this case, the modified angular interval can be used instead of the captured angular interval to apply a similar mapping approach to determine the position of the monaural audio signal in the stereo image. Thus, the generator 107 can determine the angular range of the image based on the zoom coefficient or zoom characteristics, determine the position of the monaural audio signal in the stereo image to match the corresponding angle if the direction of arrival is within the determined range, and set it appropriately to either the left channel or the right channel if it is outside the range.

[0156] This approach can improve the consistency between spatial information provided by the visual presentation of images and spatial information provided by the auditory presentation of stereo signals.

[0157] More specifically, for a monaural audio signal obtained from a particular beamformer, the generator 107 can generate an intermediate stereo signal component for that monaural audio signal so that it has a specific position in the stereo image, where the position is determined based on the direction of the beam generating the monaural audio signal. However, the determined position may be constrained by the zoom characteristics.

[0158] The generator 107 may be configured to generate intermediate stereo signal components for a given monaural audio signal (for example, one determined to be within the range of an image) such that the monaural audio signal is positioned at the corresponding angle in the stereo image, by determining, for example, the relative amplitudes of the monaural audio signal in the left and right channels corresponding to a determined angle. On the other hand, if the direction of arrival is outside the visible range of the image in the current zoom characteristics, the position may be set to correspond to the left or right channel.

[0159] Thus, the generator 107 typically generates multiple intermediate stereo signal components, each component usually corresponding to a single sound source, and especially in video conferencing applications, each intermediate stereo signal component typically represents a specific speaker. Furthermore, the position within the stereo image in each intermediate stereo signal component is represented by the intermediate stereo signal component and is determined according to the direction of the sound, typically indicated by the beam direction. Since speakers are usually located in different positions, their direction estimations will also differ, and the positions within the stereo image in each generated intermediate stereo signal component will also differ from one another. This improves the perception of sound and facilitates, for example, the separation or individual identification of speakers. However, this approach is further combined with a flexible and adaptive approach to maintain consistency between sound and image.

[0160] One of the advantages of this specific approach is that sound sources can be placed within the stereo image with greater freedom, rather than being limited to specific, precise positions determined by stereo recording.

[0161] The position of a particular monaural audio signal / source within the stereo image of the corresponding intermediate stereo signal component and stereo audio signal can be determined as a function of the determined source direction (direction of arrival / beam direction). In some embodiments, the generator 107 may first determine the desired position, for example, the angular position of the intermediate stereo signal component within the stereo image, as a function of the beam direction, and then determine the level / scale coefficients for the left and right channels. In other embodiments, the level / scale coefficient corresponding to the desired position may be, for example, the function f(φ) in the above example. i It may also be determined directly as a function of direction φ, as reflected by ).

[0162] The relationship between the direction of sound captured by a specific beam (hereinafter referred to as beam direction φ for brevity) and the position of the monaural sound signal within the stereo image (hereinafter referred to as stereo image position) may depend on the preferences and requirements of each embodiment and application. Furthermore, the reliance on zoom characteristics, particularly constraint effects, may also depend on the preferences and requirements of each embodiment.

[0163] For example, various mapping functions can be applied, and some examples of approaches are shown below. These approaches may be performed, for example, when the arrival angle is included within the range / scene region represented by the image determined based on zoom characteristics.

[0164] For example, if the beam direction φ increases monotonically from -π / 2 to +π / 2, the stereo image position may also be configured to increase monotonically, except when outside the image angle range. Therefore, the function / relationship mapping the beam direction to the stereo image position can be a monotonic function. When amplitude panning is used, the level / amplitude / scale coefficients are also monotonic functions of the beam direction φ; specifically, one function increases monotonically with respect to φ, and the other function decreases monotonically with respect to φ.

[0165] This can result in a desirable user experience in many situations. For example, in a video conferencing application, each detected speaker may be positioned within the stereo image of the stereo audio signal so as to be sufficiently separated and in the same order as the captured environment. For example, speakers may be placed in fixed positions within the stereo image, or they may be positioned, for example, starting from the left (or right) channel and then at intervals of π / (N-1), in the same order as the captured environment. Remote participants listening to the stereo audio signal may be provided with a spatial experience where each speaker is sufficiently separated, while maintaining the same spatial relationships as the captured room (for example, allowing remote participants to determine the seating order of participants).

[0166] In some embodiments, the stereo image position may be a linear function of the beam direction. For example, the beam direction angle can be directly mapped to the stereo image position angle. This allows for a closer correspondence with the captured environment and provides remote participants with information about the relative positions of speakers (e.g., whether they are close to each other). In some cases, the mapping between the beam direction and the stereo image position may involve scaling. For example, if the beam direction is expressed as an angle within the range / interval of -π / 2 to +π / 2 and the stereo image position is expressed as an angle within the range / interval of -π / 2 to +π / 2, the function between them may be a linear function with a gradient different from 1.

[0167] For example, in many embodiments, scaling can be configured to expand / widen the stereo image by applying a linear function with a gradient greater than 1. For instance, if the beam direction of the target sound source is known to be (most likely) within the range of -π / 3 to +π / 3, a function with a gradient of 3 / 2 can be used to map this to the entire stereo image range of -π / 2 to +π / 2. This can be achieved by a direct mapping to the scale coefficient, for example, by setting one scale coefficient as a linear function such that f(-π / 3)=0 and f(π / 3)=1, and the other scale coefficient as a linear function such that f(-π / 3)=1 and f(π / 3)=0. It will be understood that other functions, including nonlinear functions for nonlinear amplitude panning approaches, can also be used as functions to determine the scale coefficient.

[0168] Therefore, in many embodiments, this approach can broaden at least a portion of the stereo image. Thus, the mapping from beam direction to stereo image location can be configured such that the range / interval of angles representing the beam direction is mapped to a wider range / interval of angles in the stereo image location (the entire range of angles / intervals is the same in both the beam direction and the stereo image location).

[0169] In some embodiments, the gradient is less than 1, and the generator 107 may be configured to compress the stereo image of the stereo audio signal compared to the captured range. For example, in many embodiments, scaling may be configured to reduce / narrow the stereo image by applying a linear function with a gradient of less than 1. For example, if the beam direction of the sound source of interest is known to be (most likely) in the entire range of -π / 2 to +π / 2, but we want to provide a more focused stereo image to a remote participant where all speakers are in the range of -π / 3 to +π / 3, this can be achieved by using a function with a gradient of 2 / 3. This can be achieved by a direct mapping to the scaling factor.

[0170] Therefore, in many embodiments, this approach can narrow at least a portion of the stereo image. Thus, the mapping from beam direction to stereo image position can be configured such that the angular / interval range representing the beam direction is mapped to a smaller angular range / interval of the stereo image position (the entire angular range is the same for both the beam direction and the stereo image position).

[0171] In some embodiments, the stereo image position may be a piecewise linear function of the beam direction. For example, different gradients can be used for different ranges in the beam direction. As an example, the central portion can be enlarged / widened and the side portions can be compressed accordingly to ensure that the entire range is covered. Thus, the gradient may be greater than 1 in some sections and less than 1 in others.

[0172] For example, in some embodiments, the central interval in the beam direction -π / 4 to +π / 4 is extended to the interval -π / 3 to +π / 3, while the peripheral interval -π / 2 to -π / 4 is mapped to the interval -π / 2 to -π / 3, and the peripheral interval +π / 4 to +π / 2 is mapped to the interval +π / 3 to +π / 2.

[0173] Such an approach can provide advantageous performance and an improved user experience in many embodiments. For example, in many video conferencing applications, multiple speakers are often positioned centrally relative to the video conferencing device, and in some cases, one or more speakers may be positioned around the periphery. By compressing the stereo image in the lateral portions and expanding the central portion, the experience is often significantly improved.

[0174] In some cases, other nonlinear functions can be used, for example, to make the transition between the enlarged and reduced portions of the stereo image smoother. For example, a sigmoid function can often be suitably used to map the entire beam direction to the entire stereo image position.

[0175] In many embodiments, the coupler 109 may be configured to generate a stereo audio signal by directly summing / including intermediate stereo signal components while maintaining the relative audio levels captured by the microphones.

[0176] However, in some embodiments, the coupler 109 may be configured to adjust the amplitude of one intermediate stereo signal component in the stereo audio signal relative to the amplitude of another intermediate stereo signal component (from the second beam / beamformer). Thus, in some embodiments, the relative amplitudes of different beam / monaural audio signals may be adjusted by the device.

[0177] For example, in some embodiments, the relative gain can be determined for each intermediate stereo signal component using a gain function of the beam direction. Subsequently, the gain of the beam can be determined by finding the gain for the corresponding beam direction, and the determined gain can be applied to both channels of the corresponding intermediate stereo signal component.

[0178] The gain function may be, for example, a function that has a higher gain towards the center than on either side, or a function that has a constant level except for a small range where certain relevant or important sound sources exist (or may exist).

[0179] Therefore, this device can improve the user experience by being configured to emphasize, for example, a particularly important area or sound source. For example, in a video conferencing application, this approach can emphasize a specific area. For instance, it can emphasize speakers near the center or the position of a specific speaker (e.g., the chairperson of a meeting being held in a video conference).

[0180] In some embodiments, the coupler 109 may be configured to exclude a monaural audio signal / intermediate stereo signal component if the beam direction of the beam capturing a particular monaural audio signal satisfies a predetermined criterion. The criterion may specifically be that the beam direction falls within a predetermined exclusion interval. For example, if the beam direction of a particular monaural audio signal / intermediate stereo signal component falls within the exclusion interval, that monaural audio signal component is not included in the resulting stereo audio signal.

[0181] In some embodiments, this criterion is dynamically adjustable and can be changed, for example, in response to user input. For instance, unwanted or disruptive speakers can be excluded by specifying an angle range to be excluded during a video conference.

[0182] In some embodiments, it will be understood that the generator 107 and the coupler 109 may operate sequentially. For example, the generator 107 may generate N separate stereo signals (and thus N × 2 channel signals), input them to the coupler 109, and the coupler 109 may mix them to form a single stereo audio signal. However, in other embodiments, these two functions may be integrated, and intermediate stereo signal components may be handled intrinsically or implicitly within the combined operation. For example, in some embodiments, a stereo output may be generated by applying a single matrix multiplication to an input vector containing all mono audio signals. For example, at each sample time, a stereo audio signal sample is generated by the following matrix multiplication:

number

number

number

number

[0183] In this example, each formed beam provides a single monaural audio signal, and therefore each sound source selected and separated by the formed beam is represented by a monaural audio signal that essentially has no spatial characteristics or features. However, a stereo signal is generated by combining these individual monaural audio signals, and each signal is given a spatial characteristic in the form of its position within the stereo image. This "reintroduction" of spatial characteristics depends on the spatial information of the captured signal, i.e., the sound source, because the position depends on the direction of arrival (specifically, estimated by the beam direction).

[0184] Therefore, this approach synergistically combines the use of beams to isolate and isolate sound sources within a scene (and essentially extract the audio components) with the "artificial" generation of a stereo image to provide spatial acoustic perception. Furthermore, these functions work together so that the output spatial image depends on the input spatial data.

[0185] This approach can offer significant advantages in many scenarios compared to conventional approaches that simply capture stereo audio representing the spatial information of a scene from a capture location. In particular, this approach allows the stereo image of a stereo audio signal to be adjusted and optimized to specific desired performance. For example, by placing different sound sources, such as speakers, further apart from each other, listeners can more easily distinguish them, similar to a face-to-face conversation. Furthermore, because sound sources can be inserted into the stereo image of a stereo audio signal based on a single monaural signal, sound sources can be spatially clearly defined. This allows sound sources / speakers to be represented as point sources, even when, for example, a stereo microphone captures sound (e.g., by capturing reflections arriving at the stereo microphone from various directions), the sound source / speaker may be perceived as more spread out. Therefore, even when the device attempts to maintain the original spatial relationship between sound sources / speakers by, for example, setting the stereo image angle equal to the beam direction angle, a clearer stereo image can be achieved that allows for easier distinction and separation of sound sources / speakers.

[0186] This approach, rather than simply capturing the spatial characteristics of an acoustic scene, utilizes the interaction between beamforming and artificial positioning to generate an output stereo image optimized for desired performance, without being constrained by the actual acoustic characteristics of the captured environment. Furthermore, this approach can also reflect or retain the spatial characteristics of the environment within the stereo audio signal.

[0187] Furthermore, this approach enables dynamic adjustments to enhance the consistency between the spatial information provided by image signals and stereo audio signals.

[0188] The specific algorithms and functions for determining the position of a particular monaural audio signal within a stereo image may vary depending on the embodiment. As described above, the position typically depends on both the zoom characteristics and the acoustic direction of arrival of the monaural audio signal, and in particular on the beam direction of the beam generating the monaural audio signal.

[0189] In some embodiments, the position may be determined such that the distance from the center of the stereo image increases when the zoom characteristics exhibit an increasing zoom level / coefficient. Thus, the position can be determined such that the distance from the center is a monotonically increasing function of the zoom level / coefficient.

[0190] One example of such an approach is the one described above, which determines the position within the stereo image by scaling the position with a value that reflects the size difference between the image range and the range of the direction of arrival, for a given direction of arrival. This provides particularly suitable behavior in scenarios where only the zoom coefficient changes with zooming, the camera / capture direction remains constant, and the center of the image does not change. In this case, as a speaker moves off-center due to zooming in, the speaker moves towards both ends of the image, and in the above approach, the monaural audio signal corresponding to those speakers may move correspondingly to the edges of the stereo image. Furthermore, this may automatically adapt so that the movement continues until the audio reaches the edge of the stereo image, i.e., the position corresponding to the line of sight of the edge of the image at the current zoom coefficient. Thus, when zooming in to the center of the screen / scene, the effect is obtained in which both the representation of the speaker in the image and the corresponding sound in the stereo image move in a consistent manner until the speaker reaches the edge of the image, after which the sound is maintained in the corresponding right or left channel.

[0191] In other embodiments, when the zoom characteristics show an increase in zoom level / coefficient, the position within the stereo image may be determined such that the distance to the center of the stereo image decreases. Thus, the position can be determined such that the distance from the center is a monotonically decreasing function of the zoom level / coefficient.

[0192] For example, given a given direction of arrival of a monaural audio signal, the generator 107 may be configured to position the monaural audio signal within the stereo image such that the monaural audio signal is positioned more centrally as the camera or image generator 113 zooms in. This effect can be desirable in many embodiments. For example, in some embodiments, the zoom direction may be configured to be toward the sound source of the monaural audio signal. For example, the zoom control may not only zoom in but also align the zoom direction with the direction in which the monaural audio signal is detected (for example, the highest level extracted monaural audio signal may be selected as the monaural audio signal to control the zoom). Thus, the visual effect may be zooming in toward the sound source of the monaural audio signal, i.e., the speaker. This may involve the corresponding monaural audio signal moving to the center of the image, thereby providing consistency of auditory and visual spatial information.

[0193] It will be understood that in many embodiments, the above approaches can be combined with each other. For example, in some embodiments, the stereo image position of the first monaural audio signal may be determined such that the distance to the center decreases, while the stereo image position of the second monaural audio signal may be determined simultaneously such that the distance to the center increases. This may depend, for example, on the zoom direction. For example, if the zoom is directed towards the sound source of the first monaural audio signal, as the zoom coefficient increases, the first monaural audio signal moves towards the center, and the other monaural audio signal moves away from the center (assuming the sound source of the first monaural audio signal is closest to the center of the image).

[0194] In some embodiments, the generator 107 may be configured to move the stereo image position toward the center by considering only the zoom coefficient, without explicitly considering the zoom direction. For example, in less complex embodiments, the mono audio signal is placed at the center of the stereo image when the zoom coefficient exceeds a predetermined threshold, and when the zoom coefficient is less than the threshold, the mono audio signal is placed at a position corresponding to its direction of arrival, or at a predetermined position offset from the center within the stereo image.

[0195] Such an approach can provide a highly advantageous experience in many embodiments while keeping implementation complexity low. For example, a video conferencing device is designed for two adjacent speakers. The device can detect who is speaking and, when both are speaking simultaneously, or when neither is speaking, it can zoom out to include both speakers. In this case, both are displayed in the image, and the audio-video device adjusts to position the corresponding monaural audio signals on the left and right, respectively. However, if the speech detector detects that only one speaker is speaking, the image is adjusted to zoom in on the active speaker (for example, by the image generator 113 selecting an appropriate region of the received frame, by changing the camera's zoom coefficient and capture direction, or by using a multi-sensor camera that acquires three different images and allows selection from among them). In this case, the zoom coefficient is determined to be above a threshold, and the active speaker's monaural audio signal is positioned in the center of the stereo image (in some cases, the monaural audio signal corresponding to the inactive speaker may be smaller than the monaural audio signal of the active speaker, so both monaural audio signals may be positioned in the center of the stereo image).

[0196] In some embodiments, the generator 107 may be configured to adjust the gain / level (acoustic / volume / sound level) of intermediate stereo signal components in a stereo audio signal according to the zoom gain / level and / or zoom direction indicated by the zoom characteristics. In particular, in many embodiments, the zoom coefficient and / or direction is used to determine the line of sight range of a scene contained in an image. In this case, the gain / level of each intermediate stereo signal component may be set according to the direction of arrival of the corresponding monaural audio signal with respect to the determined line of sight. For example, the gain / level may be set to decrease as the direction of arrival moves away from the center of the line of sight range of the image. The gain / level of an intermediate stereo signal component for a given mass may be determined as a monotonically decreasing function with respect to the distance between the direction of arrival of the monaural audio signal of the intermediate stereo signal component and the center of the line of sight range of the image based on the (current) zoom characteristics (value). On the other hand, in some embodiments, the gain / level of an intermediate stereo signal component for a given monaural audio signal may be determined as a monotonically increasing function with respect to the distance between the direction of arrival of the monaural audio signal of the intermediate stereo signal component and the edge of the line of sight range.

[0197] In many embodiments, the audio level can be dynamically adjusted to emphasize the sound source in the center of the image. A specific example is a configuration in which the gain / level of all intermediate stereo signal components corresponding to a monaural audio signal with an incoming direction whose distance from the center of the line of sight is less than a predetermined threshold is kept constant, while the gain / level of those with distances outside the threshold is reduced according to the distance.

[0198] In many embodiments, the generator 107 may be configured to reduce the level of the intermediate stereo signal component in the stereo audio signal if the direction of arrival is outside the field of view or range of the image at the current zoom setting. For example, if it is determined that the direction of arrival is outside the field of view, the gain of the intermediate stereo signal component can be reduced, and in many embodiments, the gain / level can be set to zero to substantially remove the corresponding monaural audio signal from the stereo signal. In such embodiments, the acoustic representation represented by the stereo signal includes only the sound source presented in the image signal.

[0199] As described above, in some embodiments, it is possible to retain the intermediate stereo signal component of a monaural audio signal outside the field of view / range within the stereo image, in which case it is perfectly positioned at the right or left channel, i.e., the endpoint of the stereo signal. Thus, the stereo signal can represent a sound source located outside the image as if it were originating from just outside the image.

[0200] In many embodiments, the generated audio-video signal may include information indicating a monaural audio signal outside the image's field of view. Therefore, the audio-video device may be configured to include information indicating that the sound source is outside the image if it detects that the direction of arrival is outside the image's field of view at the current zoom. In many embodiments, the audio-video signal may include information indicating that a particular sound source / monaural audio signal has been excluded from the provided stereo signal. This allows, for example, a remote user to be notified that one or more sound sources are missing (e.g., by overlaying text on the image).

[0201] In many embodiments, monaural audio signals from sound sources outside the image are not included in the stereo signal, and often not in the audio-video signal either. However, in some embodiments, the audio-video device may be configured to include such monaural audio signals as a separate signal. This allows the remote device to choose whether or not to include the monaural audio signal in the rendered audio. For example, the information above indicates whether the monaural audio signal is outside the left or right of the image, and the remote device can add the monaural audio signal to the corresponding playback audio channel.

[0202] The above discussion has focused primarily on examples where one-dimensional judgment and consideration are performed, specifically by considering only the horizontal effect of zoom. This is practical in many situations, such as video conferencing applications where sound sources are often located in roughly the same vertical position. It is also highly compatible with the generation of stereo images, which are often perceived as one-dimensional images in the horizontal direction.

[0203] However, it will be understood that the principles, approaches, and examples described can be appropriately extended to two-dimensional cases, for example, where the vertical direction is also considered. For example, when determining whether a monaural audio signal represents a sound source outside the viewport at the current zoom, both the vertical and horizontal components may be considered. Thus, instead of considering a one-dimensional line-of-sight range, a two-dimensional field of view can be considered. For example, if it is determined that the direction of arrival of a given monaural audio signal is outside the field of view at the current zoom, the level of that monaural audio signal may be reduced or, for example, set to zero (equivalent to removing the monaural audio signal from the resulting stereo signal).

[0204] As described above, the image generator 113 is configured in many embodiments to control zoom, and the control includes controlling the zoom direction and / or zoom coefficient. In some embodiments, the image generator 113 can also control the camera and / or perform digital zoom.

[0205] In many embodiments, zooming is performed according to the direction of arrival. For example, the zoom coefficient may be set so that all determined directions of arrival are (exactly) included. Thus, if one speaker is speaking and only one monaural audio signal is detected, the image generator 113 may be configured to zoom in on that speaker. The direction may be set to match the direction of arrival of the monaural audio signal, and the zoom coefficient may be set to the maximum level. If two or more monaural audio signals are detected corresponding to two active speakers, the zoom level may be set to a size large enough to just cover both directions of arrival, and the zoom direction may be set to the midpoint of the two ends of the directions of arrival. This allows zooming out to include both active speakers. Thus, the image generator 113 controls the zoom to include all active sound sources, i.e., all active speakers.

[0206] In many embodiments, the image generator 113 may be configured to control the zoom in accordance with the number of monaural audio signals generated / detected, i.e., the number of detected / active sound sources. Specifically, the zoom can be set depending on whether one or more active sound sources / speakers are detected. For example, if one active sound source / speaker is detected, an image zooming in on that single sound source may be generated, for example, by setting the zoom coefficient to an appropriately high value and corresponding the zoom direction to the direction of arrival. If there are no active sound sources, or if multiple active sound sources are detected, the zoom may be set to a default level that captures the entire scene as wide as possible. Thus, if there are no active speakers or multiple active speakers, the entire scene / room is displayed, and if only one speaker is active, the image generator 113 can zoom in on that speaker.

[0207] The above explanation primarily assumes a situation where the origins of the video and audio coincide, which applies to many embodiments and scenarios. However, the origins do not necessarily have to be the same. For example, the conversion can be performed if the distance between the origins, i.e., between the center of the camera and the center of the linear array, is known.

[0208] For example, Figure 5 illustrates one such example, showing a semicircle with radius r whose center point represents the origin of the video. The center point of the audio locator is located to the right at a distance of dav.

[0209] Next, applying trigonometric functions, the angle of incidence of the sound is φ. a and the field of view of the image φ v The relationship is expressed by the following equation. φ a =g(φ v ) = arctan((r sin φ v +dav) / (r cos φ v ))

[0210] f(φ a ) is as follows (where f(φ a Similarly, the beam direction / incident angle φ of the beamformer is also relevant. a It is a function of , and its range is restricted to the range of 0 to 1. f(φ a ) = g(φ v,l )=1 f(φ a ) = g(φ v,r )=0 f(φ a )=g((φ v,l +φ v,r ) / 2) = 0.5

[0211] The sound device may be implemented in one or more appropriately programmed processors. Different functional blocks may be implemented in separate processors, and / or, for example, in the same processor. Examples of suitable processors are provided below.

[0212] Figure 6 is a block diagram showing an example of a processor 600 included in embodiments of the present disclosure. Processor 600 may be used to implement one or more processors that implement the above-mentioned device or its elements (in particular, including one or more artificial neural networks). Processor 600 may be any suitable processor type, including, but not limited to, a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable array (FPGA) programmed to form a processor, a graphics processing unit (GPU), an application-specific circuit (ASIC) designed to form a processor, or a combination thereof.

[0213] The processor 600 may include one or more cores 602. A core 602 may include one or more arithmetic units (ALUs) 604. In some embodiments, the core 602 may include, in addition to or instead of, an ALU 604, a floating-point arithmetic unit (FPLU) 606 and / or a digital signal processing unit (DSPU) 608.

[0214] The processor 600 may include one or more registers 612 that are communicatively coupled to the core 602. The registers 612 may be implemented using dedicated logic gate circuits (e.g., flip-flops) and / or any memory technology. In some embodiments, the registers 612 may be implemented using static memory. The registers can provide data, instructions, and addresses to the core 602.

[0215] In some embodiments, the processor 600 may include one or more levels of cache memory 610 communicably coupled to the core 602. The cache memory 610 can provide computer-readable instructions to the core 602 for execution. The cache memory 610 can provide data for processing by the core 602. In some embodiments, computer-readable instructions may be provided to the cache memory 610 by local memory, for example, local memory attached to the external bus 616. The cache memory 610 may be implemented using any suitable cache memory type, such as metal-oxide-semiconductor (MOS) memory, for example, static random-access memory (SRAM), dynamic random-access memory (DRAM), and / or any other suitable memory technology.

[0216] The processor 600 may include a controller 614, which may control inputs to the processor 600 from other processors and / or components included in the system, and / or control outputs from the processor 600 to other processors and / or components included in the system. The controller 614 may control data paths in the ALU 604, FPLU 606, and / or DSPU 608. The controller 614 may be implemented as one or more state machines, data paths, and / or dedicated control logic. The gates of the controller 614 may be implemented as standalone gates, FPGAs, ASICs, or other suitable technologies.

[0217] Registers 612 and cache memory 610 may communicate with controller 614 and core 602 via internal connections 620A, 620B, 620C, and 620D. Internal connections may be implemented as buses, multiplexers, crossbar switches, and / or other suitable connection techniques.

[0218] The inputs and outputs of the processor 600 may be provided via a bus 616 which may include one or more conductive wires. The bus 616 may be communicatively coupled to one or more components of the processor 600, such as a controller 614, a cache 610, and / or registers 612. The bus 616 may be coupled to one or more components of the system.

[0219] Bus 616 may be coupled to one or more external memories. External memories may include read-only memory (ROM) 632. ROM 632 may be mask ROM, EPROM (Electronically Programmable Read Only Memory), or any other suitable technology. External memories may include random access memory (RAM) 633. RAM 633 may be static RAM, battery-backed static RAM, dynamic RAM (DRAM), or other suitable technology. External memories may include EEPROM (Electrically Erasable Programmable Read Only Memory) 635. External memories may include flash memory 634. External memories may include magnetic storage devices such as disk 636. In some embodiments, external memories may be included in the system.

[0220] It will be understood that in many embodiments, this approach may be even more advantageous if it further includes an adaptive canceller configured to cancel signal components of the beamformed speech output signal that correlate with at least one noise reference signal. For example, as in the example in Figure 1, the adaptive filter may receive a noise reference signal as input and subtract its output from the beamformed speech output signal. This adaptive filter may be configured, for example, to minimize the signal level in time intervals where no speech is present.

[0221] For clarity, the above description illustrates embodiments of the invention with reference to several different functional circuits, units, and processors. However, it will be understood that functions can be appropriately distributed among different functional circuits, units, or processors without impairing the invention. For example, a function described as being performed by several separate processors or controllers may be performed by the same processor or controller. Therefore, references to specific functional units or circuits should be considered not as indicating a strict logical or physical structure or organization, but as references to appropriate means for providing the described function.

[0222] The present invention can be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. The present invention may be implemented at least partially as computer software running on one or more data processors and / or digital signal processors. Elements and components of embodiments of the present invention can be implemented physically, functionally, and logically in any suitable way. In practice, the functionality may be implemented as a single unit, multiple units, or as part of other functional units. Thus, the present invention may be implemented as a single unit, or it may be physically and functionally distributed among multiple different units, circuits, and processors.

[0223] Although the present invention has been described in relation to several embodiments, the present invention is not limited to the specific forms described in the specification. The scope of the present invention is limited only by the appended claims. Furthermore, even if a certain feature appears to be described in relation to a particular embodiment, those skilled in the art will recognize that various features of the above embodiments can be combined in accordance with the present invention. In the claims, terms such as "equipment," "includes," etc., do not preclude the presence of other elements or steps.

[0224] Furthermore, even if listed individually, multiple means, elements, circuits, or method steps may be carried out by, for example, a single circuit, unit, or processor. Moreover, even if individual features are included in different claims, they can be suitably combined, and their inclusion in different claims does not mean that the combination of features is unfeasible and / or unfavorable. Also, the inclusion of a feature within one claim category does not mean that the feature is limited to that category; features are equally applicable to other claim categories as appropriate. Furthermore, the order of features in a claim does not indicate a specific order in which the features should act, and in particular, the order of individual steps in a method claim does not mean that the steps must be performed in that order. The steps can be performed in any suitable order. Also, singular expressions do not preclude plural forms; therefore, expressions such as "first," "second," etc., do not preclude plurals. Reference numerals in the claims are merely examples for clarity and do not in any way limit the scope of the claims.

[0225] The following can be provided:

[0226] A device for generating audio and video signals, the device is An audio receiver (101) that receives at least one microphone signal from a microphone configuration that captures a scene with multiple sound sources, An extractor (103) that extracts at least one monaural audio signal representing one of several sound sources from at least one microphone signal, A video receiver (102) that receives a video signal from a camera that captures a scene, An image generator (113) that generates an image signal including an image generated from a video signal, A zoom determiner (115) that determines the zoom characteristics of the image, Audio component generator (107) that generates an intermediate stereo signal component for each of at least one monaural audio signals, wherein the first intermediate stereo signal component of the first monaural audio signal of at least one monaural audio signal includes a first monaural audio signal positioned at a first position within the stereo image of the first intermediate stereo signal component, A coupler (109) that generates a stereo audio signal containing at least one intermediate stereo signal component, It includes an output section (111) that generates an audio / video signal including a stereo audio signal and an image signal, The audio component generator (107) is a device that determines a first position according to the zoom characteristics.

[0227] A method for generating audio and video signals, wherein the method is A step of receiving at least one microphone signal from a microphone configuration that captures a scene with multiple sound sources, The steps include extracting at least one mono audio signal representing one of several sound sources from at least one microphone signal, The steps include receiving a video signal from a camera that captures the scene, A step of generating an image signal that includes an image generated from a video signal, The steps include determining the zoom characteristics of the image, A step of generating an intermediate stereo signal component for each of at least one monaural audio signals, wherein the first intermediate stereo signal component of the first monaural audio signal of at least one monaural audio signal includes a first monaural audio signal positioned at a first position within the stereo image of the first intermediate stereo signal component; A step of generating a stereo audio signal that includes at least one intermediate stereo signal component, A step of generating an audio-video signal including a stereo audio signal and an image signal, A method comprising the step of determining a first position according to zoom characteristics.

[0228] Zoom characteristics can be described as the field of view of an image or the characteristics of the change in the field of view.

Claims

1. A device for generating audio and video signals, wherein the device is An audio receiver that receives at least one microphone signal from a microphone configuration that captures a scene with multiple sound sources, An extractor that extracts at least one monaural audio signal representing one of the multiple sound sources from the at least one microphone signal, A video receiver that receives a video signal from a camera that captures the aforementioned scene, An image generator that generates an image signal including an image generated from the aforementioned video signal, A zoom determiner for determining the zoom characteristics of the aforementioned image, wherein the zoom characteristics include at least one of a zoom level and a zoom direction, An audio component generator that generates an intermediate stereo signal component for each of the at least one monaural audio signals, wherein the first intermediate stereo signal component of the first monaural audio signal of the at least one monaural audio signal includes the first monaural audio signal positioned at a first position within the stereo image of the first intermediate stereo signal component, A coupler that generates a stereo audio signal containing at least one intermediate stereo signal component, The system includes an output unit that generates an audio / video signal including the stereo audio signal and the image signal, The audio component generator is a device that determines the first position according to the zoom characteristics.

2. The acoustic device according to claim 1, wherein the zoom characteristics include a zoom level.

3. The apparatus according to claim 1 or 2, wherein the zoom characteristics include the zoom direction.

4. The apparatus according to any one of claims 1 to 3, further comprising a direction determining device for determining the direction of arrival of sound from the sound source, wherein the sound component generator determines a first position according to the direction of arrival.

5. The apparatus according to claim 4, wherein the sound component generator sets the first position to the position of an endpoint in the stereo image when the direction of arrival is outside the field of view of the image in the zoom characteristics.

6. The apparatus according to claim 4 or 5, wherein the output unit includes information indicating that the sound source is located outside the image when it is detected that the direction of arrival is outside the field of view of the image in the zoom characteristics.

7. The apparatus according to any one of claims 4 to 6, wherein the image generator adjusts the zoom control parameters of the image according to the direction of arrival.

8. The apparatus according to any one of claims 1 to 7, wherein the at least one microphone signal includes a plurality of microphone signals, the extractor includes a plurality of audio beamformers that receive the plurality of microphone signals, each audio beamformer generates a beam audio signal representing the sound captured by the beam formed by the audio beamformer, the first monaural audio signal is the beam audio signal of a first beam formed by a first audio beamformer among the plurality of beamformers, and the audio component generator determines the first position according to the direction of the first beam.

9. The apparatus according to any one of claims 1 to 8, wherein the audio component generator determines the first position such that the distance from the center of the stereo image increases when the zoom characteristic shows an increase in the zoom level.

10. The apparatus according to any one of claims 1 to 8, wherein the audio component generator determines the first position such that the distance from the center of the stereo image decreases when the zoom characteristic shows an increase in the zoom level.

11. The apparatus according to any one of claims 1 to 10, wherein the audio component generator adjusts the audio level of the intermediate stereo signal component in the stereo audio signal according to the zoom level indicated by the zoom characteristic.

12. The apparatus according to claim 11, wherein the audio component generator adjusts the audio level of the intermediate stereo signal component in the stereo audio signal according to the first position of the image relative to the field of view at the zoom level.

13. The apparatus according to any one of claims 1 to 12, wherein the apparatus is a video conferencing apparatus, and the output unit transmits the audio and video signals to a remote video conferencing device.

14. A method for generating audio and video signals, wherein the method is The steps include receiving at least one microphone signal from a microphone configuration that captures a scene with multiple sound sources, The steps include extracting at least one monaural audio signal representing one of the multiple sound sources from the at least one microphone signal, The steps include receiving a video signal from a camera that captures the aforementioned scene, A step of generating an image signal that includes an image generated from the aforementioned video signal, A step of determining the zoom characteristics of the aforementioned image, wherein the zoom characteristics include at least one of a zoom level and a zoom direction; A step of generating an intermediate stereo signal component for each of the at least one monaural audio signals, wherein the first intermediate stereo signal component of the first monaural audio signal of the at least one monaural audio signal includes the first monaural audio signal positioned at a first position within the stereo image of the first intermediate stereo signal component; A step of generating a stereo audio signal that includes at least one intermediate stereo signal component, A step of generating an audio / video signal including the stereo audio signal and the image signal, A method comprising the step of determining the first position according to the zoom characteristics.

15. A computer program comprising computer program code means that, when executed on a computer, performs all the steps of the method according to claim 14.

Images